As you said, the ninja or shinobi was first and foremost a master of disguise.
This was known as Hensojutsu, the art of disguise.
From Hatsumi, Masaaki. The way of the ninja: secret techniques. Kodansha International, 2004. p92
For most missions a sterotypical black ninja outfit would not have been the most prudent option. The ninja then attempted to look like whatever the situation called for. Some examples below.
Undercover operations are to be found throughout Japanese history, and
one component of the modern ninja cult has been to exaggerate this
fact by crediting certain historical figures with being ninja or
“proto-ninja.” The legendary Prince Yamato Takeru resorts to
subterfuge on at least one occasion including dressing up as a woman,
making him a ninja in some eyes (Aston, 1972, p. 201)
He sent two Matsukura retainers as spies to Manila disguised as
merchants, ostensibly to discuss reopening trade.
From Turnbull, Stephen. "The Ninja: An Invented Tradition?." Journal of Global Initiatives: Policy, Pedagogy, Perspective 9.1 (2015): 3. p12
For their commbat missions, however, a ninja fell back on armor that blended with his environment. The color needed to be flexible since black is not a good choice for blending in at all times. Colors like earth brown, watery blue and even snow white were utilised.
The author below descibes the ninja's armor as rainbow colored to fit in with the colors of nature.
One day, I was asked, "What color is Shinobi color?" I answered
immediately, "Rainbow-color of course, the colors of a prism. They
blend with nature's colors, don't they?
From Hatsumi, Masaaki. The way of the ninja: secret techniques. Kodansha International, 2004. p90
The author even refers to clan colors on page 92. You might notice that there is indeed a reference to black as the color of the Genji.
Certain old documents refer also to the "armor of the four clans,"
saying, "the head of the Genji wore black, the Heike violet, the
Fujiwara light green, and the Tachibana yellow." This means using
colored armor for Henso-jutsu— not merely techniques for changing
one's appearance, but methods for changing the whole flow of battle.
A ninja indeed used a specific outfit that blended in when infiltrating.
The color black was used, but it was only a small part of the ninja apparel.
Their clothes' color was adapted to their surroundings