The back and forth trunks arguments are so tedious. He's worn them for most of his history. An alien flying around performing miracles in a bright spandex suit and cape, I doubt a pair of red trunks would be what people focus on.
At the same time when he doesn't have them it's not a big deal...