I'll tell you exactly why I don't have an issue with Roshi: he doesn't WANT to be a fighter anymore. He's comic relief. He fulfills the role of the old pervy mentor and it's funny. DB is in part a comedy and Roshi has a good role in that. Yamcha however ADDS NOTHING. I don't give a **** about what he says, he has no unique funny aspect to him beyond just being a bit of a derp.
Fact of the matter is that Toriyama himself turned Yamcha into the joke fighter
^ Yamcha was MADE to be a joke fighter. The problem is that because Toriyama can't let characters go, people got attached to him when they honestly just shouldn't have. Yamcha was a moron desert bandit who when confronted with a girl he liked morally shifted. He had some fighting capability and nothing to do with his life so he started training. Yamcha wasn't made to be anything else except a joke, the loser. The problem is: in DBZ Krillin usurped his role after becoming less significantly powerful. Yamcha was the "krillin of db" as we would now say, but in Z krillin is the krillin. That made Yamcha wholly obsolete and he still is. Yamcha is not relevant in ANY way. Roshi is, for comic relief, hell Oolong is for comic relief. Yamcha isn't even actually funny. He's a character that has no raison d'être as the french say. (Do the french actually say that?) A writer should either let him die, for real, in a last stand, with honour. Or devise a way to make him relevant again. But Yamcha tagged along without relevancy too long and people (god knows why) became a fan. I completely respect people liking someone, but that does not change the former. In fact, I'd sooner just feel sorry for yamcha fans.
The fact that Toriyama has an issue with making everyone ultimately lesser than Goku and the main villain is separate from this. Vegeta is VERY relevant throughout all of the story. He's Zod to Goku's Kal-el, he's an influential force on namek that actually causes gohan and krillin to still be alive (imagine if he wasn't there when the ginyus landed). Vegeta was a key plot component in letting Cell reach completion (apart from just being ****in awesome and he COULD have wiped out Cell). And in the Buu era, without vegeta in the end, everyone would've died. But above all Vegeta is entertaining. As I said though, from the buu saga onward, the issue started to become a serious one for everyone else but vegeta and goku. And that has grown worse since. But Yamcha lost most of his relevance when bulma broke up with him in DB (not DBZ, DB) and lost all of it when he stopped being relevant power-wise, because he has no added value as a character. He's not particularly funny or impacts the plot. He's irrelevant. He could suddenly stop showing up and I wouldn't notice and wouldn't lose out on anything.