My collection, small as it is now, is kept in my office. My bosses (which happens to be my parents, as it's a family business) would probably prefer it if my office wasn't festooned with figures, framed Calvin and Hobbes prints, goalie masks, and such. But they're okay enough with it.
As for other people - if they feel the need to disparage you, **** 'em.
Usually when people walk by my office, they spot something that causes them to say something to me about it. Whether it's a figure, some Disney stuff, or whatever - most people seem to love the opportunity to talk about or at least look at something fun.
In my experience, the percentage of people who scoff at this are getting smaller. It's so much more acceptable now to like "uncool" stuff. If comics, toys, and Star Wars were as socially acceptable when I was in high school as they are now, I'd have had a much easier time, socially.