I'm thinking if it were to happen at all, continuation with mostly new characters is the way to go...
Firstly if you pick up after Serenity, Wash and Book are dead. That changes the group dynamic seriously right there.
I think it would be far more interesting to keep Serenity (the ship) the constant, pick it up ten years or so down the road, and follow new characters with the originals coming back in guest or recurring roles as older and wiser (ok, Jayne will probably just be older ) versions of themselves (as actor availability permits).
Am I the only one who doesn't see Mal giving up Serenity to someone else? He's the kind of guy that will go down with that ship.
I'm against a reboot because of what the show means to me. It is fine if some of you are okay with new cast members. Good for you, seriously. For me, the current characters with the stories told are far too important to me to brush them aside and start anew. I prefer the show to stay gone, rather than come back and be something different.
There's a new show called Defiance starting, maybe those of you who want a new Firefly can go watch that.