Whedon had already solidified his legacy well before his directing involvement with Marvel. I can't count how many lists 'best of' lists Buffy has made. Angel was a success in its own right. Firefly got canceled because Fox was being Fox. And Dollhouse was before its time, IMO. I really got a kick out of that one. Again, give it to ANY other network and I can almost guarantee that it would have lasted longer.
The Avengers movie would've been big without Joss, BUT it wouldn't have been THAT big. Marvel was smart by seeking out Joss. He knows the source material better than any other director out there, IMO. He's been writing for Marvel for years, so it should've been a no brainer.
Not quite sure where all of this commotion is about though.
Marvel will continue to do its own thing, and so will Joss, with or without each other.