My issue is, those references don't really tie the stories together, they're nods for fans but not overly relevant to the characters.
Iron Man introduced Nick Fury and SHIELD.
Hulk is a thin connection but alluded to Nick Fury needing Stark to go after Hulk, thus the Hulk's cell on the Hellicarrier.
Iron Man 2 introduces Black Widow and draws Stark into the Avengers.
Thor introduced Hawkeye as a member of SHIELD and establishes Loki's thirst for power and his place in the universe leading to him becoming the Avengers villain.
Captain America brings Cap into SHIELD.
Avengers took those pieces and used them to give us the versions of the characters in that film as formed by those previous experiences.
Personally, I felt every character in Avengers was a fitting continuation of where their characters would go based on their individual plot lines, and Phase 1 was about bringing the team together, solo films worked fine because they were solo people coming together.
Phase 2 starts after Avengers with Fury saying they're going separate ways but will come back together when the world needs them, great, except, how are the events of Iron Man 3, Dark World or Winter Soldier not serious enough to warrant at least some of them coming back together.
Let's just look at Banner and Stark's relationship. Bruce goes off working with Tony, he gives him therapy in Iron Man 3, Age of Ultron establishes Veronica among collaborative projects.
How does Bruce go from wanting to develop Veronica as a precaution and avoiding becoming Hulk to being willing to change and working with Widow on ways to bring him out of transformation and do it so often the team has a label for the situation as Code Green? Stark needed an army of suits to fight in Iron Man 3, that would have been a good time to call Banner in, but he didn't, but by AOU we're to believe Bruce will change anytime they need him to.
Thor could have used help in Dark World but only utilized resources that were available to him in the original Thor pre-Avengers.
Stark at the least could have been valuable in Winter Soldier and Fury, Widow and Cap would know he'd be an option.
Phase 2 seemed to go from actually weaving the characters story into one giant cohesive tale by the conclusion in the Avengers film of the Phase to just casual name dropping. Fury mentions Stark helping with tech in Winter Soldier, ok, so he'll help you with Tech but you can't call him into a fight.
I understand that because these are movies, you're not going to bring Iron Man into every film, etc, but you could at least draft your stories to explain that absence. It's like once Avengers made a billion dollars, they said **** it, we can just crank out movies again and again that don't connect. What I loved about Phase 1 was how they were individual stories but all came together and it's a great evolution of characters. The films of Phase 2 are all stand alone including Age of Ultron, if you watch AOU without seeing any other Phase 2 movie, you won't be missing anything.