Loki shot pretty much everyone in the movie an "evil grin" at some point, but only "possessed" people by touching his staff to their chest. Your argument demands that the film be internally inconsistent in which case you can't actually use any events to support your theory because by definition nothing would then be credible.
He never stated he was going to unleash the Hulk upon them, he stated that *they* brought the monster, ie "they" brought the uncontrollable beast that wreaks havoc not because he needs to be "possessed," but simply because he's a "monster" all by himself. All Loki had to do was rile people up enough for nature (in Banner) to take its course.
During a scene when everyone in the room was acting out of character due to magical influences, yet were still not possessed.
Which of course was a lie. Loki's plan was simply to get Banner to "hulk out" in a giant pressurized metal container, which Banner himself recognized as being a real possibility before any hint of Loki's involvement ever manifested. The Avengers knew the danger, took a calculated risk, and paid the price.