I disagree that Vader was "turning" or starting to turn at the end of ESB when he watched the Falcon escape into hyperspace. I think he was beyond angry (and all of his officers could practically feel it in their very souls) but so much so that he decided to just bury it and process this new turn of events in charting his next move for the throne.
It's not about "turning" as much as it is about a conflict within him getting stronger. But I want to start with that final scene on the Executor because I think it was deliberately designed to contrast with how he dealt with every prior instance of anger with his Imperial subordinates. Like JAWS just posted, I just cant see it as a display of anger.
When Vader is angry, he doesn't do double-takes and put his hands behind his back to stride off casually. There's no story purpose to having Vader react that way if it's supposed to give the audience an impression that Vader was seething. From how he handled Captain Antilles, to Motti's mocking, to the incompetence of Ozzel and Needa... Vader's expressions of anger were well established. His demeanor at the end of ESB doesn't convey anything remotely related to that, IMO.
As you alluded to in your post, Anakin's redemption was going to be part of the ultimate resolution to the saga at every stage of writing ROTJ. That includes the planning stages when ESB was being filmed. The redemption can't happen without some type of inner conflict, and seeds of that would need to be planted. This doesn't mean that such an inner conflict needs to be 50/50 between Vader's "Sith" side and Anakin's "Jedi" side. It could be something closer to 95/5. But I believe the seeds of whatever conflict existed were purposely illustrated in that final ESB Vader scene to build upon later.
And yeah, I obviously prefer a seamless flow, and prefer to think that Luke was right in sensing the conflict in Vader before ever turning himself in on Endor. But I honestly believe elements of ESB were absolutely setup for the sequel(s). And Vader's budding inner conflict was one of those.
You can trace everything Vader would've been feeling to the first appearance of the Emperor in ESB. Palpatine wanted Luke killed. It was Vader who suggested that Luke be converted instead, and thereby spared. By the time Vader became familiar with Bespin, his intention was still to freeze/preserve Luke to personally deliver to the Emperor. When their duel starts, Vader follows through exactly with that intention in a fashion that he deemed "all too easy." It wasn't until Luke showed much more potential that I think Vader's wheels started turning on a new ambition for his son. Instead of "he will join us, or die" now Vader was recognizing a new option. Something more like "join *me*" (and overthrow Palpatine).
The longer that duel went on, and the longer Vader had to evaluate Luke, the more he had to deal with the fact that this was his son. Prior to that, Vader was very willing to either turn Luke over to the Emperor or kill him. There was enough disconnect. But on that gantry, after having violently dismembered his son, I think "Don't make me destroy you" had more than one layer to it.
Vader ended up reaching out to Luke to join him in defeating the Emperor. Luke wasn't going to do that, and it became abundantly clear. So, the options returned to 1.) have the Emperor convert Luke, or 2.) have Luke die. The inner conflict would obviously lead to preferring one of those options over the other.
When Palpatine made it clear at the end of ROTJ that turning Luke to the dark side was no longer on the table, Vader had to watch his son being killed. And that was always the line he couldn't cross, and therefore the seed of inner conflict that started when Luke earned Vader's respect as the Bespin duel went on. And I choose to believe that it was more than just respect that Vader discovered himself feeling for Luke. That's why Vader connected with Luke via the Force and their final ESB exchange started: "Luke." "Father!" "Son... come with me."
