Speaking of capturing emotion - I always find that Vader somehow looks sad in this part of RotJ (pic below - MF's of course!) or you can feel the 'conflict within him' as Luke says... dont ask me how I can see anything at all given he's wearing that mask but I always felt he looked pensive, contemplating etc did they use a different mask in that part?
It's just a matter of angle, lighting and posture, plus Vader's movement. In the seen, Vader walks over to the railing Luke leaned on very slowly, as though a great weight is on his back, which emotionally it is, then he rests on the railing and tilts his head down. Such actions are common to emotional distress, so even though there's no expression to the Vader mask, the music and actions define and emotion within the viewer and you project it onto his face.