Actually that episode was in Season 3. Off the top of my head, I want to say that the episode name was "Relic".
Basically Jor-El briefly visited earth in the late 50s and had a romance with Lana's great-aunt (of course). After he is wrongfully accused of her murder, John Kent's father helps him escape town. This act of kindness leads Clark to speculate that his parents might have been chosen to raise him. Despite a large part of the episode essentially being Clark/Lana fan fiction; it wasn't bad overall.
I believe there was also an episode in Season 7 with a flashback to when Lara and Kara visited the Kent farm to "check-out" the family before they sent Clark to them. This scene didn't make any sense for many reasons and was par for the course from an era of Smallville where no one cared anymore.
There were a few good moments in the later seasons, but overall you bailed right on time.