I can't see them going to woodbury for the same reason I can't see them staying at the prison -they need to shake up the scenery to keep things interesting. They had to create a huge disaster to move them off the farm, and now they have TWO locations that are both viable that need to be turned into a bad choice. I'm very interested in seeing just how they do that.
where else are they gonna go? I think if they stay within the walls of the prison and not out in the open, there is no way the woodbury crew is gonna be able to over take them unless they have rocket launchers. After all, it's a prison. Rick's crew knows the prison, I can't imagine it would be very difficult to trap a lot of the woodbury crew in a room full of zombies or something.