I have to think it'd be easier for the Governor and a few guys to sneak into Woodbury than the prison. Secure from walkers, but people have snuck in and out, Rick's group got in no problem. Although Tyrese's group got into the prison too. Also i'm sure the showers at the prison work fine, and its one single complex to keep everyone in. And the people keeping the electricity and other amenities going in Woodbury may have been gunned down. Not everyone is an electrical engineer, plumber, etc.
My guess is the Governor will rally some other smaller groups on the road, take in a few more guys, and than start ambushing Rick's smaller groups that come out for supplies. Maybe build back up to an all out assault with a Tank or some mortars or something that just levels the entire prison.
But cmon, wouldn't it be awfuly similar to the S.2 finale if it was just overrun and they had to leave again. We'd be expecting every season to go just like a video game where they end up settling in only to have to go somewhere new next season. I like they caught most of us off guard.
I guess you make a lot of sense.