See, while I accept that X3 was weaker than X2, I do think the Beast was awesome. Multiple Man was fun. The action scenes had a pacing that the other movies lack and I find them to be among the most exciting in the franchise.
But there were a lot of other things that were really annoying. Hollowing out Jean's story was a mistake. So many cameos that got disintegrated for shock value. Callisto being tweaked for casual plot points. Angel should have got in on the action and ended up in a uniform at the end. X4 should have followed on with Mr Sinister to lay the groundwork for Cable, Rogue gaining super strength, the Marauders, the fall of Angel, and Apocalypse. It was a natural progression that would have showcased the actual X-men.
If you leave out X3, you lose the Wolverine as well, which was good fun, albeit I don't think Viper or Silver Samurai were well done, Mariko was note perfect and the I Dream of Jeannie sections were great.
First Class was a great movie but cocked up the time frame, so the movies I have to leave out are Wolverine Origins and First Class on my rewatches.
DOFP the Rogue cut missed an opportunity to give Rogue her strength. I'm guessing Anna Paquin wasn't available long enough to do any great special effects but a stunt woman could have done the action scenes.
Despite all this, in my head, I'm building my own X-movie head canon. The movies only provide a snapshot. Lots happens in between and Sinister has use for Banshee, Havok, and Apocalypse...