I think almost all comic movie fans understand that liberties will be taken with comic films--and they should, for the purpose of making the material more suitable for the medium. But all other things being held equal, if something from the comic can be brought to the screen to make the character more accurate to the source, then it should usually be done IMO.
Ask yourself this--what is it that makes Wolverine distinctive? For comic fans, the size is part of it. His aggression, his crazy hair, the metallic skeleton, the claws, the healing factor, his name, his association with the X-Men, hell, his classic costumes, these are all also components of this character. Some of them may be more important than others, but ideally, they will all be part of Wolvie in any medium.
If the creators of films don't think these things matter, then they can pull a Chris Nolan and just keep the superficial parts of characters, and films can still be good or even great as a result. But many comic fans won't be totally satisfied, and that's understandable.
Getting a good actor to represent the spirit of the character is more important than anything else, but size of Wolvie does matter, and it would be better if he were short rather than tall.