That's my position exactly. I refuse to participate in the fantasies of other people, but I can respect them within the bounds of reason.
I understand that the argument from biology is that the brain presumably develops separately from the reproductive organs and something gets lost in the mix, resulting in a female brain with a male body. I have no idea if that's true. I am not a specialist in fetal development. If it is true, how often does it happen that a brain chemistry will persist in generating endocrine conditions to support a uterus and ovaries when there are none to be found? Seems like too major an error for any significant portion of a population to be affected. It seems more likely that behavioral psychology is the routine culprit. A young boy (even a toddler) doesn't need to have a brain that's deluded regarding the identity of his genitals to begin identifying as female. All he has to do is begin a chain of value judgments that lay a foundation for future emotional responses defining his relationship to the world as feminine.
In either case, I still think the individual is free to proceed as they like, believing as they like. I do not think that either situation qualifies the individual as female when they were born with a male body. Nothing changes with gender reassignment surgery, other than the destruction of their body. A man can be feminine and a woman can be masculine. They are still a man and a woman.