The answer is really as simple as they didn't have the budget for it.
Then there's also the problem that they had with Luke in the sequels; he's simply too powerful. If he shows up right before a big fight, he could basically win it single-handedly. They couldn't have him there for budgetary AND story reasons. Although they picked a hell of a time to apply "logic" to their story.
I don't think him sticking Grogu in an auto-piloted X-Wing is the worst thing. He obviously has some way to communicate with Grogu, even if it's non-verbal. It's probably how Grogu knew where to go on Tattooine...he remembered Amy Sedaris and somehow conveyed that to Luke, and Luke told R2 to go fly there. Also, apparently all the starships are monitored by the New Republic, hence Mando being so insistent on getting pre-Empire ships, so someone somewhere was tracking R2's flight the whole time.
Grogu and R2 were never in any danger, and Luke wasn't a **** for not wanting to go all the way back to Tattooine.
I always laugh at the name of the planet he's on though....the one Ahsoka was on.....Takodana or something? There's a famous part of Tokyo on the west side called Takadanobaba...is that supposed to be some kind of reference?