The Mushroom Kingdom was ruled by tyrannical human dictators from another world in the recent past
They were vicious and cruel and killed the Toads for fun on the weekends, but not too many, as they were valuable as slaves. These wads were friends with the Koopa Kingdom and stuff.
Peach didn't like her parents for some reason and really didn't like Bowser for another (maybe both kingdoms were arranging Bowser and Peach's marriage to unite both into a grotesque world-conquering villain fiesta) and so she killed her parents and granted the Shroom Men a reasonable amount of liberty and turned the kingdom into the rainbows-and-lollipops land it is now.
This could have occurred anytime. If you can drive a go kart, you can kill. Peach murdered her parents as a baby!
The tension between the Mushroom and Koopa Kingdoms is a relatively new thing, as they were bros up until this point, and that is why anyone from before Mario and Luigi appeared is not dead or kidnapped or whatever. They'd never fought before.
With Peach's evil ******* parents out of the way, the Toads were inspired to kill all the other human oppressors. That's why there are so few of them. Peach was their new, benevolent princess, Mario and Luigi are heroes, Rosalina has lived in space through all this and E. Gadd is just too fun an old man to want to hurt. Il Piantissimo disguised himself to evade his punishment.
The generic humans from the Golf games are from this era and have since been killed, crossed over by ways that don't require logic; same way the baby characters, Metal Mario, and the bosses Mario has defeated can appear in spin-offs. THEY'RE EVIL BLOODTHIRSTY TOAD-TORTURERS, EVERY ONE OF 'EM