I've always thought of Boos as the ghosts of the millions of Goombas and Koopas Mario has stomped on, though I do remember one theory that somebody (not here) had: that the Boos are of the Shyguys, and King Boo (Luigi's Mansion) is actually Wart transformed into a Boo.
That theory made little sense to me, as there were already ghosts that were clearly Shyguys (or had been Shyguys) roaming about (in Luigi's Mansion and Yoshi's Island).
Maybe the Boos are just the projections of evil magic, very much how Redeads (Zelda) are animated magic.
Here's a Shyguy related question: What's behind a Shyguys' mask?