Boos are either a form that any species can take upon death under certain circumstances, or they're more poltergeists than ghosts.
Or maybe we could say that they're Koopa Troopa ghosts. That probably won't get disproven, considering there's already Dry Bones so Nintendo wouldn't be able to justify making Koopa ghosts.
I used to think they were Shy Guys (similar shape, and they're shy), but then I saw the Shy Guy Ghosts in LM. And then I found out about Boo Guys, which are also Shy Guy ghosts, but shaped and colored different, and they shew up in games both before and after LM. Sooooo yeah.
What are we to make of the Boo in TTYD that says she's 18 years old, used to be a model, and gets hit on by Arfur the Doogan (who is alive)? How do Boos measure their age? Would a corporeal living person and a Boo actually be able to... do anything (Mario can spin-jump on Boos, after all), or are Doogans just stupid? Or maybe he just wanted to be friends?