Best: Hmm.. Mario 64 had.. trees, and water, Luigi's Mansion was nice, full of things to scan/interact with, Delfino had Piantas (or Pinatas, heh) to annoy/stuff to interact with, Galaxy had the Observatory, which looked nice and had few things to interact with, but the music was nice. Both PM games were had good ones, as did all of the Mario Parties..
Worst: Partners in Time, because the castle is huge, the time holes are spread out, and there is only one shop in the castle (present). That and the fact that it screwed with the already messed up continuuity of the series.. or lack thereof. (but that's a rant for another time..) Also, I want to vote for Flipside in SPM. It's confusing to navigate at times.
Nominees: best Bowser castle music. Fortresses are only halfway points.. Ok, so that's only a technicality, but still..