I'd say my most favorite would be the Stone Tower Temple from Majora's Mask, having to go through the temple right-side up and upside down, and using the skills of Link's other three forms, Deku Scrub, Goron, and Zora. The music's also eerily calm in upside-down form.
In Ocarina of Time, I'd say the Spirit Temple was also pretty fun, going through it both as Young Link and Adult Link. Anyone see a pattern going on here?
I wouldn't say I liked the Wind Temple in WindWaker, but I did like the music in it quite a bit, with the acoustic guitars coming in every so often for a few bars. I wouldn't say I liked the Earth Temple much either, but the boss, Jalhalla(?), was fun to defeat.
In Twilight Princess, that mansion owned by the Yeti and his wife was quite fun. I also really enjoyed the Temple of Time.