Took me years before I found most of Pilotwings 64 to be a cakewalk (I consider the "rite of passage" moment being getting a perfect score on Chicken Dive). And Wave Race 64 extremely easy too for that matter. Wonder if having played Wave Race: Blue Storm helped with that.
That said, my top 5 N64 games (Rayman 2's not on the list because, as beautiful as it is, I'm bad at the game):
1. Banjo-Kazooie (best adventure game ever)
2. Tetrisphere (The N64 version of crack. Sure, half the game is the amazing music, but I also enjoy how easy it is to get points such that it feels rewarding to score 50 million)
3. Blast Corps (destroying buildings is fun!)
4. Turok: Dinosaur Hunter (for some reason, I still think the original's the best of the series)
5. Road Rash 64 (it's stupid fun, but I never get tired of whacking guys with a 2x4 and seeing bikes fly through the air. And the soundtrack rocks)
That means I'm waiting for Tetrisphere, Turok, and Road Rash 64 to show up.