I guess what you have to consider concerning the Paper Mario art shift is that it wasn't really clear what the vast majority of the enemies in Paper Mario's actual designs were prior to the original's release. Besides a very few common enemies such as Goombas, Koopas, and the like, not many of the enemies had 3D models at the time, and many of the enemies hadn't appeared in a game for at least half a decade.
Take Buzzy Beetles, for example; I can't think of any game after All-Stars (although not a new game, it did implement some of the newer enemy designs and add some of its own) that had Buzzy Beetles in it, and what previously existing design do the Paper Mario Buzzy Beetles most closely resemble? I'd say the SMAS-SMB ones, even if the palette's more reminiscent of SMAS-SMB3/SMW. The only major difference is that PM's Buzzy Beetles don't have sclerae, but then again, neither does Mario. When NSMB came around, it had the first official 3D rendering of a Buzzy Beetle (at the very least, the first in-game model), so I guess that could be a valid reason why they changed it in PM3DS, even if it doesn't resemble most in-game earlier designs.