Way too many plugins can slow down your page, yes. The more widgets you add, the more coding and other stuff you're putting on a page, and that adds to load time.
I've heard too that going over-board with the widgets increases your chances of getting hacked. Something about a fatal security flaw in PHP that can be exploited just right for a malicious user to get into your site and delete or replace whatever they want. So I've heard from a few who use WordPress and got their sites royally screwed up by a prankster. I don't use WordPress, so I've never tried to verify it.