I have SoundFX from videocopilot (you know andrew Cramer's After Effects training ground (which isnt actually that useful IMO)) and while it has a good assortment of swishes and drums, i am recommending www.freesound.org
which has thousands more high quality samples all free under creativecommons.
http://www.freesound.org/packsView.php has many sound packs you can get as well.



