I think society (at least, my society's) sees anything that doesn't involve movement as nerdy.
Watching TV too much is nerdy
Reading books is nerdy
Playing Magic the Gathering past the age of 17 is nerdy
Playing Video games is nerdy
Watching anime is nerdy
Building models is nerdy
Masterbation is nerdy... wait a second...
Watching hentai is nerdy...
Watching hentai while making love to your wife on a chair is nerdy... oh shit...
Having video game parties on saturdays with friends visiting is nerdy
Being able to build your own computer and mod your own Xbox Arcade hori Stick is nerdy
The list is too fecking long and it's full of idiots describing nerdy things. Of course, I'm a martial artist, rock climber, swimmer, and runner, so I fall under the world's best blend of human, NERDY JOCK!! I loved chemistry, sci-fi, and have recently moved into the realm of finance and stock options business.
Anime is nerdy to the uninformed and uneducated masses who think with things aside from their brains. The best way to avoid this condescending statement is to look at yourself and do something that's non-nerdy. I hear exercise and fitness can add that extra 'punch' to a nerd that's needed to be 'da man / woman' that society needs.
Anime nerdy? HELL NAW...

"Strength lies in action. Let the weak react to me..." - Kamahl, Pit Fighter from Magic: the Gathering
"That is a mistake many of my enemies make. They think before they act. I act before I think!" - Vortigern from Merlin ('98)
"I AM REBORN!" - Dark Schneider Bastard!! OAV