I know you do it in your free time without geting payed and I'm aware that it gets annoying... oh and I don't want to be a mod in this forum too, but: It's somehow your job. I understand you if it discourages you, but: It's your duty. If you don't do something because you fear that people will blame you than you do something wrong. Just saying. I don't want to say that you feel or do this way.
and seriously... it gets disturbing when Godix is one of the more polite people here. Man... I sometimes really disliked you because I thought you were such a bad model for other users and their behavior, but today...
In some cases, for example, it's true that spam and especially trolling dies out by itself if people (including of course the mods) don't react to it. Then there's also times where dealing with everything and tearing into every post that is potentially against rules is just inefficient.
I'm the opinion (I can't tell how much you do now): Draw a line (I know that it will be more subjective in that case), react on people who overdo it. Show them that they steped over the line. Edit posts, delete posts in hard cases, give out warnings, give temp bans, give perma-bans. Just show that not everything is alright what they do.
As I said, be too excessive about preventing that, and people just leave.
On the other hand: Take no action and new people will leave or hardly stay. I think you can have fun with meme pics and stuff. Look at the General AMV forum. There is a lot of fun (minor) spam and hardly anybody cares and it hardly goes out of hand. That's fun too and no one gets insulted. Just harmless joking around.
We can't stifle, we need wiggle room, but at the same time people will not moderate themselves.
I believe that a lot of people can if they get shown that what they do is not alright and that they often really hurt people and often piss them of big times.
For example: If I would have known that Code was already that much pissed with Project Editor II trashtalk. I would have never ripped his final nerves with continueing the spam in the Project Gateway thread. What was meant as a small silly fun on my side seriously made him explode and made him start phase 1 of the end of Project Editor. and I'm very sorry for it.