Are you trying to put x264 in the AVI container by any chance? I'm not too familiar with that myself, but I believe that VirtualDubMod doesn't support that option. I found that the best way for me to use x264 is in the MP4 container (you can use MKV and OGM of course, but I have never tried it with VirtualDubMod). For MP4, there is a great GUI called Zarx264gui and a good guide explaining how to use it here: Compressing to H.264 using Zarx264gui. I hope this helps you out.
virtualdub/mod is not a good choice for encoding with an h.264 codec (something about it being vfw and not supporting the features that make h.264 so good IIRC)
command-line, meGUI and Zarx264GUI are much better choices for encoding with an h.264 codec