Capturing in virtualdub is rather simple.
Change mode to "Capture Mode:
Set the Audio compression:
It's best to just set it to PCM 44.1Khz 16bit Stereo. This is if you plan on captureing audio, if you don't then don't worry about these settings:
Setup Video Compression:
Here you can select what codec you want to use and set it up properly:
Setup Capture Settings:
You can adjust things like frame rate, if you want to capture audio and some other stuff. Really those are the only things you need to worry about. Unless you have special needs you'll want the frame rate at 29.97:
Set the capture file:
This is the file that the video will be stored to. You'll want lots of free space
Then finally you need to capture video:
Before you do that though you probably want to make sure you can actually see the video in the little box in the upper left of the window.
If can't, then you may need to goto Video|Source and fiddle with the settings in there until you can see the video.
To stop captureing, hit the Esc key. While the video is capturing you'll get updates on the right side of the window with neato information.