If you can play back the file, then you are halfway there. To get VirtualDubMod to open it, you need to run the video through AviSynth. Use
DirectShowSource("C:\path\video.mkv") in an AVS script where C:\path\video.mkv is the path and name of your video file (keep the quotation marks in the parentheses). Then open the AVS file in VirtualDubMod.
Most modern codecs are designed to work with Microsoft's DirectShow API instead of the older Video for Windows (VfW) framework. VirtualDub/Mod unfortunately uses VfW which limits its usefulness. To get around this, we use
AviSynth to "frameserve" the video into VDM. The video is played by AviSynth using the DirectShow codecs and the output is fed into VDM as what is essentially a fake VfW AVI file.
A handy tutorial to AviSynth is here:
http://www.animemusicvideos.org/guides/ ... synth.html