It sounds like you have blended frames. Because of a framerate shift at some point (most likely) or just a really bad source, it's when inbetween every two frames, there's a frame where they're 50% blended together. This can create havoc with masking, since this makes massively deformed frames. Also, IIRC, in vegas you don't usually notice the frames unless it's rendered in some form, so it might look like it's masked okay, but it's really not.
The only way to correct this is to actually remove the blended frames from your source, I'm not familiar with the process, but it can be something of a bitch. I'm sure someone else will pop in in a minute who knows how to do it.
