Here's the thing. YUV and RGB are very different ways to represent colors. YUV is much like the human eye: the Y represents the luma (you could say a b/w image), the UV the chroma.
In the chroma red is opposite of blue and yellow is opposite of green. On the other hand, RGB represents colors by additively combining red, green, and blue information for each pixel.
What this means is that some colors that a colorspace can represent aren't necessarily standard or representable in the other colorspace. This then means that doing colorspace conversions from one colorspace to another will introduce VERY slight roundings in these values to represent valid colors, thus why colorspace conversion is not lossless. Of course, if kept to a minimum, this should be visually unnoticeable, but if you can go without converting colorspace entirely, it is for the better.
Of course, though, you'll have to convert YUV to RGB eventually. LCDs are RGB, for once, so in order to show the video on the screen you'll eventually need to convert these colors from YUV to RGB.
This means that you have to make sure as best as you can that the colors are being converted as properly as possible. The one thing that matters the most in this is the colormatrix. It basically is a matrix that says how colors should be weighted and represented in the conversion. Getting this wrong will introduce the green alteration you're noticing, for example.
The banding is introduced for similar reasons, namely if a conversion to rgb is not done properly, then you'll have steps in the luma (and possibly chroma too, but luma is obviously more noticeable to the eye).
As a rule of thumb for colormatrix, SD video is generally bt601, whereas HD video is generally bt709. Also, H.264 encodes generally do specify this in the SEI info as well (if it was properly set when encoding), so you can even check and make sure that what you're working with is of a given colormatrix just to be sure (though THORA encodes have the SEI zeroed, so the colormatrix info is prolly removed as well ─ and I don't see it in your screenshot, so yet again further proof).
As for virtualdub, you should be selecting "Fast recompress" under compression. If you don't do so, it will internally convert colorspace, afaik, and possibly colormatrix too, thus why you get the slight chroma change and slight banding as well.