The programming difficulties in doing some things in YUV space are that in order to calculate an actual color value, you have to internally convert to RGB sometimes anyways. Some things work better in YUV, some things work better in RGB. In theory almost anything can be done in either, but the mathematical simplicity of RGB is far better for certain operations. Then again, having the luminance (the Y channel) already there for you makes other things really easy (like changing say, the brightness
)
And a note about RGB<->YUV conversions - the loss of video quality is an exponential function. I.e. every time you convert back and forth you lose MORE data than you dude the previous time. So every single color conversion that you can prevent is a definate benefit.
That's why I'm really excited about AVISynth 2.5 - native YV12 colorspace! Now THAT'S super-cool.
