HuffYUV RGB->YUY2 conversion: TV/PC scale?

[Scintilla]

PC scale is silly if your content is TV scale already. Just leave it alone.

Okay, maybe I'm misunderstanding things here, but my impression was that when AVISynth converts YV12 (DVD) to RGB (in the NLE), it defaults to using the TV scale matrix -- so that, say, white pixel with a Y of 235 becomes (255, 255, 255) in RGB, and black with a Y of 16 becomes (0, 0, 0).

And then, once it's RGB (assuming the conversion was done using the right matrix), it's full scale -- and you can't tell what scale it was originally. In that case, there would be no harm in keeping it full scale when converting back to YV12 for playback on a computer monitor. Was I wrong in my understanding of this?

I don't think it's important enough for you to have to worry about given you're already doing it wrong by using RGB at any stage to begin with. When it's wrong in one place, who cares if it's wrong in others. Nobody will notice.

Hey, it's not my fault that a major prosumer NLE only works in RGB colorspace. I was given to understand that there still weren't any NLEs out there that handled YUY2 or YV12 natively. That's the only reason I need to worry about all this in the first place.

And this whole thread was kicked off precisely because a contest coordinator *did* notice, when I converted my YUY2 HuffYUV file to MPEG-2 with TMPGEnc without realizing I had the "Basic YCbCr" (full scale) option checked.

[Mister Hatt]

You are most definitely misunderstanding it then. It uses what you tell it to when you convert to RGB<resolution>. You can define whether it uses PC or TV luma models along with which chroma matrix. Using Rec{whatever} makes it use TV levels. PC.709 f.e. will use PC levels. Also you seem to think that it's rescaling your video when it isn't. If your source vid uses TV levels and you convert to PC levels, it will NOT change a black defined as 16 to 0 for you. It will still be 16 on the PC scale. That statement may in fact be wrong for the values of 16 and 235 only, but all values in between WILL stay the same. Best thing to do is use avisynth to do your convertions so you have full control over them.

Use a better NLE. Almost every professional NLE I have seen uses YV12, YUY2, and RGB40 (er yeaaaah) quite well, although the user being able to do so is another question. Professional video is mostly YUY2, at least in my industry experience.

I am very surprised a contest coordinator can notice anything that small given that they are also asking for MPEG-2 and therefore have no right to talk about video whatsoever.

[Scintilla]

It uses what you tell it to when you convert to RGB<resolution>. You can define whether it uses PC or TV luma models along with which chroma matrix. Using Rec{whatever} makes it use TV levels. PC.709 f.e. will use PC levels.

I get that part; and according to the documentation, it defaults to Rec601.

Also you seem to think that it's rescaling your video when it isn't. If your source vid uses TV levels and you convert to PC levels, it will NOT change a black defined as 16 to 0 for you. It will still be 16 on the PC scale. That statement may in fact be wrong for the values of 16 and 235 only, but all values in between WILL stay the same.

The idea that it rescaled the video came to me straight out of the AVISynth documentation (emphasis mine):

In v2.56, matrix="pc.601" (and matrix="pc.709") enables you to do the RGB <-> YUV conversion while keeping the luma range, thus RGB [0,255] <-> YUV [0,255] (instead of the usual/default RGB [0,255] <-> YUV [16,235]).

If that line doesn't say that (for example) black with Y of 16 in YUV becomes (0,0,0) in RGB under the usual/default behaviour (= Rec601), then what is it saying?
(Setting aside PC-levels YUV for the moment. I'm only concerned right now with converting TV-levels YUV to RGB; I'd like to get that cleared up first.)

[mirkosp]

Why are you still using v2.5.6? At least get 2.5.8... Hatt will say to get 2.6a, though.

[Scintilla]

According to the Version script I just ran, I *am* using 2.5.8, built 12/22/2008.

[mirkosp]

My bad, seems that convert hasn't had updates in 2.5.7 and 2.5.8 (I was asking that cause I saw 2.5.6 in the documentation bit you quoted). There's some new stuff in 2.6a however, but I'm not sure if defaults behaviour has been touched.
Still, with the "Also you seem to think that it's rescaling your video when it isn't." line, I think he was referring to the NLE and not to AviSynth anyway. >_>

[Mister Hatt]

The new Convert stuff in 2.6a is totally worth upgrading for alone, ConvertToYV24() f.e. will let you crop odd pixels in YV12 video without destroying colourimetry.

As far as the docu there and the confusion between RGB[0,255]<->YUV[16,235], it's actually written really badly. If you are converting RGB to YUV with PC levels then yes, it will use [0,255]. If you are converting RGB to YUV with TV levels, it will use [16,235]. If you are converting YUV[0,255] to RGB with PC levels it will give you [0,255]. However, if you are converting YUV[16,235] to RGB with PC levels it will NOT auto rescale everything other than pure 0 and 255. 17 f.e. will stay as 17, it won't be dropped down to 1. That's as far as I understand it anyway, from testing and RTFS etc.

[Scintilla]

I think I've finally managed to answer my own question after a fair deal of testing. Major takeaways are in blue.

I started off by exporting the same video from Adobe Premiere Pro two ways: to HuffYUV in RGB mode, and then to HuffYUV in YUY2 mode.
I then wrote up an AVISynth script to convert them to YV12 (using the default (Rec601) matrix for the RGB clip) and mt_binarized them at a few different threshold values, including 234, so that any slight differences in the luma values of the two clips would be obvious.

There weren't any -- the results of the binarize were identical. In other words, it appears that HuffYUV uses the Rec601 (TV scale) matrix to do its RGB-to-YUY2 conversion.

I compared these back to my source episode's script, YV12 straight from the VOB with no colorspace conversion, also binarized, at the same frames. There were only the tiniest of differences; it wasn't until I tried converting the RGB export clip to YV12 with the PC.601 matrix that the luma differences were noticeable. For all intents and purposes, the HuffYUV files matched the original source.

For good measure, I tried using the Histogram function too, which confirmed that all of the videos I was looking at stayed in-bounds with the luma -- nothing in the invalid zones outside [16,235].

However, the contest coordinator was right. My encode out of TMPGEnc DID go out of the TV range, as I saw when I slapped a Histogram on that. The culprit is TMPGEnc's "Output YUV data as Basic YCbCr not CCIR601" option, on the Quantize Matrix tab. So, this is something I might want to add to the MPEG-2 encoding guide, so as not to blind audiences.

The upshot of all this is twofold:
1) I can stop worrying that HuffYUV is converting to YUV wrong
2) I can make my white mattes (255, 255, 255) in Premiere Pro, not (245, 245, 245) like I've been doing for years, because they'll get scaled properly later on


... in fact, I just might do that to my Otakon video and re-export. It was kind of embarrassing to see the values on the right edge of the histogram back off when going into my white flashes. That's not intense or exciting at all.

[Mister Hatt]

So when I said that TV scale colours would look the same on PC scale, that didn't make sense? You seem to have made that exact same conclusion. Using PC scale for extra stuff is fine as long as you understand that the TV scaled stuff won't rescale itself. Everything will still look normal at PC levels. Converting your PC scale stuff to TV however will be lossy.

[Scintilla]

See, now you have me going back over the previous posts and trying to figure out where the misunderstanding happened. I think you thought I was talking about converting TV YUV to RGB using the PC matrix; I always intended to convert to RGB with the TV matrix, it's just how to convert back to YUV afterwards that was the question.

Because as far as I see, everything you said was correct, but so was everything I said -- with the exception of: "In that case, [...] there would be no harm in keeping it full scale when converting back to YV12 for playback on a computer monitor, correct?".
What I didn't realize when I said that was that whatever video decoder is handling the output in VirtualDub is apparently rendering YUV according to the TV matrix... so if I convert RGB to YUV using the PC scale, all whites above 235 (as verified by the Histogram) just show up as (255, 255, 255) when I screencap and view in The GIMP.

So, it's not a good idea to convert back to YUV using PC scale, because my display doesn't use the extra luma range anyway -- it clips.