by Scintilla » Mon Jul 19, 2010 8:19 pm
I think I've finally managed to answer my own question after a fair deal of testing. Major takeaways are in blue.
I started off by exporting the same video from Adobe Premiere Pro two ways: to HuffYUV in RGB mode, and then to HuffYUV in YUY2 mode.
I then wrote up an AVISynth script to convert them to YV12 (using the default (Rec601) matrix for the RGB clip) and mt_binarized them at a few different threshold values, including 234, so that any slight differences in the luma values of the two clips would be obvious.
There weren't any -- the results of the binarize were identical. In other words, it appears that HuffYUV uses the Rec601 (TV scale) matrix to do its RGB-to-YUY2 conversion.
I compared these back to my source episode's script, YV12 straight from the VOB with no colorspace conversion, also binarized, at the same frames. There were only the tiniest of differences; it wasn't until I tried converting the RGB export clip to YV12 with the PC.601 matrix that the luma differences were noticeable. For all intents and purposes, the HuffYUV files matched the original source.
For good measure, I tried using the Histogram function too, which confirmed that all of the videos I was looking at stayed in-bounds with the luma -- nothing in the invalid zones outside [16,235].
However, the contest coordinator was right. My encode out of TMPGEnc DID go out of the TV range, as I saw when I slapped a Histogram on that. The culprit is TMPGEnc's "Output YUV data as Basic YCbCr not CCIR601" option, on the Quantize Matrix tab. So, this is something I might want to add to the MPEG-2 encoding guide, so as not to blind audiences.
The upshot of all this is twofold:
1) I can stop worrying that HuffYUV is converting to YUV wrong
2) I can make my white mattes (255, 255, 255) in Premiere Pro, not (245, 245, 245) like I've been doing for years, because they'll get scaled properly later on
... in fact, I just might do that to my Otakon video and re-export. It was kind of embarrassing to see the values on the right edge of the histogram back off when going into my white flashes. That's not intense or exciting at all.