Why do I get a massively huge HuffYUV file sometimes?

[dazza1008]

Hi all!
I'm new to Premiere pro, and therefore converting video formats... and I'd like to start with the most major problem.

My raws are divX and I've been trying to convert them to HuffYUV using VirtualDub.

Episodes 1, 2 and 3 were all converted to HuffYUV @ about 4 gigs each. It's OK, I think I have enough disk space for this. (each of these was divX 704x396, 192kbps audio, mp3, video: 23frames/s, 194kbps, 24 bit - and they were about 250 megs each originally)

Episodes 6, 19 and 20 were all converted to HuffYUV @ about 26 gigs each. Now disk space is a problem! (each of these was divX 640x360, 128kbps audio, mp3, video: 119frames/s, 136kbps, 24 bit - and they were about 170 megs each originally)

The thing which I don't understand is that I'm doing exactly the same thing in VirtualDub.

Come to think of it, that 119 frames/second looks a bit worrying... <.< I don't think it could actually be that!

Any help would be greatly appreciated!

[Autraya]

try lagarith?

[Scintilla]

Come to think of it, that 119 frames/second looks a bit worrying... <.< I don't think it could actually be that!

Nevertheless, some fansub groups do actually release episodes at such a high bitrate.

If you open up one of those huge HuffYUVs in VirtualDub(/Mod) and check the file information and it still tells you 120fps, then you know that it's right <s>cause I hear it in the ni-ee-ight</s>. In that case, you'd better remove interlacing, if there is any, and do some serious decimation* to get your frame rate back down to either 23.976 or 29.97 fps.


* Yes, I know it's not really decimation in the strictest sense, because you'd be killing off a lot more than one frame in every ten, but that's what people appropriate the word to mean when it comes to AVISynth filters: namely, arbitrary removal of X frames out of every Y.

[dazza1008]

Thanks guys. ^^

try lagarith?

I could, but I'm too stubborn. I didn't quit using Windows Movie Maker just to get another program with issues as well (although I'm sure once I know Premiere CS3's personality it would be more fun). Plus I heard Lagarith was more heavy on the processing power, which my computer doesn't need. I'm seriously considering using AVIsynth stuff, then I don't have to convert it beforehand and use disk space (but I'm not toooooooo sure how it works).

If there's a way, I'd want to use AVIsynth to convert the divXs to HuffYUV, and all 704x396, because I'm about 1/3 into my hopefully next AMV, and I didn't realise my raws were different sizes.

Come to think of it, that 119 frames/second looks a bit worrying... <.< I don't think it could actually be that!

Nevertheless, some fansub groups do actually release episodes at such a high bitrate.

If you open up one of those huge HuffYUVs in VirtualDub(/Mod) and check the file information and it still tells you 120fps, then you know that it's right <s>cause I hear it in the ni-ee-ight</s>. In that case, you'd better remove interlacing, if there is any, and do some serious decimation* to get your frame rate back down to either 23.976 or 29.97 fps.


* Yes, I know it's not really decimation in the strictest sense, because you'd be killing off a lot more than one frame in every ten, but that's what people appropriate the word to mean when it comes to AVISynth filters: namely, arbitrary removal of X frames out of every Y.

^^
I wouldn't have pulled you up on the decimation point. Actually, that's laughable, considering where I'm at. X]
Yes, it says it's 119 point something frames per second in VirtualDub... thanks.
What I don't understand is how the originals could be that frame rate. It's like an error in the original divX. It just doesn't make sense to me that a larger frame size (740x396) would convert to a smaller file size than one with a smaller aspect ratio (640x360) (I'm taking the file size into consideration too). I don't see how there could really be 119 frames per second for the 640x360 one (and the file size to be 170 megs), while the 740x396 one is 250 megs (with 23 frames/second). It's like VirtualDub copied many frames and really increased the file size so it really became 119 frames/second for the HuffYUV file. And also I'm not sure whether fansub means subtitles, but I assume not... but all those divX raws were done by the same person. >.<

I'd like to try using AVIsynth with Premiere Pro CS3 first, then I can hopefully skip the converting issues, and I can also instantly convert the aspect ratio to what the first ones are. If you have any warnings, please let me know. Cheers.

[prYzm]

one problem could of course be that the fansubbers got their files from 2 different sources. eg if 1,2,3 are all from the same disk, and ur later eps were from a disc bought elsewhere (for whatever reason) it could mean that it was done in a diff format thus the problems?

[Qyot27]

119fps is what crazy Japanese RAW cappers do to deal with the problems of using AVI for content that's VFR. I'd be willing to bet such encodes are assuredly not sourced from DVD.

The issue is that in those streams, there are real frames and dummy frames. Sections with 23.976 or 29.97, and loads of dummy frames inserted between them in order to equal them out to the least common denominator (nevermind the fact that in reality, the only sections that would feasibly need dummy frames are the 23.976 sections, to bring them up to 29.97). The dummy frames are minuscule, because there's nothing in them. When played back, these frames are just skipped over by the decoder - if you open the original file up in WMP 6.4 and check statistics, the Frame rate field will read 119 or 120 but the Actual rate will be much much lower.

Now, what happens in this is that VirtualDub can't distinguish between dummy frames and real frames (at least, this is true in older versions of VDub and in VDubMod - VDub 1.7.6 shows options for 'Smart rendering' and 'Preserve empty frames', which I would take as being evidence of it being aware of it now). So when you encode to a different format without decimating the video stream using AviSynth or correctly removing the dummy frames with avi_tc, you get VirtualDub outputting a full 119fps stream of real frames instead - which causes the huge bloat, and is generally an encoding and playback nightmare.

In terms of seeing this in fansubs, the only series I've ever seen that fell prey to this were the remaining Happiness! subs that got released not too long ago. 7 & 8 were incorrectly processed and have the 119fps real frames issue, whereas 9-12 were just softsubbed and still have their dummy frames intact. RAWs, however, exhibit this is spades, for many many series. In reality, the issue would be solved by using a container like MKV that can deal with VFR correctly, but they do it out of overreliance on AVI - much like the early efforts of putting H.264 in AVI (which thankfully are almost non-existent now).

The recommended method of correcting the issue is to use avi_tc to remove the dummy frames, leaving you with a new AVI file that has none (it will assume a constant framerate, however - this can be dealt with by simply altering it during pre-processing or in the editing program).

You can find avi_tc here:
http://bengal.missouri.edu/~kes25c/#c3

[Qyot27]

Now, what happens in this is that VirtualDub can't distinguish between dummy frames and real frames (at least, this is true in older versions of VDub and in VDubMod - VDub 1.7.6 shows options for 'Smart rendering' and 'Preserve empty frames', which I would take as being evidence of it being aware of it now). So when you encode to a different format without decimating the video stream using AviSynth or correctly removing the dummy frames with avi_tc, you get VirtualDub outputting a full 119fps stream of real frames instead - which causes the huge bloat, and is generally an encoding and playback nightmare.

An addendum:

Actually, older versions of VirtualDub and VDubMod can detect the presence of such frames, they just can't do anything about them. Open up the original AVI - even if it reports 119 or 120fps, look at the frame counter at the bottom of the window next to the controls. Track through your video stream one frame at a time. You'll probably see [], [K], and another indicator, [D] - [D] is the one that tells you that you've hit a dummy frame (note, the [D] indicator also shows up on some 'normal' 23.976 or 29.97 AVIs, because of the way codecs like DivX, XviD, and yes, even H.264, deal with B-frames in AVI - the difference, though, is that in those cases the framerates are still manageable and still pretty much what the input fps was).

[prYzm]

all hail qyot - the technical dictionary?

[DayWalker B.]

all hail qyot - the technical dictionary?

x2

[dazza1008]

one problem could of course be that the fansubbers got their files from 2 different sources. eg if 1,2,3 are all from the same disk, and ur later eps were from a disc bought elsewhere (for whatever reason) it could mean that it was done in a diff format thus the problems?

Yeah, I can understand that... Looks like the larger ones are from a disc and the smaller 119fps ones are from a streaming source... (e.g. TV)
Thanks!

119fps is what crazy Japanese RAW cappers do to deal with the problems of using AVI for content that's VFR. I'd be willing to bet such encodes are assuredly not sourced from DVD.

The issue is that in those streams, there are real frames and dummy frames. Sections with 23.976 or 29.97, and loads of dummy frames inserted between them in order to equal them out to the least common denominator (nevermind the fact that in reality, the only sections that would feasibly need dummy frames are the 23.976 sections, to bring them up to 29.97). The dummy frames are minuscule, because there's nothing in them. When played back, these frames are just skipped over by the decoder - if you open the original file up in WMP 6.4 and check statistics, the Frame rate field will read 119 or 120 but the Actual rate will be much much lower.

Now, what happens in this is that VirtualDub can't distinguish between dummy frames and real frames (at least, this is true in older versions of VDub and in VDubMod - VDub 1.7.6 shows options for 'Smart rendering' and 'Preserve empty frames', which I would take as being evidence of it being aware of it now). So when you encode to a different format without decimating the video stream using AviSynth or correctly removing the dummy frames with avi_tc, you get VirtualDub outputting a full 119fps stream of real frames instead - which causes the huge bloat, and is generally an encoding and playback nightmare.

In terms of seeing this in fansubs, the only series I've ever seen that fell prey to this were the remaining Happiness! subs that got released not too long ago. 7 & 8 were incorrectly processed and have the 119fps real frames issue, whereas 9-12 were just softsubbed and still have their dummy frames intact. RAWs, however, exhibit this is spades, for many many series. In reality, the issue would be solved by using a container like MKV that can deal with VFR correctly, but they do it out of overreliance on AVI - much like the early efforts of putting H.264 in AVI (which thankfully are almost non-existent now).

The recommended method of correcting the issue is to use avi_tc to remove the dummy frames, leaving you with a new AVI file that has none (it will assume a constant framerate, however - this can be dealt with by simply altering it during pre-processing or in the editing program).

You can find avi_tc here:
http://bengal.missouri.edu/~kes25c/#c3

^^ Thanks very much - I'm new to all this and I'm interested to know how things work... and I think I can only handle it as it becomes relevant to me. So thank you SO MUCH for making your comment detailed.
Um, I'm not sure what to do... when I clicked the GUI, I got an error message, and when I ran the command lines (what I thought was properly) I got "frame counts don't match" I made sure all the unzipped files and the file I wanted to convert was in c root directory, then typed
cfr2tc c:\raw06huffy.avi c:\raw6conv.avi c:\timecodes.txt 1
(raw06conv.avi didn't exist - I thought it was supposed to create a new file. I have no idea what the timecodes text is, so I just copied the sample syntax)
So, "frame counts don't match"???? How can it not match against something which doesn't exist? And I thought I didn't want the frame counts to match - that was the whole point, get rid of the null frames.

Actually, scratch all that - I was trying to convert the huffy file, and of course it couldn't detect any null frames because they were all changed to real frames.

Converting the divX - Now it's much smaller (160 megs approx... before it was 185 megs approx). Same aspect ratio (640x480). Frame rate 29/s

I have to now convert this to 23.976fps and 704x396 aspect ratio. Can I do this in VirtualDub? Is that the best approach?

BTW - a problem. NO SOUND!!!! @_@ I'd like to have sound available because I might want to use the original soundtrack as an intro, but that may be why the size was reduced so much... (btw in order to have sound I have opened up a file in WMM, saved it as an mp3, memorised who said what when, and matched it up with the image. Not recommended.)

And btw what do you mean by pre-processing because this program assumes a constant frame rate (and my avi probably doesn't have that, hence the need for 119 fps with some dummy frames)? This was the first thing I did... I thought I would have to create another divX file with null frames removed. But exactly how would I go about doing that?

Thanks! ^-^