Instrumental Anime Project

[pen-pen2002]

I finally finished with the encoding. The filtering is much better now and it's a first generation encode. Aslo I fixed the AR and cropped it correctly, which was off before.

Bakadeshi, I already have a creditless file ready for compression so I should have that done soon.

Rose - You could also try checking out #AMV, as apparently Ian Roberts is on a good amount of the time and you could ask him directly about what he thinks is the best way to go about it and about hosting. Kalium apparently talks to Ian Roberts a good bit so I'll ask him when Ian is around.

[Brain-Carnival]

OMG i cant wait to see the final video guys why do you let us wait so long^^

..is there already an fixed release date or something??

[Otohiko]

Well, we were hoping to have it released around christmas, but some nasty encoding/compatibility issues popped up in the process. rose4emily has been working to get those, so as soon as he has an encode that doesn't get messed up when played on a Windows system - it'll be out.

That should be very soon we hope.

[NeoQuixotic]

Just posting to wonder what the progress is so far? I assume we still have no good way to get the mpeg2 files for the DVD via FTP. If this is stopping us, lets just mail cds. As far as the encoding problems, I think rose is pulling his hair out (at least I would be.) This must be the longest time this thread has gone without a post! I'm ready and willing to submit my mpeg2 file. I hope we get some activity back ^_^!

[Bakadeshi [AuN Studios]]

Just posting to wonder what the progress is so far? I assume we still have no good way to get the mpeg2 files for the DVD via FTP. If this is stopping us, lets just mail cds. As far as the encoding problems, I think rose is pulling his hair out (at least I would be.) This must be the longest time this thread has gone without a post! I'm ready and willing to submit my mpeg2 file. I hope we get some activity back ^_^!

Those who are willing to mail the cds with the mpeg2s on them can PM me and I'll PM you an address to send them to. (no mailbombs please ;p)

As for the status on the project encode and widescreen sections... I'm interested also. Update us Rose!

[ooshna]

I was just looking at the catergories for the VMAs and seen best use of simplicity any of you know how to get your video added to be nominated.

[downwithpants]

ooshna, i didn't see your videos in the nominatable videos list. did you make sure all your videos were eligible before the cutoff date, which i think was monday or tuesday? to check your videos' eligibility, go to the my qualified videos link in the vcas section, and make sure all the potential disqualifying factors say no.


i emailed rose4emily on monday, asking how the project was going, but he has not yet responded. i hope he's still alive.

[ooshna]

it said that only one of mine was nominatable. Which is Simplicity.

[rose4emily]

Wow - It's been a while...

Anyhow, the reason for my extended dissapearence was my Software Engineering project - which, due to an egregious mistake I made with architectural design, was both "foo" and "bar" when Release 1 was due. If you aren't a computer geek, I'll elaborate that "foo" and "bar" are commonly-used placeholder words tossed into example code when the person who wrote the example wasn't in the mood to come up with more sensible names for their hypothetical variables. It also happens to have originated from the "unofficial" WWII acronymn "FUBAR" - which is what something often is after you take a sledgehammer to it, detonate a bomb under it, or fail to RTFM before attempting to operate it.

In essence, I built the application design around a relational database server that had to be accessed through SQL and JDBC. This would've been a good idea - except for the fact that I was the only member who knew SQL, I had to learn the JDBC as I went along, and none of the other members had any experience or significant interest in becoming familiar with either. Consequentially, I found myself doing almost all of the coding, and the project managed to both suck up about 40-50 hours of my time per week in coding and research, and to fall horribly behind schedule.

Anyhow, for R2 we're going forward with a custom DB design I came up with that's going to require a lot more code, but which all four of us will be able to effectively work on.

In short, the assumption that everyone can "just learn it" is apparently not a very safe one to bet a key design point on. I guess I've just be spoiled by working on all of my past group development projects with groups of experienced professionals under the direction of a skilled manager. I am not that manager - at least not now or any time in the near future.

The good news is that my team, while inexperienced, has the saving grace of being both bright and well-motivated. Thanks to the design refactoring and hard kick in the ass that came with R1's failure, we're actually recovering quite nicely.

---

Okay, so that's my excuse for dropping off of the face of the earth. Now, to business:

As a side effect of my SE project, I had to re-image my desktop to convert it from an all-Linux server to a Linux/Windows dual boot. Bassically, I needed the Windows partition as a testing platform.

This was yet another delay in getting anything done for this project, but it gave me a chance to get a newer Linux distro and install all my editing/transcoding software in the right order to support DV encoding and DVD burning.

The DVD burning works beautifully. That's some very good news, as I finally have a "safe" (as in not on some flaky external hard-drive) backup of all the project files that were too large to fit on a CD. I was also able to completely clean out my hard disk to make room for the additional oversized video files that I'm going to have to make with my Nth (I've forgotten how many times I've tried to get this to work) compilation process. Finally, I now have the capability to actually make those source disks for the DVD compilation - and to create copies of the DVD compilation.

The DV encoding, on the other hand, has me a bit perplexed. Using MEncoder with the "libdv" codec, I can produce DV that joins nicely, but is ugly as all hell. Specifically, it looks blocky (not in the low-bitrate MPEG sense - more in the pixelated 8-bit video game sprite sense). Since DV is THE standard for digital video editing, and is encoded at a bitrate unheard of even on DVDs, it really shouldn't be doing that.

Using FFMPEG, which has it's own DV codec, should work. At least the FFMPEG documentation states that DV encoding is supported, and most of the forum/newsgroup posts I've run into suggest using it rather than MEncoder/Transcode for DV encoding - though none seem to give a specific reason - or specific instructions on how to do this.

So far, I've been having problems with it complaining that the supplied video is in the wrong format. This makes sense, sort-of, as DV is really picky about what it will encode. Think the "multiple of 16" restriction imposed by the MPEG family of codecs is restrictive? DV only does two things: 25 fps PAL at 720x576, and 29.97 NTSC at 720x480. These two formats happen to be those natively supported by most of the world's TV sets (which tend to be either PAL or NTSC, though the whole "HDTV" thing is throwing a big monkey wrench into that assumption). They also happen to be the ones most commonly seen on DVDs, which tend to be NTSC in both America and Japan. PAL is actually the better broadcast standard thanks to its superior color reproduction (and, IMHO, the greater spatial resolution - though some prefer the [barely perceptably] increased refresh rate of NTSC). NTSC, however, is what both American and Japanese TVs are designed to support. Because of this, Anime is almost always encoded in NTSC for both domestic (Japan) and foreign (America) markets. PAL playback on an NTSC television requires a lossy format conversion involving some frame copying and discarding the extra spatial resolution. Of course, NTSC to PAL conversion is also lossy, as the extra frames are tossed out and the extra spatial information has to be "made up" by copying or interpolating some of the horizontal lines. So, maybe, French-translation releases of anime are distributed in PAL, instead of NTSC.

Anyhow, back to my encoding problem, I obviously can't encode a 24 fps video at 512x288 (or 512x384, or 640x480, or 640x360, or 720x405, or 720x540, or any of the other resolutions I have source video or encoding experiments in) to DV. Neither the file format or the codec supposed to do that - so it's not going to happen.

I somewhat expected that to happen as soon as I realized DV was the only thing that was going to work for the compilation process (since HuffYUV files, when joined, seem to go all screwy with funny patterns and colors - probably due to the fact that no two files have the same Huffman encoding tree, which is the key needed to properly interperet the Huffman-encoded data during the decoding process, and uncompressed video is just too damn big for me - or most professional studios - to work with). So the need for a format conversion was no big surprise. Problem is, I can't seem to get FFMPEG to convert the video format and encode to DV in one step. This, as it adds another generation to the already multi-generational video, is unacceptable.

So I have to try a couple more possibilites. One is a tool called 'smilutils'. I haven't used it before, but I've heard it suggested as a "wrapper" program around FFMPEG's DV codec that should help me with my MPEG1/MPEG2/MPEG4/HuffYUV (really, could you guys have worked in any more submission formats?) to DV conversion.

The other is the possibility of using MEncoder to make an NTSC HuffYUV encoding of each video and then using FFMPEG to encode those more "DV-friendly" files to DV.

I'll also be posting this little encoding question to the "Video Software Help" forum - on the off-chance that someone here knows something about video encoding that the rest of the Linux users and video experts at RIT and on the Internet don't know or haven't mentioned. At first, I consedered the thought a little crazy (seing as how I live with a pair of filmmakers and work amongst a few thousand Linux geeks) - but, then again, the desire to transcode MPEG-family videos with odd resolutions and framerates to DV is probably something only AMV makers would ever have. The real filmmakers and animators get DV (or - better yet - actual film, cels, or pristine digital renders) to work with in the first place, and would never touch "source" in a distribution format like MPEG2 DVDs, much less lossy re-encodes of footage from an MPEG2 DVD. The Linux geeks around here, on the other hand, all seem to be too busy coding, gaming, getting drunk, or watching TV to mess with stuff like video editing.

---

The farther I get into this project, the more things I know I should've done differently. For starters, I should've asked for NTSC DV in the first place - back when you were still making the videos and could have produced NTSC DV renders as easily as anything else. I also should have edited my own video in DV - ripped from the FLCL DVDs. Instead I edited DivX4 footage re-encoded from a WMV9 "fansub" (which I later learned was actually created after the American DVD release - pretty much invalidating the whole "fansub" thing by removing both the hours of tedious translation work and any reasonable excuse for its distribution), rendered to PNG, encoded a copy to XviD, and lost the PNG render to a disk failure. If you're going to spend 40 hours staring at the same 13 minutes of anything, it's worth paying for the DVD. I'm almost tempted to do that one again - the right way - now that I've learned enough to do it twice as well in a quarter of the time. I also should've made no attempt at setting a release schedule, since it's pretty clear that the combination of my academic schedule and this project's technical roadblocks are making any kind of progress estimation impossible. I also should've tested things like the AVI joining process with some arbitrary video before several months went by for the submission/creation process to come to an end. I could've found those audio dropout problems in a joined version of those crappy DivX4 re-encodes of the shameless WMV9 FLCL rips I had at the time - and found a better way to do this while I was on summer vacation and actually had a fair amount of free time for this sort of thing.

I'll definately be writing a "How To" (or maybe a "How Not To") guide for these feature-length group projects. Of course, people could just read the thread - but that'd take a while, and expose them to far more of my convoluted thought process than any human being should be subjected to within a short span of time. I think this and AMV Hell 3 (now there's an idea! I should ask Zarxrax how he's planning to do the compilation...) have garnered enough attention that future editors will either flock to feature-length projects or, realizing how much work they are, avoid them like the plague.

I'm no giant - but I still imagine one could see farther by standing on my shoulders. One normal-sized person, that is. Giants, sumo wrestlers, American football players, those Scottish guys who throw logs for fun, etc. are pretty much on their own.

[rose4emily]

BTW, I think I mentioned this somewhere in that last monstrosity of a post, but:

rose, what happens if you do the join via a lossless video codec (such as Huffy) and then reencode the final version to divx? so theres little to no loss quality wise in the second gen encode? Was there some obsticle in doing it this way?

The HuffYUV files get scrambled in the joining process. It seems Maverick managed to get it to work in Premiere (which probably does some sort of frameserver thing instead of a direct stream copy), but both MEncoder and VirtualDub gave me nothing but a really big video of attractive (but unrecognizable) abstract colour patterns. The last time I saw anything like that, I was writing a PNG decoder and screwed up the portion of the code that mapped the decoded colour values to pixels in the image. Since the data was written to the pixel buffer in a completely messed up order, the resulting image wasn't the picture in the PNG file, but rather an "alternative" data representation that looked a lot like a colourful cross between a fractal and a Moire pattern. The really wierd thing about that messed up PNG decoder, however, was the fact that each of those patters actually contained a very clear geometric structure - yet the geometric structure produced by each image was very different from that produced by any other image. I wish I still had the source so I could figure out how it did that, because I later tried and failed to reproduce this frustrating but fascinating mistake.

The joined-Huffy color patterns looked more like a colourful version of the "white dots" you can see when an old TV is turned on but the UHF (remember UHF?) cable is loose or disconnected. Much more entertaining than the blue screen all the newer TVs seem to produce under similar circumstances, but hardly a meaningful representation of your work.

I did get a copy to work where I used the failed XviD join (the one with the audio droputs on Windows), encoded to HuffYUV, and then re-encoded to XviD. This however added yet one more generation - and the quality drop was noticable. It also, when I thought about it, was a pretty senseless waste of time - since I could have gotten the exact same ugly encode from a direct XviD to XviD re-encode, bit for bit. I could, I suppose, distribute the HuffYUV encode of the joined film - but I doubt many people would appreciate the 20 Gigabyte download. Especially if they're on dial-up. 20 GiB might take a while if you're on dial-up. You could distribute on a set of five DVDs, I guess, but then you're talking about $15 and 70-80 minutes of burning time for each copy - not to mention the packaging, trips to the post office, and four disk changes in the middle of viewing the film. I'm sorry, but I have to keep anything that wonderfull all to myself.

---

On another note that I should have thought of earlier:

As I recall, Bakadeshi's requirements for the DVD included that the footage be submitted in NTSC (29.97fps, 720x480), with the 16:9 videos presented as 4:3 video letterboxed with "black bars" rather than in an anamorphic format.

Since I need to work in NTSC as well, now, there's a lot to be gained, in terms of quality, if you can send me NTSC copies as well. Unless, of course, all you have is the copy you sent to me and the NTSC encoding is just a re-encode of that copy, rather than a rendering from a HuffYUV archive file or your original source and project file. Since almost all anime footage is NTSC, and most editing software performs edits in the source format and then renders to the frame rate and resolution you specify, there's a fair chance at least some of the segments can be spared two format conversions and one generation of lossy encoding if you render directly to NTSC DV (which every video editing app I've ever heard of can do - even the Windows Movie Maker) and send that to me. Even if you don't have the project, but do have a Huffman in the original resolution (and, better yet, the original frame rate as well), a DV encode from that will look better than one from the 512x288 24fps copies I have.

Unlike MPEG4, DV is a codec and format that's both simple and strict enough that perfect compatability is pretty much guaranteed between apps. Actually, the DV standard not only specified a bitstream, but also every bit of the encoding algorithm (and, since DV is a so-called "symmetric system", the decoding algorithm is explicitly defined by the encoding algorithm). It is also limited to one colour space and two video formats (though some "DV-like" codecs exist for other color spaces and formats, like 4:2:2 YUV instead of 4:1:1 YUV, or high-resolution HDTV signals). So, while every MPEG4 codec on earth is both incomplete (in the same sense as every web browser on earth is incomplete - the standards involved are so complex that "good enough" implementations are all that any sane group of developers will ever produce) and unique (in that MPEG4 leaves most of the math and some of the bitstream and quantization details up to the people writing codec implementations), every DV codec is pretty much the same as any other - with performance optimizations (in terms of encoding time, not video quality) being the only selling point of expensive commercial versions. DV is also used as the native format of a wide variety of hardware devices, and one DV file often sees at least two or three different devices throughout the capture and editing process. Because of this, bugs in a DV implementation aren't just glossed over as "quirks" or "features" the same way they are in distrubution codecs like MPEG4 (the most popular versions of which are actually probable patent violations, hence XviD only offering the source to their codec - or, worse, known illegal hacks of proprietary implementations, such as the original Divx ;) codec).

A DV file is 25Mbits per second. That's about 10 times the bitrate of most DivX videos, but much smaller than a HuffYUV of the same video. Actually, that's about the same bitrate as you see with the 512x288 24fps HuffYUV files - but those actually contain about 1/3 of the data, due to the reduced resolution and frame rate. And this is anime. The only more Huffman friendly video source I can think of is Daria. Actually, I did once see an animation called "The Cat and The Moon" that was entirely in black and white - by which I mean BLACK and WHITE. That would Huffman quite nicely, 'though a RLE or LZW type encoder would actually work even better. BTW "The Cat and The Moon" is a really cool short animation, and you'd be amazed by how much it can express with nothing but two colors.

NOTE - I'd actually prefer to get the 16:9 segments as 720x480 anamorphic segments, rather than 4:3 720x480 video letterboxed with "black bars". Bakadeshi needs those because a DVD can be 4:3 or 16:9 anamorphic, but not part one and part the other. At least not through any means I've heard of that doesn't involve calling all the 4:3 sements "menus" (which can cause some funky side effects in the behavior of the viewer's playback controls). I, on the other hand, don't have this limitation for the online distribution - and would like to take advantage of the fact that anamorphic widescreen has a sharper picture than letterboxed widescreen.