AMV Encoders - A/V Sync Test

[Quu]

Some of you are aware of the testing I have been doing on some other forums (Zarxrax is even passing on some of my preliminary results to the tool vendors). As a hint so far, don't trust anything except uncompressed video and uncompressed audio as a "master" even with AVISynth, avoid re compressing if you can, always try to re encode from the master.

I have gotten a couple of queries about it, so I figured I would explain what I am doing. And to get a bit of peer review. This is meant for AMVers to show AMVs. This is for you. Review my testing methods and plan, find flaws, suggest additional tools, methods. I could use an uncompressed AMV or two, as I am using two synthetic videos currently in my testing.

The long version is here
http://quu.livejournal.com/819616.html?view=2905504

The short version:
I am working on a new "playback platform" for conventions based on VLC 1.2.X. No it is not done yet, it is something for the future. It is meant to be future proof, and to be flexible, adapting to convention needs, instead of forcing the convention to adapt to it. It is also meant to be cross platform, and consistent, again this is more for convention needs, but it helps editors.

But, it also needs to be tested and verified (I tested the hell out of NS2K cards), and that is the process I am in right now. I am defining the platform, and the tuning of the platform. After that I will be testing the various encoding methods against this platform, and making a test flow to verify the results. I should be able to come up with various options for encoders.

What I ultimately want is a way for a convention to be able to use the "playback platform", and for an AMV creator to feel confidant in preparing videos for that platform. I am focusing on "free", as in beer, software, and currently focusing on the Windows platform. Most of the encoders I am testing are multi-platform, so it may not matter.

given time and resources, I could expand my testing to Mac and Linux, and to various non free platforms, but that is out of my initial scope.

[Mister Hatt]

>Based on VLC.
With all due respect because I think you have a good thing going, please stop talking like you know what you are doing. Review this one decision you've made first.

[Zarxrax]

>Based on VLC.
With all due respect because I think you have a good thing going, please stop talking like you know what you are doing. Review this one decision you've made first.

It helps if you can give people constructive advice instead of always just saying that what they are doing is bad or wrong. Why specifically shouldn't he target vlc?
Yea I've heard you mention in the past that it does some things wrong. But where could he reasonably be expected to find that information?
Also as I understand it, vlc has a few issues with specific types of files. If his target encodes do not involve those specific issues, should it not be a moot point? (though I can't remember what specific things vlc has problems with, so I'm obviously no help there)

[Mister Hatt]

I'll go with the biggest point then. VLC is unable to decode AVC properly. All AMVs should be encoded with AVC, preferably via x264 with a constant ratefactor. If VLC cannot decode that, then it isn't worth using VLC at all.

[Quu]

I am open to other suggestions as the rendering engine, I can add them to my test setup, I am still early in my design phase, so can make changes. VLC provides allot of the framework that I am looking for, and with my testing, i will see if it provides the sync and playback. I can always code my own, but I would rather utilize an existing framework that I can build to save time and effort.

I do take one issue with what you said... "All AMVs should be encoded with AVC"
I am looking at a very specific target... convention playback... in that case, AMVs should be encoded with what will allow them to be played back with the highest reasonable quality and sync, while still being of manageable size.

I would love to be able to store and playback Lagarith(or FFV1) and flac encoded AMVs, but we are not there yet. MPEG-2 video can handle HD resolutions, and can maintain the same quality as H.264... of course it is nearly double the bitrate of H.264 at the same quality level. Hard drive space is cheap.

And, why do you assume I don't know what I am doing? What I am doing, right now, is testing... finding out what works and what does not. Can you give me examples of VLC and AVC, and an objective way to test and confirm any issues you have? You would not believe the amount of issues I have already found with my testing, and that was based primarily on AVISynth. Review my test plan, give suggestions on what I should be looking for, or example videos that I can use as a reference.

I have a rule at work, which amuses my boss to no end.... nobody can tell me I am wrong. They can tell me how I am wrong and what do do to fix it. They can tell me what I am doing wrong and what I need to do it right, and they can tell me why I am wrong, and explain what is right.

[Mister Hatt]

I didn't realise you intended to write an actual player of sorts. Really all that is needed is a build of mplayer-uau such as Kovensky compiles. The other important thing is to get con managers to enforce people use decent codecs/formats. The best possible setup would be x264 crf with FLAC in matroska.

Regarding what I read on your LJ about sync, that seems more like an incompetent sound technician than anything else. The speakers should be configured for HRTF which would prevent the issue you're describing entirely, regardless of position in the room. Additionally, the human eye/ear is unable to detect a desync shorter than one tenth of a second, so single frame offsets shouldn't be a problem.

Regarding VLC playback:
http://doom10.org/index.php?topic=38.0
http://forums.animesuki.com/showthread.php?p=1910683 (slightly old)

The best systems for playback are currently the latest CCCP Beta and DivX7 if you're going the VFW/DirectShow route. If you are going for a more platform agnostic and self contained route, then mplayer-uau from the build repo or doing it via system links the way the Ophion project does. Kovensky maintains to a degree a build of mplayer-uau for windows.

As far as codecs go, I would really suggest that rather than making a system that works with everything, conventions enforce specific sets. As I wrote above, FLAC in matroska, no header compression, so maybe use MKVToolNix 3.4.0 instead of newer versions, along with a decent build of x264. Force CRF. Good basic options in a guideline would be --preset veryslow --tune animation --crf 18.

I am somewhat impressed you found L-SMASH to be honest. It isn't well known however already beats GPAC. Unfortunately, there are many formats that use the same mp4 extension which makes mp4 support very shonky on a playback system. L-SMASH is nice and all but it's still better to just use matroska.

All in all, rather than making a full system for playback, you just need a decent GUI wrapper or something for mplayer-uau I think. Preventing people from using stupid formats or codecs should also ease the issues conventions seem to be having.

[Quu]

I am one of those con managers. We talk. If i can provide a reference player platform, and provide proof on why they should move to it, I assure you, more than a couple will.

If I am going to add FLAC to my list of codecs, which encoders should I include, ffmpeg and the main FLAC binaries at http://flac.sourceforge.net? Should I add Vorbis and Theora? (not going to touch snow, since it is not bitstream locked yet) Dirac? I am guessing that since I added MKV, I should add FLAC, but where do I stop. I like sticking to the MPEG family, i just wish the MP4 container platform was more stable, instead of some muxers being "Part 1", others being mostly "Part 12", and others being "Part 14" and "Part 15". The "MP4 File Format" has 4 different definitions, based on the time the encoder was written, and what the encoder wanted to support. There is a reason I like MKV. I found out about L-SMASH by asking for help, providing examples, and engaging the developers.

(I can't download the mplayer-aua from his site, my work firewall is blocking it based on malicious software flag, it will have to wait until i get home)

Right now my testing is on Sync, not on quality. If the encoders are breaking sync between audio and video, then i don't care how the quality is. Once I have a shorter list of acceptable encoders, I will then test for quality, which is an entire other set of issues. Bitrate can solve for quality. Like i said, hard drives are cheap.

I can't "force" people to do allot of things, even when managing the contests. If I limit it to H.264 CFR with FLAC audio and mkv container (I have had no problems with header compression, not since FFMpegSource2 was updated), what is to stop someone from using QuickTime to encode the video, or with using a two pass encoding, Nero, etc? FLAC being lossless, i don't care what encodes it, or do I? Do I start inspecting the bitstreams to enforce my choice of encoders and settings?

What I have found is that the best thing is to define an open and assessable playback platform, and then a recommended encode path that is optimal to that. Maybe provide a set of compatible choices. Most creators will follow that encode path, and the hard core ones will replicate the playback platform to test their videos, giving them more control. I have been around a while, trust me on that. (The Netstream 2000 was "pushing it" on accessibility, sorry) The best thing I can do is publish the playback platform, and provide a "reference" encode/preparation path.

There is also an existing library of MPEG-2 encoded videos that I need to maintain backwards compatibility with. I originally started into this assuming that timing and the like was defined in the format spec... boy was I wrong. The most bizarre part is that AUDIO is the part giving me the most grief. Video is pretty easy to test, it either works, and decodes to the same amount of frames as the original, with the frame contents matching, or it does not. rather simple and easy to test. The audio, it depends on the decoder, and the encoder agreeing on how many empty sample to include/consume at the beginning. This is why I have to define my playback platform.

I am using VLC 1.2.0-DEVL, which I am building from source. The current public release is 1.1.2 (though 1.1.3 is coming out soon). I have been talking with j-b, and a couple other devs, about VLC. As I find things and fix them, they are willing to review my patch, and if it works, include it into the trunk of VLC, meaning I don't have to maintain it. I have kind of been derailed from this as I started researching the sync issues in the encoders. If you have a video that breaks in VLC, give it to me, I can take it to cormish, j-b, and the like and we can work on a fix. Or then I have an example that I can use to move to another player and test it to.

I like VLC because it is cross platform, hardware accelerated, plays all the test videos I have thrown at it, and I can write a skin, lua plugin, and replace the http interface, without having to patch or recode VLC. IE I can take a stock VLC add a .zip file of my additions, and it is ready to go on any machine. I also like that VLC has no external dependencies. It does not use DirectShow and does not care what filters you have installed.

I had originally (a year ago) started under the assumption that I would be writing my own software from scratch, using ffmpeg and DxVA2, in probably VC++. Then work got stupid busy for me, time became a premium, so I started looking around at more options. As I learned more, my goals changed. Future proofing became more important, along with removing as many dependencies as possible, I do not want anything based on DirectShow. The filters I have installed may/will be different from an AMV creator, so their playback experience might be different. I want consistency. CCCP and DivX7 are not options for me, since other filters installed on a users machine could over ride those with out them knowing.

After I decided to explore other options, I bought an Acer Revo 3610 and installed XBMC on it, thinking I could write a set of plug ins for it, controlling stuff on a second computer via a GWT web app. At the time my imagined "convention setup" was a set of these little computers with a single control computer managing them. Finally VLC 1.1.0 was launched, it was, on paper, everything I wanted (all of your problem references were pre 1.0.0). It is based on ffmpeg and a small set of internal codecs (like FAAD2). It is cross platform, and very extensible, and supports hardware acceleration. There are some warts, but I have source access and am fixing the ones I know about. I have a skin almost finished that allows me to control playback on a dual monitor computer, one being the projector. I have the skeleton of an http interface written with GWT that replicates this skin, allowing me to control the player from a remote interface (such as an ipad on the stage during a panel?). I have also been poking around the lua plug in interface, seeing if I can define new play list modes.

Am I completely sold on the VLC based solution, no, but it is the best I have right now that solves all my needs. It is part of a whole. I can always go back to coding my own solution, or somewhere in between. But for everything I do, it is based on providing the best possible playback at a conventions. Right now, any issues with VLC, i can fix, and at worse, I fork it. (I really don't want to fork it, as the devs have been very open to my changes... they are busy and told me "code it yourself and give us a patch")

I have talked to allot of AMV creators over the years, thought many of the new ones don't know who I am. Many of them can spot a 1 frame error, specially when it is their own video. Also, these errors become cumulative. If your encoder adds a frame, the decoder adds a frame, and then the sound system/video system at the convention adds a frame, you are a 1/10th of a second off. It is in my best interest to minimize any loss during the process, so that when there are unavoidable sync issues, they are alone, and not multiplied by other issues.

HRTF is an audio location system, not a timing thing. It allows for the placement of audio cues and perceptual location tricks to make a listener think an audio source is to the left or to the right, behind them, etc. It would not solve the fact that for a medium sized room (where the VAT is) the audio cues will reach the front row 70 milliseconds before it reaches the back row. That is physics. For larger rooms (such as the main events room at Otakon), the back row will be more than a 10th is a second away from the speakers as the speed of sound is. This is one of those unavoidable delays, which we can only mitigate, not remove.

Take a look at my testing methodology, tell me what I need to improve, what is wrong, what is right. Provide some sample files I can use in my testing (right now I have two synthetic videos). Become part of the process to help me make it better. When I am done, we should have some definitive, repeatable, results that we can then use to make things better. And, as new versions of existing tools come out, and new tools are created, we can add it to the test and check the results. We will "know" instead of "think" what is best, and why.

I will defiantly take a look at mplayer-aua when i get home, thought probably not until after this weekend will i delve deep into it. But do me a favor, and look at vlc 1.1.2 (at least) and help me. Another option I was told to look at was MPC-HT?

The biggest problem I have, is objectively testing the sync of the playback with the player (any player). Is the camcorder method the best you can think of?

[Quu]

For reference, what I do for a living is programing and testing. Primarily my programing is in tools, but I do get to write large apps at work, and I do allot of code review of other people's stuff when it breaks to help them find why. My testing is primarily in load testing and stability performance, but I am also given the new projects that don't have a testing methodology defined yet, and I get to define it. I am also given the weird projects at work, since I am good at breaking things, and being a programmer my self, the development teams can't lie to me.

[SailorDeath]

OMG Quu! I mean you're like the 3 person to join the org when it was first created. Hey wait I think I was the 4th

[SailorDeath]

OMG Quu! I mean you're like the 3 person to join the org when it was first created. Hey wait I think I was the 4th

Heheh acutally you were 44th and I was 46th but still hehe