Multi-threaded compression on multiple boxes

[DJ_Izumi]

I had this idea in my head and I'm wondering if it's stupid. To me, it sounds like a good idea and that it's almost suprising that no one has done it before, which leads me to believe that I'm failing to understand something and the idea is infesable.

The idea is to design a Windows based video encoder, probably for x264 that can operate in a sort of Server/Client situation. You install the server/encoder on the 'main' computer and install the client program on other computers on your network, the encoding job will then be shared by the other computers on the network like cluster computing only idiot proof for the lamen, and you're encoding is boosted by the work being shared by any computers available on the network.

So tell me why this is a stupid idea, cause there just has to be a huge hole in it. :/

[Coderjo]

This sort of thing has been done before. At some point someone had an MPEG2 encoder that could be run in a cluster. Also, x264 already has a multithreaded mode. It won't compress as well as using single-threaded mode, though, since it splits the video into pieces somewhat randomly, and the video encoded by one thread cannot reference frames in video encoded by the other thread. Also, I think it would be really hard to idiotproof the cluster mode, and would wind up being fragile.

[Kalium]

It would be difficult. Before any actual encoding could be done, the i-frames would have to be determined. Once you had that, you could pass the segments intelligently around.

The biggest problem is that video encoding with inter-frame compression is not easily parallelized without loss. So yeah, you're right Izumi: there is a huge hole in it - the amount of pre-processing that would need to be done, although perhaps some of that could be parallelized.

[Zarxrax]

Look up TCPDeliver in avisynth documentation.

Do filtering on one machine and then encode on another.

[TaranT]

Sony Vegas already supports "render farms" over a network. When you buy it you also get two render-only licenses. And their update to v 6.0d apparently added H264 encoding.

[DJ_Izumi]

Interesting! o.o

But I don't use Vegas. XD

[Coderjo]

It would be difficult. Before any actual encoding could be done, the i-frames would have to be determined. Once you had that, you could pass the segments intelligently around.

With h264, you would have to find the IDR frames. Nothing can refer to frames before the last IDR frame, but they can refer to frames before the last I frame. I don't know whose crazy-ass idea this was...

But otherwise, yeah. you could do one pass to determine where the IDR frames are and then slice it up into each IDR chunk and have different systems encode the video between the IDR frames. You shouldn't lose too much in the way of efficiency this way, since you are splitting on a boundary anyway.

[DJ_Izumi]

How long does the first pass take? I've only tip toed into h264 using RealAnime which uses x264, even at it's highest settings the first pass is fairly quick compared to the other passes.

[Kalium]

But otherwise, yeah. you could do one pass to determine where the IDR frames are and then slice it up into each IDR chunk and have different systems encode the video between the IDR frames. You shouldn't lose too much in the way of efficiency this way, since you are splitting on a boundary anyway.

I'm willing to bet that there's no good way to parallelize the first pass, too...

Which means you do a nasty serial job, and then you can parallelize the second part within limits. Seems like a lot of work.

[DJ_Izumi]

Would it practical to use one computer to quickly scan the entire video for obvious scene changes and divide the work up by those sections without experiancing losses? You'd have fairly large sections to share between the computers but it'd be quick and easy.