Compiling x264 on Linux

This forum is for questions and discussion of all the aspects of handling your footage. If you have questions about capturing/ripping footage, AviSynth, or compression/encoding/converting, look here.

Compiling x264 on Linux

Postby Qyot27 » Mon Mar 23, 2009 10:28 am

While I know many of you that use Linux around here probably don't blink at compiling your own builds, I feel that it could be of use to have a thread detailing how to do so for those that want to, but don't have much (or any) experience with compiling programs. Heck, I'm just about clueless myself, although I have managed to do so when I need to.

One of the biggest benefits, at least to me, of using x264 under Linux is because the sole 64-bit capable machine I have access to only has a 32-bit version of Windows on it. For people in this kind of situation, Linux is a good way to use the full 64-bit power of the computer while not needing to buy another OS (unless you go with Red Hat, et al). And the difference in encoding time between 32- and 64-bit versions of x264 can be pretty dramatic.

The best approach, in my opinion, would be to set this up by distribution (and version, if you feel the need, and the process differs so greatly). For instance:

Ubuntu 8.10 [Intrepid Ibex]:
0. This process uses GCC 4.3.2 to do the compiling. I don't know how this might change under a different version of GCC. Just be aware of this fact. Also be aware that this may require that you have the Universe and/or Multiverse repositories open.

1. Install the necessary dependencies (you can do this in Synaptic if you want, but I'm going to use the Terminal commands for ease of doing all this in one go). You only need to do this once.
Code: Select all
sudo apt-get update
sudo apt-get install build-essential git-core checkinstall yasm texi2html

2. Download, compile, and install zlib. zlib is required for building GPAC, which provides the MP4 output support in x264.
Code: Select all
wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar -zxf zlib-1.2.3.tar.gz
cd zlib-1.2.3
./configure
make
sudo make install

3. Download, compile, and install GPAC 0.4.5. Once again, you'll only need to do this once (although feasibly when GPAC updates it might be a good idea; the reason I don't suggest this for zlib is because, well, the main site's source tarball hasn't been updated since 2005, and I don't know how easily this works using a newer version).
Code: Select all
wget http://superb-east.dl.sourceforge.net/sourceforge/gpac/gpac-0.4.5.tar.gz
wget http://voxel.dl.sourceforge.net/sourceforge/gpac/gpac_extra_libs-0.4.5.tar.gz
tar -zxf gpac-0.4.5.tar.gz
tar -zxf gpac_extra_libs-0.4.5.tar.gz
cd gpac_extra_libs
cp -r * ../gpac/extra_lib
cd ../gpac
chmod +x configure
./configure --disable-opengl --use-js=no --use-ft=no --use-jpeg=no --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no
make lib
make apps
sudo make install-lib
sudo make install
sudo cp bin/gcc/libgpac.so /usr/lib

(The string of disables I used with the configure step is a result of my paranoia about getting GPAC to build. The only ones indicated on the help forum over at Sourceforge as needing to be disabled were opengl, jpeg, and png. However, since what I put down above worked, then I don't mind. If I get confirmation on how the others don't affect the build process, I'll amend that step. But until then, I'd just rather not take the chance)

4. Download, compile, and install x264. If you want to use some of the common patches* (see links below), download the diff files and copy them to the x264 directory (the one git creates). Then replace file.diff in line #4 to the actual name of the patch and repeat for as many of the patches as necessary.
Code: Select all
cd ~/
git clone git://git.videolan.org/x264.git
cd x264
patch -p1 <file.diff
./configure --enable-shared --extra-cflags="-march=pentium3"
make
sudo checkinstall --fstrans=no --install=yes --pkgname=x264 --pkgversion "1:0.svn-$(grep X264_VERSION config.h | cut -d' ' -f 4)-0.0ubuntu1" --default
sudo ldconfig

When using ./configure, be sure to set the --extra-cflags option to the proper CPU type you have. I use Pentium 3 on mine because I have a Celeron 2; on another computer I use -march=athlon64, since it has an Orleans processor. You can check the list of proper CPU flags here:
http://gcc.gnu.org/onlinedocs/gcc-4.3.2 ... 64-Options

*Patches:
x264_win_zone_parse_fix_05.diff:
http://skystrife.com/x264/

x264_hrd_pulldown.10_interlace.diff:
http://forum.doom9.org/showpost.php?p=1 ... count=1726

Keep up with patches here:
http://forum.doom9.org/showthread.php?t=130364


Now then, as far as using x264 is concerned, I'm going to assume you already know how to use x264 from the command line. If necessary, fire up MeGUI or Zarx264gui under Windows, make them expose the command line options, copy those down, and then bring that text file over to your Linux install.

What I'm going to explain instead is how to feed x264 your video so that you don't need to bother with trying to get raw video encoded correctly (which can be a real bitch - see my second post for a quick fix of sorts).

Linux builds of x264 don't support AVI or AviSynth input, because of fairly obvious reasons (lack of VFW and subsequently, lack of a *working* AviSynth 3.0). This presents a challenge, because otherwise you would need to output your video in raw format and then give this to x264 - however, doing that eliminates both the framerate and resolution information from the file, and requires you to specify this in x264's options. Quite frankly, this can be annoying. But there is a solution. Let's go into more detail below.

You'll want to make sure that mplayer is installed first. The best option for this is to go to Applications->Add/Remove Programs, and find SMPlayer (which I prefer because of the GUI; if you want standard mplayer, you can get it from Synaptic or something). Install it.

The video should be in a lossless format that mplayer can open. This could be any number of things; personally, I use YV12-mode HuffYUV (ffvhuff, if you wanna be technical about it). Uncompressed YV12 would be fine as well.

Now that that's done, we're all set. What's going to happen is that we'll make a named pipe using mkfifo, then have mplayer open the video, and pipe it over to x264.
Code: Select all
mkfifo input.y4m
x264 input.y4m <options> | mplayer -nosound -vo yuv4mpeg:file=input.y4m inputfile

In reality, this would probably look something like this:
Code: Select all
mkfifo input.y4m
x264 input.y4m --crf 18.0 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-adapt 2 --b-pyramid --weightb --direct auto --deblock 1:1 --subme 9 --trellis 2 --partitions all  --8x8dct --scenecut 100 --threads auto --thread-input --sar 1:1 --aud --progress --no-dct-decimate --no-psnr --no-ssim --output "dawn.mp4" | mplayer -nosound -vo yuv4mpeg:file=input.y4m dawn.avi




Most of this information is cobbled together from various other sources (primarily the Doom9 forums and Ubuntu forums) I found online or asked about directly. If you want to customize your builds with special flags then you can check with ./configure --help on a per-program basis.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Compiling x264 on Linux

Postby Qyot27 » Mon Mar 23, 2009 10:36 am

For making the proper raw files to give to x264, you can process the files using mencoder. The easiest way is to edit mencoder.conf to always be set to a certain type of compression.

x264 only takes I420 format video. So, you'd make mencoder.conf look like this, more or less:
Code: Select all
ass=no
subcp=enca:ru:cp1251
fontconfig=no
nosound=yes
autoexpand=no
of=rawvideo=yes
ovc=raw=yes
rawopts="vf format:I420"
ffourcc=I420
force-avi-aspect=1/1

This will allow you to use this simple command-line to convert to the right format:
Code: Select all
mencoder inputfile -o outputfile


This can be used under either a Windows version (requires Wine) or the version that comes with SMPlayer. Using Wine, you'll want to put mencoder.exe and the mplayer subfolder that holds mencoder.conf in Wine's windows folder. You'll also need to make sure to call wine mencoder when you use the Terminal rather than simply mencoder. For the native version, you'll stick mencoder.conf in ~/.mplayer and go on your merry way.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Compiling x264 on Linux

Postby dragontamer5788 » Mon Mar 23, 2009 1:00 pm

Woah woah woah... someone other than me uses Linux? o.O

Do you use Cinelerra for editing too? :-p

------------

Anyway, I run Debian, which is what Ubuntu is apparently based on. I get all of my multimedia programs from the apt repository at debian-multimedia. Just add:

Code: Select all
deb http://www.debian-multimedia.org lenny main
deb-src http://www.debian-multimedia.org lenny main


Then of course...

Code: Select all
aptitude update
aptitude install x264


Which installs x264, no sweat :D. I'm unsure if this will work on Ubuntu (those are debian packages afterall...) but Ubuntu apparently has decent compatibility with Debian. Anyway, I just get mencoder, cinelerra, x264 and so on from there. Unless you want the latest bleeding edge builds... I don't think it really matters to go about compiling the development branch... unless... there is something that I'm missing? The version of x264 in the debian-multimedia repository is 0.65.x. Is there anything that I'm missing by sticking with this slightly older version?
dragontamer5788
 
Joined: 07 Oct 2006

Re: Compiling x264 on Linux

Postby Qyot27 » Tue Mar 24, 2009 12:54 am

Nah, I could never get into Cinelerra. I still do my editing in Premiere under Windows. Once Saya (or Lumiera) gets released I'll take a look at those, though.

Well, the bleeding edge stuff is basically the reason I posted this, as those accustomed to using MeGUI know that it's the current revisions that get added to the update chain. Unfortunately, having builds that new (and really, some repositories aren't exactly *new* considering they could be holding a build several months old - and with x264's fairly rapid development pace, it means things get outdated quickly) on Linux means compiling it yourself. The 0.65 part of the version number doesn't tell me much because I pay attention to the revision number, which is the final part - currently at 0.67.1129. 'r1129' tells me more than '0.67' does.

Anyway, are the x264 builds in the Debian repos enabled for MP4 output? There's also other minor issues such as optimizing for one's processor and including relevant patches (for Blu-ray authoring compatibility, as an example) that can be addressed with doing the compiles yourself. On Windows, others supplying up-to-date builds is a pretty well-oiled machine and has been for a couple years now. But that doesn't exist so much on other platforms because it's assumed you can compile them yourself if you want them. It's just that sometimes it's difficult to bring all the relevant info on doing so together so that those starting out can get a handle on it.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Compiling x264 on Linux

Postby dragontamer5788 » Mon Mar 30, 2009 10:21 am

MP4 is a good reason... but more seriously... they've already went up a minor version? Wow, you're right, x264 is pretty fast. Generally, I prefer to avoid the hassle of compiling, even if it isn't fully up-to-date. Also, I haven't bothered to learn x264 yet... and just do all my processing from MEncoder. (which uses x264 library, so I can get h.264 encodes but it is probably even less up-to-date). More of a "yeah, I know the work in compiling stuff... and it ain't pretty :sweat: "

Looking over a few things however...

When using ./configure, be sure to set the --extra-cflags option to the proper CPU type you have. I use Pentium 3 on mine because I have a Celeron 2; on another computer I use -march=athlon64, since it has an Orleans processor. You can check the list of proper CPU flags here:


Erm... you have a ~10 year old computer? I somehow doubt that. It is sometimes difficult to figure out the true core of a Celeron... which can be anything from a Pentium 2 up to a Core 2 Duo! Aside from the cache, they behave like whatever core they are, except they have smaller cache. So run...

Code: Select all
cat /proc/cpuinfo


and you'll get the output of your CPUinfo. From there, you can select the real architecture of your machine (or at least... the one hardwired into your CPU)
dragontamer5788
 
Joined: 07 Oct 2006

Re: Compiling x264 on Linux

Postby Qyot27 » Tue Mar 31, 2009 8:59 am

Erm... you have a ~10 year old computer? I somehow doubt that. It is sometimes difficult to figure out the true core of a Celeron... which can be anything from a Pentium 2 up to a Core 2 Duo! Aside from the cache, they behave like whatever core they are, except they have smaller cache. So run...

Code: Select all
cat /proc/cpuinfo


and you'll get the output of your CPUinfo. From there, you can select the real architecture of your machine (or at least... the one hardwired into your CPU)

Yes, my computer is that old. It's an eMachines T1110 - 256 MB RAM, 1 GHz Celeron 2/Pentium 3 Coppermine (which I was aware of not only from CPU-Z, but also from the fact it's what mencoder and mplayer spit out at me), 100 MHz FSB, i810e chipset. It's from 2001, actually. I'm sorely in need of an update, I know. And yes, it's also my editing setup, and has been since mid-2003. I have it set up as a dual-boot between XP Home SP3 and Ubuntu 8.10. The only reason for XP Home is because that's what the restore disc the computer came with uses. I bought XP Pro SP2 (since slipstreamed to include SP3) last year just before the retail deadline went into effect, but save for very short spans of time in a virtual machine, I'm saving that for when I get a better setup.

Heck, my family even still has the NEC computer we bought about 14 years ago...runs Windows 95 on a Pentium-166 (or maybe it was a -133). And it still works fine, although the monitor is a tad screwy.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Miscellaneous

Postby trythil » Thu Apr 02, 2009 10:54 pm

Qyot27 wrote:patch -p1 <file.diff


Doing that doesn't take advantage of git's versioning capabilities, which is something that you'd want to do if you plan to upgrade to newer versions of the software tools. Instead, do:

Code: Select all
git am file.diff


git-am applies patches generated by git format-patch (and any other patch mechanism that generates similarly-formatted patches). The advantage of doing it this way is that you append to your repository's history, which can make it easier to merge patches later on. It also allows you to back out patches that don't apply cleanly, and all the other fun stuff that you get with version control.

Re: mplayer: you can also use raw YV12 output (-of rawvideo -ovc raw -nosound), which can be handy if yuv4mpeg messes up on you.

Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Re: Miscellaneous

Postby Qyot27 » Fri Apr 03, 2009 12:03 am

trythil wrote:
Qyot27 wrote:patch -p1 <file.diff


Doing that doesn't take advantage of git's versioning capabilities, which is something that you'd want to do if you plan to upgrade to newer versions of the software tools. Instead, do:

Code: Select all
git am file.diff


git-am applies patches generated by git format-patch (and any other patch mechanism that generates similarly-formatted patches). The advantage of doing it this way is that you append to your repository's history, which can make it easier to merge patches later on. It also allows you to back out patches that don't apply cleanly, and all the other fun stuff that you get with version control.

Well, I just found the reference to using patch in one of Dark Shikari's posts that's buried in the Current Patches thread, and just went with that. Didn't know about git's internal patching capabilities. When you say 'append to your repository's history', that refers to what exactly? When I go to update x264 I actually completely remove it first, then completely remove the source directory and download everything from git again, since I can't see as many issues arising that way. Then again I tend to be paranoid on some issues and not on others that are seemingly just as important.

Or newer software tools in reference to git and gcc, rather than x264?

Re: mplayer: you can also use raw YV12 output (-of rawvideo -ovc raw -nosound), which can be handy if yuv4mpeg messes up on you.

Those options are actually covered in the mencoder.conf in the second post. -vf format=I420 was also specifically required in the tests I ran, unless that's what it automatically defaults to in the case of YV12 (I can only remember it kept telling me I need to declare -vf format). -vf format=YV12 causes color errors.

Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.

True, but the cflags make sure that everything (or nearly everything?) else is optimized also.

In my tests, I used just --crf 18 and --progress (dunno if --progress really affects speed, though). Running a non-cflags-enabled build and one with them enabled netted an increase on average from 0.5 to 1.0 frame per second, which considering my computer's age is pretty significant (the actual numbers were something like 3.65->4.02fps or 3.65->4.53fps). I noticed similar little jumps on the Athlon64 machine I tested on using -march=athlon64, although there it seemed like less of a difference (2-3 frames, but only from say, 12-14fps to 15-17fps, respectively). Normally I don't stick to using just --crf 18, though, and any sort of percentage boost can help when dealing with average fps render values in the 0.2-0.3 range or lower.

At this point I can't remember if I was testing 720p or 480p, as I had done both a few times.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Miscellaneous

Postby trythil » Fri Apr 03, 2009 2:10 am

Qyot27 wrote:Well, I just found the reference to using patch in one of Dark Shikari's posts that's buried in the Current Patches thread, and just went with that. Didn't know about git's internal patching capabilities. When you say 'append to your repository's history', that refers to what exactly? When I go to update x264 I actually completely remove it first, then completely remove the source directory and download everything from git again, since I can't see as many issues arising that way. Then again I tend to be paranoid on some issues and not on others that are seemingly just as important.


git's a version control system. It's used to manage (among many other projects) the Linux kernel codebase. (Can you imagine deleting and re-fetching all that every time you wanted to update?)
git-pull is designed to bring your local repository up-to-date, and merge any commits that you haven't pushed upstream. It is very rarely necessary to delete and re-fetch.

When you commit changes (i.e. apply patches), that change is recorded. A series of changes is a git history.

A good git tutorial is here: http://www.kernel.org/pub/software/scm/ ... orial.html

Qyot27 wrote:Those options are actually covered in the mencoder.conf in the second post. -vf format=I420 was also specifically required in the tests I ran, unless that's what it automatically defaults to in the case of YV12 (I can only remember it kept telling me I need to declare -vf format). -vf format=YV12 causes color errors.


If it's color errors like swapped U and V channels, you can throw swapuv in the filter chain. That usually does it for me, anyway.

Qyot27 wrote:Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.

True, but the cflags make sure that everything (or nearly everything?) else is optimized also.[/quote]

How did you isolate the cflags changes from every other factor? (Are you absolutely sure it was compiler optimizations making the difference?)

While you might get a small performance boost out of tweaking compiler flags, overly-aggressive optimization can cause also subtle bugs in program behavior. The default compiler flags are best unless you're sure you know what you're doing.
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Re: Compiling x264 on Linux

Postby trythil » Fri Apr 03, 2009 2:23 am

overly-aggressive optimization can cause also subtle bugs in program behavior

Actually, that probably isn't such a big deal anymore.
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Re: Miscellaneous

Postby Qyot27 » Fri Apr 03, 2009 11:35 am

trythil wrote:When you commit changes (i.e. apply patches), that change is recorded. A series of changes is a git history.

A good git tutorial is here: http://www.kernel.org/pub/software/scm/ ... orial.html

I suppose this means that different versions of the same patch are treated similarly? For instance, the HRD/Pulldown patch was recently updated from 10 to 11, apparently from a major re-write. Or would that require removing the previous version first and then applying the new version?

Thanks for the link to the tutorial. I'll make sure to read through that since beyond the really basic act of retrieving the data, I have next-to-zero experience with git (or other revision control systems, for that matter).

trythil wrote:If it's color errors like swapped U and V channels, you can throw swapuv in the filter chain. That usually does it for me, anyway.

I'm pretty sure it is swapped channels - things start looking blue and orange when they definitely shouldn't. I think the reversed channels are the only thing that really differentiates I420 from YV12 (aside from I420 actually using 4:2:0 in its name).

trythil wrote:How did you isolate the cflags changes from every other factor? (Are you absolutely sure it was compiler optimizations making the difference?)

What else could it be? Everything else was exactly the same, as far as I can remember - same video file in the same method of input (either raw or through piping, as I did separate tests on those), same options, same revision of x264 - I didn't include the patches because at that point I'd not found the information to use them yet. Nothing else was running, except maybe Nautilus and gedit (far more likely to still have gedit open than both it and Nautilus, though). But I usually close all other programs when encoding, no matter what OS I'm using.

I'll probably go and re-run all those tests again so I can get a more definite picture, though.

trythil wrote:While you might get a small performance boost out of tweaking compiler flags, overly-aggressive optimization can cause also subtle bugs in program behavior. The default compiler flags are best unless you're sure you know what you're doing.

I chose to include the optimization because of the general trend on the Current Patches thread for the custom builds there to have them (including Skystrife's builds that are included in MeGUI's update chain, which are -march=pentium2).
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Compiling x264 on Linux

Postby Qyot27 » Mon Apr 06, 2009 4:12 pm

Ok, well, I'm going to eat my words a little bit now. I re-ran tests to determine the speed differences between a CFLAGS-enabled build and one without. The results are posted below.

Source video: scrotal.avi / scrotal.i420
AVI:
Code: Select all
mkfifo scrotal.y4m
x264 --crf 18.0 --progress --output "scrotal-avi.mp4" "scrotal.y4m" | mplayer -nosound -vo yuv4mpeg:file=scrotal.y4m scrotal.avi

RAW:
Code: Select all
x264 --crf 18.0 --progress --output "scrotal-raw.mp4" "scrotal.i420" 848x480 --fps 23.976

x264-r1136M, Apr 6 2009, gcc 4.3.2
x264_win_zone_parse_fix_05.diff
x264_hrd_pulldown.11_interlace.diff

Stats for "-march=pentium3" build:
AVI: 2.77fps
RAW: 2.91fps

Stats for non-CFLAGS build:
AVI: 2.86fps
RAW: 3.09fps

Values were the final reported average fps. The AVI file was compressed with ffvhuff, 95 frames long, and contained mostly static imagery. It would probably be good to run a subsequent test with a high-motion, much longer source, but I needed something quick for right now.

The only thing open was the Terminal. Nautilus, gedit, etc. were all closed at the time.

Also, trying to use git-am for the win_zone_parse_fix and hrd_pulldown patches failed with warnings about not having email/mailbox information or whatever. Those patches didn't come from the mailing list (or at least, the copies I have of them didn't come from the mailing list - I downloaded them straight from Skystrife's host server and the attachments on the Doom9 patches thread, respectively). Is there some other set of options needed to make git-am work on those patches? Because it seems - and I somewhat expected it, considering the description of git-am in the first place - that it'll only accept patches that adhere to its formatting standards, otherwise you're SOL. Which means you'd need to either know where the patches came from to ensure that they do meet said requirements, or that you know how to correct the patches to that standard yourself.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs


Return to Footage Help

Who is online

Users browsing this forum: No registered users and 1 guest