Time and experience, mostly. I actually had an interest in video encoding and disc authoring before I ever joined this site (I cut my teeth by authoring Video CDs
and playing around with converting the videos that came on Enhanced CDs between different video/audio formats for really no reason other than to do it), but AMVs naturally hooked into that interest.Doom9's Forum
is probably the single largest resource for the technical knowledge, and it's also where a tremendous
amount of the development and knowledge base gathers for open source media tools. To wit, actually following the development process of some of the tools themselves can give insight on this too; it's not just about what commands to use, the ability to test against different versions from date A to date B also provides the opportunity to see how parts of the encoding pipeline grow and evolve, and this can then have an effect on the user's workflow. For instance, the difference between what encoding with x264 was like prior to the introduction of mbtree (or the preset/tune system) and afterward.
There's also simply experimenting with things. The AVTECH guides are a starting point, and a generally excellent one for trying to distill the core concepts into a form that beginners can easily absorb (this actually becomes more apparent when you do just experiment on your own, because then the things the guides say will just start to 'click').
Concerning Zarx264gui, this happens mostly because a lot of the encoding tools are actually command-line tools, so frontends naturally pop up to provide users that don't like the CLI the ability to still encode (and when you actually look at x264's --fullhelp, you'll realize that what Zarx264gui does is more-or-less directly expose x264's existing presets system to the end user). The same things largely apply to any random GUI out there.
In general, you will need to get familiar with command-line tools if you want to really get into the bleeding edge development aspects or really gain an understanding of more intricate encoding features, as frontends may be extremely rudimentary (and they can be pretty clumsy at times, making using the CLI faster and more agile). Windows doesn't quite lend itself too well to getting a handle on command-line usage, and it places more of a burden on the end user to know what to do with the PATH, which is pretty damn annoying. Even though I'd have reasons to use the command line every so often since I was a child, really getting accustomed to it and making it second nature only really started happening around 2005 or 2006. Dual-booting between Windows and Linux also really helped (it also really helps that many of those open source tools I mentioned get developed under Linux). I've also come out on the other side of that, since I've taken to building some of this stuff from source and occasionally contribute patches back upstream and file bug reports.
In a sense, if you look at AviSynth usage, it's not hard to then make the jump to using the command line, especially since you can script that too - .bat files or shell scripts are exactly the means to do that, and like an average AviSynth script, you can see it as one command per line, executed in sequence, to get your result (obviously, both AviSynth and shell scripting can get more advanced than that, with environment variables and for loops and all that fun stuff, but the simplest scripts of both types have similarities that make it look not as threatening).
Another thing to keep in mind is that, whether it's looking around on Doom9, or experimenting, or reading X or Y guide, the way to 'do it right' in this sense is to take it one step at a time. Understand or get a grasp on one part before moving onto the next, since looking for help on a particular subject makes it easier to get clear answers than trying to get it all right the first time. Trying to absorb everything all at once leads to burnout.