First of all, make sure you export your AMV from your editing program (Premiere/Vegas/Magix...) using a lossless codec like UTVideo, Lagarith, HuffYUV; you could also go Uncompressed (The file size will be even bigger than the other lossless options, but you know exactly what you are getting: raw data, untouched, unchanged). This will give you a gigantic master copy of your file with all the details so nothing is lost. This will also ensure compression artifacts are not introduced into your video (which would explain why file size sometimes goes up after re-encoding).
Exporting losslessly is important, because if you don't do this, you will be re-compressing an already-compressed file. This would be roughly equivalent to making a photocopy of a photocopy: bad idea.
So, once you have your lossless master copy, run it through zarx264gui.http://www.amvhell.com/stuff/zarx264gui/
You'll have to download and unzip it first.
Also note: with the latest version, it helps if you have Quicktime installed (Audio quality is better, and synch is more accurate.)