Did you read any of the stickies at the top of this forum, including
Avisynth on Intel Macs?
Do not use Handbrake. The staff is well aware of my complaints regarding the lack of adaptive processing. However, at this point, they are still evaluating several filter combinations from doom9 to integrate by porting from Avisynth. In the mean time, we are not their demographic, and we never have been. Handbrake is designed for quick (and "dirty") encodes for people who want to watch their media on the go, usually on a laptop or portable device.
If you use Handbrake, at the very least, it will have trouble with the telecining. At the worst, you will all the awful problems you mention above. Do not use JES Deinterlacer, it is intended for live action footage shot by digital cameras and hard-encoded directly in an interlaced format. It is simply not designed to deal with the way anime is placed on DVDs.
I was the one who discovered how to get Avisynth working on Macs (at least, in as far as this community, I am sure others have discovered it independently, possibly at Doom9) so, it definitely exists. I am surprised you missed the sticky at the top of the page.
Quicktime, if you mean, MOV, is a container NOT a codec. Plenty of the compression codecs you would be used to in AVI are compatible with MOV, and the end result is no different.