Looking for suggestions on End of Evangelion improvement

[wightwizard8]

I'm working on filtering and improving my rip of End of Evangelion (which is unfortunately the R1 version) in AviSynth, and I've got a fairly good-looking version by my standards, but my only problem is that the final result looks a little too "cel-shaded-ish", for lack of a proper adjective.

This is the script I'm using:

Code: Select all

Import("C:\Program Files\AviSynth 2.5\plugins\FastLineDarken.avs")
mpeg2source("X:\Source\VTS_01_1.d2v",cpu=4)
AMVDeint(mode=3)
SmoothD(quant=1, num_shift=4, adaptive_shift=0, zero_weight=1)
DeRainbow(10)
aWarpSharp()
LimitedSharpenFaster(strength=200)
FastLineDarken(strength=50,luma_cap=225,thinning=0)
Crop(8,0,-8,0).Spline36Resize(640,480).Crop(8,16,-14,-16).Spline36Resize(640,480)
ConvertToRGB32()

This is using filters that are found either in the AMVApp or in Scintilla's guide to postprocessing here: http://www.aquilinestudios.org/avsfilters/index.html.

Here's an example:
Original - http://yfrog.com/0kendofeva64627originalfrp
Edited - http://yfrog.com/msendofeva64627remasteredp

I'm concerned about the reduction in detail, especially in the trees, but I'm not sure how to keep the detail and still smooth out all the ugly.
If anyone has any suggestions on how to make it look better, please let me know. Thanks for the input!

[Mister Hatt]

Is it even interlaced as such, or is it telecined? I forget.

To be brutally honest: your original looks FAR better than your filtered. The filtered one has MASSIVE chroma blurring, is oversmoothed, straight lines are deformed hideously, and it's oversharpened. I have no idea why you double crop+resized on it either.

If I was filtering, ignoring interlacing status, I would start with some really minor derainbowing, although at this time I'm not sure what I would use to do so. I would then removegrain(1) it and possible run some dirt removal, otherwise I would do an fft3d contrasharpen followed by EE(). If it really needs it I guess some freq domain smoothing is in order but I doubt it. The aliasing you have should go away with proper treatment of the fields coding depending on how it's done. It might be better to derainbow post-noise filtering though, depends on which filter you use however.

[wightwizard8]

Yes, it's interlaced, at least the version I have is, and there's also a lot of black space around the video that I didn't want to have in the final product. So instead of doing the math, I just cropped on top of the first crop that was recommended in Zarxrax's "Guide to all things audio and video" (8 pixels from left and right and resize to 4:3).

Thanks for the critique, but your suggested solution has a lot of methods that I can't find, don't seem to do anything, or can't get to work. Could you point me to where I could find out how to get the FFT3D filters working, or tell me what specific filters you'd recommend, as far as dirt removal? And what is EE()?

This is my first time doing anything like this, so thanks for the help.

[Mister Hatt]

EE() is a filter that is part of Kintaro's Useless Filterscripts, you can google for it or I can upload it later. FFT3D filters are called FFT3DFilter or FFT3DGPU depending on whether you have access to DirectX9 or not. RemoveDirt is a pretty decent filter but you could try undot or using FFT3D for it, or even removegrain is capable of doing similar. ContraSharpening is kind of annoying and I don't think I could explain it here well enough.

And again, are you sure it's interlaced and not telecined? Telecine video is encoded interlaced but it isn't same-field matching like regular interlacing. I have never heard of End of Eva being interlaced.

[mirkosp]

EE() is a filter that is part of Kintaro's Useless Filterscripts, you can google for it or I can upload it later.

http://kaze-tenshu.pastebin.com/f91ndpGM

function EE(clip clip, int "str", string "cspace"){

str = default(str,80)
cspace = default(cspace,"useless")

yv12 = clip.isYV12() ? clip : clip.converttoyv12()
aa_clip = yv12.LanczosResize(width(clip)*2,height(clip)*2). \
UnsharpMask(strength=str,radius=2,threshold=2). \
LanczosResize(width(clip),height(clip))
maske = yv12.EdgeMask(7,7,5,5,"sobel").inflate()
semi = MaskedMerge(yv12,aa_clip,maske)
semi = overlay(yv12,semi,mode="darken")

aa_clip2 = semi.LanczosResize(width(clip)*2,height(clip)*2). \
TurnLeft().SangNom().TurnRight().SangNom(). \
LanczosResize(width(clip),height(clip))
maske2 = semi.EdgeMask(7,7,5,5,"sobel").inflate()
new = MaskedMerge(semi,aa_clip2,maske2)
new = overlay(semi,new,mode="darken").MergeChroma(yv12)

orig_cspace = clip.isRGB32() ? new.converttoRGB32() :
\ clip.isRGB24() ? new.converttoRGB24() :
\ clip.isYUY2() ? new.converttoYUY2() :
\ new

final = (cspace == "YV12") ? new :
\ (cspace == "YUY2") ? new.converttoYUY2() :
\ (cspace == "RGB24") ? new.converttoRGB24() :
\ (cspace == "RGB32") ? new.converttoRGB32() :
\ orig_cspace

return final }

It's still available on Kaze-Tenshu's pastebin!

As for FFT3D: http://forum.doom9.org/showthread.php?t=89941 get the exe of the version 0.8.2 and follow the easy installation steps. It's FFT3DGPU, 'cause I don't feel like writing up how to "install" fft3dfilter right now and 'cause if you have a proper gpu it's faster...

[Mister Hatt]

Incidentally I pastebinned it to Tenshu like a year ago on IRC, lol :3

Also incidentally, if you use a better compiled version of FFTW but the same lib version as what is packaged, you can run FFT3DFilter significantly faster. I might poke some people for actual specs. Compiling is good for you.

[wightwizard8]

Thanks for all the help!

But unfortunately, I've still got problems.

Here's my code now:

Code: Select all

Import("C:\Program Files\AviSynth 2.5\plugins\EE.avs")
mpeg2source("X:\Source\VTS_01_1.d2v",cpu=4)
Checkmate()
AMVDeint(mode=3)
EE()
DeRainbow()
FFT3DGPU()
Crop(8,0,-8,0).Spline36Resize(640,480).Crop(8,16,-14,-16).Spline36Resize(640,480)
ConvertToRGB32()

And here's what it gets me:
Original - http://img713.imageshack.us/img713/4885 ... iginal.png
Filtered - http://img706.imageshack.us/img706/6484 ... ltered.png

Now the main problems I've got are the fact Unit 02's red color seeps into everything around it, like you can see along her arms, around her head and shoulders, and around the handle of the other blade. There's also still a lot of noise in the flat spaces and around the edges, is there some sort of smoother that won't over-blur it? Is there some parameter in one of the filters I'm using that I should change? And what exactly does EE() do? I can tell it's doing something, but it's such a tiny change that I can't tell what it's doing.

[wightwizard8]

Okay, sorry, just figured out how to stop the bleeding - the default Rainbow() settings are too strong. But the rest of the problems still apply.

[mirkosp]

Don't call fft3dgpu like that. Try to call it like so, instead:

Code: Select all

strength = 0.5
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

Adjust the strength as needed. The default would be 2, but since apparently 0.8 is enough for rather noisy anime, I think 0.5 should be a good starting point for eva.
Also, EE should go after the denoising and derainbowing.

[Mister Hatt]

Crop/resize before filtering. You don't need checkmate(). Put EE() after everything else. The last line should be the ConvertToRGB one but tell it what matrix to use. Are you sure you need RGB32 and not RGB24? It will make avisynth use 30% more RAM than necessary, and you don't have an alpha channel to begin with anyway. You are having SEVERE chroma bleeding issues, find out why. You don't need AMVDeint, do it properly. You still haven't confirmed whether this is telecine or pure interlacing. Do so.