Bad banding and grain, plus blur after filters

[Taite]

So I'm currently making my script for my next amv.
So I have three ugly things going on: noise/grain, banding, and blur as an after effect of filters.

This is my script as it stands (that is, before the end of this post):

Code: Select all

AVISource("C:\Users\Alex!\Documents\FINAL2eyesonfire.avi")
ConvertToYV12()
LanczosResize(720, 400)
TTempSmooth
deen("a2d",3,4,6)
Toon()
Deblock(quant=25)
gradfun2db

FYI, the visual quality, besides all the teerrible banding, was perfectly fine when exported (lagarith) from vegas, except there were small blocks that needed to be removed, and while in the process of removing those blocks and the banding, everything got screwed up.


This is an image with none of the filters. The image is of a fade in (probably at 50% opacity) of Ryou from Clannad.:


Problems that I can see: Terrible banding, if that's the correct term. On other parts of the video (not necessarily this cap), there is some grain and blocks, so I now add filters to remove those. However...

An image with all of the filters I add (without gradfun2db):

Newly created problems: It's blurred now, and the banding seems to be worse because of it. However, those parts with the grain and blocks look smoother (all except for the parts where the video fades in from black, aka this.)

Now, I add gradfun2db....:

Problems: The banding has decreased, however terrible noise has replaced it. I know that that's basically the point of debanding, but it still looks awful. Still blurry.

I can now conclude that: The filters before gradfun2db didn't really help. They helped when the video was at 100% opacity (still not perfect though), but as it fades in it looks like crap. Gradfun2db did help the banding issue, but the noise is nearly unbearable.

And so now I try just gradfun2db, and the issues from the previous filters are all, of course, gone now. But there is still a lot of noise and banding, plus those blocks on the other parts of the video.

What I can conclude from all of this: I hate banding.

Here is an example of the blocks that I was trying to remove to begin with.


Notice his hair. There're blocks on other parts of the video as well.

This is what it looks like with all those filters (plus gradfun2db)

The blocking has decreased, but it still looks iffy. Kinda blurry now as well.

Now my script looks like this:

Code: Select all

AVISource("C:\Users\Alex!\Documents\FINAL2eyesonfire.avi")
ConvertToYV12()
LanczosResize(720, 400)

So I'm kind of looking for some advice as to how to remove those blocks. Also, how to effectively use gradfun2db, because it's the only filter I know of that debands. If you can point me to another that would work better, that'd be great. Anything that you can think of that might help my situation. (Of course, I'm still going to be working too.)

I realize I can't get my video to look perfect, but I know for darn sure it can look a lot better than this

You may or may not have to be too patient with me. I know what the filters I were using do, I just don't know how to use them as effectively . I've tried rearranging them as well, because I know that you can get different results by doing so (just like how you crop before you resize and such.) I know how to make a script and where to place downloaded filters on your computer and stuff, so nothing needs to be explained in baby-talk. I'm still a kid in this area though. I've been studying and searching for filters and trying to figure out why it's all weird, but I still need advice every now and then.

Thanks a lot for reading through this lengthy post o_O
Any recommendations are appreciated! (Even the bashes of how I could be so stupid to use x-filter. Making a fool out of myself is the best way to learn from my mistakes.)

-runs off to play around with the script for the next 2 hours-

[mirkosp]

What I can conclude from all of this: I hate banding.

What I can conclude from all this: quit using downloaded footage and start using the DVDs/Blu-rays (Clannad blu-rays will be out this month).
You wouldn't get the random blocks with straight from DVD footage... and bluray footage would be better than that too, even if just downscaled.
In any case, if the banding appears only in the fade, it's not worth using debanding while encoding (you shouldn't use gradfun2db for encoding anyway, but gradfun2dbmod). Just let people deband over playback like they already should. As for the random blocks, they might even get away with default deblock settings of x264 although you shouldn't rely on that. But in any case, it's not worth screwing the quality of the whole thing for a couple blocks. In a case like this, the crispiness of the image comes first, and you'd be blurring out a lot. Of course, the blur is due to the bad filters you are using there, but still it wouldn't be worth it even if you were to use better filters. I blame the lame plugins of the amvapp.

TL;DR: Keep it like it is and leave the debanding to the user for playback. You better have a couple blocks rather than blurring out the source, too. Start using actual DVDs/Blu-rays in the future, as those don't have random blocks.

PS: To be 100% honest I don't see the blocks you're referring to in the image you posted, tho.

[EvaFan]

I'd say try a temporal smoother for the blocks but it looks like your doing post processing (though I don't see much if any, hard to tell without full screening)... If you use a temporal smoother your fades will obviously be ruined to a degree. Your best bet is to just try some very low spatial smoothing (and if abosolutely necessary very VERY low temporal) and hope for the best otherwise like mirko said, you will just ruin it for the sake of some miniscule things.

I may be coming off as a one trick pony by posting this everywhere but seriously, fft3dgpu has never let me down... Versatile and fast if you have good GPUs (if you have a good processor but so-so GPUs then try the original fft3d plugin). I won't say leave it as is but if you insist on trying more with it before you release it then try fft3d but just remember less is more for smoothing. If the edges are still there after you get rid whatever your problem areas are then great. If not .

I don't really feel like explaining the plugin but you can read up on it, scintilla's got a good easy to understand explanation somewhere for it that I can't remember but you can PM him.
http://avisynth.org.ru/fft3dfilter/fft3dfilter.html

If you want to look for other options then here's a place you can start looking. No harm in trying and learning more options before just giving in and releasing it as is, I guess just make sure your happy with it first.
http://avisynth.org/warpenterprises/

[mirkosp]

scintilla's got a good easy to understand explanation somewhere for it that I can't remember

http://www.aquilinestudios.org/avsfilte ... p.html#fft

[Taite]

Well, when you have a 30gb HD you can't really get much on it, so I had a friend send me the episode, and it, whether it was a dvd rip or download, was already compressed so I could store it, so Idk :/ The visual quality of it is actually really superb, but I think you're right. I was probably overreacting at little things I saw. I'm not gunna have to worry about quality issues as much in the future though, because I'm getting a new comp this May, and trust me, I've more than enough dvds to keep me busy.
I definitely wasn't going to go with all those filters after seeing the blur, so I just did more research. I tried no filters like you suggested, and just compressed it quickly in zarx, and the banding doesn't look nearly as bad as it did in avsp. Shoulda tried that earlier So thanks! If I can't manage to find any filters as EvaFan had said, I'll know just to go with nothin' lol

I actually found fft3d while looking through Scintilla's guide, so thanks for bringing it up! I had downloaded it before, but completely forgot about it. I tried out just regular fft3d, because I don't have that great of GPUs, and it worked wonders! With a low sigma, it didn't over smooth anything, and everything still looks crisp. I think I'm gunna hang on to this filter. I've also looked around on the second link you posted with the list of filters (not that I need any more on this video, I don't want to over-smooth it), and have found some other useful ones that I can use in the future.

I think for now I'm not gunna need to be worrying about more filters for this vid. Thanks EvaFan and Mirkosp!

[Mister Hatt]

All your banding and blurring is caused by your crappy selection of filters. Grain is good for you, noise is not. Deen is an insanely bad filter, it kills all detail and blurs things. You have no sharpening going on but your use of Lanczos is going to introduce ringing which may be the noise you're seeing. The deblocking filter you're using is also overly aggressive. The banding is caused by oversmoothing and making bad gradients and then gradfun is adding too much dither to clean that up, while the quantizer isn't able to give the bitrate to save that and so it get's banded. Get rid of all your filtering.

I have no idea why you want to remove grain as it's not a defect. As for replacement filtering, removegrain(1), then fft3dfilter or fft3dgpu depending on your setup, settings (sigma=0.6,bt=4,plane=4) followed by EE(str=120) should get you a nice sharp image without any shit on it. If you still have blocking, turn up fft3d a bit, maybe sigma=1.2 I guess. For your resize, never use anything but spline36resize unless you have a really good reason (you don't). Just for the record, removegrain is a dirt removal filter, it's mostly good for cleaning shit in between thin lines.