Artifacts in Anime/Toon Footage

[Anno-san]

Haali renderer might be slightly better. I can't really tell.

Some questions.

1. If haali eliminated the chroma error should there be no jagged edges at all? I still see some.

2. What would happen if I encoded the video anyway?

3. Am I only seeing the artifacts on MY screen, so that if others viewed the same video on their computers they WOULDN'T see the problem? Or is it an issue with the source file itself?

4.Renderers only affect how the video looks in MPC, correct? Or does a renderer affect how video will look when it is copied to the computer via DVDDecrypter?

[Mister Hatt]

Why do you keep saying you don't know what the problem is and that people are giving you contradictory advice? I've fully explained exactly what it is AND told you what to do about it. Every decent encoder I threw your screenshots to said exactly what I already have. The chroma subsampling error (there is no UPSAMPLING error at all and anyone who says it obviously doesn't know what upsample means) is part of the footage and your splitter choice will not fix it.

Renderers affect how your GFX card outputs each pixel. It has nothing to do with software for the most part. Rather it takes colours, applies display operations to figure out what each should be represented as in output, and then outputs it. It's done mostly on the hardware side by your video driver; however different renderers in your player will send slightly different information to your hardware.

[Anno-san]

@Mister Hatt

So the problem can be fixed with either Deblock_QED etc. or Edgecleaner or another masking filter? I'd prefer to solve the issue in as little lines as possible. If your script really was 4000 lines, I clearly can't do that as I have only been using AviSynth for a > a year.

[Anno-san]

Also, I have Edgecleaner v. 1.03 and when I try to run it it give the error "there is no function called aWarpsharp". Usually this error almost always means that the wrong version of said filter (in this case aWarpsharp) is being used. So what aWarpsharp do I need.

[Anno-san]

The other problem with the red jaggies/blocking on line edges is due to terrible chroma subsampling. The chroma issue can be fixed with EdgeCleaner I guess, although contrasharpening might be a better option.

Edgecleaner did not help much. what is contrasharpening and what filters can do this?

The blocking you need to find the golden settings of whatever your favourite deblocker is but I would consider dfttest masking or highly tweaked FFT3D as another option; I ended up using a dfttest mask on Cobra which had similar issues and it turned out pretty sugoi.

What were you saying about finding the chroma's frequency? Is that the sigma? Basically, will FFT3DFilter fix the jagged chroma edges on red objects or will EdgeCleaner? If FFT3D then what parameters?

[Mister Hatt]

The sigmas in frequency domain filters are the frequency sums, yes. DFTTest is a somewhat global sigma, while FFT3D lets you define smoothing strengths for various sigmas. The default in FFT3D is for larger blocking, about what you'd be throwing Deblock_QED_MT2 at. Sigma2 is for slightly smaller blocks, common in poor quantization. Sigma3 is probably the one you want; it's for stuff that is dirt or larger grain sized. Sigma4 is for mosquito noise. I guess something like FFT3DFilter(sigma=0.6, sigma2=1.2, sigma3=2, sigma4=0.4, bt=4, plane=4) should so it.

Contrasharpening is a method of smoothing detail as little as possible on edges, then masking it and applying a difference mask to the unfiltered video. It basically cleans up blocky edges. You can get it here. EdgeCleaner is probably giving you the error because you don't have awarpsharp loaded. You might also have a version it doesn't like, and tbh if that happened then you get what you deserve for being dumb and auto-loading standard filters which should NEVER be done.

I don't think you need Deblock_QED at all really. I'd just find a lowish sigma and dft nuke it or something. If you really care though, your best bet is probably a two-stage FFT3D followed by some EE or something just to clean it up a bit. I would avoid lsfmod but my reluctance to use that filter is based more on a personal dislike for it than any justifiable problems it has.

I just checked my script, it is 4091 lines long but I hope you don't think I wrote that manually. It was created with YATTA of course; how else would one make black magic so agreeable?

[Anno-san]

Here is a better sample of the footage.

[Mister Hatt]

Direct link please. Or someone else like mirko can take a look. This isn't really interesting enough for me to go out of my way and follow crappy-download-provider-x links.

[mirkosp]

I'm downloading it, but it has no extension. Is this an m2v or what? I'll reupload it to voodoo-powered with an extension, so Hatt won't have excuses to not take a look at it.

[Anno-san]

I'm downloading it, but it has no extension. Is this an m2v or what?

Source VOB.