Fixing grain and deinterlacing? (Code Geass)

[Dext3r]

Also how do I get the HQDering filter to work? I've found the .dll's for SmoothD2 and NLMeansCL (given they're right in those threads); but for HQDering all there is is that wiki page with the source code?

HQDering is a script, so you make an .avs file, paste the code you can find in that page inside, save the .avs, rename the extension to .avsi and place the .avsi in the plugins folder of avisynth.

Also you don't have to wait the whole preview (if you mean File > Preview)... that's just slow. You can directly do File > Save Project. That just indexes as fast as your cpu allows, so it won't take too long. Take the screenshot after it's done.

Ok, I setup the HQDering script now; and I'm just waiting for my comp. to index the file.

[l33tmeatwad]

OT, but it might be useful in future: if you hold Alt while pressing Stamp, it'll only save the highlighted panel (so you could make a screenshot of just the information panel).

Anyway, that information panel looks indeed like a soft pulldown source. Which would mean force film should be the correct choice.
However you made the screenshot before it finished indexing the whole thing, so it would be better if you took a screenshot of after the whole source has been indexed. There might be some stray 60i bit amidst the soft pulldown footage, so that might be the cause.
If not, then I'd assume the green issue is either a mastering issue, or a MPEG-2 fuckup.

Oh christ, now you're just making me feel bad, I'm supposed to wait for that whole preview to finish? XD

I've NEVER waited for the DGIndex preview to finish, I only ever made sure I wasn't looking at the opening or ending (for obvious reasons since the opening and ending of the show can be running at different settings, like in Code Geass the opening is interlace).

Just to add on (I know the conversation moved a bit past this already), you definitely should wait until you preview the entire video before selecting Force Film because you never know when there could be mixed soft and hard telecine material. If you are coming across issues, handling all the interlacing with AviSynth is the safer choice. Also, as mentioned by mirkosp, there is always the possibility that it's a mastering or encoding problem (which is quite common for a lot of anime).

[Dext3r]

Ok so a few things.

Here's the settings after its entirely indexed (Forced Film)



Another slight problem I've noticed is apparently cropping or something strange going on in the edges of my source (particularly on the top, and whatever all that is going on the right side as well).



Are these problems I'm causing myself with my other unnecessary filters you already pointed out?

[Dext3r]

Also, for that second one, make sure you pull up the image in a different window so you can see what's going on the right side.

[l33tmeatwad]

That is normal for most videos, they often have "blank space" on some sides that you will need to crop off. Just keep in mind, when you crop, crop evenly to avoid distorting the image when you resize it. The amount cropped from the top & bottom added together should equal that of what is cropped from the left & right added together. The parts of the video you are seeing are usually outside of the "TV safe" area so you won't see it on most TV sets as it naturally cut it off.

[Dext3r]

ok, I'll play around with that a bit to get rid of those parts.

Still waiting to hear back from @mirkosp about the others though.

[mirkosp]

The information box you need to show us is the one of honor pulldown, as that is the reliable one.
That said, even with that forced film information, you can see it's not 100% film. 99.85% is very high though so, if at the end of the video you have something like staff credits for this particular dvd release or some studio logo, that could be what's offsetting it. Otherwise, perhaps one or two scenes might not be film, in which case I'll have to ask you whether the chroma issue you pointed out in the beginning happens all over the place or just in that specific scene.
I have to admit, though, that I find it likely that it is some sort of encoding problem either on master or on the DVD. If it's just there you might just find it easier to deal with by opening photoshop and manually fixing the frames, in case you need to use that scene in your AMV.

Now onto the other issue. As I explained, nlmeanscl uses opencl to run faster than it would on the cpu. Assuming you placed the cloo and avsfilternet .dll files in the plugins folder as well, then I'd guess that perhaps your GPU doesn't have OpenCL support (which is quite likely if you have just on-board graphics instead of a dedicated card). You could run the filter on the cpu but at that point it'd be quite slow. Since blocking is the only issue at hand, I think smoothd2 should be enough. I'd suggest properly setting the qtype value of smoothd2, btw, as that tremendously helps out the resulting efficiency of the filter.

[Dext3r]

The information box you need to show us is the one of honor pulldown, as that is the reliable one.
That said, even with that forced film information, you can see it's not 100% film. 99.85% is very high though so, if at the end of the video you have something like staff credits for this particular dvd release or some studio logo, that could be what's offsetting it. Otherwise, perhaps one or two scenes might not be film, in which case I'll have to ask you whether the chroma issue you pointed out in the beginning happens all over the place or just in that specific scene.
I have to admit, though, that I find it likely that it is some sort of encoding problem either on master or on the DVD. If it's just there you might just find it easier to deal with by opening photoshop and manually fixing the frames, in case you need to use that scene in your AMV.

Now onto the other issue. As I explained, nlmeanscl uses opencl to run faster than it would on the cpu. Assuming you placed the cloo and avsfilternet .dll files in the plugins folder as well, then I'd guess that perhaps your GPU doesn't have OpenCL support (which is quite likely if you have just on-board graphics instead of a dedicated card). You could run the filter on the cpu but at that point it'd be quite slow. Since blocking is the only issue at hand, I think smoothd2 should be enough. I'd suggest properly setting the qtype value of smoothd2, btw, as that tremendously helps out the resulting efficiency of the filter.

Oh, I didn't think about how since I use honor pulldown flags for the preview, I obviously need to use it to get the accurate results in the end too.

I have an Nvidia 480 GTX card, so I should definitely have a card that has that kind of support. So if I just use SmoothD2 and HQDering; how do I use SmoothD2 specifically then for that given it has so many parameters? Is it just something like SmoothD2(10)?

Or do I have to account for some of the others too?

[mirkosp]

Yeah your card should definitely be supported... I'll just blame it on opencl in avisynth being buggy. It's not too important anyway.
As for smoothd2, the documentation is pretty good.
As I said, the qtype parameter's pretty useful.

Qtype (1, 2, 3, 4): Quant Type. Selects the type of quant-dequant processing that is used in the filtering. They are ordered by their deblocking strength from least strong to strongest. h263 intra and h263 inter, Qtype’s 2 and 4, ignore the matrix argument and use their own built in matrix.

1. mpeg intra
2. h263 intra
3. mpeg inter
4. h263 inter
The intra in Qtypes 1 and 2 referes to the encoding done for I-Frames. The inter in Qtypes 3 and 4 refers to the encoding done for P-frames and B-Frames. See http://www.fh-friedberg.de/fachbereiche ... ginnzi.htm for a good introduction to mpeg coding. Different matrices are usually used when encoding I-Frames then when encoding P-Frames and B-Frames. All the information about how a particular frame was originally encoded, I-Frame, P-Frame, B-Frame, and Matrix value, has been lost by the time SmoothD2 is passed a frame to process, SmoothD2 does not do automatic intra-inter Qtype or matrix switching.

Generally you are supposed to manually set the qtype for the frame, but if you don't feel like spending all that effort, in the case of mpeg-2 DVDs, set it to 3 (which is mpeg inter).
That way it'll be much more effective in removing the blocking and mosquito noise at lower quant values, which is its filtering strength parameter. The default of 3 might not be enough for some of the most action intense scenes, but I really would suggest you to at least filter only those scenes at that point, since filtering clean scenes will just ruin them.
As for the other parameters, feel free to read the documentation, but I personally haven't found to be nearly as necessary and useful as these two, though I guess in some cases the matrix parameter could make wonders.

[Dext3r]

Yeah your card should definitely be supported... I'll just blame it on opencl in avisynth being buggy. It's not too important anyway.
As for smoothd2, the documentation is pretty good.
As I said, the qtype parameter's pretty useful.

Qtype (1, 2, 3, 4): Quant Type. Selects the type of quant-dequant processing that is used in the filtering. They are ordered by their deblocking strength from least strong to strongest. h263 intra and h263 inter, Qtype’s 2 and 4, ignore the matrix argument and use their own built in matrix.

1. mpeg intra
2. h263 intra
3. mpeg inter
4. h263 inter
The intra in Qtypes 1 and 2 referes to the encoding done for I-Frames. The inter in Qtypes 3 and 4 refers to the encoding done for P-frames and B-Frames. See http://www.fh-friedberg.de/fachbereiche ... ginnzi.htm for a good introduction to mpeg coding. Different matrices are usually used when encoding I-Frames then when encoding P-Frames and B-Frames. All the information about how a particular frame was originally encoded, I-Frame, P-Frame, B-Frame, and Matrix value, has been lost by the time SmoothD2 is passed a frame to process, SmoothD2 does not do automatic intra-inter Qtype or matrix switching.

Generally you are supposed to manually set the qtype for the frame, but if you don't feel like spending all that effort, in the case of mpeg-2 DVDs, set it to 3 (which is mpeg inter).
That way it'll be much more effective in removing the blocking and mosquito noise at lower quant values, which is its filtering strength parameter. The default of 3 might not be enough for some of the most action intense scenes, but I really would suggest you to at least filter only those scenes at that point, since filtering clean scenes will just ruin them.
As for the other parameters, feel free to read the documentation, but I personally haven't found to be nearly as necessary and useful as these two, though I guess in some cases the matrix parameter could make wonders.

Ok awesome, so how should I put my SmoothD2 command then? Because the first parameter is Clip or ClipZmask, so I can't just skip that one to get to the quantum?