Improving Visual Quality and Compressibility
How to use this page:
This page is organized based on a number of things that you can do to fix up your video footage. Most of the things listed here are solutions to problems that can exist in your footage. If your footage doesn't exhibit a particular problem, then please don't toss on unnecessary filters! Filtering when none is necessary can hurt the quality of your video.
On this page, you will encounter images with some links below it. Clicking on these links will change the image, so you can easily see the effect of a particular filter.
Jump to a particular section:
Blocking / Ringing
Light/Invisible Noise (Important!)
Bringing it all together
Blocking and Ringing are types of artifacts that are caused by certain types of video compression. Both video from DVD, and just about and sort of video that you download off the internet will suffer from blocking and ringing. If the video is encoded well, then it may be so minor that you don't even notice it. On the other hand, it may be VERY apparent, as seen in the image below.
Blocking is easy to spot. It appears as sharp, blocky edges in your video, which shouldn't be there. It is particularly apparent in the red flowers shown here.
Ringing is a little more difficult to describe. It is also caused by video compression, and tends to be tougher to remove than blocking. Ringing is also referred to as Mosquito Noise. It manifests itself as specs around edges. It can be seen all throughout the image below. Both blocking and ringing tend to become worse on scenes that contain high amounts of motion, and the problems can subside or disappear on scenes with little or no motion.
MPEG2Source Postprocessing: The MPEG2Source command contains a parameter called cpu which allows it to deblock and dering your video. This is the best way to deblock/dering your video! Because it is part of the MPEG2Source command, this must be used when you load your d2v file, before you ever edit your amv. If you have already exported your amv as an AVI file, or if you weren't working from DVD source to begin with, then you will have to use one of the other two methods below. This parameter can be set to values from 0 to 6, with 0 being completely disabled, 4 being full deblocking, and 6 being full deblocking and deringing. You can try other values if you like, but 4 and 6 are by far the most useful. What makes this method so much better than the other methods is that the MPEG2Source filter can determine how much filtering should be applied to a particular frame and adjusts it's strength accordingly. This means that frames that contain heaving blocking and ringing will be filtered much more agressively than frames that have light or no blocking. To use it, simply insert the cpu parameter into the command, like so:
I usually recommend using CPU=4 for nice to medium quality DVDs, and CPU=6 for medium to ugly DVDs. Of course the decision is up to you. By clicking the links above, you can see how both cpu=4 and cpu=6 changed this frame. Using stronger settings can cause a loss of detail in your video, so be careful. You probably don't want to use cpu=6 on a really nice looking dvd, because its probably just overkill. If your dvd is really nice, I guess you can opt for no deblocking at all, but in that case, I would strongly suggest you at least do a small amount of smoothing (discussed further down this page).
BlindPP: If you can't use the method above, then its either this or Deblock. BlindPP works very much like the postprocessing in the MPEG2Source command, but with one important difference: While MPEG2Source can automatically adjust its strength for each frame, blindPP can't. This means you will need to specify a strength yourself, and it will filter each frame the same amount. It has the same CPU parameter that was in MPEG2Source. CPU=4 means full deblocking, and CPU=6 means full deblocking and full deringing. Additionally, there is a parameter called quant, which sets the strength it should filter at. The default is 2, and the strongest value is 31. An example usage with only deblocking, with a strength of 8 would look like this:
Thats all there is to it!
Deblock: This is a different type of deblocking based on a method used to deblock H.264 video. You simply need to set the parameter quant, which determines the strength of the deblocking, like so:
The default quant value is 25, and the maximum value is 51. While this filter can be quite effective, be careful because it does tend to remove detail and cause blurriness at stronger settings.
BlindDeHalo3: This is a function that can significantly reduce mosquito noise. In the second example above, you can see that it's so bad that cpu=6 barely touches it! In a case like this, you might want to try BlindDeHalo3. Look at how well it removes the noise around the edges. Also be sure to notice how badly the edges have become blurred now though. That is the cost of using this filter--it will blur sharp edges and detail. For this reason, it should only be used in extreme circumstances like the one shown above.
You think your video source looks nice and clean already? You got it from a nice DVD, and the quality looks absolutely perfect to you. Well, I'm here to tell you, it's NOT clean. In fact, it has a ton of noise in it. You might be thinking "What are you talking about? I don't see any noise in it!" But just because you can't see it, that doesn't mean its not there! As you will see below, even the cleanest sources are absolutely covered in noise!
Now, let me explain what's going on here. First, we have the original image, straight off of the DVD. Looks clean, right? Not quite. On the second image here, I have enhanced the noise that exists in the original image so you can see it better. I did not add this noise. It already exists in the image! Oh, and there is plenty of noise on the cat too, you just can't see it here. The entire frame is covered in this noise. But wait, that's not even the worst part! Although I can't show you here, all of this noise is changing on every single frame! Now when you go to encode this video, the encoder is going to see all these pixels changing, and it will try to encode them... and this results in larger filesizes, all because of this noise that we can't even see! But don't fret, there is still hope! If you look at the third image here, I have used Deen("w3d",3,3,5) on the video. I'll discuss this filter in more detail shortly, but it essentially smooths the video, reducing the noise. I doubt you can really tell any difference between the original frame and the one that I used Deen on, except some slight variations in the color. That's because I used very light settings on it, and these settings generally are too light to visibly hurt your video, but are still very effective at reducing all of this invisible noise! Granted, this isn't a very detailed scene, so you can't tell for sure that it's not hurting the video much, but just go try it out yourself, the difference is barely visible. The final image I have here is the frame that has been smoothed with deen, enhanced the same way the original frame was in order to make the hidden noise visible. Now, you can see that the noise has been "clumped together" into large, solid patches. Although you can't tell from just a single image, the changes between frames are also significantly reduced. In simple terms, this means that your video will now compress much better. Now, let's look at this Deen filter in greater detail, as well as some other filters that you might find useful.
Deen: Deen is a tried and true smoothing filter that I have been using for years now. It can perform 2 jobs: Spatial Smoothing and Temporal Smoothing. A spatial smoother smooths the image in each frame. A temporal smoother, on the other hand, works by removing noise across a series of frames. I strongly recommend that you use BOTH types of smoothing rather than just one. Lucky for us, Deen is capable of both of these!
The first parameter in deen is the mode. Deen offers a variety of modes, but I recommend using either "w2d" or "w3d". Some other modes are c2d/c3d and a2d/a3d. c2d and c3d are essentially limited versions of a2d and a3d, so they aren't particularly useful. The a2d and a3d modes aren't bad, but they are known to cause "banding" in sources. The w2d and w3d modes work very similar to a2d and a3d, but they can preserve gradients and don't cause banding. For this reason, I prefer the w2d/w3d modes above the others. The w2d mode does spatial smoothing only, and w3d does both spatial and temporal smoothing. I personally recommend w3d, but feel free to use the 2d mode if you prefer it. If you use the 2d mode though, you need to make sure to use a seperate temporal smoother along with it. The next parameter that deen takes is the radius. This sets the area that the filter looks at when it is processing each pixel of the image. Larger values are slower and cause it to smooth more. I think a value of 3 works pretty well for light filtering. With a 3d filtering mode you can set the radius as high as 4, and with a 2d mode you can go as high as 7. The next parameter is the luma threshold. This value sets how strongly you want to filter the luminance (lightness) channel of the video. Higher values smooth more, but smoothing too much can eliminate details from the image and give it a blurry or washed out look, so be careful! For light settings I recommend a 3 or 4 for this value, but please try a few values and judge it for yourself. The final parameter is the chroma threshold. This works just like the luma threshold, but it controls how much you want to smooth the chrominance (color) channels of the video. Generally, you can set this value a higher than the luma threshold without a visible difference. So to sum all of this up, you have the mode (enclosed in quotation marks), the radius, the luma threshold, and the chroma threshold. Here are a couple samples you might want to try:
or if you want to use a seperate temporal smoother, and you want to increase the strength of the spatial smoothing:
Pretty simple right? Just remember to be careful if you increase the thresholds, because you can lose details and make the video look washed out.
Undot: This is a pretty simple (and fast) filter that does a very simple type of spatial smoothing. It essentially clips each pixel to be within the range of its 8 neighbors. What this means, is it eliminates fine noise that is only 1 pixel in size. The effect of this filter is usually not even visible to the naked eye--far less so than the light Deen settings that are posted above. Some people say that they use this filter on every source. I can't say thats a bad idea. While it's usually not a good idea to just blindly toss filters on your video, the effect of this one is so minor that I seriously doubt it would cause any adverse effect. If you don't use any other kind of spatial smoothing, then this filter can give you a rather nice compressibility boost. If you are already using another smoother such as Deen though, you probably aren't going to see much difference, if any, from using Undot. Using undot is very simple:
That's it. Nothing to configure. Once again, I strongly recommend this filter if you aren't using any other smoother, like Deen. Even if you are using another smoother, this isn't likely to hurt anything, so toss it on anyways if you want to. Since this doesn't really create a visible effect, I'm not going to bother to post a screenshot.
TemporalCleaner: This is a rather simple and fast temporal smoother. I used to use this all the time, but recently started to prefer the temporal smoothing in Deen. This is still a good temporal smoother though, and you definately need to make sure that you do at least some temporal smoothing on your video. This filter takes two parameters. First, is the pixel lock threshold. If the change in a pixel from one frame to the next is less than this threshold, then that pixel is "locked" to the same value keeping it from changing any. Next is the pixel blend threshold. If the change in a pixel is greater than the pixel lock threshold, but smaller than this value, then the pixel in the new frame is blended with the pixel from the previous frame. The default values for this filter (4,6) are quite good, and are what I usually use.
One thing to watch out for, I have known this filter to cause ghosting, meaning that residue from a previous frame will remain on screen for 1 or more additional frames. This seems to be a rare occurance with this filter, but it can still happen. So when using this, just make sure to look over your video carefully to make sure that it hasn't been hurt any.
Fluxsmooth This is another temporal smoother that works quite well. I don't personally have a lot of experience with it, so I'll just copy what AbsoluteDestiny wrote in the previous version of this guide:
This filter has a spatial component but the spatial component is there to find out what a pixel should be averaged to. This is a very powerful filter and definitely a good addition to almost any filter chain. The default settings are FluxSmoothST(7,7) where the first 7 is the temporal threshold and the last 7 is the spatial threshold. Increasing these values can improve the smoothing but this filter needs very little tweaking.
So to sum all of this up: You want to perform both spatial and temporal smoothing on your video. Deen can function solely as a spatial smoother, or it can do both spatial and temporal smoothing. Undot is a very light spatial smoother. TemporalCleaner and Fluxsmooth are both temporal smoothers. Just make sure not to overdo it, because with too strong settings, these filters can all end up making your video look worse than it started out! But with reasonable settings, these filters can significantly help your final video quality. On that note, don't use all of these in the same script. Just make sure that you have some spatial smoothing going on, and that you have some temporal smoothing going on.
Alright, so what do you do if your video has a LOT of noise in it? You want to get rid of it, right? Well, hold on there, because it's not so easy. Well actually, thats a lie. Its actually very easy to get rid of noise. The trick is getting rid of noise WITHOUT also getting rid of real details in your video. No matter what denoising filter you try, this is the problem that will come up each time. The more agressively you try to remove noise, the more you are going to remove real details from your video, the more blurry you will make your video, and the more your video will get that "washed out" look. In some cases, you are probably better off just leaving the noise there than you are trying to get rid of it. In fact, some types of noise are actually put into the video on purpose sometimes. Worst case scenario, you will end up with a blurry, oversmoothed video that is STILL full of noise.
That's not to say that efforts put towards removing noise are pointless. In fact, sometimes noise can be removed very effectively. This all works on a case-to-case basis though, and for each source, you will need to experiment to find the best ways of reducing the noise. Now, there are a LOT of AVISynth filters out there that have been designed to reduce noise. I'm just going to give an overview of my favorite three, which I think should give you pretty good results under most circumstances.
If you would like to see more denoising filters in action, Scintilla has written up a nice comparison of some of them here.
Deen: I wrote an explaination of all the Deen settings above, so refer to that to figure out how to use it. To remove more noise, we basically just need to use stronger settings. Remember that you cant use a radius higher than 4 with the 3d modes, so if you want to use a higher radius you will need to fall back to a 2d mode, and then you should use a seperate temporal smoother along with it. Notice how deen kills some details when used with stronger settings. Especially notice on the detailed image aboe, how it totally smears all the texture in the ground! This is just ugly, so don't overdo it like this.
VagueDenoiser: This is an interesting filter that can be pretty effective at reducing noise. When used with the default settings, it has the ability to automatically detect how much noise is in the video, and tweak its settings. For this reason, I often like to call VagueDenoiser with the defaults, just to see how it looks. Sometimes it works out pretty well. As you can see in the detailed image above though, sometimes it doesn't work very well. In that image, it totally kills the detail in the ground, and it makes everything else look kinda "weird". If you want to tweak the strength of this filter manually, use the threshold parameter, like so:
This filter has a number of other settings which you can find in the filter's documentation. This is a spatial smoother only, so you should use some type of temporal smoothing along with it.
FRFun7: At the time of this writing, this is a brand new filter, but I'm already liking it a lot. It is quite effective at reducing noise while still retaining detail. It has three parameters, Lambda, T, and Tuv. The documentation describes these parameters as follows:
Lambda adjusts the power of the local denoising. The default value is 1.1
T limits the max luma denoising power for edges. The default value is 6.0
Tuv does the same as T, but for the chroma. The default value is 2.0
The lambda parameter seems to have the most effect on the noise removal, and I haven't had much success with changing the other two parameters. This filter does not do any temporal smoothing, so be sure to use a temporal smoother along with it.
Now that you have removed some of the noise from your video, do you think it might need a little sharpening? Most anime could benefit a bit from a small amount of sharpening to make the edges more defined. Of course, maybe you think it looks sharp enough already, and that may be so. Don't feel like you have to sharpen your video, just because you can. In a lot of cases however, it can really make your footage stand out, as long as you don't overdo it.
While conventional wisdom tells us that sharpening will make your video less compressable, this is actually only the case with one of the sharpeners that I'm going to show you. With some non-conventional sharpening methods that we will try, there will be very little effect on the compressibility, or it may even make your video MORE compressible!
vmToon: Those of you who read the previous version of this guide may remember two functions by the names of FastLineDarken and MFToon. Now imagine if those two functions got together and gave birth to a new function that combines their abilities into something far better, and you have vmToon!
vmToon basically does two different things. First, it can darken the lines in anime. While this doesn't really sharpen the image any, it can make the image appear sharper by making the edge outlines more defined. Next, it can do mild sharpening of the edges using a technique called super-sampled xsharpening.
I have supplied a number of different images above on the right side, which show off the effects of both the line darkening and the sharpening. First of all, we have vmToon called with its default settings:
As you can see, this causes the image to be much more well-defined. I don't know about you, but I really like the way it looks, except maybe the lines have been darkened too much.
The next image shows off the line darkening by itself, with the sharpening disabled. To turn off the sharpening, you simply add the sharpen=false parameter. It should be noted that the sharpening is really slow. If you only want line darkening, you can get a very nice speed boost by turning off the sharpening.
Finally, the last image shows the sharpening by itself, with no line darkening. Line darkening is controlled by the strength parameter, and the default value is 48, and can be set as high as 255. As you can see, the sharpening by itself is a very mild effect, and unfortunately it can't be increased. By default, sharpening strength, controlled by the xstren parameter, is at the maximum value of 255. If you wanted to use vmToon with half the line darkening strength, and half of the sharpening strength, you can call it like this:
Really though, I think the sharpening effect is so minor that there is generally no need to turn down the xstren. I do think the line darkening at the default settings may be too strong for some sources though, so toning it down with something like strength=24 is probably a good idea. Of course, this will vary from source to source though, so just try out some different settings and see what looks good to you. If you take a look at the default vmToon usage on the utena image on the left side, I think that one turned out very well with the default settings.
LimitedSharpenFaster: LimitedSharpenFaster is a great sharpening filter, but unfortunately it will hurt the compressibility of your video. Regardless, you may want to use it, because it can look very good. The main setting is the strength parameter. The default strength value is 100. If you want to increase the strength of the sharpening effect, you would do it like this:
This will do sharpening twice as strong, but of course at the expense of hurting compressibility. LimitedSharpenFaster is the most "traditional" of the three sharpeners mentioned here.
aWarpSharp: This is a very peculiar filter, and must be used very carefully. aWarpSharp works by warping the image in such a way that it thins lines. I feel like the default settings of this filter are way too strong for almost any purpose, because it simply warps the image way too much. When it warps the image too much, the effect can be very obvious, and it will make your footage just look strange. I personally only recommend this filter for sources that appear visibly blurry to you. Because this filter tends to make lines thinner, it can bring thick, blurry lines back closer to how they ought to look. It's probably not an altogether terrible idea to use it on the Utena source pictured above, but I would definately want to lower the strength. You can do this with the depth setting, which defaults to 16. I think a value of about half that is a good starting point.
Again, be very careful with this filter so you don't overdo it. One important thing I need to mention is that this filter is buggy and can cause green junk on the sides of your video. You can clearly see this in the image above. This appears to be related to the resolution of the video that this filter is applied to. So far, I have only encountered this problem on video that is 720x480, so if you have resized to a different resolution prior to this point, it will probably be fine.
What is rainbowing? Just take a look at the image below, and it should be painfully obvious. Rainbows manifest themselves as shimmering colors along edges. Not only is this really ugly, but the changing colors also hurt compressibility! Fortunately, we can usually remove rainbows quite effectively.
There exist a number of different filters and functions designed to combat rainbowing, but many of them are old and outdated or just don't work all that well. I have decided on just two methods of rainbow removal that I believe work very well for pretty much all situations.
DeRainbow: This is a function written a few years ago by Sh0dan which is excellant at removing rainbows. It works by creating a mask of the edges in the video, then smoothing the chroma channel of these edges both spatially and temporally. This works extremely well and with few side effects. You should note however, that it can case some slight discolorations around edges, especially if you use a high threshold. The default threshold for rainbow removal is 10, and you can increase or decrease this value as you deem necessary by calling this function like so:
BiFrost: This is a filter which operates purely temporally. This means that it shouldn't cause any discolorations around edges like DeRainbow might, and the author of this filter claims that it can actually recover detail. The downside of this filter is that it really only works on parts of the image that are static (not moving). Anime tends to have a lot of non-moving scenes though, so this can be quite effective most of the time. You should use BiFrost like this:
Sometimes though, you may not see much, if any difference at all! In those cases, you may need to use BiFrost before you IVTC/Deinterlace the source. Now if you have already edited your video, you obviously cant do this. But if you are preprocessing your footage, this is an option. You would put it directly after the MPEG2Source() line, and call it without the interlaced=false parameter.
Most footage wont need to have it's colors adjusted, because it probably already looks fine. In some cases though, especially with older anime, we can really make it look a lot nicer by adjusting the colors a little bit. With the old version of this guide, people were blindly just copying and pasting color adjustment settings for some reason. That's a really stupid thing to do. If you don't know how to properly adjust the colors (if they even need to be adjusted, which they usually don't), then you shouldn't even mess with them.
Anothing important thing that you need to note--and this applies to all filters, not just color adjustment filters--you must make sure your settings work on the entire video, not just one or two frames that you happened to look at.
Ok, so as you can see from the images, this original source was rather dull, and it had a very strong pink/red tint. Panda should not be pink! To tweak the colors, I started with ColorYUV(). As you should know, in a YUV colorspace, the color of an image is stored in the U and V channels, so by manipulating these two channels, we can change the overall color tint of the image.
The two main parameters of ColorYUV that you need to be aware of are gain_u and gain_v. By setting gain_u to a positive value, it will make your video more blue, and at negative values it makes your video more yellow. Positive values for gain_v, on the other hand, will make the video more red, and negative values more green.
For the particular source pictures above, I played around with these values until I arrived at what looked most correct. I ended up using a gain_v value of -25 in order to remove the strong red tint, then I used a gain_u value of 5 to give it just a slight bit more blue. The settings I arrived at were ColorYUV(gain_u=5, gain_v=-25).
After getting the color tint correct, I thought that the colors looked a little faded, so I needed to increase the saturation a good bit. For increasing saturation, you can use the Tweak() command. By inserting the "sat" parameter into it, we can adjust the saturation. 1.0 is the standard saturation, 2.0 means twice as saturated, 0 means completely desaturated. For this source, I used Tweak(sat=1.2). Most anime should have perfectly fine saturation, although some older titles tend to be faded and can benefit from a slight increse. Tweak also has some other parameters for things like brightness and contrast, but I strongly recommend against using these. Instead, you should use levels.
Finally, I thought the video was a bit dark, so it needed to be brightened up a little. For this, I used the Levels() command. There are probably hundreds of tutorials on the internet about using Levels, so I'm not even going to try and attempt to explain it here. If you don't know how levels works, just google for "levels tutorial" or "photoshop levels tutorial". I doubt that most sources would need any adjustments to the levels, but in order to use this filter it is suggested to use the Levels virtualdub filter in order to obtain the correct values to insert into the filter. More information on the avisynth levels filter can be found in the documentation.
There are hundreds of AVISynth filters available, all aimed at solving different problems. Here are a few other situations you might come across:
These filters are shown here with no settings. Please read the documentation on each filter to learn how to use it.
Dot Crawl: Dot crawl is one of the absolute worst problems that you can come across, but fortunately it's not all that common these days. It looks like crawling dots around edges and bright colors. It is notoriously hard to get rid of, but there is a filter called DeDot that is quite successful against it. DeDot() only works on NTSC sources BEFORE they have been deinterlaced.
Dirt/Spots/Scratches: Things like this tend to appear in older anime when they still photographed the animation cels instead of doing everything by computer. Small problems like these can be removed quite well with RemoveDirt().
Aliasing/Jagged Edges: This can be helped by a filter called TIsophote().
Shifted Chroma: Especially with TV captures, the chroma (UV) channels may be offset from the luma channel, usually in the horizontal direction. This causes the colors to bleed out of the lines on one side. With ChromaShift() you can shift these channels back into their proper location.
Edge Halos: Edge halos are white rings which appear around edges. They are caused by overly strong sharpening by traditional means (not really the types of sharpening listed in this guide). These halos can be removed by a function called DeHalo_aplha(), although it can also cause the edges to become blurred.
Even more filters can be found at Warpenterprises AVISynth Filter Collection.
Alright, so you have seen the difference that all of these filters can make, so now lets bring it all together to see how a properly filtered video should look! Here are three different samples showing how much a source can improve with proper filtering... and also how a source can be ruined by poor filtering.
Please note that I am not advocating that you use these particular scripts with these particular anime. I have not thoroughly tested these, and so a slightly different script may be optimal. Also, a lot of it comes down to personal preference, and there can be more than one way of achieving a similar effect.
Alright, so now we are done with cleaning up the video in AviSynth! That wasn't so hard, was it? Next, we just need to crop and resize!