Excessive Banding and Noise After Compressing

This forum is for questions and discussion of all the aspects of handling your footage. If you have questions about capturing/ripping footage, AviSynth, or compression/encoding/converting, look here.

Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 2:37 pm

Hey all,

I ripped all my footage, cleaned it up in Avisynth, and my footage looks great. I sent it to a few conventions and gave them either lossless copies or converted to MPEG-2 and even though they were large files, they looked good.

But when I compress my video to x264 MP4 and XviD AVI I see excessive banding and noise and discolorations in especially the dark places. I spent a long time cleaning it up before editing and have even tried gradfun2db() and various denoisers such as deen before I compress to XviD/x264 so I'm not sure what's causing it. Is it the way XviD and mp4 compresses footage?

Here are some example pictures. But please beware of SERIES SPOILERS for the first season of Clannad and Kanon 2006 which provide the best examples for darker areas:

Spoiler :
The before pictures are the Huffy and the after pictures are the compressed XviD.

Before:
Image
After:
Image

Before:
Image
After:
Image

Before:
Image

After:
Image

Before:
Image
After:
Image

Before:
Image
After:
Image


Maybe those scenes just pop out to me because I've looked at the HuffYUV version for so long. Some scenes are worse than others and some scenes even look practically as good as the original, but I'd love to make the darker scenes above look much better if I can. I am using VDub to convert to XviD: BVOPs set to 1, Quantizer ratio set to 1.00 to boost quality slightly, and on the quantization tab I alternate 1 and 31. For x264 I am using Zarx264GUI latest version: The preset is set to insane, the Quantizer is set all the way down to 16, and the mode is constant quality. I've tried it compressing for the ORG with and without gradfun2db and gradfun2dbmod() as well as with various denoisers. The best results I got were when I had gradfun2b() and that only. the x264 compresses better and looks a little better than the XviD, but not quite as good as I want it to be.

Anything I can try to do next, like any settings I can mess with or new filters I can try? Thanks. :)
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Qyot27 » Fri Jun 19, 2009 4:03 pm

I forget what the settings for Insane in Zarx264gui are concerning deblocking/in-loop, but if you set them to 1:1 (if they're not already), that might help reduce the noise in the H.264 copy.

With Xvid, several things that spring to mind: enable Trellis (on the Quantization tab), set the maximum I-frame interval to 50, rather than 300, and if possible, use 2-pass encoding. I typically have Xvid set to use 4000kbps, which in conjunction with the other settings, reaches saturation well below that level and virtually never gets close to it. The B-frames may also be contributing to it - you could try disabling them, if you don't mind the increase in filesize.


I had to crank my monitor's brightness all the way up to see some of the issues really clearly (the exception being the shot of Mai in the corridor; I could see the noise there). Then again, I prefer 'dark' displays in the first place (Contrast=100%, Brightness=60%, using the 9300 color setting - I'm also not using an LCD).
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Excessive Banding and Noise After Compressing

Postby Kariudo » Fri Jun 19, 2009 4:09 pm

I can't see anything wrong with your caps there...I think it may be just Vdub's output color depth setting causing the gradients in the preview window
ImageImage
Image
User avatar
Kariudo
Twilight prince
 
Joined: 15 Jul 2005
Location: Los taquitos unidos
Status: 1924 bots banned and counting!

Re: Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 4:36 pm

Wow, you guys are fast. :)

Qyot27 wrote:I forget what the settings for Insane in Zarx264gui are concerning deblocking/in-loop, but if you set them to 1:1 (if they're not already), that might help reduce the noise in the H.264 copy.

With Xvid, several things that spring to mind: enable Trellis (on the Quantization tab), set the maximum I-frame interval to 50, rather than 300, and if possible, use 2-pass encoding. I typically have Xvid set to use 4000kbps, which in conjunction with the other settings, reaches saturation well below that level and virtually never gets close to it. The B-frames may also be contributing to it - you could try disabling them, if you don't mind the increase in filesize.


I believe the settings for Zarx264gui are already set to 1:1. I checked the XviD settings and I changed I-frame interval to 50 from 300 and turned off B-frames. I am encoding the first pass now. I will let you know how it goes. :) I couldn't find the option to set XviD to use 4000kbps though.

Qyot27 wrote:I had to crank my monitor's brightness all the way up to see some of the issues really clearly (the exception being the shot of Mai in the corridor; I could see the noise there). Then again, I prefer 'dark' displays in the first place (Contrast=100%, Brightness=60%, using the 9300 color setting - I'm also not using an LCD).


My monitor is an LCD, so maybe some of the noise might be a bit more accentuated. The worst ones on my screen are the ones of Nagisa standing on the stage (the walls behind her show a lot of banding) and the picture of Yuuichi looking at Shiori (discoloration and banding on both their faces along with noise). I've never actually adjusted the brightness and other settings that much on my monitor. I'll try it out.

Kariudo wrote:I can't see anything wrong with your caps there...I think it may be just Vdub's output color depth setting causing the gradients in the preview window


You got me very happy there for a minute because I thought that might be the case, but I watched them in my video players (I use Windows Media Classic and VLC) and the issues are still there. Awww. :(

First pass is encoding now with the new settings and I might try out the second pass as well. I'll let you both know of the results. *Crosses fingers* Thanks. :) I just hope it isn't too big of a file size. I always have a bit of a file size problem with XviD.
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 5:39 pm

Well, as I expected the first pass of the video was too big for the org, so I did a second pass and set the target to about 85% and it's now under 100MB.

The result after the second pass is about the same size and quality as the first XviD. I'm not sure what I am doing wrong. I did discover this though:

Huffy:
Image

New XviD:
Image

See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. :) I'm willing to make a few clips to put up too, to see if it really is my monitor.
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Athena » Fri Jun 19, 2009 6:22 pm

Melichan923 wrote:See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. :) I'm willing to make a few clips to put up too, to see if it really is my monitor.


Wow. You really did lose the tatami weave. Is there any reason why XviD would be failing to account for the tatami weave? Can you post a screen cap of the first pass, I would be interested in seeing how we get from Picture 1 to Picture 2, because that is, indeed, a huge difference.
Image
User avatar
Athena
I ♥ the 80's
 
Joined: 02 Mar 2001
Location: Japan
Status: Sad Girl on Mac

Re: Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 6:44 pm

Kionon wrote:
Melichan923 wrote:See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. :) I'm willing to make a few clips to put up too, to see if it really is my monitor.


Wow. You really did lose the tatami weave. Is there any reason why XviD would be failing to account for the tatami weave? Can you post a screen cap of the first pass, I would be interested in seeing how we get from Picture 1 to Picture 2, because that is, indeed, a huge difference.


Sure I can. :) Here's a pic of the first pass:

Image

I don't see a very big difference between the first and second pass. A tad bit more detail is lost during the second pass, but I noticed it only after I looked a few times.
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Athena » Fri Jun 19, 2009 7:04 pm

Yep. That sucks. Also, here:

Image
Image
User avatar
Athena
I ♥ the 80's
 
Joined: 02 Mar 2001
Location: Japan
Status: Sad Girl on Mac

Re: Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 7:41 pm

Oh thanks, Kionon. :D Is that what Qyot27 meant by using 4000kbps? My target was a lot higher. It can't hurt trying that setting out though. If it helps, here are screenshots of most of my settings for the last encode:

Spoiler :
Image
Image
Image
Image
Image


If I'm lucky, maybe someone can point out something that I missed.
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Scintilla » Fri Jun 19, 2009 7:59 pm

Melichan923 wrote:Oh thanks, Kionon. :D Is that what Qyot27 meant by using 4000kbps? My target was a lot higher.

No it wasn't; "target size" != "target bitrate".
ImageImage
:pizza: :pizza: Image :pizza: :pizza:
User avatar
Scintilla
(for EXTREME)
 
Joined: 31 Mar 2003
Location: New Jersey
Status: Quo

Re: Excessive Banding and Noise After Compressing

Postby Melichan923 » Fri Jun 19, 2009 8:08 pm

Scintilla wrote:
Melichan923 wrote:Oh thanks, Kionon. :D Is that what Qyot27 meant by using 4000kbps? My target was a lot higher.

No it wasn't; "target size" != "target bitrate".


Ah, I see. Thank you.
Image
Image Image
User avatar
Melichan923
 
Joined: 30 May 2006
Location: New Hampshire

Re: Excessive Banding and Noise After Compressing

Postby Athena » Fri Jun 19, 2009 8:33 pm

To clarify, target size = how big the file will be, approximately, after encoding. Not at all the same.
Image
User avatar
Athena
I ♥ the 80's
 
Joined: 02 Mar 2001
Location: Japan
Status: Sad Girl on Mac

Re: Excessive Banding and Noise After Compressing

Postby Qyot27 » Fri Jun 19, 2009 11:29 pm

Aye, I should've clarified I meant Target bitrate, even though I thought that would have been obvious from the fact I appended kbps to the number. You just click on the Target size button to switch between methods. To know [roughly] what the average bitrate for the other ~94 MB encode was, you'd need to divide 93,350 by the length of the video (in seconds), and then multiply the resulting number by 8.

Not that the above calculation may matter at all when it comes down to using the 4000kbps target bitrate setting, as like I said, it can easily hit saturation below that and never spike that high. Also be aware that you might need to set the Quantization tab settings back down to 2's if the video is still way too huge. For instance, on the last video I released, that Quantization tab settings pretty much doubled the filesize between being set on 2 and being set on 1. Granted, I didn't release the video in Xvid in the end, but whatever.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Excessive Banding and Noise After Compressing

Postby Qyot27 » Fri Jun 19, 2009 11:47 pm

Also, because I didn't mention it earlier, you might also want to see about updating Xvid. Currently, it's at version 1.2.2: http://www.koepi.info/xvid.html

And now that I've taken a closer look at Zarx264gui (1.16, from http://amvhell.com/stuff/zarx264gui/), the 1:1 setting I was referring to were the Alpha & Beta sliders underneath Deblocking. The defaults are 0 for both. My suggestion was to set them both to positive 1.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Excessive Banding and Noise After Compressing

Postby Athena » Sat Jun 20, 2009 12:44 am

Qyot27 wrote:Aye, I should've clarified I meant Target bitrate, even though I thought that would have been obvious from the fact I appended kbps to the number.


I thought so too, which is why I added no explanation to my screencap.
Image
User avatar
Athena
I ♥ the 80's
 
Joined: 02 Mar 2001
Location: Japan
Status: Sad Girl on Mac

Next

Return to Footage Help

Who is online

Users browsing this forum: No registered users and 2 guests