tweaking the x264 settings to prevent banding

This forum is for video and audio help and discussion.

tweaking the x264 settings to prevent banding

Postby x265 » Wed Apr 24, 2013 7:28 pm

Which of the x264 settings should i change in order to prevent banding?
“The only true wisdom is knowing that you know nothing.”
User avatar
x265
 
Joined: 25 Dec 2012
Location: India

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Wed Apr 24, 2013 7:45 pm

x265 wrote:Which of the x264 settings should i change in order to prevent banding?

You cannot prevent banding, it's a side effect of 8-bit YUV encoding. You can reduce it from happening quite so bad through tweaking your footage through filters and adjusting the colors, but there's no way to prevent it via encoder settings.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby x265 » Wed Apr 24, 2013 11:39 pm

I'm using the 10 bit version of x264.
“The only true wisdom is knowing that you know nothing.”
User avatar
x265
 
Joined: 25 Dec 2012
Location: India

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Wed Apr 24, 2013 11:59 pm

x265 wrote:I'm using the 10 bit version of x264.

10 bit won't remove all banding, it simply reduces the amount.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby BasharOfTheAges » Thu Apr 25, 2013 6:05 am

If It's not noticeable before encoding but is after, upping the VAQ could help.
User avatar
BasharOfTheAges
Just zis guy, you know?
 
Joined: 14 Sep 2004
Location: Merrimack, NH
Status: Extreeeeeeeeeme

Re: tweaking the x264 settings to prevent banding

Postby x265 » Thu Apr 25, 2013 8:01 am

is it the aq-strength option?
“The only true wisdom is knowing that you know nothing.”
User avatar
x265
 
Joined: 25 Dec 2012
Location: India

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Thu Apr 25, 2013 8:03 am

x265 wrote:is it the aq-strength option?

If you provided more information we might be able to help you tweak your settings and maybe suggesting some filters for your footage to reduce the banding, but we do need more information to be able to do that. You existing settings and screenshots of before & after for the current problem would be very helpful.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby x265 » Thu Apr 25, 2013 9:25 am

Source : Death Note R2J
DGSource("C:\Users\Abhijith Nair\Desktop\location\Episode.dgi")
Interp=NNEDI3(nns=4, qual=2, pscrn=4)
deint = tdeint(edeint=Interp, emask=tmm())
tfm(pp=6, micmatching=2, slow=2, clip2=deint)
smartfade(dgm=true)
tdecimate(mode=1)
Episode=trim(1918, 30688)
Preview=trim(32368, 32727)
Opening=avisource("opening.avs")
Ending=avisource("ending.avs")
AlignedSplice(Opening, Episode, Ending, Preview)
#Dehalo_Alpha_mt(ss=3.0)
Dither_Convert_8_to_16()
Dither_crop16(2,4,-2,0)
#F3kdb(dither_algo=2, input_depth=16, grainY=25, grainC=25, output_depth=10)
GradFun3(thr=0.4, smode=2, lsb_in=true, lsb=true)
#Dither_Quantize(bitdepth=10, reducerange=true, mode=0)
#Dither_Out()

Screenshots : (Without filtering)

Image
Image

x264 settings i'm planning to once i'm done correcting the avs script :

--qcomp 0.7 --aq-strength 0.8 --psy-rd 0.75:0.2 --no-dct-decimate --no-fast-pskip
“The only true wisdom is knowing that you know nothing.”
User avatar
x265
 
Joined: 25 Dec 2012
Location: India

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Thu Apr 25, 2013 11:35 am

First off let me say that Death Note is a tough source considering the color scheme and will be tough to keep the banding down. Increasing your aq-strength closer to 1 would help maintain detail in low detail areas and prevent more banding from occuring, but it's also going to require more bitrate if you want to maintain the same quality and avoid noise around the edges. I'd probably recommend raising psy-rd a little as well.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby x265 » Thu Apr 25, 2013 12:56 pm

Should i raise psy-rd to 0.9?
“The only true wisdom is knowing that you know nothing.”
User avatar
x265
 
Joined: 25 Dec 2012
Location: India

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Thu Apr 25, 2013 1:03 pm

x265 wrote:Should i raise psy-rd to 0.9?

It's really hard to give you exact numbers, the best thing to do is take some of the scenes that were banding the worst mixed with ones that aren't showing banding so bad, and do some short encode tests. I would say try with it set at 1 for both and tweak based on some sample tests. That should provide a good balanced middle ground to work from. Raise aq for more banding reduction but reduce psy-rd to help with the noise.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby Mister Hatt » Fri Apr 26, 2013 11:19 pm

RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.
Mister Hatt
 
Joined: 25 Dec 2007
Status: better than you

tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Sat Apr 27, 2013 8:45 am

Mister Hatt wrote:RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.

The reason I suggested RDO was to help keep the overall picture complexity and give it more of a overall appearance of better quality. I feel like his qcomp setting is fine, it's rather balanced and I don't really see the need to tweak that. Although let's be honest, just tweaking aq-strength and qcomp is not a fix all for issues like banding, going over ALL of his setting and tweaking things for this particular source would be more ideal considering the complexity of the source material. Of the few tweaks he described above, there are more tweaks that could be done to improve the overall quality of the encode.

Additionally, if you feel like changing his qcomp setting could be helpful, it would be nice if you actually gave him a recommendation instead of just saying "qcomp is what you are looking for" as he may not know which way to tweak that setting to help. It's generally a good idea when helping to at least give some kind of recommendation for him to work with :sorcerer:

x265 wrote:Should i raise psy-rd to 0.9?

I'd recommend --psy-rdo 1.0:0.0 for initial testing on clips and tweak it down from there.
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Re: tweaking the x264 settings to prevent banding

Postby Mister Hatt » Sat May 04, 2013 2:40 am

Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding. RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation. I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.
Mister Hatt
 
Joined: 25 Dec 2007
Status: better than you

Re: tweaking the x264 settings to prevent banding

Postby l33tmeatwad » Sat May 04, 2013 10:50 am

Mister Hatt wrote:Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding.
While true for particular scenes, knowing the source and judging from the fact he's encoding the entire episode, I think it would hurt the overall quality dropping it as low as 0.6...

Mister Hatt wrote:RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation.
RDO relies on the image "complexity" not necessarily image detail (although detail can affect the complexity). Animation can sometimes bit a bit tricky to encoding software when it comes to this. To the human eye a scene may LOOK rather plain because of a static background, but with some sharply outlined character moving around in front of it, sometimes it makes the image seem a bit more complex for the encoder.

I remembered researching about this stuff when I was learning about x264 settings and how to properly use them way back in the day...referring to my old bookmarks I found this concerning RDO...but hey, maybe one of the developers could be wrong :uhoh:

Mister Hatt wrote:I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.
Ah, actual advice now, and good solid advice, you should have started off with that :sorcerer:
Guides, Scripts, & Software: AMV 101 | AMVpack | AviSynth 101 | Anime Upscale
PixelBlended Studios: Website | Facebook | Tumblr | Twitter | YouTube
User avatar
l33tmeatwad
 
Joined: 16 Feb 2005
Location: Charleston, SC
Status: Trolling

Next

Return to Video & Audio Help

Who is online

Users browsing this forum: No registered users and 2 guests

cron