tweaking the x264 settings to prevent banding

User avatar
l33tmeatwad
Joined: Wed Feb 16, 2005 3:22 pm
Location: Christiansburg, VA
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by l33tmeatwad » Thu Apr 25, 2013 1:03 pm

x265 wrote:Should i raise psy-rd to 0.9?
It's really hard to give you exact numbers, the best thing to do is take some of the scenes that were banding the worst mixed with ones that aren't showing banding so bad, and do some short encode tests. I would say try with it set at 1 for both and tweak based on some sample tests. That should provide a good balanced middle ground to work from. Raise aq for more banding reduction but reduce psy-rd to help with the noise.
Software & Guides: AMVpack | AMV 101 | AviSynth 101 | VapourSynth 101
PixelBlended Studios: Website | Twitter | YouTube

Mister Hatt
Joined: Tue Dec 25, 2007 8:26 am
Status: better than you
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by Mister Hatt » Fri Apr 26, 2013 11:19 pm

RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.

User avatar
l33tmeatwad
Joined: Wed Feb 16, 2005 3:22 pm
Location: Christiansburg, VA
Contact:
Org Profile

tweaking the x264 settings to prevent banding

Post by l33tmeatwad » Sat Apr 27, 2013 8:45 am

Mister Hatt wrote:RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.
The reason I suggested RDO was to help keep the overall picture complexity and give it more of a overall appearance of better quality. I feel like his qcomp setting is fine, it's rather balanced and I don't really see the need to tweak that. Although let's be honest, just tweaking aq-strength and qcomp is not a fix all for issues like banding, going over ALL of his setting and tweaking things for this particular source would be more ideal considering the complexity of the source material. Of the few tweaks he described above, there are more tweaks that could be done to improve the overall quality of the encode.

Additionally, if you feel like changing his qcomp setting could be helpful, it would be nice if you actually gave him a recommendation instead of just saying "qcomp is what you are looking for" as he may not know which way to tweak that setting to help. It's generally a good idea when helping to at least give some kind of recommendation for him to work with :sorcerer:
x265 wrote:Should i raise psy-rd to 0.9?
I'd recommend --psy-rdo 1.0:0.0 for initial testing on clips and tweak it down from there.
Software & Guides: AMVpack | AMV 101 | AviSynth 101 | VapourSynth 101
PixelBlended Studios: Website | Twitter | YouTube

Mister Hatt
Joined: Tue Dec 25, 2007 8:26 am
Status: better than you
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by Mister Hatt » Sat May 04, 2013 2:40 am

Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding. RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation. I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.

User avatar
l33tmeatwad
Joined: Wed Feb 16, 2005 3:22 pm
Location: Christiansburg, VA
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by l33tmeatwad » Sat May 04, 2013 10:50 am

Mister Hatt wrote:Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding.
While true for particular scenes, knowing the source and judging from the fact he's encoding the entire episode, I think it would hurt the overall quality dropping it as low as 0.6...
Mister Hatt wrote:RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation.
RDO relies on the image "complexity" not necessarily image detail (although detail can affect the complexity). Animation can sometimes bit a bit tricky to encoding software when it comes to this. To the human eye a scene may LOOK rather plain because of a static background, but with some sharply outlined character moving around in front of it, sometimes it makes the image seem a bit more complex for the encoder.

I remembered researching about this stuff when I was learning about x264 settings and how to properly use them way back in the day...referring to my old bookmarks I found this concerning RDO...but hey, maybe one of the developers could be wrong :uhoh:
Mister Hatt wrote:I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.
Ah, actual advice now, and good solid advice, you should have started off with that :sorcerer:
Software & Guides: AMVpack | AMV 101 | AviSynth 101 | VapourSynth 101
PixelBlended Studios: Website | Twitter | YouTube

Mister Hatt
Joined: Tue Dec 25, 2007 8:26 am
Status: better than you
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by Mister Hatt » Sat May 04, 2013 5:57 pm

D_S is not wrong in that link, but it is 5 years old and x264 changes frequently. RDO doesn't bias as hard now (unless you tell it to) and there are better things to do for rate optimisation and bit allocation than RDO now. FGO is a good one (if you use the patch) but only in HD anime. Otherwise, the qcomp, aq-strength, mbtree, and ip/pb settings generally do better. Deadzoning is worth looking at too in the event of grain (even dither) issues but that stuff is difficult to tweak unless you really know what's going on. The main reason I don't give hard numbers is I think people should read up on things instead of just waiting for an exact answer which is probably wrong.

User avatar
l33tmeatwad
Joined: Wed Feb 16, 2005 3:22 pm
Location: Christiansburg, VA
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by l33tmeatwad » Sat May 04, 2013 7:14 pm

Mister Hatt wrote:D_S is not wrong in that link, but it is 5 years old and x264 changes frequently. RDO doesn't bias as hard now (unless you tell it to) and there are better things to do for rate optimisation and bit allocation than RDO now.
The way I've always understood it was that the subpixel refinement level controlled how much the RDO influenced the encode. I don't recall seeing any documentation about any kind of change to this, if you wouldn't mind linking me to your sources I would like to read up on this so I can stay informed.
Mister Hatt wrote:FGO is a good one (if you use the patch) but only in HD anime. Otherwise, the qcomp, aq-strength, mbtree, and ip/pb settings generally do better. Deadzoning is worth looking at too in the event of grain (even dither) issues but that stuff is difficult to tweak unless you really know what's going on. The main reason I don't give hard numbers is I think people should read up on things instead of just waiting for an exact answer which is probably wrong.
I can't disagree there, tweaking those can help, although as I said before, you don't want to get too crazy with lowering the qcomp so much that it negatively impacts the rest of the video.
Mister Hatt wrote:The main reason I don't give hard numbers is I think people should read up on things instead of just waiting for an exact answer which is probably wrong.
It's not about giving hard numbers for them to use, it's about giving them help by saying "here is a good place to start, and you go up from here" etc. Anything short of that is not actually helpful and just serves to frustrate people by trying to work with incomplete answers.
Software & Guides: AMVpack | AMV 101 | AviSynth 101 | VapourSynth 101
PixelBlended Studios: Website | Twitter | YouTube

Mister Hatt
Joined: Tue Dec 25, 2007 8:26 am
Status: better than you
Contact:
Org Profile

Re: tweaking the x264 settings to prevent banding

Post by Mister Hatt » Mon May 06, 2013 8:35 pm

I can't think of any big changes to RDO, just small things over time like how CRF differs between revisions. I suppose you could read over the commit logs if you really wanted to. The main thing is, it's more neutral than blur-biased like it used to be. I think that's a good thing considering sources have gotten better over the years and micro-detail is more often than not grain-based rather than dither-based.

Locked

Return to “Video & Audio Help”