Blocks on black?

This forum is for questions and discussion of all the aspects of handling your footage. If you have questions about capturing/ripping footage, AviSynth, or compression/encoding/converting, look here.

Blocks on black?

Postby noelle675 » Fri Jun 12, 2009 6:09 pm

Ok, I have no idea what this problem is called so blindly trying filters is getting me no where. Could any give me a name or a solution to get rid of the blocks on the hair? A lot of my blacks usually have this on them...

Image
Original

I've tried deblock, dfftest and temporal cleaner, but that doesn't seem to work quite right, or maybe I just don't know what luma/chroma variable I need. I'm still trying to figure out all the variables for everything.

I've gotten it so far to look like this but it's still not gone yet.

Image

Code: Select all
ConvertToYV12()
deen("a2d",4,9,11)
msharpen(threshold=15,strength=150)
deblock()

Thank you in advance for any advice. Yukimura's hair thanks you, too. :sweat:
User avatar
noelle675
 
Joined: 01 Feb 2006
Location: Colorado

Re: Blocks on black?

Postby Scintilla » Fri Jun 12, 2009 10:38 pm

The hair (that is the problem area, right?) in the second picture looks decent, and I wouldn't reasonably expect to be able to do much better. The only problem is that you're smearing details in the rest of the picture.

I don't really know if this is the kind of noise that a deblocker will help you with; I'm fairly sure that deblockers are supposed to help more with this kind of blocking.
You may want to try VagueDenoiser; I think the noise in my sample image is similar to your kind of noise, just in a lighter area. Also, VagueDenoiser shouldn't smear details as much as Deen.
ImageImage
:pizza: :pizza: Image :pizza: :pizza:
User avatar
Scintilla
(for EXTREME)
 
Joined: 31 Mar 2003
Location: New Jersey
Status: Quo

Re: Blocks on black?

Postby Qyot27 » Fri Jun 12, 2009 11:54 pm

I had to crank my monitor's brightness up to the maximum just to see the problem in that first picture. Without really knowing much beyond what could be causing it, I'm guessing gradiation of a sort, in which case I'd hit it with gradfun2db(), with a value of 2.0 - or maybe more, if it really seemed to need it.

One comment, though, is that those deen settings are way too high. There's very few sources I've encountered that are truly bad enough to warrant a 4,9,11 assault, no matter what mode you're using (I stick to w3d mode).
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Blocks on black?

Postby Qyot27 » Fri Jun 12, 2009 11:59 pm

Qyot27 wrote:Without really knowing much about what could be causing it

Fixed. Also, the link to gradfun2db:
http://forum.doom9.org/showthread.php?t=108681
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Blocks on black?

Postby Kariudo » Sat Jun 13, 2009 12:45 am

Just looks like quantization noise from compressing, a temporal smoother might help out some (that type of noise fluctuates with a relatively large change in color value)...fluxsmoothST() is worth a shot.
ImageImage
Image
User avatar
Kariudo
Twilight prince
 
Joined: 15 Jul 2005
Location: Los taquitos unidos
Status: 1924 bots banned and counting!

Re: Blocks on black?

Postby mirkosp » Sat Jun 13, 2009 4:22 am

Qyot27 wrote:
Qyot27 wrote:Without really knowing much about what could be causing it

Fixed. Also, the link to gradfun2db:
http://forum.doom9.org/showthread.php?t=108681

gradfun2db is good times, but be sure to grab the gradfun2dbmod OR paste the gradfunkmirror() code into an avsi. Its code is the following:
Code: Select all
Function gradfunkmirror( clip c, float "strength" )
{
   strength   =   default( strength, 1.2 )
   w         =   c.width()
   h         =   c.height()
   vflip      =   c.FlipVertical()
   hflip      =   c.FlipHorizontal()
   
   stackhorizontal( hflip.crop( w-16, 0, 16, h ).addborders( 0, 16, 0, 16 ),
   \   stackvertical( vflip.crop( 0, h-16, w, 16 ), c, vflip.crop( 0, 0, w, 16 ) ),
   \   hflip.crop( 0, 0, 16, h ).addborders( 0, 16, 0, 16 ) )
   gradfun2db( strength )
   crop( 16, 16, -16, -16 )

   Return last
}


I'm telling you to do so because the original gradfun2db has an issue that causes the 16 margin borders to NOT get debanded.
Image
User avatar
mirkosp
MODkip
 
Joined: 24 Apr 2006
Location: Gallarate (VA), Italy
Status: (」・ワ・)」(⊃・ワ・)⊃

Re: Blocks on black?

Postby noelle675 » Sat Jun 13, 2009 4:25 pm

Image
It worked! Thank you all so much. It looks pretty close to perfect. Can't really ask for much more than that.

Code: Select all
ConvertToYV12()
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)
VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0)
gradfun2db(3)
msharpen(threshold=15,strength=120)
Function GradFunkMirror( clip c, float "strength" )
{
#counters edge effects for gradfun2db()
#gradfun2db() requires width mod8 and height mod2
strength = default( strength, 2)
w = c.width()
h = c.height()
vflip = c.FlipVertical()
hflip = c.FlipHorizontal()

stackhorizontal( hflip.crop( w-16, 0, 16, h ).addborders( 0, 16, 0, 16 ),
\ stackvertical( vflip.crop( 0, h-16, w, 16 ), c, vflip.crop(0, 0, w, 16 ) ),
\ hflip.crop( 0, 0, 16, h ).addborders( 0, 16, 0, 16 ) )
gradfun2db( strength )
crop( 16, 16, -16, -16 )

Return last
}


Scintilla wrote:The hair (that is the problem area, right?) in the second picture looks decent, and I wouldn't reasonably expect to be able to do much better. The only problem is that you're smearing details in the rest of the picture.

I don't really know if this is the kind of noise that a deblocker will help you with; I'm fairly sure that deblockers are supposed to help more with this kind of blocking.
You may want to try VagueDenoiser; I think the noise in my sample image is similar to your kind of noise, just in a lighter area. Also, VagueDenoiser shouldn't smear details as much as Deen.


Oh, the background is actually permanently like that. I should have mentioned that earlier. That's perfect. I've changed the settings on Deen and added VagueDenoiser and it's working ALOT better with my other parts with a detailed backgrounds. Thank you very much!

Qyot27 wrote:I had to crank my monitor's brightness up to the maximum just to see the problem in that first picture. Without really knowing much beyond what could be causing it, I'm guessing gradiation of a sort, in which case I'd hit it with gradfun2db(), with a value of 2.0 - or maybe more, if it really seemed to need it.

One comment, though, is that those deen settings are way too high. There's very few sources I've encountered that are truly bad enough to warrant a 4,9,11 assault, no matter what mode you're using (I stick to w3d mode).


Ah, yes!! That's it. Gradients. I never thought that would be what was causing it. That fixed it perfectly. And as an added bonus, you also helped me fix the light coming off of the moon. I'll definitely have to remember that one.

Spoiler :
Image


oh are the settings too high? I have to admit I'm still trying to wrap my brain around the different values and how to actually use them right. I took another look at Scintilla's guide and she has the values labeled. That makes things a lot easier to for me. Thanks for pointing out that they were too high.


Kariudo wrote:Just looks like quantization noise from compressing, a temporal smoother might help out some (that type of noise fluctuates with a relatively large change in color value)...fluxsmoothST() is worth a shot.


Ah, so that's how that one works. It didn't work too well with this problem, but I'm sure it'll come in handy later on now that I actually know what it does. Thanks.

mirkosp wrote:I'm telling you to do so because the original gradfun2db has an issue that causes the 16 margin borders to NOT get debanded.


Perfect. Thank you!
User avatar
noelle675
 
Joined: 01 Feb 2006
Location: Colorado

Re: Blocks on black?

Postby Qyot27 » Sat Jun 13, 2009 5:02 pm

Well, it's good there was some improvement, but when I said that deen's settings were too high, I meant that using 4 for your radius, especially with high values for the other two, was overkill. I rarely ever go above deen("w3d",3,8,10) as an upper limit, and usually use it somewhere between 2,8,8 and 3,6,8 - 2,8,8 being more common by far. If I want temporal smoothing in there too, I extend it out to deen("w3d",2,8,8,8,8) - the thing to understand about filtering is that how much to do depends on how good or bad the source footage is.

Most shows produced the last 8 or 9 years are nowhere near bad enough to require heavy smoothing or even heavy sharpening (some of the U.S. domestic releases of the 2001-2004 period might, though). And that kind of stuff will harm the source even more than it might seem to help. It's like attacking the footage with a meat cleaver when what you really need is a scalpel.

If you're wanting to use gradfunkmirror, you don't call it like that. Copy and paste the function into a separate file, gradfunkmirror.avsi, and stick that into your plugins folder, then use it like you would any other filter. The way you have the script right now means you're not even using gradfunkmirror, because you aren't calling it.
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Blocks on black?

Postby Scintilla » Sat Jun 13, 2009 9:11 pm

noelle675 wrote:I took another look at Scintilla's guide and she has the values labeled.

:lol: :lol: I swear, this never gets old!
ImageImage
:pizza: :pizza: Image :pizza: :pizza:
User avatar
Scintilla
(for EXTREME)
 
Joined: 31 Mar 2003
Location: New Jersey
Status: Quo

Re: Blocks on black?

Postby noelle675 » Sat Jun 13, 2009 9:23 pm

Qyot27 wrote:Well, it's good there was some improvement, but when I said that deen's settings were too high, I meant that using 4 for your radius, especially with high values for the other two, was overkill. I rarely ever go above deen("w3d",3,8,10) as an upper limit, and usually use it somewhere between 2,8,8 and 3,6,8 - 2,8,8 being more common by far. If I want temporal smoothing in there too, I extend it out to deen("w3d",2,8,8,8,8) - the thing to understand about filtering is that how much to do depends on how good or bad the source footage is.

Most shows produced the last 8 or 9 years are nowhere near bad enough to require heavy smoothing or even heavy sharpening (some of the U.S. domestic releases of the 2001-2004 period might, though). And that kind of stuff will harm the source even more than it might seem to help. It's like attacking the footage with a meat cleaver when what you really need is a scalpel.

If you're wanting to use gradfunkmirror, you don't call it like that. Copy and paste the function into a separate file, gradfunkmirror.avsi, and stick that into your plugins folder, then use it like you would any other filter. The way you have the script right now means you're not even using gradfunkmirror, because you aren't calling it.


Oh. I had no idea that's how you use it. There we go. Here's what I have now.

Code: Select all
ConvertToYV12()
Deen(mode="a2d", rad=4, thrY=5, thrUV=9, min=0.5)
VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0)
msharpen(threshold=15,strength=120)
GradFunkMirror(3)


Well, it's good there was some improvement, but when I said that deen's settings were too high, I meant that using 4 for your radius, especially with high values for the other two, was overkill. I rarely ever go above deen("w3d",3,8,10) as an upper limit, and usually use it somewhere between 2,8,8 and 3,6,8 - 2,8,8 being more common by far. If I want temporal smoothing in there too, I extend it out to deen("w3d",2,8,8,8,8) - the thing to understand about filtering is that how much to do depends on how good or bad the source footage is.


I feel really bad asking, but if you wouldn't mind could you explain what "radius" and "threshold" are? I tried changing the radius and the two threshold values with the code I have now just to see what happens and I'm not really seeing much of a difference. I read bits of Scintilla guides again to try and get my head to osmosis the knowledge in, but the farthest my brain got was that thrY and thrUV deal with chroma and luma. So now I don't know what either of those are. The best guess is chroma has to do with color and luma has to do with light, but that's vague and not helpful. I did notice some minor noise like smoothing on another part of the clip that is got rid of.

Scintilla wrote:
noelle675 wrote:I took another look at Scintilla's guide and she has the values labeled.

:lol: :lol: I swear, this never gets old!


You mean I'm not the first idiot you've enlightened by naming values? :wink: And while you're helping I got to thank you for your wonderful guides. They're extremely helpful.
User avatar
noelle675
 
Joined: 01 Feb 2006
Location: Colorado

Re: Blocks on black?

Postby Scintilla » Sat Jun 13, 2009 9:32 pm

noelle675 wrote:I feel really bad asking, but if you wouldn't mind could you explain what "radius" and "threshold" are? I tried changing the radius and the two threshold values with the code I have now just to see what happens and I'm not really seeing much of a difference. I read bits of Scintilla guides again to try and get my head to osmosis the knowledge in, but the farthest my brain got was that thrY and thrUV deal with chroma and luma. So now I don't know what either of those are. The best guess is chroma has to do with color and luma has to do with light, but that's vague and not helpful. I did notice some minor noise like smoothing on another part of the clip that is got rid of.

Radius refers to how big of a surrounding area around each pixel gets included in the smoothing calculations. When radius = 1, only the eight pixels immediately around the target pixel are considered (and, if you're using a 3d mode, one pixel in the previous frame and one in the next frame), and so on.

Threshold refers to how different a pixel's value can be to that of the target pixel and still be considered for the smoothing calculations. When threshold = 1, only pixels one bit darker or brighter than the target (plus pixels that are exactly the same value) will be used.

noelle675 wrote:
Scintilla wrote:
noelle675 wrote:I took another look at Scintilla's guide and she has the values labeled.

:lol: :lol: I swear, this never gets old!

You mean I'm not the first idiot you've enlightened by naming values? :wink: And while you're helping I got to thank you for your wonderful guides. They're extremely helpful.

Thank you! But what I actually meant was, well, look at where I added emphasis in your quote. :P
ImageImage
:pizza: :pizza: Image :pizza: :pizza:
User avatar
Scintilla
(for EXTREME)
 
Joined: 31 Mar 2003
Location: New Jersey
Status: Quo

Re: Blocks on black?

Postby Qyot27 » Sat Jun 13, 2009 11:17 pm

Yeah, generally speaking, the higher the radius, the more area is allowed around any particular spot that gets smoothed. With higher thresholds, that content size of the area shoots up even more (as in, even though the radius is a certain amount, some pixels can still be excluded based on how different they are; higher thresholds make deen progressively ignore more and more of the differences, thereby including more pixels in the smoothing operations), and the higher and higher the values are, the more watercolor-ey the output will look (or, perhaps more aptly, like you submersed something already painted in watercolors).

As a test, you could have the same source, and do side-by-side comparisons of the frames using different smoothing values. That'd probably be more illuminating than simply trying to explain it could be. You can even do it in one script, and just switch out the values between opening the files and saving a screenshot, or use a single script to take a small portion, hit it with the differing smoother settings, and splice them all together sequentially.

Example 1: Opening each example separately (the # sign comments out the line, so AviSynth doesn't use it - which saves you from retyping the command in if you want to revert back to it; to reactivate the line, just delete the #)
Code: Select all
ConvertToYV12()
Deen(mode="a2d", rad=4, thrY=5, thrUV=9, min=0.5)
#Deen(mode="w2d", rad=3, thrY=5, thrUV=9, min=0.5)
#Deen(mode="c3d", rad=2, thrY=5, thrUV=9, min=0.5)
VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0)
msharpen(threshold=15,strength=120)
GradFunkMirror(3)


Example 2: splicing different examples together (for ease of use, I'm going to use variable descriptors and periods to separate the filter operations rather than putting them on new lines)
Code: Select all
vid = AVISource("yourvideo.avi").Trim(100,1000)
ex1 = vid.ConvertToYV12().Deen(mode="a2d", rad=4, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex2 = vid.ConvertToYV12().Deen(mode="w3d", rad=3, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex3 = vid.ConvertToYV12().Deen(mode="c3d", rad=2, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex1 ++ ex2 ++ ex3
My profile on MyAnimeList | Quasistatic Regret: yeah, yeah, I finally got a blog
User avatar
Qyot27
Surreptitious fluffy bunny
 
Joined: 30 Aug 2002
Location: St. Pete, FL
Status: Creepin' between the bullfrogs

Re: Blocks on black?

Postby Scintilla » Sun Jun 14, 2009 12:02 pm

Qyot27 wrote:Example 2: splicing different examples together (for ease of use, I'm going to use variable descriptors and periods to separate the filter operations rather than putting them on new lines)
Code: Select all
vid = AVISource("yourvideo.avi").Trim(100,1000)
ex1 = vid.ConvertToYV12().Deen(mode="a2d", rad=4, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex2 = vid.ConvertToYV12().Deen(mode="w3d", rad=3, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex3 = vid.ConvertToYV12().Deen(mode="c3d", rad=2, thrY=5, thrUV=9, min=0.5).VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0).msharpen(threshold=15,strength=120).GradFunkMirror(3)
ex1 ++ ex2 ++ ex3

For the last line, Interleave(ex1,ex2,ex3) might be even better. Then the differently filtered copies of each frame would be right next to each other, so you could just step through to see how each frame changes from one filter chain to the next.
ImageImage
:pizza: :pizza: Image :pizza: :pizza:
User avatar
Scintilla
(for EXTREME)
 
Joined: 31 Mar 2003
Location: New Jersey
Status: Quo

Re: Blocks on black?

Postby noelle675 » Sun Jun 14, 2009 3:20 pm

Scintilla wrote:Radius refers to how big of a surrounding area around each pixel gets included in the smoothing calculations. When radius = 1, only the eight pixels immediately around the target pixel are considered (and, if you're using a 3d mode, one pixel in the previous frame and one in the next frame), and so on.

Threshold refers to how different a pixel's value can be to that of the target pixel and still be considered for the smoothing calculations. When threshold = 1, only pixels one bit darker or brighter than the target (plus pixels that are exactly the same value) will be used.


Alright that makes a lot more sense now. Thank you so much.

Scintilla wrote:Thank you! But what I actually meant was, well, look at where I added emphasis in your quote. :P


OH! Hahaha. Sorry. :sweat:

Qyot27 wrote:Yeah, generally speaking, the higher the radius, the more area is allowed around any particular spot that gets smoothed. With higher thresholds, that content size of the area shoots up even more (as in, even though the radius is a certain amount, some pixels can still be excluded based on how different they are; higher thresholds make deen progressively ignore more and more of the differences, thereby including more pixels in the smoothing operations), and the higher and higher the values are, the more watercolor-ey the output will look (or, perhaps more aptly, like you submersed something already painted in watercolors).


I'm am definitely going to keep that in mind when I'm filtering. Now I should be able to really fine tune the smoothing without randomly changing numbers.

Qyot27 wrote:As a test, you could have the same source, and do side-by-side comparisons of the frames using different smoothing values. That'd probably be more illuminating than simply trying to explain it could be. You can even do it in one script, and just switch out the values between opening the files and saving a screenshot, or use a single script to take a small portion, hit it with the differing smoother settings, and splice them all together sequentially.


Much better now I can see what's happening. I've tried it out on the clip I'm using now and I'm able to see subtle differences. I'll have to go and get a clip that needs more filtering to see the difference better. The source I have now is pretty good already so it's a little hard for my untrained eye.

Thank you so much for your help!

Scintilla wrote:For the last line, Interleave(ex1,ex2,ex3) might be even better. Then the differently filtered copies of each frame would be right next to each other, so you could just step through to see how each frame changes from one filter chain to the next.


You never cease to amaze me with your knowledge. :up:
User avatar
noelle675
 
Joined: 01 Feb 2006
Location: Colorado


Return to Footage Help

Who is online

Users browsing this forum: No registered users and 0 guests