"Average" Scores

[downwithpants]

As for the average vid deserving a 5, it depends on what a 5 means. If it's put to academic scales, a 5 means it fails. If you look at it as the midway point between having everything perfect and everything perfectly wrong, that's fine. You'd be saying it deserves a 5 because there is as much wrong as there is good. But you can't say it deserves a 5 because it's 'average' and isn't better than any other video. The only way you can do that is if every other video has as much wrong as there is good. I think the average video has at least a few more things right than the number of things that are wrong. So I can't give a video a 5 just because it doesn't seem to stand out over the average video. Until the average video here is one with as much wrong as there is right, I can't give 5s as the average - to me a 5 will *always* be the halfway point between perfect and absolutely bad (no good at all).

Okay, that's one way of looking at the scoring. However, even this method is very difficult and ambiguous. You can't simply count the number of things a person did well on his or her AMV against the bad things. Except for the very worst and very best videos, videos have a countless number of things done well and a countless number of things done wrong. I could not count every good and bad quality of the typical AMV and say "for every 7 good aspects, i found 3 bad aspects, so i'll give this a 7 out of 10." Especially on completely qualitative standards such as originality or reviewability (and i don't find Phade's quantitative standards of reviewability too reliable because I don't watch most of my favorite videos every week, yet i would never give them anything under a 7 for reviewability- and this is why i think the reviewability average is way below any other category's average).

Mathematically, the arithmetic average isn't even a very good tool to figure out what the middle of the pack is.

If you want to find the score that better denotes what a more middle-of-the-pack video is like, ask Phade to add a module to calculate and display the median score. Throw in a standard deviation calculation as well. Both data points would work better as a numerical indication of the "average" video, and are trivial to calculate.

The mean is skewed from the peak of the distribution curve if there are outliers (data points very far from the mean) and is skewed in the direction of the outliers. Since the average is about 8s, only the lowest scores (1s and 2s) could be considered outliers. In this case, the median is probably higher than the mean (probably mid-high 8's).

The median has nothing to do with the standard deviation, which is calculated off the mean. You could instead look for quartiles, deciles, or as koronoru mentioned, percentiles.

I'm as not concerned about the mean or median of the scores given on the .org as I am about the score reviewers consider an average video. I've already dismissed the high mean score as the result of a lot of people voting 10's on their favorite videos and deleting the crappy ones.

------------------------------------------------------------------------------

Thanks everyone who replied for your helpful comments:

I'll just tell you what I think of the rating system.
Existing anime music videos available to the public fall under a roughly normal distribution in regards to their skill, merit, and enjoyability. The rating system is a basically a coordinate system where we can align the average videos (in terms of skill, merit, and enjoyability) to a score of 5.5, the worst videos to a 1, and the best videos to a 10. The standard deviation is up to the reviewer, but I choose to put it somewhere around 2 (so about 70% videos fall between 3.5 and 7.5, and 95% fall between 1.5 and 9.5).

This sounds way too rigid and mathematical, and in fact it is. There is no way I can watch every AMV out there to determine which videos fall under which score of the population distribution. Additionally, opinions are subjective and qualitative, so there's no consistent guideline to scoring them. So my scoring is basically: if it's average in a category, give it a 5 or a 6, and only give it a 10 or a 1 if it really stands out from the pack. (I don't actually keep track of videos I've watched to fit the normal distribution.)

I think koronoru got the gist of my argument:

The org's opinions-scoring system would be most useful if everyone used percentile scoring (adjust things so that the bottom 10% of videos score 0.0 to 1.0, the next 10% score 1.0 to 2.0, and so on) but I don't think there's any hope of getting people to do that. It would mean that half of the videos you review must score less than 5.0 - and in the current environment, a 5.0 is considered to be a pretty strong "this is a bad video" message, reserved for really bad videos. Like it or not, I think we're stuck with something like the school grade curve, and we've even got the same problem that we see in schools - grade inflation. No individual can change things, either; if I started reviewing a lot of videos and giving scores less than 5.0 to exactly half of them, all that'd happen is I'd get a reputation as being that guy who shits on everyone's work.

Seeing as I am one of the few who consider a 5.5 to be an appropriate score for an average video, I guess I should just warn you that if I give you a 6, it doesn't mean I think it's bad. Of course, now no one is going to want me to score their videos.

[dwchang]

This is a guess, but the reason I think that schools give 70s as an average grade is because students traditionally have gotten 70% of the maximum number of points correct on tests, homework, etc. (Hence, the academic grade is a quantitative measure of the students academic ability, or at least they would be if there were no such thing as a teacher’s pet.) Because GPAs are essential indicators of academic ability when applying to college and grad school, schools must be able to show which students are proficient and which students are inept in their subjects. Since students usually get about 70% of the maximum points in a class (from tests, homework, etc.) schools assigned the 70-80% range as the average GPA of 2.0.

And since students usually get a 70%, thus the "curve" is set around there yes? That's how it works here. After 10,000+ scores, the "curve" has been set and thus that's the "stick" I use to judge. To me, it seems to fit everything I think (which Trythil already mentioned with reviewing good videos, etc.) and since I'm a nice guy, I figured I'd just use that curve as my grading scale. You can say that I'm ultimately helping this over-inflation, but at the same time as Nappy said, it's just how I grade and to me there's no real way to grade. It's up to you and ultimately this discussion is all a matter of opinion.

Your next point is that if the org’s scoring were based on the “entire world,” then as experienced editors, even our bad videos would be better than the videos that unexperienced editors would make. However, I’m guessing with high confidence that the “entire world” is not going to start making AMVs. Therefore, we don’t need to compare what we now consider average to what we would consider average if the entire world started making videos, because it’s not going to happen.

Yes, but when doing averages for things such as tests at school, the reason people get 70% of the questions right is because they're meant to be there (HS or whatever). They *should* be doing that well when compared to the rest of the world. As I said earlier, people in school are generally some delta apart from each other (not that big of a delta) and thus all of them will generally perform around that 70% average with distributions up and down (as you stated later). I will go into this later as koronoru made some good points.

I grade upper-level computer science courses. If I give you 70 out of 100, that means you probably know more about the course material than about 99.9% of the population (in this country) or 99.999% (in the entire world) or maybe about 40% (of your classmates) - it's not actually a great score (the expected class average is in the high seventies), but only a few people qualify to be in the class at all. Those three percentages are percentiles; none of them is equal to your mark of 70.

You're right. That's what I meant and was just using HS as the example since in HS (well at least mine) there is no curve. A 70% is a C and so on.

I guess college throws that out the window since I've had some senior level Circuit or Computer Engineer courses where a 20% was the average (I shit you not). I don't think I've had a test that had above a 70% average since my sophomore year.

So thanks for the correction.

What my main point is that after 10,000 opinions, we already know the averages and thus the curve to grade upon.

Now ultimately it's up to the reviewer to decide what they want to do, but myself (and others) look at that global average and grade accordingly. I guess you could say I don't wanna "rock the boat" or whatever, but again yours is yours and mine is mine.

Although, I do acknowledge what Trythil has stated and unfortunately, there isn't really a solution. People aren't going to want to rate things they don't like and thus the higher averages are here to stay. Well...maybe. I've (as well as others) talked to Phade about implementing the IMDB.com grading system that takes into account # of votes as well. He somewhat corrected this by switching the # of ops to 8 to get on the list.

[Nemoxs]

To me... an 8 is average.. I don't release a video unless I expect it to get ATLEAST an 8... if it wont get an 8.. fuck it.

[dwchang]

To me... an 8 is average.. I don't release a video unless I expect it to get ATLEAST an 8... if it wont get an 8.. fuck it.

So can I be an ass and give you a 7 everytime you release a video? That would in turn mean you'd never make a video

j/k

[Nemoxs]

To me... an 8 is average.. I don't release a video unless I expect it to get ATLEAST an 8... if it wont get an 8.. fuck it.

So can I be an ass and give you a 7 everytime you release a video? That would in turn mean you'd never make a video

j/k

Indeed... i'd end up on E true anime story...coughs...

"At the begining of his carrier he was showing potential.. untill someone gave him a 7 and it all fell apart.. he started drinking koolaid and eventually he got so water/sugar logged he exploaded"

[dwchang]

To me... an 8 is average.. I don't release a video unless I expect it to get ATLEAST an 8... if it wont get an 8.. fuck it.

So can I be an ass and give you a 7 everytime you release a video? That would in turn mean you'd never make a video

j/k

Indeed... i'd end up on E true anime story...coughs...

"At the begining of his carrier he was showing potential.. untill someone gave him a 7 and it all fell apart.. he started drinking koolaid and eventually he got so water/sugar logged he exploaded"

Y'know, you're just tempting me now

[fyrtenheimer]

Ut oh.

[downwithpants]

What my main point is that after 10,000 opinions, we already know the averages and thus the curve to grade upon.

That's the greatest danger I think the rating system faces. Right now, many experienced raters know that people usually rate off an average of 7. However, because so many people rate the best videos or hyped up, giving them 10s, the global average now sits at around 8. Some of the new reviewers might start thinking that they should give an average video an 8 when they look at the global averages, and this belief becomes increasingly popular. And at the same time, people will still mostly be giving lots of 10s for their favorite videos and their buddies videos, but now giving 8s (instead of 7s) on the average ones. Pretty soon the global average will move up to 9, and the cycle repeats until pretty much every video is squeezed between a 9 and a 10 and it is impossible to separate the good from the not so good. So we either need to make sure everybody sticks with a set average score (be it 5, 6, or 7), or Phade has to assign quantitative standards to all of the rating categories like he did to reviewability (which is going to be difficult to do well.)

[Saeth1]

I know it's hard to grasp, but if you're in HS or college, you're already smarter than a very large % of the world's population.

No, it just means you're lucky enough to be from a country where education is compulsory, free, and high-quality.

No, it means both. Yes, they are lucky to be required to go to school.


That doesn't change the fact they know more than people who are unindustrialized.


and that % isn't as large as you might think -

Japan, US, China, almost all of Europe, Canada, most of North Africa, and Australia boast about 97-99% literacy rate, which means about 89% of the population in those countries has gotten to High School level (thats an aproximation)

Now China makes up 1/6th of the world's population alone....i think yuo see where that leads

the other countries aren't that bad off either, for the most part.

[dwchang]

What my main point is that after 10,000 opinions, we already know the averages and thus the curve to grade upon.

That's the greatest danger I think the rating system faces. Right now, many experienced raters know that people usually rate off an average of 7. However, because so many people rate the best videos or hyped up, giving them 10s, the global average now sits at around 8. Some of the new reviewers might start thinking that they should give an average video an 8 when they look at the global averages, and this belief becomes increasingly popular. And at the same time, people will still mostly be giving lots of 10s for their favorite videos and their buddies videos, but now giving 8s (instead of 7s) on the average ones. Pretty soon the global average will move up to 9, and the cycle repeats until pretty much every video is squeezed between a 9 and a 10 and it is impossible to separate the good from the not so good. So we either need to make sure everybody sticks with a set average score (be it 5, 6, or 7), or Phade has to assign quantitative standards to all of the rating categories like he did to reviewability (which is going to be difficult to do well.)

That may be true, but in the year I've been a part of the community, the rating averages have been going down, not up. IIRC, it was around 8 - 8.5 when I joined.

At the same time, your argument does have a logical conclusion, but at the same time...it's just opinions.

As I said (as well as Trythil), it's up to the reviewer what they want to review and if it's only good videos, so be it. It's kind of pointless to argue about scores and so forth when ultimately it's up to us as individuals to do what we feel is right. I don't like the idea of making a universal score guideline to follow or implementing some system to "average it out." If the people say this...so be it.

I guess I'm being a defeatist