Examination of removing opinion scoring

[dwchang]

*Stuff about why we need lists*

Instead of dumping the whole system and losing lots of valuable data, perhaps additional “fixes” can be applied to the existing system. Hopefully the result will be just as pleasing as when the Bayesian average was first applied.

Agreed. This is also why I feel we should try and improve the lists so they can be used effectively by the membership.

<ol><li>Slight negative discount for videos made with only crazy-popular anime
How this could help: With ultra-popular anime, there is a much higher percentage of pure fanboy scores. Having a slight discount for these anime would counteract the fanboy factor.

Interesting. I could certainly see it taking care of some of the problem, but my question would be, how is it going to be implemented? The most obvious (and easiest from a software perspective) approach would be to use the Top 30 Anime list and weigh according to relative rank. Perhaps # of videos using a certain show? At the same time, I don't agree with that.

For example, I like Kare Kano a lot and a lot of people do (I'm not sure if it's on the Top 30). Let's be honest, are there a bunch of Rabid Kare Kano fans causing this inflation? Probably not or at least I haven't seen it. Same with shows that are much more popular like Evangelion or Cowboy Bebop. Let's be even more honest...I think most of us know which shows are suffering from such a fanboy-craze just by looking at the lists and the influx of videos from particular shows.

This would then suggest there be a procedure implemented where administrators would need to continually update the list and weights based on their "feel of the community." Obviously years down the road, there will be a new craze and *hopefully* Naruto won't be as crazy.

I guess what I'm saying is that I support this, but not along a simple useage of the Top 30 list since it's not as simple as that and there are plenty of shows on there that don't really need to be discounted since they aren't being abused or inflated (imo). I could be wrong.

<li>Members who have been here longer (older accounts) get slightly more weight than newer accounts
How this could help: Members who have been around generally have more AMV experience. These members’ opinions should help counteract newbies who think the first few AMVs they’ve ever seen are the greatest things in the AMV world. This would also combat any “sock puppet” problem.

I like this. Just like silver_moon, I too remember rating videos really high when I became a member. I later corrected this (much to the chagrin of some of the creators), but I imagine most do not. Now perhaps this will create a stigma about newbies and further divide things, but to be honest, it makes sense given the statistics and purely from a logical standpoint (experience probably means you get better at something).

At the same time, how you present this to the members is quite important and somewhat political. You don't wanna give the impression that the .org does not welcome the opinion of new folk. I realize you don't feel that way, but I imagine some people will take it that way and hence the warning.

<li>A members first opinions have less weight than newer opinions
How this could help: As a member becomes more familiar with videos, their opinion scores are likely to be more true. This would also combat any “sock puppet” problem.

Excellent. This would surely combat the sock puppet and dupe account problems. Not 100%, but every bit helps.

<li>Add an admin/creator-given usefulness score for each opinion
How this could help: Video creators would be able to identify members who do a particularly good job of giving opinions. These members would then have a higher weight for their opinions since they appear to be more thought out than normal or just particularly biased.

I've always wanted something like this. Lke Zarxrax, I rarely receive a *really* good and detailed review, but when I get them, I really appreciate it and am actually glad we do have a review system (even with all the abuse). This way the creator can also say "thank you" to their reviewers with a little review of their own .

At the same time, the concern with creators abusing this is quite high. I could easily give higher weight to only the people who give me good scores and low scores to those who give me bad scores. This would in term, mathematically, make their bad scores affect my averages less and the good scores would affect my average (positively) more. In fact, that's quite easy for creators to do imo.

See a bad review of all 4's, give it a low rating back, voila, it doesn't average in as much. See a review of all 10's, give it a high review score back and bam, it moves your averages up a bunch. Certainly on the reviewer side, this is more difficult, but perhaps it will also discourage people to leave reviews since creators could potentially abuse their opinions. I for one would probably leave less detailed reviews if I knew my "review score" was going to go down because I was critical, yet helpful.

I do like this idea, but I still do see the potential for abuse.

<li>Add a score for “I am a big fan of this anime” to “I’m not really a big fan of this anime”
How this could help: With the upcoming “anime I’ve watched” section, this score would be used as an inverse adjustment factor. If you are not a fan of the anime but score an AMV highly with that anime in it, the score gets a boost since the creator must have done something really good for you to give it a higher score.

I first did not agree with this since I thought it might not bring any benefit, but then I realized I saw an FY video recently that I enjoyed. I *hate* Fushigi Yuugi. If I had given it a review, I imagine I'd have shown that appreciation in making me like a video to a show I hate. This makes sense!

<li>Create an AMV Hall Of Fame
How this could help: By creating a so-called “hall of fame”, video creators could not directly affect any particular system but instead would truly have to create a good AMV in order to be on the list. The list would be voted upon by a set of trusted/competent members. Video age and other criteria must be met before being a potential candidate. More to follow if implemented...

I think this is the only idea I am against. Now although I agree with the *original intent* of our lists, we all know that some people see it as sort of a status within the community and "making it." Let's be honest, some people view it a lot more highly than others.

If we created yet another list of "prestige" in our community, I think we are really just introducing another factor for said people to grade themselves upon and worse, another reason to abuse the system (i.e. to get on this list).

Given, with a trusted group of people choosing, this is more difficult, but is it really right for a small group of people to decide which creators/videos go into such a list? This list should be encompass all the members and their tastes. In fact, all the lists should. What if this group has a high bias against DBZ vids (highly probable?)? Would that then mean it'd be that much more difficult for a video or creator of such vids to get in? Of course.

To be honest, I imagine the people on this "panel" will probably do a good job and my real gripe is with the generation of yet another list for people to compare themselves to, strive for and abuse. And I'll also be frank, the name alone seems a bit pretentious and would probably lead to quite an outcry from parts of the community.

With each of these score adjustment factors, they will have to be tweaked so that the final list appears to be the most correct. Certain factors can be judged by using certain benchmark videos (ex. Scott Melzer’s “This Is DBZ Life” video uses only DBZ footage (ultra-popular anime) but it is actually a very good video).

Agreed. However, I think given the Baysian system, the fact Scott has so many reviews kinda counter-acts this in that the Baysian system rewards the fact so many people agree on how good it is. I think it *may* fall a bit due to a discount, but I guarantee it'll still be near the top quarter of the list.

Hopefully by implementing some or all of these adjustment factors, abuse can be mitigated and the Top 10% List can be salvaged and more genuinely reflect the status of AMVs.

Agreed. I'm quite happy to see how quickly you responded to this and how open you are to changing things. Sadly, I do agree that things will always be open to abuse and I wouldn't be surprised if 2 years down the road we have this discussion again, but well...step-by-step I guess.

[Zarxrax]

Slight negative discount for videos made with only crazy-popular anime
How this could help: With ultra-popular anime, there is a much higher percentage of pure fanboy scores. Having a slight discount for these anime would counteract the fanboy factor.
Possible abuse: There is not much an individual can do to affect the popularity of anime in AMVs.
Abuse countermeasures: Since there is very little an individual can do to affect this factor, countermeasures are not needed.

What about when people start adding extra source into the listing to avoid this filter?

<li>Add an admin/creator-given usefulness score for each opinion
How this could help: Video creators would be able to identify members who do a particularly good job of giving opinions. These members would then have a higher weight for their opinions since they appear to be more thought out than normal or just particularly biased.

I've always wanted something like this. Lke Zarxrax, I rarely receive a *really* good and detailed review, but when I get them, I really appreciate it and am actually glad we do have a review system (even with all the abuse). This way the creator can also say "thank you" to their reviewers with a little review of their own .

At the same time, the concern with creators abusing this is quite high. I could easily give higher weight to only the people who give me good scores and low scores to those who give me bad scores. This would in term, mathematically, make their bad scores affect my averages less and the good scores would affect my average (positively) more. In fact, that's quite easy for creators to do imo.

See a bad review of all 4's, give it a low rating back, voila, it doesn't average in as much. See a review of all 10's, give it a high review score back and bam, it moves your averages up a bunch. Certainly on the reviewer side, this is more difficult, but perhaps it will also discourage people to leave reviews since creators could potentially abuse their opinions. I for one would probably leave less detailed reviews if I knew my "review score" was going to go down because I was critical, yet helpful.

I do like this idea, but I still do see the potential for abuse.

I agree, this is too easy to be abused. But more importantly, it adds yet ANOTHER layer of complexity onto people. People don't want to have to bother rating the reviews that they receive. This is extra work. People are lazy. But just for a minute think about what this is saying. A proposed plan to review the reviewers. WTF? When's the last time you saw someone rate Ebert's latest movie ratings? It just doesnt work this way! It's not natural! What's next, Reviewing the review of the reviewers? This web site is already somewhat drowning in its own complexity. Features shouldn't add more complexity, they should strive towards simplicity.

<li>Add a score for “I am a big fan of this anime” to “I’m not really a big fan of this anime”
How this could help: With the upcoming “anime I’ve watched” section, this score would be used as an inverse adjustment factor. If you are not a fan of the anime but score an AMV highly with that anime in it, the score gets a boost since the creator must have done something really good for you to give it a higher score.

Another layer of complexity. Plus it intends to rely upon yet another feature, "anime I've watched"? How many people could honestly be expected to fill out a list of every anime that they have watched? I've seen hundreds. I don't even remember some of the ones that I've watched. Plus, why does whether or not you are a fan of the anime play into the score? You are assuming that I will automatically like videos better if I'm a fan of the anime. Perhaps I am in fact more critical of videos that use anime that I like? or maybe I just have never even seen the anime and couldn't care one way or the other. Plus, what about videos that use various anime? This idea completely falls apart right there.

<li>Create an AMV Hall Of Fame
How this could help: By creating a so-called “hall of fame”, video creators could not directly affect any particular system but instead would truly have to create a good AMV in order to be on the list. The list would be voted upon by a set of trusted/competent members. Video age and other criteria must be met before being a potential candidate. More to follow if implemented...

I think this is the only idea I am against. Now although I agree with the *original intent* of our lists, we all know that some people see it as sort of a status within the community and "making it." Let's be honest, some people view it a lot more highly than others.

If we created yet another list of "prestige" in our community, I think we are really just introducing another factor for said people to grade themselves upon and worse, another reason to abuse the system (i.e. to get on this list).

Given, with a trusted group of people choosing, this is more difficult, but is it really right for a small group of people to decide which creators/videos go into such a list? This list should be encompass all the members and their tastes. In fact, all the lists should. What if this group has a high bias against DBZ vids (highly probable?)? Would that then mean it'd be that much more difficult for a video or creator of such vids to get in? Of course.

To be honest, I imagine the people on this "panel" will probably do a good job and my real gripe is with the generation of yet another list for people to compare themselves to, strive for and abuse. And I'll also be frank, the name alone seems a bit pretentious and would probably lead to quite an outcry from parts of the community.

I don't really see your problem with a list composed by a select group of people. Most cons out there do it. Plus, its just an additional list, its not taking away the public's right to voice their opinions.
for this idea.

[Zarxrax]

By the way, this is nearly off-topic, but if you want an "anime I've watched" list, Phade, it could be implemented right now with absolutely no work aside from changing a bit of the html pages. Just change "my favorite anime" into "anime I've watched", and tell people to order it according to their favorites. Instant "anime i've watched" list.

[downwithpants]

what's more to be concerned about is that selecting relative to other scores on the member's other videos will tend to promote only one video to the top list per creator (the highest outlier). Right now, 6 of AD's 25 premiered videos are on the top 10%. Under the new system, maybe one or two, depending on the distance between mean and outlier for everybody else under consideration. Normalizing scores to the creator's own standard would recognize more people, but if it's about the videos, the top list should recognize the most worthy videos on the site regardless of who made them.

i think derobert meant normalize to the opinion scores the member has given, not received. i think this would be a good way to make some sense out of opinion scores because one person's 8 might be another person's 5.

however, it would be less effective in gauging the true responses of creators who have given fewer scores (how do we know if a person who has given a single all-8s opinion thinks an 8 is average or above average?). on the other hand, this could provide a means of incentive to score more opinions (i.e. after leaving one opinion, your rating given will have no effect because it will be automatically normalized to the average, however as you give more and more opinions your standard dev would shrink and shrink, and your extreme ratings would be more and more effective [curved less towards the mean].)

<li>Slight negative discount for videos made with only crazy-popular anime

i'm opposed to this. one reason is i don't think videos should be punished just based on the anime they use. second there are op givers that go, "omg you used such-and-such anime? it's been used 3 gajillion times you get 1 for originality" to balance out the fanboys. third is what dwchang noted: it'd be really tough to determine what's being subjected to fanboydom.

<li>Members who have been here longer (older accounts) get slightly more weight than newer accounts

my only concern is this could produce a degree of elitism.

<li>A members first opinions have less weight than newer opinions

i support this. i just want to mention this is possible through derobert's idea of normalizing scores over individual opinion giver's distributions.

<li>Add an admin/creator-given usefulness score for each opinion

this was mentioned before as an alternative to the current usefulness ratings, which get abused from copying and pasting large chunks of text unrelated to the opinion. the problem people saw was that, as dwchang mentioned, creators are prone to favor high scoring op givers over low scoring op givers. furthermore, creators aren't really the right ones to judge how others perceive their videos, otherwise there wouldn't be a need for feedback. my suggestion was for other viewers or other opinion givers to rate opinions given to the same video on a amazon.com-esque "was this review helpful to you? yes/no" basis. of course, this means more codework for site programmers.

<li>Add a score for “I am a big fan of this anime” to “I’m not really a big fan of this anime”

another potential abuse would be if people start rating even higher on preferred animes knowing their scores will later be reduced.

<li>Create an AMV Hall Of Fame

biggest problem i see is selection of the "set of trusted/competent members" and associated elitism whether intentional or not. people will be complaining why this member got selected and why not that member. and then there are gonna be problems with favoritism among trusted/competent members. i seriously don't believe there is a single person who is completely impartial to all creators.

also wanted to add imdb.com's rating system is not free from abuse. i've seen plenty of instances mentioned on message boards about users rating movies low simply because they think it stakes too high of a spot on the top movies list.

[Zarxrax]

<li>Create an AMV Hall Of Fame

biggest problem i see is selection of the "set of trusted/competent members" and associated elitism whether intentional or not. people will be complaining why this member got selected and why not that member. and then there are gonna be problems with favoritism among trusted/competent members. i seriously don't believe there is a single person who is completely impartial to all creators.

So make the members anonymous. Problem solved.
Again, its just an additional list. No one is forced to view their suggestions if they dont want to.

[godix]

Members who have been here longer (older accounts) get slightly more weight than newer accounts

I'd add one line to this. "With the possibility of having that weighting removing if they are found to be abusing it". Hopefully older account members wouldn't abuse this but I'm sure someone will. When someone reports a bad opinion to the admins, and they agree, any extra weight the account giving the opinion has should be removed along with the bad op.

A members first opinions have less weight than newer opinions

Oh god yes. I've never gone and changed my fan boyish first ops because I think it'd cause more problems than it's worth but I definitely wouldn't have given those videos near the same score I would these days.

Add an admin/creator-given usefulness score for each opinion

I see too much room for abuse here. There's already enough 'you left me a low op, now I must go leave you a low op' tit for tat going on. Adding yet another way someone who received a low op could 'hurt' the person giving an op is not a good idea. At best it'll be annoying. At worse it'll drive people who leave good useful reviews to quit oping entirely. If anything along this line were done I think basing the weighting on the current usefulness score would be less prone to problems but even then it'd probably entirely to easy to game.

Add a score for “I am a big fan of this anime” to “I’m not really a big fan of this anime”

I'm a big fan of Lain. If I see a video that I think doesn't showcase Lain as well as I think it should I'm going to be a lot harsher than if the exact same video had used some other source. Being a fan of the anime doesn't always means the reviewer will give higher scores.

Create an AMV Hall Of Fame

I like this idea but I'm not sure Hall Of Fame is the way to phrase it. I'd prefer to see it phrased as 'great oldies' or something like that. If it's a hall of fame some people will get competitive and argue about why some video they didn't like was included. If it's a 'great oldie' list then people unconsciously look at it expecting a video that, by todays standards, isn't that good on video quality and doesn't have a lot of bells and whistles effects in them. In other words, good videos but nothing they'd want their video to be classified with. It's the exact same idea, just a different phrasing will influence the psychology of how it's viewed. Plus this phrasing gives more leeway to pull out some great two-vcr method era videos that are very hard to find these days without feeling pressured to add Euphoria or other currently widespread and popular vids the second they meet the age requirement.

[Vlad G Pohnert]

I in agreement that I would like to see at least improvements to the present system...

I've been thinking about at it now from a perspective of something like the record charts where we have:

- An All time chart
- A best of year XXXX chart
- A Current top 10 or 100, etc chart

I very much like the concept of a current chart that changes as popularity of videos go up and down... Ut gives a focus of new stuff and does not feel like a static list of the same video alternatives as the current chart has become more or less.

Not withstanding that the concept of how that video would be rated, as this would need more thought, but I think the parameters or "score" of the video could very with age from entry into the catalog, how often it is reviewed, etc and thus as a video gets older and less reviewed, it would fall down the current chart! Even the Star rating chart is too much similar to the all time top chart and the weekly chart I find is hard to really see the overall picture…

The yearly would just be a the highest score the "Current" chart of that year while the all time would remain more or less what it is...

The other thing I would like to bring up is maybe a "controlled" poll or list of questions on how everyone here would like to see the statistics go would be in order. I hear a lot of opinion but maybe it would make it easier if some of the good suggestions here would be put up to the rest of the members and see which way people would like to see things go.

The other consideration I had is maybe it would be more flexible if the opinion ratings and review section were separate and OPTIONAL. I personally never pay any attention to the "Fixed" category of ratings anymore . I've gotten low scores for the video quality just because the anime I used was older and not as high quality or budget as some of the stuff today (thus some of the scoring categories are too vague or interpreted differently)... IN all honesty, it has become somewhat meaningless in some respects.

BUT, I do feel it has its importance and so why not have it optional as with the review side. I like the review part since it tends to (not always) give some good insight and feedback about ones' AMVs... Nut not everyone wants to score videos and/or leave a review...

Vlad

[derobert]

but what's more to be concerned about is that selecting relative to other scores on the member's other videos will tend to promote only one video to the top list per creator

Ummm, I think I've been misunderstood. I'm referring to normalizer per voter, not per creator.

There are attacks on that, but ultimately there is only so much you can do.

[derobert]

So make the members anonymous. Problem solved.
Again, its just an additional list. No one is forced to view their suggestions if they dont want to.

Either that, or we make a selection mechanism not open to cries of elitism, such as an election.

[srocketa]

I know people people go to look at the top ten because they want to 'see the best' and since the top ten are viewed the most, just for being top ten, they will always remain at/near the top...it's not really a fair system to other amv's that are just as good but less well known