Star Scale Scores Modified

This forum is for site announcements. Please go here to read the SITE & FORUM RULES.

Star Scale Scores Modified

Postby Phade » Sun Jul 13, 2003 9:08 pm

Hey All,

I've written a script that detects if a member is giving about the same score each time they rank a local download with the star scale and then ignores that member. What this means is that if someone keeps giving almost all 1's or 5's or 3's or whatever, all of that members scores are ignored since they obviously aren't taking the scoring seriously. There are some members who have score hundreds of vids and given them all the same score: some all 5's, some all 1's, some all 3's. Of the 3000 or so members giving star scores, only about 160 are affected by this rule.

Hopefully this will make the star scale more reflective of people who actually put thought into their scoring.

Phade.
User avatar
Phade
Site Admin
 
Joined: 20 Oct 2000
Location: Little cabin in the woods...

Postby Deaths_ally » Sun Jul 13, 2003 11:18 pm

interesting Idea.
now wouldn't it be nice to do the same to the opinions [/way off track thought]

I like that idea phade. not that i'm really fussed.. I'm going to try and avoid local downloads until I donate.
User avatar
Deaths_ally
 
Joined: 10 Jan 2003
Location: A hole in the floor

Postby EarthCurrent » Mon Jul 14, 2003 12:07 am

So if I did nothing but download average videos all deserving 3s my scores are ignored?

hmmm....
User avatar
EarthCurrent
 
Joined: 12 Feb 2002

Postby TaranT » Mon Jul 14, 2003 12:14 am

So that's why all my ratings dropped. :P

Just out of curiosity, does this algorithm account for number of ratings that a person has given? Someone who has given only two ratings, both 5-stars, could be doing the right thing, as opposed to someone who has given 392 ratings, all 5-stars.
TaranT
 
Joined: 16 May 2001

Postby dwchang » Mon Jul 14, 2003 12:22 am

TaranT wrote:So that's why all my ratings dropped. :P

Just out of curiosity, does this algorithm account for number of ratings that a person has given? Someone who has given only two ratings, both 5-stars, could be doing the right thing, as opposed to someone who has given 392 ratings, all 5-stars.


Whoa good point. I also think it should take into account averages given AND # of ratings...like someone who has a rating below 1.5 or something and 50+ ratings is probably doing something malicious. Then again, I guess it is possible to download that many bad videos.
-Daniel
Newest Video: Through the Years and Far Away aka Sad Girl in Space
User avatar
dwchang
Sad Boy on Site
 
Joined: 04 Mar 2002
Location: Madison, WI

Postby Phade » Mon Jul 14, 2003 9:15 am

Hey,

The formula goes something like: You must have a ratings count greater than a certain number and a standard deviation less than a certain small number. This way, if you are just starting out and download videos starting at the top of the star scale list, your score will stay until you reach a certain number. By the time you reach the count criteria, your standard deviation should be greater than the small number unless you are just handing out all 5's. The check is updated frequently.

I hope that clears it up some.

Phade.
User avatar
Phade
Site Admin
 
Joined: 20 Oct 2000
Location: Little cabin in the woods...

Postby turboneko » Mon Jul 14, 2003 11:09 am

Phade wrote:There are some members who have score hundreds of vids and given them all the same score: some all 5's, some all 1's, some all 3's. Of the 3000 or so members giving star scores, only about 160 are affected by this rule.

Hopefully this will make the star scale more reflective of people who actually put thought into their scoring.


dwchang wrote:Whoa good point. I also think it should take into account averages given AND # of ratings...like someone who has a rating below 1.5 or something and 50+ ratings is probably doing something malicious. Then again, I guess it is possible to download that many bad videos.


Hmmm... well, we are in those 160 that received the nice email from Phade :P

I have to admit that having a star rate average around 1.50 should be enough to be catalogued as a "video basher", but let me explain the reasons behind this low scores.

First of all, I am very very critic: can't do much about that. This goes back to the times when I was giving out free reviews and hasn't really changed: when 5 is the average (2.5 in terms of stars), I can't see why I should rate 7 or 8 something that's just average.

This doesn't mean that I am not able to rate high: when the video really deserves it I am more than happy to give high marks (or stars). But honestly, of the nearly 20,000 vids that are registered on this site, how many are really enjoyable?

I'd say that we can divide them in the following percentages:

20% are really crappy
50% are just ordinary, average videos that I'll watch just once
15% are nice vids that I'll replay more than once
5% are the really good ones

Now, what makes a good video for me? First of all a good, solid concept. I am really tired of videos made just for the sake of it, without any reason and any planning behind the music/anime combo. I understand that it's not possible to make a masterpiece out of every music video (I've made quite a few senseless videos myself), but a bit of careful planning it's needed if you want to deliver something that is interesting.

Another important point that is seriously understimated is the lenght of the song. If you want to make an 8:00 min video be my guest, but be EXTRA sure that you are going to make it enjoyable. The average attention span drops after 2:50, that's unavoidable.

Now, back to the star scale. I've downloaded quite a few videos that were popping up in the first places with high scores (like 4.80) and only 5 or 6 ratings: most of them honestly sucked (here are the first 1's).

Second, I have a taste for bad videos/movies/stuff, so sometimes I download them just for fun (here they go the other 1's).

Third, the scale doesn't give you much space when you have to rate a video. If there is a video that completely sucked, but had at least one or two nice shots, I'll still rate it 1 because I feel that 2 is too much for just a couple of nice edits. Maybe allowing half scores would help and will make scoring a little more precise.

Fourth, the carrot has given to everybody what they always dreamt of: cheap cheap hosting. This gave to a lot of people the great opportunity of finally sharing their work... but also opened the door to tons of crappy videos. Let's face it: almost everybody can make a music video with little or no knowledge if he/she really wants. Hell, how many times I've put another song on an existing video and found out that it works better than the original?!?

In the overall, I admit that I may have been a little too harsh in some cases... and with the stars it's even worst because there is no comment associated with the rating (hey, what about the option of adding a two liner to your star rate? :P): in writing opinions I had the opportunity of fully explaining my scores.

Well, I hope that this explains a little bit my position: I do not give out straight ones just for the fun of it (hopefully this came out from my post :P). If anybody has pointers, comments and/or rants, I am open to any discussion.

- Marco
You should never underestimate the predictability of stupidity.
User avatar
turboneko
 
Joined: 23 Feb 2001
Location: Foxboro, MA

Postby Phade » Mon Jul 14, 2003 11:38 am

Hey turboneko,

Actually, your scores are still in because your standard deviation is above the cutoff limit. I wrote you a note because your count is high and average is quite low. Of the ~3000 members leaving a star rating, your average is the 22nd lowest, counting the 7 people who gave just one score of 1 (counting the people who have given 5 or more scores, you are 13th from the lowest). There are some people who have given ~50 scores with an average of less than 1.2. There is also a member who has given 200+ scores, all of them 5. It's those people who are being ignored.

Phade.
User avatar
Phade
Site Admin
 
Joined: 20 Oct 2000
Location: Little cabin in the woods...

Postby dwchang » Mon Jul 14, 2003 1:11 pm

Phade wrote:Hey turboneko,

Actually, your scores are still in because your standard deviation is above the cutoff limit. I wrote you a note because your count is high and average is quite low. Of the ~3000 members leaving a star rating, your average is the 22nd lowest, counting the 7 people who gave just one score of 1 (counting the people who have given 5 or more scores, you are 13th from the lowest). There are some people who have given ~50 scores with an average of less than 1.2. There is also a member who has given 200+ scores, all of them 5. It's those people who are being ignored.

Phade.


It might also be nice to have an automated system which mails people to tell them their scores are being dropped (useless?). It might lead them to remedy the situation.

I also think a comment or two would be nice (~255 characters ). That also goes along with my suggestion for non-anonymous :-P. At the same time, it *somewhat* defeats the purpose of a system (very fast), by adding 20 seconds to the time it takes to do star-ratings...which could discourage *slightly*.
-Daniel
Newest Video: Through the Years and Far Away aka Sad Girl in Space
User avatar
dwchang
Sad Boy on Site
 
Joined: 04 Mar 2002
Location: Madison, WI

Postby turboneko » Mon Jul 14, 2003 1:32 pm

Thanks for the reply Phade... it's nice to know that I'm still in :P
About the star system... I agree with dwchang and actually it could be modified in this way:

1) When you star-rate a video, you are not required to submit any comment

2) If you do not submit a comment, your rating will be anonymous

3) If you do submit a comment, there should be a checkbox that allows you to choose between anonymous/non anonymous

4) It should be possible to see for each video all the non-anonymous star "comments" that people left: making this stuff public may help in understanding if the video suits the personal taste before downloading it without completely spoil it (as it could happen with opinions).

The checkbox at point 3 is mainly to avoid "revenge" ratings... if stars were not anonymous I would have already collected tons of them :P

On the other end, if you have the possibility of justifying your low score with a small comment, people would not take it so bad (hopefully :?) and it may actually help the feedback system.

What do you guys think of it?
You should never underestimate the predictability of stupidity.
User avatar
turboneko
 
Joined: 23 Feb 2001
Location: Foxboro, MA

Postby dwchang » Mon Jul 14, 2003 1:56 pm

turboneko wrote:Thanks for the reply Phade... it's nice to know that I'm still in :P
About the star system... I agree with dwchang and actually it could be modified in this way:

1) When you star-rate a video, you are not required to submit any comment


:nods:

turboneko wrote:4) It should be possible to see for each video all the non-anonymous star "comments" that people left: making this stuff public may help in understanding if the video suits the personal taste before downloading it without completely spoil it (as it could happen with opinions).


:nods:


turboneko wrote:On the other end, if you have the possibility of justifying your low score with a small comment, people would not take it so bad (hopefully :?) and it may actually help the feedback system.


:nods again:

The only thing I disagree with is the anonymous scoring. I am still a supporter of is non-anonymous scoring. People are generally quite a bit more honest when they have to back-up their score with their name/reputation. As you mentioned, I think there is too much "revenge" scoring (sad that people are this petty) and I'm sure you'll notice there is less within the Opinions option since your name is there.

At the same time, I think "Opinions" suffers from over-inflation which is demonstrated by the insanely high scores and "global averages."

To be honest, I do realize there is no "perfect answer," but at the same time think there is room for improvement and thus my suggestions.

As for "Opinions," as I have stated, I'd like to see things with the imdb.com scoring system and I know Phade is working on that. I look forward to see how that pans out.
-Daniel
Newest Video: Through the Years and Far Away aka Sad Girl in Space
User avatar
dwchang
Sad Boy on Site
 
Joined: 04 Mar 2002
Location: Madison, WI

Postby AbsoluteDestiny » Mon Jul 14, 2003 2:04 pm

I think I have to side with marco here.

I've given videos low star ratings where I wouldnt have without anonimty. I can be more honest with anonimity because I dont want to be flamed in return or seen as elitist or some other odd notion.

With full blown opinions I will back up what I say in detail, so it's harder to refute... but with a star opinion it's not easy to do that, even with a small comment box.
User avatar
AbsoluteDestiny
 
Joined: 15 Aug 2001
Location: Oxford, UK

Postby dwchang » Mon Jul 14, 2003 2:30 pm

AbsoluteDestiny wrote:I think I have to side with marco here.

I've given videos low star ratings where I wouldnt have without anonimty. I can be more honest with anonimity because I dont want to be flamed in return or seen as elitist or some other odd notion.

With full blown opinions I will back up what I say in detail, so it's harder to refute... but with a star opinion it's not easy to do that, even with a small comment box.


I understand that....perhaps ~1024 characters then? :-P

I hope you understand the inverse in anonymous ratings and how easiliy one could be malicious.

As I said earlier, there is no true "perfect" solution when you have a "competitive" rating system like this. Oh well, we're all having fun right? :-D
-Daniel
Newest Video: Through the Years and Far Away aka Sad Girl in Space
User avatar
dwchang
Sad Boy on Site
 
Joined: 04 Mar 2002
Location: Madison, WI

Postby turboneko » Mon Jul 14, 2003 3:24 pm

dwchang wrote:I understand that....perhaps ~1024 characters then? :-P

I hope you understand the inverse in anonymous ratings and how easiliy one could be malicious.

As I said earlier, there is no true "perfect" solution when you have a "competitive" rating system like this. Oh well, we're all having fun right? :-D


I think that anonymous is good... it helps keeping scores on a more "reasonable" (read: lower) level and in any case looks like most of the people is not abusing it. Again, it would be nice to associate your name, if you choose it, with your comments: believe me, I don't rate low just for the sake of it :P

To make an example, I rated 2 the Excel Popup Video (and here it goes my anonimity :roll:) because:

1) It uses pop ups, which have already been used (wasn't a total waste of 6min 35secs the first one?), without breaking any new ground

2) I don't find it particularly funny (I have a strange sense of humor, I know)

3) Excel Saga is becoming a little overused now

Even if I was to score it 1 without even watching it, does it make the difference over the 200+ stars it already got? The nice thing about the star system is that it's easy to collect quite a few of them... the more stars, the more precise the average will be. If people are a little more strict because they have the shield of anonymity, so be it... I don't find it particularly problematic. But absolutely it would be nice to have comments linked to the rating... and the ability to not be anonymous if you like.
You should never underestimate the predictability of stupidity.
User avatar
turboneko
 
Joined: 23 Feb 2001
Location: Foxboro, MA

Postby dwchang » Mon Jul 14, 2003 3:34 pm

turboneko wrote:Even if I was to score it 1 without even watching it, does it make the difference over the 200+ stars it already got? The nice thing about the star system is that it's easy to collect quite a few of them... the more stars, the more precise the average will be. If people are a little more strict because they have the shield of anonymity, so be it... I don't find it particularly problematic. But absolutely it would be nice to have comments linked to the rating... and the ability to not be anonymous if you like.


No, I agree on that point. That's why I think it's a great idea...very little effort to do and you can receive quite a few of them. From my statistics classes in college (and common sense), your score will make a lot more sense when you have a larger set of data. I don't tend to believe videos rated *extremely* high with less than 20 stars. At the same time, I am also pleased when I get in the hundreds since it gives me an accurate idea of how people feel.

So yeah...I agree there. I'm not at all displeased with the overall idea. At the same time, I'm not *that* upset with anonymous scoring (although it may appear otherwise) :-P.
-Daniel
Newest Video: Through the Years and Far Away aka Sad Girl in Space
User avatar
dwchang
Sad Boy on Site
 
Joined: 04 Mar 2002
Location: Madison, WI

Next

Return to Site Announcements

Who is online

Users browsing this forum: No registered users and 3 guests