VGMdb
Go Back   VGMdb Forums > Discussion > Miscellaneous Discussion
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old Feb 23, 2013, 11:59 AM
Namorbia's Avatar
Namorbia Namorbia is offline
Member
 
Join Date: Mar 2012
Location: Helsinki, Finland
Posts: 71
Default Rating systems: 5 stars vs. something else?

I've been reading a bit about rating systems lately.

Apparently 5-star rating systems are not good, since most people rate really high or really low. YouTube used to have a 5-star rating system and "the number of 5-star ratings greatly outnumbered all others, with 1-star a distant second." People also use half-stars a lot less than whole stars. On some site called MovieLens with millions of ratings "only 20% of all ratings are half-stars."

Link here: A Better Way to Rate Films
________________

I found a very simple, but effective way of using ratings. Assign a verbal meaning to each point in the rating system. This is the 10-point system that RPGnet uses:

10 - One of the Best Ever
9 - Outstanding
8 - Very Good
7 - Good
6 - Above Average
5 - Average
4 - Almost Average
3 - Some flaws
2 - Poor
1 - Worthless

(I use this now for rating my songs on iTunes (with half stars) and my mood affects the ratings less, which is good. Ratings 3 and 2 to me are "bad" and "very bad," respectively. Sadly I have less 5-star tracks than before.)

Link here: Collective Choice: Experimenting with Ratings
________________

This is also worth reading: Is there a better alternative to the 5-star rating system?
Many insightful answers with lots of examples.
________________

How do you go about ratings albums on VGMdb, movies on IMDB or anything else?
Are you happy with VGMdb's 5-star rating system and how people use it?
Reply With Quote
  #2  
Old Feb 23, 2013, 12:26 PM
Phonograph's Avatar
Phonograph Phonograph is offline
Senior Member
 
Join Date: Jun 2010
Location: 5.342.482.337.666
Posts: 3,810
Default

already discussed on the irc chan, and maybe on forums?

edit: I mean, it's been discussed and blah agreed about the fact that the actual ranking system isn't the best

maybe a system rating track by track would be better
but the problem would be how to implement that simply

Last edited by Phonograph; Feb 23, 2013 at 12:39 PM.
Reply With Quote
  #3  
Old Feb 23, 2013, 12:27 PM
Xenofan 29A Xenofan 29A is offline
Senior Member
 
Join Date: Jan 2009
Posts: 609
Default

VGMdb has a 10 point system. Half stars are functionally equivalent in all but graphical representation to points. I don't think about it differently here than at IMDb.

In my mind:
10 (5) - Best of the best
9 (4.5) - Excellent
8 (4) - Great
7 (3.5) - Good
6 (3) - Decent
5 (2.5) - Okay
4 (2) - Mediocre
3 (1.5) - Poor
2 (1) - Terrible
1 (0.5) - Absolutely horrible

Quote:
Originally Posted by Namorbia
Apparently 5-star rating systems are not good, since most people rate really high or really low. YouTube used to have a 5-star rating system and "the number of 5-star ratings greatly outnumbered all others, with 1-star a distant second." People also use half-stars a lot less than whole stars. On some site called MovieLens with millions of ratings "only 20% of all ratings are half-stars."
Any ratings system you can imagine will be abused this way. Look at any movie on IMDb, and you'll see the same kind of distribution.

Last edited by Xenofan 29A; Feb 23, 2013 at 12:31 PM.
Reply With Quote
  #4  
Old Feb 23, 2013, 01:07 PM
Dag's Avatar
Dag Dag is offline
VGMdb Staff
 
Join Date: Dec 2007
Posts: 1,279
Default

Three star rating!
2 stars: great
1 star: good
-1 star: no good

Less choices = easier to understand!
Reply With Quote
  #5  
Old Feb 23, 2013, 01:25 PM
Phonograph's Avatar
Phonograph Phonograph is offline
Senior Member
 
Join Date: Jun 2010
Location: 5.342.482.337.666
Posts: 3,810
Default

the thing I hate the most about the 5 star system is that you can't see all ratings, not necessarily who rates, but the rates themselves
because if you have 3 people with 3 stars, you can't know if 3 people have rated 3 stars or 1/3/5 stars (already spoke about it)

there should a rating page like there is a history page (for edits)
Reply With Quote
  #6  
Old Feb 23, 2013, 01:32 PM
Efendija's Avatar
Efendija Efendija is offline
VGMdb Staff
 
Join Date: Aug 2011
Location: Serbia
Posts: 1,905
Default

Yup it's like someone gives a 5 to a jpop single and then I come by and give it like 1.5-2 heh
But it's not about 5/10 stars there, just a rating history is missing.
Reply With Quote
  #7  
Old Feb 24, 2013, 04:33 AM
Namorbia's Avatar
Namorbia Namorbia is offline
Member
 
Join Date: Mar 2012
Location: Helsinki, Finland
Posts: 71
Default

Quote:
Originally Posted by Xenofan 29A View Post
Any ratings system you can imagine will be abused this way. Look at any movie on IMDb, and you'll see the same kind of distribution.
Like I said, attaching a verbal meaning to each point in the scale decreases this abuse. At least according to this article. So there is a difference in rating systems. Not everyone thinks of stars in terms of excellent, good, mediocre, etc. like you do. But if they would, that would help. RPGNet determines those labels for you and apparently it helps.

I didn't start this thread to change the rating system of VGMdb. I'm just interested in the subject in general. But if it's a viable option, of course we can think about changes to VGMdb's rating system too.

Quote:
Originally Posted by Dag View Post
Three star rating!
2 stars: great
1 star: good
-1 star: no good

Less choices = easier to understand!
I like this. Do you use this approach anywhere? Is it difficult not to have 0 stars or some option for "average"?

Last edited by Namorbia; Feb 24, 2013 at 04:36 AM.
Reply With Quote
  #8  
Old Feb 24, 2013, 07:15 AM
Datschge's Avatar
Datschge Datschge is offline
Trusted Editor
 
Join Date: Mar 2008
Posts: 702
Default

Instead ratings I personally prefer a relative ranking system purely by order. Usually I know pretty well myself what I prefer over other works at any given time, the issue is always how much value I attach to that (naturally heavily skewed to pushing the max, after all you want others to notice what you like...). Don't know how others feel about this, but I think it would be great to allow everyone to make personal ordered hit lists without any ratings attached (the order IS the rating), and the final rating value is then calculated based on the relative ranking a work has on those hit lists.
Reply With Quote
  #9  
Old Feb 24, 2013, 11:11 AM
Dag's Avatar
Dag Dag is offline
VGMdb Staff
 
Join Date: Dec 2007
Posts: 1,279
Default

Quote:
Originally Posted by Namorbia View Post
I like this. Do you use this approach anywhere? Is it difficult not to have 0 stars or some option for "average"?
Nah, just an internal scale of sorts. Personally, having the "is this a 9/10, or maybe 8.5/10?" and "this sucks so much that deserves a 1/10. Wait, there is a song I like, 3/10? Not sure, maybe I won't vote at all" situations makes me avoid voting. So less possible votes/stars would make it easier to use and understand other people's votes.

It's better if you don't have too many choices (think some movie reviewers), and "verbal" ratings like you said. So maybe - "(no vote) / lame / average / sweet / AWESOME".
Reply With Quote
  #10  
Old Feb 25, 2013, 08:59 PM
Rhythmroo's Avatar
Rhythmroo Rhythmroo is offline
Senior Member
 
Join Date: Aug 2009
Location: New England
Posts: 103
Default

Perhaps disposing of the rating system altogether in favor of posting reviews (likes/dislikes) or thoughts about the album on an entry's thread would eradicate unreasonably low scores that could deter a potential listener from an unjustly muddied, but attractive album.
Reply With Quote
  #11  
Old Feb 25, 2013, 10:40 PM
Mortavia's Avatar
Mortavia Mortavia is offline
Senior Member
 
Join Date: Jul 2012
Posts: 141
Default

GameFAQs has descriptions next to its ratings for games, and I always agreed with what they describe as a certain score (out of 10).

Spoiler:
Unplayable (could be "unlistenable" here, though that's not a word)
Terrible
Bad
Poor
Playable (could be "listenable", or "average" if that sounds too bad)
Fair
Good
Great
Outstanding
Flawless

I've always thought these words are pretty basic quality descriptions, nothing fancy like "awesome" vs. "amazing" (which one's really better?), and they're distanced enough from each other so that there's really no confusion in what you think the rating should be if you go by the descriptions.

I don't think 10 is too many choices. If I'm going to give something a rating, I want some room to distinguish it from things I don't think are quite at its level. For example, I think X is a 3.0, Y is a 3.5, and Z is a 4.0. These are very different rankings, because as it is, there will be things in the 3.0 ranking I like more than others in the same 3.0 ranking, but yet still less than anything in 3.5 - it's general but descriptive. But if we start to smush rankings into each other, now X and Y are in the same class, 3.0, and Z is way better than Y, 4.0. I like 1-10 ranking systems for this. Sure, we could have 1-20, 1-100, whatever, but 1-10 is pretty common and I think it works well.

I don't think abuse is something anybody should try too much to combat, because it's just going to exist no matter what. Some people rate things 5/5 before the album has even come out. And some people rate 5/5 if they "like" it, so a good 80-plus percent of their ratings are 5/5. That's okay, let them rate how they want, and the people who want to make meticulous use of it will. I think it will be like that regardless of any system used.

Wow... I need to get on it and give some more soundtracks ratings! And fix some of my old ones... they're from when I first joined and didn't really have enough sense of good ratings yet.

Datschge: Sometimes I can easily say "I like X soundtrack more than Y", and then that would work. But sometimes the soundtracks are too different, and I like them "equally" for different things (and I quote "equally" like that because it's too hard even to say that since they're so different, they can't really be compared). If X and Y are really different from each other, but both deserving of a 3.5, I wouldn't want to have to rank them against each other, but instead I'd want to give them a rating based on what they deserve for what they do (in my opinion of course).

Last edited by Mortavia; Feb 25, 2013 at 10:45 PM.
Reply With Quote
  #12  
Old Feb 26, 2013, 03:23 AM
Datschge's Avatar
Datschge Datschge is offline
Trusted Editor
 
Join Date: Mar 2008
Posts: 702
Default

Quote:
Originally Posted by Mortavia View Post
Datschge: Sometimes I can easily say "I like X soundtrack more than Y", and then that would work. But sometimes the soundtracks are too different, and I like them "equally" for different things (and I quote "equally" like that because it's too hard even to say that since they're so different, they can't really be compared). If X and Y are really different from each other, but both deserving of a 3.5, I wouldn't want to have to rank them against each other, but instead I'd want to give them a rating based on what they deserve for what they do (in my opinion of course).
Right, but in my experience cases of "I like both the same" are much less frequent than having ever so slight preferences. Put another way, with the current rating system you essentially have only 10 fixed levels/groups of "I like them the same" connected to a very specific value to choose from, so the more works you rate the fussier the differentiation gets (and this is already neglecting the psychological force that comes with the rating values themselves, one is likely to put similarly valued works in the same group as a whole 0.5* step is often felt too big a value difference in such cases, so the value variety actually used is reduced even further by this). With a simple ordered list the opposite is the case, the more works are included the more detailed is its informative value (conversely the smaller the list the more misleading can it be, this is were fixed values are indeed better). And naturally works that are same or similarly valued are placed close together. I'd like to rate everything. With ratings I feel it's both too much work and futile so I don't even start using it.

Last edited by Datschge; Feb 26, 2013 at 03:26 AM.
Reply With Quote
  #13  
Old Mar 7, 2013, 03:22 PM
Hellacia Hellacia is offline
Trusted Editor
 
Join Date: Jan 2010
Posts: 813
Default

I think we should just be like Facebook and have a Like button. Then you can just go around and Like all the albums you like and then there's no worries about whether or not somebody gave a fair rating. Plus then we're more like Facebook! I Like Facebook.
Reply With Quote
  #14  
Old Mar 7, 2013, 04:57 PM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

Quote:
Originally Posted by Dag View Post
Three star rating!
2 stars: great
1 star: good
-1 star: no good

Less choices = easier to understand!
This is actually pretty similar to what I use for VGM ratings. It works really well, especially since my two main goals when rating are to eliminate waste and identify the best tracks on a given soundtrack. So it's basically Pass/Fail with an extra step to give a boost to the really special tracks.
Reply With Quote
  #15  
Old Mar 7, 2013, 06:40 PM
Phonograph's Avatar
Phonograph Phonograph is offline
Senior Member
 
Join Date: Jun 2010
Location: 5.342.482.337.666
Posts: 3,810
Default

actually, instead of changing the current rating system for a better one now, this one could be tweaked a bit
removing the current average thing and limiting to 5 stars or even 3 as follows

*****_(excellent?) nb of votes
****__(good?) nb of votes
***___(ok?) nb of votes
**____(bad?) nb of votes
*_____(forget?) nb of votes

edit: I mean showing those 5/3 (perso, 3 is enough) lines, in case it's not clear enough

Last edited by Phonograph; Mar 7, 2013 at 06:47 PM.
Reply With Quote
  #16  
Old Mar 8, 2013, 12:59 AM
Namorbia's Avatar
Namorbia Namorbia is offline
Member
 
Join Date: Mar 2012
Location: Helsinki, Finland
Posts: 71
Default

I removed half-star ratings from iTunes. It takes too much thinking and looks ugly. Just look at a list of tracks like this:

***½
****½
**½
*****
***½
****

***½

It looks confusing and I can't quickly pick out the best tracks (unless I order them by rating). I replaced my thinking like this:

10/9 -> 5 (best ever OR outstanding)
8 -> 4 (great)
7 -> 3 (good)
6/5 -> 2 (average)
4/3/2/1 -> 1 (bad)

The list of stars above would now look like this:

***
*****
**
*****
***
****
*
***

My point is that "average/ok" should not be in the middle. If a track is bad, I don't care if it's very bad or not. That's why "average/ok" should be 2 stars in a 5-star or 4-star rating system.
Reply With Quote
  #17  
Old Mar 8, 2013, 07:11 AM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

See, that is the issue I started having...Average tracks should not be rewarded. They just take up too much room when they come up with very little reward. But it makes sense, rarely do I play a full game where every piece grabs my attention. So I decided to start removing them, as I would a truly "bad" track.

I think on average, when I lay down a rating on a full soundtrack, I usually end up with a near perfect split: 1/3 bad/lackluster/filler tracks, 1/3 "average" tracks and 1/3 "really good/great" tracks, which really should be the focus of any future listening I do.
Reply With Quote
  #18  
Old Mar 8, 2013, 07:38 AM
Phonograph's Avatar
Phonograph Phonograph is offline
Senior Member
 
Join Date: Jun 2010
Location: 5.342.482.337.666
Posts: 3,810
Default

Quote:
Originally Posted by Namorbia View Post
...

My point is that "average/ok" should not be in the middle. If a track is bad, I don't care if it's very bad or not. That's why "average/ok" should be 2 stars in a 5-star or 4-star rating system.
sorry, but it's mathematically incorrect
average of 1+2+3+4+5 is 3, not 2

you should simply use another word, average at 2 is confusing
Reply With Quote
  #19  
Old Mar 8, 2013, 07:49 AM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

I don't think an "average" track is the same as a mathematical average. If anything, it's weighted much lower (calling a track "average" is almost never a compliment).
Reply With Quote
  #20  
Old Mar 8, 2013, 09:23 AM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

Here are the ratings I have for Seiken Densetsu 3's first and second discs. Some bias exists (it's one of my favorite soundtracks), but that top row makes it easy to pluck out my favorites quickly. The ones on the bottom row, I delete from my computer, while the ones on the middle row, I tend not to carry them around on my portable players.

Disc 1

P! - 2, 3, 5, 6, 7, 12, 13, 15, 16, 18
P - 4, 8, 14
F -1, 9, 10, 11, 17, 19

Disc 2
P! - 7, 8, 10, 12, 14, 19, 20
P -1, 11, 13, 21
F- 2, 3, 4, 5, 6, 9, 15, 16, 17, 18
Reply With Quote
  #21  
Old Mar 8, 2013, 09:36 AM
Xenofan 29A Xenofan 29A is offline
Senior Member
 
Join Date: Jan 2009
Posts: 609
Default

Wow, you listed some of my favorite tracks from SD3 as filler! Different tastes, I guess.
Reply With Quote
  #22  
Old Mar 8, 2013, 09:57 AM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

Yeah, I tended not to favor a lot of the darker/more atmospheric tracks over the more upbeat, action-oriented tunes. Then again, I knew before I sat down to rate, I was never big on the first half of disc 2.
Reply With Quote
  #23  
Old Mar 8, 2013, 12:22 PM
yvk2000's Avatar
yvk2000 yvk2000 is offline
Junior Member
 
Join Date: Mar 2009
Posts: 20
Default

Wow, I am suprised. I mean even taking out the dark/atmospheric/somber tracks you've still got Another Winter, Harvest November and Few Paths Forbidden (1.9, 2.3, 2.4 resp) in there. Granted not the upbeat stuff, but hardly dark and definitely colorful and pure Kikuta!

Last edited by yvk2000; Mar 8, 2013 at 12:30 PM.
Reply With Quote
  #24  
Old Mar 8, 2013, 12:32 PM
Efendija's Avatar
Efendija Efendija is offline
VGMdb Staff
 
Join Date: Aug 2011
Location: Serbia
Posts: 1,905
Default

In the end it's useless to discuss about differences in personal taste.
I don't like almost all of j-pop, but when somebody praises a certain j-pop song/single/album/whatever, I won't start saying dude wtf and similar (just a random example)
To each his own.
Reply With Quote
  #25  
Old Mar 8, 2013, 01:09 PM
Xenofan 29A Xenofan 29A is offline
Senior Member
 
Join Date: Jan 2009
Posts: 609
Default

Quote:
Originally Posted by Efendija View Post
In the end it's useless to discuss about differences in personal taste.
I don't like almost all of j-pop, but when somebody praises a certain j-pop song/single/album/whatever, I won't start saying dude wtf and similar (just a random example)
To each his own.
Sometimes it can be interesting to discuss taste. There are things most people agree are bad and things many people agree are good. You can try to understand why others appreciate something that you don't.
Reply With Quote
  #26  
Old Mar 8, 2013, 01:26 PM
Hellacia Hellacia is offline
Trusted Editor
 
Join Date: Jan 2010
Posts: 813
Default

Darn it, I'm missing out on this thread being derailed! This has become about discussing personal taste? Oh, let me in on this! I love to discuss personal taste. I personally think that entire soundtrack sucks! There was not one song on it that I liked. That is my personal taste.
Reply With Quote
  #27  
Old Mar 8, 2013, 01:29 PM
Hellacia Hellacia is offline
Trusted Editor
 
Join Date: Jan 2010
Posts: 813
Default

Nah I'm just kidding. I actually liked every single song except for one. I don't know how you guys could dislike any song on that soundtrack (except for that one).

P.S. Efendija is right. It is the most utterly stupid thing EVER to discuss personal taste about art.

Example 1:
"I really like this thing!"
"So do I!"
"Okay, cool!"

Example 2:
"I really like this thing!"
"I don't, it sucks!"
"Really? Wow..."

Who cares? And you can't "understand" why somebody likes something. That like trying to "understand" why I like broccoli while you think it's the most horrible tasting thing you've ever eaten.

Me: "I think broccoli tastes really good because ______!"
You: "That's weird, I don't!"
NOT: "Oh, okay, I get it, now it tastes good to me too!"

Discussing taste = fail and this thread is about ratings systems, seriously. I vote a Facebook Like system guys. Who's with me?!

Last edited by Hellacia; Mar 8, 2013 at 01:31 PM.
Reply With Quote
  #28  
Old Mar 8, 2013, 04:36 PM
Datschge's Avatar
Datschge Datschge is offline
Trusted Editor
 
Join Date: Mar 2008
Posts: 702
Default

Quote:
Originally Posted by Hellacia View Post
Who's with me?!
I'm not. I hate Facebook. And a simple Like-system is nothing more than what's currently the amount of votes albums get disregarding the actual rating, which would only make popular works stick out even more and less known works never see the light that way.
Reply With Quote
  #29  
Old Mar 8, 2013, 04:39 PM
GoldfishX GoldfishX is offline
Senior Member
 
Join Date: Jul 2010
Posts: 266
Default

My rating system was a demonstration of showing which tracks I liked and disliked from the subject matter in basically a capsule form. Based on the follow-up and said follow-up nearly derailing the entire thread, with basically no additional input from myself, I'd call it a rousing success!

(well, I already know it is one, since I have hundreds of those little P!/P/F grids for various discs/albums, but it's the first time I've used it in public.)

Free bonus: SD Disc 3

P! - 1, 3, 4, 5, 7, 8, 14, 15, 17
P - 2, 9, 12, 13, 16, 19, 20
F - 6, 10, 11, 18
Reply With Quote
  #30  
Old Mar 9, 2013, 12:21 AM
Namorbia's Avatar
Namorbia Namorbia is offline
Member
 
Join Date: Mar 2012
Location: Helsinki, Finland
Posts: 71
Default

Quote:
Originally Posted by Phonograph View Post
sorry, but it's mathematically incorrect
average of 1+2+3+4+5 is 3, not 2

you should simply use another word, average at 2 is confusing
I know. Maybe calling it average is wrong then. But there are more good tracks than bad tracks and I need some room to distinguish the good from the great and the outstanding. Giving equally many stars to bad and good tracks is a waste, if I only use five stars. If it's a 9+ star system, then I don't mind having average in the middle, because I'll still have enough leeway for good tracks.

This is a question for everyone: Do "meh", "okay" and "average" mean the same thing and have the same rating? Or is it more like okay>average>meh?

Quote:
Originally Posted by GoldfishX View Post
The ones on the bottom row, I delete from my computer
How many listens do you give to a track before rating it and deleting it? Maybe you will start liking and appreciating the bad tracks after a few more listens? Ever regretted deleting tracks from your computer?

This brings me to the next quote, which I disagree with:

Quote:
Originally Posted by Hellacia View Post
P.S. Efendija is right. It is the most utterly stupid thing EVER to discuss personal taste about art.
Others opinions of a track, especially positive opinions, have an effect on me. When a friend points out what's good about a track, I often start liking it more. This effect can be achieved by listening to VGM podcasts also. Maybe it's simply taking the time to really listen, be attentive and realizing the good parts. The more friends I have who like VGM and point out the good parts, the broader my taste becomes!

Last edited by Namorbia; Mar 9, 2013 at 12:28 AM.
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Dysfunctional Systems: Orientation Soundtrack ff80c38 Album Discussions 0 Jul 25, 2017 09:48 AM
ARCD0001: The regret of stars, but stars shine bright. Faalagorn Album Discussions 2 Apr 9, 2011 05:09 PM
Missing systems dma Questions and Comments 4 Mar 15, 2010 03:42 PM
Album rating function Gaia Questions and Comments 6 Oct 30, 2008 07:10 AM