The forum has been archived
While the forum may not be active, the community still lives on Discord! Click here to join us.

How the CC16 rating system was a mess, and how we could fix it

General discussion about Atmosphir.
Post Reply
User avatar
zardini001
Member
Posts: 89
Joined: June 13th, 2014, 7:54 pm
Design Competitions Voted: 1

How the CC16 rating system was a mess, and how we could fix it

Post by zardini001 »

The winnings for CC16 was a very tight race. Untitled came first, Cirrus in second, and Dreamland came in third. Now did they really? Is truly Untitled the number one best entry in the CC16 competition? Or is Dreamland? Or is Cirrus? You all would say your favorite level should of been Number 1. Thats just how voting works.

If you have any other points that I should add to this article, let me know.

I will be referencing the data from the votes throughout this article, so here they are:
Fig. 1: (final votes)
Image
Fig. 2: (votes that includes chibilord_'s and Baufritz's brother)
Image

Onto my points about the CC16 rating system.

The “1-2-3” rating system can be a very untrustworthy system

What is the “1-2-3” system?
Each vote gets a certain amount of points for each entry, like so:
  • 1st place: 3 points
    2nd place: 2 points
    3rd place: 1 point
Why is it a “untrustworthy system”, and how can it be fixed?
Who here really truly went through each level, and carefully decided which level would be a very effective Character Creator, for 3 levels? Most people were very definite on their 1st place choice, but most people just picked their 2nd and 3rd place levels quickly without much thought, except papaya who did a full review on each level, and matthewbny who explained this choices. Truly rating and reviewing each level takes a great deal of time with the amount of submissions and the complexity of each submission, and most people don't want to take that time to do so. Papaya’s review on the levels changed the mind on a couple people, including mine.
Re-look at the “1-2-3” system on an ethical level more than numerical:
  • 1st place: “Awesome! Favorite level out of all submissions!”
    2nd place: “Pretty good, but not amazing”
    3rd place: “Meh, I just picked this one ‘cuz it was somewhat cool, or I just don't want to really think about it much”
With a competition as intense and important as CC16, you would seem we would want to remove as much bias as possible. The reason Untitled won was because it got 7 2nd place votes (See Fig. 1). Think about that. If I am assuming correctly, and my ethics are on point, these 2nd place votes meant that 7 people thought it was “Pretty good, but not amazing,” but should not equate to Dreamland’s 5 and Cirrus’s 6 “Amazing/Awesome” votes. Though 2 people found Untitled to be “Amazing/Awesome”, I just don’t see how it should add up.
Now taking what we’ve discussed about ethics, assuming that 1st place decisions are 100% truthful, 2nd place decisions are 66.6% truthful, and 3rd place decisions are 33.3% truthful. Now multiplying these truthful values to the original "1-2-3" system now the points for each place would equate to:
  • 1st place: 3 points
    2nd place: 4/3 points
    3rd place: 1/3 points
This weighting would help remove the feeling of obligation of the voters to just throw in the 2nd and 3rd votes, and truly pick the level they feel most dearly about. But this weighting would not compete with the action of just only allowing one vote.

Conclusions on the “1-2-3” system
The “1-2-3” system could work if every person is required to explain their choices on a level of at least what matthewbny did. Short, sweet, to the point. Weighting each vote level could also remove some bias. The point of all these adjustments is so you think before you act, and choose the most important, effective, and featurable level in your opinion.

The method of rating resulted in tactical rating strategies

Anybody could see anybody’s vote when voting for CC16. When you vote for something major like president or somebody important like that, do you get to see other people’s votes, and change them after you submitted? No. You have to decide your decision before your time to vote. You think over the pros and cons of each candidate to make sure that your vote helps choose who you think will do better than all other candidates.
Because everyone could see each others votes for CC16 voting, it was very tempting and very easy to vote in such a way that would push your desired winner to the front while not aiding those you didn’t want to win. If nobody knows who anyone else voted for, people will be a lot more honest in their decisions.
When voting for the USA president, you have to be a legal citizen of the United States of America to vote, which allows for legitimate voting matters. On the internet, you can create new accounts to boost your vote. You can also get a friend or family member to vote in your favor.
According to ThatOneFox, chibilord_ was not a valid vote because he “because it’s his only post on the forum, and he never speaks in the discord.” chibilord_ does speak on the discord from time to time, but what is a good measure of activity? How do you determine who is and isn’t active enough to get a vote?

Conclusion on rating methods
All votes for something as large as CC16 should be done via some sort of system that is a one time submission, and cannot be changed afterwards. In regards to “sketchy” people voting, I do understand where ThatOneFox and others were coming from, but to be honest, there are some people in the discord chat and in the OMB forms that haven’t touched Atmosphir in years, and would be considered valid voters by moderators. A system must be defined by the OMB community that determines if a person is an active member of this OMB community or not.

Final Thoughts:
BECAUSE we are already are going to cycle between the 3 levels, and BECAUSE people thought all 3 levels were very well put together, and BECAUSE this rating system was pretty screwed from the start, why don’t we veto the results, and give 1st place to the 3 levels that collectively truly deserved it (its just a suggestion). I personally enjoy all 3 levels, and its great to have all 3 as levels for the Character Creator.

(holy shit that was a lot of typing)
Last edited by zardini001 on February 10th, 2016, 9:00 am, edited 2 times in total.
User avatar
Entity
Editorial Staff
Posts: 3097
Joined: November 29th, 2012, 9:41 pm
Design Competitions Voted: 1

Re: How the CC16 rating system was a mess, and how we could fix it

Post by Entity »

No voting system is perfect, and there's always room for tweaks. However I think saying "the CC16 rating system was a mess" is blowing things a little out of proportion.

I'm not sure I understand where you're getting your "truthful %" values from, but I just thought I'd point out that they match up perfectly with the 3-2-1 system:

#1 vote: 100% truthful - 100% of 3 = 3 pts
#2 vote: 66% truthful - 66% of 3 = 2 pts
#3 vote: 33% truthful - 33% of 3 = 1 pt

I think you accidentally removed the explanation of Fig. 2 (I read an earlier version of this article), but Fig. 2 is what the results would be if chibilord_ and lukatzm's votes had been included. Here is what the results would be if zardini's complete system was used:

Image

I'm not saying we won't ever implement an exponential weighting system in the future, but at this time, after the winners have been announced and rewards given, I think it would be unfair to the other contestants if we implemented your changes. In the end, the top 3 levels would not have changed anyway. The staff is always willing to discuss changes to the system for future competitions.

I think my opinion on the matter can be summed up pretty well by your opening sentence:
zardini001 wrote:You all would say your favorite level should of been Number 1. Thats just how voting works.
Last edited by Entity on February 10th, 2016, 10:59 am, edited 3 times in total.
User avatar
zardini001
Member
Posts: 89
Joined: June 13th, 2014, 7:54 pm
Design Competitions Voted: 1

Re: How the CC16 rating system was a mess, and how we could fix it

Post by zardini001 »

Entity wrote: I'm not sure I understand where you're getting your "truthful %" values from, but I just thought I'd point out that they match up perfectly with the 3-2-1 system:

#1 vote: 100% truthful - 100% of 3 = 3 pts
#2 vote: 66% truthful - 66% of 3 = 2 pts
#3 vote: 33% truthful - 33% of 3 = 1 pt
The multiplier is not multiplied to 3, it's multiplied to the original series of points in the "1-2-3" system.

Entity wrote: A small excerpt from your post:
zardini001 wrote:1st place: “Awesome! Favorite level out of all submissions!”
2nd place: “Pretty good, but not amazing”
3rd place: “Meh, I just picked this one ‘cuz it was somewhat cool, or I just don't want to really think about it much”
This is not how voting works at all. When I was ranking levels for my vote I had to think hard about the things I liked and disliked in each level compared with what I thought would make a good CC level.
Yes, you may have went throughout every level to see what's good and bad about each one, but I have observed from many different voting environments where what I said was true, where people did not take the time to go through entries. If all of you did do what Entity did, good job. I assumed wrong.
Entity wrote: You seem to assume that there was only one "great" level in the competition.
As I can assume by context, you are taking about Cirrus. Oh boy. Your assuming my thoughts. I really enjoyed Baufritz's Dreamland and freekboy's Untitled, and I think it's great freekboy's level got first place.

Basically this thread is based on assumptions and some observations. We all don't stand behind one another in real life to observe if how people is voting is truthful or not, or what people say in general is truthful or not, so we have to assume. It's a big clustertruck of opinions and assumptions. After all, we are just people behind a screen.
User avatar
papaya
Member
Posts: 938
Joined: October 9th, 2012, 3:03 pm
Design Competitions Voted: 1

Re: How the CC16 rating system was a mess, and how we could fix it

Post by papaya »

I think the main thing here is that, ESPECIALLY in the case of cc16 where the winner gets atmos and gets to be the new customization level (and I know that was changed last minute), it only really matters who gets first place. 2nd and 3rd are irrelevant.

would it have made a difference if you could've voted for UP TO three people? That way those of us who only wanted one particular person to win could've voted for just that person and those who thought multiple were good could vote using the 3-2-1 system.

It's important that the next contest has secret votes, but you mentioned in the discord that's already planned so that's fine

perhaps next time the voting could be done with ranking EVERY level? I know there's a way to do this with google docs, or if you really want simply have people PM you their choices and do it in excel or some kooky way. I think it's important in future that people try every level if we're going to have community voting. Zardinis and Freekboys entries were overlooked by some people simply because they're not well-known designers who promote their stuff and talk a lot in the discord. Encouraging every level to be played (and yes, people could still vote having not played ALL the levels, but this way if they have to assign a rank to each they're more likely to) is a much healthier thing for competitions like this.
User avatar
papaya
Member
Posts: 938
Joined: October 9th, 2012, 3:03 pm
Design Competitions Voted: 1

Re: How the CC16 rating system was a mess, and how we could fix it

Post by papaya »

also the idea that both chibilord and baufritz's bro's votes dont count is a little stupid. Where exactly do we draw the line at what is and isnt inactivity? If we want to give atmosphir a thriving community again we're gonna have to include everyone in voting regardless of activity.
User avatar
Entity
Editorial Staff
Posts: 3097
Joined: November 29th, 2012, 9:41 pm
Design Competitions Voted: 1

Re: How the CC16 rating system was a mess, and how we could fix it

Post by Entity »

papaya wrote:I think the main thing here is that, ESPECIALLY in the case of cc16 where the winner gets atmos and gets to be the new customization level (and I know that was changed last minute), it only really matters who gets first place. 2nd and 3rd are irrelevant.
I agree. And that was pretty much why we made the decision to include the top 3. We were looking at the code, doing some preparation in advance, and saw how easy it would be to cycle through and it just made perfect sense.
papaya wrote:perhaps next time the voting could be done with ranking EVERY level? I know there's a way to do this with google docs, or if you really want simply have people PM you their choices and do it in excel or some kooky way.
That's not a bad idea. We have software that counts the votes and it could be easily configured to do that.
papaya (also zardini mentioned this too) wrote:also the idea that both chibilord and baufritz's bro's votes dont count is a little stupid. Where exactly do we draw the line at what is and isnt inactivity? If we want to give atmosphir a thriving community again we're gonna have to include everyone in voting regardless of activity.
You're right. It's my fault for not doing the research, I'd never seen them before in the chat or otherwise, and I incorrectly assumed they were alts. In the end, the 3 CC levels would have been the same, but the atmo distribution would have been different.

EDIT: I had a discussion in the chat with zardini, and he felt that I misrepresented the intent of this part of his post:
zardini001 wrote:1st place: “Awesome! Favorite level out of all submissions!”
2nd place: “Pretty good, but not amazing”
3rd place: “Meh, I just picked this one ‘cuz it was somewhat cool, or I just don't want to really think about it much”
I interpreted it at face value, but here's (roughly) what he actually meant.
1st place: The only level I played, it was pretty good!
2nd place: I didn't really play this level, but papaya spoke highly of it so it was probably good.
3rd place: Uh, this level had a good screenshot so it's probably worthy of 3rd place.
In light of this I have redacted the part of my original post discussing this.
Post Reply