Bungie.net Community
This topic has moved here: Subject: Ratings and Rewards Systems: An Analysis for the Truly Bored
  • Subject: Ratings and Rewards Systems: An Analysis for the Truly Bored
Subject: Ratings and Rewards Systems: An Analysis for the Truly Bored
  • gamertag: [none]
  • user homepage:

Destinypedia - The Wiki for Bungie's Destiny
Posted by: DEATHPIMP72
Anyone but Foman. He smells like cheese.

Introduction

In a recent post here on the Septagon, Achronos confirmed what many of us already knew: that the New New Hawtness v2.0 contains a latent and passive algorithm that is currently rating us all based on unknown factors. This algorithm is meant to lead to a new system which, once eventually put in place, will help to encourage members to be contributive, helpful, mature, and respectful.

From Achronos's aforementioned post (as well as his others in recent months), we can safely assume that the new system, whatever it is and if ever it is implemented, will include rewards for good behavior and potentially even punishments for bad behavior. This post is not meant to discuss the rewards or what they should be, but rather focuses solely on the ratings system.

Issue

Rewards and punishments can generally be a good thing, but Achronos mentioned the following:
Any system that uses a "rating" of a user must not allow subjective input. That also has a side benefit - users don't have to understand how their actions affect things, because they aren't actually doing anything. Ideally, even the rewards for good behavior appear to be given out magically, other than the fact that you haven't broken any rules lately.

The issue here is that objective and subjective systems both clearly have their detriments and benefits. Which kind of system would be better? Is it right to have an automated and fully objective rewards system? Would complete subjectivity be better? A mixture?

Brief Answer

I think that a ratings system should contain both objective and subjective inputs for the following reasons:

1) Fully objective systems can be gamed and can lead to uniform yet unfair results.
2) Fully subjective systems can fall prey to a higher workload, petty squabbles, grudges, power games, sycophantism (kissing up), and can lead to highly unpredictable results.
3) A mixture of the two systems would lead to the checks and balances necessary to ensure that a ratings system was both uniformly applied and wholly fair in its application.

Discussion

Fully Objective Systems

Fully objective systems, such as the one Achronos suggests, certainly have their appeal to our mechanical sides. They are undoubtedly immune to human emotions, which greatly enhances their appeal. If their methods are hidden, they can also be "mysterious" enough to cause users to fully obey the rules across the board in order to be more likely to fall within the "good behavior" profile contemplated by the system. Moreover, they are easy to administer, as they generally require a one-time implementation and can then be trusted to run on their own with minimal maintenance.

Unfortunately, a fully automated and objective ranking system also has its downsides. The first downside is that with anything that involves a computation and the internet, about a million geeks will immediately seek to decode the system. I am not much of a tech guy so I don't know how to decode such a thing, but I have been around long enough to know that whenever there is a system, someone will always try to game it. Whether it be by outright deduction or through inductive reasoning based on observing who gets rewards and looking at their past behaviors, the system will not be a mystery for long.

Second, any fully objective system that gives rewards for good behavior must be tested thoroughly for fairness. In devising exactly how a member becomes ranked as a "good" member, it becomes necessary to scrutinize the system for exceptions to the rule.

For example: an automated system that scrutinizes the ratio of posts-to-warnings/blacklistings and determines that a person who has never been warned/blacklisted over their last 1000 posts is automatically a "good member" ignores the possibility that this individual has been around long enough to know exactly how far they can push the rules without breaking them. Note that it is possible to get away with flaming in every single post as long as it is not outrageous. I have rarely seen someone blacklisted for saying something mildly flaming like, "You're an idiot." But under this (very) hypothetical example, a person could place this phrase in every post and still qualify as a "good" member.

This example can be expanded to include many potential exceptions to the rule:
• An objective system that takes number of posts into account fails to include the "good" member who posts infrequently and includes the "bad" member who posts frequently.
• An objective system that takes into account which forum a user posts in fails to include "good" or "bad" members who stick to only one forum.
• An objective system that takes into account how many times the user's posts trigger the "blam" filter fails to include the many users who purposely evade this feature.
• An objective system that gives weight to the length of an account is both under- and overinclusive, as it fails to account for "bad" members with old accounts and "good" members with new accounts.

I understand and concede that the "algorithm" is likely to take many more factors into account than these in order to minimize these negative effects. But I don't think that it is possible for any algorithm to anticipate and cover every exception to the rules. For this reason, the results of such an algorithm could end up being uniform but unfair.

Fully Subjective Systems

The problem with fully subjective systems is clear enough on its face that this section will be understandably and mercifully short. Fully subjective systems lead to several problems. In fact, the problems with a fully subjective system whereby ALL members could participate are so obvious that I won't even go into them. This section assumes that the fully subjective system at issue here is administered solely by people at the "Forum Ninja" level and higher.

The first problem with such a system is clearly evident in that this would create a TON of new work for moderators and up. This would add to their already heavy duties by forcing them to decide when and where to "tag" new users for "good" or "bad" marks. They would not be excited about such a system.

The second problem is one of fairness: an "average" member who happens to be friends in real life with the right people could easily receive credit over another, more deserving "good" member. This is not to say that the moderators are subject to such corruption, but rather to say that "birds of a feather flock together," and that it is much easier to notice someone who is hanging out with you at school or work every day than it is to notice an anonymous person on the internet.

The third problem with a fully subjective system is its unpredictability. An objective system is the same, day in and day out. But subjective input is subject to the inputter's emotions, whims, desires, and moods. While such a system might, on the whole, help to eliminate the unfair results caused by an objective system, it clearly is not ALWAYS fair, nor is it predictable.

The Combination and Conclusion

Undoubtedly, a system is needed to encourage good behavior and discourage bad behavior. But rather than choose the lesser of two evils in terms of a fully subjective or fully objective system, it seems better to include a combination of subjective and objective factors in the ratings system.

I know that I've been speaking somewhat metaphysically and unspecifically this whole time, and I apologize. This has been because I do not know the specifics of Achronos's proposed system and because I am hoping to speak broadly enough to cover many different types of behaviors and systems. But I think that it is clear that a combined system would be far superior to either other system in standalone form.

But to satisfy your thirst for specifics, here is just an example of a ratings system that would allow both factors to be included and could lead to much more uniform and fair results:

Example (NOT a suggestion!)
To get "bumped up to the next level" or to receive whatever rewards are contemplated for good behavior the following steps would take place:

• A SECRET, automated, and objective system weeds out users based on wholly objective factors. Users who have been blacklisted more than twice or warned more than 4 times are automatically ineligible. Users who have been active less than six months are automatically ineligible. Users whose median post length is under ten words are automatically ineligible. Users whose IP addresses/cookies/however they are tracked leads to more than two active, alternate accounts are automatically ineligible.

• From this list, moderators are asked to approve or disapprove members for "promotion" based on rigid, predefined rules issued by site administrators and actively supervised by Master Forum Mods. Users' most-often-forums-posted-in are matched to moderators who tend to moderate those respective forums to increase the likelihood that the moderator is familiar with the particular user. Moderators have indicated many times here in the Septagon that they are familiar with most users, so this should not be much extra work.

Anyway, that's just a small example of a system that would potentially be better than either a fully subjective or fully objective system, and could clearly use some tweaking and more thought. But I just wanted to throw it out there.

Feel free to post what you think about "ratings" systems - I look forward to your opinions.

  • 03.26.2007 5:24 PM PDT

It is the soldier, not the reporter, who has given us freedom of the press. It is the soldier, not the poet, who has given us freedom of speech. It is the soldier, not the campus organizer, who has given us the freedom to demonstrate. It is the soldier, not the lawyer, who has given us the right to a fair trial. And it is the soldier who salutes the flag, who serves the flag, whose coffin is draped in the flag that allows the protester to burn the flag.Father Dennis Edward O'Brien, USMC

Well you did a great job of taking a fiew dozen threads and making them into one.

Good work.

I still doubt we will ever get anything other then member status.

[Edited on 03.26.2007 5:36 PM PDT]

  • 03.26.2007 5:31 PM PDT

You never cease to amaze. Being truely bored because I'm stuck home and ailing, I decided to read your post. It was very well thought out, and written, and I feel pretty much the same way. I don't see how one can completely use an objective, mechanized system to judge who is a "good" member and who is not. If it searches for key phrases, then it will undoubtedly become exploited, and length of activity can be faked as well. I know I don't post a lot, but I'm usually always on Bungie.net. I always have a Bungie tab available. I would rather it be fully or partially subjective because what makes a "good" member how they participate in the community, not whether or not they participate frequently or infrequently. A "post-counter" would be useless because that would just lead to more spam, but if there's also a "word-counter" people can just post a big fat heaping chunk of spam. There's too many flaws for an objective system. But on your "Half and Half" system, I agree. There should be a certain amount of input from each party. That way no one can pass by undetected. But the only thing I don't like about your "Half and Half" system is the fact that there's no way for Mods to select members. What about those good members that didn't get picked up by the algorithm? Do they just get left in the dust? Or what if the algorithm accidentally selects a "good" member and classifies him as a "bad" member? What then?

I just don't see how something like this can be fully automated. If there is a Ninja selection base, do they all need to be approved by all the mods? Most people don't really interact with Mods on a daily basis. There's just too many unknowns to actually form an opinion. I would say that Achronos should give more specifics on the functioning of it, but then that may give away too much and people will exploit the system. I can't really be sure.

If I sounded redundant, I apologize. I'm not really in the healthiest of states.

Summary: Very good post, Foman.

  • 03.27.2007 8:25 AM PDT
  • gamertag: [none]
  • user homepage:

Destinypedia - The Wiki for Bungie's Destiny
Posted by: DEATHPIMP72
Anyone but Foman. He smells like cheese.

So what is my point? Simply this. While it is good to speak about the system, I would caution that we not speculate too deeply or attempt to gain more insight into the function of this system than we really require to be elements in it. In order to allow or make a trust-based rewards/restriction system work, the people who are affected by it must not be aware of HOW it works. Knowing that it is there, seeing examples of it in action, those contribute to the effectiveness. But as soon as someone understands the parameters, their actions will be influenced by that knowledge and in some cases, people will manipulate the system for personal gain.

Morpheus may not have said it, but I think that he would agree. "Once someone truly understands the nature of the Matrix, they are no longer a part of it."

No one who is a member of this site should have access to the "red pill".


This is a great point.

I think that it is very important that, although we can speak about the merits of fully objective versus fully subjective systems, a fully objective system, once decoded, will be of lesser use in encouraging overall good behavior. Although this factor may or may not serve to highlight the need for some subjective elements in any eventual rating-and-rewards system, Achronos has nevertheless wisely been very secretive about the specifics of the new system.

And as for those "secrets," I personally have no desire to learn them. As I have said about myself, I am not much of the kind of guy who would either desire or know how to begin trying to decode such a system. I would hope that most of us are the same.

For most of us who already try to practice good behavior, it is a terrible idea to try to change your behavior based on what you speculate or think the new system will care about. Rather, it's a wise policy to obey the forum rules, follow the guidance of the forum rules, master mods, and admins, and engage in (as evilcam said yesterday) "Business as usual FTW!"



[Edited on 03.27.2007 9:46 AM PDT]

  • 03.27.2007 9:41 AM PDT

MY LOVE FOR YOU IS LIKE A TRUCK...
Posted by: Langley
--on another note, I think MLG Chewhatever is an idiot.

Posted by: Achronos
There is a reason I am user ID 1 and my account creation date is before this site came online.

Jebus Christ Foman, get a friggin' hobby!!

Is my example of how if an objective system only counts the amount of posts (which is mainly how the Xbox.com system works) That previous statement would go towards me being considered a "good member" or "trustworthy member" even though it is pretty much flaming.

I also see that the issue of how this should be considered to be a job of both an objective and subjective system. Look at another objective system, say for example, I dunno...off the top of my head...the Halo 2 ranking system. Everyone knows that if you win a game in Halo 2, your rank (experience) will go up, and if you lose a game it will go down. A completely objective system. And exactly how long did it take before people were able to abuse the system to allow themselves to get a high rank without fairly earning it?? Not too terribly long. I would imagine that eventually if the system on B.net were basically the same, it wouldn't take too long until things were exploited.

At the same time, the system shouldn't be just a subjective system either. Even if it is based merely upon the input of the Mods, there are plenty of us "normies" who are friends if not acquaintances with several members of Mod-Club. For me, a lot of these connections were with them even before they got Hippo-Licked. So, I am not so naive to say that when I post something it doesn't get the exact same treatment from the Mods than compared to some completely unknown member. This could lead to a problem if the Mods like my posting compared to another, even though the content of my post is not superior to the other person's. I might get credit where it was obviously not warranted.

Here's my solution: Have at the bottom of each post (as only viewable by the Mods and Online team) a box that says either "Good" or "Bad". If the mod likes the post, they could click the "good" button, and if they thought it was bad click the "Bad" button and move forward with either warning them or blacklist, or whatever the hell you mods do, I don't pay attention. For every time a bell rings...wait that's not it...um, for every time you get a "Good" click from a Mod or Online team member, it adds a small positive value to the algorithm that is already running. That way, it is both subjective and objective. You can still become a "good member" or "trusted member" without ever having influence from a Mod, but at the same time you could get there quicker if you "find favor in the eyes of the Mods" [/Charlton Heston voice]

That's my solution, and I don't pretend that it is really any different than that kid Foman, who desperately needs to find a hobby, had suggested. Or countless others have suggested. That is just the way that makes the most sense to me.

Oh, and I deserve some reward for having to read all of Foman's original post, that's all I'm saying!!

***Edit: Apparently Recon and I were thinking about the same example, I still maintain the idea that "I THOUGHT OF IT FIRST!!!"

~B.B.

[Edited on 03.27.2007 9:53 AM PDT]

  • 03.27.2007 9:45 AM PDT
  • gamertag:
  • user homepage:
  • last post: 01.01.0001 12:00 AM PDT

I don't see the point of having a rating and rewards system. Why do members feel they need to prove themselves over the internet? I could just as easily *give you a cookie.* But more to the point, the less its speculated, (like recon said) the better. Rewarding and rating members isn't a great way to bring the community together. It will make people elitist, and others looked down upon.

  • 03.27.2007 10:22 AM PDT

Tom Achronos
Bungie.net Overlord
twitter: http://twitter.com/Achronos

"I have no words that would do justice to the atrocities you commit to the English language, as well as your continued assaults on the concepts of basic literacy and logical reasoning."

It has nothing to do with "rating" members. It has everything to do with counteracting the effect of unlimited free accounts.

And, by the way, the author of this topic is incorrect. The system I proposed and implemented a prototype of is not completely objective - bans and warnings are not objective, yet they are one of the MANY datapoints used by the system.

It will be a while before I'm satisifed with the system... while it rates people quite well right now, it takes far too long for it to have enough data to get an accurate trust rating (i.e., newer members get screwed for no good reason). The math needs a bit of work to fix that. But, hey, that's the point of prototyping it first.

Posted by: Thing1
I don't see the point of having a rating and rewards system. Why do members feel they need to prove themselves over the internet? I could just as easily *give you a cookie.* But more to the point, the less its speculated, (like recon said) the better. Rewarding and rating members isn't a great way to bring the community together. It will make people elitist, and others looked down upon.

  • 03.27.2007 11:03 AM PDT
  • gamertag: [none]
  • user homepage:

Destinypedia - The Wiki for Bungie's Destiny
Posted by: DEATHPIMP72
Anyone but Foman. He smells like cheese.

Posted by: Achronos
And, by the way, the author of this topic is incorrect. The system I proposed and implemented a prototype of is not completely objective - bans and warnings are not objective, yet they are one of the MANY datapoints used by the system.

This is true, I was slightly incorrect in that I was too vague. My original post should have clarified that I was speaking of systems with completely objective input, as you have stated this system is:Posted by: Achronos
Any system that uses a "rating" of a user must not allow subjective input. That also has a side benefit - users don't have to understand how their actions affect things, because they aren't actually doing anything. Ideally, even the rewards for good behavior appear to be given out magically, other than the fact that you haven't broken any rules lately.

Posted by: Achronos
I've said it before: systems that require manual input (so-called karma systems) are flawed and subjective.

Posted by: Achronos
You're right - an automated system cannot make value judgements about the content of posts... but it can use information about an account to determine a different thing: trust.
. . .
Admittedly, the math is quite complex, but that's why it is in prototype state right now, just collecting data. I hope that eventually we'll be able to tweak it to be fair, and then turn the consequences on.


Hey, I want to be clear that I fully support this new feature or algorithm and I trust you to tweak it to the maximum fairness possible. I was simply arguing that there must be some level of subjective input - even if it is just careful oversight - in order to ensure that this fairness does indeed occur.

Just as a fully subjective system is "flawed" and unfair, so too is a fully objective system.


[Edited on 03.27.2007 12:03 PM PDT]

  • 03.27.2007 11:57 AM PDT

MY LOVE FOR YOU IS LIKE A TRUCK...
Posted by: Langley
--on another note, I think MLG Chewhatever is an idiot.

Posted by: Achronos
There is a reason I am user ID 1 and my account creation date is before this site came online.

Posted by: Thing1
It will make people elitist, and others looked down upon.


You say "elitist" like it is a bad thing. Being an elitist doesn't mean that you will necessarily be mean to other people, it is merely a type of sociology. I am tired of people on this site, and I'm not directly saying Ben Grimm here, that cry "elitist" every time that something like this is mentioned. Being an elitist is not against the rules on B.net, being what I like to call "ass-clown" is. I'm an elitist, as I have admitted countless times, but that doesn't mean that I am an ass-clown to people. I'd like to think that I am quite the opposite of that, and that I'm actually a helpful member. Even if people start to get rewards, it wouldn't lead to any more people being ass-clowns (or what he calls 'elitist') than pretty much anything else on this site does. The people that would receive the rewards could just as easily be stripped of them if they crossed the line between elitist (such as my case) and being a total ass-clown. People can be ass-clowns (elitists) in a wide variety of ways, but that doesn't mean that we should all be nameless, faceless, people on an internet forum. You have just the same oppertunity to become a "trusted" or "good" member as anyone else based upon how I understand the new system. If you don't become one, you can only blame yourself.

I say let people get rewards, and then lets see if they start using them as leverage on the site. I for one can't wait for that first "trusted" member to get hammered by the Mods or Online team for crossing the line over to ass-clownville!! I've said before, getting rewards always carries with it higher responsiblity, and I'm sure you'll be watched much more closely than your average "normie".

Just my thoughts on it, sorry if they were quite redundant.

~B.B.

  • 03.27.2007 2:46 PM PDT
  • gamertag:
  • user homepage:
  • last post: 01.01.0001 12:00 AM PDT

I'm not sure I fully understand this system, and that may be because I haven't really understood what anyone is saying because I'm a complete doofus.

Anyway, I'm just wondering if this system will rate us and reward us solely on what type of punishments, and the frequency of these punishments, given by the Mods? Or does actual posting play a factor. Because I know that there are people who can be complete jerks without breaking the rules, and there are others who rarely post at all. How can this system accurately and effectively reward people when there are so many (I believe your last census said something around 2 million) users and so few actively posting users?

  • 03.27.2007 3:49 PM PDT

MY LOVE FOR YOU IS LIKE A TRUCK...
Posted by: Langley
--on another note, I think MLG Chewhatever is an idiot.

Posted by: Achronos
There is a reason I am user ID 1 and my account creation date is before this site came online.

You aren't suppose to completely understand the system, Achronos wants to keep some of the "smoke and mirrors" stuff to himself so things stay on the up and up. We are just suggesting or commenting on the way the system is hypothetically, not speaking on it as an authority. I would be very surprised if Achronos ever announced or divulged the entire algorithm, or even a good portion of it. Basically he is trying to counteract the "Hawthorne Effect" and let us lab mice go on thinking that no one is watching.

~B.B.

  • 03.27.2007 4:26 PM PDT
  •  | 
  • Fabled Mythic Member
  • gamertag: [none]
  • user homepage:

argh....math...it hurts my brain!

  • 03.27.2007 5:47 PM PDT