- x Foman123 x
- |
- Master Forum Ninja
- gamertag: [none]
- user homepage:
Introduction
In a recent post here on the Septagon, Achronos confirmed what many of us already knew: that the New New Hawtness v2.0 contains a latent and passive algorithm that is currently rating us all based on unknown factors. This algorithm is meant to lead to a new system which, once eventually put in place, will help to encourage members to be contributive, helpful, mature, and respectful.
From Achronos's aforementioned post (as well as his others in recent months), we can safely assume that the new system, whatever it is and if ever it is implemented, will include rewards for good behavior and potentially even punishments for bad behavior. This post is not meant to discuss the rewards or what they should be, but rather focuses solely on the ratings system.
Issue
Rewards and punishments can generally be a good thing, but Achronos mentioned the following:
Any system that uses a "rating" of a user must not allow subjective input. That also has a side benefit - users don't have to understand how their actions affect things, because they aren't actually doing anything. Ideally, even the rewards for good behavior appear to be given out magically, other than the fact that you haven't broken any rules lately.
The issue here is that objective and subjective systems both clearly have their detriments and benefits. Which kind of system would be better? Is it right to have an automated and fully objective rewards system? Would complete subjectivity be better? A mixture?
Brief Answer
I think that a ratings system should contain both objective and subjective inputs for the following reasons:
1) Fully objective systems can be gamed and can lead to uniform yet unfair results.
2) Fully subjective systems can fall prey to a higher workload, petty squabbles, grudges, power games, sycophantism (kissing up), and can lead to highly unpredictable results.
3) A mixture of the two systems would lead to the checks and balances necessary to ensure that a ratings system was both uniformly applied and wholly fair in its application.
Discussion
Fully Objective Systems
Fully objective systems, such as the one Achronos suggests, certainly have their appeal to our mechanical sides. They are undoubtedly immune to human emotions, which greatly enhances their appeal. If their methods are hidden, they can also be "mysterious" enough to cause users to fully obey the rules across the board in order to be more likely to fall within the "good behavior" profile contemplated by the system. Moreover, they are easy to administer, as they generally require a one-time implementation and can then be trusted to run on their own with minimal maintenance.
Unfortunately, a fully automated and objective ranking system also has its downsides. The first downside is that with anything that involves a computation and the internet, about a million geeks will immediately seek to decode the system. I am not much of a tech guy so I don't know how to decode such a thing, but I have been around long enough to know that whenever there is a system, someone will always try to game it. Whether it be by outright deduction or through inductive reasoning based on observing who gets rewards and looking at their past behaviors, the system will not be a mystery for long.
Second, any fully objective system that gives rewards for good behavior must be tested thoroughly for fairness. In devising exactly how a member becomes ranked as a "good" member, it becomes necessary to scrutinize the system for exceptions to the rule.
For example: an automated system that scrutinizes the ratio of posts-to-warnings/blacklistings and determines that a person who has never been warned/blacklisted over their last 1000 posts is automatically a "good member" ignores the possibility that this individual has been around long enough to know exactly how far they can push the rules without breaking them. Note that it is possible to get away with flaming in every single post as long as it is not outrageous. I have rarely seen someone blacklisted for saying something mildly flaming like, "You're an idiot." But under this (very) hypothetical example, a person could place this phrase in every post and still qualify as a "good" member.
This example can be expanded to include many potential exceptions to the rule:
• An objective system that takes number of posts into account fails to include the "good" member who posts infrequently and includes the "bad" member who posts frequently.
• An objective system that takes into account which forum a user posts in fails to include "good" or "bad" members who stick to only one forum.
• An objective system that takes into account how many times the user's posts trigger the "blam" filter fails to include the many users who purposely evade this feature.
• An objective system that gives weight to the length of an account is both under- and overinclusive, as it fails to account for "bad" members with old accounts and "good" members with new accounts.
I understand and concede that the "algorithm" is likely to take many more factors into account than these in order to minimize these negative effects. But I don't think that it is possible for any algorithm to anticipate and cover every exception to the rules. For this reason, the results of such an algorithm could end up being uniform but unfair.
Fully Subjective Systems
The problem with fully subjective systems is clear enough on its face that this section will be understandably and mercifully short. Fully subjective systems lead to several problems. In fact, the problems with a fully subjective system whereby ALL members could participate are so obvious that I won't even go into them. This section assumes that the fully subjective system at issue here is administered solely by people at the "Forum Ninja" level and higher.
The first problem with such a system is clearly evident in that this would create a TON of new work for moderators and up. This would add to their already heavy duties by forcing them to decide when and where to "tag" new users for "good" or "bad" marks. They would not be excited about such a system.
The second problem is one of fairness: an "average" member who happens to be friends in real life with the right people could easily receive credit over another, more deserving "good" member. This is not to say that the moderators are subject to such corruption, but rather to say that "birds of a feather flock together," and that it is much easier to notice someone who is hanging out with you at school or work every day than it is to notice an anonymous person on the internet.
The third problem with a fully subjective system is its unpredictability. An objective system is the same, day in and day out. But subjective input is subject to the inputter's emotions, whims, desires, and moods. While such a system might, on the whole, help to eliminate the unfair results caused by an objective system, it clearly is not ALWAYS fair, nor is it predictable.
The Combination and Conclusion
Undoubtedly, a system is needed to encourage good behavior and discourage bad behavior. But rather than choose the lesser of two evils in terms of a fully subjective or fully objective system, it seems better to include a combination of subjective and objective factors in the ratings system.
I know that I've been speaking somewhat metaphysically and unspecifically this whole time, and I apologize. This has been because I do not know the specifics of Achronos's proposed system and because I am hoping to speak broadly enough to cover many different types of behaviors and systems. But I think that it is clear that a combined system would be far superior to either other system in standalone form.
But to satisfy your thirst for specifics, here is just an example of a ratings system that would allow both factors to be included and could lead to much more uniform and fair results:
Example (NOT a suggestion!)
To get "bumped up to the next level" or to receive whatever rewards are contemplated for good behavior the following steps would take place:
• A SECRET, automated, and objective system weeds out users based on wholly objective factors. Users who have been blacklisted more than twice or warned more than 4 times are automatically ineligible. Users who have been active less than six months are automatically ineligible. Users whose median post length is under ten words are automatically ineligible. Users whose IP addresses/cookies/however they are tracked leads to more than two active, alternate accounts are automatically ineligible.
• From this list, moderators are asked to approve or disapprove members for "promotion" based on rigid, predefined rules issued by site administrators and actively supervised by Master Forum Mods. Users' most-often-forums-posted-in are matched to moderators who tend to moderate those respective forums to increase the likelihood that the moderator is familiar with the particular user. Moderators have indicated many times here in the Septagon that they are familiar with most users, so this should not be much extra work.
Anyway, that's just a small example of a system that would potentially be better than either a fully subjective or fully objective system, and could clearly use some tweaking and more thought. But I just wanted to throw it out there.
Feel free to post what you think about "ratings" systems - I look forward to your opinions.