Valve tackles dodgy devs cheating Steam review scores
Now only players who bought games directly through Steam will affect review scores.
Mark Walton - Sep 13, 2016 2:50 pm UTC
with 54 posters participating
In an effort to stop developers from artificially inflating review scores on Steam, Valve has changed the system so that reviews written by users who received a game via a developer- or CD-distributed licence key will no longer count towards the review rating. The change will also affect reviews from Kickstarter backers that receive a Steam key as part of their backer rewards.
The Steam review system was updated in May so that an overall review rating (positive, negative, or mixed) is displayed at the top of each game page. The idea was to better show the current state of a game, particularly as more of them enter early access and evolve over time. Unfortunately, Valve claims that many developers have been abusing the system by giving out keys to friends or paid-for review services in order to bolster the overall review score.
"The review score has also become a point of fixation for many developers, to the point where some developers are willing to employ deceptive tactics to generate a more positive review score," wrote Valve in a blog post.
"The majority of review score manipulation we're seeing by developers is through the process of giving out Steam keys to their game, which are then used to generate positive reviews. Some developers organise their own system using Steam keys on alternate accounts. Some organisations even offer paid services to write positive reviews."
Valve lays out new rules and guidelines for Early Access developers
Few Steam "Early Access" games have actually been completed
Valve estimates that "at least 160 titles have a substantially greater percentage of positive reviews by users that activated the product with a CD key, compared to customers that purchased the game directly on Steam." While it admits there can be legitimate reasons for the discrepancy—including strong audiences outside of Steam and popular Kickstarter campaigns—for the most part it says "the abuse is clear and obvious."
Valve will not be removing the ability for developers to issue Steam keys. But in cases where it finds developers have duplicated or generated reviews in large batches, or found review accounts linked to the developer in question, it "will be ending business relationships." Gamers that purchase or receive a game outside of Steam will still be able to write a review, but it will no longer contribute to the overall review score.
"This does mean that the review score category shown for about 14 percent of games will change; some up and some down," continues Valve's blog post. "Most changes in the review score category are a result of games being on the edge of review score cut-offs such as 69 percent positive or 70 percent positive. A change of 1 percent in these cases can mean the difference between a review score category of 'Mixed' and 'Positive.' About 200 titles that only had one or two reviews will no longer have a score at all until a review is written by a customer that purchased that item via Steam."
While the change is largely positive for consumers, developers that have large fanbases outside of Steam or rely on crowdfunding could be negatively affected. Valve is also working on other issues that include a mismatch between the most helpful reviews and the review score, which can be caused users intentionally trying to skew the system, or where reviews are helpful "simply because they are funny."
Mark Walton Mark is Consumer Editor at Ars Technica UK by day, and keen musician by night. He hails from the UK, the home of ARM, heavy metal, and superior chocolate.
Email mark.walton@arstechnica.co.uk // Twitter @markalexwalton
DarkSkyForever Ars Tribunus Militum
Valve should also consider placing a "This user received a copy of this game for free" message on the top of these reviews, like they do for early access reviews.
Chmilz Ars Scholae Palatinae
Totally OK with this. I've looked at a number of dubious titles that had a very favorable rating only to look elsewhere and get vastly differing opinions. I'm sure those are the titles that will be affected.
Otus Ars Tribunus Militum et Subscriptor
DarkSkyForever wrote:
But that's not what it means. I have a lot of Steam games I haven't bought from there and almost none of them I've received for free.
Mostly they come from publishers' web sites, other third party stores selling Steam keys and from Humble Hundles. I've paid for all of those.
UserIDAlreadyInUse Ars Tribunus Militum et Subscriptor
The reviews written for laughs can be a hoot at times; Valve could add a "Not helpful, but amusing" button for those that want to show their appreciation for a well-written but irrelevant review.
Edit: Thanks, Voldenuit. I'm not one to comment often on Steam so I didn't know that existed already. Good to know.
Last edited by UserIDAlreadyInUse on Tue Sep 13, 2016 10:09 am
Voldenuit Ars Praefectus
I don't know if silencing Kickstarter users is the best way to go about stacked review scores.
Seems a bit drastic if Valve is silencing users to fix dodgy dev behavior.
UserIDAlreadyInUse wrote:
There are already separate categories for 'helpful' and 'funny'. You can tick one and not the other.
It's a big hammer, and I hope they can find a way to reduce its impact; but stopping spam reviews from inflating game scores has to be the top priority.
Shudder Ars Legatus Legionis
I get the rest, but what's with the Kickstarter reviews? Something like the new Torment will probably be largely bought up-front by the backers, which I'd trust more than a bunch of people who bought it on their own after the launch. The kickstarters - If it's good you're going to know about it, if it's not they'll be the first to trash it.
24025 posts | registered Oct 23, 2000
Otus wrote:
I hear ya. I've Kickstarted a few, which means I paid money for them. And I also buy from gog.com.
Valve may have limited data on how third party keys are transacted. Perhaps assigning different serial numbers to developers and resellers could help?
KAL1989 Ars Scholae Palatinae
The problem for Valve is identifying place of purchase.
The only way I can see that happen is if Valve implements a unique coding system for the keys where steam keys from various sources have distinct identifiers that are distinguished from kickstarter, early access, third party website, etc.
That sounds like a notable undertaking that requires cooperation from both Valve and the distributors. Don't know if that will happen. We have that for physical products if they come with unique items that are distinct from other distributors, but physical and digital goods are quite different in that digital goods are not inherently unique in that sense.
Edit: It is worth pointing out that this policy by Valve is to take away the ability for distributors I.E. game creators to game the system. Giving distributors unique codes usually means they must come from somewhere. That somewhere being the developer/publisher. Giving them control to determine the source of the purchase puts us back in square one for those with intent to game the system. It seems like an enormous undertaking to police a review system for post sale.
Last edited by KAL1989 on Tue Sep 13, 2016 10:10 am
Psyker01 Smack-Fu Master, in training
I wonder it will effect games you buy from bundles like the Humble Bundle. I have an lot of games I have brought from bundles and it would be bull if I couldn't review them
9 posts | registered Jul 2, 2016
Psyker01 wrote:
All games purchased outside of Steam are likely to be affected. Valve can only verify legitimate purchases off its own store front, unless they already planned for this in advance.
As an FYI, you can still review them.
Your positive or negative opinion numerically will not affect the percentage score based on buyer feedback. Your review will just sit there as a review.
Rhialto Smack-Fu Master, in training
Is there a system in place preventing buying a game, writing a review then asking for a refund? Will the review remain in place? Hope I'm not giving tips to devs. I believe Steam would adjust.
ZenBeam Ars Scholae Palatinae
Shudder wrote:
I get the rest, but what's with the Kickstarter reviews? Something like the new Torment will probably be largely bought up-front by the backers, which I'd trust more than a bunch of people who bought it on their own after the launch.
If you're part of the KickStarter, the reviews don't mean much to you beyond bragging rights for a game you supported.
If you're not part of the KickStarter, but are part of the general game-buying population, the non-KickStarter reviews will tend to have more meaning to you, since they're also from the general game-buying population.
1212 posts | registered Apr 8, 2014
Lonyo Ars Tribunus Militum
I doubt it's possible to have large scale buying, reviewing, refunding in place because Valve would stop the cards from being used to purchase games, or stop refunding games on the accounts, so you would need a large number of accounts and bank cards. The bank side would probably be more hassle than it's worth. Plus the additional effort of buying and refunding the game.
It's just not a viable method for large scale abuse IMO.
pappypappy Ars Scholae Palatinae
Rhialto wrote:
It definitely was/is a potential issue. I can't recall whether Steam have closed the loophole. It's a tough one though, as a genuinely broken game should get bad reviews even if you need to refund. But it's very hard to prevent abuse using the refund system, especially if there are companies willing to spread it across many dummy accounts.
1461 posts | registered Jul 9, 2007
nononsense Ars Scholae Palatinae et Subscriptor
They mark reviews that have been made by people who receive games for free. This is my favorite NMS review:
pappypappy wrote:
That's not really an issue, though. If your game has a significant number of positive reviews on accounts that refunded it, that's a pretty blatantly obvious warning sign.
xdeathknightx Smack-Fu Master, in training
Hmm this is a hard nut to crack. On one hand ensuring everyone that bought the game on another site can't review the game is a tad much (and reeks of telling people to only use their store), but they do need to somehow curb the fake positive reviews.
What does surprise me is that they are coming down on these reviews and, to my knowledge, never did anything against steam curators who were publically ousted as cheating the system.
xdeathknightx wrote:
People who bought the games on other sites can still review them, and their reviews are still listed. They just aren't used for calculating the game's overall score.
Frosty Grin Ars Legatus Legionis
It's important to note that reviews like that are also hidden by default - you need to opt in to be able to see them.
16219 posts | registered Apr 24, 2006
Korpo Ars Praefectus
Good. Now to get Amazon to crack down on the "I received this product for free or at a discount in exchange for my honest and unbiased review" nonsense.
jwbaker Ars Praefectus
Why is Goat Simulator pictured? Is it one with notoriously biased reviews?
Magnus Dredd Ars Scholae Palatinae
KAL1989 wrote:
I'm pretty sure that to third party stores, a key is simply a string of text that's provided them by Steam. If different third party stores are provided different lists of keys, it would require no support on their end.
On the other hand it might be important to require them to use keys they are issued for the purpose they're provided for.
Examples: Steam generated "kickstarter" keys are only available to those who pay to back a kickstarter campaign and not released in any other way. "GOG" keys, available only for those who pay money on GOG and not for promotional use. ...and so on.
I don't what method Steam uses to generate keys. However, depending on the method used, it could be fairly easy to implement different keys for different vendors.
Example: If they use SSL certificates or some other such crypto strings as the basis for their keys, they could implement multiple key-spaces. In effect, a system that supports multiple SSL certificates or crypto strings (domains).
To do this:
Their online platform would need to support metadata that allows for games to be tagged as being purchased from differing sources.
They'd need to implement a system which could determine which of the key domains (Kickstarter, GOG, etc) that the license key belongs to.
They'd need to modify their validation routines to use this information to validate the license using the correct SSL certificate or crypto key.
They'd need to communicate which domain the license key is a part of to their online service in order to tag it appropriately.
Lastly, the algorithm that tracks ratings would have to be adjusted to support multiple domains.
While this wouldn't be quick to implement, it's certainly within the realms of possibility.
885 posts | registered Sep 3, 2003
Balderstrom Ars Tribunus Militum
Valves "mixed" rating is ambiguous and somewhat unfair, I think it ranges between 69% and ~40% - which aren't even close.
100%: Flawless Victory
80-99%: Overwhelmingly Positive
60-79%: Mostly Positive
40-59%: Almost Positive
20-39%: Negative
00 -19%: Critical Failure
Also, the article is wrong here,
The Steam review system was updated in May so that an overall review rating (positive, negative, or mixed) is displayed at the top of each game page. The idea was to better show the current state of a game, particularly as more of them enter early access and evolve over time.
Steam recently updated it's review system so a "Current|Recent" rating is provided. Steam always had the "Overall".
Last edited by Balderstrom on Tue Sep 13, 2016 10:47 am
jwbaker wrote:
Why is Goat Simulator pictured?
Because everyone loves goats.
slymenstra hymen Ars Tribunus Militum
Same issue crops up on IMDB, every low budget movie has friends and family pumping up movies not even good enough for late night TV.
Magnus Dredd wrote:
But where do the keys come from?
Who generates them? Valve or the publisher of the game? I know they originate from Steam, but who opts to get them distributed?
From what I am aware of, sites like GMG get keys from publishers, who get keys from steam. Therefore you are expecting the publisher to legitimately tell Valve the source of the sale. That doesn't work for people who intended to game the system to begin with ergo my last paragraph.
It is not so simple. Valve has to be the facilitator for all key distributions, not just the source of creation for this to work. Valve doesn't trust publishers/creators to do the right thing, that is why this policy was created. They have to take over the roll of facilitator for key distribution as well as the source under the current system. That is the part that makes it an enormous undertaking. They would need to negotiate with all the third party suppliers to generate their own keys for that to work and with so many stores, is Valve interested in negotiating with all of them? Or does Valve start their own Approved by Valve store partnership system? It is a complicated mess.
Balderstrom wrote:
If I'm not mistaken, "Mixed" is used to signify that the ratings don't follow a standard bell curve and instead follow more of a "bathtub" distribution of ratings.
In effect, it implies that users either love or hate the game. Sometimes this is a clear sign of ratings manipulation.
Perhaps, but it implies negative, even its color is shaded towards negative.
slymenstra hymen wrote:
I don't think we need conspiracy theories to explain the fact that a lot of people have bad taste in film.
rafadavidc Ars Tribunus Angusticlavius
Because it's hilarious and AWESOME
dmoan Ars Praetorian
Plus i doubt pumping up ratings in IMDB really makes any $$ for these low budget movies. People use critic reviews (or RT) or word of mouth to determine whether to watch these movies in theater not IMDB.
Gledster Seniorius Lurkius
This is already having an effect on smaller devs. See Simon Roth on Twitter: https://twitter.com/SimoRoth/status/775620744399192065
Another Dev also has a take on how it's going to affect smaller developers more: https://twitter.com/randomnine/status/7 ... 7573456896
1 post | registered Sep 15, 2010
bicarb Ars Tribunus Militum
I agree, the mixed/negative is borked. I saw a post from an early access game begging for folks yo go give positive reviews because reviews have slowed as they are close to going gold, after getting 4 negative reviews in one week (hundreds previously, maybe more) they were showing as mixed reviews because those were the only 4 that week. It's so bad that I'm mostly in the habit of ignoring it & reading some reviews if it looks at all interesting
onkeljonas Ars Tribunus Angusticlavius
How do you get a bell curve distribution of a binary rating?
The review system is love/hate by design.
Head Pirate Ars Scholae Palatinae
This is an interesting thing to devote effort to, as both steamspy and internal steam reports show that review scores have only a minor effect on a games sales, and even then it's only with review scores in the 4 to 7 range that we see any real impact. Basically, if you're game rates under 4.0 (or 40% of people like it) it looks like people still buy the game, but no more frequently then if it rated 2 (or 20%). At the same time, once you get over 7 (or 70%), score doesn't seem to have an impact on sales. There is a clear positive correlation between sales and scores from 4-7, but from 1-4 and 7-10 it completely disappears and it looks just like noise.
Plus 160 games game, some of which steam said themselves in the press release might have valid reasons seems like a small problem.
Not saying this is a bad thing at all, but a much clearer driving force behind how well your game sells on steam is discounts, and the practice of listing a games "regular price" on steam as something much higher then it's retail price even on a company's web site so that you can show a dramatic "discount" is widespread, well documented, and effect way more people's buying decisions.
2017 posts | registered Jan 6, 2016
ZenBeam wrote:
Kickstarter backers aren't legitimate users?
They put their money down to support development, they tend to have high expectations.
bicarb wrote:
That's not how it works. "Mixed" reviews on steam mean that 41% to 59% of the reviews are positive. Recent reviews, as of the new system, count for 1 to 3 times the numbers of reviews based on an automatic algorithm that checks how often the game is updated.
If I game has, say, 1000 negative reviews in the past and 100 positive"recent" reviews even in the best possible case that review is "overwhelmingly negative". Also if you are every skeptical and want to judge a game by it's pervious mistakes you can just click on the store page, where an un-bias "overall" review score is separated from it's "recent" review score.
In most cases, this is a very good thing for consumers. Games get much better as they are patched and the develop shows support, and when I buy a game I want to know if the reviews of the game in it's current state are different from the reviews of the game previously because that tells me it has strong developer support. And I don't know why I would care about how the game was 2 years ago unless I have a time machine.
Last edited by Head Pirate on Tue Sep 13, 2016 1:54 pm