Role community score dramatically affects overall score, often negatively

[geerlingguy]

Bug Report

SUMMARY

I was glancing at a few of my roles on Galaxy as I was doing some manual cleanup work, and noticed that one role in particular (geerlingguy.java) suddenly had an overall rating of 2.5/5.

Now, I don't like Java as much as the next person, but I've spent dozens of hours on that role, making sure it works across a huge variety of OSes and versions, and works with a lot of other major Java projects downstream... and was surprised to see such a poor rating!

Anyways, it seems that there has been only one community review so far... and that review was just "0, 0, 0, 0, 0" for everything. I can imagine I could quickly put the kibosh on any new role that is submitted to Galaxy by going in and rating it as 0's across the board, thus dropping it's overall score to at most 2.5/5, then making it much less likely someone would consider trying it out.

STEPS TO REPRODUCE

1. Go to any role on Galaxy with 0 community ratings.
2. Rate it as '0' and 'N' for quality scores.

EXPECTED RESULTS

This malicious rating should not dramatically impact the chances this role would be evaluated by those using Ansible Galaxy.

ACTUAL RESULTS

This single malicious rating drops the role from a possible 5.0 rating (assuming the linting passes muster) to a maximum 2.5 rating, turning the rating value orange, which visually indicates the role is not that good (to a user evaluating roles on Galaxy).

[newswangerd]

@geerlingguy one of the goals when we set up this system was to make it so that it's possible for people to filter out low quality roles on Galaxy, so being able to drop someone's score to 2.5 with a 0 is basically the intended behavior.

With that said, this is clearly an abuse of the system, which is unfortunate. Ideally we want a system where people do have the power to indicate when something is low quality, but still make it difficult to abuse the system. Right now the score on the search page is calculated by taking the average of the community score and the quality score (if both of them exist), which makes it so that one person can drop a repo with a quality score of 5 down to an overall rating of 2.5. With all of this in mind I'd like to propose the following solution:

Rather than use an unweighted average for the combined score, use a weighted average that's proportional to the number of surveys that have been submitted:

Where

• q = quality score
• c = community score
• x = the number of surveys submitted (up to 3)

This would result in the following for a role that has a quality score of 5 and community score of 0:

• 0 surveys -> 5
• 1 survey -> 4.17
• 2 surveys -> 3.34
• 3+ surveys -> 2.5

This would make it so that receiving one malicious review would still allow for a role to maintain the green check, but a pattern of several negative reviews would seriously affect the score. How does this sound to you?

[geerlingguy]

@newswangerd - That sounds better, because it would take at least two or three people working together to intentionally harm another role's ranking (and this kind of brigading would likely be easier to counteract by finding patterns with accounts on Galaxy).

User reviews are hard in any context... especially when they can be gamed (because, like spam, it's just a back-and-forth game of trying to outsmart the other party), so I don't envy the Galaxy team the work!

[geerlingguy]

Looks good to me!