2018: a year in moderation

Question

It's New Year's Day in Stack Exchange land...

A distinguishing characteristic of these sites is how they are moderated:

We designed the Stack Exchange network engine to be mostly self-regulating, in that we amortize the overall moderation cost of the system across thousands of teeny-tiny slices of effort contributed by regular, everyday users.
-- A Theory of Moderation

While there certainly are Moderators here, a significant amount of the moderation is done by ordinary people, using the privileges they've earned by virtue of their contributions to the site. Each of you contributes a little bit of time and effort, and together you accomplish much.

As we enter a new year, let's pause and reflect, taking a moment to appreciate the work that we do here together. To that end, here is how the moderation done here on Geographic Information Systems breaks down by activity over the past 12 months:

                 Action                  Moderators Community¹
---------------------------------------- ---------- ----------
Users suspended²                                  1          6
Users destroyed                                  11          0
Users deleted                                     1          0
Users contacted                                  10          0
Tasks reviewed³: Suggested Edit queue           545      6,987
Tasks reviewed³: Reopen Vote queue              383      4,034
Tasks reviewed³: Low Quality Posts queue        143      4,185
Tasks reviewed³: Late Answer queue               83      2,281
Tasks reviewed³: First Post queue                84     10,112
Tasks reviewed³: Close Votes queue              211     12,802
Tags merged                                     117          0
Tag synonyms proposed                            46          5
Tag synonyms created                             47          0
Tag highlight language set                       10          0
Questions unprotected                             3          0
Questions reopened                              538        138
Questions protected                              13         33
Questions migrated                               38          0
Questions merged                                 29          0
Questions flagged⁴                               10      1,051
Questions closed                              3,843      2,255
Question flags handled⁴                         361        700
Posts unlocked                                    7          8
Posts undeleted                                  38        273
Posts locked                                     47         98
Posts deleted⁵                                1,767      7,192
Posts bumped                                      0      8,731
Escalations to the CM team                        3          0
Comments undeleted                               19          0
Comments flagged                                  7      3,243
Comments deleted⁶                             3,127      7,648
Comment flags handled                           548      2,702
Answers flagged                                  93      3,586
Answer flags handled                          3,023        656
All comments on a post moved to chat             14          0

Footnotes

¹ "Community" here refers both to the membership of Geographic Information Systems without diamonds next to their names, and to the automated systems otherwise known as user #-1.

² The system will suspend users under three circumstances: when a user is recreated after being previously suspended, when a user is recreated after being destroyed for spam or abuse, and when a network-wide suspension is in effect on an account.

³ This counts every review that was submitted (not skipped) - so the 3 suggested edits reviews needed to approve an edit would count as 3, the goal being to indicate the frequency of moderation actions. This also applies to flags, etc.

⁴ Includes close flags (but not close or reopen votes).

⁵ This ignores numerous deletions that happen automatically in response to some other action.

⁶ This includes comments deleted by their own authors (which also account for some number of handled comment flags).

Wishing you all a happy new year...

Answer 1

It was claimed in comments moderators in GIS SE have played a major role closing questions than the community, while in other Stack Exchange sites (Stack Overflow SE, Mathematics SE and Database Administrators SE) it is the opposite:

I don't believe we can compare GIS to Stack Overflow and Mathematics because they are much larger communities than GIS. In such communities the proportion of questions per moderator is much greater:

Action (in 2018)            SO  Math    CV   GIS   DBA
----------------------- ------ ----- ----- ----- -----
Questions asked          2889k  216k   38k   22k   15k
Number of moderators        26     9     7     8     4
----------------------- ------ ----- ----- ----- -----
Questions per moderator   111k   24k  5.4k  2.7k  3.7k

Cross Validated (CV) SE was added in the analysis which is a site I used to participate and has more or less the size of GIS.

Now let's see the impact of each actor in closing, with metrics standardized by the number of questions asked in 2018:

Action (in 2018)                       CV   GIS   DBA
------------------------------------ ---- ----- -----
Questions asked                       38k   22k   15k
------------------------------------ ---- ----- -----   
Q closed by moderator (%)            23.2  17.8   3.5
Q closed by community (C) (%)         5.9  10.4    19
Q closed by both (total %)           29.1  28.2  22.5
-----------------------------------  ---- ----- -----               
Close votes queue (C) / closed Q (C)  8.5   5.7   4.2

Data is not conclusive. Moderators from GIS SE closes much more questions than mods in DBA, and considerably less than mods in CV. The same in relation to the community, but with an inverse order (DBA community closes the most, then GIS, and CV). All 3 websites close more than 20% and less than 30% questions in relation to questions asked in 2018.

Regarding DBA SE, I started to participate there recently as a spectator. I know guys there are affectionate for keeping a high signal to noise ratio in content; they sometimes promote campaigns to purge content with large involvement from the community (see here and here, for example). It helps explaining why the 'community' there closes much more in relation to moderators. They also have the lowest ratio between 'close votes queue by community' (votes casted in the review queue) per 'closed questions by community' (4.2) which means they are more rigorous in casting 'close' votes (rather than 'leave open' votes) than in GIS and CV.

About CV SE I was once active there. The highest ratio among 'close votes queue' (community) per 'closed questions' (community) is in line with a feeling I used to have that many users were prioritizing welcoming newcomers over keeping a high signal to noise in content. I have a hypothesis that such behaviour had a side effect there which overloaded moderators, the ones responsible for closing the majority of their questions in 2018 (23.2% versus 5.9% from community). Also, for those interested, CV has the third worst answer rate (proportion of questions answered) across the entire Stack Exchange network (around 62%, they recognize it is sort of a problem for them; GIS and DBA have 80%).

I acknowledged data used for this discussion is inconclusive, but my personal opinion is that moderators in GIS SE are doing a good job with respect to 'closing' and that the community could do more.

Sometimes a mod will close a question unilaterally as unclear and if I understand it and want to post an answer (given the Q is on-topic), I usually get a positive feedback for reopening. Commenting in favour of reopening explaining why it shouldn't have been closed in the first place may also work.

Another thing we could do better in my opinion is tying duplicate questions more frequently; there are lots of unclosed dupes in popular tags, for example, r (only 2.5% of closed dupes), postgresql (2.3%) / postgis (2.6%), geoserver (3%), etc. Compare to tags (which I consider to be benchmarks) qgis (5.5%), arcgis-desktop (4.7%), lidar (10.4%). And one interesting thing about qgis (our top tag) it has 8 users with gold tag badges (no other tag has this amount of 'gold users') who can unilaterally close questions as duplicates, but none of them is top ranked in the close review queue (i.e., it does not help explaining the 5.5%).

The pattern of voting among users will vary which is expected (people diverge/disagree, that is why there is the vote count thresholds before a decision is reached for closing or leaving open a question). But I have a suspicion there is at least one case of robo reviewing in GIS SE which is affecting/skewing our data on this matter. Small sites don't have review audits like the ones in Stack Overflow, and it would be nice if we had them.

---

A cliché in Stack Exchange is that we are all volunteers (true); everyone do what it is possible and we move on.