[RFC] Stale bot

[mratsim]

It seems like Nim now has a stale bot.

Advantages:

• Less open issues

Disadvantages:

• Feeling of having issues "dismissed"
• Unaddressed issues

Opinion

I don't think this is a right approach when there are so few core contributors to Nim:

• We don't have a large number of contributors dedicated to closing issues with proper fixes.
• Most issues are stale because of lack of time, not because they are solved.
• Someone interested enough will look in the open issues to find something to work on, for example documentation or good first issue. Closed/Stalled and Closed/Fixed are extra barriers for those first contributions.
• On the benefits/disavantages scale, the stale bot is definitely a heavy minus for me.

[disruptek]

Please make suggestions for a better bot. I am happy to contribute it.

[timotheecour]

Stale bot optimizes for a vanity metric of minimizing number of opened issues, but does not actually contribute to fixing anything, and makes things harder / more confusing (eg increases likelyhood of duplicate issues since you're not going to check into closed issues before submitting an issue). Old issues can be still relevant even if the person who authored them is no longer an active contributor.

Please make suggestions for a better bot. I am happy to contribute it.

harder to implement than a stale bot but much more useful would be the following:

• implementing nim's equivalent of D's digger (see https://github.com/CyberShadow/Digger), which would allow (among other things) building nim from an arbitrary git hash; it's not hard to implement but requires taking care of finding the correct csources version for a given git commit hash, as well as skipping over bad commits
• implementing nim's equivalent of D's dustmite (see https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/) which minimizes test cases.
  I've used it extensively when I was using D (as have many others), it's an incredibly useful tool for fixing bugs, in particular in combination with git bisect.

these can be simply command line tools, but could also be integrated in a github bot, eg:

• automate git bisect on opened issues (for issues following a template) to figure out when was a regression introduced
• automating running test embedded in an issue to figure out whether problem still occurs on devel

links

• [TODO] nim-dustmite source code minimization tool for automated bug reduction #8276 nim's dustmite
• bisect bot juancarlospaco/nimbug#8 bisect bot

[genotrance]

tissue already can download and test issues with working snippets, post nim_temp stack traces as comments to Github and create testament test cases and PRs out of snippets once an issue is fixed. I know @narimiran used it off and on but it will be great to expand this method of root causing and fixing issues quickly. It can detect various flags mentioned in issues (ssl, threads) but is 2 years old and obviously could do with some improvements if people used it and provided some feedback.

All this is only possible with issues that have good structured information and snippets to reproduce a problem. Getting to that point is not easy though and time invested in minimizing issues sufficiently to create good issues should not be devalued and marked stale. I also agree that issues without sufficient steps to reproduce should get flagged and rejected with good guidelines on how to create minimal snippets.

Automation can be valuable:

• If an issue is marked as waiting for info or feedback or equivalent by maintainer, and there's no update in a month, then sure, post an auto-reminder and then auto-close it.
• When issue is opened, bot can check if it has a valid snippet (based on tissue or similar) and if not, it can post a comment saying that issue might get rejected. In a follow-up run, it can auto-close such issues.
• Bisecting is only sensible if it is a true regression and might need some user intervention before a bot can do it automatically.

I'm sure there are other such automation tasks that can keep the issue count reasonable. I do agree that 1.7k open issues and 140 PRs is daunting and you want to get rid of useless issues but I am not totally convinced with using stale time as the metric.

Lastly, I did some gitting and the last 100 commits had 20 odd authors, last 1000 had 80 odd authors and the last 10k commits had 400 odd authors (ballparking since some authors show up multiple times with different names). We aren't so short of contributors. Also, the last six months has seen 1000 commits with 260 of them explicitly referencing an issue fix. Perhaps 30% of contributions are bug fixes. That falls down to 20% over 10k commits. Maybe we should reduce the issue count by focusing more on fixing open bugs than new features for a few months.

[Skrylar]

Oh good, inbox zero bots. Why bother to solve anything when you can just #wontfix.

[Clyybber]

@Skrylar I think in this case upvoting this issue means being against Stale bot, at least thats how I interpreted it.

[Skrylar]

@Clyybber i received an auto response on an issue i bothered to box up a neat little test case for. on top of the code still doesn't work 3 years later (although hey, the error is different now!) a human couldn't even be bothered to #wontfix it by hand.
But then I haven't been able to use the forums due to a bug for about as many or more years, and that's never been fixed either, so I'm not really sure why we even have an issue tracker.

[Clyybber]

@Skrylar To be clear, I hate stalebot and would like to see it removed :)
I think your issue shouldn't be #wontfix'ed. AFAICT stalebot is intended to close issues that even the author doesn't care about anymore, but I think its stupid regardless.

[timotheecour]

so what's the conclusion? can we safely ignore the warning This issue has been automatically marked as stale because it has not had recent activity. If you think it is still a valid issue, write a comment below; otherwise it will be closed. Thank you for your contributions. ?

[mratsim]

Another thing is that, we strongly request issue raisers to give us a minimal reproducing example. It may takes a couple hours to get one in complex codebases or when it touches complex parts of Nim.

Closing those after 3 months is not respectful to people hard work (point raised by @arnetheduck)

[zah]

The list of open issues on Github is a useful database. When you start working on particular sub-system in Nim, it's often useful to take a look at the list of open issues that might be related. This helps you understand the overall requirements better and sometimes leads to more comprehensive fixes, closing several issues at once.

Now, if we really want to reduce the number of Github issues for whatever reason, I think a better strategy is available. If the Nim test suite had a category of "known problems", we can close issues by importing the reported snippets into the known problems set. These would be tested in CI and we'll be aware when accidental fixes resolve some old long standing problems.

[dom96]

I wonder how many commenters here also keep hundreds of tabs open in their browser because they are afraid that one of them might come in handy in the future? Then after years of this they end up with the same tabs opened over and over again, the old tabs long lost but there to give you comfort.

Unlike tabs, when an issue is closed it is not lost forever, so I don't buy the "it's useful to see issues that may be related" argument. You'll see these issues even if they have been closed and can very rightly ask for them to be reopened.

I wonder if everyone is on the same page when it comes to how this Stale bot should operate, personally here is what I envision:

• When there is no activity on an issue in 3 months the stale bot comments with "This issue hasn't been active in 3 months, comment or add a DoNotClose label otherwise I will close this issue after another 3 months"
• Anyone can comment on the issue to stop it from being closed.
• For any issues that have been closed by the Stalebot, anyone should be able to comment on the issue with a @Stalebot reopen command to reopen it.

I think that's perfectly reasonable. It is a means of prioritising our issues and that is what's important here. Issues which nobody cares about anymore should be forgotten, just like your hoarded tabs. Issues which multiple people agree are important and should never be closed should get a DoNotClose tag.

I probably have issues which I opened 4 years ago, if nobody else has ran into that issue then they will not comment on them to stop them being closed, in that case it's right that the issue is forgotten. If someone has run into that issue then we will get another piece of signal that it affects others, the issue won't be closed, and maybe someone will be inspired to fix it.

[arnetheduck]

the tissue tool, while valuable, is a little bit too far away from sight - it would be more useful that issues that have a clear automated repro should be included in the test suite and counted as "known broken" - this helps keep an eye on a statistic that is in many ways more important than the number of tests that worked: the number of tests that fail and are known to fail - in particular, this would give a number to the low quality of certain nim features which usually helps focus attention on them.

in such a world, tissue would import the issue from the ticket and add it to the test suite in repo automatically - then it doesn't matter that much if the issue itself stays open or not.

[jibal]

I don't see any value at all to auto-closing issues. There's a Low Priority tag--why not use that? Close means something very different. (And if a bot started closing my browser tabs on me I would be quite unhappy ... not that I think that analogy is valid).

Issues which nobody cares about anymore should be forgotten

Lack of a recent comment does not mean that nobody cares about it.

[narimiran]

The stale bot now only works on pull requests, notifying the authors of the PRs which haven't had any activity in last two years to please rebase on the latest devel.

[Araq]

One minor addition to the discussion: We don't know how many bugs remain unreported because there are 1700 open issues and reporters cannot be bothered to check for a duplicate of their bug. That would mean we care more about preserving history than about bugs newcomers actually encounter.

[arnetheduck]

If good (reproducible) issues were automatically added to a known-fail test-suite, duplicates wouldn't matter: the fix would make all relevant tests pass, thus resolve all duplicates.

[Skrylar]

there isn't much automatic about this; someone has to fit the test in to a framework and spot check it isn't trying to compile abuse code.

[dom96]

I think the general consensus is:

• community doesn't want this
• core devs do

I think this may be a result of things that only the core devs have experienced, but I also believe that listening to the community is a good way to develop Nim so I support the decision to follow what appears to be the community consensus.

I think we can close this now.

[MatthewScholefield]

Just to add a perspective, as an outside developer looking at the project, I am discouraged from creating new issues because looking at the 1.8k open ones, I feel that it's like a drop in the ocean and will likely be ignored (which of course there is no obligation for it not to be ignored so this is totally fine).

Not sure if this is a benefit or drawback, but that's my perspective on the implicit effect of having many open issues.

Also, having 1.8k issues (and of course always growing), I don't think there's much hope for most issues to get solved as it is a constant ongoing battle. Given this assumption, it is my inclination that issue prioritization is one of the most important details to get right. We already have data points like number of comments and last updated, but having one more data point of "this issue still affects at least one person" to me is very valuable.

So, to me having some sort of stalebot to either close issues or mark them with a tag would seem to be valuable.