Update flakiness script (#30)

* update test names

* increase default runs number

* use most common responses for each test id as comparison

* also use presentationNumber from API responses

* Update stress-test.mts

* logging cleanup - now with markdown!

* use a nice human label for workflow id in markdown

* output final summary as markdown table

* markdown reformatting progress (#34)

* WIP: markdown reformatting progress

* fix for header section

* some changes based on carmen feedback

* Total unequal % - carmen request

* remove old code comment

* remove old code comment

* link to header

* adding some additional notes to stress test readme

* add notes about using a personal non-network fork

* simplify join in formatResponses

* remove old settle tracking

we got this from p-limit

* Update stressor/stress-test.mts

* Apply suggestions from code review

Co-authored-by: Mx Corey Frang <[email protected]>

---------

Co-authored-by: cypress evelyn masso <[email protected]>

* Update stressor/README.md

Co-authored-by: jugglinmike <[email protected]>

* Update stressor/README.md

Co-authored-by: jugglinmike <[email protected]>

* Add try...finally for clearInterval

* use _.isEqual instead of JSON.stringify

---------

Co-authored-by: Mx Corey Frang <[email protected]>
Co-authored-by: jugglinmike <[email protected]>
Co-authored-by: Mx. Corey Frang <[email protected]>

.gitignore

@@ -1,4 +1,5 @@
 # Logs
+stressor/stressor-run.json
 logs
 *.log
 npm-debug.log*

stressor/README.md

@@ -15,8 +15,10 @@ It works sequentially through the list of test plans, completing the runs for on
 
 ## Running
 
-Run it with `npm run stress-test`.
-
-Running the script can take a while, as it is constrained by GitHub Actions availability and speed.
+1. It is prefered for you to run the stress test against your own personal "non-fork" of this repo (create a personal repo and push to it instead of using "fork" so it isn't part of the "network") to limit the number of action runs against the main branch.
+2. Update the stress-test.mts file `owner`, `repo`, and `defaultBranch` definitions near the top, as well as setting up the tests / matrix you want to test.
+3. Run it with `npm run --silent stress-test | tee some-output-file.md`.
+4. Running the script can take a while, as it is constrained by GitHub Actions availability and speed.
+Will need the occasional manual job restart on GitHub when the ngrok tunnel sometimes fails (maybe 1 out of 20 runs).
 
 Set an environment variable `DEBUG` to `1` or `true` to get extra logging

stressor/package.json

@@ -11,10 +11,14 @@
   "license": "MIT",
   "dependencies": {
     "@octokit/rest": "^21.0.1",
+    "@types/lodash.isequal": "^4.5.8",
     "jest-diff": "^29.7.0",
+    "lodash.isequal": "^4.5.0",
     "ngrok": "^5.0.0-beta.2",
+    "p-limit": "^6.1.0",
     "ts-node": "^10.9.2",
-    "typescript": "^5.5.4"
+    "typescript": "^5.5.4",
+    "word-wrap": "^1.2.5"
   },
   "devDependencies": {
     "@octokit/types": "^13.5.0",