Trac wiki migration, improve markup conversion code

[soehms]

A. According to the sage-devel thread of the same title, the following tasks must be completed:

1. • Create links with URLs of Trac tickets (to be removed after the migration of the tickets).
     Addressed in PR Add links to Trac tickets for the wiki migration #15.
2. • Add table alignment instructions.
     Addressed in PR Add table alignment instructions #16.
3. • Sections with fragment names
     Example: == Legacy sage-trac Account Request == #legacy-account-request following section Account Names Mapped to Real Names in Home.md. Addressed in PR Fix for item A.3. of issue #18 #21.
4. • Various bad links
     Examples: PageOutline and https://groups.google.com/forum/#!forum/sage-packaging in Distribution page. Addressed in PR Fix for items 4 and 7 and completion of item 5 of issue #18 #19.
5. • Various missing links
     Examples: [wiki:symbolics/maxima] and [https://en.wikipedia.org/wiki/Symbolic_computation] on Symbolics page
     Addressed in PR Add missing links #17
6. • Lines containing nested bullets are not converted.
     Examples: Missing Trac ticket links in nested bullets after abs_integrate on Symbolics page. Addressed in PR Fix for item A6 and A8 of issue #18 #23.
7. • Non working queries to the Trac ticket database.
     Examples: [query:?status=needs_review... on Symbolics page. Addessed in PR Fix for items 4 and 7 and completion of item 5 of issue #18 #19.
8. • First lines and headings of pages are omitted.
     Examples: Distribution page (heading is - Distribution and packaging of SageMath -). Infrastructure page (first line is This page aims to keep information public about Sage's infrastructure.)
     Addressed in PR Fix for item A6 and A8 of issue #18 #23.
9. • Global references (by URL) to Trac tickets should be localized in order to be converted more easily to their corresponding GitHub issue after ticket migration.
     Examples: Reference to ticket 25206 on Cygwin64port page. Addressed in PR Fix for items 4 and 7 and completion of item 5 of issue #18 #19.

B. Tasks to be performed manually after wiki migration:

1. • Replace explanations and references to the Trac wiki syntax with explanations and references to the GitHub wiki syntax.
     Examples: Reporting a bug

C. Tasks to be done after successful ticket migration:

1. • Remove the link to the Trac ticket so that they point to the corresponding GitHub issue.
2. • Replace requests to the Trac database with requests to GitHub issues (where possible). But maybe it's better to keep the queries on Trac and add corresponding queries on GitHub.
3. • Find analogues to Trac macros and insert them at the corresponding hints produced be the script in the new GitHub wiki.

[soehms]

Remarks

Concerning A. 3.

The script doesn't recognize headings that have an anchor (#legacy-account-request) in the definition. Should we just ignore these anchors? Apparently, GitHub Markdown defines anchors automatically to each heading. A link to the above anchor would look like [#legacy-account-request Legacy sage-trac Account Request] in the Trac wiki (but in this case it is not even used) and would work as [Legacy sage-trac Account Request](#legacy-account-request) in GitHub Markdown, as well. So it should be sufficient to implement the latter conversion.

Concerning A. 4.

Unfortunately Trac uses the syntax [[Name]] with two meanings. The first meaning is a relative link in the wiki. The second meaning is a macro that implements a plugin to the wiki page. The PageOutline macro creates the table of contents plugin for the current page. It seems that the only way to distinguish between these two meanings is to check the name against a corresponding list, which must be created before starting the migration.

The next question is: what to do with the macros? Do you just ignore? Or do they have useful replacements in GitHub? Perhaps we can turn them into links to the appropriate Trac documentation for now.

Concerning A. 6.

This is caused by the following line in trac2markdown of migration.py:

        if not line.startswith('    '):

I have no idea about the purpose of this line. Removing it would fix 6 and the second example of item 4. Furthermore, the lines

    text = re.sub(r'^             * ', r'****', text)
    text = re.sub(r'^         * ', r'***', text)
    text = re.sub(r'^     * ', r'**', text)
    text = re.sub(r'^ * ', r'*', text)

don't seem to make any changes:

>>> text
'abs_integrate`-related errors (see #12731 for possibly disabling or raising warning always with it)\n     * #13097 - integral of ln(1+4/5*sin(x))\n     * #11590 - `sgn()` fu'
>>> re.sub(r'^     * ', r'**', text3)
'abs_integrate`-related errors (see #12731 for possibly disabling or raising warning always with it)\n     * #13097 - integral of ln(1+4/5*sin(x))\n     * #11590 - `sgn()` fu'

If it were, the result would be even worse since "**" is not Markdown syntax for nested bullets. Does anyone have any idea what the purpose of these lines could be? I tend to remove them as well as they just create confusion.

Concerning A. 8.

This is due to the boolean multilines passed as True to the trac2markdown function when called from convert_wiki. Setting this boolean to False fixes the problem.

Concerning B. 1.

There is a lot of text explaining the functionality of the Trac syntax. Do we want to migrate that too? A lot of wiki pages are all about that (I think most pages that have Trac or Wiki in their names). For example there are a couple of WikiProzessors that are not just syntax highlighting and that have no equivalent in Markdown (e.g. #!td, see also the macros mentioned concerning 4).

[kwankyu]

Where is the final destination of the migrated wiki? That is, what is the repository that the migrated wiki will be placed?

Unlike migration of trac tickets, I think we can migrate trac wiki pages one by one, so that we fix some broken things manually on the migrated wiki pages, instead of perfecting the migration script and make one shot migration. Perhaps we should do both (fixing manually and improving the script).

If the plan is to keep trac running in read-only mode, then we may do the wiki migration gradually, independent of trac tickets migration.

What is the plan?

[kwankyu]

I guess the final destigation is sage.wiki. Who will have the write permission to the wiki? As a wiki, every developer should have the write permission. Is this possible?

[soehms]

Where is the final destination of the migrated wiki? That is, what is the repository that the migrated wiki will be placed?

I assume it to be https://github.com/sagemath/sage/wiki.

... Perhaps we should do both (fixing manually and improving the script).

Yes, I think so. Improving the script helps also for the ticket migration since surely there are references to wiki pages using all possible syntax variations, as well.

What is the plan?

From the migration document it is not completely clear. I think, we should consider the Wiki migration as a test case for the ticket migration and thus it should be completed before. For the procedure of the Wiki migration I have the following in mind:

1. Improve the script as long as there are obvious issues that can be fixed easily. Maybe we should create a list of pages that can be completely skipped. For example all pages that explain Trac functionality (having Trac or Wiki in their names).
2. Run the migration and merge the result into https://github.com/sagemath/trac_to_gh/wiki
3. Let the community check the result.
4. If there are still issues to be fixed generically go back to 1.
5. Do the remaining things manually. At least with respect of item B.1. there is still a good deal of manual work left to replace Trac-functionality specific explanations with appropriate links to the GitHub markdown documentation.
6. Let the community check the result again.
7. If there are still issues to be fixed manually go back to 5.
8. Move the result to it's final destination.

After that ticket migration might happen into an appropriate temporary repository at which the community can have a look at it. An iteration similar to 1.-4. can follow.

[soehms]

I guess the final destigation is sage.wiki. Who will have the write permission to the wiki? As a wiki, every developer should have the write permission.

In principal I agree! On the other hand reviews would improve the quality (but consumes resources).

Is this possible?

See this page.

[kwankyu]

Thanks. All you said sounds reasonable.

Are all the items in A of your first comment "obvious issues that can be fixed easily"? Or any of them is hard to fix but should be implemented because we cannot do it manually (that is hard to do manually)?

[soehms]

Are all the items in A of your first comment "obvious issues that can be fixed easily"? Or any of them is hard to fix but should be implemented because we cannot do it manually (that is hard to do manually)?

I think the hardest are done ore addressed, now. What remains are issues caused be some unreasonable lines of code. Deactivating these lines wil fix them. It's just a matter of understanding the purpose of them (see the corresponding remarks in the second comment).

[kwankyu]

Okay. I will see if I can help you with some easy items.

[kwankyu]

I am working on A.9. You want to convert, for instance, https://trac.sagemath.org/ticket/25206 to #25206. Right? And these #... will be converted to github issues later. Right?

[kwankyu]

I worry that my intervention may make things screwed up. Just let me know when you need a hand on any item.

[soehms]

I am working on A.9. You want to convert, for instance, https://trac.sagemath.org/ticket/25206 to #25206. Right? And these #... will be converted to github issues later. Right?

Both answers: Yes!

[soehms]

I worry that my intervention may make things screwed up. Just let me know when you need a hand on any item.

No problem! That can be treated in the PRs. The more people involved the better.

[soehms]

All issues known to me are done or addressed, now!

[kwankyu]

Great! It looks almost perfect. I looked through all topics, and found the following glitches. Some may need fixing the script and other may need manual edits.

(Open the links in a new tab to see the highlighted text. Clicking the github link does not work for me.)

E. Need more edits

1. • A link to an attachment does not work
     https://github.com/soehms/trac_to_gh/wiki#:~:text=!%5B%5D(ticket_badges.png%2C%20547px)
     trac - https://trac.sagemath.org/attachment/wiki/WikiStart/ticket_badges.png
2. • The whole table is broken
     https://github.com/soehms/trac_to_gh/wiki#:~:text=Displays%20as-,%5C,-%7B%7B%7B%0A%7B%7B%7B%0A%23!python%0Adef%20spam
     trac - https://trac.sagemath.org/#:~:text=name%20diff.-,Markup,-Displays%20as
3. • A link to the Octocat (github icon) is broken
     https://github.com/soehms/trac_to_gh/wiki/GitHub#:~:text=!%5B%5D(https%3A//github.githubassets.com/images/modules/logos_page/Octocat.png%2C%20height%3D200%2C%20right)
     trac - https://github.githubassets.com/images/modules/logos_page/Octocat.png
4. • It should be a link to wiki/GitLab
     https://github.com/soehms/trac_to_gh/wiki/GitHub#:~:text=There%20is%20also%20a%20Sage%20mirror%20on%20GitLab.
     trac - https://trac.sagemath.org/wiki/GitHub#:~:text=There%20is%20also%20a%20Sage%20mirror%20on%20GitLab.
5. • It should be a link to wiki/Mathematica-Graph-Theory https://github.com/soehms/trac_to_gh/wiki/GraphTheoryRoadmap#:~:text=%5BThis%5D(Mathematica%20Graph%20Theory%20)%20page
     trac - https://trac.sagemath.org/wiki/GraphTheoryRoadmap#:~:text=graph%20theory%20software-,This%20page,-aims%20at%20comparing
6. • Perhaps safe to delete the whole topic
     https://github.com/soehms/trac_to_gh/wiki/Mapple-Graph-Theory-#:~:text=See%20%5BMaple%20Graph%20Theory%20%5D(Maple%20Graph%20Theory%20).
     trac - https://trac.sagemath.org/wiki/Mapple%20Graph%20Theory%20#:~:text=See%20Maple%20Graph%20Theory%20.
7. • Fixed in trac wiki
     https://github.com/soehms/trac_to_gh/wiki/patchbot#:~:text=See%20also%20%5B%2333253%5D(https%3A//trac.sagemath.org/ticket/33253).
     trac - https://trac.sagemath.org/wiki/patchbot#:~:text=multiple%20workflows%20simultaneously.-,See%20also%20%E2%80%8B%2333253.,-Last%20modified%208
8. • Image links are broken https://github.com/soehms/trac_to_gh/wiki/patchbot#:~:text=!%5B%5D(https%3A//patchbot.sagemath.org/svg/New%2C%20width%3D48)
     trac - https://patchbot.sagemath.org/svg/New
9. • Totally broken
     https://github.com/soehms/trac_to_gh/wiki/PatLeSmithe
     trac - https://trac.sagemath.org/wiki/PatLeSmithe
10. • Broken links to wiki
      https://github.com/soehms/trac_to_gh/wiki/SageMath#:~:text=wiki%3Asage.%20The%20full%20(formal)%20name%20of%20the%20software%20that%20is%20nicknamed%20wiki%3Asage.
      trac - https://trac.sagemath.org/wiki/SageMath#:~:text=wiki%3Asage.%20The%20full%20(formal)%20name%20of%20the%20software%20that%20is%20nicknamed%20wiki%3Asage.
11. • Does color code work in github?
      https://github.com/soehms/trac_to_gh/wiki/sd51#:~:text=%3Cspan%20style%3D%22color%3A%20blue%22%3EBlue%3C/span%3E%3A%20New
      trac - https://trac.sagemath.org/wiki/sd51#:~:text=Color%20codes%3A-,Blue%3A%20New,-Orange%3A%20Needs%20review
12. • Broken
      https://github.com/soehms/trac_to_gh/wiki/TicketReports
      trac - https://trac.sagemath.org/wiki/TicketReports
13. • Groupings are gone: ReleaseTours, Sage, combinat, patchbot, sd, Symbolics
      trac - https://trac.sagemath.org/wiki/TitleIndex

[mkoeppe]

Thanks a lot for #19, #21, #23! These are great improvements.

[mkoeppe]

https://github.com/sagemath/trac_to_gh/wiki/sage-9.6#cylp has two links where the conversion did not work right.

[mkoeppe]

https://github.com/sagemath/trac_to_gh/wiki/sage-9.6#sage-development-in-the-cloud-with-gitpod has broken inline image

[mkoeppe]

What is the plan?

As soon as we are happy with the quality of the migration of the wiki, we can do a test migration of Trac tickets as well and see if more work on the markup translation is necessary.

Doing the actual final migration and switchover of the Trac wiki is not urgent.

[kwankyu]

Doing the actual final migration and switchover of the Trac wiki is not urgent.

Okay. Would the final migration of wiki occur well before the migration of tickets or at the same time? I think there's no problem in doing the wiki migration separately from the tickets migration.

[mkoeppe]

I agree, the two migrations can be done separately.

[kwankyu]

https://github.com/sagemath/trac_to_gh/wiki/sage-9.6#cylp has two links where the conversion did not work right.

Fixed in trac side.

[mkoeppe]

The script needs a patched version of PyGithub now; use pip install -r requirements.txt

[kwankyu]

I needed to remove trac_cache directory after that.

[mkoeppe]

Thanks very much for fixing this! Looking great

[mkoeppe]

the issue number "26574" does not match with the actual trac ticket number "26811"

https://trac.sagemath.org/ticket/26811

Is this a bug of the script?

Fixed in f9a937d; see for example https://github.com/sagemath/trac_to_gh/blob/main/Issues-26xxx/26811.md

[kwankyu]

Thanks. Very nice to look around and check converted issues.

[kwankyu]

We may name md files with 5 digit number so that they are listed in order.

[kwankyu]

We may name md files with 5 digit number so that they are listed in order.

Perhaps not. This inconvenience applies only to issues 00xxx.

[mkoeppe]

Many converted tickets have the text "Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:" but then no commits are listed. What's going on there?
Compare https://trac.sagemath.org/ticket/31580 with https://github.com/sagemath/trac_to_gh/blob/main/Issues-31xxx/31580.md

[kwankyu]

I am working on this issue. Almost fixed.

[mkoeppe]

Another nested-codeblock problem:
https://github.com/sagemath/trac_to_gh/blob/main/Issues-27xxx/27996.md

[mkoeppe]

Also https://github.com/sagemath/trac_to_gh/blob/main/Issues-34xxx/34393.md#comment:4

[mkoeppe]

And @ quoting misbehaves within code-quotes - https://github.com/sagemath/trac_to_gh/blob/main/Issues-34xxx/34393.md#comment:16

[kwankyu]

And @ quoting misbehaves within code-quotes - https://github.com/sagemath/trac_to_gh/blob/main/Issues-34xxx/34393.md#comment:16

I don't know why @ is converted to be quoted in the first place. Do you know?

[mkoeppe]

I think it's to avoid creating accidental mentions - see also #49

[mkoeppe]

problem with a codeblock within a bulleted list
https://github.com/sagemath/trac_to_gh/blob/main/Issues-18xxx/18231.md#comment:3

[mkoeppe]

Also https://github.com/sagemath/trac_to_gh/blob/main/Issues-18xxx/18938.md#comment:6

[mkoeppe]

Also https://github.com/sagemath/trac_to_gh/blob/main/Issues-27xxx/27870.md#comment:27

[mkoeppe]

Bad table conversion: https://github.com/sagemath/trac_to_gh/blob/main/Issues-15xxx/15239.md#comment:2 and https://github.com/sagemath/trac_to_gh/blob/main/Issues-15xxx/15415.md#comment:2 and https://github.com/sagemath/trac_to_gh/blob/main/Issues-15xxx/15734.md#comment:6

[mkoeppe]

These are all fixed by #56, thanks a lot @kwankyu !