Merge Ray Train XGboost tutorials

[crypdick]

Why are these changes needed?

This PR merges the two existing Ray Train XGBoost tutorials, which were largely duplicated, into one.

• created new distributed-xgboost-lightgbm.ipynb which is a superset of the two existing tutorials
• ported .rst with transcluded code snippets since it was a tangled nightmare
• updated pointers in train/examples.yml to this tutorial
• kept orphaned code in doc_code/preprocessors.py since it is part of our CI

NOTE: I am not sure that I updated the pointers to this tutorial correctly. Please double check my work in examples.yml

Out of scope: merging the third Ray Tune XGBoost tutorial.

Related issue number

#50232 (comment)

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[pcmoritz]

@crypdick Unfortunately this doesn't work since now none of the links to the .rst that were converted to .ipynb are functional any more (look at the test failures in https://buildkite.com/ray-project/microcheck/builds/11000) -- these are set by the anchor .. _train-gbdt-guide: for example. Either we keep the file in .rst, or we need to figure out how to set the anchor in the .ipynb :)

There was also a small error about a non-existant use case that I pushed a fix for :)

[crypdick]

@crypdick Unfortunately this doesn't work since now none of the links to the .rst that were converted to .ipynb are functional any more (look at the test failures in https://buildkite.com/ray-project/microcheck/builds/11000) -- these are set by the anchor .. _train-gbdt-guide: for example. Either we keep the file in .rst, or we need to figure out how to set the anchor in the .ipynb :)

@pcmoritz I am not sure why the link didn't work. For example, the Ray Data examples page links to an ipynb tutorial via the examples.yml links using the same pattern. What am I missing?

[pcmoritz]

The problem is that the tests are failing with this:

[2025-02-08T01:34:27Z] /ray/doc/source/cluster/kubernetes/examples/ml-example.md:14: WARNING: undefined label: 'train-gbdt-guide'
--
  | [2025-02-08T01:34:27Z] /ray/doc/source/cluster/kubernetes/examples/ml-example.md:180: WARNING: undefined label: 'train-gbdt-guide'
  | [2025-02-08T01:34:28Z] /ray/doc/source/cluster/vms/examples/ml-example.md:14: WARNING: undefined label: 'train-gbdt-guide'
  | [2025-02-08T01:34:28Z] /ray/doc/source/cluster/vms/examples/ml-example.md:120: WARNING: undefined label: 'train-gbdt-guide'
  | [2025-02-08T01:34:38Z] /ray/doc/source/ray-overview/use-cases.rst:175: WARNING: unknown document: '/train/examples/xgboost/xgboost_example'
  | [2025-02-08T01:34:50Z] /ray/doc/source/train/more-frameworks.rst:43: WARNING: unknown document: 'distributed-xgboost-lightgbm'

(you should be able to see this in the CI run linked from my comment)

[crypdick]

@pcmoritz ok, I've fixed 3 of the error categories.

I'm stuck on the undefined label errors from the missing anchors. The Sphinx cross-referencing docs don't mention ipynb support. However, it is possible to link to arbitrary webpages, would that be a satisfactory work-around?

Alternatively, I found an ugly hack to create anchors by converting the ipynb notebooks, which I don't recommend.

[pcmoritz]

Does it work if you replace the labels train-gbdt-guide with /train/examples/xgboost/distributed-xgboost-lightgbm? Unfortunately using link to arbitrary webpages is not a valid workaround (first, it doesn't check the webpage exists, and more importantly, it doesn't handle versioning of the docs correctly)

[2025-02-10T21:53:31Z] /ray/doc/source/cluster/kubernetes/examples/ml-example.md:14: WARNING: undefined label: 'train-gbdt-guide'
--
  | [2025-02-10T21:53:31Z] /ray/doc/source/cluster/kubernetes/examples/ml-example.md:180: WARNING: undefined label: 'train-gbdt-guide'
  | [2025-02-10T21:53:32Z] /ray/doc/source/cluster/vms/examples/ml-example.md:14: WARNING: undefined label: 'train-gbdt-guide'
  | [2025-02-10T21:53:32Z] /ray/doc/source/cluster/vms/examples/ml-example.md:120: WARNING: undefined label: 'train-gbdt-guide'

[crypdick]

@pcmoritz now getting WARNING: undefined label: '/train/examples/xgboost/distributed-xgboost-lightgbm' 😭 I feel like there has to be a way

Also, I'm not sure why it says that I requested reviews, all I did was push a new commit

[pcmoritz]

Very nice! The only small wrinkle is that now this is displayed twice in the example gallery, maybe we can deduplicate it going forward (but not a big deal):