Add Python 3.12 to the CI and update Ubuntu

[gdlg]

Summary

This PR adds support for Python 3.12 in the project configuration and CI. I have left the min Python version unchanged (3.9).

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added the description of my changes into CHANGELOG.​
• I have updated the documentation accordingly

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
#
# SPDX-License-Identifier: MIT

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.14%. Comparing base (676bd10) to head (0a76ae9).
Report is 8 commits behind head on releases/1.10.0.

Additional details and impacted files

@@               Coverage Diff                @@
##           releases/1.10.0    #1701   +/-   ##
================================================
  Coverage            81.14%   81.14%           
================================================
  Files                  284      284           
  Lines                32891    32891           
  Branches              5299     5299           
================================================
+ Hits                 26688    26690    +2     
+ Misses                4751     4750    -1     
+ Partials              1452     1451    -1     

Flag	Coverage Δ
ubuntu-20.04_Python-3.10	?
ubuntu-24.04_Python-3.12	81.12% <100.00%> (?)
windows-2022_Python-3.10	?
windows-2022_Python-3.12	81.12% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[djdameln]

Thanks a lot for the PR. I only have some minor comments.

[djdameln]

Somewhat out of scope of this PR, but my concern with this implementation is that it may be unclear to users when to pass just the tokenizer, and when to pass both a tokenizer and a vocab. I think a proper docstring would address this.

Adding the complete dosctring is out of scope, but maybe you could add a brief docstring, or comment, explaining that the vocab arg is not needed when the tokenizer already returns the token ids instead of the raw tokens, and that this depends on which NLP framework is used. Just a brief 1-2 sentence would be sufficient. It's mainly to help whoever adds the full docstring for this class in the future.

[gdlg]

Done but I think that the construction of those classes needs to be refactored to be simplified. The parameters of the to_framework function depends on which framework is used which makes it difficult to document.

[djdameln]

Yes, I fully agree. We'll address this at a later point.