Features/context

[steffencruz]

• Refactors dataset.py into a submodule with each dataset type having its' own file
• Adds a base Dataset class which enables context generation via 3 methods
  • get (specific context retrieval, fully deterministic)
  • search (dataset-specific search algorithm, generally deterministic)
  • random (produces a random context, can be seeded)
• Removes existing requests based wikipedia dataclass in favor of the wiki python api. Closes Improve wikipedia retrieval mechanism #76
• Adds Selector class and variants, which enable customizable selection from sets of link-like objects
• Adds Context dataclass which all datasets now return.
  • Enforces consistent schema.
  • Makes development, logging and testing more robust and efficient.
• Adds MaxRetryError exception class for handling api call errors

[steffencruz]

625 tests passing for each python version :)

[steffencruz]

Tracked experiment

[p-ferreira]

I see that you define the page sections as internal_links, I believe that would be more than enough for our use case. Do we still need this comment?

[p-ferreira]

It seems that we are not handling possible leap year scenarios, which could break the code with ValueError: day is out of range for month, even though its probability of happening is small

[p-ferreira]

Formatting is a bit weird (which will be solved once black PR is merged) but overall this PR seems to add a very nice structure around the tasks and rewards organization, not to mention the dataset modularization + extra refactoring.

The only consideration I have is towards date qa not taking leap years in consideration, which could cause an exception but it wouldn't affect the flow that much considering that we have a task creation retry policy in place.