Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/mx-1502 wikidata results refactor #39

Merged
merged 38 commits into from
May 16, 2024

Conversation

mr-kamran-ali
Copy link
Contributor

@mr-kamran-ali mr-kamran-ali commented Mar 25, 2024

Changes

  • receive one or None organization from wikidata aux extractor
  • adjust Timestamp usage to TemporalEntity
  • move quotation marks (") filtering to mex-common from requested wikidata label (MX-1526)
  • also upgrade mex-common to 0.25.0

@mr-kamran-ali mr-kamran-ali changed the title [WIP] Feature/mx 1502 wikidata results refactor Feature/mx 1502 wikidata results refactor Mar 27, 2024
Copy link
Contributor

@cutoffthetop cutoffthetop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wikidata updates looking workable. thanks for taking on the timestamp-migration as well!

some questions and some confusion in the tests though.

CHANGELOG.md Outdated Show resolved Hide resolved
mex/datscha_web/extract.py Outdated Show resolved Hide resolved
mex/ff_projects/extract.py Outdated Show resolved Hide resolved
mex/ff_projects/extract.py Outdated Show resolved Hide resolved
mex/international_projects/extract.py Outdated Show resolved Hide resolved
tests/ff_projects/test_extract.py Outdated Show resolved Hide resolved
tests/ff_projects/test_extract.py Show resolved Hide resolved
tests/ff_projects/test_transform.py Show resolved Hide resolved
tests/international_projects/test_transform.py Outdated Show resolved Hide resolved
tests/international_projects/test_transform.py Outdated Show resolved Hide resolved
Copy link
Contributor

@erichesse erichesse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I investigated the ODK load issue, unfortunately I'm more confused than before. Anyway I summed up my thoughts on it.

mex/odk/transform.py Outdated Show resolved Hide resolved
@mr-kamran-ali
Copy link
Contributor Author

I investigated the ODK load issue, unfortunately I'm more confused than before. Anyway I summed up my thoughts on it.

If I have your ok, I can take care of this as well by making the process very similar to other extractors wikidata implementation. that is to separate extraction, transformation and load in main file.

@mr-kamran-ali mr-kamran-ali requested a review from erichesse April 10, 2024 07:41
tests/ff_projects/test_transform.py Show resolved Hide resolved
tests/international_projects/test_transform.py Outdated Show resolved Hide resolved
tests/international_projects/test_transform.py Outdated Show resolved Hide resolved
mex/odk/transform.py Outdated Show resolved Hide resolved
Copy link
Contributor Author

@mr-kamran-ali mr-kamran-ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cutoffthetop Temporal entity stuff in international projects extract and its testing, is still not perfect yet, have a look at changes in mex-common(robert-koch-institut/mex-common#157), they need to be merged as well, its a breaking change if this PR is merged into mex-extractors main.

mex/odk/transform.py Outdated Show resolved Hide resolved
@cutoffthetop cutoffthetop changed the title Feature/mx 1502 wikidata results refactor feature/mx-1502 wikidata results refactor May 14, 2024
@cutoffthetop cutoffthetop merged commit 8a9c9a2 into main May 16, 2024
7 checks passed
@cutoffthetop cutoffthetop deleted the feature/mx-1502-wikidata-results-refactor branch May 16, 2024 12:57
rababerladuseladim added a commit to robert-koch-institut/mex-common that referenced this pull request May 21, 2024
# PR Context
- blocked by:
robert-koch-institut/mex-extractors#39 as
testing is only possible after the mex-common dependency in
mex-extractors is updated to v0.24.0
- Bug: the first test that is executed does not receive isolated
settings. It therefore receives the production data as they are in
mex-assets. If the test uses data from mex-assets, it will get a
different result than if it is not executed first.
- Reproduce: in mex-extractors, execute `pdm run pytest
.\tests\sumo\test_extract.py::test_extract_cc1_data_model_nokeda`
  - passes if not executed as first test
  - fails if executed in isolation / as first test

# Fixed
- isolate settings context before first test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants