Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/mx-1502 wikidata results refactor #39

Merged
merged 38 commits into from
May 16, 2024
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
36e63dc
receive only one wikidata org
mr-kamran-ali Mar 25, 2024
e04b09a
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Mar 25, 2024
fecfec3
handle wikidata dagster asset for rki
mr-kamran-ali Mar 25, 2024
0d4e781
Timestamp updates in tests
mr-kamran-ali Mar 25, 2024
c8ddf10
update odk wikidata integration
mr-kamran-ali Mar 25, 2024
d743add
fix odk organization merged id extraction
mr-kamran-ali Mar 26, 2024
d99bbbd
fix temporal entity precisoin
mr-kamran-ali Mar 26, 2024
306d054
adjust temporalentity artificial data
mr-kamran-ali Mar 26, 2024
39008f6
update changlog and version
mr-kamran-ali Mar 27, 2024
073b1f6
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Mar 27, 2024
1d2bd47
move quotation mark filtering to mex-common
mr-kamran-ali Mar 28, 2024
c20fd94
update changelog
mr-kamran-ali Mar 28, 2024
4850e73
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Mar 28, 2024
29af6bc
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Apr 2, 2024
f4e2cf4
update changelog
mr-kamran-ali Apr 4, 2024
64f740c
adjust variable declaration
mr-kamran-ali Apr 4, 2024
c6c2984
rename get_timestamp to get_tempral_entity
mr-kamran-ali Apr 4, 2024
958cdd0
reverse ff projects temporal entity precision
mr-kamran-ali Apr 4, 2024
53f0753
remove TODO comment
mr-kamran-ali Apr 4, 2024
01c677b
fix datscha_web mocking result
mr-kamran-ali Apr 4, 2024
29f8bc9
reverse ff projects test assert
mr-kamran-ali Apr 4, 2024
bf6dc59
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Apr 5, 2024
3c1fbe4
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Apr 8, 2024
b5dee7c
update mex-common dependency
mr-kamran-ali Apr 8, 2024
ae3a045
temporal entity changes
mr-kamran-ali Apr 16, 2024
ac729f3
move odk wikidata extraction out of transform
mr-kamran-ali Apr 16, 2024
1c2a39c
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
mr-kamran-ali Apr 16, 2024
c821c92
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
cutoffthetop May 13, 2024
75391a1
Update mex/international_projects/extract.py
cutoffthetop May 13, 2024
d54bdfa
Apply new mex-common version
cutoffthetop May 14, 2024
d11f185
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
cutoffthetop May 14, 2024
17cf946
Fix tests
cutoffthetop May 14, 2024
00468ef
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
cutoffthetop May 14, 2024
05d07ef
Merge branch 'feature/mx-1502-wikidata-results-refactor' of https://g…
cutoffthetop May 14, 2024
e91b848
Update versions
cutoffthetop May 14, 2024
d111226
hold back version bump
cutoffthetop May 14, 2024
b032707
Fix tests
cutoffthetop May 16, 2024
fe83682
Merge branch 'main' of https://github.com/robert-koch-institut/mex-ex…
cutoffthetop May 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
handle wikidata dagster asset for rki
  • Loading branch information
mr-kamran-ali committed Mar 25, 2024
commit fecfec31b8adfe1378e8c092e0d4e6f83bfd1714
15 changes: 8 additions & 7 deletions mex/pipeline/wikidata.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
from collections.abc import Generator

from mex.common.exceptions import MExError
from mex.common.models import ExtractedOrganization, ExtractedPrimarySource
from mex.common.wikidata.extract import search_organization_by_label
from mex.common.wikidata.models.organization import WikidataOrganization
Expand All @@ -11,21 +10,23 @@


@asset(group_name="default")
def wikidata_organization_rki() -> WikidataOrganization | None:
def wikidata_organization_rki() -> WikidataOrganization:
"""Extract WikidataOrganization for Robert Koch-Institut."""
return search_organization_by_label("Robert Koch-Institut")
if org := search_organization_by_label("Robert Koch-Institut"):
return org
raise MExError("RKI not found on wikidata, cannot proceed.")


@asset(group_name="default")
def extracted_organization_rki(
wikidata_organization_rki: WikidataOrganization,
extracted_primary_source_wikidata: ExtractedPrimarySource,
) -> Generator[ExtractedOrganization, None, None]:
) -> ExtractedOrganization:
"""Transforms RKI organization data to extracted organizations and load result."""
extracted_organization_rki = (
extracted_organization_rki = list(
transform_wikidata_organizations_to_extracted_organizations(
[wikidata_organization_rki], extracted_primary_source_wikidata
)
)
load(extracted_organization_rki)
return extracted_organization_rki
return extracted_organization_rki[0]
Loading