Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update DW Version #342

Merged
merged 6 commits into from
Feb 12, 2024
Merged

Update DW Version #342

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
909395d
Update DW Version
MaxDall Jan 30, 2024
70fb945
update _paragraph_selector
MaxDall Feb 1, 2024
68a2168
Adds comment about `_paragraph_selector`
MaxDall Feb 6, 2024
40fd1a8
add author byline regex
MaxDall Feb 8, 2024
a9c44e2
add regex to paragraph selector
MaxDall Feb 12, 2024
17cee4c
changed regex slightly
MaxDall Feb 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/fundus/parser/utility.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,10 @@ def get_meta_content(tree: lxml.html.HtmlElement) -> Dict[str, str]:
return meta


def strip_nodes_to_text(text_nodes: List[lxml.html.HtmlElement]) -> Optional[str]:
def strip_nodes_to_text(text_nodes: List[lxml.html.HtmlElement], join_on: str = "\n\n") -> Optional[str]:
if not text_nodes:
return None
return "\n\n".join(([re.sub(r"\n+", " ", node.text_content()) for node in text_nodes])).strip()
return join_on.join(([re.sub(r"\n+", " ", node.text_content()) for node in text_nodes])).strip()


def apply_substitution_pattern_over_list(
Expand Down
15 changes: 15 additions & 0 deletions src/fundus/publishers/de/dw.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,14 @@
generic_date_parsing,
generic_text_extraction_with_css,
generic_topic_parsing,
strip_nodes_to_text,
)


class DWParser(ParserProxy):
class V2(BaseParser):
VALID_UNTIL = datetime.date(2024, 1, 18)

_paragraph_selector = CSSSelector("div.rich-text > p")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DW seems to sometimes have additional tags at the end of the article (https://www.dw.com/de/trump-in-verleumdungsprozess-gegen-e-jean-carroll-zu-83-millionen-dollar-schadenersatz-verurteilt/a-68100499, https://www.dw.com/de/us-regierung-genehmigt-verkauf-von-f-16-kampfjets-an-türkei/a-68100064), unfortunately I didn't see an easy fix at the first glance, since they aren't any special tags and not always there.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They also sometimes mention an update of the article at the end, this should be able to be handled easily since it's in italics: https://www.dw.com/de/esc-2024-schwedische-k%C3%BCnstler-gegen-teilnahme-israels/a-68011968

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DW seems to sometimes have additional tags at the end of the article

Are you talking about this jj/sti (dpa, afp, rtr)?
I can only think of regular expressions. But I don't know how to get them working in XPath.
Anyways, that would be a regular expression matching the above string.
.*\((rtr, |dpa, |afp, ){0,2}(rtr|dpa|afp)\)$

They also sometimes mention an update of the article at the end, this should be able to be handled easily since it's in italics

I added it to the selector.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, although it's not restricted to those: here are some that I found: AR/jj (dpa, rtr, ap) kle/jj (kna, dpa, rtr, afp) pg/AR/kle (dpa, afp) pg/AR/haz (afp, dpa) kle/haz (dpa, rtr, afp) not really sure what it's limited to though

Copy link
Collaborator Author

@MaxDall MaxDall Feb 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So unless we can use reg exp in XPath there is nothing much to do about it anyways, but updating the parser would be very important because it currently does not work.

Update: i added a comment about the author line.

_summary_selector = CSSSelector("header > p")
_subheadline_selector = CSSSelector("div.rich-text > h2")
Expand Down Expand Up @@ -53,6 +56,18 @@ def title(self) -> Optional[str]:
def topics(self) -> List[str]:
return [node.text_content().strip() for node in self._topic_selector(self.precomputed.doc)]

class V2_1(V2):
VALID_UNTIL = datetime.date.today()

_topic_selector = CSSSelector("header > div.kicker > span")

@attribute
def topics(self) -> List[str]:
topic_nodes = self._topic_selector(self.precomputed.doc)
if (topic_string := strip_nodes_to_text(topic_nodes, join_on=", ")) is not None:
return topic_string.split(", ")
return []

class V1(BaseParser):
VALID_UNTIL = datetime.date(2023, 6, 12)

Expand Down
11 changes: 11 additions & 0 deletions tests/resources/parser/test_data/de/DW.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,16 @@
"Long COVID",
"Coronavirus"
]
},
"V2_1": {
"authors": [
"Jennifer Pahlke"
],
"publishing_date": "2024-01-30 14:13:12.269000+00:00",
"title": "Russland-Wahl: Nadeschdin setzt auf ein Ende der Putin-Ära",
"topics": [
"Politik",
"Russische Föderation"
]
}
}
Binary file added tests/resources/parser/test_data/de/DW_2024_01_30.html.gz
View file
Open in desktop
Binary file not shown.
4 changes: 4 additions & 0 deletions tests/resources/parser/test_data/de/meta.info
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@
"url": "https://www.dw.com/de/post-vac-syndrom-nach-covid-19-impfung-was-wissen-wir/a-65897191?maca=de-rss-de-all-1119-xml-mrss",
"crawl_date": "2023-06-13 16:57:08.558047"
},
"DW_2024_01_30.html.gz": {
"url": "https://www.dw.com/de/russland-wahl-nadeschdin-setzt-auf-ein-ende-der-putin-%C3%A4ra/a-68117411?maca=de-rss-de-all-1119-xml-mrss",
"crawl_date": "2024-01-30 16:12:24.867249"
},
"DieWelt_2023_04_28.html.gz": {
"url": "https://www.welt.de/wirtschaft/article245055596/BIP-Diese-Grafiken-zeigen-wie-schlecht-es-um-Deutschlands-Wirtschaft-steht.html",
"crawl_date": "2023-04-28 20:22:32.033988"
Expand Down
Loading