Eroshareripper now uses mirror for data #29
Conversation
There was a problem hiding this comment.
Looks good, other than a few comments.
Also, rather than relying on a rehost, we could actually set up the redirects to internet archive / wayback machine on our own. There's basically two things to look for: the site pages (which are backed up in wayback machine) and the content, which is also hosted on the internet archive, and we can pull the link format directly out of a rehost like eroshae and extrapolate the redirect.
This will work for now and unblock ripping of eroshare links so that's awesome!
| @@ -51,22 +51,34 @@ public void downloadURL(URL url, int index) { | |||
| } | |||
| @Override | |||
| public boolean canRip(URL url) { | |||
| Pattern p = Pattern.compile("^https?://[w.]*eroshare.com/([a-zA-Z0-9\\-_]+)/?$"); | |||
| Pattern p = Pattern.compile("^https?://eroshae.com/([a-zA-Z0-9\\-_]+)/?$"); | |||
There was a problem hiding this comment.
Should probably stick to allowing optional www (which is what the [w.]*was accomplishing although(www[.])?` documents the intent better and would likely be more efficient under the regex engine.
| @@ -79,12 +91,14 @@ public Document getNextPage(Document doc) throws IOException { | |||
| // Find next page | |||
| String nextUrl = ""; | |||
| Element elem = doc.select("li.next > a").first(); | |||
| logger.info(elem); | |||
There was a problem hiding this comment.
Why remove the logger line?
There was a problem hiding this comment.
I assume I did it because I was trying to debug and it was clogging up the log/output
Feature/log4j2
Category
This change is exactly one of the following (please change
[ ]to[x]) to indicate which:Description
This allows users to rip eroshare (Now down and never coming back) albums using a mirror. Some content will 404 because it wasn't archived before eroshare went down
Testing
Required verification:
mvn test(there are no new failures or errors).Optional but recommended: