Fix bug in class used by OAI Harvest #35
Conversation
…g tag, rather than just closing tag
|
Nathan, this looks good to me but I need to test it first... as soon as I get my PC back together at home (just moved) I'll run it and merge it. If you can provide output of you testing it I'll go ahead and merge it without my own testing. |
|
First attempt at testing it freezes at the output step in NLMJournalFetch testing. These passed on master branch so I'm wondering what is happening. I'll look into it later this week and try to help you get this merged. Sorry its taken almost 4 months for me to finally get around to testing this. 2015-02-04 20:58:59.854 INFO [o.v.h.f.n.NLMJournalFetch] Fetching 500 to 1000 records from search |
Fix bug in class used by OAI Harvest
|
Was a memory issue that happens occasionally when testing the harvester. Tests completed now and std tests for PubMed failed, but the rest past |
The original method used was just detecting when a record was found via the close tag, which resulted in the occasional corruption of raw-records due to the OAI-PMH harvest data being included. Improved but not foolproof method waits for opening tag to be hit before writing to the buffer that generates the record. I have not tested this with multiple tags to split on. That situation would complicate the harvest especially if the tags were nested inside each other.