Tweak Data Collection

[maxachis]

There are several possible ways we could tweak our data collection efforts to potentially be more efficacious for machine learning components. For example:

Add Bag-Of-Words data to Div Text Collection

A Bag-Of-Words model disregards word order and focuses solely on how many of which words are present. It would look something like this:

{"John":1,"likes":3,"to":2,"watch":2,"movies":2,"Mary":2,"too":1,"also":1,"football":1,"games":1};

This would enable us to balance data collection (especially for pages with more div content) by focusing on keywords rather than accounting for every single word sequence in the text. Presumably certain web pages for certain record types will be more likely to have certain keywords than others.