Code

Spidercache#85

Code

Merged

stooit merged 31 commits intodevelopsalsadigitalauorg/merlin-framework:developfrom feature/issue-74-spider-cachesalsadigitalauorg/merlin-framework:feature/issue-74-spider-cacheCopy head branch name to clipboardSep 25, 2019

+1,012-245

Commits

Commits on Aug 6, 2019

• Add crawler test.

  steveworley

  committed
  e2c977a

  Copy full SHA for e2c977a

• Add group testing.

  Show description for 3bef07d
  steveworley

  committed
  3bef07d

  Copy full SHA for 3bef07d

• Use sys_get_temp_dir().

  steveworley

  committed
  65f9b98

  Copy full SHA for 65f9b98

• Remove cleanup to debug CI

  steveworley

  committed
  44a1fab

  Copy full SHA for 44a1fab

Commits on Aug 18, 2019

• Add sleep and php server from the right directory.

  steveworley

  committed
  f8a0c69

  Copy full SHA for f8a0c69

Commits on Sep 5, 2019

• Strip trailing slash if exists on domain for cache store dir

  

  Andrew Rowlands

  committed
  8417bd9

  Copy full SHA for 8417bd9

Commits on Sep 6, 2019

• WIP but functional spider-crawler cache

  Show description for d3d2cb3
  

  Andrew Rowlands

  committed
  d3d2cb3

  Copy full SHA for d3d2cb3

• A missing ingredient

  

  Andrew Rowlands

  committed
  3d47ce8

  Copy full SHA for 3d47ce8

Commits on Sep 9, 2019

• Resolved URL count.

  stooit

  committed
  251a6c9

  Copy full SHA for 251a6c9

• Added no-cache runtime flag.

  stooit

  committed
  e67c368

  Copy full SHA for e67c368

• Code standards cleanup.

  stooit

  committed
  581f0ab

  Copy full SHA for 581f0ab

Commits on Sep 13, 2019

• cache_enabled flag docs

  

  Andrew Rowlands

  committed
  91d8451

  Copy full SHA for 91d8451

• Fixes cache write before grouping logic

  

  Andrew Rowlands

  committed
  40d64a7

  Copy full SHA for 40d64a7

• Detect urls with duplicate content in crawler-spider

  

  Andrew Rowlands

  committed
  53719f8

  Copy full SHA for 53719f8

• Detect urls with duplicate content in crawler-spider

  

  Andrew Rowlands

  committed
  0982cbc

  Copy full SHA for 0982cbc

• crawler duplicate flag docs

  

  Andrew Rowlands

  committed
  e16efda

  Copy full SHA for e16efda

Commits on Sep 14, 2019

• Remove newline

  

  Andrew Rowlands

  committed
  347107d

  Copy full SHA for 347107d

Commits on Sep 20, 2019

• Merge branch 'develop' into feature/issue-74-spider-cache

  

  Andrew Rowlands

  committed
  5d6798d

  Copy full SHA for 5d6798d

• Refactoring to allow common php server base for tests

  

  Andrew Rowlands

  committed
  8ef47af

  Copy full SHA for 8ef47af

• Merge branch 'develop' into feature/issues-54

  

  Andrew Rowlands

  committed
  0321e39

  Copy full SHA for 0321e39

• Merge crawler tests from issue 54

  

  Andrew Rowlands

  committed
  c09a260

  Copy full SHA for c09a260

Commits on Sep 21, 2019

• Crawler tests using common php server class

  

  Andrew Rowlands

  committed
  27c67cd

  Copy full SHA for 27c67cd

• phpdoc fixes

  

  Andrew Rowlands

  committed
  8551423

  Copy full SHA for 8551423

Commits on Sep 22, 2019

• Cache update to allow recursive removal of a domains cached content

  

  Andrew Rowlands

  committed
  7111759

  Copy full SHA for 7111759

• Crawler tests based on common php server including caching and duplicate options

  

  Andrew Rowlands

  committed
  23d9c49

  Copy full SHA for 23d9c49

• clear output dirs on runs

  

  Andrew Rowlands

  committed
  3986f0a

  Copy full SHA for 3986f0a

• Stop server on start to reset if running

  

  Andrew Rowlands

  committed
  f60ed64

  Copy full SHA for f60ed64

• Move sleep, increase

  

  Andrew Rowlands

  committed
  e165364

  Copy full SHA for e165364

Commits on Sep 24, 2019

• Merge branch 'develop' into feature/issue-74-spider-cache

  stooit

  committed
  0a9fdc7

  Copy full SHA for 0a9fdc7

• readd html pages

  

  Andrew Rowlands

  committed
  e0d9a0d

  Copy full SHA for e0d9a0d

• Merge branch 'feature/issue-74-spider-cache' of github.com:salsadigitalauorg/merlin-framework into feature/issue-74-spider-cache

  stooit

  committed
  3ac87c6

  Copy full SHA for 3ac87c6