Code

Treebank word tokenizer from NLTK#83

Code

Draft

rth wants to merge 6 commits intomainrth/vtext:mainfrom TreebankWordTokenizerrth/vtext:TreebankWordTokenizerCopy head branch name to clipboard

Commits

Commits on Jun 18, 2020

• WIP

  rth

  committed
  2aad6d1

  Copy full SHA for 2aad6d1

Commits on Jun 22, 2020

• WIP naive implementation

  rth

  committed
  71c6c20

  Copy full SHA for 71c6c20

• Python wrapper

  rth

  committed
  aad62af

  Copy full SHA for aad62af

Commits on Jun 23, 2020

• Rename to NLTKWordTokenizer

  rth

  committed
  8db4a70

  Copy full SHA for 8db4a70

• Update evaluation scripts

  rth

  committed
  5a90a90

  Copy full SHA for 5a90a90

• More unicode tests

  rth

  committed
  9c96295

  Copy full SHA for 9c96295