Skip to content

ZhongHistory

Jump to bottom
SanghounSong edited this page Mar 23, 2015 · 22 revisions

Note

columns

  1. ordinary: the ordinary grammar (using neither unknown word handling nor bridge rules)

  2. unk: unknown word handling (the YY mode (ZhongYYMode))

  3. br: robust parsing with bridge rules

  4. unk+br: unknown word handling and robust parsing with bridge rules

  5. generation: using the result of [1] as the input MRSes (# of generated items / # of parse items)

  6. end-to-end: # of generated items / # of input sentences

numbers

After using gTest, the numbers are described in the following format. Before that, only the total coverages are presented.

coverage-of-grammatical (readings-or-grammatical) / coverage-of-ungrammatical (readings-or-ungrammatical)

For example, 4.94 (1.5) / 7.89 (2.6667) means the coverage for grammatical items is 4.94%, the coverage for ungrammatical items is 7.89%, the average number of readinfs for grammatical items is 1.5, and the average number of readings for ungrammatical items is 2.6667.

Not all datasets have ungrammatical items. If there is no ungrammatical item in the dataset, the part after "/" is missing.

Note that with the numbers for grammatical items, the higher the better, while with those for ungrammatical ones the lower the better.

abbreviations

  1. auto: lexical items automatically constructed from the Sinica treebank
  2. punct: robust text handling, such as punctuations, symbols, interjection

options

  1. nbest: 5
  2. timeout: 5 sec.
  3. max-chart-megabytes: 256mb
  4. max-unpack-megabytes: 256mb

Development Data

mrs
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 93.46 (3.27) 93.46 (3.27) 100.0 (5.00) 100.0 (5.00) 99.0 92.53
AABB 2015-03-13 93.5 88.8 100 94.4 N/A N/A
interjections 2015-03-13 93.5 88.8 100 94.4 N/A N/A
auto 2015-03-12 93.5 88.8 100 94.4 N/A N/A
w/o auto 2015-03-12 38.3 38.3 97.2 93.5 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

ntumc
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 0.32 (1.92) 4.04 (4.14) 1.13 (4.49) 9.7 (4.39) 100.0 0.32
AABB 2015-03-13 0.8 1.5 1.0 3.0 N/A N/A
interjections 2015-03-13 0.8 1.5 1.0 2.7 N/A N/A
auto 2015-03-12 0.8 1.5 1.0 2.8 N/A N/A
w/o auto 2015-03-12 0.0 1.4 0.0 1.9 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

pctb-dev (the first 5,000 sentences)
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 0.04 (1.50) 3.08 (4.68) 0.06 (5.00) 7.3 (4.84) 100.0 0.04
AABB 2015-03-13 0.0 2.5 0.1 4.1 N/A N/A
interjections 2015-03-13 0.0 2.5 0.1 4.1 N/A N/A
auto 2015-03-12 0.0 2.5 0.1 4.1 N/A N/A
w/o auto 2015-03-12 0.0 2.3 0.0 3.0 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

sinica-dev (the first 5,000 sentences (nltk_data))
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 1.36 (3.06) 23.06 (4.70) 2.8 (4.46) 30.86 (4.83) 95.59 1.3
AABB 2015-03-13 1.2 8.7 2.5 11.1 N/A N/A
interjections 2015-03-13 1.2 8.6 2.5 11.1 N/A N/A
auto 2015-03-12 1.2 8.6 2.4 11.1 N/A N/A
w/o auto 2015-03-12 0.1 8.1 0.7 10.0 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

Test Data

jec
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 2.77 (2.29) 21.66 (3.88) 6.26 (5.00) 37.03 (5.00) 87.5 2.42
AABB 2015-03-13 2.7 20.5 4.5 31.1 N/A N/A
interjections 2015-03-13 2.5 20.2 4.3 30.4 N/A N/A
auto 2015-03-12 2.5 20.2 4.3 30.6 N/A N/A
w/o auto 2015-03-12 0.0 18.3 0.1 22.7 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

fu-berlin
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 4.94 (1.5000) / 7.89 (2.67) 37.04 (2.67) / 21.05 (2.38) 12.35 (4.10) / 36.84 (3.50) 74.07 (4.85) / 76.32 (4.28) 100.0 4.94
AABB 2015-03-13 4.9 39.5 9.9 65.4 N/A N/A
interjections 2015-03-13 4.9 40.7 9.9 65.4 N/A N/A
auto 2015-03-12 4.9 40.7 9.9 65.4 N/A N/A
w/o auto 2015-03-12 1.2 48.1 3.7 64.2 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

mcg-smallest
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 24.32 (1.89) / 16.67 (2.33) 38.74 (2.81) / 22.22 (2.00) 45.05 (5.00) / 33.33 (5.00) 67.57 (5.00) / 44.44 (5.00) 100.0 24.32
AABB 2015-03-13 27.9 44.1 44.1 64.0 N/A N/A
interjections 2015-03-13 26.1 42.3 42.3 62.2 N/A N/A
auto 2015-03-12 26.1 42.3 42.3 62.2 N/A N/A
w/o auto 2015-03-12 4.5 46.8 7.2 56.8 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

mcg-wxl
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 21.36 (2.21) / 23.0 (1.40) 34.97 (3.06) / 31.0 (2.19) 45.89 (5.00) / 45.0 (5.00) 70.09 (5.00) / 55.0 (5.00) 100.0 21.36
AABB 2015-03-13 23.9 37.8 44.5 66.3 N/A N/A
interjections 2015-03-13 23.4 37.3 43.8 65.5 N/A N/A
auto 2015-03-12 23.4 37.3 43.8 65.5 N/A N/A
w/o auto 2015-03-12 3.8 46.6 5.5 56.2 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

pctb-test (the second 5,000 sentences)
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 0.0 3.68 (4.41) 0.08 (5.00) 7.38 (4.89) 0.0 0.0
AABB 2015-03-13 0.0 8.5 0.0 9.9 N/A N/A
interjections 2015-03-13 0.0 8.5 0.0 9.9 N/A N/A
auto 2015-03-12 0.0 8.5 0.0 9.9 N/A N/A
w/o auto 2015-03-12 0.0 9.2 0.0 8.5 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

sinica-test (the second 5,000 sentences (nltk_data))
Changes Date ordinary unk br unk+br generation end-to-end
gTest 2015-03-20 0.0 5.14 (4.95) 0.02 (5.00) 3.74 (4.68) 0.0 0.0
AABB 2015-03-13 0.0 1.8 0.1 1.5 N/A N/A
interjections 2015-03-13 0.0 1.7 0.1 1.5 N/A N/A
auto 2015-03-12 0.0 1.7 0.1 1.5 N/A N/A
w/o auto 2015-03-12 0.0 1.7 0.0 1.1 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

Home | Forum | Discussions | Events

Clone this wiki locally