so I ran CYK with Viterbi without smoothing out unknown words and as a result I got:
-- len<=15 --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 31.71
Bracketing Precision = 41.81
Bracketing FMeasure = 36.06
Complete match = 0.66
Average crossing = 1.42
No crossing = 51.41
2 or less crossing = 75.79
Tagging accuracy = 83.71
does this match your findings so far? I want to know before I start smoothing :)
Update:
-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 77.30
Bracketing Precision = 78.17
Bracketing FMeasure = 77.73
Complete match = 21.56
Average crossing = 0.73
No crossing = 70.48
2 or less crossing = 89.39
Tagging accuracy = 92.66
I get "Bracketing FMeasure = 67.41" for the baseline method (no smoothing, generate flat tree with bogus POS-tags for un-parsable sentences).
ReplyDeletehth,
Andrei
What are the parameters you used for evaluation? Khalil told me that these are probably the problem for me :)
ReplyDeleteFor anyone wondering:
ReplyDeleteMy precision is correct according to Gideon but he recommends removing all sentences longer than 15 from the test set and the gold standard
here is a (very ugly) python program that does just that:
#run it with: python removetoolong.py test.sentence.23 test.sentence.15 test.trees.23 test.trees.15
import sys
# f is test.sentence.23
# f2 is the file to write the sentences to
# f3 is test.trees.23
# f4 is the file to write the trees to
f = open(sys.argv[1], 'r')
f2 = open(sys.argv[2], 'w')
f3 = open(sys.argv[3], 'r')
f4 = open(sys.argv[4], 'w')
lines = f.readlines()
lines2 = f3.readlines()
for line,line2 in zip(lines,lines2):
if(len(line.split())<16):
f2.write(line)
f4.write(line2)
I didn't use the parameter file option. If I use "-p COLLINS.prm" I get better results, but I think that's misleading, because I think some options there make it indirectly skip some of the sentences.
ReplyDeleteBtw, I wouldn't blindly follow all of Gideon's remarks.
- Andrei
Just realized my last comment may sound a bit weird. What I meant was, while Gideon's remarks usually provide good insight, it happened to me a couple of times to find them misleading. Filtering the data though is of course helpful.
ReplyDelete=== Summary ===
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 77.26
Bracketing Precision = 78.02
Bracketing FMeasure = 77.64
Complete match = 21.39
Average crossing = 0.73
No crossing = 70.32
2 or less crossing = 88.89
Tagging accuracy = 92.61
Quite good results... I did nothing special though, wonder why it's so good o_O
...
ReplyDeletewut?
no but seriously, is this without smoothing? because this is way over what the parser should be capable of :)
Totally without smoothing. Only thing I did was adding all non-terminals that can lead to a terminal for non-existing words, with 10e-20 as a uniform probability.
ReplyDeleteSurprisingly, the parser guesses a lot of the unknown words correctly this way. I manually checked some of my results, and it really scores 100% on a lot of the trees ^_^
Also: I used your script to remove the long sentences Robrecht.
That be smoothing right there!
ReplyDeleteanyhow: did you add the preterms as (NP (PN name)) or as (NP-PN name)
I guess I am most surprised about the tagging precision, which is higher for you. This leads me to believe that one of us has erred somewhere... probably me. I vill investigate!
Unary reproductions were added as (NP (PN name)) ^_^
ReplyDeleteI found something, remove in python did not work as I intended :) using pop now and everything seems better, will post new results in a bit :)
ReplyDeleteThe program is now also way faster since pruning does work correctly :D
ReplyDeleteNice!!!! XD Show the results! ^_^
ReplyDeletecurrent results:
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 42.28
Bracketing Precision = 42.48
Bracketing FMeasure = 42.38
Complete match = 1.82
Average crossing = 1.82
No crossing = 32.67
2 or less crossing = 69.15
Tagging accuracy = 86.38
Which is better than before and match Gideon his results.
Any more people who have results?
I'd love it if more people posted their results!!!! ^_^
ReplyDeleteThis is what we got:
ReplyDelete=== Summary ===
-- All --
Number of sentence = 603
Number of Error sentence = 12
Number of Skip sentence = 0
Number of Valid sentence = 591
Bracketing Recall = 71.98
Bracketing Precision = 71.48
Bracketing FMeasure = 71.73
Complete match = 17.43
Average crossing = 0.75
No crossing = 68.02
2 or less crossing = 88.66
Tagging accuracy = 92.03
I too am a bit surprised at our results, I don't think we did anything really special with respect to smoothing. However, we do have a dozen error sentences so maybe we can polish it up a bit more. But we'll only do that if we have time to spare...
Oh and a small tip: if you're also getting several error sentences and the evaluation script stops running before it reaches the last tree, just change the maximum number of errors allowed in the parameter file. Its 10 by default, so if you go over this amount it terminates the program.
ReplyDeleteIt is better to run it without the parameter file: it is what is causing most errors :)
ReplyDeleteAnother bug another sense of utter stupidity :)
ReplyDeleteI made the reverse look up table in such a way that it didn't return a list of possible rules that produced the two symbols...
The only logical next step is Seppuku :)
First couple of trees now look identical to your results Tijmen
Lovely! I think Thomas got results similar to Taco and mine's, so probably the results I posted are a rough indication for what a proper implementation should look like for everyone ^_^. Hope everything works out well for you robrecht!
ReplyDeleteFamiliar results:
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 77.30
Bracketing Precision = 78.17
Bracketing FMeasure = 77.73
Complete match = 21.56
Average crossing = 0.73
No crossing = 70.48
2 or less crossing = 89.39
Tagging accuracy = 92.66
Finally I got my results too :P
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 72.88
Bracketing Precision = 75.87
Bracketing FMeasure = 74.35
Complete match = 17.74
Average crossing = 0.67
No crossing = 70.81
2 or less crossing = 89.72
Tagging accuracy = 93.12
Similar, even though the bracketing stuff is a bit lower than yours
We also finnaly have our results.Curiously enough we obtained the highest complete match seen here but not the best precision or recall.
ReplyDelete-- All --
Number of sentence = 2416
Number of Error sentence = 0
Number of Skip sentence = 1813
Number of Valid sentence = 603
Bracketing Recall = 75.27
Bracketing Precision = 76.43
Bracketing FMeasure = 75.84
Complete match = 23.38
Average crossing = 0.66
No crossing = 72.31
2 or less crossing = 89.72
Tagging accuracy = 91.38
=== Summary ===
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 77.70
Bracketing Precision = 77.98
Bracketing FMeasure = 77.84
Complete match = 23.38
Average crossing = 0.74
No crossing = 70.48
2 or less crossing = 88.23
Tagging accuracy = 92.76
We had some trouble with our probabilities becoming to low. The new results are a bit better. They look comparable to the best scores here.
ReplyDelete-- All --
Number of sentence = 2416
Number of Error sentence = 0
Number of Skip sentence = 1813
Number of Valid sentence = 603
Bracketing Recall = 77.37
Bracketing Precision = 77.92
Bracketing FMeasure = 77.64
Complete match = 23.71
Average crossing = 0.59
No crossing = 73.47
2 or less crossing = 91.71
Tagging accuracy = 91.84
Our final results. We learned a decision tree on infrequent words from the training set for smoothing, using features like the ascii values of letters in the words, common suffixes, numericals etc. This improved our tagging accuracy by only a small bit... But the tree looks cool, so >.<
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 77.93
Bracketing Precision = 78.46
Bracketing FMeasure = 78.19
Complete match = 22.72
Average crossing = 0.72
No crossing = 70.65
2 or less crossing = 89.05
Tagging accuracy = 93.26
Using the smoothing method as proposed by Khalil yielded the following (using 8 features):
ReplyDelete-- All --
Number of sentence = 603
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 603
Bracketing Recall = 75.20
Bracketing Precision = 76.96
Bracketing FMeasure = 76.07
Complete match = 19.90
Average crossing = 0.79
No crossing = 67.50
2 or less crossing = 88.39
Tagging accuracy = 92.92
So tagging accuracy has gone up whilst bracketing precision has gone down