Missing repeating article links

[jodaiber]

Hey guys,

there was one issue I ran into when using your tool for Spotlight: In Wikipedia, only the first occurrence of a surface form within an article is linked. However for training, you want to have all occurrences within each single article (subsequent occurrences of a SF are assumed to link to the same page as the first occurrence). In pignlproc, these are artificially introduced into the training. Your tool is missing these so far and hence you're missing a lot of tokenCounts and the sfCounts are incorrect.

I fixed this in our fork, but I also introduced quite a few Spotlight-related changes. I can send a separate pull request if you want. It makes the extraction a bit slower, but there is plenty of room for improvement if you want to make it faster/more memory-friendly.

Here's the relevant commit.

Jo

[samhumeau]

It looks good.

I will try it later today to see if it is fast enough, and then merge if you want to.

[jodaiber]

Cool! Keep in mind though that I also adapted/butchered Launcher.java to make it more easy to use with our index_db.sh.