Deduplicate repeated tweets?

[grinnellian]

I often see cases where multiple people attempt to RT w/ #defendPDX
(Heck, I've done it myself before I learned to get better at checking other retweets first)
However, slowing down to check for other RTs reduces the utility of the repeater somewhat.

How would you like to handle duplicates and deduplication?
Some ideas/cases to think about:

• Could a sufficient number of tweets from accts not on the allow-list cause a repeater RT?
• How far down a chain of RTs should a dupe be considered a dupe?
  e.g. if Alice tweets "King Bun on 3rd and Salmon" and Bernice RTs with "DefendPDX" then some time later (after more retweets) Francie retweets the lot saying "King Bun moved to 5th & Main #DefendPDX" we might want to RT the updated information.

[jmlingeman]

I could see this being implemented as a simple “90% of tokens are the same” or similar, but anything more sophisticated falls into the NLP realm where information duplication detection is a field of ongoing research and probably outside the scope of this bot.