General Discussion
Related: Editorials & Other Articles, Issue Forums, Alliance Forums, Region ForumsWhat you need to know about email de-duplication
I work in a field called Electronic Discovery, and we deal with processing, reviewing, and producing emails all of the time. De-duplication of emails has been around for years. There are several existing applications that can quickly de-duplicate an email set. The process is described below:
Most e-Discovery service providers would allow you to use a custom hash that includes your choice of metadata fields in addition to document contents for de-duplication. For example, you could choose to include the file name field in your custom hash if you would like to make sure that documents can be considered duplicates only when their file names are also identical.
You can read more about it here:
http://www.meridiandiscovery.com/articles/frequently-asked-questions-about-de-duplication/
--OnEdit--
I post this so that you will have reference materials to combat your right wing relatives at Thanksgiving when they go off about "rigged" system and you cannot review 650,000 emails.
Demit
(11,238 posts)For those of us who don't know much computer science-wise, how long would a search of 650K emails take? Are you able to guesstimate that?
Yavin4
(35,445 posts)A powerful application like Nuix could probably de-duplicate that many emails in two or three days.
BlueStreak
(8,377 posts)only about 0.001% of that 650,000 would be of interest. Most of the pictures were probably Anthony sending out pictures of his junk.
By simply filtering down to the emails sent to and by his wife, and then further selecting emails where Hillary was the author or recipient, you end up with a few hundred emails -- a set that can be manually perused in a few hours.
The question is not how they could get through that process in a week, but why it took more than a day. Why did they wait until 2 days before the election to admit there was nothing there?
freebrew
(1,917 posts)of 650,000(minus .001%) of pictures of Weiner's junk.
(No more comment)
duncang
(1,907 posts)Where did that number come from? Was it actually confirmed by the fbi that it was the total number of emails? I bet flynn heard there were 650,000 + files total on the computer. Since she was Secretary of State for the entire 4 years. That would still be around 450 emails everyday being sent and received. Just seems like that is unlikely.
Wounded Bear
(58,706 posts)that there are 10 agents in a room somewhere comparing paper documents.
Electronically comparing 600k documents would take a few seconds.