http://www.businessweek.com/magazine/palantir-the-vanguard-of-cyberterror-security-11222011.htmlPalantir, the War on Terror's Secret WeaponA Silicon Valley startup that collates threats has quietly become indispensable to the U.S. intelligence communityBy Ashlee Vance and Brad Stone
....
An organization like the CIA or FBI can have thousands of different databases, each with its own quirks: financial records, DNA samples, sound samples, video clips, maps, floor plans, human intelligence reports from all over the world. Gluing all that into a coherent whole can take years. Even if that system comes together, it will struggle to handle different types of data—sales records on a spreadsheet, say, plus video surveillance images. What Palantir (pronounced Pal-an-TEER) does, says Avivah Litan, an analyst at Gartner, is “make it really easy to mine these big data sets.” The company’s software pulls off one of the great computer science feats of the era: It combs through all available databases, identifying related pieces of information, and puts everything together in one place.
....
Depending where you fall on the spectrum between civil liberties absolutism and homeland security lockdown, Palantir’s technology is either creepy or heroic. Judging by the company’s growth, opinion in Washington and elsewhere has veered toward the latter. Palantir has built a customer list that includes the U.S. Defense Dept., CIA, FBI, Army, Marines, Air Force, the police departments of New York and Los Angeles, and a growing number of financial institutions trying to detect bank fraud. These deals have turned the company into one of the quietest success stories in Silicon Valley—it’s on track to hit $250 million in sales this year—and a candidate for an initial public offering. Palantir has been used to find suspects in a case involving the murder of a U.S. Immigration and Customs Enforcement special agent, and to uncover bombing networks in Syria, Afghanistan, and Pakistan. “It’s like plugging into the Matrix,” says a Special Forces member stationed in Afghanistan who requested anonymity out of security concerns. “The first time I saw it, I was like, ‘Holy crap. Holy crap. Holy crap.’ ”
....
Palantir has found takers for its data mining system closer to home, too. Wall Street has been particularly receptive. Every year, the company holds a conference to promote its technology, and the headcount swelled from about 50 people at past events to 1,000 at the most recent event in October. “I saw bankers there that don’t go to any other conferences,” says Gartner’s Litan. The banks have set Palantir’s technology loose on their transaction databases, looking for fraudsters, trading insights, and even new ways to price mortgages. Guy Chiarello, chief information officer for JPMorgan Chase, says Palantir’s technology turns “data landfills into gold mines.” The bank has a Palantir system for fraud detection and plans to use the technology to better tailor marketing campaigns to consumers. “Google unlocked the Internet with its search engine,” Chiarello says. “I think Palantir is on the way to doing a similar thing inside the walls of corporate data.”
....
Using Palantir technology, the FBI can now instantly compile thorough dossiers on U.S. citizens, tying together surveillance video outside a drugstore with credit-card transactions, cell-phone call records, e-mails, airplane travel records, and Web search information. Christopher Soghoian, a graduate fellow at the Center for Applied Cybersecurity in the School of Informatics and Computing at Indiana University, worries that Palantir will make these agencies ever hungrier consumers of every piece of personal data. “I don’t think Palantir the firm is evil,” he says. “I think their clients could be using it for evil things.”.....
....
(much more at link, including examples of use, the company's strategies for avoiding misuse of information, and critical response)