Yes, I want Palantir to munch my public data
Government incompetence is not a civil rights protection
Here’s an open and obvious secret: the U.S. government collects vast data across every activity on every person living in the country. If you bank, use transportation, walk outside your home, pay taxes, or do any regulated activity from fishing to investing — there is a vault with your name on it (either digitally or frankly more frightening, in meatspace). There’s a reason that “off the grid” in the privacy sense is an all-encompassing lifestyle choice — it’s hard not to radiate your every action in front of the government’s panopticon.
This accumulation doesn’t matter much, though, because the U.S. government is wholly incompetent at connecting any of it together. Data is spread across local, county, state and federal governments. It is also fragmented across dozens of disparate agencies, none of which know how to cooperate. At a recent Riskgaming meetup, a former top data director at the Pentagon described the Kafkaesque process it took just to create standardized data interchange formats to integrate two databases. Multiply that by infinity, and that’s the government’s big data problem.
In individual cases, of course, incompetence can be overcome through sheer will. When the authorities want to track down, say, a terrorist, they can flood enough manpower into the byzantine warrens of databases and actually synthesize a complete portrait. The data is just sitting there, ready for whoever asks for it (sometimes with — but often without — a court order).
Much as cybersecurity experts will say that security through obscurity isn’t a great plan, privacy through incompetence isn’t a surefire way to maintain our civil rights.
Thus my bemusement at the extraordinary level of vitriol directed toward Palantir and the work of the Trump administration to update the government’s IT stack. The most recent salvo came late last week in the form of a The New York Times investigation that detailed Palantir’s contract victories across a range of agencies:
The push has put a key Palantir product called Foundry into at least four federal agencies, including D.H.S. and the Health and Human Services Department. Widely adopting Foundry, which organizes and analyzes data, paves the way for Mr. Trump to easily merge information from different agencies, the government officials said.
Trump’s ultimate objective of building a centralized database has led to intense online blowback from activists on both the left and the right. The main thrust of their arguments is that a unified system is the slippery path to a dystopian social credit system, à la China. The backlash is particularly notable from the right, since it combines an intense hatred of the so-called deep state with a profound desire for that very deep state to fulfill Trump’s policies.
Left unsaid amidst the arguments is what exactly all of this work in improving the government’s databases would actually lead to. Simplifying tax payments. Massively speeding up benefits applications. Improving criminal justice efficiency and accuracy. More easily uncovering waste, fraud and abuse. The kinds of automated activities that are quotidian in the private sector but verboten in the public sector. As I wrote last summer in City Journal on adding more AI into government:
Rather than enter a technological cul-de-sac, federal, state, and local governments must stay competitive with the private sector’s best practices. That means taking more humans out of the decision-making loop. Humans and machines are both ultimately black boxes; decision systems can be intentionally designed for optimal transparency and due process. Far from a computational dictator usurping the powers of free citizens, AI, properly implemented, is just another extension of a well-functioning republic.
I realize we aren’t a well-functioning republic, and so the hope is to just cordon the authorities off from our lives as much as possible. Yet … it’s the U.S. government, and it’s here to stay. The best protection we can afford ourselves is to upgrade not just our data infrastructure, but the institutions that manage it too. Once again, incompetence is not a credible protection against the violation of privacy.
What would a more credible system look like? Laurence and I just recorded a Riskgaming podcast episode with the technologist Joel Burke about his new book Rebooting a Nation, which chronicles the rise of Estonia’s digital government (we’ll publish the episode in a few weeks). How did the citizenry of a former Soviet republic once terrorized by the secret police allow its newly-democratic government to combine so many disparate data sources together into a mostly unified whole?
The answer from our conversation was essentially transparent legibility. Decisions around data were made deliberately and openly, ensuring citizen participation. Robust privacy controls and notifications were carefully architected to ensure that everyone felt they had even more control of their data under the new system than before. Also, Estonia’s digital government didn’t just suddenly materialize into existence, but accreted more and more data sources over time as part of a lengthy transition. The government made early useful gains that encouraged people to continue supporting its expansion over time.
In short, the country simultaneously gained efficiency and privacy as part of its digitalization efforts. We shouldn’t be surprised by this outcome, since keeping everything as paperwork is hardly a guarantee of either goal.
One crucial difference between Estonia and America though is that the U.S. government lacks the capability to do this data engineering, and must thus outsource it to the private sector in the form of Palantir. That’s a typical issue with American state capacity these days, particularly in technology. It’s a worthy debate on whether the government should have this capability in-house, but the reality is that it doesn’t. There is no agency capable of doing this work, since no agency has the political wherewithal or the workforce to cut through endless debates on interchange formats and just deliver an excellent product.
That leaves Palantir as an almost sole-source provider of these services. For left and right activists, Palantir, alongside CEO Alex Karp and founder and board member Peter Thiel, seem to represent some sort of four horsemen of the apocalypse (I’ll leave it as an exercise to the reader on which horsemen the two represent). Peel off the layers of invective and counter-invective though, and the controversy remains all smoke and no fire. Palantir is a full-stack data software and consulting firm that integrates databases and offers visualization and analysis functions on demand. It’s a good product, but hardly the sort of Orwellian super-system that its detractors see in the glint of Karp’s eyes.
That criticism has remained unchanging for more than a decade. Years ago as a journalist, I fought a skirmish with Palantir during its direct listing to trade publicly. I leaked its S-1 filing, and then wrote a series of more than a dozen stories on how the company was forming its corporate governance structure. I cared less about the Palantir bit of the story, mind you, and far more about what its novel governance model meant for the future of shareholder rights for publicly-traded equities (we all have our thing). Palantir mostly relented at the prompting of the SEC. What I learned from that experience though is that no one looked at Palantir as a business like any other.
Instead, activists turned Palantir into the political linchpin on whether the government should use data effectively or not. I strongly think the government — scaffolded by careful privacy protections and transparent due process procedures — should be able to offer much more robust and efficient services by leveraging its massive data for my benefit and the benefit of everyone. So yes, I want Palantir to munch all of my public data that the government already stores on me.
Unlike most activists, I have zero faith that keeping data lying dormant in a vault holding boxes of papers will protect me. That’s a passive model of privacy that can’t be relied upon. Only through active efforts to create robust institutions as impervious as possible to foul actors can we be assured that the government’s powers are properly circumscribed.
The line to draw and how to build such institutions is a worthy debate. Unfortunately, data privacy is one of the toughest policies to work on in Washington. Data privacy proposals are almost always an immediate assault on the digital advertising industry, which makes tens of billions of dollars in profits annually and has an extraordinarily high budget for lobbying at the federal and state levels. Rules on privacy for the government are held up for the same reason, lest the momentum transfers from the public sector over to private sector regulation.
But that lethargy must be overcome. Centralized databases are coming to the government, and so is robust artificial intelligence. It is an inevitability that, for instance, small claims courts will adjudicate decisions algorithmically in the relatively near future, even just as triage. If you don’t believe me, consider how AI will help litigants file lawsuits with more documentation faster to the courts. The courts will need to automate just as much as their petitioners.
Bad databases aren’t a veto on good policy. We just get bad public services, which intensify the sense among U.S. citizens that their government can’t do anything well. We need to go the opposite direction. As I never tire of reminding you, the word state and the word statistics descend from the same Latin root. Bad statistics means bad government, as has been true since the dawn of civilization. The panopticon already exists — let’s actually start to benefit from it.
A lot of those databases/paper records are left separate by design. https://www.nbcnews.com/news/s-just-insanity-atf-now-needs-2-weeks-perform-routine-gun-trace-rcna39606 Not saying I agree with every record that isn't connected, but some of that is the definition of privacy.
Enjoyed reading your perspective on this Danny!