That spurred the engineers of the Defense Digital Service the so-called “SWAT team of nerds” that tackles the Pentagon’s thorniest IT problems to make patching the vulnerability a top priority. Even then, it took nearly a year to complete what engineers consider a minor technical fix.

It’s a saga that illustrates the massive logistical challenges facing the world’s most powerful military as it tries to keep up with hackers intent on pilfering some of the country’s most sensitive data. As China, Russia and profit-seeking criminals ramp up their efforts to tunnel into U.S. systems, the federal government’s bureaucracy often stands in the way of its own efforts to be nimble on cybersecurity.

Informed of the fix by POLITICO, an aide to Sen. Ron Wyden (D-Ore.) called it welcome but long overdue.

Anything that we can do to make life more difficult for our adversaries is a good thing, the aide said. Wyden, who serves on the Intelligence Committee, called out the Pentagon four years ago for failing to protect employees emails from hackers and foreign spies.

The aide noted that Wydens office had recently reached out to DoD for an update on its efforts.

The flaw didnt compromise the Pentagons classified communications or internal mail.mil emails. But it meant that DoDs unclassified electronic conversations with outsiders were essentially naked as they traveled server to server across the internet.

That posed a risk for the vaccine push, opening the door for hackers to read trade secrets or launch spearphishing email attacks aimed at gaining access to other parts of DoDs network. The Pentagon was already breached in such an attack in 2015, when suspected Russian hackers compromised an unclassified email server used by Joint Chiefs.

The root of the problem: The Pentagon never fully implemented a widely used security protocol, known as STARTTLS, that makes it easier for email servers to exchange encrypted messages. The protocol was created in 2002, but over the years the department enabled it only for communications with a handful of external agencies.

Even when the Pentagon overhauled its email safeguards in 2017 and 2018, its Defense Information Systems Agency opted not to buy a security certificate that would vouch for the authenticity of DoD emails instead creating its own, less universally accepted version.

The setup ensured that Pentagon emails could be encrypted as long as they remained within the departments networks. But messages lost that protection once they reached the outside world, where most email systems didnt trust the departments homegrown certificate.

The pandemic changed all that, by hastening efforts to adopt STARTTLS for all traffic crossing DoDs email gateway.

Government bureaucracy is often on a slippery slope that slides into the outdated reasoning that Because weve always done it this way outweighs the better logic: Because this is the right answer, said Goldstein, whose team highlighted the lack of basic email encryption in 2019. Solutions that might otherwise seem obvious can get sidelined and forgotten, often because it is unfamiliar and foreign.

Goldsteins team got the go-ahead and the resources it needed in the early days of the pandemic. He assigned three engineers to the effort and recruited the Pentagons CIO for extra muscle to cut through layers of bureaucracy.

Cleghorn, the lead engineer, said that even then there were lots of stop-and-go and odd hurdles that we had to overcome.

They called the effort Project Groot, after a character from Marvels Guardians of the Galaxy movies.

Groot is a tree-like character that’s resilient to fire and has the ability to regenerate, which is fitting for this project, DDS chief Brett Goldstein said in an email. He also has excellent taste in music!

Even with buy-in from on high, enabling STARTTLS something that should take minutes became a nearly yearlong effort of testing and editing policies that hadnt been implemented with a government-wide pandemic fight in mind.

DDS ultimately spent $3,000 to purchase a certificate from a company called Entrust. Spending $3,000 to secure over 2 million email accounts was a drop in the bucket to resolve a lingering issue and significantly improve our security posture, Goldstein said.

From a technical perspective this is like an hours worth of work, said Cleghorn. It’s getting a certificate and installing it on the mail gateway which is just File, Browse, Click, Click, Upload and then attaching it to that profile.

Roger Greenwell, the risk management executive at the Defense Information Systems Agency responsible for signing off on the change, said most of the holdup wasnt about instituting the fix, but in analyzing what impact hitching a new commercial certificate would have on DoDs existing email system and network architecture.

For all intents and purposes you can almost think of it as somewhat a relatively minor software upgrade, Greenwell said.

The shift by DoD drew applause from people who have urged wider adoption of STARTTLS following former NSA contractor Edward Snowdens revelations of government mass surveillance in 2013. But some had only limited praise for the departments decision to finally catch up with the rest of the world.

Alexis Hancock, a technologist at the Electronic Frontier Foundation, said the move warrants only a golf clap because calls for adopting STARTTLS became more urgent and widespread post-Snowden.

DoDs conversion also looks long overdue considering Google started an effort to shame organizations into switching to the protocol in 2014.

But now that it has adopted email encryption for itself, Hancock argued, DoD should support encryption efforts for the government and the public.

For now, she had just one message for the Pentagon: Welcome to the encryption party.