The Thorny Problem of Keeping the Internet’s Time

In 1977, David Mills, an eccentric engineer and computer scientist, took a job at COMSAT, a satellite corporation headquartered in Washington, D.C. Mills was an inveterate tinkerer: he’d once built a hearing aid for a girlfriend’s uncle, and had consulted for Ford on how paper-tape computers might be put into cars. Now, at COMSAT, Mills became involved in the ARPANET, the computer network that would become the precursor to the Internet. A handful of researchers were already using the network to connect their distant computers and trade information. But the fidelity of that exchanged data was threatened by a distinct deficiency: the machines did not share a single, reliable synchronized time.

Over decades, Mills had gained wide-ranging expertise in mathematics, engineering, and computer science. In the early seventies, as a lecturer at the University of Edinburgh, he’d written programs that decoded shortwave radio and telegraph signals. Later, largely for fun, he’d studied how the clocks in a power grid could wander several seconds in the course of a hot summer’s day. (The extent of their shifts depended not just on the temperature but on whether the grid used coal or hydropower.) Now he concentrated on the problem of keeping time across a far-flung computer network. Clock time, Mills learned, is the result of an unending search for consensus. Even the times told by the world’s most precise government-maintained “master clocks” are composites of the readings of several atomic clocks. The master clocks, in turn, are averaged to help create international civil time, known as Coördinated Universal Time and initialized as U.T.C.

To solve the problem of time synchronization on the ARPANET, Mills built what programmers call a protocol—a collection of rules and procedures that creates a lingua franca for disparate devices. The ARPANET was experimental and capricious: electronics failed regularly, and technological misbehavior was common. His protocol sought to detect and correct for those misdeeds, creating a consensus about the time through an ingenious system of suspicion. Mills prided himself on puckish nomenclature, and so his clock-synchronizing system distinguished reliable “truechimers” from misleading “falsetickers.” An operating system named Fuzzball, which he designed, facilitated the early work. Mills called his creation the Network Time Protocol, and N.T.P. soon became a key component of the nascent Internet. Programmers followed its instructions when they wrote timekeeping code for their computers. By 1988, Mills had refined N.T.P. to the point where it could synchronize the clocks of connected computers that had been telling vastly differing times to within tens of milliseconds—a fraction of a blink of an eye. “I always thought that was sort of black magic,” Vint Cerf, a pioneer of Internet infrastructure, told me.

Today, we take global time synchronization for granted. It is critical to the Internet, and therefore to civilization. Vital systems—power grids, financial markets, telecommunications networks—rely on it to keep records and sort cause from effect. N.T.P. works in partnership with satellite systems, such as the Global Positioning System (G.P.S.), and other technologies to synchronize time on our many online devices. The time kept by precise and closely aligned atomic clocks, for instance, can be broadcast via G.P.S. to numerous receivers, including those in cell towers; those receivers can be attached to N.T.P. servers that then distribute the time across devices linked together by the Internet, almost all of which run N.T.P. (Atomic clocks can also directly feed the time to N.T.P. servers.) The protocol operates on billions of devices, coördinating the time on every continent. Society has never been more synchronized.

For decades, Mills was the person who decided how N.T.P. should work (though he disputes the suggestion that he acted with total sovereignty). Quirky, prickly, authoritative, and sometimes opaque—“He does not suffer fools gladly,” one longtime collaborator said—he has served as the Internet’s Father Time. But his tenure is coming to an end. Mills was born with glaucoma. When he was a child, a surgeon was able to save some of the vision in his left eye, and he has always worked using very large computer displays. Around a decade ago, his vision began to fail, and he is now completely blind. Examining computer code and writing out explanations and corrections have become maddeningly tedious. Drawing diagrams or composing complex mathematical equations is nearly impossible.

A couple of years ago, I visited Mills in his unassuming house in the Delaware suburbs. He and his wife, Beverly, have lived there since 1986, when Mills became a professor at the University of Delaware, a position he held for twenty-two years until his retirement. While we sat in his kitchen, our conversation was regularly interrupted by an automated voice announcing the time from the next room. The oven and microwave clocks were out of synch. Mills, who has a snow-white beard and wore a charcoal fisherman sweater, tracks the time for himself using a speaking wristwatch, which connects by radio signals to a master clock in Colorado.

He led me upstairs to his office, slowly making his way through the house by feeling for a series of memorized “navigation points.” At his desk, where a cat lay atop some crackling ham-radio equipment, Mills sat down at his computer. He used the keyboard to pull up a research paper he was working on, with suggestions for improvements to N.T.P. (He asks his wife and daughter to proofread what he types.) As he used the arrow keys to scroll, the computer spoke aloud. “This memo explores new security and protocol enhancements,” a voice said. “Blank. Table of contents. Blank. One. Two. Two point. . . . Three. Three. Four. Four point one. . . .” Soon, he got lost. “I do what I can using the voice that you hear,” Mills said. “But I observe myself and comment on the following: man was made to do English composition by eyeball.”

Technology doesn’t stand still. The Internet continues to grow in both scale and complexity; even as its infrastructure ages, our world depends upon its functioning to an ever-increasing degree. The continued evolution of the Internet’s time-synchronization system is essential. And yet Mills’s inability to swiftly contribute to N.T.P. has sapped his authority over it. In his absence, only a few people appear to be both capable and willing to oversee the critical yet overlooked software. A contest for influence over how clocks are kept in synch across the Internet has begun.

Mills was born in 1938 in Oakland, California, eleven years after the development of the first quartz clock and nine years before the construction of the first transistor. He took a steam-powered train to a school for the visually impaired, in San Mateo, and marvelled at the engineers who ran it. In his teens, he became a model-railroad and ham-radio enthusiast, communicating with friends and patching Navy Seabees at the South Pole through to their wives. His father, an engineer and salesman, co-founded National Oil Seal, a company that manufactured equipment to prevent leakage within machinery. (“You might not know what it is, but there are at least two of them in the engine of your car,” his father told him, of the seals.) His mother trained as a pianist at the Toronto Conservatory of Music before staying home to raise him and his two younger brothers.

The family moved around, and Mills’s teachers didn’t always accommodate his visual impairment. Mills recalls an eleventh-grade teacher telling him, “You’re never going to get to college”—a remark that was “like waving a flag in front of a bull,” he said. In 1971, Mills earned a Ph.D. in computer and communication sciences at the University of Michigan; after a two-year stint lecturing in Edinburgh, he moved with his wife and two children to the University of Maryland, which denied him tenure after five years. “It was the best thing that ever happened to me,” Mills said. He started work at COMSAT, where he had access to funding from the Department of Defense, some of which was earmarked for the ARPANET. “It was a sandbox,” he later told an interviewer. “We just were told, ‘Do good deeds.’ But the good deeds were things like develop electronic mail, and protocols.” Part of the allure of the time-synchronization work, he told me, was that he was just about the only one doing it. He had his own “little fief.”

In N.T.P., Mills built a system that allowed for endless tinkering, and he found joy in optimization. “The actual use of the time information was not of central interest,” he recalled. The fledgling Internet had few clocks to synchronize. But during the nineteen-eighties the network grew quickly, and by the nineties the widespread adoption of personal computers required the Internet to incorporate millions more devices than its first designers had envisioned. Coders created versions of N.T.P. that worked on Unix and Windows machines. Others wrote “reference implementations” of N.T.P.—open-source codebases that exemplified how the protocol should be run, and which were freely available for users to adapt. Government agencies, including the National Institute of Standards and Technology (NIST) and the U.S. Naval Observatory, started distributing the time kept by their master clocks using N.T.P.

A loose community of people across the world set up their own servers to provide time through the protocol. In 2000, N.T.P. servers fielded eighteen billion time-synchronization requests from several million computers—and in the following few years, as broadband proliferated, requests to the busiest N.T.P. servers increased tenfold. The time servers had once been “well lit in the US and Europe but dark elsewhere in South America, Africa and the Pacific Rim,” Mills wrote, in a 2003 paper. “Today, the Sun never sets or even gets close to the horizon on NTP.” Programmers began to treat the protocol like an assumption—it seemed natural to them that synchronized time was dependably and easily available. Mills’s little fief was everywhere.

N.T.P. works by telling computers to send tiny, time-stamped messages to time-checking devices superior to them in a hierarchy. The hierarchy’s uppermost layer consists of servers that are closely connected to highly accurate clocks kept in tight synchronization with Coördinated Universal Time. The time then trickles, from strata to strata, to the machines at the bottom of the hierarchy, such as ordinary laptops. The protocol tracks the instants that elapse as a time-checking message is sent, received, returned, and received again by its original sender. All the while, a collection of algorithms—the “popcorn spike suppressor,” the “huff-n’-puff filter”—sifts through the data, singling out falsetickers and truechimers and instructing the clocks on how to adjust their times based on what the time-stamped messages tell them.