How Google took over the world

Smaller companies claim the search tech is actively suppressing possible competition, and there are mounting calls for an antitrust probe, writes Charles Duhigg

SHIVAUN Moeran and Adam Raff met, married and started a company — thereby sparking a chain of events that might, ultimately, take down this age of internet giants as we know it — because they were both huge nerds. In the late 1980s, Adam was studying programming at the University of Edinburgh, while Shivaun was focused on physics and computer science at King’s College London. They had mutual friends who kept insisting they were perfect for each other. So one weekend, they went on a date and discovered other similarities: They both loved stand-up comedy. Each had a science-minded father. They shared a weakness for puns.

In the years that followed, those overlapping enthusiasms led to cohabitation, a raucous wedding and parallel careers at big technology firms. The thing is, though, when you’re young and geeky and fall in love with someone else young and geeky, all your nerdy friends want you to set them up on dates as well. So Adam and Shivaun, who took Adam’s last name after marriage, approached the problem like two good programmers: They designed a dating app.

The app was known as MatchMate, and the idea was simple: Rather than just pairing people with similar interests, their software would put together potential mates according to an array of parameters, such as which pub they were currently standing in, and whether they had friends in common, and what movies they liked or candidates they voted for, and dozens of other factors that might be important in finding a life partner (or at least a tonight partner). The magic of MatchMate was that it could allow a user to mix variables and search for pairings within a specific group, a trick that computer scientists call parameterization. “It was like asking your best friend to set you up,” Shivaun told me. “Someone who says, ‘Well, you probably think you’d like this guy because he’s handsome, but actually you’d like this other guy because he’s not as good-looking, but he’s really funny’.”

Within computer science, this kind of algorithmic alchemy is sometimes known as vertical search, and it’s notoriously hard to master. Even Google, with its thousands of PhDs, gets spooked by vertical-search problems. “Google’s built around horizontal search, which means if you type in ‘What’s the population of Myanmar,’ then Google finds websites that include the words ‘Myanmar’ and ‘population,’ and figures out which ones are most likely to answer your question,” says Neha Narula, who was a software engineer at Google before joining the MIT Media Lab. You don’t really care if Google sends you to Wikipedia or a news article or some other site, as long as its results are accurate and trustworthy. But, Narula says, “when you start asking questions with only one correct answer, like, Which site has the cheapest vacuum cleaner? — that’s much, much harder.”

For search engines like Google, finding that one correct answer becomes particularly difficult when people have numerous parameters they want satisfied: Which vacuum cleaner is cheapest but also energy-efficient and good on thick carpets and won’t scare the dog? To balance those competing preferences, you need a great vertical-search engine, which was something Adam and Shivaun had thought a lot about.

Soon the Raffs began daydreaming about turning their idea into a moneymaker. They didn’t have the funds to compete with huge dating sites like Match.com, so they applied for a couple of patents and began brainstorming. They believed that their vertical-search technology was good — better, in fact, than almost anything they had seen online. Best of all, it was built to work well on almost any kind of data set. With just a bit of tinkering, it could search for cheap airline tickets, or great apartments, or high-paying jobs. It could handle questions with hard-to-compare variables, like what’s the cheapest flight between London and Las Vegas if I’m trying to choose between business class or leaving after 3pm?

As far as they could tell, their search technology performed better on such problems than Google did, which Adam discovered when he tried to buy an iPod online. “I spent half an hour searching Google for the lowest price, and it drove me completely mad,” he told me. It was impossible for him to figure out which sites were selling iPods and which were selling accessories, like headphones or charging cords. Or Google would show Adam one price, but then the actual price was completely different. Or there was an extra charge for shipping. It seemed to Adam his technology would do a much better job.

Google executives, had they known of Adam’s frustrations, probably wouldn’t have been surprised. For years, Google had been trying to build a tool for comparing online prices. “The idea was you should be able to input any item, and we’d show you the best place to buy it,” says Brian Larson, a technical lead for what was then named Froogle and today is called Google Shopping. Larson’s team was small — just himself and one other programmer at first, and roughly a dozen people at its height — and Larson would regularly test how Froogle compared with other online price-comparison services. “Sometimes we were neck and neck; sometimes, not so much,” Larson said. “We had a hundred million product listings, which was better than competitors.” But they were often outperformed by sites like PriceGrabber.com, which had many more employees devoted to price comparisons.

Froogle’s limitations tended to pop up particularly when users included too many search parameters. For a while, Larson had a specific test search that Froogle kept failing, something like “white running shoes and cheap and free shipping”. Inevitably, the first result would be a Christmas elf wearing running shoes that some guy was selling online. No matter how Google’s engineers fiddled with their coding, they couldn’t stop the elf from appearing as the top link. Eventually, a manager bought the elf so it wouldn’t appear in the search results anymore. “We made elf T-shirts,” Larson told me. “It became our mascot.”

Adam and Shivaun’s technology was good enough to tell the difference between an elf wearing running shoes and an actual pair of running shoes. It was good enough, in fact, to figure out which websites charged hidden shipping fees and which offered truly good deals. So the Raffs quit their jobs, hired a few programmers, spent months perfecting their technology and, in early 2006, unveiled Foundem.com, a vertical-search engine for finding cheap online prices, to a small group of friends and associates. Each time someone used Foundem to buy something, the Raffs would receive a small payment from the website making the sale. Adam and Shivaun weren’t sure their company would succeed — there were already a couple of other big price-comparison search engines, like PriceGrabber, NexTag and, of course, Google itself — but they figured this was how the internet was supposed to work: Two people with a new idea can take on giants and, if their technology is good enough, grow into colossi themselves.

The Raffs knew they would have to rely on Google to find customers. For one thing, as evidenced by the name Foundem, they weren’t marketing geniuses. (“It’s like we found ‘em for you, you know?” Shivaun explained.) But early tests indicated that Foundem usually came up high in Google’s search results whenever people submitted queries like “compare prices xr-1000 motorcycle helmets”. Six months later, they opened Foundem to the world, and initial traffic was encouraging. “Search engines liked the site,” Shivaun told me. “That’s supposed to be the recipe for success.” As long as their vertical-search technology was strong, the Raffs figured, Google would guide shoppers to their door.

GOOGLE has succeeded where Genghis Khan, communism and Esperanto all failed: It dominates the globe. Though estimates vary by region, the company now accounts for an estimated 87% of online searches worldwide. It processes trillions of queries each year, which works out to at least 5.5bn a day, 63,000 a second. So odds are good that sometime in the last week, or last hour, or last 10 minutes, you’ve used Google to answer a nagging question or to look up a minor fact, and barely paused to consider how near-magical it is that almost any bit of knowledge can be delivered to you faster than you can type the request. If you’re old enough to remember the internet before 1998, when Google was founded, you’ll recall what it was like when searching online involved AltaVista or Lycos and consistently delivered a healthy dose of spam or porn. (Pity the early web enthusiasts who innocently asked Jeeves about “amateurs” or “steel”.)

In other words, it’s very likely you love Google, or are at least fond of Google, or hardly think about Google, the same way you hardly think about water systems or traffic lights or any of the other things you rely on every day. Therefore you might have been surprised when headlines began appearing last year suggesting that Google and its fellow tech giants were threatening everything from our economy to democracy itself. Lawmakers have accused Google of creating an automated advertising system so vast and subtle that hardly anyone noticed when Russian saboteurs co-opted it in the last election. Critics say Facebook exploits our addictive impulses and silos us in ideological echo chambers. Amazon’s reach is blamed for spurring a retail meltdown; Apple’s economic impact is so profound it can cause market-wide gyrations. These controversies point to the growing anxiety that a small number of technology companies are now such powerful entities that they can destroy entire industries or social norms with just a few lines of computer code. Those four companies, plus Microsoft, make up America’s largest sources of aggregated news, advertising, online shopping, digital entertainment and the tools of business and communication. They’re also among the world’s most valuable firms, with combined annual revenues of more than half a trillion dollars.

In a rare display of bipartisanship, lawmakers from both political parties have started questioning how these tech giants grew so powerful so fast. Regulators in Missouri, Utah, Washington DC, and elsewhere have called for greater scrutiny of Google and others, citing antitrust concerns; some critics have suggested that our courts and legislatures need to go after tech firms in the same way the trustbusters broke up oil and railroad monopolies a century ago. But others say that Google and its cohort are guilty only of delighting customers. If these tech leviathans ever fail to satisfy us, their defenders argue, capitalism will punish them the same way it once brought down Yahoo, AOL and Myspace.

At the core of this debate is a question that is more than a century old: When does a megacompany’s behaviour become so brazen that it violates the law? In the early 1900s, just after the Industrial Revolution, the federal government provided an answer by suing one of America’s largest companies, Standard Oil, on the novel theory that big becomes bad when a giant uses its dominance not only to defeat its competitors but also to extinguish the possibility that competition might occur.

In its technological innovation, Standard Oil was the Google of its day. The company’s founder, John D Rockefeller, had become the richest man in America by spending millions of dollars hiring scientists to transform how oil was refined and transported. And those innovations earned the public’s admiration. In 1858, before Standard Oil was founded, lighting a home required whale oil, which cost up to $3 a gallon, putting illumination out of reach for all but the wealthiest of households. By 1885, after Standard Oil figured out how to refine kerosene, it cost just 8c a gallon to brighten the night. “Let the good work go on,” Rockefeller wrote to a partner. “We must ever remember we are refining oil for the poor man and he must have it cheap and good.”

Standard Oil’s technological discoveries gave the company huge advantages over its rivals, and Rockefeller exploited those advantages ruthlessly. He cut secret deals with railroads so that other firms had to pay more for transportation. He forced smaller refineries to choose between selling out to him or facing bankruptcy. “Rockefeller and his associates did not build the Standard Oil Co. in the boardrooms of Wall Street,” wrote Ida Tarbell, a muckraking journalist of the day. “They fought their way to control by rebate and drawback, bribe and blackmail, espionage and price cutting, and perhaps more important, by ruthless, never slothful efficiency of organization.”

In 1906, President Theodore Roosevelt ordered his justice department to sue Standard Oil for antitrust violations. But government lawyers faced a quandary: It wasn’t illegal for Standard Oil to be a monopoly. It wasn’t even illegal to compete mercilessly. So government prosecutors found a new argument: If a firm is more powerful than everyone else, they said, it can’t simply act like everyone else. Instead, it has to live by a special set of rules, so that other companies get a fair shot. “The theory was that competition is good, and if a monopoly extinguishes competition, that’s bad,” says Herbert Hovenkamp, co-author of a seminal treatise on antitrust law. “Once you become a monopoly, you have to start acting differently, and if you don’t, then what you’ve been doing all along starts breaking the law.”

The Supreme Court agreed and split Standard Oil into 34 firms. (Rockefeller received stock in all of them and became even wealthier.) In the decades following the Standard Oil breakup, antitrust enforcement generally abided by a core principle: When a company grows so powerful that it becomes a gatekeeper, and uses that might to undermine competitors, then the government should intervene. And in the last century, as courts have censured other monopolies, academics and jurists have noticed a pattern: Monopolies and technology often seem intertwined. When a company discovers a technological advantage — like the innovations of Rockefeller’s scientists — it sometimes makes that firm so powerful that it becomes a monopoly almost without trying very hard. Many of the most important antitrust lawsuits in American history — against IBM, Alcoa, Kodak and others — were rooted in claims that one company had made technological discoveries that allowed it to outpace competitors.

For decades, there seemed to be a consensus among policymakers and business leaders (though not always among targeted companies) about how the antitrust laws should be enforced. But around the turn of this century, a number of tech companies emerged that caused some people to question whether the antitrust formula made sense anymore. Firms like Google and Facebook have become increasingly useful as they have grown bigger and bigger — a characteristic known as network effects. What’s more, some have argued that the online world is so fast-moving that no antitrust lawsuit can keep pace. Nowadays even the biggest titan can be defeated by a tiny start-up, as long as the newcomer has better ideas or faster tech. Antitrust laws, digital executives said, aren’t needed anymore.

Consider Microsoft. The government spent most of the 1990s suing Microsoft for antitrust violations, a prosecution that many now view as a complete waste of time and money. When Microsoft’s chief executive, Bill Gates, signed a consent decree to resolve one of its monopoly investigations in 1994, he told a reporter that it was essentially pointless for the company’s various divisions: “None of the people who run those divisions are going to change what they do or think.” Even after a federal judge ordered Microsoft broken into separate companies in 2000, the punishment didn’t take. Microsoft fought the ruling and won on appeal. The government then offered a settlement so feeble that nine states begged the court to reject the proposal. It was approved.

What eventually humbled Bill Gates and ended Microsoft’s monopoly wasn’t antitrust prosecutions, observers say, but a more nimble start-up named Google, a search engine designed by two Stanford PhD dropouts that outperformed Microsoft’s own forays into search (first MSN Search and now Bing). Then those two dropouts introduced a series of applications, like Google Docs and Google Sheets, that eventually began to compete with almost every aspect of Microsoft’s businesses. And Google did all that not by relying on government prosecutors but by being smarter. You don’t need antitrust in the digital marketplace, critics argue. “When our products don’t work or we make mistakes, it’s easy for users to go elsewhere because our competition is only a click away,” Google’s co-founder, Larry Page, said in 2012. Translation: The government ought to stop worrying, because no online giant will ever survive any longer than it deserves to.

Google office in Dublin

ONCE Foundem.com was available to everyone, the company’s honeymoon lasted precisely two days. During its first 48 hours, the Raffs saw a rush of traffic from users typing product queries into Google and other search engines. But then, suddenly, the traffic stopped. Alarmed, Adam and Shivaun began running diagnostics. They quickly discovered that their site, which until then had been appearing near the top of search results, was now languishing on Google, mired 12 or 15 or 64 or 170 pages down. On other search engines, like MSN Search and Yahoo, Foundem still ranked high. But on Google, Foundem had effectively disappeared. And Google, of course, was where a vast majority of people searched online.

The Raffs wondered if this could be some kind of technical error, so they began checking their coding and sending email to Google executives, begging them to fix whatever was causing Foundem to vanish. Figuring out whom to write, and how to contact them, was a challenge in itself. Although Google’s parent company bills itself as a diversified firm with about 80,000 employees, almost 90 percent of the company’s revenues derive from advertisements, like the ones that show up in search. As a result, there are few things more important to Google’s executives than protecting the firm’s search dominance, particularly among the most profitable kinds of queries, such as those of users looking to buy things online. In fact, at about the same time the Raffs were starting Foundem.com, Google executives were growing increasingly concerned about the threats that vertical-search engines posed to Google’s business.

“What is the real threat if we don’t execute on verticals?” one Google executive emailed his colleagues in 2005, according to internal documents later shared with the Federal Trade Commission. “Loss of traffic from Google.com because folks search elsewhere for some queries,” he wrote, in answer to his own question. “If one of our big competitors builds a constellation of high-quality verticals, we are hurt badly,” the internal documents continued. Another executive put it more bluntly: “Google’s core business is monetizing commercial queries. If users go to competitors such as Amazon to do product queries, long-term revenue will suffer.”

Google executives began holding battle-plan meetings for the vertical war. Shortly after Foundem.com went online, one executive issued an order: Henceforth, Google’s own price-comparison results should appear at the top of many search pages, as quickly as possible, even if that meant disregarding the natural results of the company’s search algorithm. “Long term, I think we need to commit to a more aggressive path,” a high-ranking Google employee wrote to colleagues. Eventually, a mandate came from the chief executive: “Larry thought product should get more exposure,” a senior official wrote.

One way to get that exposure was to influence the rules governing how Google displayed search results. In 2006, Google instituted a shift in its search algorithm, known as the Big Daddy update, which penalized websites with large numbers of subpages but few inbound links. A few years later, another shift, known as Panda, penalized sites that copied text from other websites. When adjustments like these occurred, Google explained to users, they were aimed at combating “individuals or systems seeking to ‘game’ our systems in order to appear higher in search results — using low-quality ‘content farms,’ hidden text and other deceptive practices.”

Left unsaid was that Google itself generates millions of new subpages without inbound links each day, a fresh page each time someone performs a search. And each of those subpages is filled with text copied from other sites. By programming its search engine to ignore other sites doing the same thing that Google was doing, critics say, the company had made it nearly impossible for competing vertical-search engines, like Foundem, to show up high in Google’s results.

Shivaun and Adam sent email after email to Google executives, but no one responded with anything useful. So the Raffs started making phone calls. Those didn’t help much, either. Adam and Shivaun had worked in technology for decades. They were well known and had connections to important people inside Google and at other big firms. But none of that seemed to matter.

As the months went by and Foundem’s bank accounts dwindled, the Raffs, desperate, began approaching other websites, offering to adapt their technology to power those sites’ internal search engines. Soon they were providing back-end technology for a popular motorcycle site and a large magazine publisher. Eventually, about 2.5m people were seeing Foundem’s search results each month. Foundem was named one of Britain’s best travel comparison sites by The Times of London and celebrated on a popular British gadget show. But without traffic from Google, the Raffs were barely holding on.

Three years passed this way. Some nights, Shivaun would sit at her computer, exhausted, Googling phrase after phrase — How do you lift a Google website penalty? Who at Google reviews mistakes? Google and deindexed and phone number and help — hoping that some magic combination of words might yield a new solution. “It just felt so unfair,” Shivaun told me. “We had great technology. It was winning awards. But we couldn’t even get an explanation from Google about why we weren’t showing up.” Eventually, they sought out a public relations firm, in the hope that a newspaper article might get Google’s attention. The PR firm had an additional suggestion: Why not file an antitrust complaint? To Adam and Shivaun, that seemed like a waste of time. If Microsoft had been able to shrug off the antitrust attacks of the United States government, why would Google care about a complaint filed by some small firm?

But they didn’t see many other options. So Adam and Shivaun pulled out their laptops and began assembling a long document detailing everything they had experienced. Then they went to Brussels, to the headquarters of the European Commission, the agency charged with regulating competitive behavior, and filed a complaint accusing Google of violating antimonopoly laws.

AS THE YEARS passed, Shivaun and Adam got into the habit of visiting message boards where people obsessively discussed Google’s many peculiarities. They began to notice an interesting pattern among companies complaining about the search giant: Often, the aggrieved parties had, in some way, posed some kind of threat to Google’s business. And they seemed to have suffered dire consequences.

There was, for instance, Skyhook Wireless, which had invented a new navigation system that competed with Google’s location software and had signed major deals with the cellphone manufacturers Samsung and Motorola. Skyhook’s accuracy “is better than ours”, one Google manager speculated in an internal email later revealed in a lawsuit filed by Skyhook against Google. Not long after that note was written, according to the lawsuit, a high-ranking Google official pressured Samsung and Motorola to end their relationships with Skyhook — and implied that if they didn’t, Google could make it impossible for them to ship their phones on time. (Google has denied doing anything inappropriate.) Soon, Samsung and Motorola cancelled their Skyhook contracts. Skyhook sued Google, and though one suit was dismissed, Google ended up paying $90m to settle a patent-infringement claim. But by then it was too late. Skyhook’s founders, bereft of other partnership options, had been forced to sell their company at a large discount.

Then there was Yelp, a website with millions of user-generated reviews of local brewpubs, auto-body shops and other businesses. Yelp grew quickly as local queries — like “best nearby steakhouse” — became a third of all online searches. For years, Yelp appeared near or at the top of millions of Google searches. Google, hoping to capitalise on that traffic, tried to buy Yelp in 2009, but Yelp’s founders rejected those advances. Then Google started pulling Yelp’s content into its own results, which meant many users didn’t have to visit Yelp’s website. Yelp complained — to Google and later to the FTC — but Google said the only alternative was for Yelp to remove its content from Google altogether, according to documents filed with federal regulators. The same thing happened at other fast-growing review sites like TripAdvisor and Citysearch, which also complained to the FTC. “We still exist,” says Luther Lowe, a vice president at Yelp, “but Google did everything it could to ensure that we’d never present a threat to them. It’s bullying, but they’re the 800lb gorilla.”

The more Adam and Shivaun looked, the more examples they found. Getty Images had created a popular search engine to help users comb through the firm’s 170m photographs and other visual art. Then, in 2013, Google adjusted how it displayed images so that rather than directing people to Getty’s website, users could easily see and download Getty’s high-definition images from Google itself. “Our traffic immediately fell 85 percent,” says Yoko Miyashita, Getty’s general counsel. “We wrote to Google, and said, Hey, this isn’t cool. And their response was, ‘Well, if you don’t agree to these terms, we’ll just exclude you’ “ — by letting Getty remove itself from the search engine entirely, Miyashita said. “That’s not really a choice, because if you aren’t on Google, you basically don’t exist.”

TradeComet.com, which operated a vertical-search engine for finding business products, initially prospered by buying ads on Google, but as the site grew, Google “raised my prices by 10,000 percent, which strangled our business virtually overnight,” the company’s CEO at the time, Dan Savage, said when he filed an antitrust lawsuit in 2009. KinderStart.com, a vertical-search engine for parents, sued Google after it received a “PageRank” of zero, making it essentially unfindable. (TradeComet.com’s suit was dismissed on a technicality; KinderStart.com’s was dismissed for insufficient evidence.)

Shivaun and Adam filled notepads with the names of companies that had complained about Google’s tactics — eJustice, a vertical-search engine for legal information; NexTag, the fellow price-comparison site; BDZV, a group of German newspapers. They printed out lawsuits and regulatory complaints until their living room was a maze of paper.

Eventually the Raffs reached out to the FTC, which, they knew, was the American equivalent of the European Commission’s antitrust office, and the U.S. regulators invited them to visit. The FTC’s staff, it turned out, had been quietly collecting complaints about Google for years. In 2012, those officials wrote a confidential 160-page report that said Google had “adopted a strategy of demoting, or refusing to display, links to certain vertical websites in highly commercial categories”. That memo, about half of which was accidentally sent to reporters at The Wall Street Journal after they submitted a Freedom of Information Act request, said that “Google’s conduct has resulted - and will result - in real harm to consumers and to innovation.”

“Google has strengthened its monopolies over search and search advertising through anticompetitive means,” which “will have lasting negative effects on consumer welfare,” FTC officials wrote. They cited instances in which Google seemed purposely to be privileging less useful information, substandard search results and suboptimal links. “Although it displays its flight search above any natural search results for flight-booking sites, Google does not provide the most flight options for travellers,” the regulators wrote. Whereas a decade earlier someone searching for steakhouses would have seen a long list of websites, now the most noticeable results pointed to Google’s own listings, including Google maps, Google local search or advertisers paying Google. Some FTC staff recommended “that the Commission issue a complaint against Google” for copying material and certain advertising and contract practices, though not search-engine bias.

Google responded to the report’s claims by arguing that the changes it made to the search engine benefited users. “Our testing has consistently showed that users want quick answers to their queries,” Google said in a statement when contacted about this article. “If you are searching for weather, you probably want a forecast, not just links to weather sites.” And when it comes to online shopping, the statement read, “if someone is searching for products, they likely want information about price and where they can buy it. They probably don’t want to be taken to another site where they have to enter their search again ... We absolutely do not make changes to our search algorithm to disadvantage competitors.” Claims to the contrary, like those made by Foundem, are untrue, Google maintained. “We make hundreds of changes to search every year, all with the same goal: Delivering users the best, most relevant search results,” the company continued. “Each change, large and small, affects millions of sites, some who see their rankings improve, others who drop.” And, Google concluded, “our ultimate responsibility is to deliver the best results possible to our users, not specific placements for sites within our results”.

When the FTC’s politically appointed leadership considered the staff’s recommendations, they declined to sue Google, surprising many inside the agency. “While not everything Google did was beneficial, on balance, we did not believe that the evidence supported an FTC challenge,” the agency’s chairman at the time, Jon Leibowitz, said when he announced the decision in 2013.

The FTC’s decision, according to agency insiders, was motivated in part by a debate that has also sparked battles within antitrust courts over the last 40 years: Should the law protect consumers or encourage competition? They’re not always synonymous. “It wasn’t consumers who were complaining about Standard,” says Hovenkamp, the antitrust scholar. “It was the other oil companies.” Similarly, few users are kvetching about Google; it’s primarily other tech firms. United States judges have increasingly held that the government must show consumer harm to win in court.

Adam and Shivaun didn’t have to wait for the official FTC announcement to know that their case was going nowhere. Meeting with officials in Washington, they could tell: These people were not going to prosecute. They had come to the United States at their own expense. They had written memo after memo arguing that Google was treating them unfairly and as a result hurting users. They had done everything they were asked. Standard Oil controlled 64 percent of the market for refined petroleum when the Supreme Court broke it into dozens of pieces. Google and Facebook today control an estimated 60% to 70% of the U.S. digital advertising market. And the FTC seemed happy to let them keep doing it. To the Raffs, it felt as if history was repeating itself, as if the pointless, ineffectual Microsoft case was happening all over again. It felt as if nobody cared.

Bill Gates: His company, Microsoft, was sued by the US department of justice for antitrust issues but prosecutors threw in the towel when the tech firm was successful in appealing the judgement.

IF YOU ARE younger than 29 — which just happens to be the average age of a Google employee, according to a survey done by PayScale — then odds are good you don’t remember much about the Microsoft antitrust battles of the 1990s. So, a quick primer: For almost a decade, starting in 1993, federal and state prosecutors besieged Microsoft in courtrooms across the nation, arguing that the company had acted in ways that were predatory and dishonest to preserve its software monopoly. One Microsoft executive was quoted in court as threatening to “cut off” the “air supply” of a competitor. “Is Bill Gates the ‘90s answer to Don Corleone?” Time magazine asked. “I expected to find a bloody computer monitor in my bed,” a witness told investigators.

Along the way, Microsoft was accused of widespread bullying, coercion and general obnoxiousness. And Microsoft basically said: Whatever. “There’s one guy in charge of licenses,” Bill Gates told reporters after he signed a consent decree with the department of justice in 1994. “He’ll read the agreement.” Everyone else, the implication was, would ignore it.

Even when a judge ruled in 2000 that Microsoft was violating antitrust law, conventional wisdom held that the victory was largely Pyrrhic. Microsoft successfully appealed, and prosecutors eventually threw in the towel, agreeing to abandon their attacks and settle if Microsoft agreed to token reforms, such as making its products more compatible with competitors’ software and giving three independent observers unfettered access to the company’s records, employees and source code. Microsoft’s executives thought that three observers, versus 48,000 employees, sounded like pretty good odds.

This was the history the Raffs recalled when they heard the FTC was abandoning its investigation. But then, they also remembered a discussion they had once had with a lawyer named Gary Reback, who told them that everything they’d heard about the Microsoft trials was wrong. Reback is something of a legend in Silicon Valley, both because of his accomplishments as an antitrust provocateur and because of his anxious - some might say paranoid - worldview. Reback has been known to call other lawyers late at night and leave long, obsessively detailed voice mail messages about legal arguments and economic theories. He was featured on a 1997 cover of Wired magazine with the headline “This Lawyer Is Bill Gates’s Worst Nightmare,” a boast that wasn’t far-off: Working on behalf of clients like Netscape and Sun Microsystems, Reback had browbeaten the department of justice into suing Microsoft for antitrust.

By the time Adam and Shivaun started visiting the FTC, Reback had exchanged his antipathy of Microsoft for a disdain of Google and had accompanied them on their visits with regulators. There’s a loose coalition of economists and legal theorists who call themselves the New Brandeis Movement (critics call them “antitrust hipsters”), who believe that today’s tech giants pose threats as significant as Standard Oil a century ago. “All of the money spent online is going to just a few companies now,” says Reback (who disdains the New Brandeis label). “They don’t need dynamite or Pinkertons to club their competitors anymore. They just need algorithms and data.”

Reback had told Adam and Shivaun that it was important for them to keep up their fight, no matter the setbacks, and as evidence he pointed to the Microsoft trial. Anyone who said that the 1990s prosecution of Microsoft didn’t accomplish anything — that it was companies like Google, rather than government lawyers, that humbled Microsoft — didn’t know what they were talking about, Reback said. In fact, he argued, the opposite was true: The antitrust attacks on Microsoft made all the difference. Condemning Microsoft as a monopoly is why Google exists today, he said.

Surprisingly, some people who worked at Microsoft in the 1990s and early 2000s agree with him. In the days when federal prosecutors were attacking Microsoft day and night, the company might have publicly brushed off the salvos, insiders say. But within the workplace, the attitude was totally different. As the government sued, Microsoft executives became so anxious and gun-shy that they essentially undermined their own monopoly out of terror they might be pilloried again. It wasn’t the consent decrees or court decisions that made the difference, according to multiple current and former Microsoft employees. It was “the constant scrutiny and being in the newspaper all the time”, said Gene Burrus, a former Microsoft lawyer. “People started second-guessing themselves. No one wanted to test the regulators anymore.”

In public, Bill Gates was declaring victory, but inside Microsoft, executives were demanding that lawyers and other compliance officials — the kinds of people who, previously, were routinely ignored — be invited to every meeting. Software engineers began casually dropping by attorneys’ desks and describing new software features, and then asking, in desperate whispers, if anything they’d mentioned might trigger a subpoena. One Microsoft senior executive moved an extra chair into his office so a compliance official could sit alongside him during product reviews. Every time a programmer detailed a new idea, the executive turned to the official, who would point his thumb up or down like a capricious Roman emperor.

In the early 2000s, Microsoft’s top executives told some divisions that their plans would be proactively shared with competitors — literally describing what the company intended to create before software was even built — to make sure it wouldn’t offend anyone who was likely to sue. Microsoft’s engineers were outraged. But they went along with it.

And most important, as Microsoft lived under government scrutiny, employees abandoned what had been nascent internal discussions about crushing a young, emerging competitor: Google. There had been informal conjectures about reprogramming Microsoft’s web browser, the popular Internet Explorer, so that anytime people typed in “Google”, they would be redirected to MSN Search, according to company insiders. Or, perhaps a warning message might pop up: “Did you know Google uses your data in ways you can’t control?”

Microsoft was so powerful, and Google so new, that the young search engine could have been killed off, some insiders at both companies believe. “But there was a new culture of compliance, and we didn’t want to get in trouble again, so nothing happened,” Burrus said. The myth that Google humbled Microsoft on its own is wrong. The government’s antitrust lawsuit is one reason that Google was eventually able to break Microsoft’s monopoly.

“If Microsoft hadn’t been sued, all of technology would be different today,” Reback told me. We’ve known since Standard Oil that advances in technology make it easier for monopolies to emerge. But what’s less recognized is the importance of antitrust in making sure those new technologies spread to everyone else. In 1969 the justice department started a lawsuit against IBM for antitrust violations that lasted 13 years. The government eventually surrendered, but in an earlier attempt to mollify prosecutors, IBM eliminated its practice of bundling hardware and software, a shift that essentially created the software industry. Suddenly, new start-ups could get a foothold simply by writing programs rather than building machines. Microsoft was founded a few years later and soon outpaced IBM.

Or consider AT&T, which was sued by the government in 1974, fought in court for eight years and then slyly agreed to divest itself of some businesses if it could keep its most valuable assets. Critics complained AT&T was getting the deal of a lifetime. But then start-ups like Sprint and MCI made millions building on technologies AT&T championed, and AT&T found itself struggling to compete. It’s completely wrong to say that antitrust doesn’t matter, Reback argues. “The internet only exists because we broke up AT&T. The software industry exists because Johnson sued IBM.”

It was critical that the Raffs continue fighting, Reback told them. Social embarrassment and sustained attacks have the power to succeed when courtrooms or political agencies fail. After their FTC disappointment, the Raffs flew back to England to consider their options. And then one night they were at home watching television when the phone rang. Someone they had met in Brussels was calling to share some remarkable news. The European Commission had issued a decision on the complaint they filed six years before.

Danish politician named Margrethe Vestager

WHAT changed everything was a middle-aged Danish politician named Margrethe Vestager, who had recently been named the European Union’s commissioner for competition. Vestager was an unusual choice for the post. She wasn’t a populist crusader or a pro-business acolyte; she was, instead, a moderate whose claim to fame, at that point, was having served as an inspiration for the television show Borgen, a fictional series about a Danish politician. But Vestager was awarded the commissioner’s post in 2014 after arguing that European marketplaces needed to do a better job of giving everyone an equal chance to succeed. Since assuming her office, Vestager has become, unexpectedly, the most prominent antitrust official in the world, invited to speak at conferences and mobbed by autograph seekers.

By the time Vestager took office, Google had already transitioned its price-comparison service to its present incarnation, which is effectively an advertising system that prominently features links only from companies that pay for the promotion. (Users are notified by a small logo that says “sponsored.”) After reviewing the complaints submitted by the Raffs and others, Vestager announced she intended to formally charge Google with antitrust violations. (She has also embarked on investigations into the European tax practices of Starbucks, Amazon and Apple, as well as anticompetitive tactics at Qualcomm, Facebook and Gazprom.)

Over the next two years, Vestager’s staff reviewed data from 1.7bn Google queries. They scrutinized how people fared when they conducted searches on topics in which Google had a vested interest, versus those where the company had nothing to gain. Then, in June of last year, the commission issued its final verdict: “What Google has done is illegal under EU antitrust rules,” Vestager said in a statement released at the time. “It denied other companies the chance to compete on the merits and to innovate. And most important, it denied European consumers a genuine choice of services and the full benefits of innovation.” Google was ordered to stop giving its own comparison-shopping service an illegal advantage and was fined an eye-popping $2.7bn, the largest such penalty in the European Commission’s history and more than twice as large as any such fine ever levied by the United States.

The verdict rocked Silicon Valley. Some think Europe’s assertiveness makes it more likely American regulators will act as well. And there’s evidence that’s already starting. Donald Trump appealed to voters, in part, by attacking the tech monopolies. In a case of truly odd bedfellows, that puts him in alignment with Elizabeth Warren and Bernie Sanders, who have long called for greater scrutiny of technology companies. Last year, a group of Democratic lawmakers in Congress, led by Senator Amy Klobuchar of Minnesota, sponsored legislation to boost antitrust enforcement by forcing companies to assume the burden of showing that a merger won’t hurt the public.

Meanwhile, a bipartisan assortment of state attorneys general have urged the FTC to reopen its investigation of Google. Most major antitrust battles, including the federal suits against Microsoft and Standard Oil, have begun as state actions. A Missouri investigation is particularly notable because the state’s Republican attorney general, Josh Hawley, who is running for the United States Senate, has subpoenaed information to see if Google has manipulated searches to disadvantage potential competitors. “The Obama-era FTC did not take any enforcement action against Google, did not press this forward and has essentially given them a free pass,” Hawley told reporters after revealing his inquiry in November. “I will not let Missouri consumers and businesses be exploited by industry giants.”

As attacks against Google have escalated, the company has tried to limit the damage. After Yelp complained to the FTC about Google’s stealing its content, Google promised to make it easier for websites to opt out of automatic copying, a pledge it reaffirmed a few months ago. And earlier this month, in exchange for Getty Images’ withdrawing its complaint to the European Commission, Google signed a licensing agreement with Getty promising to more clearly display images’ copyright information. Other titans like Facebook are similarly trying to get ahead of criticisms, voluntarily pledging greater transparency and promising to work more cooperatively with regulators.

The implication is clear enough: Google and the other tech titans understand that the landscape is shifting. They realize that their halos have become tarnished, that the arguments they once invoked as a digital exception to American economic history — that the internet economy is uniquely self-correcting, because competition is only a click away — no longer hold as much weight. “When you get as big as Google, you become so powerful that the market bends around you,” Vestager told me. The notion that antitrust law isn’t needed anymore, that we must choose between helping consumers or spurring competition, no longer seems sufficient reason to exempt the tech giants from century-old legal codes. If anything, Vestager’s verdict and state investigations indicate that companies like Google may have more in common with the monopolists of old than most people thought. Silicon Valley’s bigwigs ought to be scared. “If Europe can prosecute Google, then we can as well,” says William Kovacic, a law professor and former Republican-appointed chairman of the FTC. “It’s just a question of willingness now.”

IF THE internet’s potentates are frightened, however, they’re doing a good job of hiding it. Google has appealed the European Commission’s decision and has vigorously defended itself online. The company’s arguments are the same ones that it was putting forth on company blogs over the course of the investigation. “We disagree with the European Commission’s argument that our improved Google Shopping results are harming competition,” Google’s top lawyer wrote in one post. The commission “drew such a narrow definition around online shopping services that it even excluded services like Amazon,” undermining the contention that Google is dominant. “Google delivered more than 20bn free clicks to aggregators over the last decade,” he wrote in another post. Forcing it to “direct more clicks to price-comparison aggregators would just subsidize sites that have become less useful for consumers.” Google’s data indicates that users appreciate how the search engine has shifted over the years. “That’s not ‘favoring’” Google’s interests, the company said. “That’s giving customers and advertisers what they find most useful.”

Some legal theorists think that Google might have a point. “To what extent are consumers, rather than competitors, being harmed by Google?” says Hovenkamp, the antitrust scholar. “If the answer is ‘not much,’ then I’m suspicious of an antitrust remedy.” Others say the risks are too high. “There are very real costs associated with suing a company like Google,” says Geoffrey Manne, executive director of the International Center for Law & Economics, a nonpartisan research center. “You’re potentially impairing a firm that provides vital services to millions of people, and potentially benefiting competitors who don’t deserve that support.”

Those are fair arguments. But they are also, in some ways, beside the point. Antitrust has never been just about costs and benefits or fairness. It’s never been about whether we love the monopolist. People loved Standard Oil a century ago, and Microsoft in the 1990s, just as they love Google today.

Rather, antitrust has always been about progress. Antitrust prosecutions are part of how technology grows. Antitrust laws ultimately aren’t about justice, as if success were something to be condemned; instead, they are a tool that society uses to help start-ups build on a monopolist’s breakthroughs without, in the process, being crushed by the monopolist. And then, if those start-ups prosper and make discoveries of their own, they eventually become monopolies themselves, and the cycle starts anew. If Microsoft had crushed Google two decades ago, no one would have noticed. Today we would happily be using Bing, unaware that a better alternative once existed. Instead, we’re lucky a quixotic antitrust lawsuit helped to stop that from happening. We’re lucky that antitrust lawyers unintentionally guaranteed that Google would thrive.

Put differently, if you love technology — if you always buy the latest gadgets and think scientific advances are powerful forces for good — then perhaps you ought to cheer on the antitrust prosecutors. Because there is no better method for keeping the marketplace constructive and creative than a legal system that intervenes whenever a company, no matter how beloved, grows so large as to blot out the sun. If you love Google, you should hope the government sues it for antitrust offences—- and you should hope it happens soon, because who knows what wondrous new creations are waiting patiently in the wings.

For the Raffs, however, it’s probably too late. By the time Vestager announced her verdict and record-setting fine last year, it had been 12 years since Adam and Shivaun started Foundem.com. During that time, their lives slowly but inexorably became devoted to battling Google. They had spent thousands of hours corresponding with regulatory agencies across the globe. They had filed a civil suit against Google in British court, a case that is ongoing. They basically shut Foundem, creating more time for them to give advice to other companies and regulators fighting Google. This consulting work, some of which was funded by Google’s competitors, has helped to keep the Raffs afloat. And if the Raffs win their lawsuit against Google, it could be worth millions. “But it’s a different business model than we expected,” Adam told me. “It’s also deeply frustrating, because we became technologists in order to build new technologies. We never intended to be professional plaintiffs or antitrust crusaders.”

One of the most difficult things for the Raffs over the past decade has been figuring out how to explain this journey to themselves and others. Even friends and family didn’t fully understand what was going on. “It feels really good to be validated like this, to be told we were right,” Shivaun told me, referring to Vestager’s verdict. “But that doesn’t turn back the clock and give us another chance. Even if we win in Brussels, or win our lawsuit, in some ways, we were still defeated. We were still beaten by Google.”

Charles Duhigg is the author of Smarter Faster Better and The Power of Habit. Copyright: New York Times