Yield

DoubleClick

Within the span of twenty years, ads went from being bought and sold by humans, over the stereotypical “three martini lunches,” to globally traded commodities bought and sold like stocks.

Google really modernized the advertising industry. This book is the story of how.

Brian’s insight was that digital advertising was essentially a valuation problem. If you could value every ad properly, you could figure out the best one to show at any time, on any website. Previously everyone in advertising thought this was a matching problem, where the goal was to “show the right ad to the right person at the right time.” From Brian’s point of view, the right ad was the one willing to pay the most, making publishers the most money and giving the advertiser the most value.

[sic]

Once you could value every ad, you could also trade them. Like the Nasdaq, an ad exchange could match ad buyers and sellers in near real time, based on price, providing better results for everyone. The first ad exchange transaction took place within Right Media in April 2005 and would revolutionize the entire industry.

[sic]

In financial terms, the market went from one entirely based on futures contracts to a spot market.

This is a very important insight. It’s important to note that this was contrarian at the time! People simply did not think of ads in this way.

He frequently interrupted a speaker giving a long PowerPoint presentation with a pointed question about the implication that the presenter had not even considered. He had a habit of asking presenters to ignore all their slides and instead just say the two or three things that had to be done to be successful. This style proved valuable in the world of advertising and amid the dot-com era, where so much nonsense could cover up the important facts and truths.

Surprise surprise, Brian would not work out in a traditional “big tech” setting.

Brian, Matt, and their team were tasked with building out what was probably the first ad server designed specifically for performance. They delivered an initial version in about six weeks. The real breakthrough came when calculating the expected value of each impression and doing it in near real time.

Wild to think that there was a time that performance didn’t matter for ad servers.

The other insight was more cultural. “Google was easily distractable and fickle,” a former employee would recall. “If DoubleClick puffed out its chest and looked hard to beat it, they would eventually give in.”

I wonder how much Google has truly changed in areas outside of ads.

DoubleClick was not bidding first; it was waiting to see what other bids were eligible and at what prices and then bidding last, once it knew the price to win.

[sic]

DoubleClick was setting up a two-tier system for demand to compete: one that ran a smart auction against direct demand and one that was stuck guessing the right price and priority.

Really smart approach.

Clicker = DoubleClick employee

There just… has to be a better name than this.

Of course, the Yahoo term sheet asked for the deal to not be shopped to others, and, of course, that’s exactly what Neal and David proceeded to do.

LOL. “Please don’t advertise this.” “No.”

It may be a little hard to imagine today, but in the late 2000s, the advertising business was dominated by what were colloquially known as the “Big Three”: Yahoo, AOL, and Microsoft (MSN). These companies captured the lion’s share of ad dollars and were considered must-buys by most brands on digital media plans.

Crazy that all of them fell off as far as ads are concerned.

The Big Three were being challenged on all fronts by a technically sophisticated but decidedly awkward newcomer to the display business: Google. Google was dominant in search advertising and was flush with cash from its blockbuster IPO yet was struggling to break into display and video advertising. (The company bought YouTube in 2006 but did not start monetizing it until a couple of years later.)

Google would constantly be underestimated for years by the incumbents.

The rationale for buying DoubleClick was clearly laid out in bulleted form by David Drummond, the company’s general counsel: acquire the customers, prevent Microsoft from buying it, and unblock Google’s progress on building its own ad server and display ad network. The plan was to buy the company, reduce the head count, make ad serving free, and migrate the customers to Google products in about two years. This was not exactly how it played out.

The main reason Google was interested was first and foremost to prevent Microsoft from getting it.

The model was tweaked slightly before each presentation to better match the inventory and monetization assets of the target company. In one unfortunate incident, Susan Wojcicki interrupted Neal’s presentation to ask why the spreadsheet she was looking at said “Microsoft,” to which he coolly but improbably answered, “Typo” and moved on.

Amazing. World-class FOMO generation.

The news of DoubleClick’s sale process had leaked in the Wall Street Journal, with Microsoft identified as the probable buyer. This only served to accelerate the auction, with multiple escalating bids from all four of the YMAG companies.

Who do you think leaked it? DoubleClick CEO is running a masterful process.

“The motivating factor for Microsoft was that Google was interested. They thought, Let’s just do it and figure it out later. They didn’t really have a strategy,” recalled Rosenblatt.

The two spidermen pointing at each other.

Amusingly, the same rationale was driving the bidding at Google’s Mountain View headquarters. Google’s David Drummond wrote, “Our interest became more acute after learning that Microsoft had already made a serious offer” and speculated the deal would go for over $2 billion in cash.

This is too funny.

Google moved fast and acted decisively. They were able to get a signed letter of intent, or LOI, to buy DoubleClick for $3.1 billion.

Considering everything DoubleClick would become, it was easily worth ten times this price.

There was an incoming message from Microsoft’s corporate development team. They were willing to match the offer for DoubleClick, and the message included an email from CEO Steve Ballmer in which he opened the door for a much higher offer. Ballmer wrote that if the offer match was not acceptable, DoubleClick should simply mark the paper up to meet its needs and sign it, then Microsoft would review and rapidly countersign to close the deal with minimal negotiation required. Without saying so, Ballmer was communicating “Here’s a blank check—tell me what closes the deal.”

Gotta feel bad for Ballmer. He mostly did everything he could here.

There was radio silence and no communication about moving forward. A day before the LOI was set to expire, David and the team got an updated term sheet from Google. The financial terms had not changed, but now the deal included a “hell or high water” clause, meaning that Google was committed to closing the deal without any substantive diligence or other conditions. It was money in the bank. David and Hellman & Friedman took it and ran. At a price of $3.1 billion, Hellman & Friedman had gotten a return of eight or nine times their money in two years.

Wild. The Acquired Podcast covers this whole auction rather wonderfully, but the key takeaway here is that DoubleClick was really meandering without a vision for years prior to this and had gone through several ownership transfers. It was only just prior to this sale that it had begun to find true traction.

Thirty minutes later they had the outline of the acquisition—Yahoo would buy the rest of Right Media, the part it did not own, for $680 million. Brian O’Kelley would not come along for the ride. He was fired soon after the deal was announced. Michael Walrath would later say of Brian, “He was 95% genius and 5% disaster,” adding, “He wouldn’t have lasted five minutes at Yahoo.” Brian made a reported $25 million on his vested shares, but his firing left on the table tens of millions more and put a chip on his shoulder the size of Mount Everest. He got a very small measure of revenge by throwing water on the deal, telling the New York Times, “I have to say I giggled … there is no way we quadrupled the value of the company in six months.” This quote infuriated Jerry Yang and the Yahoo team in Sunnyvale. Asked whether the firing was deserved, Brian would later admit, “I’m not saying that everyone didn’t want to fire me.”

I love that last sentence. Incredibly self aware. But yeah, this guy was a visionary. He would never do well in a big tech environment. As a reminder, that third quotation in this post was about Brian.

Despite these drawbacks, four weeks after the DoubleClick announcement, on May 18, Microsoft announced it would acquire the smaller aQuantive for $6.1 billion.23 Internally, the Google team would joke that Microsoft got “half the company for twice the price.” One Google executive even prank-called Brian McAndrews, aQuantive’s CEO, to say, “You’re welcome.”

Absolute FOMO-driven acquisition.

In just four months, from April to August 2017, over $10 billion had been spent to acquire advertising technology companies. The common vision was to build liquid marketplaces for ads that would better bring together buyers and sellers—with the winner taking the spoils. The race to the finish line was more like the race to the starting line.

It’s important to remember just how much of a marathon these new initiatives are. It really was a race to the starting line.

Integrating with Google

Google had been betting on the traditional media business prior to the DoubleClick deal. Lost to history are products like Google Print, Google Radio, and Google TV Ads, all efforts to bring the efficiency of the auction-based search business to unsold inventory in traditional analog media.

Yikes. Google’s strategy prior to DoubleClick was just flat out bad.

The teams got along pretty well, other than one Googler whom the DoubleClick side termed “Febreze Boy” because of his endless boasting of working on the launch of that odor-reducing spray.

Why does every tech company circa this era hacve some story about body odor?

In a strange coincidence, the two companies’ New York headquarters were in the same building but on different floors. While legally employees were not allowed to communicate before the deal closed, they might have shared the elevators every morning. Google was known for giving its employees free food—a rare perk at the time—so Tom arranged a couple of lunches where the Googlers abandoned their cafeteria to avoid any contact and the whole New York DoubleClick office came down the stairs to enjoy the famous gourmet lunch.

They were in the same building! Insanity.

DoubleClick had roughly thirteen hundred employees at the time the deal was announced. The Performics sell-off would bring that down to about one thousand, but there was a strong push to trim that down substantially. Tom was paired with the exceptionally poorly named Patricia Severynse (pronounced “severance”) to model out where to make cuts and where to retain talent.

The best and funniest case of nominative determinism.

The DoubleClick business was an ongoing enterprise software concern, servicing hundreds, if not thousands, of customers. Between the institutional knowledge and the required operations of the company, cutting head count in this way would have been an abject disaster. This whole experience shed light on how unprepared Google was to enter a customer-centric business like the one they were acquiring. The company’s culture of engineering-led innovation, with limited input from the commercial side, was poison to the types of long-term relationships required to sell and maintain enterprise software.

Google had to learn the hard way that, even though the ads were digital, advertising was still (and is still) primarily a relationship-driven business. There are a few customers that account for the bulk of the ad spend. You cannot engineer your way around these relationships. Elon Musk found this out the hard way as well when he first acquired Twitter. Ad revenue fell precipitously after he alienated the publishers.

The goal ultimately would be to automate and optimize to make the ad serving businesses as close to self-service as possible, despite the facts that customers paid millions of dollars a year for the software and the complexity of the offering really necessitated hands-on help.

This is a classic engineering mindset. It would take a very long time before that would ever be the case.

In a foreshadowing of the attitude that eventually dominated Google’s approach to customers, the layoffs were born from the belief that the ad serving business was a means to an end rather than a product customers paid for and expected to work reliably.

Yup.

Over the coming years, most of the DoubleClick products would be rebuilt from scratch on the Google stack.

This is not so unusual; virtually every acquisition into Google goes through the same fate.

The existing DoubleClick ad server desperately needed to be rebuilt. It had been deployed around 2001, when the web was still nascent, and used nonstandard technologies that only worked on Windows PCs using the Internet Explorer browser. The product literally could not be operated in any way on a Mac or using any other browser.

Too funny. At least DoubleClick took Elon’s advice and chose to use Windows over Linux.

Enterprise software is very different from the kinds of tools Google was used to building. Unlike free tools, enterprise customers are extremely demanding and will not simply do what they are told when features or roadmaps change. The GAM team and their engineering leaders both massively underestimated the amount of functionality and nuance that the DoubleClick system had built over the years and overestimated the willingness of customers to live without some of those features.

GAM = Google Ad Manager.

This is a very typical flaw of engineers - they assume the systems are everything and ignore the people factor.

There was a natural inclination to dismiss the DoubleClick server as a yardstick for the future product. An email from one engineer told the team that the ad server only had about six major features that GAM lacked and that the migration would start in just six months. At one meeting, when Jonathan expressed his view that “I think you’re underestimating how big this project is,” the retort was the deeply arrogant “I think you’re underestimating Google engineers.”

DoubleClick was the leader for a reason. There are a lot of edge cases they accounted for.

The consensus was it would take about two years to complete XFP. Five years later, the new ad server, dubbed DoubleClick for Publishers, was complete, and the final customers migrated to the new system.

Five years into a two year project. Frame that. Maybe they weren’t underestimating Google engineers then?

Google CEO Eric Schmidt challenged Neal and Scott to think bigger. He wanted to know how the tiny-at-the-time AdX business could reach $10 billion in revenue (a level it took thirteen years to achieve, likely in 2021).

Wow.

Rajiv recognized there was a sea change coming in advertising. When you could target ads to individual users, it made no sense to buy big, static, direct advertising campaigns where half or more of the audience was wasted. He also felt there were a lot of start-ups trying to solve this problem on the advertiser side but there were not as many trying to help publishers.

It’s crazy to imagine that this personalization revolution didn’t come until 2009! That’s very late!

Six weeks after being fired from Right Media, O’Kelley started AppNexus, a cloud computing company optimized for use cases that required real-time data: advertising, finance, and health care. This was also a helpful way to ride out his one-year noncompete agreement. Brian had a huge chip on his shoulder after being fired and losing out on tens of millions of dollars from the Yahoo acquisition and, in his own words, was “so fucking angry.”

Call Brian O’Kelley what you want, but the man is no quitter.

Letting buyers choose which individual impressions they wanted to purchase would fundamentally tilt the balance of power and deteriorate much of publishers’ control over their pricing and sales channels. It was also not lost that the agencies controlled the vast majority of advertising dollars around the world—and they wanted in.

Yup. And whine as you might, the past is not coming back. This is like variety stores longing for the days of 45% margins when Walmart is coming. This is the books industry wishing to be the gatekeepers before Kindle Direct Publishing came out.

Publisher profiles are just not as valuable as those collected by advertisers. Knowing that a user spent time in the “Sports” section of your newspaper website is slightly interesting but frankly of little commercial value. Knowing someone was shopping for your product, on your website, just ten minutes ago, on the other hand, is enormously valuable! This disparity in the value of the user profiles proved to be a critical problem for publishers entering the programmatic era. If a buyer could use a DSP to find the most valuable users on the publisher’s site, they could just buy those users and skip wasting money on all the others. Those “other users” were the bread and butter of the publishing business. They were the majority of the publisher’s audience, and now buyers could simply choose to avoid paying for them.

The economics of the industry would never be the same.

The product manager for the product used to joke, “They bought DoubleClick for $3.1 billion and got DFA for free.”*

Yup - this is the real value of DoubleClick, and it took the insights of Rajiv (and others) to realize it.

O’Kelley, as was his fashion, told the duo that it was the “dumbest fucking idea he’d seen in his life.” Rather than just sending them on their way back to Philly, O’Kelley whiteboarded for them how the emerging world of ad exchanges worked and explained how they could potentially build a system to allow advertisers to bid on ad inventory and take control over the money they were sending to ad networks. “He just told us what to do,” recalls Weinberg.

I really like this guy. I’m not sure if that’s the intention in this book, but man do I love people like Brian O’Kelley. He is someone that is absolutely willing to truth nuke. Keep those people around.

The Squeeze

The final shoe would drop in 2015, when access to buying YouTube ads would be cut off from all competitive DSPs. If you wanted to buy on the largest video site in the world, you only had one choice—the Google DSP.

Google begins securing the monopoly on ads.

“The reality is we missed the [Yield Management] threat—both on the AdX side as well as the DFP side.”3 Google feared a world where most of the indirect revenue was coming through yield management, leaving AdX as an outlier, primarily representing the Google ad network’s demand and not other third-party demand. If AdX became less important to publishers, then so could the ad server. You could even imagine what might happen if a company that owned a competitive ad server decided to acquire one of the yield managers!

Imagine that as late as 2015, Google still did not have a secure monopoly!

Neal suggested “picking up the one with the most traction and parking it somewhere.” It is very unclear what the verb “parking” means in this context. A generous interpretation is that Google would build its own solution while the existing solution that was acquired would continue to work and keep customers satisfied. A less generous reading, and one that would be used as evidence in later lawsuits, was they intended to effectively neuter a competitor. Jonathan responded to Neal’s parking comment with an equally ambiguous statement: “If we bought one and parked it, it would let us solve the problems from a position of strength (market share, knowledgeable team members).” Once again, you could read this in many ways.

This reminds me of Hoover telling Dyson they wish they had bought him out so they could park his bag-less vaccuum forever. I think we all know exactly what “parking” meant.

Wary from his last experience being acquired by Google, Brian Kane steadied himself for arrogance and layoffs. When the first integration meetings between the companies took place, he was surprised to see Neal, Scott, and Jonathan on the other side of the table. The DoubleClick crew was clearly in charge, and things would be different this time.

Great job, Google. You finally did something right.

Nevertheless, the threat from AdMeld has been neutralized. Google had bought a direct competitor and effectively shut it down, and no one seemed to have noticed. The leading yield manager was effectively neutered, and while it was “parked,” Google bought itself more time to move everything onto its ad exchange.

The monopoly allegations would only grow.

The RFP, he told his Microsoft contact, was dumb and shortsighted. It only focused on optimizing ad networks and unsold inventory, while Google, post the AdMeld acquisition, was building a stack to allow all types of ads to compete in a unified auction. Microsoft needed to get into the game and commit if it wanted to win. Microsoft was not speaking with one voice at this point.

It’s amazing that despite all the brilliant moves Google did, this was still really Microsoft’s game to fumble. The Google monopoly could very well have been the Microsoft monopoly.

O’Kelley would continue lobbying over the next year in an attempt to strengthen the partnership and give advertisers a single auction across all the premium supply. He felt it was the only way to compete with what Google was building on top of DoubleClick. He started pressing Yahoo on their technology problems and making the case that they should sell Right Media to him and switch the rest of their business over to AppNexus technology. For Brian, this would be the ultimate victory, reclaiming his lost prize while putting his new company at the center of the anti-Google alliance.

He was exactly right, of course, but the game theory of it was too hard to pull off. He would never have the type of cooperation needed to successfully oppose Google.

He read the news that Marissa Mayer had been named Yahoo’s new CEO and that his boss Ross had been passed over. The meeting ended abruptly, and the whole partnership would soon as well.

Womp womp.

Marissa was not an ads person (her counterpart Susan ran all the ads products), and her closeness with Google seemed like a very bad sign. As a former Googler, she also felt like good engineering could solve any problem, so a divestiture of a technology asset would not sit well. Predictably, she killed the Right Media deal on her first day. Marissa spent most of her efforts on the consumer side, with an ill-advised billion-dollar acquisition of Tumblr. By 2014 she was admitting on an earnings call that she needed the company’s ad tech stack to get to “parity” with others. It took a couple of years to capitulate, but in early 2015 Yahoo officially killed Right Media. Moving forward, Yahoo would offer the Yahoo Ad Exchange, which was primarily their own sites, plus some limited inventory from other publishers. Other than its foray with newspapers, Yahoo was never committed to doing business with the open web publishers, and as David Rosenblatt predicted years earlier, their revenue dwindled hand in hand with their audience.

And thus ended any chances of a united anti-Google wing.

Despite billions in investment, Microsoft failed to become a meaningful counterweight to Google in the advertising market. The Bing search engine remains in the single digits of market share. The AppNexus partnership was plagued by distracting demands to add support for ads on the Xbox, to Windows Phones, to Nokia phones, and to Skype. Microsoft even had a “call option” in its contract with AppNexus, allowing them to acquire the company for a cool $1 billion, but they let it expire, unexercised. The aQuantive acquisition would turn out to be the largest corporate write-off of all time with virtually no value extracted by Microsoft.

Welp.

Jonathan Bellack laid it all out, asking, “Is there a deeper issue with us owning a platform, the exchange, and a huge network? The analogy would be if Goldman or Citibank owned the NYSE.”

Google is 100% an ads monopoly. The greatest business in business history. A marketplace where they own both sides, supply and demand. Insane.

There were two fairly large loopholes in these commitments. It was not stated that the revenue share would be calculated evenly on every single impression. Really there was not any detail given at all about how these shares might be implemented. Google would take actions to make these take rates averages over many impressions rather than a specific calculated rate to give it more flexibility per auction, which it would then use to advantage itself. The revenue shares also excluded any mention of the calculations on the advertiser side. AdSense only really received demand from Google Ads, so the 32% total take rate could be allocated between buyers and sellers in a nontransparent manner, as suited the company. AdX was a sell-side product, so again, the buy-side fee could be manipulated as desired without breaking any contractual pledge. Buy-side fees do not really matter to sellers since they only generally care about the net revenue they are paid. Google could thus raise or lower these buy-side fees to win more auctions, even when using competitive publisher data from the ad server and the auction. Winning more auctions helped grow market share for Google’s products, which gave them more data and more scale, which then fed further advantages. It was a feedback loop no one else could match.

The “virtuous” cycle begins.

“If Adwords submitted only 1 bid, the publisher will not be monetized well. The simplest solution was adopted—Adwords would submit 2 bids to Adx to prop up publisher payout.”14 The ad network ultimately would “second price itself” on over 80% of winning bids,15 artificially reducing the buy-side margin from 57%(!) to just 15% and overpaying publishers by an estimated $500 million annually.

The pricing power is so unmatched. How is it possible for anyone to compete?

By Google’s estimates, 60% of publishers using multiple calls stopped as soon as they were subject to this treatment, yet another example of the power Google had to get publishers to do what they wanted. Those that played by Google’s rules would get more of the spoils.

That’s only short term, of course. Play by Google’s rules and you will be the last to get squeezed. But you will still be squeezed.

The difference between Google’s manipulations and those of some of their competitors was that Google could pursue theirs within their closed-loop system without anyone knowing better or having any way of definitively testing.

There really needs to be a shakeup here.

Procter & Gamble is the world’s largest advertiser, so what it says matters. The company’s chief brand officer, Marc Pritchard, took the industry to task in a 2017 speech at an industry conference. He did not hold back any punches, saying, “We serve ads to consumers through a non-transparent media supply chain with spotty compliance to common standards, unreliable measurement, hidden rebates and new inventions like bot and … fraud.” The advertising supply chain, he asserted, was “murky at best, and fraudulent at worst.”

If that isn’t an indictment of a whole industry, I don’t know what is.

Criteo ran tests where they took a set of user cookies that had high value and bid both through their direct connections and through AdX to see how efficiently they could buy the inventory even accounting for AdX’s fees. AdX would consistently underperform, even at very high prices. It was time for a more radical experiment. Criteo tried bidding only $10 into their direct integration but $100 (!) through AdX. One would expect the AdX bid would always win, yet the direct integration continued to outperform. Clearly there were things going on in the AdX black box that were not well understood and likely gave Google secret advantages.

This is as close to a smoking gun as it is possible to have.

One former senior executive drew the line at Global Bernanke, saying, “That program was fucking evil.”

Even Google employees are apalled.

Open Source?

Brian reasoned that if his company’s greatest challenge was Google, then open source made a ton of sense, since it would align all the independents. If their greatest challenge was competing with companies like PubMatic and Rubicon, though, keeping it proprietary to their own ad server would be the most advantageous. After some discussion, the plan was clear—open would win. If this turned into an AppNexus-only solution, it would defeat the whole purpose of evening the playing field with Google.

The same open versus closed debate is playing out in the AI sphere. Open source is leveraged as a weapon against rent-seeking megacorps.

It was not enough for prebid.js to be open source. It would also have to have transparent governance to gain the industry’s trust. If the primary contributor, AppNexus, could change the code at a whim, then having the open-source code was merely a fallback and not a real reassurance of continuing independence.

A smart insight.

Header bidding was the first crack in Google’s decade-long hegemony over media companies. With some JavaScript and some creative use of ad technology, publishers could finally optimize yield on their own terms, without Google’s permission. If that wasn’t enough, the eye-popping revenue improvements were the icing on the cake.

[sic]

One publisher executive complained that the Google ad server was actually the slowest part of his entire stack. “Given the latency of [the ad server], I don’t think they have a leg to stand on talking about header bidding latency,” he added.7 Another example of latency hypocrisy related to the ad network bidding into AdX. While other bidders, including DSPs, were given strict time-outs on every auction, when the ad network bid into the exchange, it would get extremely loose time-out restrictions, a fact that was not disclosed.

Insane that Google’s server was the slowest part of the stack! What the absolute fuck.

Ultimately the Google response to header bidding was Project Jedi, a new ability to accept bids into the ad server directly, in exchange for a lower fee, generally 5% to 10%. Jonathan had pushed to make this product good enough to entirely kill off header bidding but was directly told not to do so by more senior executives in the group. He was told to make Jedi just “slightly better than header” so it didn’t entirely remove the advantages of AdX.

Incredible anti-competitive incentives at play here.

A big flaw in the strategy for Open Bidding was an underlying assumption that it would replace header bidding. Commenting on the initial release, Bellack noted that if buyers bid in multiple ways on the same impression, they might experience higher prices, since they might “second price” themselves. This is exactly what you would expect if the publisher maintained their existing header setups while also adding Open Bidding to the mix: the duplication of auctions would—and did—go through the roof!

Genius.

OpenX, was seeing a precipitous decline in transactions. It didn’t take a lot of digging to find the problem. Google had abruptly started to bid less—a lot less—for the same ad inventory. Google’s DSP was the largest buyer on ad exchanges, so a change in its bidding strategies would send ripples, or really waves, of change to the sellers and the publishers they represent.

[sic]

“We believe this is about ~$100K/day opp loss due to this change.”

[sic]

Tim emailed Google AdX product manager Sam Cox with graphs showing the precipitous decline in revenue but never heard back. Tim was in the midst of a nine-figure deal to move all his technical infrastructure to the Google Cloud, and he still could not get any meaningful response from the company.

Think about how much money is at stake if Google can say “fuck off” to someone spending 9 figures on their compute product.

She would be informed that sfgate.com, for example, was the single most common domain to buy on all of AppNexus, despite being a medium-sized local Hearst newspaper website that did not actually sell through that exchange. “I was obsessed,” she said. “I yelled at everyone my industry was turning into a criminal enterprise, a cesspool.” The most egregious form of fraud common in the mid-2010s was the phenomenon of “domain spoofing.”

Insane!

This is a prisoner’s dilemma—if a single DSP took the stand that it did not want “bad” inventory, it would achieve lower results and would lose against the competition; if every DSP stood against this fraud, it would be wiped out and the whole ecosystem would be better off.

Game theory at play.

“Let’s shut down all your ads for fifteen minutes and see what happens.” He would not tell her exactly why, but she trusted him and was game. Shutting down the ads was not actually that hard. Using the vaunted priority system in the ad server, Susan could just set up blank ads in every size available on the Times website to run from 4:00 a.m. to 4:15 a.m. and set them all to priority 1, which means “serve ahead of everything, no matter what.”

I love this experiment.

…the analytics group at Google, led by David Goodman, looked at the auctions seen by the Google DSP and reported back that there was no real difference in volume during the shutoff period. That did not make much sense.

Sure it does. There’s an insane volume of fake data.

The fake traffic vastly outweighed the real traffic.

Surprise surprise.

With the evidence in hand, Sam wanted to create a sustainable solution. He knew that if it came from Google, it would be seen as some kind of power grab or manipulation—the other side of the coin for being a dominant industry player.

At least they’re self aware!

The requirements were simple: it could not require any coding, and it could not come from Google. The idea came up to model the solution on the ubiquitous technology known as robots.txt, wherein a publisher can post a simple text file that tells search engines and other web crawlers whether they want their content to be indexed. Why not do the same thing for ad sales? From this insight was created the spec for ads.txt, a simple text file that allowed publishers to declare on their own websites which exchanges and publisher ids on those exchanges were authorized and official.

This is an absolutely brilliant solution.

Within a year of the standard being published, more than half of all publishers had adopted it. Even Facebook published an ads.txt file, in their case just an empty one to prove the inventory was not available for sale. The ubiquity of this simple tool has largely stamped out spoofing and brought millions, or maybe even billions, of ad dollars back to the publishers that deserve them. Google made this happen and did not even take the credit.

This really is the biggest under-popularized story in all of tech. Absolutely remarkable solutioning by Google.

It is not clear why so much of the low-quality and fraudulent ad business is based in Israel.

I’m just going to leave that as is here.

Youtube

Google had been trying to create its own competitive product, Google Video, but it failed to take off, ironically because they took pains to review every uploaded clip before setting it live instead of throwing caution to the wind like YouTube.

This is a very important aspect of getting acquired. The first thing any respectable competitor will do is to try and beat you. Unless you beat them and prove you’ve won, why would they bother acquiring you?

YouTube was an early innovator in creating ads that balanced the user experience with monetization; adding the “Skip” button to many ads reduced the annoyance factor but also lowered prices significantly.

The skip button was a genius innovation.

Google even made a not-so-serious attempt to acquire FreeWheel for $95 million but after a quick rejection did not pursue at a higher price.

[sic]

FreeWheel was thriving but also saw that the video market was going to be a brutal battle between enormous media firms looking to control the future of television. Perhaps with this in mind, they made the decision to sell out to one of the world’s largest traditional media companies, Comcast, for $360 million. Comcast was not going to let Google do to TV what it had done to the web—they wanted to do it themselves! With NBC as the “anchor tenant” and the FreeWheel system allowing most of the major broadcasters to interoperate, this seemed like a viable ambition.

This could have been the real competitor to Youtube’s ads engine. But Google continues to get incredibly lucky and their competitors just shoot themselves in the head.

The lawyers at YouTube looked at their processes for serving ads and sharing data and decided the relationship with FreeWheel could no longer be supported. They decided, unilaterally and without any communication to the team at Comcast, to send out letters to all customers saying they would have to switch ad servers to Google’s to continue serving ads on YouTube. Left without any recourse, Comcast brought this issue to the attention of Congress and European antitrust authorities.

Whining to Congress is how you know you’ve lost.

It is not even clear if Google did any of this out of malice; it seems more likely it was a mixture of indifference and fear of a privacy backlash. Regardless, as was usually the case, Google came out on top.

Yup.

The fact that Google’s ad server is not dominant in the video market and FreeWheel continues to hold significant market share has ended up having little effect on the outcome. FreeWheel, meanwhile, remains a healthy business within Comcast but has not achieved the strategic goal of creating an independent video ecosystem out of Google’s control.

Comcast fumbling the bag, nothing new.

”Privacy” Wars

Apple restricted ad load and capabilities significantly and did not allow cookies or other identity data to be used. While Apple News does support a robust subscription capability, which in some cases makes publishers meaningful revenue, ad monetization is consistently poor.

Complain as you might about Google, there is no better way to make money. Apple’s “privacy” focus only serves to hurt the web! It reduces monetization and drives players into apps.

Google jumped in later that same year with their typical strategy of being technology first. Instead of a reader app, like Apple, they designed an open-source technical specification, Accelerated Mobile Pages, or AMP. AMP was designed to “dramatically improve the performance of the mobile web.” From the start Google gave AMP pages advantages in search results, showing a “Top Stories” carousel featuring content in the new format. In 2018, when Google launched a dedicated news app, AMP support was a requirement for publisher content to be shown and was also required to be featured on the Google News web experience. Publishers did not find great success with AMP. The pages did not monetize as well, and the benefits of traffic acquisition were questionable. When it launched, AMP did not support header bidding, and it has been alleged that Google intentionally slowed down non-AMP pages and ads to benefit its own monetization.

AMP was such a disaster.

The AppNexus ad server was $500,000 cheaper than Google. AppNexus’s open exchange rate was only 5.7%, compared to Google’s 20%. But News Corp generated $83 million in programmatic revenue, and 53% of that was from AdX. The risk to that $44 or so million vastly exceeded savings on technology fees.7 She simply could not guarantee to the senior executives at her company that they were not going to lose significant money if they switched. Cinderella did not get her glass slipper, and the transition to AppNexus—despite the $10 million investment—would not happen.

If that isn’t monopoly pricing power I don’t know what is.

Susan Wojcicki, the longtime head of advertising and a key advocate of the display business, moved to a new role as the CEO of YouTube during this time.

[sic]

Neal Mohan had toyed with leaving the company in 2013, when he was close to joining Twitter only to be retained with a whopping $100 million pay package.

All the key ad talent starts to leave for greener pastures.

The details of how Google retained Neal this second time are not known, but it likely resulted in his internal transfer to become the chief product officer of YouTube under Susan later that year.18 With his move, the two most senior executives looking after the display business were gone.

Good for Neal.

…the senior leadership was now dominated by Mountain View–based search executives with little direct experience of the display ads business and a decidedly “Googley” worldview. Some of the more aggressive missteps the company would make in the second half of the decade could be traced back to the loss of senior executives who really cared about the business they were running.

Absolutely insane how even a large company like Google can simply lose the plot on their primary business line.

Midway through the keynote in 2017, Craig introduced a new concept within the Safari browser called ITP, or Intelligent Tracking Prevention. ITP would move beyond blocking third-party cookies by default and instead would actively seek out so-called “trackers” on the browser and delete them proactively. Craig proudly announced that “it’s not about blocking ads; the web behaves as it always did, but your privacy is protected.” That statement is true, if you define “the web” as not including ads, which inconveniently had been a big part of it since 1994.

Apple is only interested in privacy as a check against Google an Facebook. It’s a brilliant maneuver by them. But let’s not ascribe any noble intentions here.

There is little evidence that the data from the cookies has caused real-world harm, other than a general sense that they violate privacy or facilitate “surveillance capitalism,” as some more radical advocates have dubbed it.

Agreed. It just “feels” bad, and Apple capitalized on that feeling.

The fundamental problem has always been that less consumer data corresponds to lower prices for publishers and worse results for advertisers.

Yup. Every single privacy initiative just makes the open web less profitable for publishers.

Two lessons were learned from the Do Not Track fiasco: the companies that control browsers have enormous power, and many of them would choose privacy over advertising given the choice.

And so Google really had to double down on Chrome.

Safari always blocked third-party cookies by default, but as Apple increasingly saw ad-supported Google as their primary competitor, the company started turning the screws on the advertising community. Once Craig launched ITP at the developer conference, Safari was fully blinded for most advertising use cases and loopholes companies were using to track results were closed.*

[sic]

The net result of Apple’s aggressive push for privacy was undoubtedly bad for advertising. Publisher webpages monetize on the Safari browser at 40% the pricing of the more privacy-agnostic Chrome browser. The ATT rollout at one point cost Facebook $10 billion in lost annual advertising revenue.

Bot notice how Apple never gets heat from publishers. They blame Google instead.

Chrome was the largest browser by market share and Android was a powerhouse in the smartphone market, but both had done little to visibly protect consumer privacy. Apple, meanwhile, was hammering the privacy message home with billboards and TV ads claiming, “What happens on your iPhone, stays on your iPhone.”

It’s amazing to peel back the curtains and look at the real motivations behind Apple’s moves.

The Google Chrome team ended up trying to split the difference. First, they announced that third-party cookies would be eliminated “within two years,” which is quite a long time frame and was clearly designed to soften the blow to the advertising businesses. This is probably the only time Google had announced anything at all with a timeline longer than a couple of months.

They’re in a tough position.

Google’s announcement tells you everything you ever need to know about the company two decades aged from Susan Wojcicki’s garage. Instead of boldly disrupting the status quo like Gmail had with virtually unlimited storage or search had with its simple design and fast response, they tried to split the difference and build an inherently compromised product. The Chrome engineers wanted to do what they perceived as the “right” thing to do about privacy, which was to get rid of third-party cookies. They could not do what Apple did and just remove them because it would hurt their advertising business. Instead, they decided to engineer their way out of the problem, with complex proposals designed on blank whiteboards, without much of any regard for the ways the current advertising world operated. They then unveiled the proposals to the world, thinking they would be hailed as saviors. In fact, they were highly conflicted and naive, especially in the assumption of how long it would take to gain adoption of these nascent technologies.

My only guess is that Google got complacent. Somewhere along the way, they simply assumed nobody could truly compete with them on this revenue front. But over time, competitors are beginning to chip away - not at Google as much as the entire open web ecosystem.

…the long-term trends are undeniable. The web is becoming an increasingly difficult place for publishers to survive.

This is regardless of who “wins”, Google or Facebook or Apple.

Facebook

Unlike Google, which was dependent on placing ads through the exchange on other publishers’ websites, these formats fit naturally in properties like the Instagram and Facebook apps and produced great results for advertisers.

Social media really revolutionized the ad game.

“There is a risk that FB becomes the ‘starting point’ of the internet.”

This was a very real risk.

A decision was made by COO and former Googler Sheryl Sandberg to concentrate the company on a “closed” ad strategy rather than an “open” one.

Sheryl knew Google’s strategies quite well, so this is an astute move by her.

Despite its enormous presence in the ecosystem, Google had very little of its own ad inventory for display since Google Search, Gmail, and Google Maps only run text ads.

[sic]

Facebook essentially had so much inventory and so much data that it did not need to run the Three Pillars strategy; it could just match its own demand to its own supply and become the second-largest advertising company in the world.

And here we have the main advantage Facebook has. It owns the ad platform, so it can decide how and who to show the ads to.

Facebook realized that, like most other ad networks, one of FAN’s key challenges was getting access to inventory. Internal documents from the company bemoan the fact that they would always have the challenge of being intermediated: “Google sits between us and the impressions we want to buy.”

Even the all powerful Facebook runs up against limitations.

They turned to the same solution, header bidding. In 2017 Facebook announced it would partner with six different providers, including AppNexus, to access inventory more directly through header bidding.15 This fueled Google’s fears since the only thing that could break Google’s sway on publishers was replacing their ad network demand, and Facebook was the only company that could match or exceed that demand.

Facebook is slowly getting ready to wage war.

The app marketplace had evolved very separately from the web, with different publishers, advertisers, and technologies. It also moved a lot more slowly, since new technology adoption was dependent on publishers installing new code…

Publishers would have to update apps or modify SDKs. The web was instantly deployed.

Facebook ended up building its own solution to the mediation problem, called the “bidding kit.” This technology would allow app publishers to compare multiple sources of demand from Facebook and others in a single auction, integrated with non-Google exchanges including MoPub, Fyber, and a small start-up that had pioneered in-app header bidding called MAX.

MoPub! One of the few successful Twitter acquisitions.

Publishers were very excited about getting Facebook demand. Facebook claimed to increase overall revenue by 10% to 30% when implemented into the header—not a small amount!

There we go! Competition.

Chris LaSala at Google wrote that his number-one personal priority in 2017 was to “fight off the existential threat posed by Header Bidding and FAN.” Google estimated that Facebook’s share of the overall display advertising market that year (including social) was 35% to Google’s 19%; it was “the dominant player,” and “the gap [was] expected to widen.”

This is wild. Facebook was winning!

…the industry was somewhat shocked to hear that FAN would bid directly into Google’s new header bidding alternative, Open Bidding.

[sic]

FAN was a demand source and Google had specifically laid out that only exchanges or other intermediaries would be allowed to bid into Open Bidding.

But eventually, the monopoly becomes an oligoly. Facebook and Google sit down and hash out an agreement.

…the announcement was the end result of a secret project called Jedi Blue that involved grueling negotiations between the parties to protect against data loss, competitive positioning, and possible government scrutiny. The whole thing was done with complete secrecy. The negotiations would often rely on paper notes and documents and avoid electronics for fear of creating evidence that could later be subpoenaed. The deal was approved by the highest-level executives of both companies, with Sheryl Sandberg, Mark Zuckerberg, and Google CEO Sundar Pichai all directly signing off.

[sic]

The project was the first one managed entirely by Facebook’s London office, and the Google team involved commuted from New York almost every month to get the program operational.

This is crazy.

…the final agreement between the two companies specified that Google would “not advantage itself or any other third-party demand” in the auction. Facebook was using its power to assure they would not be played by Google like everyone else. The overall agreement read like a tentative truce between two deeply distrustful enemies. Every provision had caveats and carve outs, and the procedures for dispute resolution were detailed to the letter.

It’s a binary star system, with each star keeping the other in check.

Open web advertising was a bit of a moot point, as FAN stopped bidding entirely on this inventory in April 2020 for business reasons unrelated to Google. The risk of fraud and other complexities of the open web were too much to stomach in the aftermath of the Cambridge Analytica scandal.

Oh man I had totally forgotten about that. Feels like a lifetime ago.

It is difficult to argue that Jedi Blue hurt advertisers or publishers. Facebook advertisers got more reach for their campaigns. Publishers got more revenue, though with an additional 5% tax going to Google through Open Bidding.

And this is the key - this might have genuinely been great for the entire ecosystem. Even though it was kind of a case of collusion, it also was truly the first moment of competition Google had in a long time.

Other demand sources, like Criteo or the AppNexus DSP,* were only allowed to bid into AdX and incur a 20% fee, whereas Facebook, through Open Bidding, was charged a lower 5% fee. The agreement also set minimum performance standards for the auction traffic, such as a guarantee that 80% of all auctions would have enough data to allow Facebook to identify the users for bidding. Other partners would get no such guarantee.

[sic]

Web advertising was of declining importance to Facebook, and that portion of FAN was shut down soon after Jedi Blue went live. Several former executives from both sides who were involved in crafting Jedi Blue would speculate that Facebook was not really interested in web in the first place but included it in the deal to play on Google’s paranoia about header bidding competition. “They wanted AdMob access; the rest was a foil,” said one person knowledgeable about the deal.

If so, Facebook played their hand beautifully.

Conclusion

“I felt like they were holding us hostage.” Even though she represented News Corp, one of the largest and most powerful media companies in the world, she had no choice but to accept whatever Google decided, a pattern that was becoming painfully familiar to everyone in the publishing business.

The deletion of chats was a running theme in the DOJ’s case against Google. Google employees extensively use a proprietary chat program built into Gmail to have conversations, both substantive and otherwise.

This is not a good look.

Only if an employee actively and voluntarily switched to “history on” would the messages be kept. This policy was in place until February 2023, well after most of the antitrust litigation was underway.

Clearly they were doing stuff that they believed the government would declare illegal.

A full breakup would sever the tie between the ad network’s demand and the exchange and the ad server, thereby allowing other exchanges and ad servers to compete. But would this be good for publishers? Publishers are apprehensive about the prospect of a spinout of Google’s advertising assets. Like it or not, they are still highly dependent on the publisher ad server, and the prospect of it having uncertain ownership entails significant risk to their business’s operations. While the ad server is a meaningfully sized software business on its own and likely could be made break-even or profitable on a standalone basis, in its current state, it is heavily subsidized by margin from AdX. If AdX had never existed, it is unlikely Google would have spent all the time and money modernizing the software, let alone acquiring it in the first place! If the ad server was spun without AdX, that subsidy would disappear, resulting in fewer resources to maintain and improve the technology, and perhaps higher prices to publishers.

Publishers are doomed either way. The world has changed, and there is no going back, even if you broke up Google.

One publisher executive made a stark historical analogy: “Like it or not, Google is a stable empire. Bring down the empire, and you might have the Dark Ages with no indoor plumbing. It could get worse before it gets better.”

Yup.

“If a private equity firm were to buy the ad server business and suddenly hike up prices, it could put some marginal publishers out of business.”

Exactly - who is the alternative? And why do you think the alternative would be better for you?

“If Google no longer has a stake in the open web, why do they have an incentive to send anyone to the open web?”

This is probably the biggest thing. Google has single handedly been pushing the web forward, to the benefit of society as a whole. If the open web is not profitable, then like it or not it just languishes. Apps come to dominate, and now Facebook or Apple have the same monopoly status. What then?

Optimistic views of a breakup or spinout assume that the remaining Google business continues as usual and that the outcome is mostly a change in economics. This view discounts Google’s increasing emphasis on YouTube and declining dependence on the Network business. What if Google simply picked up their toys and went home?

Yup.

…it is important to consider that a spinout of the Google assets could also be dangerously abused if not also regulated. After the AT&T breakup of the 1980s, the Baby Bells largely recreated the monopoly through acquisitions. An independent ad serving company would still have near monopoly power over publishers, opening the opportunity for abuse. The spun-out company could raise prices, manipulate the auction for its own benefit, acquire related companies, or do any kind of thing that might not be in the interests of its customers. Does anyone think that the ad server would be less abusive if owned by Microsoft, Meta, or Oracle?

Completely agreed. Google is clearly a monopoly, but what do we do about that? It’s not clear to me.

---

This book is an incredible read. The author is simultaneously technical enough and a skilled enough writer to convey complex concepts to the layperson. Ads is far more complex than I had realized, but for the first time I truly understand the history and the rough details of what is happening. Google is absolutely a monopoly, but it’s just not clear to me what, if anything, can be done. Or even should be done. Detractors will argue that somehow breaking up Google will return us to the old days when any blog could make real money via ads. That simply won’t happen. Publishers on the open internet are not returning to power. The model needs to fundmanetally change. Maybe AI will do just that.