I just want to serve 5 terabytes.

Ha ha ha. Have you seen the Broccoli Man short? And that was a satire of what Google's infrastructure looked like in 2010 . [VIDEO]

Regarding the phrase "I don't know how to count that low", here's the video where it originated. Frankly I'm astonished this hasn't been shared before, and I hope I don't get in trouble for posting it (it's like 11 years old, surely nobody cares at this point, right?) [VIDEO]

> At a different (unnamed) FAANG I'm guessing Google, on the basis of the recently published (to the public) "I just want to serve 5TB"[1] video. If it isn't Google, then the broccoli man video is still a cogent reminder that unyielding multi-region rigor comes with costs. 1. [VIDEO]

Yeah, when I was there I saw plenty of <1% optimizations saving REDACTED gobs of money, and people were rewarded for it. I don't think it's applicable to most teams though. Imagine a foo/bar/widget app that only serves 20B people (obvious exaggeration to illustrate the point) and is only necessary up to a few hundred times per day. You can handle that sort of traffic on a laptop on my home router and still have enough hootzpah left to stream netflix. I mean, you are Google, and you need to do something better than that [0], but the hardware for your project is going to be negligible compared to other concerns unless you're doing FHE or video transcoding or something extraordinarily expensive. Walk that backward to, how many teams have 20B users or are doing extraordinarily expensive things? I don't have any clue, but when you look at public examples of cheap things that never got much traction and probably had a suite of engineers [1], I'd imagine it's not everyone in any case. You're probably mostly looking at people with enough seniority to be able to choose to work on core code affecting most services. [0] [VIDEO] [1] [LINK]

I am a Xoogler. The source document/presentation ("Coordination Headwinds") and idea are many years old, and from what I recall were not aimed to be critical of the company and its organization, rather to give its employee readers a feeling of solace and understanding for why getting seemingly small things done can feel overly hard and slow… a less humorous, and less SWE-specific version of "Broccoliman" if you will. Refer to the red panda that merely wants to serve 5TB of data. - [VIDEO] - [LINK] I’ll try to continue using the organism-metaphor in my commentary. While the “slime mold” story provides a convenient and charitable explanation of Google's bureaucracy and dysfunctions, another explanation is: - As an organism, Google's executive decision-making functions are severely impaired by lack of accurate information, dysfunction of its middle-management “organs” or tiers, and impedance mismatches arising from genetic encodings or mental models associated with the slime-mold idea. - Despite being on a mission to “organize the world's information and make it universally accessible and useful,” Google is far less successful at doing this with its own information. These characteristics are not unique to Google, but some of Google’s cultural history makes this a particularly challenging problem. It may be convenient to think of Google as a slime-mold organism, and in an individual contributor’s day-to-day engineering activities it may very well be, but as a corporation it is no

There are only so many TPUs available and they're kept at 100% utilization. Google has had complex "marketplaces" around company resources for years: [VIDEO] It's a cultural value, like playing board games.

This video [0] was from 2010s, and seems like they had similar issues even back then. (never worked at Google) [0] [VIDEO]

Maybe fear killed engineering. Even in a decently managed company, I can see so many engineering effort gets stalled, compromised, bastardized, or killed. Like your manager asks you to get sign-offs from 10 different orgs. Like you want to do a POC, yet somehow an infra team demanded that you integrate with their system even though your targeted users didn't give a shit. Like you just want to submit a workflow to Temporal Cloud yet a product manager demands that you create an abstraction because he is afraid of X and Y. Like you just want to serve 5TB of data, and you got sucked into this: [VIDEO] . I'm not sure how companies battle this kind of fear. It looks Amazon's Working Backwards, Netflix's Freedom and Responsibility, Uber's Let Builders Build more or less can counter such fear, but ultimately it's about finding the right people who have good sense of product and who strive to make progress.