Good videos, with context.
Zebras All the Way Down - Bryan Cantrill, Uptime 2017

Zebras All the Way Down - Bryan Cantrill, Uptime 2017

Code & Supply
6 HN recommenders · discussed 2018–2025
Open YouTube ↗

About this video

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Why it appears in the archive

A perennial HN favorite whenever firmware, disks, ECC or opaque hardware come up, valued as both entertaining and sobering.

AI-generated summary and labels from the source dataset; verify important details against the video. Source & credits.

From the Hacker News discussion

gt2 · 2018-10 · rec

Zebras All the Way Down - Bryan Cantrill, Uptime 2017 [VIDEO] Lots of great tangents that add to the subjects. I love talks like this.

Read original comment ↗
ar0b · 2018-10 · rec

One of my favorite conference talks is semi related to problems like this. [VIDEO] Brian Cantrill - Zebras all the way down

Read original comment ↗
kbenson · 2019-07 · rec

On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: [VIDEO]

Read original comment ↗
kbenson · 2020-01 · rec

There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: [VIDEO]

Read original comment ↗
ignoramous · 2020-02 · rec

In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2]. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3]. [0] [LINK] [1] [LINK] [2] [LINK] [3] [VIDEO]

Read original comment ↗
Bayart · 2021-01 · rec

>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. [1] [1] [VIDEO]

Read original comment ↗
thomasjudge · 2025-04 · rec

For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" [VIDEO]

Read original comment ↗
floatboth · 2018-01 · reference

On The Turing Completeness of PowerPoint [LINK] Compiling C to printable x86, to make an executable research paper [LINK] HDR Photography in Microsoft Excel?! [LINK] Zebras All the Way Down [VIDEO] Solving Layout Problems with CSS Grid and Friends [LINK] Console Security - Switch [LINK] Let's move SMM out of firmware and into the kernel [LINK] Bringing Linux back to server boot ROMs with NERF and Heads [LINK] Sharing the Chisel [LINK] (looks like the actual conference video wasn't released)

Read original comment ↗