Two lines in the same function, both shaped like a call into a library:
clock_gettime(CLOCK_MONOTONIC, &ts);
syscall(SYS_getpid);
On the laptop I measured these on, the first costs about 17 nanoseconds and the second about 107. Same syntax, same headers, same place in the source. One of them drops into the kernel and climbs back out, the other runs start to finish inside your own process, and nothing at the call site tells you which is which.
I wrote up the mechanism behind that gap for freeCodeCamp, down to the instruction and the register file. This is the other half of it: what it means to build on an abstraction whose cost you can measure but never see.
Three kinds of cost
We handle two of them well.
The first is visible cost. A network call looks like a network call. It takes a URL, it returns an error type with a timeout in it, and everyone in review knows what they’re looking at. Visible cost gets budgeted, so it turns up on architecture diagrams and in the argument about whether this should be one service or two.
The second is unmeasurable cost. What does a bad abstraction cost you over six months of maintenance? Real, and nobody’s putting a number on it. Costs like that get handled by judgment, and by people who have been burned before, and the arguments never settle even though they’re worth having every time.
The third kind sits between the two and gets handled worst of all. You can measure it, in that a benchmark you could write before lunch will give you the figure to two significant digits. You can’t see it, in that nothing at the call site separates it from the cheap thing on the line above. Nobody budgets for it, because budgeting starts with noticing. Nobody argues about it either, since an argument needs something to point at.
Almost everything expensive about a boundary lands in that third category.
Somebody measured, once
A hundred nanoseconds is nothing. Multiply it by the million tiny read() calls a chatty parser makes over a large file, though, and you’ve burned a tenth of a second on transitions alone, which is the sort of p99 that takes a week and three people to account for.
What happened next is the part worth keeping. Somebody measured, and then the interfaces changed. sendfile() moves bytes from a file to a socket without the round trip through your buffer that the obvious version performs. Buffered I/O in every standard library is the same trick at smaller scale. io_uring goes furthest, letting you queue a batch of work behind a single transition, which is why the interface looks so unlike the one it replaced. Those aren’t three optimizations. They’re one optimization applied in three places, and each came out of somebody putting a number on a cost the API had been hiding for years.
Everyone downstream inherits the answer. Almost nobody inherits the reason. You learn to buffer your writes because buffering is the idiom, and the idiom is correct, and you can go most of a career without knowing that the thing underneath it happens to be a privilege transition on your CPU. That holds up until you hit a case the idiom doesn’t cover, and then you’ve got nothing to reason from, because the reasoning was never handed to you.
The abstraction is doing its job
It’s worth saying plainly that write() hiding the privilege transition is good design, not a leak or an oversight. That was the whole point of it. A programmer who had to think about swapgs and the kernel stack switch every time they logged a line would never finish anything. Every layer you build on is a promise that you can stop thinking about something, and the promise is what makes large systems possible at all.
So this isn’t an argument for fewer abstractions, or for reading kernel source before you write a web handler. The abstraction earns its place, and what I’m after is the specific thing it takes from you in exchange, because the exchange is usually fine and occasionally ruinous, and the difference between those two cases has a shape you can learn.
Flat regions and cliffs
Here’s the shape. These costs are almost never linear, in that they stay flat across a wide range of inputs and then fall off a cliff, and the interface is identical on both sides of it.
read() into a 64 KB buffer amortizes one crossing across sixty-five thousand bytes. read() into a four-byte buffer pays the entire crossing for four. Same function, same argument shapes, three orders of magnitude apart in cost per byte. There’s no type distinction between them, no separate name, no warning from the compiler. Both call sites look the same because both call sites are the same.
Once that shape is in your head you start finding it everywhere. An ORM turns a loop over an association into N queries, and the loop looks like a loop over a list. A method call on a remote object looks like a method call. String concatenation inside a hot loop looks like concatenation. An allocation looks like a variable. In each case the abstraction is doing exactly what it promised at the level of syntax, which is precisely why nothing on the screen tells you which side of the cliff you’re standing on.
Some tools try to close that gap. Go will report its escape analysis if you ask it to. Rust puts a copy in the type system where you can see it. async colors functions in a way people complain about constantly, and the complaint is partly the point, since the color is telling you a call might not return for a while. Those are all attempts to drag an invisible cost back onto the screen, and they’re the exception rather than the rule.
What breaks is the note
The failure here was never the hidden cost. Somebody measured. That keeps being true, in every example above. A person sat down with a benchmark, found the cliff, and changed how the interface gets used, so the knowledge existed and it was right.
What usually goes missing is the step afterward: writing down the number, the machine it came from, and the date, somewhere the next person will actually find it. So the idiom propagates and the reason evaporates. Five years later a team is buffering writes without knowing why, which is harmless, and reaching for the same idiom in a situation where it buys nothing, which isn’t.
This is the same failure I’ve written about in other clothes. It’s the constraint nobody wrote down, the one that holds until an ordinary change breaks it. It’s the measurement a rewrite throws away along with the code, because the number lived in somebody’s head and the code was the only thing in version control. It’s also, at a longer range, why verification tools ask you to leave the language you ship in: the moment a property has to live somewhere other than the code, it starts drifting away from it.
A cost that nobody records is a cost that has to be rediscovered, by somebody with less context, under more pressure, usually during an incident.
Measuring once
Three things, then, that are cheap enough to actually do.
Find your cliffs, and not by reading. Take the three operations your hot path performs most often and establish, for each one, whether it’s a function call or a boundary crossing. That’s usually an afternoon. The answer is frequently surprising, and it stays true for years, since the boundaries in a system move far more slowly than the code around them.
Record the machine along with the number. My 107 nanoseconds is close to a floor. That CPU reads Not affected for Meltdown, which means it skips the page table isolation work on every return, and it answers Spectre v2 with Enhanced IBRS in hardware instead of paying for retpolines on every indirect branch. If your own machine reads Mitigation: PTI, it’s doing work at that boundary mine never does, and a few generations back the gap widens considerably. The number is a property of one CPU, one kernel, and one set of mitigations, so written down without them it turns into folklore inside a year.
grep . /sys/devices/system/cpu/vulnerabilities/*
Trust the ratio further than the absolute. Six times moves slowly. A hundred and seven nanoseconds moved about fifteen percent between runs on the same laptop in the same afternoon. Ratios survive a hardware refresh; absolutes rarely survive the quarter. When you write the note, lead with the ratio and keep the absolute as a dated reading from a named machine.
None of this asks anyone to become a kernel developer, only for one afternoon and one paragraph in a document, once, per boundary that matters.
The exchange
Every abstraction is a trade. You give up knowing how something works, and in return you get to build something larger than you could otherwise hold in your head. That trade is nearly always correct, and I wouldn’t want to write software under any other terms.
What the trade costs is a number you can no longer see and can still pay. The number is usually small. Occasionally it’s the whole performance story, and you can’t tell which of those you’re looking at from the call site, because the call site is doing its job.
So the abstractions that hold up over decades aren’t the ones that hide the least. They’re the ones that arrived with a note about what they cost.