A number you can't evaluate
Nobody writes a crawl config from scratch. You inherit the defaults and change a few numbers — a depth limit from 4 to 6, a page cap, a delay; sometimes an absolute, sometimes a fraction of whatever the frontier turned out to be.
It isn't blind. You watch new URLs per level collapse into pagination, so depth 6 looks like enough. You watch the duplicate ratio climb, so the recrawl interval looks too tight. Those readings are real and you read them correctly. They describe the run.
They say nothing about the record. What the number actually decides is which parts of the present still exist in twenty years, and no line in the log reaches that far. So the judgment is sound, on a genuine signal, about an adjacent quantity — it seems plausible because it is plausible. And then the defaults survive, because nothing at the layer you can see ever pushes back.
Space is spent on change - ideally
What that number allocates is space, and space is owed only where the world moved. The value of a stored byte is not a property of the byte: the tenth identical capture of an unchanged page carries nothing. What carries value is that the page changed — or, just as usefully, that it didn't, at a moment when it might have.
WARC admits this in the format. Fetch a resource and find the world unmoved, and you write a revisit record: no payload, just a pointer and a timestamp saying same as before, still true now. Real bytes get spent only where something out there became different.
You occupy space here in proportion to change that happened elsewhere. An archive isn't a copy of the web; it's a record of the web's derivative, sampled at times you chose in advance and mostly chose wrong. Which is also why cadence should follow volatility rather than importance — importance tells you whether you'd like the record, volatility tells you whether missing it costs anything.
What spending space is
Strip the web away and the act underneath hasn't changed since someone pressed a wedge into wet clay.
You take a region of space that was doing something else — an unmagnetized patch of platter, a floating gate with no charge, a slab of quartz — and put matter into a specific configuration. Then you hold it there. That is the entire technology, cuneiform to LTO tape.
And the configuration isn't about itself. It's about a different point in spacetime: this arrangement of charge in a datacenter outside Stockholm is a claim about what a webserver in São Paulo emitted one Tuesday in 2019. The archive is a piece of the present deputized to speak for a piece of the past.
That's the trade. Information about time is bought with occupancy of space.
The rent
It isn't bought once. Holding a configuration in place is a continuous argument with a universe that would rather smooth it out — domains relax, charge leaks off floating gates, oxide degrades — and long before any of that physics matters, someone keeps paying for the rack, the power, and the person who swaps the failed drive.
So the unit of memory isn't bytes. It's byte-seconds, and byte-seconds have a price. An archive converts a one-time event into a permanent recurring liability: every capture decision you make today writes a line into every future year's budget, forever, on your behalf, without asking again.
Which is why "just keep everything" isn't the conservative option. It's an aggressive bet that the rent stays cheap.
And the rule for reading it
Matter in a configuration is not information. It's just matter in a configuration.
A magnetized patch means nothing unless something knows to read the patch as a bit, the bits as bytes, the bytes as a gzip stream, the stream as a WARC record, the record as an HTTP response, the response as HTML in a declared charset — and the HTML as a document that once needed forty other resources to look like anything at all. So an archive is always two stored things: the occupied space, and the rule for reading it as a statement about somewhere and somewhen else. The rule lives in matter too, and pays the same rent.
Which makes the corollary sharp. A perfectly preserved bitstream whose decoder has been forgotten is not a damaged archive — it has stopped being an archive. Nothing decayed; every bit is exactly where you left it. What decayed is the relationship between the occupied space and the event it was about. The matter is fine. The reference is gone.
All of this gets set, in practice, by someone nudging a depth limit to fit a quota. They are choosing which events get a piece of the present held in place, at continuous cost, on the bet that somebody downstream will still know how to read it — deliberately, with reasons, on evidence from one layer down. The instrument that would actually help doesn't count bytes. It measures how much change you witnessed, and how much slipped past.
Archiving is the art of choosing where to spend space so that time has somewhere to leave a mark.