What the Crawl Missed

Every cell is a page that changes at its own rate. The crawler comes back on a fixed interval and can only ever learn one thing per visit: whether the page differs from last time. Everything that happened in between is gone.

unchanged since last visit unseen changes piling up just crawled
Changes happened
0
Witnessed
0
Never witnessed
0
Seen
—

A visit is worth at most one observation. If a page changed six times since you last looked, you see one difference — five events leave no trace anywhere, and no amount of storage recovers them. Push the interval up and watch the witnessed share collapse while the archive keeps growing.

Coverage below 100% means each crawl samples only some pages; the rest keep accumulating in the dark. That is the quantity worth instrumenting — not how many bytes a crawl cost, but how much change it let you witness.

From An Archive Pays Space for Time.