thepragmaticquant — tsbootstrap

Count the bytes, not the FLOPs

Sat, 11 Jul 2026 00:00:00 GMT

TL;DR: tsbootstrap’s fast path used to build every bootstrap replicate into one giant batched tensor: on a routine job, a 160 MB tensor pushed out to main memory and hauled back to produce one summary number per replicate. We rebuilt the hot path around two rules: never materialize an array whose only consumer is a reduction, and never carry state you can derive. Every head-to-head benchmark cell now favors the fused path; the settled benchmark grid sustains 3.1x to 20x on the longer series. The last stubborn cell, where the runtime was mostly Python building random-number seeds, dropped from 13.1 ms to 0.55 ms in the redesign’s validation run once we deleted the seed objects.

The full replicate tensor is still available when you want it, and on that materializing path our default backend remains slower than the incumbent. That row is printed below with the rest.

Early this summer, in the middle of a performance push I thought was going well, we ran tsbootstrap’s block-bootstrap engine head-to-head against arch, a mature econometrics library whose bootstrap loop has had years of polish, for the first time. We lost. At a realistic workload, a few thousand replicates of a few-thousand-point series, the whole call came back several times slower end-to-end, on the machine we developed on, before the compiled engine this article describes existed. Every release we had ever shipped carried that loss; users hitting exactly that workload had been paying it the whole time, and nothing in our tooling would ever have told us.

I had not seen it coming: our benchmark suite had only ever compared the library against its own history. A regression suite tells you when you get slower than yesterday; it cannot tell you that you ship slower than the other library, at the exact workload where users would notice. The day a head-to-head existed, the loss appeared. It had been there all along.

The profiler’s first answer was the ordinary one. Most of the measured deficit was removable interpreter tax: a per-replicate Python index loop, thousands of seed objects spawned per call, stacked object wrapping. But beneath it sat a second, structural problem, the reason the reflexive fix could not win either. The reflexive fix for Python overhead is to batch: replace the loop with a single vectorized NumPy operation. Batching removes the interpreter from the inner loop and swaps it for B-fold traffic through main memory (B being the number of bootstrap replicates, the count every cost in this story scales with) because the batched design materializes every replicate at once. The incumbent’s replicate loop is ordinary Python too. It wins by keeping its working set (the data its loop actively touches) small enough to live in cache, consuming each resample the moment it exists. We were not going to beat that by batching harder.

Here is the machine’s side of the argument. A CPU core computes only on what it holds in its registers. Think of that as your hands. Behind them sit the caches: L1, a desk you can reach without looking up; L2, the shelf behind you; L3, shared with the other cores, a cabinet down the hall. Behind all of that is DRAM: main memory, the warehouse across the street, bigger and slower at every step, orders of magnitude in latency top to bottom. The hardware hides none of that from your runtime; it hides it only from your source code.

Now put a bootstrap on that machine. A bootstrap, one time for anyone who has not run one: it re-draws your data thousands of times to see how much a statistic would wobble across plausible alternate samples, and each redrawn copy is a replicate. Take a modest, entirely realistic job: a series of n = 10,000 observations, B = 2,000 bootstrap replicates, float64 (8 bytes per number). Build all the replicates at once, the way a batched NumPy design wants to, and you have asked for a B x n tensor: 10,000 x 2,000 x 8 bytes = 160 MB. That is far beyond the tens of megabytes of L3 a core complex sees. Every byte of that tensor is written out to main memory, then hauled back in so the statistic can read it once, and if the statistic is a mean, the answer you keep is 2,000 numbers. Sixteen kilobytes, bought with 160 MB of round trips.

Every replicate is a copy of your data.

The tensor is B times larger than the data. It lives in DRAM.

Gather and reduce in the same pass. Keep only the answer.

20 MB where 1.94 GB stood, B = 50,000. measured 2026-07-04 on v0.4.0; the streaming path is unchanged since

The batched design's tensor: assembled, flooding past the cache boundary, and collapsed into the answer. The boundary is schematic; the byte counts are arithmetic on the worked example. At a larger job (fifty thousand replicates of a shorter series) the materializing path peaked at 1.94 GB where the streaming path held 20 MB, roughly 97x (measured 2026-07-04 on version 0.4.0; the streaming path is unchanged since).

data table

quantity	value
replicate tensor, worked example (n = 10,000, B = 2,000, float64)	160 MB
answer kept (one mean per replicate)	16 kB
peak memory at B = 50,000 on an n = 2,000 series: materialize vs streaming	1.94 GB vs 20 MB (~97x), measured 2026-07-04 on v0.4.0
peak resident memory at the worked example’s shape, stationary-block resample (measured 2026-07-11): materialize vs streaming	617 MB vs 234 MB, where 234 MB is the bare process baseline; the streaming run is indistinguishable from a run that computes nothing. The materialize delta exceeds the 160 MB tensor because the path also holds a returned copy and the index matrix

The tsbootstrap that ships today never builds that tensor on its fast path. The story of how it stopped: The first wall is memory traffic: the bytes themselves, moving through the hierarchy. The second took longer to see, because it hides inside the language runtime and the library’s own design, not the hardware: the cost of state (allocated objects and seeded generators) that exists only to make randomness happen in the right order. We stopped materializing arrays, and we stopped carrying state.

Wall one: memory traffic¶

Start with what the batched design actually does. To bootstrap a series with blocks (contiguous runs resampled whole, so the short-range dependence that makes a series a series survives inside each block; the first piece in this series is about what happens when it doesn’t), you draw random block starts, expand them into resample indices, gather the data through those indices, and compute your statistic on each replicate. The batched version does each of those as one array operation over all B replicates: build a (B, n) index matrix, gather into a (B, n, d) values tensor (d is just the number of columns in your series), then reduce along the middle axis.

Count the work in each cell of that tensor. Eight bytes written on the way out. Eight bytes read back for the reduction. Roughly one floating-point addition, one addition per sixteen bytes moved, where a compute-bound kernel like a matrix multiply does hundreds of operations for every byte it pulls. That ratio, the arithmetic intensity (how much computing you do per byte you move), is the number that decides whether your workload is limited by the processor or by the memory system. Ours is about as low as arithmetic intensity gets. The cores were never the constraint; they spent the benchmark waiting on memory. The gather is slow not because the code is interpreted but because its arithmetic intensity is far too low to keep the processor busy: it is a bytes problem being graded as a FLOPs problem.

The incumbent’s loop, seen this way, stops looking old-fashioned and starts looking correct. One replicate at a time means one (n, d) resample at a time: a working set that fits in cache, is consumed by the statistic immediately, and is gone before the next one arrives. That working set never grows with B, so DRAM barely enters its story. The batched design’s traffic grows linearly with B, and once the tensor outgrows the last cache level, every additional replicate’s bytes take the round trip through main memory. NumPy’s programming model quietly couples two decisions that are really separate: batching the dispatch and batching the bytes. The loop was winning because it never built the thing we were building.

Here is the whole argument as accounting, on the worked example: n = 10,000, B = 2,000, float64, one mean per replicate. Redo it on a napkin for your own workload; the ratio is the point.

quantity	materialize path	fused, streaming path
intermediate allocated	160 MB, the full replicate tensor	one `(n,)` index row + one `(n, d)` scratch per thread (~80 kB each)
bytes through DRAM	at least twice the tensor: written out, read back	consumed in cache, per work item
useful output kept	16 kB, one mean per replicate	16 kB, the same answer
bytes moved per byte of answer	~20,000 : 1	~1 : 1

None of this is a new insight: compilers have fused loops to avoid intermediate arrays for decades, and every numerical programmer eventually meets the memory wall. What is worth knowing is that the same wall now dominates transformer training and serving. FlashAttention, the kernel that reshaped both, describes its own core move in the caption of its first figure: it uses “tiling to prevent materialization of the large N × N attention matrix”, and that one refusal, plus fusing the surrounding operations into a single kernel, yielded a 7.6x speedup on GPT-2’s attention. The hierarchy it plays against is the GPU’s version of ours: an A100’s main memory streams at up to 2.0 TB/s, while the tiny on-chip SRAM (192 KB per streaming multiprocessor) runs roughly ten times faster. Same shape, same conclusion: keep the working set in fast memory; never write the big intermediate at all. FlashAttention is the proof that the old compiler principle still binds at datacenter scale.

One pass, and nothing left behind¶

The fix is a single compiled kernel that does everything the batched pipeline did, per replicate and in one pass, without ever building the B-wide tensor. Per-worker scratch still exists, but it is n-sized and transient, not B times the data. The standard name for this is kernel fusion: instead of separate passes that hand each other arrays, one loop body builds, gathers, and reduces for one replicate and emits only the statistic.

Each replicate is one unit of parallel work. The kernel gives it a private (n,) index buffer and a private (n, d) scratch, builds that replicate’s resample indices into the buffer, gathers values through them column by column, applies the reducer (mean, variance, standard deviation, or a quantile) in the same loop body, and writes one row of the (B, d) output. The 160 MB tensor from the opening never exists; peak memory is the source data, per-worker scratch, and the 16 kB we actually wanted. The kernel is compiled to machine code the first time it runs, a just-in-time (JIT) compilation, and that first-call cost is paid off the hot path by an explicit warm-up sweep when the compiled backend loads.

The same byte-mindedness shows up in smaller knobs: resample index tensors are int32 rather than int64, halving their footprint (with an explicit raise the moment an index would exceed what 32 bits can address, rather than a silent overflow), and the returned simulation tensor can be requested in float32 while every model fit and every reduction stays float64. Storage-dtype choices, both of them, made for traffic.

From the outside it is one argument. The materializing call and the fused call take the same method spec and the same seed:

python

import numpy as np
from tsbootstrap import MovingBlock, bootstrap, bootstrap_reduce

x = np.random.default_rng(0).standard_normal(10_000)

# the materializing path: every replicate, as arrays
res = bootstrap(
    x, method=MovingBlock(block_length="auto"),
    n_bootstraps=2_000, random_state=0)

# the fused path: same resampling, one pass, only the statistic survives
red = bootstrap_reduce(
    x, method=MovingBlock(block_length="auto"), statistic="mean",
    n_bootstraps=2_000, random_state=0, backend="compiled")
lo, hi = red.quantile(0.05), red.quantile(0.95)

The memory consequence is not subtle. In a peak-memory sweep over B (measured 2026-07-04 on version 0.4.0; the streaming path is unchanged since), holding the full replicate tensor at B = 50,000 on an n = 2,000 series cost 1.94 GB; the reduce path over the same fifty thousand replicates peaked at 20 MB, roughly 97x lighter. The reduce path’s peak grows with the number of statistics you keep, not with the length of the paths (B rows of answers instead of B full series, which is the whole point), so fifty thousand replicates stops being a capacity-planning question.

Fusing the pass won the big workloads. It did not win everything. The cell it refused to win is where the second wall shows itself.

Wall two: the state you carry¶

After the fused kernel shipped, one benchmark cell refused to move: the smallest one, an i.i.d. resample (independent draws, no blocks, the simplest bootstrap there is) of a short series: 200 observations, 999 replicates. Still losing to the incumbent, run after run, while its neighbors fell by multiples. The cell’s total time held at 13.1 milliseconds like it was nailed there. A kernel this small finishes in microseconds; something else owned the clock. I finally profiled the entire call, not the part I was proud of.

The kernel was innocent. It was barely running.

About 95% of that cell’s wall time was spent before the kernel launched, in Python, setting up random-number seeds. And this is not a fixed dispatch cost: it is an O(B) scaling tax, a serial object-allocation loop that dominates the clock on small jobs and grows linearly to tax the biggest ones before the kernel even starts.

The same workload, twice, in the state redesign's validation run. On the left the per-replicate seed objects are built one by one in Python before any resampling happens, about 95% of the elapsed time, while the kernel waits. On the right each replicate's key is a pure function of one root key and its own index, derived inside the kernel: 13.1 ms becomes 0.55 ms (23.8x end-to-end at n = 200) with identical kernel arithmetic.

data table

design	state per stream	this cell (n = 200, B = 999)
Mersenne Twister (the classic default)	~2.5 kB carried per stream	n/a
B seed objects materialized in Python	one object per replicate, built serially	13.1 ms (~95% seed setup)
keys derived in-kernel from one 128-bit root	none, a pure function of (root, replicate)	0.55 ms

Reproducibility in tsbootstrap is a per-replicate contract: replicate i is bound to its own random stream, stable no matter how the work is chunked or how many threads run it. The obvious implementation is to materialize the streams: spawn B seed objects up front, one per replicate, and hand them out. Correct, and O(B) Python objects allocated on every call. For big workloads the kernel dwarfed it; for small ones the setup was the runtime. We had stopped materializing the data tensor and were still materializing a tensor of state.

Terminology, one time: a PRNG, short for pseudorandom number generator, is the deterministic algorithm behind every “random” draw in scientific computing. Give it the same seed and it replays the same sequence, which is what makes a bootstrap reproducible at all.

Wall two has two faces. The first is ordinary Python overhead: allocating B objects, looping B times, on every call. The second is deeper than Python, and it is the reason GPU people solved this a decade before we needed it. Conventional PRNGs are, in the words of the Random123 paper that changed that corner of computing, “designed as sequentially dependent state transformations”: each draw mutates a state block that the next draw depends on. State means memory: a Mersenne Twister instance drags around about 2.5 kB of it. Worse than the bytes is the sequencing. State that must evolve in order resists being handed to ten thousand independent workers.

The alternative the Random123 authors proposed, counter-based generators that “require little or no memory for state,” inverts the design. Stop treating randomness as a machine you advance; treat it as a pure function: hand it a key and a counter, get back bits, no state anywhere. Any worker can compute draw k of stream b without computing any other draw first.

So we deleted the seed objects, and the nailed-down cell fell: 13.1 ms before, 0.55 ms after, on the same box in the redesign’s validation run, with identical kernel arithmetic. Nothing about the math changed. The setup work simply no longer exists.

Here is what replaced it. The executor now takes a single 128-bit root key and the replicate count, and each backend derives its streams from there; inside the compiled kernel, each replicate’s key is computed from the root and the replicate index by a few integer mixing rounds: nothing allocated, nothing carried, no Python in sight. Our disease was Pythonic, object-allocation cost rather than per-thread kilobytes, but it was exactly the disease Random123 diagnosed: state carried in sequence where a pure function should be. The key for replicate b is now a pure function of (root, b); any specific draw is a pure function of that key and a counter. That sentence is the whole design, and its consequences are the guarantees: run the replicates on one thread or eight, in any order, and the output is bitwise identical, because there is no shared state for a schedule to touch. That guarantee is not pedantry: it is what makes a CI failure mean a real regression rather than thread-schedule noise, and what lets a published interval replay byte for byte months later on a machine with a different core count.

The generator doing the drawing deserves its precise name. The compiled kernels run Philox-4x32-10, the counter-based generator from that same Random123 paper (Salmon et al., SC 2011): ten rounds, the canonical multiplier constants. It is the identical algorithm, with the identical constants, that NVIDIA ships as cuRAND’s Philox device generator, and the same splittable-key philosophy JAX built its entire random API around. The 32-bit variant was chosen deliberately so the integer index streams can one day be made bit-identical between this CPU kernel and an array-backend GPU port. A statistics library on a laptop and a datacenter training run, drawing their randomness from the same ten rounds of the same function, for the same reason: state does not parallelize; functions do. None of this touches the default: the shipping default backend is unchanged and byte-for-byte compatible with previous releases, and the Philox path is opt-in via backend="compiled": equal to the default in distribution, not bit-identical to it.

A hand-rolled generator inside a compiled kernel is also exactly the kind of code that fails silently: a wrong-but-plausible RNG sails through smoke tests and quietly skews every interval you ever publish. So the kernel is never trusted on its own word. The key-derivation math lives twice (once inside the kernel, once in a deliberately slow, dependency-free reference module that takes a different arithmetic path), so a bug in one implementation cannot hide behind agreement with itself; the round function is pinned against the published Random123 known-answer vectors (the paper’s own input-output pairs, which the algorithm must reproduce exactly); and the fused, materializing, and index-only paths all share one index-generation code path, so they cannot drift apart per seed.

The default backend’s own machinery: plain NumPy PCG64 streams, one per replicate; even its chunking uses a fixed chunk size (never derived from available RAM), so accumulation order and results cannot vary across machines. And the compiled path raises loudly if the compiled extra is missing rather than falling back silently.

Two optimizations we measured and deleted¶

The rebuild was not a straight line. Two other ideas were measured during development, on the development machine, and both were reverted; the verdicts, not the multipliers, are what matters.

hypothesis	verdict	why
shared scratch: pre-allocate one buffer block per chunk of replicates instead of per-work-item temporaries	implemented, proven bitwise-identical, net regression on every cell, reverted	the allocator was already pooling per-iteration buffers; the shared scratch defeated SIMD auto-vectorization
cache-blocked gather: sort or tile the resample reads for locality	every locality fix measured as a net loss, disqualified	the stall is bandwidth-bound, and any reorder breaks bitwise identity

The shared scratch was classic advice (allocation is overhead, pooling is virtue), and it lost even single-threaded, because the JIT runtime already pools small per-iteration allocations, and rows of a shared block are strided views (rows spaced apart in memory rather than contiguous), which the auto-vectorizer, the compiler pass that processes several array elements per instruction, refuses to touch. The cache-blocked gather’s diagnosis even measured out (the gather really is memory-stall-bound), but every locality-improving reorder cost more in bookkeeping than it recovered in bandwidth, and because floating-point addition is not associative, reordering the accumulation changes the last bits of the result, which the bitwise-reproducibility contract forbids.

Both hypotheses were plausible enough that a design debate would have shipped them. The measurements killed them in an afternoon, and both verdicts are pinned so neither gets re-attempted. Burying an idea is far cheaper than maintaining one.

The scoreboard, losses included¶

Everything above was measured on dedicated hardware so the numbers mean something: 8-core AMD EPYC Milan cloud instances with dedicated vCPUs, 30 GB of RAM, Ubuntu 24.04 (kernel 6.8), exact package versions pinned in each run’s committed manifest. Sixteen cells: the observation-resampling methods crossed with a range of series lengths and replicate counts, our fused reduce path against the incumbent’s equivalent loop. Every cell’s time is the median of 15 settled timed runs after an explicit warm-up, on the shipping versions of both libraries (tsbootstrap 0.6.1 against arch 8.0.0), one receipt, measured 2026-07-11.

All sixteen cells favor the fused path, and the concession below holds beside them. On the sustained workloads (the longer series where the engines are past their fixed overheads) the settled grid holds 3.1x to 20x. The settled timing exists because single-draw timing failed us first: when we re-timed the grid with one draw per cell, the incumbent’s own control numbers moved 22 to 74 percent between runs of identical code on identical instance types (the box noise exceeded the effect), so nothing measured single-draw is quoted here. The smallest cells post far larger multiples still, and we do not quote them either: at that size the comparison measures the incumbent’s per-replicate dispatch overhead as much as it measures our kernel.

One more disclosure that belongs next to the range: the fused path runs parallel across all eight cores, and the incumbent’s loop is single-threaded by construction. That concurrency is part of what the fused design buys, not a handicap we imposed on the comparison, but you should know the multiple contains it.

And one path still loses. If you want the full (B, n, d) replicate tensor as plain NumPy arrays (the materializing path, the thing the whole article is about not doing), the incumbent’s loop is faster than our default path in every cell of the grid, by 1.2x to 3.1x on the settled 2026-07-11 grid, depending on the cell. That row is a design position, printed with the rest. Materializing is the incumbent’s home game, and it plays it well; we spent our budget on the path where the statistic is the product, because for the statistic-shaped questions a bootstrap exists to answer, nobody needed the tensor in the first place.

The grid, wins and losses on one chart, every series from the settled 2026-07-11 grid (median of 15 timed runs per cell). Fused reduce path: all sixteen cells favor tsbootstrap, 3.1x to 20x on the sustained workloads, a multiple that includes the fused path's eight-core parallelism against the incumbent's single-threaded loop. The squares left of parity are the concession: ask for the full replicate tensor on the default path and the incumbent is faster in every cell.

data table

path	cells	result
fused reduce (compiled, parallel across 8 cores; the incumbent’s loop is single-threaded)	16 / 16	3.1x to 20x sustained on the settled 2026-07-11 grid, median of 15 timed runs per cell
the once-stubborn cell (i.i.d., n = 200, B = 999)	n/a	13.1 ms -> 0.55 ms in the state-redesign validation run (0.49 ms in the settled grid’s own timing)
materializing path (default backend)	0 / 16	incumbent faster in every cell, 1.2x to 3.1x

What the hardware counted while both paths ran the same workload (a stationary-block resample at the worked example's shape, *n* = 10,000, *B* = 2,000) on the same box (measured 2026-07-11; empty-loop baseline subtracted; the miss rate is derived from the null-subtracted counts). The materializing path touches memory 27x more often, misses cache 116x more often, and misses the TLB (the small hardware cache that translates program addresses to physical memory pages) 342x more often. These are the receipts for every claim in this section.

data table

raw counter (per iteration, null-subtracted)	materialize path	fused path
cache references	159,490,000	5,844,000
cache misses	6,171,000	53,100
dTLB (data TLB) misses	47,112	138
derived / measured separately
L1 dcache miss rate (from the null-subtracted counts)	8.6%	2.9%
peak resident memory, the RAM the process actually occupies (234 MB = bare process baseline; the materialize delta exceeds the 160 MB tensor because the path also holds a returned copy and the index matrix)	617 MB	234 MB

The whole rebuild, in three moves:

the move	the reason
never materialize an array whose only consumer is a reduction	the tensor’s bytes cost more than its arithmetic; the answer was 16 kB, the intermediate was 160 MB
fuse the pass: build, gather, reduce in one loop body	a working set that fits in cache is the difference between cache latency and DRAM round trips
derive state, never carry it	B seed objects were 95% of a workload; a key derived from (root, replicate) is free, order-independent, and bitwise reproducible

Frequently asked questions

Why would a bootstrap be limited by memory rather than by computation?

Because its arithmetic intensity is very low: resampling moves eight-byte values around and then does roughly one floating-point operation per value for a statistic like the mean. When a batched implementation materializes all *B* replicates at once, the resulting tensor is far larger than the caches, so every byte takes a round trip through main memory. The processor spends the workload waiting on that traffic, not computing. The fix is to process one replicate at a time inside a fused kernel so the working set stays cache-resident.

What is a counter-based random number generator?

A conventional pseudorandom generator is a state machine: each draw updates internal state that the next draw depends on, which forces sequential execution and per-stream memory. A counter-based generator such as Philox is a pure function instead: it maps a (key, counter) pair straight to random bits, with no carried state. That means any parallel worker can compute any draw of any stream independently and in any order. tsbootstrap's compiled backend derives a Philox-4x32-10 key for each replicate from a single 128-bit root key inside the kernel.

Does the compiled backend change my results?

The default backend is unchanged: NumPy PCG64 streams, one per replicate, reproducing previous releases byte for byte. The compiled backend is opt-in and uses its own counter-based streams, so its replicates are equal in distribution to the default path but not bit-identical to it. Within the compiled backend, results are bitwise reproducible for a given seed regardless of thread count or execution order, and the library raises an explicit error if the compiled extra is unavailable rather than silently falling back.

When is the fused compiled path the wrong choice?

When you actually need the full materialized tensor of resampled series (for a custom diagnostic that inspects whole paths, for example), the materializing path exists for exactly that, and the incumbent's per-replicate loop is quicker at producing the fully materialized array than our default path. The fused path is built for the common case where the bootstrap's output is a statistic per replicate, and there it wins the benchmark grid in every cell.

The fastest code in this library is the code that no longer runs: the tensor nobody builds, the seed objects nobody allocates. Performance work sounds like adding cleverness; the two biggest wins here were subtractions. Somewhere in your own hot path there is probably a 160 MB array whose only job is to be read once and averaged, and next to it a few thousand small state objects built on every call, because carrying state was the design that came to mind first, and main memory bills you by the byte for both, whether or not you ever look. You find these things the way we did: profile the whole call, count the bytes. The FLOPs were never the bill.

The first piece in this series was about a resampler quietly deleting the structure in your data. This one was about a design quietly deleting your performance, byte by byte, in arrays you never needed and state you never had to carry. What happens when the data refuses to be a rectangle? tsbootstrap already packs panels of unequal-length series into one flat array behind a row-offset table (compressed-sparse-row bookkeeping, the layout sparse linear algebra has leaned on for decades), and how a thousand ragged series share one fused pass without padding a single one is where this track goes next.

Your bootstrap is lying to you

Mon, 06 Jul 2026 00:00:00 GMT

TL;DR: The ordinary bootstrap resamples your observations at random, which silently assumes they carry no information about each other. On a time series that assumption fails, and the price is a confidence interval that is far too narrow: on an autocorrelated series I built with a true mean of exactly zero, the i.i.d. bootstrap’s standard error for the mean came out 2.4x too small, and in a repeated coverage study on fresh series from the same process its nominal 90% interval covered the true mean only 49.6% of the time: a “90% interval” that is really a coin flip.

Resampling contiguous blocks instead of single points keeps the dependence and lifts that coverage to 79.7%. It does not fully fix it, and by how much is part of the story.

I built a time series whose mean I knew exactly, because I chose it: zero. Then I asked a bootstrap for a 90% confidence interval on that mean, the way you would ask any statistics library, and it answered [-0.51, -0.19]. Read that back. A ninety-percent interval that does not contain zero is a ninety-percent-confident claim that the mean is negative (a real, signed effect) on data I generated to have no such thing. The interval was tight, it was confident, and it was wrong.

Nothing in the call was exotic. The series was an AR(1) process, the simplest autocorrelated series there is: each value is 0.7 times the previous value plus a fresh random shock. Autocorrelation just means today’s value carries information about tomorrow’s, the thing that makes a time series a time series and not a bag of independent draws. I used the default bootstrap, the one every tutorial reaches for first. And it lied to me by a factor of two, with a straight face.

This is the failure the whole tsbootstrap library exists to fix. Here is where the lie enters, and what to do about it.

What the bootstrap actually does¶

The bootstrap is one of the most useful ideas in applied statistics: to find out how uncertain a number is, resample your data with replacement many times, recompute the number on each resample, and look at how much it wobbles. You have one dataset, but the bootstrap manufactures thousands of plausible alternate datasets out of it and lets the number’s spread across them stand in for the uncertainty you could not otherwise measure. No formula, no parametric model you have to name out loud. It is close to magic, and it usually works.

The usually is doing a lot of work. The ordinary bootstrap (i.i.d., for independent and identically distributed, the label you will see in the figures) draws each resampled observation independently and uniformly from your data: pick a row at random, put it back, pick another. That independence is the entire assumption. Drawing observations independently is a mathematical statement that the order of your data does not matter, that the rows are interchangeable. For a bag of independent measurements, they are. For a time series, they are not, and shuffling them destroys the one property that made the data worth collecting in order.

The same series, resampled two ways. The block bootstrap lifts out contiguous chunks, so the resampled series still moves like the original; the i.i.d. bootstrap shuffles single points and shreds the wave into noise. The shredded version looks more ‘random,’ and that is why it reports a smaller standard error (the typical miss you should expect between the sample mean and the true mean) than the truth (grey line).

data table

quantity	SE of the mean	reads as
i.i.d. shuffle	0.097	the collapse (dependence discarded)
moving-block	0.176	dependence kept
analytic truth	0.236	the yardstick

Watch the two panels. On the right, the i.i.d. shuffle takes my autocorrelated wave and scatters its points at random; the resampled series looks like static. On the left, the block bootstrap lifts out contiguous runs of the series and lays them back down intact, so the resampled version still rises and falls in the same lazy way the original did. That static-looking version is the one that understates uncertainty, the opposite of what your gut says.

Why shredding the wave shrinks the interval¶

When observations are positively autocorrelated, they move in the same direction for a while. A run of above-average values tends to be followed by more above-average values before the series wanders back. That persistence means your sample does not contain as much independent information as its length suggests: two hundred points that each nudge their neighbor are worth fewer than two hundred points drawn fresh and independent. Fewer effective observations means the sample mean is a shakier estimate of the true mean than the raw count of points implies.

You can put a number on “fewer.” For an AR(1) at phi = 0.7 (phi is that 0.7 multiplier from earlier, how strongly each value leans on the last), the effective sample size (the number of independent points that would carry the same information about the mean) is n / ((1 + phi) / (1 - phi)) (the denominator counts how much each point merely repeats its neighbors’ information), which for my two hundred points is 200 / 5.67 = 35. Two hundred autocorrelated readings are worth about thirty-five independent ones. The i.i.d. bootstrap computes the uncertainty of a mean built from two hundred independent points; the truth is the uncertainty of a mean built from thirty-five. That shrinkage, two hundred down to thirty-five, is where the factor of 2.4 comes from: uncertainty in a mean shrinks with the square root of the number of independent points, so the inflation is the square root of the ratio, sqrt(200 / 35) = 2.4.

The i.i.d. bootstrap never sees any of this. By drawing points independently, it builds resampled series that have no autocorrelation at all: it manufactures the “two hundred independent points” fiction whole, and it measures the wobble of the mean under that fiction. So it reports the uncertainty you would have if your data were independent, which is smaller, often much smaller, than the uncertainty you actually have. The scattered series on the right of the figure is not a bug in the resampling; it is the resampling doing just what it was designed to do, on data that violates its one assumption.

The block bootstrap fixes this by resampling contiguous blocks (chunks of consecutive observations) instead of single points. Because each block is a run of real, consecutive data, it carries the local ups and downs with it. Glue enough blocks together and the resampled series keeps the autocorrelation of the original, so the wobble of its mean reflects the real, reduced information content of dependent data. It resamples in a way that respects the assumption the i.i.d. bootstrap breaks.

The version in the figure is the moving-block bootstrap: slide a window of fixed length along the series, and resample whole windows with replacement. It is the oldest and simplest of the block methods; the others differ in how they choose and glue the blocks, and they get their own article later in this series.

How wide should the interval actually be?¶

I do not have to guess at the truth here, because I built the data. For an AR(1) series you can write the honest standard error of the mean in closed form. The standard error is just the typical size of the gap between your sample mean and the true mean, how far off you should expect to be. For independent data it is sigma / sqrt(n), the data’s typical spread divided by the square root of the count, where sigma is the standard deviation of the series itself, not of the shocks driving it. For an AR(1) with autocorrelation phi, dependence inflates it by a factor that depends only on phi:

SE_true = (sigma / sqrt(n)) * sqrt((1 + phi) / (1 - phi))

That square-root factor is the whole story. At phi = 0.7 it equals sqrt(1.7 / 0.3) = 2.38. The honest standard error of the mean is nearly two and a half times larger than the one you would get by pretending the data were independent, and the i.i.d. bootstrap, by construction, targets the independent one. It is 2.4x too confident, before you have done anything wrong yourself.

The factor is a property of how autocorrelated the data is, not of which block method you pick. At phi = 0.3 it is only 1.36, so the lie is mild; at phi = 0.9 it is 4.36, and a naive 90% interval misses the truth most of the time. Stronger dependence, bigger lie. My phi = 0.7 sits in the range plenty of real economic and sensor series actually occupy.

Put the standard errors side by side and the collapse is obvious. On my one example series, the i.i.d. bootstrap’s standard error for the mean is 0.097. Every block method lands far higher, 0.16 to 0.20, and the analytic truth is 0.24. The i.i.d. bar is about two-fifths the height of the truth.

Standard error of the mean, one series, six ways to measure it. The naive i.i.d. bootstrap (orange) collapses to 41% of the true long-run standard error (grey rule). Every block method (blue) reaches most of the way to it, though notice none of them quite gets there, which will matter in a moment. The gaps between the four block bars are single-series noise, not a ranking.

data table

method	SE of the mean	vs. true (0.236)
i.i.d. (naive baseline)	0.097	41%, collapsed
stationary block	0.163	69%
tapered block	0.170	72%
moving block	0.176	75%
circular block	0.197	84%
analytic true long-run SE	0.236	100% (reference)

I have shipped this exact mistake¶

I did not learn this from a textbook warning. I learned it because I once reported a result that was not there.

Years ago I had a run of daily measurements and a question about whether their average had shifted from a baseline of zero. I did the responsible-looking thing: I bootstrapped a confidence interval instead of trusting a formula, the interval excluded zero, and I wrote down that the shift was real. It was not. The measurements were autocorrelated (of course they were, they were daily readings of a slow physical process), and my “responsible” bootstrap had shredded that dependence the same way the figure above shredded the wave. The effect I found was nothing more than the interval’s collapse. Nobody caught it, because nothing looked wrong; a tight interval that excludes zero is the most ordinary output in the world. That is the quiet part of this failure that unsettles me most. It does not announce itself.

So how often does it actually lie?¶

One series and one anecdote are not evidence, so I ran the experiment properly. Generate a fresh AR(1) series with phi = 0.7, length 200, true mean zero. Compute the i.i.d. bootstrap’s 90% percentile interval and the moving-block bootstrap’s 90% percentile interval for the mean, the same simple interval construction for both, so the only thing that differs between them is the resampler. Check whether each one contains the true mean of zero. Do that two thousand times and count.

This is a coverage study, the honest test of any confidence interval. A method that advertises 90% coverage is making a falsifiable promise: across many datasets, its intervals should contain the truth 90% of the time. Fewer than that and the interval is too narrow; the “confidence” is inflated.

The i.i.d. bootstrap’s nominal 90% interval covered the true mean 49.6% of the time.

Not 90%. Not 85%. A coin flip. Half the time the interval you would have reported, and believed, does not contain the answer, and it never tells you which half you are in.

Consider what that does to everyday inference. A 90% interval that excludes zero is the standard way people declare an effect “significant” at the 10% level. If that interval is really covering the truth half the time on autocorrelated data, then every such call on the mean of a series with this much dependence carries a one-in-two error rate, not the one-in-ten it advertises. The analyst did nothing careless; the default resampler shipped them the wrong width. I am not claiming this voids any particular published result; I am claiming the gap between 90% and 49.6% is large enough that the burden is on the interval to prove it earned its confidence, and the ordinary bootstrap on a time series has not.

The moving-block bootstrap, on the same two thousand series, covered the truth 79.7% of the time. Its intervals were about twice as wide on average. That extra width is the uncertainty that was really there all along, the uncertainty the i.i.d. bootstrap deleted.

The 49.6% is not a fluke of the simulation. You can predict the expected i.i.d. coverage from a closed-form normal approximation using the same phi = 0.7: it comes out to 51.0%. The measured 49.6% lands within 1.5 percentage points of that (consistent with Monte-Carlo noise), so the undercoverage is structural, not a simulation artifact.

The promise against the delivery, on one scale. Both methods advertise the same 90% (grey rule). The i.i.d. bootstrap delivers 49.6%, a coin flip wearing a confidence interval; the moving-block bootstrap delivers 79.7%, most of the way back and measurably not all of it.

data table

quantity	coverage	reads as
nominal (the promise)	90%	the yardstick
i.i.d. bootstrap	49.6%	a coin flip
moving-block bootstrap	79.7%	most of the way back

But 79.7% is not 90% either. The block bootstrap recovers most of the gap the i.i.d. bootstrap opens, and it stops short. At a series this short, block resampling still understates the truth: you can see it coming in the chart above, where even the tallest block bar fell shy of the grey line. The two shortfalls are not the same kind of wrong: give the block bootstrap a longer series and its coverage climbs toward the 90% it promises, while the i.i.d. bootstrap’s gap never closes, because it is baked into the resampler rather than the sample size. Block bootstrapping is the right tool here, and it is honest about its own limits; a piece that sold it as a cure would be committing a smaller version of the same sin I am describing. An interval that delivers half its promised coverage is a different category of wrong.

The fix is contiguity, and it is one argument¶

Everything above reduces to a single call. The library takes the observed series, the statistic you care about, and a method (a typed description of how to resample) and returns the interval:

python

import numpy as np
from tsbootstrap import IID, MovingBlock, conf_int

# an AR(1) series, phi=0.7, n=200, whose true mean is exactly 0
def ar1(n, phi, seed, burnin=500):
    rng = np.random.default_rng(seed)
    e = rng.standard_normal(n + burnin)
    x = np.empty(n + burnin)
    x[0] = e[0]
    for t in range(1, n + burnin):
        x[t] = phi * x[t - 1] + e[t]
    return x[burnin:]                      # drop the burn-in; true mean = 0

x = ar1(n=200, phi=0.7, seed=42)

# the lie: resample observations independently
lo, hi, point = conf_int(
    x, "mean", method=IID(),
    alpha=0.10, random_state=0)
# -> (-0.508, -0.188)
# a 90% interval that excludes the true mean, 0

# the fix: resample contiguous blocks, length chosen automatically
lo, hi, point = conf_int(
    x, "mean", method=MovingBlock(block_length="auto"),
    alpha=0.10, random_state=0)
# -> (-0.556, 0.022)
# now the truth is inside

The only thing that changed between the lie and the fix is the method. That is deliberate. There is a small family of block bootstraps (moving, circular, stationary, tapered) and they differ in how they cut the series into blocks and stitch them back together. Each is the right answer to a slightly different question about your data’s structure. The one that chose the block length above did it with a standard automatic rule (Politis and White’s spectral plug-in estimator, the textbook default), so you get a defensible answer with no tuning; a hand-picked length could squeeze out more of the residual gap, but that is a refinement.

The headline is the thesis this whole series is built on, and it is worth stating plainly, because every later piece is a variation on it: match the resampler to the assumption your data violates. The i.i.d. bootstrap violates independence on a time series and pays for it in coverage. Later you will meet data that breaks other assumptions (errors whose variance shifts over time, models whose residuals are themselves dependent), and each one has a resampling scheme built to respect what it breaks. The bootstrap is not one tool. It is a question you have to answer honestly: what does my data actually do?

What I now check before I trust an interval¶

I still bootstrap everything. But I no longer read a tight interval as good news, and I no longer reach for the default resampler on data that has an order.

what I read as true	what was actually true	the rule it became
a tight interval means a precise estimate	on autocorrelated data, a tight interval means the resampler deleted the dependence	narrowness is a red flag on a time series, not a comfort
the bootstrap is assumption-free	the ordinary bootstrap assumes your observations are interchangeable	name the assumption before you resample, because the default has one
a 90% interval covers the truth 90% of the time	this one covered it 49.6% of the time	test coverage against a known truth before you trust a method on an unknown one
the block bootstrap fixes it	it recovered coverage to 79.7%, not 90%	the right tool can still undercover; honesty about the residual gap is part of the fix

The interval I opened with, [-0.51, -0.19], still sits in my terminal history. It is a perfectly ordinary-looking result: a signed effect, a tight band, ninety percent confidence. It is also a statement about data with a mean of zero that the mean is not zero. Nothing about the number looks like a lie, which is why the ordinary bootstrap gets away with telling it. Change one argument (resample the blocks, not the points) and, on this series, the truth climbs back inside the interval, where it was the whole time.

Frequently asked questions

What is tsbootstrap?

tsbootstrap is a time series bootstrap library for Python. It resamples a series while preserving the temporal dependence that an ordinary bootstrap destroys, using the methods the statistics literature recommends (block, residual, sieve, and wild resampling) plus a conformal layer for prediction intervals. The public API is one bootstrap call configured with a typed method spec, and a conf_int helper that runs the bootstrap and reads the interval in one call.

Why does the ordinary bootstrap fail on time series data?

The ordinary (i.i.d.) bootstrap resamples observations independently, which assumes they carry no information about each other. A time series violates that: consecutive values are correlated. Resampling single points at random destroys the autocorrelation, so the resampled series looks more random than the real one and the bootstrap reports a standard error that is too small. On an AR(1) series with autocorrelation 0.7, the i.i.d. standard error of the mean comes out about 2.4 times too small.

What is a block bootstrap and how does it fix this?

A block bootstrap resamples contiguous chunks (blocks) of consecutive observations instead of single points. Because each block is a run of real, adjacent data, it carries the local dependence with it, so the resampled series keeps the autocorrelation of the original. That gives a standard error and confidence interval that reflect the real, reduced information content of dependent data. In tsbootstrap you select it by passing method=MovingBlock(block_length="auto") to bootstrap or conf_int; the "auto" length comes from Politis and White's spectral plug-in estimator, the textbook default rule.

Does the block bootstrap give perfectly correct confidence intervals?

No, and it should not be sold as if it does. In a 2,000-run coverage study on AR(1) data (*phi* = 0.7, *n* = 200), the moving-block bootstrap's nominal 90% interval covered the true mean 79.7% of the time, versus 49.6% for the i.i.d. bootstrap. It recovers most of the gap but still undercovers at short series lengths. The block bootstrap is the right tool, an honest improvement that falls short of exact coverage.

The ordinary bootstrap, pointed at a time series, breaks the coverage promise by a wide margin and without a warning. The next piece takes a different broken assumption (errors whose variance shifts over time) and the resampler built to respect it.

When your errors aren’t equal

Mon, 06 Jul 2026 00:00:00 GMT

TL;DR: The residual bootstrap resamples your model’s errors by shuffling them across time, which silently assumes every error was drawn from the same distribution. When the error variance shifts over time (larger in some regimes than others) that shuffle averages the variance profile away, and the price is a standard error that is too small exactly where it matters. On a heteroskedastic autoregression I built, the shuffle recovered on average only 78% of the true standard error of the slope, and its nominal 90% confidence interval covered the truth 79.1% of the time.

The wild bootstrap (keep each residual at its own time position, multiply it by a random sign) recovered 100% of the true standard error and lifted coverage to 88.8%. Same model and data; one argument changed.

I built a time series whose errors I controlled completely, because I wrote the noise myself: an AR(1) with slope 0.5 (each value is half the previous value plus a fresh random shock, so the 0.5 is a slope you can fit like any regression), six hundred points long, where the shocks in the second half are three times louder than in the first, nine times the variance. Then I fit the model, asked a residual bootstrap for the standard error of that slope (the typical miss between the fitted 0.5 and the truth) and compared it to the truth, which I can compute to four decimal places by brute force because I own the data-generating process. The truth is 0.0453. The bootstrap said 0.034.

That is not a rounding disagreement. It is a quarter of the uncertainty, gone, on a series where nothing was misspecified and the bootstrap did exactly what it was designed to do. In the first piece in this series, the ordinary bootstrap lied by shredding autocorrelation, and the block bootstrap fixed it by resampling contiguous chunks. This is the second failure mode, and it lives one layer deeper: not in how you resample the observations, but in how you resample the model’s errors.

The bootstrap under the bootstrap¶

The block methods from the first article resample the observed series directly. There is a second family, the residual bootstrap, that goes through a model instead: fit the model, collect the leftover errors it could not explain (the residuals), resample those, and rebuild synthetic series by running the fitted model forward on the resampled errors. It is the natural choice when you trust the model’s structure and want uncertainty on its parameters, which is precisely the situation of anyone bootstrapping a regression slope, an autoregression coefficient, or a fitted forecast.

And how does it resample the residuals? The default answer is the same default as always: independently and uniformly, with replacement. Pick a residual at random from anywhere in the series, put it back, pick another. Which is a mathematical statement that the residuals are exchangeable, that the error the model made at time 12 could just as well have happened at time 580, because they all came from one common distribution.

There is a name for the property that assumption requires: homoskedasticity, from the Greek for “same scatter”: the error variance is constant over the whole series. Its violation, heteroskedasticity (“different scatter”), shows up everywhere: financial returns in calm and turbulent regimes, sensor noise at different temperatures, forecast errors between weekdays and holidays. If your data was collected over any interesting stretch of time, the odds are good its noise level did not hold still. The check is cheap: plot the squared or absolute residuals in time order and look for a level that drifts or steps.

The tight number I believed¶

I did not go looking for this failure. It turned up in a mean-reversion estimate I once made on a spread whose noise regime changed partway through the sample: a venue migration, roughly doubling the tick-to-tick noise from that date on. I knew better than to trust a textbook formula, so I bootstrapped the reversion coefficient’s standard error from the fitted model’s residuals and got a satisfyingly tight number. What I did not register was that my resampler was lifting residuals from the loud months and planting them in the quiet months, and vice versa, building synthetic histories in which the noise regime never changed at all. The tight standard error was a measurement of those counterfeit series. The strategy sized off that estimate was, in hindsight, sized off roughly three quarters of the real uncertainty; it survived, but by luck rather than arithmetic, and I only worked out why long after the trade was closed.

As in the first article, nothing looked wrong: a residual bootstrap on a fitted model is the responsible choice. The failure has no error message.

Shuffling averages the variance away¶

The mechanism fits in one sentence: shuffling residuals averages the variance away; flipping their signs keeps it in place.

In my built series, the second-half residuals are three times larger than the first-half ones, and, this is the load-bearing detail, the second half is also where the series itself swings widest, since louder shocks push the process further from its mean. The uncertainty of a regression slope is dominated by the errors that sit on the extreme values of the regressor (the input on the x-axis of the fit, which for an AR(1) is simply yesterday’s value). A big error on a big swing tilts the fitted line hard. Heteroskedasticity that is correlated with the regressor level concentrates the largest errors exactly where they do the most damage, and an honest standard error has to account for that.

The i.i.d. shuffle (independent and identically distributed) destroys the correlation. It takes the loud residuals and scatters them uniformly across time, so in the synthetic series a big error is no likelier to land on a big swing than a quiet one is. Averaged over replicates, every time position experiences the average error variance: the profile is flattened and the slope wobbles less than it truly would. The bootstrap then reports that reduced wobble as your standard error. On my series: 0.034 reported, 0.0453 true.

The same fitted residuals, resampled two ways. The wild bootstrap keeps every residual at its own time position and only randomizes its sign, so the loud half of the series stays loud in every replicate, and the refitted slopes spread out to the honest width. The i.i.d. shuffle relocates the residuals, flattening the variance profile, and reports a standard error a quarter too small.

data table

quantity	SE of the slope	reads as
i.i.d.-residual shuffle	0.034	the collapse (variance profile averaged away)
wild (Rademacher flips)	0.045	variance kept where it lives
Monte-Carlo truth (200k draws)	0.0453	the yardstick

Flip the signs instead¶

Now the second half of the sentence. The wild bootstrap refuses to move anything. Each synthetic error is the original residual, at its original time position, multiplied by a fresh random draw:

math

e_t^* = v_t \, \hat{e}_t

where the multiplier $v_t$ is drawn independently at each $t$ from a distribution satisfying two moment conditions:

math

\mathbb{E}[v] = 0, \qquad \mathbb{E}[v^2] = 1 .

Mean zero makes the synthetic errors centered, like errors should be. Variance one is the trick: it means $e_t^*$ has exactly the variance of $\hat{e}_t$ , at every single $t$ (the multiplier is drawn independently of the residual, so $\operatorname{Var}(v_t \, \hat{e}_t) = \mathbb{E}[v^2] \, \hat{e}_t^2 = \hat{e}_t^2$ ). A loud residual stays loud, a quiet one stays quiet, and the whole variance profile, whatever shape it has, known or unknown, is preserved by construction. You never estimate the heteroskedasticity; you just decline to destroy it.

The simplest distribution meeting both conditions is the Rademacher: $v = +1$ or $-1$ , a fair coin. Under it, the wild bootstrap literally just flips residual signs, which sounds too crude to work, except that a sign flip is exactly as much randomness as a resampler needs: the refit cares only about which way each error pushes the line and how hard, and the flip randomizes the direction while the residual keeps the how-hard. Some multipliers add a third condition, $\mathbb{E}[v^3] = 1$ , which additionally preserves each residual’s skewness (the asymmetry of the errors, not just their size). That is why the Mammen two-point distribution exists, with its odd-looking golden-ratio values: it is the two-point law that satisfies all three conditions at once. The working answer: use the coin, and reach for Mammen only when the errors are strongly skewed and the asymmetry of the interval matters.

The coin flip is the actual literature recommendation, crude as it looks. Davidson and Flachaire (2008) found the Rademacher version generally performs best in practice even on skewed errors, which is why it is the default here and in most implementations. Mammen (1993) supplied the theory for the skewness-preserving alternative.

Three multiplier distributions, one contract. All satisfy the two moment conditions; Mammen alone adds the third, preserving skewness. On this article’s data all three are statistically indistinguishable, so read this as a glossary of mechanisms; on this data there is no leaderboard.

data table

distribution	values	probabilities	E[v²]	E[v³]
Rademacher	−1, +1	1/2, 1/2	1	0
Mammen	−0.618, +1.618	0.724, 0.276	1	1
Gaussian	N(0,1)	n/a	1	0

Five hundred series, one verdict¶

One series proves nothing, so here is the experiment in full, self-contained enough to check. Generate 500 fresh series from the heteroskedastic AR(1) above (slope 0.5, n = 600, error variance 9x larger in the second half). For each, fit the model and bootstrap the standard error of the lag-1 slope four ways (the i.i.d.-residual shuffle and the wild bootstrap under each of the three multipliers) with 999 replicates each. Divide every bootstrap standard error by the true one, 0.045326, computed from 200,000 independent Monte-Carlo draws of the process. A perfect resampler averages 1.0. All of it ran on tsbootstrap 0.6.1 on one 8-core Linux workstation, measured 2026-07-06.

The i.i.d.-residual bootstrap recovered on average 78% of the true standard error (mean ratio 0.783). The wild bootstrap with Rademacher multipliers recovered 100% (mean ratio 1.001); Mammen and Gaussian multipliers landed at 0.997 and 0.990. And the miss was unanimous: the i.i.d. standard error came out below the wild one in 500 of 500 series: the wild estimate ran 1.28x larger on average, and the gap is exactly the uncertainty the shuffle deleted.

Mean recovered fraction of the true slope standard error, 500 heteroskedastic AR(1) series, 999 replicates each. The i.i.d.-residual shuffle (orange) stalls at 78% of the truth; every wild variant (blue) sits at the grey truth line. The three multipliers are indistinguishable here; the whole gap sits between shuffling and flipping.

data table

residual resampler	mean SE / truth	reads as
i.i.d.-residual (naive baseline)	0.783	78%, collapsed
wild, Rademacher	1.001	100%
wild, Mammen	0.997	100%
wild, Gaussian	0.990	99%
Monte-Carlo truth (200k draws)	1.000	reference

Two scope notes. The 78% belongs to this process, where the loud errors sit on the big regressor values; heteroskedasticity that is uncorrelated with the regressor bites far less. And on a genuinely homoskedastic, well-specified model the i.i.d.-residual bootstrap is fine: the shuffle only lies when there is a variance profile to flatten. The wild bootstrap removes the question: flipping works whether or not the shuffle would have failed.

What a collapsed standard error costs you downstream¶

A standard error is an intermediate quantity; the thing you act on is the interval. So: 2,000 more fresh series from the same process, a nominal 90% percentile confidence interval (read straight off the replicates’ percentiles) for the slope from each method, 999 replicates, and a count of how often the interval contains the true 0.5.

The i.i.d.-residual intervals covered the truth 79.1% of the time. The wild (Rademacher) intervals covered 88.8%, running 1.27x wider on average, width the shuffle had been withholding.

And no, 88.8% is not 90%. Part of the residual gap has nothing to do with variance at all: the least-squares slope of an AR(1) is biased slightly downward in finite samples (across the 200,000 truth draws it averaged 0.4947 against a true 0.5), and percentile intervals at 999 replicates carry their own small-sample wobble. Both leftovers belong to the estimator and the percentile rule, not to the resampling. The wild bootstrap recovers the standard error and most of the coverage; it does not repair the estimator it is wrapped around. If the first article taught suspicion of intervals that are too tight, the habit worth keeping from both is asking which assumption each resampler restores, and checking coverage when the answer matters.

The fix is one argument¶

In tsbootstrap the residual bootstrap is a method spec, and how the residuals are resampled is its innovation, a typed argument:

python

import numpy as np
from tsbootstrap import (
    AR, IID, Wild,
    ResidualBootstrap, bootstrap)

# hetero AR(1): slope 0.5, n=600,
# shock sd 1 then 3 (9x variance)
def hetero_ar1(n, phi, seed):
    rng = np.random.default_rng(seed)
    z = rng.standard_normal(n)
    late = np.arange(n) >= n // 2
    sd = np.where(late, 3.0, 1.0)
    x = np.empty(n)
    x[0] = z[0]/np.sqrt(1 - phi**2)
    for t in range(1, n):
        x[t] = phi*x[t-1]+sd[t]*z[t]
    return x

x = hetero_ar1(600, 0.5, seed=42)

def slope_se(paths):
    # refit the slope per replicate
    slopes = [
      np.polyfit(p[:-1], p[1:], 1)[0]
      for p in paths]
    return np.std(slopes, ddof=1)

# the lie: shuffle residuals in time
res = bootstrap(
    x,
    method=ResidualBootstrap(
        model=AR(order=1),
        innovation=IID()),
    n_bootstraps=999,
    random_state=0,
)
se = slope_se(res.values())
print(round(se, 3))
# slope SE: 0.034   (truth 0.0453)

# the fix: keep each residual in
# place, flip its sign at random
res = bootstrap(
    x,
    method=ResidualBootstrap(
        model=AR(order=1),
        innovation=Wild()),
    n_bootstraps=999,
    random_state=0,
)
se = slope_se(res.values())
print(round(se, 3))
# slope SE: 0.045   (truth 0.0453)

Wild() defaults to Rademacher multipliers; Wild(distribution="mammen") and "gaussian" are one keyword away. Everything else (model, data, replicate count, seed) is identical between the lie and the fix. As of this writing (July 2026), none of arch, darts, MAPIE, or TorchCP ships a wild-bootstrap innovation spec or a block-wild variant as a first-class resampling option.

When the residuals themselves are dependent¶

One assumption is still hiding in the wild bootstrap: drawing each $v_t$ independently asserts that the residuals, whatever their variances, are at least uncorrelated with each other. For a well-specified model that is what residuals should be. But when the model is a little wrong (an underfit autoregression, structure at a horizon the model does not see) the leftover errors are themselves locally dependent, and independent sign flips would shred that dependence the same way the i.i.d. bootstrap shredded the wave in the first article. The concrete picture: fit an AR(1) to a series that is truly AR(3), and the two lags the model never sees leave ripples behind: each residual is correlated with its own recent past, and the residual autocorrelations show it at short lags.

The extension is the block-wild bootstrap: partition the time axis into contiguous blocks and give every residual in a block the same multiplier. Within a block, relative signs (and therefore local correlation) are preserved exactly; across blocks, the flips are independent. You can measure the machinery doing its job directly, on the multiplier process itself: at block length 10, neighboring multipliers correlate at 0.904, multipliers five steps apart at 0.513, and twelve steps apart (past the block edge) at 0.007, indistinguishable from zero. The measured points sit on the triangular law $\max(0,\, 1 - h/L)$ (correlation at lag h for block length L) that the block construction implies.

This block-constant scheme is the piecewise-constant case of Shao’s (2010) dependent wild bootstrap, and the same construction as the wild cluster bootstrap of Cameron, Gelbach and Miller (2008), with time blocks playing the role of clusters.

What a block multiplier preserves. With block length 10 (Rademacher multipliers, 2,000 replicates), the sign flips applied to nearby residuals are almost perfectly correlated (0.904 at lag 1), half-correlated at lag 5, and independent past the block edge; the measured correlations track the triangular law from the construction. This is what lets block-wild keep local residual dependence that independent flips would destroy.

data table

lag h	measured correlation	theory 1 − h/10
1	0.904	0.9
5	0.513	0.5
12	0.007	0

Block-wild is not a free upgrade over plain wild, though. On this article’s process the residuals are independent by construction, so there is no local dependence to preserve, and block-wild duly buys nothing. Its automatic block-length rule resolved to a length of 2 on my showcase series, and its standard-error recovery there, 0.856 of the truth, actually sat below plain wild’s. Blocking the multipliers spends resampling variety to protect a structure, so when the structure is absent it costs variety for no gain. So the order of operations is explicit. First, fix the model: dependent residuals are a misspecification symptom before they are a resampling problem, and a refit that absorbs the missing lags beats any bootstrap patch. When you cannot refit, block-wild is the patch that respects what the residuals still carry. And when the residuals pass the checks (no drift in their spread, no short-lag correlation) plain wild is all you need.

Which resampler, then?¶

The decision this piece adds to the last one’s, in one table:

your data’s situation	the assumption at risk	reach for
dependent observations, no model you trust	independence of observations	block bootstrap (moving, circular, stationary)
trusted model, steady error variance	none (defaults hold)	i.i.d.-residual bootstrap
trusted model, error variance shifts over time	homoskedasticity of errors	wild bootstrap, Rademacher default
shaky model, residuals locally dependent	independence of errors	block-wild, then question the model

The first article’s thesis holds here unchanged: match the resampler to the assumption your data violates. Only the assumption moved. That failure lived in the observations; this one lives in the errors of a model you already trusted, which is a more dangerous address, because trusting the model is what made the analysis feel careful. My tight little 0.034 came from a fitted model and 999 replicates of honest-looking work. The truth, 0.0453, was never hidden. It was flattened, averaged into the quiet half of a history that never happened. The wild bootstrap’s whole contribution is refusing to build that history: every residual stays home, and the only randomness is a sign.

Frequently asked questions

What is the wild bootstrap?

The wild bootstrap is a resampling method for regression and time series models with heteroskedastic errors, meaning error variance that shifts over the sample. Instead of shuffling the fitted residuals across time, it keeps each residual at its own position and multiplies it by an independent random draw with mean zero and variance one (by default a Rademacher sign flip: +1 or −1 with equal probability). That preserves each residual’s variance and position exactly, so the bootstrap reproduces the true variance profile without ever estimating it. In tsbootstrap you select it by passing innovation=Wild() to ResidualBootstrap.

When does the ordinary residual bootstrap fail?

When the errors are heteroskedastic (their variance shifts over time), and especially when the loud errors coincide with extreme regressor values. Shuffling residuals across time averages the variance profile away, so the bootstrap understates the uncertainty. On a heteroskedastic AR(1) (slope 0.5, *n* = 600, error variance 9x larger in the second half), the i.i.d.-residual bootstrap recovered on average only 78% of the true standard error of the slope across 500 simulated series, and its nominal 90% interval covered the truth 79.1% of the time (tsbootstrap 0.6.1, 999 replicates per bootstrap, measured 2026-07-06). The wild bootstrap recovered 100% of the true standard error on the same series, with 88.8% coverage.

Which wild bootstrap multiplier should I use: Rademacher, Mammen, or Gaussian?

Rademacher (random sign flips) is the standard default, following Davidson and Flachaire (2008), who found it generally performs best in practice. Mammen’s two-point distribution additionally preserves the skewness of each residual (its third moment equals 1), which theory favors for strongly asymmetric errors; the Gaussian multiplier is a smooth alternative. In the measured experiment here all three recovered 99 to 100% of the true standard error; the choice that matters is wild versus shuffle, not the multiplier.

What is the block-wild bootstrap and when is it needed?

The block-wild bootstrap gives every residual within a contiguous time block the same multiplier, preserving local correlation among residuals, useful when the model is misspecified enough that its residuals are still dependent. It is the piecewise-constant case of Shao’s (2010) dependent wild bootstrap. It is not a free upgrade: on a well-specified model with independent residuals, plain wild is the better choice, and in tsbootstrap BlockWild(block_length="auto") will resolve to a very short block in that case.

The next failure in this series has nothing to do with statistics: when you need a million replicates, you have to be able to build them at all.