A 4K albedo texture is 64 MB as RGBA8. The dungeon ships dozens of material sets, each with an albedo and a companion normal-plus-height map, and all of it has to sit in VRAM while the GPU samples it every frame. Uncompressed is not an option.
BC7 is the answer the hardware already provides: a fixed 4:1 block format that every GPU since about 2010 decodes in silicon, at full speed, with no shader cost. Each 4×4 pixel block becomes exactly 16 bytes. The question is not whether to use it — it is how well you can pack those 16 bytes, and that is entirely the encoder's problem.
There have now been three encoders. The first was written in an afternoon and used one of BC7's eight modes. The second trialled two and stopped the material edges smearing. The third trials four — but the interesting part of the third is not the extra modes. It is that it came with a test that can fail, and the test kept telling me I was wrong. There is no fourth encoder for the same reason: the fifth mode was measured before it was written, and the measurement said don't.
The problem with one mode
BC7 mode 6 is the sensible default: one subset, RGBA endpoints at 7.7.7.7 bits plus a per-endpoint p-bit, and 4-bit indices. Sixteen pixels, one colour line through the block, each pixel placed somewhere along it at one of sixteen positions.
That is excellent for a block sitting inside a single material — smooth stone, a gradient, anything whose colours genuinely lie on a line. It falls apart at a boundary. A block straddling brick and mortar contains two clusters of colour that are nowhere near collinear, and mode 6 has to run a single line between them, averaging through the gap in the middle. Every pixel lands somewhere on that compromise line. The result is muddy, smeared edges exactly where the eye is most likely to be looking.
The decision
Trial every eligible mode per block and keep the lowest-error result.
That is the whole architecture, and it has not changed since the second encoder — only the number of candidates has. A mode is a way of dividing the same 128 bits, and each division is a different bet about where the block's information is. Mode 6 bets on index precision: one line, sixteen positions. Mode 1 spends bits on a partition field to buy a second colour line, paying for it with coarser endpoints and eight positions each. Mode 3 buys the same second line and pays differently — full-precision endpoints, four positions. Mode 5 keeps one subset but splits its indices into two independent sets, so one channel can be described entirely separately from the other three.
None of them wins everywhere, which is the point. The encoder does not guess which block is which — it encodes every eligible way and measures.
That rule is only meaningful because every mode reports the same quantity: the sum of squared differences over the same sixteen pixels and four channels. Not "error in its own units". Making that true by construction is what lets a new mode be added without touching mode selection, and it is the reason the fourth mode was a day's work rather than a rewrite.
How a subset is solved
Every mode reduces to the same sub-problem: given a set of pixels, find the best line and place everyone on it.
- Find the axis. Build the covariance matrix of the subset's pixels and run power iteration — eight rounds of multiply-and-normalise — to recover the principal axis. That is the direction the colours actually vary along.
- Seed the endpoints. Project every pixel onto that axis; the two extremes become the initial endpoints.
- Refine, three times. Quantize the endpoints to the mode's bit depth, assign each pixel the index that lands it closest, then least-squares refit the endpoints against those fixed indices. Repeat.
The refit step is what makes it more than a PCA estimate: once the indices are pinned, recovering the endpoints that minimise total error is a small least-squares solve, and the improved endpoints then let some pixels pick better indices. Three rounds is where it stops paying.
The two-subset modes have an extra problem — which of the 64 partition shapes to use. Fully evaluating all 64 would mean 128 subset solves per block, per mode. Instead each shape gets a cheap prescore and only the best few are solved properly.
The second encoder scored a shape by summing the per-subset bounding-box extents, and left an open question behind: nobody had checked whether the shortlist ever excluded the shape that would have won. It does, on 31% of blocks. The reason turned out to be a hole in the score itself — bounding-box extent is blind to how many pixels are in a subset, so a partition that isolates a single pixel scores beautifully, because a one-pixel box has no extent at all.
Scoring total within-subset scatter instead — the sum of n × variance, which is the k-means objective — costs the same and measures what the solve is actually about to go and do:
| shortlist | misses the best shape | excess error against exhaustive |
|---|---|---|
| bounding box, best 8 | 30.5% | 15.93% |
| bounding box, best 16 | 20.6% | 7.43% |
| scatter, best 8 | 25.5% | 6.56% |
| scatter, best 16 | 16.8% | 3.32% |
The second column is the one that matters, and it is why both are there. A miss that costs a hair is not the same failure as missing the only good shape, so the miss rate alone would have made the old score look better than it was. Fixing the score beat doubling the search: scatter at 8 shapes leaves less error than bounding-box at 16, for half the work.
What the measurements changed
Three things went into the third encoder. Two of them I would have got wrong by reasoning.
Mode 3 was the obvious one — mode 1's opposite trade, and the natural next mode. It also does something mode 6 cannot: it makes flat blocks lossless. A tile of one solid colour went from 54.15 dB to 99.99. Mode 6 could not manage a flat block because its p-bit is shared across a whole endpoint, RGB and alpha together, and an alpha of 255 wants the opposite bit from the colour that tile happened to be. That had been sitting in the encoder since the first afternoon, invisible, because nothing measured a flat tile.
Mode 5 gives one channel its own endpoints and its own indices. The engine's normal maps carry a parallax height field in alpha that has nothing to do with the normal in RGB, and mode 6 has to run a single four-dimensional line through both — so every height step drags the normal off the line, and every normal change drags the height.
Mode 5's channel rotation is the one I nearly did not implement, on an argument that sounded solid: a rotation swaps alpha with one of R, G or B before encoding, so it only helps content whose odd channel out is a colour — and this project's odd channel is the height, which mode 5 already handles. I wrote that down as a reason to skip it.
It is worth +0.89 dB, and the biggest beneficiary is precisely the content the argument was about. One scanned normal map went from 35.77 dB to 39.94, and the share of its blocks choosing mode 5 went from 2% to 93%. The awkward channel in a normal map is usually blue: z is derived from x and y, so it behaves nothing like either. Decoupling alpha was never the point. Decoupling whichever channel refuses to lie on the line is the point, and I had mistaken one instance of the idea for the idea.
The alpha gate, revisited
The second encoder's write-up made a claim I want to correct rather than quietly drop.
Mode 1 is RGB-only — it forces alpha opaque — so it is only ever a candidate for fully-opaque blocks, and the encoder gates it on exactly that. Because the normal maps have a height field in alpha, mode 1 is never a candidate there, and I described this as normal maps being protected by construction: a correctness requirement met by the format's own constraint rather than by a rule someone has to remember.
That is true, and it was also hiding something. Those blocks were not just protected, they were stranded — they had exactly one legal mode, so the entire multi-mode design did nothing for them. Every normal map in the project sat at mode 6 by default, not by merit. The gate was doing real work as a safety property while quietly marking out the largest untouched patch of quality in the encoder, and I read only the first half of that.
Mode 5 is what un-strands them. It carries alpha, so it is eligible everywhere, and on those same normal maps it now takes 79–93% of the blocks.
What changed downstream: nothing
Three encoders, no format change and no runtime change. BC7's mode is encoded per block, in the block itself, and the GPU's fixed-function decoder handles every mode. The DDS writer, the loader, the texture upload path, and the shaders are byte-for-byte identical across all three versions.
That asymmetry is the quiet luxury of a hardware format. The decoder is fixed, universal, and free; the encoder can be made arbitrarily smarter, at any time, and nothing downstream needs to know. It is the opposite of the usual situation, where improving a format means shipping a reader that understands both versions.
The partition tables, anchor indices, and interpolation weights are BC7 hardware constants, taken from Microsoft's DirectXTex. They have to be exact — the GPU indexes the same tables when it decodes, so a single wrong value produces garbage that no amount of encoder logic can fix.
How it's proved
This is the part that actually changed.
The second encoder's evidence was a PSNR number from a tile, measured once, by hand, in a session that no longer exists. Nothing re-checked it. I said so at the time and left it as the open item, which is a way of being honest about a gap without closing it.
The encoder already knew something useful: every mode returns the error it believes its block carries, because that number is what mode selection compares. So if that number is wrong, the encoder is not choosing the better mode — it is choosing at random, and every quality claim downstream is void. That makes it the one thing most worth testing, and it is testable, because the estimate is computed from the endpoints the solver fitted and never from the packed bytes.
So the harness decodes the packed bytes with a separate decoder, written from the format's field tables rather than from the encoder's bit writers, and requires the two numbers to be exactly equal. Not within a tolerance — exactly. Every term is a small integer, so the sums are exact whichever order they are added in, and a tolerance would only have been somewhere for a bug to hide. Across the corpus that is 44,032 blocks and zero disagreements.
Two more checks ride along. The block fan-out must not change a single byte, so every image is encoded once serially and once in parallel and the buffers compared. And each image's PSNR is held against a recorded baseline, so a refactor that quietly costs a decibel fails now rather than being noticed a year later.
The harness has an inverted mode: -SelfTest corrupts the encoded bytes on purpose and the run has to come back FAIL. A regression test that cannot fail is not evidence of anything.
What none of it can catch is a misreading of the format shared by both sides. The encoder and the decoder are independent of each other, but not of the spec — write a field in the wrong order, read it back the same wrong way, and they agree with each other while the GPU does not. Modes 1 and 6 had already retired that risk by rendering correctly since June. Modes 3 and 5 had not, so both were cleared the old way: rebake every texture, walk the level, look. The trick that made that check worth anything was picking the surface by its measured mode mix rather than by assumption — for mode 5 the obvious candidate, the brick wall, turned out to be 1% mode 5 while the floor was 21%, so the obvious check would have proved nothing.
What it cost
Encoding is about eight times the work of mode 6 alone, and roughly three times the two-mode encoder. Four modes, mode 5 solved four times over for its rotations, and eight partition trials each needing two subset solves and three refinement rounds — for every block of every mip level.
Block encoding is now parallel, which is what makes that affordable: blocks are independent, so the encoder fans out over block rows for about a 14× speedup. A full bake of the installed texture set — 639 images at every resolution from 1K to 4K — takes 33.7 minutes, against 23.6 for the previous encoder. Deliberately not built on this engine's worker-thread manager: that registry exists for long-lived workers with cadences, watchdogs and a supervisor, and an offline batch wants none of it. Plain threads over an atomic counter.
Writing your own encoder means owning its correctness. There is no library between this code and the hardware. The harness now catches most of what that implies, and the bit packer asserts it can neither leave the 128-bit block nor stop short of it, so a mis-specified field width is a loud failure rather than a smashed stack — but the final word still belongs to a GPU and a person looking at it.
Half the modes are still missing, and the case for each is now weaker than it was, because the four that exist overlap heavily.
What it bought
- A quarter of the VRAM and bandwidth versus RGBA8, which is the whole reason for the exercise, and 384 MB of uncompressed DDS became 96 MB on disk.
- Mean quality across the corpus went from 42.80 dB to 51.39 — the middle, two-mode encoder sat at 44.41, so the third version is worth +7.0 dB on the one it replaced.
- Material edges stopped smearing, which was the second encoder's whole purpose and remains fixed.
- Normal maps stopped being stranded. The blocks that had exactly one legal mode now choose between three, and gained 4 dB doing it.
- A quality claim that can be re-checked. The number above is a command, not a memory.
- Nothing downstream moved. The runtime, the on-disk format, and the shaders were untouched across all three versions, so none of it carried compatibility risk at all.
What the numbers kept getting wrong
Three of the decisions above were made twice, because the first measurement lied. All three failures were mine rather than the code's, and they are the most portable thing here.
Averaging PSNR by pooling squared error is a report on your worst image. An incompressible noise tile sits about a thousand times higher in mean squared error than a smooth one, so pooling hands it the microphone. Pooled, one knob measured +0.01 dB and I turned it off as not worth its cost. Per image, turning it off costs 1.35 dB on the brick tile and a quarter of a decibel on real scanned stone — the exact content this dungeon is built out of. The audit now reports a worst image column alongside the mean, because a knob that helps on average and hurts one kind of content is the failure mode worth catching.
Search breadth and mode coverage buy overlapping things. The number of partition shapes evaluated went 8 → 16 → 8. It was raised when three modes existed and the step was worth +0.22 dB; adding two more modes made the same step worth +0.07, because a block the shortlist mis-partitions usually has another mode that suits it, and the extra modes get there first. A knob tuned before the system around it changed is a knob that needs re-measuring, not ratcheting.
A mode's worth is not a property of the mode. Mode 3 measures +0.31 dB with mode 5's rotations enabled and +3.40 dB without them. They are competing to solve the same problem from opposite directions. Quoting either number alone would be true and useless.
The mode I measured and did not build
The previous version of this page ended by saying the honest next step was to measure the headroom before writing another encoder. So here is that measurement, and the useful part is that the cheap version of it said yes.
Mode 7 was the strongest candidate left: the only BC7 mode with two subsets and alpha. That matters because modes 1 and 3 force alpha opaque, so a block whose alpha varies is stuck on a single colour line no matter how good the encoder gets. Mode 7 is the only thing in the format that changes that.
The bound. How much error currently sits in blocks mode 7 could address — non-opaque, and structured enough that a partition would explain their spread? If every one of those blocks fell to zero error:
| mean corpus PSNR | |
|---|---|
| now | 51.39 dB |
| every addressable block at zero error | 54.44 dB (+3.05) |
Around +4 dB of ceiling on each of the real scanned normal maps. Read alone, that says build it.
The mode. Then I ran mode 7's actual search — the same solver the other two-subset modes use, at mode 7's precision — and let it compete for the win on every block, which is exactly what shipping it would do:
| mean corpus PSNR | |
|---|---|
| with mode 7 competing | 51.47 dB (+0.08) |
It wins 2.3% of blocks and reaches 3% of its own ceiling. Best real texture: +0.29 dB.
The gap is not a measurement error, it is the format. Mode 7 buys its second subset and its alpha by being the coarsest two-subset mode there is — five colour bits plus a p-bit, spread across four channels instead of three, and four index positions. Look at its bar in the diagram above: the same endpoint budget as mode 3, asked to cover an extra channel. On exactly the blocks it was meant to rescue, mode 5's decoupled channel — 7-bit RGB, 8-bit alpha, two independent index sets — is still worth more than a second region at that precision. Mode 7 addresses the right blocks and then cannot afford them.
So it was declined: +0.09 dB against a packer, a decoder, a GPU verification pass and permanent maintenance. For scale, mode 5 was +2.33 dB and the prescore fix was about twenty lines.
The transferable part is the factor of forty. A ceiling computed from block statistics counts the error a mode could address; a real mode also has to pay for the structure it adds, in precision taken from the same 128 bits. Bounds rule things out well and rule things in badly. The cheap analysis was worth running — it would have killed a weak idea instantly — but it could not license building one. That needed the solver, which is perhaps a day's work and the reason there is no fifth encoder.
The measurement stays in the tree rather than being reverted, because the answer is a function of the content. If this project ever grows a lot of two-material alpha-cutout art, mode 7's case changes, and re-asking is one command.
Still open
Every item that used to sit here has been dealt with — the reference decode built, the modes added, the prescore validated and then replaced, the baker parallelised, and the last plausible mode measured and turned down. What follows is shorter still, and mostly says stop.
- The remaining three modes are narrower than the one that lost. Modes 0 and 2 buy a third subset, and they pay for it in exactly the currency mode 7 could not afford — 4- and 5-bit endpoints — on a rarer kind of block. Mode 4 is close enough to mode 5 to be a refinement rather than a new capability. If anyone revisits this, the move is to extend the headroom tool with a three-subset estimate, not to reason about it. The tooling is the durable part.
- The prescore is still a clustering score for a line-fitting problem. Scatter measures spread about a subset's mean; the solver fits a line, so the error it actually leaves is the residual after the principal component. Subtracting that would score the real thing, but an eigenvector per shape per block is far too slow. Whether a cheap approximation exists is an open question rather than a plan.
- GPU-side verification is the one structural hole, and it has now been deferred twice on the same reasoning: the manual check takes five minutes and has been sufficient both times. That argument weakens with every mode added, because new modes are getting rarer and harder to find on screen — the last one needed a measurement just to choose which wall to look at. It is the thing I would build if a mode ever does justify itself.
- Nothing runs the harness but me. It is a command that exits non-zero, which is most of the way to a gate, and it is not a gate.