Linear Light Upscaling Is Correct, and It Cost Me 18 Shadow Levels

Every resampling article opens with the same demo: downscale a black-and-white checkerboard in sRGB, get a muddy grey that should have been mid-grey, then do it in linear light and get the right answer. The demo is correct. I have reproduced it.

Then I took that conclusion — "always resample in linear light" — and applied it to a real 512×512 photograph, and to a shadow gradient, and to an unsharp mask. It is not that simple, and the reasons are specific enough to measure.

Seven experiments below. Two of them contradict the advice I started with. All of them run on numpy 1.26.4 and Pillow 12.2.0, and every number comes from the snippet at the end.

Why the theory says linear light wins

sRGB is a non-linear encoding. A stored value of 128 is not 50% of the light — it is about 21%. The encoding exists because human contrast perception is roughly logarithmic, so spending code values where the eye is sensitive wastes fewer bits. Good decision, for storage.

Most image operations, though, are physical operations on light. Averaging two pixels means averaging the light they emitted. Average the encoded values instead and you are averaging logarithms, and the average of logarithms is not the logarithm of the average. The error is systematic, not random, so it does not average away at higher resolutions.

That argument is airtight, as far as it goes. It is an argument about averaging. An upscaler does more than average, and an 8-bit file has to store the result somewhere. Those two facts are where the tidy story breaks.

Method

One 512×512 crop from a 1080×1620 photo, converted to 8-bit greyscale so the colour-space question is not confounded by chroma. Two pipelines:

• sRGB pipeline: resize the 8-bit buffer directly. This is what every naive Image.resize() call does.
• Linear pipeline: convert to linear float, resize with the same kernel, convert back to sRGB, store 8-bit.

Both pipelines use the same filter, because the filter is not the variable being tested. Between the downscale and the upscale each pipeline stores its intermediate as 8-bit, which is what actually happens when a tool writes a temp file or an application holds a texture.

High-frequency retention is mean |Laplacian| of the result divided by mean |Laplacian| of the ground truth, both computed in sRGB code values so the two pipelines are compared in the same units the eye sees.

Experiment 1: linear wins on detail, loses on PSNR

Downscale 4× to 128×128, then back up to 512×512.

Two things point in opposite directions, and both are real.

Linear light keeps more high-frequency detail: +1.12 percentage points at bilinear, +3.52 at lanczos, a 12% relative improvement. That is the theoretical prediction confirmed — resampling in the correct space preserves structure better.

Linear light also has worse PSNR, by 0.98 dB at bilinear and 1.19 dB at lanczos. And worse worst-case error: 172 code values against 151.

The resolution is that PSNR is a mean squared error over the whole image, and the linear pipeline spends its error where the eye is most sensitive: the shadows. MSE does not know that a 6-code-value error at value 12 is far more visible than a 20-code-value error at value 200. It weights them by the same squared distance.

So which pipeline is "better" depends on whether you believe your metric or your optics. That is not a satisfying answer, so I went looking for where the error actually lives.

Experiment 2: the metric and the pipeline disagree by brightness

Same round trip, lanczos, absolute error bucketed by the brightness of the source pixel.

The crossover sits at value 128. Below it, sRGB resampling is more accurate in code values; above it, linear is. The 0–32 bucket is the extreme case, where linear is 78% worse.

That is the mechanism behind the PSNR result: this photograph is a high-key image, but even so, the dark pixels carry enough absolute error to lose 1.19 dB.

Experiment 3: quantising a linear intermediate is expensive

The linear pipeline converts back to sRGB before storing its intermediate — so it stores 8-bit sRGB, and no precision is lost beyond that of the sRGB path.

But some libraries do not. They keep the buffer linear and store it as 8-bit, because the format says "8-bit" and the bit depth metadata says nothing about encoding. To find out what that costs, I took a smooth dark gradient — sRGB 0 to 40 across 256 pixels, which is ordinary shadow detail — and upscaled it 4× bicubic, once with a linear buffer rounded to 8-bit before the second step, once keeping it as float.

A 4×-uplifted gradient with 41 distinct values needs 41 distinct codes. The 8-bit linear intermediate delivers 23. 18 of 41 levels are gone, and the gaps between survivors reach 4 code values.

The panels are contrast-stretched the same way, so the comparison is fair; the counts in the caption are the measurement, the picture is only there to make them visible.

This is the banding you see in a dark sky after a "high quality" resample, and it is not a maths problem — the linear maths is correct. It is a storage problem: linear light puts too few 8-bit codes in the shadows, exactly where banding is most visible. Keeping the buffer in float fixes it completely, with no change to the algorithm.

The practical rule: if you resample in linear light, every intermediate must be float or 16-bit. A linear 8-bit intermediate is the one configuration worse than not converting at all.

Experiment 4: convert on one side, save on the other

Real pipelines mix this up, usually because one library converts on load and another on save. Four combinations of {sRGB, linear} for the downscale and the upscale:

I expected the mixed rows to be the worst of both. They are not, and the pattern is more interesting than that.

PSNR decreases monotonically with the number of linear steps: 27.309 for pure sRGB, 26.930 for one linear step, 26.790 for the other one, 26.123 for two. That is consistent with Experiment 1 — each linear step costs about half a decibel.

But detail retention does not follow the same order. Almost the entire HF gain comes from the downscale: sRGB→linear retains 32.62%, against 32.97% for the fully linear pipeline — it captures 99% of the benefit for one of the two steps. Putting the linear step only on the upscale gets 30.95%, less than half the available gain.

So if you can only afford to convert once, convert on the downscale. That is the step where distinct pixels are being averaged into each other and where the averaging actually happens; the upscale is interpolating between values that are already correct, so getting the space right there buys much less.

Experiment 5: the kernel matters more than the space

Same round trip, four kernels, both pipelines:

Switching kernel from nearest to lanczos is worth +3.69 dB. Switching colour space is worth at most ∓1.19 dB. If you have one decision to spend, spend it on the kernel.

The nearest-neighbour row is the control, and it is worth staring at: the delta is exactly 0.000 dB. Nearest-neighbour does no arithmetic — it copies pixels — so there is no averaging step for the colour space to affect. That zero is the check that the whole harness is measuring what it claims to. When I first ran this experiment every kernel reported 3.647 dB, which is physically impossible; the bug was converting an already-linear float buffer with a function that divides by 255. The zero at nearest is what caught it.

Experiment 6: an unsharp mask in linear light rings harder

If linear light is correct for averaging, it should be correct for sharpening too. It is not, or at least not in the direction I expected.

Unsharp mask with amount 1.0, sigma 1.0, applied in each space, measuring overshoot beyond the local 3×3 maximum and undershoot below the local 3×3 minimum of the source, in sRGB code values:

Linear light produces 33% more total halo, and it is almost entirely undershoot — 1.015 against 0.498, roughly double.

The mechanism is the encoding itself. Adding a fixed fraction of the high-pass in linear space, then re-encoding to sRGB, gets amplified in the shadows by the derivative of the transfer curve, which is steepest near black. Overshoot gets compressed in highlights, undershoot gets expanded in shadows, and the result is asymmetric ringing that is more visible than the symmetric ringing you get by sharpening in sRGB.

This is a negative result for the "linear light is always right" hypothesis. Sharpening is not an averaging operation, and it depends on the perceptual distribution of error, which is exactly what the non-linear encoding was designed to provide. Sharpening in sRGB is the pragmatic choice.

Experiment 7: what I would actually do

The measurements point to a decision table rather than a rule:

Two caveats I cannot measure my way out of. This is one photograph; a low-key image would push the shadow penalty much harder and could easily turn a 1.19 dB loss into several. And PSNR is not a perceptual metric — the HF retention numbers and the brightness-bucketed MAE are closer to what the eye does, but neither is a substitute for looking.

Reproduce it

import numpy as np from PIL import Image def srgb_to_linear(u8): c = np.asarray(u8, dtype=np.float64) / 255.0 return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4) def linear_to_srgb(lin): lin = np.clip(lin, 0.0, 1.0) c = np.where(lin <= 0.0031308, lin * 12.92, 1.055 * lin ** (1 / 2.4) - 0.055) return np.clip(np.round(c * 255.0), 0, 255).astype(np.uint8) def resize_lin(f, size, k): return np.asarray(Image.fromarray(f.astype(np.float32), "F").resize(size, k)) def resize_srgb(u8, size, k): return np.asarray(Image.fromarray(u8, "L").resize(size, k)) def psnr(a, b): mse = np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2) return 99.0 if mse == 0 else 10 * np.log10(255.0 ** 2 / mse) def hf(a): a = a.astype(np.float64) lap = 4*a[1:-1,1:-1] - a[:-2,1:-1] - a[2:,1:-1] - a[1:-1,:-2] - a[1:-1,2:] return float(np.mean(np.abs(lap))) img = Image.open("your512x512.jpg").convert("L") gt = np.asarray(img, dtype=np.uint8) # exp 1 + 5, lanczos, 4x down then 4x up K = Image.LANCZOS s = resize_srgb(resize_srgb(gt, (128, 128), K), (512, 512), K).astype(np.uint8) l_dn = resize_lin(srgb_to_linear(gt), (128, 128), K) l = linear_to_srgb(resize_lin(srgb_to_linear(linear_to_srgb(l_dn)), (512, 512), K)) print("psnr srgb ", round(psnr(gt, s), 3)) # 27.309 print("psnr linear", round(psnr(gt, l), 3)) # 26.123 print("hf srgb % ", round(100*hf(s)/hf(gt), 2)) # 29.45 print("hf lin % ", round(100*hf(l)/hf(gt), 2)) # 32.97 

Note the ordering in the linear path: linear_to_srgb before the upscale, so the intermediate is stored as 8-bit sRGB exactly like the other pipeline. Converting twice (once on the way out, once on the way back in) is the bug that produced the impossible 3.647 dB.

The pipeline I use for client work — where the resample happens in linear light with a float intermediate, and the sharpening happens last, in sRGB — is the default at upgolab.com, along with the presets that skip sharpening entirely for sources where the ringing would be worse than the softness.

The measurements behind this post, with the browser tests reproduced: gamma space vs linear light when resampling, measured.
Everything else in the cluster: AI upscaling notes.