Overview
A near-sighted eye takes a sharp image and blurs it. That blurring is a well-defined mathematical operation, and mathematical operations can be run backwards. So instead of putting a corrective lens in front of the eye, you can apply the inverse blur to the image on screen — the eye then applies its own blur to that pre-distorted image, the two cancel, and the viewer sees something sharp without wearing anything.
The idea isn't new; the reason it hasn't shipped is that every prior attempt needed special hardware (a light-field display with a pinhole mask) or was far too slow to run live, and the corrected images came out washed out and low-contrast. Akhilesh Balaji and I built a version that runs on an ordinary screen with an ordinary webcam, in real time, at full contrast: it tracks where your face is, recomputes the correction as you move, and deconvolves whatever is on screen at up to 90 FPS. It scores 83.04% on structural similarity against the reference image, compared with 50.59% for the best prior light-field method, and preserves 100% of image contrast where earlier approaches lost half of it. We presented it at ISEF 2024 and later posted it to arXiv.
Context
About 2.2 billion people have a near or distance visual aberration, and treatment carries an annual global cost above 411 billion USD. Myopia alone is projected to affect half the world's population by 2050. Glasses and contacts solve most of this well — but not all of it. They can't be worn under a VR headset or with 3D anaglyph glasses, contacts contribute to chronic dry eye, and for higher-order aberrations (anything beyond simple defocus) the corrective lens becomes heavy, asymmetric, or simply impractical to manufacture.
The alternative is to move the correction off the face and onto the screen. Prior work took two routes. Huang's light-field displays place a pinhole mask over a pre-distorted image so the diffracted wavefronts interfere into the correct light field — it works, but it requires a physical hardware layer and heavy pre-processing. Holographic approaches capture the corrected light field on film, but only for still scenes; temporally evolving holograms remain an open research problem. Both give up either the hardware-free requirement or the real-time one, and this project is about giving up neither.
Question
Can refractive vision correction be performed entirely in software, on unmodified display hardware, fast enough to be interactive — and without the contrast loss and ringing artefacts that have made deconvolution-based approaches unusable in practice?
Method
The optical model. The eye's defocus is characterised by a point spread function (PSF) — the shape a single point of light smears into on the retina. For simple defocus this is well modelled as a uniform disk of radius . Blurring an image is then convolution with that kernel, so if is the sharp image and the PSF, the eye perceives . Correction means finding a pre-distorted image to put on screen such that the eye's own blur turns it back into the original:
which makes the deconvolution of by . For higher-order aberrations the disk model is replaced by a PSF reconstructed from Zernike polynomial coefficients, which a wavefront aberrometer can measure directly — we verified the pipeline still works by correcting a simulated oblique trefoil aberration.
Why the correction has to be live. The blur radius isn't a fixed property of the eye — it depends on how far away the screen is:
where is pupil diameter, the distance to the display, and the distance to the viewer's plane of focus (from the thin lens equation). Lean in or sit back and the correct PSF changes, so a static pre-distortion is wrong the moment the viewer moves. The prototype closes that loop with the webcam: dlib's face recognition model (99.38% on Labeled Faces in the Wild) locates the face each frame, distance is recovered from the similar-triangles relationship between real face width and its pixel width given the camera's 80° horizontal FOV, and horizontal and vertical angles come from the face midpoint's deviation from frame center. Off-axis viewing also distorts the PSF itself — the circle of confusion must stay circular from the viewer's perspective — so the PSF is passed through a perspective transform built from projection, rotation, and translation matrices, and the left-inverse is applied to recover the on-screen kernel.
Three problems that make naive deconvolution unusable — and what we did about them.
Ringing artefacts. Inverting the PSF in the frequency domain amplifies noise wherever the kernel's spectrum is near zero, producing halos that swamp fine detail. We edge-detect the blurred image to build a mask, apply the deconvolved result only inside the masked (in-focus, detailed) regions, and composite it against the untouched original elsewhere. Text is the worst case because it is nothing but edges, so segments containing text — detected with Tesseract OCR — are deconvolved separately with a higher regularization constant.
Contrast and colour loss. Deconvolving red, green, and blue independently triples the cost and produces visible colour bleeding. Instead we convert to YUV/YCbCr and deconvolve only the luma channel, leaving chroma untouched — the eye reads colour as filling the shapes defined by luminance, so one deconvolution replaces three and the colour stays clean. This is the single change most responsible for the contrast result below.
Speed. The Wiener deconvolution used here is dominated by the FFT and inverse FFT, so it costs in the data size. Slicing the frame into tiles reduces the cost per tile quadratically, and summing over all tiles gives
for division into tiles — a substantial saving that also parallelises cleanly, since tiles are independent. On top of that, the display layer is double-buffered: the frame being deconvolved lives in a back buffer and the two are swapped on completion, which avoids the flicker you get from hiding and re-showing the layer each cycle. Video runs on a separate caching thread that pre-deconvolves roughly three seconds ahead.
Result
Fidelity. Against the reference image, the corrected-and-reconvolved result reached 83.04% SSIM — versus 50.59% for Huang 2014, 47.28% for Pamplona 2012, and 27.89% for Huang 2012. Normalized cross-correlation was 0.742 against 0.423, 0.380, and 0.206 respectively. Absolute error came to a 10.67% pixel-wise difference from the original, and mapping those differences spatially showed them concentrated almost entirely at edges rather than distributed across the image — consistent with residual ringing rather than a global degradation.
Contrast. The prototype preserved 100% of image contrast, where the light-field approaches retained 45% and 15%. This is the practical difference between a demo and something you would actually look at, and it follows directly from confining deconvolution to the luma channel.
Speed. On an i9 with 32 GB of RAM, 30 seconds of video (1800 frames) deconvolved in 20 seconds — about 90 FPS, comfortably real-time. A more modest i7 with 16 GB sustained 27 FPS, still interactive.
Limitations. The approach degrades on small images: at low pixel counts the rasterized PSF is coarse and ringing dominates the little detail there is, so deconvolution would ideally operate on continuous vector content. Tile seams are faintly visible, which padding the tiles would fix. Tiling also breaks down for higher-order aberrations, where the blur displaces content across tile boundaries and reassembly misaligns — segmenting by content contours instead of a fixed grid would avoid this. And human-subject testing so far is limited; the planned protocol is to test on clear-sighted viewers with myopia induced by lenses, so the PSF entered into the display exactly matches the aberration being corrected.
Reflection
The satisfying part of this project was that the hard problems turned out not to be optical. The physics is a single convolution, and inverting it is textbook. What actually stood between that and a working display were three engineering problems the theory doesn't mention — ringing that makes text illegible, colour bleeding from per-channel processing, and an algorithm too slow to keep up with a moving head. Each fix was small and unglamorous on its own: mask the artefacts, pick a better colour space, tile the FFT. Together they moved SSIM by more than thirty points over the prior state of the art.
The colour-space decision is the one I'd generalise. Deconvolving three channels wasn't just three times slower, it was actively worse — the channels drifted apart and the colours bled. Moving to YUV and touching only luma made it faster and better at the same time, because it matched how human vision actually encodes an image rather than how the file happens to store it. Choosing the representation to fit the perceptual system, not the data format, is a move that transfers well beyond this problem.