Project Vermont: Improving camera performance through curved image sensors

By Richard Stoakley (opens in new tab), Microsoft Research

It’s hard to remember when our world wasn’t full of cameras. They’re in our phones and tablets, posted on street corners and in our buildings and cars. We rely on them not just to capture moments and memories (opens in new tab), but for security, navigation (opens in new tab), 3D spatial mapping (opens in new tab) and processing medical data (opens in new tab).

While significant advances have been made to camera sensors and the algorithms that process images (opens in new tab), optical systems have not seen the same radical improvements. Our small team of Microsoft researchers, including Brian Guenter (opens in new tab) and Neel Joshi (opens in new tab), set out to solve one of the most well-known optical issues in conventional photography: field curvature.

Microsoft Research Podcast

AI Frontiers: Models and Systems with Ece Kamar

Ece Kamar explores short-term mitigation techniques to make these models viable components of the AI systems that give them purpose and shares the long-term research questions that will help maximize their value. 

Listen now

Opens in a new tab

Field curvature exists because lenses don’t want to focus on a plane. As illustrated to the right, the lens delivers sharpest focus along a curved path, rather than on a flat surface. The result: objects appear out of focus across the image. By curving the focal surface, we eliminate the need for the lens to address field curvature and can use the resulting lens design freedom to improve other aspects of the system.

How can we curve the focal surface? There are multiple approaches to this and some work well in specific circumstances (for example, curved arrays of flat sensors can be useful in astronomical telescopes), but most known techniques fail in conventional, human-scale cameras. An obvious approach would be to bend the sensor itself, but since silicon sensors are monocrystalline, they are prone to shattering rather than bending.

Enter Project Vermont

Several companies have attempted to bend silicon, but attaining useful amounts of curvature has remained elusive. To address these limitations, Brian, Neel and I joined together on Project Vermont (opens in new tab)— a project designed to push the limits of silicon sensor curvature and deliver substantial camera performance improvements.

We designed numerous optical models and built several early prototypes to validate our hypotheses regarding these improvements. Those early successes helped us identify key opportunities to attain higher degrees of curvature. We then collaborated with scientists at HRL Laboratories to capitalize on these opportunities. In our new paper, Highly curved image sensors: a practical approach for improved optical performance  (opens in new tab), published today in Optics Express, the team outlines the various techniques we developed to attain useful amounts of curvature and shows empirical prototype results.

The Future

What does Project Vermont mean for the future of cameras? Curving the image surface can dramatically improve performance along many axes — resolution, light-gathering, and illumination uniformity — while also reducing system size, cost, and complexity. While the amount of improvement is dependent on many camera system variables, including focal length, sensor size, and lens complexity, Project Vermont showed substantial progress over previous approaches.

In our most recent research prototype, we significantly outperformed a premium 35mm digital single- lens reflex camera (DLSR) in both sharpness and uniformity of illumination, using a much smaller cellphone-sized sensor. At present, our prototype camera system is sharper (in terms of line pairs per picture height) than any known commercially available consumer camera.

Applying this technique to cellphones will require additional research and engineering. However, for more conventional camera systems, we see a near-term opportunity for significant improvement and are working with third parties to advance the state of the art. We hope this work inspires further investigation into new optical systems and an exciting new world of cameras.

For more information, please visit Project Vermont. (opens in new tab)

Related:

May 30, 2017, The Optical Society: Breakthrough Curved Sensor Could Dramatically Improve Image Quality Captured with Digital Cameras (opens in new tab)

Editor’s Note: Richard is a principal program manager with Microsoft Research and the project manager for Project Vermont.