Computational Imaging and Microscopy

This is an excellent talk, and takes one of the most complex subjects and breaks it down simply from the beginning.  

     14:38 in to the video. 

  It took be the better part of 25 years to learn the secret of zooming in to a license plate from an impossibly Zoomed image.  Something that when shown in Sci-Fi TV shows in the 70's and 80's - 90's...  I just assumed was Bullshit.  

 I eventually learned the secret from one of the top digital imaging experts that's I've known 25 years, and well after his retirement.  He implemented it on super top secret hardware for satellite imaging, when I was still in grade school.  It used the fact that every pixel in your image represents a sine cardinal from the whole image.  Also known as Sinc function it is the continuous inverse Fourier transform of a rectangular pulse and can be thought of a Gaussian modulated sine wave, although I am not really sure if these are the exact equivalent though I see it implemented in RF applications like this. 

In his case the Lens had to be extremely well understood, and in the end it generated convolution filters that were able to be ran quickly and efficiently. 

What is interesting is to see this generalized in to a generic computational imaging problem.  It may actually yield better results with more information, but most likely will be much more computation. 

Computational imaging involves the joint design of imaging system hardware and software, optimizing across the entire pipeline from acquisition to reconstruction. This talk will describe new methods for computational microscopy with coded illumination, based on a simple and inexpensive hardware modification of a commercial microscope. Traditionally, one must trade field-of-view for resolution; with our methods we can have both, resulting in Gigapixel-scale images with resolution beyond the diffraction limit of the system. Our reconstruction algorithms are based on large-scale nonlinear non-convex optimization procedures for phase retrieval. Laura Waller leads the Computational Imaging Lab, which develops new methods for optical imaging, with optics and computational algorithms designed jointly. She holds the Ted Van Duzer Endowed Professorship and is a Senior Fellow at the Berkeley Institute of Data Science (BIDS), with affiliations in Bioengineering and Applied Sciences & Technology. Laura was a Postdoctoral Researcher and Lecturer of Physics at Princeton University from 2010-2012 and received BS, MEng and PhD degrees from MIT in 2014, 2015 and 2010, respectively. She is a Moore Foundation Data-Driven Investigator, Bakar fellow, Distinguished Graduate Student Mentoring awardee, NSF CAREER awardee and Packard Fellow.

new A.I. camera that can spot you from 28 miles away

the lidar-based system can cut through city smog to resolve human-sized features at vast distances.

https://www.technologyreview.com/s/613457/a-new-camera-can-photograph-you-from-45-kilometers-away/?_ga=2.245828740.1840581567.1557426276-1747270102.1554128552

A new camera can photograph you from 45 kilometers away
Developed in China, the lidar-based system can cut through city smog to resolve human-sized features at vast distances.
by Emerging Technology from the arXiv
May 3, 2019

Long-distance photography on Earth is a tricky challenge. Capturing enough light from a subject at great distances is not easy. And even then, the atmosphere introduces distortions that can ruin the image; so does pollution, which is a particular problem in cities. That makes it hard to get any kind of image beyond a distance of a few kilometers or so (assuming the camera is mounted high enough off the ground to cope with Earth’s curvature).

But in recent years, researchers have begun to exploit sensitive photodetectors to do much better. These detectors are so sensitive they can pick up single photons and use them to piece together images of subjects up to 10 kilometers (six miles) away.

Nevertheless, physicists would love to improve even more. And today, Zheng-Ping Li and colleagues from the University of Science and Technology of China in Shanghai show how to photograph subjects up to 45 km (28 miles) away in a smog-plagued urban environment. Their technique uses single-photon detectors combined with a unique computational imaging algorithm that achieves super-high-resolution images by knitting together the sparsest of data points.

The new technique is relatively straightforward in principle. It is based on laser ranging and detection, or lidar—illuminating the subject with laser light and then creating an image from reflected light.

The big advantage of this kind of active imaging is that the photons reflected from the subject return to the detector within a specific time window that depends on the distance. So any photons that arrive outside this window can be ignored.

This “gating” dramatically reduces the noise created by unwanted photons from elsewhere in the environment. And it allows lidar systems to be highly sensitive and distance specific.

To make the new system even better in urban environments, Zheng-Ping and co use an infrared laser with a wavelength of 1550 nanometers, a repetition rate of 100 kilohertz,  and a modest power of 120 milliwatts. This wavelength makes the system eye-safe and allows the team to filter out solar photons that would otherwise overwhelm the detector.

The researchers send and receive these photons through the same optical apparatus—an ordinary astronomical telescope with an aperture of 280 mm. The reflected photons are then detected by a commercial single-photon detector. To create an image, the researchers scan the field of view using a piezo-controlled mirror that can tilt up, down, and side to side.

In this way, they can create two-dimensional images. But by changing the gating timings, they can pick up photons reflected from different distances to build a 3D image.

The final advance the team has made is to develop an algorithm that knits an image together using the single-photon data. This kind of computational imaging has advanced in leaps and bounds in recent years, allowing researchers to create images from relatively small sets of data.

The results speak for themselves. The team set up the new camera on the 20th floor of a building on Chongming Island in Shanghai and pointed it at the Pudong Civil Aviation Building across the river, some 45 km away.

single pixel resolution imaging

Conventional images taken through the telescope show nothing other than noise. But the new technique produces images with a spatial resolution of about 60 cm, which resolves building windows. “This result demonstrates the superior capability of the near-infrared single-photon LiDAR system to resolve targets through smog,” say the team.

That’s also significantly better than the conventional diffraction limit of 1 meter at 45 km, and certainly better than other recently developed algorithms. The image here shows the potential of the technique in images taken in daylight from a distance of 21 km. ”Our results open a new venue for high-resolution, fast, low-power 3D optical imaging over ultralong ranges,” say Zheng-Ping and co.

That’s interesting work that has a wide range of applications. The team mention remote sensing, airborne surveillance, and target recognition and identification. Indeed, the entire device is about the size of a large shoebox and so is relatively portable.

And Zheng-Ping and co say it can be significantly improved. “Our system is feasible for imaging at a few hundreds of kilometers by refining the setup, and thus represents a significant milestone towards rapid, low-power, and high-resolution LiDAR over extra-long ranges,” they say.

So keep smiling—they may be watching.

Ref: arxiv.org/abs/1904.10341 : Single-Photon Computational 3D Imaging at 45 km

This AI Learned To See In The Dark

The paper "Learning to See in the Dark" and its source code is available here:

• http://cchen156.web.engr.illinois.edu/paper/18CVPR_SID.pdf
• https://arxiv.org/abs/1805.01934
• https://github.com/cchen156/Learning-to-See-in-the-Dark

Computational cameras

Computational Cameras: Convergence of Optics and Processing

 Changyin Zhou, Student Member, IEEE, and Shree K. Nayar, Member, IEEE

Abstract—A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. In the last decade, computational imaging has emerged as a vibrant field of research. A wide variety of computational cameras has been demonstrated to encode more useful visual information in the captured images, as compared with conventional cameras. In this paper, we survey computational cameras from two perspectives. First, we present a taxonomy of computational camera designs according to the coding approaches, including object side coding, pupil plane coding, sensor side coding, illumination coding, camera arrays and clusters, and unconventional imaging systems. Second, we use the abstract notion of light field representation as a general tool to describe computational camera designs, where each camera can be formulated as a projection of a high-dimensional light field to a 2-D image sensor. We show how individual optical devices transform light fields and use these transforms to illustrate how different computational camera designs (collections of optical devices) capture and encode useful visual information. Index Terms—Computer vision, imaging, image processing, optics.


http://www1.cs.columbia.edu/CAVE/publications/pdfs/Zhou_TIP11.pdf

Workshop on Light Field Imaging to be held at Stanford on February 12, 2015

Workshop on Light Field Imaging 

February 12, 2015 

MacKenzie Conference Room, Huang Engineering Center

Stanford University

  

We invite you to join us on February 12, 2015 at Stanford University to explore the exciting area of research and product development in Light Field Imaging. 

The Workshop on Light Field Imaging will include a summary of the state-of-the-art research and a glimpse into the future of technologies designed to capture and create light rays in a three dimensional scene. Participants will leave with a better understanding of the concept of a light field as it is used in geometric optics, computer vision, computer graphics and computational photography.  The Workshop will include talks that summarize recent advances in light field cameras and light field displays, as well as applications of these technologies in entertainment, consumer devices, industrial applications and medical imaging. The Workshop will also include an interactive session with experts from industry and academics addressing questions about the killer applications and challenges in product development, new areas for research and graduate training, and the future of light field imaging.   There will also a technology demo session that will include presentations by research labs and startup companies.   

You can now register for the Workshop on Light Field Imaging that will be held at Stanford on February 12, 2014.  Registration is limited to 200 people, and we are rapidly approaching this limit, so you should register now if you intend to participate.

Visit our website to get updates on the program.  A list of companies that will participating in the Interactive Demo Session will be published in the coming weeks.

If you would like to receive future announcements about this event, be sure to subscribe to our mailing list at https://mailman.stanford.edu/mailman/listinfo/scien_events

lensfree holographic on-chip microscopy

Actually this shouldn't be that hard to do.  It is computational photography at it's finest.

It should be able to completely put to shame normal optical microscopes.
It is volumetric and 3D viewable and could even go multi-spectral. 

There's no information on the specifics of the optics, but the sample must go directly on the imaging chip or very close to it.

So cleaning and reuse are my only questions.

Lens-free microscope can detect cancer at the cellular level

UCLA researchers develop device that can do the work of pathology lab microscopes

http://newsroom.ucla.edu/releases/lens-free-microscope-can-detect-cancer-at-the-cellular-level


 The latest invention is the first lens-free microscope that can be used for high-throughput 3-D tissue imaging — an important need in the study of disease.

“This is a milestone in the work we’ve been doing,” said Ozcan, who also is the associate director of UCLA’s California NanoSystems Institute. “This is the first time tissue samples have been imaged in 3D using a lens-free on-chip microscope.”

The device works by using a laser or light-emitting-diode to illuminate a tissue or blood sample that has been placed on a slide and inserted into the device. A sensor array on a microchip — the same type of chip that is used in digital cameras, including cellphone cameras — captures and records the pattern of shadows created by the sample.

The device processes these patterns as a series of holograms, forming 3-D images of the specimen and giving medical personnel a virtual depth-of-field view. An algorithm color codes the reconstructed images, making the contrasts in the samples more apparent than they would be in the holograms and making any abnormalities easier to detect.

Wide-field computational imaging of pathology slides using lens-free on-chip microscopy

Alon Greenbaum, Yibo Zhang,  Alborz Feizi, Ping-Luen Chung, Wei Luo, Shivani R. Kandukuri and Aydogan Ozcan

http://stm.sciencemag.org/content/6/267/267ra175

Optical examination of microscale features in pathology slides is one of the gold standards to diagnose disease. However, the use of conventional light microscopes is partially limited owing to their relatively high cost, bulkiness of lens-based optics, small field of view (FOV), and requirements for lateral scanning and three-dimensional (3D) focus adjustment. We illustrate the performance of a computational lens-free, holographic on-chip microscope that uses the transport-of-intensity equation, multi-height iterative phase retrieval, and rotational field transformations to perform wide-FOV imaging of pathology samples with comparable image quality to a traditional transmission lens-based microscope. The holographically reconstructed image can be digitally focused at any depth within the object FOV (after image capture) without the need for mechanical focus adjustment and is also digitally corrected for artifacts arising from uncontrolled tilting and height variations between the sample and sensor planes. Using this lens-free on-chip microscope, we successfully imaged invasive carcinoma cells within human breast sections, Papanicolaou smears revealing a high-grade squamous intraepithelial lesion, and sickle cell anemia blood smears over a FOV of 20.5 mm². The resulting wide-field lens-free images had sufficient image resolution and contrast for clinical evaluation, as demonstrated by a pathologist’s blinded diagnosis of breast cancer tissue samples, achieving an overall accuracy of ~99%. By providing high-resolution images of large-area pathology samples with 3D digital focus adjustment, lens-free on-chip microscopy can be useful in resource-limited and point-of-care settings.

Toward giga-pixel nanoscopy on a chip: a computational wide-field look at the nano-scale without the use of lenses

Euan McLeod,   Wei Luo,   Onur Mudanyali,  Alon Greenbaum and  Aydogan Ozcan  

http://pubs.rsc.org/en/content/articlelanding/2013/lc/c3lc50222h#!divAbstract


The development of lensfree on-chip microscopy in the past decade has opened up various new possibilities for biomedical imaging across ultra-large fields of view using compact, portable, and cost-effective devices. However, until recently, its ability to resolve fine features and detect ultra-small particles has not rivalled the capabilities of the more expensive and bulky laboratory-grade optical microscopes. In this Frontier Review, we highlight the developments over the last two years that have enabled computational lensfree holographic on-chip microscopy to compete with and, in some cases, surpass conventional bright-field microscopy in its ability to image nano-scale objects across large fields of view, yielding giga-pixel phase and amplitude images. Lensfree microscopy has now achieved a numerical aperture as high as 0.92, with a spatial resolution as small as 225 nm across a large field of view e.g., >20 mm². Furthermore, the combination of lensfree microscopy with self-assembled nanolenses, forming nano-catenoid minimal surfaces around individual nanoparticles has boosted the image contrast to levels high enough to permit bright-field imaging of individual particles smaller than 100 nm. These capabilities support a number of new applications, including, for example, the detection and sizing of individual virus particles using field-portable computational on-chip microscopes.

Paolo Favaro: Portable Light Field Imaging: Extended Depth of Field, Ali...

From ICCP11 Hosted by Carnegie Mellon University, Robotics Institute

April 8, 2011



Abstract:



Portable light field cameras have demonstrated capabilities beyond conventional cameras. In a single snapshot, they enable digital image refocusing, i.e., the ability to change the camera focus after taking the snapshot, and 3D reconstruction. We show that they also achieve a larger depth of field while maintaining the ability to reconstruct detail at high resolution. More interestingly, we show that their depth of field is essentially inverted compared to regular cameras. Crucial to the success of the light field camera is the way it samples the light field, trading off spatial vs. angular resolution, and how aliasing affects the light field. We present a novel algorithm that estimates a full resolution sharp image and a full resolution depth map from a single input light field image. The algorithm is formulated in a variational framework and it is based on novel image priors designed for light field images. We demonstrate the algorithm on synthetic and real images captured with our own light field camera, and show that it can outperform other computational camera systems.



Bio:



Paolo Favaro received the D.Ing. degree from Universita di Padova, Italy in 1999, and the M.Sc. and Ph.D. degree in electrical engineering from Washington University in St. Louis in 2002 and 2003 respectively. He was a postdoctoral researcher in the computer science department of the University of California, Los Angeles and subsequently in Cambridge University, UK. Dr. Favaro is now lecturer (assistant professor) in Heriot-Watt University and Honorary Fellow at the University of Edinburgh, UK. His research interests are in computer vision, computational photography, machine learning, signal and image processing, estimation theory, inverse problems and variational techniques. He is also a member of the IEEE Society.

The Ultimate Spy Camera Will Be Lensless And Thinner Than A Pencil Point

From:
http://thecreatorsproject.vice.com/blog/the-ultimate-spy-camera-will-be-lensless-and-thinner-than-a-pencil-point

As if today in tiny technology couldnt get any better, research scientists at Rambus have released details on the technology that could potentially revolutionize camera-imaging: utilizing a spiral-shaped sensor that maps light, and then lets a computer reconstruct the rest, these clear, glass-based cameras no longer need lenses—and can be reduced to sizes thinner than a pencil point. 

Images via

While the technology still has nowhere near the sensor-capabilities of even today's most low-end cell phone camera, it's the mechanism behind these tiny technologies that have us looking towards the future.

Instead of "seeing" the image in the way that a lens allows reflected light to hit a digital camera's CMOS ("complementary metal–oxide–semiconductor") imager, these are sensors that effectively filter available light into a spiral, which a computer is able to "reconstruct" to the likeness of the original image. Talk about a nightmare for image theory... 

Below, an image of what the lens-free cameras "see," followed by its computer reconstruction, followed by the original image. Not bad, eh?

While there's still no word on when these gadgets will become readily available to consumer technology, we've got a funny feeling when they catch on, we'll all be seeing things a little differently. 

Check out video footage of the lens-free camera technology making its debut at the Mobile World Congress. h/t Yahoo! News & MIT

Bell Labs creates a lensless camera that's always in focus

http://www.theverge.com/2013/6/3/4391106/bell-labs-lensless-camera-technology-compressive-sensing

One day, cameras may be able to capture less data but produce images that look just as good as a traditional photo. Bell Labs is the latest to attempt such a feat, and it's doing so while eschewing another major camera standby, the lens. The laboratory has developed a single-pixel camera that only uses a series of transparent openings to capture its image, without any glass to direct the light. The system uses a technology called "compressive sensing," which is still in its early stages of study. The idea is that instead of a camera capturing a full image and then whittling the data down into a small, compressed file like a JPG, a camera could instead capture almost exactly what it needs, making capture times much quicker.

NO LENS, ALWAYS IN FOCUS

This isn't a reality just yet, however. Compressive sensing cameras build their final image by comparing the differences that come in through each aperture, and right now that takes too much time for them to shoot anything other than a still life. But by removing the lens, Bell Labs adds another impressive feature to its camera: its shots are always in focus. One always-in-focus camera, the Lytro, is already on the market, but Bell Labs sees its new tech as a practical way to shrink the size and cost of future cameras.

Bell Labs' device is built with "low cost, commercially available components," which primarily amount to a semi-transparent LCD panel, a one-megapixel imaging sensor, and a computer to connect it all to. The LCD panel was placed in front of the sensor, and light came in through white "openings" in the panel. The camera measured the data separately for red, green, and blue light, and used a computer to stitch together the final image. While the images don't demonstrate the finest image quality, they emphasize what compressive sensing is capable of. The books were captured using only a quarter of the camera's total imaging capabilities, and the soccer ball was captured using even less, just one-eighth.