StereoPi : Stereo Camera Vision Board for the PI Zero Modules.

Front view:

Top view:

Dimensions: 90x40 mm
Camera: 2 x CSI 15 lanes cable
GPIO: 40 classic Raspberry PI GPIO
USB: 2 x USB type A, 1 USB on a pins
Ethernet: RJ45
Storage: Micro SD (for CM3 Lite)
Monitor: HDMI out
Power: 5V DC
Supported Raspberry Pi: Raspberry Pi Compute Module 3, Raspberry Pi CM 3 Lite, Raspberry Pi CM 1
Supported cameras: Raspberry Pi camera OV5647, Raspberry Pi camera Sony IMX 237, HDMI In (single mode)
Firmware update: MicroUSB connector
Power switch: Yes! No more connect-disconnect MicroUSB cable for power reboot!
Status: we have fully tested ready-to-production samples

That’s all that I wanted to cover today. If you have any questions I will be glad to answer.

Project website is http://stereopi.com

https://www.crowdsupply.com/virt2real/stereopi




------

StereoPi - companion computer on Raspberry Pi with stereo video support
https://diydrones.com/profiles/blogs/stereopi-companion-computer-on-raspberry-pi-with-stereo-video-1

 Compute Module 3 for playing with stereo video and OpenCV. It could be interesting for those who study computer vision or make drones and robots (3D FPV).

It works with a stock Raspbian, you only need to put a dtblob.bin file to a boot partition for enabling second camera. It means you can use raspivid, raspistill and other traditional tools for work with pictures and video.

JFYI stereo mode supported in Raspbian from 2014, you can read implementation story on Raspberry forum.

Before diving into the technical details let me show you some real work examples.

1. Capture image:

raspistill -3d sbs -w 1280 -h 480 -o 1.jpg

and you get this:

You can download original file here.

2. Capture video:

raspivid -3d sbs -w 1280 -h 480 -o 1.h264

and you get this:

You can download original captured video fragment (converted to mp4) here.

3. Using Python and OpenCV you can experiment with depth map:

For this example I used slightly modified code from my previous project 3Dberry (https://github.com/realizator/3dberry-turorial)

I used this pair of cameras for taking the pictures in examples above:

SLP (StereoPi Livestream Playground) Raspbian Image

https://wiki.stereopi.com/index.php?title=SLP_(StereoPi_Livestream_Playground)_Raspbian_Image#SLP_Admin_panel_options



https://github.com/realizator

https://github.com/realizator/StereoVision

https://github.com/realizator/stereopi-fisheye-robot

https://github.com/search?p=3&q=stereopi&type=Repositories

High Efficiency Image File Format (HEIF)

https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format

High Efficiency Image File Format (HEIF) is a container format for individual images and image sequences. It was developed by the Moving Picture Experts Group (MPEG) and is defined as Part 12 within the MPEG-H media suite (ISO/IEC 23008-12). MPEG claims that a HEIF image using HEVC requires about half the storage space as the equivalent quality JPEG. HEIF also supports animation, and is capable of storing more information^{[citation needed]} than an animated GIF or APNG at a small fraction of the size.

Introduced in 2015, HEIF was adopted by Apple in 2017 with the introduction of iOS 11, and support on other platforms is growing.

HEIF files are a special case of the ISO Base Media File Format (ISOBMFF, ISO/IEC 14496-12), first defined in 2001 as a shared part of MP4 and JPEG 2000. This file format standard covers multimedia files that can also include other media streams, such as timed text, audio and video.

https://www.tomshardware.com/news/how-to-open-heic-file,36733.html

High Efficiency Image Container (HEIC) in iOS 11 and macOS High Sierra in 2017.

HEIC And HEIF

HEIC is the container or file extension that holds HEIF images or sequences of images. HEIF borrows technology from the High Efficiency Video Compression (HEVC) codec, also known as h.265. Both HEVC and HEIF are proprietary technologies developed by the Moving Picture Experts Group (MPEG).

HEIF came into the mainstream when Apple made it the default format for its pictures on iOS11 devices and macOS High Sierra. However, other operating systems or websites don’t yet support HEIF and its HEIC file extension, so Apple’s operating systems will automatically convert the images to JPEG when users want to share them with friends who don’t use Apple products.

HEIC files can store not just multiple individual images, but also their image properties, HDR data, alpha and depth maps, and even their thumbnails.

https://techcrunch.com/2018/06/07/how-facebooks-new-3d-photos-work/

DepthKit - Depth Image Video - Free Viewpoint video. volumetric video

https://www.depthkit.tv/
DepthKit for AFrame

An A-Frame component for rendering Volumetric videos captured using DepthKit (i.e Kinect + DSLR) in WebVR. The component wraps DepthKit.js which provides a similar interface for Three.js projects.


https://github.com/juniorxsound/DepthKit-A-Frame

https://orfleisher.com/aframe

https://github.com/juniorxsound/DepthKit-for-Max

DepthKit for Max/Msp/Jitter

A sample Max patch demonstrating a workflow for playing volumetric video in Max/Msp/Jitter using DepthKit combined-per-pixel exports. Supports rendering a mesh, wireframe and points.

https://github.com/ScatterCo/Depthkit.js

A Comprehensive List of 3D Sensors Commonly Leveraged in ROS Development

http://telerobotics.blogspot.com/2018/04/a-comprehensive-list-of-3d-sensors.html

Light Field Imaging: The Future of VR-AR-MR

Published on Nov 24, 2015

 Presented by the VES Vision Committee. Presentation by Jon Karafin, Head of Light Field Video for Lytro, followed by a Q&A with all presenters moderated by Scott Squires, VES. View Parts 1-3, as well as an amazing 360 video of the Panel with all presenters at

https://www.youtube.com/playlist?list=PLkK8iVS5ZZ7TsRbdPIviKymxlGmf3t9qW


https://www.visualeffectssociety.com/events/event/event-light-field-imaging-future-vr-ar-mr-los-angeles



 










Event - Light Field Imaging: The Future of VR-AR-MR (Los Angeles)

When:
Tue Nov 17, 2015
6:30pm to 9:30pm
Type:
event


(Image from Light Field Capture device – Photo provided by Lytro)

Light Field Imaging is a technology designed to capture and re-create light rays in a three dimensional scene. It has applications in entertainment, consumer devices, industrial applications and medical imaging. The presentations will cover the latest research in this technology which promises to revolutionize virtual, augmented and mixed reality.

Check out videos of the event at the links below:

https://www.youtube.com/watch?v=Raw-VVmaXbg

https://www.youtube.com/watch?v=ftZd6h-RaHE

https://www.youtube.com/watch?v=0LLHMpbIJNA

https://www.youtube.com/watch?v=_PVok9nUxME

Watch 360 degree video here: https://www.youtube.com/watch?v=FbP9hsdnVmg&list=PLkK8iVS5ZZ7TsRbdPIviKymxlGmf3t9qW&index=5

Principal Speakers:

Paul Debevec, Chief Visual Officer, USC Institute for Creative Technologies, will present the latest technologies being developed at USC ICT on light fields and photoreal virtual actors for Virtual Reality. He will cover the areas of high-resolution face scanning, real-time photoreal digital characters, and light field capture and playback for creating breathtaking realistic and interactive VR content.

Mark Bolas, Director for Mixed Reality Research at USC Institute for Creative Technologies, will describe the MxR Lab and Studio’s recent work on: Discovering Near Field VR Stop Motion with a Touch of Light Fields and a Dash of Redirection which won the Best VR/AR competition at SIGGRAPH 2015.

Jules Urbach, Founder & CEO of OTOY will discuss OTOY’s cutting edge light field rendering toolset and platform.

Jon Karafin, Head of Light Field Video for Lytro, a company which develops light field cameras will discuss light field technologies and their application in visual effects workflows, cinematography and virtual reality as well as the next generation of state-of-the-art capture systems

Moderator: Scott Squires, VES, Academy Tech Award Winning Visual Effects Supervisor and Developer

Paolo Favaro: Portable Light Field Imaging: Extended Depth of Field, Ali...

From ICCP11 Hosted by Carnegie Mellon University, Robotics Institute

April 8, 2011



Abstract:



Portable light field cameras have demonstrated capabilities beyond conventional cameras. In a single snapshot, they enable digital image refocusing, i.e., the ability to change the camera focus after taking the snapshot, and 3D reconstruction. We show that they also achieve a larger depth of field while maintaining the ability to reconstruct detail at high resolution. More interestingly, we show that their depth of field is essentially inverted compared to regular cameras. Crucial to the success of the light field camera is the way it samples the light field, trading off spatial vs. angular resolution, and how aliasing affects the light field. We present a novel algorithm that estimates a full resolution sharp image and a full resolution depth map from a single input light field image. The algorithm is formulated in a variational framework and it is based on novel image priors designed for light field images. We demonstrate the algorithm on synthetic and real images captured with our own light field camera, and show that it can outperform other computational camera systems.



Bio:



Paolo Favaro received the D.Ing. degree from Universita di Padova, Italy in 1999, and the M.Sc. and Ph.D. degree in electrical engineering from Washington University in St. Louis in 2002 and 2003 respectively. He was a postdoctoral researcher in the computer science department of the University of California, Los Angeles and subsequently in Cambridge University, UK. Dr. Favaro is now lecturer (assistant professor) in Heriot-Watt University and Honorary Fellow at the University of Edinburgh, UK. His research interests are in computer vision, computational photography, machine learning, signal and image processing, estimation theory, inverse problems and variational techniques. He is also a member of the IEEE Society.

Intel post promotional videos for its 3D camera-based RealSense

Another Two Video Promotions from Intel
 

Intel keeps posting promotional videos for its 3D camera-based RealSense technology. The first one shows refocusing capability similar to Lytro,  Pelican Imaging and some Nokia products:

second video demos 3D scanning:



http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html


https://software.intel.com/en-us/realsense/home    Intel® RealSense™ Developer Kit.

Softkinetic depthsense time-of-flight 3D camera Teardown

Leap Motion Teardown in Macro HD

Stereo Vision and Depth Mapping with Two Raspi Camera Modules

http://hackaday.com/2014/11/03/stereo-vision-and-depth-mapping-with-two-raspi-camera-modules/




The Raspberry Pi has a port for a camera connector, allowing it to capture 1080p video and stream it to a network without having to deal with the craziness of webcams and the improbability of capturing 1080p video over USB. The Raspberry Pi compute module is a little more advanced; it breaks out two camera connectors, theoretically giving the Raspberry Pi stereo vision and depth mapping. [David Barker] put a compute module and two cameras together making this build a reality.

The use of stereo vision for computer vision and robotics research has been around much longer than other methods of depth mapping like a repurposed Kinect, but so far the hardware to do this has been a little hard to come by. You need two cameras, obviously, but the software techniques are well understood in the relevant literature.

[David] connected two cameras to a Pi compute module and implemented three different versions of the software techniques: one in Python and NumPy, running on an 3GHz x86 box, a version in C, running on x86 and the Pi’s ARM core, and another in assembler for the VideoCore on the Pi. Assembly is the way to go here – on the x86 platform, Python could do the parallax computations in 63 seconds, and C could manage it in 56 milliseconds. On the Pi, C took 1 second, and the VideoCore took 90 milliseconds. This translates to a frame rate of about 12FPS on the Pi, more than enough for some very, very interesting robotics work.

There are some better pictures of what this setup can do over on the Raspi blog. We couldn’t find a link to the software that made this possible, so if anyone has a link, drop it in the comments.

Heptagon offers smartphone camera for 3D imaging

Heptagon sorta 3D sensor, uses a camera array like Pelican imaging. how does it compare?



The TrueD H2 array camera is a 3D imaging and depth sensing microsystem for use in smart devices, such as a front-facing camera in a smartphone. It captures additional short-range depth information for gesture and user recognition, background removal, and enhanced imaging. Integration of image sensor, module, and optics allow a very small size.

The TrueD H2 has been reported to measure 5.9mm by 5.8mm by 2.25mm.

Heptagon
www.hptg.com

http://www.eetimes.com/document.asp?doc_id=1321024

http://www.laserfocusworld.com/.../heptagon-offers...

http://www.businesswire.com/news/home/20140213005424/en/Heptagon-Announces-Production-Ready-3D-Array-Camera-Smart#.U7McffldUQM

LazeeEye: Turn Your Smartphone Into a 3D Camera

Heuristic Labs' LazeeEye, is hot on the trail of PrimeSense's Capri3D, Structured sensor, and Omnivision mobile 3D scanning. Heuristics takes a different approach determining the depth points and may be less processing intensive than PrimeSense...



https://www.kickstarter.com/.../lazeeeye-turn-your...

So close but not funded!

Funding Unsuccessful This project’s funding goal was not reached on May 30.2014

683

backers

$232,822

pledged of $250,000 goal

Sony Lightfield Camera Application Promises Full Resolution Stereo Imaging

Lightfield-Forum noticed Sony patent application on lightfield camera that maintains the full image sensor resolution while providing stereo imaging. The 73-page, 43-figure US20140071244 application "Solid-state image pickup device and camera system" by Isao Hirota proposes dual level microlens and two-sided image sensor with 45-deg rotated pixels to achieve their promise:

Patent abstract:

There are provided a solid-state image pickup device and a camera system that include no useless pixel arrangement and are capable of suppressing decrease in resolution caused by adopting stereo function. A pixel array section including a plurality of pixels arranged in an array is included. Each of the plurality of pixels has a photoelectric conversion function. Each of the plurality of pixels in the pixel array section includes a first pixel section and a second pixel section. The first pixel section includes at least a light receiving function. The second pixel section includes at least a function to detect electric charge that has been subjected to photoelectric conversion. The first and second pixel sections are formed in a laminated state. Further, the first pixel section is formed to have an arrangement in a state shifted in a direction different from first and second directions that are used as references. The second direction is orthogonal to the first direction. The second pixel section is formed in a square arrangement along the first direction and the second direction orthogonal to the first direction.

For more information, check out the full patent details here: Patent US20140071244 – Solid-state image pickup device and camera system

http://lightfield-forum.com/2014/07/sony-patents-light-field-sensor-with-full-resolution-3d-stereo-output/

ESPROS Demos ToF Evaluation Kit

ESPROS Photonics publishes Youtube video showing its epc610 ToF sensor evaluation kit:

http://www.espros.ch/3d-imagers

Fully integrated Time-of-Flight imager/camera built as a system-on-chip.

      

epc610 on PCB                                         The world's smallest TOF camera based on epc610

Operation principle

The epc610 chip is based on the 3D TOF principle of light. Modulated light is emitted by a transmitter. This light is reflected by the object to be detected. The returning light is sampled by an on-chip photosensitive TOF CCD array. The receiver compares the phase difference between the emitted and the received light and computes the time difference of the "Time-of-Flight" individually per pixel. This value multiplied by the speed of light (ca. 300'000km/sec) and divided by 2 corresponds directly linearly to the distance.
The epc610 chip is designed to enable simple and cost effective 3D TOF cameras. Together with a microprocessor and few external components a fully functional TOF camera can be built.
The measurement functionality supports distance and ambient light measurement with variable integration time and on-chip temperature measurement for drift compensation.

Image Sensors World: Google Tango Project Tablets Feature pmd's 3D technology

http://image-sensors-world.blogspot.com/2014/06/google-tango-project-tablets-feature.html

Google’s Advanced Technology and Projects group (ATAP) is demonstrating their latest Project Tango Tablet Development Kits at Google I/O 2014 today. Now, for the first time, the tablets with integrated pmd-based depth sensors are shown publicly. Besides a motion tracking and a RGB camera the tablets utilize a pmd-based 3D ToF sensor to allow the tablet to sense its environment in space and motion. The ToF sensor used in the tablet has been jointly developed by Infineon and pmdtechnologies.

"We are proud that Google's ATAP group shows the world for the first time how our 3D technology is contributing to the addition of environmental awareness to mobile devices in a new and unique way. We also look forward to seeing what kind of amazing applications will be developed based on this tablet as the possibilities are endless in applications such as augmented reality, architecture, retail, gaming, and many more.", says Bernd Buxbaum, CEO of pmdtechnologies.


http://www.pressebox.com/pressrelease/pmdtechnologies-gmbh/Googles-ATAP-Group-shows-first-Tango-Tablets-at-Google-IO-live-with-pmds-3D-technology/boxid/686619

http://www.pmdtec.com/news_media/press_release/google_tango_tablet.php

Google's ATAP Group shows first Tango Tablets

http://www.pmdtec.com/news_media/press_release/google_tango_tablet.php

Duo3D AudioSight - 3D depth mapping

3D depth mapping announcement comes from Duo3D presenting AudioSight - a software-based solution turning stereo camera into a depth mapping device.

http://duo3d.com/


Youtube video 

http://duo3d.com/product/duo-mini-lv1

They have a dual camera board that does board level  3D sensing utilizing stereo vision, ready to work out of the box supporting a wide range of accessories and configurations. $200 USD

Google's Project Tango using Mantis Vision's Structured Light 3D Camera

Israel based Mantis Vision confirms that its MV4D 3D camera is used in Google's Project Tango tablet.

Mantis Vision camera consists of flash projector hardware components and includes structured light-based depth sensing algorithms like PrimeSense technology used in the Microsoft Kinect and now owned by Apple. PrimeSense is also Israel based.

Amihai Loven, CEO, Mantis Vision said:

"3D represents a major paradigm shift for mobile. We haven't seen a change this significant since the introduction of the camera-phone. MV4D allows developers to deliver 3D-enabled mobile devices and capabilities to the world,"
 "This partnership with Google offers Mantis Vision the flexibility to expand quickly and strategically. It will fuel adoption and engagement directly with consumer audiences worldwide. Together, we are bringing 3D to the masses."

Johnny Lee, Technical Product Lead at Google said:

"We are excited about working with partners, such as Mantis Vision, as we push forward the hardware and software technologies for 3D sensing and motion tracking on mobile devices,"

http://www.mv4d.com/

http://www.marketwired.com/press-release/mantis-visions-mv4d-tapped-as-core-3d-tech-behind-googles-project-tango-tablets-1917998.htm

Project Tango Tablet Development Kit

Hacking the Xbox Kinect with Johnny Chung Lee

Google "rapid evaluator" Johnny Chung Lee demonstrates how independent developers around the globe are using Microsoft's Xbox Kinect to create sophisticated full-body tracking video games.