Shutter Speed: HDR and Low Light

Shutter speed (or pixel integration timing) is a critical factor when defining the required low-light capabilities of a sensor/lens combination for a new robotics camera. The primary trade offs under consideration when considering shutter speed are motion blur and temporal noise in the image:

- Motion blur results in reduced of contrast (spatial frequency) across object space. Unfortunately, motion blur is not possible to correct for with traditional imaging methods expect through shutter speed adjustment. To visual human-viewers, motion blur results in reduced image JNDs for moving ojects, however, motion blur can also be employed for artistic 'long exposure' photography with tripods and neutral density filters. For robotic systems, newer state of the art CNN approaches can be used to infer the original de-blurred object, however, these networks require immense amounts of stationary ground-truth data from the exact same camera+ISP configuration.

- Temporal Noise is a numeric variance in a single pixel's output value which can appear as white 'speckle' or moderate white/gray/black shift. Temporal Noise is a result of numerous components within the imaging system such as semi-conductor materials, pixel architecture design, analog-to-digital converters, and lithography technologies. Many sensor manufacturers try to compile their results into a single dynamic range value which is based at room-temperature. For comparison, human vision has seemingly no noise (at any temperature!) due to the spatial density of rods/cones in the eye and advanced processing in the brain - if you have references on this topic it would be appreciated!

For robotics, shutter speed highlights one of the major disconnects in the industry as HW and SW engineers make subjective best guess selections. Noise resultant from digital gain can be catastrophic to accuracy of single-frame detection algorithms; in the worst case this is similar to a 'single pixel attack' and in the best case the denoising results in lower accuracy.

For consumers, shutter speed (1/30s-1/1000s) has a much more functional impact is typically much shorter than the human tactile reflex (1/4s) so timing between capture and human awareness will be naturally offset. The most interested daily-user can check the direct manifestation of shutter speed on their cellphone when switching between HDR and regular mode. Every phone has the the user interaction configured every-so-slightly differently, some which better represent the actual time-slice in the HDR multi-exposure pixel integration. When in HDR mode, notice that the 'click' is slightly delayed relative to the non-HDR mode - consumer phone manufacturers have to tediously dial this in during production; a related interesting insight is below!

"Digital cameras generally provide better feedback to their users than film cameras. After each shot, the photographer can see a (small) version of the image just captured. This eliminates all kinds of errors that were common in the film era, from failing to load the film properly (or at all), to forgetting to remove the lens cap, to cutting off the head of the central figure of the picture. However, early digital cameras failed on one crucial feedback dimension. When a picture was taken, there was no audible cue to indicate that the image had been captured. Modern models now include a very satisfying but completely fake "shutter click" sound when a picture has been taken. (Some cell phones, aimed at the elderly, include a fake dial tone, for similar reasons.)"

From Richard Thaler and Cass Sunstein, Nudge, page 92.