I can’t see you – it’s too loud in here.

Once upon a time I started to write a post about some night astrophotography I made and before I could post it, the whole thing blew up into an excruciatingly long and technical exposé. This is thus the first post in a series on what I call computational photography, one of my favorite activities with my camera.

I can’t help it – I’m an engineer (yes, you’re born that way, and yes, this Dilbert cartoon is exactly how it goes down in the ER).

What this means is that when I take a picture with my camera, I see numbers. Today’s discussion is about the difference between wanted and unwanted numbers. In this case, we will call the wanted numbers a signal, and the unwanted numbers noise. These are terms from the world of signal processing, which is oh so exciting!

Okay, what is noise and why is it visual?

A camera has a single job – accurately capture the image of the scene it looks at. Since cameras aren’t perfect, they get imperfect representations of the scene. We call a certain group of these imperfections noise, because just like stray noise on a street or in a factory, it appears random and conveys no important information. In a modern digital camera, we see noise in the form of strange colorations, blurry bubbles, and unexpected bright spots in our photos.

The color noise in this portrait has turned the background purple and hides the intricate patterns of the lion’s mane.

Here the noise obscures the boundary between sky, trees, and water. It also masks the ripples of the waves, making them indistinguishable from the graininess in the image.

Noise here obscures the sharp lines of the Bank of China tower and has created a strange and wavy glow around the building edge where it should be black sky.

Noise ruins color, sharpness, and can hide entire objects from our pictures, such as stars against a night sky. It’s like trying to hear someone talking during the Superbowl at the stadium. The noise drowns out the desired voice. We have simply borrowed the audio terminology because of the similarities.

What causes noise?

Heat.

Heat implies energy and that energy corresponds to little atomic particles running around like they just ate a bunch of Mexican jumping beans. These charges jump onto the camera and cause it to register a false reading. The charges originate from various sources, even from cosmic rays in outer space.

When working with photography, we further classify noise as originating from within the camera, which is completely caused by imperfections in the technology; and originating from the environment, over which we have no control.

This is an important distinction because we can limit the impact of the noise coming from the camera by making better cameras but we can’t do a darn thing about the noise from the environment – well, almost.

The impact of the noise is dependent on a very important measure we call the signal-to-noise ratio. This is where the math starts coming in. If you think back to the stadium analogy, imagine trying to hear your friend from fifty feet away who is talking in a low speaking voice verses the man shouting, “Cold beer…peanuts!” The noise in the stadium is the same in both cases, but because the salesman is louder, he has a stronger signal in comparison to the noise level, thus you hear him better.

Camera sensors are basically a big array of buckets – photo wells – that count how many photons jump in. Noise occurs when stray photons jump into those wells and register as if they came from the objects in the scene, even though they didn’t. Collisions with these atomic particles inside the well can trigger those photons to release, just as heat within the camera can trigger stray photons – thermal noise. The brighter the object, by the way, the more wanted photons it will emit, which is why most image noise can be found in the darker areas. Fewer desired photons means a lower SNR.

There are further causes of noise in the transformation from photon count into light value. The good news, mathematically speaking, is that noise tends to be random. If it’s systematic instead of random, we probably wouldn’t call it noise.

What do we do about it?

Random means that the noise is different each time we snap the shot. For static scenes, we can synthetically remove noise using simple statistical operations. There’s not much to do about moving scenes where we have but a single chance to capture the picture, unfortunately. We can do what Nasa does with the Hubble images to give us their incredible portraits of our universe: image stacking.

This is the Hubble eXtreme Deep Field, a composite of 2 millions seconds from images taken over the previous ten years! Each bright spot is an entire galaxy, the dimmest of which is less than one ten-billionth as bright as the minimum brightness that we can see with our eyes.

For scale, the box labeled XDF shows which part of the sky that image shows.

Here are a couple of examples from images I took within the past week while in Singapore and Hong Kong. It was late at night, dark, and I only had my iPhone. You can click on the pictures to see a larger version where the detail comes out.

Unaltered: unclear lines, bubbly colors in the background, limited dynamic range (difference in quality between the brightest and dimmest areas of the image), and overall blurriness plague this lowly-lit scene.

Cleaned up, the building looks shark and clear, the bright and dim regions are restored, and the colors stand out.

Although it doesn’t look too bad at a glance, this unaltered image suffers from spotty noise and grainy lines on the building facade.

Cleaned up, the building looks great and the sky is dark.

This is the power of computational photography – to uncover the unseen and capture the impossible with imperfect equipment in undesirable circumstances.

Stay tuned! We’ll uncover how to do this and how to get rid of noise in another blog post.