Signal of the Day: Resolution and Space Exploration
Digital Signal Processing is the branch of engineering that, in the space of just a few decades, has enabled unprecedented levels of interpersonal communication and of on-demand entertainment. By reworking the principles of electronics, telecommunication and computer science into a unifying paradigm, DSP is a the heart of the digital revolution that brought us CDs, DVDs, MP3 players, mobile phones and countless other devices. The goal, for students of this course, will be to learn the fundamentals of Digital Signal Processing from the ground up. Starting from the basic definition of a discrete-time signal, we will work our way through Fourier analysis, filter design, sampling, interpolation and quantization to build a DSP toolset complete enough to analyze a practical communication system in detail. Hands-on examples and demonstration will be routinely used to close the gap between theory and practice. To make the best of this class, it is recommended that you are proficient in basic calculus and linear algebra; several programming examples will be provided in the form of Python notebooks but you can use your favorite programming language to test the algorithms described in the course.
in which we will talk about camera resolution and space exploration.
We will start our discussion on camera resolution by talking about one of
the most exciting scientific missions to take place in the last ten years,
the Rosetta mission.
The Rosetta spacecraft was launched about ten years ago by the European Space Agency
with the mission of reaching the comet Churyumov–Gerasimenko.
This comet is also known with the more technical but
easier to pronounce name, 67P.
Rosetta had to fly through space for
ten years in order to gain the necessary momentum to join the comet on its orbit.
And it had to fly by Mars and Earth over four time to get what is called
the gravitational assist, basically exploiting the gravitational field of
planets to gain speed and reach the comet.
The total flight path was over 6.4 billion kilometers, and
Rosetta successfully reached the comet 67P in August 2014.
And it became actually the first man-made object to orbit a comet.
Rosetta also carried a lander along called Philae, which was supposed to descend
on the comet and analyze its physical and chemical properties.
But unfortunately, not everything went perfectly well with the lander.
And only time will tell if the lander will be able to complete
its part of the mission.
But if we go back to the main topic of our video today,
what Rosetta managed to do was to send back to Earth increasingly detailed
pictures of the comet as it was getting closer to it.
So here's the first picture from 86,000 kilometers away, and
the underlying resolution is one kilometer per pixel.
As Rosetta got closer to the comet, the quality of the pictures progressively
improved, and here we start seeing the shape of the comet for instance.
From 5,000 kilometers, we have a resolution of 100 meters per pixel.
From 285 kilometers, the resolution is now 5.3 meters per pixel.
And finally, one of the closest points that Rosetta managed to reach once it
entered orbit around the comet is a distance of six kilometers for
a resolution of 11 centimeters per pixel.
These are the most detailed pictures ever taken of a comet.
But the question is, why do we have to send a probe for
ten years into space in order to get high resolution pictures of the comet?
Can't we just point a big telescope and obtain the same result?
And this is the question that we will try to answer today.
People have been looking at the sky since time immemorial.
And it is a fact of life that you don't really expect
to see a lot of detail when you look up at the sky with the naked eye.
It has always been accepted that the limiting device
in appreciating the complexity of the cosmos is the human eye.
It is not powerful enough to see details of objects that are so far away.
But things of course changed with the invention of the telescope.
Now you had a device that could make the eye more powerful, and so
the question was, what are the technical limitations of a device like a telescope?
Another way to formulate the question is, what are the mechanics of image formation,
and what are the things that limit the resolution of an image
that I can obtain with an optical system?
Let's take a very simplified model of image formation, and
let's assume that all the points of interest are sufficiently far away
from the imaging device, whether it's a camera or
whether it's the eye, that we don't really have to take their distance into account.
If we accept this model, then the image that is formed, say, on the retina or
on the sensor of the camera is the convolution between the original light
sources and a transfer function that accounts for all the transformations
that light undergoes from the source to the imaging plane.
It is convenient to split the transfer function into two distinct components.
The first one describes the properties of the medium through which light travels.
Whereas the second one is dependent on the imaging device and
takes into account the effects of, say, lenses and
focal distance and characteristic of the imaging plane.
This is also called the point spread function, and
as we will see shortly, it characterizes the properties of an imaging device.
The reason why we perform the split is because the major enemy to image quality
for telescopes on Earth are the disturbances introduced by the atmosphere.
Atmospheric disturbances include turbulence, and scattering, and haze.
And you probably have experienced those even with your naked eye when you see
the Sun close to the horizon.
The distortion of the shape of the Sun is due to the fact that the sun
rays have to travel through a thicker
slice of the atmosphere at sunset before reaching your eyes.
Of course, today we're particularly privileged because in 1990,
NASA put into orbit the first space telescope, Hubble.
And so today we can perform telescopic observations that are immune
from defects of the atmosphere.
But even with a space telescope, our vision is limited.
We will see why in a second, but for the time being as a point of reference,
here are the best pictures of the comet P67 as seen by the Hubble Telescope.
And you can see they're a far cry from the kind of images we got from Rosetta.
So with the atmosphere out of the way, we can concentrate on the mechanics of image
formation and on the point spread function associated to the imaging device.
So let's set up a very simple model where we have a point light source.
So it's a star, say, a faraway star, that is sending light rays in all directions.
Instead of a telescope, let's put somewhere a very simple type of camera,
a pinhole camera.
A pinhole camera is a very simple device.
It's just a box with a very small aperture here,
the smallest aperture you can imagine, and an imaging sensor on the back plate here.
It could be film or it could be a digital CCD.
The properties of the pinhole camera are such that from every light point
source in space, only one ray of light can actually go through the aperture and
reach the back plate so that for each light source in space,
you have one image point on the image sensor.
If you have two light sources, well, it's easy to find geometrically
which ray ends up entering the camera through the pinhole.
And for each point is basically we'll have the correspondent point on
the image plane.
From this simple geometrical model,
it would appear that the ideal pinhole camera has a point spread function which
is just a delta function, namely nothing happens to light in the imaging process.
In other words, the pinhole camera appears to have infinite resolution.
And so, we might be tempted to say that the maximum achievable resolution
of a system is only dependent on the density of the sensor,
the digital sensor, that is put in the back plane of the camera.
In other words, if we assume that the back plane of the pinhole camera is composed of
an array of light-sensitive elements.
Let's call them pixels, like so.
We can see that in this picture where we have indicated with different
colors successive pixels, if the angular separation between points
in space is too small with respect to the width of the pixel,
two different points in space might end up mapped to the same pixel.
And therefore, we wouldn't be able to tell them apart in the resulting image.
So, in order to resolve two separate light sources in space,
we have to make sure that the density of pixels is such that each ray
that goes through the pinhole falls on a different pixel on the camera.
And furthermore, we need to have an intermediate pixel between the lit pixels
to make sure that we can distinguish between light, dark, and light, and
recognize the picture as two separate sources.
Alternatively, if we can't afford a higher pixel density,
what we can do is build a longer pinhole camera.
This is equivalent to using a zoom lens in a normal camera.
So you see that by extending the body of the camera,
we can use a lower pixel density and still achieve
resolution with respect to the same angular separation of the light sources.
However, in reality resolution is not limited by pixel density.
Resolution is limited by diffraction.
And this is when things start to get a little complicated.
When light goes through an opening, like in the case of the pinhole camera,
it starts exhibiting a wave-like nature, and the image
of the original point source will actually appear as a diffraction pattern.
The diffraction pattern through a small circular aperture is called the Airy disk.
And here we're showing how it looks.
Actually, we're showing the negative image of the diffraction pattern of light for
better contrast.
What this means is that on the back plane of the camera, whether the camera is
digital or analog, instead of seeing a single point corresponding to the original
point source, you would see this diffraction pattern.
Now, when you have two point sources,
each source will generate its own independent Airy pattern.
And so, instead of having two points here and here,
you will have two diffraction patterns that will start to interact
when the angular separation between sources is small.
The patterns will ultimately overlap so much that you will not be able to resolve
the individual sources independently of the sensor.
And so here, for instance, you have two Airy patterns
that are overlapping into one undistinguishable blob.
So you see independently on the sensor of the camera,
there is an angle of resolution between light point sources as seen by the camera,
under which no resolution is possible.
The minimum angle that guarantees resolution
is given by Rayleigh's criterion, and the formula is like so.
It will depend on the wavelength of the light that hits the camera and
on the diameter of the aperture.
It's actually inversely proportional to the diameter of the aperture, so
the smaller the aperture, the bigger the Airy disk.
And so, a pinhole camera would be an extremely bad device to obtain
sufficient angular resolution at, for very distant objects.
Okay, then.
If aperture is the problem, let's increase the aperture and
therefore increase the angular resolution of the camera.
Now, increasing the aperture is not as simple as drilling a bigger hole
in the pinhole camera because if we do so,
then each light source will have a cone of rays of light that will enter the camera.
This cone of rays of light will diverge inside the camera and
create an undifferentiated blob of light on the back plane.
So what we need to do is put a lens on the aperture so
that the cone of rays that hits the lens will be focused back on the back plane,
and one point source will be mapped onto one point on the sensor plane.
Similarly, when we have two point sources,
the lens will focus the rays again on a different point, and
we have the same situation as before, but with a larger aperture for the camera.
This incidentally is also very good because it means that more light
enters the camera, and so
the exposure time can be shorter, which is very good for moving objects.
However, now our system is dependent on an extra piece of hardware, namely, the lens.
And let me tell you right away that making precise lenses is a very, very
difficult task, which will limit the kind of apertures that we can use in real life.
But before we even consider that problem,
let's see what kind of aperture the Hubble Telescope should have
in order to be able to see the lunar landing module.
As you may have heard, there are some people out there,
especially on the internet, that claim that the Apollo missions were a hoax and
that man never landed on the Moon.
And what they say is if indeed man had landed on the Moon,
we should be able to point a telescope to the Moon and
see the leftovers from the Moon missions, such as the lunar landing module.
So, let's do the math.
The size of the LEM is about five meters across, and the distance from Earth or
the Hubble Telescope to the moon is about 380,000 kilometers.
If you do the trigonometry, you will see that the angle
subtended by the LEM is about 0.003 arc seconds, and
one arc second is 1 degree divided by 3,600.
So this is an infinitesimally small angle.
The Hubble Telescope has an aperture of 2.4 meters.
To compute the angular resolution of such an aperture, remember Rayleigh's
criterion requires us to specify the wavelength of the incident light.
Now, the visible spectrum is from 400 nanometers to 700 nanometers.
Let's pick 550 nanometers as the midpoint.
We plug these values into Rayleigh's criterion, and
we get an angular resolution for the Hubble Telescope of 0.1 arc seconds.
Proportionally, if we were to see the LEM on the moon,
the Hubble should have an aperture of 80 meters.
Now, even if we could put in space something as big as that,
we will still have to build a lens 80 meters across
with perfect focus, and this is currently way beyond the limits of our technology.
Any small imperfections in the curvature of the lens
would result in an out-of-focus and blurred image.
This is really what happens if you're shortsighted, for instance.
Your lens is too round for the length of your eye, or your eye is too long for
the thickness of your lens, and the focus will happen before the retina.
To give you an idea of the level of precision that is required
by large aperture optics,
consider that when the Hubble Telescope was launched in 1990, astronomers
realized that there was a minute error in the calibration of the optics.
The Hubble Telescope, like all large telescopes,
doesn't use lenses to focus the image, but uses mirrors.
Now, there's a mirror in the back of the Hubble Telescope that has the same
diameter as the aperture of the telescope.
So we're talking about a 2.5 meters wide mirror.
The curvature of the mirror had an imprecision of 2.2 micrometers
at its edge, and as a result, Hubble could not focus properly.
In the left-hand side of this picture, you see how a galaxy would look
before corrective optics were installed on the telescope.
And on the right-hand side of the picture, you see the picture after correction.
So you see a minute error of 2.2 micrometers over 2.5 meters
of diameter was already enough to cause this kind of blurriness.
Okay, but so, what about pixels?
How does the camera resolution impact image quality?
To understand that, we have to go back to Rayleigh's criterion for
the minimum resolvable angle in a camera.
That angle maps to an actual physical distance
on the camera sensor via this formula here.
And if you look at it in detail, you see that this is equal to theta, which is
the minimal angle given by Raleigh's criterion times the focal length.
What this formula tells us is that if the pixel separation on the camera sensor
is not less than delta x,
our camera will be resolution limited rather than diffraction limited.
Note that f over D, which appears in the formula here,
is what photographers call the f-number of a camera.
So imagine you have two point light sources that have an angular separation
that is just resolvable according to Rayleigh's criterion.
On the camera's sensor, you would observe the two diffraction patterns
relative to the two light sources.
And in order to be able to distinguish the two light sources,
your pixel size would have to be small enough to capture the light pattern here,
the darker pattern here, and the light pattern here.
If the pixels are any bigger,
then we will not be able to resolve the light sources due to a lack of resolution.
It is clear from this discussion that what is important in a camera sensor is not so
much the absolute number of pixels, but the pixel density.
So for instance,
this sensor here would have the same number of pixels as this sensor here.
But clearly, this sensor will not be able to resolve the same two point
light sources because the pixel density in this sensor is much lower.
When it comes to commercial cameras, the marketing slogans are about
the number of pixels, but hardly ever about the size of the sensor.
So, how many megapixels do you actually need in your camera?
This really depends on the size of the sensor and on the optics.
So, the minimal resolution,
the minimal pixel density, will depend on the f-number, as we showed just before.
So let's pick an f-number that works pretty much in most conditions, f over 8.
If we use the spatial Rayleigh criterion, we find that the pixel pitch,
the width of the pixel that we need to be diffraction limited
rather than resolution limited will have to be four micrometers.
So that means that the maximum pixel area will be 16 times 10 to
the minus 5 square millimeters.
Or if we flip the perspective, it means that to operate to the diffraction limit,
we need approximately 62,500 pixels per square millimeter.
So now we need to look at the size of the sensor.
High-end cameras today usually have one of these two sensor formats.
The APS-C, which is very common and has an area of 329 mi, square millimeters.
With that sensor, after you reach 20 megapixels, your camera is already
operating at the diffraction limit, so no use in increasing the number of pixels.
For a full sensor that simulates the 35-millimeter film and has an area
of 864 square millimeters, the limit, the limit will be 54 megapixels.
So in the end, don't buy more megapixels than you need.
Now let's go back to our Rosetta spacecraft.
This is the camera that is on board and
that has sent us back these beautiful pictures of the comet, and you'll be
surprised to know that the resolution of the camera is only about four megapixels.
So the moral of the story is you don't need to buy a camera with a lot of pixels
if you're willing to travel 6 billion kilometers to get close to your target.