[Blindmath] Life After Images

[Richard Baldwin]

More food for thought.

Michael wrote:

"I just wonder whether image data may be able to give us potentially other
meanings if we process it in other ways?

Yes there probably is more innovative ways of providing access to image
data, however it might be worth keeping in mind one needs to draw the same
conclusin of what it shows in the end (IE. a photo of a tree, well the
observer should still recognise a tree when viewing it in the other way)."

My response to that is, maybe so, and maybe not. Consider the following
hypothetical scenario involving a tree. You were blind from birth. During
your childhood, your family owned a nice getaway cottage in the countryside
in Vermont. You spent many happy days at that cottage during your childhood
and you have very fond memories that you would like to pass along to your
children.

Your parents and your siblings are no longer alive, but fortunately, your
mother was an avid photographer and started using a digital camera when you
were very young. Also, she was very well organized and you now have access
to a well-organized digital family photo album.

You would like to begin writing the story of your life and you plan to use
those photos, among other things, as resources in that project.

Being the organized person that she was, each time your family spent a few
days at the cottage, your mother would place you, your older sister, and
your older brother in a line in front of a beautiful maple tree and take a
picture of the three of you. She always lined you up by age with you on the
left and your older brother on the right. Now you have 30 or 40 such
photographs but there is nothing to identify when they were taken.

When someone views one of those photos, they not only see you and your
siblings in front of the tree, they also see a gorgeous wooded Vermont
mountain side in the background and a vehicle in the driveway off to the
left of the tree.

You would like to estimate the month and year that each of those photos was
taken.

As it turns out, your family owned three different vehicles during your
childhood: a sedan, a van, and a crew-cab pickup truck. By going through
your parent's written records, you are able to establish when each of those
vehicles was purchased. Therefore, if you can establish which vehicle was
parked in the driveway in each photo, that will allow you to classify each
photo to within a date span of several years.

Being the resourceful person that you are, you emboss each of the photos
and explore the silhouettes of the vehicles. (Note that you may need to
apply an outline filter to the photo before embossing to make the shape of
the vehicle stand out from the background.) Fortunately, the silhouettes of
a van, a sedan, and a crew-cab pickup truck are sufficiently different that
you can use that information to classify the photos into groups. Now you
know, to within a few years, when each photo was taken.

Next, you decide to tackle the task of estimating the time of year that
each photo was taken. You really need to be cleaver to pull this one off.
You begin by learning how to program to the point that you develop
expertise in manipulating the color data in the pixels in digital images.

Then you write a program that accepts an image file as input and produces
three output image files for each input image. Each output image file is a
simple histogram that can easily be embossed and explored with your
fingers. One of the histograms shows the distribution of the hue values of
all of the pixels in the original image. The second histogram shows the
distribution of the saturation values of all the pixels in the original
image. The third histogram shows the distribution of the brightness values
of all the pixels in the original image.

Each histogram shows the number of occurrences of a particular value in the
range from 0 to 1.0 in the original image.

So, what can you learn from these histograms?

Assume that you examine the histograms for an image and find that the
saturation histogram tends toward the very low side and the brightness
histogram tends toward the high side. In this case, you might conclude that
the photo was taken on a bright sunny day when the tree, the vehicle, and
mountain side in the background were all covered with snow. (Low saturation
and high brightness indicates a lot of white.) Although I have never been
to Vermont in the winter, I suppose that might indicate that the photo was
taken in the December through March time frame.

Next, you examine the histograms for a different image and find that the
saturation tends toward the high side and the hue histogram tends heavily
towards values that indicate a predominance of red, orange, and gold. You
conclude that photo was probably taken during the last couple of weeks of
October and possibly the first week of November when the leaves on the big
maple tree and the leaves on the mountain side in the background were awash
with bright autumn colors.

The next histogram shows a similar result, except that this time, the hue
histogram tends toward values that indicate a lot of green. When was the
photo taken? Probably sometime between late spring and early Fall.

Still another histogram might show low saturation values and low brightness
values. This result might indicate that the photo was taken when the maple
tree and the trees on the mountain side had lost their leaves but had not
yet become covered with snow. Lots of dark objects in the image.

Well, that's progress. So far you have been able to categorize the photos
into seasons within spans of several years each.

Now back to the images of you and your siblings. By examining the
silhouettes of you and your siblings, you might be able to nail the time
spans down a little closer. For example, if you are short but your sister
and brother are taller, that might mean that the photo was taken while you
were quite young.

If you are a male and you appear to be a little taller than your sister,
that might mean that you have reached your teenage years.

If your brother is missing in a group of photos, that might mean that was
the year when his Army Reserve unit was called to active duty.

If you find different groups of photos with different resolutions, you
might be able to tie those photos to the sequence of better and better
cameras that your mother purchased over the years, and for which she was
good enough to keep the dated receipts.

So, when the magician says "watch my hands," don't fall victim to
misdirection and watch his hands. Instead, watch everything but his hands.
When someone says look at the tree in the photo, it's okay to look at the
tree, but also look at everything else in the photo as well. There is no
telling what you might find there.

When law enforcement officers find a decaying human corpse in the woods,
they not only look at the corpse, they also look at everything else as
well. I have read that they can estimate about how long the person has been
dead by identifying the types of bugs that are consuming the corpse at that
point in time. They can also sometimes identify the dead person through
dental records or DNA.

When a modern digital seismic station detects an earth tremor, they may not
look initially at the waveform of the signals produced by the seismometers.
Instead, they may transform those time-domain signals into frequency-domain
signals and look at the spectral content. If there are peaks in the
spectrum at 1 hertz and below, it is probably an earthquake and that may
trigger an automatic telephone call to the chief seismologist in the middle
of the night to get her out of bed and so that she can come to the seismic
station. At that point, the seismologist will probably be interested in the
shapes of the seismic waveforms.

However, if the peaks in the spectrum are much above 1 hertz, that might
mean that the signals were generated by an explosion at a rock quarry a few
hundred miles away and that is not something that should trigger the
automatic telephone call to the chief seismologist in the middle of the
night.

Passive sonar sensors on a submarine generate waveforms that replicate
sound pressure waves generated somewhere in the ocean, which are impinging
on the sensors. However, it is unlikely that the sonar operator will be
interested in the actual waveforms of the signals. Instead, she will
probably be interested in transforming those signals into other formats,
such as spectral data or bearing data, and will examine the data in those
other formats. For example, a spectral line at a particular frequency (in
combination with other information) might indicate that a particular type
of coolant pump on a particular type of submarine that belongs to the
"other guys" may be in the area.

In other words, "watch my hands" and see me miraculously pull a rabbit from
a hat. Don't watch my hands, and you may see how I pull the rabbit from the
hat. Look at the gray scale silhouette of the tree and miss the color of
the foliage that might indicate when the photo was taken.

Food for thought.

Dick Baldwin

On Fri, Mar 23, 2012 at 5:07 AM, Michael Whapples <mwhapples at aim.com> wrote:

> The only thing which comes to mind is, if taking such an approach will one
> take the same meaning from the data?
>
> To illustrate what I mean, I think back to a humourous exert from a book
> on the radio, Alan Coren's "Core Coren", the episode name "out for lunch".
> Unfortunately I can not give a link to listen as the BBC does not have this
> currently available to listen to, but if you do a google search you may
> find an unofficial download. Alternatively I think its an exrt from Alan
> Coren's book "A bit on the side".
>
> In short in that Alan Coren discusses how historians sometimes seem to be
> able to give us such wonderful stories of how things were from so few facts
> and he muses over alternative explainations of the few facts (eg. how do we
> know some of these ancient cultures were hunters, may be the tools they
> made were things they had made and were admiring and thinking about how to
> use and the animal bones surrounding the human skelitons are simply from
> animals which came to eat these people who didn't know how to use the tools
> for fighting. Any broken animal bones, may be they were really created by
> the animals fighting each other so that they would get a larger piece of
> the feast of defenceless human. He goes on in such a style expanding it
> further and further.)
>
> I just wonder whether image data may be able to give us potentially other
> meanings if we process it in other ways?
>
> Yes there probably is more innovative ways of providing access to image
> data, however it might be worth keeping in mind one needs to draw the same
> conclusin of what it shows in the end (IE. a photo of a tree, well the
> observer should still recognise a tree when viewing it in the other way).
>
> Michael Whapples
>
> -----Original Message----- From: Richard Baldwin
> Sent: Thursday, March 22, 2012 9:57 PM
> To: BlindMath Mailing List ; accessibleimage at freelists.org
> Subject: [Blindmath] Life After Images
>
>
> A couple of years ago, I watched a very interesting series on TV named
> "Life After People." The thesis was that suddenly for unknown reasons, all
> humans disappeared from the earth. The series went on to explore what would
> happen to everything that people left behind: buildings, pets, livestock,
> plants, etc.
>
> The purpose of this post is to promote thought, discussion, and perhaps
> innovation.
>
> Assume that all of sudden, for some unknown reason, every device on earth
> capable of displaying images as we know them (regular arrays of colored
> pixels) were to become non-functional. Assume also that every picture on
> earth was suddenly erased, but the data behind those pictures were left
> intact.
>
> Also assume that devices capable of producing more image data, such as
> digital cameras, would continue to exist in a fully functional form.
>
> At that point, there would exist giga-giga-bytes of digital image data
> throughout the world, with more on the way, but there would be no way to
> display that data in the conventional sense of a visible picture.
>
> Would mankind simply allow all of the information stored in that data to
> become lost, or would mankind find alternative ways to retrieve and use the
> information in that data. I believe that the latter would be true. While
> many might throw up their hands in despair, a few really bright
> entrepreneurs might find ways to extract and use that information, some for
> the benefit of mankind and perhaps some to the detriment of mankind.
>
> Well, we know that isn't going to happen. However, we also know that there
> is a large community of very bright people who are deprived of the ability
> to extract and use the information contained in that data in the
> conventional way -- pictures. Might there be some bright entrepreneurial
> mathematicians, physicists, engineers, and digital signal processors within
> that community who are willing to think about and innovate new and
> different ways to extract and use that information.
>
> Let me sow a few seeds for thought to see if any of them will take root,
> grow, and blossom.
>
> To begin with, forget about image data as being associated with pictures.
> Instead, think of the data in an image file as a very large set of
> numerical values, which, when arranged in a particular way and presented in
> a particular format will cause a sighted person like myself to see
> recognize patterns in the data. However, there may be other arrangements
> and other formats that are as useful or possibly more useful but which
> don't ordinarily evoke human recognition of such patterns.
>
> Using Fourier transform theory, we can transform that data into the
> wave-number domain and for certain sets of data, that format might be more
> useful than the typical space-domain format.  For example, one can perform
> a forward Fourier transform on a set of image data, add a watermark in the
> wave-number domain, and then perform an inverse Fourier transform back to
> the space domain. At that point, the watermark will still be there, but it
> will be hidden. If the image is later used in violation of a copyright, as
> long as the space-domain version has not been modified modified (a very big
> IF), it can be transformed back into the wave-number domain to expose the
> hidden watermark.
>
> Along these same lines, transforms that are very similar to Fourier
> transforms are commonly used in the compression of image data and humans
> aren't expected to view those data as pictures in the transformed state.
>
> One interesting way to think about image data is as a set of 3D vectors in
> a 3D world where the axes are red, green, and blue instead of the typical
> x, y, and z. When thinking along those lines, each pixel is a vector having
> magnitude and direction. Lots of interesting things can be done with
> vectors, such as addition, subtraction, dot products, cross products, etc.
> Could these techniques be used to extract information from image data and
> present that information in a format that doesn't require sight? I don't
> know. I'm asking the question just to get you thinking.
>
> This group is advertised as being "Blind Math list for those interested in
> mathematics." Think about image data as fertile ground for the application
> of innovative mathematical concepts -- not just data for use to create
> pictures for sighted people.
>
> You can also think about the data in an image file as describing three
> different 3D surfaces which may, or may not be correlated with one another.
> If the data is represented in an RGB format, it can be thought of as
> representing a red surface, a green surface, and a blue surface. If the
> data is represented in an HSB or HSV format, it can be thought of as
> representing a hue surface, a saturation surface, and a brightness surface.
>
> What would you get if you were to convolve a red surface with a hue
> surface. I don't have the slightest idea. Probably garbage! But then again,
> maybe not garbage. Despite the low odds, such an operation might produce
> something useful.
>
> What would you get if you were to subtract the vector representing each
> pixel from each of the eight neighboring pixel vectors and keep the vector
> difference with the greatest magnitude. I do know the answer to that
> question and the result is somewhat useful, but I will "leave it as an
> exercise for the student" to think about it.
>
> What would you get if you were to perform a two-dimensional Fourier
> transform on the 3D hue surface, set the transform results near the origin
> to zero, and then perform an inverse transform back to the hue-space
> domain? Would the result be useful? I don't know. I have never done it.
>
> These questions represent only a few of the many mathematical operations
> that can be performed on existing image data in an attempt to extract
> useful information that may not require sight to be useful.
>
> Food for thought.
>
> Dick Baldwin
>
> --
> Richard G. Baldwin (Dick Baldwin)
> Home of Baldwin's on-line Java Tutorials
> http://www.DickBaldwin.com
>
> Professor of Computer Information Technology
> Austin Community College
> (512) 223-4758
> mailto:Baldwin at DickBaldwin.com
> http://www.austincc.edu/**baldwin/ <http://www.austincc.edu/baldwin/>
> ______________________________**_________________
> Blindmath mailing list
> Blindmath at nfbnet.org
> http://nfbnet.org/mailman/**listinfo/blindmath_nfbnet.org<http://nfbnet.org/mailman/listinfo/blindmath_nfbnet.org>
> To unsubscribe, change your list options or get your account info for
> Blindmath:
> http://nfbnet.org/mailman/**options/blindmath_nfbnet.org/**
> mwhapples%40aim.com<http://nfbnet.org/mailman/options/blindmath_nfbnet.org/mwhapples%40aim.com>
>
> ______________________________**_________________
> Blindmath mailing list
> Blindmath at nfbnet.org
> http://nfbnet.org/mailman/**listinfo/blindmath_nfbnet.org<http://nfbnet.org/mailman/listinfo/blindmath_nfbnet.org>
> To unsubscribe, change your list options or get your account info for
> Blindmath:
> http://nfbnet.org/mailman/**options/blindmath_nfbnet.org/**
> baldwin%40dickbaldwin.com<http://nfbnet.org/mailman/options/blindmath_nfbnet.org/baldwin%40dickbaldwin.com>
>



-- 
Richard G. Baldwin (Dick Baldwin)
Home of Baldwin's on-line Java Tutorials
http://www.DickBaldwin.com

Professor of Computer Information Technology
Austin Community College
(512) 223-4758
mailto:Baldwin at DickBaldwin.com
http://www.austincc.edu/baldwin/