[BlindRUG] Sonification of plots in R - an experiment

[Godfrey, Jonathan]

Hi Robin et al.,

There have been a few discussions about this idea before now, but your work looks like it has gone further than other attempts I've seen to date. Good stuff.
I'll send you the work Karen Vines and I did  off-list.

I think there are a few lessons from our experiences I'd like to share with you and the group.

We found it difficult to get a great handle on what the sonified result could show. We couldn't really tell the difference between a normal curve and a variety of other symmetric unimodal distributions for example, especially if we used the exact observations instead of  a smooth curve.

I'm interested in using any means, including sound, to help show a blind person what appears in a graph window. I'm sure I have the components that deliver the contents of a graph window but I haven't gone as far as the sonification yet. This should be fairly easy to do once I get the text descriptions for some content  sorted.

The main problem I encounter is that the standard plot() command that generates a scatter plot in many instances is a family of commands and the resulting graph windows can vary considerably. It would be much better if we could convince the entire R community to use xyplot() from the lattice package or some such, but that's not likely!

I would prefer to see your functions use (x, y) or (y~x) instead of the current (y, x) as they are more standard practice and match the options for plot) etc.

I'll write more on this topic after e hear from others. Off to do some teaching now.
Jonathan



From: BlindRUG [mailto:blindrug-bounces at nfbnet.org] On Behalf Of Williams, Robin via BlindRUG
Sent: Wednesday, 6 April 2016 3:48 a.m.
To: BlindRUG at nfbnet.org
Cc: Williams, Robin
Subject: [BlindRUG] Sonification of plots in R - an experiment

Dear all,

In collaboration with a colleague here at the University of Exeter, I've started to experiment with the idea of producing sound scapes of plots. The idea was motivated by my current work, which requires me to look at lines of best fit (often using polynomial smoothers) through many datapoints. Essentially, we can produce a soundscape of a plotted function, or plotted data points. The idea is similar to the Mathtrax software produced by NASA some years ago, except we can now harness the power of R. The code is based on the tuner package. Essentially, we create a signal that represents the value of a plotted function using Fourier series, and output the wave to a .wav file. This can be played from within R, using a software such as Windows Media Player under Windows, or mplayer under linux. You can choose from four forms of wave which all sound quite different to me. Datapoints can be indicated with pulses, and tick marks on the x axis are indicated with a short blast of the distinctive sawtooth wave. The soundscape moves from left to write, in order to mimic moving along the x axis. So far, I've used the sonify() function to examine distributions whose PDF I was unfamiliar with, and to look at a nonparametric approximation to a line of best fit estimated from data. I've had to look at many of these plots over the years, which have often required sighted assistance, and so this represents quite a breakthrough in terms of working efficiency.
The code can be obtained from here:

https://raw.githubusercontent.com/sieste/sonic-plot/master/Rsonify/sonify.R
For example, you could try
X=seq(-5,-5,.01)
Y=sin(x)
Sonify(y,x,ticks=seq(-4,4,1),pulse_len=0) # without pulses for the datapoints
Sonify(y,x,ticks=seq(-4,4,1),pulse_len=0.2) # to hear pulses at the datapoints

At this stage I would like to gather any feedback, and suggestions as to how the script could be made more useful still. Please send me any bugs you find in the code.

Best wishes
Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nfbnet.org/pipermail/blindrug_nfbnet.org/attachments/20160405/ef3ce6e3/attachment.html>