Processing, With Sound
Processing
The graphics capabilities of modern computers are truly amazing. Whether you're viewing an animated Mandelbrot fractal, watching a DVD, designing a highly-detailed 3D image in a CAD program, or just playing a contemporary game, impressive graphics are the rule. Until recently the creation of stunning displays depended on deep familiarity with programs and coding environments that seemed to require a Master's degree in digital image processing and an arcane knowledge of video hardware. In fact, the old ways of creating complex graphics really were almost that demanding. But the old ways are always steadily morphing into the new ways, and the new ways are bringing greater capabilities to the normal user, even in specialized fields such as computer graphics and digital signal processing.
Processing represents the new way with a vengeance. It's a programming language designed especially for beginners in the domain of computer graphics, yet it is powerful enough to attract the attentions of advanced artists and coders. Note that Processing is a graphics programming language, not a graphical language. You write lines of code in a text editor and execute that code in the Processing environment. If your code passes inspection, a display window opens and runs the coded routines. Errors are reported in a log panel, and the offending code is highlighted in the text editor. Experienced programmers will recognize Processing as an Integrated Development Environment (IDE) for the Processing programming language. Indeed, the main center for activity in the language is called the Processing Development Environment, i.e. the PDE.
Before going further, I suggest a visit to the Exhibition for a look at what's been done already with the system. Processing appeals to a wide variety of users, and the Exhibition shows off that variety with some impressive examples.
Installation And Configuration
You'll need Sun's Java SDK >= 1.5 to run Processing. The Java JRE itself is insufficient, so be sure to install the full SDK. Only the official Sun package is explicitly supported, there are no guarantees if you insist on using other versions of Java.
Processing is not available through the official software repositories for Ubuntu 10.04 (32-bit), the test system for this report. I visited the Processing site, downloaded the tarball for version 1.2.1, unpacked it in my $HOME directory, and entered my newly created $HOME/processing-1.2.1 directory. A shell script there starts the program :
$ sh processing
Alternately you can use chmod to make the script executable :
$ chmod +x processing $ ./processing
After starting the script you'll probably have to wait a few moments while Processing initializes itself. When everything is place you'll see the Processing development environment GUI shown in Figure 1. By default the system opens with a new sketch - a Processing work file - that's ready for input.
Processing has been designed with a clever "hook" that's almost guaranteed to attract the user. In its most basic form the language begins with the principle that a single line of code should make something happen on-screen. For example, running the following code will create a circle 80 pixels wide/high :
ellipse(50, 50, 80, 80)
The first two values locate the circle in its display window 50 pixels from the left and 50 pixels down from the top of the window. So begins our journey into Processing. As we advance we learn how to animate, color-fill, and multiply our images. A part of the art of Processing is the generation of great complexity from simple components, so the attraction at the basic stage is important for new users. Accomplishment fosters confidence, and Processing succeeds beautifully at the primary stages of language acquisition.
The next example from the Getting Started book shows off the same line of code used in a larger program :
void setup() { // Defines the display window parameters.
size(480, 120);
smooth();
}
void draw() { // Defines an action taken if a mouse-button is pressed.
if (mousePressed) {
fill(0);
} else {
fill(255);
}
ellipse(mouseX, mouseY, 80, 80);
}
The formerly fixed position of our simple ellipse is now subject to the movement of the mouse, i.e. if the mouse pointer is in the display window the circle follows the pointer when a mouse button is pressed. The example also nicely demonstrates various conventions in a typical Processing program. The setup() and draw() functions are usually present - I'm not sure yet if they're absolutely necessary - and our ellipse() function is now subject to the process within the draw() function. Interactivity is the norm for many Processing programs, and the example gives us a look at how simple it can be to design interactive code. Of course things can get considerably more complicated, but I leave further exploration to the reader.
Incidentally, the PDE is a pleasure to use. The package includes many excellent examples accessible from the IDE's Files menu, thus inviting the new user to explore Processing at will. At first I took a structured approach to the system by reading Getting Started with Processing and running its examples, all conveniently available from the Files menu. More recently I've become bolder, pillaging existing code and writing little sketches for my own use, learning by doing in an easily managed environment. The language syntax reminds me of Python - with a little C for spice - but I hasten to add that my impression is that of an amateur linguist. At any rate, I soon became comfortable with the Processing language and its environment, largely thanks to the inclusion of great example code.
The Library Is Open
A variety of contributed libraries extend the possibilities of Processing into the domains of networking, OpenGL rendering, animation, video, hardware interfacing, and more. Some libraries expand Processing's capabilities into the audio realm. The default package includes the Minim library, a set of audio functions that include play/record controls, an FFT display of an incoming signal, a bandpass filter, and templates for user-defined effects and signal creation. Minim employs the JavaSound API as its audio backend, a fact that may or may not be a feature. JavaSound's design restrictions disallow any support for JACK, and its soundcard detection is limited to cards with on-board mixers, ruling out the automatic deployment of my M-Audio systems. Within this limitation Minim works as advertised, and its examples provide useful starter code for audio programming in Processing.
Csounders will be pleased to know about Jacob Joaquin's csoundo, a library that connects the graphics strength of Processing with the audio power of Csound. Yes, you need to know some Csound to fully exploit csoundo's capabilities, but the installation package includes some helpful examples to provide clues to the connection. Fans of SuperCollider3 will be equally happy about oscP5 and p5_sc, libraries to connect Processing to that excellent audio synthesis/composition environment.
I've installed the Ess and SoundCipher audio libraries, the latter of which is especially promising. Most of its examples worked perfectly, and it is a combined audio/MIDI extension for Processing. The Beads library is similarly promising, and I've also discovered Sonia, an audio extension based on Phil Burk's cool JSyn plugin.
Most of these extensions employ the services of the Javasound API. That fact shouldn't concern normal desktop systems, but as I mentioned, if you have a pro-audio device as your primary card you may have problems with Javasound. The API sets the default audio device to the first one it discovers that has an on-board mixer. Unfortunately, my M-Audio Delta 66 includes no such mixer and is passed over by Javasound. However, there is a solution to this problem, at least there's one for Minim.
If your primary audio device - the one ALSA calls Card #0 - is without a mixer subsystem you can still force Javasound to use it. The following Minim code sets the output audio device to the primary device in my system :
// These declarations should be made before any code blocks. Minim minim; // For the Mixer and Mixer.Info objects. import javax.sound.sampled.*; // For setting output format, channels, buffers, etc. AudioOutput out; // Prepares an array for the data returned by mixer.Info. Mixer.Info[] mixer.Info; // Add to the setup() routine *before* the loadFile statement. // Select mixer device here. mixerInfo = AudioSystem.getMixerInfo(); int mixerIndex = 0; // My M-Audio system. Mixer mixer = AudioSystem.getMixer(mixerInfo[mixerIndex]); minim.setOutputMixer(mixer); out = minim.getLineOut(Minim.STEREO, 2048); // Set channels and buffer size.
The most recent Minim includes an example (setOutputMixer) that creates a GUI for listing and selecting your output mixer device (Figure 2). I'd like to bundle that code along with the snippet above for a neat and easy way to re-use it all as a function, but my Processing skills are still at the lowest level. I'll be working on solutions, and of course I welcome suggestions from my Processing-savvy readers. Meanwhile I'll continue to explore the many possibilities of Minim, SoundCipher, csoundo, and any other audio extensions I find. To be clear, the mixer problem is likely to occur only if you have a pro-audio system. I've had no troubles with Processing and its audio libraries on my laptop with its consumer-grade sound hardware.
One word more regarding the Minim library: Processing includes Minim with its bundled externals, but version 2.1.0 is available as a beta release with some neat new features and example code. If you decide to run the newer version be sure to install it to your $HOME/sketchbook/libraries/ directory, that's where Processing now expects to find all contributed libraries.
The news regarding broader MIDI support seems not so good for Linux users of Processing. I installed the proMIDI, rwmidi, and MidiBus libraries but had no success with their examples. Some appear to run okay but without MIDI output - which is kind of the point - while others generate error messages that are a bit difficult to decipher (I'm still trying to determine what is meant by "The type PApplet is ambiguous"). I rebuilt the needed jar files, to no avail, so I've joined the Processing forums and hope to find enlightenment there soon. [Update:] Alas, the explorer is on his or her own when it comes to Processing and MIDI. I've received no replies from the forums, but there are other relevant messages I've yet to study. However, it does seem to me that Processing could use a guide to the employment of its internal and external audio/MIDI capabilities.
I found two library extensions designed for music composition within Processing. Tactu5 adds a variety of algorithmic music functions to the system - in realtime, no less - and jm-Etude is a helper library for the creation and performance of scores in the jMusic format. Both libraries depend upon external resources for the actual sound synthesis, e.g. a standalone softsynth or the Java Sound Synthesizer. Alas, I haven't figured out how to get Tactu5 connected to anything yet, but jm-Etude automatically routes its output to Java's internal synth. It's pretty cool - I import the required library, write my jm-Etude code, and it runs and plays. Nice.
Processing includes some nice facilities for the addition of text in your sketches, so of course it must also have its own text-to-speech library. Behold, ttslib, a wrapper for the FreeTTS Java-based speech synthesizer. Alas, I haven't had time to test it, but it looks like it might be fun (and useful, of course).
I've only started my explorations of Processing, and I'm already overwhelmed. Other than merely running example sketches, I haven't even started to work on the conjunction of sound and visuals. Obviously there are enough extensions to Processing to keep me busy with it for a long time to come. Currently I'm focused on the possibilities in csoundo (of course), SoundCipher, Minim, and jm-Etude, but who knows where my explorations will lead.
And for the benefit of anyone seeking a quick overview of my progress with audio/MIDI libraries for Processing 1.2.1 on Ubuntu 10.04 :
- Audio
- Music composition
- MIDI
But Wait, That's Not All
Long-time readers of my articles are likely to be aware of my forays into the world of Linux video, so of course I looked for video tools for Processing. Unfortunately for Linux users the designers of Processing decided upon the proprietary Quicktime libraries from Apple. It's true that there is a Linux-friendly libquicktime project that aims towards full compatibility, but as far as I can tell it is not useful for Processing in its current condition. Thankfully we have the excellent GSVideo and JMyron on Linux libraries that provide basic and advanced video capabilities for Processing under Linux (Figure 3). I've tested both packages with good results and I look forward to further explorations.
I'm also starting to take an interest in the Arduino project, particularly because there is a strong connection between the Processing and Arduino projects. I haven't been involved in any experiments with hardware for a long while, but the Arduino project is just too darned attractive for the musician with a taste for unique control surfaces. If I do get into I'll certainly be sure to post the results of my investigations.
Documentation
There's plenty of documentation for Processing. Web-based material is available on the forums at the Processing home site, on media channels such as YouTube and Vimeo, and on many of its users' personal pages. Processing is well-represented in hard-copy too. If you're completely new to the fields of computer programming and computer graphics I recommend Getting Started With Processing, an outstanding introduction written the system's primary developers. After going through that excellent tutorial you should be ready to tackle larger projects and the larger books on Processing. Google is your friend for looking up more tutorials and examples, and the OpenProcessing site is another must-see location for anyone interested in advancing their knowledge of Processing. I also recommend the Learning Processing site, an excellent resource for beginners in Processing.
Processing Warts
The system does have a few problematic parts. Processing has gone through many significant changes throughout its history, with the predictable resulting inconsistencies between versions. For example, I'm using Sun's Java SDK 1.6, the latest and greatest public release that also happens to be the version installed by default on Ubuntu 10.04. Unfortunately some of the contributed library jar files have been compiled with Java 1.5, and their binary compatibility is not guaranteed for 1.6. More sadly, not all extensions include their source code, and that code matters when you start receiving error messages regarding specific functions and language primitives. The messages vary in their utility - some reports suggest fixes, others offer only terse descriptions of problems. The Processing home site includes some helpful resources for users venturing into code repair, but not all errors are covered.
It seems to me that for all its many resources Processing lacks a well-defined "middle ground", i.e. a well-understood and well-documented path that proceeds logicaly from the basics and instructs the user in the ways of constructing larger, more complex sketches. Basic examples abound, as do advanced art works and other impressive displays, but there seems to be little instructive material that takes the no-longer-new user to the next level of complexity. Of course, this opinion is based on my very brief acquaintance with Processing, and I welcome readers' comments and advice on the topic of learning materials for the intermediate-level student.
Comparisons
Miller Puckette's Pd is an audio programming environment with hooks into OpenGL via the GEM library. Pd is a graphical programming system, i.e. icons representing synthesis and/or other signal processing components are wired together into networks on a "canvas" display panel. The network can be played by external devices through MIDI and OSC or by Pd's internal sequencing tools. Pd's UI is not so shiny as the Processing IDE, but in my opinion Pd is easier to use when conjoining audio and graphics data streams.
Jean-Pierre Lemoine's AVSynthesis works along different lines. Its graphics processing is based on the JOGL OpenGL shaders, and its image processing role is limited to the creation of sequences of image transformations. AVSynthesis operates in various performance modes, including a realtime interactive mode. All audio in AVSynthesis is handled by Csound, but its implementation is indirect - the program includes its own Csound-based instruments and signal processors, and the user does not write code. Full documentation is available, and demonstration videos can be found on YouTube and Vimeo. However, AVSynthesis is remarkably deep and even with its GUI it is not so quickly learned as Processing.
Fluxus is Dave Griffiths' multimedia production environment especially designed for live coding, an event described by its Wikipedia page as
... the process of writing software in realtime, as a form of improvised time-based art ... particularly prevalent in computer music, combining algorithmic composition with improvisation. Live coding is also in use to improvise video animation, for example using the Fluxus environment.
Processing sketches are certainly capable of interactive operation, but Fluxus works at another level. The user/coder becomes a performer, and Fluxus is all about performance. Its language base - a dialect of Scheme - is not terribly difficult to learn, and the system includes documentation in a variety of formats, including a series of instructional on-line videos.
Outro
I'm eager to reach higher levels of accomplishment with Processing, but I can tell it'll take some time. Fortunately I have access to an abundance of tools for learning more about the system, including a wealth of code out there on the net, so I don't lack for guidance. Alas, time is always working against me, and the craft is long to learn, so I'll end this profile and get back to my studies. I'll return here in a couple of weeks with more news from the ever-advancing world of Linux multimedia. In the interim, please feel free to post reports of your own experience with Processing, and be sure to let us know of any pointers to more information about Processing, especially links to audio/MIDI libraries and example code. We shall all advance together.
Recent Articles
