28. Introduction to Image Processing

Image processing is a large and important topic that is all about using computers to process and manipulate images. Here are just a few of the problems that image processing deals with:

• Converting a color image to black and white or grayscale. This is very important in printing. For example, newspapers often show pictures in black and white, and so must convert color images to black and white.
• Applying special effects to images, such as sharpening, blurring, scaling, etc.
• Removing “noise” from images. For example, little “specks” and visual imperfections occasionally appear in pictures, and it is often possible to automatically remove them.
• Recognizing image features, such as faces or text. For instance, many digital cameras have facial recognition software that puts a box around faces.
• Efficiently compressing images into small files so that they can be more easily processed. This is especially important in video begin sent over a network, where you might have to send dozens of individual pictures (i.e. frames of the video) a second.

Image processing applications can be computationally intensive because typically every pixel of an image must be processed. For example, a 640x400 pixel image has 256,000 pixels, and if you wanted to, say, convert each pixel to black and white, then you must process all 256,000 pixels. If you are processing an image in real time (e.g. the output of a webcam), then things are even tougher because you might have to process, say, 24 images per second. Clearly, speed is important in real-time image processing.

In these notes we’ll see a few simple yet useful applications of image processing in Processing. We’ll use this image:

These notes are in part based on Intro to Image Processing notes on the Processing website.

28.1. Displaying an Image

First, lets see how to display a single image:

PImage img;

void setup() {
  img = loadImage("kids.jpg");
  size(img.width, img.height);
}

void draw() {
  background(0);

  image(img, 0, 0);
}

Recall that images are placed according to their upper-left corner.

28.2. Playing with tint

The easiest way to process an entire image in processing is to use the tint function:

PImage img;

void setup() {
  img = loadImage("kids.jpg");
  size(img.width, img.height);
}

void draw() {
  background(0);

  float g = map(mouseY, 0, 500, 0, 255);
  tint(g);
  image(img, 0, 0);
}

Here we pass one number, g, to tint that controls the image’s brightness. If you pass in a second value, you can control the transparency:

PImage img;
PImage bg;

void setup() {
  img = loadImage("kids.jpg");
  bg = loadImage("bg.jpg");

  size(bg.width, bg.height);
}

void draw() {
  background(bg);

  float g = map(mouseY, 0, 500, 0, 255);
  tint(255, g);
  image(img, 0, 0);
}

By passing in three parameters, you can control the amount of red, green, and blue for each pixel:

PImage img;

void setup() {
  img = loadImage("kids.jpg");
  size(img.width, img.height);
}

void draw() {
  tint(255, 0, 0);      // just red
  //tint(255, 0, 0);      // just green
  //tint(255, 0, 0);      // just blue
  //tint(100, 10, 230);   // a little bit of red, a tiny bit of
                          // green, and a lot of blue
  image(img, 0, 0);
}

28.3. How to Access Image Pixels in Processing

The tint function is useful, but many image processing algorithms need to work at the pixel level. That is, they scan through an image one pixel at a time, changing the pixels as they go.

For example, one way to convert a color image to black and white is to look at every pixel and change it to white if it is bright enough, and to black otherwise.

Every PImage object contains an array of pixels that you can access. An array is like an ArrayList, but simpler and with more compact notation for accessing its elements.

For example, suppose you have tiny 3x3 image that we’ll call img:

a b c
d e f
g h i

This little image contains exactly 9 pixel. The pixels are stored in an array like this:

             0   1   2   3   4   5   6   7    8
            --- --- --- --- --- --- --- --- ---
img.pixels | a | b | c | d | e | f | g | h | i |
            --- --- --- --- --- --- --- --- ---

The individuals pixels are accessed using square-bracket notation, e.g. img.pixels[0] is pixel a, img.pixels[1] is pixel b, and so on.

To use img.pixels in your programs you must call loadPixels and updatePixels:

img.loadPixels();

// ... code that processes img.pixels

img.updatePixels();

For example, this code converts a color image to grayscale:

img.loadPixels();

int i = 0;
while (i < img.pixels.length) {
  color c = img.pixels[i];
  float r = red(c);
  float g = green(c);
  float b = blue(c);
  img.pixels[i] = color((r + g + b) / 3);  // set the pixel to a shade of gray
                                           // proportional to the brightness
                                           // of the color
  ++i;
}

img.updatePixels();

The resulting image now consists of 256 different shades of gray, and so is ready to be printed on a black-and-white printer.

28.4. Processing Every Pixel of an Image

Here’s a complete program that is useful for experimenting with pixel- processing:

PImage img;

void setup() {
  img = loadImage("kids.jpg");
  size(img.width, img.height);
  println(img.width * img.height + " pixels");
}

void draw() {
  image(img, 0, 0);
}

void mousePressed() {
  println("mouse pressed");
  img.loadPixels();
  int i = 0;
  while (i < img.pixels.length) {
    color c = img.pixels[i];
    float r = red(c);
    float g = green(c);
    float b = blue(c);

    // the code below is the only code you should change

    img.pixels[i] = color((r + g + b) / 3);  // set the pixel's new color

    // end of code to change

    ++i;
  }
  img.updatePixels();
}

As it stands, the program converts img to grayscale when you click the mouse. Here are a few other ideas:

• Show only a single color component of each pixel, e.g.:

  img.pixels[i] = color(r, 0, 0);
  

• Re-arrange the color components, e.g.:

  img.pixels[i] = color(g, b, r);
  

• “Invert” the components, e.g.:

  img.pixels[i] = color(255 - r, 255 - g, 255 - b);
  

• Make pure black and white, e.g.:

  float br = (r + g + b) / 3.0;  // approximate brightness
  if (br < 128) {
    img.pixels[i] = color(0);
  } else {
    img.pixels[i] = color(255);
  }

  You can use colors other than black and white if you like, e.g.:

  float br = (r + g + b) / 3.0;  // approximate brightness
  if (br < 128) {
    img.pixels[i] = color(255, 255, 0);
  } else {
    img.pixels[i] = color(244, 238, 224);
  }