Linear algebra and digital image processing. Part IV. Image editor.

In the three previous posts we explained basic concepts about the use of linear algebra in the digital image processing: how an image can be expressed as a matrix, the concept of digital filters and their representation as matrix operations, and finally the affine transformations, their geometric interpretation and their algebraic representation using a 3x3 matrix.

In this post, the last one of the series, we'll show an implementation in JavaScript of all the theory we have seen. We also explain how you can use the HMTL5 canvas object to make your own implementation of the concepts of image processing.

A simple JavaScript image editor

When moving the mouse cursor over the image, you can see, under the image, the 5x5 matrix of pixels surrounding the pixel at the cursor position. Use options to apply filters and make transformations. If you want, you can choose your own image and try.

HTML5 Canvas

HTML5 element <canvas> allows you to render graphics in a powerful way using Javascript.

<canvas id="mycanvas" width="200" height="200">

The canvas element has a DOM method called getContext, used to obtain the rendering context and its drawing functions.

var mycanvas = document.getElementById('mycanvas');
var context = mycanvas.getContext('2d');

Although other contexts may provide different types of rendering, as WebGL, which uses a 3D context based on OpenGL ES, here we focus on the 2D rendering context.

Using the properties of the rendering context object you can, for example, set the line width, style and color; and using its methods you can draw lines, curves, arcs and rectangles. You can also make some transformations like scaling, rotation, translation or any other affine transformation by means of its matrix representation.

You can find the specification for the 2D context for the HTML canvas element at https://www.w3.org/TR/2dcontext/, and a good tutorial at https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial

Drawing images and manipulating pixels

In order to draw images onto the canvas, the method drawImage is provided. This method has three variants depending on its parameters. Let's look one of them:

ctx.drawImage(image, dx, dy, dWidth, dHeight);

Here, image is the element to draw into the context, and it permits any canvas image source such as an HTMLImageElement, an HTMLVideoElement, an HTMLCanvasElement or an ImageBitmap; dx and dy are the X and Y coordinate, respectively, in the destination canvas at which to place the top-left corner of the source image; and dWidth and dHeight are the width and height to draw the image in the destination canvas, which allow scaling of the drawn image.

In order to manipulate the pixels of a canvas, it's provided the method getImageData, which returns an ImageData object that represent the underlying pixel data of an area of a canvas element.

This method requires the following parameters: the x and y coordinates (in pixels) of the upper-left corner of the rectangular area you will copy, as well as the width and height of this area.

Using the property data of the ImageData object, you get a one-dimensional array containing the data in the RGBA order with integer values between 0 and 255 (Red, Green, Blue and Alpha channel. For the alpha channel, 0 is transparent and 255 is fully visible). That means that every 4 elements in the array, you can get a pixel, where the first value is the red channel, the second one is the green channel, the third one is the blue channel, and the last one the alpha channel.

In order to get a better understaing of this array, in the following javascript code we create a two-dimension matrix of pixels from the property data of the ImageData object:

function getImgColorMatrix(pxlData, width, height){
  var matrix = new Array(height);
  for (var i = 0; i < height; i++){
    matrix[i] = new Array(width);
    for (var j = 0; j < width; j++){
      matrix[i][j] = GetPixelColor(pxlData, i, j, width);
    }
  }
  return matrix;
}

function GetPixelColor(pxlData, x, y, width){
  var index = (x * width + y) * 4;
  var pixel = {
    r : pxlData[index],
    g : pxlData[index + 1],
    b : pxlData[index + 2]
  };
  return pixel;
}

Programming the Image Editor

In the image editor tool we used two basic functions, one to filter the image and the other to make affine transformation (even when there are methods for transformations, we decided to make them only by manipulating pixels, in order to demonstrate the theory explained in the previous post).

The function to transform the image receives the rendering context (ctx), the width (w) and height (h) of the image, and the transformation matrix as an array.

As explained in the previous post, affine transformation can be represented using a 3x3 matrix:

As the last row is fixed, a common way to represent the matrix is by mean of the 6-component vector [a, b, c, d, e, f] (this is also the way the transformation matrix is expressed when using the transform method of the rendering context object).

Using the concepts of projective coordinates explained in the previous post, linear filters can be expressed as the product of a 4x4 matrix by the 4-component vector, representing the RGB vector. This way, the following operations:

Can be expressed as:

We used this concept to implement the filter function, which receives the rendering context (ctx), the width (w) and height (h) of the image, and the filter matrix as a one-dimensional array [a, b, c, d, e, f, g, h, i, j, k, l].

var filter = function(ctx, w, h, matrix){
  var imageData = ctx.getImageData( 0, 0, w, h);
  var pxlData = imageData.data;
  for (var i = 0; i < pxlData.length; i += 4){
    var r = pxlData[i];
    var g = pxlData[i + 1];
    var b = pxlData[i + 2];
    pxlData[i]     = matrix[0]*r + matrix[3]*g + matrix[6]*b + matrix[9];
    pxlData[i + 1] = matrix[1]*r + matrix[4]*g + matrix[7]*b + matrix[10];
    pxlData[i + 2] = matrix[2]*r + matrix[5]*g + matrix[8]*b + matrix[11];
  }
  ctx.putImageData(imageData, 0, 0);
}

var transform = function(ctx, w, h, matrix){
  var pxlData = ctx.getImageData( 0, 0, w, h).data;
  var imageData = ctx.createImageData(w, h);
  var newPxlData = imageData.data;
  for (var x = 0; x < w; x++){
    for (var y = 0; y < h; y++){
      var nx = Math.floor(matrix[0]*x + matrix[2]*y + matrix[4]);
      var ny = Math.floor(matrix[1]*x + matrix[3]*y + matrix[5]);
      if (nx >= 0 && nx < w && ny>=0 && ny < h){
        newPxlData[(y*w + x)*4]     = pxlData[(ny*w + nx)*4];
        newPxlData[(y*w + x)*4 + 1] = pxlData[(ny*w + nx)*4 + 1];
        newPxlData[(y*w + x)*4 + 2] = pxlData[(ny*w + nx)*4 + 2];
        newPxlData[(y*w + x)*4 + 3] = pxlData[(ny*w + nx)*4 + 3];
      }
    }
  }
  ctx.putImageData(imageData, 0, 0);
}

Once the pixels have been manipulated, you can use the method putImageData to copy the image data back onto the canvas.

Notice that in the transform function, a new blank ImageData object is created using the createImageData() method. In that method the new object is created with the specified dimensions, but you can use another ImageData as a parameter to create a new one with the same dimensions.

As gamma correction cannot be expressed using matrix multiplication, it has its own function, where ctx, x and y have the same meaning that previous functions, and factor is a parameter with the grade of correction:

var gamma = function(ctx, w, h, factor){
  var imageData = ctx.getImageData( 0, 0, w, h);
  var pxlData = imageData.data;
  for (var i = 0; i < pxlData.length; i += 4){
    pxlData[i]     =  255*Math.pow(pxlData[i] / 255, 1/factor);
    pxlData[i + 1] = 255*Math.pow(pxlData[i + 1] / 255, 1/factor);
    pxlData[i + 2] = 255*Math.pow(pxlData[i + 2] / 255, 1/factor);
  }
  ctx.putImageData(imageData, 0, 0);
}

Following, the list of filters and transformations implemented, as well as the matrices we used. The parameter value is the input obtained from the corresponding slider.

Grayscale conversion

filter(ctx, w, h, [1/3, 1/3, 1/3, 1/3, 1/3, 1/3, 1/3, 1/3, 1/3, 0, 0, 0]);

Color invertion

filter(ctx, w, h, [-1, 0, 0, 0, -1, 0, 0, 0, -1, 255, 255, 255]);

Sepia

filter(ctx, w, h, [0.393, 0.349, 0.272, 0.769, 0.686, 0.534, 0.189, 0.168, 0.131, 0, 0, 0]);

Brightness

filter(ctx, w, h, [1, 0, 0, 0, 1, 0, 0, 0, 1, value, value, value]);

Contrast

var f = (259 * (value + 255)) / (255 * (259 - value));
filter(ctx, w, h, [f, 0, 0, 0, f, 0, 0, 0, f, 128*(1-f), 128*(1-f), 128*(1-f)]);

Gamma correction

gamma(ctx, w, h, value);

Red color adjustment

filter(ctx, w, h, [1, 0, 0, 0, 1, 0, 0, 0, 1, value, 0, 0]);

Green color adjustment

filter(ctx, w, h, [1, 0, 0, 0, 1, 0, 0, 0, 1, 0, value, 0]);

Blue color adjustment

filter(ctx, w, h, [1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, value]);

Vertical flip

transform(ctx, w, h, [1, 0, 0, -1, 0, h-1]);

Horizontal flip

transform(ctx, w, h, [-1, 0, 0, 1, w-1, 0]);

Scale horizontally (scale and centre)

if (value != 0){
  transform(ctx, w, h, [1/value, 0, 0, 1, w*(1 - 1/value)/2, 0]);
}

Scale vertically (scale and centre)

if (value != 0) {
  transform(ctx, w, h, [1, 0, 0, 1/value, 0, h*(1 - 1/value)/2]);
}

Move horizontally

transform(ctx, w, h, [1, 0, 0, 1, -w*value/100, 0]);

Move vertically

transform(ctx, w, h, [1, 0, 0, 1, 0, -h*value/100]);

Rotate (around the centre of the image)

var cos = Math.cos(value*Math.PI/180);
var sin = Math.sin(value*Math.PI/180);
transform(ctx, w, h, [cos, sin, -sin, cos, -cos*w/2 + sin*h/2 + w/2, -sin*w/2 - cos*h/2 + h/2]);

Skew horizontally (around the centre of the image)

var tan = Math.tan(value*Math.PI/180);
transform(ctx, w, h, [1, tan, 0, 1, 0, -tan*w/2]);

Skew vertically (around the centre of the image)

var tan = Math.tan(value*Math.PI/180);
transform(ctx, w, h, [1, 0, tan, 1, -tan*h/2, 0]);

The Blur was included as an example of a non-linear filter, using an implementation obtained from:

http://www.quasimondo.com/BoxBlurForCanvas/FastBlurDemo.html