Book Image

OpenCV with Python Blueprints

By : Michael Beyeler, Michael Beyeler (USD)
Book Image

OpenCV with Python Blueprints

By: Michael Beyeler, Michael Beyeler (USD)

Overview of this book

Table of Contents (14 chapters)
OpenCV with Python Blueprints
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Creating a black-and-white pencil sketch


In order to obtain a pencil sketch (that is, a black-and-white drawing) of the camera frame, we will make use of two image blending techniques, known as dodging and burning. These terms refer to techniques employed during the printing process in traditional photography; photographers would manipulate the exposure time of a certain area of a darkroom print in order to lighten or darken it. Dodging lightens an image, whereas burning darkens it.

Areas that were not supposed to undergo changes were protected with a mask. Today, modern image editing programs, such as Photoshop and Gimp, offer ways to mimic these effects in digital images. For example, masks are still used to mimic the effect of changing exposure time of an image, wherein areas of a mask with relatively intense values will expose the image more, thus lightening the image. OpenCV does not offer a native function to implement these techniques, but with a little insight and a few tricks, we will arrive at our own efficient implementation that can be used to produce a beautiful pencil sketch effect.

If you search on the Internet, you might stumble upon the following common procedure to achieve a pencil sketch from an RGB color image:

  1. Convert the color image to grayscale.

  2. Invert the grayscale image to get a negative.

  3. Apply a Gaussian blur to the negative from step 2.

  4. Blend the grayscale image from step 1 with the blurred negative from step 3 using a color dodge.

Whereas steps 1 to 3 are straightforward, step 4 can be a little tricky. Let's get that one out of the way first.

Note

OpenCV 3 comes with a pencil sketch effect right out of the box. The cv2.pencilSketch function uses a domain filter introduced in the 2011 paper Domain transform for edge-aware image and video processing, by Eduardo Gastal and Manuel Oliveira. However, for the purpose of this book, we will develop our own filter.

Implementing dodging and burning in OpenCV

In modern image editing tools, such as Photoshop, color dodging of an image A with a mask B is implemented as the following ternary statement acting on every pixel index, called idx:

((B[idx] == 255) ? B[idx] : min(255, ((A[idx] << 8) / (255-B[idx]))))

This essentially divides the value of an A[idx] image pixel by the inverse of the B[idx] mask pixel value, while making sure that the resulting pixel value will be in the range of [0, 255] and that we do not divide by zero.

We could translate this into the following naïve Python function, which accepts two OpenCV matrices (image and mask) and returns the blended image:

def dodgeNaive(image, mask):
    # determine the shape of the input image
    width,height = image.shape[:2]

    # prepare output argument with same size as image
    blend = np.zeros((width,height), np.uint8)

    for col in xrange(width):
        for row in xrange(height):

            # shift image pixel value by 8 bits
            # divide by the inverse of the mask
            tmp = (image[c,r] << 8) / (255.-mask)

            # make sure resulting value stays within bounds
            if tmp > 255:
                tmp = 255
            blend[c,r] = tmp
    return blend

As you might have guessed, although this code might be functionally correct, it will undoubtedly be horrendously slow. Firstly, the function uses for loops, which are almost always a bad idea in Python. Secondly, NumPy arrays (the underlying format of OpenCV images in Python) are optimized for array calculations, so accessing and modifying each image[c,r] pixel separately will be really slow.

Instead, we should realize that the <<8 operation is the same as multiplying the pixel value with the number 2^8=256, and that pixel-wise division can be achieved with the cv2.divide function. Thus, an improved version of our dodge function could look like this:

import cv2

def dodgeV2(image, mask):
    return cv2.divide(image, 255-mask, scale=256)

We have reduced the dodge function to a single line! The dodgeV2 function produces the same result as dodgeNaive but is orders of magnitude faster. In addition, cv2.divide automatically takes care of division by zero, making the result 0 where 255-mask is zero.

Now, it is straightforward to implement an analogous burning function, which divides the inverted image by the inverted mask and inverts the result:

import cv2

def burnV2(image, mask):
    return 255 – cv2.divide(255-image, 255-mask, scale=256)

Pencil sketch transformation

With these tricks in our bag, we are now ready to take a look at the entire procedure. The final code will be in its own class in the filters module. After we have converted a color image to grayscale, we aim to blend this image with its blurred negative:

  1. We import the OpenCV and numpy modules:

    import cv2
    import numpy as np
  2. Instantiate the PencilSketch class:

    class PencilSketch:
        def __init__(self, (width, height), bg_gray='pencilsketch_bg.jpg'):

    The constructor of this class will accept the image dimensions as well as an optional background image, which we will make use of in just a bit. If the file exists, we will open it and scale it to the right size:

    self.width = width
    self.height = height
    
    # try to open background canvas (if it exists)
    self.canvas = cv2.imread(bg_gray, cv2.CV_8UC1)
    if self.canvas is not None:
        self.canvas = cv2.resize(self.canvas, (self.width, self.height))
  3. Add a render method that will perform the pencil sketch:

    def renderV2(self, img_rgb):
  4. Converting an RGB image (imgRGB) to grayscale is straightforward:

    img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2GRAY)

    Note that it does not matter whether the input image is RGB or BGR.

  5. We then invert the image and blur it with a large Gaussian kernel of size (21,21):

    img_gray_inv = 255 – img_gray
    img_blur = cv2.GaussianBlur(img_gray_inv, (21,21), 0, 0)
  6. We use our dodgeV2 dodging function from the aforementioned code to blend the original grayscale image with the blurred inverse:

    img_blend = dodgeV2(mg_gray, img_blur)
    return cv2.cvtColor(img_blend, cv2.COLOR_GRAY2RGB)

The resulting image looks like this:

Did you notice that our code can be optimized further?

A Gaussian blur is basically a convolution with a Gaussian function. One of the beauties of convolutions is their associative property. This means that it does not matter whether we first invert the image and then blur it, or first blur the image and then invert it.

"Then what matters?" you might ask. Well, if we start with a blurred image and pass its inverse to the dodgeV2 function, then within that function, the image will get inverted again (the 255-mask part), essentially yielding the original image. If we get rid of these redundant operations, an optimized render method would look like this:

def render(img_rgb):
    img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (21,21), 0, 0)
    img_blend = cv2.divide(img_gray, img_blur, scale=256)
    return img_blend

For kicks and giggles, we want to lightly blend our transformed image (img_blend) with a background image (self.canvas) that makes it look as if we drew the image on a canvas:

if self.canvas is not None:
    img_blend = cv2.multiply(img_blend, self.canvas, scale=1./256)
return cv2.cvtColor(img_blend, cv2.COLOR_GRAY2BGR)

And we're done! The final output looks like what is shown here: