For now, we have some basic framework for testing image processing and computer vision algorithms. Now it's time to add OpenCV to your project and add your first call to the library. You will learn how to convert UIImage to cv::Mat, and make a call to the C++ library using Objective-C code.
First you should download the OpenCV framework for iOS from the official website at http://opencv.org. In this book, we will use Version 2.4.6. You can use the iOS Simulator to work on this recipe. Source code for this recipe can be found in the Recipe03_LinkingOpenCV folder in the code bundle that accompanies this book.
The following are the main steps to accomplish the task:
Add the OpenCV framework to your project.
Convert image to the OpenCV format.
Process image with a simple OpenCV call.
Convert image back.
Display image as before.
Let's implement the described steps:
We continue modifying the previous project, so that you can use it; otherwise create a new project with
UIImageView. We'll start by adding the OpenCV framework to the Xcode project. There are two ways to do it.You can add the framework as a resource as described in previous recipe. This is a straightforward approach. Alternatively, the framework can be added through project properties by navigating to Project | Build Phases | Link Binary With Libraries. To open project properties you should click to the project name in the Project Navigator area.
Next, we'll include OpenCV header files to our project. In order to do so, we will modify the
Recipe03_LinkingOpenCV-Prefix.pchprecompiled header. To avoid conflicts, we will add the following code to the very beginning of the file, above all other imports:#ifdef __cplusplus #import <opencv2/opencv.hpp> #endif
This is needed, because OpenCV redefines some names, for example,
min/maxfunctions.Set the value of Compile Sources As property as Objective-C++. The property is available in the project settings and can be accessed by navigating to Project | Build Settings | Apple LLVM compiler 4.1 - Language.
To convert the images from
UIImagetocv::Mat, you can use the following functions:UIImage* MatToUIImage(const cv::Mat& image) { NSData *data = [NSData dataWithBytes:image.data length:image.elemSize()*image.total()]; CGColorSpaceRef colorSpace; if (image.elemSize() == 1) { colorSpace = CGColorSpaceCreateDeviceGray(); } else { colorSpace = CGColorSpaceCreateDeviceRGB(); } CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data); // Creating CGImage from cv::Mat CGImageRef imageRef = CGImageCreate(image.cols, //width image.rows, //height 8, //bits per component 8*image.elemSize(),//bits per pixel image.step.p[0], //bytesPerRow colorSpace, //colorspace kCGImageAlphaNone|kCGBitmapByteOrderDefault,// bitmap info provider, //CGDataProviderRef NULL, //decode false, //should interpolate kCGRenderingIntentDefault //intent ); // Getting UIImage from CGImage UIImage *finalImage = [UIImage imageWithCGImage:imageRef]; CGImageRelease(imageRef); CGDataProviderRelease(provider); CGColorSpaceRelease(colorSpace); return finalImage; } void UIImageToMat(const UIImage* image, cv::Mat& m, bool alphaExist = false) { CGColorSpaceRef colorSpace = CGImageGetColorSpace(image.CGImage); CGFloat cols = image.size.width, rows = image.size.height; CGContextRef contextRef; CGBitmapInfo bitmapInfo = kCGImageAlphaPremultipliedLast; if (CGColorSpaceGetModel(colorSpace) == 0) { m.create(rows, cols, CV_8UC1); //8 bits per component, 1 channel bitmapInfo = kCGImageAlphaNone; if (!alphaExist) bitmapInfo = kCGImageAlphaNone; contextRef = CGBitmapContextCreate(m.data, m.cols, m.rows, 8, m.step[0], colorSpace, bitmapInfo); } else { m.create(rows, cols, CV_8UC4); // 8 bits per component, 4 channels if (!alphaExist) bitmapInfo = kCGImageAlphaNoneSkipLast | kCGBitmapByteOrderDefault; contextRef = CGBitmapContextCreate(m.data, m.cols, m.rows, 8, m.step[0], colorSpace, bitmapInfo); } CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), image.CGImage); CGContextRelease(contextRef); }These functions are included into the library starting from Version 2.4.6 of OpenCV. In order to use them, you should include the
ios.hheader file.#import "opencv2/highgui/ios.h"
We won't explain these functions in this recipe, because it requires from readers some knowledge about
CGImageandUIImageclasses; but the use of these methods is really simple. Let's consider a simple example that extracts edges from the image. In order to do so, you have to add the following code to theviewDidLoad()method:- (void)viewDidLoad { [super viewDidLoad]; UIImage* image = [UIImage imageNamed:@"lena.png"]; // Convert UIImage* to cv::Mat UIImageToMat(image, cvImage); if (!cvImage.empty()) { cv::Mat gray; // Convert the image to grayscale cv::cvtColor(cvImage, gray, CV_RGBA2GRAY); // Apply Gaussian filter to remove small edges cv::GaussianBlur(gray, gray, cv::Size(5, 5), 1.2, 1.2); // Calculate edges with Canny cv::Mat edges; cv::Canny(gray, edges, 0, 50); // Fill image with white color cvImage.setTo(cv::Scalar::all(255)); // Change color on edges cvImage.setTo(cv::Scalar(0, 128, 255, 255), edges); // Convert cv::Mat to UIImage* and show the resulting image imageView.image = MatToUIImage(cvImage); } }
Now run your application and check whether the application finds edges on the image correctly.
Frameworks are intended to simplify the process of handling dependencies. They encapsulate header and binary files, so the Xcode sees them, and you don't need to add all the paths manually. Simply speaking, the iOS framework is just a specially structured folder containing include files and static libraries for different architectures (for example, armv7, armv7s, and x86). But Xcode knows where to search for proper binaries for each build configuration, so this approach is the simplest way to link external library on the iOS. All dependencies are handled automatically and added to the final application package.
Usually, iOS applications are written in Objective-C language. Header files have a *.h extension and source files have *.m. Objective-C is a superset of C, so you can easily mix these languages in one file. But OpenCV is primarily written in C++, so we need to use C++ in the iOS project, and we need to enable support of Objective-C++. That's why we have set the language property to Objective-C++. Source files in Objective-C++ language usually have the *.mm extension.
To include OpenCV header files, we use the #import directive. It is very similar to #include in C++, while there is one distinction. It automatically adds guards for the included file, while in C++ we usually add them manually:
#ifndef __SAMPLE_H__ #define __SAMPLE_H__ … #endif
In the code of the example, we just convert the loaded image from a UIImage object to cv::Mat by calling the UIImageToMat function. Please be careful with this function, because it entails a memory copy, so frequent calls to this function will negatively affect your application's performance.
Note
Please note that this is probably the most important performance tip—to be very careful while working with memory in mobile applications. Avoid memory reallocations and copying as much as possible. Images require quite large chunks of memory, and you should reuse them between iterations. For example, if your application has some pipeline, you should preallocate all buffers and use the same memory while processing new frames.
After converting images, we do some simple image processing with OpenCV. First, we convert our image to the single-channel one. After that, we use the GaussianBlur filter to remove small details. Then we use the Canny method to detect edges in the image. To visualize results, we create a white image and change the color of the pixels that lie on detected edges. The resulting cv::Mat object is converted back to UIImage and displayed on the screen.
The following is additional advice.
There is one more way to add support of Objective-C++ to your project. You should just change the extension of the source files to .mm where you plan to use C++ code. This extension is specific to Objective-C++ code.
If you don't want to use UIImage, but want to load an image to cv::Mat directly, you can do it using the following code:
// Create file handle
NSFileHandle* handle =
[NSFileHandle fileHandleForReadingAtPath:filePath];
// Read content of the file
NSData* data = [handle readDataToEndOfFile];
// Decode image from the data buffer
cvImage = cv::imdecode(cv::Mat(1, [data length], CV_8UC1,
(void*)data.bytes),
CV_LOAD_IMAGE_UNCHANGED);In this example we read the file content to the buffer and call the cv::imdecode function to decode the image. But there is one important note; if you later want to convert cv::Mat to the UIImage, you should change the channel order from BGR to RGB, as OpenCV's native image format is BGR.