Now that the program works on the desktop, we can make an embedded system from it. The details given here are specific to Raspberry Pi, but similar steps apply when developing for other embedded Linux systems such as BeagleBone, ODROID, Olimex, Jetson, and so on.

There are several different options for running our code on an embedded system, each with some advantages and disadvantages in different scenarios.

There are two common methods for compiling the code for an embedded device:

• Copy the source code from the desktop onto the device and compile it directly on board the device. This is often referred to as native compilation since we are compiling our code natively on the same system that it will eventually run on.
• Compile all the code on the desktop but using special methods to generate code for the device, and then you copy the final executable program onto the device. This is often referred to as cross-compilation since you need a special compiler that knows how to generate code for other types of CPUs.

Cross-compilation is often significantly harder to configure than native compilation, especially if you are using many shared libraries, but since your desktop is usually a lot faster than your embedded device, cross-compilation is often much faster at compiling large projects. If you expect to be compiling your project hundreds of times, in order to work on it for months, and your device is quite slow compared to your desktops, such as the Raspberry Pi 1 or Raspberry Pi Zero, which are very slow compared to a desktop, then cross-compilation is a good idea. But in most cases, especially for small, simple projects, you should just stick with native compilation since it is easier.

Note that all the libraries used by your project will also need to be compiled for the device, so you will need to compile OpenCV for your device. Natively compiling OpenCV on a Raspberry Pi 1 can take hours, whereas cross-compiling OpenCV on a desktop might take just 15 minutes. But you usually only need to compile OpenCV once and then you'll have it for all your projects, so it is still worth sticking with native compilation of your project (including the native compilation of OpenCV) in most cases.

There are also several options for how to run the code on an embedded system:

• Use the same input and output methods you used on the desktop, such as the same video files, USB webcam, or keyboard as input, and display text or graphics on an HDMI monitor in the same way you were doing on the desktop.
• Use special devices for input and output. For example, instead of sitting at a desk using a USB webcam and keyboard as input and displaying the output on a desktop monitor, you could use the special Raspberry Pi Camera Module for video input, use custom GPIO push buttons or sensors for input, and use a 7-inch MIPI DSI screen or GPIO LED lights as the output, and then by powering it all with a common portable USB charger, you can be wearing the whole computer platform in your backpack or attaching it on your bicycle!
• Another option is to stream data in or out of the embedded device to other computers, or even use one device to stream out the camera data and one device to use that data. For example, you can use the GStreamer framework to configure the Raspberry Pi to stream H.264 compressed video from its camera module to the Ethernet network or through Wi-Fi, so that a powerful PC or server rack on the local network or the Amazon AWS cloud computing services can process the video stream somewhere else. This method allows a small and cheap camera device to be used in a complex project requiring large processing resources located somewhere else.

If you do wish to perform computer vision on board the device, be aware that some low-cost embedded devices such as Raspberry Pi 1, Raspberry Pi Zero, and BeagleBone Black have significantly less computing power than desktops or even cheap netbooks or smartphones, perhaps 10-50 times slower than your desktop, so depending on your application you might need a powerful embedded device or stream video to a separate computer, as mentioned previously. If you don't need much computing power (for example, you only need to process one frame every 2 seconds, or you only need to use 160 x 120 image resolution), then a Raspberry Pi Zero running some computer vision on board might be fast enough for your requirements. But many computer vision systems need far more computing power, and so if you want to perform computer vision on board the device, you will often want to use a much faster device with a CPU in the range of 2 GHz, such as a Raspberry Pi 3, ODROID-XU4, or Jetson TK1.

Let's begin by keeping it as simple as possible, by using a USB keyboard and mouse and an HDMI monitor just like our desktop system, compiling the code natively on the device, and running our code on the device. Our first step will be to copy the code onto the device, install the build tools, and compile OpenCV and our source code on the embedded system.

Many embedded devices such as Raspberry Pi have an HDMI port and at least one USB port. Therefore, the easiest way to start using an embedded device is to plug in an HDMI monitor and USB keyboard and mouse for the device, to configure settings and see the output, while doing the code development and testing using your desktop machine. If you have a spare HDMI monitor, plug that into the device, but if you don't have a spare HDMI monitor, you might consider buying a small HDMI screen just for your embedded device.

Also, if you don't have a spare USB keyboard and mouse, you might consider buying a wireless keyboard and mouse that has a single USB wireless dongle, so you only use up a single USB port for both the keyboard and mouse. Many embedded devices use a 5V power supply, but they usually need more power (electrical current) than a desktop or laptop will provide in its USB port. So, you should obtain either a separate 5V USB charger (at least 1.5 amps, ideally 2.5 amps) or a portable USB battery charger that can provide at least 1.5 amps of output current. Your device might only use 0.5 amps most of the time, but there will be occasional times when it needs over 1 amp, so it's important to use a power supply that is rated for at least 1.5 amps or more, otherwise your device will occasionally reboot, or some hardware could behave strangely at important times, or the filesystem could become corrupt and you lose your files! A 1 amp supply might be good enough if you don't use cameras or accessories, but 2.0-2.5 amps is safer.

For example, the following photographs show a convenient setup containing a Raspberry Pi 3, a good quality 8 GB micro-SD card for $10 (http://ebay.to/2ayp6Bo), a 5-inch HDMI resistive touchscreen for $30-$45 (http://bit.ly/2aHQO2G), a wireless USB keyboard and mouse for $30 (http://ebay.to/2aN2oXi), a 5V 2.5 A power supply for $5 (https://amzn.to/2UafanD), a USB webcam such as the very fast PS3 Eye for just $5 (http://ebay.to/2aVWCUS), a Raspberry Pi Camera Module v1 or v2 for $15-$30 (http://bit.ly/2aF9PxD), and an Ethernet cable for $2 (http://ebay.to/2aznnjd), connecting the Raspberry Pi to the same LAN network as your development PC or laptop. Notice that this HDMI screen is designed specifically for the Raspberry Pi, since the screen plugs directly into the Raspberry Pi below it, and has an HDMI male-to-male adapter (shown in the right-hand photo) for the Raspberry Pi so you don't need an HDMI cable, whereas other screens may require an HDMI cable (https://amzn.to/2Rvet6H), or MIPI DSI or SPI cable.

Also note that some screens and touch panels need configuration before they will work, whereas most HDMI screens should work without any configuration:

Notice the black USB webcam (on the far left of the LCD), the Raspberry Pi Camera Module (green and black board sitting on the top-left corner of the LCD), Raspberry Pi board (underneath the LCD), HDMI adapter (connecting the LCD to the Raspberry Pi underneath it), a blue Ethernet cable (plugged into a router), a small USB wireless keyboard and mouse dongle, and a micro-USB power cable (plugged into a 5V 2.5A power supply).

The following steps are specific to Raspberry Pi, so if you are using a different embedded device or you want a different type of setup, search the web about how to set up your board. To set up an Raspberry Pi 1, 2, or 3 (including their variants such as Raspberry Pi Zero, Raspberry Pi 2B, 3B, and so on, and Raspberry Pi 1A+ if you plug in a USB Ethernet dongle), follow these steps:

1. Get a fairly new, good quality micro-SD card of at least 8 GB. If you use a cheap micro-SD card or an old micro-SD card that you already used many times before and it has degraded in quality, it might not be reliable enough to boot the Raspberry Pi, so if you have trouble booting the Raspberry Pi, you should try a good quality Class 10 micro-SD card (such as SanDisk Ultra or better) that says it handles at least 45 Mbps or can handle 4K video.

2. Download and burn the latest Raspbian IMG (not NOOBS) to the micro-SD card. Note that burning an IMG is different to simply copying the file to SD. Visit https://www.raspberrypi.org/documentation/installation/installing-images/ and follow the instructions for your desktop's OS to burn Raspbian to a micro-SD card. Be aware that you will lose any files that were previously on the card.
3. Plug a USB keyboard, mouse, and HDMI display into the Raspberry Pi, so you can easily run some commands and see the output.
4. Plug the Raspberry Pi into a 5V USB power supply with at least 1.5 A, ideally 2.5 A or higher. Computer USB ports aren't powerful enough.
5. You should see many pages of text scrolling while it is booting up Raspbian Linux, then it should be ready after 1 or 2 minutes.
6. If, after booting, it's just showing a black console screen with some text (such as if you downloaded Raspbian Lite), you are at the text-only login prompt. Log in by typing pi as the username and then hit Enter. Then, type raspberry as the password and hit Enter again.
7. Or if it booted to the graphical display, click on the black Terminal icon at the top to open a shell (Command Prompt).
8. Initialize some settings in your Raspberry Pi:
   • Type sudo raspi-config and hit Enter (see the following screenshot).
   • First, run Expand Filesystem and then finish and reboot your device, so the Raspberry Pi can use the whole micro-SD card.
   • If you use a normal (US) keyboard, not a British keyboard, in Internationalization Options, change to Generic 104-key keyboard, Other, English (US), and then for the AltGr and similar questions, just hit Enter unless you are using a special keyboard.
   • In Enable Camera, enable the Raspberry Pi Camera Module.
   • In Overclock Options, set to Raspberry Pi 2 or similar to the device runs faster (but generates more heat).
   • In Advanced Options, enable the SSH server.
   • In Advanced Options, if you are using Raspberry Pi 2 or 3, change Memory Split to 256 MB so the GPU has plenty of RAM for video processing. For Raspberry Pi 1 or Zero, use 64 MB or the default.
   • Finish, then reboot the device.

9. (Optional): Delete Wolfram to save 600 MB of space on your SD card:

sudo apt-get purge -y wolfram-engine

It can be reinstalled using sudo apt-get install wolfram-engine.

To see the remaining space on your SD card, run df -h | head -2:

10. Assuming you plugged the Raspberry Pi into your internet router, it should already have internet access. So, update your Raspberry Pi to the latest Raspberry Pi firmware, software locations, OS, and software. Warning: many Raspberry Pi tutorials say you should run sudo rpi-update; however, in recent years, it's no longer a good idea to run rpi-update since it can give you an unstable system or firmware. The following instructions will update your Raspberry Pi to have stable software and firmware (note that these commands might take up to one hour):

sudo apt-get -y update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo reboot

11. Find the IP address of the device:

hostname -I

12. Try accessing the device from your desktop. For example, assume the device's IP address is 192.168.2.101. Enter this on a Linux desktop:

ssh-X [email protected]

12. Or, do this on a Windows desktop:
    1. Download, install, and run PuTTY
    2. Then in PuTTY, connect to the IP address (192.168.2.101), as the user pi with the password raspberry
13. Optionally, if you want your Command Prompt to be a different color than the commands and show the error value after each command, use this:

nano ~/.bashrc

14. Add this line to the bottom:

PS1="[e[0;44m]u@h: w ($?) $[e[0m] "

15. Save the file (hit Ctrl + X, then hit Y, and then hit Enter).
16. Start using the new settings:

source ~/.bashrc

17. To prevent the screensaver/screen blank power saving feature in Raspbian from turning off your screen on idle, use this:

sudo nano /etc/lightdm/lightdm.conf

18. And follow these steps:
    1. Look for the line that says #xserver-command=X (jump to line 87 by pressing Alt + G and then typing 87 and hitting Enter).
    2. Change it to xserver-command=X -s 0 dpms.
    3. Save the file (hit Ctrl + X, then hit Y, then hit Enter).
19. Finally, reboot the Raspberry Pi:

sudo reboot

You should be ready to start developing on the device now!

There is a very easy way to install OpenCV and all its dependencies on a Debian-based embedded device such as Raspberry Pi:

sudo apt-get install libopencv-dev

However, that might install an old version of OpenCV from one or two years ago.

To install the latest version of OpenCV on an embedded device such as Raspberry Pi, we need to build OpenCV from the source code. First, we install a compiler and build system, then libraries for OpenCV to use, and finally OpenCV itself. Note that the steps for compiling OpenCV from source on Linux are the same whether you are compiling for desktop or for embedded systems. A Linux script, install_opencv_from_source.sh, is provided with this book; it is recommended you copy the file onto your Raspberry Pi (for example, with a USB flash stick) and run the script to download, build, and install OpenCV, including potential multi-core CPU and ARM NEON SIMD optimizations (depending on hardware support):

chmod +x install_opencv_from_source.sh
 ./install_opencv_from_source.sh

It's highly recommended to build and run a few OpenCV samples every time you install OpenCV, so when you have problems building your own code, at least you will know whether the problem is the OpenCV installation or a problem with your code.

Let's try to build the simple edge sample program. If we try the same Linux command to build it from OpenCV 2, we get a build error:

cd ~/opencv-4.*/samples/cpp
 g++ edge.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui
 -o edge
 /usr/bin/ld: /tmp/ccDqLWSz.o: undefined reference to symbol '_ZN2cv6imreadERKNS_6StringEi'
 /usr/local/lib/libopencv_imgcodecs.so.4..: error adding symbols: DSO missing from command line
 collect2: error: ld returned 1 exit status

The second to last line of that error message tells us that a library was missing from the command line, so we simply need to add -lopencv_imgcodecs in our command next to the other OpenCV libraries we linked to. Now, you know how to fix the problem anytime you are compiling an OpenCV 3 program and you see that error message. So, let's do it correctly:

cd ~/opencv-4.*/samples/cpp
 g++ edge.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui
 -lopencv_imgcodecs -o edge

It worked! So, now you can run the program:

./edge

Hit Ctrl + C on your keyboard to quit the program. Note that the edge program might crash if you try running the command in an SSH Terminal and you don't redirect the window to display on the device's LCD screen. So, if you are using SSH to remotely run the program, add DISPLAY=:0 before your command:

DISPLAY=:0 ./edge

You should also plug a USB webcam into the device and test that it works:

g++ starter_video.cpp -lopencv_core -lopencv_imgproc
 -lopencv_highgui -lopencv_imgcodecs -lopencv_videoio \
 -o starter_video
 DISPLAY=:0 ./starter_video 0

Note: if you don't have a USB webcam, you can test using a video file:

DISPLAY=:0 ./starter_video ../data/768x576.avi

Now that OpenCV is successfully installed on your device, you can run the Cartoonifier applications we developed earlier. Copy the Cartoonifier folder onto the device (for example, by using a USB flash stick, or using scp to copy files over the network). Then, build the code just like you did for the desktop:

cd ~/Cartoonifier
 export OpenCV_DIR="~/opencv-3.1.0/build"
 mkdir build
 cd build
 cmake -D OpenCV_DIR=$OpenCV_DIR ..
 make

And run it:

DISPLAY=:0 ./Cartoonifier

And if all is fine, we will see a window with our application running as follows:

While using a USB webcam on Raspberry Pi has the convenience of supporting identical behavior and code on the desktop as on an embedded device, you might consider using one of the official Raspberry Pi Camera Modules (referred to as the Raspberry Pi Cams). They have some advantages and disadvantages over USB webcams.

The Raspberry Pi Cams use the special MIPI CSI camera format, designed for smartphone cameras to use less power. They have a smaller physical size, faster bandwidth, higher resolutions, higher frame rates, and reduced latency compared to USB. Most USB 2.0 webcams can only deliver 640 x 480 or 1280 x 720 30 FPS video since USB 2.0 is too slow for anything higher (except for some expensive USB webcams that perform onboard video compression) and USB 3.0 is still too expensive. However, smartphone cameras (including the Raspberry Pi Cams) can often deliver 1920 x 1080 30 FPS or even Ultra HD/4K resolutions. The Raspberry Pi Cam v1 can, in fact, deliver upto 2592 x 1944 15 FPS or 1920 x 1080 30 FPS video even on a $5 Raspberry Pi Zero, thanks to the use of MIPI CSI for the camera and compatible video processing ISP and GPU hardware inside the Raspberry Pi. The Raspberry Pi Cams also support 640 x 480 in 90 FPS mode (such as for slow-motion capture), and this is quite useful for real-time computer vision so you can see very small movements in each frame, rather than large movements that are harder to analyze.

However, the Raspberry Pi Cam is a plain circuit board that is highly sensitive to electrical interference, static electricity, or physical damage (simply touching the small, flat orange cable with your finger can cause video interference or even permanently damage your camera!). The big flat white cable is far less sensitive but it is still very sensitive to electrical noise or physical damage. The Raspberry Pi Cam comes with a very short 15 cm cable. It's possible to buy third-party cables on eBay with lengths between 5 cm and 1 m, but cables 50 cm or longer are less reliable, whereas USB webcams can use 2 m to 5 m cables and can be plugged into USB hubs or active extension cables for longer distances.

There are currently several different Raspberry Pi Cam models, notably the NoIR version that doesn't have an internal infrared filter; therefore, a NoIR camera can easily see in the dark (if you have an invisible infrared light source), or see infrared lasers or signals far clearer than regular cameras that include an infrared filter inside them. There are also two different versions of Raspberry Pi Cam: Raspberry Pi Cam v1.3 and Raspberry Pi Cam v2.1, where v2.1 uses a wider angle lens with a Sony 8 megapixel sensor instead of a 5 megapixel OmniVision sensor, has better support for motion in low lighting conditions, and adds support for 3240 x 2464 video at 15 FPS and potentially up to 120 FPS video at 720p. However, USB webcams come in thousands of different shapes and versions, making it easy to find specialized webcams such as waterproof or industrial-grade webcams, rather than requiring you to create your own custom housing for a Raspberry Pi Cam.

IP cameras are also another option for a camera interface that can allow 1080p or higher resolution videos with Raspberry Pi, and IP cameras support not just very long cables, but potentially even work anywhere in the world using the internet. But IP cameras aren't quite as easy to interface with OpenCV as USB webcams or Raspberry Pi Cams.

In the past, Raspberry Pi Cams and the official drivers weren't directly compatible with OpenCV; you often used custom drivers and modified your code in order to grab frames from Raspberry Pi Cams, but it's now possible to access a Raspberry Pi Cam in OpenCV in the exact same way as a USB webcam! Thanks to recent improvements in the v4l2 drivers, once you load the v4l2 driver, the Raspberry Pi Cam will appear as a /dev/video0 or /dev/video1 file like a regular USB webcam. So, traditional OpenCV webcam code such as cv::VideoCapture(0) will be able to use it just like a webcam.

First, let's temporarily load the v4l2 driver for the Raspberry Pi Cam to make sure our camera is plugged in correctly:

sudo modprobe bcm2835-v4l2

If the command failed (if it printed an error message to the console, it froze, or the command returned a number besides 0), then perhaps your camera is not plugged in correctly. Shut down and then unplug power from your Raspberry Pi and try attaching the flat white cable again, looking at photos on the web to make sure it's plugged in the correct way around. If it is the correct way around, it's possible the cable wasn't fully inserted before you closed the locking tab on the Raspberry Pi. Also, check whether you forgot to click Enable Camera when configuring your Raspberry Pi earlier, using the sudoraspi-config command.

If the command worked (if the command returned 0 and no error was printed to the console), then we can make sure the v4l2 driver for the Raspberry Pi Cam is always loaded on bootup by adding it to the bottom of the /etc/modules file:

sudo nano /etc/modules
 # Load the Raspberry Pi Camera Module v4l2 driver on bootup:
 bcm2835-v4l2

After you save the file and reboot your Raspberry Pi, you should be able to run ls /dev/video* to see a list of cameras available on your Raspberry Pi. If the Raspberry Pi Cam is the only camera plugged into your board, you should see it as the default camera (/dev/video0), or if you also have a USB webcam plugged in, then it will be either /dev/video0 or /dev/video1.

Let's test the Raspberry Pi Cam using the starter_video sample program we compiled earlier:

cd ~/opencv-4.*/samples/cpp
 DISPLAY=:0 ./starter_video 0

If it's showing the wrong camera, try DISPLAY=:0 ./starter_video 1.

Now that we know the Raspberry Pi Cam is working in OpenCV, let's try Cartoonifier:

cd ~/Cartoonifier
 DISPLAY=:0 ./Cartoonifier 0

Or, use DISPLAY=:0 ./Cartoonifier 1 for the other camera.

In embedded systems, you often want your application to be fullscreen and hide the Linux GUI and menu. OpenCV offers an easy method to set the fullscreen window property, but make sure you created the window using the NORMAL flag:

// Create a fullscreen GUI window for display on the screen.
 namedWindow(windowName, WINDOW_NORMAL);
 setWindowProperty(windowName, PROP_FULLSCREEN, CV_WINDOW_FULLSCREEN);

You might notice the mouse cursor is shown on top of your window even though you don't want to use a mouse in your embedded system. To hide the mouse cursor, you can use the xdotool command to move it to the bottom-right corner pixel, so it's not noticeable, but is still available if you want to occasionally plug in your mouse to debug the device. Install xdotool and create a short Linux script to run it with Cartoonifier:

sudo apt-get install -y xdotool
 cd ~/Cartoonifier/build

After installing xdotool, now is the time to create the script, create a new file with your favorite editor with the name runCartoonifier.sh and the following content:

 #!/bin/sh
 # Move the mouse cursor to the screen's bottom-right pixel.
 xdotoolmousemove 3000 3000
 # Run Cartoonifier with any arguments given.
 /home/pi/Cartoonifier/build/Cartoonifier "$@"

Finally, make your script executable:

chmod +x runCartoonifier.sh

Try running your script to make sure it works:

DISPLAY=:0 ./runCartoonifier.sh

Often, when you build an embedded device, you want your application to be executed automatically after the device has booted up, rather than requiring the user to manually run your application. To automatically run our application after the device has fully booted up and logged in to the graphical desktop, create an autostart folder with a file in it with these contents, including the full path to your script or application:

mkdir ~/.config/autostart
 nano ~/.config/autostart/Cartoonifier.desktop
 [Desktop Entry]
 Type=Application
 Exec=/home/pi/Cartoonifier/build/runCartoonifier.sh
 X-GNOME-Autostart-enabled=true

Now, whenever you turn the device on or reboot it, Cartoonifier will begin running!

You will notice that the code runs much slower on Raspberry Pi than on your desktop! By far the two easiest ways to run it faster are to use a faster device or use a smaller camera resolution. The following table shows some frame rates, frames per seconds (FPS), for both the sketch and paint modes of Cartoonifier on a desktop, Raspberry Pi 1, Raspberry Pi 2, Raspberry Pi 3, and Jetson TK1. Note that the speeds don't have any custom optimizations and only run on a single CPU core, and the timings include the time for rendering images to the screen. The USB webcam used is the fast PS3 Eye webcam running at 640 x 480 since it is the fastest low-cost webcam on the market.

It's worth mentioning that Cartoonifier is only using a single CPU core, but all the devices listed have four CPU cores except for Raspberry Pi 1, which has a single core, and many x86 computers have hyperthreading to give roughly eight CPU cores. So, if you wrote your code to efficiently make use of multiple CPU cores (or GPU), the speeds might be 1.5 to 3 times faster than the single-threaded figures shown:

Notice that Raspberry Pi is extremely slow at running the code, especially the paint mode, so we will try simply changing the camera and the resolution of the camera.

The following table shows how the speed of the sketch mode compares on Raspberry Pi 2 using different types of cameras and different camera resolutions:

As you can see, when using the Raspberry Pi Cam in 320 x 240, it seems we have a good enough solution to have some fun, even if it's not in the 20-30 FPS range that we would prefer.

We've seen that various embedded devices are slower than desktops, from the Raspberry Pi 1 being roughly 20 times slower than a desktop, up to Jetson TK1 being roughly 1.5 times slower than a desktop. But for some tasks, low speed is acceptable if it means there will also be significantly lower battery draw, allowing for small batteries or low year-round electricity costs for a server, or low heat generation.

Raspberry Pi has different models even for the same processor, such as Raspberry Pi 1B, Zero, and 1A+, which all run at similar speeds but have significantly different power draws. MIPI CSI cameras such as the Raspberry Pi Cam also use less electricity than webcams. The following table shows how much electrical power is used by different hardware running the same Cartoonifier code. Power measurements of Raspberry Pi were performed as shown in the following photo using a simple USB current monitor (for example, J7-T Safety Tester (http://bit.ly/2aSZa6H) for $5) and a DMM multimeter for the other devices:

Idle power measures power when the computer is running but no major applications are being used, whereas Cartoonifier power measures power when Cartoonifier is running. Efficiency is Cartoonifier power/Cartoonifier speed in a 640 x 480 sketch mode:

We can see that Raspberry Pi 1A+ uses the least power, but the most power efficient options are Jetson TK1 and Raspberry Pi 3B. Interestingly, the original Raspberry Pi (Raspberry Pi 1B) has roughly the same efficiency as an x86 laptop. All later Raspberry Pis are significantly more power efficient than the original (Raspberry Pi 1B).

Let's also look at the power draw of different cameras that work with Raspberry Pi:

We see that Raspberry Pi Cam v2.1 is slightly more power efficient than Raspberry Pi Cam v1.3 and significantly more power efficient than a USB webcam.

Thanks to the hardware-accelerated video encoders in all modern ARM devices, including Raspberry Pi, a valid alternative to performing computer vision on board an embedded device is to use the device to just capture video and stream it across a network in real time to a PC or server rack. All Raspberry Pi models contain the same video encoder hardware, so an Raspberry Pi 1A+ or Raspberry Pi Zero with a Pi Cam is quite a good option for a low-cost, low-power portable video streaming server. Raspberry Pi 3 adds Wi-Fi for additional portable functionality.

There are numerous ways live camera video can be streamed from a Raspberry Pi, such as using the official Raspberry Pi V4L2 camera driver to allow the Raspberry Pi Cam to appear like a webcam, then using GStreamer, liveMedia, netcat, or VLC to stream the video across a network. However, these methods often introduce one or two seconds of latency and often require customizing the OpenCV client code or learning how to use GStreamer efficiently. So instead, the following section will show how to perform both the camera capture and network streaming using an alternative camera driver named UV4L:

1. Install UV4L on the Raspberry Pi by following the instructions at http://www.linux-projects.org/uv4l/installation/:

curl http://www.linux-projects.org/listing/uv4l_repo/lrkey.asc
 sudo apt-key add -
 sudo su
 echo "# UV4L camera streaming repo:">> /etc/apt/sources.list
 echo "deb http://www.linux-
 projects.org/listing/uv4l_repo/raspbian/jessie main">>
 /etc/apt/sources.list
 exit
 sudo apt-get update
 sudo apt-get install uv4l uv4l-raspicam uv4l-server

2. Run the UV4L streaming server manually (on the Raspberry Pi) to check that it works:

sudo killall uv4l
 sudo LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so
 uv4l -v7 -f --sched-rr --mem-lock --auto-video_nr
 --driverraspicam --encoding mjpeg
 --width 640 --height 480 --framerate15

3. Test the camera's network stream from your desktop, following these steps to check all is working fine:
   • Install VLC Media Player.
   • Navigate to Media | Open Network Stream and enter http://192.168.2.111:8080/stream/video.mjpeg.
   • Adjust the URL to the IP address of your Raspberry Pi. Run hostname -I on Raspberry Pi to find its IP address.
4. Run the UV4L server automatically on bootup:

sudo apt-get install uv4l-raspicam-extras

5. Edit any UV4L server settings you want in uv4l-raspicam.conf, such as resolution and frame rate to customize the streaming:

sudo nano /etc/uv4l/uv4l-raspicam.conf
 drop-bad-frames = yes
 nopreview = yes
 width = 640
 height = 480
 framerate = 24

You will need to reboot to make all changes take effect.

6. Tell OpenCV to use our network stream as if it was a webcam. As long as your installation of OpenCV can use FFMPEG internally, OpenCV will be able to grab frames from an MJPEG network stream just like a webcam:

./Cartoonifier http://192.168.2.101:8080/stream/video.mjpeg

Your Raspberry Pi is now using UV4L to stream the live 640 x 480 24 FPS video to a PC that is running Cartoonifier in sketch mode, achieving roughly 19 FPS (with 0.4 seconds of latency). Notice this is almost the same speed as using the PS3 Eye webcam directly on the PC (20 FPS)!

Note that when you are streaming the video to OpenCV, it won't be able to set the camera resolution; you need to adjust the UV4L server settings to change the camera resolution. Also note that instead of streaming MJPEG, we could have streamed H.264 video, which uses a lower bandwidth, but some computer vision algorithms don't handle video compression such as H.264 very well, so MJPEG will cause fewer algorithm problems than H.264.

Now that you have created a whole embedded Cartoonifier system, and you know the basics of how it works and which parts do what, you should customize it! Make the video full screen, change the GUI, change the application behavior and workflow, change the Cartoonifier filter constants or the skin detector algorithm, replace the Cartoonifier code with your own project ideas, or stream the video to the cloud and process it there!

You can improve the skin detection algorithm in many ways, such as using a more complex skin detection algorithm (for example, using trained Gaussian models from many recent CVPR or ICCV conference papers at http://www.cvpapers.com), or add face detection (see the Face detection section of Chapter 5, Face Detection and Recognition with the DNN Module) to the skin detector, so it detects where the user's face is, rather than asking the user to put their face in the center of the screen. Be aware that face detection may take many seconds on some devices or high-resolution cameras, so they may be limited in their current real-time uses. But embedded system platforms are getting faster every year, so this may be less of a problem over time.

The most significant way to speed up embedded computer vision applications is to reduce the camera resolution absolutely as much as possible (for example, 0.5 megapixels instead of 5 megapixels), allocate and free images as rarely as possible, and perform image format conversions as rarely as possible. In some cases, there might be some optimized image processing or math libraries, or an optimized version of OpenCV from the CPU vendor of your device (for example, Broadcom, NVIDIA Tegra, Texas Instruments OMAP, or Samsung Exynos), or for your CPU family (for example, ARM Cortex-A9).