A Gentle Introduction to Computer Vision, Part 1: Compiling OpenCV on the Jetson Xavier NX

7 min readJan 2, 2021

Introduction to OpenCV

OpenCV is for computer vision. There are a number of programming language interfaces: C++, Python, Java and Matlab. OpenCV also runs on various operating systems, such as Windows, Linux, Android and Mac OS, as well as a number of different architectures. The library has a long history of industry adoption and help from Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, and Toyota, as well as algorithms being implemented from CVPR papers and other academic publications. With more than 2,500 optimized algorithms, it’s got to be one of the most fun libraries to toy with in your spare time or for your work project.

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code.

Compiling OpenCV

Compiling OpenCV is a long process, so let’s get started. First, we must install the package dependencies. The goal of the build is to make as complete of a build as possible, leveraging as many features as possible. I’ll also be installing libjpeg-turbo8 instead of the typical libjpeg.

$ sudo apt-get install build-essential unzip pkg-config libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libgtk-3-dev libatlas-base-dev gfortran python3-dev python3-venv libglew-dev libtiff5-dev zlib1g-dev libpng-dev libavcodec-dev libavformat-dev libavutil-dev libpostproc-dev libswscale-dev libeigen3-dev libtbb-dev libgtk2.0-dev qtbase5-dev libavresample-dev libdc1394-22-dev libopenblas-dev liblapack-dev libgflags-dev libgoogle-glog-dev libbenchmark-dev libsuitesparse-dev libmetis-dev ccache libhdf5-dev libhdf5-mpi-dev libhdf5-openmpi-dev libtesseract-dev libprotobuf-dev apt-utils libjpeg-turbo8 libjpeg-turbo8-dev# Upgrade CMake for VTK
$ sudo apt remove --purge cmake
$ hash -r
$ sudo snap install cmake --classic

Get the OpenCV and dependency sources:

$ git clone https://github.com/opencv/opencv.git
$ git clone https://github.com/opencv/opencv_contrib.git
$ git clone https://github.com/opencv/opencv_extra.git
$ git clone https://github.com/ceres-solver/ceres-solver.git
$ git clone https://github.com/Kitware/VTK.git

For compilation speed, please make sure we are running one of the most performant NVIDIA Performance Models (nvpmodel):

$ sudo /usr/sbin/nvpmodel -m 2  # 15W, 6 core, 1.4GHz each
$ sudo /usr/sbin/nvpmodel -m 0  # 15W, 2 core, 1.9GHz each

The VTK development package that Ubuntu 18.04 provides causes OpenCV compilation to fail. So, we will compile this particular dependency from source to alleviate that issue:

$ mkdir VTK-bin && cd VTK-bin
$ cmake ../VTK
$ make -j$((`nproc`-1)) && sudo make install

OpenCV doesn’t appear to detect the Ceres Solver provided by Ubuntu, so we can instead build it from source. Be sure to turn off examples and testing, as these take foreeeeeeeeeeeever.

$ cd .. && mkdir ceres-solver-bin && cd ceres-solver-bin
$ cmake -D BUILD_EXAMPLES=OFF -D BUILD_TESTING=OFF ../ceres-solver
$ make -$((`nproc`-1)) && sudo make install

Finally, configure the OpenCV CMake project and build it:

$ cd .. && mkdir opencv-bin && cd opencv-bin
$ cmake \
-D ENABLE_PRECOMPILED_HEADERS=OFF \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=OFF \
-D CPU_BASELINE_REQUIRE=NEON \
-D BUILD_TESTS=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D CMAKE_SHARED_LINKER_FLAGS=-latomic \
-D BUILD_EXAMPLES=OFF \
-D WITH_QT=ON \
-D WITH_CUDA=ON \
-D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
-D CUDA_ARCH_BIN=7.2 \
-D CUDA_ARCH_PTX="" \
-D WITH_CUDNN=ON \
-D WITH_CUBLAS=ON \
-D ENABLE_FAST_MATH=ON \
-D CUDA_FAST_MATH=ON \
-D OPENCV_DNN_CUDA=ON \
-D WITH_OPENMP=ON \
-D WITH_OPENGL=ON \
-D WITH_FFMPEG=ON \
-D WITH_GSTREAMER=ON \
-D WITH_TBB=ON \
-D BUILD_TBB=ON \
-D WITH_EIGEN=ON \
-D WITH_V4L=ON \
-D WITH_LIBV4L=ON \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_opencv_python3=TRUE \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D WITH_JPEG=ON \
-D BUILD_JPEG=OFF \
-D JPEG_INCLUDE_DIR=/usr/include \
-D JPEG_LIBRARY=/usr/lib/aarch64-linux-gnu/libjpeg.a \
../opencv

Most of these compiler flags are obvious, but I’ll highlight a few that have a bit more nuance them:

ENABLE_PRECOMPILED_HEADERS=OFF because the Jetson TX1 does not support this, and the build will take a long time, anyway.
ENABLE_NEON=ON, ENABLE_VFPV3=OFF, CPU_BASELINE_REQUIRE=NEON, CMAKE_SHARED_LINKER_FLAGS=-latomic are enabling NEON and disabling VFPV3. NEON is compatible with the Arm Cortex-A processor series and for Cortex-R52 and Cortex-R82 processors. VFPV3 is not compatible with the Jetson Xavier NX. The atomic linker flag is required when enabling NEON.
WITH_QT=ON will enable useful status bar items that show RGB values on mouseover, for example.
CUDA_ARCH_BIN=7.2 is the GPU architecture number for the Jetson Xavier NX’s Volta GPU.
JPEG_INCLUDE_DIR=/usr/include, JPEG_LIBRARY=/usr/lib/aarch64-linux-gnu/libjpeg.a is here because the JPEG library was not being detected, and I also decided to use libjpeg-turbo8 instead of the standard libjpeg. The “turbo” version will make use of NEON SIMD instructions to to achieve 2–6x faster encode/decode of JPEG images, which happens to be the most common file format.

OpenCV compilation eats up memory like a starving lemur in a nectarine farm. Note that if you use make -j, engaging all CPUs, one would need to increase the swap size by a significant amount to avoid stalling the compilation. So, only engage with $((`nproc`-1)) cores. Also, we’ll symlink the generated name (your version may vary) to the expected name, cv2, so that a Python import will work as expected (i.e. import cv2):

$ make -j$((`nproc`-1)) && sudo make install
$ sudo ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.so

It’s best to work in virtual environments so that different OpenCV versions can be worked on without interfering with each other. So, also link the library to a virtual environment.

$ ls ~/dev/envs/venv/
bin  include  lib  lib64  pyvenv.cfg  share
$ ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so ~/dev/envs/venv/lib/python3.6/site-packages/cv2.so
$ source ~/dev/envs/venv/bin/activate
(venv) $ python
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.5.1'

We’ve successfully compiled OpenCV v4.5.1 from source!

Troubleshooting

“Illegal instruction (core dumped)” when importing cv2

When importing cv2, you may encounter an issue where an illegal instruction is called:

$ python
Python 3.9.1 (tags/v3.9.1:1e5d33e9b9, Jan 11 2021, 22:02:52) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Illegal instruction (core dumped)

This is coming from a library dependency calling an illegal instruction. To find out which library is causing this particular issue, allow the kernel to dump a core file in the current directory for gdb analysis:

$ sudo sysctl -w kernel.core_pattern=core
kernel.core_pattern = core
$ ulimit -c unlimited
$ python    # Run 'import cv2' to generate a core dump
$ gdb python core
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.warning: core file may not match specified executable file.
[New LWP 17811]
[New LWP 17819]
[New LWP 17818]
[New LWP 17821]
[New LWP 17820]
[New LWP 17817]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `python'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x0000007f4c3d5f54 in gotoblas_dynamic_init ()
   from /home/username/dev/envs/venv/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-32ff4d91.3.13.so
[Current thread is 1 (Thread 0x7f84cd35c0 (LWP 17811))]

It looks like the Numpy package’s supplied compilation of OpenBLAS is causing an issue. So, I opted to build the Numpy wheel on my local system. There are several requirements to build the Numpy wheel:

Python 3.6.x or newer and header files
gcc 4.x or later
FORTRAN 77 compiler
OpenBLAS (The choice and location of these libraries as well as include paths and other such build options can be specified in a site.cfg file located in the NumPy root repository or a .numpy-site.cfg file in your home directory. See the site.cfg.example example file included in the NumPy repository or sdist for documentation, and below for specifying search priority from environmental variables)
Cython

(venv) $ sudo apt install libfftw3-dev libopenblas-dev
(venv) $ python --version   # requires >=3.6
Python 3.9.1
(venv) $ pip install cython
(venv) $ git clone https://www.github.com/numpy/numpy.git
(venv) $ cd numpy
(venv) $ git checkout -b v0.19.5
(venv) $ git reset --hard && git clean -fd
(venv) $ cp site.cfg.example ~/.numpy-site.cfg

We’ll need to tell Numpy where the libfftw3 and libopenblas are located with the .numpy-site.cfg file. Numpy will look for this configuration file in your home directory. In L4T, the relevant files are located here:

[DEFAULT] 
library_dirs = /usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib 
include_dirs = /usr/local/include:/usr/include[openblas] 
libraries = openblas 
library_dirs = /usr/lib/aarch64-linux-gnu 
include_dirs = /usr/include/aarch64-linux-gnu 
runtime_library_dirs = /usr/lib/aarch64-linux-gnu[fftw] 
libraries = fftw3

Build the Numpy wheel:

(venv) $ python setup.py build -j $(nproc) install

Now, OpenCV should be able to import with no issue:

(venv) $ python
Python 3.9.1 (tags/v3.9.1:1e5d33e9b9, Jan 11 2021, 22:59:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.5.1'

Summary

Installed dependencies and compiled Ceres Solver and VTK from source.
Compiled OpenCV from source.
Symlinked compiled library to the Python cv2 module in our virtual environment.
Resolved issues with the Numpy package installed from pip by building the Numpy wheel.