posted on Sep 27, 2017

Implementing OpenPTrack on Ubuntu 16.04

The following is a guest post by Daniel Low.

This is the second post of a multi-part series about prototyping a 3D pedestrian tracking system with ZED cameras.

Previously, we implemented the OpenPTrack software together with ZED cameras in Ubuntu 14.04. This time, we will be implementing it on Ubuntu 16.04 instead, which is not natively supported at this point in time.

Hence, we will not be using any scripts used by OpenPTrack installation, but instead install the 16.04 version of many dependencies and tools needed by OpenPTrack manually. We will also have to update the source code in OpenPTrack, as Ubuntu 16.04 and above no longer uses OpenCV2.

Dependencies

As before, ensure your hardware is up to date before beginning the installation using the below commands:

$ sudo apt-add-repository universe
$ sudo apt-add-repository multiverse
$ sudo apt-get update && sudo apt-get upgrade
$ sudo apt-get install bash-completion command-not-found locate git gitg vim
$ sudo apt-get install ntp

CUDA

Similarly, the next step is to install CUDA. Depending on your hardware, you may choose to install CUDA 8 or any version later (in our case, we would be installing CUDA 8). Follow the instructions in this link, or the instructions from the previous post on how to install CUDA and verify the installation.

Once the installation is complete, set the environment paths to the CUDA library appropriately.

$ echo "export PATH=/usr/local/cuda/bin:$PATH" >> ~/.bashrc
$ echo "export LD_LIBRARY_PATH=/usr/local /cuda/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
$ source ~/.bashrc

ROS Kinetic

Next, we will have to install the version of ROS for Ubuntu 16.04. This will be ROS Kinetic. The installation is relatively straightforward, just execute the following commands:

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
$ sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
$ sudo apt-get update
$ sudo apt-get install ros-kinetic-desktop-full
$ sudo apt-get install python-rosinstall
$ sudo rosdep init
$ rosdep update
$ echo "source /opt/ros/kinetic/setup.bash" >> ~/.bashrc
$ source ~/.bashrc

PCL

Next will be to install PCL onto your system. Previously, we installed a pre-built PCL binary via PPA. However, (as of when we did this project) there was no pre-built binary for Ubuntu 16.04. Hence, there are 2 other possible ways of installing PCL.

Using the following command:
```
$ sudo apt-get install ros-kinetic-pcl-ros
```
This should install the PCL package for ROS Kinetic. If this package is insufficient (ie. you encounter PCL errors during later parts of installation), you may need to follow the second method below to install PCL.
Compiling the PCL library from source. Instructions on how to compile the PCL library can be found here. First, download the latest PCL source code from their github page.
Then, execute the following commands to extract and compile the source code. Please ensure your terminal is at the folder you downloaded the source code to, and replace the version number with the version you have downloaded.
```
$ tar xvfj pcl-pcl-1.8.0.tar.gz
$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make -j$(nproc)
$ sudo make -j$(nproc) install
```

ROS packages

Once you have successfully installed both ROS Kinetic and PCL, we can move on to installing the necessary ROS packages. Execute the following commands to install the packages.

$ sudo apt-get install ros-kinetic-compressed-depth-image-transport ros-kinetic-compressed-image-transport ros-kinetic-pcl-ros ros-kinetic-camera-info-manager ros-kinetic-calibration ros-kinetic-image-view 
$ sudo apt-get install ros-kinetic-robot-state-publisher ros-kinetic-cmake-modules ros-kinetic-freenect-stack ros-kinetic-openni-launch ros-kinetic-camera-info-manager-py

These should complete without any issues. However, if you compare the commands with the previous installation using ROS Indigo, you will realise that we are missing the driver_base package. This is because the driver_base package is deprecated in ROS Kinetic.

Hence, we will have to download the package and install it ourselves in our catkin workspace. First, download the package from the ROS drivers GitHub repository. Next, we will create our catkin workspace.

#build the catkin workspace
$ mkdir -p ~/workspace/ros/catkin/src
$ cd ~/workspace/ros/catkin
$ catkin_make --force-cmake
$ mkdir -p ~/workspace/ros/rosbuild
$ rosws init ~/workspace/ros/rosbuild ~/workspace/ros/catkin/devel
$ echo "source ~/workspace/ros/rosbuild/setup.bash" >> ~/.bashrc
$ echo "export LC_ALL=C" >> ~/.bashrc
$ source ~/.bashrc

To build the package, just copy the downloaded folder into ~/workspace/ros/catkin/src and execute the following commands.

$ cd ~/workspace/ros/catkin
$ catkin_make

OpenCV3

While ROS Kinetic comes bundled with OpenCV3, we will still have to install a patched branch of OpenCV3, as there are some conflicting errors (as of when this project was done) with CUDA 8. The commands listed are nearly the same as the previous installation of OpenCV3, with the only difference being where the OpenCV3 library is cloned from. We will be cloning from this repository.

$ cd
$ sudo apt-get update

#install dependencies
#Some of the dependencies have changed between Ubuntu 14.04 and 16.04, hence you will be 
#prompted with suggestions after u execute the below command. Update the packages to their
#suggested names/versions and you should be able to install the dependencies without issues.

$ sudo apt-get install libopencv-dev build-essential checkinstall cmake pkg-config yasm libtiff4-dev libjpeg-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev python-dev python-numpy libtbb-dev libqt4-dev libgtk2.0-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils

$ sudo add-apt-repository ppa:mc3man/trusty-media
$ sudo apt update && sudo apt dist-upgrade
$ sudo apt-get install ffmpeg
#if there is no installation candidate for ffmpeg, it is ok to ignore it and
#move on to the next command.  
$ sudo apt-get install frei0r-plugins  

#the new directory to install OpenCV3 in
$ sudo mkdir /usr/local/opencv3
$ mkdir OpenCV  
$ cd OpenCV  
$ git clone https://github.com/daveselinger/opencv
$ git checkout 3.1.0-with-cuda8
$ mkdir release  
$ cd release  
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local/opencv3 -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D ENABLE_FAST_MATH=1 -D CUDA_FAST_MATH=1 -D WITH_CUBLAS=1 ..

$ make –j$(nproc)
$ sudo make install

$ echo “#OpenCV3” >> ~/.bashrc  
$ echo “PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/opencv3/lib/pkgconfig” >> ~/.bashrc  
$ echo “export PKG_CONFIG_PATH” >> ~/.bashrc  
$ source ~/.bashrc

ZED SDK

With the dependencies and libraries for OpenPTrack installed, we will move on to installing the ZED SDK first. Similarly, download the SDK from the Stereolabs site. Pick the latest SDK supported by Ubuntu 16.04 and ROS Kinetic (as of when this project was done, it was SDK v2.0). Plug in your ZED camera and execute the runfile to install their SDK.

As before, test the ZED Depth Viewer, Zed Explorer, and ZED Diagnostics to check for any errors after installation. If you once again encounter the libopencv_core.so.3.1 cannot be found error, please refer to the previous post on how to solve it.

ZED-ROS Wrapper

Similarly, the ROS wrapper for ZED has been updated to v2.0 when we were implementing this project. v2.0 introduces the multi_gpu and multi_cam launch files in the wrapper, which is extremely useful for us when implementing a multi-cam setup, but we will touch on at a later stage. It can be downloaded from this link.

Place the downloaded folder inside your catkin workspace, and execute catkin_make to install the wrapper. Once again, if your installation has completed without error, please execute the following commands to test if the wrapper is functioning properly.

$ roslaunch zed_wrapper zed.launch
#in a new terminal window
$ rosrun image_view image_view image:=/camera/rgb/image_rect_color

You should see a window appear with the ZED camera’s video feed.

Next, to orientate the point cloud published by the wrapper properly in OpenPTrack, we once again have to change the point cloud coordinates in the source code. Open the zed_wrapper_nodelet.cpp file in the ~/workspace/ros/catkin/src/zed-ros-wrapper/src folder. Change lines 299-302 in the publishPointCloud method to those below.

point_cloud.points[i].x = cpu_cloud[i][0];
point_cloud.points[i].y = cpu_cloud[i][1];
point_cloud.points[i].z = cpu_cloud[i][2];
point_cloud.points[i].rgb = cpu_cloud[i][3];

Furthermore, the launch files in v2.0 has increased, with an added zed_camera.launch file. The parameters for resolution and quality of the ZED camera are contained within the zed_camera.launch file instead of the zed.launch file.

The new zed.launch file simple provides a namespace for the camera and calls the zed_camera.launch file. Hence, if you want to vary the resolution of the video feed and the quality of the point cloud, please edit the values in the zed_camera.launch file.

Lastly, you may want to note that the default value of the point_cloud_topic parameter has been changed from previous versions. The topic broadcasting the point cloud is now zed/point_cloud/cloud_registered instead of camera/point_cloud_cloud in previous versions.

It is important that you remember this topic as we require it in the launch files for OpenPTrack later during the installation. (We changed the point cloud topic to zed/point_cloud/cloud, however, it is up to you what you wish to set that value to be).

Once you have made the changes to the wrapper files, you may recompile it with catkin_make again and verify that it is still functional. Now, we can move on to installing OpenPTrack again.

OpenPTrack

In order to install OpenPTrack, we need to trace the openptrack_install.sh file and execute each command individually (executing the script will result in errors as we are not using a compatible OS version).

First, calibration_toolkit_install.sh is executed, which installs the Ceres Solver library. Hence, we have to install a compatible version of Ceres Solver for Ubuntu 16.04. Go to their webpage and follow the instructions to download and install the latest version of Ceres Solver.

Alternatively, you could execute the below commands (replace the version of Ceres Solver with the latest compatible version for Ubuntu 16.04).

#install dependencies
$ sudo apt-get install cmake libgoogle-glog-dev libatlas-base-dev libeigen3-dev libsuitesparse-dev
$ mkdir /tmp/ceres-install
$ cd /tmp/ceres-install
$ wget ceres-solver.org/ceres-solver-1.12.0.tar.gz
$ tar xzvf ceres-solver-1.12.0.tar.gz
$ mkdir ceres-bin && cd ceres-bin
$ cmake ../ceres-solver-1.12.0
$ make -j${nproc}
$ make test
$ sudo make install
$ sudo rm -R /tmp/ceres_install

Once successfully installed, we clone the calibration toolkit repository into our catkin workspace.

$ cd ~/workspace/ros/catkin/src
$ git clone https://github.com/iaslab-unipd/calibration_toolkit 
$ cd calibration_toolkit 
$ git fetch origin --tags
$ git checkout tags/v0.2

Next, if we follow openptrack_install.sh, we would need to execute both libfreenect_update.sh and mesa_install.sh, which are scripts to install drivers for the Kinect and SwissRanger cameras respectively.

As we are using the ZED camera instead, we need not install these drivers and can skip these 2 scripts. Hence, we would move on to building the message packages needed for OpenPTrack. You may execute the below commands to do so.

$ cd ~/workspace/ros/catkin
$ catkin_make --pkg calibration_msgs
$ catkin_make --pkg opt_msgs
$ catkin_make --force-cmake

This will result in a large number of errors and warnings (as the code still uses OpenCV2 and many files are configured for ROS Jade and earlier instead of ROS Kinectic). We will have to address each error individually. The list of possible errors are as follows:

Eigen3 deprecated warning. If you encounter an error such as
```
The FindEigen.cmake Module in the cmake_modules package is deprecated.
```
you would need to change all instances of Eigen to Eigen3 in all the CMakelist.txt files (there is 1 CMakelist.txt file in every package of OpenPTrack, check all the files and ensure every file uses Eigen3 instead of Eigen). You would also need to add
```
set(EIGEN3_INCLUDE_DIRS ${EIGEN3_INCLUDE_DIR})
```
after each instance of find_package(Eigen3). Examples of the change is as follows:
- Replace any instance of
```
find_package(Eigen)
include_directories(${Eigen_INCLUDE_DIRS} include ${catkin_INCLUDE_DIRS})
```
  with
```
find_package(Eigen3 REQUIRED)
set(EIGEN3_INCLUDE_DIRS ${EIGEN3_INCLUDE_DIR})
include_directories(${EIGEN3_INCLUDE_DIRS} include ${catkin_INCLUDE_DIRS})
```
- Replace any instance of
```
catkin_package( ...
  DEPENDS eigen
  ... 
)
```
  with
```
catkin_package (...
  INCLUDE_DIRS ${EIGEN3_INCLUDE_DIRS}
  ...
)
```
  More information about this change can be found here http://wiki.ros.org/jade/Migration.
OpenCV dependency in CMake files. In each CMakelist.txt file, replace all instances of
```
catkin_package (...
  DEPENDS opencv2
  ...
)
```
with
```
catkin_package(...
  DEPENDS OPENCV
  ...
)
```
This will ensure that the OpenCV3 library will be looked for when making the packages instead of OpenCV2 library.

Please note that any instance of find_package(OpenCV Required) need not be changed, as it would find OpenCV3 automatically in ROS Kinetic. Furthermore, for each CMakelist.txt file that you have changed, please check that the package.xml file in the same folder has a both a run and build dependency on either cv_bridge or image_geometry or another package with that dependency already. The package.xml file should not contain a dependency on OpenCV3. More information can be found here.
Missing dependencies warning.
```
CMAKE WARNING catkin_package() DEPENDS on ‘X’ but neither ‘X_INCLUDE_DIRS’ nor ‘X_LIBRARIES’ is defined.
```
This warning states that the dependencies you have installed do not have the default names (which CMake expects) for their include directories or libraries. This warning can be safely ignored, however, if you wish to address it, you may make the following changes to all the CMakelist.txt files. Replace all instances of
```
catkin_package(...
  DEPENDS pcl ceres
  ...
)
```
with
```
catkin_package(...
  DEPENDS PCL CERES
  ...
)
```
Missing ros.h file.
```
Fatal error, ros/ros.h cannot be found.
```
If you encounter this error, please open the CMakelist.txt file in ~/workspace/ros/catkin/src/open_ptrack/optmsgs. Include the catkin_INCLUDE_DIRS by making the following change: Replace
```
include_directories(include include/open_ptrack/${PROJECT_NAME}/)
```
with
```
include_directories(include ${catkin_INCLUDE_DIRS} include/open_ptrack/${PROJECT_NAME}/)
```
Furthermore, please check the package.xml file in the same folder for a build and run dependency on roscpp.
Missing rule error.
```
No rule to make /usr/lib/liblapack.so.3gf needed for ...
```
If this error is encountered, just execute the following command to install libopenblas-base.
```
$ sudo apt-get install libopenblas-base
```

Syntax errors, for example:

/bin/sh: 1: Syntax error: "(" unexpected open_ptrack/detection/CMakeFiles/haar_disp_ada.dir/build.make:89: recipe for target 'open_ptrack/detection/CMakeFiles/haar_disp_ada.dir/apps/haardispada_nodelet.cpp.o' failed

or any other similar error.
Despite what the error shows, this is not a bash error, but instead an issue with the VTK library, as OpenPTrack uses VTK5 while we are now using VTK6. Hence, we should open the CMakelist.txt file for the folder stated in the error message (~workspace/ros/catkin/src/open_ptrack/detection in this case) and make the following changes: From

find_package(VTK REQUIRED)
if(VTK_FOUND)
include_directories(${VTK_INCLUDE_DIRS})
link_directories(${VTK_LIBRARY_DIRS})
add_definitions(${VTK_DEFINITIONS})
include (${VTK_USE_FILE})
endif()

find_package(VTK 6.0 REQUIRED NO_MODULE)
if(VTK_FOUND)
include_directories(${VTK_INCLUDE_DIRS})
link_directories(${VTK_LIBRARY_DIRS})
  #add_definitions(${VTK_DEFINITIONS})
include (${VTK_USE_FILE})
endif()

Repeat this fix on every folder which such an error appears. More information on this fix can be found at the ROS migration wiki.

Missing libraries.

/usr/bin/ld: cannot find -lvtkHybrid
/usr/bin/ld: cannot find -lvtkRendering

As mentioned in the previous point, we have to migrate from using VTK 5 to VTK 6. In VTK6, the libraries VTKHybrid and VTKRendering has been split into multiple other modules (ie. VTKRenderingCore and VTKRenderingOpenGL). To find the libraries and to keep things simple (so we do no need to identify which part of VTKRendering and VTKHybrid OpenPTrack previously used) we can make the following changes in ~/workspace/ros/catkin/src/open_ptrack/detection/CMakeList.txt from

add_executable(ground_based_people_detector apps/ground_based_people_detector_node.cpp)
SET_TARGET_PROPERTIES(ground_based_people_detector PROPERTIES LINK_FLAGS -L${PCL_LIBRARY_DIRS})
target_link_libraries(ground_based_people_detector ${PROJECT_NAME} ${catkin_LIBRARIES} ${PCL_LIBRARIES} vtkHybrid vtkRendering)

add_executable(ground_based_people_detector_sr apps/ground_based_people_detector_node_sr.cpp)
SET_TARGET_PROPERTIES(ground_based_people_detector_sr PROPERTIES LINK_FLAGS -L${PCL_LIBRARY_DIRS})
target_link_libraries(ground_based_people_detector_sr ${PROJECT_NAME} ${catkin_LIBRARIES} ${PCL_LIBRARIES} vtkHybrid vtkRendering)

add_executable(ground_based_people_detector apps/ground_based_people_detector_node.cpp)
SET_TARGET_PROPERTIES(ground_based_people_detector PROPERTIES LINK_FLAGS -L${PCL_LIBRARY_DIRS})
target_link_libraries(ground_based_people_detector ${PROJECT_NAME} ${catkin_LIBRARIES} ${PCL_LIBRARIES} ${VTK_LIBRIARIES})

add_executable(ground_based_people_detector_sr apps/ground_based_people_detector_node_sr.cpp)
SET_TARGET_PROPERTIES(ground_based_people_detector_sr PROPERTIES LINK_FLAGS -L${PCL_LIBRARY_DIRS})
target_link_libraries(ground_based_people_detector_sr ${PROJECT_NAME} ${catkin_LIBRARIES} ${PCL_LIBRARIES} ${VTK_LIBRARIES})

Migration of OpenCV2 code to OpenCV3. Lastly, we have to change the source code to use OpenCV3 instead of OpenCV2. You would need to update the module headers from
```
#include “opencv2/<module>/<module>.hpp”
```
to
```
#include “opencv2/<module>.hpp”
```
Next, you would need to update the namespaces for some of the classes (ie. CvBoost) and the API call for some of the classes (ie. CvBoost.train). The main files you would need to change are ~/workspace/ros/catkin/src/openptrack/detection/src/haardispada.cpp and ~/workspace/ros/catkin/src/openptrack/detection/include/openptrack/detection/haardispada.h

The OpenCV documentation provides more elaboration and a migration guide.

Now, if you execute the make commands again, OpenPTrack should compile successfully.

$ cd ~/workspace/ros/catkin
$ catkin_make --pkg calibration_msgs
$ catkin_make --pkg opt_msgs
$ catkin_make --force-cmake

ZED launch files

Lastly, we need to create the launch files and config files for our ZED camera in OpenPTrack. More information about how to create the files can be found in the previous post, but please note that the files you need to create are:

ground_based_people_detector_zed.yaml in ~/workspace/ros/catkin/src/open_ptrack/detection/conf.
detector_depth_zed.launch in ~/workspace/ros/catkin/src/open_ptrack/detection/launch. (Please note the pointcloud_topic and camera_info_topic which you are using now. These are provided in the zed_camera.launch file in the ROS wrapper.)
detection_ and_tracking_zed.launch in ~/workspace/ros/catkin/src/open_ptrack/tracking/launch.

With this, you should be able to run a single camera pedestrian tracking with

roslaunch tracking detection_and_tracking_zed.launch

We have successfully run OpenPTrack with Ubuntu 16.04!

Read Part 1: Setting up a 3D pedestrian tracking system
Read Part 3: Multi-camera setup with OpenPTrack