OpenVino Driver Behaviour

Table of Contents

• 1. Foreword
• 2. Introduction
  • 2.1. [Optional] Simulator
  • 2.2. [Optional] AWS (In Progress)
• 3. Bussines Logic
  • 3.1. Blink/Yawn detection
  • 3.2. 'Eyes on the road' detection
• 4. Prerequisites
  • 4.1. Hardware
  • 4.2. Software
  • 4.3. Checks
• 5. Building
  • 5.1. Basic Build
  • 5.2. Simulator Build
• 6. Usage
  • 6.1. Run
    • 6.1.1. Face Detection
      • 6.1.1.1. Other models
    • 6.1.2. Drowsiness detection
    • 6.1.3. Driver 'eyes on the road' detection
    • 6.1.4. Realtime feed
    • 6.1.5. Driver recognition
    • 6.1.6. Simulator
• 7. To Do
  • 7.1. README
  • 7.2. Development

1. Foreword

This is a follow-up on the OpenVino’s inference tutorials samples:

https://software.intel.com/en-us/articles/OpenVINO-IE-Samples#interactive-face-detection

We will focus on the "Interactive face detection" sample, and we will work on and extend this tutorial as a demo app aimed to the automotive industry.

2. Introduction

This project consists on showcasing the advantages of the Intel’s OpenVINO toolkit. We will develop a Driver Behaviour case scenario, where we will detect drowsiness based on blinking and yawning and gaze direction. For that, we will use the OpenVINO toolkit and OpenCV, all written in C++.

As mentioned previously, we will take the Interactive face detection sample as a starting point, as it provides us with the options to run and stack different models synchronously or asynchronously. We will develop the following features based on computer vision:

1. Sleep/Drowsiness Detection:

   1. Counting frecuency of blinking.

   2. Yawn detection.

2. Gaze detection.

2.1. [Optional] Simulator

To test our system with data closer to reality we added support for ETS or ATS. As the simulator is not free, you can opt whether to compile the project with this feature or not. The communication between the simulator and our program is done via a ROS2 client and it provides the following info:

1. Engine Status (On/Off)

2. Trailer Status (Connected/Disconnected).

3. Speed.

4. RPM.

5. Acceleration.

6. Position (Coordinates).

7. Gear (-1 for Reverse, >0 the rest).

2.2. [Optional] AWS (In Progress)

We also plan to send the data through MQTT using AWS IoT-Core, to produce a dashboard with the trucks positions, alarms, etc. Again, using AWS may incur in a cost, so this will also be optional for you to compile with/without it.

3. Bussines Logic

Using OpenVino’s model detection we can easily detect faces with great accuracy. We are currently using for testing 2 different face detection models that are included with OpenVino out-of-the-box:

1. face-detection-adas-0001

2. face-detection-retail-0004

3.1. Blink/Yawn detection

Using the image detected inside the face ROI (region of interest), we feed a facial landmarks detector to identify points of iterest. Using 6 points for each eye and 6 points for the mouth it is possible to calculate 'Eye Aspect Ratio (EAR)' that gives 2 values for eye/mouth open or closed (based on this paper).

At the moment of writing this guide, the facial landmarks detection model included with OpenVino (facial-landmarks-35-adas-0001) has not enough points to run this calculations. We are using dlib’s facial landmarks detector instead.

Once we have a positive detection for blink/yawn, we count frames of those events and trigger an alarm when they hit a threshold.

3.2. 'Eyes on the road' detection

Using the face’s ROI, we feed a head-pose detector model provided by OpenVino (head-pose-estimation-adas-0001). Analizing the output of that model we can easily detect when the face is not centered or not looking to the front.

4. Prerequisites

To run the application in this tutorial, the OpenVINO™ toolkit and its dependencies must already be installed and verified using the included demos. Installation instructions may be found at: https://software.intel.com/en-us/articles/OpenVINO-Install-Linux

If to be used, any optional hardware must also be installed and verified including:

• USB camera - Standard USB Video Class (UVC) camera.

• Intel® Core™ CPU with integrated graphics.

• VPU - USB Intel® Movidius™ Neural Compute Stick and what is being referred to as "Myriad"

A summary of what is needed:

4.1. Hardware

• Target and development platforms meeting the requirements described in the "System Requirements" section of the OpenVINO™ toolkit documentation which may be found at: https://software.intel.com/en-us/openvino-toolkit

Note: While writing this tutorial, an Intel® i7-8550U with Intel® HD graphics 520 GPU was used as both the development and target platform.

• Optional:

  • Intel® Movidius™ Neural Compute Stick

  • USB UVC camera

  • Intel® Core™ CPU with integrated graphics.

4.2. Software

• OpenVINO™ toolkit supported Linux operating system. This tutorial was run on 64-bit Ubuntu 16.04.1 LTS updated to kernel 4.15.0-43 following the OpenVINO™ toolkit installation instructions.

• The latest OpenVINO™ toolkit installed and verified. This tutorial was written using version 2018 R4.0. Support for R5 has recently been added and +2019.

• Git(git) for downloading from the GitHub repository.

• BOOST library. To install on Ubuntu, run:

apt-get install libboost-dev

• [Optional] ETS or ATS simulator. Install it through Steam on Ubuntu.

• [Optional] AWS Crt Cpp

4.3. Checks

By now you should have completed the Linux installation guide for the OpenVINO™ toolkit, however before continuing, please ensure:

• That after installing the OpenVINO™ toolkit you have run the supplied demo samples

• If you have and intend to use a GPU: You have installed and tested the GPU drivers

• If you have and intend to use a USB camera: You have connected and tested the USB camera

• If you have and intend to use a Myriad: You have connected and tested the USB Intel® Movidius™ Neural Compute Stick

• That your development platform is connected to a network and has Internet access. To download all the files for this tutorial, you will need to access GitHub on the Internet.

5. Building

5.1. Basic Build

1. Clone the repository at desired location:

git clone https://github.com/incluit/OpenVino-Driver-Behaviour.git

2. The first step is to configure the build environment for the OpenCV toolkit by sourcing the "setupvars.sh" script.

source  /opt/intel/computer_vision_sdk/bin/setupvars.sh

3. Change to the top git repository:

cd OpenVino-Driver-Behaviour

4. Create a directory to build the tutorial in and change to it.

mkdir build
cd build

5. Before running each of the following sections, be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models:

source ../scripts/setupenv.sh

6. Compile:

cmake -DCMAKE_BUILD_TYPE=Release ../
make

5.2. Simulator Build

In order to run the simulator you will need to install:

• ROS2 following Installing ROS2 via Debian Packages.

• colcon following Colcon Tutorial.

• ETS or ATS using Steam.

• ets_ros2-plugin from this repo.

Follow the plugin’s instructions to install everything, you can test the ros client is working through the sample application provided there. Once that is working we can build or program.

1. Clone the repository at <ros2_workspace>/src/ets_ros2 location:

<ros2_ws>/src/ets_ros2$ git clone https://github.com/incluit/OpenVino-Driver-Behaviour.git

2. Source everything!

source /opt/intel/openvino/bin/setupvars.sh
source /opt/ros/<ros-version>/setup.bash
source <ros2_ws>src/ets_ros2/OpenVino-Driver-Behaviour/scripts/setupenv.sh

The OpenVino path may vary depending on the version installed. If it’s a 2018 Release or older, the path should be:

source /opt/intel/computer_vision_sdk/bin/setupvars.sh

3. Compile:

colcon build --symlink-install --parallel-workers N --cmake-args -DSIMULATOR=ON

N being the number of cores to build (like make’s -jN flag). We recommend using 1 as it is a bit memory intensive.

4. Copy the plugin to the corresponding folder as described in the plugin repo:

   mkdir  ~/.local/share/Steam/steamapps/common/Euro\ Truck\ Simulator\ 2/bin/linux_x64/plugins
   cp install/ets_plugin/lib/ets_plugin/libetsros2.so ~/.local/share/Steam/steamapps/common/Euro\ Truck\ Simulator\ 2/bin/linux_x64/plugins/

or the ATS folder:

   mkdir ~/.local/share/Steam/steamapps/common/American\ Truck\ Simulator/bin/linux_x64/plugins
   cp install/ets_plugin/lib/ets_plugin/libetsros2.so ~/.local/share/Steam/steamapps/common/American\ Truck\ Simulator/bin/linux_x64/plugins/

5. Lastly, source our workspace and copy the data directory:

source <ros2_ws>/install/setup.bash
cd <ros2_ws>/build/driver_behavior
cp -r ../../src/ets_ros2/OpenVino-Driver-Behaviour/data ../

6. Usage

6.1. Run

6.1.1. Face Detection

1. First, let us see how face detection works on a single image file using the default synchronous mode.

./intel64/Release/driver_behaviour_tutorial -m $face132 -i ../data/img_1.jpg

2. For video files:

./intel64/Release/driver_behaviour_tutorial -m $face132 -i ../data/video1.mp4

3. You can also run the command in asynchronous mode using the option "-async":

./intel64/Release/driver_behaviour_tutorial -m $face132 -i ../data/video1.mp4 -async

4. You can also load the models into the GPU or MYRIAD:

Note: In order to run this section, the GPU and/or MYRIAD are required to be present and correctly configured.

./intel64/Release/driver_behaviour_tutorial -m $face132 -d GPU -i ../data/video1.mp4

./intel64/Release/driver_behaviour_tutorial -m $face132 -d MYRIAD -i ../data/video1.mp4

6.1.1.1. Other models

You can also experiment by using different face detection models, being the ones available up to now:

1. face-detection-adas-0001:

   • -m $face1{16,32}

2. face-detection-retail-0004:

   • -m $face2{16,32}

By default they will be loaded into the CPU, so remember to pass the corresponding argument:

• -d {CPU,GPU,MYRIAD}

6.1.2. Drowsiness detection

In order to enable drowsiness and yawn detection, we add to the pipeline a face landmarks detection.

./intel64/Release/driver_behaviour_tutorial -m $face232 -dlib_lm -i ../data/video2.mp4

6.1.3. Driver 'eyes on the road' detection

To analize if the driver is paying attention to the road, we enable the head/pose model and work with that information:

./intel64/Release/driver_behaviour_tutorial -m $face232 -m_hp $hp32 -i ../data/video3.mp4

6.1.4. Realtime feed

Removing the '-i' flag, if the computer has a video camera enabled, the programs uses its feed to run the face detection models and the following calculations.

./intel64/Release/driver_behaviour_tutorial -m $face232
./intel64/Release/driver_behaviour_tutorial -m $face232 -dlib_lm
./intel64/Release/driver_behaviour_tutorial -m $face232 -d GPU -dlib_lm -async
./intel64/Release/driver_behaviour_tutorial -m $face232 -m_hp $hp32

6.1.5. Driver recognition

We could also detect if the person sitting in front of the camera is actually an authorized driver. For that matter, we added a first stage of driver recognition that works as follows:

In drivers/ there are pictures of "authorized drivers", you can add yours by taking a picture of yourself and cropping your face as you can see in the sample pictures, name the file as name.N.png. Then navigate to the scripts/ and generate the .json.

cd scripts/
python3 create_list.py ../drivers/

You should now see a file named faces_gallery.json with your name and the path to your photo there.

Now we can run the program with the flag -d_recognition and the path to the .json file -fg ../scripts/faces_gallery.json as follows:

./intel64/Release/driver_behaviour_tutorial -m $face232 -d CPU -m_hp $hp32 -d_hp CPU -dlib_lm -d_recognition -fg ../scripts/faces_gallery.json

It will wait there until an authorized driver sits in front of the camera for a couple of seconds and then will continue with the previous features.

For this feature we are making use of the next models that are available within OpenVINO’s distribution:

1. face-reidentification-retail-0095: For R5

2. face-reidentification-retail-0071: For R4

6.1.6. Simulator

If you compiled with the simulator, you may run all together. We consider the next use-cases to show on the screen:

1. System off if Engine = Off.

2. "Eyes out of the road" enable (inferred by Head Position) when [GearStatus = Driving] and [VehicleSpeed > 5 kmh].

3. "Eyes out of the road" disabled (inferred by Head Position) when Gear Status = Reverse.

4. "Eyes out of the road" disabled (inferred by Head Position) when Gear Status = Parking.

5. "Stop looking at (…​)" detection (inferred by Head Position) when [GearStatus = Driving] and [VehicleSpeed > 2 kmh].

6. "Stop looking at (…​)" disabled (inferred by Head Position) when [GearStatus = Reverse].

7. "Stop looking at (…​)" disabled (inferred by Head Position) when [GearStatus = Parking].

8. "Drowsiness state" detection (inferred by Blink and Yawn detection) when [GearStatus = Driving].

9. "Drowsiness state" detection (inferred by Blink and Yawn detection) when [GearStatus = Reverse].

10. "Drowsiness state" disabled (inferred by Blink and Yawn detection) when [GearStatus = Parking].

7. To Do

7.1. README

• ✓ Short README with usage examples

• ✓ Travis + Sonarcloud

• ❏ Include diagrams and images

• ❏ Elaborate on the wiki

7.2. Development

• ✓ Try with different models

• ✓ Face detection

• ✓ Dlib landmark idetification integration

• ✓ Blink/Yawn detection

• ✓ Blink/Yawn time

• ✓ 'Eye out of road' detection

• ✓ Face identification

• ❏ Heart rate + speed/acceleration patterns risk