diff --git a/docs/HOWTO/Custom_Layers_Guide.md b/docs/HOWTO/Custom_Layers_Guide.md index 5eb458ab6e8389..f3a31cba8912d8 100644 --- a/docs/HOWTO/Custom_Layers_Guide.md +++ b/docs/HOWTO/Custom_Layers_Guide.md @@ -371,7 +371,6 @@ python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/benchmark_tool/benchmark_app. - [Inference Engine Extensibility Mechanism](../IE_DG/Extensibility_DG/Intro.md) - [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) - [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_intel_index) -- [Inference Engine Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic) - For IoT Libraries and Code Samples see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit). ## Converting Models: diff --git a/docs/IE_DG/protecting_model_guide.md b/docs/IE_DG/protecting_model_guide.md index 99b7836b1b25d1..2074d2230146cb 100644 --- a/docs/IE_DG/protecting_model_guide.md +++ b/docs/IE_DG/protecting_model_guide.md @@ -59,5 +59,4 @@ should be called with `weights` passed as an empty `Blob`. - Inference Engine Developer Guide: [Inference Engine Developer Guide](Deep_Learning_Inference_Engine_DevGuide.md) - For more information on Sample Applications, see the [Inference Engine Samples Overview](Samples_Overview.md) - For information on a set of pre-trained models, see the [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_intel_index) -- For information on Inference Engine Tutorials, see the [Inference Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic) - For IoT Libraries and Code Samples see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit). diff --git a/docs/doxygen/openvino_docs.xml b/docs/doxygen/openvino_docs.xml index 58df638978b506..a6e401e4523698 100644 --- a/docs/doxygen/openvino_docs.xml +++ b/docs/doxygen/openvino_docs.xml @@ -96,6 +96,19 @@ + + + + + + + + + + + + + diff --git a/docs/gapi/face_beautification.md b/docs/gapi/face_beautification.md new file mode 100644 index 00000000000000..539b0ca9b7e67e --- /dev/null +++ b/docs/gapi/face_beautification.md @@ -0,0 +1,435 @@ +# Implementing a Face Beautification Algorithm {#openvino_docs_gapi_face_beautification} + +## Introduction +In this tutorial you will learn: + +* Basics of a sample face beautification algorithm; +* How to infer different networks inside a pipeline with G-API; +* How to run a G-API pipeline on a video stream. + +## Prerequisites +This sample requires: + +* PC with GNU/Linux* or Microsoft Windows* (Apple macOS* is supported but was not tested) +* OpenCV 4.2 or higher built with [Intel® Distribution of OpenVINO™ Toolkit](https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html) (building with [Intel® TBB](https://www.threadingbuildingblocks.org/intel-tbb-tutorial) is a plus) +* The following pre-trained models from the [Open Model Zoo](@ref omz_models_intel_index) + * [face-detection-adas-0001](@ref omz_models_intel_face_detection_adas_0001_description_face_detection_adas_0001) + * [facial-landmarks-35-adas-0002](@ref omz_models_intel_facial_landmarks_35_adas_0002_description_facial_landmarks_35_adas_0002) + +To download the models from the Open Model Zoo, use the [Model Downloader](@ref omz_tools_downloader_README) tool. + +## Face Beautification Algorithm +We will implement a simple face beautification algorithm using a combination of modern Deep Learning techniques and traditional Computer Vision. The general idea behind the algorithm is to make face skin smoother while preserving face features like eyes or a mouth contrast. The algorithm identifies parts of the face using a DNN inference, applies different filters to the parts found, and then combines it into the final result using basic image arithmetics: + +![Face Beautification Algorithm](../img/gapi_face_beautification_algorithm.png) + +Briefly the algorithm is described as follows: + +Briefly the algorithm is described as follows: +- Input image \f$I\f$ is passed to unsharp mask and bilateral filters + (\f$U\f$ and \f$L\f$ respectively); +- Input image \f$I\f$ is passed to an SSD-based face detector; +- SSD result (a \f$[1 \times 1 \times 200 \times 7]\f$ blob) is parsed and converted to an array of faces; +- Every face is passed to a landmarks detector; +- Based on landmarks found for every face, three image masks are generated: + - A background mask \f$b\f$ -- indicating which areas from the original image to keep as-is; + - A face part mask \f$p\f$ -- identifying regions to preserve (sharpen). + - A face skin mask \f$s\f$ -- identifying regions to blur; +- The final result \f$O\f$ is a composition of features above calculated as \f$O = b*I + p*U + s*L\f$. + +Generating face element masks based on a limited set of features (just 35 per face, including all its parts) is not very trivial and is described in the sections below. + +## Constructing a G-API Pipeline + +### Declare Deep Learning Topologies +This sample is using two DNN detectors. Every network takes one input and produces one output. In G-API, networks are defined with macro G_API_NET(): +```cpp +G_API_NET(FaceDetector, , "face_detector"); +G_API_NET(LandmDetector, , "landm_detector"); +``` +To get more information, see Declaring Deep Learning topologies described in the "Face Analytics pipeline" tutorial. + +### Describe the Processing Graph +The code below generates a graph for the algorithm above: +```cpp +cv::GComputation pipeline([=]() +{ + cv::GMat gimgIn; // input + cv::GMat faceOut = cv::gapi::infer(gimgIn); + GArrayROI garRects = custom::GFacePostProc::on(faceOut, gimgIn, config::kConfThresh); // post-proc + cv::GArray landmOut = cv::gapi::infer(garRects, gimgIn); + cv::GArray garElems; // | + cv::GArray garJaws; // |output arrays + std::tie(garElems, garJaws) = custom::GLandmPostProc::on(landmOut, garRects); // post-proc + cv::GArray garElsConts; // face elements + cv::GArray garFaceConts; // whole faces + std::tie(garElsConts, garFaceConts) = custom::GGetContours::on(garElems, garJaws); // interpolation + cv::GMat mskSharp = custom::GFillPolyGContours::on(gimgIn, garElsConts); // | + cv::GMat mskSharpG = cv::gapi::gaussianBlur(mskSharp, config::kGKernelSize, // | + config::kGSigma); // | + cv::GMat mskBlur = custom::GFillPolyGContours::on(gimgIn, garFaceConts); // | + cv::GMat mskBlurG = cv::gapi::gaussianBlur(mskBlur, config::kGKernelSize, // | + config::kGSigma); // |draw masks + // The first argument in mask() is Blur as we want to subtract from // | + // BlurG the next step: // | + cv::GMat mskBlurFinal = mskBlurG - cv::gapi::mask(mskBlurG, mskSharpG); // | + cv::GMat mskFacesGaussed = mskBlurFinal + mskSharpG; // | + cv::GMat mskFacesWhite = cv::gapi::threshold(mskFacesGaussed, 0, 255, cv::THRESH_BINARY); // | + cv::GMat mskNoFaces = cv::gapi::bitwise_not(mskFacesWhite); // | + cv::GMat gimgBilat = custom::GBilatFilter::on(gimgIn, config::kBSize, + config::kBSigmaCol, config::kBSigmaSp); + cv::GMat gimgSharp = custom::unsharpMask(gimgIn, config::kUnshSigma, + config::kUnshStrength); + // Applying the masks + // Custom function mask3C() should be used instead of just gapi::mask() + // as mask() provides CV_8UC1 source only (and we have CV_8U3C) + cv::GMat gimgBilatMasked = custom::mask3C(gimgBilat, mskBlurFinal); + cv::GMat gimgSharpMasked = custom::mask3C(gimgSharp, mskSharpG); + cv::GMat gimgInMasked = custom::mask3C(gimgIn, mskNoFaces); + cv::GMat gimgBeautif = gimgBilatMasked + gimgSharpMasked + gimgInMasked; + return cv::GComputation(cv::GIn(gimgIn), cv::GOut(gimgBeautif, + cv::gapi::copy(gimgIn), + garFaceConts, + garElsConts, + garRects)); +}); +``` +The resulting graph is a mixture of G-API's standard operations, user-defined operations (namespace custom::), and DNN inference. The generic function `cv::gapi::infer<>()` allows you to trigger inference within the pipeline; networks to infer are specified as template parameters. The sample code is using two versions of `cv::gapi::infer<>()`: + +* A frame-oriented one is used to detect faces on the input frame. +* An ROI-list oriented one is used to run landmarks inference on a list of faces – this version produces an array of landmarks per every face. +More on this in "Face Analytics pipeline" ([Building a GComputation](@ref gapi_ifd_gcomputation) section). + +### Unsharp mask in G-API +The unsharp mask \f$U\f$ for image \f$I\f$ is defined as: + +\f[U = I - s * L(M(I)),\f] + +where \f$M()\f$ is a median filter, \f$L()\f$ is the Laplace operator, and \f$s\f$ is a strength coefficient. While G-API doesn't provide this function out-of-the-box, it is expressed naturally with the existing G-API operations: + +```cpp +inline cv::GMat custom::unsharpMask(const cv::GMat &src, + const int sigma, + const float strength) +{ + cv::GMat blurred = cv::gapi::medianBlur(src, sigma); + cv::GMat laplacian = custom::GLaplacian::on(blurred, CV_8U); + return (src - (laplacian * strength)); +} +``` +Note that the code snipped above is a regular C++ function defined with G-API types. Users can write functions like this to simplify graph construction; when called, this function just puts the relevant nodes to the pipeline it is used in. + +## Custom Operations +The face beautification graph is using custom operations extensively. This chapter focuses on the most interesting kernels, refer to G-API Kernel API for general information on defining operations and implementing kernels in G-API. + +### Face detector post-processing +A face detector output is converted to an array of faces with the following kernel: + +```cpp +using VectorROI = std::vector; +GAPI_OCV_KERNEL(GCPUFacePostProc, GFacePostProc) +{ + static void run(const cv::Mat &inDetectResult, + const cv::Mat &inFrame, + const float faceConfThreshold, + VectorROI &outFaces) + { + const int kObjectSize = 7; + const int imgCols = inFrame.size().width; + const int imgRows = inFrame.size().height; + const cv::Rect borders({0, 0}, inFrame.size()); + outFaces.clear(); + const int numOfDetections = inDetectResult.size[2]; + const float *data = inDetectResult.ptr(); + for (int i = 0; i < numOfDetections; i++) + { + const float faceId = data[i * kObjectSize + 0]; + if (faceId < 0.f) // indicates the end of detections + { + break; + } + const float faceConfidence = data[i * kObjectSize + 2]; + // We can cut detections by the `conf` field + // to avoid mistakes of the detector. + if (faceConfidence > faceConfThreshold) + { + const float left = data[i * kObjectSize + 3]; + const float top = data[i * kObjectSize + 4]; + const float right = data[i * kObjectSize + 5]; + const float bottom = data[i * kObjectSize + 6]; + // These are normalized coordinates and are between 0 and 1; + // to get the real pixel coordinates we should multiply it by + // the image sizes respectively to the directions: + cv::Point tl(toIntRounded(left * imgCols), + toIntRounded(top * imgRows)); + cv::Point br(toIntRounded(right * imgCols), + toIntRounded(bottom * imgRows)); + outFaces.push_back(cv::Rect(tl, br) & borders); + } + } + } +}; +``` + +### Facial Landmarks Post-Processing +The algorithm infers locations of face elements (like the eyes, the mouth and the head contour itself) using a generic facial landmarks detector (details) from OpenVINO™ Open Model Zoo. However, the detected landmarks as-is are not enough to generate masks — this operation requires regions of interest on the face represented by closed contours, so some interpolation is applied to get them. This landmarks processing and interpolation is performed by the following kernel: +```cpp +GAPI_OCV_KERNEL(GCPUGetContours, GGetContours) +{ + static void run(const std::vector &vctPtsFaceElems, // 18 landmarks of the facial elements + const std::vector &vctCntJaw, // 17 landmarks of a jaw + std::vector &vctElemsContours, + std::vector &vctFaceContours) + { + size_t numFaces = vctCntJaw.size(); + CV_Assert(numFaces == vctPtsFaceElems.size()); + CV_Assert(vctElemsContours.size() == 0ul); + CV_Assert(vctFaceContours.size() == 0ul); + // vctFaceElemsContours will store all the face elements' contours found + // in an input image, namely 4 elements (two eyes, nose, mouth) for every detected face: + vctElemsContours.reserve(numFaces * 4); + // vctFaceElemsContours will store all the faces' contours found in an input image: + vctFaceContours.reserve(numFaces); + Contour cntFace, cntLeftEye, cntRightEye, cntNose, cntMouth; + cntNose.reserve(4); + for (size_t i = 0ul; i < numFaces; i++) + { + // The face elements contours + // A left eye: + // Approximating the lower eye contour by half-ellipse (using eye points) and storing in cntLeftEye: + cntLeftEye = getEyeEllipse(vctPtsFaceElems[i][1], vctPtsFaceElems[i][0]); + // Pushing the left eyebrow clock-wise: + cntLeftEye.insert(cntLeftEye.end(), {vctPtsFaceElems[i][12], vctPtsFaceElems[i][13], + vctPtsFaceElems[i][14]}); + // A right eye: + // Approximating the lower eye contour by half-ellipse (using eye points) and storing in vctRightEye: + cntRightEye = getEyeEllipse(vctPtsFaceElems[i][2], vctPtsFaceElems[i][3]); + // Pushing the right eyebrow clock-wise: + cntRightEye.insert(cntRightEye.end(), {vctPtsFaceElems[i][15], vctPtsFaceElems[i][16], + vctPtsFaceElems[i][17]}); + // A nose: + // Storing the nose points clock-wise + cntNose.clear(); + cntNose.insert(cntNose.end(), {vctPtsFaceElems[i][4], vctPtsFaceElems[i][7], + vctPtsFaceElems[i][5], vctPtsFaceElems[i][6]}); + // A mouth: + // Approximating the mouth contour by two half-ellipses (using mouth points) and storing in vctMouth: + cntMouth = getPatchedEllipse(vctPtsFaceElems[i][8], vctPtsFaceElems[i][9], + vctPtsFaceElems[i][10], vctPtsFaceElems[i][11]); + // Storing all the elements in a vector: + vctElemsContours.insert(vctElemsContours.end(), {cntLeftEye, cntRightEye, cntNose, cntMouth}); + // The face contour: + // Approximating the forehead contour by half-ellipse (using jaw points) and storing in vctFace: + cntFace = getForeheadEllipse(vctCntJaw[i][0], vctCntJaw[i][16], vctCntJaw[i][8]); + // The ellipse is drawn clock-wise, but jaw contour points goes vice versa, so it's necessary to push + // cntJaw from the end to the begin using a reverse iterator: + std::copy(vctCntJaw[i].crbegin(), vctCntJaw[i].crend(), std::back_inserter(cntFace)); + // Storing the face contour in another vector: + vctFaceContours.push_back(cntFace); + } + } +}; +``` +The kernel takes two arrays of denormalized landmarks coordinates and returns an array of elements' closed contours and an array of faces' closed contours; in other words, outputs are, the first, an array of contours of image areas to be sharpened and, the second, another one to be smoothed. + +Here and below `Contour` is a vector of points. + +#### Get an Eye Contour +Eye contours are estimated with the following function: +```cpp +inline int custom::getLineInclinationAngleDegrees(const cv::Point &ptLeft, const cv::Point &ptRight) +{ + const cv::Point residual = ptRight - ptLeft; + if (residual.y == 0 && residual.x == 0) + return 0; + else + return toIntRounded(atan2(toDouble(residual.y), toDouble(residual.x)) * 180.0 / CV_PI); +} +inline Contour custom::getEyeEllipse(const cv::Point &ptLeft, const cv::Point &ptRight) +{ + Contour cntEyeBottom; + const cv::Point ptEyeCenter((ptRight + ptLeft) / 2); + const int angle = getLineInclinationAngleDegrees(ptLeft, ptRight); + const int axisX = toIntRounded(cv::norm(ptRight - ptLeft) / 2.0); + // According to research, in average a Y axis of an eye is approximately + // 1/3 of an X one. + const int axisY = axisX / 3; + // We need the lower part of an ellipse: + static constexpr int kAngEyeStart = 0; + static constexpr int kAngEyeEnd = 180; + cv::ellipse2Poly(ptEyeCenter, cv::Size(axisX, axisY), angle, kAngEyeStart, kAngEyeEnd, config::kAngDelta, + cntEyeBottom); + return cntEyeBottom; +} +``` +Briefly, this function restores the bottom side of an eye by a half-ellipse based on two points in left and right eye corners. In fact, `cv::ellipse2Poly()` is used to approximate the eye region, and the function only defines ellipse parameters based on just two points: +- The ellipse center and the \f$X\f$ half-axis calculated by two eye Points. +- The \f$Y\f$ half-axis calculated according to the assumption that an average eye width is \f$1/3\f$ of its length. +- The start and the end angles which are 0 and 180 (refer to `cv::ellipse()` documentation). +- The angle delta: how much points to produce in the contour. +- The inclination angle of the axes. + +The use of the `atan2()` instead of just `atan()` in function `custom::getLineInclinationAngleDegrees()` is essential as it allows to return a negative value depending on the `x` and the `y` signs so we can get the right angle even in case of upside-down face arrangement (if we put the points in the right order, of course). + +#### Get a Forehead Contour +The function approximates the forehead contour: +```cpp +inline Contour custom::getForeheadEllipse(const cv::Point &ptJawLeft, + const cv::Point &ptJawRight, + const cv::Point &ptJawLower) +{ + Contour cntForehead; + // The point amid the top two points of a jaw: + const cv::Point ptFaceCenter((ptJawLeft + ptJawRight) / 2); + // This will be the center of the ellipse. + // The angle between the jaw and the vertical: + const int angFace = getLineInclinationAngleDegrees(ptJawLeft, ptJawRight); + // This will be the inclination of the ellipse + // Counting the half-axis of the ellipse: + const double jawWidth = cv::norm(ptJawLeft - ptJawRight); + // A forehead width equals the jaw width, and we need a half-axis: + const int axisX = toIntRounded(jawWidth / 2.0); + const double jawHeight = cv::norm(ptFaceCenter - ptJawLower); + // According to research, in average a forehead is approximately 2/3 of + // a jaw: + const int axisY = toIntRounded(jawHeight * 2 / 3.0); + // We need the upper part of an ellipse: + static constexpr int kAngForeheadStart = 180; + static constexpr int kAngForeheadEnd = 360; + cv::ellipse2Poly(ptFaceCenter, cv::Size(axisX, axisY), angFace, kAngForeheadStart, kAngForeheadEnd, + config::kAngDelta, cntForehead); + return cntForehead; +} +``` +As we have only jaw points in our detected landmarks, we have to get a half-ellipse based on three points of a jaw: the leftmost, the rightmost and the lowest one. The jaw width is assumed to be equal to the forehead width and the latter is calculated using the left and the right points. Speaking of the \f$Y\f$ axis, we have no points to get it directly, and instead assume that the forehead height is about \f$2/3\f$ of the jaw height, which can be figured out from the face center (the middle between the left and right points) and the lowest jaw point. + +### Draw Masks +When we have all the contours needed, you are able to draw masks: + +```cpp +cv::GMat mskSharp = custom::GFillPolyGContours::on(gimgIn, garElsConts); // | +cv::GMat mskSharpG = cv::gapi::gaussianBlur(mskSharp, config::kGKernelSize, // | + config::kGSigma); // | +cv::GMat mskBlur = custom::GFillPolyGContours::on(gimgIn, garFaceConts); // | +cv::GMat mskBlurG = cv::gapi::gaussianBlur(mskBlur, config::kGKernelSize, // | + config::kGSigma); // |draw masks +// The first argument in mask() is Blur as we want to subtract from // | +// BlurG the next step: // | +cv::GMat mskBlurFinal = mskBlurG - cv::gapi::mask(mskBlurG, mskSharpG); // | +cv::GMat mskFacesGaussed = mskBlurFinal + mskSharpG; // | +cv::GMat mskFacesWhite = cv::gapi::threshold(mskFacesGaussed, 0, 255, cv::THRESH_BINARY); // | +cv::GMat mskNoFaces = cv::gapi::bitwise_not(mskFacesWhite); // | +``` + +The steps to get the masks are: +* the "sharp" mask calculation: + * fill the contours that should be sharpened; + * blur that to get the "sharp" mask (`mskSharpG`); +* the "bilateral" mask calculation: + * fill all the face contours fully; + * blur that; + * subtract areas which intersect with the "sharp" mask --- and get the "bilateral" mask (`mskBlurFinal`); +* the background mask calculation: + * add two previous masks + * set all non-zero pixels of the result as 255 (by `cv::gapi::threshold()`) + * revert the output (by `cv::gapi::bitwise_not`) to get the background mask (`mskNoFaces`). + +## Configuring and Running the Pipeline +Once the graph is fully expressed, we can finally compile it and run on real data. G-API graph compilation is the stage where the G-API framework actually understands which kernels and networks to use. This configuration happens via G-API compilation arguments. + +### DNN Parameters +This sample is using OpenVINO™ Toolkit Inference Engine backend for DL inference, which is configured the following way: +```cpp +auto faceParams = cv::gapi::ie::Params +{ + /*std::string*/ faceXmlPath, + /*std::string*/ faceBinPath, + /*std::string*/ faceDevice +}; +auto landmParams = cv::gapi::ie::Params +{ + /*std::string*/ landmXmlPath, + /*std::string*/ landmBinPath, + /*std::string*/ landmDevice +}; +``` +Every `cv::gapi::ie::Params<>` object is related to the network specified in its template argument. We should pass there the network type we have defined in `G_API_NET()` in the early beginning of the tutorial. + +Network parameters are then wrapped in `cv::gapi::NetworkPackage`: +```cpp +auto networks = cv::gapi::networks(faceParams, landmParams); +``` + +More details in "Face Analytics Pipeline" ([Configuring the Pipeline](@ref gapi_ifd_configuration) section). + +### Kernel Packages +In this example we use a lot of custom kernels, in addition to that we use Fluid backend to optimize out memory for G-API's standard kernels where applicable. The resulting kernel package is formed like this: +```cpp +auto customKernels = cv::gapi::kernels(); +auto kernels = cv::gapi::combine(cv::gapi::core::fluid::kernels(), + customKernels); +``` + +### Compiling the Streaming Pipeline +G-API optimizes execution for video streams when compiled in the "Streaming" mode. + +```cpp +cv::GStreamingCompiled stream = pipeline.compileStreaming(cv::compile_args(kernels, networks)); +``` +More on this in "Face Analytics Pipeline" ([Configuring the pipeline](@ref gapi_ifd_configuration) section). + +### Running the streaming pipeline +In order to run the G-API streaming pipeline, all we need is to specify the input video source, call `cv::GStreamingCompiled::start()`, and then fetch the pipeline processing results: +```cpp +if (parser.has("input")) +{ + stream.setSource(cv::gapi::wip::make_src(parser.get("input"))); +} + auto out_vector = cv::gout(imgBeautif, imgShow, vctFaceConts, + vctElsConts, vctRects); + stream.start(); + avg.start(); + while (stream.running()) + { + if (!stream.try_pull(std::move(out_vector))) + { + // Use a try_pull() to obtain data. + // If there's no data, let UI refresh (and handle keypress) + if (cv::waitKey(1) >= 0) break; + else continue; + } + frames++; + // Drawing face boxes and landmarks if necessary: + if (flgLandmarks == true) + { + cv::polylines(imgShow, vctFaceConts, config::kClosedLine, + config::kClrYellow); + cv::polylines(imgShow, vctElsConts, config::kClosedLine, + config::kClrYellow); + } + if (flgBoxes == true) + for (auto rect : vctRects) + cv::rectangle(imgShow, rect, config::kClrGreen); + cv::imshow(config::kWinInput, imgShow); + cv::imshow(config::kWinFaceBeautification, imgBeautif); + } +``` +Once results are ready and can be pulled from the pipeline we display it on the screen and handle GUI events. + +See [Running the pipeline](@ref gapi_ifd_running) section in the "Face Analytics Pipeline" tutorial for more details. + +## Conclusion +The tutorial has two goals: to show the use of brand new features of G-API introduced in OpenCV 4.2, and give a basic understanding on a sample face beautification algorithm. + +The result of the algorithm application: + +![Face Beautification example](../img/gapi_face_beautification_example.jpg) + +On the test machine (Intel® Core™ i7-8700) the G-API-optimized video pipeline outperforms its serial (non-pipelined) version by a factor of 2.7 – meaning that for such a non-trivial graph, the proper pipelining can bring almost 3x increase in performance. \ No newline at end of file diff --git a/docs/gapi/gapi_face_analytics_pipeline.md b/docs/gapi/gapi_face_analytics_pipeline.md new file mode 100644 index 00000000000000..83dcf4594ca953 --- /dev/null +++ b/docs/gapi/gapi_face_analytics_pipeline.md @@ -0,0 +1,325 @@ +# Building a Face Analytics Pipeline {#openvino_docs_gapi_gapi_face_analytics_pipeline} + +## Overview +In this tutorial you will learn: + +* How to integrate Deep Learning inference in a G-API graph. +* How to run a G-API graph on a video stream and obtain data from it. + +## Prerequisites +This sample requires: + +* PC with GNU/Linux* or Microsoft Windows* (Apple macOS* is supported but was not tested) +* OpenCV 4.2 or higher built with [Intel® Distribution of OpenVINO™ Toolkit](https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html) (building with [Intel® TBB](https://www.threadingbuildingblocks.org/intel-tbb-tutorial) +* The following pre-trained models from the [Open Model Zoo](@ref omz_models_intel_index): + * [face-detection-adas-0001](@ref omz_models_intel_face_detection_adas_0001_description_face_detection_adas_0001) + * [age-gender-recognition-retail-0013](@ref omz_models_intel_age_gender_recognition_retail_0013_description_age_gender_recognition_retail_0013) + * [emotions-recognition-retail-0003](@ref omz_models_intel_emotions_recognition_retail_0003_description_emotions_recognition_retail_0003) + +To download the models from the Open Model Zoo, use the [Model Downloader](@ref omz_tools_downloader_README) tool. + +## Introduction: Why G-API +Many computer vision algorithms run on a video stream rather than on individual images. Stream processing usually consists of multiple steps – like decode, preprocessing, detection, tracking, classification (on detected objects), and visualization – forming a *video processing pipeline*. Moreover, many these steps of such pipeline can run in parallel – modern platforms have different hardware blocks on the same chip like decoders and GPUs, and extra accelerators can be plugged in as extensions, like Intel® Movidius™ Neural Compute Stick for deep learning offload. + +Given all this manifold of options and a variety in video analytics algorithms, managing such pipelines effectively quickly becomes a problem. For sure it can be done manually, but this approach doesn't scale: if a change is required in the algorithm (e.g. a new pipeline step is added), or if it is ported on a new platform with different capabilities, the whole pipeline needs to be re-optimized. + +Starting with version 4.2, OpenCV offers a solution to this problem. OpenCV G-API now can manage Deep Learning inference (a cornerstone of any modern analytics pipeline) with a traditional Computer Vision as well as video capturing/decoding, all in a single pipeline. G-API takes care of pipelining itself – so if the algorithm or platform changes, the execution model adapts to it automatically. + +## Pipeline Overview +Our sample application is based on [Interactive Face Detection](omz_demos_interactive_face_detection_demo_README) demo from Open Model Zoo. A simplified pipeline consists of the following steps: + +1. Image acquisition and decode +2. Detection with preprocessing +3. Classification with preprocessing for every detected object with two networks +4. Visualization + +![Face Analytics Pipeline Overview](../img/gapi_face_analytics_pipeline.png) + +## Construct a pipeline {#gapi_ifd_constructing} + +Constructing a G-API graph for a video streaming case does not differ much from a [regular usage](https://docs.opencv.org/4.5.0/d0/d1e/gapi.html#gapi_example) of G-API -- it is still about defining graph *data* (with cv::GMat, `cv::GScalar`, and `cv::GArray`) and *operations* over it. Inference also becomes an operation in the graph, but is defined in a little bit different way. + +### Declare Deep Learning topologies {#gapi_ifd_declaring_nets} + +In contrast with traditional CV functions (see [core](https://docs.opencv.org/4.5.0/df/d1f/group__gapi__core.html) and [imgproc](https://docs.opencv.org/4.5.0/d2/d00/group__gapi__imgproc.html)) where G-API declares distinct operations for every function, inference in G-API is a single generic operation `cv::gapi::infer<>`. As usual, it is just an interface and it can be implemented in a number of ways under the hood. In OpenCV 4.2, only OpenVINO™ Inference Engine-based backend is available, and OpenCV's own DNN module-based backend is to come. + +`cv::gapi::infer<>` is _parametrized_ by the details of a topology we are going to execute. Like operations, topologies in G-API are strongly typed and are defined with a special macro `G_API_NET()`: + +```cpp +// Face detector: takes one Mat, returns another Mat +G_API_NET(Faces, , "face-detector"); +// Age/Gender recognition - takes one Mat, returns two: +// one for Age and one for Gender. In G-API, multiple-return-value operations +// are defined using std::tuple<>. +using AGInfo = std::tuple; +G_API_NET(AgeGender, , "age-gender-recoginition"); +// Emotion recognition - takes one Mat, returns another. +G_API_NET(Emotions, , "emotions-recognition"); +``` + +Similar to how operations are defined with `G_API_OP()`, network description requires three parameters: +1. A type name. Every defined topology is declared as a distinct C++ type which is used further in the program -- see below. +2. A `std::function<>`-like API signature. G-API traits networks as regular "functions" which take and return data. Here network `Faces` (a detector) takes a `cv::GMat` and returns a `cv::GMat`, while network `AgeGender` is known to provide two outputs (age and gender blobs, respectively) -- so its has a `std::tuple<>` as a return type. +3. A topology name -- can be any non-empty string, G-API is using these names to distinguish networks inside. Names should be unique in the scope of a single graph. + +## Building a GComputation {#gapi_ifd_gcomputation} + +Now the above pipeline is expressed in G-API like this: + +```cpp +cv::GComputation pp([]() { + // Declare an empty GMat - the beginning of the pipeline. + cv::GMat in; + // Run face detection on the input frame. Result is a single GMat, + // internally representing an 1x1x200x7 SSD output. + // This is a single-patch version of infer: + // - Inference is running on the whole input image; + // - Image is converted and resized to the network's expected format + // automatically. + cv::GMat detections = cv::gapi::infer(in); + // Parse SSD output to a list of ROI (rectangles) using + // a custom kernel. Note: parsing SSD may become a "standard" kernel. + cv::GArray faces = custom::PostProc::on(detections, in); + // Now run Age/Gender model on every detected face. This model has two + // outputs (for age and gender respectively). + // A special ROI-list-oriented form of infer<>() is used here: + // - First input argument is the list of rectangles to process, + // - Second one is the image where to take ROI from; + // - Crop/Resize/Layout conversion happens automatically for every image patch + // from the list + // - Inference results are also returned in form of list (GArray<>) + // - Since there're two outputs, infer<> return two arrays (via std::tuple). + cv::GArray ages; + cv::GArray genders; + std::tie(ages, genders) = cv::gapi::infer(faces, in); + // Recognize emotions on every face. + // ROI-list-oriented infer<>() is used here as well. + // Since custom::Emotions network produce a single output, only one + // GArray<> is returned here. + cv::GArray emotions = cv::gapi::infer(faces, in); + // Return the decoded frame as a result as well. + // Input matrix can't be specified as output one, so use copy() here + // (this copy will be optimized out in the future). + cv::GMat frame = cv::gapi::copy(in); + // Now specify the computation's boundaries - our pipeline consumes + // one images and produces five outputs. + return cv::GComputation(cv::GIn(in), + cv::GOut(frame, faces, ages, genders, emotions)); +}); +``` + +Every pipeline starts with declaring empty data objects – which act as inputs to the pipeline. Then we call a generic `cv::gapi::infer<>` specialized to Faces detection network. `cv::gapi::infer<>` inherits its signature from its template parameter – and in this case it expects one input cv::GMat and produces one output cv::GMat. + +In this sample we use a pre-trained SSD-based network and its output needs to be parsed to an array of detections (object regions of interest, ROIs). It is done by a custom operation custom::PostProc, which returns an array of rectangles (of type `cv::GArray`) back to the pipeline. This operation also filters out results by a confidence threshold – and these details are hidden in the kernel itself. Still, at the moment of graph construction we operate with interfaces only and don't need actual kernels to express the pipeline – so the implementation of this post-processing will be listed later. + +After detection result output is parsed to an array of objects, we can run classification on any of those. G-API doesn't support syntax for in-graph loops like `for_each()` yet, but instead `cv::gapi::infer<>` comes with a special list-oriented overload. + +User can call `cv::gapi::infer<>` with a `cv::GArray` as the first argument, so then G-API assumes it needs to run the associated network on every rectangle from the given list of the given frame (second argument). Result of such operation is also a list – a cv::GArray of `cv::GMat`. + +Since AgeGender network itself produces two outputs, it's output type for a list-based version of `cv::gapi::infer` is a tuple of arrays. We use `std::tie()` to decompose this input into two distinct objects. + +Emotions network produces a single output so its list-based inference's return type is `cv::GArray`. + +## Configure the Pipeline {#gapi_ifd_configuration} + +G-API strictly separates construction from configuration -- with the idea to keep algorithm code itself platform-neutral. In the above listings we only declared our operations and expressed the overall data flow, but didn't even mention that we use OpenVINO™. We only described *what* we do, but not *how* we do it. Keeping these two aspects clearly separated is the design goal for G-API. + +Platform-specific details arise when the pipeline is *compiled* -- i.e. is turned from a declarative to an executable form. The way *how* to run stuff is specified via compilation arguments, and new inference/streaming features are no exception from this rule. + +G-API is built on backends which implement interfaces (see [Architecture](https://docs.opencv.org/4.5.0/de/d4d/gapi_hld.html) and [Kernels](kernel_api.md) for details) -- thus `cv::gapi::infer<>` is a function which can be implemented by different backends. In OpenCV 4.2, only OpenVINO™ Inference Engine backend for inference is available. Every inference backend in G-API has to provide a special parameterizable structure to express *backend-specific* neural network parameters -- and in this case, it is `cv::gapi::ie::Params`: + +```cpp +auto det_net = cv::gapi::ie::Params { + cmd.get("fdm"), // read cmd args: path to topology IR + cmd.get("fdw"), // read cmd args: path to weights + cmd.get("fdd"), // read cmd args: device specifier +}; +auto age_net = cv::gapi::ie::Params { + cmd.get("agem"), // read cmd args: path to topology IR + cmd.get("agew"), // read cmd args: path to weights + cmd.get("aged"), // read cmd args: device specifier +}.cfgOutputLayers({ "age_conv3", "prob" }); +auto emo_net = cv::gapi::ie::Params { + cmd.get("emom"), // read cmd args: path to topology IR + cmd.get("emow"), // read cmd args: path to weights + cmd.get("emod"), // read cmd args: device specifier +}; +``` + +Here we define three parameter objects: `det_net`, `age_net`, and `emo_net`. Every object is a `cv::gapi::ie::Params` structure parametrization for each particular network we use. On a compilation stage, G-API automatically matches network parameters with their `cv::gapi::infer<>` calls in graph using this information. + +Regardless of the topology, every parameter structure is constructed with three string arguments – specific to the OpenVINO™ Inference Engine: + +* Path to the topology's intermediate representation (.xml file); +* Path to the topology's model weights (.bin file); +* Device where to run – "CPU", "GPU", and others – based on your OpenVINO™ Toolkit installation. These arguments are taken from the command-line parser. + +Once networks are defined and custom kernels are implemented, the pipeline is compiled for streaming: +```cpp +// Form a kernel package (with a single OpenCV-based implementation of our +// post-processing) and a network package (holding our three networks). +auto kernels = cv::gapi::kernels(); +auto networks = cv::gapi::networks(det_net, age_net, emo_net); +// Compile our pipeline and pass our kernels & networks as +// parameters. This is the place where G-API learns which +// networks & kernels we're actually operating with (the graph +// description itself known nothing about that). +auto cc = pp.compileStreaming(cv::compile_args(kernels, networks)); +``` + +`cv::GComputation::compileStreaming()` triggers a special video-oriented form of graph compilation where G-API is trying to optimize throughput. Result of this compilation is an object of special type `cv::GStreamingCompiled` – in contrast to a traditional callable `cv::GCompiled`, these objects are closer to media players in their semantics. + +> **NOTE**: There is no need to pass metadata arguments describing the format of the input video stream in `cv::GComputation::compileStreaming()` – G-API figures automatically what are the formats of the input vector and adjusts the pipeline to these formats on-the-fly. User still can pass metadata there as with regular `cv::GComputation::compile()` in order to fix the pipeline to the specific input format. + +## Running the Pipeline {#gapi_ifd_running} + +Pipelining optimization is based on processing multiple input video frames simultaneously, running different steps of the pipeline in parallel. This is why it works best when the framework takes full control over the video stream. + +The idea behind streaming API is that user specifies an *input source* to the pipeline and then G-API manages its execution automatically until the source ends or user interrupts the execution. G-API pulls new image data from the source and passes it to the pipeline for processing. + +Streaming sources are represented by the interface `cv::gapi::wip::IStreamSource`. Objects implementing this interface may be passed to `GStreamingCompiled` as regular inputs via `cv::gin()` helper function. In OpenCV 4.2, only one streaming source is allowed per pipeline -- this requirement will be relaxed in the future. + +OpenCV comes with a great class cv::VideoCapture and by default G-API ships with a stream source class based on it -- `cv::gapi::wip::GCaptureSource`. Users can implement their own +streaming sources e.g. using [VAAPI](https://01.org/vaapi) or other Media or Networking APIs. + +Sample application specifies the input source as follows: +```cpp +auto in_src = cv::gapi::wip::make_src(input); +cc.setSource(cv::gin(in_src)); +``` + +Please note that a GComputation may still have multiple inputs like `cv::GMat`, `cv::GScalar`, or `cv::GArray` objects. User can pass their respective host-side types (`cv::Mat`, `cv::Scalar`, `std::vector<>`) in the input vector as well, but in Streaming mode these objects will create "endless" constant streams. Mixing a real video source stream and a const data stream is allowed. + +Running a pipeline is easy – just call `cv::GStreamingCompiled::start()` and fetch your data with blocking `cv::GStreamingCompiled::pull()` or non-blocking `cv::GStreamingCompiled::try_pull()`; repeat until the stream ends: + +```cpp +// After data source is specified, start the execution +cc.start(); +// Declare data objects we will be receiving from the pipeline. +cv::Mat frame; // The captured frame itself +std::vector faces; // Array of detected faces +std::vector out_ages; // Array of inferred ages (one blob per face) +std::vector out_genders; // Array of inferred genders (one blob per face) +std::vector out_emotions; // Array of classified emotions (one blob per face) +// Implement different execution policies depending on the display option +// for the best performance. +while (cc.running()) { + auto out_vector = cv::gout(frame, faces, out_ages, out_genders, out_emotions); + if (no_show) { + // This is purely a video processing. No need to balance + // with UI rendering. Use a blocking pull() to obtain + // data. Break the loop if the stream is over. + if (!cc.pull(std::move(out_vector))) + break; + } else if (!cc.try_pull(std::move(out_vector))) { + // Use a non-blocking try_pull() to obtain data. + // If there's no data, let UI refresh (and handle keypress) + if (cv::waitKey(1) >= 0) break; + else continue; + } + // At this point we have data for sure (obtained in either + // blocking or non-blocking way). + frames++; + labels::DrawResults(frame, faces, out_ages, out_genders, out_emotions); + labels::DrawFPS(frame, frames, avg.fps(frames)); + if (!no_show) cv::imshow("Out", frame); +} +``` + +The above code may look complex but in fact it handles two modes – with and without graphical user interface (GUI): + +* When a sample is running in a "headless" mode (`--pure` option is set), this code simply pulls data from the pipeline with the blocking `pull()` until it ends. This is the most performant mode of execution. +* When results are also displayed on the screen, the Window System needs to take some time to refresh the window contents and handle GUI events. In this case, the demo pulls data with a non-blocking `try_pull()` until there is no more data available (but it does not mark end of the stream – just means new data is not ready yet), and only then displays the latest obtained result and refreshes the screen. Reducing the time spent in GUI with this trick increases the overall performance a little bit. + +## Comparison with Serial Mode +The sample can also run in a serial mode for a reference and benchmarking purposes. In this case, a regular `cv::GComputation::compile()` is used and a regular single-frame `cv::GCompiled` object is produced; the pipelining optimization is not applied within G-API; it is the user responsibility to acquire image frames from `cv::VideoCapture` object and pass those to G-API. + +```cpp +cv::VideoCapture cap(input); +cv::Mat in_frame, frame; // The captured frame itself +std::vector faces; // Array of detected faces +std::vector out_ages; // Array of inferred ages (one blob per face) +std::vector out_genders; // Array of inferred genders (one blob per face) +std::vector out_emotions; // Array of classified emotions (one blob per face) +while (cap.read(in_frame)) { + pp.apply(cv::gin(in_frame), + cv::gout(frame, faces, out_ages, out_genders, out_emotions), + cv::compile_args(kernels, networks)); + labels::DrawResults(frame, faces, out_ages, out_genders, out_emotions); + frames++; + if (frames == 1u) { + // Start timer only after 1st frame processed -- compilation + // happens on-the-fly here + avg.start(); + } else { + // Measurfe & draw FPS for all other frames + labels::DrawFPS(frame, frames, avg.fps(frames-1)); + } + if (!no_show) { + cv::imshow("Out", frame); + if (cv::waitKey(1) >= 0) break; + } +} +``` + +On a test machine (Intel® Core™ i5-6600), with OpenCV built with [Intel® TBB](https://www.threadingbuildingblocks.org/intel-tbb-tutorial) support, detector network assigned to CPU, and classifiers to iGPU, the pipelined sample outperformes the serial one by the factor of 1.36x (thus adding +36% in overall throughput). + +## Conclusion +G-API introduces a technological way to build and optimize hybrid pipelines. Switching to a new execution model does not require changes in the algorithm code expressed with G-API – only the way how graph is triggered differs. + +## Listing: Post-Processing Kernel +G-API gives an easy way to plug custom code into the pipeline even if it is running in a streaming mode and processing tensor data. Inference results are represented by multi-dimensional `cv::Mat` objects so accessing those is as easy as with a regular DNN module. + +The OpenCV-based SSD post-processing kernel is defined and implemented in this sample as follows: +```cpp +// SSD Post-processing function - this is not a network but a kernel. +// The kernel body is declared separately, this is just an interface. +// This operation takes two Mats (detections and the source image), +// and returns a vector of ROI (filtered by a default threshold). +// Threshold (or a class to select) may become a parameter, but since +// this kernel is custom, it doesn't make a lot of sense. +G_API_OP(PostProc, (cv::GMat, cv::GMat)>, "custom.fd_postproc") { + static cv::GArrayDesc outMeta(const cv::GMatDesc &, const cv::GMatDesc &) { + // This function is required for G-API engine to figure out + // what the output format is, given the input parameters. + // Since the output is an array (with a specific type), + // there's nothing to describe. + return cv::empty_array_desc(); + } +}; +// OpenCV-based implementation of the above kernel. +GAPI_OCV_KERNEL(OCVPostProc, PostProc) { + static void run(const cv::Mat &in_ssd_result, + const cv::Mat &in_frame, + std::vector &out_faces) { + const int MAX_PROPOSALS = 200; + const int OBJECT_SIZE = 7; + const cv::Size upscale = in_frame.size(); + const cv::Rect surface({0,0}, upscale); + out_faces.clear(); + const float *data = in_ssd_result.ptr(); + for (int i = 0; i < MAX_PROPOSALS; i++) { + const float image_id = data[i * OBJECT_SIZE + 0]; // batch id + const float confidence = data[i * OBJECT_SIZE + 2]; + const float rc_left = data[i * OBJECT_SIZE + 3]; + const float rc_top = data[i * OBJECT_SIZE + 4]; + const float rc_right = data[i * OBJECT_SIZE + 5]; + const float rc_bottom = data[i * OBJECT_SIZE + 6]; + if (image_id < 0.f) { // indicates end of detections + break; + } + if (confidence < 0.5f) { // a hard-coded snapshot + continue; + } + // Convert floating-point coordinates to the absolute image + // frame coordinates; clip by the source image boundaries. + cv::Rect rc; + rc.x = static_cast(rc_left * upscale.width); + rc.y = static_cast(rc_top * upscale.height); + rc.width = static_cast(rc_right * upscale.width) - rc.x; + rc.height = static_cast(rc_bottom * upscale.height) - rc.y; + out_faces.push_back(rc & surface); + } + } +}; +``` \ No newline at end of file diff --git a/docs/gapi/gapi_intro.md b/docs/gapi/gapi_intro.md new file mode 100644 index 00000000000000..aedbcd68a7bb47 --- /dev/null +++ b/docs/gapi/gapi_intro.md @@ -0,0 +1,52 @@ +# Introduction to OpenCV Graph API (G-API) {#openvino_docs_gapi_gapi_intro} +OpenCV Graph API (G-API) is an OpenCV module targeted to make regular image and video processing fast and portable. G-API is a special module in OpenCV – in contrast with the majority of other main modules, this one acts as a framework rather than some specific CV algorithm. + +G-API is positioned as a next level optimization enabler for computer vision, focusing not on particular CV functions but on the whole algorithm optimization. + +G-API provides means to define CV operations, construct graphs (in form of expressions) using it, and finally implement and run the operations for a particular backend. + +The idea behind G-API is that if an algorithm can be expressed in a special embedded language (currently in C++), the framework can catch its sense and apply a number of optimizations to the whole thing automatically. Particular optimizations are selected based on which [kernels](kernel_api.md) and [backends](https://docs.opencv.org/4.5.0/dc/d1c/group__gapi__std__backends.html) are involved in the graph compilation process, for example, the graph can be offloaded to GPU via the OpenCL backend, or optimized for memory consumption with the Fluid backend. Kernels, backends, and their settings are parameters to the graph compilation, so the graph itself does not depend on any platform-specific details and can be ported easily. + +> **NOTE**: Graph API (G-API) was introduced in the most recent major OpenCV 4.0 release and now is being actively developed. The API is volatile at the moment and there may be minor but compatibility-breaking changes in the future. + +## G-API Concepts + +* *Graphs* are built by applying operations to data objects. + * API itself has no "graphs", it is expression-based instead. +* *Data objects* do not hold actual data, only capture dependencies. +* *Operations* consume and produce data objects. +* A graph is defined by specifying its boundaries with data objects: + * What data objects are inputs to the graph? + * What are its outputs? + +The paragraphs below explain the G-API programming model and development workflow. + +## Programming Model +Building graphs is easy with G-API. In fact, there is no notion of graphs exposed in the API, so the user doesn’t need to operate in terms of “nodes” and “edges” — instead, graphs are constructed implicitly via expressions in a "functional" way. Expression-based graphs are built using two major concepts: *[operations](kernel_api.md)* and *[data objects](https://docs.opencv.org/4.2.0/db/df1/group__gapi__data__objects.html)*. + +In G-API, every graph begins and ends with data objects; data objects are passed to operations which produce (“return”) their results — new data objects, which are then passed to other operations, and so on. You can declare their own operations, G-API does not distinguish user-defined operations from its own predefined ones in any way. + +After the graph is defined, it needs to be compiled for execution. During the compilation, G-API figures out what the graph looks like, which kernels are available to run the operations in the graph, how to manage heterogeneity and to optimize the execution path. The result of graph compilation is a so-called “compiled” object. This object encapsulates the execution sequence for the graph inside and operates on real image data. You can set up the compilation process using various [compilation arguments](https://docs.opencv.org/4.5.0/dc/d1c/group__gapi__std__backends.html). Backends expose some of their options as these arguments; also, actual kernels and DL network settings are passed into the framework this way. + +G-API supports graph compilation for two execution modes, *regular* and *streaming*, producing different types of compiled objects as the result. +* Regular compiled objects are represented with class GCompiled, which follows functor-like semantics and has an overloaded operator(). When called for execution on the given input data, the GCompiled functor blocks the current thread and processes the data immediately — like a regular C++ function. By default, G-API tries to optimize the execution time for latency in this compilation mode. +* Starting with OpenCV 4.2, G-API can also produce GStreamingCompiled objects that better fit the asynchronous pipelined execution model. This compilation mode is called **streaming mode**, and G-API tries to optimize the overall throughput by implementing the pipelining technique as described above. We will use both in our example. + +The overall process for the regular case is summarized in the diagram below: + +![G-API Programming Model](../img/gapi_programming_model.png) + +The graph is built with operations so having operations defined (**0**) is a basic prerequisite; a constructed expression graph (**1**) forms a `cv::GComputation` object; kernels (**2**) which implement operations are the basic requirement to the graph compilation (**3**); the actual execution (**4**) is handled by a `cv::GCompiled` object with takes input and produces output data. + +## Development Workflow +One of the ways to organize a G-API development workflow is presented in the diagram below: + +![G-API development workflow](../img/gapi_development_workflow.png) + +Basically, it is a derivative from the programming model illustrated in the previous chapter. You start with an algorithm or a data flow in mind (**0**), mapping it to a graph model (**1**), then identifying what operations you need (**2**) to construct this graph. These operations may already exist in G-API or be missing, in the latter case we implement the missing ones as kernels (**3**). Then decide which execution model fits our case better, pass kernels and DL networks as arguments to the compilation process (**4**), and finally switch to the execution (**5**). The process is iterative, so if you want to change anything based on the execution results, get back to steps (**0**) or (**1**) (a dashed line). + + + + + + diff --git a/docs/gapi/kernel_api.md b/docs/gapi/kernel_api.md new file mode 100644 index 00000000000000..f004ea9e89d922 --- /dev/null +++ b/docs/gapi/kernel_api.md @@ -0,0 +1,188 @@ +# Graph API Kernel API {#openvino_docs_gapi_kernel_api} + +The core idea behind Graph API (G-API) is portability – a pipeline built with G-API must be portable (or at least able to be portable). It means that either it works out-of-the box when compiled for new platform, or G-API provides necessary tools to make it running there, with little-to-no changes in the algorithm itself. + +This idea can be achieved by separating kernel interface from its implementation. Once a pipeline is built using kernel interfaces, it becomes implementation-neutral – the implementation details (i.e. which kernels to use) are passed on a separate stage (graph compilation). + +Kernel-implementation hierarchy may look like: +![Kernel API/implementation hierarchy example](../img/gapi_kernel_implementation_hierarchy.png) + +A pipeline itself then can be expressed only in terms of `A`, `B`, and so on, and choosing which implementation to use in execution becomes an external parameter. + +## Define a Kernel +G-API provides a macro to define a new kernel interface `G_TYPED_KERNEL()`: + +```cpp +#include +G_TYPED_KERNEL(GFilter2D, + , + "org.opencv.imgproc.filters.filter2D") +{ + static cv::GMatDesc // outMeta's return value type + outMeta(cv::GMatDesc in , // descriptor of input GMat + int ddepth , // depth parameter + cv::Mat /* coeffs */, // (unused) + cv::Point /* anchor */, // (unused) + double /* scale */, // (unused) + int /* border */, // (unused) + cv::Scalar /* bvalue */ ) // (unused) + { + return in.withDepth(ddepth); + } +}; +``` + +This macro is a shortcut to a new type definition. It takes three arguments to register a new type, and requires type body to be present (see below). The macro arguments are: + +* Kernel interface name -- Also serves as a name of new type defined with this macro; +* Kernel signature -- An `std::function<>`-like signature which defines API of the kernel; +* Kernel's unique name -- Used to identify kernel when its type information is stripped within the system. +* Kernel declaration may be seen as function declaration -- In both cases a new entity must be used then according to the way it was defined. + +Kernel signature defines kernel's usage syntax -- which parameters it takes during graph construction. Implementations can also use this signature to derive it into backend-specific callback signatures (see next chapter). + +Kernel may accept values of any type, and G-API dynamic types are handled in a special way. All other types are opaque to G-API and passed to kernel in `outMeta()` or in execution callbacks as-is. + +Kernel's return value can only be of G-API dynamic type – `cv::GMat`, `cv::GScalar`, or `cv::GArray`. If an operation has more than one output, it should be wrapped into an `std::tuple<>` (which can contain only mentioned G-API types). Arbitrary-output-number operations are not supported. + +Once a kernel is defined, it can be used in pipelines with special, G-API-supplied method `on()`. This method has the same signature as defined in kernel, so the following code is a perfectly legal construction: + +```cpp +cv::GMat in; +cv::GMat out = GFilter2D::on(/* GMat */ in, + /* int */ -1, + /* Mat */ conv_kernel_mat, + /* Point */ cv::Point(-1,-1), + /* double */ 0., + /* int */ cv::BORDER_DEFAULT, + /* Scalar */ cv::Scalar(0)); +``` +This example has some verbosity, though, so usually a kernel declaration comes with a C++ function wrapper ("factory method") which enables optional parameters, more compact syntax, Doxygen comments, etc.: + +```cpp +cv::GMat filter2D(cv::GMat in, + int ddepth, + cv::Mat k, + cv::Point anchor = cv::Point(-1,-1), + double scale = 0., + int border = cv::BORDER_DEFAULT, + cv::Scalar bval = cv::Scalar(0)) +{ + return GFilter2D::on(in, ddepth, k, anchor, scale, border, bval); +} +``` +So now it can be used like: +```cpp +cv::GMat in; +cv::GMat out = filter2D(in, -1, conv_kernel_mat); +``` + +### Extra information +In the current version, kernel declaration body (everything within the curly braces) must contain a static function `outMeta()`. This function establishes a functional dependency between operation's input and output metadata. + +Metadata is an information about data kernel operates on. Since non-G-API types are opaque to G-API, G-API cares only about G* data descriptors (i.e. dimensions and format of `cv::GMat`, etc). + +`outMeta()` is also an example of how kernel's signature can be transformed into a derived callback – note that in this example, outMeta() signature exactly follows the kernel signature (defined within the macro) but is different – where kernel expects `cv::GMat`, `outMeta()` takes and returns `cv::GMatDesc` (a G-API structure metadata for `cv::GMat`). + +The point of `outMeta()` is to propagate metadata information within computation from inputs to outputs and infer metadata of internal (intermediate, temporary) data objects. This information is required for further pipeline optimizations, memory allocation, and other operations done by G-API framework during graph compilation. + +## Implement a Kernel +Once a kernel is declared, its interface can be used to implement versions of this kernel in different backends. This concept is naturally projected from object-oriented programming "Interface/Implementation" idiom: an interface can be implemented multiple times, and different implementations of a kernel should be substitutable with each other without breaking the algorithm (pipeline) logic (Liskov Substitution Principle). + +Every backend defines its own way to implement a kernel interface. This way is regular, though – whatever plugin is, its kernel implementation must be "derived" from a kernel interface type. + +Kernel implementation are then organized into kernel packages. Kernel packages are passed to `cv::GComputation::compile()` as compile arguments, with some hints to G-API on how to select proper kernels (see more on this in "Heterogeneity"[TBD]). + +For example, the aforementioned Filter2D is implemented in "reference" CPU (OpenCV) plugin this way (NOTE – this is a simplified form with improper border handling): + +```cpp +#include // GAPI_OCV_KERNEL() +#include // cv::filter2D() +GAPI_OCV_KERNEL(GCPUFilter2D, GFilter2D) +{ + static void + run(const cv::Mat &in, // in - derived from GMat + const int ddepth, // opaque (passed as-is) + const cv::Mat &k, // opaque (passed as-is) + const cv::Point &anchor, // opaque (passed as-is) + const double delta, // opaque (passed as-is) + const int border, // opaque (passed as-is) + const cv::Scalar &, // opaque (passed as-is) + cv::Mat &out) // out - derived from GMat (retval) + { + cv::filter2D(in, out, ddepth, k, anchor, delta, border); + } +}; +``` +Note how CPU (OpenCV) plugin has transformed the original kernel signature: + +* Input `cv::GMat` has been substituted with `cv::Mat`, holding actual input data for the underlying OpenCV function call; +* Output `cv::GMat `has been transformed into extra output parameter, thus `GCPUFilter2D::run()` takes one argument more than the original kernel signature. + +The basic intuition for kernel developer here is not to care where that cv::Mat objects come from instead of the original `cv::GMat` – and just follow the signature conventions defined by the plugin. G-API will call this method during execution and supply all the necessary information (and forward the original opaque data as-is). + +## Compound Kernels +Sometimes kernel is a single thing only on API level. It is convenient for users, but on a particular implementation side it would be better to have multiple kernels (a subgraph) doing the thing instead. An example is `goodFeaturesToTrack()` – while in OpenCV backend it may remain a single kernel, with Fluid it becomes compound – Fluid can handle Harris response calculation but can't do sparse non-maxima suppression and point extraction to an STL vector: + +A compound kernel implementation can be defined using a generic macro `GAPI_COMPOUND_KERNEL()`: + +```cpp +#include // GAPI_COMPOUND_KERNEL() +using PointArray2f = cv::GArray; +G_TYPED_KERNEL(HarrisCorners, + , + "org.opencv.imgproc.harris_corner") +{ + static cv::GArrayDesc outMeta(const cv::GMatDesc &, + int, + double, + double, + int, + double) + { + // No special metadata for arrays in G-API (yet) + return cv::empty_array_desc(); + } +}; +// Define Fluid-backend-local kernels which form GoodFeatures +G_TYPED_KERNEL(HarrisResponse, + , + "org.opencv.fluid.harris_response") +{ + static cv::GMatDesc outMeta(const cv::GMatDesc &in, + double, + int, + double) + { + return in.withType(CV_32F, 1); + } +}; +G_TYPED_KERNEL(ArrayNMS, + , + "org.opencv.cpu.nms_array") +{ + static cv::GArrayDesc outMeta(const cv::GMatDesc &, + int, + double) + { + return cv::empty_array_desc(); + } +}; +GAPI_COMPOUND_KERNEL(GFluidHarrisCorners, HarrisCorners) +{ + static PointArray2f + expand(cv::GMat in, + int maxCorners, + double quality, + double minDist, + int blockSize, + double k) + { + cv::GMat response = HarrisResponse::on(in, quality, blockSize, k); + return ArrayNMS::on(response, maxCorners, minDist); + } +}; +// Then implement HarrisResponse as Fluid kernel and NMSresponse +// as a generic (OpenCV) kernel +``` +It is important to distinguish a compound kernel from G-API high-order function, i.e. a C++ function which looks like a kernel but in fact generates a subgraph. The core difference is that a compound kernel is an *implementation detail* and a kernel implementation may be either compound or not (depending on backend capabilities), while a high-order function is a "macro" in terms of G-API and so cannot act as an interface which then needs to be implemented by a backend. \ No newline at end of file diff --git a/docs/get_started/get_started_dl_workbench.md b/docs/get_started/get_started_dl_workbench.md index 5b24c217a27d06..7c5a3cdb9dbe75 100644 --- a/docs/get_started/get_started_dl_workbench.md +++ b/docs/get_started/get_started_dl_workbench.md @@ -136,5 +136,4 @@ For detailed instructions to create a new project, visit the links below: * [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md) * [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) * [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) -* [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) -* [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic) +* [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) \ No newline at end of file diff --git a/docs/get_started/get_started_linux.md b/docs/get_started/get_started_linux.md index 672e8391ed9571..a01d5a11c674a1 100644 --- a/docs/get_started/get_started_linux.md +++ b/docs/get_started/get_started_linux.md @@ -570,4 +570,3 @@ Use these resources to learn more about the OpenVINO™ toolkit: * [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) * [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) * [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) -* [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic) diff --git a/docs/get_started/get_started_macos.md b/docs/get_started/get_started_macos.md index c36d7b58270b45..14456171d60ad4 100644 --- a/docs/get_started/get_started_macos.md +++ b/docs/get_started/get_started_macos.md @@ -529,4 +529,3 @@ Use these resources to learn more about the OpenVINO™ toolkit: * [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) * [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) * [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) -* [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic) diff --git a/docs/get_started/get_started_raspbian.md b/docs/get_started/get_started_raspbian.md index e238029b70495e..bae7078587f63a 100644 --- a/docs/get_started/get_started_raspbian.md +++ b/docs/get_started/get_started_raspbian.md @@ -106,4 +106,3 @@ Use these resources to learn more about the OpenVINO™ toolkit: * [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) * [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) * [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) -* [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic) diff --git a/docs/get_started/get_started_windows.md b/docs/get_started/get_started_windows.md index f360c80f12b0fd..0255a1bb396597 100644 --- a/docs/get_started/get_started_windows.md +++ b/docs/get_started/get_started_windows.md @@ -537,5 +537,4 @@ Use these resources to learn more about the OpenVINO™ toolkit: * [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md) * [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) * [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) -* [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) -* [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic) \ No newline at end of file +* [Overview of OpenVINO™ Toolkit Pre-Trained Models](https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models) \ No newline at end of file diff --git a/docs/img/gapi_development_workflow.png b/docs/img/gapi_development_workflow.png new file mode 100644 index 00000000000000..658fdafe87a60a --- /dev/null +++ b/docs/img/gapi_development_workflow.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a0a11bedbfe2df3352b064e80498aa39fbc3817eaf99439865a090f34501e44a +size 25936 diff --git a/docs/img/gapi_face_analytics_pipeline.png b/docs/img/gapi_face_analytics_pipeline.png new file mode 100644 index 00000000000000..31f045c5d77ca2 --- /dev/null +++ b/docs/img/gapi_face_analytics_pipeline.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:662a823fbef3be0cca1755de9118e73b4137fe7ec4b7cb6a389e64b9ec5a9c13 +size 13511 diff --git a/docs/img/gapi_face_beautification_algorithm.png b/docs/img/gapi_face_beautification_algorithm.png new file mode 100644 index 00000000000000..7693c3b0fd825e --- /dev/null +++ b/docs/img/gapi_face_beautification_algorithm.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:12fe8e0b841aa6759f3b1975d3a877e65b8d72b752d11ffd212b67d11e62e048 +size 19539 diff --git a/docs/img/gapi_face_beautification_algorithm.svg b/docs/img/gapi_face_beautification_algorithm.svg new file mode 100644 index 00000000000000..38ede92ac0eef5 --- /dev/null +++ b/docs/img/gapi_face_beautification_algorithm.svg @@ -0,0 +1,415 @@ + + + + + + + + + + + + + + + + + + Page-1 + + + + Rounded Rectangle + Landmarks detector + + + + + + + + + + + + + + + + + + + + + + Landmarksdetector + + Rounded Rectangle.4 + Generate BG mask + + + + + + + + + + + + + + + + + + + + + + GenerateBG mask + + Rounded Rectangle.5 + Input + + + + + + + + + + + + + + + + + + + + + + Input + + Rounded Rectangle.7 + Unsharp mask + + + + + + + + + + + + + + + + + + + + + + Unsharpmask + + Rounded Rectangle.8 + Bilateral filter + + + + + + + + + + + + + + + + + + + + + + Bilateralfilter + + Rounded Rectangle.9 + Face detector + + + + + + + + + + + + + + + + + + + + + + Facedetector + + Rounded Rectangle.10 + Generate sharp mask + + + + + + + + + + + + + + + + + + + + + + Generatesharp mask + + Rounded Rectangle.11 + Output + + + + + + + + + + + + + + + + + + + + + + Output + + Rounded Rectangle.13 + Generate blur mask + + + + + + + + + + + + + + + + + + + + + + Generateblur mask + + Circle + + + + + + + Circle.15 + * + + + + * + + Circle.16 + + + + + + + Circle.17 + * + + + + * + + Circle.18 + + + + + + + + + + + + Circle.20 + + + + + + + Circle.21 + * + + + + * + + Dynamic connector + + + + Dynamic connector.27 + + + + Dynamic connector.28 + + + + Dynamic connector.29 + + + + Dynamic connector.30 + + + + Dynamic connector.31 + + + + Dynamic connector.32 + + + + Dynamic connector.33 + + + + Dynamic connector.34 + + + + Dynamic connector.35 + + + + Dynamic connector.36 + + + + Dynamic connector.37 + + + + Dynamic connector.38 + + + + Dynamic connector.39 + + + + Dynamic connector.40 + + + + Rectangle + For each face + + + + + + + For each face + + diff --git a/docs/img/gapi_face_beautification_example.jpg b/docs/img/gapi_face_beautification_example.jpg new file mode 100644 index 00000000000000..eb3df6b58785bf --- /dev/null +++ b/docs/img/gapi_face_beautification_example.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fb32d3db8768ff157daeff999cc7f4361d2bca866ed6dc95b8f78d8cc62ae208 +size 176525 diff --git a/docs/img/gapi_kernel_implementation_hierarchy.png b/docs/img/gapi_kernel_implementation_hierarchy.png new file mode 100644 index 00000000000000..f910caa840d191 --- /dev/null +++ b/docs/img/gapi_kernel_implementation_hierarchy.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f291422f562825d4c5eee718b7c22e472b02a5a0a9c0be01d59b6b7cd8d756b1 +size 14603 diff --git a/docs/img/gapi_programming_model.png b/docs/img/gapi_programming_model.png new file mode 100644 index 00000000000000..2ac10dcc82c13f --- /dev/null +++ b/docs/img/gapi_programming_model.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:925f70ede92d71e16733d78e003f62cd8bfdee0790bddbf2b7ce4fc8ef3f44bf +size 171518 diff --git a/docs/install_guides/PAC_Configure_2018R5.md b/docs/install_guides/PAC_Configure_2018R5.md index 8177adb315d2b5..1378c0c6f2cb09 100644 --- a/docs/install_guides/PAC_Configure_2018R5.md +++ b/docs/install_guides/PAC_Configure_2018R5.md @@ -236,11 +236,7 @@ classification_sample_async -m squeezenet1.1.xml -i $IE_INSTALL/demo/car.png -d classification_sample_async -m squeezenet1.1.xml -i $IE_INSTALL/demo/car.png -d HETERO:FPGA,CPU -ni 100 ``` -Congratulations, You are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and are other resources are provided below. - -## Hello World Face Detection Tutorial - -Use the [Intel® Distribution of OpenVINO™ toolkit with FPGA Hello World Face Detection Exercise](https://github.com/fritzboyle/openvino-with-fpga-hello-world-face-detection) to learn more about how the software and hardware work together. +Congratulations, You are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. ## Additional Resources diff --git a/docs/install_guides/PAC_Configure_2019RX.md b/docs/install_guides/PAC_Configure_2019RX.md index 867215540e4881..5e43876ec20e00 100644 --- a/docs/install_guides/PAC_Configure_2019RX.md +++ b/docs/install_guides/PAC_Configure_2019RX.md @@ -237,12 +237,7 @@ classification_sample_async -m squeezenet1.1.xml -i $IE_INSTALL/demo/car.png classification_sample_async -m squeezenet1.1.xml -i $IE_INSTALL/demo/car.png -d HETERO:FPGA,CPU ``` -Congratulations, You are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and are other resources are provided below. - -## Hello World Face Detection Tutorial - -Use the [Intel® Distribution of OpenVINO™ toolkit with FPGA Hello World Face Detection Exercise](https://github.com/fritzboyle/openvino-with-fpga-hello-world-face-detection) to learn more about how the software and hardware work together. - +Congratulations, You are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. ## Additional Resources Intel® Distribution of OpenVINO™ toolkit home page: [https://software.intel.com/en-us/openvino-toolkit](https://software.intel.com/en-us/openvino-toolkit) diff --git a/docs/install_guides/VisionAcceleratorFPGA_Configure_2018R5.md b/docs/install_guides/VisionAcceleratorFPGA_Configure_2018R5.md index c0082ef86f62a2..328c824fa35967 100644 --- a/docs/install_guides/VisionAcceleratorFPGA_Configure_2018R5.md +++ b/docs/install_guides/VisionAcceleratorFPGA_Configure_2018R5.md @@ -319,11 +319,7 @@ The throughput on FPGA is listed and may show a lower FPS. This is due to the in ./classification_sample_async -i car.png -m ~/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 100 ``` -Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and are other resources are provided below. - -## Hello World Face Detection Tutorial - -Use the [Intel® Distribution of OpenVINO™ toolkit with FPGA Hello World Face Detection Exercise](https://github.com/fritzboyle/openvino-with-fpga-hello-world-face-detection) to learn more about how the software and hardware work together. +Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. ## Additional Resources diff --git a/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R1.md b/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R1.md index 640f5387c38fa7..8de131e8c45161 100644 --- a/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R1.md +++ b/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R1.md @@ -270,11 +270,7 @@ The throughput on FPGA is listed and may show a lower FPS. This is due to the in ./classification_sample_async -i car.png -m ~/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 100 ``` -Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and are other resources are provided below. - -## Hello World Face Detection Tutorial - -Use the [Intel® Distribution of OpenVINO™ toolkit with FPGA Hello World Face Detection Exercise](https://github.com/fritzboyle/openvino-with-fpga-hello-world-face-detection) to learn more about how the software and hardware work together. +Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. ## Additional Resources diff --git a/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R3.md b/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R3.md index 369555f35f2f8a..06d8ebbc86939a 100644 --- a/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R3.md +++ b/docs/install_guides/VisionAcceleratorFPGA_Configure_2019R3.md @@ -270,11 +270,7 @@ Note the CPU throughput in Frames Per Second (FPS). This tells you how quickly t ``` The throughput on FPGA is listed and may show a lower FPS. This may be due to the initialization time. To account for that, increase the number of iterations or batch size when deploying to get a better sense of the speed the FPGA can run inference at. -Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and are other resources are provided below. - -## Hello World Face Detection Tutorial - -Use the [Intel® Distribution of OpenVINO™ toolkit with FPGA Hello World Face Detection Exercise](https://github.com/fritzboyle/openvino-with-fpga-hello-world-face-detection) to learn more about how the software and hardware work together. +Congratulations, you are done with the Intel® Distribution of OpenVINO™ toolkit installation for FPGA. ## Additional Resources diff --git a/docs/install_guides/installing-openvino-apt.md b/docs/install_guides/installing-openvino-apt.md index 08249588623ac6..812c6195f2c9a5 100644 --- a/docs/install_guides/installing-openvino-apt.md +++ b/docs/install_guides/installing-openvino-apt.md @@ -129,6 +129,5 @@ sudo apt autoremove intel-openvino--ubuntu-.< - [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). - [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md). - For more information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). -- For information on Inference Engine Tutorials, see the [Inference Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic). - For IoT Libraries & Code Samples see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit). diff --git a/docs/install_guides/installing-openvino-conda.md b/docs/install_guides/installing-openvino-conda.md index c491c862a682dc..a53997c4901fb5 100644 --- a/docs/install_guides/installing-openvino-conda.md +++ b/docs/install_guides/installing-openvino-conda.md @@ -49,7 +49,7 @@ Now you can start to develop and run your application. ## Known Issues and Limitations - You cannot use Python bindings included in Intel® Distribution of OpenVINO™ toolkit with [Anaconda* distribution](https://www.anaconda.com/products/individual/) -- You cannot use Python OpenVINO™ bindings included in Anaconda* package with official [Python distribution](https://https://www.python.org/). +- You cannot use Python OpenVINO™ bindings included in Anaconda* package with official [Python distribution](https://www.python.org/). ## Additional Resources @@ -59,6 +59,5 @@ Now you can start to develop and run your application. - [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). - [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md). - For more information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). -- For information on Inference Engine Tutorials, see the [Inference Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic). - Intel® Distribution of OpenVINO™ toolkit Anaconda* home page: [https://anaconda.org/intel/openvino-ie4py](https://anaconda.org/intel/openvino-ie4py) diff --git a/docs/install_guides/installing-openvino-linux.md b/docs/install_guides/installing-openvino-linux.md index 9d5c39df874948..019c14d09385a3 100644 --- a/docs/install_guides/installing-openvino-linux.md +++ b/docs/install_guides/installing-openvino-linux.md @@ -85,7 +85,6 @@ This guide provides step-by-step instructions on how to install the Intel® Dist After installing your Intel® Movidius™ VPU, you will return to this guide to complete OpenVINO™ installation. 9. Run a Sample Application 10. Uninstall the Intel® Distribution of OpenVINO™ Toolkit. -11. Use the Face Detection Tutorial ## Install the Intel® Distribution of OpenVINO™ Toolkit Core Components @@ -146,6 +145,7 @@ steps: ![](../img/openvino-install-linux-02.png) **Optional:** You can choose **Customize** to change the installation directory or the components you want to install: ![](../img/openvino-install-linux-03.png) + > **NOTE**: If there is an OpenVINO™ toolkit version previously installed on your system, the installer will use the same destination directory for next installations. If you want to install a newer version to a different directory, you need to uninstall the previously installed versions. By default, the Intel® Distribution of OpenVINO™ is installed to the following directory, referred to as ``: - For root or administrator: `/opt/intel/openvino_/` - For regular users: `/home//intel/openvino_/` @@ -448,12 +448,6 @@ cd ~/inference_engine_samples_build/intel64/Release For information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). -Congratulations, you have finished the installation of the Intel® Distribution of OpenVINO™ toolkit for Linux*. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and other resources are provided below. - -## Hello World Face Detection Tutorial - -See the [OpenVINO™ Hello World Face Detection Exercise](https://github.com/intel-iot-devkit/inference-tutorials-generic). - ## Uninstall the Intel® Distribution of OpenVINO™ Toolkit Choose one of the options provided below to uninstall the Intel® Distribution of OpenVINO™ Toolkit from your system. @@ -505,7 +499,6 @@ trusted-host = mirrors.aliyun.com - [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md). - For more information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). - For information on a set of pre-trained models, see the [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_intel_index) -- For information on Inference Engine Tutorials, see the [Inference Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic) - For IoT Libraries and Code Samples see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit). To learn more about converting models, go to: diff --git a/docs/install_guides/installing-openvino-macos.md b/docs/install_guides/installing-openvino-macos.md index bceb06a811b12a..cd1e93c88faa3a 100644 --- a/docs/install_guides/installing-openvino-macos.md +++ b/docs/install_guides/installing-openvino-macos.md @@ -304,10 +304,6 @@ For example, to install the `libusb` library using Homebrew\*, use the following brew install libusb ``` -## Hello World Tutorials - -Visit the Intel Distribution of OpenVINO Toolkit [Inference Tutorials for Face Detection and Car Detection Exercises](https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_r3_0) - ## Uninstall the Intel® Distribution of OpenVINO™ Toolkit Follow the steps below to uninstall the Intel® Distribution of OpenVINO™ Toolkit from your system: diff --git a/docs/install_guides/installing-openvino-windows.md b/docs/install_guides/installing-openvino-windows.md index 2ccd706a82cab8..ccb39d4cb69d74 100644 --- a/docs/install_guides/installing-openvino-windows.md +++ b/docs/install_guides/installing-openvino-windows.md @@ -118,6 +118,8 @@ Proceed to an [easy installation from Docker](@ref workbench_docs_Workbench_DG_I ![](../img/openvino-install-windows-01.png) +> **NOTE**: If there is an OpenVINO™ toolkit version previously installed on your system, the installer will use the same destination directory for next installations. If you want to install a newer version to a different directory, you need to uninstall the previously installed versions. + 3. Click **Next**. 4. You are asked if you want to provide consent to gather information. Choose the option of your choice. Click **Next**. @@ -459,7 +461,7 @@ cd C:\Users\\Documents\Intel\OpenVINO\inference_engine_samples_build\i For information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). -Congratulations, you have finished the installation of the Intel® Distribution of OpenVINO™ toolkit for Windows*. To learn more about how the Intel® Distribution of OpenVINO™ toolkit works, the Hello World tutorial and other resources are provided below. +Congratulations, you have finished the installation of the Intel® Distribution of OpenVINO™ toolkit for Windows*. ## Uninstall the Intel® Distribution of OpenVINO™ Toolkit Follow the steps below to uninstall the Intel® Distribution of OpenVINO™ Toolkit from your system: @@ -491,7 +493,6 @@ To learn more about converting deep learning models, go to: - [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) - [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md) - [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_intel_index) -- Intel Distribution of OpenVINO Toolkit Hello World Activities, see the [Inference Tutorials for Face Detection and Car Detection Exercises](https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_r3_0) - [Intel® Neural Compute Stick 2 Get Started](https://software.intel.com/en-us/neural-compute-stick/get-started) diff --git a/docs/install_guides/installing-openvino-yum.md b/docs/install_guides/installing-openvino-yum.md index 2dab2bcdf938ab..5fc6143ae5133d 100644 --- a/docs/install_guides/installing-openvino-yum.md +++ b/docs/install_guides/installing-openvino-yum.md @@ -106,6 +106,5 @@ sudo yum autoremove intel-openvino-runtime-centos-. - [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). - [Inference Engine Developer Guide](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md). - For more information on Sample Applications, see the [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md). -- For information on Inference Engine Tutorials, see the [Inference Tutorials](https://github.com/intel-iot-devkit/inference-tutorials-generic). - For IoT Libraries & Code Samples see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit).