Brings back documentation merge removals #286 #287

Signed-off-by: Thomas Druez <[email protected]>

CHANGELOG.rst

@@ -1,6 +1,11 @@
 // Release notes
 // -------------
 
+### Unreleased
+
+- Log the outputs of run_scancode as progress indication.
+  https://github.com/nexB/scancode.io/issues/300
+
 ### v21.8.2
 
 - Upgrade ScanCode-toolkit to version 21.7.30

docs/built-in-pipelines.rst

@@ -15,12 +15,16 @@ Pipeline Base Class
     :members:
     :member-order: bysource
 
+.. _pipeline_docker:
+
 Docker Image Analysis
 ---------------------
 .. autoclass:: scanpipe.pipelines.docker.Docker()
     :members:
     :member-order: bysource
 
+.. _pipeline_docker_windows:
+
 Docker Windows Image Analysis
 -----------------------------
 .. autoclass:: scanpipe.pipelines.docker_windows.DockerWindows()
@@ -33,6 +37,8 @@ Load Inventory From Scan
     :members:
     :member-order: bysource
 
+.. _pipeline_root_filesystems:
+
 Root Filesystem Analysis
 ------------------------
 .. autoclass:: scanpipe.pipelines.root_filesystems.RootFS()

docs/index.rst

@@ -28,6 +28,7 @@ you’ll find information on:
     tutorial-2
     tutorial-3
     tutorial-4
+    tutorial-5
 
 .. toctree::
     :maxdepth: 2
@@ -42,6 +43,7 @@ you’ll find information on:
     command-line-interface
     rest-api
     scancodeio-settings
+    recognized-distros-os-images
 
 Indices and tables
 ==================

docs/recognized-distros-os-images.rst

@@ -0,0 +1,64 @@
+.. _recognized_distros_os_images:
+
+Recognized Distros, OS, and Images
+==================================
+
+Archives Formats
+----------------
+
+ScanCode.io recognizes and **can extract most archive formats**; however, it offers
+special support for VM and container image formats:
+
+- **Docker image tarbal** archives using a Docker image layout
+- **Virtual machine images** using **QCOW** and **VHDI** image format
+
+Operating Systems Detection
+---------------------------
+
+When scanning for Docker or virtual machine (VM) images, one of the first tasks
+of a pipeline after extracting an archive is to **detect the operating system**.
+For Linux, this also includes detecting the installed Linux distribution, which
+checks for certain files such as ``/etc/os-release`` on Linux.
+The detected OS—distro—is then used to determine **which system packages are
+likely installed**, such as RPM or Debian packages.
+
+For each recognized OS, a pipeline collects the following information:
+
+- OS and image details
+- Installed system packages metadata, license and their files
+- Installed application packages metadata, license and their files
+- Details for files not part of a package
+
+Installed System Packages
+-------------------------
+
+ScanCode.io recognizes the following OS technology combinations; some of which
+may be only used for certain pipelines:
+
+- **Debian-based** Linux distros: Debian, Ubuntu and Debian-derivative
+- **RPM-based** Linux distros: RHEL, Fedora, openSUSE/SUSE
+- **Alpine** Linux distros
+
+For the above three flavors, the :ref:`docker <pipeline_docker>` and
+:ref:`root_filesystems <pipeline_root_filesystems>` pipelines support comprehensive
+detection of installed system packages, their provenance, their license metadata,
+and their installed files.
+
+- For **Windows**, the :ref:`docker_windows <pipeline_docker_windows>` pipeline supports
+  Windows Docker images with extensive detection of installed Windows packages,
+  programs, and the majority of installed files.
+
+- **Distroless** Docker images system packages are detected with the
+  :ref:`docker <pipeline_docker>` pipeline; package and license metadata are also
+  detected.
+  However, some work needs to be done to achieve comprehensive support and fix
+  the issue of system packages ot tracking their installed files. Check `this
+  open GitHub issue <https://github.com/GoogleContainerTools/distroless/issues/741>`_
+  for more details.
+
+- **Yocto** and **OpenWRT** Linux VM images are partially supported; adding more support
+  is currently in progress.
+
+- Other distros and OS will be scanned; however, we might not be able to detect
+  system installed packages and may return a larger volume of file-level
+  detections.

docs/rest-api.rst

@@ -75,7 +75,7 @@ An API endpoint that provides the ability to list, get, and create projects.
     }
 
 Create a project
-^^^^^^^^^^^^^^^^
+----------------
 
 Using cURL:
 
@@ -109,18 +109,18 @@ Using Python and the **"requests"** library:
     response.json()
 
 
-When creating a project, the response will provide the project details URL value
-in the returned data.
+When creating a project, the response will include the project's details URL
+value among the returned data.
 You can make a GET request to this URL, which returns all available information
 about the project, including the status of any pipeline run:
 
 .. code-block:: json
 
     {
-        "name":"project_name",
-        "url":"/api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/",
-        "uuid":"6461408c-726c-4b70-aa7a-c9cc9d1c9685",
-        "created_date":"2021-07-21T16:06:29.132795+02:00"
+        "name": "project_name",
+        "url": "/api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/",
+        "uuid": "6461408c-726c-4b70-aa7a-c9cc9d1c9685",
+        "created_date": "2021-07-21T16:06:29.132795+02:00"
     }
 
 Project details

docs/tutorial-4.rst

@@ -4,7 +4,7 @@ Review Scan Results (Web UI)
 ============================
 
 This chapter is complementary to the :ref:`tutorial_3` tutorial, and the output
-shown here is the generated results of the tutorial's pipeline run.
+included here represents the generated results of the tutorial's pipeline run.
 The goal here is to guide you on how to understand and review your scan
 results using the ScanCode.io web interface.
 

docs/tutorial-5.rst

@@ -0,0 +1,129 @@
+.. _tutorial_5:
+
+Analyse Package Archive (REST API)
+==================================
+
+This tutorial complements the :ref:`rest_api` section, and the aim here is to
+show the API features while analyzing a package archive.
+
+.. tip::
+    As a perquisite, check our :ref:`rest_api` chapter for more details on REST
+    API and how to get started.
+
+Instructions:
+
+- First, let's create a new project called ``boolean.py-3.8``.
+- We'll be using this `package <https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip>`_
+  as the project input.
+- We can add and execute the scan_package pipeline on our new project.
+
+.. note::
+    Whether you follow this tutorial and previous instructions using cURL or
+    Python script, the final results should be the same.
+
+Using cURL
+----------
+
+- In your terminal, insert the following:
+
+.. code-block:: bash
+
+    api_url="http://127.0.0.1:8001/api/projects/"
+    content_type="Content-Type: application/json"
+    data='{
+        "name": "boolean.py-3.8",
+        "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
+        "pipeline": "scan_package",
+        "execute_now": true
+    }'
+
+    curl -X POST "$api_url" -H "$content_type" -d "$data"
+
+.. note::
+    You can set the api_url to http://localhost/api/projects/ when running with
+    Docker.
+
+.. tip::
+    You can provide the data using a json file with the text below, which will be
+    passed in the -d parameter of the curl request:
+
+    .. code-block:: json
+
+        {
+            "name": "boolean.py-3.8",
+            "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
+            "pipeline": "scan_package",
+            "execute_now": true
+        }
+
+    While in the same directory as your JSON file, here called
+    ``boolean.py-3.8_cURL.json``, create your new project with the following
+    curl request:
+
+    .. code-block:: bash
+
+        curl -X POST "http://127.0.0.1:8001/api/projects/" -H "Content-Type: application/json" -d @boolean.py-3.8_cURL.json
+
+If the new project has been successfully created, the response should include
+the project's details URL value among the returned data.
+
+.. code-block:: json
+
+    {
+        "name": "boolean.py-3.8",
+        "url": "http://127.0.0.1:8001/api/projects/11de938f-fb86-4178-870c-99f4952b8881/",
+        "[...]": "[...]"
+    }
+
+If you click on the project url, you'll be directed to the new project's
+instance page that allows you to perform extra actions on the project including
+deleting it.
+
+.. note::
+    Refer to our :ref:`rest_api` section for more information about these extra actions.
+
+Using Python script
+-------------------
+
+.. tip::
+    To interact with REST APIs, we will be turning to the requests library.
+
+- To follow the above instructions and create a new project, start up the Python
+  interpreter by typing ``python`` in your terminal.
+- If you are seeing the prompt ``>>>``, you can execute the following commands:
+
+.. code-block:: python
+
+    import requests
+
+    api_url = "http://127.0.0.1:8001/api/projects/"
+    data = {
+        "name": "boolean.py-3.8",
+        "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
+        "pipeline": "scan_package",
+        "execute_now": True,
+    }
+    response = requests.post(api_url, data=data)
+    response.json()
+
+The JSON response includes a generated UUID for the new project.
+
+.. code-block:: python
+
+    # print(response.json())
+    {
+        "name": "boolean.py-3.8",
+        "url": "http://127.0.0.1:8001/api/projects/11de938f-fb86-4178-870c-99f4952b8881/",
+        "[...]": "[...]",
+    }
+
+.. note::
+    Alternatively, you can create a Python script with the above commands/text.
+    Then, navigate to the same directory as your Python file and run the script
+    to create your new project. However, no response will be shown on the
+    terminal, and to access a given project details, you need to visit the
+    projects' API endpoint.
+
+.. tip::
+    You can check the :ref:`rest_api` section for more details on how to view
+    and download your scan results.