Merge branch 'development'

# Conflicts:
#	docs/Importing.md

.gitignore

@@ -141,3 +141,5 @@ dmypy.json
 # VSCode
 .vscode/
 .VSCodeCounter
+ACME oneM2M CSE.docset
+apidocs

CHANGELOG.md

@@ -7,6 +7,61 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+
+## [0.10.0] - 2022-05-06
+
+### Added
+- [CSE] Added *--http-port* command line argument.
+- [CSE] Added initial support for the Upper Tester protocol defined in TS-0019.
+- [CSE] Added guided setup of a configuration file when the CSE is started without a config file.
+- [CSE] Added bi-directional update of announced resources.
+- [CSE] Added remote resources support for <group>.
+- [CSE] Added support for *dataGenerationTime* attribute for <contentInstance> (R5 feature).
+- [CSE] Added "acme://" URL scheme for notifications to run ACMEScript scripts.
+- [CSE] Added always setting *Originating Timestamp* in requests and responses.
+- [CSE] Added [logging].queueSize configuration to set the internal logging queue (or switch it off).
+- [CSE] Added validation of <flexContainer>'s *containerDefinition* attribute during CREATE requests.
+- [CSE] Added check for correct Service Provider ID in absolute requests.
+- [CSE] Added support for <pollingChannel>'s *requestAggregation* attribute and functionality.
+- [CSE] Added support for BLOCKING UPDATE notification event type.
+- [CSE] Added a first support for the <timeSyncBeacon> resource type.
+- [WEB] Allow to open the WebUI of a registered CSE via the context menu.
+- [CONSOLE] Added config for dark (default) and light theme for better readability on consoles with light background.
+- [CONSOLE] Added graph plotting for <contentInstance> resources that contain numerical values (also: continuous observation of a container).
+- [SCRIPTS] Added scripting to the CSE.
+- [SCRIPTS] A dedicated startup script is now executed during startup. This is mainly used to import the base resources and resource structure, and replaces the JSON resource imports.
+- [SCRIPTS] Added executing scripts to the console
+- [SCRIPTS] Added scripts tagged with "@uppertester" can be executed as upper tester commands.
+- [SCRIPTS] Added scripts scheduling vie the "@at" meta tag.
+- [SCRIPTS] Added possibility to run scripts on notifications, when using the "acme://" URL scheme.
+- [HTTP] Added workaround for missing DELETE method in http/1.0 (by using PATCH instead).
+
+### Changed
+- [CSE] Adapted Announcements to latest R4 changes. 
+- [CSE] Adapted TimeSeries to latest R4 changes. 
+- [CSE] Changed the default release version to 4. Also, the supported and the actual release versions are now fully configurable (in the config file).
+- [CSE] Changed name of *holder* attribute to *custodian* according to R4 spec change.
+- [CSE] Transit requests will now be handled after the resolution for blocking/non-blocking was handled. Non-blocking happens in the first CSE that received the original request.
+- [CSE] Improved feedback instructions when problems during startup are encountered, e.g. how to install missing packages.
+- [CSE] Introduced a thread pool to reuse threads.
+- [TESTS] Replaced CSE test cases' re-configurations with upper tester commands / script calls.
+- [DATABASE] Optimizations when working with resource lists.
+
+### Removed
+- [HTTP] Removed the http server's configuration and reset endpoints. This functionality is now handled by the upper tester endpoint, commands and scripts.
+- [CSE] Removed import of JSON resources from the init directory during startup. This functionality is now provided by a startup script. 
+
+### Fixed
+- [CSE] Improved check that IDs contain only unreserved characters.
+- [CSE] Improved check for validating non-empty list attributes.
+- [CSE] Improved error detection and handling for RCN=7 (original-resource).
+- [CSE] Added missing "creator" attribute in notifications when the creator was set in the <subscription>.
+- [CSE] Improved support to recognize structured and unstructured resource IDs for verification notifications.
+- [HTTP] HTTP requests will not be sent with a *Date* header field. Instead, the *Originating Timestamp* will be used.
+
+### Experimental
+- [CSE] Blocking RETRIEVE notification event type: A RETRIEVE is blocked and a notification is sent to a target to give it a change to update the original RETRIEVE's target resource.
+
 ## [0.9.1] - 2021-11-09
 
 ### Fixed
@@ -41,9 +96,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [CSE] Changed optionality of and originator assignment to CSR.csi according to TS-0004 spec change.
 - [HTTP] Moved the security setting for http to the separate section *server.http.security*.
 - [IMPORTING] Changed the file extension for <flexContainer> attribute policies from ".ap" to ".fcp".
+- [IMPORTING] Updated the <flexContainer> attribute policies to the latest version of TS-0023.
 - [Logging] Changed the internal handling of log messages. Output should be more immediate than before.
 - [RUNTIME] Due to the restructuring of the project structure the CSE must now be started like this: ```python3 -m acme```
 
+
 ### Removed
 - [CSE] Removed "cse.enableNotifications" configuration option. Notifications are now always enabled.
 - [CSE] Removed "cse.enableTransitRequests" configuration option. Requests are now always forwarded.
@@ -94,6 +151,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [CSE] Refactored internal request handling to support future protocol binding developments other than http.
 - [CSE] Changed the sorting of request result lists to type and creation time for <contentInstance>, <flexContainerInstance> and <timeSeriesInstance>.
 - [CSE] Improved the validation when registering <remoteCSE> resources for *csi* and *cb* attributes.
+- [CSE] Added validation of complex attributes in general.
 - [RUNTIME] The CSE is now started by running the *acme* module (```python3 acme```)
 
 ### Fixed

README.md

@@ -3,7 +3,7 @@
 # ACME oneM2M CSE
 An open source CSE Middleware for Education.
 
-Version 0.9.1
+Version 0.10.0
 
 [![oneM2M](https://img.shields.io/badge/oneM2M-f00)](https://www.onem2m.org) [![Python](https://img.shields.io/badge/Python-3.8-blue)](https://www.python.org) [![Maintenance](https://img.shields.io/badge/Maintained-Yes-green.svg)](https://github.com/ankraft/ACME-oneM2M-CSE/graphs/commit-activity) [![License](https://img.shields.io/badge/License-BSD%203--Clause-green)](LICENSE) [![MyPy](https://img.shields.io/badge/MyPy-covered-green)](LICENSE)  
 [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/acmeCSE.svg?style=social&label=%40acmeCSE)](https://twitter.com/acmeCSE)
@@ -28,6 +28,10 @@ Please consult the following pages for further instructions.
 - [Web & Rest UI](docs/WebUI.md)
 - [Importing Resources](docs/Importing.md)
 - [Operation](docs/Operation.md)
+- [ACMEScript](docs/ACMEScript.md)
+	- [Commands](docs/ACMEScript-commands.md)
+	- [Macros](docs/ACMEScript-macros.md)
+	- [Meta Tags](docs/ACMEScript-metatags.md)
 - [Supported Resource Types and Functionalities](docs/Supported.md)
 	- [Limitations](docs/Supported.md#limitations)
 - [Roadmap](docs/Roadmap.md)
@@ -39,28 +43,30 @@ Please consult the following pages for further instructions.
 
 Please see the [Changelog](CHANGELOG.md) for the detailed list of changes.
 
-### Important 1: Example AEs removed
-This release removes the example  AEs from the project. They will become available in another project in the future. This make the distribution a bit smaller and removes the problematic dependency to the *psutil* package, which is not available on all platforms.
+### Important: Changed CSE bootstrapping
+Before this release the initial resources to bootstrap the CSE were JSON scripts that were imported at the CSE's startup phase.
+With this release this is changed to running the script *startup.as* that [imports the initial resources](docs/Importing.md#resources).
 
-### Important 2: Changed the start method (again)
-Due to the project restructuring the CSE is now started with the command ```python3 -m acme```.
 
 ### Highlights in this release
 
-- Long-necessary refactoring of the project source structure.
-- Simplified initial configuration.
-- MQTT Binding support.
-- PollingChannel support.
-- Improved evaluation of target URIs for notifications, forward URIs etc.
-- New console commands (Continuous updating tree view, resource export etc).
-- Support for delayed operation execution.
-- Lots of small improvements, bug fixes, and optimizations
+- <timeSyncBeacon> resource type support
+- Guided setup process to create the initial configuration file
+- Adapted announcements and time-series procedures to latest R4 changes
+- Support for the Upper Tester interface defined in oneM2M's TS-0019
+- Adding scripting support to the CSE that can be used through the console, the Upper Tester, CSE events and notifications, and other
+- Improved the CSE console (e.g. drawing graph diagrams from <contentInstance> resources, dark & light theme)
+- Added support for BLOCKING UPDATE notification event type
+- Improved announced resources (e.g. bi-directional update)
+- Improved validation of complex attributes
+- And, as usual, many improvements, bug fixes, and performance improvements
 
 ## Acknowledgements
 
 Thank you for contributed code, patches, testing, bug fixes, time, and more!
 
 ![reinaortega](https://github.com/reinaortega.png?size=24) [Miguel Angel Reina Ortega](https://github.com/reinaortega)  
+![BobFIV](https://github.com/BobFIV.png?size=24) [BobFIV](https://github.com/BobFIV)  
 ![YannGarcia](https://github.com/YannGarcia.png?size=24) [Yann Garcia](https://github.com/YannGarcia)  
 <img src="https://github.com/massimov.png" width="24"> [Massimo Vanetti](https://github.com/massimov)  
 

acme.ini.default

@@ -49,17 +49,17 @@
 ; IN Configuration
 ;
 
-[basic.config]
-cseType=IN
-cseID=id-in
-cseName=cse-in
-adminID=CAdmin
-dataDirectory=${baseDirectory}
-networkInterface=0.0.0.0
-cseHost=127.0.0.1
-httpPort=8080
-logLevel=debug
-databaseInMemory=False
+; [basic.config]
+; cseType=IN
+; cseID=id-in
+; cseName=cse-in
+; adminID=CAdmin
+; dataDirectory=${baseDirectory}
+; networkInterface=0.0.0.0
+; cseHost=127.0.0.1
+; httpPort=8080
+; logLevel=debug
+; databaseInMemory=False
 
 ;
 ; MN Configuration
@@ -100,9 +100,9 @@ databaseInMemory=False
 ; databaseInMemory=False
 
 ; registrarCseHost=127.0.0.1
-; registrarCsePort=8080
-; registrarCseID=id-in
-; registrarCseName=cse-in
+; registrarCsePort=8081
+; registrarCseID=id-mn
+; registrarCseName=cse-mn
 
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -141,11 +141,14 @@ enableRemoteCSE=true
 sortDiscoveredResources=true
 ; Interval to check for expired resources. 0 means "no checking". Default: 60 seconds
 checkExpirationsInterval=60
+; Indicate the preference for flexBlocking response types. Allowed values: "blocking", "nonblocking".
+; Default: blocking
+flexBlockingPreference=blocking
 ; A comma-separated list of supported release versions. This list can contain a single or multiple values.
 ; Default: 2a,3,4
 supportedReleaseVersions=2a,3,4
-; The release version indicator for requests. Allowed values: 1,2,3,3a,4. Default: 3
-releaseVersion=3
+; The release version indicator for requests. Allowed values: 2a,3,4. Default: 4
+releaseVersion=4
 ; Indicate the serialization format if none was given in a request and cannot be determined otherwise.
 ; Allowed values: json, cbor. Default: json
 defaultSerialization=json
@@ -163,6 +166,24 @@ enableACPChecks=true
 fullAccessAdmin=True
 
 
+;
+;	Settings for CSE operation
+;
+
+[cse.operation]
+; Thread Pool Management: Target balance between paused and running jobs (n paused for 1 running threads).
+; Default: 3.0
+jobBalanceTarget=3.0
+; Thread Pool Management: Number of get / create requests for a new thread before performing a balance check.
+; A latency of 0 disables the thread pool balancing.
+; Default: 1000
+jobBalanceLatency=1000
+; Thread Pool Management: The Factor to reduce the paused jobs (number of paused / balanceReduceFactor) in a balance check.
+; Example: a factor of 2.0 reduces the number of paused threads by half in a single balance check.
+; Default: 2.0
+jobBalanceReduceFactor=2.0
+
+
 ;
 ;	HTTP Server settings
 ;
@@ -172,26 +193,30 @@ fullAccessAdmin=True
 port=${basic.config:httpPort}
 ; Interface to listen to. Use 0.0.0.0 for "all" interfaces. Default:127.0.0.1
 listenIF=${basic.config:networkInterface}
-; Own address. Should be a local/public reachable address. Default: http://127.0.0.1:8080
+; Own address. Should be a local/public reachable address.
+; Default: http://127.0.0.1:8080
 address=http://${basic.config:cseHost}:${basic.config:httpPort}
 ; CSE Server root. Always provide a trailing /. Default: /
 root=/
 ; Enable an endpoint for get and set certain configuration values via a 
 ; REST interface.
 ; ATTENTION: Enabling this feature exposes configuration values, IDs and
 ; passwords, and is a security risk.
-; Default: False
-enableRemoteConfiguration=false
-; Enable an endpoint for getting a structured overview about a CSE's resource
-; tree and deployment infrastructure (remote CSE's)
-; ATTENTION: Enabling this feature exposes various potentially sensitive information.
+; See also the [cse.console].hideResources setting to hide resources
+; from the tree.
 ; Default: False
 enableStructureEndpoint=false
-; Enable an endpoint for resetting the CSE (remove all resources and import
-; the init directory again)
+; Enable an endpoint for supporting Upper Tester commands to the CSE.
+; This is to support certain testing and certification systems.
+; See oneM2M's TS-0019 for further details.
 ; ATTENTION: Enabling this feature may lead to a total loss of data.
 ; Default: False
-enableResetEndpoint=false
+enableUpperTesterEndpoint=false
+; Allow the http PATCH method to be used as a replacement for the DELETE
+; method. This is useful for constraint devices that only support http/1.0,
+; which doesn't specify the DELETE method.
+; Default: False
+allowPatchForDelete=false
 
 
 ;
@@ -303,7 +328,7 @@ allowedAEOriginators=C*,S*
 ; Wildcards (* and ?) are supported.
 ; Note: No leading "/"
 ; Default: empty list
-allowedCSROriginators=id-mn,id-asn
+allowedCSROriginators=
 ; Check the liveliness if the registrations to the registrar CSE and also from the registree CSEs. Default: True
 checkLiveliness=True
 
@@ -315,8 +340,8 @@ checkLiveliness=True
 [cse.registrar]
 ; URL of the Registrar CSE. Default: no default
 address=http://${basic.config:registrarCseHost}:${basic.config:registrarCsePort}
-; Registrar CSE root path. Default: /
-root=/
+; Registrar CSE root path. Never provide a trailing /. Default: empty string
+root=
 ; CSE-ID of the Registrar CSE. Default: None
 cseID=/${basic.config:registrarCseID}
 ; The Registrar CSE's resource name. Default: None
@@ -353,6 +378,10 @@ stackTraceOnError=False
 ; Enable logging of low-level HTTP & MQTT client events.
 ; Default: False
 enableBindingsLogging=False
+; Number of log entries that can be added to the asynchronous queue before blocking.
+; A queue size of 0 means disabling the queue.
+; Default: 5000 entries
+queueSize=5000
 
 
 ;
@@ -415,6 +444,21 @@ mbs=10000
 mdn=10
 
 
+;
+;	Resurce defaults: TimeSyncBeacon
+;
+
+[cse.resource.tsb]
+; Default timeSyncBeacon interval. This is the duration between to beacon notifications sent by the CSE
+; to an AE or CSE. The format must be an ISO8601 duration.
+; Default: PT1H = 1 hour
+bcni=PT1H
+; Default timeSyncBeacon threshold. When this time threshold is passed then a beacon notifications
+; is sent to an AE or CSE.
+; Default: 10.0 seconds
+bcnt=10.0
+
+
 ;
 ;	Resource defaults: Request
 ;
@@ -458,14 +502,38 @@ refreshInterval=2.0
 ; Wildcards are allowed.
 ; Default: Empty list
 hideResources=
-; Set the mode how resources and their content are presented in the console's tree view. 
+; Set the mode how resources and their content are presented in the console's and structure endpoint tree view. 
 ; Allowed values: normal, compact, content, contentOnly
 ; Default: normal
 treeMode=normal
+; Show virtual resources in the console's and structure endpoint's tree view.
+; Default: False
+treeIncludeVirtualResources=False
 ; Quitting the console needs to be confirmed.
 ; This may not work under Windows, so it should be switched off.
 ; Default: False
 confirmQuit=False
+; Set the color theme for the console. Allowed values are "dark" and "light".
+; Default: dark
+theme=dark
+
+
+;
+;	Scripting settings
+;
+
+[cse.scripting]
+; Add one or multiple directory paths to look for scripts, in addition to the ones in the "init" directory.
+; Must be a comma-separated list.
+; Default: not set
+scriptDirectories=
+; Enable debug output during script execution, such as the current executed line.
+; Default: False
+verbose=false
+; Set the interval to check for new files in the script (init) directory.
+; 0 means disable monitoring.
+; Default: 2.0 seconds
+fileMonitoringInterval=2.0
 
 
 ;