unified shv log format

SHV remote log

getLog

getLog({"since": dt1, "until": dt2, ... })

struct SHVIOTQT_DECL_EXPORT ShvJournalGetLogParams
{
	shv::chainpack::RpcValue::DateTime since;
	shv::chainpack::RpcValue::DateTime until;
	/// '*' and '**' wild-cards are supported
	/// '*' stands for single path segment, shv/pol/*/discon match shv/pol/ols/discon but not shv/pol/ols/depot/discon
	/// '**' stands for zero or more path segments, shv/pol/**/discon matches shv/pol/discon, shv/pol/ols/discon, shv/pol/ols/depot/discon
	std::string pathPattern;
	enum class HeaderOptions : unsigned {
		BasicInfo = 1 << 0,
		FileldInfo = 1 << 1,
		TypeInfo = 1 << 2,
		PathsDict = 1 << 3,
	};
	unsigned headerOptions = static_cast<unsigned>(HeaderOptions::BasicInfo);
	int maxRecordCount = 1000;
	bool withSnapshot = false;

	ShvJournalGetLogParams() {}
	ShvJournalGetLogParams(const shv::chainpack::RpcValue &opts);
};

Unified SHV log format

Unified log format is used when it is retrieved from any SHV device.

Example:

<
	"dateTime":d"1970-01-11T18:51:47.232Z",
	"logVersion":1,
	"params":{
		"headerOptions":15u,    
		"maxRecordCount":1000,    
		"since":d"2018-10-01T10:20:30Z",
		"withSnapshot":false
	},
	"device":{
		"id":"DS:1"
	},
	"fields":[
		{"name": "timestamp"},
		{"name": "path"},
		{"name": "value"},
	],
	"typeInfo": {
		"types" {
			"Status": {
				"type": "BitField",
				"fields": [
					{"name": "PosOff", "value": 0}, 
					{"name": "PosOn", "value": 1}, 
					{"name": "PosMiddle", "value": 2}, 
					{"name": "PosError", "value": 3}, 
					{"name": "BatteryLow", "value": 4}, 
					{"name": "BatteryHigh", "value": 5}, 
					{"name": "DoorOpenCabinet", "value": 6}, 
					{"name": "DoorOpenMotor", "value": 7}, 
					{"name": "ModeAuto", "value": 8}, 
					{"name": "ModeRemote", "value": 9}, 
					{"name": "ModeService", "value": 10}, 
					{"name": "MainSwitch", "value": 11},
				],
				"description":"status of something"
			},	
			"VehicleDir": {
				"type": "Enum",
				"fields": [
					{"name": "Left", "value": 0},
					{"name": "Right", "value": 1},
				]
			},
			"AncaData": {
				"type": "List",
				"fields": [
					{"name": "ShortTime", "type": "UInt"},
					{"name": "Voltage", "type": "Int"},
					{"name": "Current", "type": "Int"},
					{"name": "Power", "type": "Int"},
				]
			},
			"VTKEventData":{
				"type": "IMap",
				"fields": [
					{
						"name": "vehicleDir", 
						"value": 1,
						"type": "VehicleDir",
					},
					{
						"name": "VehicleID", 
						"value": 2,
						"type": "Int",
						"description":"Tram number"
					},
				],
				"description":"VTK event data"
			},
		},
		"paths":{
			"status": {"type": "Status"},
			"tramInfo1":{
				"type": "VTKEventData",
				"description":"VTK event data"
			},
			"tramInfo2":{
				"type": "VTKEventData",
				"description":"VTK event data"
			},
		},
	}
>
[
	[d"1970-01-11T15:38:50.104Z", "status", 2u],
	[d"1970-01-11T18:51:37.987Z", "status", 1u]
]

Device log

One possible device log format implementation to support the Unified SHV log format in simple way

• Devices are logging to the text file
• Every entry is one line terminated by \n
• Fields in line are separated by \t
• Minimal fields are:
  1. timestamp - monotonic timestamp with format yyyy-mm-ddThh:MM:ss.zzz
  2. uptime - number of seconds since device boot, this information is necessary to reconstruct timestamp in rows where device system time wasn't in sync with NTP
  3. path - string
  4. value - Cpon coded logged value for path

Notes:

1. It is a device responsibility to ensure monotonic timestamp. When device boots, it have to read last log entry and remember recently logged timestamp. If device time after reboot is lover than recent TS, it have to log new events with the recent timestamp. Even if new timestamps will not be correct, they can be reconstructed later on, when device will connect to internet and sync system time with NTP, using uptime value.
2. It is necessary to keep timestamp monotonic even if it is not correct to enable binary search in logs and to generate the Unified log format easily.
3. \n and \t are good line and field separators, since they never appear in Cpon packed values
4. One option to implement device logs was to use SQLite. Simple file was choosen according to Do the simplest thing that could possibly work rule from following reasons:
5. Append line to file is really cheep and fast in almost every possible OS
6. To ensure DB consistency even when device is rebooted suddenly, Sqlite will create rollback journal on SD card https://www.sqlite.org/tempfiles.html , this can produce undesirable write cycles on EEPROM.
7. restore corrupted text log file is pretty easy task comparing to do the same thing with Sqlite
8. Open Sqlite with every log entry can spend some system resources to read table structures and init SQL library. App can keep Sqlite connected all its lifetime to come over this.

log file example

1970-01-11T15:38:50.104Z--->123456--->status--->2u
1970-01-11T18:51:37.987Z--->123457--->status--->1u