Consolidate flow json spec (#417)

* Consolidate flow json spec

* json ref: update header

* fix interfacedirection xref

docs/flows-format.adoc

@@ -1,13 +1,13 @@
 // Automatically generated by 'hack/asciidoc-flows-gen.sh'. Do not edit, or make the NETOBSERV team aware of the editions.
 :_content-type: REFERENCE
-[id="network-observability-flows-format-specifications_{context}"]
-= Network Flows format specifications
+[id="network-observability-flows-format_{context}"]
+= Network Flows format reference
 
 This is the specification of the network flows format, used both internally and when exporting flows to Kafka.
 
 The document is organized in two main categories: _Labels_ and regular _Fields_. This distinction only matters when querying Loki. This is because _Labels_, unlike _Fields_, must be used in link:https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector[stream selectors].
 
-When using this specification as a reference for exporting flows to Kafka, you must ignore this distinction about _Labels_ and _Fields_, and consider all of them at the same level in the JSON structure.
+If you are reading this specification as a reference for the Kafka export feature, you must treat all _Labels_ and _Fields_ as regular fields and ignore any distinctions between them that are specific to Loki.
 
 
 == Labels
@@ -130,15 +130,15 @@ Kind of the destination matched Kubernetes object, such as Pod name, Service nam
 
 SrcPort::
 
-• *SrcPort*: `number`
+• `Optional` *SrcPort*: `number`
 
 Source port
 
 '''
 
 DstPort::
 
-• *DstPort*: `number`
+• `Optional` *DstPort*: `number`
 
 Destination port
 
@@ -208,11 +208,27 @@ Network interface
 
 '''
 
+IfDirection::
+
+• `Optional` *IfDirection*: `InterfaceDirection` (see the following section, Enumeration: InterfaceDirection)
+
+Flow direction from the network interface observation point
+
+'''
+
+Flags::
+
+• `Optional` *Flags*: `number`
+
+TCP flags
+
+'''
+
 Packets::
 
-• *Packets*: `number`
+• `Optional` *Packets*: `number`
 
-Number of packets in this flow
+Number of packets
 
 '''
 
@@ -234,9 +250,9 @@ In conversation tracking, B to A packets counter per conversation
 
 Bytes::
 
-• *Bytes*: `number`
+• `Optional` *Bytes*: `number`
 
-Number of bytes in this flow
+Number of bytes
 
 '''
 
@@ -256,6 +272,126 @@ In conversation tracking, B to A bytes counter per conversation
 
 '''
 
+IcmpType::
+
+• `Optional` *IcmpType*: `number`
+
+ICMP type
+
+'''
+
+IcmpCode::
+
+• `Optional` *IcmpCode*: `number`
+
+ICMP code
+
+'''
+
+PktDropLatestState::
+
+• `Optional` *PktDropLatestState*: `string`
+
+Pkt TCP state for drops
+
+'''
+
+PktDropLatestDropCause::
+
+• `Optional` *PktDropLatestDropCause*: `string`
+
+Pkt cause for drops
+
+'''
+
+PktDropLatestFlags::
+
+• `Optional` *PktDropLatestFlags*: `number`
+
+Pkt TCP flags for drops
+
+'''
+
+PktDropPackets::
+
+• `Optional` *PktDropPackets*: `number`
+
+Number of packets dropped by the kernel
+
+'''
+
+PktDropPackets_AB::
+
+• `Optional` *PktDropPackets_AB*: `number`
+
+In conversation tracking, A to B packets dropped counter per conversation
+
+'''
+
+PktDropPackets_BA::
+
+• `Optional` *PktDropPackets_BA*: `number`
+
+In conversation tracking, B to A packets dropped counter per conversation
+
+'''
+
+PktDropBytes::
+
+• `Optional` *PktDropBytes*: `number`
+
+Number of bytes dropped by the kernel
+
+'''
+
+PktDropBytes_AB::
+
+• `Optional` *PktDropBytes_AB*: `number`
+
+In conversation tracking, A to B bytes dropped counter per conversation
+
+'''
+
+PktDropBytes_BA::
+
+• `Optional` *PktDropBytes_BA*: `number`
+
+In conversation tracking, B to A bytes dropped counter per conversation
+
+'''
+
+DnsId::
+
+• `Optional` *DnsId*: `number`
+
+DNS record id
+
+'''
+
+DnsFlags::
+
+• `Optional` *DnsFlags*: `number`
+
+DNS flags for DNS record
+
+'''
+
+DnsFlagsResponseCode::
+
+• `Optional` *DnsFlagsResponseCode*: `string`
+
+Parsed DNS header RCODEs name
+
+'''
+
+DnsLatencyMs::
+
+• `Optional` *DnsLatencyMs*: `number`
+
+Calculated time between response and request, in milliseconds
+
+'''
+
 TimeFlowStartMs::
 
 • *TimeFlowStartMs*: `number`
@@ -280,6 +416,14 @@ Timestamp when this flow was received and processed by the flow collector, in se
 
 '''
 
+TimeFlowRttNs::
+
+• `Optional` *TimeFlowRttNs*: `number`
+
+Flow Round Trip Time (RTT) in nanoseconds
+
+'''
+
 _HashId::
 
 • `Optional` *_HashId*: `string`
@@ -310,12 +454,38 @@ Ingress::
 
 • *Ingress* = `"0"`
 
-Incoming traffic, from node observation point
+Incoming traffic, from the node observation point
+
+'''
+
+Egress::
+
+• *Egress* = `"1"`
+
+Outgoing traffic, from the node observation point
+
+'''
+
+Inner::
+
+• *Inner* = `"2"`
+
+Inner traffic, with the same source and destination node
+
+
+== Enumeration: InterfaceDirection
+
+
+Ingress::
+
+• *Ingress* = `"0"`
+
+Incoming traffic, from the network interface observation point
 
 '''
 
 Egress::
 
 • *Egress* = `"1"`
 
-Outgoing traffic, from node observation point
+Outgoing traffic, from the network interface observation point

hack/asciidoc-flows-gen.sh

@@ -20,12 +20,19 @@ echo -e "\n" >> $ADOC
 kramdoc -o - <(curl -fsSL ${MD_SOURCE}/interfaces/Fields.md) \
   | sed -r 's/^= /== /' \
   | sed -r 's/Interface: //' \
-  | sed -r '/Properties/d' >> $ADOC
+  | sed -r '/Properties/d' \
+  | sed -r 's~xref:.*InterfaceDirection\.adoc\[`InterfaceDirection`\]~`InterfaceDirection` (see the following section, Enumeration: InterfaceDirection)~' >> $ADOC
 
 # FlowDirection enum
 echo -e "\n" >> $ADOC
 kramdoc -o - <(curl -fsSL ${MD_SOURCE}/enums/FlowDirection.md) \
   | sed -r 's/^= /== /' \
   | sed -r '/Enumeration Members/d' >> $ADOC
 
+# InterfaceDirection enum
+echo -e "\n" >> $ADOC
+kramdoc -o - <(curl -fsSL ${MD_SOURCE}/enums/InterfaceDirection.md) \
+  | sed -r 's/^= /== /' \
+  | sed -r '/Enumeration Members/d' >> $ADOC
+
 sed -i -r "s/^=== (.+)/\1::/" $ADOC

hack/flows-format-header.adoc

@@ -1,9 +1,9 @@
 :_content-type: REFERENCE
-[id="network-observability-flows-format-specifications_{context}"]
-= Network Flows format specifications
+[id="network-observability-flows-format_{context}"]
+= Network Flows format reference
 
 This is the specification of the network flows format, used both internally and when exporting flows to Kafka.
 
 The document is organized in two main categories: _Labels_ and regular _Fields_. This distinction only matters when querying Loki. This is because _Labels_, unlike _Fields_, must be used in link:https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector[stream selectors].
 
-When using this specification as a reference for exporting flows to Kafka, you must ignore this distinction about _Labels_ and _Fields_, and consider all of them at the same level in the JSON structure.
+If you are reading this specification as a reference for the Kafka export feature, you must treat all _Labels_ and _Fields_ as regular fields and ignore any distinctions between them that are specific to Loki.