Skip to content

Commit

Permalink
Add dtrack.vuln-analysis.result.processed topic
Browse files Browse the repository at this point in the history 
As introduced in DependencyTrack/hyades-apiserver#637

Also removes Kafka Streams related environment variables for API server from `docker-compose.yml`

Signed-off-by: nscuro <[email protected]>
  • Loading branch information
@nscuro
nscuro committed Apr 11, 2024
1 parent 2070772 commit 62e5d75
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 23 deletions.
8 changes: 2 additions & 6 deletions docker-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,16 +133,12 @@ services:
# which can cause problems on some workstations.
# For production deployments, the default should be used.
EXTRA_JAVA_OPTIONS: "-Xmx2g"
ALPINE_DATABASE_MODE: "external"
ALPINE_DATABASE_URL: "jdbc:postgresql://dt-postgres:5432/dtrack"
ALPINE_DATABASE_DRIVER: "org.postgresql.Driver"
ALPINE_DATABASE_USERNAME: "dtrack"
ALPINE_DATABASE_PASSWORD: "dtrack"
ALPINE_METRICS_ENABLED: "true"
ALPINE_SECRET_KEY_PATH: "/var/run/secrets/secret.key"
KAFKA_BOOTSTRAP_SERVERS: "dt-redpanda:29092"
KAFKA_NUM_STREAM_THREADS: "12" # Default number of input partitions is 12
KAFKA_STREAMS_METRICS_RECORDING_LEVEL: "DEBUG"
INTEGRITY_CHECK_ENABLED: "true"
ports:
- "127.0.0.1:8080:8080"
Expand Down Expand Up @@ -237,8 +233,6 @@ services:
# REPO_META_ANALYSIS_TOPICS_RETENTION_MS: "43200000" # 12h
# VULN_ANALYSIS_TOPICS_PARTITIONS: "3"
# VULN_ANALYSIS_TOPICS_RETENTION_MS: "43200000" # 12h
# VULN_ANALYSIS_RESULT_TOPIC_PARTITIONS: "3"
# VULN_ANALYSIS_RESULT_TOPIC_RETENTION_MS: "43200000" # 12h
# VULN_MIRROR_TOPICS_PARTITIONS: "3"
# VULN_MIRROR_TOPICS_RETENTION_MS: "43200000" # 12h
volumes:
Expand Down Expand Up @@ -302,6 +296,8 @@ services:
- topicName: dtrack.vuln-analysis.result
keyProtoType: org.dependencytrack.vulnanalysis.v1.ScanKey
valueProtoType: org.dependencytrack.vulnanalysis.v1.ScanResult
- topicName: dtrack.vuln-analysis.result.processed
valueProtoType: org.dependencytrack.vulnanalysis.v1.ScanResult
- topicName: dtrack.vulnerability
valueProtoType: org.cyclonedx.v1_4.Bom
fileSystem:
Expand Down
30 changes: 15 additions & 15 deletions docs/reference/topics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

| Name | Partitions | Config |
|:--------------------------------------------------------------------------------------------------|:-----------|:------------------------------------------------------------------------------------|
| `dtrack-apiserver-processed-vuln-scan-result-by-scan-token-repartition`<sup>1A</sup> | 3 | |
| `dtrack.notification.analyzer` | 3 | |
| `dtrack.notification.bom` | 3 | |
| `dtrack.notification.configuration` | 3 | |
Expand All @@ -17,26 +16,27 @@
| `dtrack.notification.repository` | 3 | |
| `dtrack.notification.vex` | 3 | |
| `dtrack.notification.project-vuln-analysis-complete` <sup>3</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `dtrack.repo-meta-analysis.component`<sup>1B</sup> | 3 | |
| `dtrack.repo-meta-analysis.component`<sup>1A</sup> | 3 | |
| `dtrack.repo-meta-analysis.result` | 3 | |
| `dtrack.vuln-analysis.component`<sup>1C</sup> | 3 | |
| `dtrack.vuln-analysis.result`<sup>1A</sup> | 3 | |
| `dtrack.vuln-analysis.scanner.result`<sup>1C</sup> | 3 | |
| `dtrack.vuln-analysis.component`<sup>1B</sup> | 3 | |
| `dtrack.vuln-analysis.result` | 3 | |
| `dtrack.vuln-analysis.result.processed` | 3 | |
| `dtrack.vuln-analysis.scanner.result`<sup>1B</sup> | 3 | |
| `dtrack.vulnerability` | 3 | `cleanup.policy=compact` |
| `dtrack.vulnerability.digest`<sup>2</sup> | 1 | `cleanup.policy=compact` |
| `dtrack.vulnerability.mirror.command`<sup>2</sup> | 1 | |
| `dtrack.vulnerability.mirror.state`<sup>2</sup> | 1 | `cleanup.policy=compact` |
| `hyades-repository-meta-analyzer-command-by-purl-coordinates-repartition`<sup>1B</sup> | 3 | |
| `hyades-vulnerability-analyzer-completed-scans-table-changelog`<sup>1C</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-expected-scanner-results-last-update-store-changelog`<sup>1C</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-expected-scanner-results-table-changelog`<sup>1C</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-ossindex-batch-store-changelog`<sup>1D</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-ossindex-retry-store-changelog`<sup>1D</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-repository-meta-analyzer-command-by-purl-coordinates-repartition`<sup>1A</sup> | 3 | |
| `hyades-vulnerability-analyzer-completed-scans-table-changelog`<sup>1B</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-expected-scanner-results-last-update-store-changelog`<sup>1B</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-expected-scanner-results-table-changelog`<sup>1B</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-ossindex-batch-store-changelog`<sup>1C</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-ossindex-retry-store-changelog`<sup>1C</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-scan-task-internal-repartition` | 3 | |
| `hyades-vulnerability-analyzer-scan-task-ossindex-repartition`<sup>1D</sup> | 3 | |
| `hyades-vulnerability-analyzer-scan-task-snyk-repartition`<sup>1E</sup> | 3 | |
| `hyades-vulnerability-analyzer-snyk-batch-store-changelog`<sup>1E</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-snyk-retry-store-changelog`<sup>1E</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-scan-task-ossindex-repartition`<sup>1C</sup> | 3 | |
| `hyades-vulnerability-analyzer-scan-task-snyk-repartition`<sup>1D</sup> | 3 | |
| `hyades-vulnerability-analyzer-snyk-batch-store-changelog`<sup>1D</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |
| `hyades-vulnerability-analyzer-snyk-retry-store-changelog`<sup>1D</sup> | 3 | `cleanup.policy=compact`<br/>`segment.bytes=67108864`<br/>`max.compaction.lag.ms=0` |

*<sup>1X</sup> The topic is subject to [co-partitioning requirements](#co-partitioning-requirements)*
*<sup>2</sup> The partition number of this topic should not be changed*
Expand Down
4 changes: 2 additions & 2 deletions scripts/create-topics.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,13 @@ done
vuln_analysis_topics=(
"${KAFKA_TOPIC_PREFIX:-}dtrack.vuln-analysis.component"
"${KAFKA_TOPIC_PREFIX:-}dtrack.vuln-analysis.scanner.result"
"${KAFKA_TOPIC_PREFIX:-}dtrack.vuln-analysis.result"
"${KAFKA_TOPIC_PREFIX:-}dtrack.vuln-analysis.result.processed"
)
for topic_name in "${vuln_analysis_topics[@]}"; do
create_topic "$topic_name" "${VULN_ANALYSIS_TOPICS_PARTITIONS:-3}" "retention.ms=${VULN_ANALYSIS_TOPICS_RETENTION_MS:-43200000}"
done

create_topic "${KAFKA_TOPIC_PREFIX:-}dtrack.vuln-analysis.result" "${VULN_ANALYSIS_RESULT_TOPIC_PARTITIONS:-3}" "retention.ms=${VULN_ANALYSIS_RESULT_TOPIC_RETENTION_MS:-43200000}"

create_topic "${KAFKA_TOPIC_PREFIX:-}dtrack.vulnerability.mirror.command" "1" "retention.ms=${VULN_MIRROR_TOPICS_RETENTION_MS:-43200000}"
create_topic "${KAFKA_TOPIC_PREFIX:-}dtrack.vulnerability.mirror.state" "1" "cleanup.policy=compact segment.bytes=67108864 max.compaction.lag.ms=0"
create_topic "${KAFKA_TOPIC_PREFIX:-}dtrack.vulnerability.digest" "1" "cleanup.policy=compact segment.bytes=134217728"
Expand Down

0 comments on commit 62e5d75

Please sign in to comment.