diff --git a/examples/README.md b/examples/README.md index 0793ccbee94..f56c56c0ea9 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,10 +2,18 @@ This folder lists some examples to run Proton in various use cases. For more real-world examples, please check https://docs.timeplus.com/showcases -- ecommerce: a combination of Proton, Redpanda, owl-shop and Redpanda Console. Owl Shop is an imaginary ecommerce shop that simulates microservices exchanging data via Apache Kafka. Sample data streams are: clickstreams(frontend events), customer info, customer orders. [Learn more](https://docs.timeplus.com/proton-kafka#tutorial) +- awesome-sensor-logger: You can install the free app for your phone, https://github.com/tszheichoi/awesome-sensor-logger, and use a proxy server to redirect the IoT sensor data to your local Proton, or even visualize it with Grafana. - carsharing: just two containers: Proton and [Chameleon](https://github.com/timeplus-io/chameleon). It is an imginary carsharing company. Sensors are equipped in each car to report car locations. The customers use the mobile app to find available cars nearby, book them, unlock them and hit the road. At the end of the trip, the customer parks the car, locks it, and ends the trip. The payment will proceed automatically with the registered credit card. [Learn more](https://docs.timeplus.com/usecases) +- cdc: demonstrates how to use Debezium to sync database changes from MySQL to Proton, via Redpanda and show live updates(UPSERT and DELETE) in Proton via changelog stream. + +- ecommerce: a combination of Proton, Redpanda, owl-shop and Redpanda Console. Owl Shop is an imaginary ecommerce shop that simulates microservices exchanging data via Apache Kafka. Sample data streams are: clickstreams(frontend events), customer info, customer orders. [Learn more](https://docs.timeplus.com/proton-kafka#tutorial) + +- fraud_detection: demonstrates how to leverage proton to build a real-time fraud detection where proton is used as a real-time feature store. + - hackernews: just two containers: Proton and [a bytewax-based data loader](https://github.com/timeplus-io/proton-python-driver/tree/develop/example/bytewax). Inspired by https://bytewax.io/blog/polling-hacker-news, you can call Hacker News HTTP API with Bytewax and send latest news to Proton for SQL-based analysis. - grafana: an example of how to use Grafana to connect to Proton and visualize the query results. + +- jdbc: demonstrates how to connect to Proton via JDBC using DBeaver or Metabase. \ No newline at end of file diff --git a/examples/awesome-sensor-logger/.dockerignore b/examples/awesome-sensor-logger/.dockerignore new file mode 100644 index 00000000000..f965aed116d --- /dev/null +++ b/examples/awesome-sensor-logger/.dockerignore @@ -0,0 +1,15 @@ +node_modules +Dockerfile* +docker-compose* +.dockerignore +.git +.gitignore +README.md +LICENSE +.vscode +Makefile +helm-charts +.env +.editorconfig +.idea +coverage* diff --git a/examples/awesome-sensor-logger/.gitignore b/examples/awesome-sensor-logger/.gitignore new file mode 100644 index 00000000000..468f82a1aad --- /dev/null +++ b/examples/awesome-sensor-logger/.gitignore @@ -0,0 +1,175 @@ +# Based on https://raw.githubusercontent.com/github/gitignore/main/Node.gitignore + +# Logs + +logs +_.log +npm-debug.log_ +yarn-debug.log* +yarn-error.log* +lerna-debug.log* +.pnpm-debug.log* + +# Caches + +.cache + +# Diagnostic reports (https://nodejs.org/api/report.html) + +report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json + +# Runtime data + +pids +_.pid +_.seed +*.pid.lock + +# Directory for instrumented libs generated by jscoverage/JSCover + +lib-cov + +# Coverage directory used by tools like istanbul + +coverage +*.lcov + +# nyc test coverage + +.nyc_output + +# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) + +.grunt + +# Bower dependency directory (https://bower.io/) + +bower_components + +# node-waf configuration + +.lock-wscript + +# Compiled binary addons (https://nodejs.org/api/addons.html) + +build/Release + +# Dependency directories + +node_modules/ +jspm_packages/ + +# Snowpack dependency directory (https://snowpack.dev/) + +web_modules/ + +# TypeScript cache + +*.tsbuildinfo + +# Optional npm cache directory + +.npm + +# Optional eslint cache + +.eslintcache + +# Optional stylelint cache + +.stylelintcache + +# Microbundle cache + +.rpt2_cache/ +.rts2_cache_cjs/ +.rts2_cache_es/ +.rts2_cache_umd/ + +# Optional REPL history + +.node_repl_history + +# Output of 'npm pack' + +*.tgz + +# Yarn Integrity file + +.yarn-integrity + +# dotenv environment variable files + +.env +.env.development.local +.env.test.local +.env.production.local +.env.local + +# parcel-bundler cache (https://parceljs.org/) + +.parcel-cache + +# Next.js build output + +.next +out + +# Nuxt.js build / generate output + +.nuxt +dist + +# Gatsby files + +# Comment in the public line in if your project uses Gatsby and not Next.js + +# https://nextjs.org/blog/next-9-1#public-directory-support + +# public + +# vuepress build output + +.vuepress/dist + +# vuepress v2.x temp and cache directory + +.temp + +# Docusaurus cache and generated files + +.docusaurus + +# Serverless directories + +.serverless/ + +# FuseBox cache + +.fusebox/ + +# DynamoDB Local files + +.dynamodb/ + +# TernJS port file + +.tern-port + +# Stores VSCode versions used for testing VSCode extensions + +.vscode-test + +# yarn v2 + +.yarn/cache +.yarn/unplugged +.yarn/build-state.yml +.yarn/install-state.gz +.pnp.* + +# IntelliJ based IDEs +.idea + +# Finder (MacOS) folder config +.DS_Store diff --git a/examples/awesome-sensor-logger/Dockerfile b/examples/awesome-sensor-logger/Dockerfile new file mode 100644 index 00000000000..9b7bef64087 --- /dev/null +++ b/examples/awesome-sensor-logger/Dockerfile @@ -0,0 +1,37 @@ +# use the official Bun image +# see all versions at https://hub.docker.com/r/oven/bun/tags +FROM oven/bun:1 as base +WORKDIR /usr/src/app + +# install dependencies into temp directory +# this will cache them and speed up future builds +FROM base AS install +RUN mkdir -p /temp/dev +COPY package.json bun.lockb /temp/dev/ +RUN cd /temp/dev && bun install --frozen-lockfile + +# install with --production (exclude devDependencies) +RUN mkdir -p /temp/prod +COPY package.json bun.lockb /temp/prod/ +RUN cd /temp/prod && bun install --frozen-lockfile --production + +# copy node_modules from temp directory +# then copy all (non-ignored) project files into the image +FROM base AS prerelease +COPY --from=install /temp/dev/node_modules node_modules +COPY . . +COPY src src + +ENV NODE_ENV=production + +# copy production dependencies and source code into final image +FROM base AS release +COPY --from=install /temp/prod/node_modules node_modules +COPY --from=prerelease /usr/src/app/index.ts . +COPY --from=prerelease /usr/src/app/src/index.routes.ts src/index.routes.ts +COPY --from=prerelease /usr/src/app/package.json . + +# run the app +USER bun +EXPOSE 3000/tcp +ENTRYPOINT [ "bun", "run", "index.ts" ] \ No newline at end of file diff --git a/examples/awesome-sensor-logger/README.md b/examples/awesome-sensor-logger/README.md new file mode 100644 index 00000000000..bab9e561cdf --- /dev/null +++ b/examples/awesome-sensor-logger/README.md @@ -0,0 +1,15 @@ +# awesome-sensor-logger + +To install dependencies: + +```bash +bun install +``` + +To run: + +```bash +bun run index.ts +``` + +This project was created using `bun init` in bun v1.0.15. [Bun](https://bun.sh) is a fast all-in-one JavaScript runtime. diff --git a/examples/awesome-sensor-logger/bun.lockb b/examples/awesome-sensor-logger/bun.lockb new file mode 100755 index 00000000000..432bc6aa635 Binary files /dev/null and b/examples/awesome-sensor-logger/bun.lockb differ diff --git a/examples/awesome-sensor-logger/docker-compose.yml b/examples/awesome-sensor-logger/docker-compose.yml new file mode 100644 index 00000000000..e0aa130dfa1 --- /dev/null +++ b/examples/awesome-sensor-logger/docker-compose.yml @@ -0,0 +1,19 @@ +version: '3.7' +services: + proton: + image: ghcr.io/timeplus-io/proton:latest + pull_policy: always + ports: + - 3218:3218 #http port for JDBC driver, default streaming mode + - 8123:8123 #http port for JDBC driver, default batch mode + - 8463:8463 #tcp port for go driver or grafana plugin + + proxy: + image: timeplus/sensor-logger-proxy + ports: + - 8000:3000 #expose internal port 3000 as 8000, avoiding conflicts with Grafana. This will forward HTTP POST to proton:3218 + environment: + HOST: proton + STREAM: phone + depends_on: + - proton diff --git a/examples/awesome-sensor-logger/index.ts b/examples/awesome-sensor-logger/index.ts new file mode 100644 index 00000000000..4377a8fd8b1 --- /dev/null +++ b/examples/awesome-sensor-logger/index.ts @@ -0,0 +1,5 @@ +import { init } from '@stricjs/app'; + +init({ + routes: ['./src'] +}); \ No newline at end of file diff --git a/examples/awesome-sensor-logger/package.json b/examples/awesome-sensor-logger/package.json new file mode 100644 index 00000000000..2c9be3e07d1 --- /dev/null +++ b/examples/awesome-sensor-logger/package.json @@ -0,0 +1,19 @@ +{ + "name": "awesome-sensor-logger", + "module": "index.ts", + "type": "module", + "scripts": { + "start": "bun run index.ts" + }, + "devDependencies": { + "bun-types": "latest" + }, + "peerDependencies": { + "typescript": "^5.0.0" + }, + "dependencies": { + "@stricjs/app": "^1.0.0-alpha", + "@stricjs/router": "^5.0.6", + "@stricjs/utils": "^1.6.1" + } +} \ No newline at end of file diff --git a/examples/awesome-sensor-logger/src/index.routes.ts b/examples/awesome-sensor-logger/src/index.routes.ts new file mode 100644 index 00000000000..2015024e69d --- /dev/null +++ b/examples/awesome-sensor-logger/src/index.routes.ts @@ -0,0 +1,15 @@ +import { routes } from '@stricjs/app'; + +export function main() { + const protonIngestEndpoint = `http://${process.env.HOST}:3218/proton/v1/ingest/streams/${process.env.STREAM}`; + return routes() + .post('/', c => { + return c.text().then(a => { + return fetch(protonIngestEndpoint, { + method: "POST", + body: `{"columns": ["raw"],"data": [["${a.replaceAll('"', '\\\"')}"]]}`, + headers: { "Content-Type": "application/json" }, + }).then(protonResp => new Response('status code ' + protonResp.status)); + }); + }) +} \ No newline at end of file diff --git a/examples/awesome-sensor-logger/tsconfig.json b/examples/awesome-sensor-logger/tsconfig.json new file mode 100644 index 00000000000..7556e1d4b08 --- /dev/null +++ b/examples/awesome-sensor-logger/tsconfig.json @@ -0,0 +1,22 @@ +{ + "compilerOptions": { + "lib": ["ESNext"], + "module": "esnext", + "target": "esnext", + "moduleResolution": "bundler", + "moduleDetection": "force", + "allowImportingTsExtensions": true, + "noEmit": true, + "composite": true, + "strict": true, + "downlevelIteration": true, + "skipLibCheck": true, + "jsx": "react-jsx", + "allowSyntheticDefaultImports": true, + "forceConsistentCasingInFileNames": true, + "allowJs": true, + "types": [ + "bun-types" // add Bun global + ] + } +} diff --git a/examples/cdc/README.md b/examples/cdc/README.md new file mode 100644 index 00000000000..9f979d81597 --- /dev/null +++ b/examples/cdc/README.md @@ -0,0 +1,50 @@ +# Demo for CDC(Change Data Capture) with Debezium/Redpanda/Proton + +This docker compose file demonstrates how to capture live database change from a OLTP database(e.g. MySQL) and apply real-time analytics with Proton. + +## Start the example + +Simply run `docker compose up` in this folder. Five docker containers in the stack: +1. ghcr.io/timeplus-io/proton:latest, as the streaming database. +2. docker.redpanda.com/redpandadata/redpanda, as the Kafka compatiable streaming message bus +3. docker.redpanda.com/redpandadata/console, as the web UI to explore data in Kafka/Redpanda +4. debezium/connect, as the CDC engine to read changes from OLTP and send data to Kafka/Redpanda +5. debezium/example-mysql, a pre-configured MySQL + +## Create the CDC job +Perform the following command in your host server, since port 8083 is exposed from Debezium Connect. +```shell +curl --request POST \ + --url http://localhost:8083/connectors \ + --header 'Content-Type: application/json' \ + --data '{ + "name": "inventory-connector", + "config": { + "connector.class": "io.debezium.connector.mysql.MySqlConnector", + "tasks.max": "1", + "database.hostname": "mysql", + "database.port": "3306", + "database.user": "debezium", + "database.password": "dbz", + "database.server.id": "184054", + "topic.prefix": "dbserver1", + "database.include.list": "inventory", + "schema.history.internal.kafka.bootstrap.servers": "redpanda:9092", + "schema.history.internal.kafka.topic": "schema-changes.inventory" + } +}' +``` + +## Run SQL + +You can use `docker exec -it proton-client -m -n` to run the SQL client in Proton container. Or use the Docker Desktop UI to choose the container, choose "Exec" tab and type `proton-client` to start the SQL client. + +Copy all content in cdc.sql and paste in the Proton Client. + +Run +```sql +select * from customers +``` +to see the current data. + +Use a MySQL client(e.g. DBeaver) to add/update/delete some records to see the update from `select * from customers`. You can also run `select * from table(customers)` to avoid waiting for new updates. \ No newline at end of file diff --git a/examples/cdc/cdc.sql b/examples/cdc/cdc.sql new file mode 100644 index 00000000000..e4948d1f410 --- /dev/null +++ b/examples/cdc/cdc.sql @@ -0,0 +1,54 @@ +-- read the topic via an external stream +CREATE EXTERNAL STREAM customers_cdc(raw string) + SETTINGS type='kafka', + brokers='redpanda:9092', + topic='dbserver1.inventory.customers'; + +CREATE STREAM customers(id int, first_name string, last_name string,email string) +PRIMARY KEY id +SETTINGS mode='changelog_kv', version_column='_tp_time'; + +CREATE MATERIALIZED VIEW mv_customers_r INTO customers AS + SELECT to_time(raw:payload.ts_ms) AS _tp_time, + raw:payload.after.id::int AS id, + raw:payload.after.first_name AS first_name, + raw:payload.after.last_name AS last_name, + raw:payload.after.email AS email, + 1::int8 as _tp_delta + FROM customers_cdc WHERE raw:payload.op='r' SETTINGS seek_to='earliest'; +CREATE MATERIALIZED VIEW mv_customers_u INTO customers AS + WITH before AS( + SELECT to_time(raw:payload.ts_ms)-1ms AS _tp_time, + raw:payload.before.id::int AS id, + raw:payload.before.first_name AS first_name, + raw:payload.before.last_name AS last_name, + raw:payload.before.email AS email, + -1::int8 as _tp_delta + FROM customers_cdc WHERE raw:payload.op='u' SETTINGS seek_to='earliest' + ),after AS( + SELECT to_time(raw:payload.ts_ms) AS _tp_time, + raw:payload.after.id::int AS id, + raw:payload.after.first_name AS first_name, + raw:payload.after.last_name AS last_name, + raw:payload.after.email AS email, + 1::int8 as _tp_delta + FROM customers_cdc WHERE raw:payload.op='u' SETTINGS seek_to='earliest' + )SELECT * FROM before UNION SELECT * FROM after; + +CREATE MATERIALIZED VIEW mv_customers_d INTO customers AS + SELECT to_time(raw:payload.ts_ms) AS _tp_time, + raw:payload.before.id::int AS id, + raw:payload.before.first_name AS first_name, + raw:payload.before.last_name AS last_name, + raw:payload.before.email AS email, + -1::int8 as _tp_delta + FROM customers_cdc WHERE raw:payload.op='d' SETTINGS seek_to='earliest'; + +CREATE MATERIALIZED VIEW mv_customers_c INTO customers AS + SELECT to_time(raw:payload.ts_ms) AS _tp_time, + raw:payload.after.id::int AS id, + raw:payload.after.first_name AS first_name, + raw:payload.after.last_name AS last_name, + raw:payload.after.email AS email, + 1::int8 as _tp_delta + FROM customers_cdc WHERE raw:payload.op='c' SETTINGS seek_to='earliest'; \ No newline at end of file diff --git a/examples/cdc/docker-compose.yml b/examples/cdc/docker-compose.yml new file mode 100644 index 00000000000..0373d02a1d5 --- /dev/null +++ b/examples/cdc/docker-compose.yml @@ -0,0 +1,69 @@ +version: "3.3" +services: + proton: + image: ghcr.io/timeplus-io/proton:latest + pull_policy: always + ports: + # - 3218:3218 #http port for JDBC driver, default streaming mode + - 8123:8123 #http port for JDBC driver, default batch mode + - 8463:8463 #native port for proton client + redpanda: + image: docker.redpanda.com/redpandadata/redpanda:v23.2.15 + ports: + - "9092:9092" + - "29092:29092" + command: + - redpanda + - start + - --overprovisioned + - --smp + - "1" + - --memory + - "1G" + - --reserve-memory + - "0M" + - --node-id + - "0" + - --kafka-addr + - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 + - --advertise-kafka-addr + - PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092 + - --check=false + console: + image: docker.redpanda.com/redpandadata/console:v2.3.5 + entrypoint: /bin/sh + command: -c "echo \"$$CONSOLE_CONFIG_FILE\" > /tmp/config.yml; /app/console" + environment: + CONFIG_FILEPATH: /tmp/config.yml + CONSOLE_CONFIG_FILE: | + kafka: + brokers: ["redpanda:9092"] + connect: + enabled: true + clusters: + - name: cdc + url: http://connect:8083 + ports: + - 8080:8080 + depends_on: + - redpanda + connect: + image: debezium/connect + depends_on: + - redpanda + ports: + - "8083:8083" + environment: + BOOTSTRAP_SERVERS: "redpanda:9092" + GROUP_ID: "1" + CONFIG_STORAGE_TOPIC: "inventory.configs" + OFFSET_STORAGE_TOPIC: "inventory.offset" + STATUS_STORAGE_TOPIC: "inventory.status" + mysql: + image: debezium/example-mysql:2.4 + ports: + - "3306:3306" + environment: + MYSQL_ROOT_PASSWORD: debezium + MYSQL_USER: mysqluser + MYSQL_PASSWORD: mysqlpw \ No newline at end of file diff --git a/examples/fraud_detection/README.md b/examples/fraud_detection/README.md index 5a66e96c8cf..c635cc6e563 100644 --- a/examples/fraud_detection/README.md +++ b/examples/fraud_detection/README.md @@ -1,6 +1,6 @@ # Demo for real-time machine learning - fraud detection -This docker compose file demonstrates how to leverage proton to build a real-time fraud detection where proton is used as a real-time feature store +This docker compose file demonstrates how to leverage proton to build a real-time fraud detection where proton is used as a real-time feature store. ## Start the example diff --git a/examples/grafana/README.md b/examples/grafana/README.md index e621aa2a27c..aecd06baa54 100644 --- a/examples/grafana/README.md +++ b/examples/grafana/README.md @@ -21,7 +21,7 @@ Install Grafana if you haven't. Please note, this plugin is not available for Gr For Mac users, simply run `brew install grafana`. -Before the plugin is approved by Grafana, you need to set your Grafana running in development mode via changing /usr/local/etc/grafana/grafana.ini, setting `app_mode = development`. +Before the plugin is approved by Grafana, you need to set your Grafana running in development mode via changing /opt/homebrew/etc/grafana/grafana.ini, setting `app_mode = development`. Also searching for key word "plugin", and change `enable_alpha` and `allow_loading_unsigned_plugins` to ``` @@ -37,7 +37,7 @@ We have submited the plugin to Grafana. Once it's approved, you will be able to Install the timeplus-proton-datasource in your Grafana plugin folder, such as - /var/lib/grafana/plugins (for Linux) -- /usr/local/var/lib/grafana/plugins (for Homebrew on Mac) +- /opt/homebrew/var/lib/grafana/plugins (for Homebrew on Mac) Unzip the file and restart Grafana.