From a4529afb28b53d09a8f95db63944a51cf38f1de7 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 20:01:49 +0000 Subject: [PATCH 1/7] bad go commands --- packages/blobstorage/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/blobstorage/README.md b/packages/blobstorage/README.md index 4a8b5cb916e..8c28507eeaa 100644 --- a/packages/blobstorage/README.md +++ b/packages/blobstorage/README.md @@ -15,7 +15,7 @@ cd local_docker && docker-compose up -d 2. Start the `indexer` ```bash -ENV_FILE=.default.indexer.env run cmd/main.go indexer +ENV_FILE=.default.indexer.env go run cmd/main.go indexer ``` By default the above command starts the app from the latest block height. If we want to specifiy a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. @@ -23,7 +23,7 @@ By default the above command starts the app from the latest block height. If we 2. Start the `server`. ```bash -ENV_FILE=.default.server.env run cmd/main.go server +ENV_FILE=.default.server.env go run cmd/main.go server ``` ## how to test / use ? From 4f0a1185c338f78331292e344e862df62d6ffd64 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 21:18:19 +0000 Subject: [PATCH 2/7] fix some of the typos as well --- packages/blobstorage/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/blobstorage/README.md b/packages/blobstorage/README.md index 8c28507eeaa..9f3878c5b9f 100644 --- a/packages/blobstorage/README.md +++ b/packages/blobstorage/README.md @@ -18,9 +18,9 @@ cd local_docker && docker-compose up -d ENV_FILE=.default.indexer.env go run cmd/main.go indexer ``` -By default the above command starts the app from the latest block height. If we want to specifiy a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. +By default the above command starts the app from the latest block height. If we want to specify a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. -2. Start the `server`. +3. Start the `server`. ```bash ENV_FILE=.default.server.env go run cmd/main.go server @@ -28,7 +28,7 @@ ENV_FILE=.default.server.env go run cmd/main.go server ## how to test / use ? -When the `DB`, `blob-catcher` and `server` is running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, tho it is not written into the DB). Use that `blobHash` (including the 0x) in +When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, though it is not written into the DB). Use that `blobHash` (including the 0x) in 1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array): @@ -52,4 +52,4 @@ python3 python_query.py What is still missing is: -- small refinements and DevOps (prod-grade DB with creditentials, proper containerization) +- small refinements and DevOps (prod-grade DB with credentials, proper containerization) From c84d1b571fd2e32154a099fde0ba50773c6cf385 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 22:37:07 +0000 Subject: [PATCH 3/7] added docker-compose.yml for mysql --- packages/docker-compose/docker-compose.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 packages/docker-compose/docker-compose.yml diff --git a/packages/docker-compose/docker-compose.yml b/packages/docker-compose/docker-compose.yml new file mode 100644 index 00000000000..09976b8f4ca --- /dev/null +++ b/packages/docker-compose/docker-compose.yml @@ -0,0 +1,14 @@ +version: "3.8" +services: + db: + image: mysql:8.0 + cap_add: + - SYS_NICE + restart: always + environment: + - MYSQL_DATABASE=blobs + - MYSQL_ROOT_PASSWORD=passw00d + ports: + - "3306:3306" + volumes: + - ~/.docker-conf/mysql/data/:/var/lib/mysql From b15cf99aa275f545ebc84a9261a524438efeba67 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 23:26:47 +0000 Subject: [PATCH 4/7] updated README and mysql config in all files --- packages/blobstorage/.default.indexer.env | 6 +- packages/blobstorage/.default.server.env | 6 +- packages/blobstorage/README.md | 70 +++++++++++++++++------ packages/blobstorage/python_query.py | 29 ++++++---- 4 files changed, 75 insertions(+), 36 deletions(-) diff --git a/packages/blobstorage/.default.indexer.env b/packages/blobstorage/.default.indexer.env index a2fddf76ede..13073c65b35 100644 --- a/packages/blobstorage/.default.indexer.env +++ b/packages/blobstorage/.default.indexer.env @@ -2,8 +2,8 @@ RPC_URL=wss://l1ws.internal.taiko.xyz BEACON_URL=https://l1beacon.internal.taiko.xyz/eth/v1/beacon/blob_sidecars/ TAIKO_L1_CONTRACT_ADDRESS=0xC069c3d2a9f2479F559AD34485698ad5199C555f DATABASE_HOST=localhost -DATABASE_PORT=27017 +DATABASE_PORT=3306 DATABASE_USER=root -DATABASE_PASSWORD=root +DATABASE_PASSWORD=passw00d DATABASE_NAME=blobs -METRICS_HTTP_PORT=7472 \ No newline at end of file +METRICS_HTTP_PORT=7472 diff --git a/packages/blobstorage/.default.server.env b/packages/blobstorage/.default.server.env index 7c469c7f072..d7c46611e1d 100644 --- a/packages/blobstorage/.default.server.env +++ b/packages/blobstorage/.default.server.env @@ -1,7 +1,7 @@ HTTP_PORT=3282 DATABASE_HOST=localhost -DATABASE_PORT=27017 +DATABASE_PORT=3306 DATABASE_USER=root -DATABASE_PASSWORD=root +DATABASE_PASSWORD=passw00d DATABASE_NAME=blobs -METRICS_HTTP_PORT=7471 \ No newline at end of file +METRICS_HTTP_PORT=7471 diff --git a/packages/blobstorage/README.md b/packages/blobstorage/README.md index 9f3878c5b9f..4e50511602f 100644 --- a/packages/blobstorage/README.md +++ b/packages/blobstorage/README.md @@ -1,39 +1,71 @@ # blob-storage -Repo for BLOB storage (archive and serve data) +Repository for BLOB storage (archive and serve data) -## how to run ? +## Prerequisites -Prerequisite is to have docker engine up and running. +- Docker engine up and running. +- Go installed on your system. -1. Start the mongoDB +## Configuration and Setup -```bash -cd local_docker && docker-compose up -d -``` +### Setting up MySQL -2. Start the `indexer` +1. **Start MySQL**: -```bash -ENV_FILE=.default.indexer.env go run cmd/main.go indexer -``` + Navigate to the `docker-compose` directory and start the MySQL service: -By default the above command starts the app from the latest block height. If we want to specify a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. + ```bash + cd ./docker-compose + docker-compose up -d + ``` -3. Start the `server`. + This command starts your MySQL instance as defined in your `docker-compose.yml` file. -```bash -ENV_FILE=.default.server.env go run cmd/main.go server -``` +2. **Migrate Database Schema**: + + Navigate to the `migrations` directory to apply database migrations: + + ```bash + cd ./migrations + goose mysql "root:passw00d@tcp(localhost:3306)/blobs" status + goose mysql "root:passw00d@tcp(localhost:3306)/blobs" up + ``` + + These commands apply migrations to the `blobs` database. + +### Environment Configuration + +Ensure your `.default.indexer.env` and `.default.server.env` files are configured with the correct database credentials, host, and any other necessary environment variables. + +## Running the Application + +1. **Start the Indexer**: + + With the environment file configured, start the indexer: + + ```bash + ENV_FILE=.default.indexer.env go run cmd/main.go indexer + ``` + + This starts the app from the latest block height by default. Adjust the `STARTING_BLOCK_ID` in the environment file if needed. + +2. **Start the Server**: + + Similarly, start the server: + + ```bash + ENV_FILE=.default.server.env go run cmd/main.go server + ``` -## how to test / use ? +## Testing and Usage When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, though it is not written into the DB). Use that `blobHash` (including the 0x) in 1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array): ```bash -curl -X GET "http://localhost:27001/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" +curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" ``` The result will be something like this: @@ -48,7 +80,7 @@ The result will be something like this: python3 python_query.py ``` -## todos +## Todos What is still missing is: diff --git a/packages/blobstorage/python_query.py b/packages/blobstorage/python_query.py index c14d373368e..d689ca6ea33 100644 --- a/packages/blobstorage/python_query.py +++ b/packages/blobstorage/python_query.py @@ -1,20 +1,27 @@ -from pymongo import MongoClient +import mysql.connector def query_blob_hash(collection_name, blob_hash): - # Connect to MongoDB - client = MongoClient('localhost', 27017) - # Access the database - db = client['blob_storage'] + # Connect to MySQL + connection = mysql.connector.connect( + host='localhost', + user='root', # Update with your username + password='passw00d', # Update with your password + database='blobs' # Update with your database name + ) - # Access the collection - collection = db[collection_name] + cursor = connection.cursor(dictionary=True) - # Query the collection for the blobHash - result = collection.find_one({'blob_hash': blob_hash}) + # Query the table for the blobHash + query = f"SELECT * FROM blob_hashes WHERE blob_hash = %s" + cursor.execute(query, (blob_hash,)) - # Close the connection - client.close() + # Fetch one result + result = cursor.fetchone() + + # Close the cursor and connection + cursor.close() + connection.close() return result From f77ee0905104302c42789c86f91bdcc61ec413d0 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 23:34:37 +0000 Subject: [PATCH 5/7] improve README --- packages/blobstorage/README.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/packages/blobstorage/README.md b/packages/blobstorage/README.md index 4e50511602f..4322cbecf18 100644 --- a/packages/blobstorage/README.md +++ b/packages/blobstorage/README.md @@ -62,19 +62,25 @@ Ensure your `.default.indexer.env` and `.default.server.env` files are configure When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, though it is not written into the DB). Use that `blobHash` (including the 0x) in -1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array): +1. **Querying Blob Data via HTTP Request**: + + To retrieve blob data, you can execute a `curl` command. This allows for querying multiple `blobHashes` simultaneously, separated by commas. A single request can yield an array of results: ```bash curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" ``` -The result will be something like this: +**Expected Output**: ```bash {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} ``` -2. Or to backtest, use the simple python script below, after overwriting the `blob_hash` variable: +2. **Backtesting with a Python Script**: + +This script facilitates querying the database directly based on a specified `blob_hash`. Modify the `blob_hash` variable in the script to match the hash you wish to query. + +To run the script: ```bash python3 python_query.py From 6852e3b42d806521d20c9a54b72b44e674bfcaef Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Sat, 16 Mar 2024 23:38:40 +0000 Subject: [PATCH 6/7] improve README --- packages/blobstorage/README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/packages/blobstorage/README.md b/packages/blobstorage/README.md index 4322cbecf18..e7fe0e15ce9 100644 --- a/packages/blobstorage/README.md +++ b/packages/blobstorage/README.md @@ -66,25 +66,25 @@ When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is ou To retrieve blob data, you can execute a `curl` command. This allows for querying multiple `blobHashes` simultaneously, separated by commas. A single request can yield an array of results: -```bash -curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" -``` + ```bash + curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" + ``` -**Expected Output**: + **Expected Output**: -```bash -{"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} -``` + ```bash + {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} + ``` 2. **Backtesting with a Python Script**: -This script facilitates querying the database directly based on a specified `blob_hash`. Modify the `blob_hash` variable in the script to match the hash you wish to query. + This script facilitates querying the database directly based on a specified `blob_hash`. Modify the `blob_hash` variable in the script to match the hash you wish to query. -To run the script: + To run the script: -```bash -python3 python_query.py -``` + ```bash + python3 python_query.py + ``` ## Todos From 29a615bff21639bc88c4e758d66eb1b1c069eca4 Mon Sep 17 00:00:00 2001 From: SamiAlHassan <150172180+SamiAlHassan@users.noreply.github.com> Date: Wed, 20 Mar 2024 17:48:30 +0000 Subject: [PATCH 7/7] fix location of docker-compose.yml for blobstorage --- packages/{ => blobstorage}/docker-compose/docker-compose.yml | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename packages/{ => blobstorage}/docker-compose/docker-compose.yml (100%) diff --git a/packages/docker-compose/docker-compose.yml b/packages/blobstorage/docker-compose/docker-compose.yml similarity index 100% rename from packages/docker-compose/docker-compose.yml rename to packages/blobstorage/docker-compose/docker-compose.yml