Skip to content

Commit

Permalink
consolidate running
Browse files Browse the repository at this point in the history
  • Loading branch information
raulb committed May 10, 2024
1 parent b64141d commit 8abff50
Show file tree
Hide file tree
Showing 15 changed files with 199 additions and 458 deletions.
1 change: 1 addition & 0 deletions docs/connectors/connector-list.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: "Connector List"
sidebar_position: 8
---

The Conduit team and our community of developers are always adding new connectors.
Expand Down
4 changes: 4 additions & 0 deletions docs/getting-started/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "Getting Started",
"position": 1
}
File renamed without changes.
8 changes: 8 additions & 0 deletions docs/getting-started/getting-started.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: 'Getting Started with Conduit'
sidebar_position: 0
hide_title: true
sidebar_label: "Getting Started"
---

# TODO: 🐝 basic instruction pointing to each guide
139 changes: 139 additions & 0 deletions docs/getting-started/installing-and-running.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
title: "Installing and running"
sidebar_position: 0
---

## Running Conduit using a pre-built binary

The recommended way of running Conduit on a local machine is using a pre-built binary. Below we'll walk through getting Conduit installed and running on your system.

### Download

First, download the [latest Conduit release](https://github.com/ConduitIO/conduit/releases/latest) for your platform.

For an ARM based back Mac for example we'd grab the `conduit_0.10.0_Darwin_arm64.tar.gz` archive.

### unzip the archive

```shell
tar xzvf conduit_0.10.0_Darwin_arm64.tar.gz
```

### Run

In the directory in which you expanded the archive. Execute the Conduit binary:

```shell
./conduit
```

```shell
....
.::::::::::.
.:::::‘‘‘‘:::::.
.:::: ::::.
.:::::::: ::::::::.
`:::::::: ::::::::‘
`:::: ::::‘
`:::::....:::::‘
`::::::::::‘ Conduit v0.10.0 linux/amd64
‘‘‘‘
2024-02-20T21:37:45+00:00 INF All 0 tables opened in 0s component=badger.DB
2024-02-20T21:37:45+00:00 INF Discard stats nextEmptySlot: 0 component=badger.DB
2024-02-20T21:37:45+00:00 INF Set nextTxnTs to 0 component=badger.DB
2024-02-20T21:37:45+00:00 INF builtin plugins initialized component=builtin.Registry count=6
2024-02-20T21:37:45+00:00 WRN could not read plugin directory error="open /app/connectors: no such file or directory" component=standalone.Registry
2024-02-20T21:37:45+00:00 INF standalone plugins initialized component=standalone.Registry count=0 plugin_path=/app/connectors
2024-02-20T21:37:45+00:00 INF processors initialized component=processor.Service count=0
2024-02-20T21:37:45+00:00 INF connectors initialized component=connector.Service count=0
2024-02-20T21:37:45+00:00 INF pipelines initialized component=pipeline.Service count=0
2024-02-20T21:37:45+00:00 INF pipeline configs provisioned component=provisioning.Service created=[] deleted=[] pipelines_path=./pipelines
2024-02-20T21:37:45+00:00 INF grpc server started address=[::]:8084
2024-02-20T21:37:45+00:00 INF http server started address=[::]:8080
2024-02-20T21:37:45+00:00 INF
2024-02-20T21:37:45+00:00 INF click here to navigate to Conduit UI: http://localhost:8080/ui
2024-02-20T21:37:45+00:00 INF click here to navigate to explore the HTTP API: http://localhost:8080/openapi
2024-02-20T21:37:45+00:00 INF
```

You should now be able to interact with the Conduit UI and HTTP API on port 8080.

## Building from source

Before you can build Conduit from source, you need to have Go 1.21 or higher installed. The Go website has [comprehensive instructions](https://go.dev/doc/install) on how to get the language installed on your machine. Once installed, you can follow the rest of the instructions.

1. Start by downloading the source code from the latest stable release on the Conduit [Releases Page](https://github.com/ConduitIO/conduit/releases/latest). Alternatively, you can run this command to automatically download the latest stable source to your current directory:

```shell
$ TAG=$(curl -s https://api.github.com/repos/ConduitIO/conduit/releases/latest | grep "tag_name\": \"v[0-9.]*" | awk -F \" '/tag_name\": \"*\"/{print $(NF-1)}') && curl -o conduit.tgz -L https://github.com/ConduitIO/conduit/tarball/$TAG
```

A file called `conduit.tgz` will be in your current directory. The next step is to expand the source:

```shell
$ tar zxvf conduit.tgz
```

Then change directories to the appropriate folder. Keep in mind that the folder name might be different between releases since it's tied to the latest git sha for the commit.

```shell
$ cd ConduitIO-
```

Now build the project:

```shell
$ make
```

If you want to build conduit without the UI, instead of running `make`, you can run the following command:

```shell
$ make build-server
```

You will have a new binary built for your architecture and machine. All that's left is to run it!

```shell
./conduit
```

## Installing Conduit with Homebrew

On MacOS we provide the option of installing Conduit via Homebrew

```
brew install conduit
```

## Running Conduit with Docker

We distribute [Docker](https://docs.docker.com/) images to ease the use of Conduit in a containerized environment. To see details about how our images are built please feel free to take a look at the `Dockerfile` [here](https://github.com/ConduitIO/conduit/blob/main/Dockerfile).
Conduit's Docker images are hosted on [GitHub](https://github.com/ConduitIO/conduit/pkgs/container/conduit).

After you install Docker, you can pull the Conduit image by running the following command:

```shell
docker pull conduit.docker.scarf.sh/conduitio/conduit
```
Then, you can run Conduit by running:

```shell
docker run -it -p 8080:8080 conduit.docker.scarf.sh/conduitio/conduit
```

The command aboves runs Conduit with no pipelines. To run Conduit with a custom pipeline file you can mount it as a volume into the container:
```shell
docker run -it -p 8080:8080 \
-v /path/to/pipeline.yaml:/app/pipelines/pipeline.yaml \
conduit.docker.scarf.sh/conduitio/conduit
```

**Helpful Tip:** If you're using Docker, check out [Watchtower](https://containrrr.dev/watchtower/). After we push an update, Watchtower will keep your Conduit container updated by gracefully shutting down your existing container and restarting it with the same options used when initially deployed.

## Next Steps

Now that you have Conduit installed it's time to explore some other topics.

- [Pipelines](/docs/pipeline-configuration-files/getting-started)
- [Connectors](/docs/connectors/getting-started)
File renamed without changes.
47 changes: 47 additions & 0 deletions docs/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ sidebar_label: "Introduction"
slug: /
---

# TODO: 🐝 Consolidate

<img
alt="Conduit Logo"
style={{ maxWidth: "400px", marginTop: 0 }}
Expand Down Expand Up @@ -249,3 +251,48 @@ Now that you've got the basics of running Conduit and creating a pipeline covere
- [Pipelines](/docs/pipeline-configuration-files/getting-started)
- [Processors](/docs/processors/getting-started)
- [Conduit Architecture](/docs/introduction/architecture)

-----



Conduit is a new open-source project, written in [Go](https://go.dev/), that
delivers real-time data integration for developers. It aims to provide the best
user experience for building and running real-time data pipelines.

Out of the box, Conduit comes with:

- A UI
- Common connectors
- Processors
- Observability

The batteries are included 🔋

### It’s simple, yet powerful

Eliminate the multi-step process you go through today. Just download the binary
and start building. Conduit pipelines run in their own goroutines and are
connected using Go channels. This makes Conduit pipelines incredibly performant
on multi-core machines.

### It’s real-time

Conduit pipelines listen for changes to a database, data warehouse, etc., and
allows your data applications to act upon those changes in real-time.

### It’s extensible

Conduit connectors are plugins that communicate with Conduit via a gRPC
interface. This means that plugins can be written in any language as long as
they conform to the required interface. Check out
our [connector docs](/docs/connectors/installing)!

## Get involved

Here are your next steps:

- [Join the Discord Community](https://discord.meroxa.com/)
- [Request features/ ask questions about Conduit in GitHub Discussions](https://github.com/ConduitIO/conduit/discussions)
- [Send bug reports to GitHub Issues](https://github.com/ConduitIO/conduit/issues)
- [Show us love on Twitter](https://twitter.com/ConduitIO)
5 changes: 0 additions & 5 deletions docs/introduction/_category_.json

This file was deleted.

Loading

0 comments on commit 8abff50

Please sign in to comment.