-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* package.json * wip * wip * wip * wip * wip * fix * wip * wip * fix * wip * wip * wip * done * build * fixes * emails
- Loading branch information
1 parent
eb1e724
commit 1cbc5a8
Showing
69 changed files
with
1,752 additions
and
39 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,3 +5,5 @@ coverage | |
duckdb | ||
duckdb.tar.gz | ||
prebuilds | ||
temp | ||
etc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,37 +1,97 @@ | ||
# Node-DuckDB | ||
|
||
###### [API](https://github.com/deepcrawl/node-duckdb/docs/api/node-duckdb.md) | [Code Of Conduct](https://github.com/deepcrawl/node-duckdb/docs/CODE_OF_CONDUCT.md) | [Contributing](https://github.com/deepcrawl/node-duckdb/docs/CONTRIBUTING.md) | [Developing](https://github.com/deepcrawl/node-duckdb/docs/DEVELOPING.md) | ||
|
||
> Production ready DuckDB Node.js library written in TypeScript. | ||
> [<img src="https://www.deepcrawl.com/wp-content/themes/deepcrawl/images/deepcrawl-logo.svg" height="200" width="300" align="right">](https://www.deepcrawl.com/) | ||
## Overview | ||
|
||
- This is a library that adds support for [DuckDB](https://duckdb.org/) to NodeJS. | ||
- It comes preinstalled with DuckDB ver 0.2.2 with the parquet extension included. | ||
- It comes preinstalled with DuckDB ver 0.2.2 with the **parquet** extension included. | ||
- Has been tested to work with Linux and MacOS. | ||
- Currently supports NodeJS v12.17.0+. | ||
- Supports BIGINT and HUGEINT types as [BigInt](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt). | ||
- Provides an async API and a streaming one. | ||
- Provides a **Promise**-based API and a **Stream**-based one. | ||
|
||
## Using | ||
## Getting Started | ||
|
||
[See examples](https://github.com/deepcrawl/node-duckdb/tree/master/examples) | ||
### Installation | ||
|
||
## Developing | ||
To use Node-DuckDB in your project: | ||
|
||
``` | ||
npm i node-duckdb | ||
``` | ||
|
||
or | ||
|
||
``` | ||
yarn add node-duckdb | ||
``` | ||
|
||
Note: this will download the duckdb binary for your platform (currently Linux and MacOS are supported), or if it's not available will attempt to build it. | ||
|
||
### Usage | ||
|
||
Using node-duckdb is easy: | ||
|
||
```ts | ||
const db = new DuckDB(); | ||
const connection = new Connection(db); | ||
await connection.execute("SELECT * FROM mytable;"); | ||
``` | ||
|
||
#### Promise API example | ||
|
||
First build: | ||
An example using promises: | ||
|
||
1. `yarn install` - installs dependencies including downloading duckdb | ||
2. `yarn build:ts` - builds typescript | ||
3. `yarn test` - runs all tests | ||
```ts | ||
import { Connection, DuckDB } from "node-duckdb"; | ||
|
||
Other useful scripts: | ||
async function queryDatabaseWithIterator() { | ||
// create new database in memory | ||
const db = new DuckDB(); | ||
// create a new connection to the database | ||
const connection = new Connection(db); | ||
|
||
- `yarn build` - build everything | ||
- `yarn build:addon` - build just the bindings code | ||
- `yarn build:duckdb` - build just the duckdb database | ||
- `yarn build:ts` - build just the typescript code | ||
- `yarn lint` - lint the project | ||
- `yarn test` - run all tests | ||
- `yarn test csv` - run just the csv test suite | ||
// perform some queries | ||
await connection.executeIterator("CREATE TABLE people(id INTEGER, name VARCHAR);"); | ||
await connection.executeIterator("INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes'), (3, 'Bob');"); | ||
const result = await connection.executeIterator("SELECT * FROM people;"); | ||
|
||
## Publishing | ||
// fetch and print result | ||
console.log(result.fetchAllRows()); | ||
|
||
// release resources | ||
connection.close(); | ||
db.close(); | ||
} | ||
|
||
queryDatabaseWithIterator(); | ||
``` | ||
|
||
#### Streaming API example | ||
|
||
Getting a stream of data from DuckDB and piping into a destination stream: | ||
|
||
```ts | ||
import { Connection, DuckDB } from "node-duckdb"; | ||
const db = new DuckDB(); | ||
const connection = new Connection(db); | ||
const resultStream = await connection.execute("SELECT * FROM people;"); | ||
// get destinationStream somehow | ||
resultStream.pipe(destinationStream); | ||
``` | ||
|
||
#### Complete sample project | ||
|
||
You can see a complete sample project using node-duckdb [here](https://github.com/deepcrawl/node-duckdb/tree/master/examples). | ||
|
||
## API | ||
|
||
API documentation is found [here](https://github.com/deepcrawl/node-duckdb/docs/api/node-duckdb.md). | ||
|
||
## Developing | ||
|
||
- `export GITHUB_TOKEN=<your PAT>` - create a PAT in github that allows uploading artifacts to github releases | ||
- `yarn login && yarn publish` - publish will do a bunch of various stuff, including prebuilding binaries for linux/mac and publishing those | ||
Documentation for developers is found [here](https://github.com/deepcrawl/node-duckdb/docs/DEVELOPING.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
{ | ||
"$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json", | ||
"mainEntryPointFilePath": "<projectFolder>/dist/index.d.ts", | ||
"bundledPackages": [], | ||
"compiler": {}, | ||
"apiReport": { | ||
"enabled": true | ||
}, | ||
"docModel": { | ||
"enabled": true | ||
}, | ||
"dtsRollup": { | ||
"enabled": true | ||
}, | ||
"tsdocMetadata": {}, | ||
"messages": { | ||
"compilerMessageReporting": { | ||
"default": { | ||
"logLevel": "error" | ||
} | ||
}, | ||
"extractorMessageReporting": { | ||
"default": { | ||
"logLevel": "error" | ||
} | ||
}, | ||
"tsdocMessageReporting": { | ||
"default": { | ||
"logLevel": "error" | ||
} | ||
} | ||
} | ||
} |
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
## Developing | ||
|
||
First build: | ||
|
||
1. `yarn install` - installs dependencies including downloading duckdb | ||
2. `yarn build:ts` - builds typescript | ||
3. `yarn test` - runs all tests | ||
|
||
Other useful scripts: | ||
|
||
- `yarn build` - build everything | ||
- `yarn build:addon` - build just the bindings code | ||
- `yarn build:duckdb` - build just the duckdb database | ||
- `yarn build:ts` - build just the typescript code | ||
- `yarn lint` - lint the project | ||
- `yarn test` - run all tests | ||
- `yarn test csv` - run just the csv test suite | ||
|
||
## Publishing | ||
|
||
- `export GITHUB_TOKEN=<your PAT>` - create a PAT in github that allows uploading artifacts to github releases | ||
- `yarn login && yarn publish` - publish will do a bunch of various stuff, including prebuilding binaries for linux/mac and publishing those |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Home](./index.md) | ||
|
||
## API Reference | ||
|
||
## Packages | ||
|
||
| Package | Description | | ||
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| [node-duckdb](./node-duckdb.md) | Node-DuckDB is a thin wrapper on top of [DuckDB](https://duckdb.org/)<!-- -->.<!-- -->Using it involves:<!-- -->1. Creating a [DuckDB](./node-duckdb.duckdb.md) object<!-- -->2. Creating a [Connection](./node-duckdb.connection.md) object to the DuckDB object<!-- -->3. Calling [Connection.execute](./node-duckdb.connection.execute.md) or [Connection.executeIterator](./node-duckdb.connection.executeiterator.md) on the Connection object | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Node-DuckDB API](./node-duckdb.md) > [AccessMode](./node-duckdb.accessmode.md) | ||
|
||
## AccessMode enum | ||
|
||
Access mode specifier | ||
|
||
<b>Signature:</b> | ||
|
||
```typescript | ||
export declare enum AccessMode | ||
``` | ||
|
||
## Enumeration Members | ||
|
||
| Member | Value | Description | | ||
| --------- | -------------- | ----------- | | ||
| Automatic | <code>1</code> | | | ||
| ReadOnly | <code>2</code> | | | ||
| ReadWrite | <code>3</code> | | | ||
| Undefined | <code>0</code> | | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Node-DuckDB API](./node-duckdb.md) > [Connection](./node-duckdb.connection.md) > [(constructor)](./node-duckdb.connection._constructor_.md) | ||
|
||
## Connection.(constructor) | ||
|
||
Connection constructor. | ||
|
||
<b>Signature:</b> | ||
|
||
```typescript | ||
constructor(duckdb: DuckDB); | ||
``` | ||
|
||
## Parameters | ||
|
||
| Parameter | Type | Description | | ||
| --------- | --------------------------------- | --------------------------------------------------------- | | ||
| duckdb | [DuckDB](./node-duckdb.duckdb.md) | [DuckDB](./node-duckdb.duckdb.md) instance to connect to. | | ||
|
||
## Example | ||
|
||
Initializing a connection: | ||
|
||
``` | ||
import { DuckDB } from "node-duckdb"; | ||
const db = new DuckDB(); | ||
const connection = new Connection(db); | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Node-DuckDB API](./node-duckdb.md) > [Connection](./node-duckdb.connection.md) > [close](./node-duckdb.connection.close.md) | ||
|
||
## Connection.close() method | ||
|
||
Close the connection (also closes all [ResultStream](./node-duckdb.resultstream.md) or [ResultIterator](./node-duckdb.resultiterator.md) objects associated with this connection). | ||
|
||
<b>Signature:</b> | ||
|
||
```typescript | ||
close(): void; | ||
``` | ||
|
||
<b>Returns:</b> | ||
|
||
void | ||
|
||
## Remarks | ||
|
||
Even though GC will automatically destroy the Connection object at some point, DuckDB data is stored in the native address space, not the V8 heap, meaning you can easily have a Node.js process taking gigabytes of memory (more than the default heap size for Node.js) with V8 not triggering GC. So, definitely think about manually calling `close()`<!-- -->. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Node-DuckDB API](./node-duckdb.md) > [Connection](./node-duckdb.connection.md) > [execute](./node-duckdb.connection.execute.md) | ||
|
||
## Connection.execute() method | ||
|
||
Asynchronously executes the query and returns a node.js stream that wraps the result set. | ||
|
||
<b>Signature:</b> | ||
|
||
```typescript | ||
execute<T>(command: string, options?: IExecuteOptions): Promise<ResultStream<T>>; | ||
``` | ||
|
||
## Parameters | ||
|
||
| Parameter | Type | Description | | ||
| --------- | --------------------------------------------------- | ----------------------------------------------------------------------------------- | | ||
| command | string | SQL command to execute | | ||
| options | [IExecuteOptions](./node-duckdb.iexecuteoptions.md) | optional options object of type [IExecuteOptions](./node-duckdb.iexecuteoptions.md) | | ||
|
||
<b>Returns:</b> | ||
|
||
Promise<[ResultStream](./node-duckdb.resultstream.md)<!-- --><T>> | ||
|
||
## Example | ||
|
||
Streaming results of a DuckDB query into a CSV file: | ||
|
||
``` | ||
import { Connection, DuckDB, RowResultFormat } from "node-duckdb"; | ||
import { createWriteStream } from "fs"; | ||
import { Transform } from "stream"; | ||
class ArrayToCsvTransform extends Transform { | ||
constructor() { | ||
super({objectMode: true}) | ||
} | ||
_transform(chunk: any[], _encoding: string, callback: any) { | ||
this.push(chunk.join(",") + '\n'); | ||
callback(); | ||
} | ||
} | ||
async function outputToFileAsCsv() { | ||
const db = new DuckDB(); | ||
const connection = new Connection(db); | ||
await connection.execute("CREATE TABLE people(id INTEGER, name VARCHAR);"); | ||
await connection.execute("INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes'), (3, 'Bob');"); | ||
const resultStream = await connection.execute("SELECT * FROM people;", {rowResultFormat: RowResultFormat.Array}); | ||
const transformToCsvStream = new ArrayToCsvTransform(); | ||
const writeStream = createWriteStream("my-people-output"); | ||
resultStream.pipe(transformToCsvStream).pipe(writeStream); | ||
} | ||
outputToFileAsCsv(); | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
<!-- Do not edit this file. It is automatically generated by API Documenter. --> | ||
|
||
[Node-DuckDB API](./node-duckdb.md) > [Connection](./node-duckdb.connection.md) > [executeIterator](./node-duckdb.connection.executeiterator.md) | ||
|
||
## Connection.executeIterator() method | ||
|
||
Asynchronously executes the query and returns an iterator that points to the first result in the result set. | ||
|
||
<b>Signature:</b> | ||
|
||
```typescript | ||
executeIterator<T>(command: string, options?: IExecuteOptions): Promise<ResultIterator<T>>; | ||
``` | ||
|
||
## Parameters | ||
|
||
| Parameter | Type | Description | | ||
| --------- | --------------------------------------------------- | ----------------------------------------------------------------------------------- | | ||
| command | string | SQL command to execute | | ||
| options | [IExecuteOptions](./node-duckdb.iexecuteoptions.md) | optional options object of type [IExecuteOptions](./node-duckdb.iexecuteoptions.md) | | ||
|
||
<b>Returns:</b> | ||
|
||
Promise<[ResultIterator](./node-duckdb.resultiterator.md)<!-- --><T>> | ||
|
||
## Example 1 | ||
|
||
Printing rows: | ||
|
||
``` | ||
import { Connection, DuckDB, RowResultFormat } from "node-duckdb"; | ||
async function queryDatabaseWithIterator() { | ||
const db = new DuckDB(); | ||
const connection = new Connection(db); | ||
await connection.executeIterator("CREATE TABLE people(id INTEGER, name VARCHAR);"); | ||
await connection.executeIterator("INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes'), (3, 'Bob');"); | ||
const result = await connection.executeIterator("SELECT * FROM people;"); | ||
// print the first row | ||
console.log(result.fetchRow()); | ||
// print the rest of the rows | ||
console.log(result.fetchAllRows()); | ||
const result2 = await connection.executeIterator("SELECT * FROM people;", {rowResultFormat: RowResultFormat.Array}); | ||
console.log(result2.fetchAllRows()); | ||
connection.close(); | ||
db.close(); | ||
} | ||
queryDatabaseWithIterator(); | ||
``` | ||
|
||
## Example 2 | ||
|
||
Providing generics type: | ||
|
||
``` | ||
const result = await connection.executeIterator<number[]>(`SELECT CAST(1 AS TINYINT)`, { | ||
rowResultFormat: RowResultFormat.Array, | ||
}); | ||
expect(result.fetchRow()).toMatchObject([1]); | ||
``` |
Oops, something went wrong.