Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: geospatial types support #4208

Open
wants to merge 106 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
f3784ac
feat: geospatial types support
Oreilles Nov 23, 2023
e605279
chore: add postgis database to query engine test matrix
Oreilles Sep 8, 2023
0a750da
fix: mysql geometries should be serialized to wkb by quaint
Oreilles Sep 8, 2023
9844033
fix: failing inequality operator with PostGIS
Oreilles Sep 8, 2023
b267f61
fix: PostGIS tests need to happen with the public schema exposed
Oreilles Sep 8, 2023
2aaf40b
fix: add missing GeometryFiltering capability constraint to geometric…
Oreilles Sep 8, 2023
b9f411f
fix: srid introspection in postgres dialect
Oreilles Sep 9, 2023
f42f3ba
fix: handle geography default srid value inconsistency in tests
Oreilles Sep 9, 2023
c44ee7a
fix: remove unused variable
Oreilles Sep 9, 2023
c16fcd7
fix: geography srid introspection test in postgres dialect
Oreilles Sep 9, 2023
6f2db0f
fix: actually fix geography srid introspection test in postgres dialect
Oreilles Sep 9, 2023
a5097de
fix: simplify mysql srid introspection on mysql and fixes vitess tests
Oreilles Sep 9, 2023
98c913f
fix: add missing MySQL version constraint for test requiring support …
Oreilles Nov 12, 2023
e287c38
fix: ignore unsuccessful initialization of SpatiaLite in SQLite query…
Oreilles Nov 12, 2023
4e4308c
fix: replace intersection test due to MySQL 5.6 quirk
Oreilles Nov 13, 2023
16f004c
fix: formatting
Oreilles Nov 19, 2023
71dae2e
fix: make Spatialite geometry filters stricter and tighten geometry f…
Oreilles Nov 19, 2023
93f582c
fix: add SQLite test versions 3 and 3-spatialite, only run query engi…
Oreilles Nov 25, 2023
6cd70be
chore: add Spatialite tests to CI
Oreilles Nov 24, 2023
e6185be
fix: incorrect dependency order
Oreilles Nov 26, 2023
42e5914
fix: add missing Spatialite system table to the list
Oreilles Nov 26, 2023
e1da2f7
fix: null_geometry return value
Oreilles Nov 26, 2023
5a255ac
refactor: share SQLITE_SYSTEM_TABLES with sql-schema-connector
Oreilles Nov 28, 2023
1290ef7
refactor: execute InitSpatialMetaData in quaint
Oreilles Nov 28, 2023
bb7e33c
fix: Spatialite diff migration tests
Oreilles Nov 28, 2023
455b274
fix: formatting
Oreilles Nov 28, 2023
35a9f15
fix: revert Sqlite total queries to 9
Oreilles Nov 28, 2023
7b8348c
fix: install Spatialite with sudo
Oreilles Nov 28, 2023
31228b5
fix: add WGS84_ONLY option to `InitSpatialMetadata` and only call the…
Oreilles Nov 29, 2023
8a5288b
fix: formatting
Oreilles Nov 29, 2023
e0f834d
fix: skip geometric_comparison_filters test for MariaDB while we figu…
Oreilles Nov 30, 2023
f7f2892
refactor: return conversion error in GeometryValue::from_str
Oreilles Nov 30, 2023
f3fa6ef
Merge remote-tracking branch 'prisma/main' into feat/geometry
Oreilles Aug 4, 2024
858653f
refactor: simplify spatialite CI setup
Oreilles Aug 4, 2024
9feff00
Merge remote-tracking branch 'prisma/main' into feat/geometry
Oreilles Sep 25, 2024
f7f25c9
Fix toolchain, features and lint
Oreilles Sep 25, 2024
0a879b4
Add JSArgType::Geometry
Oreilles Sep 30, 2024
e8658e9
Fix: fmt
Oreilles Sep 30, 2024
f74d4de
Fix test_native_types_multifile
Oreilles Sep 30, 2024
33048e9
Update wasm toolchain
Oreilles Sep 30, 2024
56644f3
Fix PostGIS version test
Oreilles Sep 30, 2024
a3db88e
Add Geometry handling in quaint from_type_identifier
Oreilles Sep 30, 2024
a16d952
Use SRID 4326 in Sqlite introspection tests
Oreilles Sep 30, 2024
ff3e9aa
Update test unknown_type_mysql
Oreilles Sep 30, 2024
9f205ed
Add GeoJson handling in MongoDB into_bson
Oreilles Sep 30, 2024
7d6d960
Fix geometry_type_mysql
Oreilles Sep 30, 2024
2b35a94
Use SRID 4326 in spatialite describer tests
Oreilles Oct 1, 2024
fffdacc
Add cfd1 and libsql.js.wasm to spatialite tests exclude list
Oreilles Oct 1, 2024
0c912c0
Add cfd1 and libsql.js.wasm to other spatialite tests exclude list
Oreilles Oct 1, 2024
906f137
Change GeoJson default SRID to 0 for PostGIS/Spatialite
Oreilles Oct 2, 2024
2a288d0
Update tests
Oreilles Oct 2, 2024
0b98b7f
Drop extended types support because of inconsistencies between (and w…
Oreilles Oct 2, 2024
b5d96de
Minor changes
Oreilles Oct 2, 2024
f680b3d
Fix WKT to GeoJSON step
Oreilles Oct 3, 2024
53347bb
Drop support for M coordinates and make GeoJSON the only geometry IO …
Oreilles Oct 3, 2024
13d8624
Update sql-introspection-tests
Oreilles Oct 3, 2024
73f051a
Fix srid handling for sql server
Oreilles Oct 3, 2024
7c834a7
Fix spatialite introspection test
Oreilles Oct 3, 2024
41b829c
Fix spatialite migration tests
Oreilles Oct 3, 2024
c08999d
Fix more tests
Oreilles Oct 3, 2024
c7a5ed3
Fix quaint geometry as raw
Oreilles Oct 3, 2024
128c12d
Fix spatialite schema describer tests
Oreilles Oct 3, 2024
280a3d3
Revert CI spatialite setup
Oreilles Oct 3, 2024
0cdaaa2
Fix lint errors
Oreilles Oct 3, 2024
ea9e1e7
Revert CI spatialite setup
Oreilles Oct 3, 2024
fbc748e
Defer GeoJSON conversion to quaint, using is_selected column property
Oreilles Oct 5, 2024
984f9dc
Skip geometry tests on MySQL(5.6). and fix tests
Oreilles Oct 5, 2024
6bb32fe
Fix more tests
Oreilles Oct 5, 2024
6a95e06
Fix more tests
Oreilles Oct 5, 2024
424671a
Fix fmt
Oreilles Oct 5, 2024
7b24702
Refactor geometry handling in quaint visitors
Oreilles Oct 6, 2024
eb86df4
Fix SQL Server geometry test
Oreilles Oct 6, 2024
0341bdb
Merge remote-tracking branch 'prisma/main' into feat/geometry
Oreilles Oct 6, 2024
e15eaa5
Run query engine tests on ubuntu latest
Oreilles Oct 6, 2024
963f1a9
Validate GeoJSON in mongodb connector
Oreilles Oct 7, 2024
c9479dd
Remove GEoOJSON from query engine custom_types
Oreilles Oct 7, 2024
8b15e06
Remove Geometry connector capabilities to allow stable build
Oreilles Oct 7, 2024
a597901
Fix mongodb tests
Oreilles Oct 7, 2024
302537f
Update geozero feature flags
Oreilles Oct 7, 2024
b3e6d22
Move geojson and geozero dependencies to workspace
Oreilles Oct 7, 2024
87fc365
Err instead of panic in sqlite schema connector creation
Oreilles Oct 7, 2024
8ed9ffd
Replace deprecated WktStr
Oreilles Oct 7, 2024
77f8b0a
Fix Android build
Oreilles Oct 7, 2024
33836bc
Update shema-engine CI runner
Oreilles Oct 7, 2024
2a66608
Try ubuntu latest for mssql query engine tests CI
Oreilles Oct 7, 2024
5060a30
Revert schema engine tests CI runner
Oreilles Oct 7, 2024
5c10491
Disable spatialite schema engine tests for now as Spatialite >=5.0.0 …
Oreilles Oct 7, 2024
da0f8e6
Slight quaint refactor
Oreilles Oct 7, 2024
bf506e9
Revert query-engine tests CI runner update and disable spatialite tests
Oreilles Oct 7, 2024
362a665
CI workflow yaml indentation
Oreilles Oct 7, 2024
ee0a985
No idea why QE tests are failing now
Oreilles Oct 7, 2024
4210df8
CI now fails for mssql-2017 as it is the only job running on 20.04...…
Oreilles Oct 7, 2024
044ec3d
Simplify SRID handling in quaint by making 4326 the default everywhere
Oreilles Oct 8, 2024
205e672
Make SRID mandatory in db geometry type parameters
Oreilles Oct 8, 2024
a957dcc
Make db geometry type params mandatory for SQLite
Oreilles Oct 8, 2024
f103c02
Fix handling of raw geometry value in quaint
Oreilles Oct 8, 2024
477bae5
Revert changes to CI runners
Oreilles Oct 8, 2024
2ddce28
Fix cockroachdb test
Oreilles Oct 8, 2024
943c80d
Fix postgres tests
Oreilles Oct 9, 2024
7427329
Remove PrismaValue::GeoJson
Oreilles Oct 9, 2024
c7d650d
Rename input/output type functions geojson to geometry
Oreilles Oct 10, 2024
9f0203c
Fix geometry / json value / prisma value conversions
Oreilles Oct 10, 2024
ca2e009
Add JSON protocol geometry input test
Oreilles Oct 14, 2024
bdcc984
Add TODOs
Oreilles Oct 14, 2024
989b876
Validate geometries on JSON protocol
Oreilles Oct 14, 2024
e813e61
Lint
Oreilles Oct 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/test-query-engine-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ jobs:
- name: "Start ${{ inputs.name }} (${{ matrix.engine_protocol }})"
run: make start-${{ inputs.name }}

- name: Install Spatialite
if: ${{ inputs.name == 'spatialite' }}
run: sudo apt install -y libsqlite3-mod-spatialite && echo "SPATIALITE_PATH=mod_spatialite" >> $GITHUB_ENV

- run: export WORKSPACE_ROOT=$(pwd) && cargo nextest run -p query-engine-tests --partition hash:${{ matrix.partition }} --test-threads=1
if: ${{ inputs.single_threaded }}
env:
Expand Down
17 changes: 13 additions & 4 deletions .github/workflows/test-query-engine.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ jobs:
version: "10"
- name: "postgres9"
version: "9"
- name: 'postgis16'
version: '16-postgis'
uses: ./.github/workflows/test-query-engine-template.yml
name: postgres ${{ matrix.database.version }}
with:
Expand Down Expand Up @@ -81,7 +83,6 @@ jobs:
matrix:
database:
- name: "cockroach_23_1"
connector: "cockroachdb"
version: "23.1"
- name: "cockroach_22_2"
version: "22.2"
Expand Down Expand Up @@ -137,11 +138,19 @@ jobs:
relation_load_strategy: '["query"]'

sqlite:
strategy:
fail-fast: false
matrix:
database:
- name: "sqlite"
version: "3"
# - name: "spatialite"
# version: "3-spatialite"
uses: ./.github/workflows/test-query-engine-template.yml
name: sqlite
name: sqlite ${{ matrix.database.version }}
with:
name: "sqlite"
version: 3
name: ${{ matrix.database.name }}
version: ${{ matrix.database.version }}
connector: "sqlite"
relation_load_strategy: '["query"]'

Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/test-schema-engine.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ jobs:
url: "postgresql://postgres:prisma@localhost:5438"
- name: postgres16
url: "postgresql://postgres:prisma@localhost:5439"
- name: postgis16
url: 'postgresql://postgres:prisma@localhost:5440'
- name: cockroach_23_1
url: "postgresql://prisma@localhost:26260"
- name: cockroach_22_2
Expand All @@ -99,6 +101,8 @@ jobs:
url: "postgresql://prisma@localhost:26257"
- name: sqlite
url: sqlite
# - name: spatialite
# url: sqlite
- name: vitess_8_0
url: "mysql://root:prisma@localhost:33807/test"
shadow_database_url: "mysql://root:prisma@localhost:33808/shadow"
Expand All @@ -124,6 +128,10 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Install Spatialite
if: ${{ matrix.database.name == 'spatialite' }}
run: sudo apt install -y libsqlite3-mod-spatialite && echo "SPATIALITE_PATH=mod_spatialite" >> $GITHUB_ENV

- name: "Start ${{ matrix.database.name }}"
run: make start-${{ matrix.database.name }}

Expand Down
1 change: 1 addition & 0 deletions .test_database_urls/postgis_16
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export TEST_DATABASE_URL="postgresql://postgres:prisma@localhost:5440"
77 changes: 74 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ url = { version = "2.5.0" }
bson = { version = "2.11.0", features = ["chrono-0_4", "uuid-1"] }
mongodb = { git = "https://github.com/prisma/mongo-rust-driver.git", branch = "RUST-1994/happy-eyeballs" }

geojson = { version = "0.24.1", default-features = false }
geozero = "0.14.0"

[workspace.dependencies.quaint]
path = "quaint"

Expand Down
11 changes: 11 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,11 @@ start-sqlite:
dev-sqlite:
cp $(CONFIG_PATH)/sqlite $(CONFIG_FILE)

start-spatialite:

dev-spatialite:
cp $(CONFIG_PATH)/spatialite $(CONFIG_FILE)

dev-react-native:
cp $(CONFIG_PATH)/react-native $(CONFIG_FILE)

Expand Down Expand Up @@ -248,6 +253,12 @@ start-postgres16:
dev-postgres16: start-postgres16
cp $(CONFIG_PATH)/postgres16 $(CONFIG_FILE)

start-postgis16:
docker compose -f docker-compose.yml up -d --remove-orphans postgis16

dev-postgis16: start-postgis16
cp $(CONFIG_PATH)/postgis16 $(CONFIG_FILE)

start-cockroach_23_1:
docker compose -f docker-compose.yml up --wait -d --remove-orphans cockroach_23_1

Expand Down
13 changes: 13 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,19 @@ services:
networks:
- databases

postgis16:
image: postgis/postgis:16-3.4
restart: unless-stopped
command: postgres -c 'max_connections=1000'
environment:
POSTGRES_PASSWORD: "prisma"
POSTGRES_HOST_AUTH_METHOD: "md5"
POSTGRES_INITDB_ARGS: "--auth-host=md5"
ports:
- "5440:5432"
networks:
- databases

mysql-5-6:
image: mysql:5.6.50
command: mysqld --table_definition_cache=2000
Expand Down
9 changes: 7 additions & 2 deletions libs/test-setup/src/postgres.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,14 @@ pub(crate) fn get_postgres_tags(database_url: &str) -> Result<BitFlags<Tags>, St
let fut = async {
let quaint = Quaint::new(database_url).await.map_err(|err| err.to_string())?;
let mut tags = Tags::Postgres.into();
let version = quaint.version().await.map_err(|err| err.to_string())?;

match version {
if let Ok(_postgis_version) = quaint.query_raw("SELECT PostGIS_version()", &[]).await {
tags |= Tags::PostGIS;
}

let postgres_version = quaint.version().await.map_err(|err| err.to_string())?;

match postgres_version {
None => Ok(tags),
Some(version) => {
eprintln!("version: {version:?}");
Expand Down
18 changes: 18 additions & 0 deletions libs/test-setup/src/sqlite.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
use enumflags2::BitFlags;
use once_cell::sync::Lazy;
use quaint::{prelude::Queryable, single::Quaint};

use crate::{runtime::run_with_thread_local_runtime as tok, Tags};

pub fn sqlite_test_url(db_name: &str) -> String {
std::env::var("SQLITE_TEST_URL").unwrap_or_else(|_| format!("file:{}", sqlite_test_file(db_name)))
Expand All @@ -25,3 +29,17 @@ fn sqlite_test_file(db_name: &str) -> String {

file_path.to_string_lossy().into_owned()
}

pub(crate) fn get_sqlite_tags() -> Result<BitFlags<Tags>, String> {
let fut = async {
let mut tags: BitFlags<Tags> = Tags::Sqlite.into();
// The SpatiaLite extension is loaded by quaint, assuming the SPATIALITE_PATH env variable is set
// If the extension can be loaded in a dummy database, it means it will also be available for the tests
let quaint = Quaint::new_in_memory().map_err(|err| err.to_string())?;
if let Ok(_has_spatialite) = quaint.query_raw("SELECT spatialite_version();", &[]).await {
tags |= Tags::Spatialite;
}
Ok(tags)
};
tok(fut)
}
2 changes: 2 additions & 0 deletions libs/test-setup/src/tags.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ tags![
CockroachDb222 = 1 << 20,
CockroachDb231 = 1 << 21,
Postgres16 = 1 << 22,
PostGIS = 1 << 23,
Spatialite = 1 << 24,
];

pub fn tags_from_comma_separated_list(input: &str) -> BitFlags<Tags> {
Expand Down
4 changes: 2 additions & 2 deletions libs/test-setup/src/test_api_args.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{logging, mssql, mysql, postgres, Capabilities, Tags};
use crate::{logging, mssql, mysql, postgres, sqlite, Capabilities, Tags};
use enumflags2::BitFlags;
use once_cell::sync::Lazy;
use quaint::single::Quaint;
Expand Down Expand Up @@ -41,7 +41,7 @@ static DB_UNDER_TEST: Lazy<Result<DbUnderTest, String>> = Lazy::new(|| {
match prefix {
"file" | "sqlite" => Ok(DbUnderTest {
database_url,
tags: Tags::Sqlite.into(),
tags: sqlite::get_sqlite_tags()?,
capabilities: Capabilities::CreateDatabase.into(),
provider: "sqlite",
shadow_database_url,
Expand Down
1 change: 1 addition & 0 deletions libs/user-facing-errors/src/schema_engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ pub struct DestructiveMigrationDetected {
code = "P3002",
message = "The attempted migration was rolled back: {database_error}"
)]
#[allow(dead_code)]
struct MigrationRollback {
pub database_error: String,
}
Expand Down
1 change: 1 addition & 0 deletions prisma-fmt/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ pub fn validate(validate_params: String) -> Result<(), String> {
}

/// Given a list of Prisma schema files (and their locations), returns the merged schema.
///
/// This is useful for `@prisma/client` generation, where the client needs a single - potentially large - schema,
/// while still allowing the user to split their schema copies into multiple files.
/// Internally, it uses `[validate]`.
Expand Down
Loading