Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: ecosystem and resources #1468

Merged
merged 1 commit into from
Nov 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 43 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,28 +37,21 @@ Get started!
- [Join our community call](https://calendar.google.com/calendar/embed?src=27ff6dc5f598c1d94c1f6e627a1aaae680e2fac88f848bda1f2c7946ae74d5ab%40group.calendar.google.com)
- [Read the contributing guide](https://github.com/narwhals-dev/narwhals/blob/main/CONTRIBUTING.md)

## Used by

Join the party!

- [altair](https://github.com/vega/altair/)
- [hierarchicalforecast](https://github.com/Nixtla/hierarchicalforecast)
- [marimo](https://github.com/marimo-team/marimo)
- [panel-graphic-walker](https://github.com/panel-extensions/panel-graphic-walker)
- [plotly](https://plotly.com)
- [pymarginaleffects](https://github.com/vincentarelbundock/pymarginaleffects)
- [py-shiny](https://github.com/posit-dev/py-shiny)
- [rio](https://github.com/rio-labs/rio)
- [scikit-lego](https://github.com/koaning/scikit-lego)
- [scikit-playtime](https://github.com/koaning/scikit-playtime)
- [tabmat](https://github.com/Quantco/tabmat)
- [timebasedcv](https://github.com/FBruzzesi/timebasedcv)
- [tubular](https://github.com/lvgig/tubular)
- [vegafusion](https://github.com/vega/vegafusion)
- [wimsey](https://github.com/benrutter/wimsey)

Feel free to add your project to the list if it's missing, and/or
[chat with us on Discord](https://discord.gg/V3PqtB4VA4) if you'd like any support.
<details>
<summary>Table of contents</summary>

- [Narwhals](#narwhals)
- [Installation](#installation)
- [Usage](#usage)
- [Example](#example)
- [Scope](#scope)
- [Roadmap](#roadmap)
- [Used by](#used-by)
- [Sponsors and institutional partners](#sponsors-and-institutional-partners)
- [Appears on](#appears-on)
- [Why "Narwhals"?](#why-narwhals)

</details>

## Installation

Expand Down Expand Up @@ -106,6 +99,34 @@ See the [tutorial](https://narwhals-dev.github.io/narwhals/basics/dataframe/) fo

If you said yes to both, we'd love to hear from you!

## Roadmap

See [roadmap discussion on GitHub](https://github.com/narwhals-dev/narwhals/discussions/1370)
for an up-to-date plan of future work.

## Used by

Join the party!

- [altair](https://github.com/vega/altair/)
- [hierarchicalforecast](https://github.com/Nixtla/hierarchicalforecast)
- [marimo](https://github.com/marimo-team/marimo)
- [panel-graphic-walker](https://github.com/panel-extensions/panel-graphic-walker)
- [plotly](https://plotly.com)
- [pymarginaleffects](https://github.com/vincentarelbundock/pymarginaleffects)
- [py-shiny](https://github.com/posit-dev/py-shiny)
- [rio](https://github.com/rio-labs/rio)
- [scikit-lego](https://github.com/koaning/scikit-lego)
- [scikit-playtime](https://github.com/koaning/scikit-playtime)
- [tabmat](https://github.com/Quantco/tabmat)
- [timebasedcv](https://github.com/FBruzzesi/timebasedcv)
- [tubular](https://github.com/lvgig/tubular)
- [vegafusion](https://github.com/vega/vegafusion)
- [wimsey](https://github.com/benrutter/wimsey)

Feel free to add your project to the list if it's missing, and/or
[chat with us on Discord](https://discord.gg/V3PqtB4VA4) if you'd like any support.

## Sponsors and institutional partners

Narwhals is 100% independent, community-driven, and community-owned.
Expand Down
7 changes: 0 additions & 7 deletions docs/css/code_select.css

This file was deleted.

4 changes: 4 additions & 0 deletions docs/css/extra.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.md-typeset ol li,
.md-typeset ul li {
margin-bottom: 0.1em !important;
}
Comment on lines +1 to +4
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduces spacing between list items in markdown

48 changes: 35 additions & 13 deletions docs/roadmap_and_related.md β†’ docs/ecosystem.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,29 @@
# Roadmap and related projects
# Ecosystem

## Roadmap
## Used by

See [roadmap discussion on GitHub](https://github.com/narwhals-dev/narwhals/discussions/1370)
for an up-to-date plan of future work.
The following is a non-exhaustive list of libraries and tools that choose to use Narwhals
for their dataframe interoperability needs:

* [altair](https://github.com/vega/altair/)
* [hierarchicalforecast](https://github.com/Nixtla/hierarchicalforecast)
* [marimo](https://github.com/marimo-team/marimo)
* [panel-graphic-walker](https://github.com/panel-extensions/panel-graphic-walker)
* [plotly](https://github.com/plotly/plotly.py)
* [pymarginaleffects](https://github.com/vincentarelbundock/pymarginaleffects)
* [py-shiny](https://github.com/posit-dev/py-shiny)
* [rio](https://github.com/rio-labs/rio)
* [scikit-lego](https://github.com/koaning/scikit-lego)
* [scikit-playtime](https://github.com/koaning/scikit-playtime)
* [tabmat](https://github.com/Quantco/tabmat)
* [timebasedcv](https://github.com/FBruzzesi/timebasedcv)
* [tubular](https://github.com/lvgig/tubular)
* [vegafusion](https://github.com/vega/vegafusion)
* [wimsey](https://github.com/benrutter/wimsey)

If your project is missing from the list, feel free to open a PR to add it.

If you would like to chat with us, or if you need any support, please [join our Discord server](https://discord.gg/V3PqtB4VA4).

## Related projects

Expand All @@ -21,7 +41,10 @@ Array counterpart to the DataFrame API, see [here](https://data-apis.org/array-a

### PyCapsule Interface

Allows C extension modules to safely share pointers to C data structures with Python code and other C modules, encapsulating the pointer with a name and optional destructor to manage resources and ensure safe access, see [here](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) for details.
Allows C extension modules to safely share pointers to C data structures with Python code
and other C modules, encapsulating the pointer with a name and optional destructor to
manage resources and ensure safe access,
see [here](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) for details.

Narwhals supports exporting a DataFrame via the Arrow PyCapsule Interface.

Expand All @@ -30,30 +53,29 @@ Narwhals supports exporting a DataFrame via the Arrow PyCapsule Interface.
Pitched as "The portable Dataframe library", Ibis provides a Pythonic frontend
to various SQL (as well as Polars LazyFrame) engines. Some differences with Narwhals are:

- Narwhals' main use case is for library maintainers wanting to support
* Narwhals' main use case is for library maintainers wanting to support
different dataframe libraries without depending on any whilst keeping
things as lightweight as possible. Ibis is more targeted at end users
and aims to be thought of as a Dataframe library akin to
pandas / Polars / etc.
- Narwhals allows you to write a "Dataframe X in, Dataframe X out" function.
* Narwhals allows you to write a "Dataframe X in, Dataframe X out" function.
Ibis allows materialising to pandas, Polars (eager), and PyArrow, but has
no way to get back to the input type exactly (e.g. there's no way to
start with a Polars LazyFrame and get back a Polars LazyFrame)
- Narwhals respects input data types as much as possible, Ibis doesn't
* Narwhals respects input data types as much as possible, Ibis doesn't
support Categorical (nor does it distinguish between fixed-size-list and
list)
- Narwhals separates between lazy and eager APIs, with the eager API
* Narwhals separates between lazy and eager APIs, with the eager API
provide very fine control over dataframe operations (slicing rows and
columns, iterating over rows, getting values out of the dataframe as
Python scalars). Ibis is more focused on lazy execution
- Ibis supports SQL engines (and can translate to SQL),
* Ibis supports SQL engines (and can translate to SQL),
Narwhals is more focused traditional dataframes where row-order is defined
(although we are brainstorming a lazy-only level of support)
- Narwhals is extremely lightweight and comes with zero required dependencies,
* Narwhals is extremely lightweight and comes with zero required dependencies,
Ibis requires pandas and PyArrow for all backends
- Narwhals supports Dask, whereas Ibis has deprecated support for it
* Narwhals supports Dask, whereas Ibis has deprecated support for it

Although people often ask about the two tools, we consider them to be
very different and not in competition. Further efforts to clarify the
distinction are welcome πŸ™!

6 changes: 3 additions & 3 deletions docs/extending.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Extending Narwhals, levels of support
# Supported libraries and extending Narwhals

## List of supported libraries (and how to add yours!)

Expand All @@ -15,7 +15,7 @@ Currently, Narwhals has full API support for the following libraries:
It also has lazy-only support for [Dask](https://github.com/dask/dask), and interchange-only support
for [DuckDB](https://github.com/duckdb/duckdb) and [Ibis](https://github.com/ibis-project/ibis).

### Levels
### Levels of support

Narwhals comes with two levels of support ("full" and "interchange"), and we are working on defining
a "lazy-only" level too.
Expand Down Expand Up @@ -93,7 +93,7 @@ converting to Narwhals DataFrame:
- `.select(names)` (Ibis and DuckDB), where `names` is a list of (string) column names. This is useful for
selecting columns before converting to another library.

### Extending Narwhals
## Extending Narwhals

If you want your own library to be recognised too, you're welcome open a PR (with tests)!.
Alternatively, if you can't do that (for example, if you library is closed-source), see
Expand Down
5 changes: 5 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,8 @@ Anyone wishing to write a library/application/service which consumes dataframes,
completely dataframe-agnostic.

Let's get started!

## Roadmap

See [roadmap discussion on GitHub](https://github.com/narwhals-dev/narwhals/discussions/1370)
for an up-to-date plan of future work.
25 changes: 16 additions & 9 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
If you haven't, you can follow our [_setting up your environment_](https://github.com/narwhals-dev/narwhals/blob/main/CONTRIBUTING.md#option-1-use-uv-recommended) guide.
Then, run:

```console
```terminal
uv pip install narwhals
```

Expand All @@ -19,18 +19,20 @@

Then, run:

```console
```terminal
python -m pip install narwhals
```

### Verifying the Installation

To verify the installation, start the Python REPL and execute:

```python
>>> import narwhals
>>> narwhals.__version__
'1.14.3'
```

If you see the version number, then the installation was successful!

## Quick start
Expand Down Expand Up @@ -60,7 +62,7 @@ import narwhals as nw
from narwhals.typing import IntoFrame


def my_function(df_native: IntoFrame) -> list[str]:
def agnostic_get_columns(df_native: IntoFrame) -> list[str]:
df = nw.from_native(df_native)
column_names = df.columns
return column_names
Expand All @@ -72,17 +74,22 @@ df_polars = pl.DataFrame(data)
table_pa = pa.table(data)

print("pandas output")
print(my_function(df_pandas))
print(agnostic_get_columns(df_pandas))

print("Polars output")
print(my_function(df_polars))
print(agnostic_get_columns(df_polars))

print("PyArrow output")
print(my_function(table_pa))
print(agnostic_get_columns(table_pa))
```

If you run `python t.py` then your output should look like the above. This is the simplest possible example of a dataframe-agnostic
function - as we'll soon see, we can do much more advanced things.
If you run `python t.py` then your output should look like the above.
This is the simplest possible example of a dataframe-agnostic function - as we'll soon
see, we can do much more advanced things.

Let's learn about what you just did, and what Narwhals can do for you!

Note: these examples are only using pandas, Polars and PyArrow. Please see the following to find the [supported libraries](extending.md).
!!! info
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am big fan boy of mkdocs material admonitions 😁


These examples are using pandas, Polars and PyArrow, however Narwhals supports
other dataframe libraries (See [supported libraries](extending.md)).
47 changes: 47 additions & 0 deletions docs/resources.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Resources

## Sponsors and institutional partners

Narwhals is 100% independent, community-driven, and community-owned.
We are extremely grateful to the following organisations for having
provided some funding / development time:

- [Quansight Labs](https://labs.quansight.org)
- [Quansight Futures](https://www.qi.ventures)
- [OpenTeams](https://www.openteams.com)
- [POSSEE initiative](https://possee.org)
- [BYU-Idaho](https://www.byui.edu)

If you contribute to Narwhals on your organization's time, please let us know. We'd be happy to add your employer
to this list!

## Appears on

Narwhals has been featured in several talks, podcasts, and blog posts:

- [Talk Python to me Podcast](https://youtu.be/FSH7BZ0tuE0)
Ahoy, Narwhals are bridging the data science APIs

- [Python Bytes Podcast](https://www.youtube.com/live/N7w_ESVW40I?si=y-wN1uCsAuJOKlOT&t=382)
Episode 402, topic #2

- [Super Data Science: ML & AI Podcast](https://www.youtube.com/watch?v=TeG4U8R0U8U)
Narwhals: For Pandas-to-Polars DataFrame Compatibility

- [Sample Space Podcast | probabl](https://youtu.be/8hYdq4sWbbQ?si=WG0QP1CZ6gkFf18b)
How Narwhals has many end users ... that never use it directly. - Marco Gorelli

- [The Real Python Podcast](https://www.youtube.com/watch?v=w5DFZbFYzCM)
Narwhals: Expanding DataFrame Compatibility Between Libraries

- [Pycon Lithuania](https://www.youtube.com/watch?v=-mdx7Cn6_6E)
Marco Gorelli - DataFrame interoperatiblity - what's been achieved, and what comes next?

- [Pycon Italy](https://www.youtube.com/watch?v=3IqUli9XsmQ)
How you can write a dataframe-agnostic library - Marco Gorelli

- [Polars Blog Post](https://pola.rs/posts/lightweight_plotting/)
Polars has a new lightweight plotting backend

- [Quansight Labs blog post (w/ Scikit-Lego)](https://labs.quansight.org/blog/scikit-lego-narwhals)
How Narwhals and scikit-lego came together to achieve dataframe-agnosticism
9 changes: 6 additions & 3 deletions docs/why.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ print(3 in pd.Series([1, 2, 3]))
print(3 in pl.Series([1, 2, 3]))
```

Try it out and see ;) Spoiler alert: they don't. pandas checks if `3` is in the index,
Try it out and see πŸ˜‰

Spoiler alert: they don't. pandas checks if `3` is in the index,
Polars checks if it's in the values.

For another example, try running the code below - note how the outputs have different column names after the join!
Expand All @@ -30,10 +32,11 @@ print(pd_left_merge.columns)
print(pl_left_merge.columns)
```

There are several such subtle difference between the libraries. Writing dataframe-agnostic code is hard!
There are several such subtle difference between the libraries. **Writing dataframe-agnostic code is hard!**

But by having a unified, simple, and predictable API, you can focus on behaviour rather than on subtle
implementation differences.

Furthermore, both pandas and Polars frequently deprecate behaviour. Narwhals handles this for you by
testing against nightly builds of both libraries and handling backwards compatibility internally
testing against nightly builds of both libraries and handling backwards compatibility internally
(so you don't have to!).
12 changes: 7 additions & 5 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,12 @@ nav:
- pandas_like_concepts/user_warning.md
- pandas_like_concepts/column_names.md
- pandas_like_concepts/boolean.md
- overhead.md
- backcompat.md
- extending.md
- how_it_works.md
- Roadmap and related projects: roadmap_and_related.md
- Overhead: overhead.md
- Perfect backwards compatibility policy: backcompat.md
- Supported libraries and extending Narwhals: extending.md
- How it works: how_it_works.md
- Ecosystem: ecosystem.md
- Resources: resources.md
- API Completeness:
- api-completeness/index.md
- Supported DataFrame methods: api-completeness/dataframe.md
Expand Down Expand Up @@ -127,3 +128,4 @@ extra_javascript:

extra_css:
- https://unpkg.com/katex@0/dist/katex.min.css
- css/extra.css
Loading