Make it easer for developers to find Ballista documentation (#330)

Author: andygrove

DEVELOPERS.md

@@ -21,6 +21,9 @@
 
 This section describes how you can get started at developing DataFusion.
 
+For information on developing with Ballista, see the 
+[Ballista developer documentation](ballista/docs/README.md). 
+
 ### Bootstrap environment
 
 DataFusion is written in Rust and it uses a standard rust toolkit:

README.md

@@ -30,6 +30,9 @@ logical query plans as well as a query optimizer and execution engine
 capable of parallel execution against partitioned data sources (CSV
 and Parquet) using threads.
 
+DataFusion also supports distributed query execution via the  
+[Ballista](ballista/README.md) crate.
+
 ## Use Cases
 
 DataFusion is used to create modern, fast and efficient data

ballista/README.md

@@ -50,15 +50,14 @@ Although Ballista is largely inspired by Apache Spark, there are some key differ
 - The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
   in any programming language with minimal serialization overhead.
 
-# Status
+## Status
 
-The Ballista project was donated to Apache Arrow in April 2021 and work is underway to integrate more tightly with 
-DataFusion.
-
-One of the goals is to implement a common scheduler that can seamlessly scale queries across cores in DataFusion and 
-across nodes in Ballista.
-
-Ballista issues are tracked in ASF JIRA [here](https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22)
+Ballista was [donated](https://arrow.apache.org/blog/2021/04/12/ballista-donation/) to the Apache Arrow project in
+April 2021 and should be considered experimental.
 
+## Getting Started
 
+The [Ballista Developer Documentation](docs/README.md) and the 
+[DataFusion User Guide](https://github.com/apache/arrow-datafusion/tree/master/docs/user-guide) are currently the 
+best sources of information for getting started with Ballista.
 

ballista/docs/README.md

@@ -20,7 +20,7 @@
 
 This directory contains documentation for developers that are contributing to Ballista. If you are looking for 
 end-user documentation for a published release, please start with the 
-[Ballista User Guide](https://ballistacompute.org/docs/) instead.
+[DataFusion User Guide](../../docs/user-guide) instead.
 
 ## Architecture & Design
 
@@ -29,9 +29,6 @@ end-user documentation for a published release, please start with the
 
 ## Build, Test, Release
 
-- Setting up a [Rust development environment](dev-env-rust.md).
-- Setting up a [Java development environment](dev-env-jvm.md).
-- Notes on building [Rust docker images](rust-docker.md)  
+- Setting up a [development environment](dev-env.md).
 - [Integration Testing](integration-testing.md)
-- [Release process](release-process.md)
 

ballista/docs/dev-env-rust.md ballista/docs/dev-env.md

@@ -18,15 +18,11 @@
 -->
 # Integration Testing
 
-Ballista has a [benchmark crate](https://github.com/ballista-compute/ballista/tree/main/rust/benchmarks/tpch) which is
-derived from TPC-H and this is currently the main form of integration testing. 
+We use the [DataFusion Benchmarks](https://github.com/apache/arrow-datafusion/tree/master/benchmarks) for integration 
+testing. 
 
-The following command can be used to run the integration tests.
+The integration tests can be executed by running the following command from the root of the DataFusion repository.
 
 ```bash
 ./dev/integration-tests.sh
 ```
-
-Please refer to the
-[benchmark documentation](https://github.com/ballista-compute/ballista/blob/main/rust/benchmarks/tpch/README.md)
-for more information.