Merge branch 'master' into remove_repeated_line

.gitignore

@@ -35,3 +35,5 @@ npm-debug.log
 yarn.lock
 superset/assets/version_info.json
 
+# IntelliJ
+*.iml

.travis.yml

@@ -19,7 +19,7 @@ env:
     - TOX_ENV=py27-mysql
     - TOX_ENV=py27-sqlite
 before_install:
-  - npm install -g npm@'>=5.0.3'
+  - npm install -g npm@'>=5.4.1'
 before_script:
   - mysql -e 'drop database if exists superset; create database superset DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci' -u root
   - mysql -u root -e "CREATE USER 'mysqluser'@'localhost' IDENTIFIED BY 'mysqluserpassword';"

CONTRIBUTING.md

@@ -85,7 +85,7 @@ Before you start changing the docs, you'll want to
 [fork the Superset project on Github](https://help.github.com/articles/fork-a-repo/).
 Once that new repository has been created, clone it on your local machine:
 
-    git clone [email protected]:your_username/superset.git
+    git clone [email protected]:your_username/incubator-superset.git
 
 At this point, you may also want to create a
 [Python virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
@@ -97,7 +97,7 @@ to manage the Python packages you're about to install:
 Finally, to make changes to the rst files and build the docs using Sphinx, 
 you'll need to install a handful of dependencies from the repo you cloned:
 
-    cd superset
+    cd incubator-superset
     pip install -r dev-reqs-for-docs.txt
 
 To get the feel for how to edit and build the docs, let's edit a file, build

README.md

@@ -171,6 +171,7 @@ the world know they are using Superset. Join our growing community!
  - [Airbnb](https://github.com/airbnb)
  - [Amino](https://amino.com)
  - [Brilliant.org](https://brilliant.org/)
+ - [Capital Service S.A.](http://capitalservice.pl)
  - [Clark.de](http://clark.de/)
  - [Digit Game Studios](https://www.digitgaming.com/)
  - [Douban](https://www.douban.com/)
@@ -181,6 +182,7 @@ the world know they are using Superset. Join our growing community!
  - [Konfío](http://konfio.mx)
  - [Maieutical Labs](https://cloudschooling.it)
  - [Qunar](https://www.qunar.com/)
+ - [Shopee](https://shopee.sg)
  - [Shopkick](https://www.shopkick.com)
  - [Tails.com](https://tails.com)
  - [Tobii](http://www.tobii.com/)

docs/installation.rst

@@ -8,6 +8,32 @@ Superset is tested against Python ``2.7`` and Python ``3.4``.
 Airbnb currently uses 2.7.* in production. We do not plan on supporting
 Python ``2.6``.
 
+Cloud-native!
+-------------
+
+Superset is designed to be highly available. It is
+"cloud-native" as it has been designed scale out in large,
+distributed environments, and works well inside containers.
+While you can easily
+test drive Superset on a modest setup or simply on your laptop,
+there's virtually no limit around scaling out the platform.
+Superset is also cloud-native in the sense that it is
+flexible and lets you choose your web server (Gunicorn, Nginx, Apache),
+your metadata database engine (MySQL, Postgres, MariaDB, ...),
+your message queue (Redis, RabbitMQ, SQS, ...),
+your results backend (S3, Redis, Memcached, ...), your caching layer
+(memcached, Redis, ...), works well with services like NewRelic, StatsD and
+DataDog, and has the ability to run analytic workloads against
+most popular database technologies.
+
+Superset is battle tested in large environments with hundreds
+of concurrent users. Airbnb's production environment runs inside
+Kubernetes and serves 600+ daily active users viewing over 100K charts a
+day.
+
+The Superset web server and the Superset Celery workers (optional)
+are stateless, so you can scale out by running on as many servers
+as needed.
 
 OS dependencies
 ---------------
@@ -107,10 +133,40 @@ the credential you entered while creating the admin account, and navigate to
 your datasources for Superset to be aware of, and they should show up in
 `Menu -> Datasources`, from where you can start playing with your data!
 
-Please note that *gunicorn*, Superset default application server, does not
-work on Windows so you need to use the development web server.
-The development web server though is not intended to be used on production systems
-so better use a supported platform that can run *gunicorn*.
+A proper WSGI HTTP Server
+-------------------------
+
+While you can setup Superset to run on Nginx or Apache, many use
+Gunicorn, preferably in **async mode**, which allows for impressive
+concurrency even and is fairly easy to install and configure. Please
+refer to the
+documentation of your preferred technology to set up this Flask WSGI
+application in a way that works well in your environment.
+
+While the `superset runserver` command act as an quick wrapper
+around `gunicorn`, it doesn't expose all the options you may need,
+so you'll want to craft your own `gunicorn` command in your production
+environment. Here's an **async** setup known to work well: ::
+
+	gunicorn \
+		-w 10 \
+		-k gevent \
+		--timeout 120 \
+		-b  0.0.0.0:6666 \
+		--limit-request-line 0 \
+		--limit-request-field_size 0 \
+		--statsd-host localhost:8125 \
+		superset:app
+
+Refer to the
+[Gunicorn documentation](http://docs.gunicorn.org/en/stable/design.html)
+for more information.
+
+Note that *gunicorn* does not
+work on Windows so the `superser runserver` command is not expected to work
+in that context. Also note that the development web
+server (`superset runserver -d`) is not intended for production use.
+
 
 Configuration behind a load balancer
 ------------------------------------
@@ -157,6 +213,8 @@ of the parameters you can copy / paste in that configuration module: ::
 
     # Flask-WTF flag for CSRF
     WTF_CSRF_ENABLED = True
+    # Add endpoints that need to be exempt from CSRF protection
+    WTF_CSRF_EXEMPT_LIST = []
 
     # Set this API key to enable Mapbox visualizations
     MAPBOX_API_KEY = ''
@@ -172,6 +230,11 @@ Please make sure to change:
 * *SQLALCHEMY_DATABASE_URI*, by default it is stored at *~/.superset/superset.db*
 * *SECRET_KEY*, to a long random string
 
+In case you need to exempt endpoints from CSRF, e.g. you are running a custom
+auth postback endpoint, you can add them to *WTF_CSRF_EXEMPT_LIST*
+
+     WTF_CSRF_EXEMPT_LIST = ['']
+
 Database dependencies
 ---------------------
 
@@ -223,10 +286,6 @@ database you want to connect to should get you to the right place.
 (AWS) Athena
 ------------
 
-This currently relies on an unreleased future version of `PyAthenaJDBC <https://github.com/laughingman7743/PyAthenaJDBC>`_. If you're adventurous or simply impatient, you can install directly from git: ::
-
-    pip install git+https://github.com/laughingman7743/PyAthenaJDBC@support_sqlalchemy
-
 The connection string for Athena looks like this ::
 
     awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...
@@ -284,6 +343,24 @@ on top of the **database**. For Superset to connect to a specific schema,
 there's a **schema** parameter you can set in the table form.
 
 
+External Password store for SQLAlchemy connections
+--------------------------------------------------
+It is possible to use an external store for you database passwords. This is
+useful if you a running a custom secret distribution framework and do not wish
+to store secrets in Superset's meta database.
+
+Example:
+Write a function that takes a single argument of type ``sqla.engine.url`` and returns
+the password for the given connection string. Then set ``SQLALCHEMY_CUSTOM_PASSWORD_STORE``
+in your config file to point to that function. ::
+
+    def example_lookup_password(url):
+        secret = <<get password from external framework>>
+        return 'secret'
+
+    SQLALCHEMY_CUSTOM_PASSWORD_STORE = example_lookup_password
+
+
 SSL Access to databases
 -----------------------
 This example worked with a MySQL database that requires SSL. The configuration

docs/security.rst

@@ -36,7 +36,7 @@ own. Alpha users can add and alter data sources.
 Gamma
 """""
 Gamma have limited access. They can only consume data coming from data sources
-they have been giving access to through another complementary role.
+they have been given access to through another complementary role.
 They only have access to view the slices and
 dashboards made from data sources that they have access to. Currently Gamma
 users are not able to alter or add data sources. We assume that they are

setup.py

@@ -68,7 +68,7 @@ def get_git_sha():
         'simplejson==3.10.0',
         'six==1.10.0',
         'sqlalchemy==1.1.9',
-        'sqlalchemy-utils==0.32.14',
+        'sqlalchemy-utils==0.32.16',
         'sqlparse==0.2.3',
         'thrift>=0.9.3',
         'thrift-sasl>=0.2.1',

superset/__init__.py

@@ -83,6 +83,9 @@ def get_js_manifest():
 
 if conf.get('WTF_CSRF_ENABLED'):
     csrf = CSRFProtect(app)
+    csrf_exempt_list = conf.get('WTF_CSRF_EXEMPT_LIST', [])
+    for ex in csrf_exempt_list:
+        csrf.exempt(ex)
 
 utils.pessimistic_connection_handling(db.engine)
 

superset/assets/javascripts/SqlLab/actions.js

@@ -20,6 +20,8 @@ export const QUERY_EDITOR_SET_TITLE = 'QUERY_EDITOR_SET_TITLE';
 export const QUERY_EDITOR_SET_AUTORUN = 'QUERY_EDITOR_SET_AUTORUN';
 export const QUERY_EDITOR_SET_SQL = 'QUERY_EDITOR_SET_SQL';
 export const QUERY_EDITOR_SET_SELECTED_TEXT = 'QUERY_EDITOR_SET_SELECTED_TEXT';
+export const QUERY_EDITOR_PERSIST_HEIGHT = 'QUERY_EDITOR_PERSIST_HEIGHT';
+
 export const SET_DATABASES = 'SET_DATABASES';
 export const SET_ACTIVE_QUERY_EDITOR = 'SET_ACTIVE_QUERY_EDITOR';
 export const SET_ACTIVE_SOUTHPANE_TAB = 'SET_ACTIVE_SOUTHPANE_TAB';
@@ -346,6 +348,10 @@ export function refreshQueries(alteredQueries) {
   return { type: REFRESH_QUERIES, alteredQueries };
 }
 
+export function persistEditorHeight(queryEditor, currentHeight) {
+  return { type: QUERY_EDITOR_PERSIST_HEIGHT, queryEditor, currentHeight };
+}
+
 export function popStoredQuery(urlId) {
   return function (dispatch) {
     $.ajax({

superset/assets/javascripts/SqlLab/components/QuerySearch.jsx

@@ -153,7 +153,7 @@ class QuerySearch extends React.PureComponent {
               placeholder="Search Results"
             />
           </div>
-          <div className="col-sm-1">
+          <div className="col-sm-4 search-date-filter-container">
             <Select
               name="select-from"
               placeholder="[From]-"
@@ -163,8 +163,7 @@ class QuerySearch extends React.PureComponent {
               autosize={false}
               onChange={this.changeFrom.bind(this)}
             />
-          </div>
-          <div className="col-sm-1">
+
             <Select
               name="select-to"
               placeholder="[To]-"
@@ -173,8 +172,7 @@ class QuerySearch extends React.PureComponent {
               autosize={false}
               onChange={this.changeTo.bind(this)}
             />
-          </div>
-          <div className="col-sm-1">
+
             <Select
               name="select-status"
               placeholder="[Query Status]"
@@ -184,10 +182,11 @@ class QuerySearch extends React.PureComponent {
               autosize={false}
               onChange={this.changeStatus.bind(this)}
             />
+
+            <Button bsSize="small" bsStyle="success" onClick={this.refreshQueries.bind(this)}>
+              Search
+            </Button>
           </div>
-          <Button bsSize="small" bsStyle="success" onClick={this.refreshQueries.bind(this)}>
-            Search
-          </Button>
         </div>
         {this.state.queriesLoading ?
           (<img className="loading" alt="Loading..." src="/static/assets/images/loading.gif" />)

superset/assets/javascripts/SqlLab/components/SqlEditor.jsx

@@ -67,11 +67,18 @@ class SqlEditor extends React.PureComponent {
       southPaneHeight: height - this.refs.ace.clientHeight,
       height,
     });
+
+    if (this.refs.ace.clientHeight) {
+      this.props.actions.persistEditorHeight(this.props.queryEditor, this.refs.ace.clientHeight);
+    }
   }
   setQueryEditorSql(sql) {
     this.props.actions.queryEditorSetSql(this.props.queryEditor, sql);
   }
   runQuery(runAsync = false) {
+    if (!this.props.queryEditor.sql) {
+      return;
+    }
     let effectiveRunAsync = runAsync;
     if (!this.props.database.allow_run_sync) {
       effectiveRunAsync = true;
@@ -191,7 +198,7 @@ class SqlEditor extends React.PureComponent {
   }
   render() {
     const height = this.sqlEditorHeight();
-    const defaultNorthHeight = 200;
+    const defaultNorthHeight = this.props.queryEditor.height || 200;
     return (
       <div
         className="SqlEditor"

superset/assets/javascripts/SqlLab/main.less

@@ -292,3 +292,10 @@ a.Link {
 .SouthPane {
     margin-top: 10px;
 }
+.search-date-filter-container {
+    display: flex;
+
+    .Select {
+        margin-right: 3px;
+    }
+}

superset/assets/javascripts/SqlLab/reducers.js

@@ -216,6 +216,9 @@ export const sqlLabReducer = function (state, action) {
     [actions.QUERY_EDITOR_SET_AUTORUN]() {
       return alterInArr(state, 'queryEditors', action.queryEditor, { autorun: action.autorun });
     },
+    [actions.QUERY_EDITOR_PERSIST_HEIGHT]() {
+      return alterInArr(state, 'queryEditors', action.queryEditor, { height: action.currentHeight });
+    },
     [actions.ADD_ALERT]() {
       return addToArr(state, 'alerts', action.alert);
     },

superset/assets/javascripts/components/Checkbox.jsx

@@ -0,0 +1,24 @@
+import React from 'react';
+import PropTypes from 'prop-types';
+
+const propTypes = {
+  checked: PropTypes.bool.isRequired,
+  onChange: PropTypes.func.isRequired,
+  style: PropTypes.object,
+};
+
+export default function Checkbox({ checked, onChange, style }) {
+  return (
+    <span style={style}>
+      <i
+        className={`fa fa-check ${checked ? 'text-primary' : 'text-transparent'}`}
+        onClick={onChange.bind(!checked)}
+        style={{
+          border: '1px solid #aaa',
+          borderRadius: '2px',
+          cursor: 'pointer',
+        }}
+      />
+    </span>);
+}
+Checkbox.propTypes = propTypes;

superset/assets/javascripts/components/InfoTooltipWithTrigger.jsx

@@ -10,6 +10,7 @@ const propTypes = {
   className: PropTypes.string,
   onClick: PropTypes.func,
   placement: PropTypes.string,
+  bsStyle: PropTypes.string,
 };
 const defaultProps = {
   icon: 'info-circle',
@@ -18,14 +19,15 @@ const defaultProps = {
 };
 
 export default function InfoTooltipWithTrigger({
-    label, tooltip, icon, className, onClick, placement }) {
+    label, tooltip, icon, className, onClick, placement, bsStyle }) {
+  const iconClass = `fa fa-${icon} ${className} ${bsStyle ? 'text-' + bsStyle : ''}`;
   return (
     <OverlayTrigger
       placement={placement}
       overlay={<Tooltip id={`${slugify(label)}-tooltip`}>{tooltip}</Tooltip>}
     >
       <i
-        className={`fa fa-${icon} ${className}`}
+        className={iconClass}
         onClick={onClick}
         style={{ cursor: onClick ? 'pointer' : null }}
       />