Merge remote-tracking branch 'upstream/master' into 3302-pandas-connctor

CODE_OF_CONDUCT.md

@@ -0,0 +1,84 @@
+# Code of Conduct
+
+## 1. Purpose
+
+A primary goal of Apache Superset is to be inclusive to the largest number of contributors, with the most varied and diverse backgrounds possible. As such, we are committed to providing a friendly, safe and welcoming environment for all, regardless of gender, sexual orientation, ability, ethnicity, socioeconomic status, and religion (or lack thereof).
+
+This code of conduct outlines our expectations for all those who participate in our community, as well as the consequences for unacceptable behavior.
+
+We invite all those who participate in Apache Superset to help us create safe and positive experiences for everyone.
+
+## 2. Open Source Citizenship
+
+A supplemental goal of this Code of Conduct is to increase open source citizenship by encouraging participants to recognize and strengthen the relationships between our actions and their effects on our community.
+
+Communities mirror the societies in which they exist and positive action is essential to counteract the many forms of inequality and abuses of power that exist in society.
+
+If you see someone who is making an extra effort to ensure our community is welcoming, friendly, and encourages all participants to contribute to the fullest extent, we want to know.
+
+## 3. Expected Behavior
+
+The following behaviors are expected and requested of all community members:
+
+*   Participate in an authentic and active way. In doing so, you contribute to the health and longevity of this community.
+*   Exercise consideration and respect in your speech and actions.
+*   Attempt collaboration before conflict.
+*   Refrain from demeaning, discriminatory, or harassing behavior and speech.
+*   Be mindful of your surroundings and of your fellow participants. Alert community leaders if you notice a dangerous situation, someone in distress, or violations of this Code of Conduct, even if they seem inconsequential.
+*   Remember that community event venues may be shared with members of the public; please be respectful to all patrons of these locations.
+
+## 4. Unacceptable Behavior
+
+The following behaviors are considered harassment and are unacceptable within our community:
+
+*   Violence, threats of violence or violent language directed against another person.
+*   Sexist, racist, homophobic, transphobic, ableist or otherwise discriminatory jokes and language.
+*   Posting or displaying sexually explicit or violent material.
+*   Posting or threatening to post other people’s personally identifying information ("doxing").
+*   Personal insults, particularly those related to gender, sexual orientation, race, religion, or disability.
+*   Inappropriate photography or recording.
+*   Inappropriate physical contact. You should have someone’s consent before touching them.
+*   Unwelcome sexual attention. This includes, sexualized comments or jokes; inappropriate touching, groping, and unwelcomed sexual advances.
+*   Deliberate intimidation, stalking or following (online or in person).
+*   Advocating for, or encouraging, any of the above behavior.
+*   Sustained disruption of community events, including talks and presentations.
+
+## 5. Consequences of Unacceptable Behavior
+
+Unacceptable behavior from any community member, including sponsors and those with decision-making authority, will not be tolerated.
+
+Anyone asked to stop unacceptable behavior is expected to comply immediately.
+
+If a community member engages in unacceptable behavior, the community organizers may take any action they deem appropriate, up to and including a temporary ban or permanent expulsion from the community without warning (and without refund in the case of a paid event).
+
+## 6. Reporting Guidelines
+
+If you are subject to or witness unacceptable behavior, or have any other concerns, please notify a community organizer as soon as possible. [email protected] .
+
+
+
+Additionally, community organizers are available to help community members engage with local law enforcement or to otherwise help those experiencing unacceptable behavior feel safe. In the context of in-person events, organizers will also provide escorts as desired by the person experiencing distress.
+
+## 7. Addressing Grievances
+
+If you feel you have been falsely or unfairly accused of violating this Code of Conduct, you should notify Apache with a concise description of your grievance. Your grievance will be handled in accordance with our existing governing policies.
+
+
+
+## 8. Scope
+
+We expect all community participants (contributors, paid or otherwise; sponsors; and other guests) to abide by this Code of Conduct in all community venues–online and in-person–as well as in all one-on-one communications pertaining to community business.
+
+This code of conduct and its related procedures also applies to unacceptable behavior occurring outside the scope of community activities when such behavior has the potential to adversely affect the safety and well-being of community members.
+
+## 9. Contact info
+
+[email protected] 
+
+## 10. License and attribution
+
+This Code of Conduct is distributed under a [Creative Commons Attribution-ShareAlike license](http://creativecommons.org/licenses/by-sa/3.0/).
+
+Portions of text derived from the [Django Code of Conduct](https://www.djangoproject.com/conduct/) and the [Geek Feminism Anti-Harassment Policy](http://geekfeminism.wikia.com/wiki/Conference_anti-harassment/Policy).
+
+Retrieved on November 22, 2016 from [http://citizencodeofconduct.org/](http://citizencodeofconduct.org/)

docs/import_export_datasources.rst

@@ -0,0 +1,103 @@
+Importing and Exporting Datasources
+===================================
+
+The superset cli allows you to import and export datasources from and to YAML.
+Datasources include both databases and druid clusters. The data is expected to be organized in the following hierarchy: ::
+
+    .
+    ├──databases
+    |  ├──database_1
+    |  |  ├──table_1
+    |  |  |  ├──columns
+    |  |  |  |  ├──column_1
+    |  |  |  |  ├──column_2
+    |  |  |  |  └──... (more columns)
+    |  |  |  └──metrics
+    |  |  |     ├──metric_1
+    |  |  |     ├──metric_2
+    |  |  |     └──... (more metrics)
+    |  |  └── ... (more tables)
+    |  └── ... (more databases)
+    └──druid_clusters
+       ├──cluster_1
+       |  ├──datasource_1
+       |  |  ├──columns
+       |  |  |  ├──column_1
+       |  |  |  ├──column_2
+       |  |  |  └──... (more columns)
+       |  |  └──metrics
+       |  |     ├──metric_1
+       |  |     ├──metric_2
+       |  |     └──... (more metrics)
+       |  └── ... (more datasources)
+       └── ... (more clusters)
+
+
+Exporting Datasources to YAML
+-----------------------------
+You can print your current datasources to stdout by running: ::
+
+    superset export_datasources
+
+
+To save your datasources to a file run: ::
+
+    superset export_datasources -f <filename>
+
+
+By default, default (null) values will be omitted. Use the ``-d`` flag to include them.
+If you want back references to be included (e.g. a column to include the table id
+it belongs to) use the ``-b`` flag.
+
+Alternatively you can export datasources using the UI: ::
+
+1. Open **Sources** -> **Databases** to export all tables associated to a single or multiple databases. (**Tables** for one or more tables, **Druid Clusters** for clusters, **Druid Datasources** for datasources)
+2. Select the items you would like to export
+3. Click **Actions** -> **Export to YAML**
+4. If you want to import an item that you exported through the UI, you will need to nest it inside its parent element, e.g. a `database` needs to be nested under `databases` a `table` needs to be nested inside a `database` element.
+
+Exporting the complete supported YAML schema
+--------------------------------------------
+In order to obtain an exhaustive list of all fields you can import using the YAML import run: ::
+
+    superset export_datasource_schema
+
+Again, you can use the ``-b`` flag to include back references.
+
+
+Importing Datasources from YAML
+-------------------------------
+In order to import datasources from a YAML file(s), run: ::
+
+    superset import_datasources -p <path or filename>
+
+If you supply a path all files ending with ``*.yaml`` or ``*.yml`` will be parsed.
+You can apply additional falgs e.g.: ::
+
+    superset import_datasources -p <path> -r
+
+Will search the supplied path recursively.
+
+The sync flag ``-s`` takes parameters in order to sync the supplied elements with
+your file. Be careful this can delete the contents of your meta database. Example:
+
+   superset import_datasources -p <path / filename> -s columns,metrics
+
+This will sync all ``metrics`` and ``columns`` for all datasources found in the
+``<path / filename>`` in the Superset meta database. This means columns and metrics
+not specified in YAML will be deleted. If you would add ``tables`` to ``columns,metrics``
+those would be synchronised as well.
+
+
+If you don't supply the sync flag (``-s``) importing will only add and update (override) fields.
+E.g. you can add a ``verbose_name`` to the the column ``ds`` in the table ``random_time_series`` from the example datasets
+by saving the following YAML to file and then running the ``import_datasources`` command. ::
+
+    databases:
+    - database_name: main
+      tables:
+      - table_name: random_time_series
+        columns:
+        - column_name: ds
+          verbose_name: datetime
+

docs/installation.rst

@@ -181,6 +181,11 @@ If the load balancer is inserting X-Forwarded-For/X-Forwarded-Proto headers, you
 should set `ENABLE_PROXY_FIX = True` in the superset config file to extract and use
 the headers.
 
+In case that the reverse proxy is used for providing ssl encryption, 
+an explicit definition of the `X-Forwarded-Proto` may be required.
+For the Apache webserver this can be set as follows: ::
+
+    RequestHeader set X-Forwarded-Proto "https"
 
 Configuration
 -------------
@@ -498,6 +503,11 @@ look something like:
     RESULTS_BACKEND = RedisCache(
         host='localhost', port=6379, key_prefix='superset_results')
 
+Note that it's important that all the worker nodes and web servers in
+the Superset cluster share a common metadata database.
+This means that SQLite will not work in this context since it has
+limited support for concurrency and
+typically lives on the local file system.
 
 Also note that SQL Lab supports Jinja templating in queries, and that it's
 possible to overload

setup.py

@@ -5,7 +5,7 @@
 from setuptools import find_packages, setup
 
 BASE_DIR = os.path.abspath(os.path.dirname(__file__))
-PACKAGE_DIR = os.path.join(BASE_DIR, 'superset', 'assets')
+PACKAGE_DIR = os.path.join(BASE_DIR, 'superset', 'static', 'assets')
 PACKAGE_FILE = os.path.join(PACKAGE_DIR, 'package.json')
 with open(PACKAGE_FILE) as package_file:
     version_string = json.load(package_file)['version']
@@ -58,15 +58,18 @@ def get_git_sha():
         'flask-wtf==0.14.2',
         'flower==0.9.1',
         'future>=0.16.0, <0.17',
+        'python-geohash==0.8.5',
         'humanize==0.5.1',
         'gunicorn==19.7.1',
         'idna==2.5',
         'markdown==2.6.8',
         'pandas==0.20.3',
         'parsedatetime==2.0.0',
+        'pathlib2==2.3.0',
         'pydruid==0.3.1',
         'PyHive>=0.4.0',
         'python-dateutil==2.6.0',
+        'pyyaml>=3.11',
         'requests==2.17.3',
         'simplejson==3.10.0',
         'six==1.10.0',
@@ -75,6 +78,11 @@ def get_git_sha():
         'sqlparse==0.2.3',
         'thrift>=0.9.3',
         'thrift-sasl>=0.2.1',
+        'unidecode>=0.04.21',
+        # PandasConnector deps
+        # @TODO sort into main deps
+        # prior to merge. Currently,
+        # separate to minimize merge conflicts
         'beautifulsoup4==4.6.0',
         'bottleneck==1.2.1',
         'feather-format==0.4.0',
@@ -104,3 +112,4 @@ def get_git_sha():
         'Programming Language :: Python :: 3.5',
     ],
 )
+

superset/__init__.py

@@ -42,7 +42,7 @@ def parse_manifest_json():
         with open(MANIFEST_FILE, 'r') as f:
             manifest = json.load(f)
     except Exception:
-        print('no manifest file found at ' + MANIFEST_FILE)
+        pass
 
 
 def get_manifest_file(filename):
@@ -167,4 +167,10 @@ def index(self):
 module_datasource_map.update(app.config.get('ADDITIONAL_MODULE_DS_MAP'))
 ConnectorRegistry.register_sources(module_datasource_map)
 
+# Hook that provides administrators a handle on the Flask APP
+# after initialization
+flask_app_mutator = app.config.get('FLASK_APP_MUTATOR')
+if flask_app_mutator:
+    flask_app_mutator(app)
+
 from superset import views  # noqa