Skip to content

Commit

Permalink
Merge pull request #12 from qld-gov-au/QOL-9161-upstream-sync
Browse files Browse the repository at this point in the history
QOL-9161 sync from upstream and fix bugs
  • Loading branch information
ThrawnCA authored Aug 29, 2022
2 parents 548c037 + bce8bb5 commit 774331c
Show file tree
Hide file tree
Showing 30 changed files with 502 additions and 366 deletions.
1 change: 1 addition & 0 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ format = pylint

# Show the source of errors.
show_source = True
statistics = True

max-complexity = 10
max-line-length = 127
Expand Down
61 changes: 19 additions & 42 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,34 +1,29 @@
---
name: Tests
on: [push, pull_request]
on:
push:
pull_request:
branches:
- master

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v2
with:
python-version: '3.6'
- name: Cache pip
uses: actions/cache@v2
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-flake8-${{ hashFiles('requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-flake8-
${{ runner.os }}-
- name: Install requirements
run: pip install flake8 pycodestyle
- name: Check syntax
run: flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ckan
run: flake8

test:
needs: lint
strategy:
matrix:
ckan-version: [2.9, 2.9-py2, 2.8, 2.7]
ckan-version: ["2.10", 2.9, 2.9-py2, 2.8, 2.7]
fail-fast: false

name: CKAN ${{ matrix.ckan-version }}
Expand All @@ -37,7 +32,7 @@ jobs:
image: openknowledge/ckan-dev:${{ matrix.ckan-version }}
services:
solr:
image: ckan/ckan-solr-dev:${{ matrix.ckan-version }}
image: ckan/ckan-solr:${{ matrix.ckan-version }}
postgres:
image: ckan/ckan-postgres-dev:${{ matrix.ckan-version }}
env:
Expand All @@ -46,7 +41,7 @@ jobs:
POSTGRES_DB: postgres
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
redis:
image: redis:3
image: redis:3
env:
CKAN_SQLALCHEMY_URL: postgresql://ckan_default:pass@postgres/ckan_test
CKAN_DATASTORE_WRITE_URL: postgresql://datastore_write:pass@postgres/datastore_test
Expand All @@ -55,38 +50,20 @@ jobs:
CKAN_REDIS_URL: redis://redis:6379/1

steps:
- uses: actions/checkout@v2

- name: Cache pip
uses: actions/cache@v2
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-${{ hashFiles('*requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-
- uses: actions/checkout@v3
- name: Install requirements
run: |
pip install -r requirements-dev.txt
pip install -r dev-requirements.txt
pip install -e .
# Replace default path to CKAN core config file with the one on the container
sed -i -e 's/use = config:.*/use = config:\/srv\/app\/src\/ckan\/test-core.ini/' test.ini
- name: Setup extension (CKAN >= 2.9)
if: ${{ matrix.ckan-version != '2.7' && matrix.ckan-version != '2.8' }}
run: |
ckan -c test.ini db init
ckan -c test.ini report initdb
ckan -c test.ini report generate tagless-datasets
- name: Setup extension (CKAN < 2.9)
if: ${{ matrix.ckan-version == '2.7' || matrix.ckan-version == '2.8' }}
run: |
paster --plugin=ckan db init -c test.ini
paster --plugin=ckanext-report report initdb -c test.ini
paster --plugin=ckanext-report report generate tagless-datasets -c test.ini
CKAN_CLI=bin/ckan_cli
chmod u+x $CKAN_CLI
export CKAN_INI=test.ini
$CKAN_CLI db init
PASTER_PLUGIN=ckanext-report $CKAN_CLI report initdb
PASTER_PLUGIN=ckanext-report $CKAN_CLI report generate tagless-datasets
- name: Run tests
run: pytest --ckan-ini=test.ini --cov=ckanext.report --disable-warnings ckanext/report/tests
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[![Tests](https://github.com/qld-gov-au/ckanext-report/actions/workflows/test.yml/badge.svg)](https://github.com/qld-gov-au/ckanext-report/actions/workflows/test.yml)
ckanext-report
# ckanext-report
====================

ckanext-report is a CKAN extension that provides a reporting infrastructure. Here are the features offered:
Expand Down Expand Up @@ -28,7 +28,9 @@ TODO:
| 2.6 and earlier | yes |
| 2.7 | yes |
| 2.8 | yes |
| 2.9 | yes |
| 2.9-py2 | yes |
| 2.9 (py3) | yes |
| 2.10 (py3) | yes |

Status: was in production at data.gov.uk around 2014-2016, but since that uses its own CSS rather than core CKAN's, for others to use it CSS needs adding. For an example, see this branch: see https://github.com/GSA/ckanext-report/tree/geoversion

Expand All @@ -43,7 +45,8 @@ Install ckanext-report into your CKAN virtual environment in the usual way:

Initialize the database tables needed by ckanext-report:

(pyenv) $ paster --plugin=ckanext-report report initdb --config=mysite.ini
CKAN < 2.9 (pyenv) $ paster --plugin=ckanext-report report initdb --config=mysite.ini
CKAN >= 2.9 (pyenv) $ ckan -c mysite.ini report initdb

Enable the plugin. In your config (e.g. development.ini or production.ini) add ``report`` to your ckan.plugins. e.g.:

Expand Down Expand Up @@ -151,7 +154,7 @@ data - other data values, as a dict
<li>Average tags per package: {{ data['average_tags_per_package'] }} tags</li>
</ul>

<table class="table table-bordered table-condensed tablesorter" id="report-table" style="width: 100%; table-layout:fixed; margin-top: 8px;">
<table class="table table-bordered table-condensed" id="report-table" style="width: 100%; table-layout:fixed; margin-top: 8px;">
<thead>
<tr>
<th>Dataset</th>
Expand Down Expand Up @@ -230,10 +233,7 @@ class TaglessReportPlugin(p.SingletonPlugin):
The last line refers to `tag_report_info` which is a dictionary with properties of the report. This is stored in `reports.py` together with the report code (see above). The info dict looks like this:

```python
try:
from collections import OrderedDict # from python 2.7
except ImportError:
from sqlalchemy.util import OrderedDict
from collections import OrderedDict
tagless_report_info = {
'name': 'tagless-datasets',
'description': 'Datasets which have no tags.',
Expand Down
75 changes: 75 additions & 0 deletions bin/ckan_cli
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/bin/sh

# Call either 'ckan' (from CKAN >= 2.9) or 'paster' (from CKAN <= 2.8)
# with appropriate syntax, depending on what is present on the system.
# This is intended to smooth the upgrade process from 2.8 to 2.9.
# Eg:
# ckan_cli jobs list
# could become either:
# paster --plugin=ckan jobs list -c /etc/ckan/default/production.ini
# or:
# ckan -c /etc/ckan/default/production.ini jobs list

# This script is aware of the VIRTUAL_ENV environment variable, and will
# attempt to respect it with similar behaviour to commands like 'pip'.
# Eg placing this script in a virtualenv 'bin' directory will cause it
# to call the 'ckan' or 'paster' command in that directory, while
# placing this script elsewhere will cause it to rely on the VIRTUAL_ENV
# variable, or if that is not set, the system PATH.

# Since the positioning of the CKAN configuration file is central to the
# differences between 'paster' and 'ckan', this script needs to be aware
# of the config file location. It will use the CKAN_INI environment
# variable if it exists, or default to /etc/ckan/default/production.ini.

# If 'paster' is being used, the default plugin is 'ckan'. A different
# plugin can be specified by setting the PASTER_PLUGIN environment
# variable. This variable is irrelevant if using the 'ckan' command.

CKAN_INI="${CKAN_INI:-/etc/ckan/default/production.ini}"
PASTER_PLUGIN="${PASTER_PLUGIN:-ckan}"
# First, look for a command alongside this file
ENV_DIR=$(dirname "$0")
if [ -f "$ENV_DIR/ckan" ]; then
COMMAND=ckan
elif [ -f "$ENV_DIR/paster" ]; then
COMMAND=paster
elif [ "$VIRTUAL_ENV" != "" ]; then
# If command not found alongside this file, check the virtualenv
ENV_DIR="$VIRTUAL_ENV/bin"
if [ -f "$ENV_DIR/ckan" ]; then
COMMAND=ckan
elif [ -f "$ENV_DIR/paster" ]; then
COMMAND=paster
fi
else
# if no virtualenv is active, try the system path
if (which ckan > /dev/null 2>&1); then
ENV_DIR=$(dirname $(which ckan))
COMMAND=ckan
elif (which paster > /dev/null 2>&1); then
ENV_DIR=$(dirname $(which paster))
COMMAND=paster
else
echo "Unable to locate 'ckan' or 'paster' command" >&2
exit 1
fi
fi

if [ "$COMMAND" = "ckan" ]; then
# adjust args to match ckan expectations
COMMAND=$(echo "$1" | sed -e 's/create-test-data/seed/')
echo "Using 'ckan' command from $ENV_DIR with config ${CKAN_INI} to run $COMMAND..." >&2
shift
exec $ENV_DIR/ckan -c ${CKAN_INI} $COMMAND "$@" $CLICK_ARGS
elif [ "$COMMAND" = "paster" ]; then
# adjust args to match paster expectations
COMMAND=$1
echo "Using 'paster' command from $ENV_DIR with config ${CKAN_INI} to run $COMMAND..." >&2
shift
if [ "$1" = "show" ]; then shift; fi
exec $ENV_DIR/paster --plugin=$PASTER_PLUGIN $COMMAND "$@" -c ${CKAN_INI}
else
echo "Unable to locate 'ckan' or 'paster' command in $ENV_DIR" >&2
exit 1
fi
2 changes: 2 additions & 0 deletions ckanext/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# encoding: utf-8

# this is a namespace package
try:
import pkg_resources
Expand Down
37 changes: 37 additions & 0 deletions ckanext/report/blueprint.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# encoding: utf-8

import six

from flask import Blueprint, make_response

from ckan.plugins import toolkit

from .utils import report_index as index, report_view


report = Blueprint(u'report', __name__)


def redirect_to_index():
return toolkit.redirect_to('/report')


def view(report_name, organization=None, refresh=False):
body, headers = report_view(report_name, organization, refresh)
if headers:
response = make_response(body)
for key, value in six.iteritems(headers):
response.headers[key] = value
return response
else:
return body


report.add_url_rule(u'/report', 'index', view_func=index)
report.add_url_rule(u'/reports', 'reports', view_func=redirect_to_index)
report.add_url_rule(u'/report/<report_name>', view_func=view, methods=('GET', 'POST'))
report.add_url_rule(u'/report/<report_name>/<organization>', 'org', view_func=view, methods=('GET', 'POST',))


def get_blueprints():
return [report]
57 changes: 57 additions & 0 deletions ckanext/report/cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# encoding: utf-8

import click

from . import utils


def get_commands():
return [report]


@click.group()
def report():
"""Generates reports"""
pass


@report.command()
def initdb():
"""Creates necessary db tables"""
utils.initdb()
click.secho(u'Report table is setup', fg=u"green")


@report.command()
@click.argument(u'report_list', required=False)
def generate(report_list):
"""
Generate and cache reports - all of them unless you specify
a comma separated list of them.
"""
if report_list:
report_list = [s.strip() for s in report_list.split(',')]
timings = utils.generate(report_list)

click.secho(u'Report generation complete %s' % timings, fg=u"green")


@report.command()
def list():
""" Lists the reports
"""
utils.list_reports()


@report.command()
@click.argument(u'report_name')
@click.argument(u'report_options', nargs=-1)
def generate_for_options(report_name, report_options):
"""
Generate and cache a report for one combination of option values.
You can leave it with the defaults or specify options
as more parameters: key1=value key2=value
"""
message = utils.generate_for_options(report_name, report_options)
if message:
click.secho(message, fg=u"red")
Loading

0 comments on commit 774331c

Please sign in to comment.