Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to s3fs so we can have multiple instances #223

Merged
merged 6 commits into from
Apr 23, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .bp-config/httpd/extra/httpd-modules.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# This file was built from
# https://github.com/cloudfoundry/php-buildpack/blob/master/defaults/config/httpd/extra/httpd-modules.conf
# For more detail, see the description in
# https://docs-cloudfoundry-staging.cfapps.io/buildpacks/php/gsg-php-config.html#engine-configurations

LoadModule authz_core_module modules/mod_authz_core.so
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comment explaining where this file came from.

LoadModule authz_host_module modules/mod_authz_host.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule dir_module modules/mod_dir.so
LoadModule mime_module modules/mod_mime.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule remoteip_module modules/mod_remoteip.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule filter_module modules/mod_filter.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so

# Also enable proxy_http for our s3fs module
LoadModule proxy_http_module modules/mod_proxy_http.so
2 changes: 2 additions & 0 deletions .docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ RUN apt-get update &&\
rm -rf /var/lib/apt/lists/* &&\
rm -rf /var/cache/apt/*

RUN a2enmod rewrite proxy proxy_http
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention needing docker-compose build.


COPY ["install-composer.sh", "/tmp"]
RUN "/tmp/install-composer.sh"

Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ modules/*
web/modules/contrib/*
web/themes/contrib/*

# Ignore compiled twig templates
storage/php

# Ignore local database dumps.
db_dumps

Expand Down
41 changes: 20 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,32 +71,31 @@ to port to other environments.

### File storage

We've configured our file fields to store content in S3; in this way, they
persist between app restarts and deploys. Unfortunately, those configurations
are therefore also present locally, which can lead to unexpected results (we
*don't* include sensitive bucket credentials, so we'll see "The file could not
be uploaded."). If needing to work with file uploads locally, modify the
relevant field's "storage" away from "Flysystem: S3" to "Public files" (which
means the local disk). This can be configured in the Drupal administration
interface, or by editing the configuration files in
`web/sites/default/config`. Notably, all instances of
By default, we don't use S3 when running Drupal locally. If wanting to
simulate the S3 environment, we need to add our credentials into the
`VCAP_SERVICES` environment variable. Edit `docker-compose.yml` and insert
something similar to the following above "user-provided":

```yaml
uri_scheme: s3
```json
"s3": [{
"name": "storage",
"credentials": {
"access_key_id": "SECRET",
"bucket": "SECRET",
"region": "SECRET",
"secret_access_key": "SECRET"
}
}],
```

should become

```yaml
uri_scheme: public
To find the values we're using in cloud.gov, use
```
cf env web
```

and Docker restarted.

Alternatively, if testing S3 integration, it's possible to configure Docker to
use a real S3 bucket by editing `docker-compose.yml`. That file holds a series
of values under the "s3" key which will need to be modified with your access
credentials.
As with other edits to the local secrets, extra care should be taken when
exporting your config, let those configuration files contain the true secret
values rather than dummy "SECRET" strings.

### Configuration workflow

Expand Down
10 changes: 9 additions & 1 deletion bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ SECRETS=$(echo $VCAP_SERVICES | jq -r '.["user-provided"][] | select(.name == "s
APP_NAME=$(echo $VCAP_APPLICATION | jq -r '.name')
APP_ROOT=$(dirname "${BASH_SOURCE[0]}")

S3_BUCKET=$(echo $VCAP_SERVICES | jq -r '.["s3"][] | select(.name == "storage") | .credentials.bucket')
S3_REGION=$(echo $VCAP_SERVICES | jq -r '.["s3"][] | select(.name == "storage") | .credentials.region')
if [ -n "$S3_BUCKET" ] && [ -n "$S3_REGION" ]; then
# Add Proxy rewrite rules to the top of the htaccess file
sed -i "s/S3_BUCKET/$S3_BUCKET/g" $APP_ROOT/web/.htaccess
sed -i "s/S3_REGION/$S3_REGION/g" $APP_ROOT/web/.htaccess
fi

install_drupal() {
ROOT_USER_NAME=$(echo $SECRETS | jq -r '.ROOT_USER_NAME')
ROOT_USER_PASS=$(echo $SECRETS | jq -r '.ROOT_USER_PASS')
Expand Down Expand Up @@ -32,7 +40,7 @@ if [ "${CF_INSTANCE_INDEX:-''}" == "0" ] && [ "${APP_NAME}" == "web" ]; then
# Mild data migration: fully delete database entries related to these
# modules. These plugins (and the dependencies) can be removed once they've
# been uninstalled in all environments
drupal --root=$APP_ROOT/web module:uninstall masquerade workflow
drupal --root=$APP_ROOT/web module:uninstall flysystem flysystem_s3 masquerade workflow
drupal --root=$APP_ROOT/web theme:uninstall bootstrap

# Sync configs from code
Expand Down
1 change: 1 addition & 0 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
"drupal/media_entity": "^1.7",
"drupal/pathauto": "^1.1",
"drupal/rules": "^3.0@alpha",
"drupal/s3fs": "^3.0@alpha",
"drupal/scheduler": "^1.0",
"drupal/time_formatter": "1.x-dev",
"drupal/tour_ui": "^1.0@alpha",
Expand Down
61 changes: 60 additions & 1 deletion composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 0 additions & 9 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,6 @@ services:
"username": "root"
}
}],
"s3": [{
"name": "storage",
"credentials": {
"access_key_id": "SECRET",
"bucket": "SECRET",
"region": "SECRET",
"secret_access_key": "SECRET"
}
}],
"user-provided": [{
"name": "secrets",
"credentials": {
Expand Down
2 changes: 1 addition & 1 deletion manifest-prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ applications:
- name: web
<<: *defaults
memory: 256M
instances: 1
instances: 2
routes:
- route: nsf-beta.app.cloud.gov
# - route: beta.nsf.gov
Expand Down
2 changes: 1 addition & 1 deletion manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ applications:
- name: web
<<: *defaults
memory: 256M
instances: 1
instances: 2
routes:
- route: nsf-demo.app.cloud.gov
- name: cronish
Expand Down
1 change: 1 addition & 0 deletions web/.htaccess
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#
# Apache/PHP/Drupal settings:
#
RewriteRule ^s3fs-(css|js)/(.*)$ http://S3_BUCKET.s3-S3_REGION.amazonaws.com/public/$2 [P]

# Protect files and directories from prying eyes.
<FilesMatch "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock))$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$">
Expand Down
3 changes: 1 addition & 2 deletions web/sites/default/config/core.extension.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ module:
field_ui: 0
file: 0
filter: 0
flysystem: 0
flysystem_s3: 0
hal: 0
help: 0
history: 0
Expand All @@ -54,6 +52,7 @@ module:
quickedit: 0
rdf: 0
rules: 0
s3fs: 0
scheduler: 0
scheduler_rules_integration: 0
search: 0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ field_name: field_featured_image
entity_type: node
type: image
settings:
uri_scheme: s3
uri_scheme: public
default_image:
uuid: ''
alt: ''
Expand Down
4 changes: 2 additions & 2 deletions web/sites/default/config/system.performance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@ cache:
page:
max_age: 0
css:
preprocess: false
preprocess: true
gzip: true
fast_404:
enabled: true
paths: '/\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i'
exclude_paths: '/\/(?:styles|imagecache)\//'
html: '<!DOCTYPE html><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL "@path" was not found on this server.</p></body></html>'
js:
preprocess: false
preprocess: true
gzip: true
stale_file_threshold: 2592000
_core:
Expand Down
28 changes: 11 additions & 17 deletions web/sites/default/settings.cf.php
Original file line number Diff line number Diff line change
Expand Up @@ -24,23 +24,17 @@
$settings['hash_salt'] = $service['credentials']['HASH_SALT'];
}
if ($service['name'] === 'storage') {
$settings['flysystem']['s3'] = array(
'driver' => 's3',
'config' => array(
'key' => $service['credentials']['access_key_id'],
'secret' => $service['credentials']['secret_access_key'],
'region' => $service['credentials']['region'],
'bucket' => $service['credentials']['bucket'],
// Optional configuration settings.
'options' => array(
'ACL' => 'public-read',
'ServerSideEncryption' => 'AES256',
),
'protocol' => 'https', // Will be autodetected based on the current request.
'prefix' => 'flysystem-s3', // Directory prefix for all uploaded/viewed files.
),
'cache' => TRUE, // Creates a metadata cache to speed up lookups.
);
$config['s3fs.settings']['access_key'] = $service['credentials']['access_key_id'];
$config['s3fs.settings']['bucket'] = $service['credentials']['bucket'];
$config['s3fs.settings']['encryption'] = 'AES256';
$config['s3fs.settings']['public_folder'] = 'public';
$config['s3fs.settings']['private_folder'] = 'private';
$config['s3fs.settings']['region'] = $service['credentials']['region'];
$config['s3fs.settings']['secret_key'] = $service['credentials']['secret_access_key'];
$config['s3fs.settings']['use_https'] = TRUE;
$settings['s3fs.use_s3_for_public'] = TRUE;
// Twig templates _shouldn't_ be in the public dir (lest they be very slow)
$settings['php_storage']['twig']['directory'] = '../storage/php';
}
}
}
Expand Down