Merge pull request #14 from alces-software/develop

Release the 0.2.0 version of anvil

README.md

@@ -47,81 +47,38 @@ this isn't very important right now. In general we need to think about versionin
 content items such as Gridware and customizers, and also in terms of Clusterware version
 compatibilities.)
 
-# Creating a Database Snapshot
+# Importing local packages into the database
+It is possible to import packages into the database from a directory. The
+directory is searched recursively for all `.zip` files. The packages are
+directly added to the database without going through the upload and thus
+do not require sign in credentials.
 
-The database snapshot can be triggered manually using:
+The import is preformed use rake:
 ```
-# cd ./anvil
-# rake packages:snapshot
-Which IP/domain are the packages hosted on?
-> www.example.com
+rake packages:import
 ```
 
-Running this command with no other system configurations sets up the
-default server, however it still needs to be told which ip/ domain the
-packages will be hosted on. See environment setup for more details
-
-NOTE: The answer should not include the protocol as it will default to
-`http`. However the underlining `ANVIL_BASE_URL` needs to be fully
-qualified including the protocol.
-
-## Environment Setup
-
-Running the `snapshot` rake command does not alter your environment setup,
-it does however use the following environment variables internally
-
-```
-ANVIL_LOCAL_DIR: The base directory for the download. Typically this would
-                 be hosted on the rails in built `public` directory or
-                 hosted by apache/ enginx. Anvil will store the files in
-                 a `packages` sub directory
-                 DEFAULT: /path/to/anvil/public
-
-ANVIL_UPSTREAM:  The upstream anvil database to take the snapshot on. It
-                 defaults to the main production database.
-                 DEFAULT: https://forge-api.alces-flight.com
-
-ANVIL_BASE_URL:  A fully qualified URL (inc. protocol) to where the
-                 packages will be hosted. This will be stored in the db
-                 with the package metadata. Thus it does not need to be
-                 permanetly set within the environment. There is no default
-```
-
-## Drop DB and Running the Snapshot Manually
-
-If the above environment variable are set, then `rake package:snapshot`
-will automatically download and import the database in a single set.
-
-The automatic snapshot will not work if the database already exists as
-it can not recover from any errors. Instead the snapshot should be
-preformed step by step using the commands bellow.
-
-It does require the above environment variables to be set manually as well,
-refer to `lib/rake/packages.rb` for further details.
-
-```
-rake snapshot:download
-rake snashot:import
-```
-
-Alternatively the database could be dropped with:
-```
-rake db:drop
-```
-
-## Running the Server
-
-You are now ready to start the `anvil` rails server with the command bellow.
-This will likely launch in development mode if no further configurations are
-made. This means the `public` directory should be statically served without
-any futher configurations.
-
-However there is no reason why the packages need to be hosted on this
-machine, the packages can be hosted anywhere as long as the
-`ANVIL_BASE_URL` was set correctly when the import occurred.
-NOTE: Changes to the base url after the fact will be ignored
+In order for the packages to be added correctly, the base url and package
+directory must be set as environment variables:
 
 ```
-bin/rails server -p 80 -b 0.0.0.0
+ANVIL_BASE_URL:   A fully qualified URL (inc. protocol) to where the
+                  packages will be hosted. This will be stored in the db
+                  with the package metadata. Thus it does not need to be
+                  permanetly set within the environment.
+
+ANVIL_IMPORT_DIR: The directory the packages are imported from. The import
+                  command does not concern itself with the hosting of the
+                  packages. It assumes the packages will be found at:
+                  http://$ANVIL_BASE_URL/packages/<relative-package-path>
+
+                  Typically packages will be hosted from within the anvil
+                  public directory. Thus ANVIL_IMPORT_DIR would normally
+                  be set to: /<path-to>/anvil/public/packages
+
+                  However a different static file server could be used
+                  (e.g. nginx). In this case the import dir should be set
+                  accordingly. It is the responsibility of the user to
+                  ensure the package is reachable at the above address.
 ```
 

lib/tasks/packages.rake

@@ -7,29 +7,11 @@ require 'highline/import'
 require_relative File.join(ENV['FL_ROOT'], 'lib/flight_direct/version.rb')
 
 namespace :packages do
-  desc 'Downloads all the rake packages'
-  task download: :environment do
-    raise 'The ANVIL_UPSTREAM has not been set' unless ENV['ANVIL_UPSTREAM']
-    raise 'The ANVIL_LOCAL_DIR has not been set' unless ENV['ANVIL_LOCAL_DIR']
-    packages = JSON.parse(
-      Net::HTTP.get(URI("#{ENV['ANVIL_UPSTREAM']}/v1/packages")),
-      object_class: OpenStruct
-    )
-    Parallel.map(packages.data, in_threads: 10) do |metadata|
-      uri = URI.parse(metadata.attributes.packageUrl)
-      puts "Downloading: #{uri.to_s}"
-      path = package_path(URI.unescape(uri.path))
-      FileUtils.mkdir_p File.dirname(path)
-      File.open(path, "wb") do |save_file|
-        open(uri.to_s) { |line| save_file.write(line.read) }
-      end
-    end
-  end
-
   desc 'Import packages from a local source'
   task import: :environment do
-    raise 'The ANVIL_LOCAL_DIR has not been set' unless ENV['ANVIL_LOCAL_DIR']
-    raise 'The ANVIL_BASE_URL has not been set' unless ENV['ANVIL_BASE_URL']
+    ['ANVIL_IMPORT_DIR', 'ANVIL_BASE_URL'].each do |env|
+      raise "The #{env} has not been set" unless ENV[env]
+    end
     files = Dir[package_path('**/*.zip')]
     files.define_singleton_method(:delete_if_saveable) do
       self.delete_if { |f| add_package_from_zip_path(f) }
@@ -42,82 +24,13 @@ namespace :packages do
     ERROR
   end
 
-  desc 'Download and import the packages'
-  task snapshot: :environment do
-    exit_if_db_exists('snapshot')
-    raise 'The ANVIL_BASE_URL has not been set' unless ENV['ANVIL_BASE_URL']
-    ENV['ANVIL_UPSTREAM'] ||= 'https://forge-api.alces-flight.com'
-    ENV['ANVIL_LOCAL_DIR'] ||= File.expand_path(File.join(
-      File.dirname(__FILE__), '..', '..', 'public'
-    ))
-
-    # Downloads the flight direct bootstrap script
-    puts 'Downloading FlightDirect bootstrap'
-    bootstrap_url = 'https://raw.githubusercontent.com/alces-software/flight-direct/master/scripts/bootstrap.sh'
-    bootstrap_path = File.join(ENV['ANVIL_LOCAL_DIR'],
-                               'flight-direct',
-                               'bootstrap.sh')
-    download(bootstrap_url, bootstrap_path)
-
-    # Sets the anvil_url in the bootstrap script
-    bootstrap_content = File.read(bootstrap_path)
-    new_bootstrap_content = bootstrap_content.gsub(
-      /# anvil_url=/, "anvil_url=#{ENV['ANVIL_BASE_URL']}"
-    )
-    FileUtils.rm(bootstrap_path)
-    File.write(bootstrap_path, new_bootstrap_content)
-
-    # Downloads the installed version of Flight Direct from S3
-    puts 'Downloading FlightDirect tarball'
-    fd_url = "https://s3-eu-west-1.amazonaws.com/flight-direct/releases/el7/flight-direct-#{FlightDirect::VERSION}.tar.gz"
-    fd_path = File.join(ENV['ANVIL_LOCAL_DIR'],
-                        'flight-direct/flight-direct.tar.gz')
-    download(fd_url, fd_path)
-
-    # Downloads the git packages
-    ['clusterware-sessions', 'clusterware-storage',
-     'gridware-packages-main', 'packager-base', 'gridware-depots'
-    ].each do |repo|
-      url = "https://github.com/alces-software/#{repo}.git"
-      source = "/tmp/repos/#{repo}"
-      target = File.join(ENV['ANVIL_LOCAL_DIR'], 'git', "#{repo}.tar.gz")
-      print `rm -rf #{source} #{target}`
-      print `mkdir -p #{File.dirname(target)}`
-      puts `git clone #{url} #{source}`
-      puts `tar --warning=no-file-changed -C #{source} -czf #{target} .`
-    end
-    # Renames packager-base to be the volatile repo
-    FileUtils.mv File.join(ENV['ANVIL_LOCAL_DIR'], 'git', 'packager-base.tar.gz'),
-      File.join(ENV['ANVIL_LOCAL_DIR'], 'git', 'gridware-packages-volatile.tar.gz')
-
-    Rake::Task['db:setup'].invoke
-    puts 'Downloading packages...'
-    Rake::Task['packages:download'].invoke
-    puts 'Importing the packages...'
-    Rake::Task['packages:import'].invoke
-    puts 'Done'
-  end
-
   def package_path(relative_path)
-    File.join(ENV['ANVIL_LOCAL_DIR'], 'packages', relative_path)
+    File.join(ENV['ANVIL_IMPORT_DIR'], relative_path)
   end
 
   def extract_package_url(absolute_path)
-    relative_path = absolute_path&.sub(ENV['ANVIL_LOCAL_DIR'], '')
-    File.join(ENV['ANVIL_BASE_URL'], relative_path)
-  end
-
-  def exit_if_db_exists(action)
-    ActiveRecord::Base.connection
-    $stderr.puts <<~ERROR.squish
-      `rake packages:#{action}` can not be ran once the db has been setup.
-      Either run `rake db:drop` to delete the database, or perform the
-      #{action} manually
-    ERROR
-    exit 1
-  rescue ActiveRecord::NoDatabaseError
-    # The database doesn't exist so do nothing
-    return
+    relative_path = absolute_path&.sub(ENV['ANVIL_IMPORT_DIR'], '')
+    File.join(ENV['ANVIL_BASE_URL'], 'packages', relative_path)
   end
 
   def add_package_from_zip_path(zip_path)
@@ -127,12 +40,4 @@ namespace :packages do
       user: user, package_url: url, file: zip_path
     ).save
   end
-
-  def download(url, path)
-    FileUtils.mkdir_p File.dirname(path)
-    case io = open(url)
-    when StringIO then File.open(path, 'w') { |f| f.write(io.read) }
-    when Tempfile then FileUtils.mv(io.path, path)
-    end
-  end
 end