Update docker2singularity to Singularity v2.5.1 #30
Conversation
vsoch
changed the title
|
This will close #26 |
|
Note that I just revamped the argument parsing to add the option to give a custom name to the container: docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/ou
tput --privileged -t --rm docker2singularity --name meatballs ubuntu:14.04
Image Format: squashfs
Docker Image: ubuntu:14.04
Container Name: meatballs
Inspected Size: 223 MB
(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
Building image from sandbox: /tmp/meatballs.build
Building Singularity image...
Singularity container built: /tmp/meatballs.simg
Cleaning up...
(10/10) Moving the image to the output folder...
65,077,279 100% 395.10MB/s 0:00:00 (xfr#1, to-chk=0/1)
Final Size: 63MB
$ ls /tmp/test/
meatballs.simgAlso notice more verbosity up front to tell the user about the choices. This will close #29 |
|
I also just updated the usage so its a single function, and displays cleanly: Here is with $ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/ou
tput --privileged -t --rm docker2singularity --help
USAGE: docker2singularity [-m "/mount_point1 /mount_point2"] [options] docker_image_name
OPTIONS:
Image Format
--folder -f build development sandbox (folder)
--writable -w non-production writable image (ext3)
Default is squashfs (recommended)
--name -n provide basename for the container (default based on URI)
--mount -m provide list of custom mount points (in quotes!)
--help -h show this help and exit
and with no arguments, does the same tput --privileged -t --rm docker2singularity
USAGE: docker2singularity [-m "/mount_point1 /mount_point2"] [options] docker_image_name
OPTIONS:
Image Format
--folder -f build development sandbox (folder)
--writable -w non-production writable image (ext3)
Default is squashfs (recommended)
--name -n provide basename for the container (default based on URI)
--mount -m provide list of custom mount points (in quotes!)
--help -h show this help and exit
|
|
Many thanks for doing this @vsoch - especially so fast, it looks great! I just tried checking out this branch and building the container, then running a test but got lots of Any ideas what's going on here? Thanks, Phil |
|
I think we are seeing this: apptainer/singularity#1570 |
|
As in the linked Singularity issue - I'm pretty sure this is not a Singularity bug. Under certain conditions, tar run under docker, with overlayfs as the storage driver, will have this issue due to an overlayfs bug. Notice here that Mac docker is being used. Try....
|
|
Thanks @dctrud ! I was just writing you a note here but you are too speedy :) @ewels would you care to try this fix and update us on if it resolves the issue? |
|
Thanks both! I'll take a look. It makes me a little anxious though, as I'm writing this into a general use tool for others to use. Is it likely that everyone (on Mac?) will have to do the same fix? |
|
Yeah it's definitely not ideal - I use Ubuntu but I would guess that other Mac users would need to do this as well (given that most Macs are the same).For your tool, is there any reason to not try using singularity natively, or within Vagrant without the need for Docker? |
|
Nothing major - I'm trying to write a helper tool to reduce the work required to fetch singularity images for our nextflow pipelines (http://nf-co.re/). We and many of our users use clusters with no internet connection, so you have to download the pipeline and containers locally first and transfer them via whatever secure method. The majority of people don't already have singularity installed locally, and to be honest they don't really need it installed locally if all they're doing is pulling a container. So in my helper tool I try with singularity first and if that's not installed I was planning to use docker if available. But people need to start playing around with docker to get it to work with this then that sort of defeats the purpose of making it super user-friendly and easy. I'm now wondering if this is all overkill and I need to take a step back. If we start putting singularity images up on singularity-hub then presumably we could just use the API to get a download URL and pull that directly from the Python helper without any build processes. This would be far simpler. So, going off on a tangent now, but is there an easy way to set up automated builds on singularity hub from Docker build scripts? We definitely want to have support for both Docker and Singularity in the pipelines, and I'd prefer to have to maintain only one set of build scripts instead of two if possible. |
|
If you are using singularity in the nextflow pipeline, you need it installed. Having docker in there also means needing that installed (and maybe it already is?) but giving user space access introduces a security issue. I would try first the simpler of the two solutions - which is using singularity to pull the images first. As for docker and singularity compatability, I would actually host them on Docker Hub and then provide the docker URI to singularity (e.g., |
The nextflow pipeline is running on the HPC, where singularity is installed, yes. But the HPC has no internet connection, so the normal
Yup, my approach is to use whatever is installed - try using singularity first, then try docker, then throw an error.
I don't really understand this sorry 😕
I absolutely agree 😀 This is what I've done - see nf-core/tools/pull/53 and relevant code
Yup, this is also what we're doing already. It works beautifully! ✨In fact, nextflow does this for us. But this requires a
This is cool! I didn't know that this was possible, or at least as easy as the docs makes it sound! I am sort of trying to avoid using any non-freeware solutions for the nf-core project as it is a community-based collaborative effort and I'd prefer to avoid relying on any one group's resources. However, if we manage to get any money for the project (I applied for a small grant just recently) then we could perhaps set something up. |
What do you mean "non freeware?" Singularity Registry is openly available and free for use, no paying and the code is all there for you to use. Any changes / updates that you need, akin to with Singularity , you can post issues and I respond pretty quickly. Having more people to contribute and help would make it stronger! Another idea that would actually work but I haven't played around with it is to (somehow, somewhere) expose the raw download links to the containers on Singularity Hub. They are JUST files on Google storage, and the singularity client simply gets them with pull by making a request to the Singularity Hub API that has the manifests. Arguably, if we had a reasonable solution to serve this information elsewhere, there isn't any reason you couldn't do the same. For most containers, you can find the metadata endpoint at a uri like |
|
Oh! Another option (but another dependency) is to install Singularity Global Client to do the pull --> https://github.com/singularityhub/sregistry-cli it will run without Singularity and make those same API calls to pull the image. A bonus is that it acts as a little local database manager, so a user that pulls an image once can then easily see where / what images they have pulled before. |
Sorry, poor choice of words. I meant that we need to run a server for this, which takes resources. This isn't impossible, but so far we've got by with GitHub / Travis / Docker Hub etc which all work without this.
Yes - this was exactly my idea too 😁 I was playing around with exactly this earlier this morning and getting super excited at how good the API is and how easy it would be to call the API from within my little helper tool and access the raw download link. I had found my way to https://www.singularity-hub.org/api/containers/2157/ but it's the same thing I think. Singularity Global Client looks properly cool - this could definitely be useful (though could potentially be a bit overkill for what could be about five lines of python).
Sure - I don't think that we would be operating on that scale here. This is going to be one-off manual use by people before they move to another system for their analysis, so downloads will be in the tens or hundreds per month max. I got to this point after my initial post above and latter today on gitter we were discussing how to make it happen (preferably without needing to co-maintain both a |
|
@ewels I know this wasn't for your exact use case, but would you have a few moments to test the PR against some other images and let us know if it works for you? |
If it can help, I tested it on several images and it worked perfectly. |
|
@chrisfilo now given the security issues with the older versions, do you have issue merging this PR to be default for master? |
|
Please go ahead. |
|
I'm no longer maintainer on this repository - or singularity-python which is primarily for Singularity Hub. @gmkurtzer would you care to explain why? |
|
And we are back! @chrisfilo I added a review for you so you can (officially) approve and then I'll merge into master. Thanks @ewels for confirmation that things look good. |
|
@vsoch I don't have the capacity to review and test this PR, but please go ahead with merging if you wish |
|
Ok no worries! I'm going to set up a circle build and deploy for this repo and I'll use this PR to test it again. Thanks for your past review @chrisfilo! |
|
holy crap, did that just work on the first try? THIS HAS NEVER HAPPENED BEFORE. Excuse me while I go and look out the window for a flying 🐷 ... :) |
|
Note that:
|
hey @chrisfilo and @ewels and @0xaf1f (interested parties!) here is an update for Singularity 2.5.1, if you'd care to give it a test. I've closed the PR for 2.4 (and it can live on the v2.4 branch). I've also added a changelog for (human notes) to coincide with these updates.