-
Notifications
You must be signed in to change notification settings - Fork 35
Khushnaseeb Log
khushnaseeb edited this page Aug 22, 2018
·
18 revisions
On Fridays, please devote at least a half hour to reflect on the work you've done over the previous week. Please use this format:
- Took a Reprozip overview and tried unnpacking the digits-sklearn-opencv example on the local machine from the reprozip-examples repository.
- Went through Reprozip developer doc and tried to understand how to write an unpacker. Reported a broken link in the documentation on the Dynamic discovery of Services and Plugins.
- Went through the following links and videos to understand Containers and Docker
- Jumpstarted with the official Docker documentation to get hands-on on Docker.
- Learned to how to create images and containers, run a container and publishing an image to Docker Hub.
- Created services using docker-compose and scaled the application using it.
- Learned how to put placement constraints and attach volumes so as to persist data even when the container is destroyed
- Learned what a swarm is, how nodes in swarms can be managers or workers, created a swarm, and deployed an application on it
- Created and configure a Stack(a group of interrelated services) using Docker compose and added services to it
- Tried to get a basic understanding of Singularity via following videos:
- Tried to understand the key differences between Singularity and Docker and why the former is a good choice for HPC. Related links:
- Went through Singularity official documentation to try out some examples.
- Jump started with Singularity official documentation to try out the following
- building a singularity container image fro docker, shub, and directory
- building a writable image
- Understanding and writing a singularity recipe file
- Understanding the singularity workflow and building apps
- creating and running singularity services/instances
- Making persistent singularity overlays and learning how to bind mounts and paths to enable read and write files on the host system from within the container.
- learned how to make singularity apps and how they share environment variables
- understanding singularity's environment variables
- had a discussion to come with a design to leverage singularity for reprozip: -- reproducing singularity image directory from reperozip by adding additional singularity files --- using app structure in singularity for replicating runs in reprozip
- Made and ran a singularity image by hand from a reprozip package in a sandbox format.
- Wrote a bash script to automatically create and run the singularity image for a single run reprozip package
- Tried emulating multiple runs via singularity apps feature.
- Converted the bash script to python.
- Modified the script for multiple runs feature (without using singularity apps) and successfully tested it on digits_sklearn_opencv.rpz package.
- Tried to make the singularity image for the reprozip package in the TAR format.
- Successfully used Singularity's app feature to emulate multiple runs in a reprozip package for example in digits_sklearn_openc.rpz example - two separate apps for classification and prediction both which can be run independently (but in order, if no dependencies are present)
- Successfully wrote a script to automatically create and run the singularity image for the reprozip package in a tar format.
- Tested the script on various packages - single and multiple runs.
- Submitted for code review and implemented code review changes
- Reimplemented the tar functionality:
- removed any files that were generated on disk. Wrote code for creating everything in memory and adding the contents as a file to the tar file using BytesIO.
- Added empty folders directly to the tar file without having the need of them on the disk.
- Added the functionality of running the tar image of multiple runs directly and not just running the individual run as an app. This allows one to upload the singularity image of singularity hub and run without the reprozop unpacker script.
- Tested the unpacker ( both single and multiple run) on HPC and it worked!
- Fixed the upload and download issue - now able to download/upload to the singularity image from anywhere to the host to anywhere in the container and vice versa.
- Removed manual adding of bin/sh. Added the functionality to download the busybox url in memory and then add it as a file to the tar file.
- Understanding MPI and how MPI applications run with singularity. Tried running a reprozip package with MPI.
- Documented singularity unpacker for future use with reprozip.