Simplifying Batch Jobs with Moab Container Support

In this article I’ll continue my discussion of Linux containers using Docker and I will show how using Moab container jobs can help users of your system by simplifying the process of starting a batch job within a container. If you’re just coming up to speed on the notion of Linux containers, I encourage you to read my previous posts where I introduced containers here <provide link> and presented a brief Docker tutorial here <provide link>.

One issue that I didn’t elaborate on in the tutorial was the origin of Docker images. I showed how to make an image using “docker commit” but it was created from an image that came from somewhere else. That “somewhere else” is the Docker Hub, a registry for Docker images that is used to download images onto your system and then use the downloaded image with “docker run”. Let’s look at this using an example from the tutorial in my previous post.

When you execute “docker run hello-world” on the command line, Docker looks for the image “hello-world” locally on your system. If it is present, it is started immediately. However, if it is not present, Docker goes to the Docker Hub to look for the image, downloads it, and then starts it up as a container. When you are ready to move beyond hello-world, you might want to browse Docker Hub. A good place to start is a listing of official repositories: Once you’ve found an image you want, you can use it with “docker run” (again refer back to the tutorial for a refresher on using “docker run” if you need it).

I’d encourage you to try out a few images you find. Note that container images from past runs tend to accumulate on the host system so you’ll want to occasionally remove them so you don’t fill up your file system. Containers that have ended consume file system storage and can be cleaned up with “docker rm <container-id>” which frees the consumed storage. You can get a list of all of them using “docker ps -a”. Images that have been downloaded or created (by using “docker commit”, for example) also consume file system storage and can be removed using “docker rmi <image-id>”. You can get a list of them by using “docker images”.

As you can see, there is some setup and teardown required when using Docker containers. There’s also the undesirable issue of having to run as the root user within a container that I mentioned in my last post. To see how these issues can be handled gracefully, I’d like to show you some new functionality with Moab 9.0. This new functionality also greatly simplifies the use of containers for users of your system.

Using Moab, you can run a batch job within a Docker container of your choice and let Moab do all the work to coordinate setup and teardown (including removal) of the job container. Moab also takes care of the root user problem by setting up the job container to be run by the job owner rather than root. You can even specify that your container jobs may be checkpointed and restarted.

Let’s go through an example so you can see how this works.

First, an interactive job is submitted as user “adaptive” where an “id” command is executed to show the user id within the container. Next, a “cat /etc/os-release” is executed to show the distribution name.


$ msub -I -l ubuntu1404template

qsub: waiting for job 1115.dtest to start
qsub: job 1115.dtest ready

Preparing container for job.

adaptive@ubuntu1404:~$ id
uid=1000(adaptive) gid=1000(adaptive) groups=1000(adaptive),10(wheel)

adaptive@ubuntu1404:~$ cat /etc/os-release
VERSION=”14.04.2 LTS, Trusty Tahr”
PRETTY_NAME=”Ubuntu 14.04.2 LTS”
adaptive@ubuntu1404:~$ exit

Removing container for job.

qsub: job 1115.dtest completed
As you can see, the job container was started as user “adaptive” and ran in an Ubuntu 14.04 container. We can do a similar thing with a non-interactive job and output is returned to a file as with other regular Moab jobs.

In addition to using container images from the Docker Hub, users can arrange to have their own special container images setup through Moab so that future job submissions can request them. This enables users to have a custom container established with all necessary libraries, etc. for their code to run within the container job.

Using Moab with Docker container support, it is also possible to establish a local, site-specific container image registry and to preempt a running container job by checkpointing the container (using “docker commit”) and saving the image to the local registry. The job may be restarted later on a different node in a clustered system using the checkpointed image from the local registry. Since containers save their file system state (but not the process state of running processes), changes made to the container file system may be preserved across a checkpoint/restart operation. Users can leverage this functionality by setting up their jobs to pick up where they last left off when their jobs were preempted by Moab or were manually checkpointed.

In conclusion I hope you now have a better understanding of Linux containers and how valuable they can be on an HPC system. Using Moab 9.0 can simplify the integration of Docker containers with your system and can help your users unleash the power of them using the existing Moab batch job interface they may already use and understand.

Series Navigation<< An Introduction to Basic Docker Commands