[Beowulf] Docker in HPC
pc7 at sanger.ac.uk
Fri Nov 29 02:00:31 PST 2013
On 28/11/13 22:44, Christopher Samuel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> On 28/11/13 23:13, Bogdan Costescu wrote:
>> f.e. a single version of GROMACS
> But then you run into the issue of different users who say "but I must
> have GROMACS x.y.z", hence why we've already got GROMACS 4.0.7, 4.5.7
> and 4.6.3 installed on a system that was a clean install only a few
> months ago. :-/ At least the person who uses GROMACS 3.x hasn't
> asked for it (yet).
> Then there are the ones who have a patch they want, or built with
> different (mutually exclusive) options (for Ray).
This is very much our software challenge in a nutshell (ignoring our
security aspects mentioned).
Person A requires historical versions for their own reasons where as
person B really needs twig cvs release for their stack to operate. In
general though the compilation step tends to be the easy part. The
challenge often comes at run time where various groups configure their
project area to have something along the lines of:
/software/project-a/bin/mybin -> /software/project-a/mybin-2.6.1/bin/mybin
If 2 projects have similar structures then getting the runtime
environment right is really rather important and becomes more exciting
where perl / python modules are involved...
Sadly it is a constant challenge to encourage users to execute their
jobs via a wrapper script which sets specifically the environment
required for the jobs to run within. So running jobs via Docker with a
pre-defined hard configured environment becomes useful from a providence
perspective. Users would know for example that running against image-a
will give results a repeatedly both now and in say 3 years time if they
come to re-analyse their data for paper 2 or follow up checks /
confirmation or cross data analysis.
I suspect the larger local challenges will be making the integration as
seem-less as possible for our users on the one hand and reducing the
maintenance overhead for those responsible for the many applications in
use and the churn that will develop over time.
Fingers are crossed for now but ultimately time will tell how successful
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
More information about the Beowulf