Container building automation

devops_logo

HPC resources have complex and dynamic software needs that are challenging to manage and maintain. Users often want the latest software available for their research and development which drives the need for frequent installation and updates. Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows to deploy rapidly-changing hardware and software resources.

One can build a CICD pipeline to automatically build and/or deploy their HPC application either in the form of containers or as a custom image with all the software dependencies installed. Integrating CICD practices into HPC workflows increases the potential for delivering high quality and reliable software. Packaging HPC applications as containers also allows these applications to be easily ported to new platforms, delivered to others, and orchestrated across the variety of container managemenet an orchestration platforms available today.

In this lab, you are introduced to AWS Developer Tools and how to use services like AWS CodeCommit, AWS CodeBuild and AWS CodePipeline to automate application deployment with containers and CI/CD pipelines. The next lab will focus on how to take the containers built in this lab and deploy tightly-coupled simulation workflows using container orchestration.

You will be deploying the below architecture as part of this lab:

AWS CICD

You will use this architecture to create a CI/CD pipeline to build an MPI-enabled GROMACS application container. GROMACS is a molecular dynamics package mainly designed for simulations of proteins, lipids, and nucleic acids. It simulates the Newtonian equations of motion for systems with hundreds to millions of particles and is a community-driven project. It can run on central processing units (CPUs) and graphics processing units (GPUs) and was originally developed in the Biophysical Chemistry department of University of Groningen. Univerisities and research centers worlwide are now maintaining the package.

This lab includes the following steps:

  1. Create a container definition for an MPI-enabled GROMACS simulation
  2. Create a repository in AWS CodeCommit
  3. Create a build environment using AWS CodeBuild
  4. Create a pipeline using AWS CodePipeline
  5. Automate the build process with repository update

AWS Developer Tools provides a list of services to host code, build, test, and deploy your applications quickly and effectively. AWS services offered as part of the AWS Developer Tools suite helps remove the undifferentiated heavy lifting associated with DevOps adaptation and software development. You can build a continuous integration and delivery capability without managing servers or build nodes, and leverage Infrastructure as code (IaC) to provision and manage your cloud resources in a consistent and repeatable manner.

Benefits

  • Minimize downtime

    Build highly available applications on a resilient cloud infrastructure and enable your teams to respond, adapt, and recover quickly from unexpected events.

  • Automate CI/CD pipelines & Release software faster

    Remove error-prone manual processes and eliminate the need to babysit software releases. Use software release pipelines that encompass build, test, and deployment.

  • Increase developer productivity

    Manage services, provision resources, and automate development tasks without switching context or leaving your editor.

  • Monitor operations

    Build an observability dashboard to gain instant and continuous insight into your system’s operations.

  • Test and automate infrastructure

    Combine Infrastructure as code with version control and automated, continuous integration to bring scalability and consistency to provisioning and management.

This lab requires an AWS Cloud9 IDE. If you do not have an AWS Cloud9 IDE set up, complete the Preparation section of the workshop.