Google Cloud recently announced the preview of Batch, a managed service for running batch jobs at scale. The new service supports the latest T2A Arm instances and Spot VMs for large batch jobs using job parallelization.
Lot handles dynamic resource provisioning and autoscaling, parallel request execution, supports scripting and containerized workloads, and can take advantage of native Google Cloud services and cloud-based processing tools. lots. Shamel Jacobs, Product Manager at Google, and Bolian Yin, Software Engineer at Google, write:
Batch processing is as old as computing itself, with the term “batch” dating back to the punched cards used by early mainframes (…) Batch jobs are particularly prevalent in areas such as research, simulation, genomics, visual effects, fintech, manufacturing and EDA.
The new service supports common task types such as task boards and multi-node MPI applications. Jacobs and Yin point out that Batch isn’t the only service on Google Cloud that handles batch processing:
Batch is a general-purpose batch processing service and the latest in a long list of products we’ve created over the years that process jobs to help businesses move their workloads to the cloud. These services include: Cloud Life Sciences (formerly Google Genomics), Data Streams, and Cloud Execution Tasks.
The key concepts of the new service are: workrunning a compute job from execution to completion, tasks that run on instances of Compute Engine, the table taskmultiple tasks within a task that simultaneously run the same executable and resources, such as Compute Engine instances, Cloud Storage, or NFS mounts. Lewis Carroll, Director at AMD, Comments:
T2D Tau virtual machines with batch should be a behemoth for large-scale life sciences, chemicals, derivatives pricing, risk, and other large-scale parallel distributed computing tasks.
The cloud provider has released a media transcoding tutorial, which leverages Batch to transcode H.264 video files to VP9. Busybox (luggage box), a project for running a container as a batch job, primegen, an end-to-end example of using workflows and Cloud Build with Batch and wrf, a sample application to run the weather research and forecast model in a batch job with MPIB, are other examples available on GitHub.
Developers can access Batch through the API, command line tool, workflow engines, or console, setting priorities for tasks and establishing retry policies. The service can be run in the HPC Toolkit, the open-source google cloud project aimed at deploying high-performance computing environments, with the cloud provider explaining:
Using Google Cloud Batch with HPC Toolkit simplifies the configuration needed to provision and run more complex scenarios, such as setting up a shared file system and installing software for use by Google Cloud Batch jobs. It also allows sharing of tested infrastructure solutions that work with Google Cloud Batch through HPC Toolkit plans.
Currently in preview, Batch is available in a subset of Google Cloud regions: Iowa, South Carolina, Oregon, and Finland. There are no additional charges for using Batch, customers pay for the resources used to run the jobs.