Population WGS Scaling on Cloud
Automating high-throughput variant calling pipelines at multi-terabyte scale using Nextflow and AWS Batch.
The Challenge
Processing whole-genome sequencing (WGS) data for large-scale population cohorts presents massive storage and computational bottlenecks. Running these workloads on static servers leads to frequent timeouts, complex resource queues, and prohibitive compute costs.
RASA's Technical Approach
RASA engineered a cloud-native, containerized pipeline using Nextflow and AWS Batch. The pipeline utilizes AWS Spot Instances to optimize computing expenses. Raw FASTQ reads are aligned using BWA-MEM, and variant calling is performed using GATK HaplotypeCaller and DeepVariant. The resulting variants (SNVs, Indels, and CNVs) are annotated using VEP (Variant Effect Predictor) alongside public reference databases such as gnomAD and ClinVar. Automated task retry and checkpointing ensure continuous processing even if spot instances are reclaimed.
Results & Biological Insights
We processed over 24 terabytes of sequencing data across 1,500 whole genomes. The containerized pipeline achieved a 100% execution success rate. By using Nextflow for workflow orchestration and AWS Batch for dynamic spot resource allocation, we reduced cloud compute costs by 42% compared to the client's previous in-house cluster setup, accelerating the cohort analysis timeline by several months.