Terra warp-pipelines out of memory issue

Help
1

Hello~
I am using the Terra workflow [warp-pipelines/1-WholeGenomeGermlineSingleSample] to try the variants calling. The input files are the uBAMs, which size ranges from 4GB - 40GB for each. As we did high depth WGS, there are about 10 uBAM for each sample, which might be the reason that lead to the “out of memory” error.
|UnmappedBamToAlignedBam||stderr for job UnmappedBamToAlignedBam.MarkDuplicates:NA:1 contained one of the memory-retry-error-keys: [OutOfMemory,Killed] specified in the Cromwell config. Job might have run out of memory.

I submit jobs with data table and the latest try was using:
MarkDuplicates - additional_disk - 500; memory_multiplier - 10
GatherBamFiles - additional_disk - 200; memory_multiplier - 2

I would very grateful if anyone could help me to optimize the parameters for this workflow run through.

2

Hi @yxhan

I am opening a ticket with Terra support and will get back to you as soon as we get an answer.

Javier

3

You are very much appreciated, @Javier-CP

4

@yxhan From Terra support:

It sounds like the user believes their workflow is failing due to insufficient memory, which is common. If that’s the case, they can try increasing the memory_multiplier value they’re setting for the individual tasks. Unfortunately, we can’t provide any recommendations around optimization and exactly how much to increase memory by so that’ll be up to the user to test out and determine for themselves based on their inputs.
If the user is willing to share their workspace with Terra Support, we can certainly take a look at the failed submission and see if there could be anything else causing the workflow to fail.

5

Thanks, @Javier-CP
I tried once again with increased memory_multiplier as well as additional_disk but failed with similar out of memory error.
I am happy to share my workspace with the Terra team if you could suggest which account I should share with.

6

@yxhan

From Terra support:

The Share option is in the three-dots menu at the top-right.

  1. Toggle the “Share with support” button to “Yes”
  2. Click Save
    share-with-support
    share-with-support1420×142 9.32 KB

Please provide us with

  1. A link to your workspace
  2. The relevant submission ID
  3. The relevant workflow ID

If you’d prefer to share the info privately, you can email it to help@lists.anvilproject.org.

Javier