How to download data to institutional cluster

Data Access
1

I’m wondering if there is functionality to download data that I have access to to my institutional cluster using the metadata file.

2

Hi @Wesley_Au ,

We recommend using Google Cloud SDK to transfer files to and from your institutional HPC. This article provides a nice detailed overview. Briefly:

  • Log into your institutional HPC
  • Make sure Google Cloud SDK is installed (you should be able to run the gcloud command)
  • Authenticate through Google Cloud SDK
  • Identify the bucket/path for your data to use in copy commands
  • Run the command(s): e.g., gcloud storage cp WHERE_TO_COPY_DATA_FROM/FILENAME WHERE_TO_COPY_DATA_TO

If you have a list of paths in the metadata, you could call that within a script that also uses gcloud storage cp.

Best,

Ava