Use service account to access data in AnVIL_GTEx_V8_hg38 workspace

,
1

Hi all,

we are planning to perform some analyses on GCP using the BAM files from the AnVIL_GTEx_V8_hg38 workspace (terra-6c7f2bca). We have access to the files with our gmail account linked to Terra (the same I am using here) but we would like to use a service account on GCP.

I read in the Terra documentation that the best approach would be to use a group for this. So, we created a group in Terra called guigolab@firecloud.org and added the service account to it. Now, I guess we need this group to have permission to access the data in the workspace. Could anybody please help us with this?

Many thanks.

Best

2

Thanks for your question! You’re right that Groups are a good way to organize members and accounts.

Service accounts are a bit trickier than regular Terra accounts. It sounds like you were able to create the service account successfully, but still might need to register it. Can you let us know if following these steps in this post allows you to access the files?

3

Hi Ava,

thanks for your reply.

I actually already registered the service account but cannot access the files. What are the next steps?

Best

4

Great! And you have shared the workspace with the guigolab@firecloud.org group?

5

That’s the issue. I don’t have permissions to share the workspace with the group. I guess I need a workspace admin to take care of that…

6

Dear all,

I haven’t received any more feedback. Could anybody please help us with this?

Many thanks!

Best regards

7

Hi Roderic,

Can you confirm first if you are able to access public data (Terra) using a service account?

It is unclear whether service accounts are compatible with Authorization Domains but as a first step you can test if the service account can access the public data.

8

Hi Javier,

many thanks for you reply.

We just checked if we can list the contents of the public bucket with this command:

gcloud storage --billing-project guigolab-352014 ls gs://fc-ed391d18-3c0a-4499-a292-35ca51ebf381

It works running it with our gmail account but we get the following error using the service account::

ERROR: (gcloud.storage.ls) User [nf-service-account@guigolab-352014.iam.gserviceaccount.com] does not have permission to access b instance [fc-ed391d18-3c0a-4499-a292-35ca51ebf381] (or it may not exist): nf-service-account@guigolab-352014.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).

Best regards

9

Started a ticket with Terra Support:

Your request (320772) has been received and is being reviewed by our support staff.

10

From Terra Support 13:00 EDT

It looks like there was an issue with the registered service account on Terra. We’ve resolved this issue so the user should now be able to access the public workspace bucket with the SA.

Roderic, can you try again?

11

Thanks a lot Javier!

I had to add the Service Usage Consumer role to the service account for the project and eventually it worked with the public bucket.

I then tried with the AnVIL_GTEx_V8_hg38 workspace bucket but I still get Permission 'storage.objects.list' denied on resource (while it works with my main account).

12

Checking into this with Terra Support. We’ll keep you updated.

1 Like
13

From Terra Support:

Our engineers indicate that the Service Account will not be able to gain access to controlled-access GTEx workspaces. Due to security reasons, they would need to be logged into the Terra UI to do the account linking mentioned in the instructions because the NIH Auth requires a redirect back to app.terra.bio. And since you can’t login via Google with a service account, this is unfortunately, not possible.

Roderic, Is there something in particular that you can practically only do via a service account?

14

Hi Javier,

thanks a lot for your help.

While technically we can do all we need with our standard gmail account, it is strongly recommended to use service accounts when, for example, running a long computational workflow with Nextflow (which is one of the tasks we want to accomplish here).

If the use of a service account is not possible, we will move forward using our gmail account. Although, I think that this is something Terra engineers should have a look at.

Many thanks again for your help!

Best

15

Definitely something to consider! Thank you for your patience, Roderic.