# Galaxy on Terra: How to best handle 100s of files

**URL:** <https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23>\
**Category:** Help\
**Created:** [February 17, 2021, 5:00pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23 "2021-02-17T17:00:40Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![edmundotogo](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/edmundotogo/32/95_2.png) [@edmundotogo](https://help.anvilproject.org/u/edmundotogo)\
**Post date:** [February 17, 2021, 5:00pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/1 "2021-02-17T17:00:40Z")

</div>

Is there a way to automate importing files into the Galaxy instance on Terra, given a list of directories in the workspace?  
I intend to process hundreds of files and I am currently limited to clicking on them one at a time in the `Choose remote files` section.

If not automating the import of files as they are analyzed, perhaps a better solution to importing in bulk?  
Galaxy has a feature to batch import URLs from the web, is there something similar when accessing files in Terra workspaces?

---

<div class="post-metadata">

**Author:** ![edmundotogo](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/edmundotogo/32/95_2.png) [@edmundotogo](https://help.anvilproject.org/u/edmundotogo)\
**Post date:** [February 17, 2021, 5:53pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/2 "2021-02-17T17:53:01Z")

</div>

On a related note, I’m curious to know if I can increase computer resources in Galaxy (in order to process a large list of files faster).

---

<div class="post-metadata">

**Author:** ![SMO](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/smo/32/13_2.png) [@SMO](https://help.anvilproject.org/u/SMO)\
**Post date:** [February 17, 2021, 6:01pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/3 "2021-02-17T18:01:34Z")

</div>

from slack:

> lack of a ‘select all’ feature is an issue that was identified in the AnVIL dev days meeting and should def be added; for current ways to accomplish batch uploading:  
> `gxfiles://<service>/file/path` type URLs work with the ‘paste/fetch’ component of the uploader, and whole directories can be imported with the rule-based uploader.

---

<div class="post-metadata">

**Author:** ![SMO](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/smo/32/13_2.png) [@SMO](https://help.anvilproject.org/u/SMO)\
**Post date:** [February 17, 2021, 6:03pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/4 "2021-02-17T18:03:41Z")

</div>

from slack:

> click paste/fetch, and include gxfiles:// formatted URIs, then press start:

 ![Screen Shot 2021-02-17 at 9.35.37 AM](https://canada1.discourse-cdn.com/flex027/uploads/anvil/original/1X/68d602af14cac2a529e06c783a3cd1662e7e3c9a.png)

---

<div class="post-metadata">

**Author:** ![SMO](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/smo/32/13_2.png) [@SMO](https://help.anvilproject.org/u/SMO)\
**Post date:** [February 17, 2021, 6:09pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/5 "2021-02-17T18:09:32Z")

</div>

from slack:

> for whole directories, you can go to rule-based and select remote files directory

 ![Screen Shot 2021-02-17 at 9.37.32 AM](https://canada1.discourse-cdn.com/flex027/uploads/anvil/original/1X/e42ff3fd237e17d38dc271fcaceaff4d4806fd5e.png)

> the trick here is to select the little arrow box to navigate, then left click the directory name of the one you want, but in my (limited) experiments with this interface, it was a bit clunky and didn’t seem to handle subdirectories (e.g., if you click on `some/dir/` you won’t get the contents of `some/dir/a/` or `some/dir/b/`

 ![Screen Shot 2021-02-17 at 9.37.49 AM](https://canada1.discourse-cdn.com/flex027/uploads/anvil/original/1X/32d9f9869905165277cf38a047163dbbc5f9ee68.png)

> but, to circle back — doing this with a ‘select all’ checkbox is probably the way to go.  
> thinking further, 100s of datafiles (unless they are quite small) are probably not yet supported on current Galaxy VM sizes and will likely fill the provisioned volume, but it would be good to have this user experience issue fixed before bigger volumes are supported

---

<div class="post-metadata">

**Author:** ![edmundotogo](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/edmundotogo/32/95_2.png) [@edmundotogo](https://help.anvilproject.org/u/edmundotogo)\
**Post date:** [February 17, 2021, 7:39pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/6 "2021-02-17T19:39:32Z")

</div>

Thanks! I’m trying this but getting this error message:

 ![Screen Shot 2021-02-17 at 2.37.18 PM](https://canada1.discourse-cdn.com/flex027/uploads/anvil/original/1X/e7e83c9d03a360b7b1aa10e75e963ea98761cf0d.png)

> {  
> “userAgent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_16\_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36”,  
> “onLine”: true,  
> “version”: “20.09”,  
> “xhr”: {  
> “readyState”: 4,  
> “responseText”: “{"source":"leonardo","message":"TraceId(1e3b656d242b2310b19d2062c12e26b3/16382466342422451293) | Unable to proxy connection to tool on mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy","statusCode":500,"exceptionClass":"class org.broadinstitute.dsde.workbench.leonardo.http.service.ProxyException"}”,  
> “responseJSON”: {  
> “source”: “leonardo”,  
> “message”: “TraceId(1e3b656d242b2310b19d2062c12e26b3/16382466342422451293) | Unable to proxy connection to tool on mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy”,  
> “statusCode”: 500,  
> “exceptionClass”: “class org.broadinstitute.dsde.workbench.leonardo.http.service.ProxyException”  
> },  
> “status”: 500,  
> “statusText”: “error”  
> },  
> “options”: {  
> “validate”: true,  
> “parse”: true,  
> “wait”: true,  
> “emulateHTTP”: false,  
> “emulateJSON”: false,  
> “textStatus”: “error”,  
> “errorThrown”: “”  
> },  
> “url”: “[https://notebooks.firecloud.org/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galax/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/api/histories/877d7351f28c2dc0/contents?details=5418c83dc470343f%2C528eabcb6d0deb65&order=hid&v=dev&q=update\_time-ge&q=deleted&q=purged&qv=2021-02-17T19%3A36%3A14.000Z&qv=False&qv=False](https://notebooks.firecloud.org/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galax/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/api/histories/877d7351f28c2dc0/contents?details=5418c83dc470343f%2C528eabcb6d0deb65&order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2021-02-17T19%3A36%3A14.000Z&qv=False&qv=False)”,  
> “model”: {  
> “state”: “error”,  
> “deleted”: true,  
> “purged”: false,  
> “name”: “GTEX-1JMLX-0326-SM-CM2RP.Aligned.sortedByCoord.out.patched.md.bam”,  
> “accessible”: true,  
> “data\_type”: “galaxy.datatypes.data.Data”,  
> “file\_ext”: “data”,  
> “file\_size”: 0,  
> “meta\_files”: ,  
> “misc\_blurb”: “tool error”,  
> “misc\_info”: “”,  
> “tags”: ,  
> “history\_id”: “877d7351f28c2dc0”,  
> “history\_content\_type”: “dataset”,  
> “hid”: 133,  
> “visible”: true,  
> “model\_class”: “HistoryDatasetAssociation”,  
> “url”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/api/histories/877d7351f28c2dc0/contents/6f27d97c8e729e85”,  
> “type”: “file”,  
> “type\_id”: “dataset-6f27d97c8e729e85”,  
> “extension”: “data”,  
> “create\_time”: “2021-02-18T00:28:15.847Z”,  
> “update\_time”: “2021-02-18T00:35:48.308Z”,  
> “dataset\_id”: “6f27d97c8e729e85”,  
> “id”: “6f27d97c8e729e85”,  
> “hda\_ldda”: “hda”,  
> “rerunnable”: false,  
> “annotation”: null,  
> “permissions”: {  
> “manage”: [  
> “877d7351f28c2dc0”  
> ],  
> “access”:   
> },  
> “uuid”: “83f7ec86-7e62-49e3-8610-592aaf135c35”,  
> “creating\_job”: “9a7ff6b167ebf5b4”,  
> “display\_apps”: ,  
> “visualizations”: [  
> {  
> “name”: “editor”,  
> “html”: “Editor”,  
> “description”: “Manually edit text”,  
> “logo”: null,  
> “title”: null,  
> “target”: “galaxy\_main”,  
> “embeddable”: false,  
> “entry\_point”: {  
> “type”: “mako”,  
> “file”: “editor.mako”,  
> “attr”: {}  
> },  
> “settings”: null,  
> “groups”: null,  
> “specs”: null,  
> “href”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/plugins/visualizations/editor/show”  
> }  
> ],  
> “validated\_state”: “unknown”,  
> “resubmitted”: false,  
> “api\_type”: “file”,  
> “file\_name”: “/galaxy/server/database/objects/8/3/f/dataset\_83f7ec86-7e62-49e3-8610-592aaf135c35.dat”,  
> “genome\_build”: “?”,  
> “peek”: null,  
> “display\_types”: ,  
> “download\_url”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/api/histories/877d7351f28c2dc0/contents/6f27d97c8e729e85/display”,  
> “created\_from\_basename”: null,  
> “validated\_state\_message”: null,  
> “metadata\_dbkey”: “?”,  
> “urls”: {  
> “purge”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/datasets/6f27d97c8e729e85/purge\_async”,  
> “display”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/datasets/6f27d97c8e729e85/display/?preview=True”,  
> “edit”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/datasets/edit?dataset\_id=6f27d97c8e729e85”,  
> “download”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/datasets/6f27d97c8e729e85/display?to\_ext=data”,  
> “report\_error”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/dataset/errors?id=6f27d97c8e729e85”,  
> “rerun”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/tool\_runner/rerun?id=6f27d97c8e729e85”,  
> “show\_params”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/datasets/6f27d97c8e729e85/show\_params”,  
> “visualization”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/visualization”,  
> “meta\_download”: “/proxy/google/v1/apps/mtdna-mitonucl/saturn-k8-e3c4b128-2807-4a9e-8371-f54f4d24216b/galaxy/dataset/get\_metadata\_file?hda\_id=6f27d97c8e729e85&metadata\_name=”  
> }  
> },  
> “user”: {  
> “id”: “877d7351f28c2dc0”,  
> “username”: “edmundotogo”,  
> “total\_disk\_usage”: 17787974463,  
> “nice\_total\_disk\_usage”: “16.6 GB”,  
> “quota\_percent”: null,  
> “is\_admin”: true,  
> “preferences”: {  
> “favorites”: “{"tools": ["toolshed.g2.bx.psu.edu/repos/lparsons/htseq\_count/htseq\_count/0.9.1", "toolshed.g2.bx.psu.edu/repos/devteam/samtools\_idxstats/samtools\_idxstats/2.0.3"]}”  
> },  
> “tags\_used”: [  
> “name:Skeletalmuscle-AfAm”  
> ],  
> “purged”: false,  
> “deleted”: false,  
> “quota”: null  
> }  
> }

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 17, 2021, 8:19pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/7 "2021-02-17T20:19:08Z")

</div>

From Slack 2:54pm EST

> This looks like the transient client error that comes up when the web handler has problems in the back. [we’ll check] to see if there are any relevant logs

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 2:20pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/8 "2021-02-18T14:20:03Z")

</div>

From Slack 9:06am EST

> Looking at this - this user’s instance is currently `RUNNING` and the galaxy-web handler seems to be up. Looking through logs to see if there were any transient errors when the file import was tried

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 2:20pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/9 "2021-02-18T14:20:51Z")

</div>

From Terra Support 9:17am EST

> Your request (114068) has been received and is being reviewed by our support staff.

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 2:30pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/10 "2021-02-18T14:30:27Z")

</div>

From Slack 9:21am EST

> here is what I see in the `galaxy-web` pod logs:
> 
> 1. file imports happening at ~12-1pm yesterday
> 2. ~30min gap in logs at around 1pm
> 3. container restarts around 2pmWondering if the web handler could have ran out of memory or something?

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 3:31pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/11 "2021-02-18T15:31:54Z")

</div>

From Terra Support 9:33am

> Thanks for writing in. Are you experiencing a specific error in Galaxy when trying to process these files? Any screenshots you can provide would be very helpful!

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 4:41pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/12 "2021-02-18T16:41:23Z")

</div>

From Slack 11:04am EST

> Was the import so big that it could’ve filled the disk?

---

<div class="post-metadata">

**Author:** ![cutsort](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@cutsort](https://help.anvilproject.org/u/cutsort)\
**Post date:** [February 18, 2021, 4:41pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/13 "2021-02-18T16:41:51Z")

</div>

From Slack 11:30am EST

> Maybe. Not actually sure since I can’t see the data. […] would it be possible for the user to add me to their Terra workspace?

---

<div class="post-metadata">

**Author:** ![edmundotogo](https://yyz1.discourse-cdn.com/flex027/user_avatar/help.anvilproject.org/edmundotogo/32/95_2.png) [@edmundotogo](https://help.anvilproject.org/u/edmundotogo)\
**Post date:** [February 18, 2021, 5:48pm UTC](https://help.anvilproject.org/t/galaxy-on-terra-how-to-best-handle-100s-of-files/23/14 "2021-02-18T17:48:51Z")

</div>

That might have been the case. I tried using the Rule-based method and I must have picked too many at a time. At about 5-7 GB per file, I’m pretty sure I hit the 250Gb limit.

Importing in smaller batches by clicking seems to work fine.

> Was the import so big that it could’ve filled the disk?