Allen Cell Discussion Forum

Has anyone tried running the ML code on a cluster?

Hello there,
I’m trying to get this software to run on my university’s cluster (https://hpcf.umbc.edu/) and I was wondering if anyone has tried the same and if so; how did you go about getting things to work like Jupyter Notebook or getting the curator windows to pop up on your own local machine? Right now I have tried to run the curator on a little mini test (just to see if it would work) and after working through some stuff, I got it to say the following:


I think somewhere between that, a window is supposed to pop up to do the whole bad-good selection but uh…I get nothing. :sweat_smile:
Thanks.

Dear @mramsahoye ,

Indeed, it is not quite straightforward to forward pop-up windows from your remote cluster to your local machine. Depending on specific setting of your storage system, it might be possible to mount your remote storage directly to your local machine, then, you can run the curator on your local machine (curator does not require GPU, so most of the basic PCs should work).

In your specific issue, it seems like the curator does not find any valid data. I cannot see the full command line in your screenshot, can you double check how you specify the --data_type? Here, you want to specify the type of data you are working on, e.g. .tiff or .tif or .ome.tif, etc. A common mistake I usually make is mis-type .tif as .tiff.

(More details here: aics-ml-segmentation/bb2.md at main · AllenCell/aics-ml-segmentation (github.com))

Let me know if you have more questions.

Thanks,
Jianxu

Hi there! Thanks for the response; sorry, I was in the midst of a few midterms so I had to step back from what I was doing for a bit.
Yeaaaa, I actually did make that mistake with the .tif/.tiff so I will try it again. I will also look into how to mount remote storage; I haven’t heard of that so thanks for the suggestion. :slight_smile: I was also going to ask; so, you said the curator doesn’t require GPU so that’s fine to run on the normal PC. The main bulk of the GPU usage then comes out of the trainer, right?

Yes, that is correct. Curation can be done in any machine, while training and testing need to be done on GPU machines