Sharing files between all tasks in a single job

Category: azure batch


JamesThurley on Thu, 26 May 2016 08:35:30

Each job I submit to my pool may execute using some files that are common to all tasks in the job.

At the moment I use this code to share the files between tasks, using a JobPreparationTask:

CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% %AZ_BATCH_NODE_SHARED_DIR% /S) ^& IF %ERRORLEVEL% LEQ 1 exit 0"

However this copies the files to a directory shared between all jobs.  I notice in the docs ( it implies there is a folder "%AZ_BATCH_NODE_ROOT_DIR%\JOB_ID", which looks like a folder common to all tasks in a job that might be automatically cleaned up for me when a job is complete.  However this folder isn't mentioned directly, and there isn't (I think) an environment variable for it, although there is one for AZ_BATCH_JOB_ID.

Is it therefore acceptable for me to do the following?

CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% \"%AZ_BATCH_NODE_ROOT_DIR%\%AZ_BATCH_JOB_ID%\" /S) ^& IF %ERRORLEVEL% LEQ 1 exit 0"

Will that behave as I want? i.e. copying the files to a location shared between the tasks in a job, and cleaning them up when the job is complete?  Or should I leave them where they are and perhaps use variables like AZ_BATCH_JOB_PREP_WORKING_DIR to find them?


Matthew Christopher on Thu, 26 May 2016 22:26:01

You should be able to use AZ_BATCH_JOB_PREP_WORKING_DIR to find them.  That directory should live for the duration of your job and be cleaned up afterwards. It also saves you having to do any file copies which means if you have any significant amount of data your JobPrepTask will run faster.

So in short: Leave them where they are and use AZ_BATCH_JOB_PREP_WORKING_DIR.