Question

JV_SubtleTech on Fri, 25 Aug 2017 08:52:44


Hello ADF people,

I have a suspicion that this is not available but I still wanted to check.  Is it allowed (or soon to be) to use json functions (such as AddDate) to slicestart or slicestop amongst other things within the PartitionedBy area for a copy activity (specifically in the dataset area for a Azure BLOB)?  Basically I want to use offsets and such in order to run an ADF copy at around 12:30 but take data from the previous date.  In order to do this, I would like to do adddate with a negative value for yyyy, mm and dd so that I can pull the blob files from the previous day via the file structure /requests/YYYY-MM-DD/*.  If I don't do this, I cannot be assured that the process that creates the files for the current window (or really just previous day) is complete properly.

If this is not doable, is there an estimation for when window and slices will be allowed to be different?

Thanks very much for the help.


Sponsored



Replies

Gerhard Brueckl on Fri, 25 Aug 2017 09:39:21


you could simply use the offset-property to run the slice/data from 2017-08-25 e.g. one day later on 2017-08-26

so instead of shifting the SliceStart, you can use the offset-property to shift the time when the slice is actually loaded

would that work for you?

-gerhard

JV_SubtleTech on Fri, 25 Aug 2017 18:26:09


It can work as a workaround but not a real solution, of course they don't allow you to alter the existing availability for some reason so I will have to recreate a clone of it (completely stupid design) as well.  I would think scheduling would be the first thing/one of the first they would work on making sure that it works and has full capabilities but I guess it isn't for now.  We'll see if it works properly in practice this evening.

Gerhard Brueckl on Wed, 30 Aug 2017 07:47:52


unfortunately, thats the current design - scheduling cannot be changed for an existing activity/pipeline

I guess the reason is, that this might cause missing spots on the timeframe hence it is prohibited

blugri on Wed, 30 Aug 2017 09:40:11


Hi,

I solved this problem by creating a custom activity. In the activity configuration you can use the AddDate function.

Writing a custom copy activity is quite straight forward.

However, I already had the custom activity for doing some other file filtering that wasn't supported out of the box. So for me it was a minimal change to add it there.

Cheers,

Tom

Gerhard Brueckl on Thu, 31 Aug 2017 10:13:01


sure, custom activities are always an option :)