Summing previous rows over a grouping

Category: azure data lake


Misinformed DNA on Wed, 03 Aug 2016 16:53:54

For a given grouping, I want to sum values in a column from previous rows and subtract the value in the same column for the current row. This was initially coded as a custom Processor, but I can't find a way to sort and group the values. I see PRESORT exists for Reducer, but I haven't found a way to use it in a Processor as I keep getting compile-time errors.

Another option is to use OVER() with a custom Aggregator, but I haven't found any documentation on how to do that.

Thoughts on how best to do this or further documentation?


MRys on Mon, 08 Aug 2016 19:40:55

PRESORT only exists for reducers. Why can't you try using a reducer (that aggregates the values but does produce one output row per input)?

Custom aggregators are currently not supported with the OVER expression. Please file a request at

Misinformed DNA on Tue, 09 Aug 2016 15:02:46

Reducer worked great!