Inputs from Blob storage - excessive ListBlobs transactions

Category: azure stream analytics

Question

Johannes Schoeman on Mon, 19 Aug 2019 11:24:45


I recently noticed an high transaction cost for a storage account associated with stream analytics.

I have about 16 input streams from General purpose V2 blob storage.
These Inputs are for exports from application insight continuous export - about 20-30 new blobs are added every minute in total.

Altogether, these input streams result in between 2 and 3 Million ListBlobs transactions per day
This turns out to be quite expensive!

It is querying listBlobs about 100 blobs for every time it reads one!
Is there any way to configure Azure Stream Analytics to perform less listBlobs operations in this scenario?

Replies

MartinJaffer-MSFT on Thu, 22 Aug 2019 17:28:08


Hello Johannes Schoeman, and thank you for your inquiry.  Are you open to the idea of doing some pre-aggregation to reduce the quantity of blobs written?

MartinJaffer-MSFT on Mon, 26 Aug 2019 16:30:01


Johannes Schoeman, are you still in need of assistance?

Johannes Schoeman on Mon, 02 Sep 2019 12:08:26


I'm open to the idea - I just export the blobs using Azure's continuous export, and I didn't see any settings that allowed me to configure the frequency at which it exported.