Question

Bradcstevens on Thu, 14 Dec 2017 00:19:38


I was wondering if anyone has run into the issue that I am experiencing with my current ASR engagement while running the ASR DP Throughput Test for VMware to Azure:

Error in uploading files. Failed to calculate achievable throughput. 

Possible reasons:

1. STORAGE_COMMUNICATION_ERROR. Unable to communicate to Azure Storage blob. Check network connectivity to Azure and if bandwidth from on-premises to Azure is very low.

What’s got me scratching my head is the Error Output in the command line vs. Error Output the logs. The snippet from the logs show the top error repeated several times in the same way prior to the last error which generates the error in the CLI screenshot above):

12/13/2017 15:05:05:Command:UploadTest\UploadTest.exe
12/13/2017 15:05:05:Argument:"https://STORAGEACCOUNT.blob.core.windows.net/asrvhdfile54636487740604601924/asrvhdfile54.vhd?sv=2015-12-11&sr=b&sig=CUKWzc8rhhZKT35i1IBlTFYs1xTo6WWFaOhvBHHDDtA%3D&st=2017-12-13T22%3A56%3A00Z&se=2017-12-14T23%3A01%3A00Z&sp=rwl" "C:\CDW\SBCERA-NEPHLIM_ProfileData\asrvhdfile54.vhd" 8 0
12/13/2017 15:05:05:Output:Arguments before dll load filePath:C:\ASR\ProfileData\asrvhdfile54.vhd, blobUri:https://STORAGEACCOUNT.blob.core.windows.net/asrvhdfile54636487740604601924/asrvhdfile54.vhd?sv=2015-12-11&sr=b&sig=CUKWzc8rhhZKT35i1IBlTFYs1xTo6WWFaOhvBHHDDtA%3D&st=2017-12-13T22%3A56%3A00Z&se=2017-12-14T23%3A01%3A00Z&sp=rwl, numThreads:8, mountAndRead:0
GetModuleHandle failed for Library [ReplicationProviderDataMover.dll], Hr = [0x8007007e], will try LoadLibrary
Found Library [ReplicationProviderDataMover.dll], hLibrary = [00007FFC81A50000]
Calling SendFile from Plugin [00007FFC81A50000]Result of SendFile Hr = [0x00000000]
12/13/2017 15:05:05:Exit code:0
12/13/2017 15:05:06:Command:UploadTest\UploadTest.exe
12/13/2017 15:05:06:Argument:"https://STORAGEACCOUNT.blob.core.windows.net/asrvhdfile1636487740372725415/asrvhdfile1.vhd?sv=2015-12-11&sr=b&sig=sGciPqaOKO96uqcBG%2FYS5BI5ohcoYXjg1bePW2EE48g%3D&st=2017-12-13T22%3A55%3A37Z&se=2017-12-14T23%3A00%3A37Z&sp=rwl" "C:\CDW\SBCERA-NEPHLIM_ProfileData\asrvhdfile1.vhd" 8 0
12/13/2017 15:05:06:Output:Arguments before dll load filePath:C:\AZURE\ProfileData\asrvhdfile1.vhd, blobUri:https://STORAGEACCOUNT.blob.core.windows.net/asrvhdfile1636487740372725415/asrvhdfile1.vhd?sv=2015-12-11&sr=b&sig=sGciPqaOKO96uqcBG%2FYS5BI5ohcoYXjg1bePW2EE48g%3D&st=2017-12-13T22%3A55%3A37Z&se=2017-12-14T23%3A00%3A37Z&sp=rwl, numThreads:8, mountAndRead:0
GetModuleHandle failed for Library [ReplicationProviderDataMover.dll], Hr = [0x8007007e], will try LoadLibrary
Found Library [ReplicationProviderDataMover.dll], hLibrary = [00007FFC81A50000]
Calling SendFile from Plugin [00007FFC81A50000]Result of SendFile Hr = [0x80790004]
12/13/2017 15:05:06:Exit code:-2139553788
12/13/2017 15:05:06:Parsing Error Messages
12/13/2017 15:05:06:Return Code = 0x00000000
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:HVR_E_STORAGE_COMMUNICATION_ERROR
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x00000000
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x00000000
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x00000000
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x00000000
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Return Code = 0x80790004
12/13/2017 15:05:06:Error encountered in throughput calculation. 
One or more 'asrvhdfile*' files uploaded in storage account: STORAGEACCOUNT might still be present. You can delete all of them manually and then retry the operation.
To delete, go to Azure Portal -> Storage Accounts -> [Storage Account Name] -> Containers, and delete 'asrvhdfile*' files.
12/13/2017 15:05:06:Exception during throughput calculation.: Threw Exception.Type: System.ArgumentOutOfRangeException, Exception.Message: startIndex cannot be larger than length of string.
Parameter name: startIndex
12/13/2017 15:05:06:StackTrace:   at System.String.Substring(Int32 startIndex, Int32 length)
   at Bandwidth.AvailableBandwidth.ParseErrorMessages(String[] outputList, List`1& listOfUniqueErrorCode)
   at Bandwidth.AvailableBandwidth.CalculateAvailableBandwidth(String storageAccountName, String storageAccountKey, List`1 filePaths, List`1 vmList, StorageAccountEnvironment storageAccountEnvironment)
   at Bandwidth.AvailableBandwidth.GetAvailableBandwidth(Virtualization platform, String storageAccountName, String storageAccountKey, StorageAccountEnvironment storageAccountEnvironment, String directory, List`1 vmList, String vmDetailsFileName)
12/13/2017 15:05:06:Deleting 'asrvhdfile*' files from local machine...
12/13/2017 15:05:06:Successfully deleted the files.
12/13/2017 15:05:06:Get throughput operation failed.
12/13/2017 15:05:06:
2017-12-13T15:05:06.7117492-08:00 Info: Trace Session Ended

But that’s not all….. the DP throughput test actually puts vhds in Azure to the SA as it does its test (visible in Azure Storage Explorer while test is running), but continues to fail with the same output regardless of machine (physical or virtual) that I run this tool on. The customer I'm working with claims that their bandwidth pipe is 50MBps, however while looking at their Cisco ASA dashboard, we see the DP consume no more than 50Mbps or 6.25MBps. 

I am unable to advise my customer on throughput issues, or planning correctly beyond the report generated from the profile of the Deployment Planning Tools output. 

Anyone have any suggestions for resolution here? 

Thanks in advance! 

-Brad, Azure Consulting Engineer


Sponsored



Replies

Adam Smith (Azure) on Mon, 18 Dec 2017 17:03:21


Hi @BradCstevens,

This could be caused by the upload of a large file in a very low bandwidth availability, in this case we encourage implementing a retry-policy for blob Service, this post might be of help: https://github.com/Azure/azure-storage-node/issues/297

var retryPolicy = new AzureStorage.ExponentialRetryPolicyFilter();
var blobService = AzureStorage.createBlobServiceWithSas(…).withFilter(retryPolicy);

Also have you used this specific tutorial in the above scenario to make sure that you have enough bandwidth?  https://docs.microsoft.com/en-us/azure/site-recovery/site-recovery-deployment-planner 

It could be that there are throttling rules in the internal network which maybe causing issues as well