Getting this error from service bus every now and then.

Category: azure connectivity

Question

woaksie at work on Thu, 14 Feb 2013 01:08:17


We are getting random errors in our production environment. Everything works just fine most of the time, but a couple of times a day we get an error like this one.


Microsoft.ServiceBus.Messaging.MessagingException: The server was unable to process the request; please retry the operation. If the problem persists, please contact your Service Bus administrator and provide the tracking id..TrackingId:********-e7cf-4e95-b966-fa2d859ac5cf_G27_B6,TimeStamp:2/12/2013 1:51:17 AM ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: The server was unable to process the request; please retry the operation. If the problem persists, please contact your Service Bus administrator and provide the tracking id..TrackingId:********-e7cf-4e95-b966-fa2d859ac5cf_G27_B6,TimeStamp:2/12/2013 1:51:17 AM

Server stack trace: 
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)

Exception rethrown at [0]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__13(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [1]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.EndReceiveCommand(IAsyncResult result, ref IEnumerable`1 messages)
   --- End of inner exception stack trace ---

Server stack trace: 
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.EndReceiveCommand(IAsyncResult result, ref IEnumerable`1 messages)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [0]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.OnEndTryReceive(IAsyncResult result, ref IEnumerable`1 messages)
   at Microsoft.ServiceBus.Messaging.MessageReceiver.TryReceive(TrackingContext trackingContext, Int32 messageCount, TimeSpan timeout, ref IEnumerable`1 messages)
   at Microsoft.ServiceBus.Messaging.MessageReceiver.Receive(TimeSpan serverWaitTime)
   at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction(Func`1 func)
   at IQ.IR.Core.ServiceBus.AzureBus`1.Listen(IMessageProcessor`1 messageProcessor) in AzureBus.cs: line 108; TraceSource 'WaWorkerHost.exe' event

The failing line of code :

var receivedMessage = _retryPolicy.ExecuteAction(() => _messageReceiver.Receive(TimeSpan.FromMinutes(15)));

We are using Microsoft.Practices.TransientFaultHandling and the above retry policy is:

RetryPolicy<ServiceBusTransientErrorDetectionStrategy>(new ExponentialBackoff())

who the heck is our Service Bus administrator? And what does this error mean?


woaksie

Replies

Dino He on Thu, 14 Feb 2013 09:51:59


HI

I had answered this thread:

http://social.msdn.microsoft.com/Forums/en-US/windowsazureconnectivity/thread/728a5c55-e91a-440d-b325-516a4100d3b9/

Which I think you may got the same problem.

woaksie at work on Thu, 14 Feb 2013 20:14:45


That other thread sounds like what we are experiencing but contains no solution or valid explanation other than Service Bus is flaky. Still need this to be fixed.

Yuan2013 on Thu, 21 Feb 2013 02:54:51


HI

That's course by Azure, you can check the state on dashboard:

http://www.windowsazure.com/en-us/support/service-dashboard/

And your problem should be fiexed now.

woaksie at work on Thu, 21 Feb 2013 17:24:14


During the period between Dec 14th 2012 and Feb 13th 2013 we saw 3,353 of these exceptions in our logs. The service dashboard showed Service Bus to be normal for nearly all of that time.

The dashboard in our experience does not show a lot of problems, even after MS acknowledge there has been a problem.