Restart Service Fabric Cluster

Category: azure service fabric


Stratford on Fri, 25 Nov 2016 15:01:53


We have a service fabric cluster deployment among 3 VMs without internet connection.

Within service fabric explorer, the warning message comes out

Unhealthy event: SourceId='System.UpgradeOrchestrationService', Property='ClusterGoalStateReachable', HealthState='Warning', ConsiderWarningAsError=false.
Goal state is unreachable. Please check cluster internet connectivity or disable autoprovisioning of cluster upgrades by updating the cluster configuration.

I turned off the "Service Fabric Host Service" on each VM, changed ClusterConfig.json to add "fabricClusterAutoupgradeEnabled": false under the "properties", and restarted all of the three "Service Fabric Host Service".

However, the same warning message is still there.

Did I restart the service fabric cluster correctly for turning on/off the "Service Fabric Host Service"? Why does the warning not go away?



TChiang on Fri, 25 Nov 2016 17:17:31

Try change ClusterConfig.json to add "fabricClusterAutoupgradeEnabled": false under the "properties and reset cluster.

Stratford on Fri, 25 Nov 2016 19:27:09

That's what I did as described in the original post. And the warning message is still there after restart "Service Fabric Host Service" on each VM.

TChiang on Sat, 26 Nov 2016 01:10:51

Use 'Reset Local Cluster' not restart "Service Fabric Host Service" on each VM.

Stratford on Sat, 26 Nov 2016 15:45:49

What is 'Reset Local Cluster' not restart'?  it's not a service fabric cmdlet

Where could I find it?

TChiang on Mon, 28 Nov 2016 15:43:24

'Local Cluster Manager' in System Tray.

Stratford on Mon, 28 Nov 2016 21:12:08

We don't have this "Local Cluster Manager" in our UAT service cluster.

Do we have other ways to do it? The VMs are restarted over weekend but the the warning messages are still there.

Does ClusterConfig.json need to be renamed after add "fabricClusterAutoupgradeEnabled": false ?

Max Burlik on Tue, 29 Nov 2016 01:10:52

Currently, the warning will persist for 48 hours after disabling "fabricClusterAutoupgradeEnabled" through config upgrade, after which it will expire.

The refresh interval is tracked to be fixed in a coming release, such that the warning goes away once the config upgrade completes.

Stratford on Tue, 29 Nov 2016 09:50:51

We made the changes on last Friday and the warnings are still there.

Any thing else I could try to see where the problem is?

Max Burlik on Tue, 29 Nov 2016 18:57:29

Try running


and see if it reflects the fabricClusterAutoupgradeEnabled setting value. If it doesn't, run a config upgrade using the updated config.

Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath config.json

Edit: After you update this value you will need to wait 48 hours for the warning to go away.

Stratford on Tue, 29 Nov 2016 21:06:29

The FabricClusterAutoupgradeEnabled  shows True from Get-ServiceFabricClusterConfiguration.

When I run Start-ServiceFabricClusterConfigurationUpgrade, the exception is generated below.

PS C:\servicefabric> Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath .\ClusterConfigV2.json
Start-ServiceFabricClusterConfigurationUpgrade : Exception of type 'System.Exception' was thrown.
At line:1 char:1
+ Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath .\C ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Microsoft.Servi...usterConnection:ClusterConnection) [Start-Ser
   viceFa...gurationUpgrade], Exception
    + FullyQualifiedErrorId : StartClusterConfigurationUpgradeErrorId,Microsoft.ServiceFabric.Powershell.Sta

Also checked the MSDN and can't see there is a switch for cluster connection.

Stratford on Tue, 06 Dec 2016 17:04:42

Hi Max,

Could you please help the issue for running Start-ServiceFabricClusterConfigurationUpgrade below?

I already run  "Connect-ServiceFabricCluster jdev-p-v-brs01:19000" but Start-ServiceFabricClusterConfigurationUpgrade  keeps complaining the connection exception.