HDInsight cluster creating with ADLS as default store

Category: sql server hdinsight


Nagesh Subramaniam on Fri, 18 May 2018 21:00:03


My use case is to provision HDI clusters on demand with the default storage as ADLS. The files will be persisted in ADLS and metadata in an external metastore.
I have already created a service principle and have the certificate and I want to use this when i re-provision the clusters.

I am using azure automation and powershell to do it. 

The certificate and password for the certificate is in key valut.

I have the pasted the code snipped below 

# getting the certificate content from key valut 

$vaultName = "xxxxx"
$certificateName = "yyyy"
$cert = Get-AzureKeyVaultSecret -VaultName $vaultName -Name $certificateName
$certBytes = [System.Convert]::FromBase64String($cert.SecretValueText)

$objectId = "xxx-yyy-zzzz"

#Provisioning cluster here

New-AzureRmHDInsightCluster `
    -ResourceGroupName $resourceGroupName `
    -ClusterName $clusterName `
    -ClusterType $clusterType `
-Location $location `
    -ClusterSizeInNodes $clusterSizeInNodes `
    -OSType $clusterOS `
    -Version $clusterVersion `
    -HttpCredential $httpCredential `
    -DefaultStorageAccountType AzureDataLakeStore `
    -DefaultStorageAccountName "$defaultADLSName.azuredatalakestore.net" `
    -DefaultStorageRootPath $storageRootPath `
    -SshCredential $sshCredentials `
-ObjectID $objectId `
    -AadTenantId $Tenantid `
    -CertificateFileContents $certBytes `
    -config $config 

However, I am getting this error 

New-AzureRmHDInsightCluster : Cannot bind parameter 'Config' to the target. Exception setting "Config": "Object 
reference not set to an instance of an object."
At line:74 char:13
+     -config $config
+             ~~~~~~~
    + CategoryInfo          : WriteError: (:) [New-AzureRmHDInsightCluster], ParameterBindingException
    + FullyQualifiedErrorId : ParameterBindingFailed, .Commands.HDInsight.NewAzureHDInsightClusterCommand

Appreciate any help with this 




Nagesh Subramaniam on Fri, 18 May 2018 21:44:41

Please ignore the error message before - 

Below is the actual error message 

New-AzureRmHDInsightCluster : BadRequest: User input validation failed. Errors: The request payload is invalid. ADL 
configured as defaultFS ... but no Service Principal has been configured
At line:58 char:1
+ New-AzureRmHDInsightCluster `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [New-AzureRmHDInsightCluster], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCommand

CHEEKATLAPRADEEP-MSFT on Sat, 19 May 2018 08:01:07

As mentioned in the error message “ADL configured as defaultFS.. but not service principal has been configured”, you may re-check “Service Principal which is set to have access to ADLS”.

For more details, you may refer the below links:

Create HDInsight clusters with Data Lake Store as default storage by using PowerShell.

Create HDInsight Cluster with Pre-Existing Azure Data Lake Store and Hive Metastore.

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.


If this answer was helpful, click “Mark as Answer” or “Up-Vote”. To provide additional feedback on your forum experience, click here

CHEEKATLAPRADEEP-MSFT on Tue, 22 May 2018 02:51:43

Checking in to see if the above answer helped. Let me know if there are still any additional issues I can help with.