The azure powershell cmdlets such as Set-AzureSubscription, Select-AzureSubscription and Remove-azureSubscription use common file across all powershell instances for managing the azure subscriptions. For the system user (account that runs the tentacles), this path is C:\Windows\system32\config\systemprofile\AppData\Roaming\Windows Azure Powershell. We run a tentacle for each service we deploy to azure (currently 22 services) which has led to a situation where deployments fail if two or more kick off close to each other. This seems to be happening more and more often for us as we ramp up our use of Octopus. The exceptions thrown are due to file locking issues (often for set-azuresubscription) as well as subscription entries ‘going missing’ (often for remote-azuresubscription) from the common files.
We have been able to reproduce one of the issue by editing the ‘deploytoazure.ps1’ for two projects to just sleep for 60 seconds and then kicking off a deployment for the two projects off about 20 seconds apart. One will also fail with an exception in the set-azuresubscription command.
To work around this we have had to edit the bootstrapdeploytoazure script to add some retry / try catch around ‘set-azuresubscription’ and adding the –SubscriptionDataFile option to all *–azuresubscription cmdlets so each deploy uses its only subscription file.
Attached is our hacked up script if anyone is interested.
Hoping Octopus might add some retry / fault handling to future releases.
BootstrapDeployToAzure.ps1 (3 KB)