2 of 10 tentacles will not upgrade to 3.3

I have 10 tentacles. 8 upgraded from 3.3 beta 2 to 3.3 official perfectly and two are refusing to upgrade. I have tried restarting the tentacle service on the one machine I have easy access to and that did not help.
Attached is the Raw output of the task showing that it continually retries. Note that the log shows an occasional error.

This is what the UI shows while attempting the upgrade.

Running Tentacle version 3.3.0-beta0002
Beginning upgrade
Waiting for Tentacle to shut down and restart with the new version 3.3.0...
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
Running Tentacle version 3.3.0-beta0002
...

ServerTasks-289.log.txt (62 KB)

When I cancel the upgrade for these tentacles, orange ‘Cancel’ button changes to a green ‘Try Again’ but the server continues to try to push the update to the tentacle. On the tentacle machine I can watch a new job coming in to c:\octopus\work every 5 seconds.

I found the folder on the tentacle that has the upgrade logs. I think this is a permissions error because the Tentacle is running as a service account that is not an admin on the machine. The help documentation says that you MAY need to give the service account the ability to start/stop services depending on your application but it does not say that this is necessary to upgrade tentacles. You should consider fixing the documentation to clarify this. http://docs.octopusdeploy.com/display/OD/Installing+Tentacles

Error: System.InvalidOperationException: Cannot open OctopusDeploy Tentacle service on computer '.'. ---> System.ComponentModel.Win32Exception: Access is denied
   --- End of inner exception stack trace ---
   at System.ServiceProcess.ServiceController.GetServiceHandle(Int32 desiredAccess)
   at System.ServiceProcess.ServiceController.Stop()
   at Octopus.Upgrader.ServiceBouncer.StopService(ServiceController service) in Y:\work\refs\tags\3.3.0\source\Octopus.Upgrader\ServiceBouncer.cs:line 50
   at Octopus.Upgrader.ServiceBouncer.StopAll() in Y:\work\refs\tags\3.3.0\source\Octopus.Upgrader\ServiceBouncer.cs:line 24
   at Octopus.Upgrader.Program.Main(String[] args) in Y:\work\refs\tags\3.3.0\source\Octopus.Upgrader\Program.cs:line 36

Hi Kevin,

You are correct, our documentation is misleading and has been updated. In order to do a Tentacle upgrade the Tentacle account must have enough permissions to install itself and start a service.

When you push the “Cancel” button the server will continue to check if the Tentacle has been upgraded until it times out. It runs the same as all other tasks: we do not interrupt jobs in progress, the cancellation takes effect once the current job has completed.

Thanks for your feedback.

Cheers,
Shane

You said the cancellation takes effect once the current job has completed. What I saw what the a new folder would get created in the tentacles work folder, it would run, get deleted. This would repeat indefinitely. The folders were only there for a blink but I could see that they were incrementing in numbers each time. Doesn’t that mean it was really hundreds of jobs getting run, not 1, and that it should have cancelled? In this situation I could only stop it by restarting the main octopus server. This is recreateable for me.

Hi Kevin,

I’ve just tried doing a Tentacle update to a machine without permissions and the task ran for 15 minutes before timing out. That does seem like a really long time and I can understand the confusion when a bunch of folders are being created in the Tentacle work folder for so long after the task had been cancelled. Are you able to re-create it again and leave it for over 15 minutes to see if the task does eventually stop?

If it is timeout related we could decrease the timeout but I am a bit concerned about environments that have hundreds of Tentacles and actually need that time to perform updates.

Cheers,
Shane