All Taks go timeout after upgrade

I have the same issue like this :
http://help.octopusdeploy.com/discussions/problems/16710-tentacle-health-check-db-backup-tentacle-upgrade-tasks-timeout-after-23-upgrade

What was the solution?

Hi Peter, thanks for getting in touch.

Which Octopus version are you using? Have you located a “stuck” task?

Regards,
Nick

Version: Octopus Deploy 2.3.6.1385

Yes, I have.
All Task are waiting for a scheduled databse backup

backup_task.PNG

Hello,

Same server, same issue here (workmate of Peter). The backup task yesterday failed with error ‘insufficient disk space’ which is strange because there is enough free space on the backup drive. Anyway, I stopped the automatic backup in the config just to see if this will solve the issue or not.

Now, the tentacle upgrade task is queued, and nothing happens… I’ve restarted the service and checked the octopusserver.txt but I don’t see anything interesting:

2014-04-16 10:16:20.9023 INFO Web server will stop serving requests
2014-04-16 10:16:21.4504 INFO Shutting down the RavenDB document store
2014-04-16 10:16:21.4504 INFO Shutting down the embedded web server
2014-04-16 10:16:35.1388 INFO Browse your Octopus server at: https://localhost:443/
2014-04-16 10:16:41.8919 INFO You can browse the RavenDB server at: http://localhost:10931/
2014-04-16 10:16:45.8476 INFO Resolving for Pipefish.Hosting.ActivitySpace
2014-04-16 10:16:46.2747 INFO Distribution service listening on: 10943
2014-04-16 10:16:46.5508 INFO Web server is ready to process requests
2014-04-16 10:17:21.3037 INFO Delete: ServerTasks-7885
2014-04-16 10:17:21.4858 INFO Deleted: ServerTasks-7885
2014-04-16 10:17:21.4858 INFO Deletion complete
2014-04-16 10:21:51.4112 INFO Delete: ServerTasks-7876
2014-04-16 10:21:51.4342 INFO Deleted: ServerTasks-7876
2014-04-16 10:21:51.4342 INFO Deletion complete

Hello,

If you navigate to the stuck task, and click Cancel, it should cancel the task (you’ll need to be running 2.3.6).

Otherwise, can you go to the Octopus Manager (remote desktop onto your Octopus Deploy server, and find Octopus Manager in your start menu), and use the “Repair” button to repair the database? This should also forcibly cancel any stuck tasks.

Paul

Thanks, the raven db repair was helpful. Now the tentacle healt check still times out for one particular tentacle (the one I want to upgrade as the last one). Canceled the task, and started the upgrade task manually.

This seems to be stucked again, the log says ‘The tentacle received a request to shut down and restart’ since 3 minutes:

                    |     Success: Deploy the Tentacle package
11:37:08   Info     |       Beginning deployment...
11:37:08   Verbose  |       Begin deployment of package: Octopus.Tentacle.2.3.6.1385
                    |       Package file path:           e:\Octopus\Applications\.Tentacle\Packages\Octopus.Tentacle.2.3.6.1385_C5A662F3E97FAB418D7F6D2B726843D0.nupkg
                    |       Package file hash:           220100e014152fb2525b699a3daff60b617aa23e
                    |       Tentacle Agent information:
                    |       - Machine name:       BPGTS-DEPLOYSRV
                    |       - Is 64-bit:          True
                    |       - Service user name:  SYSTEM
                    |       - CLR version:        4.0.30319.18408
                    |       - Current directory:  C:\Windows\system32
                    |       - OS version:         Microsoft Windows NT 6.2.9200.0
                    |       - Tentacle version:   2.3.6.1385
11:37:08   Verbose  |       Package will be extracted to: e:\Octopus\Applications\.Tentacle
11:37:09   Verbose  |       Running procedure Delete temporary package files
11:37:11   Verbose  |       Storing a record of the deployment.
11:37:11   Info     |       Deployment complete.
                    |     
                    |       Success: Deploy.ps1
11:37:10   Info     |         Reconfiguring the Octopus Tentacle Windows Service to use path: E:\Octopus\Applications\.Tentacle\Octopus.Tentacle\2.3.6.1385\Tentacle.exe
11:37:10   Info     |         Tentacle instance: Tentacle
11:37:10   Info     |         Octopus Deploy: Tentacle version 2.3.6.1385
11:37:11   Info     |         Service reconfigured
11:37:11   Info     |         Reconfiguration complete!
                    |       
                    |     Running: Restart the Tentacle service
11:37:11   Info     |       The Tentacle has received a request to shut down and restart
                    |

Sounds like we’re making some progress! The failure to restart is most likely recorded in the Tentacle’s log file - C:\Octopus\Logs\OctopusTentacle.txt by default. Are you able to retrieve an error message from the end of that file?

You might try restarting the Octopus Tentacle service by hand on the machine and check the Windows event log if it fails to start.

Let us know what you find - hope this hits the mark…

Cheers,
Nick