Upgrade Tentacles task causes bottleneck

Hi,

We encountered an issue earlier today when someone manually triggered the Upgrade Tentacles task.
This in turn queued every other task and would not run any task until the Upgrade tentacles task was cancelled. It had 77 machines to upgrade to a new version of Calamari.

We operate a 3 node HA setup and as the Upgrade Tentacles task was running on one node it would not run any other task on the other 2 nodes that had no tasks running.

Is this expected behaviour as it affects all spaces? What is the process that happens when the Tentacle gets upgraded?

We have told users to upgrade tentacles on a machine by machine basis or schedule the task to run at the weekend.

Kind Regards,
Micheál Power

Hi @mikepower79,

Thank you for contacting Octopus Support.

I hope you don’t mind, I have a few clarifying questions:

  • Were the upgrades to the tentacle client and calamari triggered at the same time?
  • Were system tasks queued in addition to deployment tasks?
  • How long did the upgrade tentacles tasks run before canceling?
  • Out of the 77 machines, how many upgraded the tentacle client in that time?

Let me know at your earliest convenience.

Regards,
Donny

Hi @donny.bell,
Here is a screenshot of the task that was triggered:

I think it was the upgrade to tentacles that was triggered
Yes system tasks were queued along with deployment tasks
We cancelled it after 25 mins
18/77 machines but some failed. see below:

Kind Regards,
Micheál

Hi @mikepower79,

Thank you for getting back to me.

Going forward, I think the best path may be to upgrade tentacles as needed individually or in small groups as you mentioned previously. If you would like to upgrade all of the tentacles at once, I would recommend turning maintenance mode on (via Configuration -> Maintenance), then initiating the tentacle upgrades to prevent other tasks/triggers from attempting to run during the process.

Let me know if you have any additional questions.

Regards,
Donny

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.