Deployment Blocked with "Another deployment is currently uploading package"

Legacy Uncategorised
1

Hi I am having a problem with one of our Octopus deployment jobs where by a phantom job seems to blocking the intial upload of the package to the remote server. The target server is a redhat linux machine…

The deployment hangs with the message -

Another deployment is currently uploading package [xxx-service] version 80 to [target-server]

Could anybody help me to troubleshot this issue? Where can I start looking to see why this is blocked? Is there any solution short of rebooting the octopus server (we have already rebooted the target linux box)?

2

Hi!

Thanks for getting in touch! This message is displayed when the Octopus Server believes it is already uploading the same Package (ID and Version) to the same Deployment Target/Machine. It does this by attempting to acquire a named System.Threading.Semaphore, which is released when the running package upload has completed.

Some things to check with you:

  1. Does the machine belong to more than one environment?
  2. Is the machine targeted by more than one project?
  3. Is the same package used by more than one project?
  4. Is the same package used by more than one step in the same project?
  5. Does this happen frequently?
  6. Which version of Octopus Deploy are you running?
  7. Could you send the raw task logs for any relevant deployments to our support email address referencing this post?

We very rarely have reports of this kind of behaviour, but your answers may help uncover a bug we haven’t discovered yet.

In the meantime, if the process is completely blocked, you will need to restart the Octopus Server windows service, which will release the Semaphore.

Hope that helps!
Mike

3

We are having the same issue.

Answers to questions:

  1. Does the machine belong to more than one environment? No
  2. Is the machine targeted by more than one project? Yes
  3. Is the same package used by more than one project? No
  4. Is the same package used by more than one step in the same project? No
  5. Does this happen frequently? Recently upgraded our server so too early to tell
  6. Which version of Octopus Deploy are you running? 3.7.11

Raw task logs sent.

4

Hi Jan,

Thanks for that info, nothing odd jumps out at me. So I have a few more questions:

  1. Are you also deploying to a Linux machine (ie SSH target)?
  2. Is the 2.6 server still running on that server?
  3. Does it occur every time you do a deployment?
  4. Have those package versions been deployed before to that machine?
  5. Could you please enable Trace logging and send us the log file if the issue occurs again.
  6. If it does occur again, could you check the task tab to see if there are any other tasks running

I’ve added some extra logging around this process in the next release of Octopus (3.7.14), that may help diagnose the issue.

Regards,

Robert W

5

Hi Jan and Shri,

I think I’ve found the cause of the issue. If a previous deployment is cancelled, the packages may still be downloading when the new deployment runs. This is apparent when the download takes a long time, or fails after some time and retries. See this issue for more details.

I want to be sure we are not dealing with two seperate issues, so could the above scenario have happened in your case (ie a cancelled deployment and then a hung deployment in succession)?

Regards,

Robert W

6

Hi Robert,

The situation we were in matches the below cause. We had server space issue due to Octopus retention policy (which is now changed), because of which couple of jobs failed and then we had to cancel a couple of jobs as it was waiting indefinitely. It will be good if you can fix this issue.

Cheers,
Shri

7

Hi Robert,

I can confirm that a deploy was cancelled. The machine running the tentacle was having some hardware issues (BSOD) that caused the original deploy to fail.

Restarting the Octopus server was required. Neither restarting the tentacle host nor restarting the tentacle service (multiple times) was enough.

Greetings,

Jan

8

Hi Jan and Shri,

Thank you for confirming. We have release version 3.8.0, which fixes the issue.

Robert W