Deploying 100+ applications to one server

Legacy Uncategorised
1

Hello,

I am trying to trigger deployment of 100+ applications to one server, I wrote a powershell script that loops through all of the deployment projects in octopus and kicks of deployments. It usually succeeds for first couple, then the rest fails and tentakle ends up crashing.

Here are couple of errors that we usually get
An error occurred when sending a request to ‘SERVER’, before the request could begin: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

An error occurred when sending a request to ‘SERVER’, before the request could begin: No connection could be made because the target machine actively refused it.

What would be the right approach to do this kind of deployment? Is there a way to make tentakle queue the deployment and deploy one at the time?

2

Hi Alex,

Wow. That is a lot of projects to one Tentacle.

In theory you should be able to do this. They should just queue on the server.

Could you send an example of the PowerShell you are using to trigger the deployments? I will see if I can replicate on a similar scale.

One idea may be to space them a little. Perhaps trigger a couple at a time, and wait for them to complete before triggering more?

Regards,
Michael

3

Hello Michael,

Sure, here is simplified version of the script to give you an idea, all it does is it gets a list of projects, loops through them and deploys.
$projects = MakeOctopusRequest “/api/projects/all” “” “GET” | ConvertFrom-Json
foreach ($p in $projects)
{
$release = GetReleaseToDeploy $p
If($release){
$releaseID = $release.Id
$environmentID = $OctopusParameters[‘Octopus.Environment.Id’]
$DataToPost = “{”“ReleaseId”":""$releaseID"","“EnvironmentId”":""$environmentID"","“SkipActions”":[],"“SpecificMachineIds”":[],"“QueueTime”":null,"“FormValues”":{},"“ForcePackageDownload”":false,"“UseGuidedFailure”":false}"
MakeOctopusRequest “/api/deployments” $DataToPost “POST”
}

Also one thing to mention, each project contains 2 web apps and 1 DBUP project, so deployment of each project takes about 2 minutes.

Ones deployment is triggered (Post to /api/deployments) is there way to watch that particular deployment through api requests to see when it’s done?
Is there a way to increase timeout between octopus server and tentacle?

At first we had an issue with space, octopus was filling up the disk with packages, but now we increased it to 80 gb and I think it’s plenty for now. But is there a way to set retention policies in a such way that it won’t store any packages on the tentacle ? I think I was able to get it to store just one per app, but that’s still a lot of space usage,(100+ projects, 2 webapps each). I’ve looked at the way it works, I could probably write a script to remove a package from Applications and Files directories,( not sure if I also need to remove an item from DeploymentJournal?)

4

Hi Alex,

What are you doing seems reasonable. I haven’t yet had a chance to create a test environment to mimic yours, but to answer your questions:

is there way to watch that particular deployment through api requests to see when it’s done?

Yes. The deployment object has a TaskId property. With this, you can make a GET request to /api/tasks/{id}. The Task object returned has a State property, which you can inspect. I would recommend polling the task, and wait for it to complete.

Is there a way to increase timeout between octopus server and tentacle?

Yes. For Tentacle/Server communications, we use a library called Halibut. It has a number of default timeouts that can be overridden by config.

<add key="Halibut.ConnectionErrorRetryTimeout" value="00:15:00"/>
<add key="Halibut.TcpClientConnectTimeout" value="00:02:00"/>
<add key="Halibut.TcpClientHeartbeatSendTimeout" value="00:02:00"/>
<add key="Halibut.TcpClientHeartbeatReceiveTimeout" value="00:02:00"/>

The config lines above are those I believe most likely to be relevant to your situation. These should be added to the appSettings section of Octopus.Server.exe.config, located by default at C:\Program Files\Octopus Deploy\Octopus.

I’ll briefly explain what each of these value does, and you can see the default values in the code.

ConnectionErrorRetryTimeout: If an error occurs when sending message, and Halibut believes it is safe to retry, it will retry up to 5 times, or until this timeout is exceeded. Increasing this value will allow your failed messages to be retried for longer.

TcpClientConnectTimeout: When there are no connections in the pool, or the pooled connections have expired, Halibut makes a new connection. This timeout applies when making this connection.

TcpClientHeartbeatSendTimeout and TcpClientHeartbeatReceiveTimeout: When taking a connection from the pool, small heartbeat request/response messages are sent to verify the connection is still valid. These timeouts apply to this process.

is there a way to set retention policies in a such way that it won’t store any packages on the tentacle ?

No. It will always persist the packages from at least the latest release. You’re right, you should be able to remove the package files yourself. You shouldn’t need to touch the DeploymentJournal (it will skip the package file if it doesn’t exist) and in general I wouldn’t recommend it.

I hope this helps! Don’t hesitate if you have any more questions.