Retention Policy, Number of Releases, Lifecycles, and Channels (oh my)

Legacy Uncategorised
1

now that Octopus has added Channels, there is a problem with the Retention Policy and how it counts releases.

we have code in our source control that we build from two different branches each night, one for QA and one for Release. at the end of the build process, we send a command to Octopus (using Octo.exe) to create a release, so one is created in the QA channel and one in the Release channel. Each of these channels are using a different lifecycle (a QA lifecycle that includes only the QA environments in one stage, and a Release lifecycle that has staging and production stages.) pretty standard for using channels.

the retention policy for both lifecycles is set to retain 3 releases.

so here’s the problem:

when the retention policy runs, it works it way through the lifecycles. it gets to the QA lifecycle first, and starts the process, keeping a few releases because they’re on the dashboard, then starts counting the other releases to determine what it can delete. it counts up to 3, and then sees a 4th release and deletes it. so far behaving correctly.
then it moves on and gets to the Release lifecycle. when it starts looking at the releases, it skips the first few because they’re on the dashboard, but when it moves on the older releases, when it gets to the first one, it says it should be deleted because the number of releases being retained has already reached 3!

so the releases retained for the project in the QA lifecycle are being counted against the releases for the same project in the Release lifecycle. this is not normally a problem, but this morning, the retention policy happened to run between when the releases were created and when they get used (a few hours later.) and so the most recent release in the Release lifecycle got deleted because it hadn’t been deployed yet and the counter had already reached 3 because of the earlier work in the QA lifecycle. and therefore the release that was then run was yesterday’s release, today’s having just been deleted.

I think the release counter needs to be reset to 0 for each lifecycle when running through the retention policy.

does this make sense? I can attach the output of a run of the retention policy if that will help.

2

Hi Mike,

Thanks for getting in touch! Sorry to hear you are having these issues.

If you could send through the raw log of the deployments in question that’d be great, we can then investigate the issue and get a fix in to ensure it doesn’t happen.

Thank you and best regards,
Henrik

3

it’s not a deployment, it the regular run of the Retention Policy.

and, unfortunately, that particular run has disappeared from the server (I have changed the policies to Retain all releases for now to avoid the problem.)

I will need to change the policies back to Retain 3 Releases and let it run again to get a log to share. will do this some time this week.

thanks.

4

had to get past a major release before I could get back to looking into this.

but in the meantime, we upgraded our Octopus version

so when I changed the retention policies back to “3 releases”, then let the policy run then checked on the results, it’s now behaving the way I would want, with it counting releases within each channel in the project.

so, it’s not a problem at this time. if I see the earlier behavior again, I will repost a problem and attach the log.

thanks.

5

Hi Mike,

Great to hear it is working as expected (for) now, thanks for letting us know.

Best Regards,
Henrik

6

ok, it is still happening. I am attaching a tasklog of a run.

our build process, which ends in the creation of new releases in Octopus, runs at 11pm. most of the the deployments start around 1am, but the “02) JobManagement - GoLive” project in our QA environment is delayed until 5am.

note, all lifecycle phases have the same retention policy of keep 3 releases.

lines to pay attention to:

180: working on lifecycle “JobManager - Release”
181: policy Last 3 items
182: phase 1: “Staging and Production” last 3 releases
183: phase 2: “Demo” last 3 releases
192-194: releases kept, undeployed 1, 2, 3
195-197: releases kept, other reasons (dashboard, rollback)
198-200: releases kept, deployed 1, 2, 3

291: lifecycle “JobManager - QA”
292: keep 3 releases
293: phase 1: QA, keep 3 releases
303: release deleted, undeployed, 3 of 3 already being kept

304: (older) release kept, dashboard
305-7: (older) releases kept, deployed 1, 2, 3

so line 303 refers to the release that was created just a few hours prior, but because the undeployed release counter for this project but from other lifecycles had already reached three, it got deleted.
the counter for deployed releases seem to be working correctly, as it counted three of them. but the counter for undeployed doesn’t seem to be reseting.

of course, the ability of a project to be in more than one lifecycle only came about when you added channels, so retention policy would need to have been adjusted to deal with that.

please let me know if I need to explain what is going on better. thanks.

ServerTasks-20221.log.txt (48 KB)

7

Hi Mike,

I’ve raised #2418 to have this investigated and fixed as soon as we can.

Sorry for the troubles this is causing, and I hope we can resolve it in the near future.

Thank you and best regards,
Henrik

8

any developments?

this problem is really hurting us. If I leave the retention policy set to 3 releases, it is deleting new undeployed releases and leaving older releases to be deployed when they are run hours later.

If I set all policies to keep all, releases and files just keep piling up and I have to manually clean them out. work I shouldn’t have to be doing.

thanks for your help.

9

Hi Mike,

Just to jump in here. There must be a number between 3 and all that will satisfy the time gap between the retention policy running and the build up of undeployed releases. How many releases are created in a few hours? Could you set the number to something more like 10 or 20?

Yes this is just a workaround, but should mean no manual work for yourself.

Vanessa

10

FYI we’re experiencing this issue too, with releases being created on a daily basis within multiple channels in the same project:
Release Elements 1.70.6044.54614 is in phase Deploy to ST, with a policy of Last 3 items; it will be deleted. Reason: Undeployed, but 3 of 3 have been kept

We’re running 3.3.20, and have a retention policy of 3 releases set. I’ll change it to a higher value to test, but realistically having this behaviour fixed is the best option for us