Adding Environment Broke my Prod - Warning/Informational

Legacy Uncategorised
1

I just wanted to write this as a warning/informational post to other people using Octopus (and for the developers to possibly address).
Yesderday, I added a new environment for “Pre-Production” testing right before our Production environment.
Our servers are hosted in AWS and use the REST api to request a new deployment when a box comes into the Auto Scaling Group.
Because I hadn’t deployed the production release onto the “Pre-Production” environment, the rest API started returning a failure message to the script that was supposed to deploy.

We didn’t see any indication of this in the Octopus GUI, and there wasn’t really a need for this to happen as the Release had already gone to the “Production” environment, it seems unnecessary to require it to go back into all previous ones before deploying to a new box.

I’ve resolved this issue by pushing to the “Pre-Production” environment, but wanted to let others know this could happen as it had a negative impact on our prod environment and I don’t want that to happen to anyone else.

Developers: I assumed like most things that the workflow would be bound to the release, or at least the requirements for pushing would be and wouldn’t retroactively take hold. This was probably a bad assumption on my part, but one others may make in the future. I’d recommend forcing a deployment (or highly recommending one) if the requirement can’t be bypassed for things that have already passed by an environment.

Thanks!

2

Hi Jason,

Thanks for getting in touch. I’m going to make a proposal we adjust the logic to help avoid this situation from surprising people. Before I do I’d like to confirm it with you to make sure it would have helped in your situation.

Using your example, let’s say Production already had 3.4.3 deployed into it, then you added Pre-Production with nothing deployed.

So, when trying to deploy 3.4.3 onto a new machine in Production, instead of blocking the deployment, we should recognise that 3.4.3 is already in Production, and we should allow the deployment onto the new machine without considering Lifecycle at all.

In pseudo-code:

bool IsValidDeployment(environment, release, lifecycle) {
  if (release already deployed to environment) return true;
  // Now do normal lifecycle checks
}

Getting more specific, what would seem like the least surprising or least dangerous logic?

  • release has already deployed to this environment (considering all available history)
  • this is the current release in the environment (blocking if not the current release)

If you can get back to me, I can take the next steps from there.

Hope that helps!
Mike

3

That would be perfect.
Thanks!

4

Hi Jason,

Thanks for getting back to me. Here’s the GitHub Issue to track.

Which way do you think we should go here (see my previous question):

Getting more specific, what would seem like the least surprising or least dangerous logic?

a) release has already deployed to this environment (considering all available history)
b) this is the current release in the environment (blocking if not the current release)

Let me know what you think would suit your situation best: A or B?
Mike

5

For our use case A would be perfect.
If there are cases where B would be useful, I’m not sure what they are. If
anything comes to mind though, I’ll let you know. But for now, I think A
seems good.

6

Hi Jason,

Thanks for getting back to me!
Mike