We are finding ourselves working around a limitation of Octopus in terms of how it handles the nuances of our environment and wondering what you suggest.
Here we have a large number of apps and services which are deployed multiple times into production with different configurations per instance. For example for AppA we may have MachineA with instances 1 to 5 and MachineB with instances 6 and 7. Each instance has different configuration file to specify things like where to find the dependencies it needs (e.g. databases, other services, etc). In development and staging we would typically only have one instance of each application.
We usually perform piecemeal upgrades, upgrading each instance of each application one at a time, rather than upgrading them all together (because the users of the applications cannot be interrupted all at the same time).
How we manage this is to use a build step per instance of the application and using configuration variables or config transforms to handle it. This kind of works but is not ideal from an operational point of view, due to the following reasons:
-
No visibility over which version each instance is at. This is a big one and makes upgrades difficult as we want to upgrade an instance at a time.
-
No visibility over which server a specific instance of an application is deployed to, without going and reading the task log (you would have to find the task log which corresponds to the build step of the instance you want to check which takes time when you are trying to quickly troubleshoot a problem)
-
No easy way to configure onto which server each instance of each application is deployed. You can create a role per instance and set roles per server in each environment but this is clunky, especially when you need to change the topology and move stuff around.
-
If a server goes down it’s not easy to move stuff around e.g MachineB goes down we want to move instances 6 and 7 onto MachineA until we can fix MachineB
-
We like a release to be tied to a specific Nuget package. Using build steps it’s possible to have a single release with different versions of the same package. This seems wrong.
-
Adding new instances of an app is not simple as you need to add a new build step and configure it correctly. You can use custom step templates for this which helps but it’s not great as they weren’t really meant for this purpose.
-
If you do use a step template and then you change something about it then you need to go and update each instance to have the new template, which is not ideal
-
Side by side upgrades are hard. For example, say we have ServiceA and a new version is created which breaks all clients unless they upgrade. We would deploy and run the new instance side by side with the old one, we would then upgrade one application instance to talk to the new service. If that works then over time we would upgrade all the others, then when all clients were upgraded we would decommission the old instance of the service. Octopus doesn’t manage this for us and we have to do this kind of work outside of Octopus.
Just wondering what you think about this?
Thanks