We recently deployed to a ~50 machine environment and had issues with the deployment logging slowing down and eventually failing completely. The actual deployments succeeded though. As we are on the verge of deploying to another environment 7x larger, I’m curious what people are running as their hardware for the Octopus server? Are we simply logging too much information? We have fairly verbose Deploy.ps1 scripts.
Thanks for getting in touch and sorry about this. The latest version of Octopus includes some changes to make rendering the task log faster, and we’re making some improvements in 2.4 which will mean Octopus handles scripts that produce 000’s of lines of log output much better. For now, dialing down the verbosity of the scripts will help. A machine with 2-4GB of RAM should be able to handle deployments to > 100 machines fine so long as there’s not too much output from scripts being logged. I’d love to know if you’re encountering any other bottlenecks though.
Thanks for the info Paul.
We’ve taken the logging down a notch and have another ~50 deployment coming up so we’ll get some results from that.
If we were to add more metal to this problem should this be in the form of more RAM or faster disk IO or something else?
Our current setup has 6GB memory and is a 2 CPU, Windows Server 2012 64-bit machine.
Disk I/O will help a great deal, however there are message rate limitations that might still prevent full utilization of the hardware.
If the majority of verbose output is produced by a few commands, Octopus 2.4 includes a new command that can be run in Deploy.ps1:
Octopus-EnableOutputBuffer -Operation "Big XCOPY"
This will group log messages for the command into a single event, which will improve handling of it a great deal (many orders of magnitude speed-up for large outputs).
If errors are written the buffer flushes automatically, so apart from some short delay in seeing output and less resilience to power outages, there aren’t too many caveats.
Let me know if this sounds like it would help, and if you have a chance to try it. Notes on the new version are at: http://octopusdeploy.com/blog/2.4
This sounds like a good feature. We’ll try this in conjunction with the other steps and see how we go.
Thanks for being so responsive - much appreciated.