We have 5 clients we deploy to and after upgrading to 1.6.1.1718 we have been having a lot of issues with 1 of them. Deployments will appear to just hang after a while. If I let them just run for 20 to 30 min I get get timeout exceeded errors and A child activity failed: The underlying secure session has faulted before the reliable session fully completed. The reliable session was faulted. errors. The big difference between this client and the other 4 is we are deploying through a VPN tunnel to a remote data center. Stopping the deployment (or letting it error out) then restarting the octopus server service allows the deployment to move along but it hangs again. It took 14 tries and 4 hours last night to do a deployment that normally takes 30 min.
Hi,
Thanks for sharing this issue. Which version were you using prior to upgrading to 1.6.1? I take it this problem didn’t exist before the upgrade? If you let me know roughly the version you were using I can try and work out what we changed between releases that may have affected you.
Paul
We were a few versions behind. It was 1.4 something.
Are there any logs I can provide that would shed some light on this issue?
Hi Jack,
The Windows event viewer should show any warnings/errors encountered by Octopus. Also, if you could attach a full deployment log when the error occurs that would help.
We did add support for proxy servers between 1.4 and 1.6.1 - is it possible that a proxy server is being used in your organization? Octopus might be trying to send VPN-bound traffic over the proxy. This is the only change I can think of that might affect it.
Paul
I have attached the raw output of the 14 attempts the deployment took. We went to productions last night and it only took 5 but it still took almost an hour to get the packages to the tentacles when before the upgrade it had been taking 10 min.
OctoLog.zip (127 KB)
I did a deployment this morning and just let it run until it failed. I included both the raw output and all the windows errors generated.
OctoLog2.zip (7 KB)
Were there any changes made that would change the number of concurrent connections Octopus opens with the tentacles? I’m wondering if we are exceeding some limit of the VPN tunnel we weren’t exceeding before.
Hi Jack,
Thanks for the updates. I can’t find any recent changes that would affect you - the only change we’ve made in this area recently was to increase our send timeout from 1 minute to two minutes.
In the next release I think we’ll need to add some controls so that only X number of packages are downloaded/uploaded at once to allow you to control the bandwidth usage better. For now, you may need to downgrade back to a 1.4 release if that still works for you.
Paul
After much, much searching this turned out to be an hardware issues that started at the same time we upgraded. Thanks for looking into it.