Details on SSH health checks

Vern_DeHaven · August 13, 2020, 2:25pm

Hello all,

We’re trying to troubleshoot some network issues we’re having, and I wondered if we could get some details on what the default health check against an SSH endpoint does. We’re expecting to see five retries with a one second delay between, but the logs show a single attempt. Is the log masking the five attempts in this error, or is this a single failed attempt?

Regards,
Vern DeHaven

                |   == Failed: Check deployment target: HAProxy ==

08:54:13 Verbose | Performing health check on machine

08:54:13 Verbose | Establishing SSH connection…

08:54:34 Fatal | Could not connect to SSH endpoint: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

08:54:34 Verbose | Could not connect to SSH endpoint: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     Could not connect to SSH endpoint: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     Octopus.Server.Orchestration.Targets.Ssh.SshEndpointConnectionException: Could not connect to SSH endpoint: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

                |     at Renci.SshNet.Abstractions.SocketAbstraction.Connect(IPEndPoint remoteEndpoint, TimeSpan connectTimeout)

                |     at Renci.SshNet.Session.SocketConnect(String host, Int32 port)

                |     at Renci.SshNet.Session.Connect()

                |     at Renci.SshNet.BaseClient.CreateAndConnectSession()

                |     at Renci.SshNet.BaseClient.Connect()

                |     at Octopus.Server.Orchestration.Targets.Ssh.Connectivity.SshClientFactory.EstablishConnection[TClient](Func`2 clientConstructor) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\Connectivity\SshClientFactory.cs:line 98

                |     --- End of inner exception stack trace ---

                |     at Octopus.Server.Orchestration.Targets.Ssh.Connectivity.SshClientFactory.EstablishConnection[TClient](Func`2 clientConstructor) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\Connectivity\SshClientFactory.cs:line 98

                |     at Octopus.Server.Orchestration.Targets.Ssh.Connectivity.SshClientFactory.EstablishSshConnection() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\Connectivity\SshClientFactory.cs:line 63

                |     at Octopus.Server.Orchestration.Targets.Ssh.Connectivity.ConnectionScope.get_SshClient() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\Connectivity\ConnectionScope.cs:line 65

                |     at Octopus.Server.Orchestration.Targets.Ssh.SshRemoteEndpointFacade.ReadUntilComplete(StartScriptCommand command, ScriptOutputFilter filter) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\SshRemoteEndpointFacade.cs:line 105

                |     at Octopus.Server.Orchestration.Targets.Ssh.SshRemoteEndpointFacade.UploadExecuteDownload(StartScriptCommand command) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Ssh\SshRemoteEndpointFacade.cs:line 120

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.SshBashExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 boostrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, Nullable`1 isolationMutexTimeout) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\Immediate\ExecutionTargets\SshBashExecutionTarget.cs:line 80

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteRawScript(ScriptCollection scripts, IReadOnlyList`1 files, Boolean isRaw, Nullable`1 isolationMutexTimeout, ExecutionIsolation isolationLevel) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\Immediate\ImmediateExecutor.cs:line 89

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.RawShellCommandBuilder.Execute() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\CommandBuilders\RawShellCommandBuilder.cs:line 26

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 167

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.ExecuteOnDeploymentTarget(DeploymentTarget deploymentTarget, ActionHandlerInvocation actionHandler, TargetManifest targetManifest) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 145

                |     at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, VariableCollection variables) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 62

                |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.SshMachineHealthController.ConfirmBashAvailable(Machine machine) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\SshMachineHealthController.cs:line 78

                |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.SshMachineHealthController.CheckMachineHealth(Machine machine) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\SshMachineHealthController.cs:line 62

                |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.MachineHealthController.CheckHealth(Machine machine) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\Controllers\MachineHealthController.cs:line 48

                |     at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\HealthCheck\HealthCheckService.cs:line 92

                |     Octopus.Server version 2020.3.2 (2020.3.2+Branch.tags-2020.3.2.Sha.0d64b33d41be33a511fe1fab38f207d2348460d4)

08:54:34 Verbose | Recording health check results

jeremy.miller · August 13, 2020, 3:01pm

Hi Vern,

Thanks for reaching out and bringing this to my attention. I did some tests on an ssh target and I am seeing the same results myself. It seems like we only do the retries on listening tentacles. I’ll reach out to engineering and get back to you with more information.

Please feel free to reach out in the meantime with any other questions or concerns.

Thanks,
Jeremy

Vern_DeHaven · August 20, 2020, 7:29pm

Connection: keep-alive

jeremy.miller · August 20, 2020, 7:59pm

Hey Vern,

Nice one.

Sorry about the lack of communication. This one is on the engineers’ radars but I can’t make any promises about when it’s going to be fixed.

I’ve gone ahead and made an issue for you to subscribe to so you can automatically get updated as the issue progresses. https://github.com/OctopusDeploy/Issues/issues/6524

Please let me know if you have any other questions or concerns in the meantime.

Thanks,
Jeremy

system · September 20, 2020, 8:00pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.