Tentacle stops after unhandled IOException

Legacy Uncategorised
1

I found a tentacle service that was stopped and went to look at its logs. The last log entry was an unhandled IOException. I am reporting this in case that is why the service stopped and Octopus needs to handle the exception more gracefully.

2016-05-22 00:37:11.6597     47  INFO  listen://[::]:10933/             47  Unhandled error when handling request from client: [::ffff:10.240.17.139]:49379
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 93
   at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 67
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func`2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 126
2

Upon further review, I have a lot of this exception in the logs and it doesn’t appear to be causing the service to stop because I would have noticed the service stopping that frequently.

3

Hi,

Thanks for getting in touch! It is odd that the Tentacle would stop like this, would you be able to provide us with a Dump log for the Tentacle? This should be able to provide us with the information we need to get to the bottom of this.

Below are instructions on how to perform this:
http://docs.octopus.com/display/OD/Capture+a+crash+dump

This is the link for you to upload the .dump file once this is below:
https://file.ac/rZvWVdlRlcM/

Looking forward to hearing from you.

Regards,
Daniel

4

Daniel, would the dump help with any unhandled exceptions or just when the
tentacle crashes? If it only records a dump when the tentacle crashes I’m
not sure I’d get a response back to you in any reasonable amount of time
because the tentacle does not crash often (but it does IOException often).

Thanks,
Kevin

5

Hi Kevin,

Thanks for the reply. No the dump file wont help us outside of a crash, there are some other things we can look at.

Are you able to confirm that you are getting the IOException while the service is still running?
There is also a chance that there could be some network issue at play here. (possibly only intermittent)
There is a tool created by one of our developers called TentaclePing that you can use to help diagnose if there are any connection issues between the Octopus server and Tentacle. The link is below.

Looking forward to hearing from you.

Regards,
Daniel