Release gets stuck when uploading packages to polling tentacle

Legacy Uncategorised
1

Hi!

I’m having issues with a polling tentacle on our staging server. The tentacle installed without any issues and I’ve been able to run releases to it every now and then, but sometimes it gets stuck while uploading the packages. I’ve tried reinstalling the tentacle and it has worked a few times, other times not.

These are the specs of the server:
Win Server 2008 R2 Standard, SP 1 (64 bit)
ESET NOD 32 Antivirus 4
McAfee VirusScan Enterprise + AntiSpyware Enterprise 8.8

I tried running this command to allow TCP communication:
netsh advfirewall firewall add rule name=“Octopus Tentacle” dir=in action=allow protocol=TCP localport=10933 profile=domain

Didn’t help unfortunately.

I’ve also tried disabling NOD32 and doing a release, it worked well. But I can’t tell for sure if it was because NOD32 was disabled or not. As I said, it has been working every now and then and I just had 1 hour to test while NOD32 was deactivated.

Looking in the event viewer there are a few of these errors:
2014-05-05 22:19:35.6341 ERROR Error posting to: https://[OURBUILDSERVER]:10943/mx/v1
System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond IPADDRESS:10943
at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 39
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 298
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 173

What should I try now?

Thanks
Andreas Nellmon

2

I just noticed that the port command I used was set to port 10933 so I tried running the command again with 10943, but it made no difference.

3

Hi Andreas,

A firewall’s definitely the first place I’d look in this instance - I wonder if NOD implements some kind of firewalling?

You might get some clues by remoting to the Tentacle machine, then visiting the “mx” URL from the error message above using a web browser. It won’t show a nice page like the Tentacle does, but you should at least get some SSL warnings and an error in response, if you’re getting to the Octopus.

Checking the Configuration > Diagnostics tab for errors on the Octopus side may also yield some clues.

Let us know what you find!

Regards,
Nick

4

Hi!

Going to https://[OURBUILDSERVER]:10943/mx/v1 in a browser (from the server) I first get to an SSL page, I accept and then get here: http://screencast.com/t/mAokcJsV6p.

Windows firewall is off: http://screencast.com/t/Y8722qvtz3a. But there are inbound rules for ports 10943 and 10933 as well: http://screencast.com/t/a8JdZpwMyRG

Our octopus server seems to have the correct inbound rules in the firewall (which is turned on): http://screencast.com/t/WiE4Jj69FPyj

Looking at the installed software it seems like there’s no firewall installed: http://screencast.com/t/6ESjOPRKMta. Unless NOD32 does have a firewall, but I believe that’s a separate product. This version at least says it has antivirus and spyware protection. McAfee agent is installed as well but it is not in use.

I did check the event viewer on the octopus server and could not find any errors thrown in regards to our staging server.

The connectivity tab for the staging machine does not tell me anything: http://screencast.com/t/1zURrZ6UYm.

You told me to check Configuration > Diagnostics tab, how can I find this? In your documentation it should be in the web portal but I’m having a really hard time to find it.

As I’ve told you before, it is the upload step that gets stuck and it happens every now and then. But if I let it run it seems to finish in about 1-5 hours.

Do you have any clue about this?

Thanks
Andreas Nellmon

5

Hi Andreas,

The screenshot indicates that you’re getting through to the Octopus server, so it seems like we can rule out a firewall as being the cause.

If you can’t see Configuration > Diagnostics it is possible you’re running a very early 2.0 build - which version is the Octopus Server specifically?

Regards,
Nick

6

Ah - I should mention, you need to be an administrator on the Octopus instance in order to see Configuration, that might be the reason it can’t be found.

7

Hi!

Getting admin rights did it, now I can see the configuration tab. This is the only thing I’ve found from Diagnostics about the staging server:
One or more items in the current operation failed.
Fatal 13:40:07
Pipefish.Errors.PipefishCommunicationException: The message expired before it was picked up by the remote machine.
Fatal 13:40:07
Operation: Upload package to [OURSTAGINGSERVER-6876E56F] failed with error: One or more items in the current operation failed.
Fatal 13:40:07
One or more items in the current operation failed.
Fatal 13:40:07
The step failed
Fatal 13:40:07
A step failed
Fatal 13:40:07
One or more items in the current operation failed.

I also checked C:\Octopus\Logs on the staging server and this is what’s in there:

System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 94.103.199.208:10943
at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 39
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 298
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 173

2014-05-15 06:10:11.4678 ERROR Undeliverable message detected: Pipefish.Messages.Delivery.DeliveryFailureEvent from: FileSender-Q98-6Dbj88Q1@SQ-WINDOWS-G89MQNP-B38358C2 to: FileReceiver-AWA-5_wmILll@SQ-RRTVIVWS-QAS01-6876E56F

  • Body is: {“FailedMessage”:{“Id”:“3c7de4d2-c816-4daa-9561-3294919f6a0a”,“From”:{“SerializedValue”:“FileReceiver-AWA-5_wmILll@SQ-RRTVIVWS-QAS01-6876E56F”},“Headers”:{“Expires-At”:“635383410099389960”,“In-Reply-To”:“79babf0a-86c5-4396-a8dd-c106b2b6e58e”,“MessageStore-Envelope-NoFwd”:“08D13E4571A3622200000097”},“To”:{“SerializedValue”:“FileSender-Q98-6Dbj88Q1@SQ-WINDOWS-G89MQNP-B38358C2”},“MessageType”:“Pipefish.Messages.Supervision.CompletionEvent”,“Body”:{"$type":“Pipefish.Messages.Supervision.CompletionEvent, Pipefish”,“Status”:“Cancelled”,“Error”:null}},“Error”:{“Message”:“Actor does not exist”,“Detail”:null}}

2014-05-15 06:10:11.4834 ERROR Undeliverable message detected: Pipefish.Messages.Delivery.DeliveryFailureEvent from: FileSender-Q+A-6DblCOwz@SQ-WINDOWS-G89MQNP-B38358C2 to: FileReceiver-AWE-5_wmr4xd@SQ-RRTVIVWS-QAS01-6876E56F

  • Body is: {“FailedMessage”:{“Id”:“afc9ff5e-e592-4283-90db-aceefcaac811”,“From”:{“SerializedValue”:“FileReceiver-AWE-5_wmr4xd@SQ-RRTVIVWS-QAS01-6876E56F”},“Headers”:{“Expires-At”:“635383410099389960”,“In-Reply-To”:“43933e36-10af-41a6-a566-875492fd7cbd”,“MessageStore-Envelope-NoFwd”:“08D13E4571A3D78000000099”},“To”:{“SerializedValue”:“FileSender-Q+A-6DblCOwz@SQ-WINDOWS-G89MQNP-B38358C2”},“MessageType”:“Pipefish.Messages.Supervision.CompletionEvent”,“Body”:{"$type":“Pipefish.Messages.Supervision.CompletionEvent, Pipefish”,“Status”:“Cancelled”,“Error”:null}},“Error”:{“Message”:“Actor does not exist”,“Detail”:null}}

2014-05-15 06:15:24.7185 INFO Beginning transfer of C:\Octopus\Files\Octopus.Tentacle.2.4.5.46.nupkg-9979835a-e136-4714-a2e2-5ab247c7e631
2014-05-15 06:40:06.0723 ERROR Error posting to: https://build.ezyserver.se:10943/mx/v1
Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 81
— End of inner exception stack trace —
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 98
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 298
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 173

The last exception only happens every now and then, but the other errors appear frequently.

We got this from the connectivity tab of one prod machine as well (same network as the staging server):

System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Pipefish.Transport.SecureTcp.Server.SecureTcpServer.<>c__DisplayClassb.b__6(StatusCode s, ResponseHeaders h, Action1 wb) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Server\SecureTcpServer.cs:line 183 at Pipefish.Transport.SecureTcp.MessageExchange.Server.MessageExchangeModule.ExchangeMessages(IncomingRequest request, IDictionary2 parameters, OutgoingResponse response) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Server\MessageExchangeModule.cs:line 91
at Pipefish.Transport.SecureTcp.Routing.Router.<>c__DisplayClass7.b__3(IncomingRequest request, OutgoingResponse response) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Routing\Router.cs:line 71
at Pipefish.Transport.SecureTcp.Server.SecureTcpServer.ApplyProtocol(AuthorizationResult authorizationResult, EndPoint clientEndPoint, String clientThumbprint, Stream clientStream) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Server\SecureTcpServer.cs:line 188

Thanks
Andreas Nellmon

8

Thanks for the follow-up. I’m not sure what to chase here next, it still seems a lot like the network is dropping the connections. Perhaps we could catch up via Skype and put our heads together?

If that’s possible, I’m in AEST - if you’re in CEST, then your 9 AM Wednesday would work for me. I’m nblumhardt on Skype.

If you can collect any other data - e.g. if other polling Tentacles can connect to the same server, what network differences are there, and so-on - that might give us some ideas to pursue.

Regards,
Nick

9

That sounds like a good idea. I’ll add you on Skype and let’s talk Wednesday 9 AM CEST.

10

Hi Andreas - just a quick follow-up; I’ve made progress on the diagnostic tool I mentioned, there’s a little bit of work remaining on it so I’m expecting it will be ready to send in the next few days.

Regards,
Nick

11

Ok, thanks!

12

Hi,

I am also having the same issue. I have 3 polling tenticles in Test Environment. And Keep getting above error on 1 or some times on all machines. PLease guide.

13

Hi Saba,

Thanks for getting in touch. We’re working on a patch build to address what we currently believe to be the cause of this.

To determine if it is likely to apply to you, can you please let me know:

  • Is the network connection between the Octopus and Tentacle machines slow and/or unreliable?
  • Are the packages being deployed larger than a few megabytes?
  • Are you using Octopus 2.4.x on both server and Tentacles?
  • Does the deployment hang during package acquisition, or at a later stage?
  • Do you have any other network-related failures surfacing during deployment?

Many thanks for the help,
Nick