Internal NuGet server not working for pushes

Matt_Cantillon · September 27, 2016, 1:54pm

Hi,

We are having an issue with Octopus and pushing NuGet packages to the internal server. When a TeamCity build step tries to push the package it seems to time out and returns the message:

[push] The request was aborted: The request was canceled.
[push] Process exited with code 1

We have tried running the push command locally on the server via PowerShell in Admin mode and get the result in the attached screen shot. The command we used was:

C:\TeamCity\buildAgent\tools\NuGet.CommandLine.DEFAULT\tools\NuGet.exe push D:\TeamCity\buildAgent\
work\fbef427612d3cf55\Nuget\PACKAGENAME.nupkg -ApiKey APIKEY -verbosity detailed -timeout 485 -Source http://localhost:8181/nuget/packages

When TeamCity tries to push the NuGet package we are seeing the following in the Octopus server log file:

Unhandled error when processing request from client
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at Pipefish.Transport.SecureTcp.Server.SecureTcpServer.ExecuteRequest(TcpClient client) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Server\SecureTcpServer.cs:line 112

If we try to upload the same package via the “Upload Package” button in the “Packages” area of the admin website the page sits there with the spinner going and doesn’t change. In the Octopus server logs the following appears:

108 ERROR  Unhandled exception from web server: The file exists.

System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at Nancy.IO.RequestStream.CreateTemporaryFileStream()
   at Nancy.IO.RequestStream.MoveStreamContentsToFileStream()
   at Nancy.IO.RequestStream.MoveStreamOutOfMemoryIfExpectedLengthExceedSwitchLength(Int64 expectedLength)
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Octopus.Server.Web.OctopusNancyHost.ConvertRequestToNancyRequest(HttpListenerRequest request) in y:\work\refs\heads\master\source\Octopus.Server\Web\OctopusNancyHost.cs:line 291
   at Octopus.Server.Web.OctopusNancyHost.Process(HttpListenerContext ctx) in y:\work\refs\heads\master\source\Octopus.Server\Web\OctopusNancyHost.cs:line 424

The package we are pushing is 4Mb.

Our server stats are:

Windows Server 2012 Datacenter
TeamCity Enterprise 9.1.7 (build 37573)
Octopus Deploy 2.6.5.1010
TeamCity and Octopus talk to each other via http://localhost

I have tried:

Restarting Octopus Server and Tentacle
Rebooting the server
Adding a 120 days retention policy to the NuGet packages (we did have 2500+ packages now we have 1488)
Different TeamCity build configurations - They are all failing at the NuGet push stage
Checked the Firewall is allowing the required ports, and it is
Checking “C:\Octopus\OctopusServer\Repository\Packages” for the package being pushed and it’s not there. (Based on the error when uploading it via
Yesterday we were able to build and push a package and deploy a staging release to production for two different projects without issue.

Any suggestions to try or additional information requests please do let us know, we currently can not deploy any builds of our products.

Thanks,

Matt.

Vanessa_Love · September 29, 2016, 12:41am

Hi Matt,

Thanks for getting in touch! We have seen this error crop up very occasionally. I believe your tmp directory is full and needs to be cleared. Octopus stores the file in tmp while downloading it then moves it when complete.
While Octopus itself should clear up these files, other programs are not so nice about it.

This is the part of the stack trace that helps identify the problem: at System.IO.Path.InternalGetTempFileName(Boolean checkHost)

Let me know if that resolves your problem.
Vanessa

Matt_Cantillon · September 29, 2016, 4:50pm

Hi Vanessa,

Darn it, caught by that again. Why didn’t we check this before. Fixed the issue perfectly, Thank you.

We are still seeing lots of the following error messages in the logs. Seems like it might be every 30 seconds or so. Any pointers for tracking it down as the UI doesn’t have any indications that there is an issue but the logs are 4-5MB a day as a result.

Unhandled error when processing request from client
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at Pipefish.Transport.SecureTcp.Server.SecureTcpServer.ExecuteRequest(TcpClient client) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Server\SecureTcpServer.cs:line 112

Thanks for the help.

Matt.

Vanessa_Love · September 30, 2016, 7:16am

Hi Matt,

Great, glad to hear that was the issue. Generally the error you pasted refers to a Tentacle that has closed connection due to a mismatched thumbprint. Unfortunately you have pasted the lines directly under the connection check that would clarify this.
But if directly above those errors you see a Tentacle health check or a Tentacle polling you can have a look at the corresponding Tentacle logs and it should say something like Connected with X thumbprint but expected Y. We will then forcibly close the connections.
30 seconds is about the time a Polling Tentacle will check in. We suggest that you remove the machines and Tentacles or fix the thumbprints as those failed connections can degrade your performance.

I would be happy to have a look at your full logs and provide much more details or show you what to look for.

Vanessa