Hello
We just encountered an issue today where the Octopus Tentacle service would crash - this was later found out to be due to there not being any disk space in the logs directory for it to write out its logs via the Event Viewer. It seems odd that the service would fall over when not able to write out its logs - particularly when the error is reported back as a connectivity issue. Is there scope for this to be improved?
cheers
Joseph
HI Joseph - that does indeed sound like a bug; we’d expect the service to stay up under these conditions, too.
Tentacle does, however, rely on file storage for its /Actors and /Messages folders, and for package caching. Running out of space here would be an issue. Can you please elaborate on what the drive layout is, so we can rule out any other files being the problem?
If you happened to collect any kind of log of the problem, from the Tentacle’s perspective, that would be great too.
Regards,
Nick
Hello Nicholas
Basically all logs (including the Octopus logs) are stored in a separate logs-specific partition.
Here‘s a dump of the text from the Event Viewer (three separate error events, one after another):
2014-04-16 15:04:13.9024 ERROR There is not enough space on the disk.
System.IO.IOException: There is not enough space on the disk.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
at Pipefish.Transport.Filesystem.DirectoryMessageStoreSubscription.ReadQueueFiles() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Transport\Filesystem\DirectoryMessageStoreSubscription.cs:line 112
at Pipefish.Transport.Filesystem.DirectoryMessageStoreSubscription…ctor(IMessageStoreSubscription subscription, String queueFolderPath, IStorageStreamTransform storageStreamTransform, IMessageStore messageStore) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Transport\Filesystem\DirectoryMessageStoreSubscription.cs:line 52
at Pipefish.Transport.Filesystem.DirectoryMessageStore.Subscribe(IMessageStoreSubscription subscription) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Transport\Filesystem\DirectoryMessageStore.cs:line 54
at Pipefish.Hosting.ActivitySpace.Run() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Hosting\ActivitySpace.cs:line 153
at Octopus.Tentacle.Commands.RunAgentCommand.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Tentacle\Commands\RunAgentCommand.cs:line 50
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass2.b__0() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 44
Application: Tentacle.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException
Stack:
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()
Faulting application name: Tentacle.exe, version: 2.1.3.1223, time stamp: 0x5302e1ac
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1677
Exception code: 0xe0434352
Fault offset: 0x000000000000940d
Faulting process id: 0x6d44
Faulting application start time: 0x01cf59208aceff09
Faulting application path: D:\wwwdeployment.Tentacle\Octopus.Tentacle\2.1.3.1223\Tentacle.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: c9a515eb-c513-11e3-97a8-005056956a6c
Thanks for the follow-up! These frames in the stack trace:
at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
at Pipefish.Transport.Filesystem.DirectoryMessageStoreSubscription.ReadQueueFiles() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Transport\Filesystem\DirectoryMessageStoreSubscription.cs:line 112
Indicate that the failure is due to the drive containing the Tentacle’s /Messages folder - this is usually under C:\Octopus\Tentacle or a similar location.
The service isn’t resilient to this error during startup (as is happening in this message), but would normally survive it if the condition occurred during operation.
Does this match up with your observations?
Regards,
Nick
It most certainly had issues when starting up – I observed this problem when attempting to restart the service; it would show up as having restarted but would stop after a couple of minutes (presumably after failing to write to its /Messages folder). Not sure if it was fine during operation however; the only reason I attempted to restart the service was because I had inexplicably lost connectivity to the deployment server. Potentially, it could have been in this state for awhile (unbeknownst to anyone) and the service restarted (either manually or automatically?) causing the issues which I observed.
Thanks for the extra detail. Let me know if you hit any further trouble, and I’ll definitely look into this code on our end when I’m in there next.
Regards,
Nick