Failed to read a file in the Tentacle\Messages folder

simon_mercer · May 29, 2014, 2:23pm

I’ve had Octopus deploy working well across a number of servers, but I upgraded to the latest server and tentacles yesterday and since then i’ve been getting an error on one of the servers.

The deployment runs fine, all files are copied, all variable substitutions… for all intents and purposes, the deployment is complete.

However, the Deployment just keeps running and never terminates, either by Failing or anything else. And because the deployment doesn’t complete, TeamCity doesn’t report that the task has completed either.

The following is the message from the Log file on the server in question.
(you can see that the deployment does actually complete in the previous step)

2014-05-29 16:11:02.6336   INFO  Deployment complete.
2014-05-29 16:11:42.3746  ERROR  Failed reading D:\Octopus\Tentacle\Messages\SQ-DEVBUILD-03-VM-9AEA9B48\08D1495EF541896000001732.pf.read
System.InvalidOperationException: Cannot create and populate list type Octopus.Platform.Variables.VariableDictionary.
   at Newtonsoft.Json.Utilities.CollectionUtils.CreateList(Type listType, Boolean& isReadOnlyOrFixedSize)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndConstructorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectFromNonDefaultConstructor(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ConstructorInfo constructorInfo, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndConstructorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectFromNonDefaultConstructor(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ConstructorInfo constructorInfo, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Pipefish.Transport.Filesystem.DirectoryMessageStoreSubscription.TryReadMessageFile(String dotReadFile, Message& message) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish\Transport\Filesystem\DirectoryMessageStoreSubscription.cs:line 154

Any tips greatly appreciated

simon_mercer · May 30, 2014, 1:43pm

Additional information:

I upgraded the tentacles to the latest version the same day.

Dev / Test servers are running on the same domain as the Octopus Deploy server. These both work fine.

The Stage server and tentacle are running on a different network with the appropriate firewall rules opened.
Only inbound connections are allowed to the tentacle, no outbound.
The messages folder contains a file names .pf.read which is listed as UNREADABLE in the file details.

Paul_Stovell · June 2, 2014, 10:26am

Hi Simon,

Thanks for the email and really sorry you are having this problem. Can you try running the following command from an Administrator command prompt?

Tentacle.exe reset-activities

You can usually find Tentacle.exe at C:\Program Files\Octopus Deploy\Tentacle

After running the command you should be able to start the Tentacle service again.

Paul

simon_mercer · June 2, 2014, 1:03pm

Hi Paul,

So, I ran that command on the Tentacle of the server that is failing to complete the deploy. The tentacle service didn’t need to be started again, but I did restart it anyway.

It still causes the same problem where in the \Tentacle\Messages<server name> folder there is an unreadable file found. This is where the deployment freezes but still seems to be running.

(see attached picture of unreadable file)

The following error is present in the log file

Failed reading D:\Octopus\Tentacle\Messages\SQ-DEVBUILD-03-VM-9AEA9B48\08D14C79522CB5260000175E.pf.read
System.InvalidOperationException: Cannot create and populate list type Octopus.Platform.Variables.VariableDictionary.

Paul_Stovell · June 6, 2014, 2:23am

Hi Simon,

Thanks for the update. On the Tentacle in question, can you browse the GAC to see if a version of Newtonsoft.Json is installed that might be overriding our version?

You can use Windows Explorer and navigate to: C:\Windows\assembly

It might be that a version of JSON.NET in the GAC is being used instead of the version in Octopus, and it is causing problems deserializing our messages.

Paul

simon_mercer · June 6, 2014, 8:56am

Hey Paul,

I actually uninstalled and reinstalled the tentacle and it seemed to fix it, so unfortunately no closer to what might have caused it.

But the re-install process is easy and it didn’t need re configuring afterwards, so should have tried that first!

Thanks