Errors after registering new Tentacles

Legacy Uncategorised
1

Hi,

I’m trying to register new tentacles and getting the following error when I try to run any project against the new machines:

Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Octopus.Server.Orchestration.Deploy.DeploymentOrchestrator.b__12(Machine m) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Deploy\DeploymentOrchestrator.cs:line 156
at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
at Octopus.Server.Orchestration.Deploy.DeploymentOrchestrator.WarnIfMachinesWithMissingSquidsArePresent(IDocumentSession session, String environmentId) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Deploy\DeploymentOrchestrator.cs:line 158
at Octopus.Server.Orchestration.Deploy.DeploymentOrchestrator.d__e.MoveNext() in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Deploy\DeploymentOrchestrator.cs:line 119
Octopus.Server version 2.5.10.567

In the server logs, I can see the following:

2014-12-03 15:30:52.0941 WARN Failed to index documents for index: MachinesByEnvironment
System.IO.InvalidDataException: File is corrupted: the length written to the file doesn’t match the number of blocks in it.
at Raven.Bundles.Encryption.Streams.BlockReaderWriter.ReadOrWriteFooter() in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Streams\BlockReaderWriter.cs:line 119
at Raven.Bundles.Encryption.Streams.BlockReaderWriter…ctor(EncryptionSettings encryptionSettings, String key, Stream stream, Int32 defaultBlockSize) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Streams\BlockReaderWriter.cs:line 48
at Raven.Bundles.Encryption.Streams.SeekableCryptoStream…ctor(EncryptionSettings encryptionSettings, String key, Stream stream) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Streams\SeekableCryptoStream.cs:line 47
at Raven.Bundles.Encryption.Plugin.IndexEncryption.Decode(String key, Stream dataStream) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Plugin\IndexEncryption.cs:line 47
at Raven.Database.Indexing.LuceneCodecDirectory.ApplyReadCodecs(String key, Stream stream) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\LuceneCodecDirectory.cs:line 62
at Raven.Database.Indexing.LuceneCodecDirectory.CodecIndexInput…ctor(FileInfo file, Func2 applyCodecs, Int32 bufferSize) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\LuceneCodecDirectory.cs:line 119 at Raven.Database.Indexing.LuceneCodecDirectory.OpenInput(String name, Int32 bufferSize) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\LuceneCodecDirectory.cs:line 24 at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentInfos.cs:line 272 at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, StreamWriter infoStream, DocumentsWriter docWriter, HashSet1 synced) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexFileDeleter.cs:line 174
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 1305
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 1203
at Lucene.Net.Index.IndexWriter…ctor(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, MaxFieldLength mfl) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 5441
at Raven.Database.Indexing.RavenIndexWriter.CreateIndexWriter() in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\RavenIndexWriter.cs:line 122
at Raven.Database.Indexing.Index.CreateIndexWriter() in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\Index.cs:line 478
at Raven.Database.Indexing.Index.Write(Func`4 action) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\Index.cs:line 388
at Raven.Database.Indexing.SimpleIndex.IndexDocuments(AbstractViewGenerator viewGenerator, IndexingBatch batch, IStorageActionsAccessor actions, DateTime minimumTimestamp) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\SimpleIndex.cs:line 177
at Raven.Database.Indexing.IndexStorage.Index(String index, AbstractViewGenerator viewGenerator, IndexingBatch batch, WorkContext context, IStorageActionsAccessor actions, DateTime minimumTimestamp) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\IndexStorage.cs:line 843
at Raven.Database.Indexing.IndexingExecuter.IndexDocuments(IStorageActionsAccessor actions, String index, IndexingBatch batch) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\IndexingExecuter.cs:line 490

This seems pretty bad, I can’t seem to fix it by restarting services (server or tentacle) or removing and re-registering tentacles. Any suggestions how to diagnose what went wrong and then fix it?

Thanks

Chris

2

Hi Chris,

Thanks for contacting us. Con you please try to repair the DB as shown on this post?

Thanks!

Dalmiro.

3

Yep, that fixed it. Should that be our default reaction to any issues with the database? Or was that a specific fix because the error mentioned the indexes explicitly?

Thanks for the quick response

Chris

4

Hi Chris,

Glad you could sort it out! Yes the fix is due to the error mentioning the indexes. We’ve seen this RavenDB issue before and it doesn’t happen too often. It is though one of the reasons why in Octopus 3.0 we are moving from RavenDB to SQL. Check Paul’s post about our future movement to SQL

Thanks!

Dalmiro