Upgrade failure

oskaremil · October 23, 2014, 10:26am

Somehow, when I upgraded from 2.5.9 to 2.5.10 I managed to to a clean install with Octopus data folder at a new location.

I couldn’t figure out how to switch to my current data folder and I was i a hurry ,- I uninstalled, renamed the data folder and installed again with the data folder pointing to where it should be.

Now, when I try tor restore a backup from my previous Octopus Server instance it fails.

The script:

"C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" service --instance "OctopusServer" --stop
"C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" restore --instance "OctopusServer" --file "D:\_Octopus\Backups\20141023-000332.octobak" --master-key "[HIDDEN]"
"C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" service --instance "OctopusServer" --start

The result:

Service stopped
Restoring Octopus Deploy Server from backup
Purging directory D:\Octopus\RavenDB
Purging directory D:\Octopus\OctopusServer\Actors
Purging directory D:\Octopus\OctopusServer\Messages
Purging directory D:\Octopus\OctopusServer\ActivityLogs
You can browse the RavenDB server at: http://localhost:10931/
-------------------------------------------------------------------------------
Error: One or more errors occurred.
-------------------------------------------------------------------------------
Full error details are available in the log files.
See: http://g.octopushq.com/LogFiles
Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: "C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" restore --instance "OctopusServer" --file "D:\_Octopus\Backups\20141023-000332.octobak" --master-key "[HIDDEN]"

Vanessa_Love · October 23, 2014, 10:37am

Hi,

Thanks for getting in touch! Could you grab any logs in d:\Octopus\Logs if there are any?
The error says one or more errors occurred, so we will need to pin point it.

Thanks!
Vanessa

oskaremil · October 23, 2014, 11:01am

Logs attached.

Regards,
Oskar

Mvh,
Oskar Emil Skeide

RavenDB.txt (12 KB)

OctopusServer.txt (51 KB)

Vanessa_Love · October 23, 2014, 12:20pm

Hi Oskar,

Thanks for the logs. It looks like the config file in the backup is corrupted.
So to clarify do you have all your old data and files in a folder somewhere?
And what error are you seeing when you try to load the Octopus Manager?

Vanessa

oskaremil · October 23, 2014, 1:01pm

No error when I start Octopus Manager, as it is a clean install.

The error I described in the first post comes when I do "Restore a backup"
and then select the latest backup from that other folder.

Mvh,
Oskar Emil Skeide

Vanessa_Love · October 23, 2014, 1:09pm

Hi Oskar,

What I am hoping will work, is pointing the new install at your existing data directory.
Please follow these instructions and let me know how it goes.

In PowerShell run the following line by line on the Octopus Server. You will have to change the $newhome to where your existing Octopus data lives.

$newHome = "C:\YourNewHomeDir"
$octopus = "C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe"
$newConfig = $newHome + "\OctopusServer\OctopusServer.config"
& "$octopus" service --stop
 
&"$octopus" delete-instance --instance=OctopusServer
&"$octopus" create-instance --instance=OctopusServer --config=$newConfig
 
& "$octopus" configure --home="$newHome"
& "$octopus" service --start

If it doesn’t work we are going to need to see your corrupted backup to see if we can get the config file out of it.
Let me know how that goes.
Vanessa

oskaremil · October 24, 2014, 6:49am

The powershell executed without errors.

When starting the service and navigating to the website this appears in the logs

2014-10-24 08:40:29.4227   INFO  Waiting for service to start. Current status: StartPending
2014-10-24 08:40:30.5817   INFO  Waiting for service to start. Current status: Running
2014-10-24 08:40:31.5867   INFO  Service started
2014-10-24 08:40:33.1687   INFO  Browse your Octopus server at: http://localhost:80/
2014-10-24 08:40:55.8355  ERROR  Unhandled exception from web server: An exception was thrown while executing a resolve operation. See the InnerException for details. ---> Encrypted stream is not correctly salted with the document key. (See inner exception for details.)
Autofac.Core.DependencyResolutionException: An exception was thrown while executing a resolve operation. See the InnerException for details. ---> Encrypted stream is not correctly salted with the document key. (See inner exception for details.) ---> System.IO.IOException: Encrypted stream is not correctly salted with the document key. ---> System.IO.InvalidDataException: The encrypted stream's salt was different than the expected salt.
   at Raven.Bundles.Encryption.CodecSaltExtensions.ReadSalt(Stream stream, String key) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Codec.cs:line 196
   --- End of inner exception stack trace ---
   at Raven.Bundles.Encryption.CodecSaltExtensions.ReadSalt(Stream stream, String key) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Codec.cs:line 203
   at Raven.Bundles.Encryption.Codec.Decode(String key, Stream dataStream) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Codec.cs:line 97
   at System.Linq.Enumerable.Aggregate[TSource,TAccumulate](IEnumerable`1 source, TAccumulate seed, Func`3 func)
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.ReadDocumentData(String key, Etag existingEtag, RavenJObject metadata) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 100
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.<DocumentByKey>b__1(JsonDocumentMetadata metadata, Func`4 createDocument) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 39
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKeyInternal[T](String key, TransactionInformation transactionInformation, Func`3 createResult) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 70
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKey(String key, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 36
   at Raven.Database.DocumentDatabase.<>c__DisplayClass36.<Get>b__34(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 714
   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 666
   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 617
   at Raven.Database.DocumentDatabase.Get(String key, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 716
   at Raven.Bundles.Encryption.Settings.EncryptionSettingsManager.VerifyEncryptionKey(DocumentDatabase database, EncryptionSettings settings) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Encryption\Settings\EncryptionSettingsManager.cs:line 93
   at Raven.Database.Extensions.EnumerableExtensions.Apply[T](IEnumerable`1 self, Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Extensions\EnumerableExtensions.cs:line 18
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 252
   at Raven.Client.Embedded.EmbeddableDocumentStore.InitializeInternal() in c:\Builds\RavenDB-Stable\Raven.Client.Embedded\EmbeddableDocumentStore.cs:line 218
   at Raven.Client.Document.DocumentStore.Initialize() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentStore.cs:line 459
   at Octopus.Server.Storage.StorageEngine.InitializeStore() in y:\work\refs\heads\master\source\Octopus.Server\Storage\StorageEngine.cs:line 148
   at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
   at Octopus.Server.Storage.StorageEngine.get_Store() in y:\work\refs\heads\master\source\Octopus.Server\Storage\StorageEngine.cs:line 39
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass1`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)
   at Octopus.Server.Storage.StorageModule.<Load>b__3(IComponentContext c) in y:\work\refs\heads\master\source\Octopus.Server\Storage\StorageModule.cs:line 66
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass1`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)
   at Octopus.Server.Web.Infrastructure.Authentication.AuthenticationModule.<Load>b__0(IComponentContext c) in y:\work\refs\heads\master\source\Octopus.Server\Web\Infrastructure\Authentication\AuthenticationModule.cs:line 31
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass1`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)
   at Octopus.Server.Web.OctopusWebBootstrapper.RequestStartup(ILifetimeScope container2, IPipelines pipelines, NancyContext context) in y:\work\refs\heads\master\source\Octopus.Server\Web\OctopusWebBootstrapper.cs:line 0
   at Nancy.Bootstrapper.NancyBootstrapperWithRequestContainerBase`1.InitializeRequestPipelines(NancyContext context)
   at Nancy.NancyEngine.HandleRequest(Request request, Func`2 preRequest, CancellationToken cancellationToken)
   at Nancy.NancyEngineExtensions.HandleRequest(INancyEngine nancyEngine, Request request, Func`2 preRequest, Action`1 onComplete, Action`1 onError, CancellationToken cancellationToken)
   at Octopus.Server.Web.OctopusNancyHost.Process(HttpListenerContext ctx) in y:\work\refs\heads\master\source\Octopus.Server\Web\OctopusNancyHost.cs:line 426

I tried to repair the database but got this error

Service stopped
Repairing the Octopus Server storage at D:\Octopus\RavenDB.
Recovering the ESENT database...

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 6.3
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: RVN
            Log files: D:\Octopus\RavenDB\logs
         System files: D:\Octopus\RavenDB\system
   Database Directory: D:\Octopus\RavenDB

Performing soft recovery...

Operation completed successfully in 0.468 seconds.

Defragmenting the ESENT database...

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 6.3
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating DEFRAGMENTATION mode...
            Database: D:\Octopus\RavenDB\Data

                  Defragmentation Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................


Moving '.\TEMPDFRG3136.EDB' to 'D:\Octopus\RavenDB\Data'...
                     File Copy Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................


Note:
  It is recommended that you immediately perform a full backup
  of this database. If you restore a backup made before the
  defragmentation, the database will be rolled back to the state
  it was in at the time of that backup.

Operation completed successfully in 0.907 seconds.

Deleting RavenDB indexes...
-------------------------------------------------------------------------------
Error: Encrypted stream is not correctly salted with the document key.
-------------------------------------------------------------------------------
Full error details are available in the log files.
See: http://g.octopushq.com/LogFiles
Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: "C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" repair-storage --instance "OctopusServer"

For now, I’m just going to recreate my instance manually since there was only a minor set of testing projects.

If you still want to pursue this issue I can keep a copy of the Octopus data folder for a while i case you need something from it

Vanessa_Love · October 24, 2014, 8:32am

Hi Oskar,

If you are happy to start with a new install we will not bother pursuing as a few steps here could have caused these issues.
Please make sure to grab the new master key and put it somewhere safe, and if you have any issues that are even remotely similar with the new instance we will grab a backup and see if we can figure out what is going on. Sorry for the troubles.

Vanessa

oskaremil · October 28, 2014, 1:42pm

Got the new installation up and running. You may close this issue.