Octopus Server won't start

Legacy Uncategorised
1

Hi,

we’ve been trialling Octopus Deploy and have run into an issue that I can’t seem to find any mention of in the forum. Trying to open the Octopus Manager, I get and unhandled error pop up with the following error:

System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
at System.Convert.FromBase64String(String s)
at Octopus.Shared.Configuration.AbstractKeyValueStore.Get[TData](String name, TData defaultValue, Nullable`1 protectionScope) in Y:\Work\refs\tags\3.3.20\source\Octopus.Shared\Configuration\AbstractKeyValueStore.cs:line 36
at Octopus.Shared.Configuration.OctopusServerStorageConfiguration…ctor(IKeyValueStore settings) in Y:\Work\refs\tags\3.3.20\source\Octopus.Shared\Configuration\OctopusServerStorageConfiguration.cs:line 15
at Octopus.Manager.Server.OctopusConfiguration.OctopusManagerModel.Reload(ApplicationInstanceRecord applicationInstance) in Y:\Work\refs\tags\3.3.20\source\Octopus.Manager.Server\OctopusConfiguration\OctopusManagerModel.cs:line 174
at Octopus.Manager.Server.OctopusConfiguration.OctopusManagerView.LoadExistingInstanceScreen(ApplicationInstanceRecord defaultInstall) in Y:\Work\refs\tags\3.3.20\source\Octopus.Manager.Server\OctopusConfiguration\OctopusManagerView.xaml.cs:line 105
at Octopus.Manager.Server.OctopusConfiguration.OctopusManagerView.b__14_0() in Y:\Work\refs\tags\3.3.20\source\Octopus.Manager.Server\OctopusConfiguration\OctopusManagerView.xaml.cs:line 96
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(Object source, Delegate method, Object args, Int32 numArgs, Delegate catchHandler)

On restarting the service, it now fails to start, with the same errors:

2016-07-06 08:18:12.5914 7 FATAL Unhandled AppDomain exception occurred: An exception was thrown while invoking the constructor ‘Void .ctor(Octopus.Shared.Configuration.IKeyValueStore)’ on type ‘OctopusServerStorageConfiguration’. —> The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. (See inner exception for details.)
Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor ‘Void .ctor(Octopus.Shared.Configuration.IKeyValueStore)’ on type ‘OctopusServerStorageConfiguration’. —> The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. (See inner exception for details.) —> System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
at System.Convert.FromBase64String(String s)
at Octopus.Shared.Configuration.AbstractKeyValueStore.Get[TData](String name, TData defaultValue, Nullable1 protectionScope) in Y:\Work\refs\tags\3.3.19\source\Octopus.Shared\Configuration\AbstractKeyValueStore.cs:line 36 at Octopus.Shared.Configuration.OctopusServerStorageConfiguration..ctor(IKeyValueStore settings) in Y:\Work\refs\tags\3.3.19\source\Octopus.Shared\Configuration\OctopusServerStorageConfiguration.cs:line 15 at lambda_method(Closure , Object[] ) at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate() --- End of inner exception stack trace --- at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate() at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters) at Autofac.Core.Resolving.InstanceLookup.Execute() at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable1 parameters) at Octopus.Server.Web.OctopusWebBootstrapper.get_CryptographyConfiguration() in Y:\Work\refs\tags\3.3.19\source\Octopus.Server\Web\OctopusWebBootstrapper.cs:line 51 at Nancy.Bootstrapper.NancyBootstrapperBase1.GetAdditionalInstances()
at Nancy.Bootstrapper.NancyBootstrapperBase`1.Initialise()
at Octopus.Server.Web.WebServerInitializer.Start() in Y:\Work\refs\tags\3.3.19\source\Octopus.Server\Web\WebServerInitializer.cs:line 35
at Octopus.Server.OctopusServerEngine.Start() in Y:\Work\refs\tags\3.3.19\source\Octopus.Server\OctopusServerEngine.cs:line 48
at Octopus.Server.Commands.RunCommand.Start() in Y:\Work\refs\tags\3.3.19\source\Octopus.Server\Commands\RunCommand.cs:line 37
at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in Y:\Work\refs\tags\3.3.19\source\Octopus.Shared\Startup\AbstractCommand.cs:line 57
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0() in Y:\Work\refs\tags\3.3.19\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\Work\refs\tags\3.3.19\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 59
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Any idea how to resolve this other than to completely start over? With no access to the Manager console, I have no way of retrieving the master key, so presumably can’t backup (although I suspect this is some sort of data corruption, so that may not help!)

2

Hi James,

Thanks for getting in touch! Generally that error means that you have an incorrect master key and certificate. Something has been corrupted or overwritten.
You can attempt to use the tentacle.exe commands to either get out the master key or delete the instance but it might not work if you cannot load the manager.

It may come down to deleting the instance and reinstalling. You also may probably lose some of your data and getting it into the new instance will be manual unfortunately.

Sorry for the bad news!
Vanessa

3

Hi Vanessa, thanks for getting back to me. I’ve installed as a new instance and reconfigured, so that side of things is ok (thankfully, Octopus makes that a breeze!).

Do you have any idea how this could happen? My only concern is how we can avoid it in the future. Nothing was done to the server/db outside of ‘normal’ process, so I don’t know how the key/certificate could have changed/disappeared/corrupted spontaneously.

4

Hi James,

Unless someone was playing with certificates, thumbprints or maybe even security settings this shouldn’t happen. I have never heard of it being corrupted with no influence.
That being said, something such as a virus scanner or server monitoring altering the files in any way could cause a false positive as we read the file to determine the correct encryption and anything ahead of the expected data would cause that error message.

If (fingers crossed it doesn’t happen) you do have this occur again we might need to look closely at the certificate.

Did you backup the master key? :slight_smile:
Vanessa