Error upgrading to 3.3.11 (Service won't start)

dotnetdev · May 7, 2016, 4:07pm

Hi Octopus team,

On trying to upgrade to 3.3.11, the Octopus service won’t start and the following exception is logged in the Windows event log. Before the service stops, I see the “running migrations” progress bar when accessing the website (the service stops seconds later).

I believe I was on 3.3.10 previously (I usually update OD within 1-2 weeks of new versions being released). The last DB upgrade script applied according to SchemaVersions is this (following the attempted 3.3.11 update):

Octopus.Core.UpgradeScripts.Script0040RemoveTemplateScriptStepPropertiesThatDoNotApplyToTheScriptSource.cs 2016-05-01 20:27:54.243

Using a backup of my database and the 3.3.10 installer, I was able to downgrade to a seemingly functional version again. On a retry of the upgrade to 3.3.11 I got the same issue (now staying at 3.3.10 for the moment).

Application: Octopus.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.CSharp.RuntimeBinder.RuntimeBinderException
   at DynamicClass.CallSite.Target(System.Runtime.CompilerServices.Closure, System.Runtime.CompilerServices.CallSite, System.Object)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(System.String, System.Func`1<System.Data.IDbCommand>)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(System.Func`1<System.Data.IDbCommand>)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(System.Func`1<System.Data.IDbCommand>)
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(DbUp.Engine.SqlScript, System.Collections.Generic.IDictionary`2<System.String,System.String>)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()

Exception Info: System.Exception
   at Octopus.Core.Initialization.DatabaseUpgrader.Upgrade(Octopus.Core.RelationalStorage.IRelationalStore)
   at Octopus.Core.Initialization.StoreInitializer.Initialize()
   at Octopus.Server.OctopusServerEngine.Start()
   at Octopus.Server.Commands.RunCommand.Start()
   at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(System.String[], Octopus.Shared.Startup.ICommandRuntime, Octopus.Shared.Internals.Options.OptionSet)
   at Octopus.Shared.Startup.WindowsServiceHost+<>c__DisplayClass1_0.<Run>b__0()
   at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Let me know if there are any additional details I can provide or scripts you want me to run on my DB to troubleshoot?

Many thanks
Chris

dotnetdev · May 7, 2016, 4:14pm

Also here are the relevant entries from the log file:

2016-05-07 16:29:57.3118    165  INFO  Stopping the Windows Service
2016-05-07 16:29:57.3118    165  INFO  The Octopus server is shutting down...
2016-05-07 16:29:57.3118    165  INFO  Shutting down the embedded web server
2016-05-07 16:29:57.3274    165  INFO  listen://[::]:10943/            165  Listener stopped
2016-05-07 16:29:57.3430    165  INFO  The Windows Service has stopped
2016-05-07 16:30:12.5245      7  INFO  Browse your Octopus server at: http://localhost:80/
2016-05-07 16:30:12.5881      7  INFO  Browse your Octopus server at: https://localhost:443/
2016-05-07 16:30:12.5901      7  INFO  The Octopus server is starting: Initializing database and performing migrations...
2016-05-07 16:30:12.8043      7  INFO  Beginning database upgrade
2016-05-07 16:30:12.8083      7  INFO  Fetching list of already executed scripts.
2016-05-07 16:30:12.8283      7  INFO  Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs'
2016-05-07 16:30:13.6447      7 ERROR  Upgrade failed due to an unexpected exception:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
2016-05-07 16:30:13.6547      7 FATAL  Database upgrade failed: Cannot perform runtime binding on a null reference
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs'
Upgrade failed due to an unexpected exception:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()

System.Exception: Database upgrade failed: Cannot perform runtime binding on a null reference
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs'
Upgrade failed due to an unexpected exception:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
 ---> Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
   --- End of inner exception stack trace ---
   at Octopus.Core.Initialization.DatabaseUpgrader.Upgrade(IRelationalStore store) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 40
   at Octopus.Core.Initialization.StoreInitializer.Initialize() in Y:\work\refs\tags\3.3.11\source\Octopus.Core\Initialization\StoreInitializer.cs:line 22
   at Octopus.Server.OctopusServerEngine.Start() in Y:\work\refs\tags\3.3.11\source\Octopus.Server\OctopusServerEngine.cs:line 49
   at Octopus.Server.Commands.RunCommand.Start() in Y:\work\refs\tags\3.3.11\source\Octopus.Server\Commands\RunCommand.cs:line 37
   at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\AbstractCommand.cs:line 58
   at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.<Run>b__0() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
   at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 44
2016-05-07 16:30:13.6757      7 FATAL  Unhandled AppDomain exception occurred: Database upgrade failed: Cannot perform runtime binding on a null reference
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs'
Upgrade failed due to an unexpected exception:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()

System.Exception: Database upgrade failed: Cannot perform runtime binding on a null reference
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs'
Upgrade failed due to an unexpected exception:
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
 ---> Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot perform runtime binding on a null reference
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.FixVariableSetChannelScope(String projectId, Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 117
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.LoadAndRepairProjects(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 69
   at Octopus.Core.UpgradeScripts.Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.ProvideScript(Func`1 commandFactory) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\UpgradeScripts\Script0041FixOrDeleteChannelReferencesForVariablesAndReleaseCreationOptions.cs:line 29
   at DbUp.Engine.LazySqlScript.get_Contents()
   at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary`2 variables)
   at DbUp.Engine.UpgradeEngine.PerformUpgrade()
   --- End of inner exception stack trace ---
   at Octopus.Core.Initialization.DatabaseUpgrader.Upgrade(IRelationalStore store) in Y:\work\refs\tags\3.3.11\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 40
   at Octopus.Core.Initialization.StoreInitializer.Initialize() in Y:\work\refs\tags\3.3.11\source\Octopus.Core\Initialization\StoreInitializer.cs:line 22
   at Octopus.Server.OctopusServerEngine.Start() in Y:\work\refs\tags\3.3.11\source\Octopus.Server\OctopusServerEngine.cs:line 49
   at Octopus.Server.Commands.RunCommand.Start() in Y:\work\refs\tags\3.3.11\source\Octopus.Server\Commands\RunCommand.cs:line 37
   at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\AbstractCommand.cs:line 58
   at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.<Run>b__0() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
   at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 59
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

robpearson · May 9, 2016, 11:22am

Hi Chris,

Thanks for getting in touch and providing the detailed error report. It really helps! This is a known issue and we should have a fix out in Octopus 3.3.12 in the very near future.

Thanks again and let me know if you have any further issues.

Rob

dotnetdev · May 9, 2016, 1:39pm

Hi Rob,

Thanks for the quick response. I’ll hold out for 3.3.12 then!

Many thanks

Chris

dotnetdev · May 15, 2016, 9:47pm

Hi Rob,

I wanted to check if this issue is definitely now fixed in 3.3.12? I couldn’t see a reference to it in the release notes, so I just wanted to check before I try an upgrade.

Thanks
Chris

Michael_Noonan · May 15, 2016, 10:07pm

Hi Chris,

Rob will be away for a few days, so I’ll answer on his behalf. Yes that was fixed in 3.3.12 as a tweak to this upgrade script.

Please don’t hesitate to get back in touch if you have any problems.

Hope that helps!
Mike