Upgrade to latest version failed

corey · April 28, 2016, 1:58pm

We recently tried to upgrade from 3.3.4 to 3.3.10 and once installed the octopus server service fails to start up with the following error in windows event viewer:

2016-04-28 09:48:59.0497 7 FATAL Unhandled AppDomain exception occurred: Database upgrade failed: An item with the same key has already been added.
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.cs’
Upgrade failed due to an unexpected exception:
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer1 comparer) at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.GetProjectActionNameToIdMap(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 33
at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.ProvideScript(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 23 at DbUp.Engine.LazySqlScript.get_Contents() at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary2 variables)
at DbUp.Engine.UpgradeEngine.PerformUpgrade()

System.Exception: Database upgrade failed: An item with the same key has already been added.
Database upgrade logs:
Beginning database upgrade
Fetching list of already executed scripts.
Executing SQL Server script 'Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.cs’
Upgrade failed due to an unexpected exception:
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer1 comparer) at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.GetProjectActionNameToIdMap(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 33
at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.ProvideScript(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 23 at DbUp.Engine.LazySqlScript.get_Contents() at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary2 variables)
at DbUp.Engine.UpgradeEngine.PerformUpgrade()
—> System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer1 comparer) at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.GetProjectActionNameToIdMap(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 33
at Octopus.Core.UpgradeScripts.Script0038ConvertStepActionChannelRuleLink.ProvideScript(Func1 commandFactory) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\UpgradeScripts\Script0038 - Convert StepAction-ChannelRule link to use Id not Name.cs:line 23 at DbUp.Engine.LazySqlScript.get_Contents() at DbUp.Support.SqlServer.SqlScriptExecutor.Execute(SqlScript script, IDictionary2 variables)
at DbUp.Engine.UpgradeEngine.PerformUpgrade()
— End of inner exception stack trace —
at Octopus.Core.Initialization.DatabaseUpgrader.Upgrade(IRelationalStore store) in Y:\work\refs\tags\3.3.9\source\Octopus.Core\Initialization\DatabaseUpgrader.cs:line 40
at Octopus.Core.Initialization.StoreInitializer.Initialize() in Y:\work\refs\tags\3.3.9\source\Octopus.Core\Initialization\StoreInitializer.cs:line 22
at Octopus.Server.OctopusServerEngine.Start() in Y:\work\refs\tags\3.3.9\source\Octopus.Server\OctopusServerEngine.cs:line 49
at Octopus.Server.Commands.RunCommand.Start() in Y:\work\refs\tags\3.3.9\source\Octopus.Server\Commands\RunCommand.cs:line 37
at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in Y:\work\refs\tags\3.3.9\source\Octopus.Shared\Startup\AbstractCommand.cs:line 58
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0() in Y:\work\refs\tags\3.3.9\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\work\refs\tags\3.3.9\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 59
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

corey · April 28, 2016, 2:41pm

looking at the Channel table in the Octopus Deploy database I see 2 records in which the channel id number does not match the project id. I’ve attached a snippet of the database. Channels in question are Channels-270 and Channels-271. They map to Projects-271 and Projects-272. They seem to be the only ones like this.

MarkSiedle · April 29, 2016, 3:02am

Hi Corey,

Thanks for getting in touch.

Unfortunately we’re having some ongoing issues with unexpected data in one of our upgrade scripts. We have created a GitHub issue here so you can track when this will be resolved.

Since the upgrade failed, are you able install 3.3.4 so that you can be operational again until this is resolved?

Sorry for any inconvenience.

The channel and project id number differences should not be a concern, as they have independent identifiers so there will be cases where they differ. You can confirm this by giving a channel a name other than Default for a given project in the UI, then confirming it’s the correct channel/project combination via your query.

Cheers
Mark S.

corey · April 29, 2016, 11:06am

yeah I was able to go back to 3.3.4 without any issue. So I should just hang tight on this version until the issue is resolved?

MarkSiedle · May 2, 2016, 11:22pm

Hi Corey,

Yes, please continue using 3.3.4 until we can investigate this further.

It sounds like your issue may be unique however. Would it be possible for you to attach the results of this query to us please?

SELECT Id, OwnerId, IsFrozen, Version FROM [dbo].[DeploymentProcess]

We’re hoping this will shed more light on your particular issue.

Cheers
Mark

corey · May 3, 2016, 11:06am

attached is an excel spreadsheet that has the results of that query.

deploymentprocess.xlsx (45 KB)

MarkSiedle · May 4, 2016, 5:37am

Hi Corey,

Thanks for those results. It appears you have some deployment snapshots that are not frozen, which should not have happened (as all snapshots are frozen). We’ve raised a Github issue here so we’ll try and investigate what could have gone wrong.

In the meantime, to fix this for you and allow you to upgrade, you’ll need to run an update query. Before running this, please make sure you’ve done a full backup of your SQL database. We would also advise running this script when no deployments are currently running.

UPDATE [DeploymentProcess]
SET [IsFrozen]=1
WHERE ([Id] LIKE '%-snapshot-%' OR [Id] LIKE '%-s-%')

Snapshots are identified by having either “-snapshot-” or “-s-” in their Id, so all this update statement is doing is setting those records to being frozen (as they should be).

Let me know how you go.

When you do decide to upgrade, I recommend grabbing the latest 3.3.11 as this release has some other fixes to upgrades

Cheers
Mark

corey · May 4, 2016, 1:58pm

Ran that, it seemed like it was only one project in particular that this affected. We have a release this weekend so we postponed upgrading Octopus until next week. I will post back on if the upgrade is successful. Thanks.