Error Upgrading from v2018.9.3 to v2018.9.6

Legacy Containers
1

Ran into the following error multiple times trying to upgrade from 2018.9.3 to 2018.9.6:

INFO Executing .NET Database Server script ‘Octopus.Core.UpgradeScripts.Script0132CorrectDanglingPackageReferences.cs’
INFO Channel ‘Tenant.TeamA’ of project ‘Email’ contains an invalid package reference. Let’s correct that…
INFO Deployment action ‘Deploy Web.Api’ references only one package. Updating to use package ‘#{ServiceName}’ () => Input string was not in a correct format.
FATAL We encountered an error during the schema upgrade. The schema upgrade was stopped and rolled back. Don’t worry, this means we haven’t made any changes to your data, and you should be able to get back up and running quickly.
Here is the summary of the problems we encountered:

Input string was not in a correct format.
System.FormatException
at System.Text.StringBuilder.AppendFormatHelper(IFormatProvider provider, String format, ParamsArray args)
at System.Text.StringBuilder.AppendFormat(String format, Object[] args)
at Octopus.Core.Initialization.DatabaseUpgraderLogAdapter.WriteInformation(String format, Object[] args)
at Octopus.Core.UpgradeScripts.Script0132CorrectDanglingPackageReferences.SanitizeDeploymentActionPackage(Func1 dbCommandFactory, JObject deploymentActionPackage, IDictionary2 deploymentActions, String humanIdentifier, IUpgradeLog log)
at CallSite.Target(Closure , CallSite , Script0132CorrectDanglingPackageReferences , Func1 , Object , IDictionary2 , String , IUpgradeLog )
at Octopus.Core.UpgradeScripts.Script0132CorrectDanglingPackageReferences.SanitizeChannelRules(Func1 dbCommandFactory, String projectId, JObject project, String projectName, IDictionary2 deploymentActions, IUpgradeLog log)
at Octopus.Core.UpgradeScripts.Script0132CorrectDanglingPackageReferences.Execute(Func1 commandFactory, IUpgradeLog log) at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.<>c__DisplayClass5_0.<ExecuteDbCommandUpgradeScript>b__0(Func1 commandFactory)
at DbUp.Engine.Transactions.SingleTrasactionStrategy.Execute(Action`1 action)
at Octopus.Core.Initialization.DatabaseUpgrader.OctopusUpgradeScriptExecutor.ExecuteDbCommandUpgradeScript(IConnectionManager connectionManager, IJournal journal, IDbCommandUpgradeScript upgradeScript)
at DbUp.Engine.UpgradeEngine.PerformUpgrade()

This put the project in question in an unusable state, where it would just show “The given key was not present in the dictionary.” when navigating to it in the Octopus UI. I was able to determine that this was caused by a corrupt channel list by using the API, so I had to delete all releases in order to delete the channels for that project one by one. Once i was left with a single channel (the default), the project would load properly in the UI.

Hoping this was just a single corrupt project, i re-tried the upgrade but ran into the same issue on another one of our projects, so it seems to be an issue with the upgrade process.

I normally upgrade once a week to the latest version and haven’t had any issues prior to the 2018.9.6 version. Seems like ‘Octopus.Core.UpgradeScripts.Script0132CorrectDanglingPackageReferences.cs’ is a good place to start…

3

Hi Jon,

We apologize you’ve hit this issue when upgrading.

To cut to the chase, we know what the problem is, and we’re aiming to ship a new version (likely 2018.9.8) which will resolve this tomorrow (Australia Eastern time). You can follow this issue.

For a little more information, we changed the way we execute our database scripts with the DbUp library. This resulted in closing a loophole around transactions, but unfortunately introduced this issue as a side-effect where all log messages during the database upgrade are assumed to be format strings. Any messages which contain Octopus variable expressions, as you can see yours does with #{ServiceName}, cause a problem similar to what you are seeing.

Again, we are terribly sorry for the inconvenience.
I’m assuming your Octopus instance is currently in a usable state, as you were able to roll-back to the previous version?

Regards,
Michael

4

Thanks for the quick reply and explanation - yes, I was able to easily roll back to the previous version based on the recommendation in the upgrade log. I’m glad you already have a fix planned, I will try the new version once it’s out in our next maintenance window. Thanks!

5

Hi Jon,

2018.9.9 was just released. The offending script has been removed in this release.
Of course please let me know if you continue to see any problems when upgrading to 2018.9.9.

Thank-you for your patience!

6

Thanks this allowed me to upgrade to the latest version!

7

I’m very glad to hear that Jon!
Thank-you for the update.