Octopus Deploy 2.6.5 Dashboard/Overviews Broken - "Given key was not present" error

adam_durham · July 28, 2015, 6:55pm

Hello,
I’ve inherited the management of an Octopus Deploy 2.6.5.1010 instance; the person who originally configured it for us has since left the company.When trying to view the dashboard or the overview of any configured project, an error message appears on the screen: “The given key was not present in the dictionary.” Looking into the Diagnostics page reveals a page full of the following:
Error 2015-07-28 13:39:35
Unhandled error on request: http://sfsvcoctsrv1/api/dashboard by Adam_D : The given key was not present in the dictionary.
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at Octopus.Server.Web.Api.Components.DashboardBuilder.<>c__DisplayClass30.b__2b(Project project) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 178
at System.Linq.Enumerable.<>c__DisplayClass123.<CombineSelectors>b__11(TSource x) at System.Linq.Enumerable.WhereSelectListIterator2.MoveNext()
at System.Linq.Enumerable.d__142.MoveNext() at Octopus.Platform.Model.ReferenceCollection.ReplaceAll(IEnumerable1 newItems) in y:\work\refs\heads\master\source\Octopus.Platform\Model\ReferenceCollection.cs:line 35
at Octopus.Server.Web.Api.Components.DashboardBuilder.GetEnvironmentIds(ProjectGroup projectGroup, Dictionary2 lifecyclesById, List1 allProjects) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 175
at Octopus.Server.Web.Api.Components.DashboardBuilder.<>c__DisplayClass27.b__17(ProjectGroup p) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 149
at System.Linq.Enumerable.WhereSelectListIterator2.MoveNext() at System.Collections.Generic.List1…ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Octopus.Server.Web.Api.Components.DashboardBuilder.LoadFreshDashboard(ILog log, IDocumentStore store) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 149
at Octopus.Server.Web.Api.Components.DashboardBuilder.GetCachedDashboard() in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 84
at Octopus.Server.Web.Api.Components.DashboardBuilder.BuildDashboard(DashboardConfiguration configuration, IOctopusPrincipal principal) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Components\DashboardBuilder.cs:line 57
at Octopus.Server.Web.Api.Actions.DashboardViewAction.Execute() in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\Actions\DashboardViewAction.cs:line 28
at Octopus.Platform.Web.Api.Responder1.Respond(TDescriptor options, NancyContext context) in y:\work\refs\heads\master\source\Octopus.Platform.Web\Api\Responder.cs:line 163 at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2) at CallSite.Target(Closure , CallSite , Object , Object , NancyContext ) at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass5.<.ctor>b__2(Object o) in y:\work\refs\heads\master\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 47 at CallSite.Target(Closure , CallSite , Func2 , Object )
at Nancy.Routing.Route.<>c__DisplayClass4.b__3(Object parameters, CancellationToken context)

So far, I’ve tried restoring the oldest backup I have, repairing the RavenDB indexes, and reinstalling the Octopus Windows Service, none of which has helped. Is there anything else I can do to fix this error, short of uninstalling and reinstalling the server? I’d prefer not to take that step, because the boxes that the tentacles are installed on are not under my control, and I’d have to coordinate with the team that does control them to update their configuration with the new key for the reinstalled server.

Octopus_Errors.docx (174 KB)

adam_durham · July 28, 2015, 7:46pm

I’ve attached the System Report from the server’s diagnostics tab.

OctopusDeploy-635736901717900914.zip (304 KB)

Damian_Maclennan · July 29, 2015, 1:56am

Hi Adam,

Thanks for getting in touch, and sorry to hear you’ve inherited a mess.

It seems some of your projects and lifecycles have gotten into an inconsistent state (not 100% how or why that’s happened).

From looking at the code, we’re trying get a Lifecycle based on the lifecycle ID stored in a project. How many projects do you have ? Is it feasible to go to each of them and check the lifecycle is valid ?

Regards

Damian

adam_durham · July 29, 2015, 2:00pm

Damian,
There are 16 projects in the instance, only one of which has had any releases so far. It looks like all of them are currently using a basic “Default Lifecycle” definition, which currently has just a single “Any environment” phase. There are 25 environments, only one of which has had any deployments carried out from Octopus, all from the same project; the first two failed, while the third and fourth succeeded with warnings.

Thanks,
Adam

Damian_Maclennan · July 30, 2015, 12:31am

Hi Adam

Sorry for the delay (timezones)

The next step then is to probably query and find out where the mismatch is. To do that, you’ll need to generate an API Key, then I’ve written two powershell scripts for you (running them in ISE is easiest). The first will query all your projects and your lifecycles so you can see where you have an ID mismatch. The second one will update a given project to use the correct lifecycle. You’ll obviously need to make appropriate changes to API key, server address and for the second script, project id and lifecycle id.

gist.github.com

https://gist.github.com/DamianMac/00c4b7ccac698b200f13

Queryprojectslifecycles.ps1

$server = " http://yourserver/" #your octopus server
$apiKey = "API-YOURAPIKEY" #you'll need to generate an API

$command = $($server + "/api/projects")

$projects = Invoke-RestMethod -Method Get -Uri $command  -Header @{"X-Octopus-ApiKey"=$apiKey}
$projects.Items | Format-Table Id, Name, lifecycleId


$command = $($server + "/api/lifecycles")

This file has been truncated. show original

Update a specific projects lifecycle.ps1

$server = " http://yourserver/" #your octopus server
$apiKey = "API-YOURAPIKEY" #you'll need to generate an API


$command = $($server + "/api/projects/Projects-106") #use the project ID from the earlier query as the URL
$project = Invoke-RestMethod -Method Get -Uri $command  -Header @{"X-Octopus-ApiKey"=$apiKey}
$project.LifecycleId = "Lifecycles-120" #use the lifecycle ID from the earlier query and change this to the one you want
$body = $project | ConvertTo-Json

$new = Invoke-RestMethod -Method Put -Uri $command -Body $body -Header @{"X-Octopus-ApiKey"=$apiKey}

This file has been truncated. show original

Have a look at that and let me know how you go.

Regards

Damian

adam_durham · July 30, 2015, 2:28pm

Damian,
No need to apologize for the delay, I knew before I contacted you that there was half a planet between us.

As it turns out, it was an issue of case sensitivity - the single lifecycle in our instance had an ID of “lifecycle-ProjectGroups-1”, and all the projects referenced “Lifecycle-ProjectGroups-1”. I updated the projects to have the lowercase l in their reference (though perhaps I should have updated the lifecycle with the capital L instead?) and everything started working correctly.

Thank you!
Adam