Object reference error during deploy-release

aykeblackburn · March 10, 2021, 11:24pm

We just upgraded to 2020.6.4671 release. Currently when we try to deploy a release using octo.exe deploy-release command, we are getting the following:

Found environment: xxx (Environments-2)
Found project: xxx (Projects-382)
Finding release xxx
Updating the release variable snapshot with variables from the project
Found environment: xxx (Environments-2)
Octopus Server returned an error: Object reference not set to an instance of an object.
Error from Octopus Server (HTTP 500 InternalServerError)

I found the exception stack trace in system log. Any idea what the cause may be/places to check? The project requires at least 1 target and as far as I know we have a healthy server as target.

    System.NullReferenceException: Object reference not set to an instance of an object.
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentFactory.AllowDeploymentsToNoTargets(Release release) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Orchestration\ServerTasks\Deploy\DeploymentFactory.cs:line 557
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionFactory`3.CreateExecution(IOctopusQueryExecutor queryExecutor, TExecution execution, User user, DateTimeOffset queueTime, Nullable`1 queueTimeExpiry, IEnumerable`1 additionalVariables, IPromptedVariableValueProvider promptedVariableValueProvider, Boolean isScheduledDeployment, TCreateInfo createInfo) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ExecutionFactory.cs:line 139
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentFactory.CreateDeployment(IOctopusQueryExecutor queryExecutor, Deployment deployment, User user, DateTimeOffset queueTime, Nullable`1 queueTimeExpiry, ILifecycleProgressionEvaluator lifecycleProgressionEvaluator, ILifecycleService lifecycleService, IDeploymentSummaryService deploymentSummaryService, IEnumerable`1 additionalVariables, IPromptedVariableValueProvider promptedVariableValueProvider, Boolean isScheduledDeployment) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Orchestration\ServerTasks\Deploy\DeploymentFactory.cs:line 107
   at Octopus.Server.Web.Api.Actions.DeploymentCreateAction.CreateDeployment(DeploymentResource resource, DateTimeOffset queueTime, Boolean isScheduledDeployment, IOctopusPrincipalInternal principal, User user, IPromptedVariableValueProvider promptedVariableValueProvider, IEnumerable`1 additionalVariables) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Api\Actions\DeploymentCreateAction.cs:line 104
   at Octopus.Server.Web.Api.Actions.DeploymentCreateAction.ExecuteRegistered() in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Api\Actions\DeploymentCreateAction.cs:line 82
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(TDescriptor options, NancyContext context) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Infrastructure\Api\CustomResponder.cs:line 289
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Infrastructure\OctopusNancyModule.cs:line 81
   at Nancy.Routing.Route`1.Invoke(DynamicDictionary parameters, CancellationToken cancellationToken)
   at Nancy.Routing.DefaultRouteInvoker.Invoke(Route route, CancellationToken cancellationToken, DynamicDictionary parameters, NancyContext context)
   at Nancy.Routing.DefaultRequestDispatcher.Dispatch(NancyContext context, CancellationToken cancellationToken)
   at Nancy.NancyEngine.InvokeRequestLifeCycle(NancyContext context, CancellationToken cancellationToken, IPipelines pipelines)
   at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Owin\NancyMiddleware.cs:line 172
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Owin\NancyMiddleware.cs:line 108
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, Lazy`1 unitOfWork, ILog log, IActionContextAccessor actionContextAccessor) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 42
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, Lazy`1 unitOfWork, ILog log, IActionContextAccessor actionContextAccessor) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\b5c671f6c6f2b660\source\Octopus.Server\Web\Middleware\ErrorHandlingMiddleware.cs:line 70

Kenneth_Bates · March 11, 2021, 12:53am

Hi @aykeblackburn,

Thanks for getting in touch! I’m sorry for the issue you’re hitting deploying via octo.exe after upgrading. However I’ve been unable to reproduce this locally in 2020.6.4671. Which version of Octo.exe are you running? Latest is 7.4.3124, so perhaps it’d come down to needing to upgrade that since you recently just upgraded Octopus itself?

From there, could I ask which version you upgraded from, and the deploy-release command itself you’re running? I’ll be happy to give it a more accurate repro attempt and hopefully spot what’s going on.

Best regards,

Kenny

aykeblackburn · March 11, 2021, 2:42am

Hi Kenneth, thank you for responding. We are actually upgrading + moving to a new server, and I also installed the latest Octo.exe (7.4.3124). I am still trying to find out the version we upgraded from, but I am quite certain it is a few years old.

It is actually Bamboo which executes the octo.exe with the following command:

%bamboo_OCTO_EXE% deploy-release --project "projectname" --deployTo %bamboo_deploy_environment% --version %bamboo_deploy_version% --server %bamboo_OCTO_API% --apiKey %bamboo_NUGET_API_KEY% --progress

I can find the release created on Octopus, and when try to Deploy using Octopus UI, it also shows the same error “Object reference not set to an instance of an object.”

Also I have found that some projects that failed a few hours earlier can now be deployed successfully, but there are still other projects that have this problem even after the rebuilding them. It is most likely data related, but would be helpful if we have some guidance on where to check. Thank you.

Kenneth_Bates · March 11, 2021, 2:59am

Hi @aykeblackburn ,

Thanks for the follow up and additional information. That’s very interesting. The projects that failed a few hours ago, now deploying successfully, are both the failures and successes from the exact same release? Or was a new release created that succeeds where an older release (created before the server upgrade) was failing? Are you able to reproduce this issue with newly created releases?

Would you be willing to send us an export from one of these problematic projects? I’d like to try importing it locally and have a direct look at the data. If that’s okay with you, you can use the migrator.exe partial-export command, and send it through via email at support@octopus.com, or upload it on octopus.com after logging in under your Support tab.

Best regards,

Kenny

aykeblackburn · March 11, 2021, 4:01am

Hi Kenny,
The ones that are now successful are for new releases. I have figured out what makes them work now. Because we have several target servers that are currently down right now, I went to the Project’s Settings and updated the Transient Deployment Targets section to Skip and Continue Unavailable Deployment targets + Exclude Unhealthy Deployment Targets. It seems that after this update, if we build a new Bamboo release, then the deployment of that release will be successful.

Kenneth_Bates · March 11, 2021, 4:33am

Hi @aykeblackburn ,

Nice! Thanks for the update, and that’s good to hear. Let us know if we can try to help with anything else in the future.

Best regards,

Kenny

system · April 11, 2021, 4:33am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.