Experiencing .NET Runtime and Application Error about once a week

JC · February 22, 2017, 12:33am

We are using Octopus Deploy 2.6.2.845.

From time to time, maybe once a week, we experience exceptions.

.NET Runtime

Log Name: Application
Source: .NET Runtime
Date: 22/02/2017 10:52:01 AM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: *****.com.au
Description:
Application: Octopus.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException
Stack:
at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
at System.Delegate.DynamicInvokeImpl(System.Object[])
at Raven.Client.Linq.LinqPathProvider.GetMemberValue(System.Linq.Expressions.MemberExpression)
at Raven.Client.Linq.LinqPathProvider.GetValueFromExpressionWithoutConversion(System.Linq.Expressions.Expression, System.Object ByRef)
at Raven.Client.Linq.LinqPathProvider.GetValueFromExpression(System.Linq.Expressions.Expression, System.Type)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitGreaterThan(System.Linq.Expressions.BinaryExpression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitExpression(System.Linq.Expressions.Expression)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitQueryableMethodCall(System.Linq.Expressions.MethodCallExpression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitMethodCall(System.Linq.Expressions.MethodCallExpression, Boolean)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].GetLuceneQueryFor(System.Linq.Expressions.Expression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Execute(System.Linq.Expressions.Expression)
at Raven.Client.Linq.RavenQueryInspector1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].GetEnumerator() at System.Collections.Generic.List1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]…ctor(System.Collections.Generic.IEnumerable1<System.__Canon>) at System.Linq.Enumerable.ToList[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.IEnumerable1<System.__Canon>)
at Octopus.Server.Web.Api.Actions.DeploymentsByProjectReportResponder.Execute()
at Octopus.Platform.Web.Api.Responder1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Respond(System.__Canon, Nancy.NancyContext) at System.Dynamic.UpdateDelegates.UpdateAndExecute3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon, System.__Canon, System.__Canon) at Octopus.Server.Web.Api.OctopusRestApiModule+<>c__DisplayClass5.<.ctor>b__2(System.Object) at Nancy.Routing.Route+<>c__DisplayClass4.<Wrap>b__3(System.Object, System.Threading.CancellationToken) at Nancy.Routing.Route.Invoke(Nancy.DynamicDictionary, System.Threading.CancellationToken) at Nancy.Routing.DefaultRouteInvoker.Invoke(Nancy.Routing.Route, System.Threading.CancellationToken, Nancy.DynamicDictionary, Nancy.NancyContext) at Nancy.Routing.DefaultRequestDispatcher+<>c__DisplayClass2.<Dispatch>b__0(System.Threading.Tasks.Task1<Nancy.Response>)
at Nancy.Routing.DefaultRequestDispatcher.Dispatch(Nancy.NancyContext, System.Threading.CancellationToken)
at Nancy.NancyEngine+<>c__DisplayClass8.b__5(System.Threading.Tasks.Task1<Nancy.Response>) at Nancy.NancyEngine.InvokeRequestLifeCycle(Nancy.NancyContext, System.Threading.CancellationToken, Nancy.Bootstrapper.IPipelines) at Nancy.NancyEngine.HandleRequest(Nancy.Request, System.Func2<Nancy.NancyContext,Nancy.NancyContext>, System.Threading.CancellationToken)
at Nancy.NancyEngineExtensions.HandleRequest(Nancy.INancyEngine, Nancy.Request, System.Func2<Nancy.NancyContext,Nancy.NancyContext>, System.Action1<Nancy.NancyContext>, System.Action`1<System.Exception>, System.Threading.CancellationToken)
at Octopus.Server.Web.OctopusNancyHost.Process(System.Net.HttpListenerContext)
at Octopus.Server.Web.OctopusNancyHost.GotCallback(System.IAsyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)
at System.Net.ListenerAsyncResult.IOCompleted(System.Net.ListenerAsyncResult, UInt32, UInt32)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped)

Event Xml:

1026
2
0
0x80000000000000

201418
Application
*****.com.au

Application: Octopus.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException
Stack:
at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
at System.Delegate.DynamicInvokeImpl(System.Object[])
at Raven.Client.Linq.LinqPathProvider.GetMemberValue(System.Linq.Expressions.MemberExpression)
at Raven.Client.Linq.LinqPathProvider.GetValueFromExpressionWithoutConversion(System.Linq.Expressions.Expression, System.Object ByRef)
at Raven.Client.Linq.LinqPathProvider.GetValueFromExpression(System.Linq.Expressions.Expression, System.Type)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitGreaterThan(System.Linq.Expressions.BinaryExpression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitExpression(System.Linq.Expressions.Expression)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitQueryableMethodCall(System.Linq.Expressions.MethodCallExpression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].VisitMethodCall(System.Linq.Expressions.MethodCallExpression, Boolean)
at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].GetLuceneQueryFor(System.Linq.Expressions.Expression) at Raven.Client.Linq.RavenQueryProviderProcessor1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Execute(System.Linq.Expressions.Expression)
at Raven.Client.Linq.RavenQueryInspector1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].GetEnumerator() at System.Collections.Generic.List1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]…ctor(System.Collections.Generic.IEnumerable1<System.__Canon>) at System.Linq.Enumerable.ToList[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.IEnumerable1<System.__Canon>)
at Octopus.Server.Web.Api.Actions.DeploymentsByProjectReportResponder.Execute()
at Octopus.Platform.Web.Api.Responder1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Respond(System.__Canon, Nancy.NancyContext) at System.Dynamic.UpdateDelegates.UpdateAndExecute3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon, System.__Canon, System.__Canon) at Octopus.Server.Web.Api.OctopusRestApiModule+<>c__DisplayClass5.<.ctor>b__2(System.Object) at Nancy.Routing.Route+<>c__DisplayClass4.<Wrap>b__3(System.Object, System.Threading.CancellationToken) at Nancy.Routing.Route.Invoke(Nancy.DynamicDictionary, System.Threading.CancellationToken) at Nancy.Routing.DefaultRouteInvoker.Invoke(Nancy.Routing.Route, System.Threading.CancellationToken, Nancy.DynamicDictionary, Nancy.NancyContext) at Nancy.Routing.DefaultRequestDispatcher+<>c__DisplayClass2.<Dispatch>b__0(System.Threading.Tasks.Task1<Nancy.Response>)
at Nancy.Routing.DefaultRequestDispatcher.Dispatch(Nancy.NancyContext, System.Threading.CancellationToken)
at Nancy.NancyEngine+<>c__DisplayClass8.<InvokeRequestLifeCycle>b__5(System.Threading.Tasks.Task1<Nancy.Response>) at Nancy.NancyEngine.InvokeRequestLifeCycle(Nancy.NancyContext, System.Threading.CancellationToken, Nancy.Bootstrapper.IPipelines) at Nancy.NancyEngine.HandleRequest(Nancy.Request, System.Func2<Nancy.NancyContext,Nancy.NancyContext>, System.Threading.CancellationToken)
at Nancy.NancyEngineExtensions.HandleRequest(Nancy.INancyEngine, Nancy.Request, System.Func2<Nancy.NancyContext,Nancy.NancyContext>, System.Action1<Nancy.NancyContext>, System.Action`1<System.Exception>, System.Threading.CancellationToken)
at Octopus.Server.Web.OctopusNancyHost.Process(System.Net.HttpListenerContext)
at Octopus.Server.Web.OctopusNancyHost.GotCallback(System.IAsyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)
at System.Net.ListenerAsyncResult.IOCompleted(System.Net.ListenerAsyncResult, UInt32, UInt32)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped)

This is Application Error

Log Name: Application
Source: Application Error
Date: 22/02/2017 10:52:06 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: ******.com.au
Description:
Faulting application name: Octopus.Server.exe, version: 2.6.2.845, time stamp: 0x54dc19e3
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18939, time stamp: 0x55b02e7b
Exception code: 0xe0434352
Fault offset: 0x000000000000b16d
Faulting process id: 0x29b0
Faulting application start time: 0x01d28292b6d7b37e
Faulting application path: C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: bfadf6c3-f890-11e6-b5d0-005056b6605d
Event Xml:

1000
2
100
0x80000000000000

201419
Application
******.com.au

Octopus.Server.exe
2.6.2.845
54dc19e3
KERNELBASE.dll
6.1.7601.18939
55b02e7b
e0434352
000000000000b16d
29b0
01d28292b6d7b37e
C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe
C:\Windows\system32\KERNELBASE.dll
bfadf6c3-f890-11e6-b5d0-005056b6605d

JC · February 22, 2017, 12:48am

Seems that this only happens under high load.

2017-02-22 10:52AM
2017-02-01 11:51AM
2017-01-31 11:03AM
2017-01-13 11:01AM
so on…

Sorry it’s not happening on a weekly basis, but frequent enough to cause some headaches for us.

JC · February 22, 2017, 2:12am

I also believe that this happens while there are clean up tasks going on, which happens every 4 hours. Is it possible for us to adjust the duration of clean up tasks or is it possible to set a fixed time of day for clean up to kick off?

Shane_Gill · February 23, 2017, 2:46am

Hi,

Thanks for getting in touch and I’m sorry Octopus is crashing regularly on you.

The driving force behind our move to SQL Server in version 3 was the support cost of Raven. If I could entice you to upgrade I think you will find that Octopus is much more reliable (and performant and full of features that make your life easier).

The clean up task in 2.6 is controlled by an internal scheduler that is hard coded to run every 4 hours. There is no way to control the frequency or set a fixed time to run it (you can in version 3…).

One thing that does come to mind: we recently developed a Windows scheduled task that will check the status of the Octopus Deploy server and start it if it has shut down. You will still have the exceptions but hopefully they won’t be so impactful. Would you be interested in something like that?

Cheers,
Shane

JC · February 23, 2017, 10:51pm

Hi Shane, thanks for your reply.

At the moment it’s too much effort for us to upgrade all our projects as we have quite a lot of projects / environments so it’s not an easy job.

The scheduled task might help us in the meantime.

Shane_Gill · February 24, 2017, 12:35am

Hi,

The service monitor is a part of Octopus 3.7.17+ but I just tested it with 2.6.2 and it seems to work fine. You’ll need to download a v3 version of Octopus but make sure you do not run the installer as it will replace your v2.6 instance.

To get it running:

download the latest Octopus Server form https://octopus.com/downloads
extract the MSI file using msiexec /a C:\Octopus.3.11.0-x64.msi /qb TARGETDIR=C:\OctopusWatchdog
open a console in the extracted Octopus folder (C:\OctopusWatchdog\PFiles\Octopus Deploy\Octopus using the example above)
run Octopus.Server.exe watchdog --create

You can optionally provide the --interval parameter which will set how many minutes between checks. The default is 5 minutes.

A scheduled task will be created that will turn the service back on if it stops.

Cheers,
Shane