Tentacle health failure on our octopus server (enterprise license)

arvind_raghunathan · December 19, 2013, 4:37pm

Hi Paul – we have about 1000 tentacles we manage in our environment & when we started adding more i.e. about 500 new yesterday we noticed the health check failed with the following request too long.

We are on the following versions : Octopus Deploy 1.6.1.1718 for both server & tentacle.

How can we resolve this? We need this resolved ASAP as our production environments monitoring & change process approvals rely on successfully passing health checks.

ERROR One or more errors occurred. System.InvalidOperationException:

Bad Request

Bad Request - Request Too Long

HTTP Error 400. The size of the request headers is too long.

---> System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.GetResponse() at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 333 --- End of inner exception stack trace ---

Server stack trace:
at Raven.Client.Connection.HttpJsonRequest.HandleErrors(WebException e) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 480
at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 339 at Raven.Client.Connection.HttpJsonRequest.ReadResponseJson() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 226 at Raven.Client.Connection.ServerClient.DirectQuery(String index, IndexQuery query, String operationUrl, String[] includes, Boolean metadataOnly, Boolean includeEntries) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 941 at Raven.Client.Connection.ServerClient.<>c__DisplayClass53.<Query>b__52(String u) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 917 at Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func2 operation, String operationUrl, Boolean avoidThrowing, T& result) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ReplicationInformer.cs:line 564
at Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func2 operation) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ReplicationInformer.cs:line 523 at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func2 operation) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 174
at Raven.Client.Document.AbstractDocumentQuery2.ExecuteActualQuery() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 535 at Raven.Client.Document.AbstractDocumentQuery2.get_QueryResult() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 504
at Raven.Client.Linq.RavenQueryProviderProcessor1.ExecuteQuery[TProjection]() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1381 at Raven.Client.Linq.RavenQueryInspector1.GetEnumerator() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryInspector.cs:line 99
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Octopus.Server.Tasks.Health.HealthControllerActivity.<>c__DisplayClass17.<StoreResults>b__16(IDocumentSession session) in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Health\HealthControllerActivity.cs:line 84 at Octopus.Core.Persistance.UnitOfWorkFactory.InSession(Action1 callback) in c:\w\e6923628be6eaf72\source\Octopus.Core\Persistance\UnitOfWorkFactory.cs:line 33
at Octopus.Server.Tasks.Health.HealthControllerActivity.d__6.MoveNext() in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Health\HealthControllerActivity.cs:line 55

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Octopus.Shared.Activities.ActivityRuntime.d__5.MoveNext() in c:\w\e6923628be6eaf72\source\Octopus.Shared\Activities\ActivityRuntime.cs:line 56

Paul_Stovell · December 20, 2013, 4:38am

Hi Arvind,

Thanks for sharing this. Unfortunately I think you’ve hit a limit for querying Raven that we didn’t expect to reach because of the number of machines. This issue should be fixed in Octopus Deploy 2.0 RC if you are able to consider upgrading.

Paul

arvind_raghunathan · December 20, 2013, 9:24am

Hi Paul - I understand that 2.0 addresses. But we can’t move quickly enough to 2.0 due to priorities - we have an enterprise product that will require gradual adoption. Is there any way you can patch the 1.6 version with the fix as we lose this key functionality esp since we are running a licensed enterprise edition of octopus?

Best
Arvind

Paul_Stovell · December 20, 2013, 8:42pm

Hi Arvind,

Thanks for the update, I’m working on a fix now and will send you a
download URL.

Regards,

Paul Stovell
Octopus Deploy
W: octopusdeploy.com | T: @octopusdeploy http://twitter.com/octopusdeploy

Paul_Stovell · December 20, 2013, 11:58pm

Hi Arvind,

Can you try this updated Octopus Server and let me know if it fixes the issue?

http://download.octopusdeploy.com/octopus/Octopus.1.6.3.1724.msi

Paul

arvind_raghunathan · December 23, 2013, 10:39pm

Thank you Paul – we have tried your fix & it works. We have environment validations to do next several days & will keep you posted if we encounter any further issues. Appreciate the quick turnaround in this matter.

Best
Arvind

akhadka · September 16, 2014, 6:17pm

Hi Paul,

I have similar issue but I am not sure if it is same or different issue. I can create release but can’t deploy to servers. I tried to reset indexes using following link

Here is the log :

2014-09-16 11:56:27,443 [10] ERROR Octopus [(null)] - Unhandled error on web server:

Bad Request

Bad Request - Request Too Long

HTTP Error 400. The size of the request headers is too long.

System.InvalidOperationException:

Bad Request

Bad Request - Request Too Long

HTTP Error 400. The size of the request headers is too long.

---> System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.GetResponse() at System.Net.HttpWebRequest.GetResponse() at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) --- End of inner exception stack trace --- at Raven.Client.Connection.HttpJsonRequest.HandleErrors(WebException e) at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) at Raven.Client.Connection.HttpJsonRequest.ReadResponseJson() at Raven.Client.Connection.ServerClient.DirectPut(RavenJObject metadata, String key, Nullable`1 etag, RavenJObject document, String operationUrl) at Raven.Client.Connection.ServerClient.<>c__DisplayClass13.b__12(String u) at Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result) at Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation) at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation) at Raven.Client.Document.HiLoKeyGenerator.PutDocument(IDatabaseCommands databaseCommands, JsonDocument document) at Raven.Client.Document.HiLoKeyGenerator.GetNextRange(IDatabaseCommands databaseCommands) at Raven.Client.Document.HiLoKeyGenerator.NextId(IDatabaseCommands commands) at Raven.Client.Document.HiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention convention, Object entity) at Raven.Client.Document.MultiTypeHiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention conventions, Object entity) at Raven.Client.Document.DocumentStore.<>c__DisplayClass4.b__2(String dbName, IDatabaseCommands databaseCommands, Object entity) at Raven.Client.Document.DocumentConvention.GenerateDocumentKey(String dbName, IDatabaseCommands databaseCommands, Object entity) at Raven.Client.Document.GenerateEntityIdOnTheClient.GenerateDocumentKeyForStorage(Object entity) at Raven.Client.Document.InMemoryDocumentSessionOperations.StoreInternal(Object entity, Nullable`1 etag, String id, Boolean forceConcurrencyCheck) at Raven.Client.Document.InMemoryDocumentSessionOperations.Store(Object entity) at Octopus.Portal.ViewModels.Deployments.CreateDeploymentCommand.Execute(CreateDeploymentPostModel model) at Octopus.Portal.Controllers.DeploymentsController.Create(String slug, String releaseVersion, CreateDeploymentViewModel model) at lambda_method(Closure , ControllerBase , Object[] ) at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass37.<>c__DisplayClass39.b__33() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.b__49() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass37.b__36(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass25.<>c__DisplayClass2a.b__20() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass25.b__22(IAsyncResult asyncResult) at System.Web.Mvc.Controller.<>c__DisplayClass1d.b__18(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass4.b__3(IAsyncResult ar) at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass4.b__3(IAsyncResult ar) at System.Web.Mvc.MvcHandler.<>c__DisplayClass8.b__3(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass4.b__3(IAsyncResult ar) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Vanessa_Love · October 9, 2014, 2:19am

Hi Akhadka,

I am very sorry but your reply got caught in a spam filter that we only just discovered.
Are you still having this issue?

Vanessa