Error when running scheduled task: ProcessAutoDeployments

Legacy Execution
1

Hello !

None of the auto triggers work on our octopus server, all failed with following error which is spamming the log file:

It seems to be related to linq limitation but wonder if we can change anything in database etc to fix it:

The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.

Thanks

Full error:

2017-01-09 00:00:44.1293 15 ERROR Error when running scheduled task: ProcessAutoDeployments
System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at Octopus.Core.RelationalStorage.TransientFaultHandling.IDbCommandExtensions.<>c__DisplayClass5_0.b__0() in Y:\Work\refs\tags\3.4.11\source\Octopus.Core\RelationalStorage\TransientFaultHandling\IDbCommandExtensions.cs:line 67
at Octopus.Shared.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func1 func) in Y:\Work\refs\tags\3.4.11\source\Octopus.Shared\TransientFaultHandling\RetryPolicy.cs:line 215 at Octopus.Core.RelationalStorage.RelationalTransaction.<Stream>d__331.MoveNext() in Y:\Work\refs\tags\3.4.11\source\Octopus.Core\RelationalStorage\RelationalTransaction.cs:line 388
at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at System.Collections.Generic.List1…ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Octopus.Server.Schedules.AutoDeployments.MachineFinder.GetMachinesThatHaveTriggered(Nullable1 fromDate, DateTimeOffset toDate) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Schedules\AutoDeployments\MachineFinder.cs:line 52 at Octopus.Server.Schedules.AutoDeployments.MachineFinder.GetMachinesThatTriggeredWhileDeploymentsInProgress(Nullable1 fromDate, DateTimeOffset toDate) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Schedules\AutoDeployments\MachineFinder.cs:line 85
at Octopus.Server.Schedules.ProcessAutoDeployments.Trigger() in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Schedules\ProcessAutoDeployments.cs:line 68
at Octopus.Server.Schedules.Scheduler.RunTask(IRunOnASchedule task) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Schedules\Scheduler.cs:line 135
ClientConnectionId:2443e462-bc3e-43ca-95a7-df925ae3f3f9
Error Number:8003,State:1,Class:16

2

Hi Raf,

Thanks for reporting this. Could you confirm the version of Octopus you are using? I believe this issue was fixed in 3.5.1: https://github.com/OctopusDeploy/Issues/issues/2860

If you are on a lower version upgrading past that version should resolve the issue for you.

Let me know.
Vanessa

3

Thanks we are on 3.4.11.

Do you know if there is any temporary solution that we can use to fix it while we preparing for upgrade ?

We create new environments every night then adding to the octopus and auto-trigger should pick this up and start release.

Also we do not have 2100 machines at the moment maybe 100

4

Hi Raf,

There is only one workaround and it isn’t fantastic. You need to reset the date that autodeploy last successfully ran so that it is not trying to find all past events. It means that none of the previous events will run but it will stop the errors and will hopefully trigger new ones.

You need to run the following SQL query on your database. Now please backup your database as you will be running a live query.
UPDATE [Configuration] SET [JSON]='{"AutoDeployLastProcessed":"2017-01-15T12:00:00.0000000+00:00"}' WHERE [Id]='scheduleconfiguration'

When that is run it should at least stop the errors and anything after that date should trigger.

We recommend upgrading as soon as you can. Sorry that you have run into this problem.
Vanessa.

5

Thank you very much.

6

Hi Raf,

Have you had any luck with this?
I wanted to let you know a further update has been done as part of 3.8 to make auto-deploy more robust in these situations and we would recommend you upgrade to 3.8 when you do.

Vanessa

7

Thanks for update, I think update to 3.8 would be the best option so I should be doing this shortly and send update.

8

Hi Vanessa, We will be upgrading to v 3.8 at the end to fix this issue.
I wonder if you would be able to just advise if we have to also upgrade database from 3.4.11 to 3.8 or can we just re point 3.8 octopus server to existing database running on 3.4.11.

Actually I raised question for it:
http://help.octopusdeploy.com/discussions/questions/10520-moving-octopus-to-new-server-and-upgrading-from-3411-to-38

9

Hi Raf,

I have responded on the other thread :slight_smile:

Vanessa