Fatal: Collection was modified; enumeration operation may not execute

Legacy Uncategorised
1

Hi,

To speed up our deployment process that have 68 steps we recently configured several of the steps to “Run in parallel with the previous step”. As a result we now often get a new error during deployment.

Looks like the Octopus code is not thread safe and it get’s a fatal error during deployment. The stack trace is always the same so it should hopefully be easy to find the problem.

                    |   == Failed: Step 58: TextAnalytics ==
17:36:54   Verbose  |     An operation in "TextAnalytics" failed
                    |     Collection was modified; enumeration operation may not execute.
                    |     System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
                    |     at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
                    |     at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
                    |     at Octopus.Core.Model.Variables.VariableCollection.ToDictionary() in Y:\Work\refs\tags\3.4.4\source\Octopus.Core\Model\Variables\VariableCollection.cs:line 97
                    |     at Octopus.Server.Orchestration.Deploy.DeploymentPlan.CreateMachineCriteria(ICollection`1 roles) in Y:\Work\refs\tags\3.4.4\source\Octopus.Server\Orchestration\Deploy\DeploymentPlan.cs:line 177
                    |     at Octopus.Server.Orchestration.Deploy.DeploymentPlan.CreateMachineCriteria(PlannedAction action) in Y:\Work\refs\tags\3.4.4\source\Octopus.Server\Orchestration\Deploy\DeploymentPlan.cs:line 171
                    |     at Octopus.Server.Orchestration.Deploy.DeploymentTaskController.DistributeStepAcrossMachines(PlannedStep step) in Y:\Work\refs\tags\3.4.4\source\Octopus.Server\Orchestration\Deploy\DeploymentTaskController.cs:line 219
                    |     at Octopus.Server.Orchestration.Deploy.DeploymentTaskController.DispatchStep(PlannedStep step) in Y:\Work\refs\tags\3.4.4\source\Octopus.Server\Orchestration\Deploy\DeploymentTaskController.cs:line 103
                    |     Octopus.Server version 3.4.4 (3.4.4+Branch.master.Sha.ff35e6d80a68240faa886aa5940d18038c8bd45d)
17:36:54   Fatal    |     The step failed: Collection was modified; enumeration operation may not execute.
17:36:54   Verbose  |     TextAnalytics completed

I hope you will be able to provide a fix for this soon.

Regards
Jan

2

Hi Jan,

Thanks for getting in touch. Based on that stack trace we are already working on making this code deal better with concurrent actions: https://github.com/OctopusDeploy/Issues/issues/2719

Hope that helps!
Mike

3

That is good to hear :slight_smile:

I’ll monitor the issue. Hope you manage to get it resolved soon

All the best,
Jan