File lock issues during deployment

mark_wojciechowicz · April 29, 2015, 2:50pm

Hi,
I have a deployment step that keeps throwing file lock errors (not on all tentacles). The step that is failing downloads a nuget package, copies the files to a custom directory and then does a variable substitution. Originally, I had a post deployment script which would delete the real configuration files and rename the templates. That part of the installation was throwing a file lock error so I moved that to a completely separate step in the process. Now I am getting the following error:

“Failed receiving Octopus.Tentacle.Procedures.ProcedureCalledEvent
| The process cannot access the file because it is being used by another process.
| System.IO.IOException: The process cannot access the file because it is being used by another process.
| at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
| at System.IO.File.InternalReplace(String sourceFileName, String destinationFileName, String destinationBackupFileName, Boolean ignoreMetadataErrors)”

Here is the raw log:

Task ID: ServerTasks-4203
Task status: Failed
Task queued: Wednesday, April 29, 2015 9:34 AM
Task started: Wednesday, April 29, 2015 9:34 AM
Task duration: 9 seconds

                | Failed: Deploy MDR release 1.0.248.3 to Dev

09:34:14 Info | Step 5: Review Change Script does not apply to the current environment, and will not be executed
09:34:14 Info | Deploying to machine: machines-1
09:34:17 Verbose | Verify Environment completed
09:34:17 Verbose | Packages acquired
09:34:23 Verbose | First chance failure detected in: Running step “Install Application” on “”…
| One or more items in the current operation failed.
| Octopus.Server version 2.6.0.778
09:34:23 Verbose | An operation in “Install Application” failed
| Operation: Running step “Install Application” on “” failed with error: One or more items in the current operation failed.
| Octopus.Server version 2.6.0.778
09:34:23 Verbose | Step “Replace RDS files with templates” runs only when all previous steps succeeded; skipping
09:34:23 Verbose | Step “Generate Db Report and Change Script” runs only when all previous steps succeeded; skipping
09:34:23 Verbose | Step “Deploy Database Changes” runs only when all previous steps succeeded; skipping
09:34:23 Verbose | Step “Deploy Reports” runs only when all previous steps succeeded; skipping
09:34:23 Verbose | Step “Start Source File Watcher” runs only when all previous steps succeeded; skipping
09:34:23 Verbose | Install Application completed
09:34:23 Fatal | A step failed
09:34:23 Verbose | The task failed
|
| Success: Step 1: Verify Environment
09:34:14 Info | Executing step: Verify Environment
|
| Success:
09:34:14 Verbose | Starting Running step “Verify Environment” on ""
09:34:14 Info | Running “Verify Environment” on “”
|
| Success: Tentacle script execution
09:34:14 Verbose | Running PowerShell script: C:\Users<MyServiceAccount>\AppData\Local\Tentacle\Temp\8c23d3af-d1bc-4700-987a-146000495c98.ps1
09:34:16 Info | ==============================================
| PowerShell exit code: 0
| ==============================================
|
| Success: Acquire packages
09:34:17 Info | Downloading packages
09:34:17 Info | Making a list of packages to download
09:34:17 Verbose | The following packages will be downloaded by the Octopus Server:
09:34:17 Verbose | - MDR 1.0.248 (feed: TeamCity_Mdr)
09:34:17 Verbose | The following packages will be downloaded directly by Tentacles:
09:34:17 Info | All packages have been downloaded
|
| Success: Package MDR 1.0.248 from NuGet feed: TeamCity_Mdr
09:34:17 Verbose | Starting Acquire package MDR.1.0.248
09:34:17 Verbose | Checking package cache for package MDR 1.0.248
09:34:17 Verbose | Package was found in cache. No need to download. Using file: C:\Octopus\PackageCache\feeds-teamcity-mdr\MDR.1.0.248_097D4008AD507F4C952B33F2979E2C7A.nupkg
09:34:17 Verbose | SHA1 hash of package is: 2a8e3d943dd42cd41ab575777d34afaa796f48c1
09:34:17 Verbose | Determining machines that need the package
|
| Success: Upload package to https://:10933/
09:34:17 Verbose | Starting Upload package to https://:10933/
09:34:17 Info | Uploading MDR.1.0.248 (6.28 MB) to SQ--FBFF4DBF
09:34:17 Verbose | Checking to see if the package is present on the tentacle
09:34:17 Info | Package MDR 1.0.248 with hash 2a8e3d943dd42cd41ab575777d34afaa796f48c1 has already been uploaded to tentacle SQ--FBFF4DBF
|
| Failed: Step 2: Install Application
09:34:17 Info | Executing step: Install Application
09:34:23 Fatal | The step failed
|
| Failed:
09:34:17 Verbose | Starting Running step “Install Application” on ""
09:34:17 Info | Running “Install Application” on ""
09:34:23 Verbose | Guided Failure is not in use for this deployment; failing.
|
| Failed: Deploy MDR on
09:34:17 Info | Beginning deployment…
09:34:17 Verbose | Begin deployment of package: MDR.1.0.248
| Package file path: C:\Octopus\Applications.Tentacle\Packages\MDR.1.0.248_D2A807CB6F9E134E86ECDB0820143A20.nupkg
| Package file hash: 2a8e3d943dd42cd41ab575777d34afaa796f48c1
| Tentacle Agent information:
| - Machine name:
| - Is 64-bit: True
| - Service user name:
| - CLR version: 4.0.30319.18444
| - Current directory: C:\Windows\system32
| - OS version: Microsoft Windows NT 6.1.7601 Service Pack 1
| - Tentacle version: 2.6.0.778
09:34:17 Verbose | Package will be extracted to: C:\Octopus\Applications\Dev
09:34:17 Verbose | The package is already installed, a directory name suffix will be used.
09:34:17 Verbose | Installing package MDR.1.0.248 from uploaded package cache into C:\Octopus\Applications\Dev\MDR\1.0.248_4
09:34:19 Verbose | 233 files were extracted from the package
09:34:19 Verbose | Initial variable evaluation performed.
09:34:19 Verbose | Variables have been fully evaluated.
09:34:19 Verbose | Running procedure 'Delete temporary package files’
09:34:19 Verbose | Looking for target files for variable substitution…
09:34:19 Verbose | Looking for any configuration transformation files
09:34:22 Verbose | Storing a record of the deployment.
09:34:22 Verbose | Failed receiving Octopus.Tentacle.Procedures.ProcedureCalledEvent
| The process cannot access the file because it is being used by another process.
| System.IO.IOException: The process cannot access the file because it is being used by another process.
| at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
| at System.IO.File.InternalReplace(String sourceFileName, String destinationFileName, String destinationBackupFileName, Boolean ignoreMetadataErrors)
| at Octopus.Platform.Util.OctopusPhysicalFileSystem.OverwriteAndDelete(String originalFile, String temporaryReplacement) in y:\work\refs\heads\release\source\Octopus.Platform\Util\OctopusPhysicalFileSystem.cs:line 269
| at Octopus.Tentacle.Deployment.Journal.DeploymentJournalReaderWriter.Overwrite(IEnumerable1 elements) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournalReaderWriter.cs:line 71 | at Octopus.Tentacle.Deployment.Journal.DeploymentJournalReaderWriter.Append(XElement element) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournalReaderWriter.cs:line 50 | at Octopus.Tentacle.Deployment.Journal.DeploymentJournal.AppendSuccessfulDeployment(JournalEntry journalEntry, ILog log, String& retentionToken) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournal.cs:line 29 | at Octopus.Tentacle.Orchestration.Deploy.Package.TentaclePackageDeployment.StartNextProcedure(Boolean reevaluate) in y:\work\refs\heads\release\source\Octopus.Tentacle\Orchestration\Deploy\Package\TentaclePackageDeployment.cs:line 299 | at Octopus.Tentacle.Orchestration.Deploy.Package.TentaclePackageDeployment.Receive(ProcedureCalledEvent reply) in y:\work\refs\heads\release\source\Octopus.Tentacle\Orchestration\Deploy\Package\TentaclePackageDeployment.cs:line 377 | at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113 | Tentacle version 2.6.0.778 09:34:22 Fatal | The process cannot access the file because it is being used by another process. | System.IO.IOException: The process cannot access the file because it is being used by another process. | at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) | at System.IO.File.InternalReplace(String sourceFileName, String destinationFileName, String destinationBackupFileName, Boolean ignoreMetadataErrors) | at Octopus.Platform.Util.OctopusPhysicalFileSystem.OverwriteAndDelete(String originalFile, String temporaryReplacement) in y:\work\refs\heads\release\source\Octopus.Platform\Util\OctopusPhysicalFileSystem.cs:line 269 | at Octopus.Tentacle.Deployment.Journal.DeploymentJournalReaderWriter.Overwrite(IEnumerable1 elements) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournalReaderWriter.cs:line 71
| at Octopus.Tentacle.Deployment.Journal.DeploymentJournalReaderWriter.Append(XElement element) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournalReaderWriter.cs:line 50
| at Octopus.Tentacle.Deployment.Journal.DeploymentJournal.AppendSuccessfulDeployment(JournalEntry journalEntry, ILog log, String& retentionToken) in y:\work\refs\heads\release\source\Octopus.Tentacle\Deployment\Journal\DeploymentJournal.cs:line 29
| at Octopus.Tentacle.Orchestration.Deploy.Package.TentaclePackageDeployment.StartNextProcedure(Boolean reevaluate) in y:\work\refs\heads\release\source\Octopus.Tentacle\Orchestration\Deploy\Package\TentaclePackageDeployment.cs:line 299
| at Octopus.Tentacle.Orchestration.Deploy.Package.TentaclePackageDeployment.Receive(ProcedureCalledEvent reply) in y:\work\refs\heads\release\source\Octopus.Tentacle\Orchestration\Deploy\Package\TentaclePackageDeployment.cs:line 377
| at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113
| Tentacle version 2.6.0.778
09:34:22 Verbose | Storing a record of the deployment.
|
| Success: Substitute variables in Reports\Data Sources\Db2.rds.template
09:34:19 Info | Loading file ‘C:\Octopus\Applications\Dev\MDR\1.0.248_4\Reports\Data Sources\Db2.rds.template’ for Octopus variable substitution
09:34:19 Verbose | Binding variables…
09:34:19 Verbose | Evaluating template…
09:34:19 Verbose | Truncating original file and inserting new contents; utf-8 encoding will be used…
09:34:19 Info | Substitution complete.
|
| Success: Substitute variables in Reports\Data Sources\PecoAMIMDR.rds.template
09:34:19 Info | Loading file ‘C:\Octopus\Applications\Dev\MDR\1.0.248_4\Reports\Data Sources\PecoAMIMDR.rds.template’ for Octopus variable substitution
09:34:19 Verbose | Binding variables…
09:34:19 Verbose | Evaluating template…
09:34:19 Verbose | Truncating original file and inserting new contents; utf-8 encoding will be used…
09:34:19 Info | Substitution complete.
|
| Success: Copy files to E:\Peco_AMI_MDR
09:34:19 Info | Copying package contents to ‘E:\Peco_AMI_MDR’
|
| Skipped: Step 3: Replace RDS files with templates
|
| Skipped: Step 4: Generate Db Report and Change Script
|
| Skipped: Step 6: Deploy Database Changes
|
| Skipped: Step 7: Deploy Reports
|
| Skipped: Step 8: Start Source File Watcher
|

Dalmiro · April 29, 2015, 5:15pm

Hi Mark,

Thanks for reaching out. The step is failing when Octopus tries to write into one of its system files where it tracks the deployment proress. The usual suspects here are AntiVirus scanning directories when they change, and locking files while doing so. Is it possible that you have an antivirus doing this? If so, Do you thinkyou could add an exception to these directories and test again?

Thanks

Dalmiro

mark_wojciechowicz · April 29, 2015, 5:37pm

Dalmiro,
I will pursue getting an exception added - though it’s locked up tight so I will have to submit a request.
Incidentally, I am able to get a release to work if I restart the Octopus Service. However, it does soon get locked up again. Is the only exception I need on the server hosting octopus in the directory where octopus is installed?

Mark

Dalmiro · April 30, 2015, 1:21pm

Hi Mark,

The file in question here is DeploymentJournal.xml which is where Octopus logs the deployment activities as they go. This file is located at C:\Octopus\Applications on a default install and its on every tentacle. This directory would be a good start point in case you have to raise an exception.

Thanks

Dalmiro