Custom powershell and vstest

Hi. I am fairly new to Octopus, but have come across a problem that i can’t resolve.
I have a problem with custom build step. I want to run vstest.console.exe after a deploy to verify the success of the deploy. My powershell script fails on postback to TFS with this error: Error: Build “Project.Package_9107” cannot be found under team project “Git”.

The command I use is this:
vstest.console.exe" Tests.dll /inisolation /TestCaseFilter:“TestCategory=Dev” /logger:TfsPublisher;Collection=http://myserver:8080/tfs/default;BuildName=Project.Package_9107;TeamProject=Git;flavor=release

This command works if I run it on the server, either directly by powershell or by console, but if I run it from Octopus it fails. I have tried from custom build step and from script console, but yields the same result. I have upgraded to 3.2 but the result is the same.

Hi,

Thanks for reaching out. Both Script Console and Custom build step are scripts executed by the account running the Tentacle service. Do you know if that account has enough permissions to run that command? A very quick way to test this out would be to change the Tentacle service account to run under your own account, and then trying to run the command again. If it runs with your user, then you know its a permission issue.

Let me know how it goes after trying that

Warm regards

Dalmiro

I’ve been through that. Logged on as the same user that runs the tentacle service, and had none problems running the script. The results was published back to the specific build.

Absolutt worth pointing out, the tests runs perfectly. Its only the postback portion of the script that fails.

Thanks
EirikSF

Here is the result fra the custom step:
08:39:55Info Executing script on 'MSTEST’
08:39:59Info Running MsTests tests…
08:40:00Info Tests.dll /TestCaseFilter:“TestCategory=Dev” /logger:TfsPublisher;Collection=http://myserver:8080/tfs/default;BuildName=Project.Package_9107;TeamProject=Git
08:40:00Info Microsoft ® Test Execution Command Line Tool Version 14.0.23107.0
08:40:00Info Copyright © Microsoft Corporation. All rights reserved.
08:40:02Error Error: Build “Project.Package_9107” cannot be found under team project “Git”.
08:40:02Info Starting test execution, please wait…
08:40:19Info Passed DevAccess1
08:40:34Info Passed DevAccess2
08:40:34Info Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
08:40:34Info Test Run Successful.
08:40:34Info Test execution time: 31,7193 Seconds

Kind Regards
Eirik SF

Hi Eirik,

Its weird that it returns that error, but tests run ok. Doing a quick google of the message “cannot be found under team project” returned these 2 results which might be helpful (not sure really, I have 0 experience with vstest.console)

http://blog.allanglen.com/2010/04/tfs-build-error-when-publishing-test-results-to-tfs

https://social.msdn.microsoft.com/Forums/vstudio/en-US/1f633bab-a479-44a9-83b9-901987eac237/build-xxx-cannot-be-found-under-team-project-xxx-while-publishing-test-results-to-tfs-build?forum=tfsbuild

This seems to be a problem with the way VSTest is being executed, and not with Octopus itself. Something tells me you’re gonna have more luck asking in a forum dedicated to VSTest. Perhaps StackOverflow would be the best place to go.

On the other hand, if all it does is return that error but tests run perfectly like you said, perhaps wrapping the script call in a Try/Catch block on a Powershell script to handle that particular exception and still return a successful exit code wouldn’t be such a bad idea.

Regards,

Dalmiro

Vstest works fine. The test results are written to TFS and are displayed as IT should.

However, when i run these scripts from Octopus, either script console or as custom build step it fails with the above mention error.

I am a bit curious to what context Octopus launches powershell, it seems to me as the culprit.

Kind regards
EirikSF

I’ve analyzed the situation with Fiddler. See screenshots. It seems like vstest does not try to connect to TFS when run by Octopus, but when run directly, by the same user, it does.

Is there any limitations to how Octopus runs powershell commands?

Kind Regards
EirikSF

Hi Eirik,

I’m very sorry for the delay here. Somehow your reply was marked as “read” and it didn’t show up on my queue.

I’m gonna consult with one of our devs if the way we execute Powershell scripts might be causing this behavior somehow. I’ll try to get back to you by tomorrow (dev team is in Australia).

Thanks,

Dalmiro

Hi Eirik,

Could youtry to manually run your script using the Calamari executable?

The syntax would be calamari.exe run-script [path of your script]. You can find Calamari.exe on your tentacle’s app directory, which is by default C:\Octopus\Calamari

Let me know how it goes,

Dalmiro

Running the script using the Calamari executable worked like a charm.

D:\Octopus\Calamari\3.1.2>Calamari.exe run-script --script=d:\temp\postback.ps1
##octopus[stdout-verbose]
Octopus Deploy: Calamari version 3.1.2+Branch.master.Sha.08f592c501569f343a331d5
42df3878d35959e69
##octopus[stdout-default]
Running MsTests tests…
Local script
Tests.dll /TestCaseFilter:“TestCategory=Dev” /logger:TfsPublisher;Collection=htt
p://tfs.xxxxxxxx.xxx:8080/tfs/Collection;BuildName=Application.Package_9708;TeamProje
ct=Git
Microsoft ® Test Execution Command Line Tool Version 14.0.23107.0
Copyright © Microsoft Corporation. All rights reserved.

Starting test execution, please wait…
Passed DevAccess1
Passed DevAccess2
Publish completed successfully. Test Results: mtm://tfs.xxxxxxxx.xxx:8080/tfs/HVIKT /p:Git/Testing/testrun/open?id=4187.

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 4,4394 Seconds

So, where does this leave us?

Hi,

The Tentacle uses the same Calamari executable you just used to execute steps during the deployment. The only difference between these 2 methods,is that one is running interactively(manual Calamari execution which appears to work), and one as a non-interactive Windows Service (from the Octopus deployment). It appears that vstest.console.exe is doing something “unexpected” and not just a plain console call, reason why it doesn’t work from a non-interactive process.

We’ve seen users workaround similar situations by setting up a scheduled task on the server using schtasks.exe that invokes that console app.

Other than that, I’ve asked around the team and we are unfortunately out of ideas. Perhaps the Microsoft forums could help explaining why cant that executable be called from a service?

Regards,

Dalmiro