Octopus Deploy SSH target on macOS -- Required command 'mono' is not available

Hi guys,

I’ve been trying to set up an Octopus SSH target on a macOS Sierra (10.12.4) machine, by following this guide: https://octopus.com/docs/deployment-targets/ssh-targets

So far, here’s what I’ve done:

  1. Create an “octopus” admin local account on the macOS machine
  2. Configure the SSH target on the Octopus Server
  3. Install mono on macOS using the .pkg file from the official mono site: http://www.mono-project.com/download/#download-mac

A health check crashes, telling me that the “Required command ‘mono’ is not available”.

Here’s the full raw output:

                    |   == Failed: Check deployment target: IIT-AS-002 (Apple Server 2) ==
09:39:49   Verbose  |     Performing health check on machine
09:39:49   Verbose  |     Requesting upload...
09:39:49   Verbose  |     Establishing SSH connection...
09:39:49   Verbose  |     SSH connection established
09:39:50   Info     |     Beginning streaming transfer of command.sh to $HOME\.octopus\OctopusServer\Work\20170712133949-14282-255
09:39:50   Verbose  |     Establishing SFTP connection...
09:39:50   Verbose  |     SSFTP connection established
09:39:50   Info     |     Stream transfer complete
09:39:51   Verbose  |     Requesting upload...
09:39:51   Info     |     Beginning streaming transfer of health-check.sh to $HOME\.octopus\OctopusServer\Work\20170712133949-14282-255
09:39:51   Info     |     Stream transfer complete
09:39:52   Error    |     Required command 'mono' is not available.
09:39:52   Verbose  |     Disposing SFTP connection...
09:39:52   Verbose  |     Disposing SSH connection...
09:39:52   Fatal    |     The remote script failed with exit code 127
09:39:52   Verbose  |     The remote script failed with exit code 127
                    |     Octopus.Shared.Tasks.ActivityFailedException: The remote script failed with exit code 127
                    |     at Octopus.Worker.Scripting.ScriptResult.EnsureSuccessful()
                    |     at Octopus.Server.Orchestration.Health.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction)
                    |     Octopus.Server version 3.14.15926 (3.14.15926+Branch.master.Sha.8dbec332182dd6cef8f901de36416d3b965c2e5e)
09:39:52   Verbose  |     Recording health check results

It looks like mono command is available though:
(the command was run as the “octopus” account on the macOS machine)

IIT-AS-002:~ octopus$ mono --version
Mono JIT compiler version 5.0.1.1 (2017-02/5077205 Thu May 18 16:11:37 EDT 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           normal
        SIGSEGV:       altstack
        Notification:  kqueue
        Architecture:  x86
        Disabled:      none
        Misc:          softdebug
        LLVM:          yes(3.6.0svn-mono-master/8b1520c)
        GC:            sgen (concurrent by default)

Some more helpful info, here’s the path content:

IIT-AS-002:~ octopus$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands

Any ideas what I might be missing?
Thanks

Hi Guillaume, thanks for reaching out.

I suspect the issue is that the environment differs between a regular login and a SSH logon. You can confirm this by logging into the Mac instance via SSH and running

command -v mono

In a local login you will see the path to the mono executable, but in the SSH session this may not return anything, which is why the deployment is failing.

https://git-annex.branchable.com/forum/OSX__39__s_default_sshd_behaviour_has_limited_paths_set/ has some information on how to configure the SSH session to match the local one, which should allow mono to be run via SSH.

Regards
Matt C

Hi Matt,

Thanks for the tip. The path looks fine from SSH, though:

IIT-AS-002:~ octopus$ command -v mono
/Library/Frameworks/Mono.framework/Versions/Current/Commands/mono

Best,
Guillaume

Hi Guillaume,

This is a copy of the checks that are performed by the Octopus server.

SESSION_TYPE=local
if [ -n "$SSH_CLIENT" ] || [ -n "$SSH_TTY" ]; then
  SESSION_TYPE=SSH
else
  case $(ps -o comm= -p $PPID) in
    sshd|*/sshd) SESSION_TYPE=SSH;;
  esac
fi
echo $SESSION_TYPE

declare -a arr=("df" "ls" "grep" "openssl" "mkdir" "tar" "mono")
deps=0
for cmd in "${arr[@]}";
do
	echo Testing for $cmd
	command -v $cmd
	if [[ $? != 0 ]]; then
		echo >&2 "Required command '$cmd' is not available.";
		deps=1;
	fi
done;
if [[ $deps != 0 ]]; then exit 127; fi

Can you try saving this to a file called test.sh and running it against the Mac target with a command like this (you will need to replace the key name so it matches your local private key):

ssh -i privatekey.pem octopus@IIT-AS-002 'bash -s' < test.sh

This will simulate the check that is performed by the Octopus server. If you can paste in the result of this script that will help us narrow down the issue.

Regards
Matt C

Hi Matt,

Here are the results:

IIT-AS-002:~ octopus$ bash test.sh
SSH
Testing for df
/bin/df
Testing for ls
/bin/ls
Testing for grep
/usr/bin/grep
Testing for openssl
/usr/bin/openssl
Testing for mkdir
/bin/mkdir
Testing for tar
/usr/bin/tar
Testing for mono
/Library/Frameworks/Mono.framework/Versions/Current/Commands/mono

I don’t know if that changes anything, but the Octopus Server is connecting to IIT-AS-002 (the Apple server) with username/password, not with public/private keys.

I’m ran the tests throught an SSH connection started from Bitvise (my SSH client on Windows).

Best,
Guillaume

Hi Guillaume,

There is an option to use raw scripting for SSH targets that might help us here. You can enable raw scripting by creating a project using the instructions at https://octopus.com/blog/trying-raw-octopus#setting-it-up. This removes the need for mono on the target system.

If you don’t need the full capabilities provided by Calamari (which is what has the dependency on mono), raw scripting may be a valid workaround for you.

If you do need the functionality provided by Calamari, then we can test the environment that Octopus logs into by creating a script step and placing the example script above into it (see the attached screenshot for an example).

Also, what version of Octopus are you running?

Regards
Matt C

Hi Matt,

Many thanks for your help. It guided me in the right direction.
You were right when you said that the environment differs between a regular login and a SSH logon.

The solution was to create the .bashrc file in the home (~) directory of the Octopus Deploy user (named simply “octopus” in my case).

For me, on the Apple machine, the home directory was /Users/octopus.

Then, add the following content to the file:

export PATH=/Library/Frameworks/Mono.framework/Versions/Current/bin/:${PATH}

This way, the $PATH will always contain the location of the mono binaries.

I first tried doing this with the ~\.bash_profile file, but that’s for the login shell. Octopus Deploy starts an interactive, non-login shell, and thus use the shell config ~\.bashrc for Bash.

I found the first half of the answer here in the section “Bash startup files” :
https://octopus.com/docs/deployment-targets/ssh-targets

And the other half here:

And here:

Hope that might help other people that run into this issue in the future.

Cheers,
Guillaume