Secure Service Fabric deployment fails on connection

I’m pleased there was a real good progress to add first-class support for Service Fabric deployment.
However I’m still stuck with some issues. Unsecure Service Fabric deployment works well, but secure - fails. I had a lot of “fun” with certificates, but it seems I nailed it down finally.

What I configured for my deployment:

However it fails and I get:
Connect-ServiceFabricCluster : An error occurred during this operation. Please check the trace logs for more details. (not sure where to find mentioned trace logs, but there is not too much additional info).

Actually everything (certificates and secure Service Fabric itself) works deploying outside of the Octopus. My guess is that it fails on Octopus, because it’s not enough to put certificate to Octopus library, but it also should exist in Windows Certificate store on Octopus server (Connect-​Service​Fabric​Cluster cmdlet uses StoreLocation and StoreName).

To make sure I tried connecting to Service Fabric cluster from my local machine using same cmdlet as in https://github.com/OctopusDeploy/Calamari/blob/master/source/Calamari.Azure/Scripts/AzureServiceFabricContext.ps1, and using same certificate, installed to ‘LocalMachine\My’ store. I was able to connect without issues, but once I removed the certificate from local certificate store - I got same error as during Octopus deployment.

I also found out there is a special step to “Import Certificate to Windows Store” (https://octopus.com/docs/deploying-applications/certificates/import-certificate-step). However I quickly was blocked again - this step can be run only against “deployment target”, while Service Fabric deployment step is run against “Octopus server”. So it didn’t help.

Does my guess sounds like a possible cause? Actually I’m not 100% sure, because the error I get looks quite generic.
Or am I doing something wrong? From deployment guideline it sounds like it should be quite straightforward…
I would really like to avoid putting certificate to Octopus server Window Certificates Store, because it would make certificates management from Octopus itself pointless… And I’m not speaking, that it would need permissions to do it, which is a problem for bigger organization where Octopus is used for many different projects/teams.

Hi Aleksandras,

Thanks for getting in touch.

Could you please confirm whether you’ve created certificate variables for your project? The variables have to be of type “certificate”.

For example, in your project variable’s list, in the Value field, you click “Show editor” where you can then choose a variable type from a dropdown. Select “Certificate” and you should see certificates that are managed by Octopus (as shown here. Note that once saved, certificate variables are displayed with a little certificate icon next to them.

Once you save those changes in your project variables page, that certificate variable should then show up in your Service Fabric step’s dropdown list for your project.

Regarding where the certificate is installed to, please see the “Location matters” section (under step 2) of the Connecting securely with client certificates documentation. For Service Fabric, we allow you to override the location if you need, but assume LocalMachine by default.

Let me know how you go.

Cheers
Mark

Thanks for response!

Yes, variables are set correctly (with “little certficate icon” and so on). Otherwise I guess I wouldn’t be able to select it in Service Fabric deployment step, isn’t so?
More to say, in verbose output of the deployment, before the error - I see all parameters are filled in correctly:

Using ConnectionParameters:  
ServerCertThumbprint=<my thumbprint>  
ConnectionEndpoint=<my dns name>  
StoreLocation=LocalMachine   
FindType=FindByThumbprint   
X509Credential=True   
FindValue=<my thumbprint>  
StoreName=MY  
Connect-ServiceFabricCluster : An error occurred during this operation. Please check the trace logs for more details.

I read carefully once again Connecting securely with client certificates documentation
In this case, Octopus Server (the client) will be connecting to Service Fabric (the server) during a deployment. Therefore this client certificate will need to reside on your Octopus Server machine. - I thought this meant “Certificates” library, but it seems it means Octopus server/machine Certificate store. So it turns out certificate should be installed somewhere else on the server, not just included into Certificates library… Is that correct?

I see there is an option to override “Location” (in “Location matters” part), but it don’t see how could it help if we still would need to install certificate somewhere outside Octopus app (I mean by someone, who has access to Octopus machine, not Octopus end-user).
Actually I tried to set it to “CurrentUser” and saw it was applied in verbose log, but it didn’t help…
Or do you say it can be overwritten to something that could help to avoid dealing with certificates outside Octopus?

Update: better news I managed to workaround it with custom Powershell step. It takes certificate Pfx Base64 encoded value (stored in Octopus variable), saves it as temporary file to some local Octopus location and then imports it to Octopus LocalMachine certificate store. Before it, it also checks if certificate doesn’t exist there already using Test-Path -Path "cert:\LocalMachine\My\<mythumbprint>".

It works, but seems like a workaround. As Octopus end-user I would like that Octopus handle it in a more transparent way - e.g. in Service Fabric deployment step, or maybe in “Certificate” library/variables logic (having an additional option saying something like “Import certificate to Machine store”).
What do you think? :wink:

Hi Aleksandras,

That’s good news. I’m glad you got a PowerShell solution to automate this :slight_smile:

Currently, the certificate itself will still need to be installed on the Octopus Server manually.

This experience could be automated and we agree with you. We have created a GitHub issue that you can track to be notified when this feature is available. Until then, the workaround is to manually install the certificate.

Cheers
Mark

Glad to hear that you are going to implement it! :wink:

On the side node - I mentioned my solution as a workaround, not just because I had to do it manually, but also because it just very happy-path for managing certificates.
E.g. I’m checking if certificate doesn’t exist in the store and only then importing it, but I’m not checking that existing certificate can be expired and therefore overwritten. Since Octopus has good monitoring for certificates expiration and also updating/archiving functionality - it would be nice that what you are planning to implement, would re-use existing Octopus functionality as well.

Sorry, it seems my apetite grows with eating - but I hope that makes sense. :wink:

Hi Aleksandras,

Yes that makes sense. Thanks for the additional details :slight_smile:

Cheers
Mark