applicationHost.config destroyed

marchenko_alexandr · February 17, 2017, 10:07am

Hello, seems that Octopus successfully destroyed applicationHost.config thanks to gods that it is staging server and not production

There DEFINITELY should be revert action in case of unsuccessful changes to it!

marchenko_alexandr · February 17, 2017, 11:02am

If someone gets into same trouble here is way to restore file:

https://www.codeproject.com/Tips/630880/Recover-IIS-When-ApplicationHost-config-is-Ruined

Dalmiro · February 17, 2017, 12:35pm

Hi,

Thanks for getting in touch. I’m very sorry your file got into such state. We haven’t heard other reports of this behavior, so we’ll be very interested in getting all the information possible to try to reproduce it and come up with a fix.

Could you start by sending us the log of the deployment that broke the config file?

Thanks,
Dalmiro

marchenko_alexandr · February 17, 2017, 2:10pm

Hi Dalmiro

Yes we can try to investigate what happen, at least now when everything operates normal I will be glad to give you any additional info

Just need to clean up some data from logs which may be sensitive (or is there a way to send you logs directly?)

From audit logs I do see following:

there was failed deploy in 11:20

in error logs for it I do see

Running: Upload package migrator version 1.0.141-master
    ...
    System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
        ...
        System.IO.FileLoadException: Could not load file or assembly 'NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Provider DLL failed to initialize correctly. (Exception from HRESULT: 0x8009001D)

then in ten minutes there is successful deploy of same project / version

then in ten more minutes failed deploy of new version where everything broke

in logs it says that it were broken right after trying to set basic authentication enabled to false which may not be related to problem

11:41:28   Info     |       Comparing existing IIS bindings with configured bindings...
11:41:28   Info     |       Looks OK
11:41:28   Info     |       Bindings are as configured. No changes required.
11:41:28   Info     |       Anonymous authentication enabled: True
11:41:28   Info     |       Applied configuration changes to section "system.webServer/security/authentication/anonymousAuthentication" for "MACHINE/WEBROOT/APPHOST/Rabota.ua2 Mobile" at configuration commit path "MACHINE/WEBROOT/APPHOST"
11:41:28   Info     |       Basic authentication enabled: False
11:41:28   Info     |       ERROR ( message:Configuration error
11:41:28   Info     |       Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
11:41:28   Info     |       Line Number: 1563
11:41:28   Info     |       Description: Configuration file is not well-formed XML
11:41:28   Info     |       . )

Here is when detective story begins:

Looking at applicationHost.config seems that it was overriden by awssdk.s3.xml file which is going with https://www.nuget.org/packages/AWSSDK.S3/

That xml is shipped with dlls, I have checked its nuget install.ps1 and there is nothig special with it, also nuget packages are shipped inside packages already preinstalled so should no do anything to them

Also looking at your code I definitely may say that it also wont do anything like that

So by logs it seems that octopus were able to verify all bindings which makes me thinking that in that point of time applicationHost.config were ok

Then there was anonymous auth https://gist.github.com/PaulStovell/9522828#file-octopus-features-iiswebsite_beforepostdeploy-ps1-L202 which in my opinion should not brake things, but after that next step with basic auth were not able to read xml

I bed that you do not hear stories like that every day, but I also see this first time, we are using Octopus for a year now and it is first time when things go wrong

Just wondering will we be able to identify at least what happen just because from now on it becomes to scary to deploy anything to productions just because in one click whole server may be destroyed

PS: as about iis config backup seems that system makes them automatically which was our day saver

Dalmiro · February 20, 2017, 5:29pm

Hi,

Thanks a lot for sending all that detailed info. Is there any chance you can send us your full log to Support@Octopus.com? I’ll have someone in our dev team check it out.

Thanks,
Dalmiro

marchenko_alexandr · February 21, 2017, 9:05am

Thank you Dalmiro

I sent logs, packages, description to email with subject “Detailed logs for applicationHost.config destroyed [Problems #51448]”

Hope it will help in investigation

In meanwhile will not touch anything in case we will need to try reproduce this (thank to gods it was stagin server so if there will be need theoretically I can try reprpoduce it and collect some more data)

BR
Alex

nick · February 27, 2017, 4:02am

Hello Alex,
Thanks for giving us all those details, we had a discussion and some thought about this.

A popular feature: ‘parallel deploys’ makes trying to roll back changes to the ‘applicationHost.config’ not possible, as it would break the behavior for those customers, so we won’t be trying to introduce a way to recover such files automatically.

What you can do in your case to give you more confidence is introduce 2 extra scripted steps; first one being a pre-deploy step to back up that file (and any others). Followed by a deploy-failure step (you run only if needed) to revert those files if it does fail. You could also give the Guided Failures feature a try to give you extra safety: https://octopus.com/docs/deploying-applications/guided-failures

As for the issue of the file itself becoming corrupted; we had a look and with the details you supplied cannot work out went wrong. Please let us know if you experience it again and hopefully additional information might help pinpoint the problem.

Best Regards,
Nick

marchenko_alexandr · February 27, 2017, 9:02am

Hi Nick

Thanks for getting in touch, I do understand your point about parallel
deployments and it seems reasonable, also thank you for pointing out for
possible solution

BTW It seems that there is no need to manually backup file, system backs it
up every time changes occur so need just one step to recover it in case of
fail

I hope to never ever again catch such issue

Thank you in advance
Alex