External Feed matching packages Internal Server Error

[pfinigan]

Hello,
I’m getting an Internal Server error when I try to search for a package name. I’ve attached a zip file with screen shots of the process I followed before getting this error.

===================================
System.InvalidOperationException: Internal Server Error —> System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. —> System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes" ?>ServerErrorExceptionInternal Server Error
at System.Data.Services.Client.QueryResult.Execute()
at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)
— End of inner exception stack trace —
at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)
at System.Data.Services.Client.DataServiceQuery1.Execute() at System.Data.Services.Client.DataServiceQuery1.ExecuteInternal()
at NuGet.DataServiceQueryWrapper1.Execute[TResult](Func1 action)
— End of inner exception stack trace —
at NuGet.DataServiceQueryWrapper1.Execute[TResult](Func1 action)
at NuGet.DataServiceQueryWrapper1.<GetAll>d__7.MoveNext() at System.Collections.Generic.List1…ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Octopus.Core.Packages.ExternalNuGetFeedAdapter.SearchForPackagesNamedLike(String packageId, Int32 take) in Y:\work\refs\tags\3.2.13\source\Octopus.Core\Packages\ExternalNuGetFeedAdapter.cs:line 84
at Octopus.Core.Packages.ExternalNuGetFeedAdapter.GetPackagesContaining(String searchTerm, Int32& total, Int32 skip, Int32 take, Boolean allowPreRelease) in Y:\work\refs\tags\3.2.13\source\Octopus.Core\Packages\ExternalNuGetFeedAdapter.cs:line 64
at Octopus.Server.Web.Api.Actions.PackageSearchAction.Search(String feedId, String packageId, Boolean partialMatch, Boolean includeMultipleVersions, Boolean includeNotes, Boolean includePreRelease, Int32 take, IVersionSpec versionRange, Regex preReleaseTag) in Y:\work\refs\tags\3.2.13\source\Octopus.Server\Web\Api\Actions\PackageSearchAction.cs:line 75
at Octopus.Server.Web.Api.Actions.PackageSearchAction.Execute() in Y:\work\refs\tags\3.2.13\source\Octopus.Server\Web\Api\Actions\PackageSearchAction.cs:line 55
at Octopus.Server.Web.Infrastructure.Api.Responder1.Respond(TDescriptor options, NancyContext context) in Y:\work\refs\tags\3.2.13\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 162 at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2) at CallSite.Target(Closure , CallSite , Object , Object , NancyContext ) at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass0_0.<.ctor>b__0(Object o) in Y:\work\refs\tags\3.2.13\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 46 at CallSite.Target(Closure , CallSite , Func2 , Object )
at Nancy.Routing.Route.<>c__DisplayClass4.b__3(Object parameters, CancellationToken context)

octopus-external-feed.zip (243 KB)

[Michael_Noonan]

Hi!

Thanks for getting in touch. Taking a look over that error message it looks like the TeamCity server is failing to serve that query to its NuGet feed and returning a 500: Internal Server Error HTTP response. Octopus is then blindly throwing that exception which is (in my opinion) confusing. I’ve added an enhancement to our backlog for that.

Firstly, looking at the “External feed” configuration for Test build I can see you’ve got /Packages at the end of your URI. The URI should be http://teamcity.1800packrat.com/httpAuth/app/nuget/v1/FeedService.svc. See the example here.

If this doesn’t help it would be good to isolate where the problem lies in your situation so we can help get you going smoothly. Perhaps Octopus is issuing a bad request, or perhaps we are hitting a bug inside TeamCity?

1. Which version of Octopus are you using?
2. Which version of TeamCity are you using?
3. It looks like you’re issuing an empty query to TeamCity. What happens if you start typing the name of a valid package like SimpleMVC? (I’m trying to isolate whether there’s a problem with different queries)
4. What happens if you do a similar thing with nuget.exe list "SimpleMVC" -Source "http://teamcity.1800packrat.com/httpAuth/app/nuget/v1/FeedService.svc"?

Hope that helps!
Mike

[Michael_Noonan]

Hi!

I hit send a little bit early! The other thing you could try is:

5. Try using the public/unauthenticated feed from TeamCity: http://teamcity.1800packrat.com/guestAuth/app/nuget/v1/FeedService.svc

Hope that helps!
Mike

[pfinigan]

Hi Mike,

That worked. Now I have another question.

When I setup my deployment I want it to go to my TEST server. On the test server I have IIS set with 2 applications:
Temp.1800packrat.com – my SimpleMVC application
Temp-svc.1800packrat.com – my SimpleService application

My nuget feed contains the two applications:
[cid:image001.png@01D1425A.2E538390]

My tentacle on the TEST server is setup just for SimpleMVC and not the SimpleService:
[cid:image002.jpg@01D1425B.4A3305D0]

The Tentacle-SimpleMVC.config file has the following:
E:\temp.1800packrat.com

I have 2 questions:

1.   Can I use the same tentacle for both applications that should deploy to different sites and directories?
   

2.   Right now I’m just trying to get the SimpleMVC application to go into the root of my ‘temp.1800packrat.com’ folder. However, when I deploy this is what the file structure results in:
   

[cid:image005.png@01D1425B.4A28CCA0]

As you can see the release version is ending up the wrong folder. I want it to deploy directly to ‘e:\websites\temp.1800packrat.com’ instead of ‘e:\websites\temp.1800packrat.com\TEST\SimpleMVC\1.011’

Thanks,
Patrick

[Michael_Noonan]

Hi Patrick,

Thanks for getting back to me, and I’m glad you got that working OK.

I’ve read over your comments and the screen shots carefully and I think there’s a couple of fundamental concepts I can explain to help you out. Please feel free to correct me if I’m missing the point.

Take a quick read over the Octopus Key Concepts which explains how Environments, Machines, Roles and Applications all work together in the Octopus world.

1. Yes, you can use a single Tentacle for multiple applications

In your case it looks like you’ve given the Tentacle an Instance Name of SimpleMVC, but this will have no impact apart from allowing you to run multiple Tentacle instances on the same machine, which doesn’t give you much benefit

Tentacles are just an agent that runs on your target machine. You typically configure a Tentacle on a Machine, and give it Roles which determine the Application Roles that machine will host.

To answer your specific question, I would normally just install a single Tentacle on a Machine using the default instance name and add Roles to that Machine in Octopus to determine which Application Roles will be hosted on the machine.

2. Installation folders

In your case you have configured the root applications folder for the Tentacle as the location where you want the SimpleMVC website to be hosted. You should give your Tentacle a folder somewhere where it can do its work.

When Octopus deploys a package for you, it puts it in a new folder each time, like this: C:\Octopus\Applications\[Environment name]\[Package name]\[Package version]\, and executes any deployment features you configure like updating Configuration Files, or in your case deploying a Web App into IIS. In this case the deployment looks a bit like this:

1. Unpack your package into a new folder like C:\Octopus\Applications\TEST\SimpleMVC\1.011\
2. Update configuration files in that folder
3. Change the PhysicalPath of your IIS Web Site to point to C:\Octopus\Applications\TEST\SimpleMVC\1.011\ which recycles the App Pool (using overlapping rotation by default)

This is quite a nice process because it avoids a lot of problems like file locking, and ensures you aren’t hosting files that were deleted from your project, and can lead to zero-downtime deployments.

Recommendations

• I would recommend you uninstall the Tentacle on that machine and delete the Tentacle root folder, then install Tentacle again as a default instance using the default root folder.
• Then give the Machine roles at your discretion.
• Try using the default IIS deployment convention as I described earlier, but if you’re really locked into using the same folder, try the Custom Installation Directory.

Hope that helps!
Mike

[pfinigan]

Hi Mike,

Thanks for the detailed explanation and recommendations.

I’ve got it a lot closer including adding a powershell script to modify the physical path of the website after deployment. However, as you can see in the attachment, the deployment is creating a folder with the package version + “_” + the number of times the same package version has been deployed.

[cid:image001.jpg@01D144C5.18B05370]

My powershell script is using this:

Set-ItemProperty IIS:\Sites\temp.1800packrat.com -name physicalPath -value (“C:\Octopus\Applications\TEST\SimpleMVC” + $OctopusParameters[‘Octopus.Action[CW deploy].Package.NuGetPackageVersion’] + “_” + $OctopusParameters[‘Octopus.Release.CurrentForEnvironment.Number’])

Obviously I’ve attempted to determine what parameter I could use to get that added number. Is there such a thing?

Thanks again for your help. And Happy New Year.

Regards,
Patrick

[pfinigan]

Mike,

Another question:

I’ve modified the configuration in the following screen shot, however, after a deployment the release config is not being used. Am I missing something?

[cid:image003.jpg@01D14598.0BB853B0]

Thanks,
Patrick

From: Patrick Finigan
Sent: Friday, January 1, 2016 6:49 PM
To: ‘Michael Noonan’ tender2+dc4b65dd5c@tenderapp.com
Subject: RE: External Feed matching packages Internal Server Error [Problems #43585]

Hi Mike,

Thanks for the detailed explanation and recommendations.

I’ve got it a lot closer including adding a powershell script to modify the physical path of the website after deployment. However, as you can see in the attachment, the deployment is creating a folder with the package version + “_” + the number of times the same package version has been deployed.

[cid:image001.jpg@01D144C5.18B05370]

My powershell script is using this:

Set-ItemProperty IIS:\Sites\temp.1800packrat.com -name physicalPath -value (“C:\Octopus\Applications\TEST\SimpleMVC” + $OctopusParameters[‘Octopus.Action[CW deploy].Package.NuGetPackageVersion’] + “_” + $OctopusParameters[‘Octopus.Release.CurrentForEnvironment.Number’])

Obviously I’ve attempted to determine what parameter I could use to get that added number. Is there such a thing?

Thanks again for your help. And Happy New Year.

Regards,
Patrick

[Michael_Noonan]

Hi Patrick,

Thanks for getting back to me.

Config transforms

Could you please send me through the raw deployment log so I can see what might be going wrong with your config transformations?

Website deployment and custom PowerShell

I must admit, I am confused about your statement: “I’ve got it a lot closer including adding a powershell script to modify the physical path of the website after deployment.” If you just let Octopus deploy the website into IIS for you, it will update the Physical Path of the website on your behalf. I’ve got a feeling I’ve confused you somewhere during our conversation.

Typically with Octopus, when deploying a web site to IIS, you should only need a Package Step, with a few Features (like IIS Web Site and Application Pool, Configuration Transforms etc). Our customers don’t typically require much (if any) custom PowerShell in this scenario - only if they want to customize certain Application Pool settings, or do something very specific for their web site…

Perhaps the best way is to take a look over the script we use to deploy your web site in to IIS: IISWebSite_BeforePostDeploy.ps1 This way you could identify if it does what you want without the need for any Custom PowerShell.

Hope that helps!
Mike

[pfinigan]

Mike,

I’ve attached the log you mention.

As for the “Package Step”, I get it now. There were further settings inside of the package step that you mentioned which I had not enabled.

I’m getting an error in my secondary deployment of the temp-svc. You will see it in the log.

Please let me know your thoughts on the web.config use of release and the deployment error in the log.

Regards,
Patrick

ServerTasks-451.log.txt (18 KB)

[Michael_Noonan]

Hi Patrick,

Thanks for investigating that further - now it looks like you’re on the right track.

Firstly to your error, this should be easy to fix following this related post. Unfortunately we cannot easily provide a better error message and it is difficult to detect collisions in IIS.

Regarding your question about web.config use of release, I’m not quite sure what you are asking about?

I can see you are using the Config Transformation feature from these log lines, showing how you are removing the debug attribute (great idea):

10:05:12   Info     |       Transforming 'E:\TEST\SimpleMVC\1.013_2\Web.config' using 'E:\TEST\SimpleMVC\1.013_2\Web.Release.config'.
10:05:12   Verbose  |       Executing RemoveAttributes (transform line 18, 18)
10:05:12   Verbose  |       on /configuration/system.web/compilation
10:05:12   Verbose  |       Applying to 'compilation' element (source line 22, 6)
10:05:12   Verbose  |       Removed 'debug' attribute
10:05:12   Verbose  |       Removed 1 attributes
10:05:12   Verbose  |       Done executing RemoveAttributes

I can also see that we couldn’t find any matching appSettings or connectionStrings to update with variables:

10:05:12   Verbose  |       Looking for appSettings and connectionStrings in any .config files
10:05:12   Info     |       No matching setting or connection string names were found in: E:\TEST\SimpleMVC\1.013_2\Web.config
10:05:12   Info     |       No matching setting or connection string names were found in: E:\TEST\SimpleMVC\1.013_2\Web.Debug.config
10:05:12   Verbose  |       File 'E:\TEST\SimpleMVC\1.013_2\Web.Release.config' was interpreted as an XML configuration transform; variable substitution won't be attempted.
10:05:12   Info     |       No matching setting or connection string names were found in: E:\TEST\SimpleMVC\1.013_2\bin\SimpleMVC.dll.config
10:05:12   Info     |       No matching setting or connection string names were found in: E:\TEST\SimpleMVC\1.013_2\bin\roslyn\VBCSCompiler.exe.config
10:05:12   Info     |       No matching setting or connection string names were found in: E:\TEST\SimpleMVC\1.013_2\Views\Web.config

Perhaps the best next step is to look further at our documentation on configuration files.

I hope now that you’re on the path to success that our documentation will make more sense, and I’d encourage you to explore around the features available in Octopus to see what would suit your needs.

Don’t hesitate to reach out if you need anything else!

Happy Deployments!
Mike

[pfinigan]

Hi Mike,

Thanks again for the quick and detailed responses.

I’ll look at the post for the error, but wanted to tell you what I was trying to do with the web.config and see if you had suggestions. I’ve attached the 2 config files (changed to .txt). You’ll see that the release config changes the client endpoint for the system.serviceModel node:

From:

To:

I guess the real question is if Octopus is really the place to do this, or if I should be doing this in the build process in TeamCity?

Thanks again!
Patrick

Web.txt (5 KB)

Web.Release.txt (1 KB)

[Michael_Noonan]

Hi Patrick,

Thanks for getting back to me. Octopus is absolutely the right place for running configuration transforms. We have a mantra about build your binaries once, deploy many times - and on deploy is the time to configure your applications for each environment.

Take a look at our documentation on configuration files which goes into detail on the different opportunities you have for configuring your application upon deployment.

The big choice often comes down to where do you want to store your configuration data? Think sensitive details like credentials.

• Configuration Transform files are nice because they are source controlled, but they are not necessarily secure (with sensitive details stored in clear text)
• Octopus Variables are a good option because you can store sensitive details securely and control access to them, and you can source control the details securely using our data migration tooling (though it’s not a perfect solution).

Hope that helps!
Mike

[pfinigan]

Hi Michael,

We shelved the auto deploy for a few months, but I’m back on it now. I have TeamCity pulling from Git, building, pushing to Octopus, creating a release and deploying the release – a beautiful thing.

So, now the real world scenario is that we have two separate solutions that we build independently and deploy to the webserver in the same root folder, the second on top of the first. How would I be able to do this either all from TeamCity like I am now, or pushing packages to Octopus and deploying one on top of the other?

Thanks,
Patrick

[Michael_Noonan]

Hi Patrick,

It sounds like you’re having some success, great news!

To have Octopus target the exact same physical folder on the target web server you can use the Custom Installation Directory feature - however that comes with potential problems, like deploying over locked files.

My preference is to let Octopus deploy the files into a new directory and have IIS point to the appropriate physical path. This way you avoid a lot of the pitfalls of deploying into the same physical directory.

You can do this using any combination of web applications and virtual directories. If you think this is something you can do, get back to me and I’ll do my best to help.

Hope that helps!
Mike