Deploying from SVN through Jenkins to AWS Elastic Beanstalk (Part 1)

Following on from the AWS Summit last month, I figured I'd do a technical one this time round. This will fall into two parts as they are fairly detailed. But I am not going to go through to much .NET stuff and shall assume that if you follow my blog, that you can already write .NET web apps. 

Part 1: AWS Toolkit and Deploying to the Cloud

Whilst working on my current project which is going to be deployed on AWS, it became the time to look to build the release deployment process to send it to AWS. I had a choice of deployment mechanisms, including AWS CloudFormation, but for the simplicity of this environment, I plumped for deploying direct to EB.

I am running the following setup:

• VS2010 SP1
• Visual SVN (with Tortoise SVN client)
• Jenkins 1.514

Having had a little bit of experience using Go, Jenkins and TFS, I chose Jenkins for the deployment and decided to deploy to AWS Elastic Beanstalk (EB). I will assume you know how to set up and use Jenkins and have an AWS account.

What's this Beanstalk?

Elastic Beanstalk is an autoscaling platform to deploy applications into the AWS cloud. It is built on top of the EC2 PaaS setup and allows you to deploy your software, without the headache of maintaining your own Elastic Load Balancer (ELB), EC2 servers and Relational Data Services (RDS) servers etc.

How many cows do I need to sell for the magic beans to use it with Visual Studio?

None, the AWS toolkit is free. If you use the t1.micro environments for everything, you also have 750 hours of AWS use per month for the EC2 and RDS instances run by EB for free. 750 hours is almost enough to run for an entire month for nothing. Note, the minimum charging unit is per hour. Details of the free usage tier are here:

http://aws.amazon.com/free/

This lot is way above what Microsoft Azure offers customers. Not surprising given how much more mature Amazon's platform is compared to other platforms.

One thing I would advise is you consider how your account will actually use this. If you start on the free usage tier and then it kicks up a second instance, you may end up paying for the computing time of that other instance (or use your monthly free usage up faster). There are solutions around this, such as the use of reserved instances for servers that are running continuously; or setting the ELB to not scale past 1 system etc. but I won't be covering them here.

Installing the AWS for MSFT VS toolkit

In order to use the AWS platform with Visual Studio, you have to download and install the AWS Toolkit for Visual Studio. The installation will provide a combination of Visual Studio Add-ins and Command Line Interface (CLI), including the AWS Explorer shown below.

fig 1 - AWS Explorer in VS2010

It also installs context menus to build the deployment package. However, as part of the Jenkins CI process, we will be deploying partly through the CLI.

Security and Using AWS Access Keys

In order to deploy to AWS, there are a number of security steps you have to use to get passed the security of the AWS cloud.

1. Sign up for an AWS account (I shall assume you've done that)
2. Using the web based account management portion of the site, obtain the secret key and Access Key ID for the platform (in this case EB uses REST/Query API) - You will need these for the automated deployment.
3. Sign in to the AWS Management Console and set up your server as authorised to deploy to the website to AWS.

The secret key and access key are used to permit the uploading of packages to the cloud. So yes, that means you need to create a package :-) But here is where the AWS tool-kit's CLI tools actually comes in. 

fig 2 - Secret and Access Key IDs

What does AWS Explorer in VS have to do with this?

The AWS Explorer allows you to deploy applications straight from VS2010/12. It also contains the project templates which you need to use to deploy different types of project on the AWS platform. For this demo, I used the "AWS Web Project" selection, which is pretty much like a normal MS Web Project, but contains the AWSSDK.dll file. 

In the image below, you can see 5 sub-items and these are almost all sample applications to help you learn how to use AWS, or to use as a template for AWS development of your own versions of these class of project. 

fig 3 - AWS Toolkit installs new project types

So select the AWS Web Project option and build a web project like you normally would any other .NET application. In my experience, there is nothing an ASP.NET/WCF app has been able to do that it hasn't.  

Let's assume I've done that, what next?

Your easiest (maybe best) bet is to first deploy the application you have created straight from VS2010. This fulfils a number of functions:

• Allows you to create a deployment file which AWSDeploy.exe uses to deploy your application to EB
• Tests your CI server (or even PC) can connect to the AWS environment
• Makes sure your access keys are correct

fig 4 - Items added to the project context menu after AWS Toolkit is installed

After installing the AWS toolkit, you'll note that unlike VS2010, the context menu elements to deploy a project are only available at the project level, not the solution level. That means you can really only deploy one project at a time from here. I've highlighted the differences in the image above. Republishing is the just publishing step without having to enter the publishing settings again.

Publishing to the cloud eh?

Configuring the publishing process from inside VS2010 is actually pretty simple. It is a matter of using the following dialog box.

fig 5 - VS2010's AWS Deploy dialog box

The steps are pretty straightforward.

• Click the 'Add [another] account' icon next to the accounts dropdown, or select an existing  account to use for deployment. In the popup that appears, this is where the Secret Key and and Access Key ID are used from previous stages. When you have completed this box click OK.

fig 6 - Creating a new AWS account record (your existing AWS account)

• Selecting the region to deploy to. In this case, I am using the EU West region which is based in Ireland.
• For new deployments, select the "AWS Elastic Beanstalk" item form the list box. It is far too easy to attempt to click on 'Next' without doing this step, especially when it appears to be lit. So make sure it is selected. Click next and you are taken to the Application details screen.

fig 7 - Application details screen

• What this screen does is allow you to specify the name of the application you wish to deploy to EB. Take a note of the bottom checkbox. This shows that the AWS toolkit set up a local Git repository to then push changes to AWS with. 
• Clicking 'Next' then takes you to the Environment details screen. Here you should create the name of the entire EB environment. 

fig 8 - EB environment details

Important Note:  
This EB environment name is not the same as the names of any EC2 instance, any RDS instance or anything else within that EB environment. The EB environment automatically creates the names of the internal EC2, S3. RDS etc. instances whe it creates the environment. For example, the EB environment could be called AProjectSample1234 and within that, we may have an EC2 instance which it names axjkljehyhsdsds-1 and an RDS instance which it names abbdhjwiukskjkmn-1. Remember, you are deploying to an IaaS environment which is designed to autoscale. You will have automatically been assigned an ELB and at least one EC2 instance, which can scale if your processing needs scale. Hence, the underlying PaaS mechanism is transparent to you, though you can examine the instances from the usual AWS Management Console of the AWS Explorer in VS.

• Select a URL to use as the main environment. Note, this URL sub-domain is linked to your ELB not your EC2 instance, which is what you configure on the next screen in the Wizard.
• The AWS Options screen is where you configure your Amazon Machine Image (AMI). A container is the image that you use to contain your application. In this case, a web app. Given it is Windows, you have the choice of 64-bit Win2012 or 2008 R2 OS platforms, with IIS8 or 7.5 respectively. The free tier is the Micro tier (which corresponds to the t1.micro tier in AWS). If you have configured an AMI if your own in Amazon's cloud, you can select it here too. 

fig 9 - Selecting the EC2 config

• The next screen after that is the Application Options screen. This is where you can configure the .NET Framework version the app pool uses; a health-check page (think heartbeat monitor in IIS for ASP.NET) as well as the security credentials to log in with (I definitely recommend you get some :-)

fig 10 - Setting up the type of container.

• If you have an RDS instance associated with your project, the screen in fig 11 allows you to configure the security groups associated with your RDS instance. Note that to communicate between the EC2 and RDS instances, they both have to exist in the same security group. This is to save you having to deal with Windows Authentication, especially if the servers are going to scale and the load balanced, since the same authentication accounts would not necessarily be available. If you have no RDS servers, then the checkboxes can be left alone and you can click 'Next'. You can always change this later and add an RDS instance from the AWS Management console > EB or AWS Explorer in VS.

fig 11 - Security group config for RDS

• As far as we are concerned for this blog post, all those steps were really to get to the following review screen. The review screen has a critical checkbox at the bottom with the label "Generate AWSDeploy configuration". For the Jenkins deployment, this is the key screen. You need to check the box and enter the location of where you want to save your AWS configuration file. I definitely recommend you generate it in your SVN workspace, since you will have to commit it to SVN, for Jenkins to then check out into its workspace and subsequently run AWS with this configuration. 

fig 12 - AWS configuration options highlighted in the review wizard panel

When you're done, hit the 'deploy' button. The wizard will then attempt to deploy your application to AWS EB, using the supplied account credentials and .NET framework onto the elasticbeanstalk.com sub-domain you specified.  This takes some time, but with any luck, you should find it deploys successfully. If you go to the sub-domain you selected in a few minutes (for example http://AProjectSample1234.elasticbeanstalk.com), you should have your site at that address. You can monitor the site through the AWS management console or through the AWS Explorer in Visual Studio.

What's in the AWS Deploy configuration  file?

When the file has been created, opening it shows us the following structure (yes I have changed my keys, you don't think I am showing you that do you? :-)

# For detailed explanation of how these config files should be used and created please see the developer guide here:

# http://docs.amazonwebservices.com/AWSToolkitVS/latest/UserGuide/tkv-deployment-tool.html

# Edit the parameter line below to set the path to the deployment archive or use

# /DDeploymentPackage=value

# on the awsdeploy.exe command line for more flexibility.

# DeploymentPackage = <-- path to web deployment archive -->

# Instead of embedding the AWSAccessKey and AWSSecretKey to be used to deploy

# artifacts we recommend that you consider using the /DAWSAccessKey and

# /DAWSSecretKey command line parameter overrides.

AWSAccessKey = XXXXXXXXXX11223XXXXX

AWSSecretKey = alkHd1PO3lakjsdka/asdlas/1+djasdjl/zu/lkjaslkdas

Region = eu-west-1

SolutionStack = 64bit Windows Server 2008 R2 running IIS 7.5

Template = ElasticBeanstalk

aws:elasticbeanstalk:application:environment.AWS_ACCESS_KEY_ID = XXXXXXXXXX11223XXXXX

aws:elasticbeanstalk:application:environment.AWS_SECRET_KEY = alkHd1PO3lakjsdka/asdlas/1+djasdjl/zu/lkjaslkdas

Application.Description = Sandbox

Application.Name = AProject

Container.Enable32BitApplications = False

Container.InstanceType = t1.micro

Container.TargetRuntime = 4.0

Environment.CNAME = AProject

Environment.Description = Sandbox environment

Environment.Name = AProject

All the details you specified in the wizard are stored here. This file is passed to AWSDeploy.exe on the command line or in our case, through Jenkins. The comments give you some very useful tips. Decisions on the keys will be especially important. If you are deploying to different environments or platforms on client's private clouds for example (over VPC for example), you wouldn't want to use your dev keys to do that. So you would pass them in on the command line, but always take care to store them appropriately (including thinking about an encryption step in the build process).

What next?

The next blog will show you how how to link this to Jenkins and built and deployed the project to AWS. This will focus on configuring the Jenkins job to use MSBuild to build the package and AWSDeploy to deploy it to the elastic beanstalk URL we've already specified here.