Fail: AWS EFS (Preview) on AWS EC2 Windows Instance

Gah! :( Poor show this morning.

Was hoping to write up a disk performance comparison of AWS EFS against EBS on Windows. Alas it wasn't to be. I am still writing this failure up as it may be useful to someone, but this story doesn't have a happy ending.

TL;DR; AWS EFS and Windows Server 2012+ are very incompatible due to lack of NFSv4 compatibility

I was aware of the limited NFS version support by AWS (i.e. only 4, whilst Windows only supports 2, 3 and 4.1). Yet, I wanted to see what could be done. Plus, it all works fine in Linux. Let's walk through it.

AWS Elastic File System

After the AWS Summit this year, which I was disappointed I couldn't attend, I was lucky enough to attend a replay at Amazon's offices at the beginning of June in London. During it, I got to hear about the Elastic File System preview. It was billed as the missing link in the AWS cloud storage offering, allowed flexible storage options to be delivered and charged without having to re-provision storage or manually add another NAS drive.

Creating and mounting the EFS storage instance in Linux is easy enough. It's still in preview in the US-West region (Oregon).

1. Walk through the EFS wizard to create your EC2 instance. Amazon recommend creating this across all the availability zones in a region. You certainly don't have to, especially as they are charged at 30 cents (roughly 18.75p) per GB-month.

AWS Elastic File System (EFS)

2. Set the security groups on mount points. It's important to note that the security groups will manage the connection between the EFS mount point and the EC2 instance you'll create.

Adding Mount point security groups

3. Spin the shizzle! Note, the full creation process can take about 5 minutes before finally marking the EFS volume as available.

When created, it's still spinning up

Creating the mount points

4. Add the NFS port (2049) to the security group in the usual manner.

5. It was at the point of spinning up and logging in to a Windows EC2 instance that the process went wrong. You could reach the mount point through the IP address. You could also use the 'net use' command to access the NFS volume using the IP address. However, despite the picture below, you couldn't write anything to it. Hence, I couldn't really test it, let along run the performance tests on it.

All the utilities, excited and ready to run :)

6. In order to test whether it was me, I decided to spin up a Linux instance to try to access the NFS volume. I'd do this to check if I could access it from there, create a file and read it on the Windows server. The first part of that went without a hitch.

1. Spin up the EC2 instance
2. Add the EC2 Instance's security group to the EFS mount points' security group (The screen is the same as shown above) 
3. Create a private key using puttyGen from 
4. Access it using the AWS Linux ec2-user via Putty
5. Install the NFSv4 client package into the AWS Linux instance using yum via an elevated command > "sudo yum install nfs-utils"
6. Create directory to mount it to
7. Mount the file system onto that directory using "mount -t nfs4 <mount>:/ /<directory>" 

Voila! Perfect!

connecting to AWS Linux instance via Putty

NFS4 installed using Yum

The only other thing I did was chmod the directory to get access (you can take ownership of it as well) and I touched a file into the directory and sure enough...

Windows fail! :(

So side-by-side. You can see Linux and Windows don't have the same access. Refreshing or reconnecting makes no difference in Windows. I changed access to the file to everyone and still no luck. However, Linux had absolutely no issues. Just for fun, I removed the security group access and tested the mounting again and it worked fine to block access.

When you've not set security group on EFS mount points

Conclusion

At the time of writing (08/2015) this issue arises from the annoying lack of [free] support from Microsoft for NFSv4 and the choice by AWS to only support NFSv4 (the only version Windows doesn't support). Here I have to be honest, I don't know which is worse. Whilst there is the possibility this could be shared across a Samba connection, you've got to then run an EC2 instance just to do that and this will increase latency. I'd definitely say that this product isn't ready to take share from Microsoft in this arena in the enterprise, though of course it works on Linux based environment perfectly well. 

The product will continue to mature I'm sure. However, I'll have to put my Windows performance investigation on hold for now sadly.

SAMPLE: Azure v AWS - Judging Trade-Offs.

Judging cloud platforms is one of the things I find myself doing a lot these days. Working mainly but not exclusively on the Microsoft stack, this generally boils down to two main options. AWS or Azure.

Now, personally, I go for AWS by default. However, for various reasons, I refuse to tie myself to any one vendor. Plus, it allow an effective, vendor neutral position to be taken and for those that know me, it cuts straight through sales cr*p to see if what vendors are saying actually matches their promises (in the main a lot don't). I tend to do this in conjunction with the organisation procuring, since it's important not just to check that vendor systems work, but that it works in context. This is especially the case when organisations are aiming to become more agile, since they will have a much closer working relationship with vendors than most vendors may feel comfortable with. So it is another tool in the toolbox to help evaluate how the line of business as a whole (business, data, application, technology, support and security) works.

How Do I Evaluate the Difference for Stories?

Trade-off analysis doesn't start with this question, but with a previous question, which is "What is it I want to achieve?" since this then leads to the all important question "What question(s) do I need to ask to evaluate vendors?" and there may be multiple ones you need answers for.

Throughout this short blog, let's use the example goal:

"Given I have to host a new room booking platform,
 I want the highest on-demand available infrastructure for the lowest monthly cost 
 So that I can extend the application in the most cost effective way"

Once we've understood the value of terms like 'cost effective', we can now look at what the availability needs are.

Let's use Microsoft Azure's own infrastructure diagrams for this. Attached is a snip of a Microsoft Blueprint for Azure hosted infrastructure.

fig 1 - Microsoft Blueprint

Comparing OnDemand costs is simply a matter of adding up all the costs for the components of network, data-store and VM for similarly matched specifications. Comparing the market price of Azure and AWS components, we see:

fig 2 - AWS v Azure Platform On-Demand Pricing

So that's the price... and AWS is cheaper... for 'bigger' hardware (same pricing tier, though did the story contain anything about application hardware specs?). Still, it's one of the two variables you need to determine cost-effectiveness. The other variable is availability guarantee.

Measuring Availability

Using the same techniques found here, it looks like it gets worse for Microsoft when looking at systemic availability. 

Azure: 99.9996%

AWS:  99.9999 %

Note, systemic availability is actually the important thing in every platform. The availability of individual components is next to no use to you as an enterprise. It only takes one component to fail irrevocably and your platform is done. 

Heard the old adage "You're only as strong as your weakest link?" When thinking systemically, such effects are a lot worse than your weakest link, since you can never make up for one weakness without impacting other elements. This is one of the reasons we host on two different, load balanced servers. Since for one single application-services-data stack on 3 VMs, each with 99% availability, we can only have an expectation of 98.01% uptime in total.

Summary and Future Posts

I started writing up comparisons for Reserved and Up-front pricing on AWS and Azure and felt the original post getting too long, even for me. So I've split it into a couple of posts to launch bit-by-bit.

The crux of all of these is to always know the question you're trying to answer. It's not a matter of boiling the ocean on day-1. After all, that's the promise of cloud. You can scale the frying pan later. 

Also, don't forget that you have a number of other options to bring these costs down. MSDN subscriptions and BizSpark give you varying levels of Azure Cloud credits and AWS gives you 'free-tier' infrastructure for 12 months which might cover your needs entirely. So you have to consider a more holistic approach to understanding your options and constraints, since the latter is your job, not the vendors.

AWS Cloud Summit 2014

I have started writing this before the start of AWS' Cloud Summit at the ExCel in London. The AWS Summit is usually a good event and as far as the big cloud players are concerned, is my favourite of the events. There is good attention to detail, lots of vendors, free food and coffee, which is always a major bonus. You get the chance to talk to some of the AWS staff and solution architects, which is always a huge bonus. This isn't going to be a live blog, as you can follow me @EtharUK for that, but at the same time, it will allow me to record some thoughts in more detail as the day progresses.

As it happens, I was at an AWS Bootstrap workshop yesterday and I am trying to hunt down a solution architect who can answer a question I had from that, Sabastian the trainer couldn't answer at the time. Best go find such a target...

Keynote

I was worried that I wouldn't be able to sit through the keynote as it was going to be two hours long. However, Steve Schmidt's presentation was actually quite informative.

Carlos from Just-Eat showing how AWS has facilitated their agility

Carlos from Just Eat was here again, but Channel 4 put in an appearance, explaining how data drives their advertising decisions, which in turn have led to an 8-fold increase in benefit to customers.

The current AWS estate

The top-level AWS service catalogue hasn't seemingly changed much from last time. That said, the number of services, which include addtios to service offerings, which I would have liked to hear more about, has increased dramatically. So much so, they didn't fit on a standard linear graph, which could have had a much stronger impact. Oh well, they're techies :o)

Schmidt showing us the mis-scaled pace of innovaiion ;)

The one thing we heard a lot more of was governance. Whilst AWS provides enough tools to deliver a reasonable level of governance, there was much more a hint around how it was being used in the enterprise sphere.



There has recently been a lot of noise about hybrid-cloud solutions. Steve Schmidt spend a little bit of time mentioning the interoperability of cloud and on premise solutions. In my experience, there is a lot more to it.

My epic fail attempt to win a Kindle

What's the answer to my question?

Yesterday I was at an AWS workshop. In it, Sabastian the trainer, mentioned that records deployed to an AWS availability zone, with a slave availability zone, would synchronously commit changes on the master and the slave. It doesn't commit this to the master until it has been committed to the slave. This gave me some cause for concern.

The aim is if the master availability zone goes down, the slave doesn't goes down. This is sensible form the point of view of the master. However, if the slave availability goes down and the master doesn't commit unless the slave also commits, then unless there is a way for RDS to commit this to the master, you have an issue. Indeed, this situation could technically result in lower availability than running single instances of the DB.

Why?

Firstly, I am pretty sure that this is DB instance independent. Assuming you have a synchronous process that requires two activities to complete successfully. However, I have not seen or heard evidence to suggest this is a two-phase commit process. So to illustrate the issue, an example might be useful.

Supposing the two cloned platforms across the two availability zones each have a 99.95% availability. For a master-slave configuration where the commit of the master is dependent on the slave, this introduces a dependency chain and means that the whole uptime of your entire platform requires both services to be up in certain configurations. The result is this reduces the availability to about 99.90% (i.e. the probability of both systems being up). This is lower than any single server and certainly lower than systems running independently in parallel.

This doesn't mean that it is a problem. After all, you can architect to remove this risk and hence increase the availability of the data sources as a whole. However, I put this to our trainer yesterday and he said he'd go away and ask. Hence, I didn't receive an answer at the time.

I spoke to a solution architect this morning and he too didn't have an answer. So it would be good to get one. I am not too bothered whether it is positive or negative, but it would dictate the complexity of a system design and also provide a theoretical constraint to conduct trade-offs around . Known-unknowns can be troublesome, especially if you've only just discovered it, since it was an unknown-unknown before. I must get round to chasing this up.

**** UPDATE: After chasing this up, it appears that there isn't currently any documentation to corroborate the assertion from a few other SAs that the platform would prevent the saving of data in the event of an AZ failure. However, this also doesn't tell me if it wouldn't. I've had my details taken but no sticker given ;) ****

400 - EBS and EC2 Optimisation

This was a 400 track. There were some extremely useful slides in this track. AWS went through an intro explaining that EBS is basically a storage mechanism with a queue attached and is not like a normal disk. I still think it kind of is, when you include the buffers and caches. Both standard EBS and EBS PIOPS (Provisioned IOPS) were introduced and in the latter case, we briefly touched on the configuration of the IOPS provision.

However, importantly for me, the existence of a formal queue defines a specific need to understand the block size per IOP, as this can significantly affect the throughput of the system. The bigger the ECS instance, the more you can write (specifically, the faster you can write the data), the bigger the EBS queue, the faster it can write the standard 16K blocks.

This suggests that the best way to write the data to disk is to chunk them up in 16K blocks (or multiples thereof) and write them in parallel, which was suggested in yesterday's workshop.

200 - Hybrid Environments with AWS

This was an interesting track, however most of this is pretty standard. For indeed, some of my clients have done this for a while. I have a much greater appreciation of this via some of the security group work I've done since last year. So I am liking the way that hybrid and cloud solutions can work together. There didn't seem to be too much that was new though.

Hybrid Environments - Yes, I was a bit late to this one :-S

300 - Building for availability and cost

Fitz, a solution architect at Amazon presented Here.com's autoscaling solution. Through all Autoscaling demos at this conference, the mantra "scale up fast, scale down slow" was repeated. This is because it takes little time to prevent an AWS EC2 instance from receiving traffic, but it can take an age for it to get to a position to receive traffic. So that makes sense.

End of Day

Not a bad summit. I don't think I will take away as much from this as I did last year... aside from that my weight isn't appropriate for perspex chairs (Sorry Amazon). Amazon always put on a very good show. I'm sat here with a beer whilst I prep to tackle the tube 'struck' TFL public transport system before getting my train to Manchester. There is a lot to take away and I'll have to let that lot ferment as much as the beer before brewing up a new vat of ideas for the future of my architecture work with the new tools AWS provide. I am still to be convinced of the some of them, such as the need for schedule based autoscaling, which I see as a way to circumvent the 15 or 20 minute spin up of a new platform. However, they do solve some problems so are not at all without purpose. Especially in warming up environments for immediate use. 

Additionally, the EBS optimisation session has set off a few ideas around using queuing theory to try to explain some of the numbers Amazon have found in their testing. One thing that appeared time and time again was the experience of other speakers, a large proportion of whom spent a lot of time and effort creating PoC platforms to prove the viability of AWS.

Deploying from SVN through Jenkins to AWS Elastic Beanstalk (Part 2): Linking Jenkins to AWS EB

Following on from Part 1 of the blog on integrating Jenkins with AWS EB, this concludes the two part series by focussing on the AWS deployment process within Jenkins. At the end we will briefly outline the limitations of AWSDeploy to EB and what you would need to look to if your needs are more complex.

So what does Jenkins look like?

Setting up Jenkins will basically require executing the following the steps:

1. Create a 'New Job' in Jenkins 
2. Fill in the URL of  your SVN repository
3. Decide on Jenkins' polling frequency for SVN
4. Fill in the batch process details, including MSBuild.exe and AWSDeploy.exe CLI, with appropriate parameters
5. Save the job

Step 1: Create a 'New Job' in Jenkins

Nice and easy, from the main Jenkins page, click "New Job" on the left hand menu in Jenkins.

When the new job page loads, fill in the details of the job you with to create. You'll want to create a free-style software project. 

I Normally prefix the job name with 'RELEASE-' but for this demo, I am being a bit slapdash. When you are happy with it, click the OK button.

fig 1 - Creating a 'New Job' in Jenkins (click to enlarge)

Step 2: Configure the Job

Jenkins then moves to the job configuration page which allows you to set up the SCM (in this case SVN), the Build Steps and Post Build Actions.

Hence:

1. Fill in a display name under "Advanced Project Options" if you want it

2. Under "Source Code Management" select the Subversion radio button and enter your repository URL. Jenkins will poll your SCM and if you need to enter SVN credentials, a validation help link appears and you can click to enter your SVN credentials within that.

fig 2 - Links to enter your server's credentials. This server doesn't exist - just showing the link

3. Select your method of authentication. In my case, I chose Username/Password and then enter your credentials. Important note, the site runs unsecured by default. Once you have entered the credentials, click the OK button.

fig 3 - Subversion authentication in Jenkins (click to enlarge)

4. Complete the rest of Jenkins' SVN config. I always check out a fresh copy so I don't have previous builds laying around and potentially making a mess of the build. Note, you can also check out multiple URL's if you have dependent projects. This can of course, also be done through an SVN 'extern' declaration.

5. Build Triggers - This is an important feature that allows you to configure what sets off the build. For example, if you have a dependency chain of projects/solutions which are already set up as Jenkins jobs, you can choose which project(s) need to complete before this job runs. Projects are separated by commas.

Here you should select the "Poll SCM" step. This then brings up a text box where you can enter the polling frequency in a CRON like language. Jenkins polls your SCM (SVN here) and stores the latest revision that it builds. If there is a change in that revision number on a poll, it then starts a checkout and build process.  

Alternatively you can choose to build periodically, or combine the two.

6. The build steps - This is probably the key elements when it comes to deploying to AWS. For simplicity, I have put all the steps in this demo into one build step so that it completes the steps in one go. However, you would normally want to split them out into multiple steps, such as 'Build', 'Test', 'Deploy' so you can stop at any point. You can pick up the build artefacts between the steps by using the same workspace environment variables.

Putting it all together still works, as long as an errorcode is returned, but it isn't neat.

Clicking on the "Add build step" button open up a text area when you can enter a list of  Windows console commands. In my case, they are:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe "%WORKSPACE%\AProject.sln" /p:Configuration=Release /p:Platform="Any CPU" /m
cd "%WORKSPACE%\AProject"
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe /t:Package /p:Configuration=Release
"C:\Program Files (x86)\AWS Tools\Deployment Tool\awsdeploy.exe" /w /r /v /DDeploymentPackage="%WORKSPACE%\AProject\obj\Release\Package\AProject.zip" "%WORKSPACE%\AProject\AWSDeploy.txt"

You will notice the %WORKSPACE% environment variable in this series of commands. This is the Jenkins workspace that the code has been checked out to. There are a number of environment variable and they are: 

The following variables are available to shell scripts

BUILD_NUMBER

The current build number, such as "153"

BUILD_ID

The current build id, such as "2005-08-22_23-59-59" (YYYY-MM-DD_hh-mm-ss)

BUILD_DISPLAY_NAME

The display name of the current build, which is something like "#153" by default.

JOB_NAME

Name of the project of this build, such as "foo" or "foo/bar"

BUILD_TAG

String of "jenkins-${JOB_NAME}-${BUILD_NUMBER}". Convenient to put into a resource file, a jar file, etc for easier identification.

EXECUTOR_NUMBER

The unique number that identifies the current executor (among executors of the same machine) that's carrying out this build. This is the number you see in the "build executor status", except that the number starts from 0, not 1.

NODE_NAME

Name of the slave if the build is on a slave, or "master" if run on master

NODE_LABELS

Whitespace-separated list of labels that the node is assigned.

WORKSPACE

The absolute path of the directory assigned to the build as a workspace.

JENKINS_HOME

The absolute path of the directory assigned on the master node for Jenkins to store data.

JENKINS_URL

Full URL of Jenkins, like http://server:port/jenkins/

BUILD_URL

Full URL of this build, like http://server:port/jenkins/job/foo/15/

JOB_URL

Full URL of this job, like http://server:port/jenkins/job/foo/

SVN_REVISION

Subversion revision number that's currently checked out to the workspace, such as "12345"

SVN_URL

Subversion URL that's currently checked out to the workspace.

The batch commands perform the following functions.

The 1st command builds the solution containing the project, using the release configuration. In a normal project I at least use the web.Debug.config and web.Release.config files with the MS Build match and replace mark-up into the main web.config file. A tutorial blog can be found here.

The 2nd command enters into the directory of the built project. This is just so we can run the package verb for MSBuild.

The 3rd builds a standard deployment package out of the project.

The 4th line is the AWS Deployment line. This actually carries out the deployment of the package created in batch step 3 previously using the AWSDeploy.txt file we created from within Visual Studio.

Post-build actions
A post build action in Jenkins is an event that takes place after the build steps. They occur regardless of whether or not the build has completed successfully and encompass one or more of the steps shown in the screengrab below.

fig 4 - The Jenkins post-build actions (click to enlarge).

In email post-build steps, Jenkins allows users to be sent emails for unstable builds (it builds but tests fail), build failures and send separate emails to the people who broke the build.

You'll notice form the screenshot that there is the option of publishing JUnit tests. Obviously, MSTest results are not exactly in the same format. However, you can run a build step to convert MSTest's TRX to HTML files using tools such as the MS Test Report Generator. There is a similar process for NUnit results.

Recommendations:
Don't forget to set up e-mail alerts within Jenkins for job events as a "Post-build action". You can also tie events to an RSS feed. In either case, you have the option of using them for failed builds and all builds. Plus the email benefits identified above. Note though, Jenkins doesn't automatically set up an (or your) email server. You have to have a mail service/daemon running to send emails.

Also, I definitely recommend post-build steps to publish your test results. This may include a step to post to your build monitor if you have one. Maybe naming and shaming the individual breaking the build... not that you want a harassment suit on your company's hands of course ;-)

Step 3: Run the darn thing!

When you check in some code, at the intervals specified for the Jenkins cron, Jenkins will poll SVN, clean checkout the code, build, test and deploy the code. Any failure will cause the build history to include the usual red light.

fig 5 - Build history, including failures 

You can of course, run the build using the "Build Now" menu link in the top left menu to kick it off manually. In either case, there is a significant increased delay over any local process, but that is to be expected since the deployment is going over the wire and EB is incrementally building the environment (it would be longer first time out. I've had to wait over 10 minutes sometimes. However, )

The following is the example deployed to AWS using the above process. As well as using AWS Elastic Beanstalk, for my own work, I have a route 53 entry to the domain name and a back end DB.

fig 6 - Deployed EB example site (click to enlarge)

Conclusion

Jenkins has always been a really easy to use system for CI. I've found it easier to use than ThoughtWorks Go. However, both Go and TFS give you more customisation options out of the box. You can certainly expand Jenkins using custom plugins. Obviously, there is quite a bit more to do to set up a CI system from the above. But it gives you a framework to modify and work with. 

Also, the use of the AWSDeploy.exe makes it really easy to deploy fairly simple environments. 

Limitations

When you get up to the level of having to manage large, custom architectures and deployments, where detailed configuration and set-up of multiple AWS resources are requires with EC2, ELB, SQS, S3, SWF, RedShift and big data instances for example, a basic AWSDeploy text file won't cut it any more. 

At that point, you would need to consider moving from just using an EB setup to using CloudFormation to set up a more complex cloud architecture. As it happens, EB does basically generate a JSON Chef  Cookbook which is used as the CloudFomation template. You can see this if you go to the ClouseFormation stacks page in the AWS Console. I suspect there will be times when you'll need to roll your own.

Deploying from SVN through Jenkins to AWS Elastic Beanstalk (Part 1)

Following on from the AWS Summit last month, I figured I'd do a technical one this time round. This will fall into two parts as they are fairly detailed. But I am not going to go through to much .NET stuff and shall assume that if you follow my blog, that you can already write .NET web apps. 

Part 1: AWS Toolkit and Deploying to the Cloud

Whilst working on my current project which is going to be deployed on AWS, it became the time to look to build the release deployment process to send it to AWS. I had a choice of deployment mechanisms, including AWS CloudFormation, but for the simplicity of this environment, I plumped for deploying direct to EB.

I am running the following setup:

• VS2010 SP1
• Visual SVN (with Tortoise SVN client)
• Jenkins 1.514

Having had a little bit of experience using Go, Jenkins and TFS, I chose Jenkins for the deployment and decided to deploy to AWS Elastic Beanstalk (EB). I will assume you know how to set up and use Jenkins and have an AWS account.

What's this Beanstalk?

Elastic Beanstalk is an autoscaling platform to deploy applications into the AWS cloud. It is built on top of the EC2 PaaS setup and allows you to deploy your software, without the headache of maintaining your own Elastic Load Balancer (ELB), EC2 servers and Relational Data Services (RDS) servers etc.

How many cows do I need to sell for the magic beans to use it with Visual Studio?

None, the AWS toolkit is free. If you use the t1.micro environments for everything, you also have 750 hours of AWS use per month for the EC2 and RDS instances run by EB for free. 750 hours is almost enough to run for an entire month for nothing. Note, the minimum charging unit is per hour. Details of the free usage tier are here:

http://aws.amazon.com/free/

This lot is way above what Microsoft Azure offers customers. Not surprising given how much more mature Amazon's platform is compared to other platforms.

One thing I would advise is you consider how your account will actually use this. If you start on the free usage tier and then it kicks up a second instance, you may end up paying for the computing time of that other instance (or use your monthly free usage up faster). There are solutions around this, such as the use of reserved instances for servers that are running continuously; or setting the ELB to not scale past 1 system etc. but I won't be covering them here.

Installing the AWS for MSFT VS toolkit

In order to use the AWS platform with Visual Studio, you have to download and install the AWS Toolkit for Visual Studio. The installation will provide a combination of Visual Studio Add-ins and Command Line Interface (CLI), including the AWS Explorer shown below.

fig 1 - AWS Explorer in VS2010

It also installs context menus to build the deployment package. However, as part of the Jenkins CI process, we will be deploying partly through the CLI.

Security and Using AWS Access Keys

In order to deploy to AWS, there are a number of security steps you have to use to get passed the security of the AWS cloud.

1. Sign up for an AWS account (I shall assume you've done that)
2. Using the web based account management portion of the site, obtain the secret key and Access Key ID for the platform (in this case EB uses REST/Query API) - You will need these for the automated deployment.
3. Sign in to the AWS Management Console and set up your server as authorised to deploy to the website to AWS.

The secret key and access key are used to permit the uploading of packages to the cloud. So yes, that means you need to create a package :-) But here is where the AWS tool-kit's CLI tools actually comes in. 

fig 2 - Secret and Access Key IDs

What does AWS Explorer in VS have to do with this?

The AWS Explorer allows you to deploy applications straight from VS2010/12. It also contains the project templates which you need to use to deploy different types of project on the AWS platform. For this demo, I used the "AWS Web Project" selection, which is pretty much like a normal MS Web Project, but contains the AWSSDK.dll file. 

In the image below, you can see 5 sub-items and these are almost all sample applications to help you learn how to use AWS, or to use as a template for AWS development of your own versions of these class of project. 

fig 3 - AWS Toolkit installs new project types

So select the AWS Web Project option and build a web project like you normally would any other .NET application. In my experience, there is nothing an ASP.NET/WCF app has been able to do that it hasn't.  

Let's assume I've done that, what next?

Your easiest (maybe best) bet is to first deploy the application you have created straight from VS2010. This fulfils a number of functions:

• Allows you to create a deployment file which AWSDeploy.exe uses to deploy your application to EB
• Tests your CI server (or even PC) can connect to the AWS environment
• Makes sure your access keys are correct

fig 4 - Items added to the project context menu after AWS Toolkit is installed

After installing the AWS toolkit, you'll note that unlike VS2010, the context menu elements to deploy a project are only available at the project level, not the solution level. That means you can really only deploy one project at a time from here. I've highlighted the differences in the image above. Republishing is the just publishing step without having to enter the publishing settings again.

Publishing to the cloud eh?

Configuring the publishing process from inside VS2010 is actually pretty simple. It is a matter of using the following dialog box.

fig 5 - VS2010's AWS Deploy dialog box

The steps are pretty straightforward.

• Click the 'Add [another] account' icon next to the accounts dropdown, or select an existing  account to use for deployment. In the popup that appears, this is where the Secret Key and and Access Key ID are used from previous stages. When you have completed this box click OK.

fig 6 - Creating a new AWS account record (your existing AWS account)

• Selecting the region to deploy to. In this case, I am using the EU West region which is based in Ireland.
• For new deployments, select the "AWS Elastic Beanstalk" item form the list box. It is far too easy to attempt to click on 'Next' without doing this step, especially when it appears to be lit. So make sure it is selected. Click next and you are taken to the Application details screen.

fig 7 - Application details screen

• What this screen does is allow you to specify the name of the application you wish to deploy to EB. Take a note of the bottom checkbox. This shows that the AWS toolkit set up a local Git repository to then push changes to AWS with. 
• Clicking 'Next' then takes you to the Environment details screen. Here you should create the name of the entire EB environment. 

fig 8 - EB environment details

Important Note:  
This EB environment name is not the same as the names of any EC2 instance, any RDS instance or anything else within that EB environment. The EB environment automatically creates the names of the internal EC2, S3. RDS etc. instances whe it creates the environment. For example, the EB environment could be called AProjectSample1234 and within that, we may have an EC2 instance which it names axjkljehyhsdsds-1 and an RDS instance which it names abbdhjwiukskjkmn-1. Remember, you are deploying to an IaaS environment which is designed to autoscale. You will have automatically been assigned an ELB and at least one EC2 instance, which can scale if your processing needs scale. Hence, the underlying PaaS mechanism is transparent to you, though you can examine the instances from the usual AWS Management Console of the AWS Explorer in VS.

• Select a URL to use as the main environment. Note, this URL sub-domain is linked to your ELB not your EC2 instance, which is what you configure on the next screen in the Wizard.
• The AWS Options screen is where you configure your Amazon Machine Image (AMI). A container is the image that you use to contain your application. In this case, a web app. Given it is Windows, you have the choice of 64-bit Win2012 or 2008 R2 OS platforms, with IIS8 or 7.5 respectively. The free tier is the Micro tier (which corresponds to the t1.micro tier in AWS). If you have configured an AMI if your own in Amazon's cloud, you can select it here too. 

fig 9 - Selecting the EC2 config

• The next screen after that is the Application Options screen. This is where you can configure the .NET Framework version the app pool uses; a health-check page (think heartbeat monitor in IIS for ASP.NET) as well as the security credentials to log in with (I definitely recommend you get some :-)

fig 10 - Setting up the type of container.

• If you have an RDS instance associated with your project, the screen in fig 11 allows you to configure the security groups associated with your RDS instance. Note that to communicate between the EC2 and RDS instances, they both have to exist in the same security group. This is to save you having to deal with Windows Authentication, especially if the servers are going to scale and the load balanced, since the same authentication accounts would not necessarily be available. If you have no RDS servers, then the checkboxes can be left alone and you can click 'Next'. You can always change this later and add an RDS instance from the AWS Management console > EB or AWS Explorer in VS.

fig 11 - Security group config for RDS

• As far as we are concerned for this blog post, all those steps were really to get to the following review screen. The review screen has a critical checkbox at the bottom with the label "Generate AWSDeploy configuration". For the Jenkins deployment, this is the key screen. You need to check the box and enter the location of where you want to save your AWS configuration file. I definitely recommend you generate it in your SVN workspace, since you will have to commit it to SVN, for Jenkins to then check out into its workspace and subsequently run AWS with this configuration. 

fig 12 - AWS configuration options highlighted in the review wizard panel

When you're done, hit the 'deploy' button. The wizard will then attempt to deploy your application to AWS EB, using the supplied account credentials and .NET framework onto the elasticbeanstalk.com sub-domain you specified.  This takes some time, but with any luck, you should find it deploys successfully. If you go to the sub-domain you selected in a few minutes (for example http://AProjectSample1234.elasticbeanstalk.com), you should have your site at that address. You can monitor the site through the AWS management console or through the AWS Explorer in Visual Studio.

What's in the AWS Deploy configuration  file?

When the file has been created, opening it shows us the following structure (yes I have changed my keys, you don't think I am showing you that do you? :-)

# For detailed explanation of how these config files should be used and created please see the developer guide here:

# http://docs.amazonwebservices.com/AWSToolkitVS/latest/UserGuide/tkv-deployment-tool.html

# Edit the parameter line below to set the path to the deployment archive or use

# /DDeploymentPackage=value

# on the awsdeploy.exe command line for more flexibility.

# DeploymentPackage = <-- path to web deployment archive -->

# Instead of embedding the AWSAccessKey and AWSSecretKey to be used to deploy

# artifacts we recommend that you consider using the /DAWSAccessKey and

# /DAWSSecretKey command line parameter overrides.

AWSAccessKey = XXXXXXXXXX11223XXXXX

AWSSecretKey = alkHd1PO3lakjsdka/asdlas/1+djasdjl/zu/lkjaslkdas

Region = eu-west-1

SolutionStack = 64bit Windows Server 2008 R2 running IIS 7.5

Template = ElasticBeanstalk

aws:elasticbeanstalk:application:environment.AWS_ACCESS_KEY_ID = XXXXXXXXXX11223XXXXX

aws:elasticbeanstalk:application:environment.AWS_SECRET_KEY = alkHd1PO3lakjsdka/asdlas/1+djasdjl/zu/lkjaslkdas

Application.Description = Sandbox

Application.Name = AProject

Container.Enable32BitApplications = False

Container.InstanceType = t1.micro

Container.TargetRuntime = 4.0

Environment.CNAME = AProject

Environment.Description = Sandbox environment

Environment.Name = AProject

All the details you specified in the wizard are stored here. This file is passed to AWSDeploy.exe on the command line or in our case, through Jenkins. The comments give you some very useful tips. Decisions on the keys will be especially important. If you are deploying to different environments or platforms on client's private clouds for example (over VPC for example), you wouldn't want to use your dev keys to do that. So you would pass them in on the command line, but always take care to store them appropriately (including thinking about an encryption step in the build process).

What next?

The next blog will show you how how to link this to Jenkins and built and deployed the project to AWS. This will focus on configuring the Jenkins job to use MSBuild to build the package and AWSDeploy to deploy it to the elastic beanstalk URL we've already specified here.

AWS Summit 2013 - London

AWS Cloud Summit

At the beginning of the day I fell out of my bed into the cloud summit. Seeing as I was less than a stone's throw away, it was one of the easiest journeys to a conference I had managed.

We registered with our bar-codes from the original e-mail, though this proved to be somewhat of a problem technically, since the bar codes didn't seem to appear on everyone's e-mail. this caused a bit of a technical glitch which delayed some poor souls, but mine worked fine so I was in fairly uneventfully.

The geographical layout was available a day prior to the summit via the Guidebook app, which I have to admit has been one of the most solid platforms that I have encountered for this sort of thing and beat the Azure backed Eventboard app from last year hands down. AWS obviously put some work in to get this right. 

What was interesting about the layout was the location of a labs sections near the entrance, with the keynote area cordoned off with high cloth walls which hid a stage where Werner Vogels, CTO of AWS would introduce the summit. The labs areas where we got to try out some of the infrastructure, including the just released (as of today) S3 storage and RedShift storage platforms to the European market. This area also contained large yellow beanbags for the more bohemian amongst us to use to sit, write, blog, experiment etc. I didn't try them out because knowing me, I'd not be able to get back up again because of a) Sleeping or b) Imitating a turtle on it's back.

The infamous beanbags! My nemesis :-)

Upstairs, near the black cloaked keynote area, sponsoring organisations and the supporting acts were plying their trade and demonstrating their wares. in the centre of that area was the AWS exhibit itself, manned by a number of architects of the platform, whoc were showcasing some of their logical reference architectures. I had a useful conversation with Andreas about availability, since this is an area I am interested in and have blogged about before. I pointed to various potential single points of failure and and Andreas managed to explain to me the points of availability and how they have solved some of their high availability problems in the reference architecture, which I could read up on and use as case studies. All very interesting stuff.

The chaps make last minute checks for our arrival

I was also lured, stomach first, to the Smart421 exhibit by the temptation of boxes of Smarties stacked pyramid-like on their desk. What was interesting about this chat was that they reminded me that unlike the Azure platform, Amazon has a very robust mechanism for having PCI-DSS compliance. After all, they do this every day for their own business. Indeed, Stephen Schmidt went on to introduce the security model under AWS and in particular, the very inspiring and competent VPC (Virtual Private Cloud) offering which uses CloudHSM as a hardware security management option where the client holds the keys and not Amazon. Indeed should anything happen to the physical media, such as tampering or accidental damage, the keys are wiped immediately. This was something that I heard hide-nor-hare of at the Azure conference and PCI-DSS, to the best of my knowledge hasn't been adequately addressed to the same level by Microsoft and certainly isn't as mature. 

A little behind schedule, Dr Werner Vogels opened the summit with a comprehensive (read long) keynote on the AWS now and in the future, including some of the new services that have come about recently (53 in the first quarter of 2013 alone). He introduced a number of customers who have used the AWS platform, including a number of very large names in the UK, such as Shell and as well as smaller start-ups such as Shutl.

Big Werner Vogals on a very big stage

Werner outlined the strategy for the company and gave us little insights into the internal workings of Amazon. In particular, how Amazon believes in and practises lean principles at their organisation. The insight into how some of the cloud based operations at other organisations seemed to, in part, back up some of the stereotypes associated with the governance of data, such as don't put most sensitive data online. This didn't seem to be a concern at all points and was certainly not one imposed due to any technical limitation, but it did show that some organisations do still have some concerns surrounding the non-colocation of their data.

Steve Schmidt then took over and introduced a number of key information security objectives. He had a lot to say and there was a lot of detail about the AWS security processes and policies. Very impressive as it happens but given they have DoD clients, this is hardly surprising. One of the implicit things that both Steve and Werner introduced without 'saying' in both their own and case study examples was the need for good governance. Indeed, shell were very explicit about their data governance and as such, cloud should never be considered something that should allow your governance processes to become lax. Indeed, the opposite is true in cloud environments. An important lesson there, especially in VPC environments where the clients manage their own keys.

After lunch, which was very popular (So I started to write this blog post until the queue died down), the breakout sessions began. I stuck myself on the bootstrap sessions mostly, but wanted to head into the Architecting for High Availability thread. My phone's battery was being a bit 'negative' and I worried that I would run out of juice from taking all the pictures that I was taking and sure enough, by the time it got to the HA presentation by Ianni Vanvadelis, I had no more juice to take any pics of relevant slides. As it happens, for the customer case study, presented by Dan Richardson, Director of Engineering at Just-Eat, the slides were not there anyway. It was unfortunate, but he actually did a brilliant job in winging it and still managed to get his message across.

Prior to that however, I attended the Bootstrap tracks of "Your first week with Amazon EC2", "Agility and Cost Savings: Achieving the IT 'X-factor'" and the Keynote tracks of AWS OpsWorks, presenting a deep dive with the OpsWorks environment by Thomas Metschke, which was a very technical example of how to combine Chef, Ruby, Git and Jenkins with OpsWorks to deploy to different CM configs. OpsWorks is an excellent platform for this, I have to say.

Thomas takes us through an OpsWorks deep dive. I really like this tbh!

I went to the X-factor track and my phone died completely. It was tough going from that point onwards, but obviously means I should have known how long my battery would last. However, not being big on photography (with a face like mine, it isn't something that I do much of ;-). Comparing the costings of cloud versus on premise platforms is something I tend to do a lot of anyway, so there wasn't really much new there. However, there is a TCO tool that has been created by AWS for just this purpose. Also, the different costing models were introduced and made very clear. What was interesting was the use of 'Spot pricing'  models, which I finally figured out a use for, especially in non-mission critical/off-line work. Reserved instancing is also something I am going to be looking more into. However, it is good to know that I value things the same way AWS do. Having grown up in a house full of economists and financiers hasn't gone to waste ;-)

AWS Pricing Models presented to us by Dan Roger

Dan shows us the break even with the On-Demand services

What was interesting about what Dan Roger told us about was what he didn't tell us. In that session he drew   comparison of On-Demand against reserved instances at various levels. you can see the slide above, but what he also shows is the break even with Heavy, Medium and Light Utilisation Reserved instances with each other. In this example, at 8 months it makes no difference which option you choose. This is on top of the very obvious comparison with 1 or 2 month on-demand services.

I spent a lot of time getting a lot of valuable information from the AWS guys playing piggy in the middle with the vendor space about their platforms. In particular, the AWS reference architectures will provide a lot of very useful information, not necessarily in how I would do it, but in providing patterns of best practise from which to work.

As an architect, I have certainly not had anywhere near as valuable a day as I've had today. So I am quite a happy bunny. Plus, having played around with AWS EC2 already, I am much happier than when I was playing around with Azure. There are a couple of blog posts that I can see spinning off this, especially in the area of highly available systems.

All-in-all, a very productive day :-)