Thoughts on Cloud Management solutions - Cloudformation

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the second of a few posts.

In the previous post, I shared my two cents on the added value of Rightscale, while in a future post I will also discuss Opscode's Chef. But now, Amazon's own Cloudformation.

Wait, we were supposed to talk about Cloud management solutions, right? And now we are going to discuss one of Amazon's features, it itself part of a much broader management solution. Yep, that's right. And I agree, Cloudformation is not a full blown cloud management solution in itself but it is a pretty darn useful component and allows you to do some powerful stuff.

So, what is then. Well, obviously Cloudformation is AWS specific and has nothing to offer in the multi-cloud area, like Rightscale does.

Amazon Web Services has grown over time to an amazing set of Cloud services, some of them overlapping, others complementary and launching a solution in AWS typically takes a few of these services to work in concert with each other. Simply launching a cloud based server typically requires a significant set of resources such as the server instance itself, storage volumes, security groups, DNS records, alarms, possibly load balancers and auto scaling configurations and so on.

Manually configuration this kind of things quickly becomes very boring, and that is where Cloudformation kicks in. It allows you to declare the kind of resources you want (if not clear enough: in a declarative way, which makes a big difference), and create this as a fully managed stack either through a web interface or command line.

An updating simply requires updating the template and applying it to an existing stack. A snippet of a security group declared in such a cloud formation template is shown below.

"WebServerSecurityGroup": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
  "GroupDescription": "Security Group for the web server instances",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"SourceSecurityGroupName": "amazon-elb-sg",
"SourceSecurityGroupOwnerId": "amazon-elb"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
  ]
  }
}

The very, very useful thing is that it allows you to treat your IaaS configuration as code, and that you don't have to deal with state. So no checking if resources already exist and based on that defining the next steps: this is taken care for you under the hoods.

Note that you have to realise that this still have to take place. Changing a resource, for instance the instance type of a server instance will result in a stopped instance (or terminated, in case of an instance store instance) and started again with the new instance type.

At best, Cloudformation is part of a full blown cloud management solution. Cloudformation focus is on describing the cloud resources you need, not on the configuration on the server instances itself. That said however, Cloudformation has some tooling to configure these servers as well, a bit of a (very) light weight Chef or Puppet kind of thing or possibly more comparable Ubuntu's cloud-init. This is useful for some not too large systems, but is typically used to bootstrap these servers with agents that take ownership for the further provisioning of these servers.

I really recommend that in case you want to use the AWS platform to take a better look at Cloudformation. It has a bit of a learning curve but it's definitely worth the investment.

There are a few things that need attention though:

• The service is not bullet proof yet. Sometimes it throws exceptions that disappear after a few hours again and in (very) rare occasions these cloudformation stacks end up in an error state that leaves you no other option that deleting the entire stack and start all over again. Ouch! 
• Cloudformation is declarative by design and has only very limited conditional logic support. This quickly leads to very long templates with quite a bit of duplication of code. There is support for included external scripts in your stack, but in practice this doesn't work too well. In my view the best way to use cloudformation script is to wrap it in a simple generator that allows you to minimise code duplication.