The art of troubleshooting

Recently I had to spent a few hours (days, weeks) to fix a nagging issue that raised its head now and then.

As always, the issue was difficult to reproduce, sometimes it seemed to go away but unfortunately it always came back to drive users crazy. So it was a matter of time before shit hit the fan and the thing really got priority.

We got it fixed, but it still left me with a bit of a hangover. Why did it took so long and why did postponed it until we had so much frustration around and we actually got on with it.

After a bit of soul searching I'll try to define a few principles that could be followed when dealing with such issues. On purpose, I won't be talking about technology, even more, I guess this is useful in most disciplines.

Rule #1: Never ignore these recurring issues. Give it priority.
If the issue recurs once or twice, there is a very good chance that it will keep on raising it head, So acknowledge the issue and focus on finding the root cause for it, otherwise you'll be forced to wipe your agenda (typically when it doesn't suit you at all) after things have been escalated.

Rule #2: Focus
Of course you need to prioritize, but fixing problems is not a part-time job. Focus on it, wipe your agenda and do not give up until you found a solution.

Rule #3: Understand and co-operate across the entire chain
More and more, solutions are chains of services, and if you do not have the overview of the entire chain, there is a very big chance that fingers will be pointed between the various parties. You need to have the overview, work together and make sure everyone involved has the same sense of urgency.

Rule #4: Work in tandems
When working alone, there is a good chance you'll be overlooking things. When working in duos, you need to explain to your partner why something might or might not be an issue, which is a critical part of finding the root cause.

Rule #5: Brace yourself for the hangover
Fixing the problem is like a night out with friends. While you're doing it, it is great but there is a good chance it feels differently the day after. Why took it so long, why was it such a simple thing, these are the sort of questions (accusations?) you will be asking yourself. That's the way it is, I guess you just have to live with it.

Is this rocket science? Far from it! Would it be helpful to live a bit more by these rules? Most definitely, at least I'll give it a try next time :-)

Architecting for the Cloud

Utilising the cloud for disaster recovery purposes: server and data replication

Introduction

Disaster Recovery is one of the most interesting use cases for adopting cloud technology. Almost every company has to deal with it, these pesky accountants asking for how you're dealing with potential disasters and how your IT is going to survive that.

The problem with DRA is that it used to be a very costly exercise, very hard to get it (and keep it) right and all that effort and money goes into something that you hope will never be used. Difficult business case indeed!

The Pay as you Go model of Cloud computing is an excellent option for this. You can prepare your entire backup solution in advance but, depending on your RTO and RPO only a minimal set of resources has to be active all the time, and hence only a minimal part of your backup infrastructure costs will be invoiced to you. Much better!

Disaster Recovery is much more than just IT, and even for the IT part designing an either cloud based or more traditional disaster recovery system is not to be underestimated. It definitely takes a lot more than a single blog post.

However, one of the usually tricky things is how to keep the configurations of your server infrastructure, as well as the data in sync between the different environments.

I just got off a phone call with some people of Cloudleap where they demonstrated their product. This is quite an interesting product and I gathered it would be just as easy to capture my impressions in a blog post.

What is the problem?

Data is obviously not static. It changes all the time and in order to have a useful disaster recovery solution, the data in your DR solution must be as up to date as possible. Cloud providers, and quite a few technology providers provide solutions for. For instance, AWS Storage Gateway allows you to replicate on-premise data volumes to the cloud, so they can be used as backups.

However, the same, although to a lesser extent, applies to the servers itself. Servers are not static, they change because of updates, new software installed etc. and in order to have a reliable DR solution it is critical that you can rely on these configurations to be in sync with the primary systems.

Solutions

To deal with this aspect, different solutions are available. For instance, if you are using dynamic provisioning solutions such as Chef or Puppet things will be a lot easier. However, getting started with these kind of tools is not necessarily an easy thing to do, and automating everything that you have configured in the past is quite some task.

From that perspective, CloudLeap is solution requiring a lot less investments to get started as it allows you to mirror your server configurations as well as your data to a cloud provider of your choice. It does this by simply copying the disks block by block to your target provider. Obviously the mammoth AWS is supported, but quite a few others as well (Azure suport still on its way though).

CloudLeap can be used to do a one-time migration of server (and data) images, and optionally it can also be used to keep it in sync by periodically replicating it again. The big difference with the likes of AWS Storage Gateway is that it turns (where applicable of course) your mirrors into bootable volumes which can be launched upon request, basically providing a point-in-time copy of your server and data.

It requires the installation of an agent on the source server systems and uses a (single tenant) management console which controls these agents specifying by things such as which volumes to replicate and using what frequency.

This introduction still leaves quite a few questions unanswered but at first glance this definitely is an interesting piece of the puzzle, and deserves to be considered when designing a DR system. I need to get started with it.

Troubleshooting AWS' Elastic Load Balancers

Introduction

Amazon Web Services' Elastic Load Balancer are excellent, well maybe not excellent but still a very useful service. It does do the, well, load balancing, but in contrast with your own (e.g. HAProxy) load balancer it takes also care of scaling, fail-over and so on. 

So if you have multiple servers, with not too complex routing requirements, ELBs are a very convenient way forward. I've also found myself using ELBs when only exposing one single server, because it helps satisfying security requirements and it gives you some useful features on top of that. For instance it allows you to easily check the health of a server on an application level, and to monitor the roundtrip times of requests.

But anyway, that's not what I want to talk about. Using ELBs can also be a p@!n in the@$$. Troubleshooting connection problems have wasted quite a few days of my precious life, and I thought it would be good, at least for me, to have an overview of things you need to check. So here you go...

Btw, I am assuming a VPC environment as this provides the most broad choice of things that can go wrong.

Problems, possible causes

1. Are the load balancers in the right subnet(s)?
• Probably it's just me but I find the console and documentation of the ELB quite confusing. Even if the ELB is listed as internet facing, it is just possible that it lives in a private VPC subnet. Go fix it, otherwise it will never work.
2. Are the listeners properly configured?
• Do you have listeners for the ports you try to connect to? Make sure they exist and have the right protocol (see next);
3. Which protocols did you use?
• For the listeners you can select HTTP, HTTPS and TCP. Depending on how your web sites behaves you might want to switch between (e.g.) HTTP and TCP to see if that makes a difference.
• Usually this is not something that troubleshoot connection problems, but might help in letting your application behave as it should.
4. Do you have the proper ports opened in the Load Balancer's security group?
• So for each listener, typically there should be a security group entry.
5. Do you have the proper ports opened in the Server's security group?
• Same applies for the Server's security group. For each protocol in the ELB listener and security group there should be a corresponding item in the server's security group.
• However, the server's security group should typically limit access on these protocols to the load balancer only, in contrast with the security group for load balancer itself.
• Note that this is not necessarily the same port, as the ELB's listener can map the incoming port to another outgoing.
6. Do you have the health check properly configured? If not the servers will never come 'In Service'.
• Check if you have the health check page installed at your web server.
• And, the smarter this health check page is, the better of course...
• Check if your health check page is accessible from the ELB. E.g. this might run on a different port.
• Check the protocol of the health check. Preferably you use HTTP or HTTPS, but if the server runs HTTPS and you have the ELB configured in Pass-Through mode, you should use TCP for the health check configuration.
7. Are the ELB's and the Server instances in the same Availability Zone
• Until the (recent) announcement of Cross-Zone load balancing, you hade to make sure that both your load balancer and server instances live in the same AZ(s). If not it simply doesn't work.
• This is quite frustrating, even more if you realise that the VPC wizard without explicit action quite often generates a public and private in different zones.
• With Crosszone support this problem should not occur, but it might well be that you haven't activated this. So check it.
8. Did you use the proper SSL configuration?
• Just loading an SSL certificate (if you want to use that) is not enough. You need to disable SSLv2 and the weaker cipher suites to get a decent rating. Check http://www.ssllabs.com.
• Again, this is not really a connectivity issue, but while we're at the subject...

So this was a quick brain dump, I'm sure I have forgotten a few cases. If you know them, please leave them in the comments.

EC2 Instance in public VPC subnet not being able to communicate with outside world

When provisioning systems within Amazon Web Services, I often use cloudformation templates that bootstrap a Chef client for further provisioning. For that bootstrapping part, we utilise the user-data mechanism.

I have this user-data bootstrapping script that works flawlessly for it seems eternity, but all of a sudden I got a call from a customer complaining about the script to fail.

It turned out that the apt-get update statement, which was one of the first statements in the script could not reach the repositories. And the strange thing was then when I ran the script manually it was ok.

It only happened in a particular topology where we had the server instance running in the public subnet and attached to an elastic IP. Usually, we don't use that topology, as I'd like to have the server in the private subnet and load distributed via an ELB, but for development purposes this topology is still useful.

Long story short, when running in the public subnet, the server needs to have an EIP attached, otherwise it cannot communicate with the outside world. The servers in the private subnet do not have that issue, as they rely on an NAT instance to transfer their communications.

It turned out that it takes a little while before that outgoing communication path for the server in the public subnet is established, and that the user-data script was sometimes (not always!) executed before the communication path was actually there. And then, obviously, it fails for things like apt-get update. A simple sleep, although it is a workaround, solved the problem.

Patience is, as always a virtue!

Best Practices for accessing AWS accounts - Quick Reference

Introduction

Now and then I run into the question what best practices should apply to providing access to your AWS accounts. To a large extent, this information is readily available in resources across the internet (such as this and this) but I thought it would be useful to provide a quick reference to that.

Providing access to AWS services

Practice	Rationale	Priority
Always use IAM users, do not use AWS Account user (or access keys)	This AWS Account User is considered the root user, providing access to all services and data. Normal operations should not be carried out using this account, instead an IAM account with the appropriate privileges is recommended.	High
Use different AWS accounts for production and non-production purposes.	This provides strong separation, and requires re-login when moving from one account to another. This reduces risk for un intended changes, and allows tailoring access rights for devops engineers.	High
Use RBAC practices using IAM groups, using least privilege model.	Using roles (groups) provides a more transparent and manageable access right model.	Medium
Enforce strong password policy	Important to prevent passwords from being guessed or cracked.	High
Enable multi-factor authentication	Enable multi-factor authentication for both (human) IAM users as well as the account owner to increase level of security, providing an additional layer of security.	High
Implement key rotation	It is advisable to ensure that access keys for IAM users are changed once in a while (say one per three months). For more info, see this blog.	Medium
Let users manage their own password	Ensure that users can manage their own passwords, and (strongly) encourage a password rotation schedule.	Medium
Use policy conditions for extra security	By means of policy, additional security can enforced. For instance, by defining that only particular users can delete a particular AWS resource. This is useful for fundamental services which failure would have great impact on service.	Medium
Enable Audit trails	Enabling AWS’ CloudTrail option enables security analysis, resource change tracking, and compliance auditing.	Medium

Providing access to the AWS hosted environments

Practice	Rationale	Priority
Always use Virtual Private Clouds	This is the default in newly created AWS Accounts, but should be used in all cases as it adds a lot of additional options from a security perspective. For certain applications, a dedicated VPC can be considered.	High
Use temporary credentials	When providing your server instances access to AWS services, always deploy temporary credentials in combination with IAM roles.	High
Consider use of VPNs	Consider using a VPN connection between your corporate network and the VPC, as it provides a strongly secured connection. However, ensure that only qualified staff have access to that particular network zone.	Medium
Use remote access bastions	In case there is no VPN connectivity between the VPC and the corporate network, use (SSH or RDP) Bastions to get access to your server instances. Only run these bastions when access is required.	High

Is the Open Group's TOSCA the future for AWS Cloudformation?

I stumbled into TOG's TOSCA initiative (Tosca also being the name of my parent's dog, hence the picture :-), a standardisation initiative that aims at "Enhancing the portability and management of cloud applications and services across their lifecycle".

According to the first sentences in the spec, "This core TOSCA specification provides a language to describe service components and their relationships using a service topology, and it provides for describing the management procedures that create or modify services using orchestration processes. The combination of topology and orchestration in a Service Template describes what is needed to be preserved across deployments in different environments to enable interoperable deployment of cloud services and their management throughout the complete lifecycle (e.g. scaling, patching, monitoring, etc.) when the applications are ported over alternative cloud environments."

In principle this is a good thing. I believe the future is in Infrastructure-as-Code and although this is typically associated with automation efforts as done with Chef and Puppet, I personally feel it is even better demonstrated by services such as AWS' Cloudformation.

Cloudformation allows you to declare which infrastructural components (server instances, load balancers, storage components, dns registrations etc.) you want to instantiate, including their configurations and inter-relationships. It typically operates on infrastructural level, and although it has some server configuration features (through cfn-init) it typically operates in concert with tools such as the mentioned Chef, Puppet or alternatives.

Cloudformation is great. It is massively frustrating to develop complex systems in, given to the immaturity of the development ecosystem around it (which will be resolved over time), but still I am not aware of an alternative provided by the other cloud providers that comes close.

This TOSCA standard acknowledges the importance of such provisioning tools, and aims at getting a standard in place. With the promise of preserving portability and avoid (cloud vendor) lock in. So that's great isn't it?

Yes, potentially yes. I am very happy with AWS and I really think they are by far the most feature-rich cloud provider out there, but it's always good to preserve portability. For instance, someone could have particular data requirements that simply cannot be met by AWS (e.g. data cannot leave the country), or has some kind of (Microsoft anyone?) enterprise agreement that they want to leverage in the cloud. Having a solid standard in this space could facilitate that, and could also boost the development of development tools.

The reality of course is that cloud landscape is evolving at lightning speed, and that standardisations body typically have no possiblity to keep up with that. Looking at the differences in services portfolio between cloud vendors (with AWS being the undisputed front runner in both current portfolio and development speed) it's a damn hard job to do standardisation there. Standards like this run the risk to only support the lowest common denominator, which is something not a lot of people are waiting for I'm afraid.

So I am bit torn. I must admit that I haven't studied the standard in detail yet, so maybe a lot of my concerns are already addressed, but my gut feeling says it will take a while before the (leading) cloud vendors adopt this.

How to avoid circular dependencies between Cloudformation stacks

I am currently designing a cloud based solution based on AWS, Cloudformation and Chef, and using the concepts as described in an earlier post.

In short, I am a big fan of Cloudformation which is a perfect example of Infrastructure as Code, but developing and maintaining Cloudformation scripts and stack is a bit of a cumbersome process. In order to keep that manageable, we need to adopt good old design principles such as low cohesion/high coupling when deciding what to put in a particular template/stack.

Let’s assume two Cloudformation templates, one containing all database server related resources, while the other contains all web server stuff. Forget all other stuff, and assume each stack just have a server instance with a security group.

The Web server requires the database server to be present to do the provisioning of the application, while the database server security group requires the Web server security group to be created in order to provide access. Bang! Circular dependency, which is always a bad thing!

Of course, this can be solved, and the following options exists:

Forget about modularisation, put everything in one template. Of course this works, and in this simple example most likely the most pragmatic approach also. But in a real life setup, things are not as simple and you might have a multitude of service stacks having inter-dependencies. And putting these all together in one script will quickly become a head ache.

First create all security groups, then the server instances. Well, that will work as well, and is a relatively clean solution too. Few things I consider a drawback are:

• It introduces another template (and hence stack);
• It also introduces additional dependencies between these stacks, which must be catered for by means of input parameters;
• The definition of the SG, which is tightly related to the server is removed from the service stack.

The option I went for, was introducing a dedicated security group in one stack, and use it for identification purposes in another. This is shown below.

In the database stack I create an additional security group (WS-ID) which has an empty definition, and I only use it to allow the DB Security Group to provide access to it.

In the web server stack, I provide a reference to this security group (through an input param) and by this allows clean access to the database server. Basically you are using the WS-ID security group for identification purposes only.

Hence I have removed the circular dependency, kept the number of stacks the same and have limited the dependency between stacks to a minimum. In this example, there is only one dependency, but in reality there might be many more. 

These software design patterns prove to be useful even when designing infrastructure he!

Update: there is a significant drawback in this approach, as you cannot terminate and rebuild the DB Server stack without terminating the web server stack. To cope with that, you can create the WS-ID Security Group in a separate stack, and feeding the reference into both the DB Server Stack as well as the Web Server stack.

Why the headache of License Management could be a major push towards cloud adoption

I've been busy lately with various software license related assignments, not necessarily related to a cloud setup. 

And quickly I recognized all the headaches and frustration that comes with it, which are even more apparent when being active in cloud environments for the last decade or so.

Let's face it. Licenses are a major headache. Understanding the rules is ridiculously complex, and change all the time. And most of the time, these changes are not to lower your fees.

Strong software asset management discipline is a necessity, and while I will be the last one to claim that cloud will solve all these problems and make them redundant, it definitely will help in some cases.

For instance, one customer has an application (provided as a SaaS) with a very peaky behaviour. One week per month it needs massive power, the other three weeks it just cruises. Buying licenses for the peak capacity would be a very costly exercise, even more so given the SaaS characteristic of the service. You simply wants to scale with demand, both periodically as well as the growing customers, and traditional licensing models simply cope very badly in such scenarios. Cloud computing, in case licenses are included proves to be a major attraction to this types of customers.

Another customers needed to renew their enterprise agreement with another software vendor, and needed to make an inventory of what software is used, over time, and in what configurations. Should be simple enough, right? Not so, keeping track of everything, understanding all the rules and exceptions is close to impossible. Comparing that with ease in which you can launch enterprise applications in the cloud, and getting charged for using both hardware and software by the hour (or even minute), it is clear that there is something to be gained here.

Understand me correctly, I do not feel the cloud is a silver bullet that solves all such licensing issues, but the matter of fact is that it comes inherent with some features that makes life in this regard a lot easier. And in these two cases cloud computing was mentioned by the customer as an attractive setup, mainly because of the licensing restrictions.

I am convinced that this will prove to be a major battle ground, and by exploiting this cloud providers have a great opportunity to lure traditional enterprises to use their services.

Quick guide to AWS Database options

Introduction

Amazon Web Services is widely known as the leading Infrastructure as a Service provider, and let's be clear: they are by far the most powerful option in this area.

However, it would be unwise to believe that AWS is only active in this particular IaaS space, and over time they have build up a significant number of services that can be classified as Platform as a Service.

A few weeks ago I visited the AWS summit in London and attended a few (big) data sessions, and I figured it would be helpful to list the various database platform services AWS is currently providing, and give a quick introduction and guide to how and when to use these. Note, both SQL and NoSQL option will be discussed, but it will not go into the details of these two options (not to mention the various SQL and NoSQL flavours that exist).

SimpleDB

According to AWS, Amazon SimpleDB is a highly available and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests and Amazon SimpleDB does the rest. It provides a very flexible query interface, sports multi-data center replication, high availability, and offers rock-solid durability. And yet customers never need to worry about setting up, configuring, or patching their database.

The main drawback however is that its scalability is limited and performance not always predictable. It also has strict storage limitations of 10GB hence partitioning is probably required for larger systems.

So when you need the flexibility of a non-relational database and your scalability requirements are modest, this could be a good option.

DynamoDB

DynamoDB is AWS' other NoSQL database offering. It is a tremendously scalable and fast service and provisioning and scaling of databases (tables) is straight forward. The underlying technical architecture is state of the art, for instance it runs on SSD rather than good old hard drives. It integrates nicely with other AWS features such as S3, Data Pipeline and RedShift.

So when you are in the NoSQL game and really need high scalable low latency solution, DynamoDB is the way to go.

RDS

Relational databases are known territory for most of us, and the likes of Oracle, SQL Server and MySQL are big players in this area. It is perfectly possible to run your own (relational) database on AWS EC2 but if you don't want to be bothered with configuration issues such as storage configuration, backup schedules and read-write copies, RDS is certainly an option. Even more that the extra costs of RDS over running your own database on EC2 is very modest.

So realising that NoSQL, although a nice and shiny option is certainly not always the preferred option, AWS' RDS option is certainly worth a look at. Especially for the transactional processing systems as for the analytical processing AWS has...

RedShift

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools. It is optimized for datasets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
RedShift is accessible through a standard ODBC or JDBC API as it implements the PostgreSQL interface (but note: it is not PostgreSQL under the hood!).

So when large amounts of data must be analysed and (in real-time fashion) reported on, RedShift is a very attractive option.

EMR

AWS' Elastic MapReduce option is technically not really a database, but is still very applicable for processing large amounts of data. EMR is a fully configured Hadoop setup, including tools such as Pig and Hive. It is tremendously powerful although also a bit low level, and typically operates in a batch-oriented (rather than real-time) fashion.

EMR is a very good option for processing incoming data streams and move the results to other database options such as discussed before in this post.

Conclusion

This is only the tip of the iceberg and an in-depth comparison of the various options takes a lot more space. However, I do hope it provides a first impression and guidance when having to choose between the various options

Note the options discussed are really platform services, so a lot of the complexity is abstracted away from you. You may or may not like that, if you are in the latter camp rest assured that you can still use the IaaS options and run your own database solution just the way you want it utilising services such as EC2 and EBS.

AWS making strides again

Amazon Web Services is pushing out one update/innovation/product after another, last year more than 100 announcements were made. That is two a week! Some announcements are pretty small, while others are short of groundbreaking.

In the last few weeks, two announcements were made that caught my eye and showed (in my opinion) that these guys are still on another planet. While the competition is struggling to provide the basic infrastructure services, AWS is providing some very advanced stuff.

First of all: OpsWorks is really cool. If you've read some of my previous blogs, then you'll know that I am skeptical about the benefits of 3rd party cloud management tools. Not because I feel we don't need such tools, but because the standard tools (of in this case AWS) are very powerful by themselves.

With the launch of OpsWorks, basically the powerful Chef framework embedded in the AWS Console, this has become even more true. Until recently, the AWS console has focused on managing the infrastructure, with OpsWorks they make a move into managing the server configuration as well.

Sure, Opsworks is still limited and does not support too many dfferent platforms. But it clearly shows the direction AWS is taking and this is light years ahead of the competition. Impressive.

The other thing that I found noticeable is smaller but still as significant  The introduction of the CloudHSM. A HSM (hardware security module) is not a commodity in a typical application architecture, but in security sensitive solutions this is where the confidential stuff happens/resides. The HSM protects this by means of software but also hardware. For instance, it typically comes with tamper resistant housing, that deletes all sensitive stuff the minute someone starts fiddling with the housing. 

What is the number one reason cited for NOT moving to the cloud? Indeed, that's security. And by providing a managed HSM service the options in this regard are much improved.

Interesting times indeed.

Thoughts on Cloud Management Solutions - Chef

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the third, and for now last of a few posts.

In the first post, I shared my two cents on the added value of Rightscale, while in the second post I discussed Amazon's own Cloudformation. Now (a bit later than planned due to personal reasons), another piece of the puzzle, which is Opscode's Chef configuration management framework.

To make things clear from the start, I am a big Chef fan. It is a hugely powerful framework to manage a (large) set of infrastructure components and provide sufficient abstraction to achieve high productivity and enough flexibility when and where it matters.

But one step back. This is supposed to be about cloud management solutions, and while Chef has been designed with Cloud in mind, it is not necessarily a cloud management solution. It is perfectly suitable in a more traditional environment, either with virtualised or physical servers.

And in comparison with the likes of Rightscale, this is no comparison. With Rightscale you are able to define your cloud environment, such as which servers, networks, scaling algorithmes and so on, and Chef is not really focused on this. Sure, it can do certain things, but it really excels in combination with other tools that take care of the management of cloud resources. So a combination of Chef + Rightscale, Chef + Cloudformation (my preferred choice) or for that matter even combinations like Chef + Vagrant.

In the Management Stack, Chef (and for that matter its counterparts like Puppet or CFEngine) has a strong focus on the upper two layers, Server Provisioning and Service Orchestration.

Chef provides strong concepts like Roles, Environments and Data bags to design and implement a clean and maintainable configuration management solution. The primitives as provided by the framework allow for a much more declarative approach rather than the procedural approach as seen in the classic shell scripts. This is not to say that Chef based solutions are declarative by nature, as more complex functionality can be provided by programming Ruby, and then you end up with a more procedural solution after all.

Traditionally, and still often used are using pre-defined images and launch them whenever needed. However, the big advantage of using something like Chef are:

• It is Execution environment independent. It can manage resources in the cloud, on your local network as well as on your desktop.
• It is relatively platform independent, although it takes a significant amount of effort to develop and test it.
• It allows you to use clean base images, which can be updated with the latest security patches and other stuff, while it doesn't affect the configuration management solution itself.
• Running server instances can be easily updated by running the agent again.
• Chef provides much more flexibility when it comes to combining and splitting roles. E.g. for development and test purposes you would like to have combined web and database server, while for production purposes these must be separated.

Apart from managing individual servers, Chef also provides tools for composing these individual resources to a meaningful solution. For instance, web servers need to know where to find the database server, and all servers must be configured to forward their syslog messages to a particular loghost. Quite often this information is only available at runtime, and by keeping a central repository of all nodes and their configuration data, Chef allows you to query this repository and use the results in your configuration tasks.

Is it perfect? Far from it. Customising resources and keeping them separate from 'standard' resources requires quite a bit of discipline. The command line is clunky at times and for sure Chef is a complex beast. So we will celebrate a lot of improvements in the coming years, that is for sure. But for now, it is one of the most powerful solutions in this space, and I am very happy to use it.

This concludes my 2013 'State of the Cloud Management space' mini series. One of the nicest thing about our job is that it will be probably outdated the minute I hit the Publish button. And that there will be many different and non-compatible views about this subject. Let me know!

Thoughts on Cloud Management solutions - Cloudformation

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the second of a few posts.

In the previous post, I shared my two cents on the added value of Rightscale, while in a future post I will also discuss Opscode's Chef. But now, Amazon's own Cloudformation.

Wait, we were supposed to talk about Cloud management solutions, right? And now we are going to discuss one of Amazon's features, it itself part of a much broader management solution. Yep, that's right. And I agree, Cloudformation is not a full blown cloud management solution in itself but it is a pretty darn useful component and allows you to do some powerful stuff.

So, what is then. Well, obviously Cloudformation is AWS specific and has nothing to offer in the multi-cloud area, like Rightscale does.

Amazon Web Services has grown over time to an amazing set of Cloud services, some of them overlapping, others complementary and launching a solution in AWS typically takes a few of these services to work in concert with each other. Simply launching a cloud based server typically requires a significant set of resources such as the server instance itself, storage volumes, security groups, DNS records, alarms, possibly load balancers and auto scaling configurations and so on.

Manually configuration this kind of things quickly becomes very boring, and that is where Cloudformation kicks in. It allows you to declare the kind of resources you want (if not clear enough: in a declarative way, which makes a big difference), and create this as a fully managed stack either through a web interface or command line.

An updating simply requires updating the template and applying it to an existing stack. A snippet of a security group declared in such a cloud formation template is shown below.

"WebServerSecurityGroup": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
  "GroupDescription": "Security Group for the web server instances",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"SourceSecurityGroupName": "amazon-elb-sg",
"SourceSecurityGroupOwnerId": "amazon-elb"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
  ]
  }
}

The very, very useful thing is that it allows you to treat your IaaS configuration as code, and that you don't have to deal with state. So no checking if resources already exist and based on that defining the next steps: this is taken care for you under the hoods.

Note that you have to realise that this still have to take place. Changing a resource, for instance the instance type of a server instance will result in a stopped instance (or terminated, in case of an instance store instance) and started again with the new instance type.

At best, Cloudformation is part of a full blown cloud management solution. Cloudformation focus is on describing the cloud resources you need, not on the configuration on the server instances itself. That said however, Cloudformation has some tooling to configure these servers as well, a bit of a (very) light weight Chef or Puppet kind of thing or possibly more comparable Ubuntu's cloud-init. This is useful for some not too large systems, but is typically used to bootstrap these servers with agents that take ownership for the further provisioning of these servers.

I really recommend that in case you want to use the AWS platform to take a better look at Cloudformation. It has a bit of a learning curve but it's definitely worth the investment.

There are a few things that need attention though:

• The service is not bullet proof yet. Sometimes it throws exceptions that disappear after a few hours again and in (very) rare occasions these cloudformation stacks end up in an error state that leaves you no other option that deleting the entire stack and start all over again. Ouch! 
• Cloudformation is declarative by design and has only very limited conditional logic support. This quickly leads to very long templates with quite a bit of duplication of code. There is support for included external scripts in your stack, but in practice this doesn't work too well. In my view the best way to use cloudformation script is to wrap it in a simple generator that allows you to minimise code duplication.