Thoughts on Cloud Management Solutions - Chef

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the third, and for now last of a few posts.

In the first post, I shared my two cents on the added value of Rightscale, while in the second post I discussed Amazon's own Cloudformation. Now (a bit later than planned due to personal reasons), another piece of the puzzle, which is Opscode's Chef configuration management framework.

To make things clear from the start, I am a big Chef fan. It is a hugely powerful framework to manage a (large) set of infrastructure components and provide sufficient abstraction to achieve high productivity and enough flexibility when and where it matters.

But one step back. This is supposed to be about cloud management solutions, and while Chef has been designed with Cloud in mind, it is not necessarily a cloud management solution. It is perfectly suitable in a more traditional environment, either with virtualised or physical servers.

And in comparison with the likes of Rightscale, this is no comparison. With Rightscale you are able to define your cloud environment, such as which servers, networks, scaling algorithmes and so on, and Chef is not really focused on this. Sure, it can do certain things, but it really excels in combination with other tools that take care of the management of cloud resources. So a combination of Chef + Rightscale, Chef + Cloudformation (my preferred choice) or for that matter even combinations like Chef + Vagrant.

In the Management Stack, Chef (and for that matter its counterparts like Puppet or CFEngine) has a strong focus on the upper two layers, Server Provisioning and Service Orchestration.

Chef provides strong concepts like Roles, Environments and Data bags to design and implement a clean and maintainable configuration management solution. The primitives as provided by the framework allow for a much more declarative approach rather than the procedural approach as seen in the classic shell scripts. This is not to say that Chef based solutions are declarative by nature, as more complex functionality can be provided by programming Ruby, and then you end up with a more procedural solution after all.

Traditionally, and still often used are using pre-defined images and launch them whenever needed. However, the big advantage of using something like Chef are:

• It is Execution environment independent. It can manage resources in the cloud, on your local network as well as on your desktop.
• It is relatively platform independent, although it takes a significant amount of effort to develop and test it.
• It allows you to use clean base images, which can be updated with the latest security patches and other stuff, while it doesn't affect the configuration management solution itself.
• Running server instances can be easily updated by running the agent again.
• Chef provides much more flexibility when it comes to combining and splitting roles. E.g. for development and test purposes you would like to have combined web and database server, while for production purposes these must be separated.

Apart from managing individual servers, Chef also provides tools for composing these individual resources to a meaningful solution. For instance, web servers need to know where to find the database server, and all servers must be configured to forward their syslog messages to a particular loghost. Quite often this information is only available at runtime, and by keeping a central repository of all nodes and their configuration data, Chef allows you to query this repository and use the results in your configuration tasks.

Is it perfect? Far from it. Customising resources and keeping them separate from 'standard' resources requires quite a bit of discipline. The command line is clunky at times and for sure Chef is a complex beast. So we will celebrate a lot of improvements in the coming years, that is for sure. But for now, it is one of the most powerful solutions in this space, and I am very happy to use it.

This concludes my 2013 'State of the Cloud Management space' mini series. One of the nicest thing about our job is that it will be probably outdated the minute I hit the Publish button. And that there will be many different and non-compatible views about this subject. Let me know!

Thoughts on Cloud Management solutions - Cloudformation

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the second of a few posts.

In the previous post, I shared my two cents on the added value of Rightscale, while in a future post I will also discuss Opscode's Chef. But now, Amazon's own Cloudformation.

Wait, we were supposed to talk about Cloud management solutions, right? And now we are going to discuss one of Amazon's features, it itself part of a much broader management solution. Yep, that's right. And I agree, Cloudformation is not a full blown cloud management solution in itself but it is a pretty darn useful component and allows you to do some powerful stuff.

So, what is then. Well, obviously Cloudformation is AWS specific and has nothing to offer in the multi-cloud area, like Rightscale does.

Amazon Web Services has grown over time to an amazing set of Cloud services, some of them overlapping, others complementary and launching a solution in AWS typically takes a few of these services to work in concert with each other. Simply launching a cloud based server typically requires a significant set of resources such as the server instance itself, storage volumes, security groups, DNS records, alarms, possibly load balancers and auto scaling configurations and so on.

Manually configuration this kind of things quickly becomes very boring, and that is where Cloudformation kicks in. It allows you to declare the kind of resources you want (if not clear enough: in a declarative way, which makes a big difference), and create this as a fully managed stack either through a web interface or command line.

An updating simply requires updating the template and applying it to an existing stack. A snippet of a security group declared in such a cloud formation template is shown below.

"WebServerSecurityGroup": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
  "GroupDescription": "Security Group for the web server instances",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"SourceSecurityGroupName": "amazon-elb-sg",
"SourceSecurityGroupOwnerId": "amazon-elb"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
  ]
  }
}

The very, very useful thing is that it allows you to treat your IaaS configuration as code, and that you don't have to deal with state. So no checking if resources already exist and based on that defining the next steps: this is taken care for you under the hoods.

Note that you have to realise that this still have to take place. Changing a resource, for instance the instance type of a server instance will result in a stopped instance (or terminated, in case of an instance store instance) and started again with the new instance type.

At best, Cloudformation is part of a full blown cloud management solution. Cloudformation focus is on describing the cloud resources you need, not on the configuration on the server instances itself. That said however, Cloudformation has some tooling to configure these servers as well, a bit of a (very) light weight Chef or Puppet kind of thing or possibly more comparable Ubuntu's cloud-init. This is useful for some not too large systems, but is typically used to bootstrap these servers with agents that take ownership for the further provisioning of these servers.

I really recommend that in case you want to use the AWS platform to take a better look at Cloudformation. It has a bit of a learning curve but it's definitely worth the investment.

There are a few things that need attention though:

• The service is not bullet proof yet. Sometimes it throws exceptions that disappear after a few hours again and in (very) rare occasions these cloudformation stacks end up in an error state that leaves you no other option that deleting the entire stack and start all over again. Ouch! 
• Cloudformation is declarative by design and has only very limited conditional logic support. This quickly leads to very long templates with quite a bit of duplication of code. There is support for included external scripts in your stack, but in practice this doesn't work too well. In my view the best way to use cloudformation script is to wrap it in a simple generator that allows you to minimise code duplication.

Thoughts on Cloud management solutions - Rightscale

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this will be the first of a few posts. In the next posts I will also talk about Cloudformation and Chef.

A few years ago I started working with Rightscale, on top of Amazon Web Services. The main selling point (at least, from my perspective) of Rightscale is the ability to be multi-cloud. AWS is obviously supported but also Rackspace and more recently players such as Azure and Google's Compute Engine are part of the deal.

The nice thing about Rightscale is that it provides ready to use, fully configured server templates, which can be configured by attaching scripts or cookbooks to it, and passing along the right parameters. Typically they operate on top of bare server images, which are available in the different clouds and configuration is done upon boot time.

I really like that model, it provides tremendous flexibility and by having a rich set of pre-defined server templates Rightscale allows you to get started very quickly.

The downside is that it mainly uses scripting (bash, ruby, powershell and sometimes chef) which is not so easy to maintain. Also the development environment to create, deploy and test these scripts are far from user friendly, which results in a relatively cumbersome experience.

The multi-cloud thing then, is excellent. That is: if you need it. If you really need to support multiple clouds, this is the way to go, but in my view most customers are perfectly served by sticking to one public cloud vendor. Regardless which one you choose, I don't believe you'll gain a lot by hopping from one vendor to another. And, glad you mentioned it, disaster recovery can be achieved using multiple (for instance AWS) regions, as they are fully decoupled by nature and don't require throwing another vendor in the mix. But if you are a product vendor, serving customers with different preferences you probably have no choice.

In case you don't need this multi-cloud thing it might really get in your way. In the end, it restricts you to the largest common denominator and since this market is so much in flux (almost 100 product announcements in 2012 for AWS) this is not the most appealing model.

Note that Rightscale does allow you to use cloud specific features (which you really need for deploying a realistic application), but then the question pops up: why not use the AWS management console straight away. These native consoles and APIs are almost always more feature rich as well.

For example, mid 2011 Amazon launched the full-blown VPC functionality as we currently know it, and we were very tempted to use it. However in order to use it we had to upgrade our Rightscale license and even then we would have access to a very limited form of VPC functionality. We couldn't use the AWS functionality itself, as RS (at that time) didn't allow us to launch a server in a particular subnet. So basically we were significantly lagging behind, which wouldn't have been the case if we would use AWS directly.

So you really have to decide whether the multi-cloud feature makes it worth sacrificing some of the flexibility you have when using the native functionality itself. In the end, it is all about portability and how much it is worth to you. It simply depends on your needs.

Never underestimate the appeal of Microsoft to Enterprises

I have been lucky to be able to work in very diverse environments, from small, highly innovative startups to large enterprises.

And while it is absolutely true that, from a technical perspective, open source solutions rule the internet, Microsoft has a very strong foothold in the corporate world.

OS technologies such as Linux, Ruby, Python, Scala, NoSQL are the foundation of almost all of these internet services and it is surprising to see how the Windows platform is almost considered as an afterthought (if considered at all), when going through technical documentation and so on. It simply is not taken into consideration, and in my view for clear, justifiable reasons.

However, the corporate world is a different beast and ruled by Microsoft. Of course, OS solutions do exist and technologies such as Java are widespread here as well. But you'll be very hard pressed to find an organisation without a Microsoft presence whatsoever.

And that gives Microsoft a (surprising, at least for me) advantage in other areas as well. When discussing potential cloud solutions I was expecting that Amazon Web Services is considered the benchmark in the IaaS area. I am not trying to say that AWS is by default the desired option, but I was expecting that their service would be recognised as the pace setter and some kind of benchmark. And Microsoft technology is certainly a first class citizen at AWS.

I was wrong. The strong relationship MS has built with these enterprises also leaves the impression that the cloud services provided by Microsoft are also some kind of safe harbour for these enterprises exploring cloud based solutions. This is not necessarily based on an objective evaluation of services, costs and service level, it is perception.

And as I have learnt a long time ago: it is the perception that matters!

Automatic, unattended install of phpmyadmin

In this brave new world of infrastructure automation, being able to install a package without manual intervention is a bare necessity.

And, how difficult can it be, installing phpmyadmin, automatically, without manual intervention.

Of course this turned out to be slightly more difficult than I thought, and as I couldn't find a real good resource on the web I decided to put my findings in a blog. Possibly more as a future reminder for myself :-)

Installing packages such as phpmyadmin on Debian/Ubuntu is usually a breeze, thanks to the apt package manager. And by passing the -y option you can send yes to all questions that may arise in this setup.

Also,  Debian (and hence also Ubuntu) has for a long time the DEBIAN_FRONTEND parameter. By setting this to noninteractive, no questions will be asked.

# export DEBIAN_FRONTEND=noninteractive

# apt-get -q -y phpmyadmin

However, what to do with questions that really need some inputs, such as asking for the database password. Welcome to debconf-set-selections.

With debconf-set-selections you basically answer the questions that will be asked in the setup before the actual install.

So by running the following command, the main questions will be answered and then the actual install will proceed without that pesky blue screen:

echo 'phpmyadmin phpmyadmin/dbconfig-install boolean true' | debconf-set-selections
echo 'phpmyadmin phpmyadmin/app-password-confirm password your-app-pwd' | debconf-set-selections
echo 'phpmyadmin phpmyadmin/mysql/admin-pass password your-admin-db-pwd' | debconf-set-selections
echo 'phpmyadmin phpmyadmin/mysql/app-pass password your-app-db-pwd' | debconf-set-selections
echo 'phpmyadmin phpmyadmin/reconfigure-webserver multiselect apache2' | debconf-set-selections

Nice, but where do these variables come from? Welll, actually there are a lot more variables to play with, and if you really want to know run the following command after the install of the package:

debconf-get-selections | grep phpmyadmin

This will return all parameters for that particular package, most of them are self-explanatory.

For completeness, the debconf-utils package is needed, but luckily that installs with only the -y parameter provided.