Quick guide to AWS Database options

Introduction

Amazon Web Services is widely known as the leading Infrastructure as a Service provider, and let's be clear: they are by far the most powerful option in this area.

However, it would be unwise to believe that AWS is only active in this particular IaaS space, and over time they have build up a significant number of services that can be classified as Platform as a Service.

A few weeks ago I visited the AWS summit in London and attended a few (big) data sessions, and I figured it would be helpful to list the various database platform services AWS is currently providing, and give a quick introduction and guide to how and when to use these. Note, both SQL and NoSQL option will be discussed, but it will not go into the details of these two options (not to mention the various SQL and NoSQL flavours that exist).

SimpleDB

According to AWS, Amazon SimpleDB is a highly available and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests and Amazon SimpleDB does the rest. It provides a very flexible query interface, sports multi-data center replication, high availability, and offers rock-solid durability. And yet customers never need to worry about setting up, configuring, or patching their database.

The main drawback however is that its scalability is limited and performance not always predictable. It also has strict storage limitations of 10GB hence partitioning is probably required for larger systems.

So when you need the flexibility of a non-relational database and your scalability requirements are modest, this could be a good option.

DynamoDB

DynamoDB is AWS' other NoSQL database offering. It is a tremendously scalable and fast service and provisioning and scaling of databases (tables) is straight forward. The underlying technical architecture is state of the art, for instance it runs on SSD rather than good old hard drives. It integrates nicely with other AWS features such as S3, Data Pipeline and RedShift.

So when you are in the NoSQL game and really need high scalable low latency solution, DynamoDB is the way to go.

RDS

Relational databases are known territory for most of us, and the likes of Oracle, SQL Server and MySQL are big players in this area. It is perfectly possible to run your own (relational) database on AWS EC2 but if you don't want to be bothered with configuration issues such as storage configuration, backup schedules and read-write copies, RDS is certainly an option. Even more that the extra costs of RDS over running your own database on EC2 is very modest.

So realising that NoSQL, although a nice and shiny option is certainly not always the preferred option, AWS' RDS option is certainly worth a look at. Especially for the transactional processing systems as for the analytical processing AWS has...

RedShift

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools. It is optimized for datasets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
RedShift is accessible through a standard ODBC or JDBC API as it implements the PostgreSQL interface (but note: it is not PostgreSQL under the hood!).

So when large amounts of data must be analysed and (in real-time fashion) reported on, RedShift is a very attractive option.

EMR

AWS' Elastic MapReduce option is technically not really a database, but is still very applicable for processing large amounts of data. EMR is a fully configured Hadoop setup, including tools such as Pig and Hive. It is tremendously powerful although also a bit low level, and typically operates in a batch-oriented (rather than real-time) fashion.

EMR is a very good option for processing incoming data streams and move the results to other database options such as discussed before in this post.

Conclusion

This is only the tip of the iceberg and an in-depth comparison of the various options takes a lot more space. However, I do hope it provides a first impression and guidance when having to choose between the various options

Note the options discussed are really platform services, so a lot of the complexity is abstracted away from you. You may or may not like that, if you are in the latter camp rest assured that you can still use the IaaS options and run your own database solution just the way you want it utilising services such as EC2 and EBS.

AWS making strides again

Amazon Web Services is pushing out one update/innovation/product after another, last year more than 100 announcements were made. That is two a week! Some announcements are pretty small, while others are short of groundbreaking.

In the last few weeks, two announcements were made that caught my eye and showed (in my opinion) that these guys are still on another planet. While the competition is struggling to provide the basic infrastructure services, AWS is providing some very advanced stuff.

First of all: OpsWorks is really cool. If you've read some of my previous blogs, then you'll know that I am skeptical about the benefits of 3rd party cloud management tools. Not because I feel we don't need such tools, but because the standard tools (of in this case AWS) are very powerful by themselves.

With the launch of OpsWorks, basically the powerful Chef framework embedded in the AWS Console, this has become even more true. Until recently, the AWS console has focused on managing the infrastructure, with OpsWorks they make a move into managing the server configuration as well.

Sure, Opsworks is still limited and does not support too many dfferent platforms. But it clearly shows the direction AWS is taking and this is light years ahead of the competition. Impressive.

The other thing that I found noticeable is smaller but still as significant  The introduction of the CloudHSM. A HSM (hardware security module) is not a commodity in a typical application architecture, but in security sensitive solutions this is where the confidential stuff happens/resides. The HSM protects this by means of software but also hardware. For instance, it typically comes with tamper resistant housing, that deletes all sensitive stuff the minute someone starts fiddling with the housing. 

What is the number one reason cited for NOT moving to the cloud? Indeed, that's security. And by providing a managed HSM service the options in this regard are much improved.

Interesting times indeed.

Thoughts on Cloud Management Solutions - Chef

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the third, and for now last of a few posts.

In the first post, I shared my two cents on the added value of Rightscale, while in the second post I discussed Amazon's own Cloudformation. Now (a bit later than planned due to personal reasons), another piece of the puzzle, which is Opscode's Chef configuration management framework.

To make things clear from the start, I am a big Chef fan. It is a hugely powerful framework to manage a (large) set of infrastructure components and provide sufficient abstraction to achieve high productivity and enough flexibility when and where it matters.

But one step back. This is supposed to be about cloud management solutions, and while Chef has been designed with Cloud in mind, it is not necessarily a cloud management solution. It is perfectly suitable in a more traditional environment, either with virtualised or physical servers.

And in comparison with the likes of Rightscale, this is no comparison. With Rightscale you are able to define your cloud environment, such as which servers, networks, scaling algorithmes and so on, and Chef is not really focused on this. Sure, it can do certain things, but it really excels in combination with other tools that take care of the management of cloud resources. So a combination of Chef + Rightscale, Chef + Cloudformation (my preferred choice) or for that matter even combinations like Chef + Vagrant.

In the Management Stack, Chef (and for that matter its counterparts like Puppet or CFEngine) has a strong focus on the upper two layers, Server Provisioning and Service Orchestration.

Chef provides strong concepts like Roles, Environments and Data bags to design and implement a clean and maintainable configuration management solution. The primitives as provided by the framework allow for a much more declarative approach rather than the procedural approach as seen in the classic shell scripts. This is not to say that Chef based solutions are declarative by nature, as more complex functionality can be provided by programming Ruby, and then you end up with a more procedural solution after all.

Traditionally, and still often used are using pre-defined images and launch them whenever needed. However, the big advantage of using something like Chef are:

• It is Execution environment independent. It can manage resources in the cloud, on your local network as well as on your desktop.
• It is relatively platform independent, although it takes a significant amount of effort to develop and test it.
• It allows you to use clean base images, which can be updated with the latest security patches and other stuff, while it doesn't affect the configuration management solution itself.
• Running server instances can be easily updated by running the agent again.
• Chef provides much more flexibility when it comes to combining and splitting roles. E.g. for development and test purposes you would like to have combined web and database server, while for production purposes these must be separated.

Apart from managing individual servers, Chef also provides tools for composing these individual resources to a meaningful solution. For instance, web servers need to know where to find the database server, and all servers must be configured to forward their syslog messages to a particular loghost. Quite often this information is only available at runtime, and by keeping a central repository of all nodes and their configuration data, Chef allows you to query this repository and use the results in your configuration tasks.

Is it perfect? Far from it. Customising resources and keeping them separate from 'standard' resources requires quite a bit of discipline. The command line is clunky at times and for sure Chef is a complex beast. So we will celebrate a lot of improvements in the coming years, that is for sure. But for now, it is one of the most powerful solutions in this space, and I am very happy to use it.

This concludes my 2013 'State of the Cloud Management space' mini series. One of the nicest thing about our job is that it will be probably outdated the minute I hit the Publish button. And that there will be many different and non-compatible views about this subject. Let me know!

Thoughts on Cloud Management solutions - Cloudformation

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the second of a few posts.

In the previous post, I shared my two cents on the added value of Rightscale, while in a future post I will also discuss Opscode's Chef. But now, Amazon's own Cloudformation.

Wait, we were supposed to talk about Cloud management solutions, right? And now we are going to discuss one of Amazon's features, it itself part of a much broader management solution. Yep, that's right. And I agree, Cloudformation is not a full blown cloud management solution in itself but it is a pretty darn useful component and allows you to do some powerful stuff.

So, what is then. Well, obviously Cloudformation is AWS specific and has nothing to offer in the multi-cloud area, like Rightscale does.

Amazon Web Services has grown over time to an amazing set of Cloud services, some of them overlapping, others complementary and launching a solution in AWS typically takes a few of these services to work in concert with each other. Simply launching a cloud based server typically requires a significant set of resources such as the server instance itself, storage volumes, security groups, DNS records, alarms, possibly load balancers and auto scaling configurations and so on.

Manually configuration this kind of things quickly becomes very boring, and that is where Cloudformation kicks in. It allows you to declare the kind of resources you want (if not clear enough: in a declarative way, which makes a big difference), and create this as a fully managed stack either through a web interface or command line.

An updating simply requires updating the template and applying it to an existing stack. A snippet of a security group declared in such a cloud formation template is shown below.

"WebServerSecurityGroup": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
  "GroupDescription": "Security Group for the web server instances",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"SourceSecurityGroupName": "amazon-elb-sg",
"SourceSecurityGroupOwnerId": "amazon-elb"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
  ]
  }
}

The very, very useful thing is that it allows you to treat your IaaS configuration as code, and that you don't have to deal with state. So no checking if resources already exist and based on that defining the next steps: this is taken care for you under the hoods.

Note that you have to realise that this still have to take place. Changing a resource, for instance the instance type of a server instance will result in a stopped instance (or terminated, in case of an instance store instance) and started again with the new instance type.

At best, Cloudformation is part of a full blown cloud management solution. Cloudformation focus is on describing the cloud resources you need, not on the configuration on the server instances itself. That said however, Cloudformation has some tooling to configure these servers as well, a bit of a (very) light weight Chef or Puppet kind of thing or possibly more comparable Ubuntu's cloud-init. This is useful for some not too large systems, but is typically used to bootstrap these servers with agents that take ownership for the further provisioning of these servers.

I really recommend that in case you want to use the AWS platform to take a better look at Cloudformation. It has a bit of a learning curve but it's definitely worth the investment.

There are a few things that need attention though:

• The service is not bullet proof yet. Sometimes it throws exceptions that disappear after a few hours again and in (very) rare occasions these cloudformation stacks end up in an error state that leaves you no other option that deleting the entire stack and start all over again. Ouch! 
• Cloudformation is declarative by design and has only very limited conditional logic support. This quickly leads to very long templates with quite a bit of duplication of code. There is support for included external scripts in your stack, but in practice this doesn't work too well. In my view the best way to use cloudformation script is to wrap it in a simple generator that allows you to minimise code duplication.

Thoughts on Cloud management solutions - Rightscale

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this will be the first of a few posts. In the next posts I will also talk about Cloudformation and Chef.

A few years ago I started working with Rightscale, on top of Amazon Web Services. The main selling point (at least, from my perspective) of Rightscale is the ability to be multi-cloud. AWS is obviously supported but also Rackspace and more recently players such as Azure and Google's Compute Engine are part of the deal.

The nice thing about Rightscale is that it provides ready to use, fully configured server templates, which can be configured by attaching scripts or cookbooks to it, and passing along the right parameters. Typically they operate on top of bare server images, which are available in the different clouds and configuration is done upon boot time.

I really like that model, it provides tremendous flexibility and by having a rich set of pre-defined server templates Rightscale allows you to get started very quickly.

The downside is that it mainly uses scripting (bash, ruby, powershell and sometimes chef) which is not so easy to maintain. Also the development environment to create, deploy and test these scripts are far from user friendly, which results in a relatively cumbersome experience.

The multi-cloud thing then, is excellent. That is: if you need it. If you really need to support multiple clouds, this is the way to go, but in my view most customers are perfectly served by sticking to one public cloud vendor. Regardless which one you choose, I don't believe you'll gain a lot by hopping from one vendor to another. And, glad you mentioned it, disaster recovery can be achieved using multiple (for instance AWS) regions, as they are fully decoupled by nature and don't require throwing another vendor in the mix. But if you are a product vendor, serving customers with different preferences you probably have no choice.

In case you don't need this multi-cloud thing it might really get in your way. In the end, it restricts you to the largest common denominator and since this market is so much in flux (almost 100 product announcements in 2012 for AWS) this is not the most appealing model.

Note that Rightscale does allow you to use cloud specific features (which you really need for deploying a realistic application), but then the question pops up: why not use the AWS management console straight away. These native consoles and APIs are almost always more feature rich as well.

For example, mid 2011 Amazon launched the full-blown VPC functionality as we currently know it, and we were very tempted to use it. However in order to use it we had to upgrade our Rightscale license and even then we would have access to a very limited form of VPC functionality. We couldn't use the AWS functionality itself, as RS (at that time) didn't allow us to launch a server in a particular subnet. So basically we were significantly lagging behind, which wouldn't have been the case if we would use AWS directly.

So you really have to decide whether the multi-cloud feature makes it worth sacrificing some of the flexibility you have when using the native functionality itself. In the end, it is all about portability and how much it is worth to you. It simply depends on your needs.