Is the Open Group's TOSCA the future for AWS Cloudformation?

I stumbled into TOG's TOSCA initiative (Tosca also being the name of my parent's dog, hence the picture :-), a standardisation initiative that aims at "Enhancing the portability and management of cloud applications and services across their lifecycle".

According to the first sentences in the spec, "This core TOSCA specification provides a language to describe service components and their relationships using a service topology, and it provides for describing the management procedures that create or modify services using orchestration processes. The combination of topology and orchestration in a Service Template describes what is needed to be preserved across deployments in different environments to enable interoperable deployment of cloud services and their management throughout the complete lifecycle (e.g. scaling, patching, monitoring, etc.) when the applications are ported over alternative cloud environments."

In principle this is a good thing. I believe the future is in Infrastructure-as-Code and although this is typically associated with automation efforts as done with Chef and Puppet, I personally feel it is even better demonstrated by services such as AWS' Cloudformation.

Cloudformation allows you to declare which infrastructural components (server instances, load balancers, storage components, dns registrations etc.) you want to instantiate, including their configurations and inter-relationships. It typically operates on infrastructural level, and although it has some server configuration features (through cfn-init) it typically operates in concert with tools such as the mentioned Chef, Puppet or alternatives.

Cloudformation is great. It is massively frustrating to develop complex systems in, given to the immaturity of the development ecosystem around it (which will be resolved over time), but still I am not aware of an alternative provided by the other cloud providers that comes close.

This TOSCA standard acknowledges the importance of such provisioning tools, and aims at getting a standard in place. With the promise of preserving portability and avoid (cloud vendor) lock in. So that's great isn't it?

Yes, potentially yes. I am very happy with AWS and I really think they are by far the most feature-rich cloud provider out there, but it's always good to preserve portability. For instance, someone could have particular data requirements that simply cannot be met by AWS (e.g. data cannot leave the country), or has some kind of (Microsoft anyone?) enterprise agreement that they want to leverage in the cloud. Having a solid standard in this space could facilitate that, and could also boost the development of development tools.

The reality of course is that cloud landscape is evolving at lightning speed, and that standardisations body typically have no possiblity to keep up with that. Looking at the differences in services portfolio between cloud vendors (with AWS being the undisputed front runner in both current portfolio and development speed) it's a damn hard job to do standardisation there. Standards like this run the risk to only support the lowest common denominator, which is something not a lot of people are waiting for I'm afraid.

So I am bit torn. I must admit that I haven't studied the standard in detail yet, so maybe a lot of my concerns are already addressed, but my gut feeling says it will take a while before the (leading) cloud vendors adopt this.

How to avoid circular dependencies between Cloudformation stacks

I am currently designing a cloud based solution based on AWS, Cloudformation and Chef, and using the concepts as described in an earlier post.

In short, I am a big fan of Cloudformation which is a perfect example of Infrastructure as Code, but developing and maintaining Cloudformation scripts and stack is a bit of a cumbersome process. In order to keep that manageable, we need to adopt good old design principles such as low cohesion/high coupling when deciding what to put in a particular template/stack.

Let’s assume two Cloudformation templates, one containing all database server related resources, while the other contains all web server stuff. Forget all other stuff, and assume each stack just have a server instance with a security group.

The Web server requires the database server to be present to do the provisioning of the application, while the database server security group requires the Web server security group to be created in order to provide access. Bang! Circular dependency, which is always a bad thing!

Of course, this can be solved, and the following options exists:

Forget about modularisation, put everything in one template. Of course this works, and in this simple example most likely the most pragmatic approach also. But in a real life setup, things are not as simple and you might have a multitude of service stacks having inter-dependencies. And putting these all together in one script will quickly become a head ache.

First create all security groups, then the server instances. Well, that will work as well, and is a relatively clean solution too. Few things I consider a drawback are:

• It introduces another template (and hence stack);
• It also introduces additional dependencies between these stacks, which must be catered for by means of input parameters;
• The definition of the SG, which is tightly related to the server is removed from the service stack.

The option I went for, was introducing a dedicated security group in one stack, and use it for identification purposes in another. This is shown below.

In the database stack I create an additional security group (WS-ID) which has an empty definition, and I only use it to allow the DB Security Group to provide access to it.

In the web server stack, I provide a reference to this security group (through an input param) and by this allows clean access to the database server. Basically you are using the WS-ID security group for identification purposes only.

Hence I have removed the circular dependency, kept the number of stacks the same and have limited the dependency between stacks to a minimum. In this example, there is only one dependency, but in reality there might be many more. 

These software design patterns prove to be useful even when designing infrastructure he!

Update: there is a significant drawback in this approach, as you cannot terminate and rebuild the DB Server stack without terminating the web server stack. To cope with that, you can create the WS-ID Security Group in a separate stack, and feeding the reference into both the DB Server Stack as well as the Web Server stack.

Why the headache of License Management could be a major push towards cloud adoption

I've been busy lately with various software license related assignments, not necessarily related to a cloud setup. 

And quickly I recognized all the headaches and frustration that comes with it, which are even more apparent when being active in cloud environments for the last decade or so.

Let's face it. Licenses are a major headache. Understanding the rules is ridiculously complex, and change all the time. And most of the time, these changes are not to lower your fees.

Strong software asset management discipline is a necessity, and while I will be the last one to claim that cloud will solve all these problems and make them redundant, it definitely will help in some cases.

For instance, one customer has an application (provided as a SaaS) with a very peaky behaviour. One week per month it needs massive power, the other three weeks it just cruises. Buying licenses for the peak capacity would be a very costly exercise, even more so given the SaaS characteristic of the service. You simply wants to scale with demand, both periodically as well as the growing customers, and traditional licensing models simply cope very badly in such scenarios. Cloud computing, in case licenses are included proves to be a major attraction to this types of customers.

Another customers needed to renew their enterprise agreement with another software vendor, and needed to make an inventory of what software is used, over time, and in what configurations. Should be simple enough, right? Not so, keeping track of everything, understanding all the rules and exceptions is close to impossible. Comparing that with ease in which you can launch enterprise applications in the cloud, and getting charged for using both hardware and software by the hour (or even minute), it is clear that there is something to be gained here.

Understand me correctly, I do not feel the cloud is a silver bullet that solves all such licensing issues, but the matter of fact is that it comes inherent with some features that makes life in this regard a lot easier. And in these two cases cloud computing was mentioned by the customer as an attractive setup, mainly because of the licensing restrictions.

I am convinced that this will prove to be a major battle ground, and by exploiting this cloud providers have a great opportunity to lure traditional enterprises to use their services.

Quick guide to AWS Database options

Introduction

Amazon Web Services is widely known as the leading Infrastructure as a Service provider, and let's be clear: they are by far the most powerful option in this area.

However, it would be unwise to believe that AWS is only active in this particular IaaS space, and over time they have build up a significant number of services that can be classified as Platform as a Service.

A few weeks ago I visited the AWS summit in London and attended a few (big) data sessions, and I figured it would be helpful to list the various database platform services AWS is currently providing, and give a quick introduction and guide to how and when to use these. Note, both SQL and NoSQL option will be discussed, but it will not go into the details of these two options (not to mention the various SQL and NoSQL flavours that exist).

SimpleDB

According to AWS, Amazon SimpleDB is a highly available and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests and Amazon SimpleDB does the rest. It provides a very flexible query interface, sports multi-data center replication, high availability, and offers rock-solid durability. And yet customers never need to worry about setting up, configuring, or patching their database.

The main drawback however is that its scalability is limited and performance not always predictable. It also has strict storage limitations of 10GB hence partitioning is probably required for larger systems.

So when you need the flexibility of a non-relational database and your scalability requirements are modest, this could be a good option.

DynamoDB

DynamoDB is AWS' other NoSQL database offering. It is a tremendously scalable and fast service and provisioning and scaling of databases (tables) is straight forward. The underlying technical architecture is state of the art, for instance it runs on SSD rather than good old hard drives. It integrates nicely with other AWS features such as S3, Data Pipeline and RedShift.

So when you are in the NoSQL game and really need high scalable low latency solution, DynamoDB is the way to go.

RDS

Relational databases are known territory for most of us, and the likes of Oracle, SQL Server and MySQL are big players in this area. It is perfectly possible to run your own (relational) database on AWS EC2 but if you don't want to be bothered with configuration issues such as storage configuration, backup schedules and read-write copies, RDS is certainly an option. Even more that the extra costs of RDS over running your own database on EC2 is very modest.

So realising that NoSQL, although a nice and shiny option is certainly not always the preferred option, AWS' RDS option is certainly worth a look at. Especially for the transactional processing systems as for the analytical processing AWS has...

RedShift

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools. It is optimized for datasets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
RedShift is accessible through a standard ODBC or JDBC API as it implements the PostgreSQL interface (but note: it is not PostgreSQL under the hood!).

So when large amounts of data must be analysed and (in real-time fashion) reported on, RedShift is a very attractive option.

EMR

AWS' Elastic MapReduce option is technically not really a database, but is still very applicable for processing large amounts of data. EMR is a fully configured Hadoop setup, including tools such as Pig and Hive. It is tremendously powerful although also a bit low level, and typically operates in a batch-oriented (rather than real-time) fashion.

EMR is a very good option for processing incoming data streams and move the results to other database options such as discussed before in this post.

Conclusion

This is only the tip of the iceberg and an in-depth comparison of the various options takes a lot more space. However, I do hope it provides a first impression and guidance when having to choose between the various options

Note the options discussed are really platform services, so a lot of the complexity is abstracted away from you. You may or may not like that, if you are in the latter camp rest assured that you can still use the IaaS options and run your own database solution just the way you want it utilising services such as EC2 and EBS.

AWS making strides again

Amazon Web Services is pushing out one update/innovation/product after another, last year more than 100 announcements were made. That is two a week! Some announcements are pretty small, while others are short of groundbreaking.

In the last few weeks, two announcements were made that caught my eye and showed (in my opinion) that these guys are still on another planet. While the competition is struggling to provide the basic infrastructure services, AWS is providing some very advanced stuff.

First of all: OpsWorks is really cool. If you've read some of my previous blogs, then you'll know that I am skeptical about the benefits of 3rd party cloud management tools. Not because I feel we don't need such tools, but because the standard tools (of in this case AWS) are very powerful by themselves.

With the launch of OpsWorks, basically the powerful Chef framework embedded in the AWS Console, this has become even more true. Until recently, the AWS console has focused on managing the infrastructure, with OpsWorks they make a move into managing the server configuration as well.

Sure, Opsworks is still limited and does not support too many dfferent platforms. But it clearly shows the direction AWS is taking and this is light years ahead of the competition. Impressive.

The other thing that I found noticeable is smaller but still as significant  The introduction of the CloudHSM. A HSM (hardware security module) is not a commodity in a typical application architecture, but in security sensitive solutions this is where the confidential stuff happens/resides. The HSM protects this by means of software but also hardware. For instance, it typically comes with tamper resistant housing, that deletes all sensitive stuff the minute someone starts fiddling with the housing. 

What is the number one reason cited for NOT moving to the cloud? Indeed, that's security. And by providing a managed HSM service the options in this regard are much improved.

Interesting times indeed.

Thoughts on Cloud Management Solutions - Chef

I have been lucky to do some serious work with several cloud management solutions in the last couple of years. I thought it would be useful to put my thoughts on paper and this is the third, and for now last of a few posts.

In the first post, I shared my two cents on the added value of Rightscale, while in the second post I discussed Amazon's own Cloudformation. Now (a bit later than planned due to personal reasons), another piece of the puzzle, which is Opscode's Chef configuration management framework.

To make things clear from the start, I am a big Chef fan. It is a hugely powerful framework to manage a (large) set of infrastructure components and provide sufficient abstraction to achieve high productivity and enough flexibility when and where it matters.

But one step back. This is supposed to be about cloud management solutions, and while Chef has been designed with Cloud in mind, it is not necessarily a cloud management solution. It is perfectly suitable in a more traditional environment, either with virtualised or physical servers.

And in comparison with the likes of Rightscale, this is no comparison. With Rightscale you are able to define your cloud environment, such as which servers, networks, scaling algorithmes and so on, and Chef is not really focused on this. Sure, it can do certain things, but it really excels in combination with other tools that take care of the management of cloud resources. So a combination of Chef + Rightscale, Chef + Cloudformation (my preferred choice) or for that matter even combinations like Chef + Vagrant.

In the Management Stack, Chef (and for that matter its counterparts like Puppet or CFEngine) has a strong focus on the upper two layers, Server Provisioning and Service Orchestration.

Chef provides strong concepts like Roles, Environments and Data bags to design and implement a clean and maintainable configuration management solution. The primitives as provided by the framework allow for a much more declarative approach rather than the procedural approach as seen in the classic shell scripts. This is not to say that Chef based solutions are declarative by nature, as more complex functionality can be provided by programming Ruby, and then you end up with a more procedural solution after all.

Traditionally, and still often used are using pre-defined images and launch them whenever needed. However, the big advantage of using something like Chef are:

• It is Execution environment independent. It can manage resources in the cloud, on your local network as well as on your desktop.
• It is relatively platform independent, although it takes a significant amount of effort to develop and test it.
• It allows you to use clean base images, which can be updated with the latest security patches and other stuff, while it doesn't affect the configuration management solution itself.
• Running server instances can be easily updated by running the agent again.
• Chef provides much more flexibility when it comes to combining and splitting roles. E.g. for development and test purposes you would like to have combined web and database server, while for production purposes these must be separated.

Apart from managing individual servers, Chef also provides tools for composing these individual resources to a meaningful solution. For instance, web servers need to know where to find the database server, and all servers must be configured to forward their syslog messages to a particular loghost. Quite often this information is only available at runtime, and by keeping a central repository of all nodes and their configuration data, Chef allows you to query this repository and use the results in your configuration tasks.

Is it perfect? Far from it. Customising resources and keeping them separate from 'standard' resources requires quite a bit of discipline. The command line is clunky at times and for sure Chef is a complex beast. So we will celebrate a lot of improvements in the coming years, that is for sure. But for now, it is one of the most powerful solutions in this space, and I am very happy to use it.

This concludes my 2013 'State of the Cloud Management space' mini series. One of the nicest thing about our job is that it will be probably outdated the minute I hit the Publish button. And that there will be many different and non-compatible views about this subject. Let me know!