How to avoid circular dependencies between Cloudformation stacks

I am currently designing a cloud based solution based on AWS, Cloudformation and Chef, and using the concepts as described in an earlier post.

In short, I am a big fan of Cloudformation which is a perfect example of Infrastructure as Code, but developing and maintaining Cloudformation scripts and stack is a bit of a cumbersome process. In order to keep that manageable, we need to adopt good old design principles such as low cohesion/high coupling when deciding what to put in a particular template/stack.

Let’s assume two Cloudformation templates, one containing all database server related resources, while the other contains all web server stuff. Forget all other stuff, and assume each stack just have a server instance with a security group.

The Web server requires the database server to be present to do the provisioning of the application, while the database server security group requires the Web server security group to be created in order to provide access. Bang! Circular dependency, which is always a bad thing!

Of course, this can be solved, and the following options exists:

Forget about modularisation, put everything in one template. Of course this works, and in this simple example most likely the most pragmatic approach also. But in a real life setup, things are not as simple and you might have a multitude of service stacks having inter-dependencies. And putting these all together in one script will quickly become a head ache.

First create all security groups, then the server instances. Well, that will work as well, and is a relatively clean solution too. Few things I consider a drawback are:

• It introduces another template (and hence stack);
• It also introduces additional dependencies between these stacks, which must be catered for by means of input parameters;
• The definition of the SG, which is tightly related to the server is removed from the service stack.

The option I went for, was introducing a dedicated security group in one stack, and use it for identification purposes in another. This is shown below.

In the database stack I create an additional security group (WS-ID) which has an empty definition, and I only use it to allow the DB Security Group to provide access to it.

In the web server stack, I provide a reference to this security group (through an input param) and by this allows clean access to the database server. Basically you are using the WS-ID security group for identification purposes only.

Hence I have removed the circular dependency, kept the number of stacks the same and have limited the dependency between stacks to a minimum. In this example, there is only one dependency, but in reality there might be many more. 

These software design patterns prove to be useful even when designing infrastructure he!

Update: there is a significant drawback in this approach, as you cannot terminate and rebuild the DB Server stack without terminating the web server stack. To cope with that, you can create the WS-ID Security Group in a separate stack, and feeding the reference into both the DB Server Stack as well as the Web Server stack.