It is exciting to be a developer these days. You’re the talk of the town and everyone looks for ways to make you happy, competing to create new tools or new services just for you. Developing a great app in such a mesh of services shouldn’t be an issue. Still, this often proves to be a daunting task. Just like the last time you wished to deploy in the cloud and there were 15+ various different services you had to consider.
– OK, so what is the problem? Having many alternatives is good … Right?
Well, that depends…. Here’s an example…
Let’s take the app we just talked about. To develop an app for cloud “X” you will need to download the SDK or Toolkit of that cloud and start writing code to use it. But what happens if you wish to switch to another provider? In case you want to change to cloud “Y”, you will need to download the toolkit for the new cloud. Cloud “Y”’s toolkit in turn has its own structure, syntax and design.
What can you do?
1. Stay on the same cloud…
2. Switch to another cloud. That is: Remove the old cloud code; download the toolkit of the new cloud; understand how it works; learn its particulars and re-design your app around it. Quite complicated, isn’t it?
It is clear that switching to another cloud provider is becoming difficult. Should you stay or should you go? This problem can be solved by two ways: standardization or 3rd party tools.
There are currently two organizations competing to come up with a standard for all cloud providers. These two standards are the CIMI / Cloud Infrastructure Management Interface and the OCCI / Open Cloud Computing Interface. The problem is that neither CIMI or OCCI are very widely used – instead of standardising, providers are increasingly coming up with their own APIs. Which leads to the 3rd party tools solution.
Cloud providers offer REST APIs for their services, but these often differ at the details. Cloud libraries or multi-cloud toolkits try to fix this by offering an abstraction layer over these services, allowing interaction with various cloud providers. They hide away their differences, provide one view and a unified API and eventually make moving between cloud providers easier. With their help, you can avoid getting stuck with one vendor. You only have to download the toolkit and write the code to use its API, which in turn will communicate with the cloud services. To switch to another cloud, you will not need to download another SDK or toolkit. You simply need to write some code (which is what you do anyway) to “talk” to both clouds.
What is out there?
There are several open-source, multi-cloud toolkits that you can use in your projects. Which to select largely depends on your coding language, cloud provider and service you need to work with. There are 4 main types of services supported by these toolkits:
- Compute nodes (Start / Stop instances or configure)
- Data Volumes (Creating / Attaching / Delete volumes, snapshots etc.)
- Load Balancers
- DNS
Note that not all providers are supported or willing to hand you over all the above four functionalities. To choose therefore the tool that best fits your needs, you need to do some homework.
To understand why you need to do any homework, let’s consider an example.
Assume you want to deploy and integrate your application and you only have to select between Amazon AWS and Microsoft Azure. Which are the best tools for you?
Apache Libcloud – Python
In 2009, Cloudkick created a SaaS cloud monitoring and management platform as a means of “talking” to many different providers. That early version of Cloudkick was later released as an open source Apache project and was finally acquired by Rackspace and replaced by Rackspace Cloud Monitoring.
Libcloud is a library that abstracts the differences between cloud providers. It allows you to manage different cloud resources through a unified and easy API. It is an open source project with a single Python interface of 35+ cloud providers and their services. Below is a list of the features and supported providers:
| SERVICE | AWS | Azure | |
|---|---|---|---|
| Compute | 40+ | YES | |
| Block Storage | 17+ | YES | |
| Cloud Object Storage | 8+ | YES | YES |
| CDN | 6+ | ||
| Load Balancers as a Service | 7+ | YES | |
| DNS | 6+ | YES |
Libcloud is the de facto choice for devs into Python and is supported by a great community. The documentation on their website is pretty good.
Let’s see some community stats:
- Stackoverflow questions: 160
- Github repositories 176
- Most Github Stars (on single project) 518
- Github Code results 2.2M
- Google search (name + “libcloud”) 80K
Notice that Libcloud does not fully support Azure. For the time being it only supports Azure Blob Storages plus many of the AWS methods.
More about Libcloud at:
Note 1: If you exclusively work with AWS you may want to check Boto < https://github.com/boto/boto>. It is a Python library for interacting with AWS APIs which allows you to build tools in Python, change and manage your AWS resources.
The project is led by an Amazon Engineer.
Note 2: If you feel adventurous, take a look at Libcloud REST, a Google Summer of Code 2012project which exposes most of the Libcloud functionality over HTTP.
Apache Jclouds – Java, Clojure
Jclouds, now one of the most popular solutions, started off as a project from Adrian Cole who later joined the Apache incubator. Adrian Cole is considered an active community leader in the cloud interoperability, REST and DevOps space. His current title is “cloud guy” at Twitter. Jclouds is an open source library that allows you to use portable abstractions or cloud-specific features and supports Java and Clojure.
Let’s see what kind of cloud resources Jclouds can manage:
| SERVICE | AWS | Azure | |
|---|---|---|---|
| Compute | 17+ | YES | |
| Block Storage | 7+ | YES | YES |
| Cloud Object Storage | |||
| CDN | |||
| Load Balancers as a Service | 2+ | YES | |
| DNS |
Jclouds, being a Java toolkit, naturally relies on a great community to support it. They also have excellent documentation on their website.
Here are the community statistics:
- Stackoverflow questions 26
- Github repositories 52
- Most Github Stars (on single project) 531
- Github Code results 7K
- Google search (name + “jclouds”) 33K
While Jclouds supports some of the AWS services, it does very little on the Microsoft Azure side. In fact, it does not go beyond Azure Blob.
Note: There seem to be some early Azure implementations in the Jclouds labs. If interested, take a look or even contribute to their project.
More about Jclouds at:
Fog – Ruby
Wesley Beary, currently working at Heroku, is a great Rubist and the creator of Fog back when he was at Engine Yard. Fog is a cloud provider agnostic toolkit, therefore it supports a variety of cloud providers. The Fog gem is the most popular cloud SDK in the Ruby world and it is heavily used with over 4M downloads and a large and active community contributing to its development. It is an MIT-licensed open-source community project and is also used as basis for other Ruby gems like Chef.
| SERVICE | AWS | Azure | |
|---|---|---|---|
| Compute | 25+ | YES | |
| Block Storage | 9+ | YES | |
| Cloud Object Storage | |||
| CDN | |||
| Load Balancers as a Service | 2+ | YES | |
| DNS | 10+ | YES |
You can find many tutorials about Fog and its implementation online and adequate documentation on their website.
- Stackoverflow questions 279
- Github repositories 188
- Most Github Stars (on single project) 3230
- Github Code results 305K
- Google search (name + “fog”) 565K
While currently Fog supports some of the major AWS services, it supports nothing on Azure! There are various discussions around the net about whether Fog will eventually go that way but, for the time being, no ongoing relative efforts exist other than some sparse GitHub repositories. Thus, if you are a Rubist and Azure user, the official SDK is a one-way option.
More about Fog at:
Apache deltacloud
RedHat wanted to develop an open cloud, vendor neutral so they started Deltacloud in 2009 and later joined the Apache incubator. For all you Rubists outhere, Deltacloud was written in Sinatra.
While Deltacloud is not the only open source project to offer cloud abstraction, it has a remarkable difference: Libcloud and Python, Jclouds and Java, Fog and Ruby are programming language-specific libraries. Deltacloud on the other hand is independent of the programming language and can also be used as web service. It provides three different APIs for interacting with Cloud service providers (a RESTful API, a CIMI-compliant API and a proprietary API) and is backward compatible across different versions of support cloud platforms.
Let’s see what kind of cloud resources and providers it supports:
| SERVICE | AWS | Azure | |
|---|---|---|---|
| Compute | 15+ | YES | |
| Block Storage | 15+ | YES | YES |
| Cloud Object Storage | |||
| CDN | |||
| Load Balancers as a Service | |||
| DNS |
Apache Deltacloud has an active community of programmers who constantly update it. You may find adequate info both in their website and online to get you started.
- Stackoverflow questions 10
- Github repositories 37
- Most Github Stars (on single project) 80
- Github Code results 34K
- Google search (name + “deltacloud”) 23K
More about Deltacloud at:
…and 2 bonus tools
1) Do you like to play with node.js? Check pkgcloud! It is a standard library for node.js that abstracts away the differences among multiple cloud providers. It supports the usual Cloud resources and providers. Interestingly, it supports Azure Compute resources which is not the case in any of the above toolkits.
2) If you are the Erlang type of dev take a look at elibloud. An Erlang wrapper around Libcloud.
Community Statistics Comparison
Below you will find two charts: a comparison of the number of cloud providers supported by each of the above toolkits and some community statistics.
Conclusion
Since 2006 the number of cloud providers has been steadily growing, bringing more services in the game. This is good news for developers, as more options become available to choose from. Using tools that provide a single interface to work with all these services can prove of great value to you, facilitating a potential transition from one provider to another. Still, good research prior to choosing a toolkit is essential. In our example, it becomes clear that Azure support from the toolkits presented is problematic, which is not the case for AWS (there could be many reasons for that, some originating in the very nature and definition of the Azure API). It seems that the development of toolkits is still a work in progress and things will soon start moving. Support of all major cloud providers is only a matter of time – and a game changer!
Do you use any kind of multi-cloud toolkits?