Widows Azure Pack/Swift Integration
Windows Azure Pack (WAP) is Microsoft’s private cloud offering. It offers a subset of the features found in Azure, Microsoft’s public cloud. What makes WAP nice for the user is that the portal and APIs are the same in both clouds. If you know how to use Azure, you’ll know how to use WAP, and vice versa.
Microsoft has also done a very nice on their control portal. Tenants can use it to subscribe to resources and monitor them. It has a clean look and is very easy to use.
At one major bank, their user community loved the simplicity of Azure and WAP, but they also wanted blob storage. Since blob storage is offered in Azure, but not WAP, OutworX undertook a project to provide blob storage in WAP.
WAP provides access points for writing a Custom Resource Provider (CRP). By writing . NET code that follows the CRP specification, you can hook any kind of resource into the the WAP portal. We decided to write a CRP for Swift, OpenStack’s blob storage.
Swift is an enterprise-ready open source blob storage system. It handles HA issues automatically, and can be set-up in Disaster Recovery configurations. Its RESTful API is fairly straightforward to use.
We integrated Swift into WAP’s Management and Tenant Portals. In the Management Portal, IT administrators can now integrate a Swift cluster by supplying the administrative credentials for Swift’s Keystone server. (Keystone is the standard identity server in OpenStack.) The credentials are stored in a database. When a tenant sings up for a storage account on that cluster, we transparently use these admin credentials to create a new Swift storage account behind the scenes on behalf of the tenant.
Both WAP and Keystone support federated identity configurations. We may implement these in the future in our integration. However, for simplicity, we chose to simply store the Keystone admin credentials in plain text in our database. This is consistent with the default configuration for OpenStack, which often lists passwords in plaintext in configuration files. Obviously production environments would require more robust security. OpenStack would need to be hardened, and our database could be protected with some form of TDE.
When a WAP user signs up for a storage account, we create a corresponding Keystone user in the OpenStack cluster. We give that Keystone user admin privileges for the Swift account, but not for Keystone. We also create a second Keystone user called Swift-Storage-Account-Name-api-user. For example, if the Azure tenant created an account called ‘mypictures’, the second user would be called ‘mypictures-api-user’.
This second *-api-user account has only normal CRUD privileges for the Swift storage account. In particular, the user cannot add or delete other users. The idea behind this second account is that its credentials can be given to programmers who need to access the storage. This lets the Azure tenant retain administrative control over the account while giving non-admin access to the programmer.
It should be noted that programmers still have to use the Swift API. While the Azure API and Swift API have large overlaps in their functionality, there are differences. One of the most significant is that Azure blob storage offers strong consistency, while Swift offers eventual consistency. In Azure, if you update an existing blob and then immediately read that blob, you’ll get the updated data. In OpenStack, sometimes you’ll get the old data since it takes a while to update things¹.
We modeled the user experience in the WAP portal on the Azure portal. As the screenshot below shows, our storage screen looks very similar to what Azure users expect.
The end result is that WAP users can sign up and manage Swift storage accounts without ever leaving the WAP portal. This gives them a familiar and easy to use environment.
Other WAP/OpenStack integrations are also possible. For example, we could use the techniques learned on this project to integrate in Ceph to give a blob storage that has strong consistency, as well as an advanced filesystem.
¹Swift actually does offer a form of strong consistency, but it is not efficient. If you set the X-Newest flag in the HTML header, it will return data from the newest replica. Because of the expense, the OpenStack documentation warns that this should only be used when absolutely necessary.