How to run OpenStack on AWS

OpenStack is a cloud operating system that controls vast computing resources through a data centre, while AWS (Amazon Web services) offers reliable, scalable and inexpensive cloud computing services. The installation of OpenStack on AWS is an instance of Cloud-on-a-Cloud.  

This article will guide you on installing OpenStack on top of AWS EC2. Installing OpenStack on a nested hypervisor environment is not a big deal when a QEMU emulator is used for launching virtual machines inside the virtual machine (VM). However, unlike the usual nested hypervisor set-up, installing OpenStack on AWS EC2 instances has a few restrictions on the networking side, for the OpenStack set-up to work properly. This article outlines the limitations and the solutions to run OpenStack on top of the AWS EC2 VM.

Limitations

The AWS environment will allow the packets to flow in their network only when the MAC address is known or registered in the AWS network environment. Also, the MAC address and the IP address are tightly mapped. So, the AWS environment will not allow packet flow if the MAC address registered for the given IP address is different.You may wonder whether the above restrictions will impact the OpenStack set-up on AWS EC2.Yes, they certainly will! While configuring Neutron networking, we create a virtual bridge (say, br-ex) for the provider network, where all the VM’s traffic will reach the Internet via the external bridge, followed by the actual physical NIC (say, eth1). In that case, we usually configure the external interface (NIC) with a special type of configuration, as given below.The provider interface uses a special configuration without an IP address assigned to it. Configure the second interface as the provider interface. Replace INTERFACE_NAME with the actual interface name, for example, eth1 or ens224.Next, edit the /etc/network/interfaces file to contain the following:

# The provider network interfaceauto INTERFACE_NAMEiface INTERFACE_NAME inet manualup ip link set dev $IFACE updown ip link set dev $IFACE down

Due to this special type of interface configuration, the restriction in AWS will hit OpenStack networking. In the mainstream OpenStack set-up, the above-mentioned provider interface is configured with a special NIC configuration that will have no IP for that interface, and will allow all packets via that specially configured NIC. Moreover, the VM packets reaching the Internet via this specially configured NIC will have the OpenStack tenant router’s gateway IP address as the source in each packet.

As I mentioned earlier, in the limitations above, AWS will only allow the packet flow when the MAC address is known/registered in its environment. Also, the IP address must match the MAC address.

In our case, the packet from the above-mentioned OpenStack tenant router will have the IP address of the router’s gateway in every single packet, and the packet source’s MAC address will be the MAC address of the router’s interface.

Note:  These details are available if you use ip netns show followed by ip netns exec qr-<router_ID> ifconfig commands in the OpenStack controller’s terminal. Since the MAC address is unknown or not registered in the AWS environment, the packets will be dropped when they reach the AWS switch. To allow the VM packets to reach the Internet via the AWS switch, we need to use some tricks/hacks in our OpenStack set-up.

Making use of what we have

One possible way is to register the router’s MAC address and its IP address with the AWS environment. However, this is not feasible. As of now, AWS does not have the feature of registering any random MAC address and IP address inside the VPC. Moreover, allowing this type of functionality will be a severe security threat to the environment.The other method is to make use of what we have. Since we have used a special type of interface configuration for the provider NIC, you will note that the IP address assigned to the provider NIC (say, eth1) is left unused. We could use this available/unused IP address for the OpenStack router’s gateway. The command given below will do the trick:

neutron router-gateway-set router provider --fixed-ip ip_address=<Registered_IP_address*>

IP address and MAC address mismatches

After configuring the router gateway with the AWS registered IP address, each packet from the router’s gateway will have the AWS registered IP address as the source IP address, but with the unregistered MAC address generated by the OVS. As I mentioned  earlier, while discussing the limitations of AWS, the IP address must match the MAC address registered; else, all the packets with the mismatched MAC and IP address will be dropped by the AWS switch. To make the registered MAC address match with the IP address, we need to change the MAC address of the router’s interface.

The following steps will do the magic.

Step 1) Install the macchanger tool.

Step 2) Note down the actual/original MAC address of the provider NIC (eth1).

Step 3) Change the MAC address of the provider NIC (eth1).

Step 4) Change the MAC address of the router’s gateway interface to the original MAC address of eth1.

Step 5) Now, try to ping 8.8.8.8 from the router namespace.

If you get a successful ping response, then we are done with the Cloud-on-the-Cloud set-up.

Key points to remember

Here are some important things that one needs to keep track of.

Changing the MAC address: In my case, I had used the Ubuntu 14.04 LTS server, with which there was no issue in changing the MAC address using the macchanger tool. However, when I tried the Ubuntu 16.04 LTS, I got an error saying, “No permission to modify the MAC address.” I suspect the error was due to the cloud-init tool not allowing the MAC address’ configuration. So, before setting up OpenStack, try changing the MAC address of the NIC.

Floating IP disabled: Associating a floating IP to any of OpenStack’s VMs will send the packet via the router’s gateway with the source IP address as the floating IP’s IP address. This will make the packets hit the AWS switch with a non-registered IP and MAC address, which results in the packets being dropped. So, I could not use the floating IP functionality in this set-up. However, I could access the VM publicly using the following NAT process.

Using NAT to access OpenStack’s VM: As mentioned earlier, I could access the OpenStack VM publicly using the registered IP address that was assigned to the router’s gateway. Use the following NAT command to access the OpenStack VM using the AWS EC2 instance’s elastic IP:

$ ip netns exec qrouter-f85bxxxx-61b2-xxxx-xxxx-xxxxba0xxxx iptables -t nat -A PREROUTING -p tcp -d 172.16.20.101 --dport 522 -j DNAT --to-destination 192.168.20.5:22

Note: In the above command, I had NAT forwarding for all packets for 172.16.20.101 with Port 522. Using the above NAT command, all the packets reaching 172.16.20.101 with Port number 522 are forwarded to 192.168.20.5:22.

Here, 172.16.20.101 is the registered IP address of the AWS EC2 instance which was assigned to the router’s gateway. 192.168.20.5 is the local IP of the OpenStack VM. Notably, 172.16.20.101 already has NAT with the AWS elastic IP, which means all the traffic that comes to the elastic IP (public IP) will be forwarded to this VPC local IP (172.16.20.101).

In short, [Elastic IP]:522 π 172.16.20.101:522  π 192.168.20.5:22

This means that you can SSH the OpenStack VM globally by using the elastic IP address and the respective port number.

Elastic IP address: For this type of customised OpenStack installation, we require at least two NICs for an AWS EC2 instance. One is for accessing the VM terminal for installing and accessing the dashboard. In short, it acts as a management network, a VM tunnel network or an API network. The last one is for an external network with a unique type of interface configuration and is mapped with the provider network bridge (say br-ex with eth1).AWS will not allow any packets to travel out of the VPC unless the elastic IP is attached to that IP address. To overcome this problem, we must attach the elastic IP for this NIC. This is so that the packets of the OpenStack’s VM reach the OpenStack router’s gateway and from the gateway, the packets get embedded with the registered MAC address. Then, the matching IP address will reach the AWS switch (VPC environment) via br-ex and eth1 (a special type of interface configuration), and then hit the AWS actual VPC gateway. From there, the packets will reach the Internet.

Other cloud platforms

In my analysis, I noticed that most of the cloud providers like Dreamhost and Auro-cloud have the same limitations for OpenStack networking.  So we could use the tricks/hacks mentioned above in any of those cloud providers to run an OpenStack cloud on top of them.

Note: Since we are using the QEMU emulator without KVM for the nested hypervisor environment, the VM’s performance will be slow.
If you want to try OpenStack on AWS, you can register for CloudEnabler’s Cloud Lab-as-a-Service offering provides a consistent and on-demand lab environment for OpenStack in AWS.