September 7, 2012
We can model a RAC deployment with VirtualBox, but natively VirtualBox does not offer a way to build the switches that the actual production versions of our VirtualBox models would use in a real network toplogy. At my employer, we have a scenario of 2-node Oracle Enterprise Edition 64-bit 11gR2 RAC (220.127.116.11.0 Grid Infrastructure + 18.104.22.168.0 Oracle RDBMS) running on RHEL5 in a shared ethernet environment. We are using tagged VLANs and jumbo frames for the private interconnect over a shared switch (see this recent Oracle paper for more information on Oracle’s recommendations for RAC interconnect architecture in shared ethernet switching environments using tagged VLANs: ”Oracle Real Application Clusters (RAC) and Oracle Clusterware Interconnect Virtual Local Area Networks (VLANs) Deployment Considerations: An Oracle White Paper, June 2012).
In preparing for my production build, I wanted to be able to test not only the RAC build and configuration (including bond0 bonded interface for the RAC connect using jumbo frames) but also test out the proposed switching toplogy that the bonded RAC interconnect would be connected through. Our toplogy was 2 switches, sw3 and sw4 which our network administrators connected 2 bonded switch ports on each switch to form a bonded connection between the switches in case of switch port failure etc. Then my bonded interfaces eth0 and eth2 on each RAC node were connected in typical criss-cross (diagram coming for this post shortly) with one eth0 going to sw3 and eth2 going to sw4 and similar on the other RAC node.
I had VirtualBox RAC nodes where I modeled the whole build: same ip’s, same server names, same bond0 bonded interface, jumbo frames, etc. but how to model the networking piece I asked myself? A grail I had wanted for a long time – the ability to virtualize switches easily on Linux. Then I discovered openvswitch ( http://www.openvswitch.org ). After some work and searching many posts and howtos, I found the recipe to create what I needed (2 virtual network switches connected to each other by dual bonded vports) and also connected to my virtualbox RAC nodes.
The RAC nodes are connected to the openvswitch switches via taps that are accessed through the “bridged” mode of the “details –> network” configuration page of your RAC VM. To cut to the chase, below, for example, are the scripts to create 2 switches with ports for the bonded RAC eth0 and eth2 interfaces, as well as connect the 2 switches via a bond. Note that you need the “taps” so that virtualbox can see and access the virtual switches.
Also included below is an example of a small file that allows you to “turn on” and “turn off” ports on the openvswitch virtual switch so that you can test out your HA network topology for your RAC and at least verify that the logic of the design is valid. Note that when you port your design to actual production physical switches, behaviours could change because physical switches might behave differently, but getting your logical switching design to work in that case would be a matter most likely of setting features on the physical switch correctly.
Create switch sw3 (note vlan tags on ports and use of jumbo frames MTU 9000)
ovs-vsctl del-br sw3
ovs-vsctl add-br sw3
ip tuntap del mode tap n1a1sw3
ip tuntap del mode tap n2a4sw3
ip tuntap add mode tap n1a1sw3
ip tuntap add mode tap n2a4sw3
ip link set n1a1sw3 up
ip link set n2a4sw3 up
ovs-vsctl add-port sw3 n1a1sw3 tag=322
ovs-vsctl add-port sw3 n2a4sw3 tag=322
ifconfig n2a4sw3 mtu 9000
ifconfig n1a1sw3 mtu 9000
ovs-vsctl list-ports sw3
Create switch sw4
ovs-vsctl del-br sw4
ovs-vsctl add-br sw4
ip tuntap del mode tap n1a4sw4
ip tuntap del mode tap n2a1sw4
ip tuntap add mode tap n1a4sw4
ip tuntap add mode tap n2a1sw4
ip link set n1a4sw4 up
ip link set n2a1sw4 up
ovs-vsctl add-port sw4 n1a4sw4 tag=322
ovs-vsctl add-port sw4 n2a1sw4 tag=322
ifconfig n2a1sw4 mtu 9000
ifconfig n1a4sw4 mtu 9000
ovs-vsctl list-ports sw4
Create bonded switch port link sw3<–>sw4 (note how you can define trunks that will pass over the bonded switch ports. this was tested and works. unlisted vlan’s will not pass, thus your interconnect can be the only vlan on the bonded switchports which will keep all traffic except interconnect traffic off your trunk bonded switchport link). This code is owed to a post I found here:
first 2 command lines create the bonds, the next 4 lines connect the switches together over the bonded virtual switch ports.
ovs-vsctl add-bond sw4 bond0 sw4p2 sw4p1 trunks=10,322
ovs-vsctl add-bond sw3 bond1 sw3p2 sw3p1 trunks=10,322
ovs-vsctl set interface sw3p1 type=patch options:peer=sw4p1
ovs-vsctl set interface sw4p1 type=patch options:peer=sw3p1
ovs-vsctl set interface sw3p2 type=patch options:peer=sw4p2
ovs-vsctl set interface sw4p2 type=patch options:peer=sw3p2
It’s in brezular’s post, but you can also view the status of the switchports bond using the command:
ovs-appctl bond/show bond0
Running the above commands will build your switching. You connect it to Virtualbox by choosing “Bridged” option in the VirtualBox GUI, then select the switchport you want to use on the switch of your choice. Since VirtualBox calls them “Adapters” I give the switches names that help me to see what is connecting to what, e.g.
n1a4sw4 means RAC Node 1, Adapter 4, Switch sw4
Once you have it all built you can try jumbo frame ping tests:
ping -s 16384 -I 192.168.0.25 -c 3 192.168.0.24
Above we are telling ping to send a 16384 byte packet using 192.168.0.25 as the sending ip (in my case that is logical interface bond0 which is made of eth0 and eth3) and to send it to interface ip 192.168.0.24 on the other RAC node which is again bond0 with the same eth assignments.
You can test out the validity of your HA redundancy by deleting and re-adding ports in openvswitch. For example, to delete the port:
ovs-vsctl del-port sw4 n1a4sw4
to add the port back use:
ovs-vsctl add-port sw4 n1a4sw4 tag=322
I am using Ubuntu Linux 12.04 as my VirtualBox Host, and I am using the openvswitch that is in the Ubuntu package respository, so installing openvswitch is as easy as just using “apt-get install openvswitch” (or using synaptic package manager). I set my terminal to be semitransparent, and set the RAC nodes pinging each other across the private interconnect, and then I watch the pings start and stop as I test the HA configurations.
You can have all kinds of fun trying out things like changing the tag value (changing the VLAN) and showing how the packets will not make it if the VLAN’s dont match on the relevant ports. You can also dynamically destroy and recreate the switchport bonded interfaces. You can force the traffic to go over the bonded switchport link or you can force it over specific remaining switches by turning the ports off and on using add-port and del-port commands.
I am going to write this up very completely with screenshots, diagrams, etc., but I wanted to get it out there so that DBA’s modeling actual projects would have a way to build up a model of the logical switching to which their RAC’s will be connected in production to validate the switching toplogy and smooth out the actual implementation.
This albeit brief work is dedicated to Tim Hall and Jeffrey Hunter whose prolific net-published efforts in RAC virtualization are both voluminous and of excellent and unique technical quality. Of both of these authors, Mr. Hall and Mr. Hunter, I can say: ”Never in the history of Oracle have so few done so much for so many.”
Enjoy! With this virtual networking “Erector Set” provided by OpenVSwitch, the possibilities are limited only by your network toplogy imagination and your ability to construct it around your RAC or any other virtualization project.
gotta run to catch the last train home from Grand Central…Cheers