For many years Oracle Clusterware have suffered from lack of native support of cluster interconnect redundancy. In the same time we still could configure more than one network for RAC cache fusion. To prevent interconnects being single point of failure, we could use 3rd party clustering solutions for older versions of oracle, or just use solutions provided by OS like network cards teaming, this often brought additional cost and complexity to the solution.
Starting from Oracle Grid Infrastructure 11.2.0.2 few feature called HAIP (High Availability Interconnect) been introduced, allowing us to configure up to 4 private networks for interconnect use. New crs resource 'ora.cluster_interconnect.haip' have appeared crs 'kitchenware' which uses multicast protocol for initial bootstrap and rac cache fusion. This puts additional requirement to the network because often the protocol is turned off at switch level. As an another requirement, each private network must sit in its own subnet, otherwise unplugging the cable will cause node eviction.
We can configure up to 4 networks for interconnect, if we set more than one private network, lets say 2, oracle will assign and use 4 addresses anyway attached to 2 physical network interfaces. In following example I've created 3 interconnects initially, and then removed first (eth1) and third (eth3), leaving only eth2. Here we see all 4 addresses attached to the single net interface.
eth2 Link encap:Ethernet HWaddr 08:00:27:AD:90:C2
inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2334553 errors:0 dropped:0 overruns:0 frame:0
TX packets:2305159 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2497624256 (2.3 GiB) TX bytes:2474843374 (2.3 GiB)
eth2:1 Link encap:Ethernet HWaddr 08:00:27:AD:90:C2
inet addr:169.254.20.220 Bcast:169.254.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2:2 Link encap:Ethernet HWaddr 08:00:27:AD:90:C2
inet addr:169.254.91.47 Bcast:169.254.127.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2:3 Link encap:Ethernet HWaddr 08:00:27:AD:90:C2
inet addr:169.254.169.41 Bcast:169.254.191.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2:4 Link encap:Ethernet HWaddr 08:00:27:AD:90:C2
inet addr:169.254.231.159 Bcast:169.254.255.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
In terms of RAC interconnect configuration, we will see all 4 addresses configured at database level.
SQL> select * from GV$CLUSTER_INTERCONNECTS where inst_id=1;
INST_ID NAME IP_ADDRESS IS_ SOURCE
---------- --------------- ---------------- --- -----------------------
1 eth1:1 169.254.20.220 NO
1 eth2:1 169.254.91.47 NO
1 eth3:1 169.254.169.41 NO
1 eth1:2 169.254.231.159 NO
|The HAIP feature allow us to change IP of interconnect on the fly without impacting the service, as it was in previous versions (we needed to stop crs on both nodes to change interconnects). I've created small test case which generates lots of messages and block flying over interconnect. This was done by creating small table and updating same blocks, even rows from 1st instance and odd rows from 2nd instance. The test generated several hundred messages and blocks per second travelling over the interconnect network, with load of about 4MB/sec.
While the load was present, I removed 2 networks, leaving only one, then I added 2nd network and dropped first one. The interfaces wasn’t dropped immediately, few minutes passed until the relocation has occurred. According to 'gc blocks lost '/'gc current lost' and other stats, no blocks dropped during the operation.
End of part I
