NFV and Open Networking with RHEL OpenStack Platfrom

(This is a summary version of a talk I gave at Intel Israel Telecom and NFV event on December 2nd, 2015. Slides are available here)

I was honored to be invited to speak on a local Intel event about Red Hat and what we are doing in the NFV space. I only had 30 minutes, so I tried to provide a high level overview of our offering, covering some main points:

  • Upstream first approach and why we believe it is a fundamental piece in the NFV journey; this is not a marketing pitch but really how we deliver our entire product portfolio
  • NFV and OpenStack; I touched on the fact that many service providers are asking for OpenStack-based solutions, and that OpenStack is the de-facto choice for VIM. That said, there are some limitations today (both cultural and technical) with OpenStack and clearly we have a way to go to make it a better engine for the telco needs
  • Full open source approach to NFV; it’s not just OpenStack but also other key projects such as QEMU/KVM, Open vSwitch, DPDK, libvirt, and the underlying Linux operating system. It’s hard to coordinate across these different communities, but this is what we are trying to do, with active participants on all of those
  • Red Hat product focus and alignment with OPNFV
  • Main use-cases we see in the market (atomic VNFs, vCPE, vEPC) with a design example of vPGW using SR-IOV
  • What telco and NFV specific features were introduced in RHEL OpenStack Platform 7 (Kilo) and what is planned for OpenStack Platform 8 (Liberty); as a VIM provider we want to offer our customers and the Network Equipment Providers (NEPs) maximum flexibility for packet processing options with PCI Passthrough, SR-IOV, Open vSwitch and DPDK-accelerated Open vSwitch based solutions.

Thanks to Intel Israel for a very interesting and well-organized event!

 

LLDP traffic and Linux bridges

In my previous post I described my Cumulus VX lab environment which is based on Fedora and KVM. One of the first things I noticed after bringing up the setup is that although I have got L3 connectivity between the emulated Cumulus switches, I can’t get LLDP to operate properly between the devices.

For example, a basic ICMP ping between the directly connected interfaces of leaf1 and spine3 is successful, but no LLDP neighbor shows up:

cumulus@leaf1$ ping 13.0.0.3
PING 13.0.0.3 (13.0.0.3) 56(84) bytes of data.
64 bytes from 13.0.0.3: icmp_req=1 ttl=64 time=0.210 ms
64 bytes from 13.0.0.3: icmp_req=2 ttl=64 time=0.660 ms
64 bytes from 13.0.0.3: icmp_req=3 ttl=64 time=0.635 ms
cumulus@leaf1$ lldpcli show neighbors 

LLDP neighbors:
-------------------------------------

Reading through the Cumulus Networks documentation, I discovered that LLDP is turn on by default on all active interfaces. It is possible to tweak things, such as timers, but the basic neighbor discovery functionality should be there by default.

Looking at the output from lldpcli show statistics I also discovered that LLDP messages are being sent out of the interfaces, but never received:

cumulus@leaf1$ lldpcli show statistics 

Interface:    eth0
  Transmitted:  11
  Received:     0
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     0
  Deleted:      0

Interface:    swp1
  Transmitted:  11
  Received:     0
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     0
  Deleted:      0

Interface:    swp2
  Transmitted:  11
  Received:     0
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     0
  Deleted:      0

So what’s going on?

Remember that leaf1 and spine3 are not really directly connected. They are bridged together using a Linux bridge device.

This is where I discovered that by design, Linux bridges silently drop LLDP messages (sent to the LLDP_Multicast address 01-80-C2-00-00-0E) and other control frames in the 01-80-C2-00-00-xx range.

Explanation to that can be found in the 802.1AB standard which is stating that “the destination address shall be 01-80-C2-00-00-0E. This address is within the range reserved by IEEE Std 802.1D-2004 for protocols constrained to an individual LAN, and ensures that the LLDPDU will not be forward by MAC Bridges that conform to IEEE Std 802.1D-2004.”

It is possible to change this behavior on a per bridge basis, though, by using:

# echo 16384 > /sys/class/net/<bridge_name>/bridge/group_fwd_mask

Retesting with leaf1 and spine3

# echo 16384 > /sys/class/net/virbr1/bridge/group_fwd_mask
cumulus@leaf1$ lldpcli show neighbor
LLDP neighbors:

Interface:    swp1, via: LLDP, RID: 1, Time: 0 day, 00:00:02  
  Chassis:     
    ChassisID:    mac 00:00:00:00:00:33
    SysName:      spine3
    SysDescr:     Cumulus Linux version 2.5.5 running on  QEMU Standard PC (i440FX + PIIX, 1996)
    MgmtIP:       3.3.3.3
    Capability:   Bridge, off
    Capability:   Router, on
  Port:        
    PortID:       ifname swp1
    PortDescr:    swp1
cumulus@leaf1$ lldpcli show statistics 

Interface:      eth0
  Transmitted:  117
  Received:     0
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     0
  Deleted:      0

Interface:      swp1
  Transmitted:  117
  Received:     72
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     1
  Deleted:      0

Interface:      swp2
  Transmitted:  117
  Received:     0
  Discarded:    0
  Unrecognized: 0
  Ageout:       0
  Inserted:     0
  Deleted:      0


LLDP now operates as expected between leaf1 and spine3. Remember that this is a per bridge setting, so in order to get this fixed across the entire setup, the command needs to be issued for the rest of the bridges (virbr2, virbr3, virbr4) as well.