published on 23 Jun 2017

Support for SR-IOV

User should be able to deploy compute nodes with dedicated network interface into SR-IOV mode, then create direct-attached port and boot a VM with it.

https://blueprints.launchpad.net/fuel/+spec/support-sriov

Problem description

SR-IOV allows to bind network interface virtual functions to the different virtual machines like a PCI device. It gives a boost to the networking performance. This feature is already implemented in OpenStack, but requires complex discovery process and configuration.

This feature is compatible with DVR: traffic between tenant networks from or to SR-IOV enabled instances will be routed through the centralized router. L3 HA is not affected by this feature and should be working as expected.

Proposed changes

Enabling SR-IOV requires:

Web UI

On Nodes tab, in Interfaces configuration dialog for every interface should be:

  • Information whether interface is SR-IOV capable
  • Visual controls to enable SR-IOV, and to input how much virtual functions should be initialized on the interface

The following validation should be done in both UI and API:

  • In case of VLAN segmentation it should be possible to assign Private network to any NIC (with or without SR-IOV support).
  • In case of tunneling segmentation it should be possible to assign Private (mesh) network only to NIC where SR-IOV is not enabled.
  • SR-IOV can be enabled on SR-IOV capable interfaces where no networks are assigned. In such case a warning should be shown that only NIC with Private network on it is going to be configured in Nova.
  • SR-IOV enabled interface(s) cannot be part of a bond

The proposed change to Node Interfaces configuration screen will look like this:

sriov-ui.png

Nailgun

Nailgun-agent

The nailgun-agent should collect and send information about SR-IOV enabled NIC:

  • Number of available virtual functions (sriov_totalvfs)
  • SR-IOV availability (IOMMU groups should be checked)
  • PCI-ID of NIC virtual functions of this NIC (it's same for all VFs)

Collected information should be passed to nailgun in the next format

System Message: WARNING/2 (<string>, line 90)

Cannot analyze code. Pygments package not found.

.. code-block:: json

  {
    'meta': {
      'interfaces': [
        {
          "sriov" : {
            "sriov_totalvfs": Number,
            "available": Boolean,
            "pci_id": String
          }
        }
      ]
    }
  }

Bootstrap

In order to check SR-IOV availability, additional kernel parameters intel_iommu=on amd_iommu=on should be passed to bootstrap.

Data model

Information from the nailgun-agent and user input should be stored in interface_properties field of NodeNICInterface in format that described:

  • Whether SR-IOV is enabled by operator or not
  • Number of enabled VFs (sriov_numvfs)
  • Number of available virtual functions (sriov_totalvfs)
  • SR-IOV availability (IOMMU groups should be checked)
  • PCI-ID of NIC virtual functions of this NIC (it's same for all VFs)

System Message: WARNING/2 (<string>, line 128)

Cannot analyze code. Pygments package not found.

.. code-block:: json

  [
    {
      "interface_properties" : {
        "sriov" : {
          "enabled": Boolean or null,
          "sriov_numvfs": Number or null,
          "sriov_totalvfs": Number, Read only,
          "available": Boolean, Read only,
          "pci_id": String, Read only
        }
      }
    }
  ]


When operator configures interface as SR-IOV:

  • Network transformations should add port using add-port action with provider sriov and fill vendor_specific attributes as following.
  • Cluster-wide parameter supported_pci_vendor_devs will be generated and contains PCI-ID of NIC virtual functions from all interfaces where operator enabled SR-IOV. If this parameter is empty, it means SR-IOV is not enabled at all.

When Private network is assigned to SR-IOV enabled interface, deployment information (astute.yaml) will be extended and will look like this:

network_scheme:
  transformations:
  - action: add-port
    name: enp1s0f0
    provider: sriov
    vendor_specific:
      sriov_numvfs: <NUM>
      physnet: physnet2
quantum_settings:
  supported_pci_vendor_devs:
    - <PCI-ID>

where <NUM> is number and <PCI-ID> is string like "8086:1515".

When no network is assigned to SR-IOV enabled interface, deployment information (astute.yaml) will be extended and will look like this:

network_scheme:
  transformations:
  - action: add-port
    name: enp1s0f0
    provider: sriov
    vendor_specific:
      sriov_numvfs: <NUM>
quantum_settings:
  supported_pci_vendor_devs:
    - <PCI-ID>

REST API

Only payload for interfaces and node agent API handlers will be changed as described in Nailgun-agent and Data model sections.

Network Checker

Network checker will not be able to check traffic through Private VLANs when SR-IOV is enabled for the corresponding interface. So, this verification should be disabled for such nodes after deployment.

Orchestration

  • Additional kernel parameters intel_iommu=on amd_iommu=on should be passed to SR-IOV enabled nodes.
  • After compute nodes addition or removal, nova-scheduler and neutron-server configuration should be updated on controllers.

RPC Protocol

Only payload changes.

Fuel Client

None

Plugins

None

Fuel Library

  • l23network will configure interfaces virtual functions, set them up, and store network configuration into Operating System config
  • additional filters will be enabled for nova-scheduler
  • additional mechanism driver will be enabled for Neutron
  • firewall_driver for Neutron ML2 plugin will be set to Noop
  • pci_passthrough_whitelist will be configured in nova-compute. New function will be introduced, that returns this mapping between interface name and physical network from network_scheme transformations.
  • supported_pci_vendor_devs will be configured in neutron-server directly from corresponding field from astute.yaml.

Alternatives

SR-IOV is more hardware-specific feature than DPDK. However, SR-IOV should provide less overhead than DPDK.

Upgrade impact

None

Security impact

Security Groups can not currently be used with SR-IOV enabled ports.

Notifications impact

None

End user impact

User interface impact described in Web UI section.

Performance impact

Deployment impact

Developer impact

None

Infrastructure impact

This feature could not be tested on virtual environment. Special lab is required for manual verification.

Documentation impact

TBD

Implementation

Assignee(s)

Primary assignee:
yottatsa skolekonov
Mandatory design review:
xenolog

Work Items

  • Collecting information about SR-IOV interfaces
  • Enable SR-IOV configuration in fuel-library
  • Enable SR-IOV related orchestrations
  • Support of configuring SR-IOV via fuel API
  • Support of SR-IOV on UI
  • Manual testing

Dependencies

None

Testing, QA

Acceptance criteria

User should be able to deploy compute nodes with dedicated network interface into SR-IOV mode, then create direct-attached port and boot a VM with it.

References

Vladimir Eremin
at me@yottatsa.name securely
@yottatsa My LinkedIn Profile