MadNetworks Technical Blog

Networking Stuff




Nexus 1000v Installation with Layer 3 control

Nexus 1000v components

The Nexus 1000v is a software switch running Cisco NX-OS that is destined to be integrated with VMWare vCenter. The goal is to propose a virtual switch that can span across multiple ESX hosts so that networking teams can work on it as it if were a regular switch.

Technically, the Nexus 1000v is pushing the access layer inside the VMWare kernel, Virtual Machines will connect to virtual ports managed by the Nexus 1000v which in turn will control the physical ports of the ESX host.

The Nexus 1000v is composed of two main parts which are :

  • The VSM that stands for the Virtual Switching Module. This is a Virtual Machine that is used for the control plane operation. You can compare this a the SUP card of a C6500
  • The VEMs, the Virtual Ethernet Module. These are running inside the VMWare Kernel to provide switching functions. These are like the line card of C6500 and by so, they will be managed through the VSM. One VEM per ESX host will be installed.

Next part explores the installation options.

Nexus 1000v installation options

There is two main ways to install the Nexus 1000v : Layer 2 installation and Layer 3 installation. The differences between the two mode is how the VSM communicates with the VEMs. As the name implies, Layer2 requires to have the VSM in the same broadcast domain as the VEM and Layer3 will use. The best practice today is to use Layer3 as it offers more flexible design options.

Layer 2

Here is the example Cisco offers for Layer 2 installation :

image

The VEM are in the same Layer2 domain as the VSMs and communication is done through the Control and Packet VLANs (which are the same in the example). The management VLAN is used to connect to the VSMs and is represented as MGMT0 interface in the NX-OS. So with this kind of design, you can span a Nexus 1000V switch across multiple ESX hosts. The issue is that you need to extend the Control and Packet VLANs so that is can reach the VEMs.

This is not an issue for small scale deployment but if you aim for geographical redundancy across multiple datacenters you’ll need to extend your Layer2  and this is not always the best design choice even if technologies like OTV allows nice things.

Layer 3

With Layer 3 installation, the VSM does not need to be in the same broadcast domain as the VEMs. Everybody can be in it’s own subnet and communication will be done through IP. This offers more flexibility in the design as you can potentially have the ESX host anywhere in the network, as long as IP reachability is acquired, the VSM-VEM communication will be ok.

Separate ESX for the VSM

So the question is, if the VSM (which is a virtual machine) manages the VEM which is installed inside the VMKernel and the VEM manages the virtual machines so how to deal with the fact that the VSM will be managed by the VEM for the data plane ?

There is three options for this :

First, you can migrate the VSM on the VEM, so that the VSM will acquire connectivity through the VEM it manages. One issue is that if you loose the connectivity between the VSM and the VEM, you will no be able to manage anything and you could have the situation of an orphaned VSM. There is ways to recover from this but it becomes difficult to recover from this without connectivity loss.

Second, you can simply keep the VSM on the VMWare native switch, as long as the ESX host is alive then you cannot loose connectivity to the VSM.

Last but not least, you can use a 1010 appliance to host the VSMs but we will not cover that one here.

Redundant VSM

Remember the VSM are like SUP cards ? So as with SUP cards you can install a dual VSM system in HA mode. Main advantage is that the loss of one VSM does not affect connectivity. This is ideal for upgrade operation for example.

Integration with ESXI5.0

Installation Methods

Cisco offers multiple way to install the Nexus 1000V, you can use manual installation from an ISO, you can use a form based installation using an OVF or you can use a Java based installer.

Here for the sake of the understanding on how it really works, everything will be done from scratch using the ISO image.

Topology used

As I do not have enough machine to build a full physical topology, I will use VMWare workstation to virtualize ESXI hosts. Each host will have 6GB of ram and two VMNics. Connectivity is then obtained through the use of GNS3. The vCenter is installed on another VM inside VMWare workstation.

Layer 3 installation

With the Layer 3 installation we do not need to to extend the layer 2 boundaries. The topology looks like this :

image

The vCenter will be in vlan 100 using IP address : 10.0.0.100/24 gateway 10.0.0.254.

The first ESX host will have two physical NICs (well… virtually physical) and the MGMT interface will have the IP 10.0.0.101/24.

The second ESX host, same setup as the first one but hosted in a different IP subnet : 10.1.0.0.101/24.

The R1 router is here for interconnecting the two subnet and bridging to the real network.

Creation of the VSM Virtual Machine

First, installation of the VSM. You need to create a virtual machine that have :

  • At least 2G RAM
  • At least 3G hard drive space
  • Three NICs that will be installed as Control Interface (control0), Management Interface (mgmt0) and Packet Interface (Nexus does it automagically)

You should also reserve to resources to ensure that the switch always have enough to work properly.

At boot, specify the password for the admin user. Here I specify standalone mode as I don’t plan for an HA installation. The Domain ID is 1, select whatever you want here.

image

Then you can enter the basic configuration or do it all manually, let’s say no here to be directed to the CLI immediately.

Login, set a hostname and enter the interface mgmt0 configuration to allocate an IP address, here my VSM will have the 10.0.0.110/24 address :

image

From here, you should be able to access the VSM with SSH. Remember that for now the VSM is just a VM that has its connectivity through the VMWare standard switch. If you need to specify a default gateway, remember that the MGMT0 interface is in VRF Management by default.

Now we need to prepare an SVS Domain. The SVS domain defines how we will communicate with the vCenter :

svs-domain
domain id 1
control vlan 1
packet vlan 1
svs mode L3 interface mgmt0

Here I will be using the mgmt0 interface to communicate between the VSM and the VEMs. Another option is to use the control0 interface.

N1000V# sh svs domain
SVS domain config:
Domain id:    1
Control vlan: 1
Packet vlan:  1
L2/L3 Control mode: L3
L3 control interface: mgmt0
Status: Config push to VC failed (communication failure with VC).
N1000V#

For now the Nexus 1000 shows us an error because we are not communicating with the vCenter. So it’s time to setup this !

The communication between the VSM and the vCenter is done through a plugin that we need to register to the vCenter. The plugin is an XML extension that contains information like the extension key. We will take in more detail about this in the troubleshooting section. For now we need to register the plugin to the vCenter.

The plugin can be download from the VSM itself in HTTP, so activate the HTTP server of the VSM first is it’s not done.

imageDownload the XML extension and save it somewhere, then open vCenter and access the plugin manager :

image
Register the new plugin by selecting the XML file.

Once this is done, the configuration of the SVS connection can know continue on the Nexus 1000V :

svs connection VC
protocol vmware-vim
remote ip address 10.0.0.100 port 80
vmware dvs datacenter-name Cluster1
max-ports 8192

So here the important information are the remote ip address, which should be the vCenter address and the dvs datacenter-name. For me it’s Cluster1 as it is how I named my vCenter cluster.

When it’s done, type connect to initiate the Nexus 1000v and vCenter communication :

N1000V(config-svs-conn)# connect
Note: Command execution in progress..please wait
Note: Command execution in progress..please wait
Note: Command execution in progress..please wait
N1000V(config-svs-conn)# sh svs connections

connection VC:
ip address: 10.0.0.100
remote port: 80
protocol: vmware-vim https
certificate: default
datacenter name: Cluster1
admin:
max-ports: 8192
DVS uuid: 8b 24 13 50 86 be db cd-51 4a b5 19 33 46 ab 08
config status: Enabled
operational status: Connected
sync status: Complete
version: VMware vCenter Server 5.0.0 build-623373
vc-uuid: FA38B412-56AC-46FC-B9D5-D01776B35449
N1000V(config-svs-conn)#

So now if you look on vCenter you should have a new DVS that has been created with the name of your Nexus 1000V. Which means that the VSM has been installed properly. Now what we need to do is to install a VEM inside the VMKernel of the ESXi host so that traffic will be managed through it.

If you type a show module inside the 1000V here is what you should see :

N1000V# show module

Mod  Ports  Module-Type                       Model               Status

  —–  ——————————–  ——————  ————

1    0      Virtual Supervisor Module         Nexus1000V          active *

 

Mod  Sw                  Hw

  ——————  ————————————————

1    4.2(1)SV1(5.2)      0.0

 

Mod  MAC-Address(es)                         Serial-Num

  ————————————–  ———-

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

 

Mod  Server-IP        Server-UUID                           Server-Name

  —————  ————————————  ——————–

1    10.0.0.110       NA                                    NA

 

* this terminal session

The output is very similar to what you would have on a modular chassis like the C6500. Here we have the VSM acting as the SUP but we have no VEMs acting as Line Cards !

The VEM is installed inside the VMKernel by patching it with a VIB file, connect to the ESXi console and import the vib file that you can find on the Nexus 1000V HTTP server :

~ #  esxcli software vib install -v /tmp/cross_cisco-vem-v144-4.2.1.1.5.2.0-3.0.1.vib
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Cisco_bootbank_cisco-vem-v144-esx_4.2.1.1.5.2.0-3.0.1
VIBs Removed:
VIBs Skipped:

Now it’s time to ask the ESX host to use the VEM instead of the standard VMWare switch.

Before doing this we must create port-profiles inside the Nexus 1000V. A port-profile is like a template (smartports macro inside a Catalyst switch), these port-profile will then be applied to the physical network cards of the ESX hosts (to me in fact these are virtual card inside the VMWare workstation host but anyway…). What we need is  :

  • One Ethernet type port profile that will tie to one of the physical NIC and will support all the system communications
  • One Ethernet type port profile that will tie to one of the physical NIC and will support all VM communications
  • One vEthernet type port profile that will tie to the virtual kernel interface used by the VEM (we will create it just after this)
  • One vEthernet type port profile that will tie to the virtual kernel interface that is used to manage the ESX host and the VSM

So first, the port-profile which will be applied to the physical system NIC. Note that the vlan 100 that I use for management is specified as “system vlan” :

port-profile type ethernet SYS-UPLINK
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 100
no shutdown
system vlan 100
state enabled

Next, the port profile for VM communication, very similar to the system port profile except that no system vlan is specified and I have removed the vlan 100 from the trunk :

port-profile type ethernet SYS-UPLINK
vmware port-group
switchport mode trunk
switchport trunk allowed vlan except 100
no shutdown
state enabled

Then for the port profile that will be used for control, you need to create a new VMKernel interface inside VMWare. In my setup I choose to use the MGMT0 interface for control traffic so my new VMKernel will be in the same VLAN as the VLAN 100. There is the option to use the CONTROL0 interface instead which gives you the opportunity to use another VLAN/Subnet for this. In any case what you really do not want in production is to have this VLAN used for heavy traffic like VM Traffic, VMotion or whatever, because if the communication is delayed too long between the VSM and the VEM you will lost the VEM like this :

N1K# 2012 Nov 11 19:32:44 N1K %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 4 (heartbeats lost)
2012 Nov 11 19:32:46 N1K %VEM_MGR-2-MOD_OFFLINE: Module 4 is offline

No VEM, no traffic. Don’t loose your VEM !

So at the end the port profile is like this :

port-profile type vethernet VSM-CONTROL
capability l3control
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled

The very important command here is capability l3control. The port profile also needs to be an access port.

And finally the port profile for the MGMT traffic :

port-profile type vethernet VSM-MGMT
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled

What you should see now is something like this :

image

The port-profiles have been pushed to the vCenter inside the N1K DVS but nothing is using it for now. The next step is the migration of the ESX host on the 1000v VEM, here I will also migrate the VSM itself to the VEM so that we have something like this :

image

The difference between the drawing and my design is Layer 3 so there is only one VLAN for Control and Mgmt.

To migrate, right click on the DVS and select Add Hosts :

image

Select the physical VMNIC used for the System communication (for me it’s vmnic1) and select your Ethernet Port profile. Don’t pay attention to the column saying “N1K” you should have the standard VMWare switch here.

image

Select the vmk0 interface and apply the port profile used for the Management (for me it’s VSM-MGMT in VLAN 100) and the vmk1 interface that will be used for the communication between VSM and VEM is the port profile you configured with the capability L3 command.

On the last windows, click finish and pray.

You should lost your access to the VSM. This is because the VSM is still using the old profile sitting on the standard switch that is not used anymore. Edit the VSM host and change the card to apply the port profile used for the management (second card of the VM).

Now if you have everything setup correctly here is what you should see on the VSM :

N1K# sh module

Mod  Ports  Module-Type                       Model               Status

  —–  ——————————–  ——————  ————

1    0      Virtual Supervisor Module         Nexus1000V          active *

3    248    Virtual Ethernet Module           NA                  ok

4    248    Virtual Ethernet Module           NA                  ok

 

Mod  Sw                  Hw

  ——————  ————————————————

1    4.2(1)SV1(5.2)      0.0

3    4.2(1)SV1(5.2)      VMware ESXi 5.0.0 Releasebuild-623860 (3.0)

4    4.2(1)SV1(5.2)      VMware ESXi 5.0.0 Releasebuild-623860 (3.0)

 

Mod  MAC-Address(es)                         Serial-Num

  ————————————–  ———-

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA

4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA

 

Mod  Server-IP        Server-UUID                           Server-Name

  —————  ————————————  ——————–

1    10.0.0.110       NA                                    NA

3    10.0.0.101       564d44f6-1c95-bf5f-7976-e05e90430669  10.0.0.101

4    10.1.0.101       564d00e2-372c-5954-85e3-af41d9b30b81  10.1.0.101

 

* this terminal session

N1K#

Again, don’t pay strict attention to the output. If you have only one ESX host added, you should see only one VEM in slot 3.

If you want to add another ESX host, you just need to configure it with one additional VMKernel interface, install the VEM (of course) and migrate it on the N1K, the same port-profile can be reused. The two ESX hosts need not be in the same layer 2 domain (this is why layer 3 control is great) but you need to ensure that the VSM is able to ping the VMKernel that the VEM will use.

Layer 3 Conclusion

We did not follow the best practices here but the goal is just to test the Layer 3 control capabilities. On a real production design I would probably have the Control on one interface and the Management on another interface not managed by the VEM. The advantages of Layer 3 is it’s flexibility, the ESXes can be hosted anywhere, as long as IP reachability is obtained and maintained it will work.

The configuration has been done manually to show how it works behind the scene. Cisco offers a Java Based installer that is working very well and automates most of the configuration.

5 thoughts on “Nexus 1000v Installation with Layer 3 control

Leave a Reply

Your email address will not be published. Required fields are marked *