Sun Dual 10GbE XFP
PCI Express Card
User’s Guide
Part No. 820-0184-14
May 2010, Revision A
1. Product Overview
Product Description
1
1
2. Installing and Setting Up the Driver
7
7
▼
▼
To Download the Driver on a Solaris Platform
7
9
▼
3. Installing the Adapter 13
Installing the Adapter 13
▼
To Install the Adapter 14
Installing an Optical Transceiver 17
▼
To Install an Optical Transceiver 18
iii
▼
4. Network Configuration 23
Configuring the Network Host Files 23
▼
Installing the Solaris Operating System Over a 10-Gigabit Ethernet Network 27
▼
To Install the Solaris Operating System Over a 10-Gigabit Ethernet
Network 27
Booting Over the 10-Gigabit Ethernet Network for Solaris x86 and Linux
Systems 30
▼
5. Configuring the nxgeDevice Driver Parameters 33
▼
▼
▼
▼
▼
▼
▼
To Improve Performance on an UltraSPARC CPU Based Sun Platform 41
To Improve Performance on an AMD CPU Based Sun Platform 42
To Obtain Higher Throughput Using the Generic Tunables for the Solaris
TCP/IP Stack 43
iv
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Setting Parameters on a Linux Platform 44
▼
▼
6. Configuring the Jumbo Frames Feature 51
Checking Jumbo Frames Configurations 51
▼
To Show the Driver Statistics in a Solaris Environment 52
Enabling Jumbo Frames in a Solaris Environment 53
▼
▼
▼
To Check Layer 3 Configuration 54
▼
▼
7. Configuring Link Aggregation 57
Configuring Link Aggregation in a Solaris Environment 58
▼
▼
8. Configuring VLANs 61
Configuring VLANs in a Solaris Environment 63
▼
To Configure Static VLANs 64
Configuring VLANs in a Linux Environment 65
To Configure VLANs in a Linux Environment 65
▼
Contents
v
A. Specifications 67
Connectors 67
Updating SunVTS to Recognize the Adapter 72
▼
Using the SunVTS netlbtest 73
To Use the netlbtest 73
▼
Index 75
vi
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Using This Documentation
This guide provides instructions for installing both the hardware and software for
the Sun Dual 10GbE XFP PCI Express card from Oracle. This manual also describes
how to configure the nxgedriver, which controls the card.
These instructions are designed for enterprise system administrators with experience
installing network hardware and software.
Note – In this document the term x86 refers to 64-bit and 32-bit systems
manufactured using processors compatible with the AMD64 or Intel Xeon/Pentium
product families. For supported systems, see the Solaris Hardware Compatibility Guide.
Using UNIX Commands
This document might not contain information about basic UNIX commands and
procedures such as shutting down the system, booting the system, and configuring
devices. Refer to the following for this information:
■
Software documentation that you received with your system
Solaris Operating System documentation, which is at:
■
vii
Shell Prompts
Shell
Prompt
C shell
machine-name%
C shell superuser
machine-name#
Bourne shell and Korn shell
Bourne shell and Korn shell superuser
$
#
viii
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Related Documentation
The documents listed as online are available at:
Application
Title
Part Number
Format
Location
Release Notes
Sun x8 Express Dual 10 Gigabit Ethernet XFP Low 820-0185
PDF
Online
Profile Adapter Release Notes
HTML
Safety information Important Safety Information for Sun Hardware
816-7190 or
821-1590
Printed
In
Systems
shipment
Documentation, Support, and Training
These web sites provide additional resources:
Sun Function
Documentation
Support
URL
Training
Documentation Feedback
Submit comments about this document by clicking the Feedback[+] link at
http://docs.sun.com. Include the title and part number of your document with
your feedback:
Sun Dual 10GbE XFP PCI Express Card User’s Guide, part number 820-0184-14.
Using This Documentation
ix
x
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
1
Fiber XFP Low Profile Adapter, including:
■
■
■
■
“Components” on page 1
“Product Description” on page 1
“Hardware and Software Requirements” on page 3
“Features” on page 3
Components
The carton in which your Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low
Profile Adapter was shipped should contain the following items:
■
Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter with a
low profile bracket attached.
■
■
Standard height bracket
Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Getting
Started Guide.
Product Description
The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter is a low
profile x8 lane PCI-Express 10 gigabit Ethernet card utilizing the Sun ASIC with
XFP-based 10-gigabit Ethernet optics. The Sun x8 Express Dual 10 Gigabit Ethernet
Fiber XFP Low Profile Adapter requires an optical transceiver.
1
FIGURE 1-1 shows the adapter.
FIGURE 1-1 Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
Hardware and Software Requirements
Before installing the adapter, make sure your system meets the hardware and
software requirements. TABLE 1-1 lists the supported hardware and software.
TABLE 1-1
Hardware and Software Requirements for Sun SPARC and X64 Servers
Requirements
Hardware or Software
Hardware
Sun Fire™ T1000, Sun Fire T2000, Sun Fire V445, Sun Fire V245, Sun Fire
V215, Sun Fire U45, Sun Fire X4600, Sun Fire X4200, Sun Fire X4100, Sun Fire
X2100
Operating System
Solaris 10 01/07 Operating System
SuSE Linux Enterprise Server 10, RedHat Enterprise Linux
4.0_u4, RedHat Enterprise Linux 4.0_u3
Optical transceivers
X5558A SR (Short Reach) XFP Transceiver for the base board
X5560A-z LR (Long Reach) XFP Transceiver for the base board
2
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Features
The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
provides the following features:
■
Two full-duplex 10 gigabit Ethernet interface that use a 10 gigabit Ethernet small
form factor pluggable (XFP), optical transceiver with duplex LC fiber connector
■
■
■
IEEE 802.3ae 2002 compliant
Uses Sun’s own ASIC and software for innovative throughput networking design
Networking I/O virtualization supporting Solaris LDOM 1.0. (VMware support
planned for mid 07)
■
■
Hardware-based flow classification for extending parallelism and virtualization to
networking
Up to 16 Receive DMA channels and up to 24 Transmit DMA channels, multiple
receive and transmit Descriptor Rings and dedicated networking hardware
resources (DMA, interrupts, buffer, and more) for each thread or strand
■
■
■
■
■
■
CPU/thread affinity and CPU load balancing at L1,L2,L3 and L4
Dynamic Reconfiguration (DR)
Jumbo frame support (up to 9KBytes)
IPv4/IPv6 and IPMP support
TCP/UDP/IP checksum and CRC32C support
IEEE 802.1Q VLAN support
Chapter 1 Product Overview
3
Declaration of Conformity
Compliance Model Number:
ATLS2XGF
Product Family Name:
Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
EMC
USA—FCC Class A
This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions:
1. This equipment may not cause harmful interference.
2. This equipment must accept any interference that may cause undesired operation.
European Union
This equipment complies with the following requirements of the EMC Directive 89/336/EEC:
As Information Technology Equipment (ITE) Class A per (as applicable):
EN 55022:1994 +A1:1995 +A2:1997 Class A
EN 61000-3-2:2000
EN 61000-3-3:1995 +A1:2000
Pass
Pass
EN 55024:1998 +A1:2001 +A2:2003 Required Limits:
IEC 61000-4-2
IEC 61000-4-3
IEC 61000-4-4
IEC 61000-4-5
4 kV (Direct), 8kV (Air)
3 V/m
1 kV AC Power Lines, 0.5 kV Signal and DC Power Lines
1 kV AC Line-Line and Outdoor Signal Lines, 2 kV AC Line-Gnd, 0.5 kV DC Power
Lines
IEC 61000-4-6
IEC 61000-4-8
IEC 61000-4-11
3 V
1 A/m
Pass
Safety
This equipment complies with the following requirements of the Low Voltage Directive 73/23/EEC:
EC Type Examination Certificates:
EN 60950-1:2001, 1st Edition
IEC 60950-1:2001, 1st Edition
Evaluated to all CB Countries
CB Scheme Certificate No.
UL and cUL/CSA 60950-1:2001, CSA C22.2 No. 60950-00 File:E138989-A82
FDA DHHS Accession Number (Monitors Only)
Vol. 54
Supplementary Information
This equipment was tested and complies with all the requirements for the CE Mark.
This equipment complies with the Restriction of Hazardous Substances (RoHS) directive 2002/95/EC.
4
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
/S/
/S/
Dennis P. Symanski
DATE
Donald Cameron
Worldwide Compliance Engineering
Sun Microsystems, Inc.
4150 Network Circle, MPK15-102
Santa Clara, CA 95054 U.S.A.
Tel: 650-786-3255
DATE
Program Manager/Customer Quality
Sun Microsystems Scotland, Limited
Blackness Road, Phase I, Main Bldg.
Springfield, EH49 7LR
Scotland, United Kingdom
Tel: +44 1 506 672 539
Fax: 650-786-3723
Fax: +44 1 506 670 011
Chapter 1 Product Overview
5
6
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
2
Installing and Setting Up the Driver
This chapter explains how to download and install the nxgedriver. The nxge gigabit
Ethernet driver (nxge(7D)) is a multi-threaded, loadable, clonable, GLD-based
configuring the data-link interfaces and link aggregations.
This chapter contains the following sections:
■
■
“Downloading and Installing the Driver on a Solaris SPARC or x86 Platform” on
page 7
“Downloading and Installing the Driver on a Linux Platform” on page 10
Downloading and Installing the Driver
on a Solaris SPARC or x86 Platform
If your system uses the Solaris SPARC or x86 operating system you will need to
download and install the nxgedevice driver for Solaris platforms.
▼ To Download the Driver on a Solaris Platform
1. Locate and download the nxgedevice driver software at the following web
site:
7
2. Uncompress the gzipped tar file:
# gunzip nxge.tar.gz
3. Unpack the tar file:
# tar xvf nxge.tar
a. For SPARC systems, change to the following directory:
# cd 10_GigabitEthernet/Solaris_10/sparc/Packages
b. For x86 systems:
# cd 10_GigabitEthernet/Solaris_10/i386/Packages
4. For SPARC systems, determine which architecture your system is running:
# uname -m
a. For sun4v systems, install the software packages by typing the following at
the command line:
# /usr/sbin/pkgadd -d SUNWnxge.v SUNWnxgem
b. For sun4u systems, install the software packages by typing the following at
the command line:
# /usr/sbin/pkgadd -d SUNWnxge.u SUNWnxgem
5. For x86 systems, install the software packages by typing the following at the
command line:
#/usr/sbin/pkgadd -d
A menu similar to the following displays:
The following packages are available:
1 SUNWnxge Sun PCI-E 10G/1G Ethernet Adapter Driver (i386)
1.0,REV=2006.12.05.10.0
Select package(s) you wish to process (or ’all’ to process
all packages). (default: all) [?,??,q]:
8
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
6. Select the packages you want to install:
■
Press Return or type allto accept the default and install all packages.
■
Type the specific numbers, separated by a space, if you prefer not to install any
optional packages.
7. Verify that the nxgedriver is installed on the system:
Sun PCI-E 10G/1G Ethernet Adapter Driver(i386)
1.0,REV=2006.12.05.10.0 Copyright 2006 Sun Microsystems,
Inc. All rights reserved. Use is subject to license terms.
## Executing checkinstall script. Using as the package base
directory.
## Processing package information.
## Processing system information.
4 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs. This package contains
scripts which will be executed with super-user permission
during the process of installing this package.
Do you want to continue with the installation of [y,n,?] y
Installing Sun PCI-E 10G/1G Ethernet Adapter Driver as
## Installing part 1 of 1.
/kernel/drv/amd64/nxge [ verifying class ]
# Executing postinstall script.
# Installation of was successful.
▼ To Remove the Driver from a Solaris Platform
1. To discover the driver packages run the pkginfocommand:
# pkginfo | grep SUNWnxge
SUNWnxge Sun PCI-E 10G/1G Ethernet Adapter Driver
2. To remove the driver packages run the pkgrmcommand:
# pkgrm SUNWnxge plus any other packages from the previous command
Chapter 2 Installing and Setting Up the Driver
9
Downloading and Installing the Driver
on a Linux Platform
1. Login to your system.
2. Download the driver RPM for your operating system:
For example:
nxge-1.0-1.x86_64.rpm
3. Discover the network interfaces before adding the package by using the
ifconfig -acommand:
# ifconfig -a |grep eth
eth0
eth1
eth2
eth3
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DC
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DD
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DE
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DF
4. Use the rpmtool to install the driver on SuSe and RedHat Linux.
# rpm -ivh /tmp/RHEL4U4-large/RPMS/x86_64/nxge-1.0-1.x86_64.rpm
Preparing...
1:nxge
###########################################
###########################################
10
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Note – In RHEL5.0 and later releases, the driver is packaged in the kmoddriver
binary package format. This packaging allows forward and backward driver binary
compatibility within the same flavors of RHEL5 releases. Driver packages nxge
version 2.0.0 and later are not compatible with the earlier nxge-1.x-xreleases. To
upgrade from 1.x-xto2.x.xrpm package, it is necessary to remove nxge-1.x-
xpackage before installing the 2.x.xpackage.
The RHEL5 nxge2.x.xkmodpackage contains two separate packages: One for
the driver and another one for applications like nxge_config. You must install
both of the following packages:
kmod-nxge-rhel-2.0-1.x86_64.rpm (driver binary package)
nxge-apps-rhel-2.0-1.x86_64.rpm (application package)
To install the complete package, enter the following:
# rpm -ivh nxge-apps-rhel-2.0-1.x86_64.rpm kmod-nxge-rhel-2.0-1.x86_64.rpm
To ensure that the driver is loaded after the rpm installation, enter the following:
# modprobe nxge
5. Verify the new network interface instances corresponding to the Sun x8 Express
Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter:
# ifconfig -a |grep eth
eth0
eth1
eth2
eth3
eth4
eth5
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DC
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DD
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DE
Link encap:Ethernet HWaddr 00:14:4F:20:F1:DF
Link encap:Ethernet HWaddr 00:14:4F:6C:78:E8
Link encap:Ethernet HWaddr 00:14:4F:6C:78:E9
The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
instances, eth4and eth5, are shown in bold italics.
6. Add the nxgeinterfaces to the /etc/modules.conffile to automatically load
the driver after system reboot:
alias eth4 nxge
alias eth5 nxge
Chapter 2 Installing and Setting Up the Driver
11
7. Use the ethtoolcommand to check the parameter configurations that apply to
the nxgedriver.
(For 10G)
# ethtool -i eth4
driver: nxge
version: 2.0.1
firmware-version: 2XGF PXE1.47 FCode 3.9 07/04/24
bus-info: 0000:84:00.0
(Equivalent for 1G)
# ethtool -i eth4
driver: nxge
version: 2.0.1
firmware-version: QGC PXE1.47 FCode 3.9 07/04/24
bus-info: 0000:02:00.2
▼ To Remove the Driver From a Linux Platform
●
To remove the driver packages from a Linux Platform use the rpm -e
command:
# rpm -e nxge-1.0-1
12
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
3
Installing the Adapter
Fiber XFP Low Profile Adapter in your system and verify that it has been installed
correctly.
This chapter contains the following section:
■
■
■
“Installing the Adapter” on page 13
“Installing an Optical Transceiver” on page 17
“Verifying the Hardware Installation” on page 20
Note – If you are installing the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP
Low Profile Adapter in a machine running either the Solaris x86 Operating System
or the Linux x86 Operating System, Dynamic Reconfiguration (DR) is not supported.
Installing the Adapter
If you are installing the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low
Profile Adapter into a machine running Solaris 10, you must install the software
before you install the hardware.
The following instructions describe the basic tasks required to install the adapter.
Refer to your system installation or service manual for detailed PCI-Express adapter
installation instructions, specifically you must check which bracket your server
requires (low profile or standard height), and install the standard height bracket if
required.
13
Caution – Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
mechanical fit limitations:
Do not install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile
Adapter in slot-2 of a Sun Fire T2000 system. The SATA cables from the adjacent
card slot prevent the proper installation of an Sun x8 Express Dual 10 Gigabit
Ethernet Fiber XFP Low Profile Adapter in this slot. (Note that this issue does not
exist for Sun Fire T2000 that is RoHS compliant.)
Caution – Do not install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP
Low Profile Adapter in slot-5 of a Sun Fire X4600 system because the heatsink on the
adapter will sit against the daughter card of the Sun Fire X4600 system.
place.
This section contains the following topics:
■
■
■
“To Install the Adapter” on page 14
“Installing an Optical Transceiver” on page 17
“Verifying the Hardware Installation” on page 20
▼ To Install the Adapter
1. Halt and power off your system.
2. Power off all of the peripherals connected to your system.
3. Open the system unit.
4. Attach the adhesive copper strip of the antistatic wrist strap to the metal casing
of the power supply. Wrap the other end twice around your wrist, with the
adhesive side against your skin.
5. Remove the filler panel from the PCI-E opening.
14
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Note – These illustrations show the Sun x8 Express Dual 10 Gigabit Ethernet Fiber
XFP Low Profile Adapter being installed into a Sun Fire T2000, which requires a
standard height bracket for the low profile adapter.
6. Open the retaining clip at the edge of the PCI-E opening.
7. Holding the adapter by the edges, align the adapter edge connector with the
PCI-E slot. Slide the adapter face plate into the small slot at the end of the PCI-
E opening.
8. Applying even pressure at both corners of the adapter, push the PCI-Express
adapter until it is firmly seated in the slot.
Chapter 3 Installing the Adapter
15
9. Secure the retaining clip
Caution – Do not use excessive force when installing the adapter into the PCI-E
slot. You might damage the adapter’s PCI connector. If the adapter does not seat
properly when you apply even pressure, remove the adapter and carefully reinstall
it.
10. Detach the wrist strap and close the system unit.
16
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
11. Install the optical transceivers.
Note – Illustrations for installing the adapter and installing the optical transceiver
show different systems. This difference in systems provides the maximum clarity for
installing the optical transceiver and shows the most complicated adapter
installation.
Installing an Optical Transceiver
The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
requires an optical transceiver in each port to create and Ethernet connection. The
short-range optical transceiver Part Number: 135-1172-01 and the long-range optical
transceiver Part Number: 375-3495-01 are both available from Sun Microsystems.
Chapter 3 Installing the Adapter
17
▼ To Install an Optical Transceiver
1. Pull the white locking handle into the full horizontal position.
You will feel the handle click into position when it is fully opened.
Note – If you are installing an optical transceiver in a Sun Fire T1000, ensure that
the label and the edge connector is visible to you.
2. Holding the optical transceiver by the edges, align the transceiver with the slot
in the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
and slide it into the opening.
3. Applying even pressure at both corners of the transceiver, push the transceiver
until it is firmly seated in the slot.
18
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
4. Push the handle closed to lock the optical transceiver in place.
5. Repeat Step 1 through Step 4 to install the second optical transceiver
6. Detach the wrist strap.
Caution – If you pull the locking handle down when the optical transceiver is
installed, remove the optical transceiver entirely and reinstall it. The handle operates
an internal lock. Pulling the handle down can disconnect the optical transceiver,
even though it might appear to be connected.
Chapter 3 Installing the Adapter
19
7. Connect the Ethernet cables.
Verifying the Hardware Installation
After you have installed the Sun 10-Gigabit Ethernet adapter, but before you boot
your system, perform the following tasks to verify the installation. Refer to the your
Solaris documentation for the detailed instructions.
Note – Verification is not required if your system supports dynamic reconfiguration
(DR). Verification is not supported if your system is running Solaris x86 software.
▼ To Verify the Hardware Installation
1. Power on the system, and when the banner appears, press the Stop-A key
sequence to interrupt the boot process and display the OpenBoot (ok) prompt.
2. List the network devices on your system.
ok show-nets
a) /pci@7c0/pci@0/pci@8/network@0,1
b) /pci@7c0/pci@0/pci@8/network@0
c) /pci@7c0/pci@0/pci@2/network@0,1
d) /pci@7c0/pci@0/pci@2/network@0
e) /pci@780/pci@0/pci@1/network@0,1
f) /pci@780/pci@0/pci@1/network@0
q) NO SELECTION from the above list
Checking the .propertiesoutput for each device is the surest way to identify
the device. Usually /pci@7c0/pci@0/pci@8or /pci@7c0/pci@0/pci@9
correspond to PCIe slots, so look at those devices first.
a) /pci@7c0/pci@0/pci@8/network@0,1
b) /pci@7c0/pci@0/pci@8/network@0
Note – If you do not see the device listed, check that the adapter is properly seated.
If necessary, reinstall the adapter.
20
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
3. View the device that you installed.
Using the previous example, type:
ok cd /pci@7c0/pci@0/pci@8/network@0,1
4. Use the .propertiescommand to display a list of device properties.
The .propertiescommand displays the specific information about the device. If
you are using the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile
Adapter, your output will be similar to the following:
ok .properties
assigned-addresses
82080010 00000000 04000000 00000000 01000000
82080018 00000000 03500000 00000000 00008000
82080020 00000000 03508000 00000000 00008000
82080030 00000000 03600000 00000000 00100000
00 14 4f 6b ce 88
local-mac-address
phy-type
xgf
reg
00080000 00000000 00000000 00000000 00000000
03080010 00000000 00000000 00000000 01000000
03080018 00000000 00000000 00000000 00008000
03080020 00000000 00000000 00000000 00008000
03080030 00000000 00000000 00000000 00100000
2XGF 10G Ethernet Adapter FCode 1.26 06/11/14
501-7283-04
version
board-model
model
SUNW,pcie-2xgf
compatible
pciex108e,abcd.108e.0.0
pciex108e,abcd.108e.0
pciex108e,abcd.0
pciex108e,abcd
pciexclass,020000
pciexclass,0200
pci108e,abcd
address-bits
00000030
max-frame-size
00002400
network-interface-type ethernet
device_type
name
network
network
fcode-rom-offset
interrupts
class-code
subsystem-vendor-id
revision-id
device-id
00000000
00000001
00020000
0000108e
00000000
0000abcd
0000108e
vendor-id
Chapter 3 Installing the Adapter
21
CHAPTER
4
installed on your system. This chapter contains the following sections:
■
■
“Setting Up a 10-Gigabit Ethernet Network on a Diskless Client System” on
page 25
■
■
“Installing the Solaris Operating System Over a 10-Gigabit Ethernet Network” on
page 27
“Booting Over the 10-Gigabit Ethernet Network for Solaris x86 and Linux
Systems” on page 30
Configuring the Network Host Files
After installing the driver software, you must create a hostname.nxgenumber file
for the adapter’s Ethernet interface. You must also create both an IP address and a
host name for its Ethernet interface in the /etc/hostsfile.
23
1. At the command line, use the grepcommand to search the
/etc/path_to_instfile for nxge interfaces.
# grep nxge /etc/path_to_inst
# "/pci@7c0/pci@0/pci@9/network@0" 0 "nxge"
# "/pci@7c0/pci@0/pci@9/network@0,1" 1 "nxge"
#
In this example, the device instance is from a Sun x8 Express Dual 10 Gigabit
Ethernet Fiber XFP Low Profile Adapter installed in slot 1.
Be sure to write down your device path and instance, which in the example is
"/pci@7c0/pci@0/pci@9/network@0" 0. Your device path and instance will be
similar. You need this information to make changes to the nxge.conffile. See
“Setting Parameters Using the nxge.confFile” on page 36.
2. Use the ifconfigcommand to set up the adapter’s nxge interface.
Use the ifconfigcommand to assign an IP address to the network interface.
Type the following at the command line, replacing ip-address with the adapter’s IP
address:
# ifconfig nxge0 plumb ip-address up
Refer to the ifconfig(1M) man page and the Solaris documentation for more
information.
■
If you want a setup that remains the same after you reboot, create an
/etc/hostname.nxgenumber file, where number is the instance number of the
nxge interface you plan to use.
To use the adapter’s nxge interface in the Step 1 example, create an
/etc/hostname.nxge0file, where 0is the number of the nxge interface. If the
instance number were 1, the filename would be
/etc/hostname.nxge1.
■
Do not create an /etc/hostname.nxgenumber file for a Sun x8 Express Dual 10
Gigabit Ethernet Fiber XFP Low Profile Adapter interface you plan to leave
unused.
■
■
The /etc/hostname.nxgenumber file must contain the host name for the
appropriate nxge interface.
The host name must have an IP address listed in the /etc/hostsfile.
24
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
■
The host name must be different from any other host name of any other interface,
for example: /etc/hostname.nxge0and /etc/hostname.nxge1cannot share
the same host name.
The following example shows the /etc/hostname.nxgenumber file required for
a system called zardozthat has a Sun x4 PCI-Express Gigabit Ethernet driver
(zardoz-11).
# cat /etc/hostname.nxge0
zardoz
# cat /etc/hostname.nxge1
zardoz-11
3. Create an appropriate entry in the /etc/hostsfile for each active nxge
interface.
For example:
# cat /etc/hosts
#
# Internet host table
#
127.0.0.1
localhost
129.144.10.57 zardoz
loghost
129.144.11.83 zardoz-11
Setting Up a 10-Gigabit Ethernet
Network on a Diskless Client System
Before you can boot and operate a diskless client system across a 10-Gigabit Ethernet
network, you must first install the 10-Gigabit Ethernet software packages into the
root directory of the diskless client. You can find the 10-Gigabit Ethernet software
packages at the following web site:
Refer to the Solaris Advanced Installation Guide and the System Administration Guide
for more information about installing and administering diskless client systems.
Note – The Solaris x86 version of the operating system do not support diskless
clients.
Chapter 4 Network Configuration
25
▼ To Set Up a 10-Gigabit Ethernet Port on a
Diskless Client
1. Locate the root directory of the diskless client on the host server.
The root directory of diskless client system is commonly installed in the host
server’s /export/root/client-name directory, where client_name is the diskless
client’s host name. In this procedure, the root directory is:
/export/root/client-name
2. Download the software for Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP
Low Profile Adapter onto the server’s drive.
client’s root directory on the server.
Install the software packages to the client’s root directory.
4. Create a hostname.nxgenumber file in the diskless client’s root directory.
page 23 for instructions.
5. Edit the hostsfile in the diskless client’s root directory.
Edit the /export/root/client-name/etc/hostsfile to include the IP address of
the 10-Gigabit Ethernet interface. See “Configuring the Network Host Files” on
page 23 for instructions.
6. Set the MAC address on the server side and rebuild the device tree if you want
to boot from the 10-Gigabit Ethernet port.
7. To boot the diskless client from the 10-Gigabit Ethernet port, type the
following boot command:
ok boot path-to-device:link-param
26
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Installing the Solaris Operating System
Over a 10-Gigabit Ethernet Network
The Solaris Advanced Installation Guide describes the full procedure for installing the
Solaris Operating System over the network. The following procedure assumes that
you have created an install server, which contains the image of the Solaris CD, and
that you have set up the client system to be installed over the network.
Before you can install the Solaris Operating System on a client system with a 10-
Gigabit Ethernet adapter, you must first add the 10-Gigabit Ethernet software
packages to the install server. These software packages are on Sun 10-Gigabit
Ethernet Driver CD.
Note – Refer to the Solaris Advanced Installation Guide for more information about
installing the Solaris Operating System over the network.
▼ To Install the Solaris Operating System Over a
10-Gigabit Ethernet Network
1. Prepare the install server and client system to install the Solaris Operating
System over the network.
The Solaris Advanced Installation Guide describes how to create the install server
and set up the client systems.
Note – If you want to install the client system over a network that is not part of the
same subnet, you must also create a boot server. The Solaris Advanced Installation
Guide describes how to create a boot server.
Chapter 4 Network Configuration
27
2. Find the root directory of the client system.
The client system’s root directory can be found in the install server’s
/etc/bootparamsfile. Use the grepcommand to search this file for the root
directory.
# grep client-name /etc/bootparams
client_name root=server-name:/netinstall/Solaris_10/Tools/Boot
install=server-name:/netinstall boottype=:in rootopts=:rsize=32768
In this example, the root directory for the Solaris 10 client is /netinstall. In
Step 4, you would replace root-directory with /netinstall.
Note – If the root directory is not found in the /etc/bootparamsfile, refer to the
Solaris Advanced Installation Guide for configuration instructions.
3. Download the Sun x8 Express Dual 10 Gigabit Ethernet driver onto the install
server’s hard drive.
The package is a folder SUNWnxge.v or SUNWnxge.u, which you can download
from the following web site:
4. On the install server, install the Sun x8 Express Dual 10 Gigabit Ethernet
software to the client’s root directory, as determined in Step 2.
Replace root-directory with the location of the client’s root directory.
# cd location where you downloaded the packages
# ls SUNWnxge*
# pkgadd -R root-directory/Solaris_10/Tools/Boot -d . SUNWnxge.v
Note – If the commands above do not work correctly, refer to the documentation for
your version of the Solaris Operating System.
Note – Perform the following steps on the client system.
28
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
5. Shut down and halt the client system.
Use the shutdowncommand to go to the OpenBoot (ok) prompt.
# shutdown -i0 -g0 -y
. . .
(shutdown command messages omitted)
. . .
ok
6. At the okprompt, use the show-netscommand to find the device path of the
10-Gigabit Ethernet device.
The show-netscommand lists the system devices. You should see the full paths
and names of the network devices, similar to the example below.
ok show-nets
a) /pci@7c0/pci@0/pci@8/network@0,1
b) /pci@7c0/pci@0/pci@8/network@0
c) /pci@7c0/pci@0/pci@2/network@0,1
d) /pci@7c0/pci@0/pci@2/network@0
e) /pci@780/pci@0/pci@1/network@0,1
f) /pci@780/pci@0/pci@1/network@0
q) NO SELECTION from the above list
7. At the okprompt, boot the client system using the full device path of the 10-
Gigabit Ethernet device, for example:
ok boot /pci@7c0/pci@0/pci@8/network@0
8. Proceed with the Solaris Operating System installation.
Refer to the Solaris Advanced Installation Guide for more information about
installing the Solaris Operating System over the network.
9. After installing the Solaris Operating System, install the Sun x8 Dual 10
Gigabit Ethernet software on the client system.
The software installed in Step 4 is required to boot the client system over the 10-
Gigabit Ethernet interface. You now need to install the software in order for the
operating system to use the client’s 10-Gigabit Ethernet interfaces in normal
operation.
Before installing the Sun 10-Gigabit Ethernet driver, ensure that the client system
does not already have the driver installed. Use the pkginfocommand to see if
the Sun 10-Gigabit Ethernet software packages are installed on the client system.
# pkginfo | grep SUNWnxge
Chapter 4 Network Configuration
29
■
■
If the software is installed, the previous command will return the package name
you typed in. In that case, skip to Step 10.
If the software is not installed, install the software from the download center.
10. Confirm that the network host files have been configured correctly during the
Solaris installation.
Although the Solaris software installation creates the client’s network
configuration files, you may need to edit these files to match your specific
networking environment. See “Configuring the Network Host Files” on page 23
for more information about editing these files.
11. Use the dladm show-devcommand to show configuration information for
all data-links or the specified data-link. By default, the system is configured to
have one data-link for each known network device.
# dladm show-dev
e1000g0
e1000g1
e1000g2
e1000g3
nxge0
link: up
speed: 1000 Mbps
duplex: full
duplex: half
duplex: half
duplex: half
duplex: full
duplex: full
link: down
link: down
link: down
link: up
speed: 0
speed: 0
speed: 0
Mbps
Mbps
Mbps
speed: 10000 Mbps
speed: 10000 Mbps
nxge1
link: up
Booting Over the 10-Gigabit Ethernet
Network for Solaris x86 and Linux
Systems
▼ To Boot Over the Network on Solaris x86 and
Linux Systems
1. Obtain the MAC address from the target Sun x8 Express Dual 10 Gigabit
Ethernet Fiber XFP Low Profile Adapter.
2. Set up the PXE boot server with the MAC addresses .
3. Choose one of the adapter ports as the boot interface.
30
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
4. Plug the Ethernet cable to the adapter port.
5. Power on the system.
6. Press the F2 key or the Control/E keys to go to the BIOS.
7. Check and make sure that the boot order of Hard Drive is higher than network
devices.
8. Refer the boot-device-order image.
The reconfiguration boot attaches the driver to the adapter. You can now
configure the driver parameters for your Sun x8 Express Dual 10 Gigabit Ethernet
Fiber XFP Low Profile Adapter.
9. Press the F10 key to save the boot configuration changes and exit.
System should reboot after saving the boot configuration.
Chapter 4 Network Configuration
31
10. Press the F12 key to install the OS from the network.
If the cable is connected to the right port, you should see the MAC address that
you assigned to your PXE server displayed by BIOS.
image : pxe-mac-addr
PXE-E61: Media test failure, check cable
PXE-MOF: Exiting Intel Boot Agent.
NVIDIA Boot Agent 217.0513
Copyright (C) 2001-2005) NVIDIA Corporation
Copyright (C) 1997-2000) NVIDIA Corporation
PXE-E61: Media test failure, check cable
PXE-MOF: Exiting Intel Boot Agent.
NVIDIA Boot Agent 217.0513
Copyright (C) 2001-2005) NVIDIA Corporation
Copyright (C) 1997-2000) NVIDIA Corporation
PXE-E61: Media test failure, check cable
PXE-MOF: Exiting Intel Boot Agent.
Intel (R) Boot Agent GE v1.2.43 Beta-1
Copyright (C) 1997-2006) Intel Corporation
CLIENT MAC ADDR; 00 15 17 13 90 00 GUID: 00000000 0000 0000 0000
00144F26E0B7
11. You can now install the nxgedriver and configure the adapter.
32
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
5
Configuring the nxgeDevice Driver
Parameters
device in your system.
set these parameters.
■
■
■
■
■
“nxgeHardware and Software Overview” on page 33
“Setting nxgeDriver Parameters on a Solaris Platform” on page 34
“Setting Parameters Using the nddUtility” on page 34
“Setting Parameters Using the nxge.confFile” on page 36
“Setting Parameters on a Linux Platform” on page 44
nxgeHardware and Software Overview
The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter
provides two 10-Gigabit Full Duplex networking interfaces. The device driver
automatically sets the link speed to 10000 Mbit/sec and conforms to the IEEE 802.3
Ethernet standard. Each interface has 8 Receive DMA Channels and 12 Transmit
DMA Channels to allow for parallel processing of the packets. The Sun x8 Express
Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter extends CPU and OS
parallelism to networking with its support for hardware-based flow classification
and multiple DMAs. Using CPU thread affinity to bind a given flow to a specific
CPU thread, it enables a one-to-one correlation of Rx and Tx packets across the same
TCP connection. This can help avoid cross-calls and context switching to deliver
greater performance while reducing the need for CPU resources to support I/O
33
processing. The Sun 10-Gigabit Ethernet Adapter utilizes Sun’s own innovative
MAC Controller to map the 10-Gigabit XAUI interface onto the PCI Express form
factor. It supports 10 Gb/sec bandwidth using eight transmit and eight receive lanes.
Setting nxgeDriver Parameters on a
Solaris Platform
You can set the nxgedevice driver parameters in two ways:
■
Using the nddutility
■
Using the nxge.conffile
If you use the nddutility, the parameters are valid only until you reboot the system.
This method is good for testing parameter settings.
To set parameters so they remain in effect after you reboot the system, create a
/platform/sun4u/kernel/drv/nxge.conffile and add parameter values to this
file when you need to set a particular parameter for a device in the system.
Setting Parameters Using the nddUtility
Use the nddutility to configure parameters that are valid until you reboot the
system.
The following sections describe how you can use the nxgedriver and the nddutility
to modify (with the -setoption) or display (without the -setoption) the
parameters for each nxgedevice.
Noninteractive and Interactive Modes
You can use the nddutility in two modes:
■
Noninteractive
Interactive
■
In noninteractive mode, you invoke the utility to execute a specific command. Once
the command is executed, you exit the utility. In interactive mode, you can use the
utility to get or set more than one parameter value. Refer to the ndd(1M) man page
for more information.
34
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
▼ To Specify Device Instances for the nddUtility
Before you use the nddutility to get or set a parameter for a nxgedevice, you must
specify the device instance for the utility.
1. Check the /etc/path_to_instfile to identify the instance associated with a
particular device.
# grep nxge /etc/path_to_inst
"/pci@7c0/pci@0/pci@9/network@0" 0 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,1" 1 "nxge"
▼ To Specify Parameter Values Using the ndd
Utility
This section describes how to modify and display parameter values.
1. To modify a parameter value, use the -setoption.
If you invoke the nddutility with the -setoption, the utility passes value, which
must be specified, down to the named /dev/nxgedriver_instance, and assigns the
value to the parameter:
# ndd -set /dev/nxgeX parameter-value
Where X is the driver instance, for example /dev/nxge0, /dev/nxge1.
2. To display the value of a parameter, specify the parameter name and omit the
value.
When you omit the -setoption, the utility queries the named driver instance,
retrieves the value associated with the specified parameter, and prints it:
# ndd /dev/nxgeX parameter
▼ To Use the nddUtility in Interactive Mode
1. To modify a parameter value in interactive mode, specify ndd /dev/nxgeX:
# ndd /dev/nxge0
name to get/set? (Enter the parameter name or ? to view all
parameters)
Chapter 5 Configuring the nxgeDevice Driver Parameters
35
After you enter the parameter name, the nddutility prompts you for the parameter
value.
2. To list all the parameters supported by the nxgedriver, type ?.
# ndd /dev/nxge1 name to get/set ?
?
(read only)
function_number
adv_autoneg_cap
adv_10gfdx_cap
adv_1000fdx_cap
adv_100fdx_cap
adv_10fdx_cap
(read only)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
(read and write)
adv_pause_cap
accept_jumbo
rxdma_intr_time
rxdma_intr_pkts
class_opt_ipv4_tcp
class_opt_ipv4_udp
class_opt_ipv4_ah
class_opt_ipv4_sctp
class_opt_ipv6_tcp
class_opt_ipv6_udp
class_opt_ipv6_ah
class_opt_ipv6_sctp
Setting Parameters Using the
nxge.confFile
Specify the driver parameter properties for each device by creating a nxge.conffile
in the /kernel/drvdirectory. Use a nxge.conffile when you need to set a
particular parameter for a device in the system.
The man pages for prtconf(1M) and driver.conf(4) include additional details.
The next procedure shows an example of setting parameters in a nxge.conffile.
●
To access any man page, type the mancommand plus the name of the man
page.
For example, to access man pages for prtconf(1M), type:
% man prtconf
36
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
▼ To Set Driver Parameters Using an nxge.conf
File
1. Obtain the hardware path names for the nxgedevices in the device tree.
a. Check the /etc/driver_aliasesfile to identify the name associated with
a particular device:
# grep nxge /etc/driver_aliases
nxge "pciex108e,abcd"
b. Locate the path names and the associated instance numbers in the
/etc/path_to_instfile.
# grep nxge/etc/path_to_inst
"/pci@780/pci@0/pci@8/network@0" 0 "nxge"
"/pci@780/pci@0/pci@8/network@0,1" 1 "nxge"
■
In this example:
■
■
■
The first part within the double quotes specifies the hardware node name in
the device tree.
The number not enclosed in quotes is the instance number (shown in bold
italicsfor emphasis).
The last part in double quotes is the driver name.
To identify a PCI-E device unambiguously in the nxge.conffile, use the name,
parent name, and the unit-address for the device. Refer to the pci(4) man page for
more information about the PCI-E device specification.
In this example:
■ name = "pciex108e,abcd"
■ parent = "/pci@780/pci@0/pci@8/network@0"
■ unit-address= "0"
2. Set the parameters for the nxgedevices in the
/platform/sun4u/kernel/drv/nxge.conffile.
a. The following parameters can be set using the
/platform/sun4u/kernel/drv/nxge.conffile.
#
#---------------Link Configuration ----------------------
#
#
The link parameters depend on the type of the card
and the port.
Chapter 5 Configuring the nxgeDevice Driver Parameters
37
#
#
#
#
#
#
#
10 gigabit related parameters ( i.e adv_10gfdx_cap)
apply only to 10gigabit ports.
Half duplex is not supported on any NIU card.
adv-autoneg-cap
Advertise auto-negotiation capability.
default is 1
# adv-autoneg-cap = 1;
#
#
#
#
adv_10gfdx_cap
Advertise 10gbps Full duplex capability.
default is 1
# adv_10gfdx_cap = 1;
#
#
#
#
adv_1000fdx_cap
Advertise 1gbps Full duplex capability.
default is 1
# adv_1000fdx_cap = 1;
#
#
#
#
adv_100fdx_cap
Advertise 100mbps Full duplex capability.
default is 1
# adv_100fdx_cap = 1;
#
#
#
#
adv_10fdx_cap
Advertise 10mbps Full duplex capability.
default is 1
# adv_10fdx_cap = 1;
#
#
#
#
adv_asmpause_cap
Advertise Asymmetric pause capability.
default is 0
# adv_asmpause_cap = 0;
#
#
#
#
adv_pause_cap
Advertise pause capability.
default is 1
# adv_pause_cap = 1;
#
#
#------- Jumbo frame support ---------------------------------
# To enable jumbo support for all nxge interfaces,
# accept_jumbo = 1;
#
# To disable jumbo support for all nxge interfaces,
# accept_jumbo = 0;
#
# Default is 0. See the example at the end of this file for
# enabling or disabling jumbo for a particular nxge interface.
38
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
#
#
#------- Receive DMA Configuration ----------------------------
#
# rxdma-intr-time
#
#
#
#
Interrupts after this number of NIU hardware ticks have
elapsed since the last packet was received.
A value of zero means no time blanking (Default = 8).
# rxdma-intr-pkts
#
#
#
#
Interrupt after this number of packets have arrived since
the last packet was serviced. A value of zero indicates
no packet blanking (Default = 20).
# Default Interrupt Blanking parameters.
#
# rxdma-intr-time = 8;
# rxdma-intr-pkts = 20;
#
#
#------- Classification and Load Distribution Configuration ------
#
# class-opt-****-***
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
These variables define how each IP class is configured.
Configuration options range from whether TCAM lookup ie
is enabled to flow hash generation.
This parameters also control how the flow template is
constructed and how packet is distributed within RDC
groups.
supported classes:
class-opt-ipv4-tcp class-opt-ipv4-udp class-opt-ipv4-sctp
class-opt-ipv4-ah class-opt-ipv6-tcp class-opt-ipv6-udp
class-opt-ipv6-sctp class-opt-ipv6-ah
Configuration bits (The following bits will be decoded
by the driver as hex format).
0010:
0020:
0040:
0080:
0100:
0200:
0400:
0800:
use MAC Port (for flow key)
use L2DA (for flow key)
use VLAN (for flow key)
use proto (for flow key)
use IP src addr (for flow key)
use IP dest addr (for flow key)
use Src Port (for flow key)
use Dest Port (for flow key)
# class-opt-ipv4-tcp = fe0;
#
Chapter 5 Configuring the nxgeDevice Driver Parameters
39
b. The following parameters operate on a per port basis and can be set using
the /platform/sun4u/kernel/drv/nxge.conffile.
#
# ------- How to set parameters for a particular interface --------
# The example below shows how to locate the device path and set a
# parameter for a particular nxge interface. (Using jumbo support as
# an example)
#
# Use the following command to find out the device paths for nxge,
#
#
more /etc/path_to_inst | grep nxge
# For example, if you see,
#
#
#
#
#
"/pci@7c0/pci@0/pci@8/network@0" 0 "nxge"
"/pci@7c0/pci@0/pci@8/network@0,1" 1 "nxge"
"/pci@7c0/pci@0/pci@8/network@0,2" 2 "nxge"
"/pci@7c0/pci@0/pci@8/network@0,3" 3 "nxge"
# then you can enable jumbo for ports 0 and 1 and disable jumbo for ports 2
# and 3 as follows,
#
# name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address
= "0"
# accept_jumbo = 1;
# name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address
= "0,1"
# accept_jumbo = 1;
# name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address
= "0,2"
# accept_jumbo = 0;
# name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address
= "0,3"
# accept_jumbo = 0;
c. In the following example, the ports of all the Sun x8 Express Dual 10 Gigabit
Ethernet Fiber XFP Low Profile Adapter are being set for load balancing Rx
traffic based on IP source address. The default value is F80 indicating Rx
load balancing based on IP 5-tuple. Notice the semi-colon at the end of the
last parameter.
class-opt-ipv4-tcp = 100;
class-opt-ipv4-udp = 100;
40
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
d. The following example shows ports on two different cards being set. Only
one node needs to be specified.
name = "pciex108e,abcd" parent = "/pci@780/pci@0/pci@8/" unit-address = "0"
class-opt-ipv4-tcp = 0x100;
name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@9/" unit-address = "0"
class-opt-ipv4-tcp = 0x40;
3. Save the nxge.conffile.
Tuning for Maximum Performance on a
Solaris Platform
Tuning for maximum performance in a Solaris platform depends on whether you
are using an UltraSPARC CPU based platform or an AMD CPU based platform.
▼ To Improve Performance on an UltraSPARC
CPU Based Sun Platform
1. Improve performance by adding the following /etc/systemfile:
# set ddi_msix_alloc_limit=4
Increasing the MSI improves the Rx performance. The default value for MSI is 2,
but changing it to 4 improves performance (8 can be used for UltraSparc-T1 based
systems).
2. Reboot the system:
# reboot -r
Chapter 5 Configuring the nxgeDevice Driver Parameters
41
3. Add the following to a startup script, or use ndd before plumbing the interface:
# ndd -set /dev/ip ip_soft_rings_cnt 8
Utilizing more soft-rings provided by the Solaris TCP/IP stack significantly
improves bulk throughput for Rx. The default number of soft-rings is 2, but
changing it to 8 improves performance. (You can increase the number to 16 in
UltraSparc-T1 based systems).
▼ To Improve Performance on an AMD CPU Based
Sun Platform
1. Enable soft-rings and change to a higher value than the default of 2 by adding
the following to the /etc/systemfile:
set ip:ip_squeue_fanout=1
set ip_squeue_soft_ring=1
Bulk throughput for Rx can be significantly improved by utilizing more soft-rings
provided by the Solaris TCP/IP stack. Soft-rings are disabled by default in
previous and current releases of the Solaris-x86 Operating System.
2. Reboot the system:
# reboot -r
3. Set the MSI to 1 on AMD platforms by adding following to the /etc/system
file:
set ddi_msix_alloc_limit=1
4. Reboot the system:
# reboot -r
42
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
▼ To Obtain Higher Throughput Using the Generic
Tunables for the Solaris TCP/IP Stack
●
To obtain higher throughput, add the following to a startup script:
ndd -set /dev/tcp tcp_conn_req_max_q 8192
ndd -set /dev/tcp tcp_conn_req_max_q0 8192
ndd -set /dev/tcp tcp_max_buf 4194304
ndd -set /dev/tcp tcp_cwnd_max 2097152
ndd -set /dev/tcp tcp_recv_hiwat 400000
ndd -set /dev/tcp tcp_xmit_hiwat 400000
Chapter 5 Configuring the nxgeDevice Driver Parameters
43
Setting Parameters on a Linux Platform
▼ To Set Parameters Using the ethtoolUtility
1. Determine which parameters are available using the ethtoolutility:
# ethtool -help eth4
ethtool version 1.8
Usage:
ethtool DEVNAME
ethtool -a DEVNAME
ethtool -A DEVNAME \
[ autoneg on|off ] \
[ rx on|off ] \
[ tx on|off ]
ethtool -c DEVNAME
ethtool -C DEVNAME \
[adaptive-rx on|off] \
[adaptive-tx on|off] \
[rx-usecs N] \
[rx-frames N] \
[rx-usecs-irq N] \
[rx-frames-irq N] \
[tx-usecs N] \
[tx-frames N] \
[tx-usecs-irq N] \
[tx-frames-irq N] \
[stats-block-usecs N] \
[pkt-rate-low N] \
[rx-usecs-low N] \
[rx-frames-low N] \
[tx-usecs-low N] \
[tx-frames-low N] \
[pkt-rate-high N] \
[rx-usecs-high N] \
[rx-frames-high N] \
[tx-usecs-high N] \
[tx-frames-high N] \
[sample-interval N]
ethtool -g DEVNAME
ethtool -G DEVNAME \
[ rx N ] \
[ rx-mini N ] \
44
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
[ rx-jumbo N ] \
[ tx N ]
ethtool -i DEVNAME
ethtool -d DEVNAME
ethtool -e DEVNAME \
[ raw on|off ] \
[ offset N ] \
[ length N ]
ethtool -E DEVNAME \
[ magic N ] \
[ offset N ] \
[ value N ]
ethtool -k DEVNAME
ethtool -K DEVNAME \
[ rx on|off ] \
[ tx on|off ] \
[ sg on|off ] \
[ tso on|off ]
ethtool -r DEVNAME
ethtool -p DEVNAME [ %d ]
ethtool -t DEVNAME [online|(offline)]
ethtool -s DEVNAME \
[ speed 10|100|1000 ] \
[ duplex half|full ]
\
[ port tp|aui|bnc|mii|fibre ] \
[ autoneg on|off ] \
[ phyad %d ] \
[ xcvr internal|external ] \
[ wol p|u|m|b|a|g|s|d... ] \
[ sopass %x:%x:%x:%x:%x:%x ] \
[ msglvl %d ]
ethtool -S DEVNAME
Chapter 5 Configuring the nxgeDevice Driver Parameters
45
Following are some common parameters that can be changed:
# ethtool -c eth8
Coalesce parameters for eth8:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 8
rx-frames: 512
rx-usecs-irq: 0
rx-frames-irq: 512
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame--high: 0
rx-usecs and rx-frames control the RX interrupt rate per RX DMA channel. RX
interrupt will be generated after rx-frames have been received or after rx-usecs
time interval if fewer than rx-frames have been received within the interval. For
low latency applications, it is recommended to set rx-usecs to smaller value. For
bulk traffic, it is recommended to use larger values of rx-usecs and control the
rate with rx-frames.
rx-frames-irq controls the maximum number of rx packets processed with a single
RX interrupt.
46
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
2. To change RX interrupt Coalesce parameters use the ethtool -Ccommand:
# ethtool -C eth4 rx-usecs 20
# ethtool -c eth4
Coalesce parameters for eth4:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 20
rx-frames: 512
rx-usecs-irq: 0
rx-frames-irq: 512
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
3. To get status of L4 HW checksumming, use the ethtool -kcommand:
# ethtool -k eth4
Offload parameters for eth4:
Cannot get device tcp segmentation offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
Chapter 5 Configuring the nxgeDevice Driver Parameters
47
▼ To Set Parameters Using the Bundled
configtoolUtility
1. To get a list of tunable parameters, use the nxge_configif_name get
command:
# /usr/local/bin/nxge_config eth4 get
The tunable parameters exported by this device are:
class_opt_ipv4_tcp
class_opt_ipv4_udp
class_opt_ipv4_ah
class_opt_ipv4_sctp
class_opt_ipv6_tcp
class_opt_ipv6_udp
class_opt_ipv6_ah
class_opt_ipv6_sctp
Read-Write
Read-Write
Read-Write
Read-Write
Read-Write
Read-Write
Read-Write
Read-Write
These classification variables define how each IP class is configured. This
parameter also controls how the flow template is constructed and how packets are
distributed within RDC groups.
Configuration bits:
0x0010:
0x0020:
0x0040:
0x0080:
0x0100:
0x0200:
0x0400:
0x0800:
use MAC Port (for flow key)
use L2DA (for flow key)
use VLAN (for flow key)
use proto (for flow key)
use IP src addr (for flow key)
use IP dest addr (for flow key)
use Src Port (for flow key)
use Dest Port (for flow key)
Note – The classification variables are modified on an adapter basis, that is, if any of
these variables is modified for one port, the change carries over to all other ports of
that adapter.
2. To get a particular variable use the nxge_config if_name get param_name:
# /usr/local/bin/nxge_config eth4 get class_opt_ipv4_udp
class_opt_ipv4_udp
0xfe3
48
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
3. To set a particular variable, use the /usr/local/bin/nxge_config if_name
set param_name value:
# /usr/local/bin/nxge_config eth4 set class_opt_ipv4_tcp 0xfe0
Tuning for Maximum Performance on a
Linux Platform
The following tunings will improve the performance of the Sun x8 Express Dual 10
Gigabit Ethernet device driver on a system running the Linux operating system.
1. Create the conffile (for example, sysctl_nxge.conf) that will be called by
the sysctlutility.
### IPV4 specific settings
# turns TCP timestamp support off, default 1, reduces CPU use
net.ipv4.tcp_timestamps = 0
# turn SACK support off, default on systems with a VERY fast bus ->
# memory interface this is the big gainer
net.ipv4.tcp_sack = 0
# sets min/default/max TCP read buffer, default 4096 87380 174760
net.ipv4.tcp_rmem = 10000000 10000000 10000000
# sets min/pressure/max TCP write buffer, default 4096 16384 131072
net.ipv4.tcp_wmem = 10000000 10000000 10000000
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
net.ipv4.tcp_mem = 10000000 10000000 10000000
### CORE settings (mostly for socket and UDP effect)
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 524287
# maximum send socket buffer size, default 131071
net.core.wmem_max = 524287
# default receive socket buffer size, default 65535
net.core.rmem_default = 524287
# default send socket buffer size, default 65535
net.core.wmem_default = 524287
# maximum amount of option memory buffers, default 10240
net.core.optmem_max = 524287
# number of unprocessed input packets before kernel starts dropping
# them, default 300
net.core.netdev_max_backlog = 300000
Chapter 5 Configuring the nxgeDevice Driver Parameters
49
2. Set up the sysctlutility.
# sysctl -p /etc/sysctl_nxge.conf
50
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
6
Configuring the Jumbo Frames
following sections:
■
■
■
■
“Jumbo Frames Overview” on page 51
“Checking Jumbo Frames Configurations” on page 51
“Enabling Jumbo Frames in a Solaris Environment” on page 53
“Enabling Jumbo Frames in a Linux Environment” on page 54
Jumbo Frames Overview
Configuring Jumbo Frames enables the Ethernet interfaces to send and receive
packets larger than the standard 1500 bytes. However, the actual transfer size
depends on the switch capability and the ethernet adapter driver capability.
Note – Refer to the documentation that came with your switch for exact commands
to configure Jumbo Frames support.
Checking Jumbo Frames Configurations
The jumbo frame configuration checking occurs at Layer 2 or Layer 3, depending on
the configuration method.
51
▼ To Show the Driver Statistics in a Solaris
Environment
1. Use the kstatcommand to display driver statistics, for example:
# kstat nxge:1 |grep rdc_packets
rdc_packets
rdc_packets
rdc_packets
rdc_packets
rdc_packets
rdc_packets
rdc_packets
rdc_packets
798982054
792546171
803941759
805674872
798714912
799293256
806470537
805413540
The previous example displays the receive packet counts on all of the eight
Receive DMA channels on interface 1. Using the kstat nxge:1shows all the
statistics that the driver supports for that interface.
2. Use the kstat command to display driver statistics of a VLAN interface, for
example:.
# kstat nxge:38001
module: nxge
name: nxge38001
brdcstrcv
brdcstxmt
collisions
crtime
instance: 38001
class:
net
0
0
0
3842.493000352
0
ierrors
ifspeed
10000000000
2116069805
6411037101
ipackets
ipackets64
multircv
multixmt
norcvbuf
noxmtbuf
obytes
0
0
0
0
2757388874
obytes64
oerrors
23380264381002
0
opackets
opackets64
rbytes
37606022
4332573318
2937141290
rbytes64
47178857920554
52
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Enabling Jumbo Frames in a Solaris
Environment
This section describes how to enable jumbo frames in both a SPARC and an x86
environment.
▼ To Enable Jumbo Frames in a Solaris
Environment Using nxge.conf
1. Enable Jumbo Frames for a port using the nxge.conffile. For example,
name = "pciex108e,abcd" parent = "/pci@780/pci@0/pci@8/network@0"
unit-address = "0"
accept-jumbo=1;
2. Reboot the system:
% boot -r
3. Set the maximum MTU for maximum performance:
# ifconfig nxge mtu 9194
▼ To Check Layer 2 Configuration
●
View the maximum transmission unit (MTU) configuration of an nxgeinstance
at any time with the kstatcommand.
# kstat nxge:0 | grep mac_mtu
The kstatmac_mtu variable represents the complete size of the Ethernet frame,
which includes the Ethernet header, maximum payload, and crc. This value
should be equal to or less than the MTU configured on the switch.
Chapter 6 Configuring the Jumbo Frames Feature
53
▼ To Check Layer 3 Configuration
●
Check the Layer 3 configuration by using the dladmcommand with the show-
link option, as shown in the following example:
# dladm show-link
e1000g0
e1000g1
e1000g2
e1000g3
nxge0
type: non-vlan mtu: 1500
type: non-vlan mtu: 1500
type: non-vlan mtu: 1500
type: non-vlan mtu: 1500
type: non-vlan mtu: 9194
type: non-vlan mtu: 9194
type: vlan 38 mtu: 9194
device: e1000g0
device: e1000g1
device: e1000g2
device: e1000g3
device: nxge0
device: nxge1
device: nxge1
nxge1
nxge38001
Enabling Jumbo Frames in a Linux
Environment
This section describes how to enable jumbo frames in a Linux environment.
▼ To Enable Jumbo Frames in a Linux
Environment
1. Ensure that the nxgesoftware is installed.
# modprobe nxge
2. Plumb the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile
Adapter interface:
# ifconfig eth2 xxx.xxx.xx.xxx up
where xxx.xxx.xx.xxx is the IP address of the interface.
3. Set the MTU for maximum performance:
# ifconfig ethx mtu 9194
Where x is the instance number of the interface you want to configure jumbo
frames on.
54
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
▼ To Show the Driver Statistics in a Linux
Environment
1. Use the ifconfigutility to display driver statistics, for example:
# ifconfig eth4
eth9
Link encap:Ethernet HWaddr 00:14:4F:6C:78:E9
inet addr:38.38.9.42 Bcast:38.38.9.255
Mask:255.255.255.0
inet6 addr: fe80::214:4fff:fe6c:78e9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9194 Metric:1
RX packets:748274 errors:0 dropped:0 overruns:0 frame:0
TX packets:8645339 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:44983920 (42.9 MiB) TX bytes:75352303126 (70.1
GiB)
2. Use the ethtoolutility with -Soption to get more detailed information. This
option displays a large amount of statistics maintained by the driver.
# ethtool -S eth9 |grep rx_pac
rx_packets: 748274
rx_packets: 828
rx_packets: 112
rx_packets: 189088
rx_packets: 134
rx_packets: 196085
rx_packets: 177884
rx_packets: 93
rx_packets: 184050
Chapter 6 Configuring the Jumbo Frames Feature
55
3. To get more specific statistics, use the grepcommand on the output of ethtool -
S:
# ethtool -S eth9 |grep tx_pac
tx_packets: 8645339
tx_packets: 3418334
tx_packets: 26
tx_packets: 0
tx_packets: 22
tx_packets: 401791
tx_packets: 1466540
tx_packets: 0
tx_packets: 21
tx_packets: 725
tx_packets: 3357880
tx_packets: 0
tx_packets: 0
56
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
7
This chapter describes how to configure link aggregation. It contains the following
sections:
■
“Overview of Link Aggregation” on page 57
■
“Configuring Link Aggregation in a Linux Environment” on page 60
Overview of Link Aggregation
Link Aggregation allows one or more network links to be aggregated together to
form a link aggregation group. This link aggregation group appears to MAC clients
as a regular link. Link aggregation is defined by IEEE 802.2ad and it provides the
following benefits:
■
■
■
■
■
■
■
■
Increased bandwidth
Linearly incremental bandwidth
Load sharing
Automatic configuration
Rapid configuration and reconfiguration
Deterministic behavior
Low risk of duplication or mis-ordering
Support of existing IEEE 802.3 MAC clients
57
Configuring Link Aggregation in a
Solaris Environment
This section explains how to configure link aggregation in a Solaris environment.
▼ To Configure Link Aggregation in a Solaris
Environment
1. Aggregate nxge0 and nxge1 to form an aggregation and a random number 33 as
key.
a. Unplumb the interfaces to be aggregated:
# ifconfig down unplumb nxge0
# ifconfig down unplumb nxge1
b. Create a link-aggregation group with key 33 without specifying mode:
# dladm create-aggr -d nxge0 -d nxge1 33
As the command returns, one line appears in /etc/aggregation.conffile and
indicates that the default mode is off, as shown in the following example:
# tail -1 /etc/aggregation.conf
33 L4 nxge0/0,nxge1/0 auto
2
off
short
# dladm show-aggr
key: 33 (0x0021)
device
policy: L4
address
address: 0:3:ba:d8:9d:e8 (auto)
speed
duplex link
state
nxge0
nxge1
0:3:ba:d8:9d:e8 10000 Mbps full
0:3:ba:d8:9d:e9 10000 Mbps full
up
up
standby
standby
# dladm show-link aggr33
aggr33 type: non-vlan mtu: 1500
aggregation: key 33
58
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
2. Plumb up the interface aggrkey, which is aggr33is this case:
# ifconfig aggr33 plumb
# ifconfig aggr33
aggr33:flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4>mtu1500index8
inet 0.0.0.0 netmask 0
ether 0:3:ba:d8:9d:e8
# ifconfig aggr33 192.168.1.1/24 broadcast + up
# ifconfig aggr33
aggr33: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 8
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
ether 0:3:ba:d8:9d:e8
3. Show link-aggregation status again, and now the state should become attached:
# dladm show-aggr
key: 33 (0x0021)
policy: L4
address
0:3:ba:d8:9d:e8 10000 Mbps full
0:3:ba:d8:9d:e9 10000 Mbps full
address: 0:3:ba:d8:9d:e8 (auto)
device
nxge0
nxge1
speed
duplex link
state
up
up
attached
attached
4. Use the dladm show-aggr -scommand to display statistics:
# dladm show-aggr -s
key: 33
ipackets rbytes
opackets obytes %ipkts %opkts
Total
nxge0
nxge1
0
0
0
0
0
0
16
16
0
1182
1182
0
-
-
100.0
0.0
5. Use the dladm show-aggr -Lcommand to display LACP specific
information:
# dladm show-aggr -L
key: 33 (0x0021)
policy: L4
LACP mode: off LACP timer: short
activity timeout aggregatable sync coll dist defaulted expired
address: 0:3:ba:d8:9d:e8 (auto)
device
nxge0
nxge1
passive short yes
passive short yes
no
no
no no no
no no no
no
no
For more information refer to the manpages for dladm, man dladm.
Chapter 7 Configuring Link Aggregation
59
Configuring Link Aggregation in a
Linux Environment
This section explains how to configure link aggregation in a Linux environment.
▼ To Configure Bonding for Multiple nxge
Interfaces
1. Modify the /etc/modprobe.conffile for 2.6 kernels file by adding these
lines:
alias bond0 bonding
options bonding max_bonds=2 mode=4 miimon=1000
where:
■ bond0is the bonding device.
■ max_bondsis the number of bond interfaces to be created.
■ modespecifies the bonding policies.
■ miimonis the frequency in milliseconds that MII link monitoring will occur.
Refer to Linux documentation for more information.
2. Load the bonding driver:
# modprobe bonding
3. Configure bond0interface.
In this example, bond0is the master of the two interfaces eth4 and eth5.
# ip addr add 192.12.38.64/24 brd + dev bond0
# ip link set dev bond0 up
# ifenslave bond0 eth4 eth5
60
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
CHAPTER
8
Configuring VLANs
This chapter explains Virtual Local Area Networks (VLANs) in detail and provides
configuration instructions and examples. It contains the following sections:
■
■
■
“Overview of VLANs” on page 61
“Configuring VLANs in a Solaris Environment” on page 63
“Configuring VLANs in a Linux Environment” on page 65
With multiple VLANs on an adapter, a server with a single adapter can have a
logical presence on multiple IP subnets. By default, 128 VLANs can be defined for
each VLAN-aware adapter on your server. However, this number can be increased
by changing the system parameters.
If your network does not require multiple VLANs, you can use the default
configuration, in which case no further configuration is necessary.
Overview of VLANs
VLANs enable you to split your physical LAN into logical subparts, providing an
essential tool for increasing the efficiency and flexibility of your network.
VLANs are commonly used to separate groups of network users into manageable
broadcast domains, to create logical segmentation of workgroups, and to enforce
security policies among each logical segment. Each defined VLAN behaves as its
own separate network, with its traffic and broadcasts isolated from the others,
increasing the bandwidth efficiency within each logical group.
Although VLANs are commonly used to create individual broadcast domains
and/or separate IP subnets, it can be useful for a server to have a presence on more
than one VLAN simultaneously. Several Sun products support multiple VLANs on a
per port or per interface basis, allowing very flexible network configurations.
61
FIGURE 8-1 shows an example network that uses VLANs.
FIGURE 8-1 Example of Servers Supporting Multiple VLANs With Tagging Adapters
The example network has the following features:
The physical LAN network consists of a switch, two servers, and five clients. The
LAN is logically organized into three different VLANs, each representing a different
IP subnet.
■
■
■
VLAN 1 is an IP subnet consisting of the Main Server, Client 3, and Client 5. This
represents an engineering group.
VLAN 2 includes the Main Server, Clients 1 and 2 by means of a shared media
segment, and Client 5. This is a software development group.
VLAN 3 includes the Main Server, the Accounting Server and Client 4. This is an
accounting group.
The Main Server is a heavily used server that needs to be accessed from all VLANs
and IP subnets. The server has a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP
Low Profile Adapter installed. All three IP subnets are accessed by means of the
single physical adapter interface. The server is attached to one of the switch’s
62
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Gigabit Ethernet ports, which is configured for VLANs 1, 2, and 3. Both the adapter
and the connected switch port have tagging turned on. The tagging VLAN
capabilities of both devices enable the sever to communicate on all three IP subnets
in this network, yet continue to maintain broadcast separation among the three
subnets. The following list describes the components of this network:
■
The Accounting Server is available to VLAN 3 only. It is isolated from all traffic
on VLANs 1 and 2. The switch port connected to the server has tagging turned
off.
■
Clients 1 and 2 are attached to a shared media hub that is then connected to the
switch. They belong to VLAN 2 only, and are logically in the same IP subnet as
the Main Server and Client 5. The switch port connected to this segment has
tagging turned off.
■
■
■
Client 3 is a member of VLAN 1, and can communicate only with the Main Server
and Client 5. Tagging is not enabled on Client 3’s switch port.
Client 4 is a member of VLAN 3, and can communicate only with the servers.
Tagging is not enabled on Client 4’s switch port.
Client 5 is a member of both VLANs 1 and 2, and has a Sun x8 Express Dual 10
Gigabit Ethernet Fiber XFP Low Profile Adapter installed. It is connected to
switch port 10. Both the adapter and the switch port are configured for VLANs 1
and 2 and have tagging enabled.
VLAN tagging is only required to be enabled on switch ports that create trunk links
to other VLAN-aware Ethernet switches, or on ports connected to tag-capable end-
stations, such as servers or workstations with VLAN-aware adapters.
Configuring VLANs in a Solaris
Environment
VLANs can be created according to various criteria, but each VLAN must be
assigned a VLAN tag or VLAN ID (VID). The VID is a 12-bit identifier between 1
and 4094 that identifies a unique VLAN. For each network interface (nxge0 and
nxge1), 4094 possible VLAN IDs can be selected. Only 512 unique IDs can be used
simultaneously. Because IP subnets are commonly used, it is best to use IP subnets
when setting up a VLAN network interface. This means that each VID assigned to a
VLAN interface of a physical network interface will belong to different subnets.
Tagging an Ethernet frame requires the addition of a tag header to the frame. The
header is inserted immediately following the destination MAC address and the
Source MAC address. The tag header consists of two bytes of Ethernet Tag Protocol
identifier (TPID, 0x8100) and two bytes of Tag Control Information (TCI). FIGURE 8-2
shows the Ethernet Tag Header format.
Chapter 8 Configuring VLANs
63
FIGURE 8-2 Ethernet Tag Header Format
Octet
1
TPID (0x8100
3 bits
2
3
1
12 bytes
bit
4
User_priority
VID
CFI
By default, a single VLAN is configured for every port. This groups all ports into the
same broadcast domain, just as if there were no VLANs at all, VLAN tagging for the
switch port is turned off.
Note – If you configure a VLAN virtual device for an adapter, all traffic sent or
received by that adapter must be in VLAN-tagged format.
▼ To Configure Static VLANs
1. Create one hostname.nxgenumber file for each VLAN that will be configured
for each adapter on the server.
Use the following naming format, which includes both the VID and the physical
point of attachment (PPA):
VLAN logical PPA = 1000 * VID + Device PPA
nxge123000 = 1000*123 + nxge
This format limits the maximum number of PPAs (instances) you can configure to
1000 in the /etc/path_to_instfile.
For example, on a server with the Sun x8 Express Dual 10 Gigabit Ethernet Fiber
XFP Low Profile Adapter having an instance of 0, belonging to a member of two
VLANs, with VID 123 and 224, you would use nxge123000and nxge224000,
respectively, as the two VLAN PPAs.
64
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
2. Use the ifconfig(1M)to configure a VLAN virtual device, for example:
# ifconfig nxge123000 plumb up
# ifconfig nxge224000 plumb up
The output of ifconfig -aon a system having VLAN devices nxge123000
and nxge224000:
# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 129.144.131.91 netmask ffffff00 broadcast 129.144.131.255
ether 8:0:20:a4:4f:b8
nxge123000: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 199.199.123.3 netmask ffffff00 broadcast 199.199.123.255
ether 8:0:20:a4:4f:b8
nxge224000: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 199.199.224.3 netmask ffffff00 broadcast 199.199.224.225
ether 8:0:20:a4:4f:b8
3. On the switch, set VLAN tagging and set VLAN ports to coincide with the
VLANs you’ve set up on the server.
Using the examples in Step 2, you would set up VLAN ports 123 and 224 on the
switch.
Refer to the documentation that came with your switch for specific instructions
for setting VLAN tagging and ports.
Configuring VLANs in a Linux
Environment
▼ To Configure VLANs in a Linux Environment
1. Ensure that the nxgemodule is loaded:
# modprobe nxge
Chapter 8 Configuring VLANs
65
2. Plumb Oracle’s Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile
Adapter interface:
# ifconfig eth2 xxx.xxx.xx.xxx up
where xxx.xxx.xx.xxx = the IP address of the interface.
3. Insert the VLAN module:
# /sbin/modprobe 8021q
4. Add the VLAN instance (VID):
# /sbin/vconfig add eth2 5
5. Configure the nxgeVLAN (eth2in this example):
# ifconfig eth2.5 xxx.xxx.xx.xxx up
where xxx.xxx.xx.xxx = the IP address of the interface.
66
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
APPENDIX
A
XFP Low Profile Adapter. It contains the following sections:
■
■
■
■
“Connectors” on page 67
“Performance Specifications” on page 68
“Physical Characteristics” on page 69
“Power Requirements” on page 69
Connectors
FIGURE A-1 shows the connector for the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP
Low Profile Adapter.
FIGURE A-1 Sun x8 Express Dual 10 Gigabit Ethernet XFP Low Profile Adapter Connectors
67
TABLE A-1 lists the characteristics of the fiber connectors used by the Sun x8 Express Dual 10
Gigabit Ethernet Fiber XFP Low Profile Adapter.
TABLE A-1 XFP Connector Characteristics
Parameter
Description
Connector type
Distance
LC
10GBase-SR: Depending on fiber quality, up to 300 meters
for high bandwidth multimode fiber
10GBase-LR: Depending on fiber quality, 10km maximum
Fiber type
10GBase-SR = multimode
10GBase-LR = single mode
Performance Specifications
TABLE A-2 Performance Specifications
Feature
Specification
Bus type
x8 lane PCI-Express 1.1
x8 lane PCI-Express
Bus width
Bus speed (x8, encoded rate)
20 Gbit/sec uni-directional; 40 Gbit/sec bidirectional
(theoretical)
Maximum Ethernet transfer rate
Optics
10 Gbps (full-duplex)
10GBase-SR, 10GBase-LR, IEEE 802.ae 2002 compliant
68
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Physical Characteristics
TABLE A-3 Physical Characteristics
Dimension
Length
Width
Measurement
167.65 mm (6.6 inches)
68.90 mm (2.713 inches)
Height
Component side = 14.48 mm (0.57 inches) maximum
Solder side = 2.67 mm (0.105 inches) maximum
Power Requirements
TABLE A-4 Low-Profile Power Requirements
Specification
Measurement
Power consumption
20.74W typical
24.33W peak
Voltage
12V and 3.3V
Appendix A Specifications
69
70
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
APPENDIX
B
Diagnostic Software
instructions for updating the SunVTS software to recognize the adapter. This
appendix contains the following sections:
■
■
■
“SunVTS Diagnostic Software” on page 71
“Updating SunVTS to Recognize the Adapter” on page 72
“Using the SunVTS netlbtest” on page 73
SunVTS Diagnostic Software
The SunVTS software executes multiple diagnostic hardware tests from a single user
interface and is used to verify the configuration and functionality of most hardware
controllers and devices. The SunVTS software operates primarily from a graphical
user interface, enabling test parameters to be set quickly and easily while a
diagnostic test operation is being performed.
Note – SunVTS diagnostic software is not currently available for Solaris x86
Operating Systems.
The SunVTS nettestdiagnostic can be used to test all of the networking interfaces
on the system, including the interfaces on the adapter.
To use the nettestdiagnostic, you must have the SunVTS software installed on
your system. Refer to your Solaris documentation for installation instructions.
Refer to the SunVTS documentation (listed in TABLE B-1) for instructions on how to
run and monitor the nettestdiagnostic. These SunVTS documents are available
online at the following URL:
71
Select the document for the Solaris release on your system.
TABLE B-1 SunVTS Documentation
Title
Description
SunVTS User’s Guide
Describes the SunVTS diagnostic environment
SunVTS Test Reference Manual
Describes each SunVTS test (including the nettest)
and describes the various test options and
command-line arguments
SunVTS Quick Reference
Provides an overview of the user interface
Updating SunVTS to Recognize the
Adapter
Use SunVTS 6.3 or later. You will need to update the SunVTS configuration to
recognize the adapter.
▼ To Update SunVTS to Recognize the Adapter
1. Plug in a loopback cable.
2. Ensure that the SunVTS software and the nxgedriver are installed on your
system.
3. Add the following lines to the /opt/SUNWvts/lib/conf/netlbtest.conf
and /opt/SUNWvts/lib/conf/nettest.conffiles:
nxge nxge 10gbaset
72
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Using the SunVTS netlbtest
You must have the Ethernet card and the device driver installed, a loopback
connector in place, and Intervention mode enabled before running netlbtest.
netlbtestcannot run if the network interface is connected to a live network, and
requires that the Ethernet device be configured offline before running the test. Use
the ifconfig(1M)command to bring the Ethernet device down before running
netlbtest.
▼ To Use the netlbtest
1. Ensure that the SunVTS software and the nxgedriver are installed on your
system.
2. Plug in a loopback cable.
3. Unplumb the interface from the system, using the ifconfigcommand:
# ifconfig nxgeinstance down
# ifconfig nxgeinstance unplumb
Where instance is the instance number of the interface.
Refer to SunVTS documentation for instructions on how to run netlbtest.
Appendix B Diagnostic Software
73
74
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
Index
Symbols
N
A
P
assigning an IP address, 24
C
configuring VLANs, 63
setting nxge driver parameters
software
requirements, 3
D
diskless clients, 25
documentation
SunVTS, 72
E
editing the network host files, 23
H
hardware
virtual device, 65
VLAN ID, 63
VLAN naming format, 64
VLANs, 61
requirements, 3
I
ifconfig -a, 65
ifconfig command, 24
installing the adapter without DR, 13
installing the Solaris operating environment over the
network, 27
75
76
Sun Dual 10GbE XFP PCI Express Card User’s Guide • May 2010
|