Fujitsu Computer Drive ISCSI for VMware ESX User Manual

Download from Www.Somanuals.com. All Manuals Search And Download.  
This page is intentionally left blank.  
Download from Www.Somanuals.com. All Manuals Search And Download.  
Preface  
This manual briefly explains the operations that need to be performed by the user in order to  
connect an ETERNUS DX60 S2/DX80 S2/DX90 S2, ETERNUS DX410 S2/DX440 S2,  
ETERNUS DX8100 S2/DX8700 S2, ETERNUS DX60/DX80, ETERNUS DX410/DX440, or  
ETERNUS DX8100/DX8400/DX8700 Disk storage system to a server running VMware ESX  
using iSCSI Software Initiator via an iSCSI interface.  
This manual should be used in conjunction with any other applicable user manuals, such as  
those for the ETERNUS DX60 S2/DX80 S2/DX90 S2, ETERNUS DX410 S2/DX440 S2, ETER-  
NUS DX8100 S2/DX8700 S2, ETERNUS DX60/DX80, ETERNUS DX410/DX440, or ETERNUS  
DX8100/DX8400/DX8700 Disk storage system, server, OS, adapters, and drivers.  
This manual references the following documents:  
Server Support Matrix  
ETERNUS DX Disk storage systems User's Guide -Server Connection- (iSCSI)  
Disk Storage System Settings (ETERNUS DX80 S2/DX90 S2, ETERNUS DX410 S2/DX440  
S2, ETERNUS DX8100 S2/DX8700 S2)  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX60/DX60 S2, DX80  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX400/DX8000 series  
Web GUI User’s Guide  
ETERNUSmgr Install Guide  
ETERNUSmgr User Guide  
Also, note that in this manual the ETERNUS DX60 S2/DX80 S2/DX90 S2, ETERNUS DX410 S2/  
DX440 S2, ETERNUS DX8100 S2/DX8700 S2, ETERNUS DX60/DX80, ETERNUS DX410/  
DX440, and ETERNUS DX8100/DX8400/DX8700 Disk storage systems are collectively referred  
to as ETERNUS DX Disk storage systems.  
Eleventh Edition  
September 2012  
3
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Preface  
The Contents and Structure of this Manual  
This manual is composed of the following seven chapters.  
This describes how to connect the ETERNUS DX Disk storage systems to a server running  
VMware ESX.  
This describes which servers can be connected to ETERNUS DX Disk storage systems.  
This describes issues that should be noted when connecting the ETERNUS DX Disk storage  
systems and server.  
This describes how to install ETERNUSmgr.  
This describes how to set up the ETERNUS DX Disk storage systems.  
This describes how to set up the VMware ESX server.  
This describes how to operate the Virtual Machine.  
Acknowledgments  
VMware, VMware logos, Virtual SMP, and VMotion are either registered trademarks or  
trademarks of VMware, Inc. in the U.S. and/or other countries.  
Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of  
Microsoft Corporation in the U.S. and/or other countries.  
Linux is a trademark or registered trademark of Linus Torvalds in the USA and other  
countries.  
Red Hat, PRM, and all Red Hat-based trademarks and logos are trademarks or registered  
trademarks of Red Hat, Inc. in the USA and other countries.  
SUSE is a registered trademark of SUSE Linux AG., a subsidiary of Novell, Inc.  
The company names, product names and service names mentioned in this document are  
registered trademarks or trademarks of their respective companies.  
4
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Preface  
Naming Conventions  
Product names  
"VMware ESX" represents the following products.  
- VMware vSphere (ESXi 5.1, ESXi 5.0, ESX 4.1, ESXi 4.1, ESX 4.0, ESXi 4.0)  
- VMware Infrastructure 3 (ESX 3.5, ESXi 3.5)  
"Windows Server® 2003" represents the following products.  
- Microsoft® Windows Server® 2003, Standard Edition  
- Microsoft® Windows Server® 2003, Enterprise Edition  
- Microsoft® Windows Server® 2003, Standard x64 Edition  
- Microsoft® Windows Server® 2003, Enterprise x64 Edition  
- Microsoft® Windows Server® 2003 R2, Standard Edition  
- Microsoft® Windows Server® 2003 R2, Enterprise Edition  
- Microsoft® Windows Server® 2003 R2, Standard x64 Edition  
- Microsoft® Windows Server® 2003 R2, Enterprise x64 Edition  
"Linux" represents the following products.  
- Red Hat Enterprise Linux 5 (for x86)  
- Red Hat Enterprise Linux 5 (for Intel64)  
- Red Hat Enterprise Linux AS (v.4 for x86)  
- Red Hat Enterprise Linux AS (v.4 for EM64T)  
- Red Hat Enterprise Linux ES (v.4 for x86)  
- Red Hat Enterprise Linux ES (v.4 for EM64T)  
- Red Hat Enterprise Linux WS (v.4)  
- SUSE Linux Enterprise Server  
- SUSE Linux Enterprise Server 11  
- SUSE Linux Enterprise Server 11 Service Packx  
- SUSE Linux Enterprise Server 11 for x86  
- SUSE Linux Enterprise Server 11 for EM64T  
- SUSE Linux Enterprise Server 11 Service Packx for x86  
- SUSE Linux Enterprise Server 11 Service Packx for EM64T  
- SUSE Linux Enterprise Server 10  
- SUSE Linux Enterprise Server 10 Service Packx  
- SUSE Linux Enterprise Server 10 for x86  
- SUSE Linux Enterprise Server 10 for EM64T  
- SUSE Linux Enterprise Server 10 Service Packx for x86  
- SUSE Linux Enterprise Server 10 Service Packx for EM64T  
- SUSE Linux Enterprise Server 9  
- SUSE Linux Enterprise Server 9 Service Packx  
- SUSE Linux Enterprise Server 9 for x86  
- SUSE Linux Enterprise Server 9 for EM64T  
- SUSE Linux Enterprise Server 9 for x86 Service Packx  
- SUSE Linux Enterprise Server 9 for EM64T Service Packx  
5
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Preface  
Other names  
"iSCSI port" refers to an iSCSI interface module used in the ETERNUS DX Disk storage  
systems to connect to the server.  
"LAN card" refers to the iSCSI interface module normally used in the server. An "onboard  
LAN", "network interface card" (NIC), "LAN adapter", or "LAN board" may be used instead.  
"iSCSI cable" refers to the cable that is used to connect the ETERNUS DX Disk storage  
system and server over an iSCSI interface. "Ethernet cable", "LAN cable", and "twisted pair  
cable" are alternative names for this cable.  
"VMware ESX" refers to "VMware vSphere" and "VMware Infrastructure 3", which are  
datacenter solutions from VMware that virtualize the storage and networking system.  
The name "VMware vSphere" and "VMware Infrastructure 3" are used in specific datacenter  
sections.  
Italics are used to show variables such as values and characters that appear in command  
parameters and output examples.  
ETERNUS Disk storage system models  
The following naming conventions are used to describe ETERNUS Disk storage system models  
in this manual.  
ETERNUS Disk storage system model  
Naming conventions  
ETERNUS DX60 S2/DX80 S2/DX90 S2  
ETERNUS DX410 S2/DX440 S2  
ETERNUS DX8100 S2/DX8700 S2  
ETERNUS DX60/DX80  
ETERNUS DX Disk storage system  
ETERNUS DX410/DX440  
ETERNUS DX8100/DX8400/DX8700  
ETERNUS DX60/DX80  
ETERNUS DX Disk storage system (excluding S2)  
ETERNUS DX410/DX440  
ETERNUS DX8100/DX8400/DX8700  
ETERNUS DX410 S2/DX440 S2  
ETERNUS DX8100 S2/DX8700 S2  
ETERNUS DX410/DX440  
ETERNUS DX400 S2 series  
ETERNUS DX8000 S2 series  
ETERNUS DX400 series  
ETERNUS DX8000 series  
ETERNUS DX8100/DX8400/DX8700  
Microsoft product screen shot(s) reprinted with permission from Microsoft Corporation.  
6
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Table of Contents  
2.3  
Connection Compatibility of ETERNUS DX Disk Storage Systems to  
7
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Chapter 1 Workflow  
This chapter describes how to connect the ETERNUS DX Disk storage systems to a server that  
is running VMware ESX.  
The workflow is shown below.  
Required Documents List  
Server Support Matrix  
ETERNUS DX Disk storage systems User's Guide -Server Connection- (iSCSI)  
Disk Storage System Settings (ETERNUS DX80 S2/DX90 S2, ETERNUS DX410 S2/DX440  
S2, ETERNUS DX8100 S2/DX8700 S2)  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX60/DX60 S2, DX80  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX400/DX8000 series  
ETERNUS DX Disk storage systems User's Guide -Server Connection- (iSCSI)  
for VMware® ESX (this manual)  
Web GUI User's Guide  
ETERNUSmgr Install Guide  
ETERNUSmgr User Guide  
9
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 1 Workflow  
Workflow  
Install ETERNUSmgr and Set Up the ETERNUS DX Disk Storage System  
If ETERNUSmgr is to be used, install it and set up the ETERNUS DX Disk storage  
system.  
Refer  
- Checking the setup and maintenance procedures.  
"Web GUI User’s Guide  
"ETERNUSmgr Install Guide"  
"ETERNUSmgr User Guide"  
- Setting up the ETERNUS DX Disk storage system.  
ETERNUS DX Disk storage systems User's Guide -Server Connection- (iSCSI)  
Disk Storage System Settings (ETERNUS DX80 S2/DX90 S2, ETERNUS DX410  
S2/DX440 S2, ETERNUS DX8100 S2/DX8700 S2)  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX60/DX 60 S2,  
DX80  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX400/DX8000  
series  
Set Up the VMware ESX Server  
Install the LAN card in the server, then set up the VMware ESX server to recognize LUNs.  
Refer  
Start the Virtual Machine  
Notes on Virtual Machine operation and related settings are given in this manual and  
other documents.  
Refer  
10  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Chapter 2 Checking the Server Environment  
ETERNUS DX Disk storage systems can be connected in the following environments. Check the  
environment of your server.  
2.1  
Hardware  
VMware ESX connection requires the use of LAN switches.  
Use the guide at the following URL and the "Server Support Matrix" to check which servers  
are supported:  
Search the VMware Compatibility Guide  
2.2  
2.3  
LAN Cards  
Refer to the "Server Support Matrix" to check which LAN cards are supported by the ETERNUS  
DX Disk storage system.  
Connection Compatibility of ETERNUS DX Disk  
Storage Systems to VMware ESX  
Use the guide at the following URL to determine which ETERNUS DX Disk storage system  
models may be connected to VMware ESX:  
Search the VMware Compatibility Guide  
11  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
         
Chapter 2 Checking the Server Environment  
2.4 Virtual Machine  
2.4  
Virtual Machine  
Virtual Machine is a virtual machine created on the VMware ESX server. Details of how to install  
an OS on the VMware ESX Virtual Machine may be checked via the following URL:  
12  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 3 Notes  
This chapter describes notes when the ETERNUS DX Disk storage systems are connected to  
the server via iSCSI interface.  
3.1  
Connection Notes  
3.1.1  
For VMware vSphere  
Dedicated IP addresses must be allocated to the iSCSI LAN. An IP address must be allocated to  
each VMkernel that corresponds to an iSCSI Initiator, and a LAN card and VMkernel must be  
connected via a virtual switch. For example, a total of four IP addresses are required by the fol-  
lowing iSCSI connection setup.  
Example connection configuration  
VMware ESX server  
VMkernel (iSCSI Software Initiator)  
IP address (1)  
VMkernel (iSCSI Software Initiator)  
IP address (2)  
LAN card  
LAN card  
LAN switch  
LAN switch  
IP address  
IP address  
for iSCSI port (3)  
for iSCSI port (4)  
ETERNUS DX Disk storage system  
13  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
       
Chapter 3 Notes  
3.2 VMware ESX Operating Notes  
3.1.2  
For VMware Infrastructure 3  
Dedicated IP addresses must be allocated to the iSCSI LAN. An IP address must be allocated to  
each VMkernel that corresponds to an iSCSI Initiator, and a LAN card and VMkernel must be  
connected via a virtual switch. For example, a total of six IP addresses are required by the follow-  
ing iSCSI connection setup.  
Example connection configuration  
VMware ESX server  
VMkernel (iSCSI Software Initiator)  
IP address (1)  
VMkernel (iSCSI Software Initiator)  
IP address (3)  
IP address for Service Console (2)  
IP address for Service Console (4)  
LAN card  
LAN card  
LAN switch  
LAN switch  
IP address  
IP address  
for iSCSI port (5)  
for iSCSI port (6)  
ETERNUS DX Disk storage system  
3.2  
VMware ESX Operating Notes  
Refer to the following web-site for the number of LUNs that VMware ESX can recognize.  
When Windows® is used on the Virtual Machine, the registry will need to be modified after  
Windows® is installed. For details, refer to "Chapter 7 Virtual Machine" (page 38).  
The VMware ESX multipath function supports path failover, meaning that server access can  
continue unaffected by any problems that might arise in the iSCSI cables or LAN switches.  
It should be noted that path failover can fail to occur when multiple SCSI sense codes are  
repeatedly and simultaneously received.  
Set the "Path Selection Policy" (per-LUN) as follows:  
- For VMware vSphere: "RoundRobin" is recommended  
- For VMware Infrastructure 3: "Fixed"  
ETERNUS Multipath Driver and GR Multipath Driver do not need to be installed on the  
VMware ESX Virtual Machine. Multipath function of VMware ESX provides path redundancy.  
14  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 3 Notes  
3.3 ETERNUS DX Disk Storage System Setup Notes  
3.3  
ETERNUS DX Disk Storage System Setup Notes  
When connecting the ETERNUS DX Disk storage system to VMware ESX, check that the  
firmware version is as specified in the "Server Support Matrix".  
Assign Affinity Group values starting from LUN0 in ascending order.  
The server cannot recognize the ETERNUS DX Disk storage system LUNs if some other  
assignment order is used.  
When LUNs are shared among multiple physical servers (in a VMotion configuration, for  
example), an Affinity Group mapping should be used to ensure that each shared LUN is  
assigned the same LUN number across every physical server.  
When connecting to VMware ESX with multiple paths sharing a single LUN, a Reset Group  
setting is required for the ETERNUS DX Disk storage system.  
3.4  
3.5  
SAN Boot Notes  
The ETERNUS DX Disk storage systems do not support SAN Boot when they are connected via  
iSCSI Software Initiator.  
Server Startup and Power Supply Control Notes  
Before turning the server on, check that the ETERNUS DX Disk storage systems and LAN  
switches are all "Ready". Specifically, check that the Ready LED is lit for ETERNUS DX Disk  
storage systems. If the server is turned on and they are not "Ready", the server will not be able to  
recognize the ETERNUS DX Disk storage systems.  
Also, when the ETERNUS DX Disk storage system power supply is being controlled by a con-  
nected server, make sure that the ETERNUS DX Disk storage system does not shut down  
before the connected servers. Similarly, the LAN switches must also be turned off after the con-  
nected servers have been shut down. If turned off, data writes from the running server cannot be  
saved to the ETERNUS DX Disk storage systems, and already saved data may also be affected.  
15  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
     
Chapter 3 Notes  
3.6 LAN Switch Connection Notes  
3.6  
LAN Switch Connection Notes  
Since an iSCSI LAN handles large amounts of data (traffic volumes) like an FC-SAN, the  
iSCSI LAN must have its own switch and be a dedicated LAN separate from the business  
LANs.  
iSCSI LAN redundancy is achieved by the use of multipaths.  
For IP network security, the iSCSI LAN assumed to be a dedicated LAN separated from the  
management LAN (for administration).  
The iSCSI LAN must be configured as a dedicated LAN for each path from a server to the  
ETERNUS DX Disk storage system.  
Example of a LAN switch connection configuration  
LAN switch  
[Business LAN]  
Business LAN  
LAN  
LAN  
LAN  
Card#1  
Card#1  
Card#1  
Similarly to FC-SAN,  
Business server A  
Business server B  
Business server C  
a dedicated LAN is used for iSCSI,  
not the business LAN.  
LAN  
LAN  
LAN  
LAN  
LAN  
LAN  
Card#2  
Card#3  
Card#2  
Card#3  
Card#2  
Card#3  
1
2
3
1
2
3
LAN switch #1  
[iSCSI LAN]  
LAN switch #2  
[iSCSI LAN]  
LAN switches should  
not be inter-connected*1  
iSCSI LAN  
10  
11  
12  
10  
11  
12  
Management LAN  
LUN1  
LUN2  
LUN3  
Management  
LAN port  
LAN switch  
[Management LAN]  
ETERNUS DX Disk storage system  
A separate LAN segment is used for each  
Server/Storage grouping*2.  
Here, LAN ports 1, 2 and 10 comprise VLAN1,  
while LAN ports 3 and 11 comprise VLAN2.  
The iSCSI LAN and management LAN  
segments are kept separate in the iSCSI  
LAN switches, helping maintain the security  
of each.  
*1: In this system configuration, multipaths provide redundant connections between the servers and  
storage system. LAN switches #1 and #2 provide physical separation of the network paths.  
*2: A separate LAN segment is provided in the LAN switch (using the switch VLAN function) for each  
grouping of business servers and disk storage systems (equivalent to the FC zones).  
16  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 3 Notes  
3.7 Jumbo Frame Setting Notes  
Example of a network address configuration  
Server B  
Server A  
192.168.10.1/24  
192.168.20.1/24  
192.168.30.1/24  
192.168.40.1/24  
LAN switch #1  
(VLAN is  
recommended)  
LAN switch #2  
(VLAN is  
recommended)  
192.168.20.10/24  
192.168.40.10/24  
192.168.10.10/24  
192.168.30.10/24  
P0  
P1  
P0  
P1  
CM0  
CM1  
ETERNUS DX Disk storage system  
3.7  
3.8  
Jumbo Frame Setting Notes  
To enable Jumbo Frame, all the connected devices must support Jumbo Frame. Set the  
appropriate values for various parameters (such as the MTU size) on each connected device.  
For details about how to set Jumbo Frame for a LAN card and LAN switch, refer to the  
VMware ESX and each device's manuals. Rebooting the server may be required to apply the  
new settings.  
The MTU size that is supported by the ETERNUS DX60/DX60 S2, DX80/DX80 S2, and DX90  
S2 is 9000 bytes.  
LAN Switch Setting Notes  
When using a LAN switch that supports the flow control function, disable the flow control function  
according to the following notes:  
For access paths that use iSCSI, disable the flow control function on the sending and  
receiving ends of the port.  
When a cascade connection is used to connect to LAN switches, disable the function for the  
LAN switch that is directly connected to the ETERNUS DX Disk storage system.  
For details on how to set the flow control function of the LAN switch, refer to the LAN switch man-  
ual.  
17  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 4 Installing and Setting Up ETERNUSmgr  
If ETERNUSmgr is to be used, install it according to the directions given in the "ETERNUSmgr  
Install Guide". After the installation, set up ETERNUSmgr following the instructions in the  
"ETERNUSmgr User Guide".  
18  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 5 Setting Up the ETERNUS DX Disk  
Storage Systems  
Set up the ETERNUS DX Disk storage systems using Web GUI or ETERNUSmgr.  
ETERNUS DX Disk storage systems' setup can be performed independently of server setup.  
For details on how to perform these settings, refer to the following documents:  
ETERNUS DX Disk storage systems User's Guide -Server Connection- (iSCSI)  
Disk Storage System Settings (ETERNUS DX80 S2/DX90 S2, ETERNUS DX410 S2/DX440  
S2, ETERNUS DX8100 S2/DX8700 S2)  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX60/DX 60 S2, DX80  
ETERNUS DX Disk storage systems Server Connection Guide (iSCSI)  
ETERNUS DX Disk Storage System Settings for ETERNUS DX400/DX8000 series  
"Web GUI User’s Guide" or "ETERNUSmgr User Guide"  
19  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 6 Setting Up the VMware ESX Server  
This chapter describes settings related to the server iSCSI interface.  
6.1  
Setting Up for VMware vSphere  
Set up the VMware ESX server using the vSphere Client. The procedures described here are  
based on the following multipath configuration.  
Example connection configuration  
VMware ESX server  
VMkernel (iSCSI Software Initiator)  
IP address (1)  
VMkernel (iSCSI Software Initiator)  
IP address (2)  
LAN card  
LAN card  
LAN switch  
LAN switch  
IP address  
IP address  
for iSCSI port (3)  
for iSCSI port (4)  
ETERNUS DX Disk storage system  
6.1.1  
Turning on the Devices  
To turn on the connected devices, use the following procedures:  
Procedure  
1
2
3
4
Turn on the LAN switch power.  
Check that the LAN switch's Ready (or equivalent) LED is lit.  
Turn on the ETERNUS DX Disk storage systems.  
Check that the ETERNUS DX Disk storage systems' Ready LED is lit.  
20  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
       
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
5
Turn on the server.  
End of procedure  
6.1.2  
Checking the LAN Cards  
Procedure  
1
2
Log in to VMware ESX from the vSphere Client and select the [Configuration]  
tab.  
Click [Network Adapters] in the [Hardware] list.  
"vmnic1" and "vmnic2" in the following example.  
End of procedure  
6.1.3  
Creating the Virtual Switches  
Add two virtual switches (vSwitch) for iSCSI to VMware ESX. Add a "VMkernel" and "vmnic" for  
each vSwitch.  
Perform the following procedure to each vmnic that configures iSCSI SAN.  
Procedure  
1
2
Open the network window by selecting the [Configuration] tab on vSphere Client  
and click [Add Network] in the upper right of the window.  
Select [VMkernel] and click the [Next] button.  
21  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
     
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
3
Select the checkbox of the target network adapter and click the [Next] button.  
4
5
Set properties of port groups as required, and click the [Next] button.  
Set the IP address and subnet mask of "VMkernel", and click the [Next] button.  
6
7
Repeat Step 1 through Step 5 to add another virtual switch (vSwitch2).  
Confirm that a virtual switch and VMkernel are set for each vmnic.  
End of procedure  
22  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
6.1.4  
Setting the Software Initiator  
Enable Software Initiator in VMware ESX.  
Procedure  
1
2
Select the [Configuration] tab on vSphere Client and click [Storage Adapters].  
Select the target iSCSI Software Adapter and click [Properties...].  
3
4
5
Click [Configure] on the [General] tab.  
Select the [Status-Enabled] checkbox and click the [OK] button.  
Confirm that the [Enabled] checkbox is selected under "Status" on the [General]  
tab and click [Configure] again.  
6
Confirm that the iSCSI name is displayed in the "iSCSI Name" column and click  
the [Cancel] button.  
7
8
Select the [Dynamic Discovery] tab and click the [Add] button.  
Enter the IP address for the iSCSI port of the connected ETERNUS DX Disk  
storage system as the iSCSI server IP address, confirm that the port is set to  
"Port 3260" (default), and click the [OK] button.  
23  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
9
Confirm that the IP address for the iSCSI port of the ETERNUS DX Disk storage  
system is displayed as follows.  
10 If the connected ETERNUS DX Disk storage system uses multiple iSCSI ports,  
repeat the IP address addition process for each iSCSI port. (Repeat Step 7  
through Step 9.)  
End of procedure  
6.1.5  
Enabling ALUA  
This procedure is only required when VMware vSphere contains VMware  
ESX 4.0/VMware ESXi 4.0.  
IMPORTANT  
It does not need to be performed when VMware vSphere contains VMware  
ESX 4.0 Update 1/VMware ESXi 4.0 Update 1 or later versions. In this  
After the VMware ESX server has been installed, Asymmetric Logical Unit Access (ALUA)  
should be enabled according to the following procedure.  
Procedure  
1
2
Use the terminal to log in to VMware ESX as "root".  
Execute the following command in the terminal.  
When connecting to an ETERNUS DX60/DX60 S2, DX80/DX80 S2, or DX90 S2  
esxcli nmp satp addrule --satp="VMW_SATP_ALUA" --vendor="FUJITSU" --  
model="ETERNUS_DXL" --description="ETERNUS DX with ALUA" --claim-  
option tpgs_on  
When connecting to an ETERNUS DX400/DX400 S2 series  
esxcli nmp satp addrule --satp="VMW_SATP_ALUA" --vendor="FUJITSU" --  
model="ETERNUS_DX400" --description="ETERNUS DX400 with ALUA"  
--claim-option tpgs_on  
24  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
When connecting to an ETERNUS DX8000/DX8000 S2 series  
esxcli nmp satp addrule --satp="VMW_SATP_ALUA" --vendor="FUJITSU" --  
model="ETERNUS_DX8000" --description="ETERNUS DX8000 with ALUA"  
--claim-option tpgs_on  
3
Execute the following command in the terminal.  
esxcli corestorage claiming unclaim --type location  
Ignore the following error message if it appears when the preceding  
command is executed:  
Errors:  
Unable to perform unclaim. Error message was : Unable to unclaim  
paths. Busy or in use devices detected. See VMkernel logs for  
more information.  
4
5
Execute the following command in the terminal.  
esxcli corestorage claimrule run  
Check that the connected ETERNUS DX Disk storage system is shown when  
the following command is executed in the terminal:  
Command  
esxcli nmp satp listrules --satp VMW_SATP_ALUA  
Display result  
When connecting to an ETERNUS DX60/DX60 S2, DX80/DX80 S2, or DX90 S2  
VMW_SATP_ALUA FUJITSU ETERNUS_DXL tpgs_on ETERNUS_DXL with ALUA  
When connecting to an ETERNUS DX400/DX400 S2 series  
VMW_SATP_ALUA FUJITSU ETERNUS_DX400 tpgs_on ETERNUS_DX400 with ALUA  
When connecting to an ETERNUS DX8000/DX8000 S2 series  
VMW_SATP_ALUA FUJITSU ETERNUS_DX8000 tpgs_on ETERNUS_DX8000 with ALUA  
6
Log out of the terminal, and restart the VMware ESX server.  
End of procedure  
25  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
6.1.6  
Checking the LUNs  
The following procedure describes how to check LUN recognition using the vSphere Client.  
Log in to VMware ESX from the vSphere Client, and then check whether the ETERNUS DX Disk  
storage system LUNs have been recognized.  
The procedure is as follows:  
Procedure  
1
2
3
4
Use the vSphere Client to login to VMware ESX as "root".  
Select the [Configuration] tab.  
Select [Storage Adapters] from the [Hardware] list.  
Select [Rescan...].  
After selecting [Rescan...], VMware ESX should attempt to recognize  
the ETERNUS DX Disk storage systems' LUNs again.  
5
If the iSCSI Software Adapter (vmhba34 in this example) is selected from the  
[Storage Adapters] area, the recognized devices will be shown in the [Details]  
area, as follows.  
26  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
6
Check the [Path Selection] for all the LUNs in the ETERNUS DX Disk storage  
system.  
When the VMware ESX recognizes the LUN, an active path for each LUN is automatically  
specified. Even though [Path Selection] is set to [Most Recently Used (VMware)],  
changing the [Path Selection] for all the LUNs to [RoundRobin] is recommended.  
Refer to the following URL or guide for details about the command above:  
URL  
:
:
Guide  
"vSphere Command-Line Interface Concepts and Examples"  
27  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
7
For a multipath configuration, confirm that the paths of all the LUNs in the ETER-  
NUS DX Disk storage system are configured with multipath. When paths for a  
LUN are configured with multipath, multiple Runtime Names and Targets are dis-  
played in [Paths].  
End of procedure  
6.1.7  
Setting the CHAP Authentication  
CHAP and mutual CHAP authentication can be set for VMware ESX connections.  
Log in to VMware ESX from the vSphere Client, check whether the ETERNUS DX Disk storage  
system LUNs have been recognized, and then enable the CHAP authentication.  
The procedure is as follows:  
Procedure  
1
2
3
Select the [Configuration] tab on vSphere Client and select [Storage Adapters].  
Select the target iSCSI Software Adapter and click [Properties].  
Select the [General] tab and click [CHAP].  
28  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.1 Setting Up for VMware vSphere  
4
Set CHAP.  
When setting CHAP only  
Select [Required] for [Select option:] under "CHAP" and set the [Name:] and [Secret:].  
When setting CHAP and mutual CHAP  
Select [Required] for [Select option:] under "CHAP" and "Mutual CHAP" and set the  
[Name:] and [Secret:].  
End of procedure  
29  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
6.2  
Setting Up for VMware Infrastructure 3  
Set up the VMware ESX server using the VMware Infrastructure Client. The procedures  
described here are based on the following multipath configuration.  
Example connection configuration  
VMware ESX server  
VMkernel (iSCSI Software Initiator)  
IP address (1)  
VMkernel (iSCSI Software Initiator)  
IP address (3)  
IP address for Service Console (2)  
IP address for Service Console (4)  
LAN card  
LAN card  
LAN switch  
LAN switch  
IP address  
IP address  
for iSCSI port (5)  
for iSCSI port (6)  
ETERNUS DX Disk storage system  
6.2.1  
Turning on the Devices  
To turn on the connected devices, use the following procedures:  
Procedure  
1
2
3
4
5
Turn on the LAN switch power.  
Check that the LAN switch's Ready (or equivalent) LED is lit.  
Turn on the ETERNUS DX Disk storage systems.  
Check that the ETERNUS DX Disk storage systems' Ready LED is lit.  
Turn on the server.  
End of procedure  
30  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
6.2.2  
Checking the LAN Cards  
Procedure  
1
2
Select the [Configuration] tab on VMware Infrastructure Client.  
Check the iSCSI SAN configuration Network Adapters.  
"vmnic1" and "vmnic2" in the following example.  
End of procedure  
6.2.3  
Creating the Virtual Switches  
Add two virtual switches (vSwitch) for iSCSI to VMware ESX.  
Define two virtual network port types ("Service Console" and "VMkernel") for the added vSwitch.  
Also assign IP addresses on the same subnet to the defined Service Console and VMkernel.  
6.2.3.1  
Adding a Service Console  
Procedure  
1
Select the [Configuration] tab on VMware Infrastructure Client and click [Add  
Networking].  
2
3
4
Select [Service Console] and click the [Next] button.  
Select the checkbox of the target network adapter and click the [Next] button.  
Check the [Service Console] details, set an arbitrary VLAN ID, IP address, and  
subnet mask, and click the [Next] button.  
31  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
     
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
5
6
Check the [Service Console] IP address and click the [Finish] button.  
Confirm that the virtual switch (vSwitch1) has been created as follows.  
Repeat Step 1 though Step 5 to add Service Console to vSwitch2.  
End of procedure  
6.2.3.2  
Adding a VMkernel  
Procedure  
1
Select the [Configuration] tab on VMware Infrastructure Client and select the  
target virtual switch' s [Properties].  
2
3
4
Click [Add].  
Select the [VMkernel] radio button, and click the [Next] button.  
Confirm that the VMkernel is displayed, set the IP address (an optional VLAN ID  
may be set, as necessary), and click the [Next] button.  
5
6
Confirm the settings shown on the confirmation screen and click the [Finish]  
button.  
A message about Gateway settings appears.  
Perform any further settings as necessary and click the [Close] button.  
The VMkernel has been added to the target vSwitch.  
7
Repeat Step 1 though Step 6 to add VMkernel to another vSwitch.  
32  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
     
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
8
Confirm that a virtual switch and VMkernel are set for each vmnic.  
End of procedure  
6.2.4  
Setting the Software Initiator  
Enable Software Initiator in VMware ESX.  
Procedure  
1
Select the [Configuration] tab on VMware Infrastructure Client and click [Storage  
Adapters].  
33  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
2
Select the target iSCSI Software Adapter and click [Properties].  
3
4
5
Click [Configure] on the [General] tab.  
Select the [Status-Enabled] checkbox and click the [OK] button.  
Confirm that the [Enabled] checkbox is selected under "Status" on the [General]  
tab and click [Configure] again.  
6
Confirm that the iSCSI name is displayed as follows and click the [Cancel]  
button.  
7
8
Select the [Dynamic Discovery] tab and click the [Add] button.  
Enter the IP address for the iSCSI port of the connected ETERNUS DX Disk  
storage system as the iSCSI server IP address, confirm that the port is set to  
"Port 3260" (default), and click the [OK] button.  
34  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
9
Confirm that the IP address for the iSCSI port of the ETERNUS DX Disk storage  
system is displayed as follows and click [Close].  
10 If the connected ETERNUS DX Disk storage system uses multiple iSCSI ports,  
repeat the IP address addition process for each iSCSI port. (Repeat Step 7  
through Step 9.)  
End of procedure  
6.2.5  
Setting the CHAP Authentication  
For VMware ESX connections, the ETERNUS DX Disk storage system uses a one-way CHAP  
authentication to authenticate the Software Initiator access.  
Perform the following settings to enable CHAP authentication.  
Procedure  
1
Select the [Configuration] tab on VMware Infrastructure Client and select  
[Storage Adapters].  
2
3
4
5
Select the target iSCSI Software Adapter and click [Properties].  
Select the [CHAP Authentication] tab and click [Configure].  
Select the [Use the following CHAP credentials] radio button.  
Set arbitrary user name during the CHAP authentication.  
35  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
   
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
6
To set the VMware ESX iSCSI name for the user name, select the [Use initiator  
name] checkbox.  
7
Enter the password in [CHAP Secret] and click the [OK] button.  
The CHAP Name is displayed as follows.  
End of procedure  
6.2.6  
Checking the LUNs  
The VMware Infrastructure Client is used to login to VMware ESX and check the LUNs.  
Procedure  
1
Select the [Configuration] tab on VMware Infrastructure Client, and select the  
target iSCSI Adapter.  
36  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 6 Setting Up the VMware ESX Server  
6.2 Setting Up for VMware Infrastructure 3  
2
Click [Rescan...].  
The LUNs are displayed under the SCSI Target 0 in the "Details" field. In this example,  
[Path:vmhba32:0:0] is displayed as follows:  
If VMware ESX is running (requiring dynamic LUN recognition),  
added LUNs should be recognized by performing the VMware  
Infrastructure Client "Rescan" operation after the LUN has been  
added to an Affinity Group.  
3
Check the multipath. Right-click the [Path:vmhba32:0:0] described in Step 2, and  
select "Manage Paths".  
When paths are configured with multipath, multiple paths are displayed in the "Path" field.  
End of procedure  
37  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
 
Chapter 7 Virtual Machine  
This chapter describes the notes on the Virtual Machine and its settings.  
To configure the Virtual Machine, access the following web-site and check the "Guest Operation  
System Installation Guide".  
Guest Operation System Installation Guide  
7.1  
For Windows®  
7.1.1  
Setting the Disk TimeOutValue  
Check the value of the "TimeOutValue" registry key. If the "TimeOutValue" registry key does not  
exist, it should be created.  
If the "TimeOutValue" key does not exist, add a registry key with the  
following values:  
Name  
Type  
Radix  
Data  
TimeOutValue  
REG_DWORD  
Hexadecimal  
3C  
Be sure to backup the registry before editing it.  
Procedure  
1
2
Start the registry editor (regedit.exe).  
Follow the path described below:  
\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk  
3
Check the value of the "TimeOutValue" registry key.  
Check that the value of the "TimeOutValue" registry key is "0x3C". If set to a different  
value, change it to "0x3C".  
38  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
       
Chapter 7 Virtual Machine  
7.2 For Linux  
4
If the contents were modified, reboot the OS.  
End of procedure  
7.1.2  
Applying Required Patches  
Patches provided by Microsoft® are necessary for Windows Server® 2003 with Service  
Pack 1 applied and Windows Server® 2003 R2.  
Refer to VMware KB 2267 on the VMware web-site or check with Microsoft® for patch  
details.  
7.2  
For Linux  
7.2.1  
Applying Required Patches  
For the following OSes, the Virtual Machine file system may be restricted to being read-only.  
If this is a problem, refer to "VMware KB Article 51306" on the VMware web-site for details, and  
patch the Virtual Machine as required.  
Red Hat Enterprise Linux 5  
Red Hat Enterprise Linux AS v.4 Update 4  
Red Hat Enterprise Linux AS v.4 Update 3  
SUSE Linux Enterprise Server 10  
SUSE Linux Enterprise Server 9 Service Pack 3  
39  
ETERNUS DX Disk storage systems User’s Guide -Server Connection- (iSCSI) for VMware® ESX  
Copyright 2012 FUJITSU LIMITED  
Download from Www.Somanuals.com. All Manuals Search And Download.  
P3AM-3412-11ENZ0  
     
ETERNUS DX Disk storage systems User's Guide  
-Server Connection- (iSCSI) for VMware® ESX  
P3AM-3412-11ENZ0  
Date of issuance: September 2012  
Issuance responsibility: FUJITSU LIMITED  
The content of this manual is subject to change without notice.  
This manual was prepared with the utmost attention to detail.  
However, Fujitsu shall assume no responsibility for any operational problems as the  
result of errors, omissions, or the use of information in this manual.  
Fujitsu assumes no liability for damages to third party copyrights or other rights arising  
from the use of any information in this manual.  
The content of this manual may not be reproduced or distributed in part or in its entirety  
without prior permission from Fujitsu.  
Download from Www.Somanuals.com. All Manuals Search And Download.  
Download from Www.Somanuals.com. All Manuals Search And Download.  

Edelbrock Automobile Parts 2 4 W D User Manual
Estate Dishwasher TUD4700SQ User Manual
Extron electronic Stereo Receiver WPD 110 A User Manual
Fellowes Musical Toy Instrument 8036001 User Manual
Fisher Price Motorized Toy Car 74547 User Manual
Genz Benz Musical Instrument Amplifier 122 User Manual
GE Range 49 80535 1 User Manual
GE Refrigerator 197D7799P001 User Manual
GE Washer 175D1807P396 User Manual
Gitzo Camcorder Accessories GT0531 User Manual