Best Practices Guide_for XIOtech ISE Storage

X-IO Technologies 9950 Federal Drive, Suite 100 Colorado Springs, CO 80921-3686 www.x-io.com Main: 719.388.5500 Fax: 719.388.5300 Customer Support: 1.800.734.4716 IMPORTANT NOTICE This manual is provided to you by Xiotech Corporation (“Xiotech”), and your receipt and use of this manual is subject to your agreement to the following conditions: • This manual and the information it contains are the property of Xiotech and are confidential. You agree to keep this manual in a secure location and not disclose or make available its contents to any third party without Xiotech’s written permission. • Reproduction or distribution of this document, in whole or in part, may only be made with Xiotech’s written permission. • This manual is not an endorsement or recommendation of any of the products or services provided by third parties and referred to herein. • Use of this manual does not assure legal or regulatory compliance. • This document is not a warranty of any kind. The only warranties given in connection with this product are those contained in Xiotech’s standard terms and conditions. • Xiotech, Magnitude, Magnitude 3D, and TimeScale are registered trademarks of Xiotech Corporation. Emprise, ISE, Dimensional Storage Cluster, DataScale, and GeoRAID are trademarks of Xiotech Corporation. Other trademarks or service marks contained herein are the property of their respective owners. Trademarks and service marks belonging to third parties appear in this document for the purposes of identification of the goods or services of that third party only. No reproduction of any trademark or service mark is authorized by this document. No right or title or other proprietary interest in any mark is transferred because of this document. © 2012 Xiotech Corporation. All rights reserved. Publication Number: 160347-000 Rev C Janaury 2013 Table of Contents Best Practices Guide Table of Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 ISE-2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 MRC—Managed Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 DataPacs—Types and Capacities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 End-to-End Solution Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Sample Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 ISE Physical Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Fibre Channel Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Direct Attached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 ISE Host Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 FC Switched Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Switch Configuration Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Brocade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 ISE Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Open Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Host-Based Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Single Initiator Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 One-To-One Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 HBA Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Connection Options (QLogic specific) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Adapter BIOS/Boot from SAN Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 LUNs per Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Execution Throttle / Queue-Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Vendor-Specific Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 ISE Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Thrashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Minimize Thrashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Operating System Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 ISE-2 Upgrade Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Online Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Offline Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows 2003 R2 SP2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multi-pathing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disabling FUA for ISE Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Required Hot Fixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISE Firmware Upgrade Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 21 21 21 21 21 23 23 Xiotech - Proprietary and Confidential 160347-000 Rev C, 11 January, 2013 Page i Best Practices Guide Table of Contents Windows 2008 SP2 / Windows 2008 R2 SP1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multi-pathing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Native MPIO Registry Key Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disabling FUA for ISE Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Required Hot Fixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISE Firmware Upgrade Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows 2012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multi-pathing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) . . . . . . . . . . . Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VMware ESXi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Physical Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SVC Logical Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Single Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Redundant Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zoning Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISE Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Volume Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Volume Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Host Queue Depth Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Storage and Networking Industry Association (SNIA) Dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Fiber Cable Ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VMware Queue Depth Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VMware Maximum Disk Requests per VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Red Hat Enterprise Linux (RHEL) MPIO Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SUSE Linux Enterprise Server (SLES) MPIO Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . QLogic Red Hat Enterprise Linux 6.x Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 23 24 24 26 27 32 32 32 32 36 36 36 36 41 42 42 42 43 43 43 43 43 43 45 45 45 45 45 45 45 Using IBM SVC with ISE Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 External References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 Page ii 160347-000 Rev C, 11 January, 2013 Xiotech - Proprietary and Confidential ISE Best Practices Introduction Storage solutions such as the ISE-2 can be configured in many different ways, each with their own set of risks and rewards. This document presents the optimal configurations for various purposes and demonstrates options at various points from ISE-2 to the HBA and Operating Systems. Intended Audience This document is intended for IT Administrators, Desktop Specialists, Partners, and Field Services. Xiotech - Proprietary and Confidential 160347-000 Rev C, 11 January, 2013 Page 1 Best Practices ISE Page 2 160347-000 Rev C.Proprietary and Confidential . 2013 Xiotech . 11 January. ISE Best Practices Hardware Configuration ISE-2 Components Refer to the Introduction of the ISE User Guide for a more complete description of each component. 2013 Page 3 . DataPac(s) 2. Figure 1. ISE-2 Major Components 1. Power Supplies Xiotech . 11 January. Managed Reliability Controllers (MRCs) 4. This is a quick reference only. System Chassis 6.Proprietary and Confidential 160347-000 Rev C. System Status Module 3. Supercapacitors (Supercaps) 5. This is disabled by default and should be enabled on every ISE upon installation unless the customer’s business does not allow it. Managed Reliability Controllers (MRCs) The X-IO Managed Reliability Controller (two per ISE) uses built-in diagnostics and recovery tools to analyze. and the details of each are summarized in the table below. These are the same diagnostics and tools that are used by the drive manufacturer to repair and refurbish a drive that has been sent back for repairs.4TB Drive Type SAS HDD SAS HDD and SSD Page 4 160347-000 Rev C.8TB 7. This ability to take a drive offline and repair it is what X-IO calls “managed reliability. identify.” and it helps drive the 5year warranty on DataPacs. Each ISE also has SNMP capability built in. 2013 Xiotech . It is not necessary to have an ISE Analyzer at the customer site if the customer is willing to monitor their ISE products through other means because the telemetry can be sent to X-IO headquarters with or without an ISE Analyzer. DataPacs—Types and Capacities The ISE-2 has several different models of DataPacs that can be installed. which allows external monitoring tools to be used to monitor and alert on changes in state of the ISE.Best Practices ISE MRC—Managed Reliability Figure 2. X-IO also makes an ISE Analyzer product that can be installed at the customer site to capture this telemetry data from multiple ISEs and display it to the customer. 11 January.4H ISE Max Capacity 9. and recover from a large number of common drive problems.2TB ISE Model ISE-2 9.6 ISE-2 14. DataPac Capacity 4. Each MRC has the ability to send information back to X-IO headquarters and notify the services organization that something has gone wrong.6TB 14.Proprietary and Confidential . Fibre Channel Switches—Optional if doing a direct-attached storage solution 4. 20% of the drive capacity is reserved for data recovery operations (sparing). and five different applications. five dual-port HBAs (one per server).4TB Table 1: ISE Model ISE-2 19. Xiotech . five physical servers. and the lines representing Fibre Channel cables are color-coded to represent which fabric they are on. Server Applications Sample Environments A typical environment is more than just a single server and more than just one application.8 ISE Max Capacity 19. A typical list of components. 11 January.ISE Best Practices DataPac Capacity 9. from the storage solution to the applications. In the environment below there are three ISE-2s.2 ISE-2 28.8TB Drive Type SAS HDD SAS HDD DataPac Types and Capacities Note that the ISE-2 14. 2013 Page 5 .2TB 28.Proprietary and Confidential 160347-000 Rev C. Server Operating System 7. These drives are grouped together into one or two pools. Server HBA 5.6TB 14. Server Hardware 6. When the ISE is initialized. Each DataPac contains either 10 or 20 industry-standard hard drives or a mix of 10 hard drives and 10 solid state drives in the case of the Hyper DataPacs.4H also goes by the name Hyper-ISE (the “H” stands for “Hyper”) because it uses solid state disks (SSD) to provide faster access to the most frequently accessed data via X-IO’s proprietary CADP algorithm. Each switch represents a separate fabric. End-to-End Solution Diagrams Any storage solution is made of multiple components that must function together to serve the needs for which it was designed. Fiber Channel cables 3. Storage device (ISE-2 or similar) 2. is listed below in the order that they might appear: 1. This spare capacity is equivalent to hot-spare drives in a more traditional storage solution. two Fibre Channel switches for redundant fabrics. it is possible for a LUN to receive I/O requests from either MRC. there is no hard requirement to use these specific ports. Additional SFPs can be purchased through X-IO to increase available ports for increased throughput or redundancy. 11 January. X-IO provides the ISE Multipath Suite (ISE-MP).5 and later includes performance improvements that detect which MRC is preferred for each LUN and send I/O to those paths connected to the preferred MRC. an OM2 cable works for all speeds from 1 Gb/sec to 16 Gb/sec. When a LUN receives an I/O request from the non-preferred MRC.6. Page 6 160347-000 Rev C. and the diagrams and charts show just one for simplicity. For Windows servers. The non-preferred MRC receives I/O only if paths to the preferred MRC become unavailable. 2013 Xiotech . Version 6. Sample End-to-End Environment ISE Physical Connectivity Note that although the diagrams and charts below call out specific ISE MRC ports. Each LUN on an ISE has a preferred MRC. Since the MRCs are active-active (any MRC can talk to any LUN). See the IBM FC cable details link in “External References” on page 45 for exact bitrate to distance details.Best Practices ISE Figure 3. a small performance hit occurs per I/O since the non-preferred MRC tries to forward the data to the preferred MRC and then sends the appropriate response back to the server. Fibre Channel Cables For cable lengths up to 35 meters (114 feet). A lower bit-rate supports longer distances for the same cable.0.Proprietary and Confidential . Each ISE-2 MRC has four fiber-channel (FC) ports available and ships with one SFP per MRC. which is assigned at the time of LUN creation and alternates between the two MRCs. Any ISE MRC port is equivalent to any other port. with each port connected to a different MRC. X-IO strongly recommends setting the HBA ports and the ISE ports to the same data rate for direct attached configurations.Proprietary and Confidential 160347-000 Rev C. In a direct attach scenario. and the other MRC connects to the other fabric. X-IO does not recommend the use of a single switch as this type of configuration implies a single point of failure. 2013 Page 7 . Single Switch—Redundant Fabrics The default configuration for an ISE-2 is to connect one MRC to one fabric. then the number of host port WWNs used is 250 (4 VMs * 2 ports * 25 hosts + 2 ports per server * 25 servers).ISE Best Practices Direct Attached When using one of the supported Operating System (OS) versions and supported HBAs (see the lists on X-IO’s Support Matrix). it is required to have at least two ports per server. Auto-speed negotiation does not always work reliably during failover/failback scenarios or certain failure conditions. Direct Attached Cabling ISE Host Limits Each ISE-2 is limited to 256 host port WWNs logged into the ISE at any one time. Xiotech . 11 January. If a customer environment has 25 physical servers and none are running virtual machines. a host can connect directly to the MRC ports with a Fibre Channel cable. Figure 4. then the total number of host port WWNs used is 50. FC Switched Fabric Single Switch—Non-Redundant It is possible to connect both MRCs and all hosts to a single fiber switch. each with two HBAs. This ensures that the host stays online in the event of an MRC failure or a normal failover as part of an ISE-2 firmware upgrade process. If these servers each run four virtual machines with two NPIV ports presented to the ISE-2. it is preferable to use them.Proprietary and Confidential . Figure 6.Best Practices ISE Figure 5. Page 8 160347-000 Rev C. Traffic between devices on the same fabric but on different switches must share the interswitch link (ISL). 11 January. This allows easier troubleshooting when viewing event logs and vendor support tools. X-IO recommends using unique domain IDs for each switch in a solution. If the switch supports high-speed ISLs. cable one port per MRC to each switch and distribute the host ports across both switches in each fabric equally. This provides at least one path to the preferred MRC for any given LUN on each fabric. regardless of whether the switches are on the same fabric. Single MRC per Fabric (Default Configuration) The best practice for an ISE-2 is to connect at least one port from each MRC to each fabric. X-IO recommends having a minimum of two ISLs between each switch. 2013 Xiotech . Both MRCs on Each Fabric–Best Practice for Performance and Redundancy Multi-switch Fabrics If there are two switches in a given fabric. 2013 Page 9 . however. This reduces the impact of a blade failure and minimizes the number of hops data has to travel through a shared ISL. this is generally considered cost-prohibitive at any scale beyond three or four switches.Proprietary and Confidential 160347-000 Rev C. Dual Fabric Core/Edge Model Xiotech . which has a director class switch in the center of each fabric. Figure 8. In this large fabric. X-IO recommends connecting each edge switch to at least two different blades on the director and connecting two ports on each ISE-2 MRC to different blades on the director as well. X-IO recommends connecting the ISE MRCs directly to the director class switches. use a mesh topology.ISE Best Practices Figure 7. 11 January. This minimizes the number of ISL links data has to cross to reach a given host. a director class switch connects to multiple edge switches. X-IO recommends using a Core / Edge model. In the model below. which then connect to hosts. with top-of-rack edge switches forming the edge. To alleviate this condition. Dual Fabrics with Two Switches and Dual ISLs Very Large Fabrics Daisy-chaining multiple switches together in large fabrics is not advised. as the traffic between devices on different switches must share the link and daisy-chaining reduces throughput. This speeds up the switch login time for an ISE. regardless of whether the switches are on the same fabric. This allows easier troubleshooting when viewing event logs and vendor support tools. Switch Configuration Settings X-IO recommends using unique domain IDs for each switch in a solution. Brocade Port Administration Page 10 160347-000 Rev C. 2013 Xiotech . X-IO recommends disabling the Loop option to prevent attempts to use FL-Port as an option.Best Practices ISE If the servers that communicate with a given ISE are in the same rack. uncheck L Port. it is advised to connect the ISE MRC ports and the server ports to the same top-of-rack switches. From CLI. Brocade Port Settings Brocade switches allow ports to be set to different types.Proprietary and Confiden- . 11 January. This avoids any extra traffic on the core. type switch> portcfggport <port#> 1 From GUI. Figure 9. ISE Best Practices Figure 10. This happens because the ISE uses the port WWN of the first port on MRC A as the Node WWN for all ports. in which there are two ports from a server as well as the first port on each MRC connected to a Brocade switch. 11 January. Xiotech . the real number of paths found for each LUN is three. See the table below as an example: Node WWN Port WWN MRC and Port Number 2000001F93103688 2000001F93103689 2000001F9310368A 2000001F93103688 2000001F9310368B 2000001F9310368C 2000001F9310368D 2000001F9310368E 2000001F9310368F Table 2: Node WWN. which can cause some unintended paths to storage LUNs to appear on certain ISE MRC ports. This should result in a total of two paths to any LUN presented from the ISE to the host. The zoning is configured to allow one port on the server to see MRC A and the other port to see MRC B.Proprietary and Confidential 160347-000 Rev C. MRC and Port Number MRC A Port 1 MRC A Port 2 MRC A Port 3 MRC A Port 4 MRC B Port 5 MRC B Port 6 MRC B Port 7 MRC B Port 8 Consider the solution below. When Brocade’s Node WWN checking feature is enabled. Brocade Port Configuration Wizard Brocade Node WWN Checking Configuration Brocade switches have a feature known as Node WWN checking. Port WWN. 2013 Page 11 . Disable the switch by typing the following line at the switch command prompt: Page 12 160347-000 Rev C. Sample Brocade Fabric—Two Paths Expected Assuming that the host WWNs are 21000024ff26415e and 21000024ff26415f. The ISE WWN assignment mechanism uses the same value for both the node WWN and the WWN of the first port on the first MRC. which disables checking the node WWN. the zones would look similar to the following. 2000001F9310368C Zone Name and Member Port WWNs Brocade switches can build zones based on either Port WWNs or Node WWNs. MRC B Port 5 has the same node WWN as MRC A Port 1’s port WWN and so is included in the zone. therefore. When a zone is created that contains a member with identical Node and Port WWNs.nodeNameDisabled to 1.Best Practices ISE Figure 11. the zone is assumed to be using Node WWN zoning. the observed number of paths is three. One is assigned to the device and is the node WWN. all other ports with the same Node WWN are added to this zone. The Node WWN checking option impacts any zones containing the first port of the first MRC on any ISE. Use the following procedure to disable and reconfigure the switch parameter. 1. however. 2000001F93103688 21000024ff26415f. 11 January. Node WWN Checking Brocade Fibre Channel (FC) switches have a configuration option that is on by default and that can cause some unexpected zoning behaviors with an ISE (both ISE-1 and ISE-2). Zone Name Member Port WWNs MrcAPort1_HostPort1 MrcBPort5_HostPort2 Table 3: 21000024ff26415e. and the other is assigned to the FC port. A two-port FC HBA has a common node WWN for the HBA. WARNING: This procedure disrupts I/O through the switch until the configuration process is complete.check. and each port has a unique port WWN. The ISE best practice is to set zoning.Proprietary and Confiden- . Consider a Brocade fabric with one switch and Node WWN checking enabled (the default behavior) that has an ISE-2 connected and zoning set up per the following list: • HBA Port 1 to MRC A Port 1 (Port WWN) • HBA Port 2 to MRC B Port 5 (Port WWN) The expected number of paths to a LUN should be two. HBA Port 1 sees both MRC A Port 1 and MRC B Port 5 because the zoning is built by node WWN first and then by port WWNs. Each FC device has a minimum of two worldwide names (WWNs). This can be avoided by either using a different port on the first MRC of an ISE-2 or by disabling the Node WWN checking on each Brocade switch in a given fabric. 2013 Xiotech . log back in to the fabric. 2013 Page 13 . If your previous configuration utilized zones. a registered state change notification (RSCN) event occurs. then the devices do not communicate unless the switch is configured for open zones. Verify that the switch displays zoning. Figure 12. Note. For example. Configure the setting using the following steps. single-initiator. One of the largest impacts the zoning model has is reducing the number of fabric-wide events that happen when a change is made.nodeNameDisabled:1. 5. which is sent to every device already connected to the fabric. and then connect to storage devices and begin I/O. There are several Fiber Channel zoning models. At the switch command prompt. which selectively presents storage to individual hosts. Re-enable the switch by typing the following line at the switch command prompt: Switchenable 7. If there are older storage devices that do not allow selective presentation.Proprietary and Confidential 160347-000 Rev C. 6. each of which has its own set of challenges and benefits. 3. Change the Disable NodeName Zone Checking to one. 11 January. If a zone does not connect two devices explicitly. type: Configure 4.check. rebooting a single host will cause each FC port to log out of the fabric. Every time a port logs into or out of a fabric. The most common zoning models are open-zoning. Select Zoning Operation Parameters from the menu. Open zoning allows all devices to connect and communicate with all other devices since the storage products (targets) are required to implement a LUN masking model. server performance and access to storage can be negatively impacted. and one-to-one. ISE Zoning Think of Fiber Channel zones as virtual SCSI cables with two or more connectors (up to the maximum number of zone members supported by the switch).ISE Best Practices switchdisable 2. review the configuration to ensure the change correctly represents your zoning preferences. Check the setting by typing the following line at the switch command prompt: configshow | grep zoning 8. single-host. If the zoning does not handle this to prevent the broadcasts. then zoning is required to provide the isolation necessary to make things work as expected. Sample Fabric Xiotech . this model requires very little documentation to manage. the host port WWNs are 21000024ff26415e and 21000024ff26415f. which can increase the documentation and configuration back-up requirements. 2013 Xiotech . This model has more zones to manage the host-based zoning. This model reduces the RSCN broadcasts to only the zones containing the device that logged into or out of the fabric. Open Zoning Open zoning is the simplest and easiest Fibre Channel fabric model to use. 2000001F93103688. In situations where one host port has a problem and sends out a lot of packets incorrectly. there is one zone for each initiator port that contains all targets with which the initiator port should be able to communicate. 2000001F9310368C Figure 13. especially when using boot from SAN. The single initiator-based zone membership for the fabric in Figure 12 should be similar to that listed in the following table: Zone Name Members HostAPort1_ISE HostAPort2_ISE 21000024ff26415e. Host-based Zone Membership Single Initiator Zoning Single initiator zoning would be identical to host-based zoning if the host has only one FC port. 2000001F93103688. back up. In cases where the host has two or more ports. because under this model all nodes connected to the fabric can automatically see each other. 2000001F93103688. It is assumed that under each zoning model described. it is possible that the healthy host port will also be impacted because the same zone includes both host ports.Best Practices ISE In this sample fabric. each host port will see both MRC ports for redundancy. This model is most appropriate for fabrics consisting of a single FC switch due to the risk of RSCN broadcasts taking hosts offline. 21000024ff26415f. and the ISE port WWNs are 2000001F93103688 and 2000001F9310368C. This can increase the host boot times. and zone configurations do not need to be backed up. This model has the fewest possible number of zones to manage. Host-Based Zoning Host-based zoning is the next simplest to implement. Open zoning increases the number of devices that a host port must scan during boot or bus rescan operations. In contrast. 2000001F9310368C 21000024ff26415f. all target ports that a given host needs to access are included in the same zone with all initiators from the same host. This limits the impact that any host port can have on any other host port and also limits the RSCN broadcasts to the zones containing the port that is logging into or out of the fabric. The host-based zone membership for the fabric in Figure 12 would look like the following table: Zone Name Members HostA_ISE 21000024ff26415e. In this model. Single initiator-based Zone Membership Page 14 160347-000 Rev C.Proprietary and Confiden- . 11 January. and document. This model is most appropriate for smaller fabrics of just a few switches. 2000001F9310368C Figure 14. These types of zones may need to have multiple initiators in the same zone. 8. but RSCN broadcasts are kept to an extremely narrow set of devices. The One-To-One based zone membership for the fabric in Figure 1 should be similar to that listed in the following table. 2000001F9310368C 21000024ff26415f. or four ports each and can communicate at 2. 11 January. 2000001F9310368C Figure 15. 2013 Page 15 . Otherwise Point to Point. Note that this zoning covers ISE-2 to servers and does not apply to zoning requirements for clusters. or ISE Mirroring. two. 2000001F93103688 21000024ff26415e.” The ISE bypasses Loop and log in with Point to Point. This may apply to storage from different vendors or to storage products from the same vendor with different requirements. Different HBA vendors may have different terms for these parameters. Zone Name Members HostAPort1_MrcAPort1 HostAPort1_MrcBPort5 HostAPort2_MrcAPort1 HostAPort2_MrcAPort5 21000024ff26415e. Note.” and “Loop Preferred. 2000001F93103688 21000024ff26415f. Under this model. 4. HBA Settings A host bus adapter (HBA) is used to connect a server to a storage solution. or mirroring tool for the specific zoning configurations needed. One-to-One-based Zone Membership Recommendations X-IO recommends the use of Single Initiator or One-To-One zoning for most applications to prevent RSCN broadcasts from negatively impacting an environment and to reduce the risk of accidentally presenting storage to the wrong server(s). X-IO recommends that a QLogic HBA be set to “Loop Preferred. QLogic defines the connection options as follows: Xiotech .” Refer to the SNIA dictionary (http://www.” “Point to Point Only.snia. such as an ISE. please refer to vendor documentation for exact details on how to set these values. If a host has two initiator ports per fabric and there are three storage devices with two ports on each fabric.Proprietary and Confidential 160347-000 Rev C.ISE Best Practices One-To-One Zoning One-to-One zoning is the most controlled model. Refer to the appropriate user guide for the cluster. then there are 12 zones per fabric per server. QLogic offers “Loop Only. or 16 Gb per second. An HBA can have one.org/education/ dictionary) for more information on arbitrated loop and point-to-point connections. The individual parameters that are commonly adjusted for an ISE-2 are described below. each zone represents a single virtual SCSI cable with exactly two connectors. this can get to be a very large number of zones very quickly. When storage products have conflicting HBA settings we recommend that different HBAs be used. storage virtualizers such as Datacore. storage virtualizer. so the impact of a failing FC port is very small.” This option is required for direct attach solutions due to a known compatibility issue when using “Point to Point Only. Obviously. Otherwise Point to Point. An HBA has several configurable parameters that control how it behaves with storage solutions. IBM’s SVC. Connection Options (QLogic specific) An HBA can be connected to multiple types of ports. org/education/dictionary) for more on these terms. QLogic’s HBA parameters allow this to be set as high as 65. 11 January. Refer to the HBA vendor’s support documentation for instructions on doing this.048 I/O requests to be queued per MRC. An ISE-2 allows a maximum of 2. An ISE is capable of handling up to 240 LUNs and up to a maximum of 240 host port WWNs for ISE-1 and 250 host port WWNs for ISE-2. ISE-1 requires the MRC. assuming the servers and applications can generate sufficient I/O. identify which servers are generating heavy workload (databases. Otherwise Point to Point (default) 3—Point to Point. large file shares. the adapter BIOS must be enabled and configured to boot from a specific storage device. switch. The LUNs per target parameter should be raised to at least 240 so that any HBA in a server can see all possible LUNs. and HBA ports to be set to a fixed data rate. In order to boot from a SAN device. See the SNIA dictionary (http://www. The Emulex HBA tools allow a queue depth of 254 for Windows hosts and 128 for Linux. however. LUNs per Target An HBA in a server is known as an initiator since it can initiate a SCSI read or write command. X-IO recommends setting the QLogic queue depth to 256 per LUN for Linux hosts.535.096 queued requests. X-IO recommends that the MRC and HBA ports be set to a fixed data rate when directly connected together (no switch between). for a total of 4. As a best practice. also known as “Queue Depth.) and set those queue depths to a higher value. Execution Throttle / Queue-Depth The Execution Throttle value.Proprietary and Confiden- .Best Practices ISE 0—Loop Only 1—Point to Point Only 2—Loop Preferred. which exceeds the ISE-2 queue depth.snia. Most storage solutions (including the ISE-2) are targets that can respond only to requests started by an initiator. Otherwise Loop Data Rate An HBA port is normally set to auto-detect the maximum speed possible when connected to a switch or storage port. the storage returns a busy response. Adapter BIOS/Boot from SAN Settings The Adapter BIOS is disabled by default. All storage solutions have a queue depth value. An ISE-2 can use auto-speed negotiation. Page 16 160347-000 Rev C. 2013 Xiotech . etc.” controls the number of I/O requests an HBA can make per volume or target before it must wait for a previous request to complete. digital imaging. and if the combined I/O of all connected hosts exceeds that value. driver. Emulex Emulex HBA Settings Parameter Name Link Timeout Node Timeout Queue Depth Queue Target / LUN Table 5: Default Value 30 30 32 0 Fabric Value 30 30 254 0 Direct Attach Value ISE Not Supported ISE Not Supported ISE Not Supported ISE Not Supported Emulex HBA Settings Cisco UCS To date there has been no need to alter the default HBA parameters for a Cisco UCS to work with an ISE-2. The adapter defaults for the HBA are shown in the figure below: Xiotech . and software being used. 11 January.Proprietary and Confidential 160347-000 Rev C. 2013 Page 17 .ISE Best Practices Vendor-Specific Parameters QLogic QLogic HBA Settings Parameter Name Connection Option Data Rate HBA BIOS Execution Throttle LUNs per Target Table 4: Default Value 2 (Loop pref) Auto Disabled 16 or 32* 8 Fabric Value 2 2/4/8 Gbps Enabled (if Boot from SAN) 256 256 Direct Attach Value 2 (Required) 2/4/8 Gbps Enabled (if Boot from SAN) 256 256 QLogic HBA Settings *Depends on the HBA model. 0. All other values are identical between all profiles as of ISE-2 firmware 2.Proprietary and Confiden- . with the exception of VMware. 11 January. which changes the highlighted Port Down Timeout (ms) from 30000 (30 seconds) to 10000 (10 seconds).0. Page 18 160347-000 Rev C.Best Practices ISE Figure 16. UCS M81KR HBA Parameter Defaults The defaults apply to all HBA profiles. 2013 Xiotech . ISE Best Practices ISE Performance Thrashing Any array of hard disks lowers its performance characteristics as the disks are sliced up and populated with data since the heads require additional time to seek back and forth to the next requested block of data. Consider RAID-10 for read-intensive applications since the data can be accessed from two different sources at once. Xiotech . Put frequently accessed data on separate DataPacs if at all possible. All X-IO DataPacs ship with either 10 or 20 spinning drives. The data access pattern is typically the single largest contributor to thrashing. a. The common root causes of thrashing are: • Multiple applications accessing the same spinning media • Frequently accessed data in LUNs at opposite ends of the storage pool • Files at opposite ends of a file system being accessed (same LUN) • The data access pattern of each application (sequential read / sequential write / random read / random write) Note. Use separate LUNs for frequently accessed data and create these LUNs one after the other so that they are close to each other on the disks to minimize seek time. 11 January. Ideally this tool will put the most frequently accessed files together at the front of the file system to minimize seek time. 1. 4. 2. Run a file system defragmentation tool of some sort. 5. Minimize Thrashing Follow the best practices below to minimize thrashing. The ISE data placement algorithm distributes the data across these drives to maintain the performance of reading and writing to all drives in a DataPac at any point in time while maintaining the proper RAID level for each ISE volume (LUN). 3. This is commonly known as thrashing.Proprietary and Confidential 160347-000 Rev C. Avoid deleting LUNs and creating new ones or resizing LUNs as this may “fragment” the LUN across the disks and impact performance. files should be re-arranged so that all blocks of a file are next to the other blocks of the file. At a minimum. Separate the high traffic applications onto separate ISEs or DataPacs. 2013 Page 19 . 6. b. and the Hyper DataPac includes 10 solid state drives in addition to the 10 spinning drives. Best Practices ISE Page 20 160347-000 Rev C. 2013 Xiotech . 11 January.Proprietary and Confiden- . Offline Upgrade An offline upgrade is when an ISE-2 firmware upgrade is performed while the connected servers are shut down or the applications that use the storage are stopped.Proprietary and Confidential 160347-000 Rev C.aspx. 1. When an HBA sees a path go away. Disabling FUA for ISE Volumes Windows Server operating systems have a mechanism that writes directly to the disk and instructs the disk to not use any write caching mechanisms it may have. file system.” or FUA.85).ISE Best Practices Operating System Considerations Performance and upgrade options for the ISE-2 vary depending on the operating system and applications in use. Windows 2003 R2 SP2 Multi-pathing ISE Multi-pathing is required for running Windows Server 2003. it waits for some period of time (typically 30 seconds) to see if the path is restored and then moves I/O to an alternate path. This is done to protect against data loss on storage devices (USB hard drives. see http://msdn.com/enus/library/windows/desktop/ dd979523(v=vs.) that do not have battery-backed caching capability. an offline upgrade is required. etc. If the operating system. ISE-2 Upgrade Types Online Upgrade An online upgrade is when the ISE-2 firmware is upgraded while the servers connected to the ISE-2 are under load. This uses part of the SCSI block commands known as “Force Unit Access. For details on this. Xiotech . 11 January. 2013 Page 21 . and ISE-2 firmware upgrades may require downtime under certain conditions. 2. or application cannot survive without I/O for this period of time.microsoft. This timeout period and the associated lack of I/O is HBA specific and not directly related to the ISE-2. Open Disk Manager. This requires that the server and the applications be able to handle paths to a disk going away and coming back while the load transfers from one MRC to the other. so performance can be increased by disabling this mechanism. Performance can be impacted by caching mechanisms in each OS. The ISE-1 and ISE-2 products support write caching and have a battery backup system to preserve that cache. Right-click the ISE LUN (ISE-1 or ISE-2). On the Disk Device Properties dialog. 11 January. select the Policies tab.Best Practices ISE 3. Select Properties. Selecting Disk Properties in Disk Manager 4.Proprietary and Confiden- . Figure 17. 2013 Xiotech . Page 22 160347-000 Rev C. 7. Multi-Path Disk Device Properties 5. X-IO does not currently support native multipath I/O for the ISE-2.1. • ISE Multi-Path Suite (ISE MP) 6. Required Hot Fixes • Microsoft Knowledge Base Article ID 950903 ISE Firmware Upgrade Options When an ISE-2 is connected to one or more Windows Server 2003 R2 SP2 servers (the only supported versions of Windows Server 2003).0. Ensure that the Enable write caching on the disk and Enable advanced performance check boxes are selected. 11 January. a firmware upgrade is an offline process. Click OK. • All required hot-fixes should be installed. which improves path stability during path failures or interruptions between the servers and the ISE. 2013 Page 23 . Windows 2008 SP2 / Windows 2008 R2 SP1 Multi-pathing With the exception of Cisco UCS systems.ISE Best Practices Figure 18.10 or later. 6.6.0 can be upgraded. • Any ISE-2 running firmware greater than 2.Proprietary and Confidential 160347-000 Rev C. Refer to the Release Notes of the new firmware version for any additional pre-requisites. Refer to the X-IO support matrix for specific operating system versions supported. Xiotech . Repeat for each ISE volume. so performance can be increased by disabling this mechanism.” or FUA. The ISE-1 and ISE-2 products support write caching and have a battery backup systems to preserve that cache. etc. X-IO ISE MP 6.0.Proprietary and Confiden- . see http://msdn.aspx. This is done to protect against data loss on storage devices (USB hard drives. 11 January. Select Properties. Page 24 160347-000 Rev C. 3. For details on this. 2013 Xiotech . X-IO recommends the use of Native MPIO on Windows Server 2008 and newer operating systems. Parameter Name Default Value X-IO Value 0 30 50 10 1 1 25 PathVerifyEnabled PathVerificationPeriod PDORemovePeriod RetryCount RetryInterval UeeCustomPathRecoveryInterval PethRecoveryInterval Table 6: 0 30 20 3 1 0 40 HKLM /System/Current Control Set/Service/MPIO/Parameters Disabling FUA for ISE Volumes Windows Server operating systems have a mechanism that writes directly to the disk and instructs the disk to not use any write caching mechanisms it may have. 1. X-IO ISE MP is not required for Windows Server 2012.) that do not have a battery-backed caching capability. When using Active-Active Mirroring. Open Disk Manager.10 or higher is required for Windows Server 2008 SP2 and Windows Server 2008 R2 SP1.com/en-us/library/windows/desktop/ dd979523(v=vs. This uses part of the SCSI block commands known as “Force Unit Access. 2.6.Best Practices ISE Native MPIO Registry Key Changes For Cisco UCS systems. the following registry keys will need to be set as follows. Right-click the ISE LUN (ISE-1 or ISE-2). In order to enable the Windows servers to handle an MRC failover/failback scenario or a firmware upgrade.microsoft.85). select the Policies tab.ISE Best Practices Figure 19. 11 January. 2013 Page 25 . Selecting Disk Properties in Disk Manager 4. Xiotech .Proprietary and Confidential 160347-000 Rev C. On the Disk Device Properties dialog. 6. 11 January. There's no benefit to installing this on standalone Windows Servers.Best Practices ISE Figure 20. • • • Page 26 160347-000 Rev C. 2013 Xiotech . Repeat for each ISE volume.Proprietary and Confiden- . Microsoft Knowledge Base Article ID 2661794 Microsoft Knowledge Base Article ID 2522766 Microsoft Knowledge Base Article ID 2511962 Note. Required Hot Fixes Required Hot-Fixes for Windows 2008 R2 SP1 • Microsoft Knowledge Base 2718576 • X-IO recommends the following values for the registry keys specified in this hotfix: • DiskPathCheckEnabled = 1 • DiskPathCheckInterval = 25 Microsoft Knowledge Base 2718576 and the registry key bullets below it apply to clusters only. Disk Properties Dialog 5. Click OK. 7. Ensure that the Enable write caching on the disk and Enable advanced performance check boxes are selected. Check "Online Upgrade (clusters)" below to perform an online upgrade.0. if there are four ISEs that contribute storage to the cluster. the restart count would be set to 13 ((3 * 4) + 1). Online Upgrade (standalone servers) • • • ISE firmware 2. Until the cause is determined and a workaround or a fix is available.6.2.2. X-IO recommends performing offline upgrades with cluster nodes configured to boot from SAN. All hot fixes listed must be installed. Online Upgrade (clusters) • • • • ISE firmware 2. All hot fixes listed must be installed. exception for Cisco UCS. All hot fixes listed must be installed.Proprietary and Confidential 160347-000 Rev C.1 or older must be upgraded offline due to the amount of time required to perform certain parts of the upgrade. ISE Multi-Path Suite (ISE-MP) 6. 11 January.3 or later.0. exception for Cisco UCS.2. ISE Multi-Path Suite (ISE-MP) 6.6. 2013 Page 27 . exception for Cisco UCS.0.10 or newer (improves path stability).ISE Best Practices • • Microsoft Knowledge Base Article ID 2468345 Microsoft Knowledge Base Article ID 2460971 Required Hot-Fixes for Windows 2008 SP2 • • • • • • • • • • • • • • • • • Microsoft Knowledge Base Article ID 968675 Microsoft Knowledge Base Article ID 970525 Microsoft Knowledge Base Article ID 972797 Microsoft Knowledge Base Article ID 974878 Microsoft Knowledge Base Article ID 976674 Microsoft Knowledge Base Article ID 976748 Microsoft Knowledge Base Article ID 977001 Microsoft Knowledge Base Article ID 977153 Microsoft Knowledge Base Article ID 977287 Microsoft Knowledge Base Article ID 977675 Microsoft Knowledge Base Article ID 977890 Microsoft Knowledge Base Article ID 978157 Microsoft Knowledge Base Article ID 979458 Microsoft Knowledge Base Article ID 979743 Microsoft Knowledge Base Article ID 979764 Microsoft Knowledge Base Article ID 981357 Microsoft Knowledge Base Article ID 2406705 ISE Firmware Upgrade Options There are some differences between cluster nodes and standalone servers when it comes to upgrading the ISE-2. Offline Upgrade • • • ISE-2 Firmware 2.10 or newer (improves path stability). “Cluster Service Resource Maximum Restart” count set to be equal to three times the number of ISE2s + 1. As an example. • Xiotech . ISE Multi-Path Suite (ISE-MP) 6.10 or newer (improves path stability).3 or later.6. No boot from SAN volumes—Some issues have been reported that seem to occur only on cluster nodes booting from SAN. Proprietary and Confiden- . and then select Properties from the drop-down menu.Best Practices ISE • Required for the duration of the upgrade process—can be set to other values for normal operations. X-IO recommends using the cluster maintenance mode to perform upgrades. 3. Page 28 160347-000 Rev C. 2. Open the Failover Cluster Manager GUI. • Configuring Cluster Disk Resource Failover Policies Follow the steps below to configure the cluster resource failover policies. 2013 Xiotech . Select the Policies tab. Cluster Disk Properties 4. Select an ISE-2 clustered disk. 1. Select the Storage node in the left-hand navigation pane. 11 January. right-click. Figure 21. This should be equal to 1 + three times the number of ISE-2s in the cluster. Repeat for each ISE-2 disk in the cluster. Configuring Clustered Services and Applications 1. and then select Properties from the drop-down. Change the Maximum restarts in the specified period value. 2013 Page 29 . 2. right-click. 11 January. Cluster Disk Policies Tab 5. which is appropriate for a single ISE-2.ISE Best Practices Figure 22. Select the Failover tab.Proprietary and Confidential 160347-000 Rev C. 3. Select each service or application. 6. Expand the Services and Applications node in the left-hand navigation pane. Xiotech . In Figure 22 the value is set to 4. Change the Maximum failures in the specified period.Proprietary and Confiden- . In the main pane for each clustered application. 5. 2013 Xiotech . Cluster Application Failover 4. Page 30 160347-000 Rev C. set the failover properties for each resource. 11 January.Best Practices ISE Figure 23. This value should be equal to 1 + three times the number of ISE-2s in the cluster. turn off maintenance mode using the same process (select Turn Off Maintenance Mode for this disk). Once all ISE-2s are upgraded. Right-click each ISE-2 volume. Right-click each resource in the resource list (see Figure 24). select More Actions. 4. 3. Clustered Application Resource List 6. 7. 11 January.Proprietary and Confidential 160347-000 Rev C. select Properties.ISE Best Practices Figure 24. Figure 25. then select Turn On Maintenance Mode for this disk. Disk Maintenance Mode 5. Navigate to the Storage node in the left-hand navigation pane. 8. The GUI shows that the disk is in maintenance mode. Change the Maximum restarts in the specified period to the same value used for the cluster disk resource. Xiotech . Setting Maintenance Mode During an Upgrade X-IO strongly recommends placing clustered storage into maintenance mode. and then select the Policies tab. The following process demonstrates how to enable and then disable maintenance mode. Open Failover Cluster Manager. 2. 2013 Page 31 . 1. Repeat these steps for each resource in each clustered service or application. which prevents failover of cluster disks while running ISE firmware upgrades. the following MPIO DSM registry key changes are recommended: Parameter Name Default Value X-IO Value 0 30 50 10 1 1 25 PathVerifyEnabled PathVerificationPeriod PDORemovePeriod RetryCount RetryInterval UseCustomPathRecoveryInterval PathRecoveryInterval 0 30 20 3 1 0 40 Claiming MPIO disks will cause the host the reboot. To configure native MPIO to claim the ISE-1 and ISE-2 volumes. including firmware upgrade.8. For ISE-1: mpclaim -r -i -d "XIOTECH ISE1400" For ISE-2: mpclaim -r -i -d "XIOTECH ISE2400" Linux Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) The below are best practices recommendations for RHEL and SLES. and that the items in the quotes are case sensitive. Page 32 160347-000 Rev C. and the login timeout ql2xlogintimeout should be 5. Note that this assumes that the MPIO feature is already installed.0. 11 January.4. HBA Configuration For direct attached systems. follow the instructions for rebuilding the RAMDISK image in the QLogic’s Linux driver documentation from QLogic’s web site. ISE-2 will require 2.Best Practices ISE Windows 2012 ISE-1 will require 1.Proprietary and Confiden- . the queue depth parameter ql2xmaxqdepth should be 256. In order for this parameter to persist across reboots. In order to improve the availability of the storage during common path loss scenarios. Fabric connected systems should use defaults.0 firmware. Required Hot-Fixes for Windows 2012 Microsoft Knowledge Base Article 2779768 Multi-pathing X-IO recommends the use of Windows Native MPIO for ISE storage. See the “Deploying the Driver” section and find the related OS version to follow the instructions for “Automatically load the driver by rebuilding the RAM disk. use the command lines below. 2013 Xiotech .” This requires a reboot to take effect. For ISE-1.conf file.ISE Best Practices Multi-path Settings Refer to the RHEL or SLES documentation for the chosen multi-path product for additional information on these parameters. use product ID ISE1400. 11 January. Device-specific modifications must be made in the /etc/multipath. use product ID ISE2400 Xiotech .Proprietary and Confidential 160347-000 Rev C.conf file. 2013 Page 33 .conf file: Note. and for ISE-2. Verify that the polling interval is set to 10 in the defaults section of the /etc/multipath. as follows: Defaults { … } polling_interval 10 … } The following are examples of what the device entries would look like in the multipath. 11 January.1 defaults { udev_dir polling_interval selector path_grouping_policy getuid_callout prio path_checker rr_min_io max_fds rr_weight failback no_path_retry fast_io_fail_tmo dev_loss_tmo user_friendly_names { device /dev 10 “round-robin 0” multibus "/lib/udev/scsi_id --whitelisted --device=/dev/%n" alua tur 100 8192 priorities immediate fail 5 600 yes # # # # # # } devices { vendor product path_grouping_policy prio_callout path_checker rr_min_io rr_weight failback dev_loss_tmo no_path_retry user_friendly_names "XIOTECH" "ISE1400" multibus “none” tur 100 priorities immediate infinity 12 yes } } Page 34 160347-000 Rev C.x devices { device { vendor product path_grouping_policy getuid_callout path_checker prio_callout path_selector failback no_path_retry user_friendly_names “XIOTECH” “ISE1400” multibus "/sbin/scsi_id -g -u -s /block/%n" tur “none” “round-robin 0” immediate 12 yes } } RHEL/CentOS/Oracle EL 5.Best Practices ISE RHEL/CentOS/Oracle EL 4.0/6. 2013 Xiotech .x devices { device { vendor product path_grouping_policy getuid_callout path_checker prio_callout path_selector failback no_path_retry user_friendly_names “XIOTECH” “ISE1400” multibus "/sbin/scsi_id -g -u -s /block/%n" tur “none” “round-robin 0” immediate 12 yes } } RHEL/CentOS/Oracle EL 6.Proprietary and Confiden- . Proprietary and Confidential 160347-000 Rev C.ISE Best Practices RHEL/CentOS/Oracle EL 6. 11 January. 2013 Page 35 .2 defaults { udev_dir polling_interval selector path_grouping_policy getuid_callout prio path_checker rr_min_io max_fds rr_weight failback no_path_retry fast_io_fail_tmo dev_loss_tmo user_friendly_names { device /dev 10 “round-robin 0” multibus "/lib/udev/scsi_id --whitelisted --device=/dev/%n" alua tur 100 8192 priorities immediate fail 5 infinity yes # # # # # # } devices { vendor product path_grouping_policy prio_callout path_checker rr_min_io rr_weight failback fast_io_fail_tmo dev_loss_tmo no_path_retry user_friendly_names "XIOTECH" "ISE1400" multibus “none” tur 100 priorities immediate 5 infinity 12 yes } } SLES 10 SPx devices { device { vendor product path_grouping_policy getuid_callout path_checker prio_callout path_selector failback no_path_retry user_friendly_names "XIOTECH" "ISE1400" multibus "/sbin/scsi_id -g -u -s /block/%n" tur "none" "round-robin 0" immediate 12 no } } SLES 11 devices { device { vendor product path_grouping_policy getuid_callout path_checker prio_callout path_selector failback no_path_retry user_friendly_names "XIOTECH" "ISE1400" multibus "/lib/udev/scsi_id -g -u -d /dev/%n" tur "none" "round-robin 0" immediate 12 no } } defaults { udev_dir polling_interval path_selector path_grouping_policy getuid_callout path_checker rr_min_io failback no_path_retry /dev 10 "round-robin 0" multibus "/lib/udev/scsi_id -g -u -d /dev/%n” tur 100 immediate 12 Xiotech . Best Practices dev_loss_tmo user_friendly_names } 150 no ISE Unix AIX X-IO does not support online upgrade of ISE-2s and Hyper ISEs on AIX operating systems. The assumption is that virtual machines are migrated to other nodes while doing maintenance on a given cluster node. 5. • X-IO also supports the use of Round-Robin MPIO for ESXi. Refer here for more information about Boot From SAN using ESX systems.5. We currently require round-robin load-balancing to use this. the following ql2xlogintimeout parameter for QLogic HBAs is recommended for use with ESXi 5. 2013 Xiotech . This requires Qlogic driver qla2xxx-934. Verify the changes have been made and then migrate virtual machines back to their preferred nodes if needed after the reboot.4.0-855364 or the inbox driver from Emulex. Note. Page 36 160347-000 Rev C. A reboot of the ESXi host will most likely be required when changing these parameters. Note. • For Windows VMs. use the following guidelines to check the ql2xlogintimeout parameter and ensure that it is set properly: # esxcli system module parameters list -m qla2xxx | grep ql2xlogintimeout ql2xlogintimeout int 5 Login timeoutvalue in seconds. # esxcli system module parameters set -p ql2xlogintimeout=5 -m qla2xxx Following the host reboot. HBA Queue Depth Considerations Review the VMware Knowledge Base references that relate to setting HBA queue depth and configuring VM disk request limits in attempting to increase queue depth or throttling the queue depth down to prevent overloading the storage. If you are using a VMware HA configuration.x Refer to “HBA Configuration” on page 32 to ensure that the HBA parameters are set correctly. VMware ESXi Some general guidelines for both ISE-1 and ISE-2 with VMware ESXi are as follows: • X-IO recommends using fixed (preferred path) for the native multi-path settings for ISE LUNs.Proprietary and Confiden- . use the preferred allocation size settings. 11 January. Direct Attach Login Timeout Values for QLogic HBAs To facilitate faster logins following link down or link up events in Direct Attach configurations with ISE-2. cluster nodes can be rebooted one at a time. but experiments have shown improvements for diskintensive workloads. Change the parameter value to the number of your choice and click OK. This change can impact disk bandwidth scheduling. X-IO recommends setting this value to at least 120 seconds to cover certain conditions. check the ql2xmaxqdepth parameter to see if it is set properly: # esxcli system module parameters list -m qla2xxx | ql2xmaxqdepth devices. 2013 Page 37 . 2. the bandwidth is controlled by the queue depth of the storage adapter. select the host in the inventory panel. The parameter controls the maximum number of outstanding requests that all virtual machines can issue to the LUN. the SCSI timeouts on RDMs should be adjusted on Linux Guest OS versions.ISE Best Practices Changing the Queue Depth for QLogic and Emulex HBAs For example. 4. SYSFS{type}=="0|7|14". the administrator can create a script on the Linux Guest OS under the /etc/udev/ rules.SchedNumReqOutstanding. In that case. Click Disk in the left panel and scroll down to Disk. 3. The default SCSI timeouts for RDMs on Linux Guest OS versions are set to 30 seconds. RDMs look like a normal physical disk when viewed from within the Guest OS with the /usr/bin/lsscsi command: # lsscsi Xiotech . Click the Configuration tab and click Advanced Settings under Software. SUBSYSTEMS=="scsi". such as controller firmware upgrades for the ISE-2 platform. to set the ql2xmaxqdepth parameter to 256. RUN+="/bin/sh -c 'echo 120 > /sys/$DEVPATH/timeout'" A Linux Guest OS with existing RDMs may need to be rebooted for this to take effect. ACTION=="add".0. # cat 99-scsi-disk. The parameter does not apply when only one virtual machine is active. int grep qdepth 256 Maximum queuedepth to report for target Please use this parameter with caution as it CAN affect other storage used by the same adapters. use the following command on each ESXi node: # esxcli system module parameters set -p ql2xmaxqdepth=256 -m qla2xxx Following the host reboot.d directory.Proprietary and Confidential 160347-000 Rev C. the script in use is called 99-scsidisk. which are set to 180 seconds by default. As an example. change the Disk.rules #Addition by USER to test RDM scsi timeout settings. In the vSphere Client.rules. Change Maximum Outstanding Disk Requests for Virtual Machines If you adjusted the LUN queue depth. Raw Device Maps (RDMs) When using RDMs with VMware ESXi 5. Note. This does not affect virtual disk timeout values. To change these values.SchedNumReqOutstanding parameter so that its value matches the queue depth. 11 January. Procedure 1. Change this parameter only when you have multiple virtual machines active on a LUN. Up to a 6-node cluster is supported. Status: Operational (None) Member 2: ISE ID: 2000001F93100150. switch to using a fixed path on each lun that will participate in the break.1 only (no DAS). LDID: 143. Note. 5. Windows VMs are not supported at this time for this configuration. ESXi 5.00 /dev/sr0 [2:0:0:0] disk VMware Virtual disk 1. Status: Operational (None) -------------------------------------------------------------------------Matching LUN from within vCenter Server: Page 38 160347-000 Rev C.0 /dev/sda [2:0:1:0] disk XIOTECH ISE2400 A /dev/sdb [2:0:2:0] disk XIOTECH ISE2400 A /dev/sd Once the parameter is properly set.Best Practices ISE [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1. For example: # cat sys/class/scsi_disk/2:0:1:0/device/timeout120 Mirror Special Considerations ESXi 5. 3. it can be verified in the /sys/class/scsi_disk directory structure. 4. 2. 11 January.tc02esxclu_shared_DS8: Status : Operational (None) Type : High Availability (long lived) GUID : 6001F931005C8000024F000400000000 Created : Wed Dec 19 22:02:43 2012 Synch Progress: 100% SameId : yes VDisk Member Count: 2 Master VDisk Member ISE ID: 2000001F931005C8 Master VDisk Member GUID: 6001F931005C800001B6000200000000 Member VDisks: Member 1: ISE ID: 2000001F931005C8. VDisk GUID: 6001F931001500000441000200000000. VDisk GUID: 6001F931005C800001B6000200000000. The preferred path must be set on each cluster node.1 Cluster Active-Active 1. 2013 Xiotech .Proprietary and Confiden- . Before breaking an active-active mirror. The preferred fixed path must be a path on the mirror master ISE. LDID: 23. For example (selected below is a preferred path to the mirror master for the VMirror): From “show vmirror”: VMirror . 2013 Page 39 . Fixed paths must be used when breaking mirrors. 11 January.do?language=en_US&cmd=displayKC&externalId=1033696 2. http://kb.com/selfservice/microsites/ search. QLogic driver: 934. Matching LUN ESXi 5.Proprietary and Confidential 160347-000 Rev C. Xiotech .5. Must use Round Robin ESXi 5.ISE Best Practices Figure 26. Emulex driver is inbox 4.4.1 BFS Cluster Upgrades with Active-Active Mirrors Requires: 1.1 Cluster Active-Active Mirror Breaks 1.0-855364 3.vmware. Best Practices ISE Page 40 160347-000 Rev C. 2013 Xiotech . 11 January.Proprietary and Confiden- . 2013 Page 41 . Physical Topology Xiotech . Figure 27.Proprietary and Confidential 160347-000 Rev C.ISE Best Practices Using IBM SVC with ISE Storage Physical Topology An IBM SVC typically connects to one or more storage products via Fiber-Channel interfaces and presents slices of that storage to hosts as logical disks. 11 January. 2013 Xiotech . Logical Topology If any MDISK goes offline. SVC volumes will stripe across the underlying MDISKs in the MDISK POOL. If the MDISK POOL goes offline. 11 January. Single Fabric • All SVC ports • At least one port from each MRC on each ISE-2 • Both MRC Ports on each ISE-1 • All SVC ports • At least one HBA port Page 42 160347-000 Rev C. all SVC volumes hosted by the MDISK POOL are also offline and hosts will lose access. The stripe width depends on the options set by the SVC administrator. One or more MDISKs are placed in an MDISK POOL. Figure 28. Zoning Single Fabric Zone Purpose Zone Members Storage Access and Inter-Controller Communication Host access to SVC Volumes Figure 29. the entire MDISK POOL goes offline. which is where SVC virtual disks (also called volumes) are created and presented to one or more hosts.Best Practices ISE SVC Logical Topology Each LUN or volume presented from an ISE to the SVC is called an MDISK by the SVC.Proprietary and Confiden- . The sum of the queue depth settings for all hosts connected to the SVC should not exceed this number. This will not cause a problem for the SVC or the ISE as long as the LUNs are presented to ONLY one system. To maintain consistent performance characteristics of the MDISK POOL. each of which contain only ISE volumes from a single ISE DataPac. 2013 Page 43 . it is recommended to zone all connected ISE MRC ports to the SVC controller ports. it is recommended to have a host view storage from either the SVC or from the ISE but not from both at the same time. If a volume from an ISE is presented to both the SVC as an MDISK and to a host as a LUN. it is a very risky move because the failure of a single ISE or a single DataPac will take the entire configuration offline. it is recommended to use ISE volumes of equal size within the same MDISK POOL.ISE Best Practices Redundant Fabric Zone Purpose Fabric A Members Fabric B Members Storage Access and InterController Communication • SVC ports 1 and 2 • At least one port from each MRC on each ISE-2 • Both MRC ports on each ISE-1 • SVC ports 1 and 2 • At least one host HBA port • SVC ports 3 and 4 • At least one port from each MRC on each ISE-2 • Both MRC ports on each ISE-1 • SVC ports 3 and 4 • At least one host HBA port Host access to SVC Volumes Zoning Limitations • In order to prevent single points of failure. No matter how many ISEs are placed behind the SVC. Host Queue Depth Settings The SVC controller pairs can handle various maximum IOPS. Xiotech . data corruption will occur. ISE Volumes Volume Size There are no hard and fast rules on the size of ISE volumes to use. • To avoid a multi-pathing conflict.Proprietary and Confidential 160347-000 Rev C. the performance will be limited to what the SVC can handle. • It is possible to have the ISE shared with an SVC and other systems on the same fabric. depending on the model. 11 January. Volume Placement While it is technically possible to create a single large MDISK POOL of all LUNs on all ISEs behind an SVC controller pair. The recommended practice is to create multiple MDISK POOLs. Different MDISK POOLs can have different LUN sizes behind them with no performance impact. 11 January.Proprietary and Confiden- .Best Practices ISE Page 44 160347-000 Rev C. 2013 Xiotech . VMware.do?language=en_US&cmd=displayKC&externalId=1268 Red Hat Enterprise Linux (RHEL) MPIO Documentation http://docs.ibm.VMware.Proprietary and Confidential 160347-000 Rev C.ISE Best Practices External References Storage and Networking Industry Association (SNIA) Dictionary http://www.novell.org/education/dictionary IBM Fiber Cable Ratings https://www.com/developerworks/mydeveloperworks/blogs/anthonyv/entry/ don_t_say_green_say_aqua1?lang=en VMware Queue Depth Considerations http://kb.com/docs/en-US/Red_Hat_Enterprise_Linux/index.com/files/Driver/81208/readme_FC-FCoE_Inbox_driver_update. 2013 Page 45 .txt Xiotech .html SUSE Linux Enterprise Server (SLES) MPIO Documentation http://www.com/documentation QLogic Red Hat Enterprise Linux 6. 11 January.redhat.x Driver http://filedownloads.do?language=en_US&cmd=displayKC&externalId=1267 VMware Maximum Disk Requests per VM http://kb.snia.com/selfservice/microsites/search.com/selfservice/microsites/search.qlogic. Proprietary and Confiden- .Best Practices ISE Page 46 160347-000 Rev C. 11 January. 2013 Xiotech . Best Practices Xiotech .Proprietary and Confidential XXXXXX-000 Rev B. 11 January. 2013 Page 47 .