53-1000885-02January 30, 2009 Fibre Channel HBA Troubleshooting Guide Supporting Models 415, 425, 815, 825 Copyright © 2006-2009 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol, DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government. The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity with respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that accompany it. The product described by this document may contain “open source” software covered by the GNU General Public License or other open source license agreements. To find-out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd. Brocade Communications Systems, Incorporated Corporate and Latin American Headquarters Brocade Communications Systems, Inc. 1745 Technology Drive San Jose, CA 95110 Tel: 1-408-333-8000 Fax: 1-408-333-8101 E-mail:
[email protected] Asia-Pacific Headquarters Brocade Communications Singapore Pte. Ltd. 30 Cecil Street #19-01 Prudential Tower Singapore 049712 Singapore Tel: +65-6538-4700 Fax: +65-6538-0302 E-mail:
[email protected] European Headquarters Brocade Communications Switzerland Sàrl Centre Swissair Tour B - 4ème étage 29, Route de l'Aéroport Case Postale 105 CH-1215 Genève 15 Switzerland Tel: +41 22 799 5640 Fax: +41 22 799 5641 E-mail:
[email protected] Document History Title Publication number Summary of changes Date Fibre Channel HBA Troubleshooting Guide 53-1000885-01 New document December 2008 Fibre Channel HBA Troubleshooting Guide 53-1000885-02 Revised with corrections. January 2009 Contents About this Document In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . viii What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Notice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Getting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Chapter 1 Introduction to troubleshooting In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 How to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . 1 Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Chapter 2 Isolating Problems In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 General problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Resolving installation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Verifying installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Errors when installing driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Installer program does not autorun from CD (Windows only) . . 13 “Files needed for bfad.sys” message appears when removing driver 13 Cannot roll back driver on all HBA instances using Device Manager 14 Host not booting from remote LUN . . . . . . . . . . . . . . . . . . . . . . . 14 Confirming driver package installation. . . . . . . . . . . . . . . . . . . . 14 Host system freezes or crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 HCM GUI fails to connect with HCM agent . . . . . . . . . . . . . . . . . . . . 17 Verifying Fibre Channel links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Chapter 3 Tools for Collecting Data In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 iii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 HCM logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Displaying settings through HCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Virtual port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Fabric statistics . 42 HBA temperature . . . . . . . . 46 Base port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Logical port statistics. . . . . . . . . 47 Port query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Quality of service (QoS) statistics . . . . . . . . . . . . 28 Collecting data using BCU commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Remote port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Port speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Port log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Beaconing. . . 45 Collecting SFP data . . . . . . . . . . . . . . . . . . 48 Authentication settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 iv Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . . . . . . . . . . 34 Port statistics . . . . . . . . 24 Collecting data using BCU commands and HCM . . . . . . . . . . . . . . . . . . 43 Trace route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Memory test . . . . . . . . . . . . . . . . . . . . 28 Event logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 FCIP initiator mode statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Virtual port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 SCSI test . 43 Ping end points . . . . . . . . . . . . . . . . . . . . . . . 48 Displaying settings through BCU. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 PCI loopback test . . . . 45 Port power on management (POM) . . . . . 24 Collecting data using host system commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Data to provide support . . . 36 Remote port statistics . . . . . . . . . . . . . . . . . . . . . 32 Syslog support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 HBA data collection using HCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Diagnostics . . . . . . . . . . . . . . . . . . . 44 Echo test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Windows Event Log support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .For detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Loopback tests. . . . . . . . . . . . . 46 Collecting port data. . . . . . . . . . . . . . . . . . . . . . . . . 25 Using Support Save . . . . . . . . . . . . . . . . . . . . . . . . . . 45 SFP diagnostics . 46 Logical port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 IOC statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Host system logs . . . . . . . . . . . . . . . . . 47 Port list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 OS tunable parameters . 49 HCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Chapter 4 Performance optimization In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Persistent binding . . . . . 53 VMware tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Linux tuning . . 53 Index Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 v . . . . . .QoS and target rate limiting settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Solaris tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Driver tunable parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 BCU commands . . . . . . . . . . . . . . . . . . 52 Windows tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . . . . . . . . . “Tools for Collecting Data” provides a summary of diagnostic and monitoring tools available through the HCM. . . . .About this Document In this chapter • How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brocade Command Line Utility (BCU). . . . . . . . . . . . . . . . xiii How this document is organized This document is organized to help you find the information that you want as quickly and easily as possible. . . . . . . . . . “Introduction to troubleshooting” provides an introduction and approach to troubleshooting HBA problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A checklist is also provided to verify that required procedures have been followed during installation. . . . . . . . . . . . . . . . . . fabric switch operating system and host system to help you isolate and resolve HBA-related problems. . . “Performance optimization” contains guidelines for optimizing HBA performance on your host system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . “Isolating Problems” provides information on common HBA problems and procedures to diagnose and recover from these problems • Chapter 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix • Notice to the reader . Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 vii . . . . . . . . . The document contains the following components: • Chapter 1. . . . . . . . . . . . vii • Supported hardware and software. . . . . viii • What’s new in this document . . . . . . . . . . . . . . . . . xi • Getting technical help . • Chapter 4. . . . . . . . . . . . • Chapter 2. . . . . . . . . . NOTE This publication is a companion guide to be used with the Brocade Fibre Channel HBA Administrator’s Guide. . . . . . . . . . . . . . . . . ix • Document conventions . . . . xi • Additional information . . . . . . . . . . . . . . . . . That publication provides detailed information on HBA monitoring and diagnostic tools in Host Connectivity Manager (HCM) and the BCU. . . . . . . . xii • Document feedback . . . . as well as tips for gathering problem information. . . . . . . . Supported hardware and software This section describes HBA hardware and software support. BCU. storage. Fabric OS and switch support For a current list of compatible servers. and HBA drivers: • • • • • • • Windows Server 2003. Brocade 425 Dual-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP. Single-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+. HCM is supported only on the guest operating system on VMware. • • • • Brocade 815. Brocade 415. only 4 Gbps maximum port speed is possible. • Install only Brocade-branded SFPs in these HBAs. Notes: • This publication only supports the HBA models listed above and does not provide information about the Brocade 410 and 420 Fibre Channel HBAs.com/hba. refer to the Brocade HBA web site at www. switches. and applications. and HCM Agent are supported only on the VMware “console” Operating System. • Although you can install an 8 Gbps SFP+ into a Brocade 415 or 425 HBA. • Windows Vista (HCM only) • Windows XP (HCM only) viii Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Brocade 825.brocade. HBA support The following Fibre Channel host bus adapters (HBAs) are supported in this release. and SLES10 Solaris 10 (x86 and SPARC) VMware ESX Server 3. version R2 with SP2 Windows Server 2008 Windows NT (HCM support only) Windows 2000 (HCM support only) Linux RHEL4. Brocade Command Line Utility (BCU). RHEL5. Dual-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+. Host operating system support The following operating systems support Brocade Host Connectivity Manager (HCM). Single-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP. SLES9. also known as the Brocade 400 Fibre Channel HBAs.5 NOTE Drivers. Text formatting The narrative-text formatting conventions that are used are as follows: bold text Identifies command names Identifies the names of user-manipulated GUI elements Identifies keywords and operands Identifies text to enter at the GUI or CLI italic text Provides emphasis Identifies variables Identifies paths and Internet addresses Identifies document titles code text Identifies CLI output Identifies command syntax examples For readability. For further information about new features not covered in this document and documentation updates for this release. In actual examples. this manual specifically notes those cases in which a command is case sensitive. Otherwise. Document conventions This section describes text formatting conventions and important notice formats used in this document. What’s new in this document This is a new document. switchShow. command names in the narrative portions of this guide are presented in mixed lettercase: for example. refer to the HBA release notes. command lettercase is often all lowercase. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 ix .NOTE Specific operating system service pack levels and other patch requirements are detailed in the current HBA release notes. ATTENTION An Attention statement indicates potential damage to hardware or data.. Safety labels are also attached directly to products to warn of these conditions or situations. for example “member[.. see the technical glossaries on Brocade Connect. or data. x Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Example: --show -mode egress | ingress Notes. For example. cautions. NOTE A note provides a tip. guidance or advice. option Command options are printed in bold. Repeat the previous element. [] Optional element. --show WWN | Boolean. Key terms For definitions specific to Brocade and Fibre Channel. DANGER A Danger statement indicates conditions or situations that can be potentially lethal or extremely hazardous to you. --option. firmware. In the help pages. values are underlined or enclosed in angled brackets < >.]” value Fixed values following arguments are printed in plain font. Elements are exclusive.Command syntax conventions Command syntax in this manual follows these conventions: command Commands are printed in bold. . or provides a reference to related information. -argument. software. See “Brocade resources” on page xi for instructions on accessing Brocade Connect. arg Arguments. see the Brocade Fibre Channel HBA Installation and Reference Manual. For definitions specific to this document.. emphasizes important information. CAUTION A Caution statement alerts you to situations that can be potentially hazardous to you or cause damage to hardware. variable Variables are printed in italics.member. and warnings The following notices and statements are used in this manual. They are listed below in order of increasing severity of potential hazards.. Red Hat Enterprise Linux (RHEL) Novell. Inc. visit the Brocade Web site: http://www.For definitions of SAN-specific terms. Hyper V for Windows. Brocade resources For HBA resources. and documentation.brocade. software.brocadeconnect. Windows Automated Installation Kit (WAIK) Sun Microsystems. join Brocade Connect. PE for Windows.brocade. Go to http://www.com/hba. firmware. For practical discussions about SAN design.snia.com Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 xi .amazon. Corporation Referenced Trademarks and Products Microsoft Corporation Windows. Solaris Red Hat Inc.org/education/dictionary Notice to the reader This document may contain references to the trademarks of the following corporations. visit the Brocade HBA web site at www. XP. These trademarks are the properties of their respective companies and corporations.com to register at no cost for a user ID and password. visit the Storage Networking Industry Association online dictionary at: http://www. and maintenance.com For additional Brocade documentation. such as product information. you can obtain Building SANs with Brocade Fabric Switches through: http://www. Vista. These references are made for informational purposes only. Windows Server 2008. ESX Server SPARC International. implementation. Windows Server 2003. Inc SUSE Linux Enterprise Server (SLES) VMware Inc. Inc SPARC Additional information This section lists additional Brocade and industry-specific documentation that you might find helpful. To get up-to-the-minute product information. if applicable syslog message logs bfa_supportsave output. • Detailed description of the problem. visit the Technical Committee T11 Web site. You can save valuable information to your local file system and send it to support personnel for further investigation. *FT00X0054E9* FT00X0054E9 You can also display the serial number through the following HCM dialog boxes and BCU commands: • HBA Properties tab in HCM. refer to “Using Support Save” on page 25. xii Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 .t11. Select an HBA in the device tree. and software support. visit the Fibre Channel Industry Association Web site: http://www. The serial number label is located on the end of the HBA opposite from the SFP receivers. including product repairs and part ordering. internal libraries. 2.org Getting technical help Contact your Brocade HBA support supplier for hardware.org For information about the Fibre Channel industry. • Description of any troubleshooting steps already performed and the results. and other applications: http://www.fibrechannel. including the switch or fabric behavior immediately following the problem. General information: • • • • • Brocade HBA model number Host operating system version Software name and software version. For details on using this feature.Other industry resources For additional resource information. Provide the following information: 1. • BCU adapter --list command. This Web site provides interface standards for high-performance and mass storage applications for Fibre Channel. use the bfa_supportsave feature to collect debug information from the driver. and firmware. then click the Properties tab in the right pane. storage management. firmware. and specific questions. To expedite your support call. HBA serial number: The HBA serial number and corresponding bar code are provided on the serial number label illustrated below. then select Adapter Settings to display the WWNN and PWWN for the port. Determine this through the following resources: • Label located on the end of the HBA opposite the SFP receiver slots. 3. • HCM Port Properties dialog box. Forward your feedback to: documentation@brocade. • Brocade BIOS Configuration Utility. refer to the Brocade Fibre Channel HBA Installation and Reference Manual. Select the appropriate HBA port from the initial configuration utility screen. Port World-Wide Port Name (PWWN). refer to the Brocade Fibre Channel HBA Administrator’s Guide. The <ad_id> parameter is the HBA’s serial number. click a port in the device tree. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 xiii . To display this dialog box. This dialog box displays the PWWN for each HBA port. port --list <ad_id> This command lists all the physical ports on the HBA along with their basic attributes.NOTE For details on using HCM and BCU commands. we want to hear from you. including the topic heading and page number and your suggestions for improvement. then click the Properties tab in the right pane. or you think that a topic needs further development.com Provide the title and version number of the document and as much detail as possible about your comment. The <ad_id> parameter is the HBA’s serial number. This label provides the WWPN for each port. Document feedback Quality is our first concern at Brocade and we have made every effort to ensure the accuracy and completeness of this document. However. For details. • The following BCU commands: adapter --query <ad_id> This command displays HBA information. if you find an error or an omission. xiv Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . “Tools for Collecting Data” to gather data for problem isolation or resolution. and host operating system commands described in Chapter 2. . . storage devices. . the problem could also originate in another SAN component or connections between these components. . Use the BCU commands. . . . Fixes and actions often reference the BCU commands. . . . . . . . and diagnostics. “Host system freezes or crashes” on page 16. “Resolving installation problems” on page 12. . Fully describe the problem and gather complete information about the symptoms that suggest a problem exists. . causes. . Isolate or resolve the problem by first using information in Chapter 2. . . . launching HBA diagnostics. . . . . or even gathering statistics on HBA operation. HCM features. . it is important that you perform the following tasks: 1. and host operating system commands described in Chapter 3. . Consider these factors when isolating and resolving the problem: • Can the issue be resolved by using the latest supported combination of host system BIOS. and the fiber optic cabling and connections for these components. . . . . . . . such as event logs. • First study information in the general HBA problem symptoms. such as a small form factor pluggable (SFP) optical transceiver. operating statistics. . .Chapter Introduction to troubleshooting 1 In this chapter • How to use this manual for troubleshooting . and fixes or actions in Table 2 on page 6. many more are included that can provide helpful data. . Before removing and replacing HBAs. . . 2. . . . 3. “Isolating Problems” to gather data to help you isolate problems. . operating system updates. . . Although many of these tools are specifically referenced as actions for problems described in Table 2 (Troubleshooting General Problems) in Chapter 2. “Verifying Fibre Channel links” on page 20. . . . “Isolating Problems”. . . . Note that Table 5 on page 24 in Chapter 3 provides a list of useful host system commands for each supported operating system that you can use to gather data. or HBA drivers? Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 1 . . 1 • Gathering problem information . use the following sections in Chapter 3. HCM features. . . 4. Although there may be a problem in the HBA or an HBA component. . This table should provide help on many general problems that you may encounter with HBA operation. . Note that these sections are referenced from Table 2 (Troubleshooting General Problems) when appropriate to further isolate problems. . . . . . 2 How to use this manual for troubleshooting A Fibre Channel HBA is one component in a SAN consisting of Fibre Channel switches. • If you still require more information to isolate problems. operating system. Refer to “Gathering problem information” on page 2. storage device.All LEDs on HBA port flashing amber. such as slow performance or file access. 5.User experiences. . HBA not recognized as PCI device by host system operating system. and controller model and firmware levels. - Run the Fabric OS supportSave command on any Brocade switch and save output. . • Draw a topology map of the SAN from the HBAs to the storage targets. core file. or service providers of that equipment. firmware. Here are some examples: .You can save captured information to the local file system and send it to support personnel for further investigation. refer to “Using Support Save” on page 25. internal libraries. - HBA not recognized by host system BIOS. and cable? • Can this problem be reproduced on one or more HBAs. supportShow. and other information needed to diagnose suspected system issues. refer to documentation. SFP. tape. help systems. or host system BIOS? • Is the problem documented in release notes for the switch and target storage system? • Is unexpected behavior intermittent or always present? If the problem is in a Fibre Channel switch. gather and provide problem information to your HBA support provider for resolution. cabling. - Run the bfa_supportsave BCU command on the host system and save output to a file on your system.1 Gathering problem information • Does the issue persist when the HBA is installed in a different platform or is connected using a different switch port.LEDs not functioning on an HBA port that is connected to the fabric.Expected storage devices not visible from the HCM or host system’s storage management application. TRACE. . or host system? Can you identify specific steps that consistently reproduce this problem on one or more hosts? • Is the problem documented in release notes for the HBA. operating system. • Describe the symptoms that you are observing. Include the following: 2 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Gathering problem information Perform the following tasks to obtain as much information as possible before contacting technical support. This command collects RASLOG. port. If you cannot resolve the problem. • Gather information for support. Be specific. This command captures all driver. • What happened prior to the observed symptoms? • Describe all observed behavior that is unexpected and compare against expected behavior. For details on using the Support Save feature. Be sure to take careful notes for use as a record and reference. FFDC data and other support information.Use appropriate tools on storage targets to gather information such as disk. . or in connectivity between these components. Model. Port WWNs connected to all links. including changes attempted to isolate the problem. if the SAN functioned without problems before installing the HBA. and SAN fabrics. Click the Support tab on www. and storage ports. historical performance data. • List steps that have been taken to troubleshoot the problem. consider using the Brocade SAN Health products to provide information on your SAN environment. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 3 . BCU. an offline switch. or a disconnected or faulty cable between the HBA. and driver release level. • Record the time and frequency of symptoms and the period of time symptoms have been observed. and connected devices. • Determine what has changed in the SAN. WWN. firmware versions. • Determine if unexpected behavior is intermittent or always present. or HBA driver package. switches. The bfa_supportSave and FOS supportsave commands can provide current information for the topology map. and other data. “Tools for Collecting Data” to gather statistics and problem data on the HBA. then the problem is most likely in the HBA installation or configuration. zoning and switch configurations. host. HBA hardware. and Fabric OS commands summarized in Chapter 3. switch. switches.brocade. For example. • Use additional HCM.com for more information on these products. Host hardware Model and hardware revision. and Fabric OS version. Also. host system. Fibre Channel links. World-Wide Name (WWN). Fiber optic links between HBA. • Run appropriate diagnostic tools for storage targets. including an inventory of devices. or storage controller fiber optic ports. Fibre Channel switches.Gathering problem information l TABLE 1 1 Topology map details Component How to identify HBA Model. Other examples to investigate could be changes in the switch or storage system firmware. 1 4 Gathering problem information Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . . . . . . . . and your host system commands. • Fabric OS Troubleshooting and Diagnostic Guide. . . . . 12 • Host system freezes or crashes . . . . . . . . . . . . . . 16 • HCM GUI fails to connect with HCM agent. . • Fabric OS Administrator’s Guide Provides detailed information on features available on Brocade storage area network (SAN) products. . . . . . . Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 General problems Table 2 on page 6 describes general problems related to HBA operation. . . . . . . . . . and troubleshoot Brocade SAN products. . . . . . . . . . Fabric OS commands. . . . • Chapter 3. . . . . . . . Provides help with isolating problems in other Brocade SAN components. • Your host’s operating system documentation and help system. and recommended actions that may fix the problem. . . . . .Chapter Isolating Problems 2 In this chapter • General problems . 17 • Verifying Fibre Channel links. . “Tools for Collecting Data” in this manual. . . . . . HCM options. . . . . . . Recommended actions may refer you to information in the following locations as appropriate to gather information to further isolate and resolve the problem. . . . . . . . . Provides details on commands for gathering information and isolating problems. Contains procedures to perform HBA diagnostics. . . and how to configure and administer these products • Fabric OS Command Reference Manual. . . 5 • Resolving installation problems . . and collect data for troubleshooting using BCU commands. . . . . . . . maintain. . . . . . . . . . . . . . . . . . . . . . . . . . display HBA statistics and event logs. . . . . possible causes. . . . . . . . . . . . . . . . . . . . . . . . . Provides detailed descriptions of command line interface commands to help system administrators and technicians operate. . . Refer to the “List PCI Devices” row in Table 5 on page 24. go on to the next step. To find this document. The HBA driver may not be loaded. perform the following steps. 1 2 3 6 Execute host operating system command to list PCI devices. HBA not compatible with host operating system or connected storage systems. Refer to “Confirming driver package installation” on page 14 for methods to verify driver installation. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Refer to the “List PCI Devices” row in Table 5 on page 24. Server slot issues. Reseat the HBA. Refer to “HBA not reported under server’s PCI sub-system” under the “Symptom” column in this table. HBA driver not loaded. Execute your host’s operating system command to list PCI devices. If the HBA is not listed.2 TABLE 2 General problems Troubleshooting general problems Symptom Possible Cause Fix or Action HBA not reported under server’s PCI subsystem. Verify compatibility by reviewing the Brocade Server Connectivity Compatibility Matrix. 1 2 3 1 Card not seated properly. If the HBA is not listed in the output from this command. 2 3 4 No HBAs reported when using the adapter --list BCU command: 1 2 HBA not reported under server’s PCI sub-system.com. Replace the HBA with an HBA in known working condition to determine whether there is a slot malfunction. log into Brocade Connect on www. then select the Compatibility Information quick link under Documentation Library.brocade. 5 Check port speed on the HBA with the BCU port --list or port --query commands to display the current and configured speed. For details on using this command. the speed setting for both ports is 4 Gbps). If non-brocade branded SFPs are inserted on the HBA side or 8 Gbps switch or director. On the switch. refer to “Port list” on page 47. 1 2 1 3 4 5 SFP or cable problems. Link between HBA and switch is down. Check the switch port speed using the Fabric OS portCfgShow command to verify that Speed is either AUTO or matches the speed of the attached HBA port (for example. Switch port is disabled or switch is disabled. Verify HBA side link status by executing the BCU port --list command. 2 3 4 Ensure that the SFPs and cables are connected properly on both HBA and switch side. execute the Fabric OS switchShow command to verify that “Mod_Inv” (invalid module) does not display for the port state. HBA port speed or topology mis-match with the switch port.General problems TABLE 2 2 Troubleshooting general problems Symptom Possible Cause Fix or Action Port link does not come up. Check for any cable damage. Check the port topology setting on the switch using the Fabric OS portCfgShow command to ensure that Locked L_Port is OFF. Check the FC Addr field for an address and the State field for Linkup. the port link will not come up. Refer to “Port speed” on page 48 and “Port query” on page 47 for details on using these commands. Use the portCfgLport command to change the setting to OFF if required. Execute either the Fabric OS switchShow or portShow commands on the attached switch to ensure that the switch or individual port is not disabled or offline. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 7 . Non-Brocade branded SFP installed. For the HBA. execute the secAuthSecret Fabric OS command. For the HBA. refer to “Verifying Fibre Channel links” on page 20. Use the BCU auth --show <port> command on the HBA and Fabric OS authutil --show command on switch. Refer to “Port list” on page 47 and “Port query” on page 47. execute the BCU auth --show command (refer to “Authentication settings” on page 48). Check the shared secret configuration on the attached switch and HBA. For additional actions and fixes for the port link not coming up. execute the port --list and port --query BCU commands. the “Sfp” field displays “us” (unsupported SFP) for port --list and the “Media” field displays “Unsupported SFP” for port --query. For the switch. Loss of sync and loss of signal errors in port statistics (refer to “Port statistics” on page 34). If an unsupported SFP is detected. Refer to “Verifying Fibre Channel links” on page 20 Fabric authentication failures Authenticating configuration is incorrect. Ensure that the SFPs and cables are connected properly on both HBA and switch side. Check for any cable damage. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 .2 TABLE 2 General problems Troubleshooting general problems Symptom Possible Cause Fix or Action Port link does not come up (continued) On the HBA. execute the authutil --show Fabric OS command. For the switch. • • • 8 Check authentication settings on the switch and HBA. execute the auth – -secret BCU command (refer to “Authentication settings” on page 48) for details on using the auth-secret command. Possible physical link problem. General problems TABLE 2 2 Troubleshooting general problems Symptom Possible Cause Fix or Action Target not visible. 1 1 2 3 4 No fabric connectivity between HBA and target or target is not online. The target and the HBA are not on the same zone. The HBA driver is not loaded. There is a problem with remote port. 2 3 4 LUN not visible 1 2 Missing or improper storage array LUN masking setting. HBA driver not loaded. 1 2 Execute the Fabric OS nsAllShow command on the attached switch to verify that the target and the host are online in the fabric and registered in the name server. Execute the Fabric OS cfgActvShow command on the attached switch and verify that the host and target are in the same zone (either using domain area members, port area members, or port or node WWNs). The HBA driver may not be loaded. Refer to “Confirming driver package installation” on page 14 for methods to verify driver installation. Verify that the remote target port (rport) is reporting itself online by comparing rport online and rport offline statistics (refer to “Remote port statistics” on page 36). The rport online counter should be one greater than the rport offline counter. If not, clear the counters and try connecting to the remote port again. Verify the rport online and rport offline statistics again. Check LUN mapping and masking using storage array configuration tools. The HBA driver may not be loaded. Refer to “Confirming driver package installation” on page 14 for methods to verify driver installation. I/Os are not failing over immediately on a path failure in MPIO setup. Improper driver mpiomode setting. Execute the port --query <port_id> BCU command and ensure fcpim MPIO mode is enabled (which implies zero Path TOV values) or that fcpim MPIO mode is disabled with the expected “Path TOV” settings (default is 10 seconds). Unable to create more than 126 Virtual (NPIV) ports. The switch’s maximum NPIV limit has been reached. Execute the Fabric OS configure command on the attached switch and change the Maximum logins per port parameter under the F_Port login parameters menu to increase the maximum NPIV I/Ds allowed per port. On Linux, the maximum IOPS numbers are very low. The amount of disk I/O requests are Refer to “Linux tuning” on page 51 for causing low throughput and high latency. suggestions to optimize HBA performance in Linux systems. On VMware, the maximum IOPS numbers are very low. The amount of disk I/O requests are Refer to “VMware tuning” on page 53 for causing low throughput and high latency. suggestions to optimize HBA performance in Vmware systems. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 9 2 TABLE 2 General problems Troubleshooting general problems Symptom Possible Cause Fix or Action QoS performance issues 1 1 2 QoS not enabled on both switch and HBA port. QoS zones not created properly on switch for high, medium, and low priority targets. 2 3 Verify if QoS is enabled for an HBA port using the qos -query <port_id> BCU command. Verify if it is enabled on the switch using the islShow command. Verify zones on the switch using cfgActvShow command. Verify that QoS is configured on switch using instructions in the Brocade Fabric OS Administrator’s Guide. The HBA is not showing in the fabric. There is a problem in the fabric or a protocol issue between the HBA and fabric. Check fabric statistics. Refer to “Fabric statistics” on page 36 for methods to display fabric statistics for the HBA. • If counts for FLOGI sent and FLOLGI accept fabric statistics do not match, suspect fabric problem or protocol issue between HBA and fabric. • If fabric offline counts increase and fabric maintenance is not occurring, this may indicate a serious fabric problem. Refer to your switch troubleshooting guide. Errors when installing bfa_driver_linux--<version>.noarch.rpm driver package. Appropriate distribution kernel development packages are not installed on your host system for the currently running kernel. Refer to “Errors when installing driver” on page 13 for more information to isolate this problem. Installer program does not autorun (Windows only). Autorun is not enabled on your system. Refer to “Installer program does not autorun from CD (Windows only)” on page 13 for more information to isolate this problem. Host not booting from remote LUN. Boot from LUN not configured correctly. Refer to “Host not booting from remote LUN” on page 14 for more information to isolate this problem. Host system freezes or crashes. 1 Fault fiber optic cabling and connections. Faulty or unseated SFPs or unsupported SFPs. Conflicts with port operating speed or topology of attached devices. HBA not compatible with host system. Refer to “Verifying Fibre Channel links” on page 20 for more information to isolate this problem. Problem in the fabric or a protocol issue between the HBA and fabric. NPIV is not supported or is disabled on the switch Check virtual port statistics, such as FDISK sent, FDISK accept, and No NPIV support statistics. Refer to “Virtual port statistics” on page 38 for methods to display virtual port statistics. 2 3 4 Virtual devices not listed in name server. • • Operating system errors (blue screen). 10 HBA driver not loaded. Refer to “Confirming driver package installation” on page 14 for methods to verify driver installation. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 General problems TABLE 2 2 Troubleshooting general problems Symptom Possible Cause “Failed to connect to agent on host...” error when using HCM. The client application cannot connect to Refer to “HCM GUI fails to connect with HCM agent” on page 17 to isolate and the HCM Agent listening on the configured port for the following reasons: resolve the problem. • The agent is not running. • The agent not accepting connections on the expected port. • The agent is not listening on the expected port. • Communication between the client and agent is blocked by a firewall preventing access to the port (usually only a consideration for remote HCM management). The HBA not registering with the name server or cannot access storage. 1 2 3 Driver event messages appearing in host system log files. HBA cannot log in to the name server. HBA has a problem registering with the name server. HBA has a problem querying the name server for available storage. Various causes and severity levels. Fix or Action Display logical port statistics (refer to “Logical port statistics” on page 37. Check for the following errors: 1 Increasing name server port login (NS PLOGI) error rejects and unknown name server port login response (NS login unknown rsp) errors means that HBA most likely cannot log in to the name server. 2 Increasing errors of the following types indicates that the HBA has a problem registering with the name server. • Name server register symbolic port name identifier (NS RSPN_ID) errors. • Name server register symbolic port name identifier response (NS RFT_ID rsp) errors • Name server register symbolic port name identifier response rejects (NS RFT_ID rejects). 3 Increasing name server get all port ID response (NS GID_FT rsp), rejects (NS_GID FT rejects), or unknown responses (NS_GID FT unknown rsp) means that the HBA has a problem querying the name server for available storage. • • • Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 Follow recommended action in message. Resolve critical-level messages and multiple major or minor-level messages relating to the same issue as soon as possible. For details on event messages, refer to “Event logs” on page 30. 11 incompatibility between the HBA and your host system. then rolling back driver HBA instances using the Device Manager. improper fiber optic cable connected to the fabric. then subsequently removing the driver using the Device Manager. Always use the Brocade installer programs to remove the driver.sys” message appears when removing driver” on page 13 for more information. Use the driver installer program (bfa_installer. Determine if problems may exist because of these factors by reviewing your installation with information in the Brocade Fibre Channel HBA Installation and Reference Manual listed in Table 3. check the installed version of the HBA (chip revision) and driver (fw version) using the adapter --query BCU command.com/hba) for known problems relating to the HBA and driver versions.exe) or Brocade Software Installer (GUI or command-based application). Resolving installation problems This section provides a summary of common HBA problems that may occur during or immediately after installing HBAs and describes methods to resolve these problems.exe) or Brocade Software Installer (GUI or command-based application) to install or upgrade the driver. Refer to “Cannot roll back driver on all HBA instances using Device Manager” on page 14 for more information. then use the Brocade Software Uninstaller to roll back drivers on all HBA instances in one-step. refer to “Collecting data using BCU commands” on page 28. • • • Install the driver for each HBA instances using the Device Manager.exe) or Software Installer (GUI or command-based application). To use this command. Refer to ““Files needed for bfad. NOTE If troubleshooting actions in Table 2 do not resolve problems. then roll back the driver using Device Manager. 12 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Verifying installation Problems with HBA operation may be due to improper hardware or software installation. or the HBA not operating within specifications.brocade.sys” message appears when removing driver Installing the driver using the driver installer program (bfa_installer. Do not uninstall the driver using the Device Manager if you have used the Brocade installer programs to install driver instances. Cannot roll back driver on all HBA instances using Device Manager Installing the driver using the Brocade driver installer program (bfa_installer. Refer to release notes posted on the Brocade HBA web site (www.2 TABLE 2 Resolving installation problems Troubleshooting general problems Symptom Possible Cause Fix or Action “Files needed for bfad. unsupported SFPs installed on the HBA. Resolving installation problems TABLE 3 2 Installation and Reference Manual references Information Chapter Hardware and software compatibility information.noarch. Product Overview Software installation packages supported by host operating system and platforms. do not uninstall the driver using the Device Manager if you have used the Brocade installer programs to install driver instances.sys” message appears when removing driver When removing the driver with Device Manager or using the Brocade Software Uninstaller.exe) or Brocade Software Installer (GUI or command-based application). If only one driver is present in the system. These should include the gcc compiler and the kernel sources. This occurs if you perform the following sequence of steps: 1.brocade.com/hba. Always use the Brocade installer programs. Enable autorun if possible and reinsert the CD. Uninstall the Brocade HBA using Device Manager. the driver module compiles on the system during installation. Be sure to “install everything” including the developer library options. Install the driver using the driver installer program (bfa_installer. Specifications Errors when installing driver If you are installing the bfa_driver_linux--<version>. Re-install the driver using the driver installer program (bfa_installer. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 13 . program a message appears asking for “Files needed for bfad. and installer program does not autorun. make sure that autorun is enabled on your system.exe) program. If these are not installed. verify that the appropriate distribution kernel development packages are installed on your host system for the currently running kernel. then the Brocade programs also remove the Fibre Channel devices from the Device Manager. 3. 2. If driver build errors result when you install the package.cmd located on the installation the CD.exe) or Brocade Software Installer (GUI or command-based application). Try manually executing the LaunchWinSetup. 4.rpm package. Uninstall the driver using the driver installer program (bfa_installer. Installer program does not autorun from CD (Windows only) When installing software from a CD created from the ISO image downloaded from www. To avoid this problem. Installation Product specifications. “Files needed for bfad. you may need to reinstall the operating system before continuing installation. Product Overview Hardware and software installation instructions.sys” to complete the installation. An out-of-date driver may cause the following problems: • Storage devices and targets not being discovered by the device manager or appearing incorrectly in the host’s device manager.exe) or Brocade Software Installer (GUI or command-based application) to install or upgrade the driver. Host not booting from remote LUN If booting the host from a remote boot device.exe) or Software Installer (GUI or command-based application). • Use the driver installer program (bfa_installer. 14 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . then use the Brocade Software Uninstaller to roll back drivers on all HBA instances in one-step. 2. • Host operating system not recognizing HBA installation. and HBA port WWNs. NOTE If driver is not installed. • The host’s operating system. driver version. • BIOS or EFI is enabled to support boot over SAN from a specific HBA port. firmware. verify the following: • A zone is created on the attached switch that contains only the PWWN of the storage system port where the boot LUN is located and the PWWN of the HBA port. HBA driver. • BIOS or EFI is configured to boot from a specific LUN. verify whether “boot over SAN” configuration is complete and correct. use one of the following methods: • Install the driver for each HBA instances using the Device Manager. you can only roll back the driver for the first HBA instance.2 Resolving installation problems Cannot roll back driver on all HBA instances using Device Manager When using the Windows Device Manager. and other necessary files are installed on the boot LUN. try re-installing the driver or re-installing the HBA hardware and then the driver. • Operating system errors (blue screen). • Improper or erratic behavior of HCM (installed driver package may not support HCM version). This occurs if you perform the following sequence of steps: 1. Make sure that the correct package is installed for your operating system. To avoid this problem. Refer to the installation chapter in the Brocade HBA Installation and Reference Manual. You can use HCM and tools available through your host’s operating system to obtain information such as driver name. Install the driver using the Brocade driver installer program (bfa_installer. and HCM agent for specific operating systems. then roll back the driver using Device Manager. For example. Roll back driver HBA instances using Device Manager. Confirming driver package installation HBA driver packages from Brocade contain the current driver. the HBA driver is loaded to the system. 3. Click the Driver tab to display the driver date and version. 5. the driver is not installed. If you do not see this entry or Fibre Channel Controller displays with a yellow question mark under Other Devices. Select the Properties tab in the right pane to display the Properties dialog box. • # dmesg This command prints kernal boot messages. • # lspci This is a utility that displays information about all PCI buses in the system and all devices connected to them. Select Properties to display the Properties dialog box. Launch HCM. and the BIOS version currently in operation 1. Right-click the Brocade FC HBA model where you are installing the driver. firmware name and version. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 15 . • # lsmod This command displays information about all loaded modules. Click Driver Details for more information. 4. 2. Verify if the driver is installed and Windows is recognizing the HBA using the following steps. try re-installing the driver or re-installing the HBA hardware and then the driver. Open the Device Manager. Windows Use the Device Manager to determine driver installation. 3.Resolving installation problems 2 Host Connectivity Manager (HCM) Use the following steps to display the HBA PWWN. NOTE If driver is not installed. driver name and version. Linux Verify if the HBA driver installed successfully using the following commands: • # rpm -qa|grep -i bfa This command prints the names of the Brocade HBA driver package (bfa) if installed. 1. 2. If bfa appears in the list. Expand the list of SCSI and RAID controllers. For the bfa entry. HBA model and driver version should display if the hardware and driver are installed successfully. Select the HBA in the device tree. The dialog box displays HBA properties. PKGINST: NAME: CATEGORY: ARCH: VERSION: BASEDIR: VENDOR: DESC: PSTAMP: INSTDATE: HOTLINE: STATUS: bfa Brocade Fibre Channel Adapter Driver system sparc&i386 alpha_bld31_20080502_1205 / Brocade 32 bit & 64 bit Device driver for Brocade Fibre Channel adapters 20080115150824 May 02 2008 18:22 Please contact your local service provider completely installed VMware Verify if the HBA driver installed successfully using the following commands: • vmkload_mod -l This lists installed driver names. bfa_pkg should display with a “completely installed. Verify that an entry for bfa exists and that the ID loaded. Look for a bfa entry and related build number. • pkginfo -l bfa This displays details about installed Brocade HBA (bfa) drivers. Solaris Verify if the HBA driver installed successfully using the following commands. and whether the ID is loaded. use the following information to resolve the problem: - 16 Check for faulty fiber optic cable and cable connections. If bfa displays. then reboot the host system. the module has been loaded to the system. • cat /proc/vmware/version This displays the latest versions of installed drivers. Look for information as in the following example. Note that the VERSION may be different. disconnect all devices from the HBA.” status. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . R/O and R/W addresses. depending on the driver version you installed. The ARCH and DESC information may also be different. but the host system freezes or crashes during startup and does not boot.2 Host system freezes or crashes • # modprobe -l bfa This verifies that the module has loaded. • pkgchk -nv bfa This checks for and lists the installed HBA driver package files. If the HBA driver package is installed. • If the system does not freeze when rebooted and operates correctly. Host system freezes or crashes If the BIOS and the operating system recognize the HBA. depending on your host system platform. replace the HBA. It is independent of the HCM user interface (client application). NOTE Observe the LEDs by HBA ports. If the problem occurs in the new system. which is installed on the local or remote host. reinstall the HBA. - Reseat SFPs in the HBA. Determine whether the installed SFPs are faulty by observing LED operation by HBA ports. of the Brocade Fibre Channel HBA Installation and Reference Manual. • The agent not accepting connections on the expected port. Refer to the Brocade Fibre Channel HBA Administrator’s Guide for procedures to configure HBA ports. Try installing the HBA into another host system. then reinstall both. This occurs for one of the following reasons: • The agent is not running. Verify that data speed (1-8 Gbps) and connection topology (for example. HCM GUI fails to connect with HCM agent The HCM Agent installs with the HBA driver package on the host system..” when using HCM. Reseat the HBA. Also. If the system boots. this indicates that the client application cannot connect to the HCM Agent listening on the configured port . If the problem is resolved after replacement. This will help isolate any hang caused by switch and device interactions.Verify if the host system firmware supports PCIe specifications listed in the “Hardware and Software Compatibility” section. - Check for conflicts with attached devices. uninstall both the driver and HBA. note that the HBA only supports point-to-point connection topology. - On Windows systems. configuring settings manually on the HBA port and devices may allow connection. You can also verify SFP operation by replacing them with SFPs in known operating condition. - Remove the HBA and reboot the system.normally TCP port 34568. download a firmware update to support the HBA. • The agent is not listening on the expected port. If you receive an error such as “Failed to connect to agent on host. If not. the HBA may not be compatible with the original host system. point-to-point) for devices attached to the HBA are compatible with settings on the HBA port. Uninstall and reinstall the driver. “Introduction” chapter. original SFP is faulty. Illuminated LEDs indicate connection. If it freezes during hardware recognition. Refer to “LED Operation” in the Specifications chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. • If the system freezes perform the following tasks: . If the problem does not occur. link activity. Although auto may be set. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 17 . If all LEDs are flashing amber. uninstall and reinstall the driver.. If it freezes as the driver loads. and connection speed negotiated with the attached device. the SFP is invalid and may not be a required Brocade model. determine when the system freezes during the boot process.HCM GUI fails to connect with HCM agent 2 - Try rebooting the system without any connectivity to the switch. Refer to the section on modifying HCM agent operation.do If successful.https Use Windows Firewall and Advanced Service (WFAS) to open port 34568. 5. Use the following command to open port 34568: /usr/sbin/esxcfg-firewall-o 34568. but the agent will restart if the system reboots.out. the host’s firewall may be blocking TCP/IP port 34568. use the appropriate “start” command for your operating system which is also described in the Brocade Fibre Channel HBA Installation and Reference Manual. To restart. collect the following data and send to your Support representative for analysis: • Data collected from the previous step in SupportSaveController. If you receive a message that the hcmagent is stopped. 1. The data file name and location displays when the SupportSave feature runs. The localhost can be replaced with a different IP address. NOTE You can change the default communication port (34568) for the agent using procedures in the “Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. Refer to the section on modifying HCM agent operation.do (actually a zip format file) will contain the data from the HCM agent. If you are managing a VMware host system through HCM from a remote system.tcp. Verify that the agent is running by executing the appropriate “status” command for your operating system as described in the “Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. 18 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . which allows agent communication with HCM. Another command restarts the agent. Linux. wget --no-check-certificate https://admin:password@localhost:34568/JSONRPCServiceApp/ SupportSaveController. but the agent will not restart if the system reboots or the agent stops unexpectedly. the file SupportSaveController. NOTE This command is a single line. Note that one command described in the manual restarts the agent.2 HCM GUI fails to connect with HCM agent • Communications between the client and agent is blocked by a firewall preventing access to the port (usually only a consideration for remote HCM management). Execute the following command to connect to the HCM agent and force it to collect the HBA driver supportsave data. Confirm the HCM agent is responding to requests using the expected user password. 2. Solaris. If HCM is still unable to connect to the HCM agent after using the preceding steps. 4. restarting the agent should resolve the problem. 3. • Data from the HCM application SupportSave feature. and VMware systems Perform the following tasks to isolate and resolve the problem.do. Select Tools > SupportSave to generate a supportsave file. 0. use the appropriate “start” command for your operating system which is also described in the Brocade Fibre Channel HBA Installation and Reference Manual.tgz /opt/hbaagent Output collects to hbaagentfiles. Note that this identifier may be different on your system. Verify that the agent is running by executing the appropriate “status” command for your operating system described in the “Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual.0. 2.2 HCM GUI fails to connect with HCM agent • HBA agent files on the HBA host (where the HCM agent is installed). Use Windows Firewall and Advanced Service (WFAS) to open port 34568. restarting the agent should resolve the problem. Collect these files using the following command: tar cvfz hbaagentfiles. If you are managing a Windows 2008 host system through HCM from a remote system.tgz. Enter the following command to confirm that the process identifier bound to TCP port 34568 is for the hcmagent. Another command restarts the agent.exe process: tasklist /svc | findstr 1960 The following should display if the identifier from step 3 is bound to TCP port 34568: hcmagent. 1.0:34568 0. NOTE You can change the default communication port (34568) for the agent using procedures in the “Installation” chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. If the HCM agent starts. • Data collected on the HBA host from the bfa_supportsave feature using the following command: bfa_supportsave Output collects to a file and location specified when the SupportSave feature runs. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 19 . Refer to the section on modifying HCM agent operation. Refer to the section on modifying HCM agent operation.exe 1960 hcmagent 5. the host’s firewall may be blocking TCP/IP port 34568. Windows systems Perform the following tasks to isolate and resolve the problem. To restart.0. verify which TCP port the agent is listening on by executing the following command at the Windows command prompt: netstat -nao | findstr 34568 Output similar to the following should display. TCP 0.0. If you receive a message that the hcmagent is stopped. 3. Note that one command described in the manual restarts the agent. but the agent will restart if the system reboots. but the agent will not restart if the system reboots or the agent stops unexpectedly.0:0 LISTENING 1960 1960 in the last column is the process identifier for the Windows process listening on the TCP port. 4. right-click this directory and select Send To > Compressed (zipped). Common link problems can be caused by the following: • Damaged fiber optic cables. If LEDs do not illuminate to indicate an active link. execute the following command: bfa_supportsave Data collected by this command saves to subdirectory named “bfa_ss_out. Select Tools > SupportSave to generate a supportsave file. For details on HBA LED operation. For additional diagnostics commands. use appropriate Fabric OS and HBA diagnostic commands and HCM options in Table 4 on page 21. “Tools for Collecting Data” for HCM and BCU commands and the Fabric OS Administrators Guide for Fabric OS commands. NOTE Also verify LED operation on switch port that is connected to an HBA port. but you get the same “Failed to connect to agent on host. and build date. Select Help > About in HCM to display the version. refer to Chapter 3. To collect these files.” In Windows explorer. Messages also display that provide the location of the zip file created. a zip file is created in the following location if installed the application in C:\Program Files\BROCADE\FCHBA: C:\Program Files\BROCADE\FCHBA\client\data\localhost\supportsave The zip file will have a name similar to the following: SupportSave_Basic_2008723_0_50_57. For example.) 20 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . • Build information for the HCM application. • Support data from the HCM application SupportSave feature. collect the following data. • Files from the output directory created after you execute the bfa_supportsave feature. By default. This creates a zip file that you can send to your Support representative. all LEDS flashing amber for a port indicates that an invalid non-Brocade SFP may be installed. If the hcmagent is running and listening on port 34568 and there are no firewall issues (as explained in step 5). refer to the specifications chapter in the Brocade HBA Installation and Reference Manual. Send this data to your Support representative for analysis: • Copies of output from the commands in step 3 and step 4. (Note that damaged cables can also cause errors and invalid data on links.2 Verifying Fibre Channel links 6. Refer to the switch Hardware Reference Manual to analyze LED meaning... a message displays an error (Agent Support Save could not be collected) and explains that only a basic collection is possible. LED operation other than expected or LEDs may indicate link problems. build identification.zip Verifying Fibre Channel links Check for link problems by observing LED operation for HBA ports. If HCM cannot connect to the agent." error when using HCM. as well as Fabric OS commands that you can use to determine link status. You can run the BCU port --stats command to display port statistics. but is compatible with switch or vice versa. Port commands.possibly the SFP is not compatible with HBA. Try replacing the SFP. Table 4 lists HCM options and BCU commands. Refer to Fibre Optic Cable specifications in the Brocade Fibre Channel HBA Installation and Reference Manual. TABLE 4 Tools to determine link status Application Tool HCM • • • BCU Switch Fabric OS Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 • • • • • • • • • • • References Port Statistics Loopback and PCI loopback test Fibre Channel ping. • SCSI retries and timeouts determine communication between HBA and storage. “Tools for Collecting Data” switchShow portShow portStatsShow portErrShow fcpProbeShow fPortTest • • • Chapter 3. Packets can drop because of SFP issues on HBA or switch . If the errors or invalid data on the link still indicate a cable problem. replace the cable with a cable of known quality. “Tools for Collecting Data” Fabric OS Administrator’s Guide Fabric OS Troubleshooting and Diagnostics Guide 21 . if convenient. such as port --stats. such as error and dropped frames. the SFP may be faulty. and trace route tests Link Beaconing Port Properties SFP information Chapter 3. echo. Verify if an SFP is the problem by connecting a different link to the HBA port or. Dropped packets cause timeouts. Chapter 3. “Tools for Collecting Data” BCU fcdiag and diag commands.Verifying Fibre Channel links 2 • Fiber optic cables may not be rated or compatible with HBA port speeds. • Faulty switch or HBA SFPs. 2 22 Verifying Fibre Channel links Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Data to provide support . . . . . . . . . . . . . and storage systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Fabric OS Command Reference Manual Fabric OS diagnostic and monitoring commands. . . . . . . refer to the following publications: • Brocade Fibre Channel HBA Administrator’s Guide The following chapters in this guide cover HBA HCM and BCU monitoring and diagnostics tools: - Monitoring Diagnostics and Troubleshooting Command Line Utility (BCU) • Fabric OS Troubleshooting and Diagnostics Guide This guide provides detailed information on collecting troubleshooting information and isolating general SAN problems between the switch. . . . . . . . 23 24 24 25 28 30 34 40 45 46 48 48 50 For detailed information This chapter provides basic instruction on tools useful for gathering information to isolate HBA problems. . . . . . . . . . . . . . . . For more detailed information on using these tools. . . . . Host system diagnostics. . . . . . . • Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Collecting port data . . .Chapter 3 Tools for Collecting Data In this chapter • For detailed information . . • Collecting data using BCU commands and HCM . . . . . . . . . . . . . . . . host systems. . . . and system monitoring tools. • Collecting data using host system commands . . . • QoS and target rate limiting settings . . . . . . . . . . . . . . . . . . • Statistics. • Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Event logs. . . . . . . . . . . . . . . . • Persistent binding . Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Your host system’s operating system user and administrator’s guides. . . . . . . . . . . logs. . . . . . . . . . . . . . . . . . . • Collecting SFP data . . . . . . . . . . . . . . . . . . . . . . . . . • Authentication settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . For details on these commands. TABLE 5 Host system data collection commands Task Linux Windows VMware Solaris List PCI devices lspci -vv In Windows registry location HKEY_LOCAL_MACHINE \SYSTEM\CurrentContro lSet\Enum\PCI devcon find pci\* lspci -vv. sar List for driver modules lsmod driverquery vmkload_mod -l modinfo To check for Brocade Fibre Channel adapter (BFA) driver module lsmod | grep bfa driverquery /v | findstr bfad vmkload_mod -l modinfo | grep bfa 24 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . as well as output collected from the following HCM and BCU tools: • • • • • • Support Save Diagnostics Port logs Port statistics and properties HBA properties Host operating system commands Collecting data using host system commands The following table describes commands common to each supported operating system that you can use to gather information for troubleshooting problems. VM Performance: esxtop [first type 'v'. top Windows Task Manager. top ps -efl. refer to your system’s online help and documentation. tasklist. mpstat. vmstat.exe vmstat. vmstat. NOTE Output from all of these commands is captured using the Support Save feature. Disk Performance: esxtop [type 'v' then 'd']. 'e' then enter vm# in the list down].exe Click the plus sign(+) next to Components to view hardware details. iostat -nx 1 5. top Memory usage top. provide a detailed description of the problem. vmstat -m vmstat -s Performance monitoring iostat. esxcfg-info -a prtdiag -v. prtconf -pv Process information ps -efl. perfmon.3 Data to provide support Data to provide support When problems occur requiring support assistance. prtconf -pv List installed HW details lsdev msinfo32.exe ps -efl. esxcfg-info -w prtdiag -v. vmstat -m Windows Task Manager. tasklist.exe top. sar Windows Task Manager. /var/log/vmkernel*.conf • • (Release 1. Use one of the following options to launch this feature: • In HCM. WWN. Using Support Save The Support Save feature is an important tool for collecting debug information from the driver. and other details for troubleshooting using BCU commands. and firmware. • Through your internet browser (Internet Explorer 6 or later or Firefox 2. /var/log/vmkwarning*.1 and later) /devices/pci*/pci*/ bfa@0:devctl Collecting data using BCU commands and HCM You can collect a variety of information on installed Brocade HBAs. cat /etc/release BFA configuration file location /etc/bfa. You can save this information to the local file system and send to support personnel for further investigation. configuration data. PCI data.0) /devices/pci*/pci*/ fibre-channel@0:dev ctl. /var/adm/message* Show OS distribution info (SuSE) cat /etc/SuSE-release. you can collect bfa_supportsave output if you do not have root access.( RedHat) cat /etc/redhat-release systeminfo.conf Windows Registry (HKEY_LOCAL_MACHINE \SYSTEM\CurrentContro lSet\Services\bfad\Para meters\Device). /var/log/message* System Category in Windows Event Viewer (eventvwr. port speed. flash status.0 or later). • Through BCU.exe cat /etc/vmware-release uname -a. The Support Save feature saves the following information: Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 25 . do not have access to file transfer methods such as FTP and SCP.conf /etc/bfa. operational status. (Release 1. and host operating system commands. or do not have access to the Host Configuration Manager (HCM). launch Support Save through the Tools menu. refer to “Host system logs” on page 30. such as firmware version installed. internal libraries.Collecting data using BCU commands and HCM TABLE 5 3 Host system data collection commands Task Linux Windows VMware Solaris System log message location dmesg.exe) /var/log/message* . HBA Flash BFA device file location /dev/bfa* Windows Registry (HKEY_LOCAL_MACHINE \HARDWARE\DEVICEMA P\Scsi\Scsi Port x) NOTE: For more information. /dev/bfa* /kernel/drv/bfa./p roc/vmware/log dmesg. enter the bfa_supportsave command. HCM menu options. NOTE A bfa_supportsave collection can also occur automatically for a port crash event. 3 Collecting data using BCU commands and HCM • • • • • • • • • • • • • • • • • • • HBA model and serial number. HBA configuration data. memory. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . refer to the Brocade Fibre Channel HBA Administrator’s Guide. Initiating Support Save through HCM Initiate Support Save by selecting Tools > Support Save. Host model and hardware revision. All support information. HBA firmware version. All operating system and HBA information needed to diagnose field issues Information about all HBAs in the system. If not specified. If you are initiating Support Save from a remote management station and receive a warning message that support files and Agent logs could not be collected. Initiating Support Save through BCU commands Use the bfa_supportsave command to Initiate Support Save through the BCU: bfa_supportsave {output_dir] where: [output_dir] 26 An optional parameter that specifies the directory where you want output saved. Windows System Event log . Syslog message logs.xml file Vital CPU. output is saved as a directory in the current working directory as bfa_ss_out. Messages display during the Support Save operation that provide the location of the directory where data is saved. For more information and additional options for using this feature. the HCM Agent is unavailable on the remote host. HCM GUI-related engineering logs Events HBA configuration data Environment information Data. configuration) Driver logs (bfa_supportsave output) Core files NOTE Master and Application logs are saved when Support Save is initiated through HCM. but not through BCU.evt file. Firmware and driver traces. Select Tools > Backup to backup data and configuration files manually. network resources HCM Agent (logs. For more information on these logs. refer to the Host Connectivity Manager (HCM) Administrator’s Guide. and browser applications of bfa_supportsave: • BCU Collects only driver-related logs and configuration files. For more information on the bfa_supportsave feature.do file. 1. The File Download dialog box displays. refer to “HCM logs” on page 32. prompting you to save the supportSaveController. Initiating Support Save through the internet browser Initiate bfa_supportsave through an internet browser. Use the current user name and password if they have changed from the default. and HCM Agent logs and configuration files. support save data is collected at a system-wide level. driver-related. bfa_supportsave collection on a port crash event If the port crashes and triggers a port crash event. • Browser Collects driver-related and HCM Agent logs and configuration files. An Application Log message is generated with the following message: Port Crash Support Save Completed Port crash events have a CRITICAL severity and you can view the details in the Master Log and Application Log tables in HCM. Log in using the factory default user name (admin) and password (password). • HCM Collects. HCM. The directory name specifies the date when the file was saved.Collecting data using BCU commands and HCM 3 Messages display as the system gathers information. Initiating Support Save through a port crash event If the port crashes and triggers a port crash event. 2. When complete. BCU. An Application Log message is generated with the following message: Port Crash Support Save Completed Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 27 .do where localhost is the IP address of the server from which you want to collect the bfa_supportsave information. 3. Open an Internet browser and type the following URL: https://localhost:34568/JSONRPCServiceApp/SupportSaveController. Support Save differences Following are differences in data collection for the HCM. an output file and directory display. Click Save and navigate to the location where you want to save the bfa_supportsave file. support save data is collected at a system-wide level. such as the current version of the HBA (chip revision) and driver (fw version). Collecting data using BCU commands Use the BCU adapter command to list and query available HBAs seen by the driver. • cfgShow Displays zone configuration information for the switch. refer to the Fabric OS Command Reference Manual. subsystem ID. current number of lanes. You can use command output to verify target ports (by port WWN) and LUNs that are intended to be accessible from the HBA. disabled or enabled status. The bcu adapter --list command lists all HBAs on the system with a brief summary of information such as model number. pwwn. number of ports. serial number. For details on using these commands. This panel also displays PCI information for the HBA. For each adapter in the system. serial number. device ID. 1. Collecting data using Fabric OS commands Use the following Fabric OS commands on attached Brocade switches to gather information and help isolate connectivity and other problems between the HBA. 2. adapter –-query <ad_id> where: ad_id ID of the adapter (HBA) for which you want to query. installed driver name. such as vendor ID. and BIOS version. and HBA number. firmware version. and flash information (such as firmware version). and PCI generation. a brief summary line is displays. The adapter --query command displays adapter information. Enter the following command: adapter --list where: list Lists all adapters in the system. temperature. driver version. model information. 28 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . hardware path. PCI information. • authUtil Use this command to display and set local switch authentication parameters. nwwn. and storage ports. Select an HBA in the device tree 3. Click the Properties tab in the right pane. switch.3 Collecting data using Fabric OS commands Port crash events have a CRITICAL severity and you can view the details in the Master Log and Application Log tables in HCM. Launch HCM. such as its WWN. HBA data collection using HCM Use the HBA Properties panel in HCM to display information about the HBA. maximum port speed. • SecAuthSecret Use this command to manage the DH-CHAP shared secret key database used for authentication. • switchShow Use this command to display switch and port information. sets. This includes information such as the device PID.Collecting data using Fabric OS commands 3 • fcpProbeShow Use this command to display the Fibre Channel Protocol daemon (FCPd) device probing information for the devices attached to a specified F_Port or FL_Port. and state of each port on the switch. • nsShow Use this command to display local NS information about all devices connected to a specific switch. device type. ID. Use this information to determine the fabric port WWN and PID connected to an HBA port. and the port and node WWN. speed. Output may vary depending on the switch model. and removes shared secret key information from the databases • sfpShow Use this command to display detailed information about specific SFPs installed in a switch. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 29 . • portErrShow Use this command to display an error summary for all switch ports. • portLogShowPort Use this command to display the port log for a specified switch port. • zoneshow Use this command without parameters to display all zone configuration information (both defined and enabled). • portShow Use this command to display information and status of a specified switch port. • portLogShow Use this command to display the port log for ports on a switch. and WWN.This command displays. Also display topology. operating state. • portStatsShow Use this command to display hardware statistics counters for a specific switch port. including the speed. This information includes the number of successful logins and SCSI INQUIRY commands sent over this port and a list of the attached devices. type. • portPerfShow Use this command to display throughput information for all ports on the switch. These messages are captured in logs. Monitoring events in these logs allows early fault detection and isolation on a specific HBA. and in some cases. These messages contain the following information: • • • • • • Message ID Message text Message arguments Severity level Cause Recommended action Brocade HBA event message files are installed in the HBA driver installation directory for each supported operating system. and how to view them.zip loaded to your driver installation directory. The following types of logs are available: • Host operating system logs • HCM logs Host system logs Brocade HBA event messages are captured in host system log files.exe) Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Unzip this file to obtain all message catalog files. TABLE 7 30 System Event Logs Operating System Log Name Location Viewing Message Log Solaris Syslog /var/adm/messages dmesg command Windows Event Log Not applicable System category in Event Viewer (eventvwr.3 Event logs Event logs Event messages that occur during HBA and driver operation are important tools for isolating and resolving problems. Table 7 describes the logs for each supported operating system. where the logs are stored. cause and recommended actions. time and date of the event. All messages related to the Brocade HBA are identified in these logs by BFAD or bfa (Brocade Fabric Adapter). Messages provide descriptions of the event. Table 8 provides the location of the message files for each system. severity. TABLE 6 Message catalog location Operating System Catalog Location Linux /opt/bfa VMware /opt/bfa Solaris /opt/bfa Windows aen. For Guest system. Unzip this file to obtain all message catalog files.html Logical port events.Event logs TABLE 7 System Event Logs Operating System Log Name Location Viewing Message Log Linux Messages Log /var/log/message dmesg command 1VmWare Messages Log /var/log/message* . bfal_aen_audit. You can view all event messages that can display for a Brocade HBA by viewing HTML files that are loaded to your system as the driver package installs. refer to information in Windows or Linux. such as adapter added or removed. has lost connection with initiator. online. /var/log/vmkernel*. has lost fabric connectivity.html Adapter events. SFP removed from port. TABLE 9 Event message files Event Catalog File Content bfal_aen_adapter. 3 For ESX Server Console operating system.html Initiator-target nexus events. or authentication successful or unsuccessful for base port. bfal_aen_ioc. such as target is online for initiator. These files contain all message information that can display on system logs for the Brocade HBA. TABLE 8 Message file location Operating System File Location Linux /opt/bfa VMware /opt/bfa Solaris /opt/bfa Windows aen. bfal_aen_itnim. such as port online or offline. /var/log/vmkwarning*. View these files through your internet browser. offline. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 31 .html Audit events. bfal_aen_port. or is disconnected by initiator. such as a logical port is created. Table 9 lists the message file names and content. such as authentication enabled or disabled for base port.html IO controller (IOC) events. deleted.html Physical port events. or the port’s login has failed.zip loaded to your driver installation directory. bfal_aen_lport. Table 8 provides the location of the Brocade HBA message files for each supported system. such as IOC heartbeat status and IOC enabled or disabled./proc/v mware/log dmesg command 1. • The Master Log displays informational and error messages during HBA operation. port. HCM logs You can view data about HBA operation through the following HCM logs.3 Event logs TABLE 9 Event message files Event Catalog File Content bfal_aen_rport. Debug. itnim. 32 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . This log contains the severity level. the function that reported the event (such as a specific HBA port or remote target port). hba_error_codes. event description. audit. date and time of event.html Remote port (RPORT) events. such as a remote port is online. and Fatal. Master Log The Master Log displays event information in seven fields: • Sr No. Error. Adjust the logging level. or the types of messages logged to your system log that relate to HBA driver operation. Click the Master Log or Application Log to toggle between logs. or the QoS priority or flow ID changed. • Adjust logging level using this BCU command. Sequence number that event occurred in ascending order. Both of these logs display on the bottom of the HCM main window. and other information. WWN of device where event occurs. offline.doc List of error codes and meanings for adapter. Info. Select the a level on the Agent Communication Log and HCM Debug Log lists. Values are Trace. or disconnected. R_Port. Warning. refer to the Brocade Fibre Channel HBA Administrator’s Guide. L_Port. using the following HCM options and BCU commands. and IOC events. NOTE For details on adjusting logging levels. log -–level <port_id> [<Trace | Debug | Info | Warn| Error | Fatal>] • Adjust logging level using the HCM. Select Configure > HCM Logging Levels. • The Application Log displays informational and error messages related to HBA discovery or HCM application issues. click the Filter button in the Master Log section of the main HCM screen. category. Severity of the message. Examples of these could be timeouts on certain operations. Port . • Description.Events relating to a specific logical port. - Information-level messages report the current non-error status of the system components. Application Log The Application Log displays all application-related informational and error messages. The agent IP address. initiator target connectivity. Categories define the component where events occur: - Adapter . LPORT . - Critical-level messages indicate that the software has detected serious problems that will eventually cause a partial or complete failure of a subsystem if not corrected immediately. Description of the message. or failure to perform a requested operation. invalid parameters. - Minor messages highlight a current operating condition that should be checked or it might lead to a failure. ItNIM . To display the Master Log Filter dialog box. the online and offline status of a fabric port.Events relating to an initiator-target nexus. Date. Select areas that you want to filter and click OK. - Audit . or critical). RPORT .Driver and firmware events involving the I/O controller on the HBA.Event logs 3 • Severity Event severity level (informational. minor. • • • • Date and time the message occurred. • WWN World Wide Name of HBA where event occurred. and loss of connectivity.Events relating to a specific port on the HBA. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 33 . Examples of these include end to end target discovery. for example. Examples of these could be a power supply failure or rise in temperature - Major messages represent conditions that do not impact overall system functionality significantly. as well as the following attributes.Events relating to a specific remote initiator or target port. IOC . • Subcategory Subcategory of main category. major. You can block events from display in the Master Log by severity. and WWN of HBA.Audit events. and Time Brief description of event and date and time when event occurred.Events relating to the HBA (Adapter). • Category The category or type of event. failures of certain operations after retries. and patch panel connections. From most to least severity. 34 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . error.” HBA events are logged under the HCM_Event category. For procedures to configure syslog destinations. and information. Driver errors are critical. received loop initialization primitive (LIP) event counts. loss of synch and loss of signal errors indicate a physical link problem. such as transmitted and received frames and words. For more detail. NT events are error. You can display statistics for the following: • • • • • • • • • HBA ports IO controller Virtual ports (vport) Logical ports (lport) Remote ports (rport) FCP initiator mode Fabric (BCU only) Targets Security authentication This section provides an overview of these statistics and how to access them. refer to the Brocade Fibre Channel HBA Administrator’s Guide. loss of synchronization. error frames received. Windows Event Log support HBA events are logged in the Windows Event Log by the HCM agent. SFPs on the HBA or switch. link failure and invalid CRS counts. Use these statistics to isolate link and frame errors. For example. Use these statistics to monitor HBA performance and traffic between the HBA and LUNs and isolate areas that impact performance and device login. warning. Event messages appear with the date and time the event occurred and the source as “hcmAgent. warning. Port statistics Use BCU and HCM to display a variety of port statistics. check cables. Statistics You can access a variety of statistics using BCU commands and HCM. To resolve these problems. refer to the Brocade Fibre Channel HBA Administrator’s Guide. and end of frame (EOF) errors. and info. Event types display with each event to note severity level. These events will display in the operating system logs for systems such as Solaris and Linux.3 Statistics Syslog support You can configure the HCM agent to forward events to a maximum of three syslog destinations using the Syslog option on the HCM Configure menu. Click Monitor > Statistics > Port Statistics. 2.Statistics 3 Displaying statistics through BCU Use the port --stats BCU command to display statistics for a specified HBA port. IOC statistics Use BCU and HCM to display port-level statistics for the I/O controller through the BCU and HCM. Displaying statistics through HCM Use the Port Statistics dialog box to monitor a variety of port data. 3. 1. Select the base HBA port from the device tree window. Launch the HCM. 3. Displaying IOC statistics through HCM Use the IOC Statistics dialog box to monitor a variety of port data. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 35 . The I/O controller refers to the firmware entity controlling the port. 1. Select the base HBA port from the device tree window. The following types of IOC statistics are displayed: • • • • • • • IOC driver IOC firmware Firmware IO Firmware port FPG Firmware port PHYSM Firmware port LKSM Firmware port SNSM Displaying IOC statistics through BCU Use the ioc -–stats command to display IOC statistics. Launch the HCM. port --stats <port_id> where: port_id ID of the port for which you want to display statistics. ioc -–stats <ioc_id> where: ioc_id ID of the IOC controller for which you want to display statistics. 2. Click Monitor > Statistics > IOC Statistics. 1. fabric -–stats <port_id> where: port_id ID of the HBA port for which you want to display statistics. Displaying fabric statistics through HCM Use the Fabric Statistics dialog box to monitor a variety of port data. Select the base HBA port from the device tree window. Following are two examples of how to use these statistics for troubleshooting: • If the HBA is not showing in the fabric. Use HCM and BCU to display statistics for the following: • • • • • • Port login (PLOGI) activity Authentication and discovery (ADISC) activity Logout (LOGO) activity RCSNs received Process logins (PRLI) received Hardware abstraction layer (HAL) activity As an example of using these statistics for troubleshooting. Verify the rport online and rport offline statistics again. Slow fabric performance or hosts unable to address storage could also be seen.3 Statistics Fabric statistics Use BCU and HCM to display statistics for fabric login (FLOGI) activity and fabric offlines and onlines detected by the port. Remote port statistics Remote port statistics can help isolate end-to-end login problems. Use these statistics to help isolate fabric login problems. the switch or fabric may not be ready to respond. check the FLOGI sent and FLOLGI accept statistics. this could indicate a problem in the fabric or a protocol issue between the HBA and fabric. If the problem persists. 2. if the host cannot see the target. Launch the HCM. 3. you can verify that the remote port (rport) is reporting itself online by comparing the rport offline and rport offline statistics. If the counts do not match. Displaying fabric statistics through BCU Use the fabric --stats command to display fabric statistics. The rport online counter should be one greater than the rport offline counter. This is normal as long as it does not persist. Click Monitor > Statistics > Fabric Statistics. If not. 36 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . this may indicate a serious fabric problem. clear the counters and retry connecting to the remote port. • If fabric offline counts increase and fabric maintenance is not being done. Displaying FCIP initiator mode statistics through BCU Use the fcpim --stats command to display FCIP initiator mode statistics. the base port is used. rpwwn Displays the remote port’s port world wide name. Launch the HCM. Click Monitor > Statistics > Remote Port Statistics > Target Statistics. FCIP initiator mode statistics Use HCM and BCU to display Fibre Channel Protocol Initiator Mode (FCP IM) statistics such as the number of online and offline remote ports. Displaying remote port statistics (BCU) Use the rport --stats command to display remote port statistics: rport –-stats <port_id> <rpwwn> [-l lpwwn] where: port_id ID of the port for which you want to display rport statistics. Select the base HBA port from the device tree window. If the -l lpwwn argument is not specified. and HAL statistics.Statistics 3 Displaying target statistics (HCM) Use the Target Statistics dialog box to display target statistics. 1. This is an optional argument. This is an optional argument. 3. If the -l lpwwn argument is not specified. process login (PRLI) activity. lpwwn Displays the logical port world wide name. Displaying FCIP initiator mode statistics through HCM Display statistics by selecting Monitor > Statistics > Remote port statistics > FCP IM Statistics. lpwwn Logical port world wide name. 2. Logical port statistics Use HCM and BCU to display logical port statistics for the following: • Name server (NS) port logins (plogin) activity • Register symbolic port name (RSPN_ID) identifier activity • Register FC4 type identifier (RFT_ID) activity Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 37 . fcpim --stats <port_id> <rpwwn [-l <lpwwn>] where: port_id ID of the port for which you want to display statistics. rpwwn Remote port world wide name. the base port is used. Displaying logical port statistics through HCM Display logical port statistics by selecting Monitor > Statistics > Logical Port Statistics OR Right-click a logical port from the device tree and select Logical Port Statistics. number of fabrics online and offline. • If name server get all port ID response NS GID_FT rsp). Following are examples of what to check if virtual devices are not listed in the name server: • If FDISK sent and FDISK accept statistics do not match. This is normal as long as it does not persist. Following are examples of how these statistics indicate these problems: • If name server port login (NS PLOGI) error rejects and unknown name server port login response (NS login unknown rsp) errors increase. the fabric or switch may not be ready. NPIV support. rpwwn Remote port world wide name for which you want to display statistics. 38 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . the base port is used. Note that in this case FDISK retries also increase. there may be a problem in the fabric or a protocol issue between the HBA and fabric. and fabric cleanups. the HBA has a problem registering with the name server. lport –-stats <port_id> [-l lpwwn] where: port_id ID of the port for which you want to display statistics. logouts (LOGO) activity. lpwwn Logical port world wide name for which you want to display statistics. If the -l lpwwn argument is not specified. This is an optional argument. rejects (NS_GID FT rejects). Virtual port statistics Use HCM and BCU to display logical port statistics for fabric discovery (FDISK) activity. • If name server register symbolic port name identifier (NS RSPN_ID) or name server register symbolic port name identifier response (NS RFT_ID rsp) errors or rejects (NS RFT_ID rejects) are increasing. the HBA has a problem querying the name server for available storage. or unknown responses (NS_GID FT unknown rsp) are increasing.3 Statistics • • • • Register FC4 type identifier (RFT_ID) activity Get all port ID requests for a given FC4 type (NS_GID_FT) activity Retries Timeouts Use these statistics to help determine if the HBA is not registering with the name server or cannot access storage. Displaying logical port statistics through BCU Use the lport --stats command to display logical port statistics. Use these statistics to isolate NPIV login problems. then the HBA most likely cannot log in to the name server. If it does persist. vport --stats <port_id> vpwwn where: port_id ID of the port for which you want to display rport statistics. the information provided is for the base vport. Select the base HBA port from the device tree window. exchange link parameter (ELP) activity. and received QOS registered state change notifications (RSCNs). vport --stats <port_id> vpwwn where: port_id ID of the port for which you want to display rport statistics. 3. If no part WWN is specified. Displaying virtual port statistics through BCU Use the vport --stats command to display statistics. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 39 . Displaying QoS statistics through BCU Use the qos --stats command to display remote port statistics. Displaying virtual port statistics through HCM Display statistics by selecting Monitor > Statistics > Virtual Port Statistics OR Right-click a virtual port on the device tree and select Virtual Port Statistics. 2. You can display statistics for fabric login (FLOGI) activity. Quality of service (QoS) statistics Use HCM and BCU to display QoS statistics for individual ports. vpwwn Displays the statistics for the virtual port by its WWN. Launch the HCM. 1. Displaying QoS statistics through HCM Use the QOS Statistics dialog box to display QoS statistics.Statistics 3 • Check the No NPIV support statistics to verify that NPIV is supported and enabled on the switch. Click Monitor > Remote Port Statistics > QOS Statistics. available through BCU commands and HCM. When you initiate link beaconing. Use the fcdiag --linkbeacon command to enable link beaconing. Toggle beaconing on and off and set beaconing duration using the BCU or HCM. The remote port sends a command back to the originating port. on | off Toggle on or off. 40 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . or both. then click the Beacon Link check box. Initiate link beaconing to flash the LEDs on a specific HBA port and the LEDs on a connected switch port to verify the connection between HBA and switch. you can specify duration. Enabling beaconing through HCM Enable link beaconing by selecting Configure > Beacon. duration Length of time between blinks. the Beacon Port check box. If turned on. Beaconing Initiate beaconing on a specific HBA port to flash the port LEDs and make it easier to locate the HBA in an equipment room. fcdiag --linkbeacon <portid> {on | off} [<duration>] where: port_id ID of the port for which you want to run a link beacon test. that port’s LEDs flash.3 Diagnostics Diagnostics Diagnostics. When that port receives this command. duration Length of time between blinks. NOTE To initiate link beaconing. this feature must be available on the connected switch. the port’s LEDs flash. Enabling beaconing through BCU Use the diag --beacon command to enable beaconing for a specific HBA port. commands are sent to the other side of the link. diag --beacon <port_id> <on | off> [<duration>] where: port_id ID of the port for which you want to enable beaconing. All of these diagnostics can be used while the system is running. evaluate the integrity of HBA hardware and end-to-end connectivity in the fabric. When the remote port receives these commands. Default value is A5A5A5A5. duration Length of time between blinks. external. -p pattern Hex number. Default value is A5A5A5A5 5. a loopback connector is required for the port. serdes speed For 8 Gbps HBA.967. frame count Integer from 0. Click the Port Tests tab. Random data is sent to the HBA port. Default value is 8192 Data Pattern: 8 Hex number. 1. bcu diag --loopback <port_id> [-t <loopback_type>][<duration>][-s <speed>] [-c <frame_count>] [-p <pattern>] where: port_id ID of the port that you want to run the test. Loopback tests require that you disable the port.4. 4G and 8G. 2 or 4. loopback type internal. You can modify the following test parameters • • • • Subtest: three options: Internal. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 41 .4. The returned data is validated to determine port operation. • External For this test. Select Configure > Diagnostics 3.967.Diagnostics 3 Loopback tests Use the BCU or the HCM to perform a loopback test for a specific port. and External. 4. 2. For 4 Gbps HBA.295. Errors may indicate a failed port. The returned data is validated to determine port operation. 4. Performing loopback tests through BCU Use the diag --loopback BCU command test to verify port function through a loopback test.294. Default is 8192. Serdes. For 4G HBA: 1G. Click Start. 2G and 4G Frame Count: Integer from 0. then returned without transmitting through the port. Launch the HCM. this is 1. The data transmits from the port then returns.295.294. Link Speed: For 8G HBA: 2G. Errors may indicate an failed port. or 8. this is 2. Select Loopback Test. The following loopback tests are available: • Internal Random data is sent to the HBA port. Performing loopback tests through HCM Use the Port Tests tab on the Diagnostics dialog box to perform a loopback test. Performing loopback tests through HCM Use the Port Tests dialog box to perform a PCI loopback test. Click the Port Tests tab. The returned data is validated to determine PCI operation.3 Diagnostics PCI loopback test Use the BCU or HCM to perform a PCI loopback test for a specific port. In this test. You can modify the following parameters • Frame count: Integer • Data pattern: Hex number. Launch the HCM. Memory test Use the BCU or the HCM to perform a memory test for the HBA. bcu diag --pciloopback <port_id> [-p <pattern>] [-c <frame_count>] where: port_id ID of the port from which you want to run the test. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . a data pattern is sent from the host to HBA firmware through the PCI bus. diag --memtest <ad_id> where: ad_id 42 Serial number of adapter. NOTE Performing the Memory test disables the HBA. frame count Integer. 2. Select Configure > Diagnostics 3. Click Start. pattern Hex number. Performing a memory test through BCU Use the diag --memtest command to test the adapter’s memory blocks. Select PCI Loopback Test. 4. • Test cycle: The number should be positive and the default is 1. 1. Performing loopback tests through BCU Use the diag --pciloopback BCU command to perform a PCI loopback test. 5. HBA temperature Use the BCU diag --tempshow command to read the adapter’s temperature sensor registers. Ping end points through HCM Use the Protocol tab on the Diagnostics dialog box to test the connection to Fibre Channel end points. Select the HBA port and target that you wish to ping. You can specify the test cycle using a positive number. Click the Port Tests tab. Select Configure > Diagnostics 3. 4. Click Start. diag --tempshow <ad_id> ad_id . Click the Protocol Tests tab. 6. 6.Serial number of adapter. Ping end points Use the BCU and HCM to ping a Fibre Channel end point from an HBA port to determine the basic connectivity to the remote port and monitor network latency. Enter a test cycle if applicable. lpwwn Logical port WWN. Launch the HCM. 4. Select FC Ping Test. Ping end points through BCU Use the fcdiag --fcping BCU command to test the connection to a Fibre Channel end point. rpwwn Remote port WWN that you want to ping. fcdiag -–fcping <port_id> <rpwwn> [-l lpwwn] where: port_id ID of the HBA port from which you want to ping the remote port. 0 indicates the base port. 5. 2. 1. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 43 . 5.Diagnostics 3 Performing a memory test through HCM Use the Port Tests dialog box to perform a memory test. Select Memory Test. 1. Select Configure > Diagnostics 3. 2. Launch the HCM. Select the HBA port and target for which you wish to trace the route. 7. Select Configure > Diagnostics 3. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Click Start. This sends an ECHO command and response sequence between the HBA port and target port to verify connection with the target. fcdiag –-fcecho <port_id> <rpwwn> [-l lpwwn] where: 44 port_id ID of the port for which you want to perform the test. Perform echo test through BCU Use the fcdiag --fcecho BCU command to initiate an echo test between the HBA and remote port. 0 indicates the base port Trace the route through HCM Use the FC Trace Route feature to trace the route between the HBA port and a target port: 1. 4. Click Start.3 Diagnostics 7. 2. 0 indicates the base port. Select FC Trace Route. Trace route Use the BCU and HCM to report the SAN path between the HBA and remote end point. 5. rpwwn Remote port WWN that you want to ping. lpwwn Logical port WWN. Echo test Use the BCU and HCM to initiate an echo test between the HBA port and a Fibre Channel end point. Enter a test cycle if desired. rpwwn Remote port WWN that you want to ping. Trace the route through BCU Use the fcdiag --traceroute BCU command to trace the route between end points. 6. otherwise. Click the Protocol Tests tab. Launch the HCM. lpwwn Logical port WWN. fcdiag -–fctraceroute <port_id> <rpwwn> [-l lpwwn] where: port_id ID of the port from which you want to trace the route. Initiating diagnostics through BCU Use the diag --sfpshow BCU command to display detailed attributes for a specific SFP transceiver. 2. as well as details on the extended link. lpwwn Logical port WWN. diag --sfpshow <port_id> where: port_id ID of the port for which you want to display SFP attributes. on small form factor pluggable (SFP) transceivers. Click Start. 5. otherwise. 0 indicates the base port. SFP diagnostics SFP diagnostics provide detailed information on the SFP transceiver for a selected port. 6. 7. Launch the HCM. Collecting SFP data This section provides an overview of BCU commands and HCM features that provide information. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 45 . Select Echo Test. Select the HBA port and target port for the test. 4. rpwwn Remote port WWN that you want to ping.Collecting SFP data 3 Perform echo test through HCM Use the Echo Test feature to initiate an echo test between the HBA port and a Fibre Channel end point: 1. such as its health status. Select Configure > Diagnostics 3. connector type. Enter a test cycle if applicable. fcdiag –-scsitest <port_id> <rpwwn> [-l lpwwn] where: port_id ID of the port for which you want to test the SCSI link. port speed. SCSI test Use the fcdiag --scsitest BCU command to test the SCSI link between the HBA and remote port. Click the Protocol Tests tab. minimum and maximum distance. online or offline state. Select a port in the device tree. such as port WWN. voltage. WWN. 2.3 Collecting port data Initiating diagnostics through HCM Use the port SFP properties dialog box to display properties for a selected small form-factor pluggable (SFP) transceiver. FCP initiator). role of port (for example. frame data field size. and other information. Click the Remote Port Properties tab in the right panel. Port power on management (POM) Use the Port POM properties panel to monitor the SFP attributes. In the right panel. port type. Collecting port data This section provides an overview of BCU commands and HCM features that provide information on HBA ports. operating topology. click the Base Port Properties tab. select a base port. such as port number. If it is a target port. two tabs display in the right pane: Properties and LUNs. Launch the HCM. such as WWN. 1. 46 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . configured speed. temperature. and current specification policy. 1. Fibre Channel address. Click the POM tab in the right pane. QoS priority. 3. Remote port properties Use the Remote Port Properties panel to display properties that are associated with the remote port. configured topology. A notification is given for any parameters that are not within the configured power. link and port beaconing state. From the device tree. offline or online state. Select a port in the device tree. 2. QoS flow. remote device information. 1. 2. select a remote port. node WWN. and target ID. Base port properties Use the Base Port Properties panel to display information about a selected base HBA port. node WWN. operating speed. Click the SFP tab in the right pane. port name. 1. and WWN of the attached switch. 2. From the device tree. Fibre Channel address. role (such as target or initiator). node WWN. speed. and name server activity. The virtual port parameters property panel displays.Collecting port data 3 Logical port properties Use the LPorts Properties panel to display properties associated with a logical port. Port log Use the debug --portlog BCU command to display a log of Fibre Channel frames and other main control messages that were sent out and received on a specific port. 2. debug --portlog <port_id> where: port_id The ID of the port for which you want to display the port log. online or offline state. port --list <port_id> where: port_id ID of the port for which you want to display information. NOTE If the port log is disabled.ID of the port for which you want to display information. received and transmitted BB_Credits. Port query Use port --query BCU command to display port information. topology. 2. 1. From the device tree. state. such as port and node WWN. You can use this information to isolate HBA and Fibre Channel protocol problems. a warning message displays. such as WWN. Port list Use the port --list BCU command to list all physical ports on the HBA along with their physical attributes. offline or online state. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 47 . role (such as FCP initiator). 1. and state. select a virtual port. select a logical port. Virtual port properties Use the Virtual Port Parameters properties panel to display the properties associated with a virtual port. such as PWWN. Fibre Channel address. such as port and node WWN. port type. From the device tree. port --query <port_id> port_id . and beacon status. NWWN. Click the LPORTs Properties tab in the right panel. and attached switch WWN. current and configured speed. Fibre Channel address. Fibre Channel address. Use the debug -portlogctl command to enable and disable the port log. or 8 Gbps. Quality of Service (QoS) works in conjunction with the QoS feature on Brocade switches to assign traffic priority (high. <1|2|4|8|auto> The speed settings. Displaying settings through BCU Use the BCU auth --show command to display authentication settings. QoS and target rate limiting settings Target rate limiting throttles the Fibre Channel Protocol (FCP) read traffic rate to slow-draining targets to reduce or eliminate network congestion and alleviate I/O slowdowns to faster targets. Select a port from the device tree. with auto being autosensing mode. port --speed <port_id <1|2|4|8|auto> where: port_id ID of the port for which you want to display port speed. 4. 48 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . Displaying settings through HCM Use the Fibre Channel Security Protocol Configuration dialog box display authentication settings: 1. auth --show <port_id> where: port_id ID of the port for which you want to display authentication settings. The Fibre Channel Security Protocol Configuration dialog box displays.3 Authentication settings Port speed Use port --speed BCU command to display the current port speed setting. Use the following BCU commands and HCM to determine current status and configuration for these features. 2. and group value. hashing algorithm. medium (default). This displays the current CHAP secret. Authentication settings Use the Brocade CLI utility (BCU) or the HCM GUI to display the HBA authentication settings and status. low) to a given source and destination traffic flow. Select Configure > FC-SP > Authentication. such as 1. 2. select a remote port (target or initiator). QoS priority. Select a port in the device tree. 2. • Use the following BCU command to display QoS and target rate limiting enabled status and target rate limiting default speed. To open the Remote Port Properties panel: 1. Click the Properties tab in the right pane. and target rate limiting enforcement: bcu rport --query where: port_id Specifies the ID of the port for which you want to query attributes of a remote port. HCM • Use the Port Properties panel in HCM to display configured QoS parameters. port --query <port-id> where: port_id ID of the port for which you want to display port information.QoS and target rate limiting settings 3 BCU commands • Use the following BCU command to determine Target Rate Limiting speed and enabled status. From the device tree. • Use the following command to determine operating speed of the remote port. Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 49 . ratelim --query <port-id> where: port_id ID of the port for which you want to display target rate limiting settings. • Use the following command to display QoS status and information for a port. To open the Port Properties panel: 1. Click the Remote Port Properties tab in the right pane. bcu qos --query <port_id> where: port_id ID of the port for which you want to display target rate limiting settings. • Use the Remote Port Properties panel in HCM to display information on target rate limiting and QoS for the remote port. 3 Persistent binding Persistent binding Persistent binding is a feature of Fibre Channel (FC) host bus adapters that enables you to permanently assign a system SCSI target ID to a specific FC device. 50 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . 3. Select Configure > Persistent Binding. Persistent binding is available in the Windows and VmWare environments only. BCU Use the pbind --list BCU command to query the list of mappings for persistent binding on a specific port. You can also select and right-click on an HBA or port in the device tree and select Persistent Binding from the list. pwwn Port World Wide Name HCM Use the Persistent Binding dialog box to determine SCSI target ID mappings. Use the HCM or BCU to display target ID mapping for an HBA port. Select an HBA or port from the device tree. even though the device’s ID on the FC loop may be different each time the FC loop initializes. Launch the HCM. perform the following steps: 1. 2. pbind --list <port_id> <pwwn> where: port_id ID of the port for which you want to query mappings. . . . . . . . . . . • Solaris tuning . . . . . Johnson. . . . . delays. . . . . . . . . Linux 2. .6 has four different disk I/O schedulers: noop. for i in /sys/block/sd[b-z]/queue/scheduler do echo noop > $i done NOTE You must disable the default scheduler because it is not tuned for achieving the maximum I/O performance. deadline. . . . and merges requests for disk I/O to achieve better throughput and lower latency than would happen if all the requests were sent straight to the disk. . . . .. . . refer to the following publications • Workload Dependent Performance Evaluation of the Linux 2. . . Canada. . Ottawa. . ISBN: 013144753X • Linux Kernel Development Robert Love. . . . • VMware tuning . . . . . . . . . . . anticipatory and completely fair queuing. . 2nd edition. . . . . . . . . . . . . . . . . . . . . . . . . . . run the following commands on your system. . . . . . . . . . . . In this chapter • Linux tuning . . . . . . . . . . . . . . . . ISBN: 0-13-148682-9 • Performance Tuning for Linux Servers Sandra K. S. . . .6 IO Schedulers Heger. . . For performance tuning on Linux. This helps in achieving higher I/O rates by queuing multiple outstanding I/O requests to each disk. . . . . . . .Chapter 4 Performance optimization This chapter provides information and tools for optimizing your HBA performance. . . . . . . Badari Pulavarty. . . . . July 2004 • Optimizing Linux Performance HP Professional Books. . IBM Press. . 51 52 52 53 Linux tuning Linux disk I/O scheduling reorders. . . . . . . D. . . Linux Symposium. . . . . . . . . . . . . . • Windows tuning . . . 2005 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 51 . . . . . . . . . . . To enable the noop scheduler. . . . . . . . Gerrit Huizenga. . . . . Enabling the “noop” scheduler avoids any delays in queuing of I/O commands. . . . . . . . .. . . . . . . . . . Pratt. . . . ” Windows tuning Windows tuning involves configuring the driver and operating system tunable parameters. the CPU savings might increase latency. bfa_lun_queue_depth (outstanding I/O requests per LUN) The driver uses a default LUN queue depth value of 32.) • Interrupt latency Default: 1 micro second 52 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . If the queue depth is not sufficient. During performance testing with specific high-end array LUNs. The default values for the fibre channel port interrupt attributes are configured as below. set the following parameters on your system: • Set the maximum device read/write directive (maxphy”). unless interrupt moderation is performed intelligently. The queue depth can be configured at the following registry location with any value within the range 1 – 254. Please refer to Sun StorageTek SAM File System Configuration and Administration Guide document for details of the two parameters. it may be necessary to increase this I/O queue depth to a much larger value.4 Solaris tuning Solaris tuning To increase I/O transfer performance. This value is chosen to reflect the average operating I/O load in most scenarios. Microsoft recommends a value of 96 during high-performance testing scenarios. Storport manages the I/O throttling at per-LUN level to guarantee the configured queue depth. then the user will notice performance degradation. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bfad\Parameters\Device\bfa_lun_ queue_depth Interrupt moderation using interrupt coalescing feature Moderating interrupts can often result in reduced CPU load on the host but. Driver tunable parameters You can manipulate several driver parameters to increase performance. • Set the Fibre disk maximum transfer parameter (ssd_max_xfer_size). • Interrupt coalescing Default: ON • Interrupt delay Default: 1 micro second Valid Range: 0 – 1125 micro seconds (Note that the value of 0 disables the delay timeout interrupt. com: • Performance Tuning Best Practices for ESX Server 3. an eXtended version of MSI defined in PCI 3. the host CPU will frequently be interrupted. leading to a slower system response under heavy I/O load (more than 7000 I/O’s per second). The BCU ioc -intr command can be used to configure these interrupt attributes for the desired port. refer to the following publications on the VMware website at www. if the number of interrupts handled between the Interrupt delay period is relatively smaller. leading to improved overall CPU utilization.vmware. To enable MSI-X. I/O completion requests are not coalesced by the firmware.Related Publications • Fibre Channel SAN Configuration Guide. Refer to “Using ESX Server with SAN: Concepts.microsoft. the HBA will not interrupt the host until Interrupt delay duration. When this feature is turned on. set the following registry key value to 0.Storage Performance Best Practices . bcu ioc --intr <ioc_id> <--coalesce|-c> {on | off} [<Latency> <Delay>] Message signaled interrupts (MSI-X) All Brocade HBAs support MSI-X. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bfad\Parameters\Device\msix_ disable OS tunable parameters Please see the section “Storage Stack Drivers” in Disk Subsystem Performance Analysis for Windows Server 2003 optimizations located on the following website.) • Interrupt Coalescing When this feature is turned off. Refer to the following sections: .microsoft.com Please see the sections “Performance Tuning for Storage Subsystem” and “I/O Priorities” in Performance Tuning Guidelines for Windows Server 2008 located on the following website. While this helps reduce I/O latency.com VMware tuning For performance tuning on VMware. http://www.0 specification.” Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 53 . together with Interrupt latency. MSI-X helps improve overall system performance by contributing to lower interrupt latency and improved host CPU utilization. Interrupt delay. http://download. then this will result in performance degradation as the I/O completion process has slowed down. MSI-X is supported in Windows Vista and Windows Server 2008.VMware tuning 4 Valid Range: 0 – 225 micro seconds (Note that the value of 0 disables the latency monitor timeout interrupt. However. helps to reduce the number of interrupts that the host CPU processes per second. 4 54 VMware tuning Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . 45 enabling through BCU. 44 enabling through BCU. 41 enabling through HCM. 28 collecting data using event logs. 40 HBA temperature.Index A adapter list command. 28 adapter query command. 52. 40 C collecting data using BCU. 40 enabling through HCM. 44 error message when removing Windows driver. 40 enabling through BCU. 28 port list. 43 SCSI test. 25 collecting using host commands. 30 collecting data using Fabric OS commands. 15 verify using Solaris commands. 15 diagnostics. 45 enable through BCU. 40 echo test enabling through BCU. 13 error when rolling back driver. 40 enabling through HCM. 46 trace route. 53 E echo test. 13 driver installation verify using Linux commands. 45. 43 PCI loopback tests. 48 B BCU commands adapter list. 42 enabling through HCM. 43 loopback tests. 45 HBA. 40 beaconing. 48 to collect data. 24 device manager. 44 driver install errors. 42 enabling through BCU. 46 enable through HCM. 16 verify using VMware command. 40 enabling through BCU. 14 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 55 . 32. 44 enable through HCM. 43 enable through HCM. 44 enabling through HCM. 28 application log. 41 enabling through BCU. 16 driver tunable parameters for Windows. 33 authentication settings. 44 enable through HCM. 24 data to provide support. 28 D data collecting data with BCU and HCM. 43 enabling through BCU. 41 memory test. 25 beaconing. 28 adapter query. 45 SFP. 47 port query. 42 enabling through HCM. 42 enabling through BCU. 42 ping end points. 47 port speed. 30 HCM. 2 installation confirming driver installation. 32 master. 13 H HBA diagnostics. 32 memory test. viii switch support. 32 master log severity levels. 36 displaying through HCM. 32 HCM options to collect data. 41 M master log.event logs. 24 host freezes or crashes. 38 logs. 42 enabling through BCU. 37 displaying through BCU. 31 F Fabric OS commands. 35 isolating problems. 14 driver errors. viii PWWN. 40 fabric OS support. 30 HCM. 34 Windows event log support. 28 HBA statistics. 42 HBA properties. 37 displaying through BCU. 47 logical port statistics. xiii serial number. 32 application. 32 host system. 35 displaying through BCU. 37 displaying through HCM. 30 I information gathering. 43 O operating system support. 42 enabling through HCM. 36 displaying through BCU. 13 problems. 12 IOC statistics. viii supported models. 34 loopback tests. xii storage support. 41 enabling through BCU.sys message. 32 port. 32 syslog support. 47 syslog support. 12 verifying. 41 enabling through HCM. 34 event message files. 37 Fibre Channel links verifying. 28 fabric statistics. 34 HCM cannot connect with agent. 16 host system logs. viii operating system support. 31 HBA memory test. 35 displaying through HCM. 20 files needed for bfad. 30 host system logs adjust logging level. 30 master log severity levels. 34 Windows event log support. 33 event. 5 L Linux tuning. 17 HCM logs. 25 host commands for collecting data. 36 FCIP initiator mode statistics. viii 56 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . 15 HBA properties panel. 38 displaying through HCM. 51 log application. 32 host system. 32 logical port properties. viii HBA event message files. 10 problem information. 35 problem driver event messages. 11 files needed for bfad. 51 Solaris tuning. 16 I/Os not failing over on path failure. 13 general.P PCI loopback tests. 37 displaying through HCM. 10. 42 performance optimization Linux tuning. 10 port link not up. 6 operating system errors. 47 remote. 46 virtual. 10. 10 fabric authentication failures. 9 maximum IOPS numbers low. 52 VMware tuning. 50 ping end points diagnostics. 47 port log. 11 HBA not reported under PCI subsystem. 47 port properties base. 11 errors when installing driver. 10. 46 logical. 10 HBA not registering with name server. 34 enable through BCU. 39 R remote port properties. 9 unable to create NPIV ports. 50 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 57 . 10 target not visible. 48 QoS statistics. 5 loss of sync and signal. 17 host not booting from remote LUN. 43 enable through HCM. 42 enabling through BCU. 2 properties panel for HBA. 35 enable through HCM. 47 port speed command. 12 S SCSI target ID mappings. 43 port data. 39 displaying through HCM. 42 enabling through HCM. 53 Windows tuning. 46 port query command. 37 resolving installation problems. 6 HCM fails connection with agent. 8 failed to connect to agent error. 46 remote port statistics. 14 host system freezes. 48 port statistics. 12 installer program does not autorun.sys message. 9 installation. 28 Q QoS settings. 46 port list command. 13 isolating. 9 virtual devices not in name server. 5 HBA not in fabric. 43 enabling through BCU. 39 displaying through BCU. 36 displaying through BCU. 9 no HBAs reported. 47 port properties panel. 52 persistent binding settings. 8 LUN not visible. 7 QoS performance issues. 39 remote port. 39 displaying through HCM. 37 technical help for product. 37 displaying through BCU. and browser. 47 virtual port statistics. 27 on port crash event. 5 introduction. xii temperature diagnostics. 25 using through BCU. 1 V virtual port properties. 34 trace route. 38 displaying through BCU. 52 WWPN of HBA. 37 virtual port. 39 displaying through HCM. 45 enable through BCU. 34 Windows tuning. 44 troubleshooting gathering information. 27 syslog support. 35 display through HCM. BCU. 36 displaying through BCU. 36 displaying through BCU. 35 QoS displaying through BCU. 39 displaying through HCM. 38 port. 37 IOC. 35 displaying through HCM. 45 serial number location. 26 using through browser. xii SFP diagnostics. 44 enable through HCM. 46 Solaris tuning.SCSI test. 26 using through port crash event. 36 displaying through HCM. 1 using this manual. 37 displaying through BCU. 45 enable through HCM. 27 using. 38 virtual port statsitcs displaying through BCU. 34 support data to provide. 48 target statistics. 34 display through BCU. 52 Windows event log support. 37 displaying through HCM. xiii T target rate limiting settings. 36 FCIP initiator mode. 44 enabling through BCU. 43 58 Brocade Fibre Channel HBA Troubleshooting Guide 53-1000885-02 . 37 displaying through HCM. 39 VMware tuning. 52 statistics fabric. 35 displaying through BCU. 2 general problems. 53 W Windows driver tunable parameters. 35 logical port. 27 using through HCM. 39 statistics for HBAs. 38 displaying through HCM. 24 support save differences between HCM.