Ibm Svc Enhanced Stretched Cluster Guide

IBM System Storage SAN Volume Controller – Enhanced Stretched ClusterEvaluation guide v1.0 Sarvesh S. Patel, Bill Scales IBM Systems and Technology Group ISV Enablement January 2014 © Copyright IBM Corporation, 2014 ............................................................................................................................... 4 Configuring site awareness ..................................................... 14 Resources ................................................................................................................ 12 Invoking the site disaster recovery feature ........ 10 Multi-WWNN controller ....................................................................................................................................................................................... 15 Trademarks and special notices ............................................. 4 Connectivity details .......................................................... 6 Assigning sites to IBM SVC nodes ................................................................................................................................................................................................................................................................................................................ 1 Getting started .............Table of contents Abstract............................................................................................................ 12 Enabling / Disabling the site disaster recovery capability ........... 1 About IBM SAN Volume Controller Stretched Cluster services ........................................................................................... 3 Setup description ................................................................................................................................................................................................................... 12 Quorum disk placement ...................................................... 6 Disaster recovery sites..................... 16 IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster ........ 1 About this guide ................................................................................................................................................................................................................................................................................................................................................................................................... 8 Controller site assignment ................................................... 2 Assumptions ....................................................................... 15 About the authors............................................. 13 Returning to normal operation after invoking the site disaster recovery feature .......................................................................................................... 2 How Stretched Cluster works? ............................. 3 Prerequisites and configuration considerations ............................................................................................................................................. A copy of the volume (VDisk) is stored at both the locations. The host will see half of the paths active and half dead as at least one node is running. The existing SVC Stretched Cluster solution is the concept of having two locations with each location having one node from each I/O group pair. SVC always has automatic quorum to act as a tie-break. The white paper gives information regarding the stretched topology and the disaster recovery feature in the Enhanced Stretched Cluster feature. This feature is supported since version 6.3. the two nodes in an I/O group are separated by a distance between two locations. SVC will fail over internally with minimal delay. at least one copy of the volume will be accessible from the user. two buildings in a campus. These two locations can be two racks in a data center. released. a host can fail over to another SVC node and continue performing I/O. if there is an impact to just a single storage device. This configuration implies that in case of a power failure or storage area network (SAN) failure at one site. If there is a failure in a fabric element or SVC node. the procedure of creating an IBM System Storage SAN Volume Controller (SVC) cluster with the Enhanced Stretched Cluster topology is described. Getting started This section gives a brief idea IBM® System Storage® SAN Volume Controller (SVC) Stretched Cluster implementation and different topologies used to configure the Stretched Cluster implementation.Abstract In this white paper. The white paper gives a brief introduction to the Enhanced Stretched Cluster feature and provides guidance on how to assign site awareness to the controllers and other entities. The key benefit of Stretched Cluster compared to Metro Mirror is that it allows for fast non-disruptive failover in the event of small scale outages. a storage virtualization system supports a stretched implementation of a cluster. Therefore. quorum disks are usually held in a third location. 1 About IBM SAN Volume Controller Stretched Cluster services IBM SVC. The main advantage of configuring a cluster as a stretched implementation is to give more redundancy considering the failures in power domains. Both the copies of the storage are maintained in synchronization by SAN Volume Controller. It also describes a procedure of enabling and disabling the feature and the procedure to be used in case of disaster recovery. This means no external management software or human intervention is ever required to perform a failover. the loss of one location causes no disruption to the alternate location. For example. additionally. or two labs between supported distances. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 1 . This simplicity is another key advantage. 1 IBM SAN Volume Controller (SVC) or SVC cluster in the entire guide are general terms that apply for IBM SVC platform only. With Stretched Cluster implementation. 0 code levels (or above). The SVC clusters have the required licenses.pdf About this guide The purpose of this guide is to support self-guided.0. The user has the basic understanding and awareness of SVC stretched and split cluster concepts.3.ibm.) The storage SAN is configured according to the product documentation and the infrastructure to support SVC clusters in a stretched cluster using Fibre Channel 8G is properly in place. This is not meant to be a substitute for product documentation and encourages users to refer to the product documentation.software.3. The user knows the different heterogeneous SVC platforms that can be added in FC partnerships. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 2 . Assumptions Below are the assumptions considered while writing this white paper. • • For SVC documentation.dhe. refer to: http://pic. this topology has direct connections from IBM SVC nodes to the switches from different power domains. • • • • The SVC clusters are successfully installed with the latest (at the time of this publication) IBM SVC 7.0 attempts to demonstrate how to deploy Enhanced Stretched Cluster configurations with site awareness and the most important site disaster recovery feature. The same apply for IP partnerships as well.com/storage/san/sanvc/V6. information center.ibm. (No separate license is required to enable Enhanced Stretched Cluster site awareness and site disaster recovery feature. and configurations for host attach. command-line interface (CLI) guide of Enhanced Stretched Cluster for more details.0/SVC_Split_IO_Group_requirements_Errata_V1.2. This paper is intended to provide an overview of the steps required to successfully evaluate and deploy site awareness on SVC. Stretched Cluster configuration using inter switch links This implementation strategy has inter switch links between two sites having different power domains.com/infocenter/svc/ic/index. refer to the following link: ftp://ftp.The following two types of configurations are supported: • Stretched Cluster configuration without using inter switch links As explained in the product documentation. The guide version 1. • For the documentation on the solution as included in 6. SVC storage concepts. hands-on evaluation of the Enhanced Stretched Cluster feature deployments for the storage administrators and IT professionals and walk them through the different configurations considerations and workflow required to set the SVC clusters in enhanced stretched implementation.jsp Note: Refer the configuration section in above documentation for how Stretched Cluster works. that is. Image mode virtual disks (vdisks) should not be configured on the controller that has site assignments. For the first release that is 7. A single SVC system supports four I/O groups. and Global Mirror with Change Volumes SAN NAS Brief description These are the different remote copy services supported on SVC platforms. When the configuration node of the system fails. Note that no extra hardware configuration changes are required to enable the site disaster recovery feature. Fibre Channel Unless explicitly specified. CF8. The IP addresses fail over to the surviving node in the I/O group. • • • IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 3 .IP partnership and SVC terminology Metro Mirror. all failed over IP addresses are failed back from the surviving node to the rejoined node and virtual disk access is restored through this node. • The SVC Clustered System should be configured as mentioned in the information center. When the failed node rejoins the system. Storwize V3700 and IBM PureFlex™ System – storage nodes. Prerequisites and configuration considerations To enable Enhanced Stretched Cluster site disaster recovery feature. Details are given in a separate section for restrictions.2. Fibre Channel over Ethernet Failover Failback I/O group FC SVC or IBM SAN Volume Controller FCoE Table 1: Terminology and abbreviation How Stretched Cluster works? The Stretched Cluster feature enables one SVC cluster to configure itself from two sites that are in different power domains and the disaster recovery feature which can be used in case of a disaster to recover a cluster when less resources are available. 8G4). eight nodes. A cluster can be connected using FC or FCoE SAN. Enhanced Stretched Cluster site disaster recovery feature is only supported on SVC nodes without having internal SSDs.0. Global Mirror. Storwize V5000. IBM Storwize® V7000. management IPs also fail over to an alternate node. Two nodes or two canisters form an I/O group. Storage area network Network-attached storage Failure of a node within an I/O group fails causes virtual disk access through the surviving node. a general term that is used to describe all applicable SVC platforms – IBM SVC (CG8. the following details should be considered and understood. It is completely dependent on how the administrator wants to configure the system IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 4 . • • Without inter switch links With inter switch links Both the implementations will not affect the behavior of site awareness and the site disaster recovery feature.Setup description Hardware summary • Minimum two IBM SVC nodes • Brocade Fibre Channel switches (IBM Brocade 2498) • Backend storage controllers (IBM System Storage DS3400) Connectivity details The Stretched Cluster system feature is supported with two types of implementations. These two sites can be in the same rack with different power domain. two production sites are connected using inter switch links. only then the site disaster recovery feature can be invoked. across racks. With inter switch links In the implementation described in the following figure strategy. across data centers. Refer to the information center document for more details regarding the recommended connections. the connectivity will not change and the feature is optional. The administrator can choose to use the same for disaster recovery. and so on as it was supported earlier. An important thing to notice here is. Implementation details • Without inter switch links This hardware implementation will be the same as recommended for a non-stretched cluster. if the Stretched Cluster implementation has site awareness. No change in the connectivity is needed. As mentioned in the earlier sections. • IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 5 .implementation. the same concept has been named using the new CLIs.Figure 1: Enhanced Stretched Cluster implementation using inter switch links Configuring site awareness Assuming that the connectivity is completed and a Stretched Cluster is implemented. Disaster recovery sites As the earlier stretch cluster implementation needed three sites. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 6 . lssite and chsite. in this paper. They are hidden on other platforms. Note that the features described here are available on SVC systems only. you will learn how to set up site awareness for all the different entities in the cluster. A new set of 'site' objects will be defined. Figure 3: Site name allocation using “chsite” CLI After the names are assigned to the sites. and 'site3'. There will always be exactly three sites. Site3 is the optional third site for a quorum tie-break disk. (Site index 0 will never be reported). The default names for the sites are 'site1'. This is the default setting for an object. 2. Objects can also leave their 'site' value undefined. chsite. and therefore. The appropriate 'site' instance will be referenced when a site value is defined for an object. Figure 2: Site listing using “lssite” CLI The site names can be changed using the CLI. numbered 1. and 3. Figure 4: Changed site information IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 7 . These will be implicitly created for every system automatically. The only configurable attribute for each site is its name. 'site2'. the Stretched Cluster implementation is spread across two data centers. they can be viewed using the lssite command. lssite has been introduced to list down the sites. Enabling the site disaster recovery feature and correct operation of the disaster recovery feature requires assigning objects to the site. For example. Site1: Production site 1 Site2: Production site 2 Site3: A site at a different location to house the quorum disk In IBM SVC version 7. A new CLI. these three sites would be present by default.2 onwards. named the site accordingly. There is no means of deleting sites or creating extra sites. These are the mandatory three sites needed for a Stretched Cluster implementation. Site1 and site2 are where the two halves of the Stretched Cluster are located. then extra policing is added which requires that every node have its site defined. This is also the initial state on upgrade to version 7. or by using the chnode command to the existing nodes. then within an I/O group with two configured nodes. Therefore the site must be specified in addnode. The default for a node is to have its site undefined. It can be set back to undefined. consider a two-node SVC cluster.2. the user can verify the site assignment using the lsnode command in concise as well as detailed view. It can also be specified or changed after that time.0. The site can be specified when a node is added to the system. the next task is to assign sites to the nodes in the cluster. Nodes cannot be assigned to site 3. Figure 5: Assigning site attributes to IBM SAN Volume Controller nodes After assigning appropriate site awareness to nodes. Additionally when the disaster recovery feature is enabled. Now. When the disaster recovery feature has been enabled using chsystem. and disallows any changes in the site for any configured node.Assigning sites to IBM SVC nodes The site can optionally be defined for each node. one node must be assigned to each of sites 1 and 2. Nodes can only be assigned to sites 1 or 2. Here. else addnode will fail. For example. one node is assigned to site 1 and another to site 2. The site awareness to nodes can be achieved using the addnode command while adding a new node to the cluster. as the sites are renamed accordingly. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 8 . Figure 6: Listing node site attributes using the lsnode CLI IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 9 . The site for a controller cannot be changed when the disaster recovery feature is enabled i. This will be the default for pre-existing controllers on upgrade to 720. Controllers can be assigned to any of the sites (site 1. The site for a controller can be changed when the disaster recovery feature is disabled. The user is responsible for ensuring that all such 'controller' objects have the same 'site' specified. the site can optionally be specified for each controller.e. or it can be set back to 'undefined' again. This will be the default for pre-existing controllers on upgrade to 720. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 10 . A managed disk (MDisk) derives its 'site' value from the controller that it is associated with at that time. These changes are effective for any MDisk whose controller has a site defined. As specified earlier. The use of solid-state drives (SSDs) within SVC nodes is not supported in the configurations described. Some backend storage devices are presented by the SVC system as multiple controller objects. the topology is Stretched or if the controller has managed (or image-mode) disks. or it can be set back to 'undefined' again. The software does not enforce any requirement that all MDisks have a site defined. so as to ensure that any MDisk associated with that controller is associated with a well-defined single site. The site property for a controller adjusts the I/O routing and error reporting for connectivity between nodes and the associated MDisks. The site attribute can be assigned to a controller using the chcontroller command.Controller site assignment After assigning sites to the nodes. site 2 or site 3). The default for a controller is for its site to be undefined. the user needs to assign the site attributes to controllers. It can also be changed if the controller has no managed (or image-mode) MDisks. Controllers can be assigned to any of sites (site 1. The site can optionally be specified for each controller. The site assigned to a controller can be verified using the lscontroller command. The default for a controller is for its site to be undefined. site 2 or site 3). even if the disaster recovery feature is disabled. and an MDisk might be associated with any of these from time to time. or controllers with no site defined A node with no site defined and any controller And of course. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 11 . Faults are only reported if any node in site 1 has inadequate connectivity (that is. if a controller is configured to site 2. then connectivity to nodes in site 2 is not expected or required. if any node in site 1 has less than two SVC ports with visibility to the controller).Figure 7: Listing site attributes for a controller Connectivity is permitted between: • • Any node and controllers in site 3. Similarly. and is disregarded. • • • When a controller is configured to site 1. I/O is permitted in the following most important cases: • • A node configured in a site and a controller MDisk configured to the same site A node configured in a site and a controller MDisk configured to site 3 The fault reporting algorithms for raising event logs in the case of missing connectivity are also adjusted to allow for these rules. then connectivity to nodes in site 1 is disregarded. New clusters and clusters upgrading to 7. Note that before the cluster topology being changed to stretched (the activation of the disaster recovery feature) system ignores the site parameter when selecting quorum disks.2. There is no precondition on sites being configured for controllers. SVC system elects 3 quorum disks (one in each of the three sites) and makes the quorum disk at site 3 as an active quorum disk. The site disaster recovery feature can be enabled or disabled by using the chsystem command and can be checked using the lssystem command. When the disaster recovery feature is enabled and automatic quorum disk selection is also enabled. then quorum selection algorithm operates as in SVC 7. then less than three quorum disks will be configured. If a site has no suitable MDisks. The feature will not be operable on nodes that were absent until they rejoin the cluster.0 and later versions have the disaster recovery feature disabled by default. If the user is controlling quorum using chquorum. The preconditions on enabling the disaster recovery feature are: • • All nodes are assigned to a site.Mu lti-WWNN c o n trolle r When the site is changed on a multi-worldwide node name (WWNN) controller.1. All I/O groups with two nodes are assigned with one node in site 1 and one node in site 2. then the choice of the quorum disk the user selects must also follow the one disk per site rule. Quorum disk placement If the site disaster recovery feature is disabled. Figure 8: Output of the lssystem command when the feature is disabled Figure 9: Enabling the site disaster recovery feature using the chsystem command Figure 10: Output of the lssystem command when the feature is enabled IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 12 . This means that the quorum selection can change if the disaster recovery feature is enabled for an existing installation. Enabling / Disabling the site disaster recovery capability A system setting will enable and disable the disaster recovery feature. all of the affected controllers are updated with the site setting on all controllers at the same time.0 release. Alternatively. the remaining half of the cluster will stop with each of the nodes suffering a lease expiry. fix the link. Nodes that did not participate in the recovery process (and hence are still members of the old cluster) will not automatically be re-added to the cluster. For a stretch cluster that has been configured with a topology of 'stretched'. the normal recovery option is to allow the site that got the quorum to keep running. A new service CLI subcommand is defined that attempts to invoke the disaster recovery feature. then both halves of the cluster will display 'node error 551' (the new version of 'node error 550').if access to the site 3 quorum disks fails at the same time as the inter-site link from site 1 to site 2. set a flag in each node indicating that disaster recovery has been invoked and then warm start. and so on. This is achieved by using the rmnode command from within the management interface of the old cluster (if it is still operating) or by using the IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 13 . the nodes continue to display the existing 'node error 550'.Invoking the site disaster recovery feature When the inter-site link fails in a Stretched Cluster. The user must explicitly delete these nodes from the cluster (returning them to candidate state) before they can be auto added. these nodes will display 'node error 550' indicating that there are not sufficient nodes to form a quorum. and then the other nodes will be automatically added back and copies synchronized. If access to the quorum disk is restored. In the simple inter-site link failure. the two halves of the cluster race each other to use the quorum disk to resolve the tie-break situation. If there are insufficient nodes to allow the overridequorum command to be used. the inter-site link from site 1 to site 2 can be restored to allow the cluster to reform. Normally. The disaster recovery feature is only expected to be invoked if the surviving site suffers a disaster just after the inter-site link failed. The site that keeps running continues to display a cluster topology of 'stretched' and a topology status of 'dual_site' even though one site is offline. The remote nodes can only be introduced to the local system later using the following procedure (needs disconnecting all FC/FCoE connectivity of the remote node). the code determines if sufficient nodes are present and if invoking the disaster recovery feature is an alternative option and will modify the node error that is displayed (new node is error 551). There are several more-complex failure scenarios . The half that successfully reserves and updates the quorum disk keeps operating. The status will only return to "dual_site" after the user has performed the recovery actions to reintroduce the nodes from the other site. They cannot be involved in the initial recovery of the local site because their state will corrupt the consistent freeze of the local site. The topology (reported by lssystem) remains as stretched but the topology_status changes from dual_site to either recovered_site_1 or recovered_site_2 to indicate that disaster recovery has been invoked. The CLI subcommand will not affect the nodes of the remote site. leaving the other site as the only option for continuing with I/O. then whichever site regains the access first will be able to automatically restart. the node that is running the CLI subcommand generates a new worldwide unique cluster ID and informs each of the set of visible nodes of the local site to change their cluster ID to the new cluster ID. If accepted. Note that the cluster alias which by default is set to the cluster ID when a cluster is created is not changed and this ensures that the VDisk UIDs do not change. The code will then disable the disaster recovery feature in the new recovered cluster. including: • Defining quorum disks in the correct sites • Re-creating volumes that were not automatically recovered earlier • Re-creating any intra-system copy services that were deleted because their volumes were deleted • Re-creating any inter-system Metro Mirror or Global Mirror objects Note that there is no need to explicitly re-enable the disaster recovery feature. 9. This can be achieved by using rmmdisk if the MDisk no longer holds any volume copy. Access to all 'recovered site' volume copies is recovered. the alert auto fixes and any non-local site volume copies become online. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 14 . Access to all other volume copies is lost. The user must treat all such storage as suspect and potentially corrupted. 1. This includes the mirror-half of stretched volumes plus any single-copy volumes with defined local site. care is needed to not conflict with the activity of any still active nodes in the failed site. the status returns to 'dual_site' and assuming that there are online nodes at both sites. All such quorum disks must be deleted. then the process must use chquorum to select new quorum disks and prevent attempts to use the other site quorum disks.satask leavecluster -force command. The conservative approach is to delete all such volume copies. 3. 6. 7. 5. Some users might choose to retain access to such volumes to attempt to recover some data. Before running these commands. the user can address the missing nodes. 2. If there are volume copies that are being retained. the voting set will be manipulated to prepare for the next disaster recovery. 8. consistency groups. All inter-system remote copy relationships. and partnerships must be destroyed (partnerships will be in the partially-configured state). At this point. the user must disconnect all the FC/FCoE cables from all the nodes which they want to re-add in the cluster. The following sequence of steps are required. When the last offline node from the failed site is repaired. Disaster recovery feature is invoked . The process of reconstructing the system objects can begin. Access to all other site quorum disks is lost. In particular. for example. and when the event log auto fixes the cluster topology. then either unconfiguring the node using svctask rmnode (in the abandoned cluster) or satask leavecluster as described earlier or decommissioning the node so it can no longer access the shared storage and then issuing rmnode in the recovered cluster to inform it that this step has been performed. Returning to normal operation after invoking the site disaster recovery feature The user must follow a careful set of steps to ensure that the system maintains integrity as the two sites' connectivity is recovered. This requires disconnecting the FC/FCoE connectivity of the missing nodes. if power is recovered after a failure.alert is raised indicating that this process must be used. The cluster topology remains as 'stretched'. 4. Mirrored volumes with one online fresh local copy can be retained. pdf SVC Split Cluster – How it Works ibm.cgi?CTY=US IBM Redbooks® ibm. He has 6 years of experience in Storage test. he was the functional development architect and lead for the feature. Bill Scales is Software engineer in IBM Systems and Technology Group SVC and Storwize family. You can reach Sarvesh at [email protected] The following websites provide useful references to supplement the information contained in this paper: • • • • • • • IBM Systems on IBM PartnerWorld ibm.com/public/applications/publications/cgibin/pbi.ibm. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 15 .elink.ibmlink. he was the functional test lead for the feature.ibm.com/partnerworld/systems/ IBM Publications Center http://www. As part of the Enhanced Stretched Cluster. As part of the Enhanced Stretched Cluster.com.com/redbooks/redbooks/pdfs/sg248142.pdf IBM SAN Volume Controller Stretched Cluster with PowerVM and PowerHA ibm.com/redbooks/redbooks/pdfs/sg248072.com/redbooks IBM developerWorks® ibm.com/developerworks IBM SAN and SVC Stretched Cluster and VMware Solution Implementation ibm.com/developerworks/community/blogs/storagevirtualization/entry/split_cluster?lang=en About the authors Sarvesh S Patel is a Software engineer in IBM Systems and Technology Group SVC and Storwize family. function or delivery schedules with respect to any future products. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. capability. or other publicly available sources and does not constitute an endorsement of such products by IBM. the IBM logo. Some information addresses anticipated future capabilities. published announcement material.com/legal/copytrade.com are trademarks or registered trademarks of International Business Machines Corporation in the United States. or any other claims related to non-IBM products. Other company. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 16 . The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream.ibm. or service names may be trademarks or service marks of others. these symbols indicate U. Such information is not intended as a definitive statement of a commitment to specific levels of performance. Information concerning non-IBM products was obtained from a supplier of these products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice. including vendor announcements and vendor worldwide homepages. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www. Changes may be incorporated in production models. Such trademarks may also be registered or common law trademarks in other countries. Information is provided "AS IS" without warranty of any kind. Therefore. Questions on the capability of non-IBM products should be addressed to the supplier of those products. or both. and ibm. Such commitments are only made in IBM product announcements. and the workload processed. other countries. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. Actual environmental costs and performance characteristics may vary by customer. registered or common law trademarks owned by IBM at the time this information was published. Photographs shown are of engineering prototypes. IBM has not tested these products and cannot confirm the accuracy of performance. Sources for non-IBM list prices and performance numbers are taken from publicly available information.shtml. product. the I/O configuration.Trademarks and special notices © Copyright IBM Corporation 2014. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.S. the storage configuration. no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™). IBM. and represent goals and objectives only. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. IBM System Storage SAN Volume Controller – Enhanced Stretched Cluster 17 . The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk.