Isilon Best Practices Guide for iSCSI With OneFS

iSCSI and OneFSA Best Practices Guide for Isilon IQ™ Storage By Michael Kade – Global Systems Engineer An Isilon Systems Best Practices Guide ISILON SYSTEMS® ® February 2010 ISILON SYSTEMS® 1 . ............................................. 6 LUNs are Directly Associated with Targets ....................... 8 LUN Backup ............................................ 5.............................................................................................................................................................................. 3 Hardware iSCSI Adapter .............................................................................................................................................. 3 Optimal Block Size ............................................. 7 OneFS Prefetch .............. 8 iSCSI with SyncIQ ............................................................................................................................................................................................................................... 3 Isilon Best Practices for various vendor iSCSI initiators ......................... 4 Write Caching ................................ 5 Client Logical Volume Manager ..................................................................................................................................... 4 Jumbo Frames ............ 8 References................................................. 4 Dedicated Network ................. Conclusion ..................................................................................................................................................Table of Contents 1........................................................................................ 3 Snapshot Licensing ................................. 8 LUN Resizing .................................... 4 2x Protection Recommended ........................................................................................................................................................... 4 Maximum Number of Targets and Maximum Number of LUNs Per Target .................. 7 LUN Provisioning ............................................................................................................................................................................... 3.............................................................................................................. 7 Dynamically Allocated IP Pools result in LUNs not being visible ............................................................................................................................................................................................................................................................................................................................................................................................................................. Introduction ......................................................................................... 9 ISILON SYSTEMS® 2 ......... Streaming Access Pattern ................................................................................................................. 2................................................................ 7 Missing Logical Unit Number .................................................................................................................................... 7 Concurrency vs...................... 3 Isilon Best Practices for iSCSI with OneFS .............. 8 4....... g.1. Snapshot Licensing Cloning allows the user to create a new LUN based upon an exact point-in-time copy of an existing LUN. This guide reviews best practices directly related to the storage of Logical Units (LUNs) on the Isilon OneFS filesystem. Isilon Best Practices for various vendor iSCSI initiators Different best practices guides are available that include detailed guidance for setting up and configuring iSCSI on the various vendor initiators. Isilon iSCSI solutions allows applications like Microsoft Exchange. Microsoft SQL. There are three types of clones available. Hardware iSCSI Adapter Hardware iSCSI Adaptors have not been tested and are not currently supported for use with Isilon storage products. iSCSI will continue to work. although LUNs may be moved or placed anywhere within the directory hierarchy for convenience (e. Isilon Best Practices for iSCSI with OneFS The following best practices will ensure the best possible performance when using iSCSI with the OneFS filesystem. snapshot and shadow. The Isilon iSCSI implementation features the following capabilities: Implementation of RFCs 3720 and 5048 Support for Thin-Provisioned or Fully-Allocated LUNs Support for One-Way CHAP Authentication Support for dynamically growing LUNs Support for creating LUN clones. Introduction The implementation of the iSCSI protocol within Isilon IQ brings unified storage capability to Isilon. This is in addition to Isilon Installed Product Help and Isilon Online Knowledge Center. This guide is intended to be complementary to the Isilon Installed Product Help. By default. to enforce a single set of SmartQuotas or aid in SyncIQ). and block-level file systems like VMware VMFS. and other Isilon iSCSI Best Practices guides. Each of these types of clones requires that SnapshotIQ be licensed. snapshots or shadow copies (writable snapshots) Support for SCSI reserve/release LUN arbitration control Client Support for iSNS (iSCSI Name Service) Support for VSS (Volume Shadow Service) when utilizing the Microsoft iSCSI Initiator Isilon iSCSI LUNs are constructed as files that reside within OneFS. Without SnapshotIQ licensed. Each iSCSI LUN is represented as eight extents within a particular directory. but it will be impossible to clone a Logical Unit. directories lie underneath the target that the LUN is allocated to. 2. normal. Please see the References section of this guide for a more complete list of documentation covering iSCSI. to benefit from the Isilon distributed architecture. Isilon Online Knowledge Center. ISILON SYSTEMS® 3 . 3. Leveraging Isilon industry leading scale-out technology. Jumbo Frames In order to achieve maximum performance. etc) within OneFS are stored as groups of 8K blocks. and use of target masking is advised to prevent an initiator from being overwhelmed with SendTargets responses. the block size within a LUN as written by the initiating client may be different than the 8K-block size of a OneFS file. 2x Protection Recommended In most cases. it is recommended that you enable Jumbo Frames of 9K on the Ethernet ports dedicated to iSCSI on both the Isilon node(s) and the client system. format the data contained within the LUN so that the block size is a multiple of the 8K-block size of OneFS. The data contained within a LUN is solely under the control of the initiator within the client and. Best performance will be achieved if iSCSI is the only protocol accessing the assigned gigabit or 10-gigabit Ethernet port. In order to set the FlexProtect setting to advanced. it was previously recommended that the block size of the client system be right-sized to match the block size of OneFS. Since each I/O would then be one or more multiples of 8K. This will result in 6 times less I/O and consequently better performance than using the default Ethernet frame size of 1500.Optimal Block Size LUNs reside as files within the OneFS filesystem. files. When possible. More detailed information is contained within the Best Practices Guides for the Microsoft. The number of LUs per target is restricted to 256 (numbered 0-255). Users creating massive numbers of targets or LUNs should be aware of initiator restrictions. as such. Dedicated Network iSCSI is a block based protocol and. Maximum Number of Targets and Maximum Number of LUNs Per Target Isilon does not enforce any limit on the number of targets. Additionally if space is a primary concern then you may be willing to sacrifice some write performance to free up space. Setting 2X protection on a LUN may fail if the global setting for FlexProtect is different than 2X (mirrored). can be very network intensive. if your workflow is primarily reads. If the client block size within the LUN is not a multiple of the 8K-block size of OneFS. then multiple I/Os could occur and this would result in suboptimal performance. login to one node of the cluster and type isi flexprotect advanced. then 2x protection won’t provide as much benefit. Linux and VMWare initiators. as such. ISILON SYSTEMS® 4 . 2x (mirrored) protection will result in better performance since parity reads aren’t required during the write process. However. File system objects (directories. You can change the individual protection for a Logical Unit once you have changed the FlexProtect setting to advanced. but unless you know whether your client initiator has this commit technology. ISILON SYSTEMS® 5 . then it may still be possible to use Write Caching by implementing LUN protection using techniques available to Logical Volume Managers available with some client operating systems. Some software iSCSI initiators protect this data-in-flight.Write Caching Write caching on Isilon iSCSI LUNs is turned off by default. but at a risk of corruption if a node loses power or crashes while uncommitted data is in the write cache. One example is given in the next section entitled Client Logical Volume Manager. If it is discovered that your client initiator does not protect data in-flight. and turning write caching off once the LUN creation has completed. One exception would be to turn write caching on during the creation of a thickly allocated LUN. it is recommended that this setting stay in the default off setting. Turning Write Caching on can result in write performance improvements. In some cases. Large Server Target: RAID0 LUN: 0 Target: MIRROR0 LUN: 1 Z:\ Logical Volume Manager Mirrored Set Target: RAID0 LUN: 1 Target: MIRROR0 LUN: 2 Target: RAID0 LUN: 2 Target: MIRROR0 LUN: 3 Target: RAID0 LUN: 3 Target: MIRROR0 LUN: 4 RAID 0 Stripe Target: RAID0 LUN: 0 LUN: 1 LUN: 2 LUN: 3 LUN: 4 RAID 0 Stripe Target: MIRROR0 LUN: 0 LUN: 1 LUN: 2 LUN: 3 LUN: 4 Target: RAID0 LUN: 4 Target: MIRROR0 LUN: 0 RAID-0 striped logical volumes will give you higher performance. This is often done for performance. additional protection or both. they could offer less protection since the failure of any one LUN will cause data loss. This will provide for the temporary removal of any LUN (say through the reboot of a node). it is possible with many LVMs to build RAID-0 (Striped) logical volumes by combining multiple LUNs. but they offer no additional protection. then access each individual LUN from a separate node of a five node or larger cluster). To prevent this.G. whereby multiple smaller physical disks or LUNs can be grouped together to form a single larger logical disk. If you wish to have better performance. If you have five LUNs striped into a RAID-0 Logical Volume.Client Logical Volume Manager Many client operating systems support the concept of a Logical Volume Manager (LVM). The best performance for such a RAID-0 logical volume on an Isilon clustered filesystem would be to have each physical LUN accessed by a separate node in the cluster (E. This is often called RAID-10. it is possible to build two RAID-0 logical volumes and then mirror them (RAID-1). thereby keeping the entire logical ISILON SYSTEMS® 6 . Missing Logical Unit Number If you create a LUN within Isilon and set the LUN number allocation to manual. you must reboot the cluster in order to have prefetching disabled. first move each LUN to a new target. in which the IP address belongs to an Isilon SmartConnect™ IP Pool where the IP Allocation method is set to Dynamic. LUNs are Directly Associated with Targets LUNs are directly associated with targets. Very random writes within a large LUN have been shown to benefit from setting the Access Pattern to Streaming. then the client LVM will detect this and rebuild the RAID-1 mirror automatically. then scanning for additional LUNs will not occur and it will appear as if no LUNs exist. Until you are sure that turning off prefetch results in an increase in performance. OneFS Prefetch Occasionally. Moving LUNs can be ® accomplished via the command line (scripted) or via the Isilon web administration interface. To turn off prefetching temporarily.enable_prefetch=0 After you are done adding the line to the file. login to one node of the cluster and type isi_for_array ‘sysctl efs. isi_flush’ To turn off prefetching permanently. Note that the best protection occurs when the LUNs of the two RAID-0 stripes are accessed through offset nodes of the cluster.bam.bam. The previous diagram illustrates how such a RAID-10 logical volume could be built. When the node reboots or its functions are transferred to another node in the cluster. all LUNs assigned to that target will be permanently destroyed as well. login to one node of the cluster and add the following line to the file /etc/mcp/override/sysctl. it is possible that the client initiator will not be able to detect any LUNs. If a LUN with the number of zero does not exist. If you need to delete a target and preserve the LUNs within that target. You can turn off prefetching either temporarily or permanently. ISILON SYSTEMS® 7 . Streaming Access Pattern Setting the Access Pattern on a LUN can result in different performance only within a very narrow workflow. you will have an application that has an access pattern that is so random in nature. You must set the IP Allocation method to Static in order to see the LUNs.conf efs. Some initiators require a LUN with a number of zero before they will scan for any other LUNs with higher numbers. you should only disable prefetch temporarily. then delete the target only when there are no LUNs associated with it. Dynamically Allocated IP Pools result in LUNs not being visible If you create a LUN within Isilon.volume online and available to your users. This protects against node failures taking down the same LUNs in the RAID-1 (mirrored) set. This generally occurs when there is no LUN with a number of zero. If you delete a target. Concurrency vs. that the prefetching that OneFS does with each file can be too aggressive in regards to the overall performance of the application. the iSCSI client/initiator will not be able to see any LUNs. With our current testing. and try to connect to that LUN from an iSCSI Initiator. Concurrency versus Streaming has demonstrated no performance increase or decrease except for this one case.enable_prefetch=0. it is possible to reduce or remove the expense of the rewrite operation for both Thin and Thick LUNs. This method has the advantage of having the highest backup performance. and while another resize operation is still pending. Please see the Best Practices Guide for the MIcrosoft iSCSI Initiator for more information on this method. All three of these types of clones utilize snapshots within OneFS. it has been demonstrated that Thinly Provisioned LUNs may initially have higher performance than Thickly Provisioned LUNs. this improvement goes away once a block has been written once within the thin LUN. Every rewrite within a thin LUN could potentially result in the same read.LUN Provisioning Isilon supports provisioning LUNs as thin or thick. LUNs that are thinly allocated or fully allocated can be resized. have a client initiator backup the contents of the LUN with the appropriate backup software. but has the advantage of allowing the user to recover any specific piece of data within the LUN. However. This performance improvement occurs because every initial write in a thinly provisioned LUN does not result a read. This ensures that all data has been flushed from the client to the LUN prior to the creation of the snapshot. iSCSI with SyncIQ SyncIQ currently does not replicate snapshot information. iSCSI can create three different clone types: Normal. modify and write operation. One method to achieve this in a Microsoft environment would be to use Volume Shadow Service (VSS). ISILON SYSTEMS® 8 . LUN Backup There are several ways to backup the contents of a LUN. Under both methods. 4. I hope this information has been beneficial to you and will allow you to realize potential block-level access use and Unified Storage from Isilon products. As previously written in the sections entitled Optimal Block Size and 2x Protection Recommended. Conclusion This Best Practices guide is focused on helping you configure iSCSI within the OneFS filesystem and achieving the best performance possible. but recovery entails restoring the entire LUN and not just the piece of data that is missing. modify and write operation within the OneFS filesystem. Provisioning a LUN as thin will take less time at creation. Snapshot and Shadow. The Normal clone is the only type of clone that is currently certified to work with SyncIQ. but does not support shrinking of LUNs. This method is just a little bit slower. Once it is assigned to a Target. LUN Resizing Isilon supports resizing of LUNs to be larger than original size. Of course. The size of the LUN can be increased while it is being reserved. One method would be to make a snapshot of LUN and then to backup the snapshot directly from the cluster. The preferred method also involves taking a snapshot and assigning that snapshot LUN to a Target. the Normal clone type only uses a snapshot to start the clone and then discards the snapshot once the cloning has completed. The references below should be helpful and contain more in-depth information of many of the topics discussed in this paper. As previously discussed. During testing. it will be necessary to first quiesce the LUN before taking a snapshot. Other patents pending. U. workflows. SmartConnect. and processes. Patent Numbers 7.675.146. and costeffective way.isilon. FlexProtect. Isilon enables enterprises and research organizations worldwide to manage large and rapidly growing amounts of file-based data in a highly scalable. SnapshotIQ. Other product and company names mentioned are the trademarks of their respective owners.5.720. ® ® ISILON SYSTEMS® 9 .com. SyncIQ are registered trademarks of Isilon Systems.S. 7. Isilon IQ. Inc. Isilon . “HOW BREAKTHROUGHS BEGIN.386. References Isilon Best Practices Guide for the Microsoft iSCSI Initiator OneFS 5_5_4 iSCSI Module Isilon Systems (NASDAQ: ISLN) is the proven leader in scale-out NAS. Information about Isilon can be found at http://www. Isilon Systems. © 2009 Isilon Systems.” and the Isilon logo are trademarks or registered trademarks of Isilon. Isilon clustered storage and data management solutions drive unique business value for customers by maximizing the performance of their missioncritical applications. 7. Autobalance. Inc.346. easy-to-manage.524. TrueScale. SmartCache. All rights reserved. OneFS.