Teradata 2580d BYNET ConfigurationNov 4, 2009 RESTRICTED PROGRAM 2009 Teradata and Restricted 2 .Teradata 2580 Knowledge Transfer Session BYNET Agenda • Overview • Switch Configuration • Node Configuration Nov 4. 20.Teradata 2580 Knowledge Transfer Session BYNET Overview – Hardware • For 2580. Nov 4. > Port 4 “eth3” is connected to BYNET 0. SMWeb shares the same switch with BYNET • BYNET Gigabit Ethernet Switches are installed in positions 1 and 20 in the 2580 base cabinet. • The following cable lengths have been certified for BYNET: 5. In the base cabinet. dual-active Dell PowerConnect 6248 48-Port Gigabit Ethernet Switches. > Port 3 “eth2” is connected to BYNET 1. • All connections are copper (1000BASE-T). 2009 Teradata and Restricted 3 . and 30 meters. 10. > chassis 1 = BYNET 0 > chassis 20 = BYNET 1 • 2580 DELL processing node on-board Ethernet interfaces are connected to the BYNET Gigabit Ethernet Switches. BYNET switching is provided by redundant. > The Dell PowerConnect 6248 is a managed switch > It is managed by SMWeb. 2009 Teradata and Restricted 4 .Teradata 2580 Knowledge Transfer Session BYNET Overview – Dell R710 node 1 ¾-Size PCIe X4 slot and 1 ½-Size PCIe X4 slot on the Center PCI Riser 2 ¾-Size PCIe X8 slots on the Left PCI Riser Rear View 1 2 3 4 Server Management connections (1GbE) Port 1 = eth0 Port 2 = eth1 BYNET connections (1GbE) Port 3 = eth2 = BYNET 1 Port 4 = eth3 = BYNET 0 Nov 4. 2009 Teradata and Restricted 5 .Teradata 2580 Knowledge Transfer Session BYNET Overview – Cabling for 1-2 Cliques BYNET 1 BYNET 0 Nov 4. > Should not normally be used by external clients or utilities. Reconcile. update firmware. psh. • In the event of a cable or switch failure. and other common Teradata network utilities.Teradata 2580 Knowledge Transfer Session BYNET Overview – IP Subnets • BYNET IP Subnet > Used by PUT. • BYNET 0 Gigabit Ethernet IP Subnet > Used internally by BYNET Protocol Emulation to poll the BYNET 0 Gigabit Ethernet Switch’s management port. Note that these three subnets are logically independent! Nov 4. 2009 Teradata and Restricted 6 . traffic is automatically rerouted. > Can be used to access the BYNET 0 Gigabit Ethernet switch to configure flow control. • Traffic is load-balanced between available switches. > The BYNET IP subnet is a virtual subnet. pcl. etc… • BYNET 1 Gigabit Ethernet IP Subnet > Same story as BYNET 0. Packets are routed by BYNET Protocol Emulation with the BYNET Gigabit Ethernet Switches providing the physical transport. 1 Node addresses: 39. • BYNET IP Subnet: > > > Default: Class A Network ID 39.<y>.<z> • <x> = 32 + (node collective number / 32) • <y> = 8 * (node collective number % 32) • <z> = node chassis number • BYNET 1 Gigabit Ethernet IP Subnet: > > > > > Nov 4.1 Node addresses: 39.240.0.<x>.<y>.0.0 Switch management address: 39.<x>.0.0.16.0 Configured by Server Management via /etc/opt/teradata/sm3g/smainfo.<z> • <x> = 16 + (node collective number / 32) • <y> = 8 * (node collective number % 32) • <z> = node chassis number Configured manually Uses the otherwise unused System 0 SM3G Secondary IP Subnet Subnet mask: 255.Teradata 2580 Knowledge Transfer Session BYNET Overview – IP Address Conventions • All IP Subnets are based on the system’s SM3G IP configuration. Example: SMP001-8:~ # cat /etc/opt/teradata/sm3g/smainfo PMA 40 BYNETIP 39.0.32.0 Switch management address: 39. 2009 Teradata and Restricted 7 .240.0 • <x> = 64 + (node collective number / 32) • <y> = 8 * (node collective number % 32) • <z> = node chassis number • BYNET 0 Gigabit Ethernet IP Subnet: > > > > > Configured manually Uses the otherwise unused System 0 SM3G Primary IP Subnet Subnet mask: 255.0.<z> BYNETMASK 255.0.240.<y>.<x>. Teradata 2580 Knowledge Transfer Session BYNET Agenda • Overview • Switch Configuration • Node Configuration Nov 4. 2009 Teradata and Restricted 8 . Use the 2550 configuration procedure for configuring the two Dell 6248 switches. Nov 4. 2009 Teradata and Restricted 9 .Teradata 2580 Knowledge Transfer Session BYNET Switch Configuration • The switch configuration for the 2580 is identical to the 2550. 2009 Teradata and Restricted 10 .Teradata 2580 Knowledge Transfer Session BYNET Agenda • Overview • Switch Configuration • Node Configuration Nov 4. 2009 Teradata and Restricted 11 . BYNET driver version 3.2.1.2.1.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration • The Broadcom Gigabit Ethernet driver (bnx2) is used on the 2580 for the BYNET ports.2 (or later) > Configure eth2 and eth3 > > > > Make eth2 and eth3 mandatory Create /etc/opt/teradata/bynet/netdev Add BYNET host file entries Install BYNET software > Reboot > Verify that BYNET is online > Verify access to the Ethernet Management Interface Nov 4.2 is required for supporting this configuration • The following steps are required to configure BYNET on a 2580 node: > Install BYNET 3. then: x = 16 + (100/32) = 16 + 3 = 19 y = 8 * (100%32) = 8 * 4 = 32 z=5 Therefore. create an /etc/sysconfig/network/ifcfg-eth2 file: STARTMODE=„onboot‟ MTU=„‟ BOOTPROTO=„static‟ IPADDR=39.240.<z> NETMASK=255. 2009 Teradata and Restricted 12 .0 <x> = 32 + (node collective number / 32) <y> = 8 * (node collective number % 32) <z> = node chassis number • The configuration updates will be picked up on the next reboot. the IPADDR = 39.<x>.240.<x>. if the node collective number =100 and the node chassis number =5.0.32. create an /etc/sysconfig/network/ifcfg-eth3 file: STARTMODE=„onboot‟ MTU=„‟ BOOTPROTO=„static‟ IPADDR=39. • To configure eth3. Nov 4.<z> NETMASK=255.5 • To configure eth2.<y>.0 <x> = 16 + (node collective number / 32) <y> = 8 * (node collective number % 32) <z> = node chassis number For example.19.<y>.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Configure eth2 and eth3 • eth2 and eth3 need to be configured with static IP addresses.0. Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Make eth2 and eth3 mandatory devices • The eth2 and eth3 devices must be marked as mandatory to guarantee that they are available when the BYNET startup script (/etc/init.d/bynet) runs. • To mark eth2 and eth3 as mandatory. Nov 4. 2009 Teradata and Restricted 13 . edit /etc/sysconfig/network/config and add eth2 and eth3 to the list of mandatory devices: MANDATORY_DEVICES=“eth2 eth3” • The mandatory attribute will be picked up on the next reboot. and it needs to be given the IP addresses of the Gigabit Ethernet switch management ports. “eth3” must appear first since it is used for BYNET 0. The 2580 uses protocol emulation and thus requires this netdev file. 2009 Teradata and Restricted 14 .0. the BYNET driver will operate in native BYNET mode. • Create the /etc/opt/teradata/bynet directory if it doesn’t already exist: # mkdir –p /etc/opt/teradata/bynet • Create an /etc/opt/teradata/bynet/netdev file containing the interface and switch IP information. the BYNET driver will operate in protocol emulation mode. If no /etc/opt/teradata/bynet/netdev file is present. eth3 dell6248 39. > If an /etc/opt/teradata/bynet/netdev file is present.0.32. it needs to be told which interfaces to use.16.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Create /etc/opt/teradata/bynet/netdev • The BYNET driver needs to be configured to operate in protocol emulation mode. Note that the order of these lines is important. Nov 4.1 eth2 dell6248 39.1 • The updated configuration will be picked up on the next reboot. Nov 4.64. a BYNET entry should be added to /etc/hosts for each node in the system. 2009 Teradata and Restricted 15 .64.24.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Add BYNET host file entries • For convenience.9 39.16.9 byn001-8 byn001-9 byn002-8 byn002-9 byn003-8 byn003-9 byn004-8 byn004-9 • You may also want to add cop entries for Teradata.16.64.64.8.64. • Here are example BYNET host file entries for an eight node system: 39.8.64.32.32.8 39.24.9 39.64.8 39.8 39.9 39.8 39. • /etc/hosts changes take effect immediately.64. Nov 4. • The latest x86_64 BYNET rpm should be downloaded from TSS. exit any active shells and start fresh ones to ensure that the BYNET user utilities will be found in your PATH. • After BYNET has been installed.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Install BYNET Software • The SM3G node software must be installed before the BYNET software to ensure proper PMA ID and node type assignment. • Use PUT's Install/Upgrade Software operation to install the rpm. 2009 Teradata and Restricted 16 . For example: # bam -s Node is running in protocol emulation mode. Node state: attached Net # 0 1 Nodes 4 4 Routes 4 4 Net name eth3 eth2 > Run netecho and verify that all nodes respond. reboot the node so configuration changes can take effect. • After the node reboots. 2009 Teradata and Restricted 17 . For example: # netecho byn001-8 byn001-9 byn002-8 byn002-9 Nov 4.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Reboot and verify that BYNET is online • After all the configuration steps have been completed. check to make sure that the BYNET is functional: > Run bam –s and verify that the correct number of interfaces and nodes are online. 32.0.0.1: icmp_seq=1 ttl=64 time=0.1) 56(84) bytes of data.678 ms # ping 39.1: icmp_seq=2 ttl=64 time=0.0.1 (39.0.32.16.667 ms 64 bytes from 39.1) 56(84) bytes of data.1 PING 39. verify that the nodes can ping each switch > Run ping <switch IP> to confirm that the switches are configured with the correct IP address and can be reached from the node.16. 64 bytes from 39. After configuration is complete.1 (39.0.0.Teradata 2580 Knowledge Transfer Session BYNET Node Configuration Verify access to the Ethernet Management Interface • BYNET accesses the management interface of the Dell switches to determine the state of the switches. 2009 Teradata and Restricted 18 .667 ms 64 bytes from 39.16.1: icmp_seq=2 ttl=64 time=0.32.0.16.0.32.0.1 PING 39.16.0.1: icmp_seq=1 ttl=64 time=0. 64 bytes from 39.678 ms Nov 4.32. For example: # ping 39.