5 Things You Dont Know About AWS

5things you don’t know about Amazon Web Services Simone Brunozzi Sr. Technology Evangelist, AWS @simon 1 AWS CLI 01 Delete a bucket and all its content with just one command 2 3 aws s3 rb s3://bucket-name --force 4 . 5 . 6 . AWS CLI 02 Recursive copy of a directory and its subfolders to Amazon S3 (and vice versa) 7 . 8 . aws s3 cp MyFolder s3://bucket-name -recursive [--region us-west-2] 9 . 10 .Autoscaling 03 Hmm... S3 DynamoDB ELB (Elastic Load Balancing) Glacier EB (Elastic Beanstalk) 11 .Autoscaling Amazon Amazon Amazon Amazon Amazon Etc. EC2 Autoscaling 03 Automatically scale out EC2 within ELB (old version) 12 . AutoScaling (AS): Four main components Get the AS Command Line Tools: aws.com/developertools/2535 1) Launch config: AMI to be used 2) Autoscaling group: where/how to launch 3) Autoscaling policy: what should AS do? 4) Autoscaling trigger: what will activate AS 13 13 .amazon. 14 AutoScaling tools 14 . 15 CloudWatch API Tools 15 . %AWS_CLOUDWATCH_HOME%\bin 16 16 .%AWS_AUTO_SCALING_HOME%\bin.pem set EC2_CERT=c:\ec2\509Certificate.%EC2_HOME%\bin set JAVA_HOME=C:\Program Files\Java\jre6 "%JAVA_HOME%\bin\java" -version set JAVA_HOME=C:\Program Files (x86)\Java\jre6 (Windows 7) set EC2_PRIVATE_KEY=c:\ec2\PrivateKey.Windows setup FILE: ec2.pem set AWS_AUTO_SCALING_HOME=c:\ec2\AutoScaling set AWS_CLOUDWATCH_HOME=c:\ec2\CloudWatch set PATH=%PATH%.bat @echo off set EC2_HOME=c:\ec2 set PATH=%PATH%. bash_profile export EC2_HOME=~/ec2 export PATH=$PATH:$EC2_HOME/bin export EC2_PRIVATE_KEY=pk-[removed].framework/Home/ export AWS_AUTO_SCALING_HOME=~/ec2/AutoScaling export AWS_CLOUDWATCH_HOME=~/ec2/CloudWatch export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin:$AWS_CLOUDWATCH_HOME/bin 17 17 .pem export JAVA_HOME=/System/Library/Frameworks/JavaVM.Linux setup FILE: .pem export EC2_CERT=cert-[removed]. 61.Test if it works % as-version Amazon AutoScaling CLI version 1.0.3 (API 2011-01-01) 18 18 . any running EC2 as-describe-auto-scaling-groups --headers as-describe-auto-scaling-instances --headers 19 19 .small [--region us-east-1] --group SG STEP 2: Autoscaling with ELB as-create-auto-scaling-group ASG1 --launch-configuration LC1 --availabilityzones us-east-1b us-east-1c --min-size 2 --max-size 20 --load-balancers aws201 Check if group is created. 2) STEP 1: Create Launch Config as-create-launch-config LC1 --image-id ami-31814f58 --instance-type m1.AutoScaling + ELB (1. AutoScaling + ELB “Emergency break” Scale in EC2 instances to ZERO. if needed as-update-auto-scaling-group ASG1 --min-size 0 --max-size 0 20 20 . AutoScaling + ELB (3) STEP 3: Create Scaling Out Policy as-put-scaling-policy SOut1 --auto-scaling-group ASG1 --adjustment=1 --type ChangeInCapacity --cooldown 120 arn:aws:autoscaling:useast-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca lingGroupName/ASG1:policyName/SP1 21 21 . AutoScaling + ELB (4) STEP 4: Create Metric Out Alarm mon-put-metric-alarm MOutA1 --comparison-operator GreaterThanThreshold -- evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" -period 60 --statistic Average --threshold 70 --alarm-actions arn:aws:autoscaling:useast-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca lingGroupName/ASG1:policyName/SOut1 --dimensions "AutoScalingGroupName=ASG1" 22 22 . AutoScaling + ELB (5) STEP 5: Create Scaling In Policy as-put-scaling-policy SInP1 --auto-scaling-group ASG1 --adjustment=-1 --type ChangeInCapacity --cooldown 120 arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417fb025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1 If you are using Windows. wrap the --adjustment parameter in quotation marks: "--adjustment=-1". 23 23 . AutoScaling + ELB (6) STEP 6: Create Metric In Alarm mon-put-metric-alarm MInA1 --comparison-operator LessThanThreshold -evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" -period 60 --statistic Average --threshold 30 --alarm-actions arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417fb025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1 --dimensions "AutoScalingGroupName=ASG1" 24 24 . 254/latest/meta-data/instanceid) aws_region=$(wget -q -O.html 25 25 .254/latest/meta-data/hostname) aws_region=${aws_region#*. at boot #!/bin/bash aws_instance=$(wget -q -O.http://169.*} aws_zone=`ec2-describe-instances $aws_instance --region $aws_region` aws_zone=`expr match "$aws_zone" ".169.} aws_region=${aws_region%%.Script you can use within each EC2 instance.254.254.169.http://169.*\($aws_region[a-z]\)"` print "Instance: $aws_instance<br>Region: $aws_region<br>Availability Zone: $aws_zone<br><br>END" > index. EC2 Autoscaling 03 Automatically scale out EC2 within ELB (NEW version) 26 . 27 . EC2 metadata 04 How to get details related to the EC2 machine that I’m running (PHP) 28 . 29 . 30 . VPC + Cloudformation 05 Using CF to create a 3tier architecture within VPC. (first steps) 31 . x/ x. "Tags" : [ {"Key" : "Application".3})\\." ! ! } }.(\\d{1.0.3})\\. ! ! ! "Type" : "String".0. ! ! ! "MaxLength": "18".3})/(\ \d{1. ! ! ! "Default" : "10. ! ! ! "MinLength": "9".x. ! "Parameters" : { ! ! "VpcCidr" : { ! ! ! "Description" : "CIDR for the VPC". "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}.3})\\. ! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC". "Value" : {"Ref" : "AWS::StackId"} } ] } } ! } } 32 .(\\d{1.x.2})".(\\d{1.{ "AWSTemplateFormatVersion" : "2010-09-09". ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.0/16". ! ! ! "AllowedPattern" : "(\\d{1. ! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture". 2})".0. "Value" : {"Ref" : "AWS::StackId"} } ] } } ! } } Do you hate me? Too small? 33 .3})\\. ! ! ! "MaxLength": "18".0/16"." ! ! } }.(\\d{1. ! ! ! "Default" : "10. ! ! ! "AllowedPattern" : "(\\d{1.x. ! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC".3})\\.x. "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}.(\\d{1. "Tags" : [ {"Key" : "Application".3})\\.(\\d{1. ! "Parameters" : { ! ! "VpcCidr" : { ! ! ! "Description" : "CIDR for the VPC". ! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture".{ "AWSTemplateFormatVersion" : "2010-09-09". ! ! ! "MinLength": "9". ! ! ! "Type" : "String".0. ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x/ x.3})/(\ \d{1. 3})\\. ! ! ! "MaxLength": "18". "Value" : {"Ref" : "AWS::StackId"} } ] } } ! } } 34 .0/16". ! ! ! "Type" : "String".0. "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}.0. ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x. ! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC". "Tags" : [ {"Key" : "Application". ! "Parameters" : { ! ! "VpcCidr" : { ! ! ! "Description" : "CIDR for the VPC".{ "AWSTemplateFormatVersion" : "2010-09-09".(\\d{1.3})\\.(\\d{1.x/ x. ! ! ! "MinLength": "9". ! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture"." ! ! } }. ! ! ! "Default" : "10.3})\\.3})/(\ \d{1.2})". ! ! ! "AllowedPattern" : "(\\d{1.x.(\\d{1. ! ! ! "Type" : "String". ! ! ! "AllowedPattern" : "(\\d{1. "Tags" : [ {"Key" : "Application". ! ! ! "MaxLength": "18". "Value" : {"Ref" : "AWS::StackId"} } ] } } ! } } 35 .x/ x. "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}.3})\\.(\\d{1. ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.(\\d{1. ! "Parameters" : { ! ! "VpcCidr" : { ! ! ! "Description" : "CIDR for the VPC".2})".x.0.0.3})\\. ! ! ! "Default" : "10.(\\d{1." ! ! } }.0/16".{ "AWSTemplateFormatVersion" : "2010-09-09".3})\\. ! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC". ! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture". ! ! ! "MinLength": "9".x.3})/(\ \d{1. Value=temp" --template-file stack_cfn_learn_001_vpc_step1.template cfn-describe-stacks cfn-describe-stacks CustomerLittleNetwork cfn-list-stack-resources CustomerLittleNetwork 36 .cfn-create-stack CustomerLittleNetwork --tag "Key=env. 37 . 38 . template cfn-describe-stacks CustomerLittleNetwork cfn-list-stack-resources CustomerLittleNetwork 39 .cfn-update-stack CustomerLittleNetwork --template-file stack_cfn_learn_001_vpc_step2. { "AWSTemplateFormatVersion" : "2010-09-09". ! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture". ! ! "Parameters" : { ! "VpcCidr" : { 40 . 3})\\.0/16". ! ! ! "MinLength": "9"."Parameters" : { ! ! "VpcCidr" : { ! ! ! "Description" : "CIDR for the VPC".2})".x/x. ! ! ! "Default" : "10.0. ! ! ! "AllowedPattern" : "(\\d{1.(\\d{1. ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x." ! ! }.3})\\. ! ! ! "Type" : "String".x.x.(\ \d{1. ! ! ! "MaxLength": "18".0.3})/(\\d{1.3})\\.(\\d{1. 41 . x.x.(\\d{1.0/24".3})\\. ! ! ! "MinLength": "9".3})\\.1.0.(\\d{1.3})\\. ! ! ! "Type" : "String". ! ! ! "AllowedPattern" : "(\\d{1. 42 .! ! "PublicSubnetCidrA" : { ! ! ! "Description" : "Subnet for public resources in the first AZ". ! ! ! "MaxLength": "18"." ! ! }.3})/(\\d{1. ! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.(\ \d{1.2})".x/x. ! ! ! "Default" : "10. 43 . CloudFormer 06 CloudFormation templates from an existing deployment 44 . 45 . Custom errors 07 Enabling custom error pages on CloudFront 46 . 47 . 48 . 49 . ZFS on Amazon EC2 08 How to install and run ZFS on Amazon EC2 (yes. you can!) Thanks to: Constantin Gonzalez Dougal Ballantyne (demo coming soon) 50 . protection against data corruption .snapshots and copy-on-write clones .continuous integrity checking and automatic repair .RAID-Z . 51 .support for high storage capacities .integration of filesystem and volume management .native NFSv4 ACLs. Features: . OpenZFS is an Open Source implementation of it.What is ZFS? ZFS is a file system + logical volume manager. How can I use it? Launch an EC2 instance with an AMI which supports ZFS. 1) Linux 2) OmniOS (native ZFS support based on a fork of the original OpenSolaris project that open-sourced ZFS) 3) FreeBSD (native ZFS support based on a port of the OpenSolaris ZFS code) 52 . 1) Linux Linux AMI: https://aws.com/amazon-linux-ami/ http://zfsonlinux.org/ 53 .amazon. 2) OmniOS 54 3) FreeBSD 55 And now what? It works the same as with any other hardware platform: 1) Attach a few EBS volumes to your EC2 instance; 2) use them as you would use regular disks with ZFS An example follows. 56 $ sudo modprobe zfs [attach 3 EBS volumes to the instance] $ sudo zpool create test raidz sdf sdg sdh $ sudo zpool status pool: test state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 errors: No known data errors 57 . gid=5.6G 6.xattr) $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.noatime) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw.9G 1.3G 20% / tmpfs 15G 0 15G 0% /dev/shm test 196G 0 196G 0% /test 58 .$ mount /dev/xvda1 on / type ext4 (rw.mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) test on /test type zfs (rw. What can I do with ZFS? Many things! Here’s one example: Use an EBS Provisioned IOPS volumes as ZFS cache and log (ZIL) devices to accelerate your zpools. 59 . This works much like you would use SSDs in the real world to accelerate ZFS reads and writes. For instance. you could provision an EBS volume with 1000 IOPS and use it as a ZIL device for your zpool in order to accelerate database writes. Learn more http://pthree.org/2012/04/17/install-zfs-on-debiangnulinux/ 60 . 61 . etc. nice people.Extra stuff 09 Friends. cool things. 62 . 63 . 64 . 65 . 66 . Thank you! 67 . AWS @simon 68 .5 things you don’t know about Amazon Web Services Simone Brunozzi Sr. Technology Evangelist.