vCenter Performance Counters

VMware Communities: vCenter Performance Countershttp://communities.vmware.com/docs/DOC-5600?decorator=print vCenter Performance Counters Created on: May 29, 2008 5:15 PM by drummonds - Last Modified: Jan 6, 2011 12:35 AM by jeanchen VERSION 19 Introduction The following table of vCenter (VC) performance counters lists the counters with a description of their purpose. This page has been updated for vSphere 4, so the counter levels will differ slightly on older versions of VC. Remember, with the exception of ready time, statistic levels one and two are the only ones needed for 99% of the performance monitoring and analysis out there. Don't spend many of your own cycles worrying about levels three and four! For information on enabling VC to display and archive these counters see the Understanding vCenter Performance Statistics article. Understanding vCenter Measurement Windows Before you continue, you should know that all total count metrics reported by VC are reported over the sample window. When you're looking at live stats, this sample window is 20 seconds. When you're looking at archive stats, it will depend on the interval duration. That duration could be five minutes, 30 minutes, two hours, or one day. This causes a lot of confusion when comparing esxtop results to live VC results to archived VC results. As an example, ready time might be reported as 10% in esxtop. In live VC results this amount of ready time would be reported as 2000 ms (10% of the 20s window.) In one day archive results, the same number would be reported as 30,000 ms (10% of the five minute interval duration.) All of these numbes reflect the same amount of ready time. CPU Statistics Level Counter name in API 1 Description Units Ready time is the time spend waiting for CPU(s) to become cpu.ready.summation millisecond available in the past update interval. The CPU utilization. The maximum possible value here is the frequency of the processors times the number of cpu.usagemhz.average cores. As an example, a VM using 4000 MHz on a system megaHertz with four 2 GHz processors is using 50% of the CPU (4000 / (4 * 2000) = 0.5) The CPU utilization. This value is reported with 100% representing all processor cores on the system. As an cpu.usage.average percent example, a 2-way VM using 50% of a four-core system is completely using two cores. cpu.reservedCapacity.average CPU Reserved Capacity megaHertz cpu.idle.summation CPU Idle millisecond Swap wait time is time that the world spent waiting for cpu.swapwait.summation memory to be swapped in. When the VM is waiting for millisecond memory, it is not doing work. System time is the time spent in VMkernel during the last cpu.system.summation update interval. This does not include guest code millisecond execution. Wait time is the time spent waiting for hardware or VMkernel cpu.wait.summation millisecond lock thread locks during the last update interval. CPU extra is the time above the statically calculated entitlement. Entitlement is the share of processing time that a VM should get as a result of its vCPU count and cpu.extra.summation millisecond assigned shares. You should not use or care about this counter in any of your own analysis. cpu.used.summation CPU Used millisecond 1 1 2 2 2 3 3 3 3 1 of 7 13/11/2011 4:50 PM usagemhz.VMware Communities: vCenter Performance Counters http://communities.none cpu.granted. 2 GHz host. The amount of memory currently claimed by the balloon driver. Shared memory represents the entire pool of memory from which sharing savings are possible. The swap out rate reports the rate at which a kiloBytesPerSecond VM's memory is being swapped out to disk. This is the "true" number of how much memory the VM currently kiloBytes has need of. 1 mem. there are 80.000 ms.average 2 mem. The average amount of shared memory. This is not a performance problem.consumed. but represents the host starting to take memory from less needful VMs for those kiloBytes with large amounts of active memory. Of the 2 GB left. if 2000 MHz have been reserved for the VM on an four-way. that's 25% of the CPU resource. As an example.usagemhz.vmware. it might have kiloBytes only touched half of that. total saving due to memory sharing equals shared memory minus shared common memory.usage.minimum cpu.overhead.000 ms of time has been reserved.average 2 of 7 13/11/2011 4:50 PM . The amount of memory used by the VM in the past small window of time.maximum cpu.com/docs/DOC-5600?decorator=print 3 cpu. CPU Usage (None) CPU Usage (Minimum) CPU Usage (Maximum) CPU Usage in MHz (None) CPU Usage in MHz (Minimum) CPU Usage in MHz (Maximum) millisecond percent percent percent megaHertz megaHertz megaHertz Memory Statistics Level Counter name in API Description units The amount of machine memory that is in use by the VM. While a VM may have been configured to use 4 GB of RAM. In a 20s update interval. half of that might be saved from memory sharing. That means 20.shared.maximum Guaranteed time is reported as the amount of the reservation time that the VM used in the past update interval. per se. The percentage of memory used as a percent of all available machine memory.latest 4 4 4 4 4 4 cpu.vmmemctl. So.000 ms available on this four-way system.usage. unused memory may be swapped out or ballooned with no impact to the guest's performance.average mem.average mem.average 2 mem. But if the host is ballooning.average mem.active. Available for host percent and VM.average 1 mem.average 1 1 1 1 mem. If a VM used only half of its available cycles. The swap in rate reports the rate at which a VM's kiloBytesPerSecond memory is being swapped in from disk. The memory used by the VMkernel to maintain kiloBytes and execute the VM.swapoutrate. Additional. check swap rates (swapin and swapout) which would be indicative of performance problems.none cpu. The amount of memory that this has been condensed kiloBytes to is reported in shared common memory. That would result in 1 GB of consumed memory.guaranteed. as an example.usagemhz.minimum cpu.swapinrate. The amount of memory that was granted to the VM by the host.usage.usage. the guaranteed time is 10.average 2 mem. Memory is not granted to the kiloBytes host until it is touched one time and granted memory may be swapped out or ballooned away if the VMkernel needs the memory. none Memory Usage (None) mem. total saving due to memory sharing equals shared memory minus shared common memory.average Memory Swap Target (Average) The rate at which memory is being swapped in from disk.maximum Memory Granted (Maximum) mem.active.average Memory Heap (Average) mem.swapunreserved.minimum Memory Unreserved (Minimum) mem.none Memory Shared Common (None) kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes number kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes megaBytes percent percent percent kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes 3 of 7 13/11/2011 4:50 PM .swapused.average Memory Balloon Target (Average) mem. mem.latest Memory State mem.swapin.granted.unreserved. swapout) to see if the guest is actively in need of more memory than is available.vmware.average problem with lack of memory and a clear indication that performance is suffering as a result.minimum Memory Active (Minimum) mem.shared.active.heap.maximum Memory Zero (Maximum) mem.minimum Memory Granted (Minimum) mem.none Memory Granted (None) mem.minimum Memory Swap Used (Minimum) mem.average Memory Reserved Capacity mem. mem.none Memory Shared (None) mem. The rate at which memory is being swapped out to disk. mem.maximum Memory Shared (Maximum) mem.average Memory Swapped (Average) mem.minimum Memory Swap Unreserved (Minimum) mem.com/docs/DOC-5600?decorator=print 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 The amount of zero pages in the guest.none Memory Unreserved (None) mem.none Memory Swap Used (None) mem.heapfree.swapunreserved.maximum Memory Usage (Maximum) mem. Zero pages are not represented in machine memory mem.state.average problem with lack of memory and a clear indication that performance is suffering as a result. So.sharedcommon. This could be memory mem.minimum Memory Zero (Minimum) mem.reservedCapacity.unreserved.usage.usage.shared.VMware Communities: vCenter Performance Counters http://communities.zero.minimum Memory Usage (Minimum) mem.maximum Memory Swap Unreserved (Maximum) mem.unreserved.swapused. Shared memory represents the entire pool of memory from which sharing savings are possible. The amount of memory that this has mem.average so this results in 100% savings when mapping from the guest to the machine memory.granted.zero.average Memory Unreserved (Average) The amount of swap memory currently in use.swapped.zero.none Memory Active (None) mem.sharedcommon.average been condensed to is reported in shared common memory.granted.swaptarget.maximum Memory Unreserved (Maximum) mem.vmmemctltarget.average Memory Heap Free (Average) mem.maximum Memory Swap Used (Maximum) mem. A large number here represents a mem.average Memory Used by vmkernel mem.swapused. Check the swap rates (swapin.shared.zero. mem.minimum Memory Shared (Minimum) mem.none Memory Zero (None) mem.sysUsage.maximum Memory Active (Maximum) mem.none Memory Swap Unreserved (None) mem. A large number here represents a mem.active. A large amount of swap memory is not a performance problem.usage.swapunreserved.swapout.unreserved.swapunreserved.swapused.average that the guest doesn't need.average Memory Swap Unreserved (Average) The average amount of shared common memory. minimum mem.overhead.maximum mem.average period. Abort commands are issued by the guest when the storage system disk.consumed.none mem.swapped.maximum mem.swapin.maximum mem. Average disk throughput due to write disk.summation Disk Commands Issued The number of aborts that have occurred in the last window of time.minimum mem.sysUsage.average physical device from the HBA to the platter takes to service an IO request.minimum mem.swapout.overhead.vmmemctltarget.maximum mem.none mem.none mem. This is the time between the guest OS and the device.read.vmmemctl.maximum mem.none mem.minimum mem.sysUsage.heap.swaptarget. Average disk throughput over the sample disk.VMware Communities: vCenter Performance Counters http://communities.consumed.swaptarget.maxTotalLatency kernel times) in the sample window.overhead.heap.none mem.heapfree. This is the time the disk.sharedcommon.none mem.none mem.consumed.minimum mem.average VMkernel takes to service an IO.swapin.minimum mem.minimum mem. Average disk throughput due to read disk.minimum mem.none mem. disk.commandsAborted.sharedcommon.none mem.vmmemctl. Kernel read latency.sysUsage.vmmemctl.deviceReadLatency.swapped.minimum mem.write.com/docs/DOC-5600?decorator=print 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 mem.heapfree.usage.average operaitons over the sample period. units milliseconds kiloBytesPerSecond kiloBytesPerSecond kiloBytesPerSecond number 2 number 2 2 number millisecond 2 millisecond 4 of 7 13/11/2011 4:50 PM .maximum mem.heap.minimum Memory Shared Common (Minimum) Memory Shared Common (Maximum) Memory Heap (None) Memory Heap (Minimum) Memory Heap (Maximum) Memory Heap Free (None) Memory Heap Free (Minimum) Memory Heap Free (Maximum) Memory Swapped (None) Memory Swapped (Minimum) Memory Swapped (Maximum) Memory Swap Target (None) Memory Swap Target (Minimum) Memory Swap Target (Maximum) Memory Swap In (None) Memory Swap In (Minimum) Memory Swap In (Maximum) Memory Swap Out (None) Memory Swap Out (Minimum) Memory Swap Out (Maximum) Memory Balloon (None) Memory Balloon (Minimum) Memory Balloon (Maximum) Memory Balloon Target (None) Memory Balloon Target (Minimum) Memory Balloon Target (Maximum) Memory Overhead (None) Memory Overhead (Minimum) Memory Overhead (Maximum) Memory Consumed (None) Memory Consumed (Maximum) Memory Consumed (Minimum) Memory Used by vmkernel Memory Used by vmkernel Memory Used by vmkernel kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes kiloBytes Disk Statistics Level Counter name in API 1 1 2 2 2 Description The highest reported total latency (device and disk.maximum mem.maximum mem.average operations over the sample period. This is the time the disk.commands.summation has not responded within an acceptable amount of time (as defined by the guest OS or application.none mem.swapout.swaptarget.swapout.maximum mem.maximum mem.swapped.vmmemctltarget.minimum mem.maximum mem.busResets.summation Disk Bus Resets Device read latency.maximum mem.none mem.vmware.kernelReadLatency.swapin.vmmemctltarget.heapfree.minimum mem.) disk. average physical device from the HBA to the platter millisecond takes to service an IO request. disk. Total write latency.average Physical Device Command Latency millisecond disk.summation previous sample period.actav1.usage.queueWriteLatency.received. Note that these number operations may be variable sized up to 64 KB.usage.summation Network Packets Transmitted net.maximum Disk Usage (Maximum) kiloBytesPerSecond Network Statistics Level Counter name in API 1 net.average Memory Swap Out (Average) 3 rescpu.latest Current failover level 3 sys.packetsTx. The number of IO write operations in the disk.average Queue Command Latency millisecond The number of IO read operations in the disk.usage.average millisecond kernel and device latency for both read and write commands. average) units second number number number megaHertz megaBytes number megaHertz kiloBytes kiloBytes kiloBytesPerSecond kiloBytesPerSecond percent 5 of 7 13/11/2011 4:50 PM .transmitted. This is the time the disk.average Resource CPU Usage (Average) 3 managementAgent.usage.maximum Network Usage (Maximum) units kiloBytesPerSecond number number kiloBytesPerSecond kiloBytesPerSecond number number kiloBytesPerSecond kiloBytesPerSecond kiloBytesPerSecond Other Statistics Level Counter name in API Description 1 sys.none Network Usage (None) net.average millisecond kernel write latencies.queueReadLatency.summation previous sample period. The sum of the device and millisecond kernel read latencies.usage.average Queue Read Latency millisecond Device write latency. Kernel write latency.summation over the sample period. This is the millisecond time between the guest OS and the device.swapIn.minimum Disk Usage (Minimum) kiloBytesPerSecond disk.latest Memory Fairness 1 clusterServices.usage.totalWriteLatency.summation Network Packets Received net.average Kernel Disk Command Latency millisecond disk.average Average network throughput for received traffic.heartbeat.write.usage.summation over the sample period.deviceLatency.average Average network throughput for transmitted traffic.kernelWriteLatency.numberWrite.memfairness.average Queue Write Latency millisecond disk.swapOut.average Disk Write Rate kiloBytesPerSecond disk. This is the average total latency over the sample window.uptime.average 2 2 2 2 3 3 4 4 4 Description Network Usage (Average) The number of received packets that were dropped net.effectivecpu.com/docs/DOC-5600?decorator=print 2 2 2 disk.average Memory Used (Average) 3 managementAgent.droppedTx.droppedRx.packetsRx. disk.cpufairness.queueLatency.effectivemem.deviceWriteLatency.latest CPU Fairness 1 clusterServices.average 2 2 2 2 2 2 3 3 3 3 4 4 4 Total read latency.minimum Network Usage (Minimum) net.latest Uptime 1 sys.average Memory Swap Used (Average) 3 managementAgent.average VMkernel takes to service an IO. Total latency is the sum of disk.VMware Communities: vCenter Performance Counters http://communities. net. disk.average Effective CPU Resources 1 clusterServices.numberRead. This is the time the disk.none Disk Usage (None) kiloBytesPerSecond disk. net.totalLatency.vmware.failover. net.average Effective Memory Resources 1 clusterServices.summation Heartbeat 1 clusterServices. The number of transmitted packets that were dropped net.average Memory Swap In (Average) 3 managementAgent.swapUsed.kernelLatency. Note that these number operations may be variable sized up to 64 KB.totalReadLatency.latest CPU Active (1 min.memUsed.resourceCpuUsage. The sum of the device and disk. average) percent CPU Active (5 min.latest rescpu.actpk5. average) percent CPU Active (15 min.minimum CPU Active (1 min.latest rescpu. average) percent CPU Running (1 min.latest rescpu.maxLimited15.maxLimited1.latest rescpu.latest rescpu. disk .sampleCount.latest rescpu.latest rescpu. storage .runav15. it means that for 2000 ms of the previous 20.runav1.none sys.runpk5. peak) percent CPU Throttled (1 min. peak) percent CPU Running (5 min.latest rescpu.latest rescpu. average) percent CPU Active (15 min. So.latest rescpu.resourceCpuUsage. average) percent Group CPU Sample Count number Group CPU Sample Period millisecond Resource CPU Usage (None) megaHertz Resource CPU Usage (Maximum) megaHertz Resource CPU Usage (Minimum) megaHertz 36.actpk1.2 / 5 (5 ratings) Comments (7) Feb 4.maximum sys. cpu . peak) percent CPU Running (1 min.runpk15. average) percent CPU Active (5 min. when "ready time" reports a number of 2000 ms.resourceCpuUsage.latest sys.resourceCpuUsage.vmware.runpk1.actav15.actav5. 6 of 7 13/11/2011 4:50 PM . peak) percent CPU Throttled (15 min. average) percent CPU Running (15 min. average) percent CPU Running (5 min.maxLimited5.000 ms sample period the vCPU was ready to run and not getting resources.com/docs/DOC-5600?decorator=print 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 rescpu.latest rescpu.latest rescpu.VMware Communities: vCenter Performance Counters http://communities. memory . the 20 second interval means "the average of the last 20 seconds" or "the value on a particular second.latest rescpu. peak) percent CPU Running (15 min. 2009 9:56 PM Iwan Rahabok says: Hi Scott.latest rescpu.685 Views Tags: virtualcenter .actpk15. 2009 9:29 PM Iwan Rahabok says: Great job Scott! Feb 4. 2009 1:07 PM drummonds says in response to Iwan Rahabok: The 20-second interval means that the values recorded were accrued or averaged over 20 seconds.samplePeriod.latest rescpu. with interval taken every 20 second"? Feb 5.runav5. network . analysis Average User Rating 4. peak) percent CPU Throttled (5 min. com/docs/DOC-5600?decorator=print Apr 6. a VM using 4000 MHz on a system with four 2 GHz processors is using 50% of the CPU (4000 / (4 * 2000) = 0.average You write: "The CPU utilization. 2009 8:37 AM olegarr says: Scott. The maximum possible value of the sum of cpu.5 times more powerful).even when they are running at maximum load. As an example.Submit product feedback © Jive Software 7 of 7 13/11/2011 4:50 PM ." I'd say: "The CPU utilization in percent.vmware. This also explains why you set reservations and limits in MHz and not in %" VMware Communities powered by Jive SBS ® 4. Nov 18.5 times higher (VMware assumes these systems are 1. metric yyy never should exceed limit 321. As an example. I have not seen this.average You write: "The CPU utilization. olegarr Apr 9. there is no document today that provides this guidance. this is a great question and one that we've been pondering a bit lately.13 community software . Sep 22. We're looking into building something like this now but would like to back it with a deep investigation using data from real deployments.VMware Communities: vCenter Performance Counters http://communities. and rather stick to the MHz values reported. 2010 12:50 PM heitorvital says in response to drummonds: Any news about guidance from VMware on thresholds? I´m looking for this kind of data. some clarifications: cpu. The maximum possible value here is the frequency of the processors times the number of cores. 2010 8:17 AM frank_wegner says: Scott. For this reason I would be careful with percentages.5)" I'd say: "The CPU utilization. a 2-way VM using 50% of a four-core system is completely using two cores. It'll take us some time. So if each VM is running at 100% utilization the corresponding MHz values add up to this capacity. There is a good deal of demand for guidance from VMware on thresholds for these metrics to advise customers of "yellow" and "red" levels for these counters. The maximum possible value for a single VM is the frequency of the processors times the number of vCPUs of the VM. It’s great document and I am using it (along with another your docs) all the time… Just 1 quick question: is there any doc available that will give some kind of guidance for most important metrics? Something like that: in normal condition parameter xxx should be not more than 123.usagemhz. For hyperthreaded systems the 100% mark of the ESX host is 1.0. Example: Assume you have an ESX host with 8 cores and 2 GHz each. I hardly see VMs with more than 75% utilization on a hyperthreaded systems . otherwise… Thanks. this means the host has a capacity of 16 GHz.usage." cpu. This value is reported with 100% representing all processor cores on the system. 2009 3:45 PM drummonds says in response to olegarr: No.usagemhz of all VMs on one ESX host is the frequency of the processors times the number of cores of that host. For non-hyperthreaded systems this is 100%. However.