https://support.oracle.com/CSP/main/article?cmd=show&id=225349.1&t...Implementing Address Windowing Extensions (AWE) or VLM on Windows Platforms [ID 225349.1] Modified 01-JUL-2009 Type BULLETIN Status PUBLISHED PURPOSE ------To address the growing need for use of more memory on 32-Bit Windows platforms, and explain how AWE is implemented by Oracle on Windows. SCOPE & APPLICATION ------------------Oracle DBA's running on the Microsoft Windows platform. Oracle Support Analysts, Field Engineers troubleshooting problems related to AWE and/or memory issues on Windows. AWE Memory implementation on Windows 2000 -----------------------------------------A common question on the Windows NT/Windows 2000 platform revolves around how to take advantage of systems with more than 4 GB of RAM. As discussed in Metalink NOTE:46001.1 and NOTE:46053.1, the 32-Bit process address space for any process on Windows equates to a total of 4GB of addressable RAM. Of this, by default, 2GB is reserved for the process itself, and 2GB for the kernel. On systems running either Windows 2000 Advanced Server, or Windows NT 4.0 Enterprise Edition, this ratio can be changed by adding the /3GB switch to the boot.ini, allowing a process to address 3GB and reserving 1GB for the kernel. However, the total addressable memory for a single process is still only 4GB. See also Note:1036312.6 : Utilizing Up to 3GB Virtual Memory on Windows NT Server 4.0 __________________________________________________________________ What can be done to address memory beyond 4GB?: =============================================== The answer is to take advantage of Physical Address Extensions (PAE), or Address Windowing Extensions (AWE)(These two terms are used interchangeably, so the rest of this document will refer to this simply as AWE). AWE support is available if you are running on a machine with more than 4GB of physical RAM which is running any of the below Windows operating systems: * * * * Windows Windows Windows Windows 2000 2000 2003 2003 Datacenter Server Advanced Server Data Center Edition (32-Bit) Enterprise Edition (32-Bit) On the above operating systems, AWE support is built into the OS. No special drivers are needed to take advantage of the additional memory. AWE CANNOT be used on the following Operating Systems: * * * * * * Windows Windows Windows Windows Windows Windows 2000 Server (Standard) 2000 Professional XP Home Edition XP Professional 2003 Standard Edition 2003 Web Edition NOTE Also that on 64-Bit Windows operating systems, there is no need for AWE implementation support, because the directly addressable memory for a single process on 64-Bit Windows is 8 Terabytes. __________________________________________________________________ Oracle versions that can use AWE: ================================= Oracle can take advantage of AWE in the following 32-Bit RDBMS releases: * * * * * Oracle Oracle Oracle Oracle Oracle 8.1.6.x 8.1.7.x 9.2.x 10.1.x 10.2.x 1 de 8 20/05/2010 12:06 p.m. Operating System Privileges Needed at the OS Level: ==================================================== In order to take advantage of the additional memory afforded through PAE. Add the appropriate user and click 'Ok'. Refer to BUG#2945011 for more detail. It is important to note that once either or both of these switches are in place in the boot. product=RDBMS. and will be fixed in 9. To rectify this.2. click on 'Domain Security Policy' instead of 'Local Security Policy') Double-click on the 'Lock Pages in memory' policy. you will need to choose between the two. but that is no longer the case on Windows 2000. .oracle.x AWE support is available on both the Enterprise Edition of Oracle and the Standard Edition of Oracle. but VLM functionality is still not enabled.0.2.. you will not receive the above errors. you must grant the 'Lock pages in memory' privilege to the user that the OracleService<SID> starts as.0. if you change the OracleService<SID> to logon as a user OTHER than LocalSystem.2 and 9. you may receive the following error if trying to start the database with USE_INDIRECT_DATA_BUFFERS=TRUE: ORA-439 . there was the concept of pointers pointing to the extended memory area.ini as such: multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /PAE It IS possible to have BOTH the /PAE and /3GB switch in place on the same machine.0. However.m.1. With VLM on Windows NT 4.ini. By default. ALL processes running can take advantage of these switches. This BUG is fixed in 9. If you are working with a server with more than 16GB of RAM. Restart the OracleService<SID> __________________________________________________________________ Understanding the Oracle implementation of AWE support: ======================================================= What the PAE switch allows you to do from the Oracle perspective is to increase the amount of memory that can be used for the Oracle Database Block Buffer Cache.2.0.FEATURE NOT ENABLED: VERY LARGE MEMORY In Standard Edition 9.2. in a case where multiple Oracle instances are running on the same server. ALL instances can take advantage of the additional memory afforded by these switches. the operating system user account which is used to start the OracleService<SID> must be granted the 'Lock Pages in Memory' system privilege at the operating system level. 2 de 8 20/05/2010 12:06 p.0. up to the amount of physical memory on the box.0.. However. It is important to note that this additional memory can ONLY be used by Oracle in the form of an increased value for DB_BLOCK_BUFFERS. the OracleService<SID> starts as the LocalSystem account.1&t. you may see the following errors when attempting to start the database with USE_INDIRECT_DATA_BUFFERS set to TRUE : SQL> startup pfile=c:\temp\initscott. as such: multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /3GB /PAE However.2.3. There is still confusion on the old style of VLM versus AWE on Windows 2000.https://support.1.ora ORA-27102: out of memory OSD-00010: Message 10 not found.4 as well. be aware that if BOTH switches are in place.com/CSP/main/article?cmd=show&id=225349. on Standard Edition of 9. the server will only be able to recognize up to 16GB of RAM. Thus. click on: Start -> Programs -> Administrative Tools -> Local Security Policy (on a Domain Controller.3 Patch 2. facility=SOSD O/S-Error: (OS 1300) Not all privileges referenced are assigned to the caller.0. __________________________________________________________________ Enabling support at the OS level: ================================== AWE can be enabled at the OS by adding the /PAE switch to the boot. The LocalSystem account has the privilege to Lock Pages in Memory granted to it by default. To do this. Oracle does NOT implement AWE support in release 9. as was the syntax under Oracle8i and earlier. and you cannot specify the DB_CACHE_SIZE parameter to define the size of the buffer cache. The window will be drawn around the block (i. However. The 9. the address space for ORACLE. It is important to realize that any database blocks accessed by Oracle (or any user/background thread within Oracle. so if this value is not set in the registry. Instead. While this mapping and unmapping of blocks does add some cost. the windowing technology as described in these articles is being used. The value for AWE_WINDOW_MEMORY must be specified in BYTES.https://support. The physical location of these blocks does not matter.1&t.com/CSP/main/article?cmd=show&id=225349. the database blocks must still be accessed from within a ‘window’.EXE will be 3GB. This will be discussed further down in the section on troubleshooting. so all databases that are running out of the HOMEx key where AWE_WINDOW_MEMORY is set will inherit the same value.exe process includes 3 de 8 20/05/2010 12:06 p. this allows the process(es) (in this case ORACLE. refer to Microsoft's website. this value is 1GB. The value for AWE_WINDOW_MEMORY must come from the normal address space used by the ORACLE. the block will be mapped) wherever it is located in memory. However.2 feature allowing for Multiple block sizes in a database will be disabled if you set USE_INDIRECT_DATA_BUFFERS=TRUE. By default. Note: Keep in mind that if there are multiple instances on a machine with the /PAE switch enabled. .e.2. AWE_WINDOW_MEMORY will be 1GB. Memory that comes from that 3GB address space addressable by the oracle.there is no need to be concerned with where the blocks are physically residing. which exists (logically) in that regular 3GB process address space. or a REG_SZ. The size of this window is defined by a registry setting in the HOME key for Oracle (HKLM\Software\Oracle\Homex) called AWE_WINDOW_MEMORY. with AWE enabled. If you add the registry key yourself. no matter what version of the RDBMS you are running. ALL instances can take advantage of the additional memory.EXE process. it is still faster than incurring an I/O operation to read the block from disk.exe) must first be mapped into the 'window' defined by AWE_WINDOW_MEMORY. __________________________________________________________________ AWE_WINDOW_MEMORY Within the 3GB Process Address Space: ======================================================= If you are using /PAE and the /3GB switch together. As mentioned previously.oracle. in order to accommodate the new block. the datatype should be a string value. In this scenario. If the block is in memory but has not been mapped into the ‘window’.. it does not matter where the blocks are physically located . you must set the following parameter in the init file (or spfile) used to start the instance: USE_INDIRECT_DATA_BUFFERS=TRUE Note again that the buffer cache MUST be defined using the parameter DB_BLOCK_BUFFERS. __________________________________________________________________ Enabling AWE Support at the Database/Instance Level: ==================================================== To enable the AWE implementation on Oracle.m. For more information on AWE/PAE implementation on the Windows platform. then it may be necessary to unmapped another block that IS in the window.EXE) to use memory above and beyond the 4GB mark defined by a 32-Bit Process Address space. On 9.. AWE_WINDOW_MEMORY cannot be set on a per-instance basis. if you attempt to startup a database with this combination of parameters: USE_INDIRECT_DATA_BUFFERS=TRUE DB_CACHE_SIZE=xxxxx (Any number) The startup will fail with the following error: SQL> startup ORA-00385: cannot enable Very Large Memory with new buffer cache parameters You must change DB_CACHE_SIZE to use DB_BLOCK_BUFFERS instead. your maximum buffer cache size will be somewhat less than this.. log_buffers. so assuming an 8K block size (8192).com/CSP/main/article?cmd=show&id=225349. your total buffer cache could theoretically be as high as 9GB: (Total RAM . remaining SGA components and all user connection memory (stack + uga + pga) noted above). This was changed under Oracle9i Release 2. to calculate the value for _DB_BLOCK_LRU_LATCHES.7.oracle.2 is calculated as such: MIN(AWE_WINDOW_MEMORY)=(4096 * DB_BLOCK_SIZE * _DB_BLOCK_LRU_LATCHES)/8 Starting with 9. while still allowing enough memory within the 3GB address space for the rest of the process memory that MUST fit within the 3GB (i. and depends on whether or not VLM is enabled.e. This change was done to help improve performance by enforcing a larger window size. this parameter was changed to be a hidden parameter. another possible cause for these errors is noted further down.If you are on Release 9.4GB + 1GB = 9GB In reality.1. java_pool.4GB + AWE_WINDOW_MEMORY) = 12GB .2 by changing/setting the value of the parameter _DB_BLOCK_LRU_LATCHES. large_pool.https://support. Under 8.facility =SOSD O/S Error: (OS 8) Not enough storage is available to process this command (Note .1&t. the following: ·The Value for AWE_WINDOW_MEMORY ·The rest of the SGA (shared_pool. Attempting to startup the database with a buffer cache larger than the maximum value as calculated above may result in the following errors: ORA-27102 out of memory OSD-00034 Message 34 not found. However.7 we do NOT enforce a minimum value for AWE_WINDOW_MEMORY to be able to start the database. this parameter was named DB_BLOCK_LRU_LATCHES. etc) ·Overhead for Oracle. SETS_PER_POOL = 2* CPU_COUNT (if VLM is enabled) SETS_PER_POOL= CPU Count /2 (If VLM is NOT enabled) 4 de 8 20/05/2010 12:06 p. overhead. you would set the following init parameters: DB_BLOCK_BUFFERS = 1179648 DB_BLOCK_SIZE = 8192 __________________________________________________________________ Troubleshooting AWE_WINDOW_MEMORY implementation: ========================= ========================= Minimum Value Required for AWE_WINDOW_MEMORY in 9. we need this formula: _DB_BLOCK_LRU_LATCHES = (Max buffer pools * SETS_PER_POOL) Max Buffer Pools is a constant = 8 SETS_PER_POOL is variable.m. 2. On a machine with 12GB of RAM. in the troubleshooting section) As mentioned above.2.2. such that we DO enforce a minimum value for AWE_WINDOW_MEMORY. using the default value of 1GB for AWE_WINDOW_MEMORY. under 9.x. plus AWE_WINDOW_MEMORY. allowing for some overhead and additional processes running on the system. . The total size of the buffer cache can then be set to the amount of physical memory remaining above the 4GB barrier. You can alter the minimum required value for AWE_WINDOW_MEMORY under 9.1. The minimum value for AWE_WINDOW_MEMORY starting with 9.2 and Above: ============================================================== Here are key points to understand when using AWE_WINDOW_MEMORY: 1.. the buffer cache must be specified using DB_BLOCK_BUFFERS rather than DB_CACHE_SIZE. unless orastack is used) ·PGA and UGA memory for all user sessions Therefore. Product=RDBMS. to get a 9GB buffer cache.exe and DLL’s (65-100M depends on version & options) ·Stack space for all threads (Defaults to 1MB/thread. 3. the value for AWE_WINDOW_MEMORY should be tuned such that mapping and unmapping operations are avoided as much as possible. Under Oracle 8. then you can explicitly set _DB_BLOCK_LRU_LATCHES to a value lower than the calculated value. the OS the system over what you actually the calculations. Therefore. < AWE_WINDOW_MEMORY) that may also result in the ORA-27102 error. and the instance still does not start. UNLESS you explicitly set _DB_BLOCK_LRU_LATCHES to a lower value. _DB_BLOCK_LRU_LATCHES must be at least 8 (Equal to the maximum number of buffer pools) Note #1 .e. /* Recall that VLM is enabled by setting USE_INDIRECT_DATA_BUFFERS=TRUE So.Recall from the earlier section that these errors may also occur if you are trying to start up with a buffer cache that is too large for the physical memory available.1&t. Note #2 . as you can see. a hyperthreaded system.https://support. and the more CPU's in a system.The same errors above have also been observed with a buffer cache that is too small. Note#3 .Also. When USE_INDIRECT_DATA_BUFFERS is set to TRUE the value for DB_BLOCK_BUFFERS should equate to a buffer cache that is AT LEAST equal to AWE_WINDOW_MEMORY. If you attempt to start up with a buffer cache that is too small (i.oracle. keep in mind that when calculating you have to take hyperthreading into account. and this is the number that must be used in the # of CPU's in the system. you will receive the following errors: ORA-27102 out of memory OSD-00034 Message 34 not found. the value for _DB_BLOCK_LRU_LATCHES in 9. Note#4 . If you cannot increase the value for AWE_WINDOW_MEMORY.m. The larger the Block Size.It has been observed on some systems that you may need to add a few additional meg to AWE_WINDOW_MEMORY to calculate for overhead. On will think that you have double the # of CPU's in have.2 and above is dependent on the number of CPU's in the box. In most cases. How to calculate the maximum used memory ========================================= With respect to awe_window_memory the following maximum amount of memory can be used from physical memory: The SGA size is composed from: ((db_block_buffers * block size) + (shared_pool_size + large_pool_size + java_pool_size + log_buffers) + 1MB 5 de 8 20/05/2010 12:06 p. If AWE_WINDOW_MEMORY is not set to the minimum value. . and retry the startup...com/CSP/main/article?cmd=show&id=225349.facility =SOSD O/S Error: (OS 8) Not enough storage is available to process this command If you receive these errors when trying to start the database under 9. try adding an additional 10 Meg or so to the calculated value. if you go through the above calculations. the total buffer cache size will be greater than AWE_WINDOW_MEMORY. and therefore MIN(AWE_WINDOW_MEMORY) is dependent on the # of CPU's as well as the DB_BLOCK_SIZE. Product=RDBMS. Here are a couple of example configurations and caclulations showing MIN(AWE_WINDOW_MEMORY). this may be because the AWE_WINDOW_MEMORY value in the registry is set too low for the calculated minimum value. Example #1: ---------------# of CPU's = 8 DB_BLOCK_SIZE = 8192 Total RAM = 8GB SETS_PER_POOL = 2 * CPU_COUNT = 16 _DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*16 = 128 MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 = ( 4096 * 8192 * 128) / 8 = 536870912 bytes = 512 MB Example #2: --------------# of CPU's = 16 DB_BLOCK_SIZE = 8192 Total RAM = 16 GB SETS_PER_POOL = 2 * CPU_COUNT = 32 _DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*32 = 256 MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 = ( 4096 * 8192 * 256) / 8 = 1073741824 bytes = 1024 MB These values above are the minimum values required for AWE_WINDOW_MEMORY to be set to. the higher the value for MIN(AWE_WINDOW_MEMORY).2 or 10g. This spin will consume CPU cycles until enough buffers can be Mapped/Unmapped to satisfy the request. so the below query will allow you to determine the statistic# for your particular DB version. this example is from a 10gR2 database: SQL> select statistic#. etc. In some cases.2 and later releases: ======================================================== Starting with 9. Note that the statistic#'s change from version to version. that DBWR will consume 100% of cycles on all CPUs. This is enabled by setting the parameter SGA_TARGET to a non-zero value.. The size of buffer cache depends on the available virual memory and can be calculated with buffer cache = db_block_buffer * db_block_size CPU Spins Possible When Using AWE Implementation: ================================================= Use caution when setting _DB_BLOCK_LRU_LATCHES or AWE_WINDOW_MEMORY too low. we have added additional statistics which can be measured in v$sesstat (sesssion-level stats) and v$sysstat (system-wide stats): STATISTIC# ---------154 number 155 number NAME -----------------------------of map operations of map misses This query below will give you system-wide information on map operations and map misses: SQL> select * from v$sysstat where statistic# in (154.x: ============================================== The above stats are not available in 8. STATISTIC# ---------168 169 NAME ---------------------------------------------------------------number of map operations number of map misses So simply substitute in the correct statistic#. and a new buffer can be mapped in. then we spin and wait.com/CSP/main/article?cmd=show&id=225349. you must use DB_CACHE_SIZE for the buffer cache.1.. this may be an indication that the value for AWE_WINDOW_MEMORY is set too low. with AWE_WINDOW_MEMORY enabled. 155).1. JAVA_POOL_SIZE. The size of SGA + Oracle's overhead must not exceed the available virtual memory.oracle. Diagnosing Spins Associated With AWE in 8. If we are unable to map a requested buffer into the window because all of the space defined by AWE_WINDOW_MEMORY is in use with buffers already actively being accessed. whereby the Oracle RDBMS will dynamically adjust SGA parameters such as SHARED_POOL_SIZE. When setting USE_INDIRECT_DATA_BUFFERS=TRUE on Oracle10g. SGA_TARGET should not be set when using AWE . checking every so often until an existing buffer in the window can be unmapped. depending on your DB version. However. DB_CACHE_SIZE. you cannot set DB_CACHE_SIZE. there may be so many buffers needing to be mapped into the window. name from v$sysstat where name like '%map %'. You can start by identifying and monitoring the thread associated with DBWR via the following query: 6 de 8 20/05/2010 12:06 p. it is more difficult to diagnose. so if you are encountering problems with CPU spins. and is simply an indication that AWE_WINDOW_MEMORY is too small. Therefore. as noted above. . When setting USE_INDIRECT_DATA_BUFFERS. effectively locking up the machine. you should also set SGA_TARGET to 0.m. Monitoring Mapping Operations in 9.2.https://support.these two features are mutally exclusive. in order for this to work properly. or particularly of the # of map misses increases consistently over time. If the # of Map misses is relatively high. This is normal behavior under some circumstances. Dynamic Memory Management/Automatic Memory Management with AWE Enabled ============================================================= Oracle10g introduces the concept of Automatic Memory Management.7.1&t. SQL> oradebug dump errorstack 3 Statement processed.com If your monitoring shows that DBWR is consuming excessive CPU.ORA-439 TRYING TO ENABLE VLM IN STANDARD EDITION OF ORACLE . due to BUG#3042660/BUG#2215894. you can attempt to get an errorstack from that thread using oradebug: SQL> oradebug setospid 1956 Oracle pid: 3.4 BUG#2945011 . . buffers are not unmapped and eventually a spin of the process will result.com/docs/cd/B10501_01/win. this is an indication that DBWR is working to map/unmap database block buffers.m.1 .IO SLAVES DON'T UNMAP BUFFERS ON LINUX IN VLM MODE (APPLIES TO WINDOWS AS WELL) RELATED DOCUMENTS ----------------BUG:1803377 .spid where p. which will equate to the Thread ID of that thread within the Oracle executable. SQL> select b. v$bgprocess b 10 rows selected.0.2.CAN'T MAP THE BUFFER Note:46001.Windows NT Memory Architecture Overview Oracle 8.1 . or DBWR may crash with ORA-471 on 9.0.1.0.2.oracle.the fix is NOT backported to 8.ORA-600[KCBVMAP] may occur with AWE.LONG SHUTDOWN TIME WITH AWE_WINDOW_MEMORY: FIXED IN 8.FIXED IN 9.1 BUG#1406194 .7.920/a95490/architec. This should dump the errorstack to the DBWR trace file. you cannot use DBWR_IO_SLAVES in combination with USE_INDIRECT_DATA_BUFFERS..Oracle Database and the Windows NT memory architecture Note:46053.1) for Windows Part Number B10113-01 Chapter 1: Oracle Database Architecture on Windows http://otn..7 of the RDBMS. which is available as a free download from http://www.VLM DOES NOT WORK ON STANDARD EDITION ORACLE 9.2.0. As you can see.paddr.name.7.SHOW SGA DOES NOT SHOW CORRECT # OF DB BUFFERS WITH AWE BUG#1412485 . p.AWE_WINDOW_MEMORY NOT RELEASED WHEN DB SHUTDOWN: FIXED IN 8. DBWR has an SPID of 1956.4 .oracle.com/CSP/main/article?cmd=show&id=225349.1.oracle.1. If the errorstack contains the function SKGMMAP. You must leave DBWR_IO_SLAVES at its default value .EXE SQL> oradebug unlimit Statement processed.otherwise.1.2. Windows thread id: 1956.com/pls/db10g/portal. found in BDUMP.0.1 BUG#2520796 .1&t.1.2 ON WINDOWS . image: ORACLE.pdf Oracle9i Database Getting Started Release 2 (9.1 .FIXED 9.com/docs/pdf/A85305_01. This problem is resolved in 9. NAME ----PMON DBW0 LGWR CKPT SMON RECO SNP0 SNP1 SNP2 SNP3 SPID --------1900 1956 572 1908 1808 920 1784 1892 1896 1844 from v$process p.sysinternals. Note: In 8.https://support.oracle.4 PATCH 2 BUG#3042660 / BUG#2215894 .portal_ Related 7 de 8 20/05/2010 12:06 p.7 Release Notes for Windows: http://download-west.7 KNOWN ISSUES -------------------BUG#2461474 .2) for Windows Part Number A95490-01 Chapter 4: Oracle9i Architecture on Windows http://download-west.FIXED IN 9.addr=b.2.2.4 BUG#3120033 .0.WINDOWS AWE MEMORY AND CPU . This thread can then be monitored using Performance Monitor and/or the PSLIST utility.htm#1005809 Oracle® Database Platform Guide 10g Release 1 (10. oracle...https://support.Enterprise Edition Keywords USE_INDIRECT_DATA_BUFFERS Back to top Rate this document 8 de 8 20/05/2010 12:06 p.m. . Products Oracle Database Products > Oracle Database > Oracle Database > Oracle Server .com/CSP/main/article?cmd=show&id=225349.1&t.
Report "Implementing Address Windowing Extensions (AWE) or VLM on Windows Platforms [ID 225349.1]"