NSD Basics

March 28, 2018 | Author: porpatham | Category: Ibm Notes, Command Line Interface, Databases, Operating System, Process (Computing)
Share
Report this link


Comments



Description

Using NSD – A Practical Guide Hands On Track Session HND107 Lotusphere 2008 Presented by: Elliott Harden and JoeWallace In Memory of Jonathan Sir Henry A great friend with a kind heart. Words along can not express his kindness. Rob Gearhart/Elliott Harden/Joe Wallace 1 Lotusphere 2008 Session HND107 Using NSD – A Practical Guide Overview Introduction This lab is intended to provide Domino Administrators a rough idea of the data that can be used in examining a server crash. Needless to say, a complete analysis of NSD is not possible within the allotted time. This lab focuses on using Call Stack data and Memcheck data perform a preliminary analysis and get you started in the right direction. Contents This lab contains the following topics: OVERVIEW.................................................................................................................................................. 2 SECTION I – NSD BASICS ........................................................................................................................... 3 SECTION II – CALL STACKS ........................................................................................................................ 8 SECTION III: MEMCHECK – SHARED MEMORY ........................................................................................ 15 LESSON II – TOP 10 SHARED MEMORY BLOCK USAGE............................................................................. 18 SECTION IV – PRIVATE MEMORY ............................................................................................................. 28 SECTION V – RESOURCE USAGE SUMMARY ............................................................................................. 35 SECTION VI – CORRELATING MEMCHECK OUTPUT .................................................................................. 38 SECTION VII – NSD CHECKLIST............................................................................................................... 42 SECTION VIII - SEMAPHORE DEBUGGING 101 .......................................................................................... 43 SECTION VIII - SEMAPHORE DEBUGGING 101 – CHEAT SHEET ................................................................ 44 SECTION VIII - SEMAPHORE DEBUGGING 101 – CHEAT SHEET ................................................................ 45 SECTION VIIII – CASE STUDIES ................................................................................................................ 46 APPENDIX – ANSWER KEY ....................................................................................................................... 56 Original vs. Newer Format Within the ND7 timeframe, NSD has undergone many format and behavioral changes. Many of these changes have been back ported to ND6.x (as of ND6.5.5). Within this lab, we refer to both formats, where the original format in ND6 is applies to versions 6.5.4 and earlier, and the newer format applies to versions 6.5.5 and later, including ND7.x. Where appropriate, we indicate the different KEYWORDs that can be used for searching within an NSD in the Newer vs. Original Format. Please refer to page 3 for more discussion on the NSD back port for existing versions of ND6.x and ND7.x, referred to as the NSD Update Strategy. Focus on Domino Server Due to time constraints, this lab is focused on troubleshooting Domino Server crash and hang/performance issues. However, many of these techniques can also be applied to Notes Client crashes and hangs. Please feel free to discuss these additional troubleshooting techniques with the lab instructors. Rob Gearhart/Elliott Harden/Joe Wallace 2 Lotusphere 2008 Session HND107 Section I – NSD Basics What is NSD? NSD (Notes System Diagnostic) is one of the primary diagnostics used for the Lotus Domino Product Suite. It is used to troubleshoot crashes, hangs, and severe performance problems for such products as: • Domino Server & Notes Client • Quickplace, DomDoc, Domino Workflow • Sametime In ND6 and ND7, NSD is used on all Notes Client and Domino Server Platforms except for the Macintosh. • On UNIX – NSD is a shell script (nsd.sh), with Memcheck compiled as a separate binary • On W32 – NSD is a compiled binary (nsd.exe), with Memcheck built into nsd.exe • On iSeries – ND6 is a compiled binary. The source for NSD differs both in nature and in output from the other platforms. However, in ND7, NSD on iSeries has been modified to more closely match the output of NSD on the other platforms. NSD Basics NSD can be run under two contexts: 1). Manually - run from a command prompt to troubleshoot hangs and performance problems 2). Automatically as part of Fault Recovery to troubleshoot crashes – this is enabled in server document (enabled by default) The NSD.EXE/NSD.SH file is located in the program directory. When running NSD manually, in order for NSD to run properly, you must run NSD from the desired data directory. With no switches, NSD will collect full set of data, creating a log file in the IBM_TECHNICAL_SUPPORT directory with the following filename format: • ND6 - nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log • ND7 - nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 3 Lotusphere 2008 Session HND107 Section I – NSD Basics, Continued NSD Update Strategy Between ND6 and ND7, NSD underwent numerous important changes, including format changes to better represent the data, as well as behavioral changes to resolve previous issues and improve NSD’s reliability as a diagnostic tool. In order to allow existing customers to take advantage of improvements to NSD in their current environments, IBM has made available an updated version of NSD for existing versions of Domino on W32, AIX, Solaris, and Linux. This updated version includes most of the features available in ND7.x The NSD Update Strategy means that IBM will periodically provide an NSD Update with the latest set of NSD fixes/enhancements for existing versions of Domino. This is done so that IBM Support and customers can leverage the latest fixes and enhancements for First Failure Data Collection. This will be done for existing versions of ND6.x as well as ND7.x. For more information regarding the NSD Update Strategy, including a list of fixes to NSD, refer to the following URL: Technote #1233676 - http://www.ibm.com/support/docview.wss?uid=swg21233676 You may also refer to a recently published Knowledge Collection for NSD: Technote # 7007508 - http://www.ibm.com/support/docview.wss?uid=swg27007508 Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 4 Lotusphere 2008 Session HND107 • • • • • • • • • • • • • Common NSD switches on W32 nsd –detach (detaches from all processes without killing them . NSD will only be invoked for crashes occurring in a Domino process (any process that makes use of the Domino API). but you can manually detach from all processes with the command “detach”. Continued W32 NSD When NSD is run manually on W32 using no switches.log nsd –nomemcheck (collects all but memcheck) => nsd*. due to a limitation with the Operating System that does not allow for a debugger to detach from a process without exiting that process as a result.log nsd –monitor (attaches and waits for exceptions) => nsd*.log At NSD prompt Can use all of these commands plus: dump (dumps all call stacks out) quit –f (forces a quit – will bring down all Domino processes) Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 5 Lotusphere 2008 Session HND107 .XP & 2003 only) nsd –stack (collects only call stacks.log nsd –kill (kills all notes process and associated memory) => nsd_kill*. it attaches to all Domino processes. allowing NSD to quit without affecting the Domino Server. speeds execution) => nsd*. On Windows 2003 & XP.log nsd –p (runs against a specific process – call stacks only) => nsd*. NSD is no longer run from JIT interface on W32.log nsd –nohandles (collects all but handle info) => nsd*. the Domino server will continue operation. dumps all call stacks. When run as part of Fault Recovery.Section I – NSD Basics.log nsd –info (collects only system info) => sysinfo*.log nsd –perf (collects process memory usage) => perf*. then display an NSD prompt. As long as the NSD prompt remains displayed.log nsd –noinfo (collects all but info) => nsd*.log nsd –handles (collects OS level handle info) => handles*.log nsd –memcheck (collects only memcheck info) => memcheck*. where it remains attached to all Domino processes. On Windows 2000. runs Memcheck data and system information.log nsd –noperf (collects all but performance data) => nsd*. quitting NSD kills all Domino processes. NSD displays a prompt. As a result. dumps data.log => nsd_lsof*.log Note: There is no NSD prompt on Unix. In ND7. DumpSrvStacks is a separate utility available for dumping call stacks. changes have been made to NSD on iSeries to function more closely to NSD on the other platforms. Unix platforms include AIX. NSD behaves differently on Unix than W32 because the Unix debugging interface allows for NSD to detach from all processes without causing any problems. • • • • • • • • • • Common NSD Switches on UNIX nsd –batch (runs nsd with no output to console) nsd –info (collects only system info) nsd –noinfo (collects all but system info) nsd –memcheck (collects only memcheck info) nsd –nomemcheck (collects all but memcheck) nsd –kill (kills all notes process and associated memory) nsd –ps (lists running processes) nsd –lsof (list of open Notes file) nsd – nolsof (collects all but lsof) nsd –user (runs nsd as specific unix user) => nsd_sysinfo*. Continued UNIX NSD On Unix. NSD on iSeries NSD is a different tool in ND6 on iSeries from the other platforms. since NSD does not have the detach problems on Unix that it does on W32.Section I – NSD Basics. and zSeries (OS390). NSD attaches to all Domino processes.log => nsd_kill*. Support is in the process of updating Knowledge Base to reflect these new capabilities. Solaris. Linux.ini Last 2000 lines of Console log Note: NSD takes no switches on iSeries.log => nsd_memcheck*.log => nsd_all*. and detached automatically (returning to a system prompt). NSD output includes: • • • • • • • Call stacks Environment Info Job Log of current jobs Status of current jobs Thread/Mutex info Notes. In ND6.log => nsd_all*. when run manually with no switches. only runs for fault recovery.log => nsd_ps_*.log => nsd_all*. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 6 Lotusphere 2008 Session HND107 . each major section being composed of subsections. These call stacks provide Support with insight into the code path involved in a particular problem. this lab will not focus on this section. network connections. While this section is important under a number of different cases. Rob Gearhart/Elliott Harden/Joe Wallace 7 Lotusphere 2008 Session HND107 . this lab does not focus on information within this section. open documents. Memcheck output is composed of a series of minor sections which summarize memory usage and provide a list of open resources such as open database. In order to more easily visualize the contents of NSD. the most important aspect in this section is the list of the Call Stacks for all threads of each Domino Process. referred to as minor sections. open view. the notes. Again.Section I – NSD Basics. you can consider that there are four major sections within an NSD log. • Memcheck (Domino Memory Objects) – The Memcheck section dumps information about Domino-specific structures. Hence. kernel configurations. and memory usage.ini. Memcheck is a runner up for most important part of NSD. • Environment Information – The section rounds out NSD output. This section is arguable the most important section within NSD. Memcheck tells Support which resources are involved in a given problem. providing the user environment. • System Information – This section provides information regarding version of OS. followed by a list of all Domino specific processes. NSD output is rather verbose. Continued NSD Major Sections As you may or may not know. patch information. connected users. disk information. The four major sections within NSD are: • Process Information (Call Stacks) – Process Information is composed of the list of all running processes system wide. Memcheck steps through all Domino allocated memory pools (both private and shared). and a list of the Domino executables and file located within the Domino Data directory and sub-directories. However. and open files. This summary.arguments may show as "???". and return addresses for each caller (i. it is often necessary to augment call stack data with other forms of debug to develop a clearer picture of the problem. is constantly changing depending on thread activity. W32 For the fatal thread. Namely. return address. It is this summary that NSD dumps for each and every Domino thread. variables that are local to each function. But what exactly is a call stack? In brief. showing arguments. platforms fall into two categories: W32 and UNIX. NSD provides one pass for call stack. AIX. the Call Stack section is the single most important section of NSD. With complex problems. Stacks frames show current conditions.e. In general. they provide limited information. but not necessarily a history of how those conditions were reached. meaning if a function is passed a pointer. where to resume activity). each thread has its own area of memory referred to as its stack region which is reserved for the execution of functions. meaning the module in question was code not compiled with the necessary levels of debug to show argument information. Linux & z/OS). NSD dumps out the contents of memory that the pointer references UNIX This includes Linux. De-referenced pointer arguments for each function. z/OS and i5/OS.Section II – Call Stacks What is a Call Stack? For most server or client crashes or hangs. On AIX . Solaris. This summary is invaluable in troubleshooting crashes and hangs. and currently does not break down stack frames. Granular break down of stack frames. providing a snap-shot of the code path of activity for each thread. NSD makes 3 passes (2 passes in the original format): • 1). The summary of return addresses listed in a thread’s stack region essentially provides an indication of the code path of execution for that thread at any given moment. basic register information • 3). also referred to as a stack trace summary. call stacks provide the current state of thread activity – not a cumulative history of thread activity. Limited register data is provided for certain platforms (AIX. Note: As important as call stacks are. Call Stacks in NSD The level of information that NSD provides for call stacks differs for each platform. Dumps complete call stack (divided into "before" and "after" frames) • 2). arguments to each function. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 8 Lotusphere 2008 Session HND107 . The region of memory is used as a temporary scratch-pad to store CPU register values. 5a4cc160) (not mangled) @[ 6] 0x626412d2 nlsxbe.ae5f894) @[10] 0x6014f59d nnotes. SS=0x00000023 ### DS=0x00000023. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 9 Lotusphere 2008 Session HND107 .WaitForSingleObject+15 (7e4e5a0.77e8ae88.0.0.0.6010ee14) (not mangled) @[ 2] 0x6018e8ec nnotes.0) @[ 3] 0x601a7046 nnotes.7e4ec0c) [ 5] 0x1000e596 jvm.e9641f4.0._AgentRun@16+313 (2ea34ec.10) C Function Name (mangled) DLL Name Frames that are pre-fixed with the ‘@’ symbol mean these are functions that NSD was able to annotate using the Domino symbol files.DLLNode::Register+35 (5a4cc160. FS=0x0000003b.0._MemoryInit1@0+212 (0.ae5f8a0.0.0) The “before” frames are all thread activity leading up to an including the crash.5a4cc160.ae5f894.e967674. SP=0x0ae5e2f8 ### stkbase=0ae60000.LSISession::RegisterClassLibrary+19 (e967674. NSD provides 2 passes of information or the fatal thread._OSFaultCleanup@12+342 (0.321. Below are excerpts from an NSD for each pass. CS=0x0000001b. EBX=0x00000000.e953ff4) @[ 8] 0x6014faf9 nnotes.ae5f7c8._JVM_FindSignal@4+180 (7e4ec0c. For non-fatal threads. total stksize=262144._LsxMsgProc@12+130 (2. PC=6018eb23.7e4ec14.1d78ffa0) @[14] 0x605b0379 nnotes. ES=0x00000023. NSD first lists the “after” frames – these are all frames that indicate how the thread attempts to handle the fatal condition. ECX=0x00ba0000. used stksize=2424 ############################################################ [ 1] 0x77f83786 ntdll.60a2bae4.e9548f4.626412d2) @[ 5] 0x600ef05e nnotes. The “before” frames are actually listed second.0. PASS ONE This pass provides a stack trace summary for the thread.e980000.60ab10d0.ae5f8a0._LSCreateScriptSession@20+125 (e9641f8. EDI=0x00001d34. you can ignore the “after” frames.36ee80.0.5a4cc160.6014f59d) @[ 9] 0x6014f5f3 nnotes.e9641f4. These frames do NOT indicate the fatal condition itself. Continued W32 Call Stacks On W32.e953ff4.ae5eeb8. GS=0x00000000 Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ C++ Class Name @[ 1] 0x6018eb23 nnotes.LSISession::RegisterClassLibrary+141 (e967674. In essence.0) @[ 3] 0x6010ee14 nnotes.605b08f0. ############################################################ ### thread 67/135: [ nSERVER:0908: 2692] ### FP=0ae5ec70._LSLotusScriptInit@4+26 (e9641f8.LSIClassRegModule::AddLibrary+105 (ae5f8a0.0) [ 6] 0x77ea8e90 KERNEL32.e9754f4._OSNTUnhandledExceptionFilter@4+145(7e4ec0c. These frames are the real meat of the crash – this is the part you want to examine: ############################################################ ### FATAL THREAD 67/135 [ nSERVER:0908: 2692] Process ID: Thread ID ### FP=0x0ae5ec70. SP=0ae5e2f8 ### stkbase=0ae60000.0. total stksize=262144.0._Panic@4+483 (609d0013. only PASS ONE is provided.ZwWaitForSingleObject+11 (560. divided into “before” and “after” frames.601a7c06) [ 2] 0x77e87837 KERNEL32.ae5ef5c) C++ Function Name @[ 4] 0x6010e1a4 nnotes. or 3 passes in ND7.Section II – Call Stacks.0. so don’t be confused.7e4ec0c.0.0._Halt@4+28 (107._OSInit@4+14 (0.6ef1ab5.7e4ec0c) @[ 4] 0x601b07b1 nnotes.e9641f8) @[11] 0x60935bdd nnotes. used stksize=2424 ### EAX=0x010d088c.77ea18a5.e953ff4) @[ 7] 0x6014fe93 nnotes. PC=0x6018eb23.ae5f884.CloseProfileUserMapping+161 (0.7e4ec0c.5a4c8288.0) @[12] 0x60935c4a nnotes.CLSIDocument::Init+30 (e9641f4._OSInitExt@8+52 (0.e9540f4) @[13] 0x6093040e nnotes. EDX=0x00ba0000 ### ESI=0x0ae5e904. or user name....Lotus. While these variables are important to examine....._MemoryInit1@0+212 (0.Data..ae5eeb8......`?.0... etc.:W|.`...s\Dom| |ino\nlsxbe.... This ASCII column can provide insight such as an error message. a database name.0.. stksize=2424 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ # ---------... arguments.0) # # # # # 0ae5ec80 0ae5ec90 0ae5eca0 0ae5ecb0 0ae5ecc0 0ae5eeb8 00000000 314d4d24 6e696d6f 5c6f6e69 6010ee14 00000000 64243439 61442e6f 78736c6e 00000107 53495249 746f4c2e 73006174 442e6562 00000000 4d454d24 442e7375 6d6f445c 00004c4c |.PANIC: Ins| # 0ae5e358 69666675 6e656963 656d2074 79726f6d |ufficient memory| @[ 1] 0x6018eb23 nnotes.| |.J.....| |.`..._OSInitExt@8+52 (0..l...D| |omino..626412d2) # 0ae5f578 0ae5f588 600ef05e 00000000 00000000 |..ae5f7c8...IRIS$MEM| |$MM194$d..| ASCII Column @[ 2] 0x6018e8ec nnotes... ############################################################ ### PASS 2 : FATAL THREAD with STACK FRAMES 67/135 [ nSERVER:0908: 2692] ### FP=0ae5ec70... St| # 0ae5e338 206b6361 657a6973 37203d20 20323935 |ack size = 7592 | # 0ae5e348 65747962 41500a73 3a43494e 736e4920 |bytes. showing local variables......DLL.| |.5a4cc160... SP=0ae5e2f8.... be careful not to jump to conclusions based on this information..`.... Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 10 Lotusphere 2008 Session HND107 ......`........Stack bas| # 0ae5e328 78303d65 36454130 34393030 7453202c |e=0x0AE60094.LZ\..........| @[ 5] 0x600ef05e nnotes..?...... To the right is an ASCII representation of the stack frame contents.... Continued W32 Call Stacks (continued) PASS TWO This pass provides a break down of stack frame contents.`..@.._Panic@4+483 (609d0013...| |....Section II – Call Stacks. T| # 0ae5e308 61657268 305b3d64 3a383039 35453130 |hread=[0908:01E5| # 0ae5e318 3841302d 530a5d34 6b636174 73616220 |-0A84].`.....`| |.ae5ef5c) # # # # # 0ae5eeb8 0ae5eec8 0ae5eed8 0ae5eee8 0ae5eef8 0ae5f578 5a4cc160 ffffffff 00000000 00000000 6010e1a4 0ae5ef5c 0ae5ef6c 000a4a10 00000000 00000000 00000000 7c57685c 00000001 0ae5f29c 00000000 00081240 00070000 0e9754f4 0ae5f2a5 |x.| |..Top of the Stack ---------# 0ae5e2f8 00000001 00000107 7c573a9d 54200a0a |..| @[ 3] 0x6010ee14 nnotes...0...0._OSInit@4+14 (0...db....4K.e9754f4.\hW|.`..LZ..... The values you see are variables that are local each function..._Halt@4+28 (107.T.`....6... they may not be directly relevant to the root-cause of a given crash.....| @[ 4] 0x6010e1a4 nnotes...| |`.T.^.5a4cc160) # # # # 0ae5f588 0ae5f598 0ae5f5a8 0ae5f5b8 0ae5f7c8 0e9754f4 4b340016 0000cdff 626412d2 5a4cc160 00000000 600011df 00000000 60003fbc 0000c436 6000123f 00000000 60c6b5e0 0000c000 010c16f2 |.....6010ee14) # 0ae5ec70 0ae5ec80 6018e8ec 609d0013 00000000 |....| Be Careful While the ASCII section in PASS TWO is helpful..| |.......0...0. PC=6018eb23. Linux. However. Note: For iSeries NSDs. class (if any). just as is the case with W32. which includes the function name. and iSeries. and does not indicate the exception itself. Below are some examples of call stacks for each platform. “abort” or “terminate” line. so they can be difficult to read. Within NSD. “signal handler”.Section II – Call Stacks. and argument types. the syntax for function decorations is very similar for AIX. “raise. NSD provides only the stack trace summary (PASS ONE). In order to uniquely identify one function from another. Function Mangling Function mangling is an artifact of the compile process for software. As a result. The exact syntax for this line will depends on the version of UNIX. Solaris has a different decoration syntax. call stacks are read from bottom (most recent function) to top (previous callers). it takes a bit of work to extract the function name from the function decoration. or mangles the function. the compiler generates a function decoration. whereas all other platforms are read the opposite. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 11 Lotusphere 2008 Session HND107 . zSeries function names are not mangles within NSD output. from top (most recent function) to bottom (previous callers). Continued UNIX Call Stacks On Unix. C++ function names are mangled for all UNIX platforms except zSeries (OS390). as well as the nature of the problem itself. the upper part of the stack shows thread activity that attempts to handle the fatal exception. Look at portion of the call stack below the “fatal”. In addition.raise”. 0x3038d150. ??. 2. k-id= 275121 . 6. 0. ??. . ??. ??. . 4bc08d5c) 0x494ee8a1: sysSignalCatchHandler + 0x5d (6. ??) at 0xd0199cf0 pth_signal. 4bc08d48. 4bc08d5c. ??. 4bc08d5c. 4bc08a1c.nsleep(??. 7c0c61cc. 4bc132e8. ??) at 0xd3e928c4 NSFItemLookupByName(??. b9. 0. 0) + 8e58 0x4272e809: ShimmerCalPrint__5HaikuP5NNoteiPPvPUlPUiT4 + 0x2d899 (4bc121d4. ??) at 0x10054cb0 DoProcessRequest__21CreateIMAPDelegationsFv(??) at 0x1005372c AdminpProcessNewRequest(??. ??) at 0x10018ddc EntryThread(??) at 0x10002f80 ThreadWrapper(??) at 0xd3c5f4ac pth_pthread. ??.nsleep(??. 0) + 354 0x406a2421: OSRunExternalScript + 0x15d (0. 4bc08d5c. 4bc08cc8. 494ee844. 0x3038d250. ??.Section II – Call Stacks. ??) at 0xd01e6490 sleep(??) at 0xd02515b0 OSRunExternalScript(??) at 0xd3e2bcf8 OSFaultCleanup(??. . 8142178. ??. ??) at 0xd3e2cf54 fatal_error(??. Continued AIX Call Stack ################################### ## thread 6/15 :: adminp pid=70336. 4bc08cc8. pthr-id=1286 ## stack :: k-state=wait. 81422f8. 0. 1. ??. 4bc14c38. 0. 81422e8. 8142178. Function Name Class Name Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 12 Lotusphere 2008 Session HND107 . ??. ??. 4bc13478) + 35c 0x40d93f18: ExtensionProc__18CompGeneralContextR7ComputeUlPP9CompValueUl + 0x154 (7c0c2ff8) + 1a8 0x40d61a7a: Execute__13ExtensionProc + 0x12a (7c0c61cc. 4bc08d5c) + 28 0x494ef0ba: intrDispatch + 0xba (6. 1. 4bc08cc8) + 20c 0x40682d35: fatal_error + 0x12d (6._pthread_body(??) at 0xd018a5a4 Function Name Class Name Linux Call Stack TID ----. 421effd4. 31d7. 6. ??. 0) + 10 0x494ef31c: intrDispatchMD + 0x60 (6. ??. 6. 8142178. 6) 0x494ef12c: userSignalHandler + 0x68 (6. 4bc08d5c. ??) at 0xd3c698dc OSLockObject(??) at 0xd3c6a358 ItemLookupByName(??. stk max-size=262144. 0. ??) at 0x1001ca0c AdminpRequestAndResponse(??. 7c0c6120) . ??. ??. ??) at 0xd3e9250c AdminpCompileResponseStatus(0x3b003b. 20. cur-size=9484 ################################### ptrgl. 4bc13608) + 120 0x4252e686: GetHaikuDatum__5HaikuP5NNoteiPPvPUlPUiT4 + 0xa76 (4bc121d4. 4bc08d5c. 0) + 110 0x42102549: abort + 0x199 (428e4d8c) Read stack from here down 0x42749c15: __default_terminate + 0x15 (428e4d8c) 0x4274a72a: terminate__Fv + 0x1a (428e4d8c. ??. 41f9ac54. 4bc08608. 2. ??. ??) at 0xd3e92d6c NSFItemInfo(??. ??. ??. ??. 0x100a2340) at 0x10006324 CompileResponseStatus__12AdminRequestFUsPcT2PCc(??. ??. 0) + 37c 0x42100dc0: __libc_sigaction + 0x120 (64033. 494fa4e0) 0x494af892: panicSignalHandler + 0xea (6. ??) at 0xd01e6490 raise.pthread_kill(??. 4bc08cc8.$PTRGL() at 0xd01d7a10 PID/TID raise._p_raise(??) at 0xd01992ec Read stack from here down raise. fffffe64. 4bc08d5c.raise(??) at 0xd01e688c Panic(??) at 0xd3c621b8 LockHandle(??. 6. ??. 4bc08cc8. 31d7) 0x4001cb13: pthread_sighandler_rt + 0x63 (6. 4bc13608) + 10c8 0x42445c49: ExtensionProc__8NFormulaUsUsPUlPPvPUiT3 + 0xaed (4bc14c38. 4bc12070. 8141f6c. 4bc08d48. ??. ??. 4bc08b68) + 8c 0x494ee80f: sysUnwindSignalCatchFrame + 0x77 (494ee844. 0x0. 0. 61660000. 41f9ac54. 41f9ac54. ffff) + 1c8 0x406a14e2: OSFaultCleanup + 0x4b2 (0. 0. b9. ??. 4bc08cc8. 4001afe0) 0x4001ce2b: raise + 0x2b (6. ??. 4281db00.Thread 12759 ----0x42174771: __nanosleep + 0x11 (1. ??. 4bc08cc8. 0. 4bc09088. ??. 4bc08b68. ??. 3150. 40043afc. ??) at 0xd4d3f5cc pth_signal. 4bc13608) + 1c 0x42444bbd: INotesCompExtProc + 0x59 (7c0c5ff8. 4bc08a1c. . 4bc08cc8. e. 0) + 234 [10] ff07ff88 sigacthandler (b. 11. de07bd20. 100. de07bd20. . 0. . 0. de07bd20) + 7e4 [8] ff085fec __sighndlr (b. de07b220. de07ba68. fd9daa74. 2b0708. de07ba68. de07ba68) + 9c [7] f820a7ac JVM_handle_solaris_signal (0. de07cc0c) + 4 [14] fd2e7f18 __1cFHaikuHGetForm6FrnHSafePtr4nGHuForm (de07cc08. dfff80. lwp=50.called from signal handler with signal 11 (SIGSEGV) --[12] fd2fcc58 __1cJURLTargetJGetDbFile6M_pc_ (de07f27c. fd63900c) + 298 . 40. fd639028. de07b1b0) [2] ff07e230 sleep (1. de07ba68. de07bd20. 0. 125800. f820a8cc. d. de07ba68. 7e68. 0. de07ba68. 26c00. 0. 1393a28. 0) + 1a4 [6] f83c7964 __1cCosHSolarisPchained_handler (1. b. fed925ac) + 3e8 [5] fd9dac18 fatal_error (b. 0. 0. de07b337. 13a6814) + 64 here down [11] --. de07bd20. fc800000) + 58 [3] fd9fe978 OSRunExternalScript (0.Section II – Call Stacks. Continued iSeries Call Stack JOB: 001099/QNOTES/HTTP THREAD: 0x34 LE_Create_Thread2__FP12crtth_parm_t ThreadWrapper HTThreadBeginProc ThreadMain__14HTWorkerThreadFv PID/TID CheckForWork__14HTWorkerThreadFv StartRequest__9HTSessionFv ProcessRequest__9HTRequestFv ProcessRequest__21HTRequestExtContainerF19HTAppl ProcessRequest__15HTInotesRequestFv InotesHTTPProcessRequest InotesHTTPProcessRequestImpl__FP18_InotesHTTPreq Execute__3CmdFv Handler__10CmdHandlerFP3CmdPv PrivHandle__10CmdHandlerFP3Cmd PrivHandle__14CmdHandlerBaseFP3CmdT1 HandleOpenImageResourceCmd__10CmdHandlerFP20Open TryIfModifiedSinceWithDb__3CmdFP9NDatabasei GetTimeLastMod__9NDatabaseFR11tagTIMEDATET1 NSFDbModifiedTime InitDbContext InitDbContextExt Function/Class HANDLEDereferenceToNSFBLOCK Names HANDLEDereference Halt Panic fatal_error Read stack from here up OSFaultCleanup (on iSeries) OSFaultCleanupExt OSRunExternalScript 20 18 9 9 75 150 330 126 3 3 256 5 31 13 115 9 194 2 1 1 20 1 8 2 29 33 1 86 39 QLECRTTH QLESPI THREAD LIBNOTES HTTHREAD LIBHTTPSTA HTWRKTHR HTSESSON HTREQUST HTEXTCON HTINOTES INOTESIF LIBINOTES CMD CMDHAND CMDHANDB OPIMGHD CMD NDB NSFSEM2 LIBNOTES DBLOCK DBHANDLE OSPANIC BREAK CLEANUP Solaris Call Stack TID ################################### ###### thread 50/61 :: http. pid=11903. f853ce2c. de07cc0c) + 70 [15] fd28c7a8 __1cOCustomResponseQAttemptToProcess(de07d028. ab6c80. 1393d8c. f84c8000. 3) + 4 [13] fd2d9448 __1cFHaikuDCtxQGetFormsCachePtr6F_pnMHuFormsCache (de07bff4. tid=50 ###### ################################### [1] ff2195ac nanosleep (de07b1b8. de07c0ac. 2e7f. 0. Class Name Function Name Continued on next page PID Rob Gearhart/Elliott Harden/Joe Wallace 13 Lotusphere 2008 Session HND107 . 0) + c Read stack from [9] ff07fdd8 call_user_handler (b. 26d40) + 15c [4] fd9fd68c OSFaultCleanup (0. f9910900. fed925ac. . stk max-size=0.Section II – Call Stacks.() at 0x9e67426 CGtrPosWork::ReadNext(unsigned char)() at 0x9e67426 CGtrPosShort::InsertDocs(CGtrPosSh)() at 0x1650036a CGtrPosBrokerNormal::Externalize(KEY_R)() at 0x164ae9ec gtrMergeMerge() at 0x164aacc8 gtr_MergePatt() at 0x1648738a GTR__mergeIndex() at 0x1648f66e GTR_mergeIndex() at 0x16428f36 cGTRio::Merge()() at 0x163fb848 FTGMergeGTRIndexes(FTG_CTX*. pthr-id=8 ## stack :: k-state=activ. k-id=0x12850600. cur-size=0 TID ################################### sleep() at 0x12370204 PID OSRunExternalScript() at 0x12a3255c OSFaultCleanup() at 0x12a35226 fatal_error() at 0x12a20eb8 __zerros() at 0x1238d164 Read stack from here down . Continued zSeries Call Stack ################################### ## thread 1/4 :: update pid=340.int)() at 0x163f2210 FTGIndex() at 0x163df398 FTCallIndex() at 0x141b3910 . Class Name Function Name . (not mangled) (not mangled) Rob Gearhart/Elliott Harden/Joe Wallace 14 Lotusphere 2008 Session HND107 . Rob Gearhart/Elliott Harden/Joe Wallace 15 Lotusphere 2008 Session HND107 .Top 10 Shared Memory Block Usage Lesson III . The Memcheck section of NSD dumps out verbose information regarding Shared Memory Usage and Shared Memory structures.Open Databases Lesson V . Memory usage by LotusScript.Shared OS Fields (MM/OS Structure) Lesson IV . Lesson Lesson I . such as a crash or performance problem. OS diagnostics are typically needed. In order to evaluate total memory usage for a process. or PerfPMR on AIX). you will be dealing with NSD in the context of an issue. Note about Memory Usage The summarization of shared (and private) memory usage in Memcheck includes memory allocated ONLY by Domino. Usually. We touch on the more important aspects of Memcheck of which you as an administrator should be aware.Section III: Memcheck – Shared Memory Overview Shared Memory is one of the most important aspects of troubleshooting Domino issues. You should use Memcheck in conjunction with other NSD components (namely call stacks) as well as other Domino diagnostics (such as console or semaphore output) or OS diagnostics (such as PerfMon on W32.Open Documents Lesson IV – Open Views See Page 15 18 19 20 22 25 Using Memcheck Output Realize that you will never be looking at NSD output in a vacuum.Summary of Shared Pools Lesson II . as well as other components such as Java and other third party components is not included in this summary. or “Static DPOOL”. therefore the amount of memory usage for every server will also be different. The higher the %used.2 GB (see note below).5 GB – 1.5.750 MB give or take.2 GB of shared memory usage. as a rule of thumb. Certain platforms such as Solaris can use more shared memory without detriment. • Search on KEYWORDs "Shared Memory:" (ND6) to find the total shared memory usage across all shared Domino Pools • Search on KEYWORDs "Shared Memory Stats" (ND7) to find the total shared memory usage across all shared Domino Pools How to Find This Section Original Format (ND6. the better – we want to be making good use of what we have allocated.0 GB to 1. While a higher number for shared memory usage may not indicate a server defect. and will match the figures for “Overall” Indicates the total amount of memory allocated by the Domino Memory Manager. 500 .4 and earlier) Shared Memory: TYPE : Count SIZE ALLOC FREE S-DPOOL: 294 1595000000 952976300 641584608 Overall: 294 1595000000 952976300 641584608 FRAG OVERHEAD %used %free 0 753136 59% 40% 0 753136 59% 40% Newer Format (ND6. you should usually see around 1.0 to 1. In almost all cases.Lesson I – Summary of Shared Pools • Search on KEYWORD "memcheck" until you come to the beginning of the memcheck section. It’s a good thing to have percentages in the 90% range.5. not all of which may be in use. How much shared memory? Every server’s configuration and usage are different.Notes Memory Analyzer (memcheck) -> Shared Memory Stats (Time 17:45:55) -----.6 GB. This total should be no higher than about 1.5 and later) <@@ -----. Indicates the percentage used of memory (ALLOC/SIZE). upwards of 1. all pools will be labeled as “S-DPOOL”. it should be investigated to see if any configuration changes need to be made. not a bad thing. However. A majority of this memory should be in the form of the UBM (0x82cd).@@> TYPE : Count SIZE ALLOC Static-DPOOL: 35 125829120 116479928 Overall : 35 125829120 116479928 FREE 9334512 9334512 FRAG OVERHEAD 0 19408 0 19408 %used 92% 92% %free 7% 7% Field Label TYPE SIZE ALLOC %used Description Indicates the various types of pools that are allocated. Indicates the amount of the “SIZE” that is actually in use (or sub-allocated) at any given time. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 16 Lotusphere 2008 Session HND107 . 2 GB) • you find a low %used figure (for instance. Upon examination of the Summary of Shared DPOOLs: IF… • you find lots of DPOOLs (more than 500 depending on platform) – OR • you find high total shared memory usage (more than about 1. or some problem resulting from high shared memory usage. Call into Support for assistance examining files you may be dealing with poor DPOOL utilization Note: In the excerpt above. This figure is above the threshold of 1. the server has 294 Shared DPOOLs allocated. While this section gives you the total amount of memory allocated for all DPOOLs. with overall shared memory usage over 1.Lesson I – Summary of Shared Pools. Call into Support for assistance examining files collect memory dumps. see the Top 10 section).2 GB and should be investigated. You may also use this section if you simply want to establish the amount of shared memory allocated by the Domino Server at any given time. below 85%) THEN… you may be dealing with high memory usage or a leak AND You Should… check the Top 10 Shared Memory Block Usage. Continued How to Use This Section Use this section of Memcheck when dealing with high shared memory usage. it does NOT provide a break down memory usage per block (for that. You will need memory dumps to determine if fragmentation is occurring for sure. Needless to say. you want to initiate a call with Support Rob Gearhart/Elliott Harden/Joe Wallace 17 Lotusphere 2008 Session HND107 .5 GB. The Top 10 Shared Memory Block Usage is usually the next place to go. Notes Memory Analyzer (memcheck).5..Lesson II – Top 10 Shared Memory Block Usage • KEYWORD " Top 10 Shared Memory" How to Find This Section Original Format (ND6.5. -----.| ---------------------------0x82cd 783314944 199 | 0x841c 2673 3803910 0x82cc 14519040 199 | 0x841b 2673 107708 0x8219 5250452 491 | 0x8405 663 1578194 0x8252 5242880 5 | 0x8001 609 824586 0x841c 3803910 2673 | 0x8219 491 5250452 0x834a 3670016 4 | 0x82d2 424 1054622 0x82df 2097152 1 | 0x8439 318 450932 0x826c 1765962 27 | 0x815f 300 56686 0x8405 1578194 663 | 0x8231 278 127260 0x8f56 1111902 17 | 0x8232 278 2506 ----------------------------------------------------------- Newer Format (ND6. Note: the above excerpt for the Newer Format has been truncated for clarity.5 and later) <@@ -----.Top 10 Shared Memory Block Usage: BY SIZE | BY HANDLE COUNT Type TotalSize Handles | Type Handles TotalSize ---------------------------.4 and earlier) #-----.-> Top 10 Shared Memory Block Usage .@@> BY SIZE Type TotalSize Handles Typename ----------------------------------------------------------0x82cd 637673472 162 BLK_UBMBUFFER 0x8252 20971520 20 BLK_NSF_POOL 0x834a 18350080 18 BLK_GB_CACHE 0x82cc 10511340 161 BLK_UBMBCB 0x824b 9810466 160 BLK_OPENED_NOTE 0x8a03 6760070 1604 BLK_NETBUFFER 0x8311 5242880 5 BLK_NIF_POOL 0x890b 4578420 70 BLK_EXECPOOL 0x8a05 2460000 1 BLK_NET_SESSION_TABLE 0x8a01 2289210 35 BLK_NETPOOL ----------------------------------------------------------- Top 10 Shared Memory Block Usage shows a list of the top 10 highest users of memory blocks by total size and by number of blocks... Field Label Type TotalSize Handles Description The hexadecimal block type designation used by the component that allocated the block Amount of shared memory allocated across all blocks of a given type Number of blocks allocated for a given type (each block uses one handle) Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 18 Lotusphere 2008 Session HND107 .. This output ONLY shows block usage. and is perfectly normal and expected. not pool usage. who is using what) Look for either a large amount of overall memory coming from one block type (exhausting the user address space) or a large number of one block type (causing “Out of shared handle” error messages). you will not see the problem here. which will typically be 500 MB to 750 MB. you should collect a memory dump. This usage is from the UBM. Hence. You should come here: • if you suspect that a shared block is leaking • if you get "Out of Shared Handles" errors • if you want to see shared block usage (i. In Domino 8 32-bit version the UBM is sized at 512MB. Continued How to Use This Section This section augments the Shared Pool Summary by breaking down memory usage by block type. Note: From the Top 10 excerpt above.e. Memory usage from any other single block type should usually be an order of magnitude lower than UBM (for instance 75 MB) Rob Gearhart/Elliott Harden/Joe Wallace 19 Lotusphere 2008 Session HND107 . you see the largest amount of shared memory comes from block 0x82cd (BLK_UBMBUFFER). Under those cases.Lesson II – Top 10 Shared Memory Block Usage. if a pool is not well utilized. 5.5.4 and earlier) • KEYWORD “MM/OS Structure Information” (Newer Format . the ‘Thread’ field actually reflects the last crashing thread ID. you may determine the first fatal thread by examine the OS Process Table. and any PANIC messages if present.5 and later) <@@ -----. For a workaround.5. (*)CAUTION For ND6.ND6. Users=-1.4 (and earlier) on W32. This issue has been addressed as of ND6.@@> Start Time = 12/13/2005 01:15:02 PM Crash Time = 12/13/2005 01:15:32 PM Error Message = PANIC: LookupHandle: handle out of range SharedDPoolSize = 4194304 FaultRecovery = 0x00010013 Cleanup Script Timeout= 300 Crash Limits = 3 crashes in 5 minutes StaticHang = [ nhttp: 2752: 10]/[ nhttp: 2752: 3500] (0xac0/0xa/0xdac) ConfigFileSem = ( SEM:#0:0x010d) n=0. Users=-1. “StaticHang” in ND7. It also provides the thread ID (both physical and virtual) that crashed the server. In theory.5. wcnt=-1. Use this section if you want to establish specific crash information. This will be the process that caused the server to crash.5. with the text “Thread” in ND6. near the top of the NSD. Rob Gearhart/Elliott Harden/Joe Wallace 20 Lotusphere 2008 Session HND107 .Shared OS Fields ------Start Time = 03/23/2004 19:59:40 Crash Time = 03/23/2004 20:18:01 Error Message = PANIC: Removing Peer that was never added SharedDPoolSize = 8388608 FaultRecovery = 0x00010013 Thread [ diiop:17796: 13]/[ diiop:17796: 1800] (4584/d/708) caused Static Hang to be set ConfigFileSem = ( SEM:#0:0x010d) n=0. such as the NSD time and console output.5.Lesson III – Shared OS Fields (MM/OS Structure) • KEYWORD "Shared OS Fields" (Original Format . refcnt=0 Writer=[] n=12. wcnt=-1 Newer Format (ND6.Notes Memory Analyzer (memcheck) -> MM/OS Structure Information (Time 13:15:45) -----. although this information is available through other means.4 and earlier) -----. you can consult this section when you have multiple fatal threads or panics to determine which thread crashed first (*).5.5 and later) How to Find This Section Original Format (ND6.ND6. and looking for the process that invoked the NSD process. crash time. Owner=[] FDSem = ( RWSEM:#12:0x410f) rdcnt=-1. in the case where there are two or more fatal threads. wcnt=-1. wcnt=-1 How to Use This Section This section gives you information about server start time. refcnt=0 Writer=[0:0]. n=11. Owner=[0:0] FDSem = ( RWSEM:#11:0x410f) rdcnt=-1. WarningThreshold = 0 ReplicaID = 86256c87:00664fb0 bContQueue = NSFPool [1935: 58788] FDGHandle = 0xf01c2a98.. WarningThreshold = 0 ReplicaID = 86256df1:006d2839 bContQueue = NSFPool [2014: 52900] FDGHandle = 0xf01c0cf6. Owner=[] By: [ nhttp:0fe4: 34] DBH= 63883. waiters=0. process ID. Dirty = N DB Sem = (FRWSEM:0x0244) state=0. Dirty = Y DB Sem = (FRWSEM:0x0244) state=0. Users=-1. refcnt=0 Writer=[] n=0. wcnt=-1.nsf Version = 41. nlrdrs=0 Writer=[] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1. nlrdrs=0 Writer=[] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1. Owner=[] By: [ nhttp:0fe4: 53] DBH= 63808. refcnt=0. nlrdrs=0 Writer=[] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1. Owner=[] By: [ nSERVER:0768: 132] DBH= 63984. If the Database name contains “!!CN=Server/O=ACME.@@> D:\Lotus\Domino\Data\HR\projnav. Users=-1. User=Anonymous D:\Lotus\Domino\Data\manufacturing\Suppliers.0 SizeLimit = 0. refcnt=0. including local databases that are opened using the server name Version of the database’s ODS Self-explanatory Indicates the process name. waiters=0. the database has been opened locally. and virtual thread ID of the server thread that has opened the database Database handle for each instance opened (can be correlated to Open Documents) Name of the user that has the database open. User=CN=Rhonda Smith/O=ACME/C=US By: [ nSERVER:0768: 132] DBH= 63006. Users=-1. Each user can have multiple instances of a database opened (multiple DBHs) Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 21 Lotusphere 2008 Session HND107 . refcnt=0 Writer=[] n=0. waiters=0. refcnt=0.nsf Version = 43. RefCnt = 1. Dirty = N DB Sem = (FRWSEM:0x0244) state=0. it means the database has been opened via the network.nsf Version = 41. User=CN=Rhonda Smith/O=ACME/C=US D:\Lotus\Domino\Data\finance\Finance. User=CN=Rhonda Smith/O=ACME/C=US By: [ nSERVER:0768: 132] DBH= 64127. When an absolute path is given.Notes Memory Analyzer (memcheck) -> Open Databases (Time 11:45:21) -----. wcnt=-1. User=CN=Mary Peterson/O=ACME/C=US Field Label (Database Name) Version Replica ID By: [ proc:PID:VTID] DBH User Description Name of the database. RefCnt = 1.”. RefCnt = 3. WarningThreshold = 0 ReplicaID = 862568fe:0019c2ad bContQueue = NSFPool [120: 7236] FDGHandle = 0xf01c0928.0 SizeLimit = 0. refcnt=0 Writer=[] n=0. wcnt=-1.0 SizeLimit = 0.Lesson IV – Open Databases • KEYWORD "Open Databases" How to Find This Section <@@ -----.. This section gives you an indication as to what databases are opened. this section gives more verbose information regarding database information. views. Continued How to Use This Section The Open Databases section is one of the more important parts of Memcheck.Lesson IV – Open Databases. However. or need specific information about a database. and what users are accessing them. you will likely find that the Resource Usage Summary is better equipped to provide immediate answers as to what databases. The main information you will find helpful is: Database Name ODS Version Replica ID List of Users While this section is valuable. or documents a crashing or hung thread is accessing. Use this section to establish the total number of open databases (you must estimate this count manually). Rob Gearhart/Elliott Harden/Joe Wallace 22 Lotusphere 2008 Session HND107 . size=2984 . Open By: CN=Jane Doe/O=ACME/C=US Flags2 = 0x0404 Flags3 = 0x0000 OrigHDB = 1135 First Item = [ 17391: 836] Last Item = [ 17391: 1780] Non-pool size : 0 Member Pool handle=0x43ef. 70694 0x43ef 0x0001 0x0200 Yes 1 6 .4 and earlier) ----------.nsf Class denotes document type 1135 4156 d:\notedata\MAIL3\jdoe. size=4444 .@@> DBH 531 NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items 7330 0x24ff 0x0001 0x0200 Yes 1 4 .. Open By: CN=BES/ACME/C=US Flags2 = 0x0004 Flags3 = 0x0000 OrigHDB = 1676 First Item = [ 21459: 836] Last Item = [ 21459: 948] Non-pool size : 0 Member Pool handle=0x53d3. 2398 0x53d3 0x0001 0x0200 No 1 6 . size=4156 .Notes Memory Analyzer (memcheck) -> Open Documents (BLK_OPENED_NOTE): total=352 .Lesson V – Open Documents • KEYWORD "Open Documents" How to Find This Section Original Format (ND6.5 and later) <@@ -----.nsf 1353 4444 d:\notedata\MAIL2\dfish.5. Open By: CN=John Smith/O=ACME/C=US Flags2 = 0x0404 Flags3 = 0x0000 OrigHDB = 531 First Item = [ 9471: 836] Last Item = [ 9471: 1228] Non-pool size : 0 Member Pool handle=0x24ff..nsf Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 23 Lotusphere 2008 Session HND107 .-----. Size Database 2984 d:\notedata\drmail\jsmith.Open Documents --------DBH 1051 391 2175 754 2216 870 1309 1926 NOTEID 79442 79442 79442 79442 79442 79442 79442 79442 HANDLE 0x2f3d 0x2b33 0x2d34 0x2d33 0x2c07 0x3550 0x342a 0x3545 CLASS 0x0001 0x0001 0x0001 0x0001 0x0001 0x0001 0x0001 0x0001 FLAGS 0x0300 0x0300 0x0300 0x0300 0x0300 0x0300 0x0300 0x0300 FirstItem [ 12093: 768] [ 11059: 768] [ 11572: 768] [ 11571: 768] [ 11271: 768] [ 13648: 768] [ 13354: 768] [ 13637: 768] LastItem 12093:3052] 11059:3052] 11572:3052] 11571:3052] 11271:3052] 13648:3052] 13354:3052] 13637:3052] [ [ [ [ [ [ [ [ Newer Format (ND6.5. Continued Field Label DBH NOTEID CLASS IsProf (Newer Format) Pools (Newer Format) Size (Newer Format) Items (Newer Format) Database (Newer Format) Opened By (Newer Format) Description Database handle for each instance opened (can be correlated to Open Databases to determine database name) NOTEID in decimal for the opened Note Class of the note – this indicates if a note is a form note.Lesson V – Open Documents. view note. agent note. or document note (see table below) Indicates if the document is a profile document Indicates how many POOLs the document is spread across in memory Indicates the size of the document meta data held in the above mentioned POOLs Indicates how many items/fields the document contains Indicates which database the document belongs to Indicates which user opened the document Note Class 0x0001 0x0004 0x0008 0x0040 0x0200 0x0800 Description Data Note (document) Form Note View note ACL Note Agent Note Replication Formula Note Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 24 Lotusphere 2008 Session HND107 . g. you may potentially see as many as three open document lists in shared memory and three open documents lists under each process. due to the fact that Memcheck dumps out different types of open documents as separate lists. view note. the server opens the note just long enough to pass it back to the client. there may also be documents opened in shared or private memory depending on the needs of the component that opened the document. You can examine this section to determine NoteID information. So while a client may have a document open for long periods of time. or new notes. About Fast Note Opens Whenever a server opens a note on behalf of a client. the server performs what is called a “fast note open”. etc). Multiple Open Document Sections Within NSD. for instance. regardless of the type of note (design note. In addition to these different note types. Hence. or data note. etc. open notes that are listed within a server NSD are typically in fact not opened on behalf of clients. which is information not reflected in Resource Usage Summary. or establish which database a document belongs to. You are most interested in: • • • • • • • • • DBH – database handle NOTEID – noteID CLASS – note class (e. This information is also held in the Resource Usage Summary. In essence. the placement of the Open Documents section within Memcheck indicates whether the notes are opened in private or shared memory. It is also quite possible that you will see no open document lists at all (since no notes may be open at the time that NSD is run). Rob Gearhart/Elliott Harden/Joe Wallace 25 Lotusphere 2008 Session HND107 . However. Continued How to use this section You will probably find the Open Documents section one of the more helpful sections of NSD. and then closes the note. and is better organized there.Lesson V – Open Documents. you may encounter multiple “Open Document” sections. notes opened locally. notes opened remotely. NOTE_CLASS_DOCUMENT) FLAGS – note flags (e. the server keeps the note open for only a fraction of a second. Therefore. This section shows the list of all open “notes” in memory on the server (or client). agent. but rather are notes that are open for the servers own use (such as during the running of an agent or mail delivery).g. NOTE_FLAG_UNREAD) Pools – how many POOLs the document is spread across in memory Items – how many items/fields the document contains Size – the size of the document meta data held in the above mentioned POOLs Database – which database the document belongs to Opened By – which user opened the document The note's "CLASS" will tell you if a note is a document. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 26 Lotusphere 2008 Session HND107 . CollectionVB Remote OFlags ViewNoteID Data HDB/Full View HDB/Full -----------.. [ nserver: 09d8: .. refcnt=0.-------... -----------. nlrdrs=0 Writer=[ : 0000] NumCollations = 2 bCollationBlocks = [ 001e72e5] bCollation[0] = [ 00117005] bCollation[1] = [ 001a2205] CollIndex = [ 00012a09] Collation 0:BufferSize 26.” indicates where output has been removed for clarity.. by KEY.------------[ 00239805] . [ 0023d005] NO 0x0082 786 1219/1874 1219/1874 CurrentCollation = 0 [ 0013a805] . -------------. [ 0020e005] NO 0x00c2 1518 551/1432 551/1432 CurrentCollation = 0 ..------.Items 1.-----.------. waiters=0...Notes Memory Analyzer (memcheck) -> NIF Collection Users (hash) (Time 12:48:33) -----.---------.-----.. summary# 2 ResponseIndex [ 0010e4b6] NoteIDIndex [ 0010e385] UNIDIndex [ 0010e5e7] <@@ -----..--. summary# 2 CollIndex = [ 00012c09] Collation 1:BufferSize 26. "StartDateTime".------..------------..Flags 0 0: Ascending.-----.---...-----..Flags 0 0: Descending.-----.-----------[ 0020e005] 1518 1356a8 358710 1 0x0000 00000008 NO NO NO NO 0 MyNotices CIDB = [ 0253cc05] CollSem (FRWSEM:0x030b) state=0.Items 1...@@> CollUserVB .. [ 00136005] NO 0x00c2 11122 886/785 886/785 CurrentCollation = 0 [ 0028d805] . by KEY.... [ nserver: 09d8: 09d8: 04ca] 0266] 03b0] Note: The string “..Lesson VI – Open Views • KEYWORD “NIF Collections” • KEYWORD “NIF Collection Users” How to Find This Section <@@ -----.-------.. [ nserver: .@@> CollectionVB ViewNoteID UNID OBJID RefCnt Flags Options Corrupt Deleted Temp NS Entries ViewTitle -----------.. Open By .---------. "StartDateTime".Notes Memory Analyzer (memcheck) -> NIF Collections (Time 12:48:35) -----. but since multiple view notes can have the same NoteID in different databases. and by which field Database handle for user & full server access for the database where the view in contained – good for correlating data Indicates the process name. you will refer to these two sections. such as number of collations. NIF Collection Users provides you information about the users of the views opened on the server (such as which thread has a view open. If contention exists on a view. but if you need to take a deeper dive. contains reader and waiter information (same as semdebug) Number of collations in the view (hints at view complexity) Indicates how the collation is sorted. or if you want detailed information about the view. The main reason to use the otherwise rather cryptic “CollectionVB” value is to cleanly correlate NIF Collection User information back to the NIF Collections information. etc). where as CollectionVB is definitive (since it is an in-memory location for collection info). For a summary of information about views being used by a specific thread. PID & TID of thread that has view opened Rob Gearhart/Elliott Harden/Joe Wallace 27 Lotusphere 2008 Session HND107 . Sometimes you can use ViewNoteID to do this. These fields are bolded in the excerpt on the previous page. this is not a definitive match. it is easier to refer to Resource Usage Summary. Below is a short description for the more important fields of information that you will routinely investigate. what collation it is using. Continued How to Use This Section NIF Collections provides you information about all collections opened server-wide. Use this section when you want to get an idea of the total number of open views. such as investigating which threads are waiting on a semaphore for a given view.Lesson V – Open Documents. Field Label CollectionVB (Collections & Collection Users) ViewNoteID (Collections & Collection Users) RefCnt (NIF Collections) ViewTitle (NIF Collections) CollSem (NIF Collections) NumCollations (NIF Collections) Ascending/Descending (NIF Collections) View HDB/Full (NIF Collection Users) Open By (NIF Collection Users) Description In-memory location of collection info – good for correlating data between Collections and Collection Users NoteID for view note design – sometimes – good for correlating data between Collections and Collection Users Number of users that have the view open Name of the View/Collection Info about the Collection Semaphore. In contrast. you should call into Support to assist in this investigation. if private memory usage for a task exceeds these thresholds. each server task should use considerably less private memory through the Domino Memory Manager.Top 10 Memory Block Usage See Page 28 29 30 32 Finding the beginning of Private Memory • Search on KEYWORD “memcheck” until you come to the beginning of the Memcheck section • Search on KEYWORD “attach” – this will bring you to the beginning of the Private Memory section where Memcheck attaches to the first process • Subsequent searches on KEYWORD “attach” will take you to the next process • Search on KEYWORD “detach” to get to the end of a specific process’ information How much private memory? As stated before. particularly when high memory usage occurs in process-private memory. http task. Rob Gearhart/Elliott Harden/Joe Wallace 28 Lotusphere 2008 Session HND107 . the majority of memory usage for a Domino Server should be from shared memory.Open Documents Lesson III . which may use upwards of 200-300 MB.Section IV – Private Memory Introduction Private Memory can also be important in troubleshooting issues.TLS Mapping Lesson II . including the server task. between 50 MB and 100 MB. While it may not indicate a server defect. Important section in Private Memory include: Lesson Lesson I .Process Heap Memory Lesson IV . an investigation should be made to determine the nature of this memory usage. and if any configuration changes are required. and router task. there are a few exceptions to this rule. Under these cases. on average. You can also use this section if you know the virtual thread ID and need the physical thread ID. This is done primarily to facilitate scalability for the server when dealing with network I/O. but not virtual Thread ID). you have physical Thread ID. For many server tasks. but this can also be found in the Resource Usage Summary along with all the open resources for that virtual or physical thread. Rob Gearhart/Elliott Harden/Joe Wallace 29 Lotusphere 2008 Session HND107 . What are Virtual Threads? Virtual thread IDs are numbers generated by Domino that allow for an additional abstraction layer between Domino and OS threads. a given physical thread will constantly be switching from one virtual thread ID to another during normal operations.Lesson I – TLS Mapping • KEYWORD “TLS Mapping” – find the appropriate Process ID and Thread ID -----.TLS Mapping ----NativeTID VirtualTID [ nSERVER:0514:0510] [ nSERVER:0514:0002] [ nSERVER:0514:0504] [ nSERVER:0514:0004] [ nSERVER:0514:05d4] [ nSERVER:0514:0005] [ nSERVER:0514:0600] [ nSERVER:0514:0006] [ nSERVER:0514:0604] [ nSERVER:0514:0007] [ nSERVER:0514:0608] [ nSERVER:0514:0008] [ nSERVER:0514:060c] [ nSERVER:0514:0009] [ nSERVER:0514:0610] [ nSERVER:0514:000a] [ nSERVER:0514:06ac] [ nSERVER:0514:000b] [ nSERVER:0514:06b0] [ nSERVER:0514:000c] [ nSERVER:0514:06b4] [ nSERVER:0514:000d] [ nSERVER:0514:06b8] [ nSERVER:0514:000e] [ nSERVER:0514:06e8] [ nSERVER:0514:000f] [ nSERVER:0514:06f4] [ nSERVER:0514:0010] [ nSERVER:0514:06f8] [ nSERVER:0514:0011] [ nSERVER:0514:06fc] [ nSERVER:0514:0012] [ nSERVER:0514:0700] [ nSERVER:0514:0013] [ nSERVER:0514:0704] [ nSERVER:0514:0014] [ nSERVER:0514:0708] [ nSERVER:0514:0015] [ nSERVER:0514:070c] [ nSERVER:0514:0016] [ nSERVER:0514:0710] [ nSERVER:0514:0017] PrimalTID nSERVER:0514:0002] nSERVER:0514:0004] nSERVER:0514:0005] nSERVER:0514:0006] nSERVER:0514:0007] nSERVER:0514:0008] nSERVER:0514:0009] nSERVER:0514:000a] nSERVER:0514:000b] nSERVER:0514:000c] nSERVER:0514:000d] nSERVER:0514:000e] nSERVER:0514:000f] nSERVER:0514:0010] nSERVER:0514:0011] nSERVER:0514:0012] nSERVER:0514:0013] nSERVER:0514:0014] nSERVER:0514:0015] nSERVER:0514:0016] nSERVER:0514:0017] How to Find This Section [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ How to Use This Section The TLS Mapping is crucial for mapping virtual thread ID used throughout Memcheck with the correct physical thread ID used in other portions of NSD such as call stacks. You should consult the TLS Mapping when you wish to determine which virtual thread ID a particular physical thread is running under (i.e. Rob Gearhart/Elliott Harden/Joe Wallace 30 Lotusphere 2008 Session HND107 . While the Resource Usage Summary also lists all these documents.Lesson II – Open Documents • KEYWORD “Open Documents” – find the appropriate process How to Find This Section How to Use This Section Formatted in the same manner as the Open Document Sections from shared memory. that section does not give an indication as to the scope each note. You can use this section to establish the scope of each note if needed. You should use this section when determining how many notes a specific process has open. this section lists open documents that are opened in private memory. The higher the %used. It’s a good thing to have percentages in the 90% range. so you should examine the stats for “S-DPOOL”.5. not all of which may be in use.@@> TYPE : Count SIZE ALLOC FREE FRAG OVERHEAD %used %free Static-DPOOL: 12 6291456 3795788 2489080 0 9486 60% 39% VPOOL : 2 130808 8994 117628 0 4210 6% 89% POOL : 3 86348 58790 24468 0 3114 68% 28% Overall : 12 6291456 3653692 2631176 0 16810 58% 41% Field Label TYPE SIZE ALLOC %used Description Indicates the various types of pools that are allocated. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 31 Lotusphere 2008 Session HND107 .5 and later) <@@ -----.5 and later) – find the appropriate process How to Find This Section Original Format (ND6. or “Static DPOOL” (other pool types such as POOL and VPOOL are actually sub-allocated from S-DPOOL.5. Indicates the percentage used of memory (ALLOC/SIZE).4 and earlier) Process Heap Memory: TYPE : Count SIZE ALLOC FREE S-DPOOL: 719 376963072 211225436 165542364 VPOOL : 1 64512 456 62428 POOL : 187 2740396 1786784 889772 Overall: 719 376963072 210273236 166494564 FRAG OVERHEAD 0 239594 0 1636 0 70088 0 311318 %used 56% 0% 65% 55% %free 43% 96% 32% 44% Newer Format (ND6. so this stat will reflect overall usage).Notes Memory Analyzer (memcheck) -> Process Heap Memory Stats (Time 17:46:00) -----.4 and earlier) – find the appropriate process • KEYWORD “Process Heap Memory Stats” (ND6. Indicates the amount of the “SIZE” that is actually in use (or suballocated) at any given time. the better – we want to be making good use of what we have allocated. not a bad thing. the overall total is currently calculated incorrectly. For private memory.5. Indicates the total amount of private memory allocated by the Domino Memory Manager.Lesson III – Process Heap Memory • KEYWORD “Process Heap Memory” (ND6. This total should be no higher than about 100 MB (see note on next page).5. and router task. each server task should use considerably less private memory usage (50-100 MB) than shared memory (1.0-1.Lesson III – Process Heap Memory. the nature of this memory usage should be investigated. when private memory exceeds these thresholds. The information provided in this section of Memcheck is identical in nature to the Shared Memory pool section. While it may not indicate a server defect. this section can be used to establish the total amount of private memory Domino has allocated for a process. Use this section of Memcheck when dealing with high private memory usage. For instance: • if a crash is due to high private memory usage • if you suspect a leak in private memory • if you want to evaluate the amount and efficiency of private memory usage As mentioned earlier. which is augmented by the Top 10 [Process] Block Usage sections. there are a few exceptions to this rule. http task. Continued How to Use This Section Similar to the “Shared Memory” Summary. or some problem resulting from high private memory usage or high private handle counts. only this time memory is private to each process. which may use upwards of 200300 MB. including the server task.2 GB). Rob Gearhart/Elliott Harden/Joe Wallace 32 Lotusphere 2008 Session HND107 . 5..Notes Memory Analyzer (memcheck).5 and later) <@@ -----..-> Top 10 [ nSERVER: BY SIZE Type TotalSize Handles Typename ----------------------------------------------------------0x4129 20447232 39 BLK_LOCAL 0x0a04 10595772 162 BLK_NET 0x028b 3327954 53 BLK_FOLDERREPLOPS 0x0910 1999180 1126 BLK_SRV_NAMES_LIST 0x093c 1219526 242 BLK_SRV_HASH_TBL 0x024b 1131334 19 BLK_OPENED_NOTE 0x0221 930818 96 BLK_NEW_NOTE 0x0130 562418 1545 BLK_TLA 0x0149 548834 101 BLK_PHTCHUNK 0x030a 319190 1 BLK_LOOKUP_THREAD ----------------------------------------------------------09d8] Memory Block Usage..5.@@> Field Label Type TotalSize Handles Description The hexadecimal block type designation used by the component that allocated the block Amount of private memory allocated across all blocks of a given type Number of blocks allocated for a given type (each block uses one handle) Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 33 Lotusphere 2008 Session HND107 .4 and earlier) #-----.Lesson IV – Top 10 Process Memory Blocks • KEYWORD “Top 10” – look for appropriate process in [brackets] How to Find This Section Original Format (ND6.| ---------------------------0x4129 2883584 11 | 0x0130 95 34580 0x29c5 349084 1 | 0x0380 67 31222 0x29c4 131072 1 | 0x030a 47 15604 0x29c2 65406 1 | 0x0f02 22 2420 0x29c3 65406 1 | 0x0275 22 2554 0x29c1 49152 1 | 0x0910 20 5332 0x0130 34580 95 | 0x4129 11 2883584 0x0149 32604 6 | 0x0149 6 32604 0x0380 31222 67 | 0x0f5d 2 220 0x024b 24908 1 | 0x0128 1 10480 ----------------------------------------------------------- Newer Format (ND6. -----..Top 10 [ nSERVER:0514] Memory Block Usage: BY SIZE | BY HANDLE COUNT Type TotalSize Handles | Type Handles TotalSize ---------------------------. Hence. with a few exceptions such as server. or router (which may use upwards of 200-300 MB). Rob Gearhart/Elliott Harden/Joe Wallace 34 Lotusphere 2008 Session HND107 . you will not see the problem here. You should come here: • if you suspect that a private block is leaking • if you get "Out of Private Handles" errors • if you want to see private block usage Just as in shared memory. look for either a large amount of total memory usage for one block type. or a large count of one block type. Again. large amounts of private memory over 100 MB should be a concern. you should consult the Process Heap Memory Summary and collect memory dumps (opening a ticket with Support). Under those cases. Continued How to Use This Section Use this section for a quick method of establishing process-private block usage broken down by block type. not total pool usage.Lesson IV – Top 10 Process Memory Blocks. http. This ONLY shows block usage. if a pool is not well utilized. allowing for one-stop shopping. this section of Memcheck lists all the major resources broken out per thread ID (virtual and physical thread). With the exception of memory usage. The format of this section is cleanly organized. As has already been eluded. This is perhaps the most important section of Memcheck next to memory usage. Important output is: • VThread (Virtual Thread) To PThread (Physical Thread) Mapping Rob Gearhart/Elliott Harden/Joe Wallace 35 Lotusphere 2008 Session HND107 . you will spend most of your time here.Section V – Resource Usage Summary Introduction No doubt you have noticed that this documentation has repeatedly mentioned the importance of the Resource Usage Summary. Database: D:\Lotus\DominoR65\Data\mayhem. SOBJ: addr=0x01130004.Description: Server for Rob Gearhart/SET on TCPIP .. doc: HDB=84. ID=2310... By: CN=Rob Gearhart/O=SET . SOBJ: addr=0x00000001. TaskVar: id: [ 70: 8392]. Task: TaskID=[ 70: 8392]..nsf .. . h=0xf0104002 h=0xf010400f h=0xf01c0043 h=0x00000000 h=0xf01c0036 h=0x00000000 t=8128 t=8a18 t=1381 t=8310 t=0a22 t=9508 (BLK_PCB) (BLK_NET_PROCESS) (BLK_SCT_MGR_CONTEXT) (BLK_NIF_PROCESS) (BLK_LOCINFO) (BLK_EVENT_PROCESS) Match crashing thread to PThread (physical thread) User with resource open Note: A user can be a server ** VThread [ nserver:289c: 77] .. file: fd: 1060. ....... SOBJ: addr=0x00000001. DBH: 91.. . PRThread: [ nserver:289c: 77] .... SOBJ: addr=0x022d2628. h=0xf01040c4 t=c130 (BLK_TLA) . using: IOCP h=297271297.. SOBJ: addr=0x02670338. sessID: [ 12: 3732] All Documents|($All) .. using: Primal Thread [ nserver:289c: 58] ..Resource Usage Summary ---** Process [ nserver:289c] .. by: CN=Rob Gearhart/O=SET . st=6. h=0xf01040c6 t=c275 (BLK_NSFT) . class=0001...Mapped To: PThread [ nserver:289c:10684] .. D:\Lotus\DominoR65\Data\mayhem. VThread [ nserver:289c: 10] . SOBJ: addr=0x0345918c.nsf Note ID of document 1 line shows a document was open with the extension doc... SOBJ: addr=0x0231d4b0. .. flags=4000 .. view: hCol=105. SOBJ: addr=0x010ede50. H=6705... transID=5457... By: CN=Rob Gearhart/O=SET Database associated with this thread . DBH: 84.Section V – Resource Usage Summary – VThread Info • Search on KEYWORDs “Resource Usage” or “Resource Usage Summary” • Search on KEYWORD “Process” until you located the necessary process ID • Search on the PHYSICAL THREAD ID of the thread you are interested in (come to this section after you have looked at call stacks) How to Find This Section ---.. 2nd line shows a view was opened st Class denotes type of document open Refer to open document section of this document for a listing of class types Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 36 Lotusphere 2008 Session HND107 . .. cg=N noteID=322. Func=59.. You can find: • • • • • • Database – establish the database name DBH – correlate the instance of the database user against other sections of Memcheck By – to determine the user of the database view – establish any open views doc – establish a list of notes open by the thread (keep an eye on class) file – establish a list of any files the thread had open through Domino file management Keep in mind that this list of resources is specific to the virtual thread ID. There are many cases where a crash or hang is a result of conditions that are completely unrelated to the resource being used. there will be multiple databases. views. which a suspect physical thread happens to be mapped to at that moment. Avoid This Pitfall Be careful not to jump to conclusions regarding a problem database. this should not affect your investigation. view or document was open at the time of the crash doesn’t mean the root-cause of the crash is due to accessing that resource. Rob Gearhart/Elliott Harden/Joe Wallace 37 Lotusphere 2008 Session HND107 . etc opened by the thread. This means that you will need to make an educated guess at to which resource was actually involved in a problem (based in part on arguments on the stack). view or document. Get an idea from the call stack about the nature of the crash. However. views. only one of which may responsible for the problem. in fact most aren’t. Not all of these resources were originally opened or even in use by this physical thread at the time of the crash. Always approach a crash or hang with a grain of salt. Just because a database. Often times. Continued How to Use This Section Use this section when you want to know what databases. Not every database involved in a crash is corrupt.Section V – Resource Usage Summary – VThread Info. notes or files a physical thread has open. Section VI – Correlating Memcheck Output Introduction We have seen throughout Memcheck Anatomy. Lesson Lesson I – Finding Physical/Virtual Thread Lesson II – Correlating Database Handle with Open Documents Lesson III – Correlating View Information See Page 38 39 40 Rob Gearhart/Elliott Harden/Joe Wallace 38 Lotusphere 2008 Session HND107 . but it is helpful to be familiar with how to correlate this directly. This section will discuss how to go about tying certain pieces of information together. The Resource Usage Summary correlates much of the important pieces together. how Memcheck lists the same information in multiple contexts. as well as in this unit. Knowing how to correlate all this information together to find what you need is very important. 0) @[14] 0x1004392a nhttpstack.0.17603a0.MHMProcessRawRequest+317 (17603e8.0) @[13] 0x10034c13 nhttpstack.37dccc4.1002544f) @[12] 0x1002d087 nhttpstack.90dfd08. Replica. etc. you will almost always be starting with a physical (native) thread ID and working your way back to the virtual thread ID using the TLS Mapping section for the process in question.90df9fc.0._LockHandle@12+59 (ffffffff._ThreadWrapper@4+251 (0.6000118b.MHMDispatchMayhem+228 (82c70a0.176023c.1. SP=0x090dece4.1760000) [ 4] 0x082d24dc mayhem.8043147b.HTDsapiRequest::RawRequest+379 (a4. ECX=0x017f088c.0) physical @[ 3] 0x60002f17 nnotes. EBX=0x00000000.90df69c.HTWorkerThread::ThreadMain+126 (37dccc4. PC=0x6027f491.90dfd30) [ 9] 0x082c10a6 mhmdsapi. ES=0x00000023.64.HttpFilterProc+31 (17603e8.a) thread ID [ 5] 0x082d1cd6 mayhem.HTSession::StartRequest+1299 (6f530.6e) [ 8] 0x082c11ee mhmdsapi.0.0.1.90dfd08.HTRequestExtContainer::RawRequest+164 (1760a68.82f22f8.6f530._OSLockObject@4+23 (ffffffff.a. CS=0x0000001b.10017341) @[11] 0x100174a4 nhttpstack.64. FS=0x00000038.6000118b._Panic@4+625 (90df35c.HTRequest::ProcessRequest+1447 (6f530.6e.2.TlsSetValue+240 thread ID -----. EDX=0xffffffff ### ESI=0x6000118b.1003f000) @[17] 0x601c0b5b nnotes.10043e63.6000118b. since there can be multiple processes with the same name.0.3f.64) [ 6] 0x082d1171 mayhem.HTWorkerThread::CheckForWork+538 (37dccc4. like AMGR. be sure to use process ID's and thread ID's for locating the process you are troubleshooting.TLS Mapping ----[ [ [ [ [ [ [ [ [ NativeTID nHTTP:113c: 1852] nHTTP:113c: 2560] nHTTP:113c: 2556] nHTTP:113c: 4152] nHTTP:113c: 1800] nHTTP:113c: 2580] nHTTP:113c: 4284] nHTTP:113c:10840] nHTTP:113c: 3500] [ [ [ [ [ [ [ [ [ VirtualTID nHTTP:113c: 5] nHTTP:113c: 6] nHTTP:113c: 7] nHTTP:113c: 8] nHTTP:113c: 9] nHTTP:113c: 10] nHTTP:113c: 11] nHTTP:113c: 12] nHTTP:113c: 13] [ [ [ [ [ [ [ [ [ PrimalTID nHTTP:113c: 5] nHTTP:113c: 6] nHTTP:113c: 7] nHTTP:113c: 8] nHTTP:113c: 9] nHTTP:113c: 10] nHTTP:113c: 11] nHTTP:113c: 12] nHTTP:113c: 13] Rob Gearhart/Elliott Harden/Joe Wallace 39 Lotusphere 2008 Session HND107 .MHMServerCrash+387 (6e.90dffb4.90df8fc.0) @[ 2] 0x6000304b nnotes. GS=0x00000000 Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ @[ 1] 0x6027f491 nnotes.6000118b.90dfd04) @[10] 0x10012a2b nhttpstack. SS=0x00000023 ### DS=0x00000023. For instance: Fatal Call Stack: ############################################################ ### FATAL THREAD 16/20 [ nHTTP:113c: 4284] ### FP=0x090df664.Lesson I – Finding Physical/Virtual Thread PIDs/TIDs When using a text editor.601c0b5b. EDI=0x090dfafc.90df6a4.2) @[15] 0x1004367e nhttpstack.0) virtual [18] 0x7c57438b KERNEL32. stksize=2432 ### EAX=0x00000002.37dccc4) @[16] 0x1003f03f nhttpstack. When using a thread ID.MHMNotesPanic+641 (90df7fc.2) [ 7] 0x082c131f mhmdsapi._HTThreadBeginProc@4+63 (37dccc4.6f530.MHMDispatchFilter+273 (17603e8.90dfd08. @@> DBH 531 NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items 7330 0x24ff 0x0001 0x0200 Yes 1 4 .0 SizeLimit = 0. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 11] DBH= 104. Users=-1. size=2984 Size Database 2984 d:\notedata\drmail\jsmith. User=CN=Sithlord/O=SET By: [ nserver:0874: 110] DBH= 80. RefCnt = 15. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 9] DBH= 97. In the original format for NSD. as well as the name of the user that has the document opened (squared in blue): <@@ -----. nlrdrs=0 Writer=[] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1. User=CN=Sithlord/O=SET ----------.-----. Dirty = N DB Sem = (FRWSEM:0x0244) state=0.Notes Memory Analyzer (memcheck) -> Open Documents (BLK_OPENED_NOTE): . refcnt=0 Writer=[] n=0.nsf Version = 43.Open Documents --------- Database handle DBH 100 103 100 100 103 103 NOTEID 2310 2310 2306 2302 2306 2302 HANDLE 0x1a32 0x1a35 0x1a33 0x1a34 0x1a36 0x1a37 CLASS 0x0001 0x0001 0x0001 0x0001 0x0001 0x0001 FLAGS 0x0100 0x0100 0x0100 0x0100 0x0100 0x0100 FirstItem [ 6706: 816] [ 6709: 816] [ 6707: 816] [ 6708: 816] [ 6710: 816] [ 6711: 816] [ [ [ [ [ [ LastItem 6706:3120] 6709:3120] 6707:2988] 6708:3160] 6710:2988] 6711:3160] The Newer Format for NSD lists the database name directly in the Open Documents section. wcnt=-1. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 10] DBH= 100. waiters=0. you can correlate an open note to its database handle using the following two sections (the info of interest is squared in blue): • Open Databases (Database Handle & Database Name) • Open Documents (Database Handle & NoteID) -----. Open By: CN=John Smith/O=ACME/C=US Flags2 = 0x0404 Flags3 = 0x0000 OrigHDB = 531 First Item = [ 9471: 836] Last Item = [ 9471: 1228] Non-pool size : 0 Member Pool handle=0x24ff.Open Databases ------D:\Lotus\DominoR65\Data\mayhem. refcnt=0. Owner=[] By: [ nserver:0874: 110] DBH= 77. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 9] DBH= 98.nsf Rob Gearhart/Elliott Harden/Joe Wallace 40 Lotusphere 2008 Session HND107 .. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 10] DBH= 101.. User=CN=Sithlord/O=SET By: [ nHTTP:113c: 11] DBH= 103.Lesson II – Database Handle Database Handle Locations Each users DBH is listed in several different sections within Memcheck. WarningThreshold = 0 ReplicaID = 86256de1:0082e471 bContQueue = NSFPool [ 2: 40868] FDGHandle = 0xf01c01a8. [ 0020e005] NO 0x00c2 1518 551/1432 551/1432 CurrentCollation = 0 .-----..Flags 0 0: Ascending..Notes Memory Analyzer (memcheck) -> NIF Collections (Time 12:48:35) -----.--.Items 1.-----.Items 1..------..Notes Memory Analyzer (memcheck) -> Open Databases (Time 12:47:58) -----. "StartDateTime". Open By ... wcnt=-1. The sections that allows you to correlate view name and database name are as follows (the info of interest is squared in blue): • NIF Collections (ViewNoteID or Collection VB) • NIF Collection Users (ViewNoteID or Collection VB. waiters=0. summary# 2 CollIndex = [ 00012c09] Collation 1:BufferSize 26.@@> D:\Lotus\Domino\Data\HR\projnav.. Dirty = N DB Sem = (FRWSEM:0x0244) state=0. nlrdrs=0 Writer=[] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1. refcnt=0.-----------[ 0020e005] 1518 1356a8 358710 1 0x0000 00000008 NO NO NO NO 0 MyNotices CIDB = [ 0253cc05] CollSem (FRWSEM:0x030b) state=0. Database Handle. CollectionVB Remote OFlags ViewNoteID Data HDB/Full View HDB/Full -----------..------------. by KEY..Notes Memory Analyzer (memcheck) -> NIF Collection Users (hash) (Time 12:48:33) -----. Thread ID) • Open Databases (Database Handle. [ 0023d005] NO 0x0082 786 1219/1874 1219/1874 CurrentCollation = 0 [ 0013a805] .-------.------------[ 00239805] .. Users=-1.-----. -------------.nsf Version = 41.-------.Flags 0 0: Descending. Owner=[] By: [ nSERVER:09d8: 03b0] DBH= 551. "StartDateTime".------.------. [ 00136005] NO 0x00c2 11122 886/785 886/785 CurrentCollation = 0 [ 0028d805] .---------. [ nserver: 09d8: .. Thread ID) <@@ -----.-----... since the view name is listed in one place. refcnt=0 Writer=[] n=0.Lesson III – View Information View Name and Handle Info Determining view information can be a bit challenging.---... [ nserver: 09d8: 09d8: 04ca] 0266] 03b0] <@@ -----. Database Name.@@> CollectionVB ViewNoteID UNID OBJID RefCnt Flags Options Corrupt Deleted Temp NS Entries ViewTitle -----------.. and the DBH to the database is listed in another. and the name of the database is listed in yet another.@@> CollUserVB . [ nserver: . waiters=0.. -----------. nlrdrs=0 Writer=[ : 0000] NumCollations = 2 bCollationBlocks = [ 001e72e5] bCollation[0] = [ 00117005] bCollation[1] = [ 001a2205] CollIndex = [ 00012a09] Collation 0:BufferSize 26.---------. User=CN=Rhonda Smith/O=ACME/C=US By: [ nSERVER:09d8: 02ae] DBH= 128. refcnt=0.. User=CN=Rhonda Smith/O=ACME/C=US Rob Gearhart/Elliott Harden/Joe Wallace 41 Lotusphere 2008 Session HND107 ..-----. WarningThreshold = 0 ReplicaID = 862568fe:0019c2ad bContQueue = NSFPool [120: 7236] FDGHandle = 0xf01c0928. User=CN=Rhonda Smith/O=ACME/C=US By: [ nSERVER:09d8: 132] DBH= 2023.. summary# 2 ResponseIndex [ 0010e4b6] NoteIDIndex [ 0010e385] UNIDIndex [ 0010e5e7] <@@ -----. by KEY. RefCnt = 3..0 SizeLimit = 0. We have given you just a taste of what you can do with NSD. and what resources it was doing it do. Rob Gearhart/Elliott Harden/Joe Wallace 42 Lotusphere 2008 Session HND107 . views. Once you know how to use NSD to your advantage. NSD can give you surprising insight into the nature of a crash or hang.LOG. While NSD is not the proverbial magic bullet. Needless to say. SEMDEBUG. NSD will get you moving solidly in the right direction. and tells you information about what a server or client was doing at the time of a problem.TXT and relevant OS diagnostics. Used in conjunction with other diagnostic files such as CONSOLE. but hopefully this checklist can give you a broad overview for what to review: Call Stacks – locate crashing process/thread • KEYWORD “fatal” or “panic” • What is the physical thread ID? • What was the crash point? Memcheck – Shared Memory – determine Domino shared memory usage • KEYWORD “Shared Memory” • KEYWORD “Top 10 Shared Block Usage” • KEYWORD “Open Databases” • KEYWORD “Open Documents” • KEYWORD “NIF Collections” & “NIF Collection Users” • KEYWORD “Shared OS” or “MM/OS” to determine Panic Message Memcheck – Private Memory – determine Domino private memory usage • KEYWORD “Process Heap Memory” • KEYWORD “Top 10” Process Block Usage • KEYWORD “TLS Mapping” Resource Usage Summary – determine databases. you can use it to determine the next steps in troubleshooting a problem. and reduce time to resolution while improving your grasp of the issue. we cannot capture the entirety of NSD analysis here.Section VII – NSD Checklist NSD Check List Below is a suggested checklist that you as an administrator might put together for looking over an NSD. if not the root-cause itself. documents opened by thread • KEYWORD “Mapped to: PThread” Parting Words of Wisdom What should you as an administrator remember about NSD? It’s one of our most important Domino diagnostics. it will have to wait for the indexer to finish the rebuild and then unlock the semaphore.TXT located in the IBM_TECHNICAL_SUPPORT directory. a task might only wait a few microseconds and hence not time-out.txt. As a result. this is considered a semaphore timeout and in debug mode a message will be logged to the console. timing out every 30 seconds. an admin should be concerned. Semaphores are commonly used in software programs to claim resources and synchronization. Semaphores can occur during daily housekeeping depending on the size of the database.Section VIII . If under 5 minutes. Based upon the scope of the work. If the semaphores lasted longer then 5 minutes. Semaphores are not bad. only allowing one train on the track at a time. it does not mean this is a bad thing. With a complicated view on a large database. If a Domino task has a semaphore locked for over 30 seconds.Semaphore Debugging 101 What are semaphores? (Taken from knowledge base article #1094630) In a multitasking environment there is often a requirement to synchronize the execution of various tasks or ensure one process has been completed before another begins. Domino will record this information in the file SEMDEBUG. The function of this is to work in much the same way a railway signal would. The task will continue to wait for the semaphore. This requirement is facilitated by the use of a software switch known as a Semaphore or a Flag. Semaphores can occur when a large task is processing a job such as performing name changes on 500 people in the middle of a production day. A semaphore timeout is where the railway signal has been set in one state too long. For most operations. it locks a semaphore so that other tasks cannot use the index until it is rebuilt. If it is stuck for more than 30 seconds. it could indicate a large job is being processed. until the semaphore is unlocked or the task is ended. the task may have to wait several minutes for the index semaphore. the user task is stuck until that semaphore is unlocked. While it is stuck waiting for the semaphore. While semaphores can occur during the day. Typically an admin would only concern themselves with semaphores that happen during a production day. maybe because the train has broken down. etc. Before we continue lets debunk the myth "Semaphores are bad!!" NO. If a user task now tries to open that index while it is being rebuilt. Rob Gearhart/Elliott Harden/Joe Wallace 43 Lotusphere 2008 Session HND107 . If semaphores are seen during the production day. this is an indication that the job being processed should be reevaluated to determine if this is the right server to process this job or move the job during off hours. it may take longer than 30 seconds to complete which would result in this information being recorded in semdebug. Semdebug Example: An example of this in Notes/Domino is when the indexer needs to completely rebuild an index. it keeps track of how long it has been waiting. iii. you will find one DBG section before the callstacks were dumped and another after the callstacks were dumped. Identify the semaphores which occurred during the time that the callstacks where dumped into the NSD.Section VIII . If Process Info is not found skip to the next step. If neither of the above are not found. Your only choice is to utilize the time the NSD started and ended. In order to get a clearer picture of what is occurring during the time of the semaphore timeout. Open the NSD and search for the keyword DBG. Convert Time date stamps within semaphore debug from Julian date to calendar date . 1. i. Open the NSD captured during the occurrence of the semaphore timeouts and search for the keyword "Process Info". Below you will find a cheat sheet that details instructions on making the correlation. an older version of NSD is installed. This is found at the time end of the NSD output. An admin can identify semaphores which occurred during the time frame of the callstack dump. The line you are searching for looks similar to this DBG(10d8) 15:27:17.Semaphore Debugging 101 – Cheat Sheet Semaphore Timeouts – Cheat Sheet In order to troubleshoot semaphore timeouts. 2. IBM provides a tool to PSP level customers to assist in this conversion. Time stamps are notated differently based upon which version of NSD is utilized. the semdebug output must be cross referenced against the NSD taken during the semaphore occurrence. it helps to have a NSD that was taken during the occurrence of the semaphore timeouts. the sections are time stamped. Typically DBG is found before the NSD section "Process Table" and after "MEMCHECK". The time stamp will be listed here. 1. This version will not allow you to definitively locate the exact time the callstacks were dumped. Started at: Tue Jun 19 13:20:38 2007 Ended at: Tue Jun 19 13:22:24 2007 Time Spent: 00:01:46 Continued on Next Page Rob Gearhart/Elliott Harden/Joe Wallace 44 Lotusphere 2008 Session HND107 . a. <@@@@@@@@@@@@@@@@@@> Section: Notes Process Info (Time 13:21:20) <@@@@@@@@@@@@@@@@@> ii. As each section is outputted in the newer versions of NSD. or reader section and locate the callstack which belong to the process id and thread id in the NSD. 4. the focus should not be placed on the thread waiting on the other thread to complete. Use the knowledge base to see if this is a known issue by searching for the callstacks in questions.1STREADER=18A0:1D20) FOR 30000 ms Rob Gearhart/Elliott Harden/Joe Wallace 45 Lotusphere 2008 Session HND107 . The goal is to see if the multiple threads are waiting on the same process id and thread id.txt file with the time and date stamps converted to calendar date and locate the semaphore timeouts which occurred during the time window of the callstacks being dumped out. Typically you will not use semaphore timeouts which occurred after or before the time and date stamps of the callstacks from the NSD. an admin should pay particular attention to the process id and thread id of the threads which own the semaphore.Semaphore Debugging 101 – Cheat Sheet Semaphore Timeouts – Cheat Sheet 3.W=0. NOTE: Once the semaphores that occurred during the NSD output time frame are identified. take the process id and thread id from the owner. Example to denote what this looks like in a semdebug: WAITING FOR READ LOCK ON FRWSEM b.Section VIII . While this information is important. a. i. The focus should be placed on the thread which needs to complete to free up the resource for other threads to use.WRITER=0000:0000. If an admin sees a multiple thread waiting pattern. Next. Example to denote what this looks like in a semdebug: WAITING FOR SEM c. An admin may see output which shows a semaphore waiting on anther semaphore. The waiting thread is an innocent bystander at this point i. open the Semdebug. the admin can start the focus of this investigation with this particular thread. An admin should place primary focus on threads which have a read or write lock within the semdebug file. writer. Once the semaphores which occurred during the callstack output are identified. Semaphore Example 04/17/2007 09:02:18 AM EDT sq="0003CE47" THREAD [1DC8:000C-11DC] WAITING FOR SEM 0x1149 Router DB Cache semaphore (@05603F73) (OWNER=1DC8:1568) FOR 30000 ms Time and Date Call stacks used to cross reference with NSD 04/17/2007 09:02:18 AM EDT sq="0003CE48" THREAD [18A0:00D5-1B70] WAITING FOR READ LOCK ON FRWSEM 0x03CE Monitor manager: Overall control semaphore (@02FA7B48) (R=1. and determine next steps. but where you can use what use see in the file to narrow down the problem. You are given a brief explanation of the scenario along with the NSD from the server. we may only ask you what the next step would be in a particular situation. Instead. you are given an NSD file. and should not be taken with you when you leave the lab.Section VIIII – Case Studies Case Studies The following are a series of scenarios intended to improve your working knowledge of NSD. To help get your feet wet. and in some cases. supporting files. we will go over an example scenario as a class.but no peeking! The provided NSD files should remain in the lab. We are not asking you to resolve a problem top to bottom. See the answer key at the end . where you may not be able to definitively solve a problem from one NSD. This echoes real life. and asked a series of questions intended to guide you through the files. In each scenario. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 46 Lotusphere 2008 Session HND107 . using “FormatFromVariant” as our search criteria. you will need to convert the NoteID from decimal format (as taken from the NSD) to hexadecimal format (as inputted into the Admin Client). then from tools select Find Note. class=0200. within the Admin client. you would search the database for NoteID 1416 (that’s 5142 in hex). Question 1: What is the process ID and thread ID of the crashing thread? Answer: Process= AMgr.nsf. and opened by user name SERVER01/SVR/ACME is instance of the same agent note opened by the server in preparation to run the agent (either one gives you the correct note ID). Question 5: From the fatal call stack. When searching the database. you can read the “$TITLE” field to establish the agent name. Locate someDB. Password is "password". Example 1 Details The note listed with HDB=35 (database handle). TN 1187989.nsf’. Process ID=0B04.log Problem: AMGR has crashed on an agent. Thread ID= 1288 Question 2: What is the note ID and database name for the crashing agent? Answer: Database=’SomeDB. Enter the converted NoteID and locate the $Title field in order to get the Agents name of NotesRocks.Example 1 Example 1 File: example_1_nsd. Open the Notes Admin Client located on your Desktop. The note with HDB=16. Agent NoteID=5142 Question 3: What is the virtual thread ID for the crashing thread? Answer: Virtual Thread ID=0002 Question 4: How would you establish the name of the agent that crashes and what is the agent's name? Answer: Use the Admin Client and search the database using the NoteID to pull up the agent design note. from there. class=0200 (agent note) and opened by AGTSIGNER/AGT/ACME (signer of the agent) is the agent note we are interested in. Navigate to the Files section. The two notes have the same Note ID which would be the disguising fact of the note referencing the same agent or a different agent. can you match this against any known SPRs? Answer: Yes. SPR HRON65SL9C. Continued on next page Rob Gearhart/Elliott Harden/Joe Wallace 47 Lotusphere 2008 Session HND107 . In this case. 0) @[10] 0x6096d745 nnotes...LSsThread::NRun+6683 (58014b0. doc: HDB=502.41564154) [32] 0x7c598989 KERNEL32. doc: HDB=502.0.c26268. doc: HDB=502. By: CN=AGTSIGNER/OU=AGT/O=ACME ...12e4b8) @[ 2] 0x6265a763 nlsxbe.78652e72) @[28] 0x0040102f nAMgr..0.LScObjCli::SetPropEXP+53 (5792368. By: CN=SERVER01/OU=SVR/O=ACME .....5802050. CS=0x0000001b.12e9b8. flags=0000 .12e4c0...12e9d4. DBH: 502. noteID=1194. h=0xf0104037 t=c30a (BLK_LOOKUP_THREAD) . PC=0x60683e2d. ID=966._ProcessRunMessage@4+65 (12fb54. DBH: 113.3a0043.._main+22 (3. EDI=0x00000000.LSsInstance::AdtCallBack+226 (c26268. flags=0000 ..7b2df8.400000.628b2fc6._AddInMain@12+316 (400000. By: CN=AGTSIGNER/OU=AGT/O=ACME .. h=0xf0104001 t=c130 (BLK_TLA) ..78652e72.3. class=0001..log Agent Note Rob Gearhart/Elliott Harden/Joe Wallace 48 Lotusphere 2008 Session HND107 .626427e0._main+246 (0. SOBJ: addr=0x01482b7c.ANNote::ANSetProp+252 (12e9b8. F:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.c226d4.14ad978..3c6._NotesMain@8+47 (3.nsf . Database: X:\Lotus\Domino\Data\Apps\SomeDB.0) @[12] 0x600231b6 nnotes.15d3bbbc) @[ 4] 0x6265ec49 nlsxbe.5802050. SOBJ: addr=0x014c8884.. DBH: 485.ANItem::ANItem+105 (15d3bbcc.f36. GS=0x00000000 Flags=0x00010293 Exception code: c0000005 (ACCESS_VIOLATION) K-Base ############################################################ @[ 1] 0x60683e2d nnotes..LsConv::FormatFromVariant+541 (c26268.ANotes::ANFormatFromVariant+67 (5802050.... DBH: 504.12ea04) @[ 7] 0x62643309 nlsxbe. H=44.12e4b8.. Continued Example 1: Correlating the Data Process ID: Thread ID Fatal Call Stack: ############################################################ ### FATAL THREAD 1/4 [ nAMgr:0b04:1288] ### FP=0x0012e464.3d550065) @[29] 0x00401136 nAMgr. H=40..0._tag_NotesADTControl::ClassControl+25 (15d385dc.0) @[ 5] 0x6266e2fc nlsxbe.. h=0xf0104035 t=c275 (BLK_NSFT) .. SOBJ: addr=0x014deea4. doc: HDB=113. Forecast .c226e0) @[26] 0x628b2fc6 namgrdll. H=16._mainCRTStartup+227 (78652e72. ID=6370.0. file: fd: 2780.. cg=N.. ID=966.57923a8) @[ 9] 0x60091715 nnotes..nsf . SS=0x00000023 Search in ### DS=0x00000023.FindFirstChangeNotificationA+16359 (401212.12e4bc) @[ 3] 0x6265f0da nlsxbe. h=0xf0104013 t=c176 (BLK_SDKT) . flags=0100 . DBH: 503.3...3d550065.108.57923c4) @[ 6] 0x6264290a nlsxbe. H=35.. ID=0.ANItem::ANICreateFromVariant+42 (15d3bbcc..403000) @[31] 0x004012f5 nAMgr. class=0200.c226d4. SOBJ: addr=0x014c8580. FS=0x00000038..0.108.108. class=0200.c26268) <stack frames removed> @[23] 0x628b403d namgrdll.Mapped To: PThread [ nAMgr:0b04:1288] .._ANCLASSCONTROL@16+298 (c26268. By: CN=AGTSIGNER/OU=AGT/O=ACME . stksize=64 ### EAX=0x00000006.0) @[11] 0x60022abb nnotes.0) @[24] 0x628b37e1 namgrdll. SP=0x0012e424.12eb24._ExecutiveMain@4+230 (0.12fc84...._ProcessMessage@4+62 (12fb54...7b27e0..5802050.1.15d385b0. By: CN=AGTSIGNER/OU=AGT/O=ACME .7ffdf000. class=0001. flags=0000 .7ffdf000) @[27] 0x628b5b4c namgrdll..57005c) From Resource Usage Summary: ** VThread [ nAMgr:0b04:0002] Virtual Thread ID .LScObjCli::SetPropertyData+117 (5792368..12e4c0.. file: fd: 2252. EDX=0x3b9c0280 ### ESI=0x05802050. ECX=0x00000000.3) @[30] 0x00401056 nAMgr.. view: hCol=505..15d3874e.. EBX=0x00000000.f36. SOBJ: addr=0x15ce00b4..12fb54) @[25] 0x628b377e namgrdll.6095176d.12e9b8) @[ 8] 0x60059212 nnotes.5802050. X:\Lotus\Domino\Data\Apps\SomeDB._RunTask@32+2045 (1.Example 1 .3.LSsThread::Run+182 (58014b0. h=0xf0104015 t=c436 (BLK_LSITLS) .0. ES=0x00000023.0.
Copyright © 2024 DOKUMEN.SITE Inc.