Kernel Internals Student Notes

V2.0.0.3 cover Front cover AIX 5L Kernel Internals (Course Code BE0070XS) Student Notebook ERC 4.0 eServer UNIX Technical Education IBM Certified Course Material Student Notebook Trademarks The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies: IBM® is a registered trademark of International Business Machines Corporation. The following are trademarks or registered trademarks of International Business Machines Corporation in the United States, or other countries, or both: AIX® Chipkill™ Electronic Service Agent™ LoadLeveler® pSeries™ S/370™ zSeries™ AIX 5L™ DB2® IBM® NUMA-Q® PTX® Sequent® AS/400® DFS™ iSeries™ PowerPC® RS/6000® SP™ ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States, other countries, or both. Intel is a trademark of Intel Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States and other countries. Other company, product and service names may be trademarks or service marks of others. June 2003 Edition The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. © Copyright International Business Machines Corporation 2001, 2003. All rights reserved. This document may not be reproduced in whole or in part without the prior written permission of IBM. Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. V2.0.0.3 Student Notebook TOC Contents Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Course Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Agenda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Unit 1. Introduction to the AIX 5L Kernel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 Operating System and the Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3 Kernel Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5 Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7 Mode and Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-9 Context Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-11 Interrupt Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-13 AIX 5L Kernel Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16 AIX 5L Execution Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18 System Header Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-20 Conditional Compile Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-22 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-24 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-25 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-26 Unit 2. Kernel Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2 What tools will you be using in this class? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3 The Major Functions of KDB are: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4 Enabling the Kernel Debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6 Verifying the Debugger is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8 Starting the Debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9 System Dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10 kdb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-13 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-15 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-16 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17 Unit 3. Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2 Parts of a Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 1:1 Thread Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7 M:1 Thread Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8 M:N Thread Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9 Creating Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-11 Creating Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13 © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Contents iii Student Notebook Process State Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-15 The Process Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-18 pvproc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-20 pv_stat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-21 Table Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-22 Extending the pvproc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-24 PID Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-26 Finding the Slot Number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-28 Kernel Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-29 Thread Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-31 pvthread Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-33 TID Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-34 u-block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-35 Six Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-37 Thread Scheduling Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-39 Thread State Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-40 Thread Priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-43 Run Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-45 Dispatcher and Scheduler Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-46 Dispatcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-47 Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-48 Preemption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-49 Preemptive Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-51 Scheduling Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-53 SMP - Multiple Run Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-56 NUMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-58 Memory Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-60 Global Run Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-62 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-64 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-65 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-66 Unit 4. Addressing Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-2 Memory Management Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-3 Pages and Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-4 Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-6 Translating Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-8 Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-9 Segment Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-11 32-bit Hardware Address Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-13 64 Bit Hardware Address Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-15 Segment Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-16 Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-19 shmat Memory Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-21 Memory Mapped Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-23 32-bit User Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-26 32-bit Kernel Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-28 iv Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook TOC 64-bit User/Kernel Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-29 4-31 4-32 4-33 Unit 5. Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2 Virtual Memory Management (VMM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 Object Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 Demand Paging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-7 Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-10 Hardware Page Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-12 Page not in Hardware Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13 Page on Paging Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-15 External Page Table (XPT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-16 Loading Pages From the File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-18 Object Type / Backing Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-20 Paging Space Management Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-21 Paging Space Allocation Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-23 Free Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25 Clock Hand Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-27 Fatal Memory Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-29 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-30 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-31 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32 Unit 6. Logical Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-3 Physical Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-5 Logical Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7 Components Required for LPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9 Operating System Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13 Virtual Memory Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14 Real Address Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15 Real Mode Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-17 Operating System Real Mode Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19 Address Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21 Allocating Physical Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23 Partition Page Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-25 Translation Control Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-27 Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-29 Dividing Physical Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34 Unit 7. LFS, VFS and LVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-2 © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Contents v Student Notebook What is the Purpose of LFS/VFS? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-3 Kernel I/O Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-5 Major Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-7 Logical File System Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-9 User File Descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-11 The file Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-13 vnode/vfs Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-15 vnode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-17 vfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-19 root (l) and usr File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-21 vmount . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-23 File and File System Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-25 gfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-27 vnodeops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-29 vfsops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-31 gnode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-33 kdb devsw Subcommand Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-35 kdb volgrp Subcommand Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-37 AIX lsvg Command Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-39 kdb lvol Subcommand Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-40 AIX lslv Command Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-44 kdb pvol Subcommand Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-46 AIX lspv Command Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-48 Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-49 Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-50 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-51 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-52 Unit 8. Journaled File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-2 JFS File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-3 Reserved Inodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-7 Disk Inode Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-9 In-core Inodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-11 Direct (No Indirect Blocks) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-15 Single Indirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-17 Double Indirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-18 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-19 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8-20 Unit 9. Enhanced Journaled File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-2 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-3 Aggregate and Fileset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-4 Aggregate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-6 Allocation Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-9 Fileset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-11 Inode Allocation Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-13 vi Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook TOC Extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Increasing an Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Binary Tree of Extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Inodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Inline Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Binary Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . More Extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Continuing to Add Extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Another Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fsdb Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Directory Root Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Directory Slot Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Small Directory Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adding a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adding a Leaf Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adding an Internal Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-14 9-16 9-18 9-20 9-26 9-27 9-28 9-29 9-30 9-32 9-34 9-35 9-37 9-39 9-41 9-42 9-43 9-44 9-45 9-46 9-47 Unit 10. Kernel Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1 Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2 Kernel Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3 Relationship With the Kernel Nucleus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-5 Global Kernel Name Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-6 Why Export Symbols? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9 Kernel Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-11 Configuration Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-13 Compiling and Linking Kernel Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-15 How to Build a Dual Binary Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-19 Loading Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-21 sysconfig() - Loading and Unloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-22 sysconfig() - Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-23 sysconfig() - Device Driver Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-24 The loadext() Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-26 System Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-28 Sample System Call - Export/Import File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-30 Sample System Call - question.c . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-31 Sample System Call - Makefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-32 Argument Passing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-33 User Memory Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-35 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-38 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-39 Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-40 © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Contents vii Student Notebook Appendix A. Checkpoint Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 Appendix B. KI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Crash Dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Process Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . About This Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1 B-2 B-3 B-5 B-6 viii Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. product and service names may be trademarks or service marks of others. LANDesk. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Trademarks ix . or both. other countries. The following are trademarks or registered trademarks of International Business Machines Corporation in the United States. MMX.3 Student Notebook TMK Trademarks The reader should recognize that the following terms.V2. 2001. Intel is a trademark of Intel Corporation in the United States. Windows NT. Pentium and ProShare are trademarks of Intel Corporation in the United States. or both: AIX® Chipkill™ Electronic Service Agent™ LoadLeveler® pSeries™ S/370™ zSeries™ AIX 5L™ DB2® IBM® NUMA-Q® PTX® Sequent® AS/400® DFS™ iSeries™ PowerPC® RS/6000® SP™ ActionMedia. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company. Windows. are official trademarks of IBM or other companies: IBM® is a registered trademark of International Business Machines Corporation. which appear in the content of this training document. Linux is a registered trademark of Linus Torvalds in the United States and other countries. Microsoft.0. © Copyright IBM Corp.0. or both. Java and all Java-based trademarks are trademarks of Sun Microsystems. Inc. or both. other countries. other countries. or both. in the United States. or other countries. other countries. and the Windows logo are trademarks of Microsoft Corporation in the United States. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook x Kernel Internals © Copyright IBM Corp. 2001. These skills can be obtained by attending the following courses or through equivalent experience: — Introduction to C Programming . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.AIX/UNIX (Q1070) — AIX 5L System Administration II: Problem Determination (AU16/Q1316) In addition.V2.1 and 5. the following courses are helpful: — KornShell Programming (AU23/Q1123) — AIX Application Programming Environment (AU25/Q1125) © Copyright IBM Corp. 2001.0. pipes. working knowledge of AIX system calls. and Input/Output (I/O) redirection.0. Additionally knowledge of basic system administration skills is required. such as the use of SMIT. and user-level working knowledge of AIX/UNIX. It is designed to provide background information useful to support engineers and AIX development/application engineers who are new to the AIX 5L Kernel environment as implemented in AIX releases 5. configuring file systems and configuring dump devices. This course also provides background knowledge helpful for those planning to attend the AIX 5L Device Driver (Q1330) course. including editors. Audience — AIX technical support personnel — Application developers who want to achieve a conceptual understanding of AIX 5L Kernel Internals Prerequisites Students are expected to have programming knowledge in the C programming language. shells.3 Student Notebook pref Course Description AIX 5L Kernel Internals Concepts Duration: 5 days Purpose This is a course in basic AIX 5L Kernel concepts.2. Course Description xi . and the relationships between them — Describe the layout of the segmented addressing model. . and how logical to physical address translation is achieved — Describe the operation of VMM subsystem and the different paging algorithms — Describe the mechanisms used to implement logical partitioning — Understand the purpose of the logical file system and virtual file system layers and the data structures they use — List and describe the components and function of the JFS2 and JFS file systems — Identify the steps required to compile.Student Notebook Objectives At the end of this course you will be able to: — List the major features of the AIX 5L kernel — Quickly traverse the system header files to find data structures — Use the kdb command to examine data structures in the memory image of a running system or system dump — Understand the structures used by the kernel to manage processes and threads. link and load kernel extensions xii Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. Enhanced Journaled File System .LFS.Enhanced Journaled File System .Enhanced Journaled File System .Kernel Analysis Tools lecture Exercise 2 .Topic 2 lecture Exercise 8 .Addressing Memory lecture Day 3 Daily review Exercise 4 .Introduction to the AIX 5L Kernel lecture Exercise 1 .Kernel Analysis Tools Day 2 Daily review Unit 3 .0. 2001.0.Logical Partitioning lecture Day 4 Daily review Unit 7 .Topic 1 lecture Exercise 7 .3 Student Notebook pref Agenda Day 1 Welcome Unit 1 .Introduction to the AIX 5L Kernel Unit 2 . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Memory Management lecture Exercise 5 .Kernel Extensions lecture Exercise 9 . VFS and LVM Unit 8 .Addressing Memory Unit 5 .LFS.Topic 2 Day 5 Daily review Unit 10 . VFS and LVM lecture Exercise 6 .V2.Enhanced Journaled File System .Process Management lecture Exercise 3 .Journaled File System lecture Unit 9 . Agenda xiii .Kernel Extensions © Copyright IBM Corp.Process Management Unit 4 .Topic 1 Unit 9 .Memory Management Unit 6 . . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook xiv Kernel Internals © Copyright IBM Corp. identify data element types for each of the available kernels in AIX 5L How You Will Check Your Progress Accountability: • Exercises using your lab system • Check-point activity • Unit review References The Design of the UNIX Operating System.com/pseries/en_US/infocenter/base/aix.0. Introduction to the AIX 5L Kernel 1-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. you should be able to: • Describe the role the kernel plays in an operating system • Define user and kernel mode and list the operations that can only be performed in kernel mode • Describe when the kernel must make a context switch • Describe the role of the mstsave area in a context switch • Name the execution environments available on each of the platforms supported by AIX 5L • Using the system header files. Introduction to the AIX 5L Kernel What This Unit Is About This unit describes the purpose. Bach. concepts and features of the AIX 5L kernel. 2003 Unit 1. ISBN: 0132017997 AIX Online Documentation: http://publib16. .0.ibm. by Maurice J.boulder.3 Student Notebook Uempty Unit 1.V2.htm © Copyright IBM Corp. What You Should Be Able to Do After completing this unit. 2001. 2001.0 Notes: 1-2 Kernel Internals © Copyright IBM Corp. Unit Objectives BE0070XS4. identify data element types for each of the available kernels in AIX 5L Figure 1-1. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .Student Notebook Unit Objectives At the end of this unit you should be able to: Describe the role the kernel plays in an operating system Define user and kernel mode and list the operations that can only be performed in kernel mode Describe when the kernel must make a context switch Describe the role of the mstsave area in a context switch Name the execution environments available on each of the platforms supported by AIX 5L Using the system header files. Operating System and the Kernel BE0070XS4. © Copyright IBM Corp. 2001. The kernel prioritizes these requests and manages the hardware through its hardware interface.0 Notes: Operating system The principal purpose of the AIX operating system is to provide an environment where application programs can be executed. Kernel The kernel is the base program of the operating system.3 Student Notebook Uempty Operating System and the Kernel Process system call Interface Process Process Kernel hardware Interface CPU CPU tty CPU Figure 1-2.V2. It provides the system call interface allowing programs to request use of the hardware. Introduction to the AIX 5L Kernel 1-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. . 2003 Unit 1.0. CPU and IO. It acts as intermediary between the application programs and the computer hardware. This mainly involves the management of hardware resources including memory. Student Notebook The kernel is the key program The operating system is made up of many programs including the kernel. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 1-4 Kernel Internals © Copyright IBM Corp. . if the kernel is not running nothing else in the operating system can function. It is safe to say that the kernel is the most important part of the operating system. This class discusses the internal working of the kernel in the AIX 5L operating system. 0 Notes: Introduction The kernel may be broken up into several sections based on the services provided to applications programs.V2. Virtual memory management The Virtual Memory Management (VMM) function of the kernel is responsible for managing all aspects of virtual and physical memory by processes and the kernel.3 Student Notebook Uempty Kernel Components Applications user kernel Buffered I/O Raw I/O File systems Disk space managment (LVM) I/O Subsystem Buffered I/O Process managment Device driver Device driver Virtual memory managment CPU CPU Disk tty Figure 1-3. Each of these sections are discussed in this class. 2003 Unit 1. along with scheduling threads on CPUs. . Introduction to the AIX 5L Kernel 1-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.0. Process management The process management function of the kernel is responsible for the creation.0. and termination of processes and threads. Kernel Components BE0070XS4. providing space for file © Copyright IBM Corp. The kernel components are shown in the visual above. This includes allocating physical page frames to virtual pages. I/O subsystem Parts of the kernel that interact directly with I/O devices are called device drivers. File system AIX supports several types of file systems including JFS. Typically each type of device installed on the system will require its own device driver. 2001. This class covers the JFS and JFS2 file systems. 1-6 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. JFS2. The file system software interacts with the disk space management software. Device drivers are covered in detail in a separate class on writing device drivers. Disk space management The management of disk space in AIX is handled by a layer above the disk’s drivers. . The Logical Volume Manger (LVM) provides the function of disk space management.Student Notebook system buffering and keeping track of which process memory is resident in physical memory and which is stored on disk. NFS and several CD-ROM file systems. The address translation tables are controlled by the kernel. 2001. .V2. Addresses referenced by a user program do not directly reference physical memory. A process’ address space contains both user. Virtual address space By using the concept of virtual memory. 2003 Unit 1. instead they reference a virtual address. Translation tables are used by the hardware to map virtual to physical addresses. each process on the system can appear to have its own address space that is separate and isolated from other processes.and kernel-memory addresses. One set of address translation © Copyright IBM Corp.0. Introduction to the AIX 5L Kernel 1-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction AIX implements a virtual memory system.0. Memory management Virtual addresses are mapped by the hardware to a physical memory address.3 Student Notebook Uempty Address Space Process A Process B Process C Address space Address space Address space user kernel Figure 1-4. Address Space BE0070XS4. To switch from one process’ address space to another. 1-8 Kernel Internals © Copyright IBM Corp.Student Notebook tables is kept for each process. 2001. . the kernel loads the appropriate address translation table into the hardware. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Mode The computer hardware provides two modes of execution. Mode and Context BE0070XS4. a privileged kernel mode and a less-privileged user mode. 2003 Unit 1.3 Student Notebook Uempty Mode and Environment Process Environment Application code System Call Interrupt Environment Invalid combination . runs in kernel mode. Application programs must run in user mode thus are given limited access to the hardware. Introduction to the AIX 5L Kernel 1-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2.0. 2001. as you would expect. The following table compares these two modes. The kernel.0 Notes: Introduction Two key concepts of mode and environment are described in this section.0. © Copyright IBM Corp.interrupts always run in kernel mode User mode Kernel mode Kernel code Hardware interrupt Figure 1-5. . 1-10 Kernel Internals © Copyright IBM Corp. . In this context the kernel cannot access the user address space or any kernel data related to the user process that was running on the processor just before the interrupt occurred. Interrupts must be handled in kernel mode. although it is also possible to create a kernel-mode only process. Kernel memory is not accessible.Student Notebook User mode Memory access is limited to the user’s private memory. I/O instructions are blocked. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. Can’t modify hardware registers related to memory management. Environment The AIX kernel may execute in one of two environments: process environment or interrupt environment. Memory management registers may be modified. the kernel is running on behalf of a user process. In process environment. When the kernel responds to an interrupt. Kernel mode Can access all memory on the system. This generally occurs when a user program makes a system call. All I/O is performed in kernel mode. it is running in the interrupt environment. 3 Student Notebook Uempty Context Switches CPU context switch Thread 1 mstsave Saved: y CPUs registers y stack pointer y instruction pointer Thread 2 mstsave Saved: y CPUs registers y stack pointer y instruction pointer Figure 1-6. © Copyright IBM Corp.V2.0. Introduction to the AIX 5L Kernel 1-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 1. Context Switches BE0070XS4.0 Notes: Introduction A context switch is the action of exchanging one thread of execution on a CPU for another. The AIX kernel manages many threads of execution by switching the CPUs between the different threads on the system. Thread of execution Threads of execution are simply logical paths through the instructions of a program. . 2001.0. This context includes information such as the values of the CPU registers. the instruction address register and stack pointer. mstsave The context of the running thread must be saved when a context switch occurs. The CPU then performs a branch instruction to the address of the saved instruction pointer. A hardware interrupt occurs.Student Notebook Context switches Context switches can occur at two points: a. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. This information is saved in a structure called the mstsave (machine state save) structure. 1-12 Kernel Internals © Copyright IBM Corp. the system register values stored in the mstsave of the thread are loaded into the CPU. Each thread of execution has an associated mstsave structure. . Execution of the thread is blocked waiting for the completion of an event. Restoring a context When a thread is restored (switched in). b. 2001. however an interrupt is a transient entity and does not have its own thread structure. therefore.0. . This is because a thread structure has an mstsave structure.0. © Copyright IBM Corp.0 Notes: Introduction A hardware interrupt results in a temporary context switch. mstsave pool Interrupts can occur when the CPU is currently processing an interrupt. multiple mstsave areas are needed to save the context of each interrupt. the current context of the processor must be saved so that processing can be continued after handling the interrupt. Interrupt Processing BE0070XS4.3 Student Notebook Uempty Interrupt Processing current save area csa mstsave mstsave mstsave threads mstsave unused (next interrupt goes here) high priority interrupt low priority interrupt base interrupt level Figure 1-7.V2. 2003 Unit 1. 2001. AIX keeps a pool of mstsave areas to use. Introduction to the AIX 5L Kernel 1-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Each time an interrupt occurs. and place the best runnable thread at the end of the MST chain. 3. The steps to restore a context are shown in this table. or csa pointer. Working backwards from the highest priority interrupt to the lowest and finally to the base-level mstsave. This pointer is called the current save area. 1. Reload the registers from the processing the context. 1-14 Kernel Internals © Copyright IBM Corp. Branch to the instruction referenced by the instruction address register. Step Action If returning to the base interrupt level and the interrupt has made a thread runnable. Update the CPU’s csa pointer to point to the new mstsave area. 5. Link the just used mstsave to the new mstsave. Interrupt history When AIX receives an interrupt that is of higher priority than the one it is currently handling it must save the current state in a new mstsave area linking the new save area to the previous one. . Get the next available mstsave area from the pool. 4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Action Save the current context in the mstsave area pointed to by the CPU’s csa. Return the current mstsave area to the pool. Interrupt processing Saving context When an interrupt occurs. The last or base-level mstsave in the chain is the mstsave of the thread that was running when the first interrupt occurred. This forms a history of interrupt processing. Unwinding the interrupts As the processing of each interrupt is completed the chain of mstsave areas are unlinked. 2. 4. The dispatcher will move the thread originally on the end of the MST chain back to the run queue. 3.Student Notebook csa pointer Each processor has a pointer to the mstsave area it should use when an interrupt occurs. the steps AIX takes to save the currently running context are: Step 1. 2001. 2. Set the CPU’s csa pointer to the previous mstsave area. invoke the dispatcher. 2001.0. © Copyright IBM Corp.3 Student Notebook Uempty Finding the current mstsave The csa always points to an unused mstsave area. The data in this mstsave will not be valid except for its pointer to the next mstsave in the chain. Introduction to the AIX 5L Kernel 1-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. 2003 Unit 1. The last used mstsave area can be located by following the prev pointer from the mstsave pointed to by the csa. This mstsave will be used if a higher-priority interrupt occurs. .V2. Student Notebook AIX 5L Kernel Characteristics Preemptable kernel Pageable kernel memory Dynamically extensible kernel Figure 1-8. AIX improves real time processing by allowing for preemption in kernel mode. Many other UNIX kernels will not allow preemption to occur when running in kernel mode. Preemptable Preemptable means that the kernel can be running in kernel mode (running a system call for example) and be interrupted by another more important task. These features are listed above. AIX 5L Kernel Characteristics BE0070XS4. As an example. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction The AIX kernel was the first mainstream UNIX operating system to implement several important features. 1-16 Kernel Internals © Copyright IBM Corp. This can result in long delays in the processing of real time threads. Preemption causes a context switch to another thread inside the kernel. 2001. Linux does not support preemption when in kernel mode. and kernel-address space. Extensible The AIX kernel is dynamically extensible. AIX supports paging both user. Pinning memory Some areas of the kernel’s memory must stay resident meaning they may not be paged to disk. Most kernels support the paging of user-virtual-address space. This means that not all the code required for the kernel needs to be included in a single binary (/unix). Portions of the kernel memory may be paged out to disk when not needed. the kernel memory of the Linux operating system is resident in physical memory at all times.Extended system calls . Portions of the kernel’s code will be loaded at runtime. Introduction to the AIX 5L Kernel 1-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.V2. As an example.3 Student Notebook Uempty Pageable Not all of the kernel’s virtual memory space needs to be resident in physical memory at all times. Kernel extensions can include: . The ability to page kernel memory is a feature not found in all UNIX kernels.File systems © Copyright IBM Corp.0. Dynamically loaded modules are called kernel extensions. This keeps the kernel smaller and requires less memory. This allows for better utilization of physical memory. Kernel extensions typically add functionality that may not be needed by all systems.Device drivers . 2003 Unit 1. portions of device drivers must be pinned in memory.0. Areas of memory that are not subject to paging are called pinned memory. for example. . there is an added cost to managing a 64-bit address space. using left zero fill of addresses for the 32-bit kernel environment.0 Notes: Introduction AIX 5L supports both 32-bit and a 64-bit execution environments. 2001. The key to this 64-bit platform flexibility is that a 64-bit VMM (Virtual Memory Manager) is run in both cases. Not all applications will require the increased address space of the 64-bit kernel.Student Notebook AIX 5L Execution Environment 32-bit Hardware 64-bit Hardware 64-bit Hardware 32-bit Applications 32-bit Applications 64-bit Applications 32-bit Applications 64-bit Applications User Kernel 32-bit Kernel 32-bit Kernel 64-bit Kernel Figure 1-9. . a 32-bit kernel is provided. In these cases. 1-18 Kernel Internals © Copyright IBM Corp. However. On 32-bit hardware platforms only the 32-bit environment can be used. but on 64-bit platforms either can be used. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. AIX 5L Execution Environment BE0070XS4. 32-bit and 64-bit kernel The primary advantage of the 64-bit kernel is the increased kernel address space. This allows systems to support increased workloads. 1. some commands require both a 32-bit and a 64-bit version. Only 32-bit kernel extensions are supported under the 32-bit kernel. For example.3 Student Notebook Uempty Selecting a kernel The file /unix is a link to the kernel image file that is loaded at boot time. and in AIX 5. regardless of the kernel that is running. If a 64-bit kernel is detected. However. 2003 Unit 1. Depending on the hardware type and kernel type (32-bit or 64-bit) the link will point to the appropriate file as shown in this table. If a 32-bit kernel is detected.0.V2. For these commands. Action 32-bit version of command is run by user. The 32-bit command checks the kernel type (32. If a 64-bit kernel is detected. then the 64-bit version of the command is run. In later versions of AIX 5. Step 1.0. 3. User commands User level commands included with the AIX 5L operating system are designed to work with either the 32-bit or 64-bit kernel. . Hardware platform 32-bit or 64-bit 64-bit Kernel type 32-bit 64-bit Kernel file /usr/lib/boot/unix_mp /usr/lib/boot/unix_up /usr/lib/boot/unix_64 User applications Both 32-bit and 64-bit applications are supported when running on 64-bit hardware. the 32-bit version of the command will determine if a 32-bit or 64-bit kernel is running. under the initial release of AIX 5. Funneling is not supported on the 64-bit AIX 5L kernel. then a 64-bit version of the command is started. These are typically commands that must work directly with the internal structures of the kernel. Kernel extensions Only 64-bit kernel extensions are supported under the 64-bit kernel. 2001. 2.2. Introduction to the AIX 5L Kernel 1-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Earlier versions of AIX supported running non-SMP safe kernel extensions on SMP hardware using a mechanism called funneling. The steps are shown in this table.or 64-bit). the 32-bit command completes its execution. vmstat (along with other performance commands) uses a performance tools API.1 the command vmstat would run the command vmstat64. All kernel extensions must be SMP safe. © Copyright IBM Corp. 4. Student Notebook System Header Files / (root) usr include stdio.h j2 j2-btree.0 Notes: Introduction The system header files contain the definition of structures that are used by the AIX kernel.h inode.h jfs dir. We will reference these files throughout this class.h utherad.h signal.h fcntl.h types. 2001. .h sys proc.h jfsmount. since they contain the C language definitions of the structures we will be describing. System Header Files BE0070XS4.. Finding header files The drawing above shows the location of the system header files. 1-20 Kernel Internals © Copyright IBM Corp.h filsys. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.h ino.h j2-dinode.h thread.h Figure 1-10.h user.h j2-types.h j2-inode.h mode. .V2.3 Student Notebook Uempty Location of header files The /usr/include directory contains several sub-directories containing header files. 2001. Introduction to the AIX 5L Kernel 1-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 1.0. Some of the sub-directories are described in this table. Header file directories /usr/include /usr/include/sys /usr/include/jfs /usr/include/j2 Description General program header files Header files dealing directly with the operations of the system Header files for the JFS file system Header files for the JFS2 file system © Copyright IBM Corp.0. . The compiler directive #ifndef __64BIT_KERNEL is used to create different definitions for the 32-bit and 64-bit kernels. This value should always be used for 64-bit kernel extensions and device drivers. 1-22 Kernel Internals © Copyright IBM Corp. 32-bit or 64-bit). 2001. Example Shown here is a portion of the definition of a struct thread. Conditional Compile Values Notes: Conditional compile values Several conditional compiler directives are used in the system header files to select the platform and environment (32-bit or 64-bit kernel). This value should always be used when compiling kernel code. Code is being compiled for a 64-bit kernel.0 _KERNSYS _KERNEL _64BIT_KERNEL _64BIT Figure 1-11. This value should always be used when compiling kernel code. Code is being compiled in 64-bit mode. Compiling kernel extension or device driver code. BE0070XS4. Enable kernel symbols in header files. This is because certain data types have different sizes depending on the execution environment (for example. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Conditional Compile Values Value _POWER_MP Meaning Code is being compiled for a multiprocessor machine. This value is automatically defined by the compiler if the -q64 option is specified. . /* user addr .V2. /* sigctx location in user space*/ #else long t_ulock. 2001. /* owner process' ublock (const)*/ } t_uaddress. /* user addresses */ #ifndef __64BIT_KERNEL uint t_ulock64. /* high order 32-bits if 64bit mode */ char *t_stackp.lock or cv */ long t_uchan. /* local data */ struct user *userp. /* sigctx location in user space*/ #endif . /* key of user addr */ uint t_userdata64. 2003 Unit 1. /* user-owned data */ long t_cv.3 Student Notebook Uempty struct thread { /* identifier fields */ tid_t t_tid. /* unique thread identifier */ tid_t t_vtid. . /* high order 32-bits */ uint t_uchan. /* high order 32-bits if 64-bit mode */ int t_userdata. /* User condition variable */ char *t_stackp.0. /* high order 32-bits */ uint t_ulock. /* saved user stack pointer */ uint t_scp64. /* Virtual tid */ /* related data structures */ struct pvthread *t_pvthreadp. /* my pvthread struct */ struct proc *t_procp. /* key of user addr */ long t_userdata. . . /* user addr . /* User condition variable */ uint t_stackp64.lock or cv */ uint t_uchan64. Introduction to the AIX 5L Kernel 1-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. /* user-owned data */ uint t_cv64. /* high order 32-bits if 64-bit mode */ int t_cv. © Copyright IBM Corp. /* high order 32-bits if 64bit mode */ struct sigcontext *t_scp.0. /* owner process */ struct t_uaddress { struct uthread *uthreadp. /* saved user stack pointer */ struct sigcontext *t_scp. 2.Student Notebook Checkpoint 1. 4. 3. The 32-bit kernel supports 64-bit user applications when running on ________hardware. 2001. The processor runs interrupt routines in ______mode. Checkpoint BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Figure 1-12. The AIX kernel is _______.0 Notes: 1-24 Kernel Internals © Copyright IBM Corp. ________ and __________. and only runs on _______ hardware. 5. . The______ is the base program of the operating system. The 64-bit AIX kernel supports only _______kernel extensions. 0.V2. © Copyright IBM Corp.0 Notes: Turn to your lab workbook and complete exercise one. Introduction to the AIX 5L Kernel 1-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.3 Student Notebook Uempty Exercise Complete exercise one Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Use the cscope tool to examine system header files Figure 1-13.0. 2003 Unit 1. Exercise BE0070XS4. . 0 Notes: 1-26 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . 2001.Student Notebook Unit Summary Describe the role the kernel plays in an operating system Define user and kernel mode and list the operations that can only be performed in kernel mode Describe when the kernel must make a context switch Describe the role of the mstsave area in a context switch Name the execution environments available on each of the platforms supported by AIX 5L Using the system header files. identify data element types for each of the available kernels in AIX 5L Figure 1-14. Unit Summary BE0070XS4. Kernel Analysis Tools 2-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Unit 2.V2. you should be able to: • List the tools available for analyzing the AIX 5L kernel • Use KDB to display and modify memory locations and interpret a stack trace • Use basic kdb navigation to explore crash dump and live system How You Will Check Your Progress Accountability: • Exercises using your lab system References AIX Documentation: Kernel Extensions and Device Support Programming Concepts © Copyright IBM Corp.0. What You Should Be Able to Do After completing this unit.0. Kernel Analysis Tools What This Unit Is About This unit describes the different tools that are available to debug the AIX 5L kernel. 2003 Unit 2. . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.0 Notes: 2-2 Kernel Internals © Copyright IBM Corp. . Unit Objectives BE0070XS4.Student Notebook Unit Objectives At the end of this unit you should be able to: List the tools available for analyzing the AIX 5L kernel Use KDB to display and modify memory locations and interpret a stack trace Use basic kdb navigation to explore crash dump and live system Figure 2-1. V2. Kernel Analysis Tools 2-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.0 Notes: Kernel Analysis Tools Several tools are available in AIX 5L that are used to examine and debug the kernel. . and lowercase kdb is used when referring to the image analysis command. © Copyright IBM Corp. 2001.3 Student Notebook Uempty What tools will you be using in this class? Figure 2-2. Description Kernel debugger for live system debugging Used for system image analysis KDB kdb Tool Typographic conventions In this class an uppercase KDB will be used when referring to the kernel debugger. This table list the primary tools we will be covering in this unit. 2003 Unit 2. What tools will you be using in this class? BE0070XS4.0. Student Notebook The Major Functions of KDB are: Set breakpoints within the kernel or kernel extensions Execution control through various forms of step execution commands Format display of selected kernel data structures Display and modification of kernel data Display and modification of kernel instructions Modify the machine state through alteration of system registers Figure 2-3. .0 Notes: Introduction This section covers describes the kernel debugger available in AIX 5L. Overview The kernel debugger is built into the AIX 5L production kernel. Interfacing with the debugger Once started the kernel debugger is operated from a terminal connected to a native serial port of the system. The Major Functions of KDB are: BE0070XS4. For the debugger to be used it must be enabled prior to booting. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. or from a serial terminal connected via an 8-port or 128-port adapter. 2-4 Kernel Internals © Copyright IBM Corp. The debugger cannot be operated from the LFT graphics display. © Copyright IBM Corp. 2001.0.3 Student Notebook Uempty Concept When KDB is invoked. 2003 Unit 2. Though this requires that kernel code be duplicated within the debugger. it is the only running program until you exit the debugger. this means it is possible to set breakpoints anywhere within the kernel code.V2.0. all processes continue to run unless the debugger was entered via a system halt. In addition. All processes are stopped and interrupts are disabled. When exiting the kernel debugger. the kernel debugger does not run operating system routines. Kernel Analysis Tools 2-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The kernel debugger runs with its own Machine State Save Area (mst) and a special stack. . Boot the new image (shutdown -Fr) 4. 2-6 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Building a new boot image The bosboot command is used to build boot images. Enabling the Kernel Debugger BE0070XS4. After the boot image has been built the system must be re-booted for the new options to take effect.Student Notebook Enabling the Kernel Debugger Perform these steps to enable the kernel debugger: 1. Set Kernel boot Flags (bosdebug -D) 2. . After changing these flags you must create a new boot image and reboot the system to use this new image. Build a new boot image (bosboot -ad /dev/ipldevice) 3. Verify the debugger is enabled (Check dbg_avail) Figure 2-4. 2001.0 Notes: Kernel flags The kernel debugger feature is enabled by setting flags in the boot image prior to booting the kernel. Arguments supplied to the bosboot command will set flags in the boot image causing the kernel debugger to be enabled or disabled. The bosdebug command is used to view or set these attributes.0. Example The following command will build a new boot image with the kernel debugger loaded: # bosboot -a -D -d /dev/ipldevice The system must be rebooted for the change to take effect. 2003 Unit 2. Creates complete boot image. 2001. Kernel Analysis Tools 2-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. To view the setting of the debug flags in the ODM database use the command: # bosdebug Memory debugger Memory sizes Network memory sizes Kernel debugger Real Time Kernel off 0 0 on off To set the kernel debugger attribute on use the command: # bosdebug -D To set the kernel debugger attribute off use the command: # bosdebug -o Note: All this command does is to set attributes in the SWservAt ODM database. © Copyright IBM Corp. Loads and invokes the kernel debugger. . The current boot disk is represented by the device: /dev/ipldevice Loads the kernel debugger. The bosboot command reads these values and sets up the boot image accordingly. bosdebug Attributes in the SWservAt ODM database can be set so that bosboot will enable the kernel debugger regardless of the command line argument used when building the boot image.0. The kernel debugger will be invoked immediately on boot.3 Student Notebook Uempty bosboot syntax The syntax of the bosboot command is: bosboot -a [-D | -I] -d device Argument Description -d device -D -I -a Specifies the boot device.V2. The kernel debugger will not automatically be invoked when the system boots. Mask Description Do invoke at bootup.0 Notes: Verifying the kernel debugger is enabled Once the kernel is booted. 2-8 Kernel Internals © Copyright IBM Corp. Don't invoke at boot. Debugger is not ever to be called 3 0x00000000 0x00000001 0x00000002 Figure 2-5. .Student Notebook Verifying the Debugger is Enabled Step 1 2 Action Start the kdb command #kdb View the dbg_avail memory flag (0)> dw dbg_avail 1 dbg_avail + 000000: 00000002 Compare the value of dbg_avail against the mask value in this table. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. but debugger is still invokable. Verifying the Debugger is Enabled BE0070XS4. you can use the following procedure to verify that the kernel debugger has been enabled. 2001. load only When the kernel debugger is configured to be invoked (the -I option) the debugger will start immediately after booting. Starting the Debugger BE0070XS4.3 Student Notebook Uempty Starting the Debugger From a native serial port. type the key sequence: Ctrl-alt-Numpad4 A kernel extension or application makes a call to brkpoint() A breakpoint previously set using the debugger has been reached A fatal system error occurs Figure 2-6. © Copyright IBM Corp. 2003 Unit 2. If configured to be loaded but not invoked (the -D option) one of the conditions listed above must occur after the system is booted for the debugger to be started. Kernel Analysis Tools 2-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. type the key sequence: Ctrl-\ From the LFT keyboard. .0 Notes: Invoke vs.V2.0. 2001. The dump contains: . and the uthread and ublock structures of the running thread are pinned as well. 2-10 Kernel Internals © Copyright IBM Corp.0 Notes: What is in a system dump Typically. System Dumps BE0070XS4.Operating system (kernel) code and data . 2001. Normally this is not a problem since most of the kernel data structures are in memory. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Most of the kernel extensions code and data Paged memory The dump facility cannot page in memory. The process and thread tables are pinned.Some data from the current running application . . an AIX 5L dump includes all of the information needed to determine the nature of the problem. so only what is currently in physical memory can be dumped.Student Notebook System Dumps A dump image is not actually a full image of the system memory but a set of memory areas copied out by the dump routines. What is in a system dump? What is the effect of kernel paging? What is the role of the Master Dump Table? What tools are used to analyze system dumps? Figure 2-7. 2. On AIX 5. Process overview The following steps are used to write a dump to the dump device: Step 1. • After the first call to a Component Dump routine. the kernel processes the CDT that was returned For each CDT entry. Kernel extensions can specify a routine to be called to include data in a system dump. 3.1 this is done with the dmp_add() kernel service. Dump Creation Process Introduction This section describes the dump process. Kernel specific areas to be included in the dump are pre-loaded at kernel initialization. Interrupts are disabled 0c9 or 0c2 are written to the LED display. Analyzing dumps System dumps can be examined using the kdb command. the bitmap.0. if present Header information about the dump is written to the dump device The kernel steps through each entry in the Master Dump Table. . 2003 Unit 2.0. Kernel Analysis Tools 2-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. the kernel : • Checks every page in the identified data area to see if it is in memory or paged out • Builds a bitmap indicating each page's status • Writes a header. 4.V2. calling each Component Dump routine twice: • Once to indicate that the kernel is starting to dump this component (1 is passed as a parameter). AIX 5.2 uses the dmp_ctl() kernel service. • Again to say that the dump process is complete (2 is passed as a parameter).3 Student Notebook Uempty The master dump table The system dump function captures data areas by processing information returned by routines registered in the Master Dump Table. and those pages which are in memory to the dump device Action © Copyright IBM Corp. 2001. Student Notebook Step 5. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . the kernel enters an infinite loop. Action Once all dump routines have been called. displaying 0c0 or flashing 888 2-12 Kernel Internals © Copyright IBM Corp. 2001. 2001. It is imperative that the /unix file supplied is the one that was running at the time the memory image was created. Kernel Analysis Tools 2-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. BE0070XS4. .0.0. The /unix file provides the necessary symbol mapping needed to analyze the memory image file. 2003 Unit 2.3 Student Notebook Uempty kdb The kdb command allows examination of an operating system image Requires system image and /unix Can be run on a running system using /dev/mem Typical invocations: # kdb -m vmcore.0 Notes: kdb Command Files needed The kdb command requires both a memory image (dump device.X -u /usr/lib/boot/unix or # kdb Figure 2-8.V2. vmcore or /dev/mem) and a copy of /unix to operate. kdb .0) must not be compressed. The memory image (whether a device such as /dev/dumplv or a file such as vmcore. . # kdb 2-14 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. Add the kernel_modules listed View XCOFF object Print CDT entries Print help Disable in-line more.X -u /unix To run kdb against the live (running kernel) no parameters are required. In this case root permissions are required. Use the image file provided Use the kernel file.Student Notebook Parameters The kdb command may be used with the following parameters: Parameter Description no parameter -m system_image_file -u kernel_file -k kernel_modules -w -v -h -l Use /dev/mem as the system image file and /usr/lib/boot/unix as the kernel file. useful when running noninteractive session Example To run kdb against a vmcore file use the following command line: # kdb -m vmcore. This is required to analyze a system dump on a different system. A system dump image contains everything that was in the kernel at the time of the crash. 2003 Unit 2.0. 3. 2.0 Notes: © Copyright IBM Corp. The value of the _______kernel variable indicates how the debugger is loaded. . _____is used for live system debugging. 2001. True or False? Figure 2-9.V2. 4. Kernel Analysis Tools 2-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Checkpoint BE0070XS4.3 Student Notebook Uempty Checkpoint 1. _____is used for system image analysis.0. 0 Notes: Introduction Turn to your lab workbook and complete exercise two. They will provide you with information needed to do the exercise. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Read the information blocks included with the exercises. 2-16 Kernel Internals © Copyright IBM Corp.Student Notebook Exercise Complete exercise two Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Enable and start the kernel debugger Display and interpret stack traces Display and modify variables in kernel memory Perform basic kdb navigations on live system and crash dump Figure 2-10. Exercise BE0070XS4. 2001. 3 Student Notebook Uempty Unit Summary List the tools available for analyzing the AIX 5L kernel Use KDB to display and modify memory locations and interpret a stack trace Use basic kdb navigation to explore crash dump and live system Figure 2-11. . Kernel Analysis Tools 2-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: © Copyright IBM Corp.0. Unit Summary BE0070XS4.0. 2003 Unit 2. 2001.V2. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. .Student Notebook 2-18 Kernel Internals © Copyright IBM Corp. user and u_thread • Use the kernel debugging tools in AIX to locate and examine a process’ proc. Process Management 3-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. you should be able to: • List the three thread models available in AIX 5L • Identify the relationship between the six internal structures: pvproc. user and u_thread data structures • Identify the states of processes and threads on a live system and in a crash dump • Analyze a crash dump caused by a run-away process • Identify the features of AIX scheduling algorithms • Identify the primary features of the AIX scheduler supporting SMP and large system architectures • Identify the action the threads of a process will take when a signal is received by the process How You Will Check Your Progress Accountability: • Exercises using your lab system • Check-point activity • Unit review References AIX Documentation: Performance Management Guide AIX Documentation: System Management Guide: Operating System and Devices © Copyright IBM Corp. thread. thread. What You Should Be Able to Do After completing this unit.0. . 2001. pv_thread. proc.0. 2003 Unit 3. Process Management What This Unit Is About This unit describes how processes and threads are managed in AIX 5L.3 Student Notebook Uempty Unit 3. 0 Notes: 3-2 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. pv_thread. Unit Objectives BE0070XS4.Student Notebook Unit Objectives At the end of this unit you should be able to: List the three thread models available in AIX 5L Identify the relationship between the six internal structures: pvproc. 2001. thread. user and u_thread Use the kernel debugging tools in AIX to locate and examine a process’ proc. proc. user and u_thread data structures Identify the states of processes and threads on a live system and in a crash dump Analyze a crash dump caused by a run-away process Identify the features of AIX scheduling algorithms Identify the primary features of the AIX scheduler supporting SMP and large system architectures Identify the action the threads of a process will take when a signal is received by the process Figure 3-1. . thread. such as a user application. 2001.0.0.V2.A collection of resources . 2003 Unit 3.A set of one or more threads © Copyright IBM Corp. . Process Management 3-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Processes and threads A process is a self-contained entity that consists of the information required to run a single program.3 Student Notebook Uempty Parts of a Process Process y y y y Resources Address space Open files pointers User credentials Management data Thread Stack CPU registers Thread Stack CPU registers Thread Stack CPU registers Figure 3-2. Parts of a Process BE0070XS4. Process A process can be divided into two components: . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The resources are: . Each thread has a private execution context that includes: .Address space (program text.User credentials .A stack .Student Notebook Resources The resources making up a process are shared by all threads in the process. .Management data Threads A thread can be thought of as a path of execution through the instructions of the process.A set of open files pointers .CPU register values (loaded into the CPU when the thread is running) 3-4 Kernel Internals © Copyright IBM Corp. data and heap) . 2001. . Threads BE0070XS4.3 Student Notebook Uempty Threads Three type of threads are available in AIX: Kernel Kernel-managed User Three thread programming models are available for user threads: 1:1 M:1 M:N Figure 3-3.0.V2. Kernel threads Kernel threads are not associated with a user process and therefore have no user context. Process Management 3-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. © Copyright IBM Corp.0 Notes: Threads Threads provide the execution context to the process. 2003 Unit 3. 2001. Kernel threads run completely in kernel mode and have their own kernel stack. They are cheap to create and manage thus are typically used to perform a specific function like asynchronous I/O. The scheduling and running of kernel-managed threads is managed by the kernel. Each thread is scheduled to run on a CPU independent of the other threads of the process. 2001. They are managed by a user-level threads library and their scheduling and execution are managed at the user level. On SMP systems. . Programming models AIX 5L provides three models for mapping user threads on top of kernel-managed threads. The kernel has no knowledge of their existence. M:1 and M:N models. Each user process contains one or more kernel-managed threads. The application developer can chose between 1:1.Student Notebook Kernel-managed threads Kernel-managed threads are sometimes called ”Light Weight Processes” or LWPs and are the fundamental unit of execution in AIX. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3-6 Kernel Internals © Copyright IBM Corp. User threads User threads are an abstraction entirely at the user level. the threads of one process can run concurrently. . 2003 Unit 3.V2. each user thread is mapped to a single kernel-managed thread: © Copyright IBM Corp.0. Process Management 3-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.3 Student Notebook Uempty 1:1 Thread Model User Thread User Thread User Thread Thread Library Kernelmanaged Thread Kernelmanaged Thread Kernelmanaged thread Figure 3-4.0.0 Notes: 1:1 Model In the 1:1 model. 1:1 Thread Model BE0070XS4. . M:1 Thread Model BE0070XS4.0 Notes: M:1 In the M:1 model all user threads are mapped to one kernel-managed thread. The scheduling and management of the user threads are completely handled by the thread library. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.Student Notebook M:1 Thread Model User Thread User Thread User Thread Library Scheduler Thread Library Kernelmanaged Thread Figure 3-5. 3-8 Kernel Internals © Copyright IBM Corp. The following will select the 1:1 model: © Copyright IBM Corp.0. 2003 Unit 3. .3 Student Notebook Uempty M:N Thread Model User Thread User Thread User Thread User Thread Thread Library Library Scheduler Kernelmanaged Thread Kernelmanaged Thread Kernelmanaged Thread Figure 3-6. user threads are mapped to a pool of kernel-managed threads. Note that the thread model is selectable. The default for AIX 4. the term “thread” refers to a kernel-managed thread.3. the 1:1 model is discussed. Unless specified. M:N Thread Model BE0070XS4. A user thread may be bound to a specific kernel-managed thread.0. Primarily. Thread model for this unit This unit focuses on the management and scheduling of kernel-managed-threads.1 and higher is the M:N model. 2001. Using the 1:1 model can improve performance. Process Management 3-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. An additional “hidden” user scheduler thread may be started by the library to handle mapping user threads onto kernel managed threads.0 Notes: M:N In the M:N model.V2. See the Performance Management Guide in the AIX online documentation. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.Student Notebook #export AIXTHREAD_SCOPE=S #<your_program> There are many similar options available for thread tuning. 3-10 Kernel Internals © Copyright IBM Corp. . 2001.3 Student Notebook Uempty Creating Processes When a process is created it is given: A process table entry Process identifier (PID) An address space (its contents are copied from the parent process) User-area Program text Data User and kernel stacks A single kernel-managed thread (even if the parent process had many threads) Figure 3-7.0.0 Notes: Creating processes A new process is created when an existing process executes a fork() system call. .V2. the creating process is the child’s parent. 2003 Unit 3.0. Process Management 3-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The new process is called a child process. Exec When a process is first created it is running the same program as its parent. Creating Processes BE0070XS4. One of the exec() class of system calls is normally used to load a new program into the process’ address space. © Copyright IBM Corp. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. } exit(0).Student Notebook Example Here is an example of fork and exec to start a new program: main(){ pid_t child."-l". if ( (child=fork()) == -1){ perror("could not fork a child process"). } if ( child==0 ) { /* child */ /* exec a new program */ if (execl("/bin/ls".NULL) == -1 ){ perror("error on execl"). /* all done end the new process */ } else { /* parent */ wait(NULL). exit(1). 2001. /* Ensure parent terminates after child */ } } /* main */ 3-12 Kernel Internals © Copyright IBM Corp. exit(1). 2003 Unit 3. Process Management 3-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Thread library AIX provides a thread library to assist programers with the creation and management of threads.0. When created the thread is assigned: A thread table entry A thread identifier An execution context (stack pointer and CPU registers) Figure 3-8. A process can create additional threads using the thread_create() system call. Typically. The thread library allows for creation and management of both kernel-managed threads and user threads using the same interface. the library function pthread_create() is used to create threads rather than calling thread_create() directly.3 Student Notebook Uempty Creating Threads A new thread is created by the thread_create() system call. © Copyright IBM Corp. .V2. 2001.0 Notes: Creating threads When a process is first created it contains a single kernel-managed thread. Creating Threads BE0070XS4.0. /* start up a new thread */ if (pthread_create (&threadId. } /* main thread code here */ } void *new_thread(void *arg) { /* new thread code here */ } 3-14 Kernel Internals © Copyright IBM Corp. NULL )) { perror ("pthread_create"). exit (errno).Student Notebook pthread_create example Here is an example of the creating a new thread using pthread_create: #include <pthread. new_thread. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.h> #include <errno. int main () { int i. NULL.h> void *new_thread(void *arg). pthread_t threadId. 2001. . .Zombie © Copyright IBM Corp.Active . 2001.V2.Swapped . In AIX a process can be in one of five states: . 2003 Unit 3.0.3 Student Notebook Uempty Process State Transitions Process creation fork() Idle Swapped Active Stopped Zombie Non-existent Figure 3-9.Stopped .0 Notes: Process states This illustration above shows the states of a process during its life. Process State Transitions BE0070XS4. Process Management 3-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.Idle . A stopped process can be restarted by the SIGCONT signal. When a process receives a SIGSTOP signal. When a process terminates. all its threads are stopped and will not be scheduled on a CPU. This is the normal process state. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. During creation the process is in the idle state. Once the creation of the process is done it is placed in the active state. A swapped process has lost its memory resources and its address space has been moved onto disk. A process is placed in the zombie state until its parent cleans up after it frees the resources. it is placed in the stopped state. If a process is stopped. One example of this situation could be that a process has exited. some of its resources are not automatically released. Active Stopped Swapped Zombie Zombie process Sometimes a Zombie process will stay in the process list for a long time. 2001. The parent must execute a wait() system call to retrieve the process’ exit status before the process will be removed from the process table.Student Notebook States The five process states are described in this table: State Idle Description A process is started with a fork() system call. This state is temporary until all of the necessary resources have been allocated. the init process (PID 1) frees the remaining resources held by the child. The threads of the process can now be scheduled to run on a CPU. If the parent process no longer exists when a child process exits. 3-16 Kernel Internals © Copyright IBM Corp. but the parent process is busy or waiting in the kernel and unable to read the return code. It cannot run until swapped back into memory. 3 Student Notebook Uempty Process state on a running system The state of a process can be found on a running system using the ps command. Process Management 3-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. # ps -l F 240001 200001 200001 200011 S A A A T UID PID 201 0 0 0 17670 19172 19392 19928 PPID 16390 17670 19172 19172 C 0 0 3 0 PRI NI ADDR SZ 496 496 308 436 WCHAN TTY pts/3 pts/3 pts/3 pts/3 TIME CMD 0:00 ksh 0:00 ksh 0:00 ps 0:00 vi 60 20 61f4 60 20 59da 61 20 2605 60 20 4dff S Flag O I A T W Z Nonexistent Idle Active Stopped Swapped Zombie State Process state in a crash dump The state of a process can also be found in a crash dump using kdb: # kdb (0)> proc * SLOTNAME pvproc+000000 0 pvproc+000200 1 pvproc+000400 2 pvproc+000600 3 STATE PID PPID PGRP UID ADSPACE swapperACTIVE 00000 00000 00000 00000 00004812 init wait netm ACTIVE 00001 00000 00000 00000 0000342D ACTIVE 00204 00000 00000 00000 00004C13 ACTIVE 00306 00000 00000 00000 0000282A © Copyright IBM Corp. 2003 Unit 3.0.0.V2. 2001. . A process identifier . NPROC pvproc proc Figure 3-10. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. pvproc proc pv_procp . .A description of the process’ address space . . Each entry contains: .Student Notebook The Process Table Process Table Slot Number 0 1 2 3 . . Each process is represented by one entry in the table.Other process management data 3-18 Kernel Internals © Copyright IBM Corp.The process state .0 Notes: The process table The kernel maintains a table entry for each process on the system. This table is called the process table. . 2001. .A list of threads . . pvproc pvproc pv_procp proc pv_procp . The Process Table BE0070XS4. proc structure The proc structure is an extension on the pvproc structure. 2003 Unit 3.V2. The pv_procp in the pvproc points to its associated proc structure. © Copyright IBM Corp. Slot number Each entry in the process table is referred to by its slot number.3 Student Notebook Uempty Process table The process table is a fixed-length array of pvproc structures allocated from kernel memory. Table Management). The proc and pvproc structures are split to accommodate large system architectures. .0. this table is divided into a number of sections called zones. For the 64-bit kernel. 2001. At system startup.0. Process Management 3-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. one zone is allocated on each SRAD (see later topic. 3-20 Kernel Internals © Copyright IBM Corp.Student Notebook pvproc Element pv_pid pv_ppid pv_uid pv_stat pv_flags *pv_procp *pv_threadlist *pv_child *pv_siblings Description Unique process identifier (PID) Parents process identifier (PPID) User identifier Process state Process flags Pointer to the proc entry Head of list of threads Head of list of children NULL termintated sibling list Figure 3-11. Some of the key elements are shown above.h. 2001. .0 Notes: pvproc structure The definition of the pvproc structure can be found in /usr/include/sys/proc. pvproc BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Values for pv_stat are defined in /usr/include/sys/proc. The size of the table is defined as NPROC in the file /usr/include/sys/proc. .0 Notes: pv_stat The process state is stored in the pvproc->pv_stat data element. pv_stat . © Copyright IBM Corp. Process Management 3-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.h.h as shown in this table.3 Student Notebook Uempty pv_stat Values SNONE SIDL SACTIVE SSWAP SSTOP SZOMB Meaning Slot is not being used Process is being created Process has at least one active thread Process is swapped out Process is stopped Process is zombie Figure 3-12.0.V2. BE0070XS4. Process table size The size of the process table determines how many processes the system can have.0. 2003 Unit 3. 2001. At system startup. . . . . .e. Zone 0 Slot 0 Pinned pages High water mark Slot 8192 Zone 32 Figure 3-13. . . therefore. Table Management BE0070XS4. . one zone is allocated on each SRAD in the system. . . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . defined in the header file <sys/pmzone. . . When a zone on an SRAD fills up (i.Student Notebook Table Management Process Table Zone 0 Zone 1 . In reality the entire table is rarely needed. is version dependent. Zones The process table used in the 64-bit kernel is split into equal sized sections called zones. . Each zone contains a fixed number of process slots. 2001. The number of zones. only a portion of the table is pinned into memory at one time. all of the process slots in that zone are used) then another zone is 3-22 Kernel Internals © Copyright IBM Corp. . and number of process slots per zone. The details can be determined by examining the value of PM_NUMSRAD_ZONES.0 Notes: Table management If the entire process table were pinned in memory it would consume a significant amount. . . . . .h>. . Pinning pages of the processes table Each zone of the process table contains a high water mark indicating the highest number of slots in the zone that have been in use. Both are defined in /usr/include/sys/pmzone. A single high water mark is used and pages are pinned as explained above. The memory pages containing the slots up to the high water mark are pinned in memory. one for each zone in the table. 32-bit kernel The process table on 32-bit kernels has only one zone encompassing the entire process table.0.3 Student Notebook Uempty allocated to the SRAD and added to the pool. © Copyright IBM Corp. Details Two structures are used to manage the process table.h. The table is defined by a struct pm_heap_global. The high water mark for the zone is found in the pm_heap. . Process Management 3-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. Large systems On some systems (64-bit kernel only) a zone would typically be associated with a single RAD (a group of resources connected together by some physical proximity). 2001. there is only one SRAD per system. As the table grows the high water mark is moved and additional pages of the table are pinned. At the moment.0. This structure has pointers to several pm_heap structures. 2003 Unit 3. . Large systems In some systems physical memory is divided into pools that have a degree of physical proximity to particular processors. History In older versions of AIX. each process is represented by two structures. Extending the pvproc . In AIX 5L.Student Notebook Extending the pvproc SRAD proc proc pvproc table zone SRAD proc proc pvproc table zone CPU CPU CPU CPU CPU CPU CPU CPU SRAD proc proc pvproc table zone CPU CPU CPU CPU Figure 3-14. The AIX 3-24 Kernel Internals © Copyright IBM Corp. Access speed to memory hosted from another processor may be slower than accessing memory hosted from the local processor. 2001. the proc and a smaller pvproc. the process table was made from an array of proc structures. BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: proc structure The proc structure is an extension to the pvproc structure. Using one large proc structure table could result in many "remote" accesses. © Copyright IBM Corp.0. 2003 Unit 3.0. An SRAD (scheduler RAD) is a RAD large enough to warrant a dedicated scheduler thread. Process Management 3-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. a collection of resources grouped by some degree of physical proximity.3 Student Notebook Uempty 5L design allows the use of RADs (Resource Affinity Domains).V2. which allows each zone to reside on its own SRAD. and refer to proc structures for processes running on that SRAD. . The table of pvproc structures is separated into zones. 3-26 Kernel Internals © Copyright IBM Corp. It is composed of the process table slot number and a generation count. The generation count is incremented each time the process table slot is used. 0 Generation count 0 Figure 3-15. This means a process table slot can be used 128 times before a process ID is reused. . PID Format BE0070XS4.Student Notebook PID Format 32-bit Kernel 31 26 25 8 7 1 0 000000 Process table slot index Generation count 0 64-bit Kernel 63 26 25 13 12 8 7 1 0 Low order bits of Process table slot index SRAD (upper bits of index) 00 .0 Notes: Process identifier The process identifier or PID is a unique number assigned to a process when the process is first created. . PID format The format of a PID is shown above. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . AIX 5. 2003 Unit 3. These bits are used to select the zone on the process table. Set to zero. A generation count used to prevent the rapid re-use of PIDs.0.1 uses 5 bits.2 currently uses 4 bits. The process table slot number. . Process Management 3-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. apart from init. and defined by PM_NUMSRAD_BITS defined in <sys/pmzone.0. which is a special case and always has process ID 1.3 Student Notebook Uempty Bits Bit 0 Generation count Process table slot index SRAD (Scheduler Resource Affinity Domain) Remaining bits Description Always set to zero making all PIDs even numbers. The number of bits used for the SRAD is version dependent. AIX 5. 2001.V2. © Copyright IBM Corp.h>. pid_t Process identifiers are stored internally using the pid_t typedef. the SRAD field is 4-bits long. . This makes calculating the slot number in your head a little difficult. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.2. Why are the fields swapped? The SRAD and index bits are shifted around so that indexing is partitioned by zones. Finding the Slot Number BE0070XS4.1. 3-28 Kernel Internals © Copyright IBM Corp. the SRAD field is 5-bits long. 2001. On AIX 5. the index bits do not line up on an even nibble boundary.Student Notebook Finding The Slot Number 000000 Process table SRAD index bits Generation 0 count SRAD Process table index bits pvproc table slot number Figure 3-16. so the calculation is a little easier. In a 64-bit kernel the slot number is a combination of the SRAD bits with the index bits as shown above.0 Notes: Finding the slot number In a 32-bit kernel the process table slot number can easily be found from a PID by shifting the PID 8 bits to the right. therefore. On AIX 5. Kernel processes are created by the kernel itself and execute independently of user thread action. as can user processes Are scheduled like user processes. Kernel Processes BE0070XS4.3 Student Notebook Uempty Kernel Processes Kernel processes: Are created by the kernel Have a private u-area and kernel stack Share text and data with the rest of the kernel Are not affected by signals Can not use shared library object code or other user-protection domain code Run in the Kernel Protection Domain Can have multiple threads. 2003 Unit 3.0. 2001.0 Notes: Kernel Processes Some processes in the system are kernel processes. Process Management 3-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . © Copyright IBM Corp. but tend to have higher priorities Figure 3-17.0.V2. 98334 114718 163968 172074 TTY TIME CMD .11:20 wait . 2001.0:02 swapper . .0:00 rtcmd . .0:00 j2pg .0:00 dog 3-30 Kernel Internals © Copyright IBM Corp.5681:27 wait . .Student Notebook Listing kernel processes You can list the current kernel processes with the ps -k command. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0:00 lvmbb . # ps -k PID 0 16388 24582 . Process Management 3-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Thread Table BE0070XS4. 2001. . Each kernel-managed thread is represented by one table entry which contains: . 2003 Unit 3.Thread management data The thread table is similar to the process table. pvthread pvthread tv_threadp thread tv_threadp . . pvthread thread tv_threadp . © Copyright IBM Corp. just as with the process table.V2.0. . . It is an array of pvthread structures allocated from kernel memory.3 Student Notebook Uempty Thread Table Thread Table Slot Number 1 2 3 .0. The thread table for 64-bit systems is divided into zones and the zones are allocated on different SRADs. Each entry in the table is referred to by its slot number. NTHREAD pvthread thread Figure 3-18. . .0 Notes: Thread Table The kernel maintains a thread table.A thread identifier (TID) .A thread state . 2001.Student Notebook thread structure The thread structure is an extension on the pvthread structure. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The thread and pvthread structures were split to accommodate large system architectures. The tv_threadp item in the pv_thread points to its associated thread structure. 3-32 Kernel Internals © Copyright IBM Corp. . pvthread Elements BE0070XS4. Process Management 3-33 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: pvthread and thread structures Definitions for the pvthread and thread structures can be found in /usr/include/sys/thread. 2001. © Copyright IBM Corp. and the number of zones are version dependent. The size of each zone.V2. The thread table is split into multiple zones. Each zone contains a high water mark representing the largest slot number used since system boot.0.0. All memory pages for the slots up to the high water mark are pinned.3 Student Notebook Uempty pvthread Elements Element tv_tid *tv_threadp *tv_pvprocp *tv_next thread *tv_prevthread tv_state Description Unique thread identifier (TID) Pointer to thread structure Pointer to pvproc for this thread Pointer to next thread (pvthread) in the process Pointer to previous thread (pvthread) in the process Thread state Figure 3-19. 2003 Unit 3. Table management The memory pages for the thread table are managed using the same mechanism that was described for the process table. Elements Some of the key element of the pvthread structure are shown above. .h. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. tid_t Thread identifiers are stored internally using the tid_t typedef. 2001. The format of a TID is similar to that of a PID except that all TIDs are odd numbers and PIDs are even numbers. 3-34 Kernel Internals © Copyright IBM Corp. TID Format BE0070XS4. .Student Notebook TID Format 32-bit Kernel 31 27 26 8 7 1 0 000000 Thread table slot index Generation count 1 64-bit Kernel 63 27 26 13 12 8 7 1 0 Low order bits of thread table slot index SRAD (upper bits of index) 00 . The format of a TID is shown above. 0 Generation count 1 Figure 3-20.0 Notes: Thread identifier Introduction The thread identifier or TID is a unique number assigned to a thread. . . .0 Notes: Introduction Each process (including a kernel process) contains a u-block area.process private memory segment y Definition . Process Management 3-35 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. therefore.V2. Access The u-block is part of the process private memory segment. and unpinned when the process is swapped out.h uthread y Thread private data y stack pointers y mstsave uthread uthread uthread uthread user y shared between all threads in the process user Figure 3-21. It need not be in memory when the process is swapped out. The u-block is made up of a user structure (one per process) and one or more uthreads (one per thread).0. It is pinned when the process is swapped into memory. 2003 Unit 3. u-block BE0070XS4. It maintains the process state information which is only required when the process is running./usr/include/sys/user. it need not be accessible when the process is not running. however.3 Student Notebook Uempty u-block y Location . it is only accessible when in kernel mode.0. 2001. Student Notebook Definitions The u-block is described in the file /usr/include/sys/user.h. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. When execution of the thread continues the stack pointers and registers are loaded from the mst-save area. When a thread is interrupted or a context switch occurs the stack pointers and CPU registers of the interrupted thread are stored in the mst-save area of the uthread. the uthread holds execution-specific items like the stack pointers and CPU registers. For example. uthread Each thread of a process has its own uthread structure. 2001. Information stored in the user structure is global and shared between all threads in the process. . Threads are responsible for storing execution context. 3-36 Kernel Internals © Copyright IBM Corp. user Each process has one user structure. the file descriptor table and the user credentials are kept in the user structure. therefore. Six Structures BE0070XS4.0. 2003 Unit 3. The © Copyright IBM Corp. Diagram The above diagram depicts the structures for a single process containing three kernel-managed threads. Process Management 3-37 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. proc and thread From the pvproc structure the first pvthread can be found by following the pv_threadlist pointer.3 Student Notebook Uempty Six Structures tv_pvprocp pv_threadlist tv_nextthread pvproc pvthread pvthread pvthread tv_threadp pv_procp t_pvthreadp t_procp proc thread thread thread t_uthreadp t_userp U_procp uthread uthread uthread u-block user Figure 3-22. This section describes how these six structures are tied together. 2001.0 Notes: Introduction This unit has discussed the AIX 5L data structures: pvproc. proc. pvthread.V2. thread. All the pvthread structures for the process are linked via a circular doubly-linked list (see pointers tv_nextthread and tv_prevthread). . uthread and user. u-block The u-block is divided into uthread sections. Data that is private to the thread-like stack pointers are kept in the uthread. 3-38 Kernel Internals © Copyright IBM Corp. 2001. This allows all threads in a process to share the same open files. the file descriptor table.Student Notebook pvproc is extended in to the proc structure via the pv_procp pointer. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Similarly. Pointers in the thread structure point to both of these sections. . the pvthread structures are extended into the thread structures via the tv_threadp. for example. Process-wide data is kept in the user area. one per thread and one process-wide user structure. © Copyright IBM Corp.0 Notes: Introduction The object of thread scheduling is to manage the CPU resources of the system.0. Process Management 3-39 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 3. 2001. sharing these resources between all the threads.3 Student Notebook Uempty Thread Scheduling Topics Thread states Thread priorities Run queues Software components of the kernel Scheduler Dispatcher Scheduling algorithms Support for SMP and large systems Figure 3-23.0. .V2. Thread Scheduling Topics BE0070XS4. Student Notebook Thread State Transitions Idle Ready to Run Sleeping Running Stopped by a signal Zombie Figure 3-24. Thread State Transitions BE0070XS4. A thread typically changes its state between running. 2001. 3-40 Kernel Internals © Copyright IBM Corp. . State transitions Threads can be in one of several states. The diagram above shows all the state transitions a thread can make. ready to run.0 Notes: Introduction In AIX. The thread state shows if a thread is currently running or is inactive. sleeping and stopped several times during its lifetime. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. the kernel allows many threads to run at the same time. but there can be only one thread actually executing on each CPU at one time. Process Management 3-41 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Once the new thread creation is completed. the thread then goes to the zombie state. The defined values for this flag are: Flag TSNONE TSIDL TSRUN TSSLEEP TSSWAP TSSTOP TSZOMB slot is available being created (idle) runable (or running) awaiting an event (sleeping) swapped stopped being deleted (zombie) Meaning © Copyright IBM Corp.V2. The zombie state is an intermediate state for the thread lasting only until all the resources owned by the thread are given up.0. Sleeping Stopped Swapped Zombie tv_state The thread state is kept in the tv_state flag of the pv_thread structure.0. the thread table is updated whenever the thread is swapped. . Though swapping takes place at the process level and all threads of a process are swapped at the same time. 2003 Unit 3. Whenever the thread is waiting for an event. A thread in the running state is the thread executing on a CPU.3 Student Notebook Uempty States All the thread states are described in this table: State Idle Ready to Run Running Description When first created a thread is placed in the idle state. Stopped threads can be restarted by the SIGCONT signal. The thread state will change between running and ready to run until the thread finishes execution. This state is temporary until all of the necessary resources for the the thread have been allocated. 2001. A stopped thread is a thread stopped by the SIGSTOP signal. The thread waits in this state until the thread is run. it is placed in the ready to run state. the thread is said to be sleeping. A thread that is actually running has a state of TSRUN. 2001.e. and a wait type of TWCPU. therefore a flag is not necessary. . The running state is implied when a thread is currently being run. and a wait type of TNOWAIT. 3-42 Kernel Internals © Copyright IBM Corp. i. A thread must be ready to run before it can be run. The value of the tv_state flag for running threads will be shown as ready to run (TSRUN).Student Notebook Running threads No tv_state flag value has been defined for the running state. the thread is waiting for CPU access. A thread that is ready to run has a state of TSRUN. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 0.V2.3 Student Notebook Uempty Thread Priority 0 kernel PUSER = 40 Highest priority user 255 Priority values Figure 3-25. CPU time is made available to threads according to their priority number. Priorities above PUSER (example: numerically lower) are used for real-time threads. . Precedence is given to the thread with the lowest priority number. © Copyright IBM Corp.0. Thread priority Each thread is assigned a priority number between 0 and 255. 2003 Unit 3. The highest priority a thread can run in user mode is defined as PUSER or 40.0 Notes: Introduction All threads are assigned a priority value and a nice value. 2001. Process Management 3-43 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The dispatcher examines these values to determine what thread to run. Thread Priority Lowest priority BE0070XS4. Student Notebook Lower number means high priority Do not confuse a high priority value with a high priority thread. 2001. The two are inversely related. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3-44 Kernel Internals © Copyright IBM Corp. A process’ nice value is saved in the proc structure as p_nice=nice+PUSER. . a thread with a numerically low priority value is more important than one with a larger value. The nice value is used to adjust thread priority. nice Each process is assigned a nice value between 0 and 39. In other words. The default value for nice is 20. The nice value of a process can be set using the nice command or changed using the renice command. 100 . . Wait thread The wait thread is always ready to run. 40 . A run queue is arranged as a set of doubly-linked lists. 20 . it will run the wait thread. © Copyright IBM Corp. A single CPU system has one run queue. . . and has a priority value of 255.V2.0. 80 .0 Notes: Introduction All runnable threads on the system (except the currently running threads) are listed on a run queue. . . Process Management 3-45 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. a single run queue consists of 256 linked lists. . Since there are 256 different thread priorities. . It is the only thread on the system that will run at priority 255. numerically lowest) runnable thread. 60 .0. with one linked list for each thread priority. 2001. 2003 Unit 3.3 Student Notebook Uempty Run Queues Run Queue 0 . 255 wait thread thread thread thread thread thread Figure 3-26. AIX selects the next thread to run by searching the run queues for the highest priority (example. If AIX finds no other ready to run thread. Run Queues BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.e. 2001. 3-46 Kernel Internals © Copyright IBM Corp. Clock ticks A clock tick is 1/100 of a second.Student Notebook Dispatcher and Scheduler Functions Dispatcher Searches the run queues for the highest priority thread Dispatches the most-favored thread (highest priority) Invoked at various points in the kernel. the priority value will grow larger). including: By the clock interrupt (every 1/100th of a second) When the running thread gives up the CPU Scheduler Runs once a second Recalculates thread priority for all runnable threads based on: The amount of CPU time a thread has received The priority value The nice value Figure 3-27. . AIX is designed to handle many simultaneous threads. (i. Dispatcher and Scheduler Functions BE0070XS4.0 Notes: Introduction The scheduling and running of threads are the jobs of the dispatcher and scheduler. Generally. a thread that has accumulated many clock ticks will have its priority decreased. The number of clock ticks a thread has accumulated will be used to calculate a new priority for the thread by the scheduler. . then increment the t_cpu element of the currently running thread.3 Student Notebook Uempty Dispatcher Step Action 1 If invoked because a clock tick has passed.A time interval has passed (1/100 sec).A thread has voluntarily given up the CPU. place the currently running thread back on the run queue. 2003 Unit 3.A thread has been made runnable by an interrupt and the processor is about to finish interrupt processing and return to INTBASE. . 2001. 3 If the selected thread is different from the currently running thread.0 Notes: Dispatcher The dispatcher runs under the following circumstances: .0. . Process Management 3-47 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.A thread (from a non-threaded process) that has been boosted is returning to user mode from kernel mode. 2 Scan the run queue(s) looking for the highest priority read-to-run thread. The steps the dispatcher takes are listed above. 4 Resume execution of the thread at the end of the MST chain. . and place the selected thread at the end of the MST chain. © Copyright IBM Corp. if (thread->t_cpu < T_CPU_MAX) thread->t_cpu++. Dispatcher BE0070XS4. t_cpu is limited to a maximum value of T_CPU_MAX.V2. Figure 3-28.0. . r impacts how severely a process is penalized by used CPU time.60 ) + 60 2 Calculate the new priority using the equation: r · § new_nice + 4 · ----------------------------------priority = new_nice + t_cpu × § ¹ © 32¹ × © 64 Degrade the value of t_cpu so that ticks the thread has used in the past have less affect as recent ticks.d is 16. Scheduler BE0070XS4. 3-48 Kernel Internals © Copyright IBM Corp. The priority of a sleeping thread will not be changed.0 Notes: Scheduler The scheduler runs every second. r and d The values of r and d can be set using the schedo command. The r and d values control how a process is impacted by the run time. double its value. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.d <= 32 The default value for r. Recall that the value of p_nice is: nice+PUSER. Given: PUSER=40 and 0<=nice<=40. while d controls how fast the system “forgives” previous CPU consumption. 0 <= r. 3 d ---t_cpu = t_cpu × 32 Figure 3-29. The steps the scheduler uses to calculate thread priorities are shown in the table above. if ( p_nice > 60 ) new_nice = 2 × ( p_nice .Student Notebook Scheduler Step 1 Action If the value of nice is greater than the default value of 20. Its job is to recalculate the priority of all runnable threads on the system. making it possible to more strongly discriminate against upwardly nice'd threads. 0 Notes: Preemption Definition When the dispatcher runs and finds a runnable thread with a higher priority than the current running thread the running context is switched to the higher priority thread.V2.3 Student Notebook Uempty Preemption What is preemption? Non-preemptive kernel vs. Non-preemptive kernel Most UNIX systems will not allow pre-emption to occur when running in kernel mode.0. The thread that was displaced before it’s time slice expired is said to have been preempted.0. 2003 Unit 3. This can result in long delays in processing high-priority or real-time threads. Process Management 3-49 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. If the current running thread is in kernel mode and a higher priority thread becomes ready to run. . preemptive kernel Preventing deadlock in preemptive kernels Priority boost Figure 3-30. © Copyright IBM Corp. Preemption BE0070XS4. it will not be granted CPU time until the running thread returns to user mode and voluntarily gives up the CPU. This feature supports real-time processing where a real-time thread must respond to an action in a known time-frame. 2001. .Student Notebook Preemption in kernel mode AIX allows thread pre-emption in kernel mode. 3-50 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2. Process Management 3-51 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Thread B. When the dispatcher runs. Preemptive Kernels BE0070XS4. © Copyright IBM Corp.3 Student Notebook Uempty Preemptive Kernels Thread A Low priority Holding lock Thread B High priority Waiting for lock Thread C Medium priority Running Figure 3-31. In this example threads A. 2001. 2003 Unit 3. is waiting to obtain the same resource lock. thread C pre-empts thread A. Thread A. 3. has obtained access to an exclusive resource lock.B and C are all running in kernel mode. Step Action 1.0. Thread C’s priority is higher than thread A’s and is ready to run. running at a higher priority. a low priority thread. This thread cannot continue until thread A releases the lock.0 Notes: Problems with preemptive kernels The above scenario demonstrates the problem that AIX has solved to make kernel preemption work. . 2.0. it changes the priority of the thread that is holding the lock to its own priority. — The boosted thread returns to user mode from kernel mode. The priority is set back to the original value when either: — The scheduler notices that the boosted thread is no longer holding any locks. . Priority boost applies to both kernel locks and user (pthreads library) locks.When a high priority thread has to wait for a lock.The priority boost only applies to the “low priority” thread when it is holding the lock. Priority boost increases the priority of threads holding locks. 3-52 Kernel Internals © Copyright IBM Corp. Thread A is not running so it can’t release the lock. 2001. A thread running in kernel mode must release any kernel locks it holds before returning to user mode. . Even though thread B is the highest priority thread on the system it can’t proceed until it obtains the resource held by thread A. priority boost was added to AIX. — The high priority thread that was waiting for the lock obtains the lock. Priority boost To resolve this situation. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Step Action 4. Thread A is still holding the resource lock. Process Management 3-53 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. .V2.0.0 Notes: Introduction AIX has 3 main types of scheduling algorithms that will affect how a threads priority is calculated by the scheduler. 2001.0. Scheduling Algorithms BE0070XS4.3 Student Notebook Uempty Scheduling Algorithms SCHED_RR Fixed priority Threads are timesliced SCHED_FIFO Fixed priority Threads ignore timeslicing SCHED_OTHER Default policy Priority based on CPU time and nice value Figure 3-32. The main algorithms as defined in <sys/sched. 2003 Unit 3.h> are listed in the visual above. SCHED_FIFO2.A thread using SCHED_FIFO must have root authority to use it.Student Notebook SCHED_RR This is a round robin scheduling mechanism in which the thread is time-sliced at a fixed priority. . The amount of CPU time and the nice value have no affect on the threads priority. The FIFO policies differ in how they return threads to the run queue. real-time process.It is possible to create a thread with SCHED_FIFO that has a high enough priority that it could monopolize the processor if it is always runnable.It is possible to create a thread with SCHED_RR that has a high enough priority that it could monopolize the processor if it is always runnable and there are no other runnable threads with the same (or higher) priority. Thread priority is constantly being adjusted based on the value of nice and the amount of CPU time a thread has received.The thread runs at fixed priority and is not time-sliced. . 3-54 Kernel Internals © Copyright IBM Corp. however: . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . priority. .int policy.The thread must have root authority to be able to use this scheduling mechanism. 2001. SCHED_FIFO3 and SCHED_FIFO4. SCHED_OTHER This is the default AIX scheduling policy that was discussed earlier.This scheme is similar to creating a fixed-priority. See the Performance Management Guide of the AIX online documentation for more details. Choosing scheduling algorithms By default a thread will run with the SCHED_OTHER scheduling algorithm. and thereby provide a way of differentiating between their effective priorities. Priority degrades with CPU usage. SCHED_FIFO Similar to SCHED_RR. or until a higher priority thread is made runnable.It will be allowed to run on a processor until it voluntarily relinquishes by blocking or yielding. int thread_setsched (tid. policy) tid_t tid. . There are actually three other related policies. . Threads running as the root user can change scheduling algorithms using the thread_setsched() subroutine. . int priority. 2003 Unit 3.V2. © Copyright IBM Corp. . 2001.0. Process Management 3-55 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.3 Student Notebook Uempty t_policy The scheduling policy a thread is using is stored in: thread->t_policy. .Multiple Run Queues BE0070XS4. .Student Notebook SMP . . . . Memory cache Each CPU in a symmetric multi-processing system has its own memory cache. CPU 0 .0 Notes: Introduction On Symmetric Multi-Processing systems (SMP) per-CPU run queues are used to compensate for the multiple memory caches used on these systems. . 2001. . . CPU 2 . CPU 1 . . The best performance is achieved when a thread runs on a 3-56 Kernel Internals © Copyright IBM Corp. . . threads can be scheduled onto any CPU.Multiple Run Queues Globle Run Queue . . The purpose of the cache is to speed up processing by pre-loading blocks of physical memory into the higher speed cache. Cache warmth A thread is said to have gained cache warmth to a CPU when a portion of the process memory had been loaded into the CPU’s cache. Figure 3-33. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. SMP . . In an SMP system. This is called hard affinity. Process Management 3-57 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The AIX thread scheduler takes advantage of cache warmth by attempting to schedule a thread on the same CPU it ran on last.V2. As long as the thread is in the same CPU run queue it will run on the same processor. The bindprocessor() subroutine is used to give a single thread or all threads of a process hard affinity to a CPU. Soft cache affinity By having a run queue for each processor. If t_cpuid is set to PROCESSOR_CLASS_ANY=-1 the thread is not using hard affinity (note that t_cpuid=0 means bound to cpu 0). RT_GRQ If a thread has exported the environment variable RT_GRQ=ON. we allow for some measure of soft cache affinity.3 Student Notebook Uempty CPU where it has gained some cache warmth. Hard affinity (or binding) is recorded in thread->t_cpuid. Multiple run queues In addition to a global run queue each CPU has given its own run queue. Each CPU draws work from its own run queue. it will sacrifice soft cache affinity. 2003 Unit 3. © Copyright IBM Corp. The thread will be placed only in the global run queue and hence run on the first available CPU. .0. Hard affinity Threads can be bound to a single CPU meaning they are never placed in the global run queue. 2001. Load balancing The system uses load balancing techniques to ensure that work is distributed evenly between all of the CPU’s in the system. selecting the highest priority work from its queue.0. In a NUMA architecture. NUMA stands for Non-Uniform Memory Access. This means that any CPU can access any piece of memory with virtually the same cost in terms of latency and bandwidth. and the amount of memory. A good example of this is the NUMA-Q systems developed by Sequent. Local memory. The SMP architecture has a limit on the size that it can grow to. One approach that has been taken in the past to allow the development of large systems is to use building blocks of SMP systems. however this is still a point at which adding more CPUs. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and Remote Memory. both in terms of the number of CPUs. there are relatively large differences in access latency (approximately 1 order of magnitude) and bandwidth between local and remote memory. The limits grow over time as individual technologies improve (such as processor speed and memory bandwidth).Student Notebook NUMA Node CPU CPU Node local memory CPU I/O CPU remote cache Memory Interconnect remote cache CPU remote cache CPU CPU Node local memory CPU Node local memory CPU CPU I/O CPU CPU I/O Node Figure 3-34. the S stands for symmetric.0 Notes: In a true SMP architecture. which is located on a different system building block. . The memory in a NUMA system is effectively divided into two classes. 3-58 Kernel Internals © Copyright IBM Corp. or adding more memory actually degrades performance. 2001. which is on the same system building block as the CPU trying to access it. NUMA Node BE0070XS4. and couple them together into a single system. 2003 Unit 3. Process Management 3-59 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .3 Student Notebook Uempty Local vs.V2. Accessing memory on a different node is defined as remote access. 2001. © Copyright IBM Corp. remote and local accesses are identical with the exception of speed.0. remote memory access Access to memory on the same node as the device requesting the access is defined as local access.0. To the device (CPU or I/O) accessing the memory. remote memory access may be slower. since these resources have a degree of physical proximity when compared to other parts of memory or other processors. we could consider this architecture to be a single system (since all of the components are inside a single cabinet). This system is an SMP system that has some characteristics of a NUMA system. Looking at this diagram. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3-60 Kernel Internals © Copyright IBM Corp.Student Notebook Memory Affinity GX Slot L3 L3 GX Mem Slot GX Slot Mem Slot Mem Slot L3 L3 GX P L2 P P L2 P L3 L3 GX L3 L3 GX P L2 P Mem Slot GX Slot L3 L3 GX P L2 P L3 L3 GX MCM 2 P P P L2 P P L2 P L2 MCM 3 P L2 P L3 L3 GX L3 L3 GX L3 L3 GX P L2 P P L2 P P L2 P P L2 P L3 L3 GX L3 L3 GX MCM 1 P L2 P P L2 P P L2 P MCM 0 P L2 P L3 L3 GX GX GX GX GX L3 L3 L3 L3 L3 L3 L3 L3 Mem Slot Mem Slot GX Slot Mem Slot Mem Slot Figure 3-35. However if we examine the diagram more closely. we can see that each MCM has two attached memory cards. Memory Affinity BE0070XS4. We could consider an MCM and its two memory cards to be a RAD.0 Notes: The visual above shows the system architecture of the pSeries 690. 2001. Some memory is 'local' to a processor. . and other parts of memory are 'remote'. The major difference between this architecture and a true NUMA one is that the latency and bandwidth differences between local and remote access are much smaller. © Copyright IBM Corp. 2001. usually a physical node.3 Student Notebook Uempty Definitions This section defines some additional terms: . . 2003 Unit 3. The SDL determines how small the RAD will be. or atomic. .SDL . The top level is the entire system.System Decomposition Level . level consists of individual CPUs and memory.SRAD .Scheduler RAD is the RAD that the scheduler will operate on.RAD . .0.A RAD exists at multiple levels. Process Management 3-61 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. is a group of resources connected together by some physical proximity. the bottom.Resource Affinity Domain.0.V2. . . Process placement For most applications the most frequent memory access is to the process’ text. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . 2001. . stack and some kernel data structures. . . . .The goal of the thread scheduler is to balance the process load between all the CPUs in the system and reduce the amount of time a runnable thread waits to be run when other CPUs are idle. . . .Student Notebook Global Run Queues Global Run Queue . . . SRAD SRAD . . . . . CPU 6 . . . . .0 Notes: Introduction This section talks about design enhancements to facilitate future systems. . . To 3-62 Kernel Internals © Copyright IBM Corp. . Run queues The design of the AIX 5L thread scheduler has been extended to allow per-node run queues and one global run queue. CPU 3 . . . CPU 7 CPU 0 CPU 1 CPU 4 CPU 5 Figure 3-36. Other frequent accesses include private data. . CPU 2 . . . . . . Global Run Queues BE0070XS4. . . . . . . . . An API is provided for the control of logical attachments. AIX will occasionally migrate a process between SRADs. RAD affinity scheduling The purpose of RAD affinity scheduling is to exploit RAD local memory and RAD level caches by allocating a process private memory and text on the RAD(s) where it will be executed. Process migration In order to keep the system efficient. Physical attachment Processes can be attached to a physical collection of resources (CPU and memory) called an RSet. data. by attempting to execute process threads on CPUs where there is cache warmth. This RAD or set of RAD’s is called the process home RAD. Logical attachment Processes that share resources may be logically attached. © Copyright IBM Corp. stack and kernel data structures are allocated from memory on the RAD containing the CPUs that will execute the threads belonging to that process. . Processes attached to an RSet can only migrate between members of the RSet.V2. 2001.0. Process Management 3-63 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Logically attached processes are required to run on the same RAD. its memory must be copied to the process’ new home RAD.0.3 Student Notebook Uempty minimize memory access time the process text. 2003 Unit 3. For a process to migrate. and conversely. All process IDs (except pid 1) are _____. . 5. The process table is an _____ of _______ structures. Checkpoint BE0070XS4. AIX provides _____ programming models for user threads. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 4.Student Notebook Checkpoint 1. True or False? 6. A thread holding a lock may have its priority _______. 2.0 Notes: 3-64 Kernel Internals © Copyright IBM Corp. A new thread is created by the __________system call. A thread table slot number is included in a thread ID. 2001. 3. Figure 3-37. © Copyright IBM Corp.0.V2.0 Notes: Introduction Turn to your lab workbook and complete exercise three. Process Management 3-65 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. They provide you with information you need to do the exercise. . Exercise BE0070XS4.3 Student Notebook Uempty Exercise Complete exercise 3 Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Examine the process and thread structures using kdb Apply what you learned to the analysis of a crash dump Learn about and configure system hang detection Explore how signal information is stored and used in AIX Figure 3-38. 2003 Unit 3. Read the information blocks contained within the exercise.0. 2001. user. pv_thread. Figure 3-39. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. M:1. SCHED_FIFO. 2001. SCHED_OTHER The six structures of a process are: pvproc. u_thread. Unit Summary BE0070XS4. proc. .0 Notes: 3-66 Kernel Internals © Copyright IBM Corp. M:N The dispatcher Selects what thread to run The scheduler adjusts thread priority based on: nice CPU time Scheduling algorithms are SCHED_RR. thread. AIX has three thread programing models available: 1:1.Student Notebook Unit Summary The primary unit of execution in AIX is the thread. threads can mask signals. Processes can handle or ignore signals. V2.ibm. What You Should Be Able to Do After completing this unit. you should be able to: • List the types of addressing spaces used by AIX 5L • List the attributes associated with each segment type • Given the effective address of a memory object. 2001.nsf/productfamilies/PowerPC © Copyright IBM Corp.0. Addressing Memory What This Unit Is About This unit describes how memory is organized and addressed in AIX 5L.3 Student Notebook Uempty Unit 4. .com/chips/techlib/techlib.0. Addressing Memory 4-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. identify the segment number and object type How You Will Check Your Progress Accountability: • Exercises using your lab system • Unit review References PowerPC Microprocessor Family: The Programmers Reference Guide Available from http://www-3. 2003 Unit 4. . Unit Objectives BE0070XS4.0 Notes: 4-2 Kernel Internals © Copyright IBM Corp. List the attributes associated with each segment type. Given the effective address of a memory object.Student Notebook Unit Objectives At the end of this lesson you should be able to: List the types of addressing spaces used by AIX 5L. Figure 4-1. 2001. identify the segment number and object type. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. . 2003 Unit 4.0 Notes: Memory Management Definitions Introduction To explore how AIX 5L addresses memory we must first define the terms and concepts listed above.V2. © Copyright IBM Corp. Addressing Memory 4-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Memory Management Definitions BE0070XS4.0.0.3 Student Notebook Uempty Memory Management Definitions Page Frame Address Space Effective address space Virtual address space Physical address space Figure 4-2. 0 Notes: Introduction AIX manages memory in 4096-byte chunks called pages. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Pages stay separate from each other. AIX 5L uses a fixed page size of 4096 bytes. Pages and Frames BE0070XS4.Student Notebook Pages and Frames Real (physical) Memory Page size = 4096 bytes Page frame Figure 4-3. 4-4 Kernel Internals © Copyright IBM Corp. Pages are organized and stored in real (physical) memory chunks called frames. . 2001. Page A page is a fixed-sized chunk of contiguous storage that is treated as the basic entity transferred between memory and disk. The smallest unit of memory managed by hardware and software is one page. they do not overlap in virtual address space. AIX 5L can be configured to allow a number of large page segments. Frame The place in real memory used to hold the page is called the frame. a frame is the place in memory to hold that information. © Copyright IBM Corp. Addressing Memory 4-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.0. Whereas a page is a collection of information.V2. 2001.3 Student Notebook Uempty Large page support POWER4 processors can handle 16MB pages. 2003 Unit 4. See the AIX online documentation for more information. . The effective address space is the range of addresses defined by the instruction set.Effective address space . AIX 5L defines several different address spaces: .0 Notes: Introduction An address space is memory (real or virtual) defined by a range of addresses. 2001. The effective address space is mapped to physical address space or to disk files for each process. However.Virtual address space . Address Space BE0070XS4. programs and processes ‘see’ one contiguous address space.Physical address space Effective address space Effective addresses are those referenced by the machine instructions of a program or kernel. 4-6 Kernel Internals © Copyright IBM Corp. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Address Space Physical Memory Virtual address space Process 1 Effective address Process 2 Filesystem pages Paging space Figure 4-4. 0. the paging space is mainly used to hold the pages from working storage (process data pages). In AIX. Physical address space The physical address space is dependent on how much physical memory (DRAM) is on the machine. however depending on how much memory is installed. rather than a single 0-8GB address range. . Physical addresses in the range 3GB-4GB would be used to access devices connected to PCI host bus controllers. it may not be referenced in a single contiguous range. For example. Paging space The paging space is the disk area used by the memory manager to hold inactive memory pages with no other home. 2001. The physical address space is mapped to the machine’s hardware memory.V2. There is more information about this subject in a later unit covering the implementation of LPAR. and the number of PCI host bus controllers in the machine. The virtual address space is bigger (since it is addressed using more address bits) than the effective address. © Copyright IBM Corp. If a memory page is not in physical memory. it may be loaded from disk. Processes have access to a limited range of virtual addresses given to them by the kernel.3 Student Notebook Uempty Virtual address space The virtual address space is the set of all memory objects that could be made addressable by the hardware. Addressing Memory 4-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Writing a modified page to disk is called a page-out. 2003 Unit 4.0. a system with 8GB of memory installed may use the ranges 0-3GB and 4GB-9GB to reference physical memory. this is called a page-in. 4-8 Kernel Internals © Copyright IBM Corp. Translating Addresses BE0070XS4.0 Notes: Introduction When a program accesses an effective address. If the page is currently located on disk a free frame is found in physical memory and the page is loaded into this frame. 2001. The memory operation requested by the process or kernel is completed on the physical memory. 2 3 4 5 Figure 4-5. . the hardware translates the address into a physical address using the above process. The hardware translates the address into a system wide virtual address. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Translating Addresses Step 1 Action The effective address is referenced by a process or by the kernel. The page containing the virtual address is located in physical memory or on disk. Segments BE0070XS4.0. Available memory A process can control how much of its effective address space is available in two ways. © Copyright IBM Corp. A process can adjust the number of pages in a single segment (up to 256 MB). . A process can create or destroy segments in its address space.V2. Segments The maximum number of segments available to a process depends on the effective address space size (32-bit or 64-bit). 2003 Unit 4. . 2001.0 Notes: Introduction Effective memory address space in AIX 5L is divided into 256 MB objects called segments. Segment number n Figure 4-6. .0. Addressing Memory 4-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Segments 256 MB Segment Segment number 0 Effective address space Segment number 1 . A segment can be mapped into more that one process’s effective address space allowing the same physical memory to be shared. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Once a shared segment is defined it can be attached or detached by many processes. 4-10 Kernel Internals © Copyright IBM Corp. .Student Notebook Sharing address space The benefit of the segmented addressing model is the high degree of memory sharing that can occur between processes. 2001. Segment Addressing BE0070XS4. Addressing Memory 4-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. In the 64-bit model. 2001. 36 bits are used for the ESID. In the 32-bit model. In this case the value identifies an entry in the STAB table. allowing for 16 segments. Each segment has a Segment number or Effective Segment ID (ESID).3 Student Notebook Uempty Segment Addressing An effective address is broken down into the following three components Segment # 4/36 bits Virtual Page Index Byte Offset 16 bits 12 bits The first 4 bits (32 bit address) or 36 bits (64 bit address) is called an ESID and selects the segment register or STAB table slot The next 16 bits select the page within the segment The next 12 bits select the offset within the page Figure 4-7.V2. . In both cases the main item in the register/table entry is called a Virtual Segment ID (VSID). In this case the ESID identifies one of 16 Segment Registers. which is pointed to by the ASR (Address Space Register).0 Notes: Introduction This section discusses how memory segments are addressed. © Copyright IBM Corp. The address resolution information that follows describes this process. Segment addressing Both the 64-bit and 32-bit effective address spaces are divided into 256 MB segments.0. The virtual page index and byte offset are used together with the VSID to resolve the effective address. this number is 4 bits long. 2003 Unit 4. allowing for 236 (more than 64 million) segments.0. 4-12 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. only the first 16 segments (ESID 0 . . When running a 32-bit application the 64-bit hardware will zero extend the 32-bit effective address. 2001.Student Notebook 32 bit process on 64 bit hardware Keeping a consistent segment size in both the 32-bit and 64-bit execution modes allows for a 32-bit environment that is compatible with 64-bit hardware. Therefore. This is consistent with the 32-bit hardware model which only has 16 segments.15) can be accessed by a 32-bit application. The segment register contains a 24 bit Virtual Segment (VSID). This value together with the remaining effective address information (segment page number and page offset) is used to resolve our effective address to a machine-usable address.0.3 Student Notebook Uempty 32-bit Hardware Address Resolution Segment # Virtual Page Index Page Offset 16 Segment Registers 24 Segment ID 16 12 Page Offset 52-bit Virtual Address Virtual Page Number 40 Translation Look-Aside Buffer (TLB) Hash Anchor Table (HAT) Hardware Page Frame Table (PFT) Software Page Frame Table Real Page Number 20 32 Real Address Figure 4-8. 32-bit Hardware Address Resolution BE0070XS4. and it is 24/52 bits long for 32/64 bit hardware.0 Notes: Introduction As already noted. We call this table value the Virtual Segment ID (VSID).0. Note that the virtual address space is larger than the effective or real address spaces (it is 52/80 bits wide on 32/64 bit hardware platforms. Addressing Memory 4-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. as well as the following visual illustrate this process. . 2001. the effective address segment number identifies a register or table value. respectively). © Copyright IBM Corp. each 32 bit effective address uses the first 4 bits to select a segment register. 32-bit Hardware Address Resolution On 32-bit hardware. 2003 Unit 4. This visual. Combine this with the 12 bit page offset. 2001. .Student Notebook These 24 bits are used with the 16 bit segment page number from the original address to yield a 40 bit virtual page number. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and we get a 52 bit virtual address which is used internally by the processor. The 40 bit virtual page number is then used in a lookup mechanism to find a 20 bit real page number. 4-14 Kernel Internals © Copyright IBM Corp. which is combined with the 12 bit page offset to end up with a 32 bit real address. Combine this with the 12 bit page offset. Note that it is completely analogous to the preceding 32 bit platform illustration.0 Notes: 64-bit Hardware Platform Address Resolution The visual above illustrates the address resolution process for 64-bit hardware platforms. 2001. 2003 Unit 4. The segment number is mapped to a 52 bit Virtual Segment ID (VSID) (using either a segment lookaside buffer (SLB) or a segment table (STAB)). which is combined with the 12 bit page offset to end up with a 64 bit real address.0. The 68 bit virtual page number is then used in a lookup mechanism to find a 52 bit real page number. 64-bit hardware allows the operating system to define a virtual memory space that is significantly larger than the maximum amount of real memory that can be addressed. Addressing Memory 4-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and we get an 80 bit virtual address which is used internally by the processor. These 52 bits are used with the segment page number from the original address to yield a 68 bit virtual page number.0. 64 Bit Hardware Address Resolution BE0070XS4. © Copyright IBM Corp.V2.3 Student Notebook Uempty 64-bit Hardware Address Resolution Segment # 36 Virtual Page Index Page Offset 16 Segment Lookaside Buffer 52 Segment Table (STAB) 12 Page Offset Segment ID 68 80-bit Virtual Address Virtual Page Number Translation Look-Aside Buffer (TLB) Hash Anchor Table (HAT) Hardware Page Frame Table (PFT) Software Page Frame Table Real Page Number 52 64 Real Address Figure 4-9. . This is accomplished via the use of a segment table. Each 64 bit effective address uses the first 36 bits as a segment number. shared Memory in a shared segment may be mapped to the same virtual address in more than one process. Memory in private segments is only mapped to a single process’ address space. Segment Types BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Private vs.Student Notebook Segment Types Kernel Segment User Text Process Private Shared Library Text Shared Data Shared Library Data Figure 4-10. 4-16 Kernel Internals © Copyright IBM Corp. This prevents one process from accessing or altering another process’ private memory.0 Notes: Introduction Several segment types are used in a process’s address space. The segment types are listed in the visual above. These segments can only be accessed by code running in the kernel protection domain. 2001. This allows the sharing of data between processes. Kernel segments Kernel segments are segments that are shared by all process on the system. The user data .Kernel per-process data such as the u-block (accessible only in kernel mode) . Running a debugger When running a debugger. This protection allows a single copy of a text segment to be shared by all processes associated with the same program. .0.3 Student Notebook Uempty User text The user text segments contain the code of the program. This allows debuggers to set breakpoints directly in code. In that case. Addressing Memory 4-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. It shares its contents with the process private segment of the parent process.V2. the process private segment of the child process is created as a ‘copy-on-write’ segment. For example.The primary kernel thread stack (accessible only in kernel mode) . Threads in user mode have read-only access to text segments to prevent modification during program execution. if two processes in the system are running the ls command. Whenever the parent or child process modifies a page that is part of the process private segment. This results in a major performance advantage for the kernel. Process private segment The process private segment is not shared among other processes. © Copyright IBM Corp.Per-process loader data (accessible only in kernel mode) Performance advantage When a process calls fork. the page is actually copied into the segment for the child process. a private read/write copy of the text segment is used. The process private segment contains: . then the instructions of ls are shared between them.Text and data from explicitly loaded modules (for 32-bit programs) . especially in the (very common) situation where the newly created child process immediately performs an exec() call to start running a different program. 2001.0.The user stack (for 32-bit programs) . the status of the text segment is changed from shared to private. 2003 Unit 4. Shared data Mapped memory regions. 4-18 Kernel Internals © Copyright IBM Corp.Addresses of data items are generally the same across processes. .Each process has one shared library data segment. Or. can serve as large pools for exchanging data among processes. . also called shared memory areas. . These elements are placed in the shared library data segment. A shared library segment: .A shared data segment can represent a single memory object or a collection of memory objects. The shared library text is loaded into this segment when a module is loaded via the exec() system call. a program may issue load() calls to get additional shared modules. . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Contains a copy of the program text (instructions) for the shared libraries currently in use in the system.Are added to the user address space by the loader when the first shared library is loaded.Student Notebook Shared library text The shared library text segment contains mappings whose addresses are common across all processes.Shared memory can be attached read-only or read-write. . . .Data itself is not shared.A process can create and/or attach a shared data segment that is accessible by other processes. . The shared library data segments act as extensions of the process private segment. Shared library data segment Functions in shared libraries can define variables and other data elements that are private to a process. Executable modules list the shared libraries they need at exec() time. Each process using text from the shared library text segment has a copy of the corresponding data in the per-process shared library data segment. 0. . or when many processes maintain a common large database. 2003 Unit 4.Mapping file data into the process address space (mmap() services). © Copyright IBM Corp. Addressing Memory 4-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.3 Student Notebook Uempty Shared Memory Memory Segments Process A effective address space Virtual memory Process B effective address space Figure 4-11.0 Notes: Introduction Shared memory areas can be most beneficial when the amount of data to be exchanged between processes is too large to transfer with messages. Shared memory address The shared memory is process-based and can be attached at different effective addresses in different processes. Methods of sharing The system provides two methods of sharing memory: . Shared Memory BE0070XS4. 2001.V2. Therefore.Student Notebook . Serialization There is no implicit serialization support when two or more processes access the same shared data segment. . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 4-20 Kernel Internals © Copyright IBM Corp.Mapping to anonymous memory regions that may be shared (shmat() services). processes using shared memory areas must set up a signal or semaphore control method to prevent access conflicts and to keep one process from changing data that another is using. The available subroutines do not provide locks or access control among the processes. shmat functions A program can use the following functions to create and manage shared memory segments. and.Attaches a shared memory segment shmdt () .0. shmat Memory Services BE0070XS4. when supporting objects larger than 256 MB shared memory regions.V2.Gets or creates a shared memory segment shmat () .3 Student Notebook Uempty shmat Memory Services shmctl () . Using shmat The shmget() system call is used to create a shared memory region.Detaches a shared memory segment disclaim () .0.Controls shared memory operations shmget () . creates multiple segments.0 Notes: Introduction The shmat services are typically used to create and use shared memory objects from a program. 2003 Unit 4. . © Copyright IBM Corp. Addressing Memory 4-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Removes a mapping from a specified address range within a shared memory segment Figure 4-12. 2001. 2001. EXTSHM The environment variable EXTSHM=ON allows shared memory regions to be created with page granularity instead of the default segment granularity.Student Notebook The shmat() system call is used to gain address ability to a shared memory region. 4-22 Kernel Internals © Copyright IBM Corp. . This allows more shared memory regions within the same-sized address space. with no increase in the total amount of shared memory region space. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2. This single-level store approach can also greatly improve performance by creating a form of © Copyright IBM Corp. Advantages Memory mapped files provides easy random access. 2001. 2003 Unit 4.0.0 Notes: Introduction Memory segments can be used to map any ordinary file directly into memory. Instead of reading and writing to the file.3 Student Notebook Uempty Memory Mapped Files Virtual memory Effective Address space Disk File Figure 4-13. using system calls. This avoids the system call overhead of read() and write(). as the file data is always available. however. . Memory Mapped Files BE0070XS4. shmat()can also be used to map disk files. the program would just access variables stored in the segment. Addressing Memory 4-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. mmap () The mmap() service is normally used to map disk files into a process address space. the mmap() subroutine extends this capability beyond that provided by the shmat() subroutine by allowing a relatively unlimited number of such mappings to be established. the file data is mapped directly into the user’s address space. 4-24 Kernel Internals © Copyright IBM Corp. Determines residency of memory pages. . mmap services The mmap() services are typically used for mapping files. although they may also be used for creating shared memory segments. 2001. Un-maps a mapped memory region. Maps an object file into virtual memory. Synchronizes a mapped file with its underlying storage device. Modifies the access protections of memory mapping. Instead of buffering the data in the kernel and copying the data from kernel to user. Of course. .Only a portion of a file needs to be mapped.Many files are mapped simultaneously. Service madvise() mincore() mmap() mprotect() msync() munmap() Description Advises the system of a process' expected paging behavior. even if some are using mapping and others are using the read/ write system call interface. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Private mapping is required. . this may require synchronization between the processes. When to use mmap() Use mmap() under the following circumstances: . When to use shmat() Use the shmat() services under the following circumstances: . Both the mmap()and shmat() services provide the capability for multiple processes to map the same region of an object so that they share addressability to that object. However.Student Notebook Direct Memory Access (DMA) file access.When mapping files larger than 256 MB. Shared files A mapped file can be shared between multiple processes.Page-level protection needs to be set on the mapping.Portability of the application is a concern. . . Mapping types There are a 3 mapping types: .When mapping shared memory regions which need to be shared among unrelated processes (no parent-child relationship). 2003 Unit 4.Read-only mapping .Deferred-update mapping Read-write mapping allows loads and stores in the segment to behave like reads and writes to the corresponding file.0. Any storing into the segment modifies the segment. . The difference between this mapping and read-write mapping is that the modifications are delayed. and allows the application to avoid a costly temporary file that may otherwise be required.0. Read-only mapping allows only loads from the segment.Read-write mapping . the application can begin modifying the file data (by memory-mapped loads and stores) and then either commit the modifications to the file system (via fsync()) or discard the modifications completely. a thread that loads beyond the end of the file loads zero values.V2. when eleven or fewer files are mapped simultaneously and each is smaller than 256 MB. The operating system generates a SIGSEGV signal if a program attempts an access that exceeds the access permission given to a memory region. . but does not modify the corresponding file. . If all processes that have a file open with the O_DEFER flag set close that file before an fsync() or synchronous update operation is made against the file then that file is not updated.3 Student Notebook Uempty . Just as with read-write access. Deferred-update mapping also allows loads and stores to the segment to behave like reads and writes to the corresponding file. 2001.For 32-bit applications.When mapping entire files. This can greatly simplify error recovery. With deferred update (O_DEFER flag set on file open). Addressing Memory 4-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. read-only shared. Process Private Segment (Data. Segment zero contains the first kernel segment. segment numbers (Effective Segment IDs) have different uses in user and kernel modes. stack. User Text . 2001. 32-bit user mode The table above shows the segment layout of a user mode 32-bit process.0 Notes: Introduction For the 32-bit hardware platform. 4-26 Kernel Internals © Copyright IBM Corp. The user program text (application code) is located in segment 1. read-only private. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The kernel segment contains the system call table and the kernel text. BSS. stack and heap for the program. Ublock. read-write Figure 4-14. The u-block for the process is also located in segment 2. BSS (uninitialized data). Segment 2 contains the data. read-write shared. heap) Shared data (shmat or mmap) NOTE: for big data programs segments 3-10 can optionally be used as additional heap. 32-bit User Address Space BE0070XS4. uthread.applications text (code). This represents how a process running in user mode will see its effective memory. . read-only shared.Student Notebook 32-bit User Address Space Segment Number (SID) 0 1 2 3-12 Segment Type and Use Kernel Segment. read-write 13 14 15 shared. read-write private. Shared library text Shared data (shmat or mmap) Shared library data segment Attributes shared. 0. Such a model is required for programs which exceed the limit imposed by the normal 32-bit address space (i. and stack. Segment 13 contains the text for shared libraries (library code). Segment 14 provides an additional segment for shmat() and mmap(). a single 256MB segment for heap and data and stack). Addressing Memory 4-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Big data model A big data model is supported for 32-bit applications.e.3 Student Notebook Uempty Segments 3 -12 are used for shmat() and mmap() areas. . Segment 15 holds the library data. To accomplish this segments 3 . Eliminating these segments as shmat() and mmap() areas.0. © Copyright IBM Corp. This allows an application to use more segments for heap. 2001.12 are added to the heap.V2. 2003 Unit 4. data. Any data passed between kernel and user will occur by copying data in and out of this segment. 4-28 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook 32-bit Kernel Address Space Segment Number 0 1 2 3 7-10 14 15 Segment type and use Kernel and kernel extension text and data. 2001. The segment layout for the 32-bit kernel is shown above. 32-bit Kernel Address Space BE0070XS4. This allows the kernel access to this section of the user address space. . Extended kernel address space (file system and network data) Private process segment Kernel heap segment MBUF segments Kernel address space Kernel thread segment Figure 4-15. Private process segment Segment 2 the (private process segment) is mapped the same for both user and kernel modes.0 Notes: 32-bit kernel mode When a process switches into kernel mode (32-bit kernel) the mapping of segments is changed so that the kernel may access its entire address space. 0XEFFF_FFFF_F 0xF000_0000_0 . 2003 Unit 4. data. BSS and heap can occupy from segments 0x10 through segment 0x6FFFFFFF.3 Student Notebook Uempty 64-bit User/Kernel Address Space Segment Number (hex) 0x0000_0000_0 0x0000_0000_1 0x0000_0000_2 0x0000_0000_3 .0x09FF_FFFF_F 0x0A00_0000_0 . © Copyright IBM Corp. 64-bit User/Kernel Address Space BE0070XS4. bss. for the 64-bit case one segment layout applies to both user and kernel modes.0x08FF_FFFF_F 0x0900_0000_0 . This allows for 64-bit programs to be significantly larger than 32-bit programs.0. text. kernel text Reserved for system use Reserved for user mode loader (process private segment) Shmat or mmap use Reserved for user mode loader shmat or mmap use Reserved for user mode loader Application text. Addressing Memory 4-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Also.0x0000_0000_C 0x0000_0000_D 0x0000_0000_E 0x0000_0000_F 0x0000_0001_0 . data.0.0x0FFF_FFFF_F 0x1000_0000_0 . stack and heap A program’s text. 2001. BSS and heap Default application shmat and mmap area Application explicit module load area Shared library text and per-process shared library data Reserved for future use Application primary thread stack Reserved for future use Additional kernel segments Figure 4-16.V2.0x07FF_FFFF_F 0x0800_0000_0 .0x0EFF_FFFF_F 0x0F00_0000_0 .0 Notes: Segment 64-bit layout The 64-bit model adds many more segments to the effective address space.0x06FF-FFFF_F 0x0700_0000_0 . .0xFFFF_FFFF_F Segment usage System call tables. data. 4-30 Kernel Internals © Copyright IBM Corp. The segments from 0xF00000000 and up may be used for additional kernel segments.Student Notebook shmat() and mmap() For 64-bit applications the default segments for shmat() and mmap() are segments 0x70000000 .0x7FFFFFFF. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . 2001. Kernel segments Segment 0 is the first kernel segment. Note that segments 0x3-0xC and segment 0xE are also reserved for shmat() and mmap(). this mirrors the 32-bit segment model. True or False? Figure 4-17.0 Notes: © Copyright IBM Corp. AIX divides physical memory into ______.V2. 2. Addressing Memory 4-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.3 Student Notebook Uempty Checkpoint 1. 2003 Unit 4. 3.0. Shared library data segments can be shared between processes.0. The _____________ provides each process with its own _______address space. A 32-bit effective address contains a ______segment number. Checkpoint BE0070XS4. True or False? 6. The 32-bit user address space layout is the same s the 32-bit kernel address space layout. 2001. 4. . A segment can be up to ______ in size. Student Notebook Exercise Complete exercise four Consists of theory and hands-on Ask questions at anytime Activities are identified by a What you will do: Given the address of a memory object you will identify what segment the address belongs to and speculate as to how the object was created. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Exercise. 4-32 Kernel Internals © Copyright IBM Corp. . 2001. Figure 4-18.0 Notes: Turn to your lab workbook and complete exericse four. BE0070XS4. 0 Notes: © Copyright IBM Corp. Unit Summary BE0070XS4. 2003 Unit 4.0.3 Student Notebook Uempty Unit Summary Pages size = 4096 Virtual memory management Address spaces effective virtual physical Segment size = 256 MB 32-bit vs 64-bit segment layout Figure 4-19. .V2. 2001.0. Addressing Memory 4-33 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .Student Notebook 4-34 Kernel Internals © Copyright IBM Corp. 2001. .3 Student Notebook Uempty Unit 5. you should be able to: • Identify the key functions of the AIX virtual memory manager • Given a memory object type identify the location of the backing store the VMM system will use for this object • Describe the affect that different paging space allocation policies have on applications and the system • Find the current paging space usage on the system • Identify the paging characteristics of a system from a vmcore file How You Will Check Your Progress Accountability: • Exercises using your lab system • Unit review References PowerPC Microprocessor Family: The Programmers Reference Guide Available from http://www-3. What You Should Be Able to Do After completing this unit. 2001. Memory Management 5-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.com/chips/techlib/techlib.0.nsf/productfamilies/PowerPC © Copyright IBM Corp.ibm.V2. Memory Management What This Unit Is About This unit describes how AIX 5L manages memory using demand paging.0. 2003 Unit 5. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .0 Notes: 5-2 Kernel Internals © Copyright IBM Corp.Student Notebook Unit Objectives At the end of this lesson you should be able to: Identify the key functions of the AIX virtual memory management system Given a memory object type identify the location of the backing store the VMM system will use for this object Describe the affect that different paging space allocation policies have on applications and the system Find the current paging space usage on the system Identify the paging characteristics of a system from a vmcore file Figure 5-1. Unit Objectives BE0070XS4. 0.0 Notes: Introduction In the Addressing Memory lesson we saw how AIX 5L manages the effective address space for both the user and kernel.V2.3 Student Notebook Uempty Virtual Memory Management Physical Memory Virtual address space Process 1 Effective address Process 2 Filesystem pages Paging space Figure 5-2. Memory Management 5-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Function of the VMM The VMM is responsible for keeping track of which program pages are resident in memory and which are on secondary storage (disk). © Copyright IBM Corp.0. Virtual Memory Management (VMM) BE0070XS4. . When all of the physical memory is in use. It handles interrupts from the address translation hardware in the system to determine when pages must be retrieved from secondary storage and placed in physical memory. the VMM decides which programs’ pages are to be replaced and paged out to secondary storage. 2003 Unit 5. 2001. This lesson focuses on the management of the virtual address space by the Virtual Memory Manager (VMM). the virtual address is mapped (if it is not already mapped) by the VMM to a physical address (where the data is located). 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. Access protection also allows programs to set up memory that may be shared between processes.Student Notebook Each time a process accesses a virtual address. . 5-4 Kernel Internals © Copyright IBM Corp. This function protects programs from incorrectly accessing kernel memory or memory belonging to other programs. Access protection Another function of the VMM is to provide for access protection that prevents illegal access to data. Working objects Working objects (also called working storage and working segments) are temporary segments. 2003 Unit 5. Memory Management 5-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Object Types BE0070XS4.0. 2001.0 Notes: Memory Object Types Introduction Memory objects in AIX 5L are classified based how the object is used. All memory objects are assigned one of five classification types. . Process data is created by the loader at run time and is paged in and out of paging space. such as stack and data areas.V2.3 Student Notebook Uempty Object Types Working objects Persistent objects Client objects Log objects Mapping objects Figure 5-3. The Virtual Memory Management system manages each memory object based on its type. The working storage segment holds the amount of paging space allocated to © Copyright IBM Corp. used during the execution of a program.0. Remote program text pages (read-only pages) page-out to paging space. the page is marked as modified and eventually paged-out directly to the original disk location. When the contents of a file changes. Persistent objects The VMM is used for performing I/O operations of file systems.Student Notebook pages in the segment. Persistent pages do not use paging space. When remote pages are modified. . Persistent objects are used to hold file data for the local file systems. File data pages and program text are both part of persistent storage. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. they are paged-in. and never paged-out to disk. the data pages are paged-in. from where they can be paged-in later if needed. they are marked and eventually paged-out to the original disk location across the network. When the process opens the file. 2001. Log objects Log objects are used for writing or reading journaled file systems file logs during journaling operations. 5-6 Kernel Internals © Copyright IBM Corp. Part of the AIX kernel is also pageable and is part of the working storage. Mapping objects Mapping objects are used to support the mmap() interfaces. File system reads and writes occur by attaching the appropriate file system object and performing loads/stores between the mapped object and the user buffer. Client objects Client objects are used for pages of client file systems. The program text pages are read-only pages. which allows an application to map multiple objects to the same memory segment. Memory Management 5-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 5.0. © Copyright IBM Corp.0 Notes: Introduction AIX is a demand paging system.3 Student Notebook Uempty Demand Paging Physical Memory Virtual address space Kernel or user effective address space Pinned Page Fault Filesystem pages Paging space Backing store Figure 5-4. Page faults A page fault occurs when a thread tries to access a page that is not currently in physical memory. Paging is done on-the-fly and is invisible to the program causing the page fault. Physical pages (frames) are not allocated for virtual pages until they are needed (referenced). 2001. .0.V2. Demand Paging BE0070XS4. How it works Data is copied to a physical page only when referenced by a program or by the kernel. References to a non allocated page results in a page fault. 2. The steps to resolve a page fault are: Step 1. The actions of the VMM in this case are described over the next few pages of the class. Page validity The VMM checks to ensure that the effective address being referenced is part of the valid address range of the segment that contains the effective address. 5. however the page containing the effective address has not been instantiated. In this case. 3. It then updates the hardware page frame table to reflect the physical location of the page. 5-8 Kernel Internals © Copyright IBM Corp. the VMM allocates a physical frame for use by the page. If the processor is running in user mode. it raises a page fault condition. In this case. an unresolveable page fault results in a system crash. and the page containing the effective address has already been instantiated. 4.The effective address is within the valid address range for the segment. There are a number of possible scenarios. and then updates the segment information to indicate that the page has been allocated. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. then the unresolved page fault results in the running process being sent either a SIGSEGV (Segmentation violation) or SIGBUS (Bus error) depending on the address being referenced. Action The hardware detects a page fault and raises the page fault condition. .Student Notebook The mapping of effective addresses to physical addresses is done in the hardware on a page-by-page basis. the page fault cannot be resolved. The page is loaded from disk into physical memory. If the processor is running in kernel mode. 2001. The pages for the malloced space are not instantiated until they are referenced for the first time. . For example. and allows the faulting thread to continue. Execution of the faulted thread is resumed. . When the hardware finds that there is no mapping to physical memory. Execution of the faulting thread is suspended.The effective address is outside the valid address range for the segment. Control is transferred to the page fault handler (part of the virtual memory management system). Page fault handler The job of a virtual memory management system (VMM) is to handle page faults so that they are transparent to the thread using effective memory addresses. this happens when an application performs a large malloc() operation. .The effective address is within the valid address range for the segment. Only some of the kernel is in physical memory at one time.These pages are said to be pinned. Pageable kernel AIX’s kernel is pageable.0. Kernel pages that are not currently being used can be paged out.0. Data that has not been recently used is kept in paging space.V2. 2003 Unit 5. valuable physical memory will never be used.3 Student Notebook Uempty Advantages The demand paging system in AIX allows more virtual pages to be allocated than can be stored in physical memory. Demand paging also saves much of the overhead of creating new processes because the pages for execution do not have to be loaded until they are needed. and continues to do so until a high-water mark threshold is reached. If a process never makes use of a portion of its virtual space. The interrupt processing portion of a device driver is pinned. Memory Management 5-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. If the number of pages available goes below a low-water mark threshold. Only a small part of the kernel is required to be pinned. A pager daemon attempts to keep a pool of physical pages free. 2001. . Pinned pages Some parts of the kernel are required to stay in memory because it is not possible to perform a page-in when those pieces of code execute. the pager frees the oldest referenced pages. Physical memory management Data that has been recently used is kept in physical memory. Address translation requires both hardware and software components. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Data Structures Effective address space Hardware Page Frame Table Physical memory Software page frame table External page tables (XPT) Segment ID and page number Paging space SID table File inode Filesystem pages Figure 5-5. 2001. This section covers the relationship between the hardware and software components of the VMM. Data structures The diagram above shows the overall relationships between the major AIX data structures involved in mapping a virtual page to a physical page or to paging space. 5-10 Kernel Internals © Copyright IBM Corp. Data Structures BE0070XS4. .0 Notes: Introduction The main function of the VMM is to make translations from the effective address to the physical address. V2.Paging disk (working object) .0. If the page is valid. recovers the page if necessary and updates the hardware’s frame page frame table with the location of the page.3 Student Notebook Uempty Page faults A page fault causes the AIX (VMM) to do the bulk of its work. A faulted page will be recovered from one of the following locations: . the VMM determines the location of the page.0. Memory Management 5-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.File system object (persistent object) © Copyright IBM Corp. 2003 Unit 5.Physical memory (but not in the hardware PFT). . . It handles the fault by first verifying that the requested page is valid. 2001. an effective address refers to a piece of memory that is currently in real memory. If a translation is found in this table the physical page is returned to the requestor. We say the memory is paged in. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Illustration The flow of the best case address translation is illustrated above. These tables only contain a subset of all available translations for the contents of physical memory. 2001.Student Notebook Hardware Page Mapping Effective address space Hardware Page Frame Table Physical memory Software page frame table External page tables (XPT) Paging space SID table File inode Filesystem pages Figure 5-6. Hardware Page Mapping BE0070XS4. There is no need for a page fault to be generated. . 5-12 Kernel Internals © Copyright IBM Corp. Hardware Page Frame Table (PFT) A hardware Page Frame Table (PFT.0 Notes: Introduction In a normal situation. sometimes “HWPFT”) of address translations is used to make the conversions from effective addresses to physical addresses. therefore.0. The VMM software must supplement the hardware table with a software-managed page table.0. . the translation entry is not in the hardware table. however. © Copyright IBM Corp.0 Notes: Introduction The size of the hardware Page Frame Table is limited. Page not in Hardware Table BE0070XS4.V2. 2001. The procedure is shown in the table above. Memory Management 5-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Page not in Hardware Table Effective address space Hardware Page Frame Table Physical memory Software page frame table External page tables (XPT) Paging space SID table File inode Filesystem pages Figure 5-7. The VMM must be called to update the hardware tables. Illustration When a translation cannot be found in the hardware table. a page fault is generated. 2003 Unit 5. the hardware can not satisfy all address translation requests. The physical page may be resident in memory. SWPFTs contain information connected with a page as well as page-in flags. Only some parts of the software PFT are pinned. page-out flags. 5-14 Kernel Internals © Copyright IBM Corp. Step 1. The faulted thread just continues the execution at the instruction that caused the fault. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Action The hardware Page Frame Table is searched for a page translation and none is found. a kernel exception is generated. 2001. The software PFT is big enough to contain translation information for every page resident in physical memory. 2. free list flags. and is used and managed by the VMM software. Software Page Frame Table Software Page Frame Table (SWPFT) is an extension of the hardware Page Frame Table. since it is already in memory. The VMM first verifies that the requested page is valid. the VMM searches the software PFT for the page. and the process resumes execution. . If the page is not valid. 3. 4. This process resembles hardware processing. The hardware generates a page fault causing the VMM to be called. If the page is valid. but uses a software page table instead. Procedure These steps assume that the memory page is in memory but not in the hardware Page Frame Table.Student Notebook What happened to the thread? When this type of page fault is resolved the dispatcher is not run. and the block number. They contain the device information used to obtain the proper page from disk. If the page is found: 5. • The hardware Page Frame Table is updated with the real page number for this page. • No page-in of the page occurs. the VMM determines whether it is on paging space or elsewhere on disk.0. Page on Paging Space BE0070XS4. Illustration Working pages are mapped to disk blocks in the paging space.V2. .3 Student Notebook Uempty Page on Paging Space Effective address space Hardware Page Frame Table Physical memory Software page frame table External page tables (XPT) Paging space SID table File inode Filesystem pages Figure 5-8. and the page is loaded into a free memory page. Any process or thread waiting for a page fault to be handled is put to sleep until the page is available. The procedure for loading a page from paging space is shown in the visual on the previous page.0. then the disk block containing the page is located. Memory Management 5-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. Waiting for I/O Copying a page from the paging space to an available frame is not a synchronous process. 2001. If the page is in paging space. 2003 Unit 5.0 Notes: Introduction If a page is not found in physical memory. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Each XPT direct block covers 1 MB of the 256MB segment. . It contains the page’s state and disk block information. Each word in a direct block represents a single page in the segment.0 Notes: External Page Table (XPT) The XPT maps a page within working storage segments to a disk block on external storage. page 65280 255 page 65535 255 . The second level consists of 256 direct blocks. . Figure 5-9. . Description The first level of the tree is the XPT root block. Structure Each segment that is mapped to paging space has the following XPT structure. Each word in the root block is a pointer to one of the direct blocks. The XPT is a two-level tree structure. . 5-16 Kernel Internals © Copyright IBM Corp. External Page Table (XPT) BE0070XS4. . .Student Notebook External Page Table (XPT) XPT Direct Block #0 0 Disk blocks in paging space page 0 XPT Root 0 page 255 1 MB 255 XPT Direct Block #255 0 . 2001. . There is one XPT for each working storage segment. 2001. When the I/O completes. 4. 2. 2003 Unit 5. © Copyright IBM Corp.3 Student Notebook Uempty Procedure In this procedure the faulting thread must be suspended until I/O for the faulting page has completed.0. the VMM is notified. 8. The VMM finds the correct XPT (direct block from XPT root). The VMM issues an I/O request to the device with the logical block and physical address of the page to be loaded. Step 1. The VMM gets the paging space disk block number from the XPT direct block. Memory Management 5-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.V2.0. 6. 3. The free list contains one entry for each free frame of real memory. The thread waiting on the frame is awakened and resumes at the faulting instruction. The net effect is that the process or thread has no knowledge that a page fault occurred except for a delay in its processing. The VMM updates the hardware PFT. . The VMM takes the first available frame from the free frame list. Action The thread causing the fault is suspended. The VMM looks up the object ID for this address in the Segment ID table and gets the External Page Table (XPT) root pointer. 7. 5-18 Kernel Internals © Copyright IBM Corp. Loading Pages From the File System BE0070XS4. allowing the VMM to find and page-in the faulting block. The effective address for the mapped page of the local file is indexed in the Segment Information Table (SID). The inode is pointed to by the SID entry. Procedure Persistent pages are mapped to local files located on file systems.Student Notebook Loading Pages From the File System Effective address space Hardware Page Frame Table Physical memory Software page frame table External page tables (XPT) Paging space SID table File inode Filesystem pages Figure 5-10. . The VMM uses the information contained in a file’s inode structure to locate the pages for the file.0 Notes: Introduction Persistent pages do not use external page tables. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. The VMM schedules the modified persistent pages to be written to their original location on disk when: . File system objects File system reads and writes occur by attaching the appropriate file system object and performing loads/stores between the mapped object and the user buffer. Memory Management 5-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. A local file has a segment allocated and has an entry (SID) in the segment information table. Scheduling a page to be written does not mean that the data is written to disk immediately. A sync() operation flushes all scheduled pages to disk. 2001. The pages for files compete for storage the same way as other pages.V2. Persistent pages AIX uses a large portion of memory as the file system buffer cache. . .3 Student Notebook Uempty File System I/O Introduction The paging functions of the VMM is also used to perform file “reads” and “writes” by processes. It means that file objects are not directly addressable in the current address space but instead are temporarily attached.The VMM needs the frame for another page. 2003 Unit 5.The file is closed. A file gnode contains information about which segment belongs to the particular file. The sync() operation is performed by the syncd daemon every 60 seconds by default or by a user running the sync command. .The sync operation is performed. © Copyright IBM Corp.0. Object Type / Backing Store BE0070XS4.0 Notes: Introduction Paging provides automatic backup copies of memory objects on disk. Working B. A regular disk file 2. Questions Using what you know about memory object types. 2001. a regular disk file. 5-20 Kernel Internals © Copyright IBM Corp. or even on a network accessible disk file. Client Backing Store 1. match the object types on the left with the location of its backing store on the right in the visual above. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. This copy is called the backing store and can be located on a paging disk. An NFS disk file 3. Persistent C.Student Notebook Object Type A. Paging disk Figure 5-11. . Memory Management 5-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and the UID is not less than nokilluid. SIGDANGER Application programs can ask AIX to notify them when paging space runs low by registering to receive a SIGDANGER signal.3 Student Notebook Uempty Paging Space Management Process Steps Condition 1 If the number of free paging space blocks falls below the paging space warning level (npswarn).0 Notes: Introduction Proper management of paging space is required for the system to perform. Low paging space can result in failed applications and system crashes.V2. This feature allows applications to release memory or take other appropriate actions when paging space runs low. The default action for SIGDANGER is to ignore the signal. Figure 5-12. Paging Space Management Process BE0070XS4. 2 If the number of free paging space blocks falls below the paging space kill level. Action SIGDANGER is sent to all process (except kprocs) that have registered to handle the signal A SIGKILL is sent to the newest process that does not have a signal handler for SIGDANGER. .Paging space warning level © Copyright IBM Corp.0. Threshold AIX has two paging space thresholds. (npskill) 3 If paging space is still below the paging space kill threshold. 2001. 2003 Unit 5. they are: .0. SIGKILL will continue to be sent to eligible processes until the free paging space rises above the kill threshold. Both thresholds are set with the vmtune (AIX 5. which means processes owned by root are eligible to be sent a SIGKILL. unload(L_PURGE).1) and vmo (AIX 5. "Paging space low!\n"). The value of nokilluid is 0 by default. which can be set with the vmtune (AIX 5.2) commands. } 5-22 Kernel Internals © Copyright IBM Corp. Nokilluid The SIGKILL signal is only sent to processes that do not have a handler for SIGDANGER and where the UID of the process is greater than or equal to the kernel variable nokilluid.1) and vmo (AIX 5. Example The init process (pid 1) registers a signal handler for the SIGDANGER signal.Student Notebook . int danger(void) { if (own_pid == SPECIALPID) { console(NOLOG. Process The table above describes the actions AIX takes when paging space becomes low. The handler prints a warning message on the system console and attempts to free memory by unloading unused modules. M_DANGER. 2001.Paging space kill level Application programs can monitor these thresholds and free paging space using the psdanger() function. /* unload and remove any * unused modules in kernel or * library */ } return(0). Age of the process The kernel send the SIGKILL signal to the youngest eligible process. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .2) commands. This helps to prevent long-running processes from being terminated due to a low paging space condition caused by a recently started process. 1. Note that this is a system-wide policy and applies to all processes running on the system. © Copyright IBM Corp.V2. Finding a process allocation policy Use kdb to examine the process flags in the proc structure to determine a process’s current paging space allocation policy. PSALLOC A process that has the environment variable PSALLOC=early will cause the VMM to allocate paging space for any memory which is requested. . This helps to ensure that the paging space will be available if it is needed. Note that this policy holds only for this process and is not system-wide.0 Notes: Introduction Individual processes may select when paging space will be allocated for them. which means that paging space will not be allocated until a page Out occurs. The system wide default applies.1}.3.2 and later releases the system default is Deferred Paging Space Allocation (DPSA). This is called a paging space policy. which means paging space will be allocated when requested memory is accessed. whether or not the memory is accessed. 2001.3 Student Notebook Uempty Paging Space Allocation Policy Policy Early allocation PSALLOC=early Description Causes paging space to be allocated as soon as the memory request is made. Paging Space Allocation Policy BE0070XS4. This can be controlled with vmtune -d {0. Memory Management 5-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. For AIX 4. This is the algorithm that was used on AIX v3. VMtune -d 0 will turn DPSA off. PSALLOC= Figure 5-13.0.0. 2003 Unit 5. .h as: #define SPEARLYALLOC 0x04000000 /* allocates paging space early */ This flag can be seen through kdb by running the p <slot_number> subcommand. 2001. This flag is defined in proc. 5-24 Kernel Internals © Copyright IBM Corp. If the flag is set it will show up in the second set of “FLAGS” indicated by the name: “SPEARLYALLOC”.Student Notebook When early allocation is selected the SPEARLYALLOC flag will be set in proc->p_flag. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Free Memory BE0070XS4.0.0. Page stealer The page stealer is invoked when the number of memory pages on the free list drops below the threshold defined by the value of minfree. the VMM always wants some physical memory to be available for page-ins.3 Student Notebook Uempty Free Memory Free memory list free pages < minfree maxfree Run page stealer minfree free pages no => maxfree Figure 5-14. 2003 Unit 5. This section describes the free memory list and the algorithms used to keep pages on the list.V2. 2001. The page stealer attempts to © Copyright IBM Corp. Memory Management 5-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Free memory list The VMM maintains a linked list containing all the currently free real memory pages in the system.0 Notes: Introduction To maintain system performance. the VMM just takes the first page from this list and assigns it to the faulting page. When a page fault occurs. The values of maxfree and minfree can be viewed or adjusted on AIX 5. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The page replacement algorithm used in AIX is called the clock-hand algorithm. Page replacement algorithm The method used by the page stealer to select a page which should be placed on the free list is called the Page Replacement Algorithm.Student Notebook replenish the free list until it reaches the high threshold defined by maxfree.1 with the vmtune command (/usr/samples/kernel/vmtune). and on AIX 5. Evidence The page stealer is visible as the lrud kernel process. 5-26 Kernel Internals © Copyright IBM Corp.2 with the vmo command. 2001. . 2001. . © Copyright IBM Corp. the clock-hand algorithm writes the page to disk (to paging space or a file system) before stealing the page. The clock-hand advances whenever the algorithm advances to the next frame.V2.0 Notes: Clock hand The algorithm is called the clock-hand algorithm because the algorithm acts like a clock hand that is constantly pointing at frames in order.3 Student Notebook Uempty Clock Hand Algorithm Physical page Reference = 1 The reference bit is changed to zero when the clock hand passes rotation Reference = 0 Reference = 0 This page is eligible to be stolen Reference = 1 Figure 5-15. Clock Hand Algorithm BE0070XS4.0. Memory Management 5-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 5.0. If a modified page is stolen. then it is likely that all frames would have been referenced by the time the algorithm starts its second pass. Action Each time a page is referenced the hardware sets the referenced bit in the PTE (Page Table Entry) for that page. The hardware automatically sets the reference bit for a page translation whenever the page is referenced. . If the reference bit is found set the bit is reset. 5-28 Kernel Internals © Copyright IBM Corp. If the reference bit is found reset the page will be stolen. If it were to examine all memory frames in the system in one cycle. The number of frames considered in each cycle is known as the lrud bucket size. Step 1. 2001. 3. The process continues until the number of free pages reaches maxfree.Student Notebook Process This algorithm is commonly used in operating systems when the hardware provides only a reference bit for each page in the physical memory. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 5. The clock hand algorithm scans all PTEs checking the reference bit. Bucket size The clock hand algorithm examines a set of frames at a time. 4. 2. Memory Management 5-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. An I/O error occurs when paging in kernel data. Figure 5-16.V2. 2001.0 Notes: Introduction Not all page and protection faults can be handled by the OS. . An instruction storage exception occurs while in kernel mode. Fatal Memory Exceptions BE0070XS4. A memory exception occurs while in kernel mode without an exception handler set up. © Copyright IBM Corp. A protection fault occurs while in kernel mode on kernel data.3 Student Notebook Uempty Fatal Memory Exceptions In all of the following cases.0. the VMM bypasses all kernel exception handlers and immediately halts the system: A page fault occurs in the interrupt environment. 2003 Unit 5. A page fault occurs with interrupts partially disabled. the system will panic and immediately halt. When a fault occurs that cannot be handled by the OS. Student Notebook Checkpoint 1. Each ______ _______ has an XPT. Checkpoint BE0070XS4. . The system hardware maintains a table of recently referenced ______ to ______address translations. Figure 5-17. 2001. A ______ ______ when interrupts are disabled will cause the system to crash. The ________environment variable can be used to change the paging space policy of a process. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The S_____ P____ F____ T____ contains information on all pages resident in _______ _______. 2.0 Notes: 5-30 Kernel Internals © Copyright IBM Corp. 5. 4. A _________ signal is sent to every process when the free paging space drops below the warning threshold. 3. 6. © Copyright IBM Corp.0 Notes: Turn to your lab workbook and complete exercise five. Memory Management 5-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .3 Student Notebook Uempty Exercise Complete exercise five Consists of theory and hands-on Ask questions at anytime Activities are identified by a What you will do: Observe the effect of the AIX paging space allocation policies on an application program Investigate what effect running out of paging space has on applications and the system Diagnose a crash dump from a system with paging space depletion Figure 5-18.0. Exercise BE0070XS4.0. 2001.V2. 2003 Unit 5. clock hand Figure 5-19.Student Notebook Unit Summary Virtual memory management Memory objects types Demand paging system Backing store Paging space allocation policies Free memory list . Unit Summary BE0070XS4. 2001. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: 5-32 Kernel Internals © Copyright IBM Corp. V2.0.0.3 Student Notebook Uempty Unit 6. Logical Partitioning What This Unit Is About This unit describes the implementation of logical partitioning (otherwise known as LPAR) on pSeries systems. What You Should Be Able to Do After completing this unit, you should be able to: • Describe the implementation of logical partitioning • List the components required to support partitioning • Understand the terminology relating to partitions How You Will Check Your Progress Accountability: • Checkpoint questions • Unit review References AIX Documentation: AIX Installation in a Partitioned Environment Hardware Management Console for pSeries Installation and Operations Guide Available from http://www-1.ibm.com/servers/eserver/pseries/library/hardware_docs/hmc.html © Copyright IBM Corp. 2001, 2003 Unit 6. Logical Partitioning 6-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Unit Objectives At the end of this lesson you should be able to: Describe the implementation of logical partitioning List the components required to support partitioning Understand the terminology relating to partitions Figure 6-1. Unit Objectives BE0070XS4.0 Notes: 6-2 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Partitioning Subdivision of a single machine to run multiple operating system instances Collection of resources able to run an operating system image Processors Memory I/O devices Physical partition Building blocks Logical partition Independent assignment of resources Figure 6-2. Partitioning BE0070XS4.0 Notes: Introduction Partitioning is the term used to describe the ability to run multiple independent operating system images on a single server machine. Each partition has its own allocation of processors, memory and I/O devices. A large system that can be partitioned to run multiple images offers more flexibility than using a collection of smaller individual systems. © Copyright IBM Corp. 2001, 2003 Unit 6. Logical Partitioning 6-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Reasons for partitioning Partitioning is intended to address a number of pervasive requirements, including: - Server consolidation: The ability to consolidate a set of disparate workloads and applications onto a smaller number of hardware platforms, in order to reduce total cost of ownership (administrative and physical planning overhead). - Production and test environments: The ability to have an environment to test and migrate software releases or applications, which runs on exactly the same platform as the production environment to ensure compatibility, but does not cause any exposure to the production environment. - Data and workload isolation: The ability to support a set of disparate applications and data on the same server, while maintaining very strong isolation of resource utilization and data access. - Scalability balancing: The ability to create resource configurations appropriate to the scaling characteristics of a particular application, without being limited by hardware upgrade granularities. - Flexible configuration: The ability to change configurations easily to adapt to changing workload patterns and capacity requirements. Partitioning types In the UNIX market place, there are two main types of partitioning available: - Physical partitioning - Logical partitioning There are a number of distinct differences between the two implementations. 6-4 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Physical Partitioning Interconnect SMP Building Block SMP Building Block SMP Building Block Operating System Dedicated CPU, Memory and I/O Dedicated CPU, Memory and I/O Operating System Dedicated CPU, Memory and I/O Physical Partition Physical Partition Figure 6-3. Physical Partitioning BE0070XS4.0 Notes: Introduction Physical partitioning is the term used to describe a system where the partitions are based around physical building blocks. Each building block contains a number of processors, system memory and I/O device connections. A partition consists of one or more physical building blocks. The diagram shows a system that contains three building block units. The system currently is configured to run two partitions. One partition consists of all of the resources (CPU, memory, I/O) on two physical building blocks. The other partition contains of all of the resources on the remaining building block. © Copyright IBM Corp. 2001, 2003 Unit 6. Logical Partitioning 6-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Properties A system that implements physical partitioning has the following characteristics: - Multiple memory coherence domains, each with an OS image. A memory coherence domain is a group of processors that are accessing the same physical system memory. Memory coherence traffic (such as cache line invalidation, and snooping) is shared between the processors in the domain. - Separation controlled by interfaces between physical units. Memory coherence information stays within the physical building blocks allocated to the partition. A processor that is part of one building block cannot access the memory on another building block that is not part of the memory coherence domain (partition). - Strong software isolation, strong hardware fault isolation. Applications running inside an operating system instance have no impact on applications running inside another partition. A failure of a component on one system building block will not (or should not) impact a partition running on other building blocks. However the system as a whole still contains components that could impact multiple partitions in the event of failure, for example a failure of the backplane interconnect. - Granularity of allocation at the physical building block level. A partition that does not have enough resources can only be grown by incorporating whole building blocks, and therefore will include all of the resources on the building block, even though they may not be desired. For example, a partition that needs more processors will need to add another building block. By doing so, the partition will also incorporate the memory and I/O devices on that building block. - Resources allocated only by contents of complete physical group. The granularity of growing individual resources (CPU, memory, I/O) is determined by the amount of each resource on the physical building block being added to the partition. For example, in a system where each building block contains 4 processors, a partition that required more CPU power would receive an increment of 4 processors, even though perhaps only 1 or 2 would be sufficient. Example The Sun Enterprise 10000 and Sun Fire15K are examples of systems that use physical partitioning. In the case of Sun machines, the term domain is used instead of partition. 6-6 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Logical Partitioning RS232 RS422 OS OS Hypervisor OS Processors Hardware Management Console (HMC) Managed System Memory I/O adapters and devices Up to 16 LPARs LPAR 1 LPAR 2 LPAR 3 Ethernet Figure 6-4. Logical Partitioning BE0070XS4.0 Notes: Introduction Logical partitioning is the term used to describe a system where the partitions are created independently of any physical boundaries. The diagram shows a system configured with three partitions. Each partition contains an amount of resource (CPU, memory, I/O slots) that is independent of the physical layout of the hardware. In the case of pSeries systems, an additional system, the Hardware Management Console for pSeries (HMC), is required for configuring and administering a partitioned server. The HMC connects to the system through a dedicated serial link connection to the service processor. Additionally, applications running on the HMC communicate over an Ethernet connection with the operating system instances in the partitions to provide service functionality, and in the case of AIX 5.2, dynamic partitioning capabilities. © Copyright IBM Corp. 2001, 2003 Unit 6. Logical Partitioning 6-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Properties A system that implements logical partitioning has the following characteristics: - One memory coherence domain with multiple OS images. This basically means that all processors in the system are aware of the physical memory addresses being accessed by the other processors, even if they are in a different partition. Since each partition is allocated its own portion of physical memory, this has no real performance impact. - Separation controlled mainly by address mapping mechanisms. Rather than using physical boundaries between components to control the memory access available to each partition, a set of address mapping mechanisms provided by hardware and firmware features is used. The operating system running in each partition is restricted in its ability to access physical memory, and is only permitted to access physical memory that has been explicitly assigned to that partition. - Strong software isolation, fair-to-strong hardware fault isolation. Applications running inside an operating system instance have no impact on applications running inside another partition. The failure of the operating system in one partition has no impact on the others. - Granularity of allocation at the logical resource level (or below). In the case of pSeries systems, the current unit of allocation for each resource type is: • One CPU • Individual I/O slot • 256MB memory - Resources allocated in almost any combinations or amounts. The amount of memory allocated to a partition is independent of the number of CPUs or I/O slots. Each resource quantity is based on the system administrator’s understanding of the needs of the partition, rather than the physical layout of the machine. - Some resources can even be shared. In the case of pSeries systems, some resources are shared by all partitions. These are divided into two classes: • Physical resources (such as power supplies) that are visible to each partition. • Logical resources, where each partition is given its own “instance”, for example, the operator panel and virtual console devices provided by the HMC. 6-8 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Components Required for LPAR BE0070XS4.0. Logical Partitioning 6-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The hardware must be capable of recognizing the source of an interrupt and determining which partition should receive the interrupt notification.3 Student Notebook Uempty Components Required for LPAR Hardware Interrupt controller hardware Processors require RMO. it is a combination of features provided by different components. Hardware The following hardware features are required for LPAR support: . RML and LPAR ID registers Processors require Hypervisor support Means no LPAR support on older machines Firmware Global firmware image Partition specific firmware instance Hypervisor code Operating System Use of Hypervisor callout by VMM Means no LPAR support for older operating systems (e. .3) All 3 required for LPAR operation Figure 6-5.Interrupt controller hardware The interrupt controller hardware on the system directs interrupts to a CPU for processing. the interrupt controller hardware must be capable of maintaining multiple global interrupt queues.g. all of which must be present. In the case of a partitioned system. AIX 4.0. For © Copyright IBM Corp. one for each partition. Rather. 2001.V2. 2003 Unit 6.0 Notes: Introduction No single feature determines whether a pSeries system is capable of implementing LPAR or not. 2001.Locate an operating system boot image . A processor implements hypervisor support by recognizing the HV bit in the Machine Status Register (MSR). The hypervisor is described in detail later. Firmware The job of firmware in a system is to: . The use of the register is described in detail in a later part of this unit. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . All processors in the same partition have the same value loaded in the LPI register. along with the Problem State bit indicates if the processor is in hypervisor mode. If the interrupt is sent to a CPU that is part of a different partition. The use of the register is described in detail in a later part of this unit.In order to implement the required isolation between partitions.Identify and configure system components . • Real Mode Limit (RML) register The RML register is also used when the processor is referencing an address in real mode.Processor support A processor requires 3 new registers in order to be used in a partitioned environment. The POWER4 processor is the first CPU used in pSeries systems that has the required capabilities. the CPU would be unable to access the device to process the interrupt. a processor must have hypervisor support. . Hypervisor mode can only be invoked from Supervisor State. only kernel code can make hypervisor calls. Hypervisor mode is implemented in a similar fashion to the system call mechanism used to transition the processor between Problem State (user mode) and Supervisor State (kernel mode). . All processors in the same partition have the same value loaded in the RML register.Initialize/Reset system components .Create a device tree . an interrupt from a SCSI adapter card must be sent to the partition that controls the card and the devices connected to it.Load the boot image into memory and transfer control 6-10 Kernel Internals © Copyright IBM Corp. The HV bit of the MSR. In other words. • Logical Partition Identity register The LPI register contains a value that indicates the partition to which the processor is assigned.Student Notebook example. All processors in the same partition have the same value loaded in the RMO register. The registers are: • Real Mode Offset (RMO) register The RMO register is used by the processor when referencing an address in real mode. and Partition firmware. that run in hypervisor mode on the processor. The RTAS functionality provided by partition firmware performs validation checks and locking to ensure that the partition is permitted to access the particular hardware feature being used. An additional component of firmware required for LPAR support is the hypervisor function. The functionality of firmware is now divided into two parts. each partition requires an I/O device to act as the console. . and that its use does not conflict with that of another partition. Logical Partitioning 6-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.When the operating system image is terminated. it has control over the hardware. Since firmware in a partitioned system now has to deal with multiple operating system images. 2001. . control is returned to firmware. known as Global firmware. so it would be impractical to insist that each partition have its own native serial port.Virtual console serial device When multiple partitions are running on a system. The hypervisor code performs partition and argument validation before allowing the requested action to take place. or enforce the addition of additional serial adapters.3 Student Notebook Uempty . The global firmware is initialized when the system is first powered on. 2003 Unit 6. it uses a component of firmware called Run-Time Abstraction Services (RTAS) to interact with the hardware. It then continues with the task of locating an operating system image and loading it. The RTAS functions are provided by pSeries RISC Platform Architecture (RPA) platforms to insulate the operating system from having to know about and manipulate a number of key functions which ordinarily would require platform-dependent code.0. a special version is required that provides additional functionality. It identifies and configures all of the hardware components in the system.Page Table access Page tables are described later in this unit when we examine the changes in translating a virtual address to a physical address in the LPAR environment. The hypervisor provides the following functions: . Examples of RTAS functions include accessing the time-of-day clock. Most pSeries systems have two or three native serial ports. The partition specific instance contains a device tree that is a subset of the global device tree. a partition specific instance of firmware is created.V2. . In order to allow AIX to run on different hardware platform types. The hypervisor provides a © Copyright IBM Corp. When a partition is started. The OS calls these functions rather than manipulating hardware registers directly. and contains only the devices that have been assigned to the partition. reducing the need for hard-coding the OS for each platform. Hypervisor The hypervisor can be considered as a special set of RTAS features. and creates a global device tree that contains information on all devices.0. and updating the boot list in NVRAM.When the operating system is running. The hypervisor is trusted code that allows a partition to manipulate physical memory that is outside the region assigned to the partition. . since there is no need for any changes. A few other low level kernel components are aware of the fact that the OS is running inside a partition. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.Student Notebook virtual serial console interface to each partition.3) will not work in a partition. . regardless of whether it is running in a partition or running as the only operating system on a regular standalone machine. rather than maintain the table directly in memory. The net effect of the required changes is that an operating system not designed for use in a partitioned environment will fail to boot. This means that older operating systems (such as AIX 4.Debugger support The hypervisor also provides support that permits the debugger running on the system to access specific memory and register locations. The I/O from the virtual device is communicated to the HMC via the serial link from the service processor in the partitioned system. Operating system The operating system that will run in a partition needs to be modified to use hypervisor calls to manipulate the Page Frame Table (PFT). 6-12 Kernel Internals © Copyright IBM Corp. This allows the operating system to present a consistent interface to the application layer. The vast bulk of the kernel however is unaware. 3 Student Notebook Uempty Operating System Interfaces Applications Operating System AIX Boot/Config VMM Kernel Virtual Debugger TTY Dev & Dump Driver Register & Memory Access TTY Data Streams Platform Adaptation Layer (PAL) Hardware Service Calls Device Tree Subset Firmware "Partitioned" Open Firmware "Global" Open Firmware Hypervisor Virtual Page Mapping Partition Validation Run-Time Abstraction Services (RTAS) Figure 6-6. © Copyright IBM Corp. The Platform Adaptation Layer (PAL) is an operating system component similar in function to the RTAS layer provided by firmware. .0.0 Notes: Introduction The diagram summarizes the interfaces used by the operating system to interact with the hardware platform. 2003 Unit 6. Logical Partitioning 6-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. Operating System Interfaces BE0070XS4. It details the different components of the OS that interact with each function provided by the platform firmware. In other words. 2001. its job is to mask the differences between hardware platforms from other parts of the kernel.0. . 2001. 6-14 Kernel Internals © Copyright IBM Corp. and ensure that pages are mapped to physical memory when required so that they can be accessed by the processors. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. since there are now multiple operating system images co-existing in a single machine. The translation of a virtual address to a physical address is an area of the operating system that has undergone some changes to allow the implementation of a partitioned environment.Student Notebook Virtual Memory Manager Physical Memory Virtual address space Process 1 Effective address Process 2 Filesystem pages Paging space Figure 6-7.0 Notes: Introduction The job of the Virtual Memory Manager (VMM) component of the operating system is to manage the effective address space of each process on the system. Virtual Memory Manager BE0070XS4. rather than being stored in the DRAMS or other components used to implement physical memory. we first take a closer look at the memory layout on a non-LPAR system.3 Student Notebook Uempty Real Address Range Non-LPAR system with 2 PCI busses Processor View Sys Mem 1 I/O Adapters 4GB HB1 I/O Adapters HB0 Sys Mem 0 0 = Invalid for load/store Figure 6-8. 2001. Data written to the Host Bridge is passed to a specific I/O adapter card. the data is passed to the Host Bridge. © Copyright IBM Corp. The Host Bridge device allocates portions of its address space to each I/O adapter plugged into a slot it controls. A system has at least one Host Bridge (HB). which is mapped to a region in the address map. . Logical Partitioning 6-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Each HB is allocated a unique portion of the system address space.V2. based on the address being written. 2003 Unit 6. Real Address Range BE0070XS4.0. Device I/O The hardware provides memory mapped access to I/O devices.0 Notes: Introduction Before examining the changes in address translation for the LPAR environment.0. When the processor writes to specific addresses. As an example. 6-16 Kernel Internals © Copyright IBM Corp.Student Notebook Physical memory Another feature of the diagram that is worth noting is that the address range of physical memory in the system is not necessarily contiguous. the physical address range may be divided into multiple components. In other words. however depending on the total amount of memory. there appears to be ‘holes’ in the physical address range used by the system. . and the remaining part of memory using addresses 4. a system with 8GB total of physical memory may address 3GB of that memory using physical addresses in the range 0 to 3GB. 2001. This is perfectly normal. The physical memory in the system always starts with address zero. and the number of Host Bridge devices in the system. and the VMM system of AIX (and most other modern operating systems) is designed to cope with this.5GB to 9.5GB. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Address translation can be enabled or disabled. 2001. Logical Partitioning 6-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. real address = physical address On LPAR systems real address != physical address Only one physical address zero in the system Physical address zero used by hypervisor Each partition requires its own address zero Requires mapping from real address generated by partition to physical address used by memory hardware Figure 6-9.0. Typically real addresses are used by specialized parts of kernel code. such as the boot process (before the VMM is initialized) or interrupt/exception handler code. Real mode memory starts at address zero. and the status of this is indicated by bits in the MSR. Address translation for instructions and data can be enabled or disabled independently. .0. another important distinction to make is the type of access being performed. The size of real mode © Copyright IBM Corp. Real Mode Memory BE0070XS4. Real address A real address is an address that is generated by the processor when address translation is disabled.0 Notes: Introduction In addition to considering the ranges used when addressing memory.3 Student Notebook Uempty Real Mode Memory Real address = address generated when translation disabled Used by system startup code that runs before VMM is configured Used by interrupt vector mechanism Used by VMM itself to maintain tables Real mode memory normally starts at address zero Size of real mode memory region depends on operating system On non-LPAR systems. 2003 Unit 6.V2. The function of the VMM is to translate a virtual address into a real (or physical) address. each partition expects to be able to access address n. We explain things later. partition page tables are used to translate the partition specific address n into a system-wide physical address. a real address is equivalent to a physical address. just know that: . Obviously they can’t all access the same physical address n. 2001. so something needs to be done to accommodate this. but there is only one true physical address zero inside a system. 6-18 Kernel Internals © Copyright IBM Corp. but we can generalize the statement as: For any given address n. In actual fact. Each partition requires its own address zero.For real mode addresses. since a system only has a single overall physical address range (although it may be split into multiple sections). physical address zero is used by the hypervisor. LPAR changes The assertion that a real address is the same as a physical address no longer holds true in the partitioned environment however. Another important thing to note is that on a non-LPAR system.For virtual addresses. .Student Notebook memory is dependent on the requirements of the operating system. this is where the RMO register of the processor is used. . but for now. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 1 VMM accesses many tables with translation disabled Some of these tables scale with amount of memory in partition Therefore AIX 5. since it is used by the hypervisor. For example.1 Real mode Requirements Real mode region size 256MB 1GB 16GB Figure 6-10. Logical Partitioning 6-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 16GB.16GB 16GB . a 16GB real mode region must be aligned on an address boundary divisible by 16GB (i. 1GB real mode region aligned on 1GB address boundary AIX 5. © Copyright IBM Corp. As we will see later.1 real mode requirement scales with memory in partition AIX 5. 2003 Unit 6. address 0 cannot be used. 48GB. Alignment Physical memory allocated in a partitioned environment for use as Real Mode memory by a partition must be contiguous.2 & Linux require 256MB of real mode memory VMM requires fixed size real mode region Most VMM tables accessed only with address translation enabled AIX 5.0.256GB BE0070XS4.e.V2.0. .0 Notes: Introduction The amount of real mode memory required by a partition depends upon two factors.). 32GB.4GB 1GB . Operating System Real Mode Issues Supported partition sizes 256MG .g.3 Student Notebook Uempty Operating System Real Mode Issues Real mode memory aligned on same size address boundary e. 1) The version of the operating system. and aligned on an address boundary that is divisible by the size of the real mode region. 2) The amount of memory allocated to the partition. 64GB etc. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .Student Notebook AIX 5. 6-20 Kernel Internals © Copyright IBM Corp.2 and linux Both AIX 5.1 may need 256MB.2 and Linux require only 256MB of memory be accessible in real mode. In these situations.1 The VMM in AIX 5. AIX 5.2 and Linux. Sometimes the alignment requirements of the 1GB and 16GB real mode regions can cause problems on systems that are using a large percentage of their physical memory. since the VMM only uses real mode to maintain tables that do not scale with memory size. The result of this is that partitions running AIX 5. 2001. 1GB or 16GB of real mode memory. sometimes the order in which partitions are started can have an impact on whether all partitions can be started. rather than the 256MB required by AIX 5.1 maintains tables in real mode memory that scale with the total amount of memory allocated to the partition. The VMM then performs an additional step. and converts the partition-specific real address into a system-wide physical address. 2003 Unit 6. VMM converts virtual address to real address Treats address as segment ID. but because a real address is the same as a physical address. the VMM is in charge. Translation enabled When address translation is enabled. In a normal non-LPAR system.V2. Address Translation BE0070XS4. It accomplishes this using partition page tables. value of RMO register added to address All processors in the same partition have the same value in RMO register RMO value set by firmware when partition is activated Figure 6-11.0. . the VMM uses a slightly different method to convert a virtual address into a true system-wide physical address. page number and page offset Determines physical page starting address from segment ID and page number Non-LPAR systems use software PFT (page frame table) LPAR systems use partition page tables (stored outside partition) Adds page offset to physical page address Value of RMO register is not used If address translation disabled. Logical Partitioning 6-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. the VMM is effectively translating the virtual address to a real address. The VMM converts the virtual address into a real address. however the real address is a “logical” address within all of the memory assigned to the partition. 2001. © Copyright IBM Corp. In a partitioned environment. there is no problem.3 Student Notebook Uempty Address Translation If address translation enabled.0.0 Notes: Introduction The method used by a partition to interpret an address depends if virtual address translation is currently enabled or disabled. . 6-22 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Translation disabled When address translation is disabled. The processor knows when it is dealing with a real address. When dealing with a real address. 2001. the processor automatically (and without the knowledge of the operating system) adds the value loaded in the RMO to the address to convert the partition specific real address into a true system-wide physical address before submitting it to the memory controller hardware as part of the request to read or write the memory location. as indicated by the status bits in the MSR. The RML register is used to limit the amount of memory that a partition can access in real mode. the RMO (Real Memory Offset) register of the processor is used in the address calculation. 3 Student Notebook Uempty Allocating Physical Memory Memory divided into 256MB Physical Memory Blocks (PMB) Each PMB has a unique ID Multiple PMBs assigned to provide the logical address space for a partition e. The PMBs assigned to a partition need not be contiguous. .g.V2. and is associated with a PMB Some PMBs are used for special purposes.0 Notes: Introduction The physical memory of a partitioned system must be divided up between the partitions that are to be started. Each PMB has a unique ID within the system. The partition views the memory assigned to it as a number of logical memory blocks (LMBs). and cannot be allocated to partitions Partition page tables TCE space Hypervisor Figure 6-12. 2GB partition requires 8 PMBs assigned to a partition need not be contiguous Logical Memory Block (LMB) is the name given to a block of memory when viewed from the partition perspective Each LMB has a unique ID within a partition. Logical Partitioning 6-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 2003 Unit 6. Allocating Physical Memory BE0070XS4. Terminology The physical memory of the system is divided up into 256MB chunks called Physical Memory Blocks (PMBs). Each LMB has an ID that is unique within the partition. © Copyright IBM Corp. a partition will be allocated sufficient PMBs to satisfy the minimum memory requirement as indicated by the partition profile being activated.0. so that the hypervisor can track which PMBs are allocated for specific purposes.0. In order to be activated. and cannot be allocated for use by partitions. 6-24 Kernel Internals © Copyright IBM Corp. . The number of PMBs allocated for these special purposes depends upon many factors.Student Notebook Some PMBs in the system are used for special purposes. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. In other words.0 Notes: Introduction As mentioned previously.3 Student Notebook Uempty Partition Page Tables Used when translating virtual address to physical address Stored outside memory area allocated to partition Under control of Hypervisor VMM makes hypervisor call to read and update partition page table Scale with size of partition memory Four 16 byte entries per 4K page of memory assigned to partition (rounded up to power of 2) Equivalent to 1/64th of partition memory Placed in contiguous physical memory Aligned on address boundary divisible by table size e. Four 16 byte entries are required in the page table for each 4K page in the partition. Page table requirements The page table space for a partition is under the control of the hypervisor. The table is used by the VMM in the partition to translate a partition specific virtual address into a system-wide physical address. 2003 Unit 6. This equates to a size equal to 1/64th of the memory allocated to the partition. . A page table © Copyright IBM Corp. but instead must make a hypervisor call to perform the requested action. 64MB page table aligned on 64MB address Figure 6-13. Page tables are allocated in sizes that are powers of two. For example.0.V2. Logical Partitioning 6-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. each partition is allocated space for a partition page table.g. 2001. the operating system instance cannot read or write the page table entries directly. a partition with 1GB of memory requires a partition page table of 16MB in size.0. Partition Page Tables BE0070XS4. So a partition that has 2. since this means the page tables are being accessed frequently. This performance penalty is only experienced when a virtual page is mapped into physical memory. then the hypervisor will allocate more PMBs for page table use. then the VMM can perform the virtual to physical address translation by accessing the Translation Lookaside Buffer (TLB). The maximum memory amount is an attribute of the partition that is used in limiting the extent of dynamic LPAR operations.5GB of memory has a page table requirement of 40MB. The performance penalty is only really noticeable when a partition is performing heavy paging activity. Page tables must be allocated on an address boundary that is divisible by the size of the page table. 2001. page tables must be allocated in contiguous physical memory. If the virtual page is already in physical memory. the next power of two. . a processor specific cache of the most recently accessed virtual to physical translations. In addition. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 6-26 Kernel Internals © Copyright IBM Corp. but this would be rounded up to 64MB. Performance penalty There is a small performance penalty associated with the action of the VMM in a partition accessing the partition page tables. If existing PMBs allocated for page table use do not contain sufficient space (or sufficient contiguous space). The hypervisor will attempt to place multiple page tables of 128MB or smaller inside a single PMB that has been allocated for page table use.Student Notebook requirement that is not a power of two is rounded up to the next size that is a power of two. The size of the page table allocated to a partition is large enough to handle the maximum memory amount the partition may grow to. the operating system controls all host bridge devices in the system. The translation entries are used to convert the 32-bit I/O address generated by the adapter card on the I/O bus into a 64-bit address that the host bridge will submit to the system memory controller. therefore all PCI slots are controlled by a single operating system instance. 2003 Unit 6. an address in the range 0 to 4GB) to access system memory above the 4GB address range. 2001. Translation Control Entries BE0070XS4.3 Student Notebook Uempty Translation Control Entries Used to allow 32-bit PCI adapters to access 64-bit memory space Similar in concept to partition page tables. TCE tables TCE tables contain information on the current TCE mappings for each host bridge device.0. but used for device I/O Provided as a function of the PCI Host Bridge device TCE space controlled by hypervisor Outside the control of a single partition Single PCI Host Bridge may have slots in different partitions Hypervisor required for dynamic LPAR operations TCE space allocated at the top of physical memory Amount of TCE space depends on number of PCI slots/drawers 512MB for 5-8 I/O drawers on p690 256MB for all others Figure 6-14.e. In this case.0.V2.0 Notes: Introduction Host bridge devices use Translation Control Entries (TCEs) to allow a PCI adapter that can only generate a 32-bit address (i. the TCE tables exist within the memory image of the operating system. In a standalone system. . Logical Partitioning 6-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. As an example. which is a requirement for the ability to dynamically reassign an I/O slot from one partition to another with a DLPAR operation. they are placed under the control of the hypervisor. the hypervisor performs the requested action on the TCE table entry. The partition makes a hypervisor call (similar to a system call). The memory locations are not under the control of any specific partition. and after validating the permissions and arguments. . that relate to the adapter slots that have been assigned to the partition. there is no requirement for all of the slots of a single host bridge device to be under the control of a single partition. and all other LPAR capable pSeries systems use 256MB (1 PMB) for TCE space. Since the TCEs need to be manipulated by the operating system as it establishes a mapping to the adapter card. p690 systems with less than 5 I/O drawers. 6-28 Kernel Internals © Copyright IBM Corp. The hypervisor allocates each partition valid “windows” into the TCE address space. and each slot may be assigned to a different partition. 2001. Currently a p690 system that has between 5 and 8 I/O drawers will use 512MB of memory (2 PMBs) for TCE space. we now have a situation where multiple partitions require to access adjacent memory locations. Access to the TCE tables is performed by the partition in a manner similar to accessing partition page tables. Rather than having the TCE tables under the control of a special partition. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook LPAR changes In the partitioned environment. Another benefit of having the TCE space under the control of the hypervisor is that it allows the “windows” that are valid for each partition to be changed on the fly. TCE space is always located at the top of physical memory. The amount of memory allocated for TCE space depends on the number of host bridge devices (and PCI slots) in the system. a single host bridge device may support 4 PCI slots. When a user application makes a system call.V2.0 Notes: Introduction The hypervisor is the name given to code that runs under the hypervisor mode of the processor.e. Logical Partitioning 6-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .g. The transition to hypervisor mode can only be made from Supervisor State (i. 2001.0. It is loaded in the first PMB in the system.0. Making a hypervisor call from user mode results in a permission denied error. The hypervisor code is supplied as part of the firmware image loaded onto the system. and the kernel segment becomes visible. Hypervisor BE0070XS4. the processor state transitions between Problem State (user mode) and Supervisor State (kernel mode). starting at physical address zero. kernel mode).3 Student Notebook Uempty Hypervisor Similar to system call mechanism Hypervisor bit in MSR indicating processor mode Can only be invoked from Supervisor (kernel) mode Used by operating system to access memory outside the partition e. Hypervisor mode Hypervisor mode is entered using a mechanism similar to that used when a user application makes a system call. partition page tables Hypervisor code validates arguments and ensures each partition can only access its allocated page table & TCE space Checks tables of PMBs allocated to each partition Prevents a partition from accessing physical memory not assigned to the partition Figure 6-15. © Copyright IBM Corp. 2003 Unit 6. Student Notebook The HV bit in the MSR indicates if the processor is in hypervisor mode. Purpose The hypervisor is trusted code that allows a partition to manipulate memory that is outside the bounds of that allocated to the partition. The operating system must be modified for use in the LPAR environment to make use of hypervisor calls to maintain page frame tables and TCE tables that would normally be managed by the OS directly if it were running on a non-LPAR system. 6-30 Kernel Internals © Copyright IBM Corp. The hypervisor routines first validate that the calling partition is permitted to access the requested memory before performing the requested action. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . This means that the parts of the VMM used for page table management and device I/O mapping are aware of the fact that the operating system is running within a partition. 2003 Unit 6. and the PMB at the top of physical memory is allocated for TCE space.5 Physical Memory LPAR 1 M 0 RMO = N Partition page tables Hypervisor N Physical Address 0 Figure 6-16. This means the first set of PMBs allocated to the partition must be contiguous for at least 1GB. which may or may not be contiguous.1.5GB of memory allocated to the partition.0. The partition needs to run AIX 5.3 Student Notebook Uempty Dividing Physical Memory TCE space 2 LPAR 2 0 RMO = M 4. 2001. Dividing Physical Memory BE0070XS4. The remaining 3.0 Notes: Introduction The diagram above shows a sample system that has two active partitions. © Copyright IBM Corp. . which must be contiguous. and aligned on a 1GB address boundary. LPAR 1 LPAR 1 has 4. The first PMB is allocated to the hypervisor. Logical Partitioning 6-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.5GB allocated to the partition consists of 14 PMBs. so this means it has a real mode memory requirement of 1GB.0.V2. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. LPAR 2 is permitted to access the portions of TCE space required for mapping the I/O slots assigned to the partition. This is already a power of 2. it effectively means that a partition running AIX 5. The algorithms used by the firmware to allocate PMBs try to make best use of those available.5GB of memory has a page table requirement of 72MB. and are careful to avoid encroaching on a 16GB aligned 16GB contiguous group of PMBs if it can be avoided. since these are required for AIX 5. This means the 32MB page table for LPAR 2 shares the same PMB as the 128MB page table for LPAR 1. the firmware allocates a page table of 32MB. The allocated PMBs need not be contiguous. and so at partition activation time. there was one PMB allocated for page tables. Typical example The example shown in the diagram shows a situation where multiple partitions may have been activated and then terminated.2. Since this is the same size as the PMB.Student Notebook A partition with 4. which will be rounded up to 128MB.1 partitions that are 16GB or larger in size. In this example. It is running AIX 5. 6-32 Kernel Internals © Copyright IBM Corp. resulting in the seemingly sparse allocation of PMBs. It only allocates a new PMB for page tables if free space inside a PMB already being used for page tables cannot be found. . LPAR 2 LPAR 2 has 2GB of memory assigned. A partition with 2GB of memory (and an attribute of a maximum of 2GB) requires a page table of 32MB. If this is the first partition to be activated. however the system firmware will allocate them in a contiguous fashion where possible. and only 128MB was being used. the page table will be placed in a PMB that is marked by the hypervisor for use as page table storage. so is quite happy with just 256MB of real mode memory.2 can consist of the required number of PMBs to satisfy the requested memory amount. The partition will be permitted to access the portions of TCE space that are used to map the I/O slots that are assigned to the partition. 2001. True or False? 6) Which physical addresses in the system can a partition access? Figure 6-17. Checkpoint BE0070XS4.0. True or False? 5) Any piece of code can make hypervisor calls.V2. Logical Partitioning 6-33 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction Answer all of the questions above. 3) All partitions have the same real mode memory requirements. © Copyright IBM Corp. True or False? 4) In a partitioned environment. We will review them as a group when everyone has finished.0. a real address is the same as a physical address. 2003 Unit 6. .3 Student Notebook Uempty Checkpoint 1) What processor features are required in a partitioned system? 2) Memory is allocated to partitions in units of __________MB. . Unit Summary BE0070XS4. IO slots) are allocated to partitions independently of one another A partition can receive as much (or as little) of each resource as it needs Multiple partitions on a single machine imply changes to the addressing mechanism used by the operating system Can't have all partitions using the same physical address range Hypervisor is special code called by the operating system that allows it to modify memory outside the partitions Figure 6-18. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.Student Notebook Unit Summary Hardware and software (operating system) changes are required for LPAR Can't run LPAR on just any system Can't use just any OS inside a partition Resources (CPU.0 Notes: 6-34 Kernel Internals © Copyright IBM Corp. memory. LFS. How You Will Check Your Progress Accountability: • Exercises using your lab system • Unit review References AIX Documentation: Kernel Extensions and Device Support Programming Concepts © Copyright IBM Corp. 2001. logical and physical volumes. Identify the kdb subcommands for displaying these structures. VFS and LVM 7-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. VFS and LVM What This Unit Is About This unit describes the organization and operation of the logical and virtual file system. you should be able to: • List the design objectives of the logical and virtual file systems. LFS. locate the file and the file system the descriptor represents. and the LVM structures used by the kernel.3 Student Notebook Uempty Unit 7. • Identify the data structures that make up the logical and virtual file systems. • Identify the basic kernel structures for tracking LVM volume groups. 2003 Unit 7.0. • Use kdb to identify the data structures representing a mounted file system.V2. • Use kdb to identify the data structures representing an open file. .0. • Given a file descriptor of a running process. What You Should Be Able to Do After completing this unit. . Given a file descriptor of a running process. Unit Objectives BE0070XS4. logical and physical volumes. Identify the basic kernel structures for tracking LVM volume groups. Identify the kdb subcommands for displaying these structures.0 Notes: 7-2 Kernel Internals © Copyright IBM Corp. Use kdb to identify the data structures representing a mounted file system. 2001. Identify the data structures that make up the logical and virtual file systems. locate the file and the file system the descriptor represents.Student Notebook Unit Objectives At the end of this lesson you should be able to: List the design objectives of the logical and virtual file systems. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Use kdb to identify the data structures representing an open file. Figure 7-1. ..Network File System (NFS) © Copyright IBM Corp. services and data structures that are provided by the Logical File System (LFS) and the Virtual File System (VFS). special files.Enhanced Journaled File System (JFS2) .0 Notes: Introduction This unit covers the interface. What is the Purpose of LFS/VFS? BE0070XS4. VFS and LVM 7-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. Supported file systems Using the structure of the logical file system and the virtual file system. sockets. LFS. 2003 Unit 7.3 Student Notebook Uempty What is the Purpose of LFS/VFS? Provide support for many different file systems types simultaneously Allow for different types of file systems to be mounted together forming a single homogenous view Provide a consistent user interface to all file type objects (regular files.Journaled File System (JFS) .. 2001. AIX 5L can support a number of different file system types that are transparent to application programs.0.0. The following physical file system implementations are currently supported: . These file systems reside below the LFS/VFS layer and operate relatively independently of each other.) Support the sharing of files over the network Provide an extensible framework allowing third party file system types to be added into AIX Figure 7-2. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. High Sierra and Rock Ridge formats Extensible The LFS/VFS interface also provides a relatively easy means by which third party file system types can be added without any changes to the LFS.A CD-ROM file system. which supports ISO-9660. .Student Notebook . 7-4 Kernel Internals © Copyright IBM Corp. © Copyright IBM Corp. . 2003 Unit 7.0. Kernel I/O Layers BE0070XS4. LFS. write() Logical File System Virtual File System File System Implementation JFS.0. Hierarchy Access to files and directories by a process is controlled by the various layers in the AIX 5L kernel. as illustrated above.0 Notes: Introduction Several layers of the AIX kernel are involved in the support of file systems I/O as described in this section. 2001.3 Student Notebook Uempty Kernel I/O Layers System call interface read(). JFS2 VMM Fault handler VMM LVM Device Driver Device Figure 7-3.V2. VFS and LVM 7-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. It is invoked by the page fault handler. Different physical file systems can handle the request (JFS.Student Notebook Layers The layers involved in file I/O are described in this table: Level System call interface Logical file system Virtual file system Purpose A user application can access files using the standard interface of the read() and write() system calls. I/O to a file causes a page fault and is resolved by the VMM fault handler. The system call interface is supported in the LFS with a standard set of operations. Files are mapped to virtual memory. 2001. The VFS defines a generic set of operation that can be performed on a file system. The file system type is invisible to the user. NFS). Device driver code to interface with the device. File system VMM fault handler Device drivers 7-6 Kernel Internals © Copyright IBM Corp. The LVM is the device driver for JFS2 and JFS. . JFS2. 2001.0. close(). . such as open().0.V2. The system calls implement services that are exported to users to provide a consistent user-mode programming interface that is independent of the underlying file system type. Major Data Structures BE0070XS4. read() and write(). Logical file system The LFS is the level of the file system at which users can request file operations by using system calls. 2003 Unit 7.3 Student Notebook Uempty Major Data Structures u-block inode vnode gnode User File Descriptor Table System File Table vnodeops vfs gfs vmount vfsops Logical File System Virtual File System (Vnode-VFS Interface) File System Figure 7-4. © Copyright IBM Corp. The illustration is repeated throughout the unit highlighting the areas being discussed. VFS and LVM 7-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. LFS.0 Notes: Introduction This illustration shows the major data structures that will be discussed in this unit. 7-8 Kernel Internals © Copyright IBM Corp. File system Each file system type extension provides functions to perform operations on the file system and its files. . NFS or CD-ROM).Student Notebook Virtual files system The Virtual File System (VFS) defines a standard set of operations on an entire file system. Pointers to these functions are stored in the vfsops (file system operations) and vnodeops (file operations) structures. In this way. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. J2. 2001. Operations performed by a process on a file or file system are mapped through the VFS to the file system below. the process need not know the specifics of different file systems (such as JFS. 0.V2. Logical File System Structures BE0070XS4.0. LFS. 2003 Unit 7. The system open file table has entries for open files on the system. . 2001. VFS and LVM 7-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction The user file descriptor table and the system file table are the key data bases used by the LFS. These memory structures and their relationship to vnodes are discussed in this section.3 Student Notebook Uempty Logical File System Structures f_data u-block fp vnode read(0) 0 1 User File Descriptor Table vnode System File Table n=open("file") n vnode Process private Global One per file Figure 7-5. Each entry in the system file table points to a vnode in the virtual file system. Structures in the LFS The user file descriptor table (one per process) contains entries for each of the process’ open files. © Copyright IBM Corp. a vnode for that object is created. The vnode will be covered in more detail later. 2001. When a process opens a file. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook User file descriptor table The user file descriptor table is private to a process and located in the processes u-area. Each time an object is located. an entry is created in the user’s file descriptor table. device. or socket in the system. vnode The vnode provides the connection between the LFS and the VFS. . If multiple processes have the same file open (or one process has opened the file several times) a separate entry exists in the table for each unique open. The index of the entry in the table is returned to open() as a file descriptor. One unique entry is allocated for each unique open of a file. It is the primary structure the kernel uses to reference files. System open file table The system file table is a global resource and is shared by all processes on the system. 7-10 Kernel Internals © Copyright IBM Corp. VFS and LVM 7-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Table management One or more slots of the file descriptor table are used for each open file.0. Descriptor table definition The user file descriptor table consists of an array of user file descriptors as defined in /usr/include/sys/user. The file descriptor table can extend beyond the first page of the u-block. unsigned short count.0 Notes: Introduction The user file descriptor table is private to a process and located in the process’ u-area.h in the structure ufd.3 Student Notebook Uempty User File Descriptor struct ufd { struct file * fp. #ifdef __64BIT_KERNEL unsigned int reserved. When a process opens a file. unsigned short flags. User File Descriptor BE0070XS4. . an entry is created in the users file descriptor table.0. #endif /* __64BIT_KERNEL */ }.V2. 2003 Unit 7. The index of the entry in the table is returned to open()as a file descriptor. LFS. Figure 7-6. and is pageable. There © Copyright IBM Corp. 2001. This value is fixed. and may not be changed. .Student Notebook is a fixed upper limit of 65534 open file descriptors per process (defined as OPEN_MAX in /usr/include/sys/limits. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 7-12 Kernel Internals © Copyright IBM Corp.h). 0 Notes: Introduction The system file table is a global resource and is shared by all processes on the system. © Copyright IBM Corp.V2. or socket in the system.000. they are added back onto the free list. . . It grows on demand and is never shrunk. Structure definition The file structure is described in /usr/include/sys/file. VFS and LVM 7-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The head of the free list is pointed to by ffreelist. 2001. Simple_lock f_lock.000 entries and is not configurable. Once entries are freed.0. device. LFS. /* next entry in freelist */ } f_up. /* read/write character pointer */ off_t f_dir_off. /* pointer to vnode structure */ struct file *f_unext. Table management The system file table is a large array of file structures. /* see fcntl. 2003 Unit 7.h */ int f_count. . One entry is allocated for each unique open of a file. /* next quick move chunk on free list*/ } f_cp. The table can contain a maximum of 1. /* file flags not passed through vnode layer */ short f_type. In the visual above the fileops definitions for __FULL_PROTO have been omitted for clarity. /* file structure offset field lock */ caddr_t f_vinfo. /* reference count */ short f_options. The array is partly initialized. Figure 7-7. int (*fo_select)(). offset_t f_offset. /* descriptor type */ union { struct vnode *f_uvnode.0. #else int (*fo_rw)().3 Student Notebook Uempty The File Structure struct file { long f_flag. }. /* BSD style directory offsets */ union { struct ucred *f_cpcred.h. int (*fo_close)(). The file Structure BE0070XS4. int (*fo_fstat)(). /* any info vfs needs */ struct fileops { . int (*fo_ioctl)(). /* file structure fields lock */ Simple_lock f_offset_lock. #endif /* __64BIT_KERNEL || __FULL_PROTO */ } *f_ops. /* process credentials at open() */ struct file *f_cpqmnext. Defined as f_up. Several of the key members of this data structure are described in this table: Member Description A reference count field detailing the current number of opens on the file. it is a pointer to another data structure representing the object (typically the vnode structure). Once the reference count is zero.Student Notebook Table entries The file table array consists of struct file data elements. the slot is considered free. and may be re-used. . ioctl. This value is increased each time the file is opened. and decremented on each close(). A structure containing pointers to functions for the following file operations: rw (read/write).h A type field describing the type of file: f_type /* f_type values */ #define DTYPE_VNODE #define DTYPE_SOCKET endpoint */ #define DTYPE_GNODE #define DTYPE_OTHER 1 2 3 -1 /* file */ /* communications /* device */ /* unknown */ f_count f_flag f_offset f_data A read/write pointer. f_ops 7-14 Kernel Internals © Copyright IBM Corp. Various flags described in fcntl. close and fstat. select. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.f_uvnode. V2. 2001. 2003 Unit 7.3 Student Notebook Uempty vnode/vfs Interface u-block inode vnode gnode User File Descriptor Table System File Table vnodeops vfs gfs vmount vfsops Logical File System Virtual File System (Vnode-VFS Interface) File System Figure 7-8. vfs and vmount structures are given in this table: © Copyright IBM Corp. . VFS and LVM 7-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. vnode/vfs Interface BE0070XS4. Description Descriptions of the vnode.0.0 Notes: Introduction The interface between the logical file system and the underlying file system implementations is referred to as the vnode/vfs interface. LFS. and the file system specific objects that the underlying file system implementation must manage. This interface provides a logical boundary between generic objects understood at the LFS layer. Data structures vnodes and vfs structures are the primary data structures used to communicate through the interface (with help from vmount). Student Notebook Part vnodes vfs vmount Function Represents a single file or directory Represents a mounted file system Contains specifics of the mount request 7-16 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty vnode struct vnode { ushort ulong32int64 int Simple_lock struct vfs struct vfs struct gnode struct vnode struct vnode struct vnode union v_data { void * struct vnode * } _v_data; char * */ }; v_flag; v_count; v_vfsgen; v_lock; *v_vfsp; *v_mvfsp; *v_gnode; *v_next; *v_vfsnext; *v_vfsprev; _v_socket; _v_pfsvnode; v_audit; /* the use count of this vnode */ /* generation number for the vfs */ /* lock on the structure */ /* pointer to the vfs of this vnode */ /* pointer to vfs which was mounted over /* this vnode; NULL if not mounted */ /* ptr to implementation gnode */ /* ptr to other vnodes that share same gnode */ /* ptr to next vnode on list off of vfs /* ptr to prev vnode on list off of vfs /* vnode associated data */ /* vnode in pfs for spec */ /* ptr to audit object Figure 7-9. vnode BE0070XS4.0 Notes: Introduction A vnode represents an active file or directory in the kernel. Each time a file is located, a vnode for that object is located or created. Several vnodes may be created as a result of path resolution. Structure definition The vnode structure is defined in /usr/include/sys/vnode.h. vnode management vnodes are created by the vfs-specific code when needed, using the vn_get kernel service. vnodes are deleted with the vn_free kernel service. vnodes are created as the result of a path resolution. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Detail Each time an object (file) within a file system is located (even if it is not opened), a vnode for that object is located (if already in existence), or created, as are the vnodes for any directory that has to be searched to resolve the path to the object. As a file is created, a vnode is also created, and will be re-used for every subsequent reference made to the file by a path name. Every path name known to the logical file system can be associated with, at most, one file system object, and each file system object can have several names because it can be mounted in different locations. Symbolic links and hard links to an object always get the same vnode if accessed through the same mount point. 7-18 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty vfs struct vfs { struct vfs struct gfs struct vnod struct vnode struct vnode int caddr_t unsigned int int #ifdef _SUN short unsigned short #else short unsigned short #endif /* _SUN */ struct vmount Simple_lock }; /* vfs's are a linked list */ /* ptr to gfs of vfs */ /* pointer to mounted vnode, */ /* the root of this vfs */ *vfs_mntdover; /* pointer to mounted-over */ /* vnode */ *vfs_vnodes; /* all vnodes in this vfs */ vfs_count; /* number of users of this vfs */ vfs_data; /* private data area pointer */ vfs_number; /* serial number to help distinguish between */ /* different mounts of the same object */ vfs_bsize; /* native block size */ vfs_exflags; vfs_exroot; vfs_rsvd1; vfs_rsvd2; *vfs_mdata; vfs_lock; /* for SUN, exported fs flags */ /* for SUN, " fs uid 0 mapping */ /* Reserved */ /* Reserved */ /* record of mount arguments */ /* lock to serialize vnode list */ *vfs_next *vfs_gfs; *vfs_mntd; Figure 7-10. vfs BE0070XS4.0 Notes: Introduction There is one vfs structure for each file system currently mounted. The vfs structure connects the vnodes with the vmount information, and the gfs structure that help define the operations that can be performed on the file system and its files. Structure definition The vfs structure is defined in /usr/include/sys/vfs.h. Key elements Several key elements of the vfs structure are described in this table: Element *vfs_next Description The next mounted file system. Unit 7. LFS, VFS and LVM 7-19 © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Element vfs_mntd Description The vfs_mntd pointer points to the vnode within the file system which generally represents the root directory of the file system. The vfs_mntdover pointer points to a vnode within another file system, usually represents a directory, which indicates where the file system is mounted. The pointer to all vnodes for this file system. The path back to the gfs structure and its file system specific subroutines through the vfs_gfs pointer. The pointer to vmount providing mount information for this file system vfs_mntdover vfs_vnodes *vfs_gfs vfs_mdata 7-20 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty root (/) and usr File Systems 1 2 vfs_next rootvfs vfs for root file system vfs_mntd 2 v_mvfsp vfs_mntdover vfs for usr file system vfs_mntd vfs_mntdover v_vfsp v_vfsp 4 3 3 v_vfsp vnode for /usr Null vnode for / vnode for root of usr root file system usr file system Figure 7-11. root (l) and usr File Systems BE0070XS4.0 Notes: Relationship between vfs and vnodes This illustration shows the relationship between the vfs and vnode objects for mounted file systems. This example shows the root (/) and usr file systems. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Description The numbered items in the table match the number in the illustration. Item 1. 2. 3. 4. Description The global address rootvfs points to the vfs for the root file system The vfs_next pointers create a linked list of mounted file systems The vfs_mntd points to the vnode representing the root of the file system The vfs_mntdover points to the vnode of the directory the file system is mounted over 7-22 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty vmount struct vmount { uint vmt_revision; uint vmt_length; fsid_t vmt_fsid; int vmt_vfsnumber; uint vmt_time; uint vmt_timepad; int vmt_flags; int vmt_gfstype; struct vmt_data { short vmt_off; /* I offset of data, word aligned short vmt_size; /* I actual size of data in bytes } vmt_data[VMT_LASTINDEX + 1]; }; /* I revision level, currently 1 */ /* I total length of structure & data */ /* O id of file system */ /* O unique mount id of file system */ /* O time of mount */ /* O (in future, time is 2 longs) */ /* I general mount flags */ /* O MNT_REMOTE is output only */ /* I type of gfs, see MNT_XXX above */ */ */ Figure 7-12. vmount BE0070XS4.0 Notes: Introduction The vmount structure contains specifics of the mount request. The vfs and vmount are created as pairs and linked together. Structure definition The vmount structure is defined in /usr/include/sys/vmount.h. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The vmount subroutine then creates the vfs structure. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and invokes the file system dependent vfs_mount subroutine. . which completes the vfs structure and performs any operations required internally by the particular file system implementation. partially populates it.Student Notebook vfs management The mount helper creates the vmount structure and calls the vmount subroutine. 2001. 7-24 Kernel Internals © Copyright IBM Corp. 2001. Data structures For each file system type installed. LFS. VFS and LVM 7-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .0. © Copyright IBM Corp.3 Student Notebook Uempty File and File System Operations u-block inode vnode gnode User File Descriptor Table System File Table vnodeops vfs gfs vmount vfsops Logical File System Virtual File System (Vnode-VFS Interface) File System Figure 7-13.0 Notes: Introduction Each file system type extension provides functions to perform operations on the file system and its files. one group of these three data structures shown above will be created. Pointers to these functions are stored in the vfsops (file system operations) and vnodeops (file operations) structures. File and File System Operations BE0070XS4. 2003 Unit 7.0.V2. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.) 7-26 Kernel Internals © Copyright IBM Corp. .) Contains pointers to file system dependent operations on the file system (mount. etc. close. and vfsops are given in this table: Part gfs vnodeops vfsops Function Holds pointers to the vnodeops and the vfsops structures Contains pointers to file system dependent operations on files (open.Student Notebook Structure descriptions Descriptions of gfs. write. read. umount. etc. 2001. vnodeops. LFS. gfs BE0070XS4.3 Student Notebook Uempty gfs ops gn_ vnodeops vfs vfs_gfs gfs gfs _op s vfsops Figure 7-14. 2001. .V2. 2003 Unit 7.0.0. VFS and LVM 7-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction gfs is used as a pointer to the vnodevops and the vfsops structures. © Copyright IBM Corp. 7-28 Kernel Internals © Copyright IBM Corp. and only one gfs entry of a given gfs_type can be inserted into the array."nfs")*/ int (*gfs_init)(). This is usually done within the CFG_TERM section of the configuration code of the file system kernel extension. /* name of vfs (eg. /* ( gfsp ) . int gfs_type. "jfs".Student Notebook Structure definition The gfs structure is defined in /usr/include/sys/gfs. The gfs entries are removed with the gfsdel() kernel service. The gfs entries are inserted with the gfsadd() kernel service. /* gfs private config data*/ int (*gfs_rinit)(). struct vnodeops *gn_ops. Generally.h: struct gfs { struct vfsops *gfs_ops. . */ /* called once to init gfs */ int gfs_flags. int gfs_hold /* count of mounts */ } gfs management The gfs structures are stored within a global array accessible only by the kernel. 2001.if ! NULL. gfs entries are added by the CFG_INIT section of the configuration code of the file system kernel extension. /* type of gfs (from vmount.h) */ char gfs_name[16]. /* flags for gfs capabilities */ caddr_t gfs_data. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3 Student Notebook Uempty vnodeops vn_link() vn_mkdir() vn_open() vn_close() vfs vfs_gfs gfs gn_ops vnodeops vn_remove() vn_rmdir() vn_lookup() Figure 7-15.0 Notes: vnodeops The vnodeops structure contains pointers to the file system dependent operations that can be performed on the vnode. VFS and LVM 7-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. close and remove. . 2001.0. open. 2003 Unit 7. LFS. vnodeops BE0070XS4. such as link. mkdir.V2.0. © Copyright IBM Corp. mknod. struct ucred *). int32long64_t. struct vnode *. char *. int (*vn_mkdir)(struct vnode *. struct ucred *). int (*vn_mknod)(struct vnode *.Student Notebook Structure definition The vnodeops structure is defined in /usr/include/sys/vnode. 2001. Due to the size of this structure. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. int (*vn_rename)(struct vnode *.caddr_t. int (*vn_remove)(struct vnode *.struct ucred *). struct ucred *). struct vnode *. struct vnode *. char *. struct ucred *). int32long64_t. char *. dev_t. only a few lines are detailed below: struct vnodeops { /* creation/naming/deletion */ int (*vn_link)(struct vnode *. caddr_t.struct vnode *. caddr_t.h. struct vnode *. . 7-30 Kernel Internals © Copyright IBM Corp. 2003 Unit 7.0. unmount. . vfsops BE0070XS4. or sync.V2. LFS.0 Notes: vfsops The vfsops structure contains pointers to the file system dependent operations that can be performed on the vfs. 2001. © Copyright IBM Corp.0. VFS and LVM 7-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty vfsops vfs_mount() vfs_unmount() vfs_root() vfs_sync) vfs vfs_gfs gfs gfs_ops vfsops vfs_vget() vfs_cntl() vfs_quotactl() Figure 7-16. such as mount. }.h: struct vfsops { /* mount a file system */ int (*vfs_mount)(struct vfs *. /* get file system information */ int (*vfs_statfs)(struct vfs *. struct ucred *). int. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. struct ucred *). 2001. struct ucred *). int. struct fileid *. /* get the root vnode of a file system */ int (*vfs_root)(struct vfs *. uid_t. struct ucred *). /* get a vnode matching a file id */ int (*vfs_vget)(struct vfs *. caddr_t. . struct statfs *. size_t. /* do specified command to file system */ int (*vfs_cntl)(struct vfs *. caddr_t. struct vnode **. /* manage file system quotas */ int (*vfs_quotactl)(struct vfs *.Student Notebook Structure definition The vfsops structure is defined in /usr/include/sys/vfs. struct ucred *). 7-32 Kernel Internals © Copyright IBM Corp. struct ucred *). /* sync all file systems of this type */ int (*vfs_sync)(). struct vnode **. int. struct ucred *). /* unmount a file system */ int (*vfs_unmount)(struct vfs *. V2. BE0070XS4. NFS files have gnodes contained within rnodes. . gnode . 2001. LFS.3 Student Notebook Uempty gnode in-core inode vnode v_gnode gnode specnode vnode v_gnode gnode rnode vnode v_gnode gnode Figure 7-17. Location The gnode is contained in an in-core-inode for a file on a local file system. /* type of object: VDIR. have gnodes contained in specnodes.0 Notes: Introduction gnodes are generic objects pointed to by vnodes but may be contained in different structures depending on the file system type.0.0. VFS and LVM 7-33 © Copyright IBM Corp. Structure definition The gnode structure is defined in /usr/include/sys/vnode.h: struct gnode { enum vtype gn_type. Special files (such as /dev/tty). 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.VREG etc */ Unit 7. Thus the gnode structure substitutes for whatever structure the file system implementation may have used to uniquely identify a file system object. A gnode is needed. Points to the start of the inode the gnode is imbedded Detail Each file system implementation is responsible for allocating and destroying gnodes. /* attributes of object */ ulong gn_seg. This is normally immediately followed by a call to the vn_get kernel service to create a matching vnode. Calls to the file system implementation serve as requests to perform an operation on a specific gnode. /* count of map for write */ long32int64 gn_mrdcnt. 2001. their "chan". Identifies the set of operations that can be performed on the object Segment number to which the file is mapped Pointer to private data. /* count of map for read */ long32int64 gn_rdcnt. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and block. /* total opens for write */ long32int64 gn_excnt. struct vnode *gn_vnode. /* for devices. 7-34 Kernel Internals © Copyright IBM Corp. as needed by file system specific code at the same time as implementation specific structures are created./* event list for file locking */ struct filock *gn_filocks. Some examples are directory. /* ptr to private data (usually contiguous) } Key elements Some of the key elements of the gnode are described below: Element gn_type gn_ops gn_seg gn_data Description Identifies the type of object represented by the gnode. their "dev_t" */ chan_t gn_chan. or when the implementation specific structure is being reused for another file. /* total opens for exec */ long32int64 gn_rshcnt. /* lock for filocks list */ int gn_reclk_event. /* ptr to list of vnodes per this gnode*/ dev_t gn_rdev. /* locked region list */ caddr_t gn_data. /* total opens for read */ long32int64 gn_wrcnt.Student Notebook short gn_flags. because some file system implementations may not include the concept of an inode. character. /* total opens for read share */ struct vnodeops *gn_ops. gnodes are created. in addition to the file system inode. The gnode structure is usually deleted either when the file it refers to is deleted. . minor’s minor*/ Simple_lock gn_reclk_lock. /* segment into which file is mapped */ long32int64 gn_mwrcnt. /* for devices. 0. This architecture is discussed in other classes. 2003 Unit 7. The structures are volgrp. and the kdb commands that display these structures. The kdb subcommands to display these structures have corresponding names: volgrp. Here we would like to introduce three kernel structures which maintain LVM data. All definitions are from src/bos/kernel/sys/dasd.nodev (000E12EC) config: 020795E8 print: .0. LFS. .h unless otherwise noted. In the above visual we will illustrate the structure definitions with example output from the kdb subcommands and corresponding AIX commands.V2.nodev (000E12EC) dump: 020A7530 mpx: . kdb devsw Subcommand Output BE0070XS4. VFS and LVM 7-35 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. which is not distributed with the AIX product).h.0 Notes: Introduction The file systems discussed earlier in this unit are contained within Logical Volume Manager (LVM) Logical Volumes. 2001. lvol and pvol. lvol and pvol (defined in src/bos/kernel/sys/dasd.nodev (000E12EC) dsdptr: selptr: opts: (0)> 310E3000 DEV_DEFINED DEV_MPSAFE 00000000 0000002A Figure 7-18. The data defining LVM entities (including Volume Groups. © Copyright IBM Corp.3 Student Notebook Uempty kdb devsw Subcommand Output (0)> devsw 0xa Slot address 30057280 MAJOR: 00A open: 0207DC40 close: 0207D694 read: 0207CDC0 write: 0207CCF4 ioctl: 0207B4DC strategy: 02095914 ttys: 00000000 select: . Logical Volumes and Physical Volumes) is maintained both on disks and in the ODM.nodev (000E12EC) revoke: . . ./* physical volume struct array */ . /* pointer to next volgrp structure */ .open_count This is the count of active logical volumes in this volume group. .*nextvg This is the volgrp linked list item. . . This table is displayed with the kdb subcommand.vg_id This is the 32 character volume id./* logical volume struct array*/ struct pvol*pvols[NEW_MAXPVS]. devsw.*lvols[NEW_MAXLVS] points to the array of lvol structures for this volume group. 7-36 Kernel Internals © Copyright IBM Corp. }. The array is indexed by logical volume minor number. A value of zero means this is the last or only volume group. At this point we introduce it only to obtain a volgrp address. The kernel describes this in the volgrp structure.Student Notebook volgrp structure The administrative unit of LVM is a volume group./* volume group id */ intmajor_num. . the major number of rootvg on this system. . short . ./* count of open logical volumes */ . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM./* lock for all vg structures struct unique_idvg_id. struct volgrp *nextvg. The Items in bold are defined as: . In the command output. We the devsw subcommand with a single parameter 0xa./* major number of volume group . . . the dsdptr: field is the address of rootvg’s volgrp structure. struct lvol*lvols[NEW_MAXLVS]. Portions of the structure definition follows: struct volgrp { Simple_lockvg_lock. 2001. volgrp kdb subcommand volgrp addresses are registered in the devsw table. . The array is indexed by physical volume minor number.*pvols[NEW_MAXPVS] points to the array of pvol structures for this volume group. . */ */ open_count. V2.0.0.3 Student Notebook Uempty kdb volgrp Subcommand Output (0)> volgrp 310e3000 VOLGRP............. 310E3000 . . . lvols............... @ 310E302C pvols............... @ 310E382C major_num............. 0000000A vg_id................. 0001D2CA00004C00000000F11C1697A0 nextvg................ 00000000 opn_pin............. @ 310E3A2C . . . sa_hld_lst............ 00000000 vgsa_ptr.............. 31107000 config_wait........... FFFFFFFF sa_lbuf............. @ 310E3B10 sa_pbuf............. @ 310E3B68 . . . LVOL[007]....... 31108180 work_Q.......... 3110BE00 lv_status....... 00000002 lv_options...... 00001000 nparts.......... 00000001 i_sched......... 00000000 nblocks......... 00010000 parts[0]........ 31108300 pvol@ 310E4600 dev 00190000 start 00DE1100 parts[1]........ 00000000 parts[2]........ 00000000 . . . LVOL[009]....... 31108380 . . . Figure 7-19. kdb volgrp Subcommand Output BE0070XS4.0 Notes: Output of volgrp command The visual above shows partial output of the kdb subcommand, volgrp, using the address just obtained with devsw. The volgrp subcommand formats volgrp structure data in a helpful way. The pointer values for pvol and lvol arrays are provided (“pvols” and “lvols”), but in addition the subcommand formats each lvols array entry. So we see an “LVOL” entry for each logical volume in our rootvg. In the example above there were 10 entries for lvols array data. We have shown only the entry for minor device number 7 which is the entry for hd3, the /tmp file system. We will examine this logical volume with other commands, and describe the bold items at that time. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-37 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Other items above in bold give: - major_num = 0xA means this is the rootvg volume group. - The vg_id value is rootvg’s volume group id. *nextvg=0 means this volume group is the last or only one on the volgrp linked list. 7-38 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty AIX lsvg Subcommand Output # lsvg rootvg VOLUME GROUP: rootvg VG IDENTIFIER: 0001d2ca00004c00000000f11c1697a0 VG STATE: active PP SIZE: 32 megabyte(s) VG PERMISSION: read/write TOTAL PPs: 542 (17344 megabytes) MAX LVs: 256 FREE PPs: 497 (15904 megabytes) LVs: 9 USED PPs: 45 (1440 megabytes) OPEN LVs: 8 QUORUM: 2 TOTAL PVs: 1 VG DESCRIPTORS: 2 STALE PVs: 0 STALE PPs: 0 ACTIVE PVs: 1 AUTO ON: yes MAX PPs per PV: 1016 MAX PVs: 32 LTG size: 128 kilobyte(s) AUTO SYNC: no HOT SPARE: no BB POLICY: relocatable # Figure 7-20. AIX lsvg Command Output BE0070XS4.0 Notes: AIX lsvg command view of the same data Now that we have seen the kernel’s view of rootvg data, it is interesting to look at what our command line interface shows. The lsvg command provides a summary of volume group information. This visual shows lsvg output for the same rootvg that we just examined with volgrp. The items in bold print above correspond to kdb volgrp items described on the prior slide: “VOLUME GROUP: rootvg” corresponds to major_num=0xA “VG IDENTIFIER” corresponds to the vg_id value. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-39 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook kdb lvol Subcommand Output (0)> lvol 31108180 LVOL............ 31108180 work_Q.......... 3110BE00 lv_status....... 00000002 lv_options...... 00001000 nparts.. 00000001 i_sched......... 00000000 nblocks......... 00010000 parts[0]..31108300 pvol@ 310E4600 dev 00190000 start 00DE1100 parts[1]........ 00000000 parts[2]........ 00000000 maxsize......... 00000000 tot_rds......... 00000000 complcnt........ 00000000 waitlist........ FFFFFFFF stripe_exp...... 00000000 striping_width.. 00000000 lvol_intlock. @ 311081BC lvol_intlock.... 00000000 (0)> Figure 7-21. kdb lvol Subcommand Output BE0070XS4.0 Notes: 7-40 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty The lvol structure Each active logical volume is represented by an lvol structure. The lvol structure is defined as follows: struct lvol { struct buf short ushort short **work_Q; /*work in progress hash table */ lv_status; /*lv status:closed,closing,open */ lv_options;/*logical dev options (see below)*/ nparts; /* num of part structures for this*/ /* lv - base 1 char char ulong struct part int ulong int i_sched; lv_avoid; nblocks; */ */ */ /* initial scheduler policy state /* online backup mask indicator /* LV length in blocks */ */ */ *parts[3]; /*partition arrays for each mirror*/ maxsize; tot_rds; /* max number of pp allowed in lv /* total number of reads to LV parent_minor_num;/*if this is an online backup copy*/ /*this is the minor number of the ’real’*/ /* or ’parent’ logical volume */ /* These fields of the lvol structure are read and/or written by * the bottom half of the LVDD; and therefore must be carefully * modified. */ int tid_t struct file unsigned int unsigned int Simple_lock uchar struct io_stat unsigned int unsigned int }; complcnt; waitlist; *fp; * completion count-used to quiesce */ /* event list for quiesce of LV */ /*file ptr for lv mir bkp open/close */ stripe_exp; /* 2**stripe_block_exp = stripe */ /* lvol_intlock; lv_behavior;/* special conditions lv may be under */ *io_stats[3];/* collect io statistics here */ syncing; blocked; /* Count of SYNC requests */ /* Count of blocked requests */ block size */ striping_width; /* number of disks striped across */ © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-41 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Items shown in bold: lv_status: 0=> closed, 1=> trying to close, 2=> open, 3=> being deleted lv_options is a flag word. Some of the flags are: 0x0001=>write verify, 0x0020=>read-only, 0x0040=>dump in progress to this logical volume, 0x0080=>this logical volume is a dump device, 0x1000=>original default (not passive) mwcc (mirror write consistency check) on. nparts: Number of copies (1=>no mirror, 2=>single mirror, 3=>two mirrors). This gives the number of *parts array elements that are meaningful. i_sched: Scheduling policy for this logical volume values include: 0=>regular, non-mirrored LV, 1=>sequential write, sequential read, 2=>parallel write, read closest, 3=>sequential write, read closest, 4=> parallel write, sequential read, 5=>striped n_blocks: Number of 512 byte blocks in this logical volume *parts[3]: Each parts element is a part structure pointer, which points to an array of part structures, which define the physical volume storage for one logical volume copy. - Each of these part structures points to a pvol structure and disk start address for one part of the logical volume data. The structure is defined as follows: struct part { struct pvol daddr_t int char char *pvol; start; sync_trk; ppstate; sync_msk; /* containing physical volume /* starting physical disk address /* current LTG being resynced /* physical partition state /* current LTG sync mask */ */ */ */ */ kdb lvol subcommand The kdb subcommand, lvol, formats lvol structure data. The visual above shows the lvol output for lvols[7], from the rootvg volume group. This is the logical volume with minor # 7: hd3. Items above in bold give: - lv_status = 2 means the logical volume is open. - nparts=1 means there is only one parts structure for this logical volume. - i_sched=0 means the scheduling policy for this logical volume is “regular, non-mirrored”. - n_blocks=0x10000 is the number of 512 byte blocks in this logical volume. This translates to 65536 decimal. The single part structure is at location 0x31108300. The lvol subcommand summarizes this part structure: 7-42 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty - It points to the pvol structure at 0x310e4600. The physical volume major/minor numbers are 0x19 (decimal 25)/0. The disk start address is 0x00DE1100. The ls -l command on /dev/hd* tells us this is the major/minor number of hdisk0. © Copyright IBM Corp. 2001, 2003 Unit 7. LFS, VFS and LVM 7-43 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook AIX lslv Command Output # lslv hd3 LOGICAL VOLUME: hd3 VOLUME GROUP: rootvg LV IDENTIFIER: 0001d2ca00004c00000000f11c1697a0.7 PERMISSION: read/write VG STATE: active/complete LV STATE:opened/syncd TYPE: jfs WRITE VERIFY: off MAX LPs: 512 PP SIZE: 32 megabyte(s) COPIES: 1 SCHED POLICY: parallel LPs: 1 PPs: 1 STALE PPs: 0 BB POLICY: relocatable INTER-POLICY: minimum RELOCATABLE: yes INTRA-POLICY: center UPPER BOUND: 32 MOUNT POINT: /tmp LABEL: /tmp MIRROR WRITE CONSISTENCY: on/ACTIVE EACH LP COPY ON A SEPARATE PV ?: yes Serialize IO ?: NO Figure 7-22. AIX lslv Command Output BE0070XS4.0 Notes: lslv command output The visual above shows lslv command output for rootvg logical volume hd3, the /tmp logical volume. The items in bold print above correspond to kdb lvol items described on prior slide: - “LV STATE: opened/syncd” corresponds to lvstatus=2 - “Write Verify: off” corresponds to lv_options=00001000 (flag is 0x0001 for write verify) - “PP SIZE: 32”, ”LPs: 1 and “PPs: 1” correspond to nblocks=00010000 (1 pp x 32 MB/pp = 65536 blocks x 512 bytes/block, and 65536 decimal = 10000 hexadecimal.) - “MIRROR WRITE CONSISTENCY: on/ACTIVE” corresponds to lvoptions=00001000 (flag is 0x1000 for original default mwcc) 7-44 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. VFS and LVM 7-45 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 2003 Unit 7.3 Student Notebook Uempty .0. LFS.“SCHED POLICY: parallel” is technically incorrect here. The i_sched=00000000 value from kdb correctly reflects this (SCH_REGULAR = 0 => regular.0. © Copyright IBM Corp. But it has no meaning because this logical volume is not mirrored.V2. non-mirrored logical volume). . . 00001100 beg_relblk..../* first blkno in reloc pool */ 7-46 Kernel Internals © Copyright IBM Corp.. 0000000A fp...... /* LVM PV number 0-31/0-127 */ int vg_num... /* place to hold flags */ short num_bbdir_ent.... 2001.......... kdb pvol Subcommand Output BE0070XS4. 00000000 vg_num. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. /* dev_t of physical device */ struct unique_idpvid....... ./* first available block on the PV */ daddr_t /* for user data*/ daddr_t beg_relblk............ /* PV state */ short pvnum..... /* VG major number*/ struct file * fp.. @ 310E4640 pv_pbuf. 00000000 pvstate... 00000000 pvnum.................... short pvstate..... 021E6B9Fl max_relblk.... This structure is defined as follows: struct pvol { dev_t dev.. 021E6B9F next_relblk............ 10000C60 flags. /* file pointer from open of PV */ char flags.. @ 310E46F0 Figure 7-23............................0 Notes: pvol structure The basic hardware unit for LVM is a physical volume........./* current number of BB Dir entries */ fst_usr_blk...... @ 310E4648 oclvm.............. 310E4800 sa_area[0].............. The kernel describes this in the pvol structure.. @ 310E4638 sa_area[1]. 021E6C9E defect_tbl..Student Notebook kdb pvol Subcommand Output (0)> pvol 310e4600 PVOL.......... 310E4600 dev. 00190000 xfcnt...... 00000000 fst_usr_blk. 00000000 num_bbdir_ent.... pvstate=0 means normal./* largest blkno avail for reloc */ struct defect_tbl *defect_tbl.0.vg_num= 0xA is the major number of this volume group (rootvg) This can be confirmed by executing ls -l in /dev: this shows /dev/rootvg as having major number 10 (decimal). LFS. Defined in /usr/include/sys/types./* blkno of next unused relocation */ /* block in reloc blk pool at end */ /* of PV */ daddr_t max_relblk. /* SA logical sector number . 3=> pv involved in snapshot) . .dev=00190000 means major/minor #s are 25/0 (decimal). /* flag set if SA to be deleted */ } sa_area[2]. and is for hdisk0./* changed to 1 on first bad read */ */ #ifdef CLVM_2_3 struct clvm_2_3pv *oclvm.3 Student Notebook Uempty daddr_t next_relblk. The parameter used is from our volgrp output for rootvg.vg_num: volume group major number pvol kdb subcommand The visual above shows output of the kdb pvol command. accessible physical volume. . /* one for each possible SA on PV */ struct pbuf short pv_pbuf. /* pointer to defect table */ struct sa_pv_whl { /* VGSA information for this PV */ daddr_t lsn. dev(15-0) = minor). 2001. 1=>cannot be accessed./* ptr to old CLVM pv struct #endif /* CLVM_2_3 */ int xfcnt. © Copyright IBM Corp.V2. 2=> No hw/sw relocation allowed./* SA wheel sequence number */ char nukesa.h.LV 0 */ ushort sa_seq_num. . */ */ Items shown in bold: . /* transfer count for this pv }. .pvstate: Physical volume state (0=>normal.0. VFS and LVM 7-47 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. /* pbuf struct for writing cache bad_read.pvnum=0 means physical volume number 0 in this volume group. 2003 Unit 7. . Items above in bold give .dev: major/minor device number for this disk (dev(31-16) = major. the LVM number for hdisk0 in rootvg ... 2001.28. The “PV IDENTIFIER” is maintained in the ODM class CuAt.“PHYSICAL VOLUME: HDISK0” corresponds to pvnum=0. AIX lspv Command Output BE0070XS4.80.“VOLUME GROUP: rootvg” corresponds to vg_num=0xA.109 USED DISTRIBUTION: 01.. the rootvg major number. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. It is also maintained in ODM class CuAt..00..00 # Figure 7-24.. The “VG IDENTIFIER” is maintained in the volgrp structure which points to this pvol structure..108. 7-48 Kernel Internals © Copyright IBM Corp.0 Notes: AIX lspv command The visual above shows output of the AIX lspv command for hdisk0.. The items in bold print above correspond to kdb pvol items described on prior visual: .Student Notebook AIX lspv Command Output # lspv hdisk0 PHYSICAL VOLUME :hdisk0 VOLUME GROUP: rootvg PV IDENTIFIER: 0001d2ca308b4251 VG IDENTIFIER 0001d2ca00004c00000000f1c1697a0 PV STATE: active STALE PARTITIONS: 0 ALLOCATABLE: yes PP SIZE: 32 megabyte(s) LOGICAL VOLUMES: 9 TOTAL PPs: 542 (17344 megabytes) VG DESCRIPTORS: 2 FREE PPs: 497 (15904 megabytes) HOT SPARE: no USED PPs: 45 (1440 megabytes) FREE DISTRIBUTION: 108.16.92. . 3 Student Notebook Uempty Checkpoint (1 of 2) Each user process contains a private F___ D______ T____. True or False? The three kernel structures __________. 2003 Unit 7. __________ and __________ are used to track LVM volume group.V2. Checkpoint (1 of 2) BE0070XS4.0 Notes: © Copyright IBM Corp. 2001. . LFS. respectively.0. There is one gfs structure for each mounted file system. The kdb subcommand __________ and the AIX command _________ both reflect volume group information. VFS and LVM 7-49 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The kernel maintains a _______structure and a _______structure for each mounted file system. Figure 7-25. logical volume and physical volume data.0. Student Notebook Checkpoint (2 of 2) There is one vmount/vfs structure pair for each mounted filesystem. Checkpoint (2 of 2) BE0070XS4. True or False? The inode number given by ls -id/usr is _____.0 Notes: 7-50 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Figure 7-26. Why? Each vnode for an open file points to a _______structure. . True or False? Every open file in a filesystem is represented by exactly one file structure. 2001. .0 Notes: Turn to your lab workbook and complete exercise six.V2. VFS and LVM 7-51 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp.3 Student Notebook Uempty Exercise Complete exercise six Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Test what you have learned about the LFS and VFS Locate the LFS/VFS structures for an open file Identify what file a process has opened Figure 7-27. 2001. LFS.0. Exercise BE0070XS4. 2003 Unit 7.0. Operations are defined by the vnodeops and vfsops structures. . lvol and pvol. Figure 7-28.0 Notes: 7-52 Kernel Internals © Copyright IBM Corp. Unit Summary BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Unit Summary The LFS and VFS provide support for many different file systems types simultaneously The LFS/VFS allows for different types of file systems to be mounted together forming a singe homogenous view The LFS services the system call interface for read()write() The VFS defines files (vnodes) and file systems (vfs) Each file system type provides unique functions for file and file system types operations. 2001. The gnode is a generic object connecting the VFS with the file system specific inode kdb has special subcommands for viewing LFS/VFS structures The kernel tracks LVM data in structures volgrp. There are kdb subcommands for displaying these structures. superblock.3 Student Notebook Uempty Unit 8. Journaled File System What This Unit Is About This unit describes the internal structures of the Journaled File System (JFS). 2003 Unit 8. 2001. you should be able to: • Describe basic concepts of the JFS disk layout • Describe JFS elements: inodes. allocation groups. What You Should Be Able to Do After completing this unit. indirect block and double indirect block • Contrast on disk and incore inode structures • Describe the relationship between JFS and LVM in performing I/O How You Will Check Your Progress Accountability: • Unit review References AIX Documentation: System Management Guide: Operating System and Devices © Copyright IBM Corp. Journaled File System 8-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2.0. .0. superblock. indirect block and double indirect block Contrast on disk and incore inode structures Describe the relationship between JFS and LVM in performing I/O Figure 8-1.0 Notes: 8-2 Kernel Internals © Copyright IBM Corp. Unit Objectives BE0070XS4.Student Notebook Unit Objectives At the end of this lesson you should be able to: Describe basic concepts of the JFS disk layout Describe JFS elements: inodes. allocation groups. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. These components include inodes. © Copyright IBM Corp.3 Student Notebook Uempty JFS File System Boot Block Super Block Inodes Indirect Blocks Data Blocks Figure 8-2. JFS File System BE0070XS4. Journaled File System 8-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 8. 2001. Each JFS file system occupies one logical volume.0 Notes: Journaled File System Introduction AIX 5L supports two main native file system types: JFS and JFS2. . The visual illustrates some of the basic components of a JFS. super blocks a boot block and one or more allocation groups An allocation group contains disk inodes and fragments.0. JFS maintains file data and components that identify where a file or directory's data is located on the disk. The actual on-disk layout of a JFS file system can be viewed with the fsdb command.V2. data blocks. JFS (Journaled File System) is the original native file system for AIX.0. JFS2 (Enhanced Journaled File System) is a more recent development and is discussed in a following unit. Student Notebook Boot Block The boot block occupies the first 4096 bytes of a JFS starting at byte offset 0. Superblock The superblock is 4096 bytes in size and starts at byte offset 4096. JFS fragments are the basic allocation unit and the disk is addressed at the fragment level.A flag indicating the state . and 4096 bytes. Different Doffs can have different fragment sizes.Number of data blocks . 1024. 2048. For this reason a backup copy of the superblock is always written in block 31. The functional behavior of JFS fragment support is based on that provided by Berkeley Software Distribution (BSD) fragment support. Blocks A block is a 4096 byte data allocation unit. 2001.Size . This area is from the original Berkeley Software Distribution (BSD) Fast File System design. but only one fragment size can be used within a single file system. JFS fragment support allows disk space to be divided into allocation units that are smaller than the default size of 4096 bytes. Fragments The journaled file system is organized in a contiguous series of fragments. Smaller allocation units or fragments minimize wasted disk space by more efficiently storing the data in a file or directory's partial logical blocks. . The default fragment size is 4096 bytes. and is not used in AIX. 8-4 Kernel Internals © Copyright IBM Corp. The allowable fragment sizes for JFS are 512. Specifying fragment size The fragment size for a JFS is specified during its creation. The superblock maintains information about the entire JFS and includes the following fields: .Allocation group sizes The superblock is critical to the JFS and if corrupted will prevent the file system from being mounted. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The inode number maps an inode to its location on the disk or to an inode within its allocation group. Allocation groups The set of fragments making up a JFS are divided into one or more fixed-sized units of contiguous fragments. The inode records file information such as size.The default allocation group size is 8 MB. Allocation group sizes Allocation groups are described by three sizes: . © Copyright IBM Corp. Journaled File System 8-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. For the first allocation group. it can be as large as 64 MB. These three values are stored in the file system superblock.V2. There is a one to one correspondence between a disk inode.Beginning in Version 4. However. Inodes are 128 bytes in size and are identified by a unique inode number. Each allocation group contains disk inodes and free blocks. For subsequent groups. the inodes are found at the start of each group. 2001. . . it is disjoint from the name since many different names can be refer to the same inode via the inode number. a disk inode can be located using a simple formula based on the i-number and the allocation group information contained in the super block. 2003 Unit 8. the inodes occupy the fragments immediately following the reserved block area. and a file.0.2.0. an i-number. allocation. Despite the fact that the inodes are distributed through the disk. and so on. These are called allocation groups. The first 4096 bytes of the first allocation group holds the boot block and the second 4096 bytes holds the superblock. This permits inodes and data blocks to be dispersed throughout the file system and allows file data to lie in closer proximity to its inode. owner. The collection of disk inodes can be referred to as the disk inode table. An allocation group is similar to BSD cylinder groups.The fragment allocation group size and the inode allocation group size are specified as the number of fragments and inodes that exist in each allocation group.3 Student Notebook Uempty Inodes The disk inode is the anchor for files in a JFS. . and they are set at JFS creation. When a page fault occurs on a mapped file object. and initiate a page in to transfer the data from the file system into memory. including user data blocks. examine the inode to determine where the data is. which contains a collection of XPT blocks. The read and write operations are much simplified in that they merely initialize the mapping and then copy the data. the VMM is able to determine what file is being accessed. disk files are made to look contiguous to the user program even though the physical disk blocks may be very scattered. 2001. it creates an External Page Table (XPT). Disk inodes (and indirect blocks). and XPT blocks make their respective user-level resources appear contiguous. Once completed. Likewise. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The I/O function is handled by the Virtual Memory Manager (VMM). The JFS maps all file system information into virtual memory. oblivious to the fact that a memory mapped access caused a disk operation. When the physical file system creates a file. the faulting process can be resumed and the operation continues. Just as virtual memory looks contiguous to a user program but may be scattered about real memory or paging space. 8-6 Kernel Internals © Copyright IBM Corp. a directory lookup operation merely maps the directory into virtual memory and then goes walking through the directory structure. When AIX needs to create a segment of virtual memory. it creates a disk inode and possibly indirect blocks to describe the file. This greatly simplifies the code by dividing the algorithmic problem of searching directory entries from the task of performing disk I/O operations and managing a buffer cache. .Student Notebook Virtual memory AIX exploits the segment architecture to implement its JFS physical file system. 3 Student Notebook Uempty Reserved Inodes 0 1 2 3 4 5 6 7 8 9-15 Not used Superblock (. 2003 Unit 8. This is done by manipulating the inodes so they do not require a directory entry to support their link count value. But. © Copyright IBM Corp.0. Reserved Inodes Notes: Reserved Inodes Introduction A unique feature of the JFS implementation is the implementation of file system data as unnamed files that reside in the file system. Every JFS file system has inodes 0-15 reserved. Most of these files names begin with a dot (“. but are only present in the VMM when a file system is mounted.superblock) Root directory of file system Disk inodes (.V2.inodes) Indirect blocks (. these ‘hidden’ files do not appear in any directory.”) because they are hidden files.0 Figure 8-3.diskmap) Disk inode extensions (. . Journaled File System 8-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.inodexmap) Reserved BE0070XS4.0.indirect) Disk inode allocation map (. Every open file is represented by a segment in the VMM. Most of these reserved inodes never actually exist on the disk. 2001.inodemap) Disk block allocation map (.inodex) Inode extension map (. inodemap. The most common JFS object is a regular file. The superblock holds a concise description of the JFS: its size allocation information. This bit map is used to keep track of free and allocated inode extensions. Disk inode allocation map Inode 5 is reserved for a virtual file named . rather than an array. Disk inode extensions Inode 7 is reserved for a virtual file named . The intermediate nodes of this tree are the indirect blocks.diskmap. For a regular file. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001.superblock. Disk inodes Inode 3 is reserved for a file named . Future use Inodes 9 through 15 are reserved for future extensions. This file contains information about inode extensions which are used by access control lists. the inode holds a list of the data blocks which compose the file. This bit map indicates whether each block on the logical volume is in use or free. Each disk inode is a fixed size: 128 bytes. Disk block allocation map Inode 6 is reserved for a virtual file named .inodexmap. Indirect blocks Inode 4 is reserved for a file named . . Data block 31 is a spare copy of the superblock at data block 1. 8-8 Kernel Internals © Copyright IBM Corp.inodex.inodes. 1 and 31. It would be impractical to allocate inodes large enough to directly hold this entire list. Inode extension map Inode 8 is reserved for the virtual file named . and an indication of the consistency of on-disk data structures. The list of physical blocks are held in a tree structure. Root directory Inode 2 is always used for the JFS root directory. Every JFS object is described by an disk inode. The inode points to two data blocks.Student Notebook Superblock Inode 1 is reserved for a file named . This allocation map has bit flags turned on or off showing if an inode is in use or free.indirect. . . . . . di_mtime_ts. Journaled File System 8-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Disk Inode Structure BE0070XS4. di_ctime_ts._di_file.h. uint di_nblocks. # define di_rdaddr _di_info. . di_atime_ts.id_raddr }. . . © Copyright IBM Corp. . . The allocation group size (default 8MB by default). . # define di_rindirect _di_info. mode_t di_mode. uid_t di_uid. The on disk inode structure is defined in /usr/include/jfs/ino._di_indblk.0.V2. . . . . . and the number of bytes per inode ratio (4096 by default). . . ._di_rdaddr .0. . Figure 8-4. The most basic elements or this structure are shown on the slide above and described in the text that follows. gid_t di_gid. . 2001. 2003 Unit 8. The number of inodes in a JFS file system depends on its size.0 Notes: Disk inode structure Introduction Inodes exist in a static form on disk and have access information for the file in addition to pointers to the real disk addresses of the file’s data blocks._di_file. . ushort di_nlink.3 Student Notebook Uempty Disk Inode Structure struct dinode { uint di_gen. . The AIX file system always allocates full blocks to data files. S_IFREG S_IFDIR S_IFBLK S_IFCHAR S_IFLNK S_IFSOCK S_IFIFO 8-10 Kernel Internals © Copyright IBM Corp. .h and contains: Symbol di_gen di_nlink di_mode di_uid di_gid di_size di_nblocks di_mtime di_atime di_ctime di_rdaddr[8] di_rindirect Description The disk inode generation number The number of directory entries which refer to the file The file type. Inode types are: Type Description Regular file. if any Inode types The private portion of the inode depends on its type. Character device inodes have only the dev_t Symbolic link A UNIX domain socket FIFO. This does not include indirect blocks. Directory.Student Notebook Inode header file The inode structure is defined in /usr/include/jfs/ino. 2001. The private portion of a directory inode is identical to that of a regular file. Time at which the contents of the file were last modified Time at which the file was last accessed by read Time at which contents of disk inode were last updated Real disk addresses of the data Real disk address of the indirect block. The format of the private portion of an inode for a data file (including some symbolic links and directories) depends on the size of the file. A FIFO inode has no persistent private data. The types are defined in /usr/include/sys/mode. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Block device. Block device inodes have only the dev_t Character device.h and compose portions of the di_mode field. access permissions and attributes User ID of owner Group ID File size Number of blocks used by file. V2.First portion of data structure relevant only while the object is accessed . The in-core inode contains a copy of all the fields defined in the disk inode in addition to fields for keeping track of the in-core inode. an in-core inode is created in memory The in-core inode structure is defined in /usr/include/jfs/inode.0 Notes: In-core inodes Introduction When a JFS file is opened.h. Journaled File System 8-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0.0.h In-core inodes include: An exclusive-use lock Use count Open counts State flags Exclusion counts Hash table links Free list links Mount table entry In-core inode states Active Cached Free Figure 8-5. In-core inode header file The in-core inode structure is defined in /usr/include/jfs/inode. 2003 Unit 8. . There are two parts to each in-core inode: . 2001.The last 128 bytes is a copy of the disk inode © Copyright IBM Corp.3 Student Notebook Uempty In-core Inodes When a file is opened. In-core Inodes BE0070XS4. an in-core inode is created in memory. it can be placed on a wait list for the inode (if the O_DELAY open flag was specified) Exclusion counts 8-12 Kernel Internals © Copyright IBM Corp. and decremented at close • A process which has opened the file for both reading and writing is counted as both a reader and writer Exclusive-use lock Use count Open counts State flags Maintain miscellaneous in-core inode state • A bit indicates that the file has been opened for exclusive access • A separate count of the number of readers who have specified read-only sharing (precluded writers) is also maintained • If a process attempts to open the inode with a mode which conflicts with the current open status.Student Notebook In-core inode header file The in-core inode includes: Item Notes • Must be held before the in-core inode is updated • Actually implemented with a simple lock The in-core inode cannot be destroyed while it has a non-zero use count • Separate reader and writer counts are maintained in the gnode in the in-core inode • Are incremented at each open. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . The data the in-core inode and associated segment holds is still valid and may be reused if the inode is reopened.0. There is currently a vnode that refers to this inode. If an inode is iput() and has no references to it. © Copyright IBM Corp. Entries are marked as unused by iput(). then it is placed on the free list. it can be easily reacquired should a process need the inode again. It contains the inode number and a file system number.Active.Cached. Entries are accessed by iget(). a hash table of in-core inodes for recently accessed files. accessed by device and index • Allows finding an inode by file handle. This avoids extra disk I/O. but still has a non-zero link count. and assures that multiple inodes are not created for the same object All unused in-core inodes are kept in a free list • If an object is in use. The structure is available for immediate use.V2. then it is placed in the cache list. . .0. Journaled File System 8-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. If an entry is not already in the table. 2003 Unit 8. its underlying device must currently be mounted • Each in-core inode points back to its mount table entry to avoid searching the mount table to find the entry for this object Hash table links Free list links Mount table entry In-core inode states There are three states for every in-core inode: . From this list.3 Student Notebook Uempty Item Notes • All existing in-core inodes are kept in a hash table. In-core inode table Active in-core inodes are maintained in the inode table. iget() will call iread to obtain the entry. If an inode is iput(). There is no vnode that refers to this inode.Free. it no longer has other references to it. . and it has a zero link count. This implies that the corresponding file is not open anywhere on the system. 2001. This implies that a process has the corresponding file open. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 4. This prevents deadlock conditions. For all operations which require locking more than one inode. the kernel searches the hash queue to see if there is an in-core inode already associated with the file. 2.Student Notebook In-core inode creation The steps for in-core inode creation are: Step 1. The ilocklist() routine sorts these into a descending order before locking (highest inode number is locked first). all involved inodes are known at the start of the operation. Inode locking The JFS serializes operations by obtaining an exclusive lock on each inode involved in the operation. . If an inode is found in the hash queue. Otherwise. 3. Note: The iget() routine does not return a locked inode. an in-core inode is removed from the free list and the disk inode is copied into the in-core inode. The in-core inode is then placed on the hash queue and remains there until the reference count is zero (no processes have the file open). the reference count of the in-core inode is incremented and the file descriptor is returned to the user. 8-14 Kernel Internals © Copyright IBM Corp. nor does iput() free any lock on the inode. 2001. Action When a file is opened. Direct . 2001.V2. file systems enabled for large files allow a maximum file size of slightly less than 64 gigabytes (68589453312).0 Notes: Indirect blocks Introduction JFS uses indirect blocks to address the disk space allocated to larger files. Direct (No Indirect Blocks) BE0070XS4.Single indirect .0. and all subsequent double indirect blocks contain (32 X 4096 = © Copyright IBM Corp. .Double indirect Beginning in AIX 4.3 Student Notebook Uempty Direct (no Indirect Blocks) Inode Disk Addresses for File Size <= 32KB di_raddr[0] di_raddr[7] Inode (Logical volume block numbers) data block 0 data block 7 Figure 8-6.0. The first double indirect block contains 4096 byte fragments.2. Journaled File System 8-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 8. There are three methods for addressing the disk space . . 8-16 Kernel Internals © Copyright IBM Corp. (8 x 4KB = 32 KB). The following produces the maximum file size for file systems enabling large files: (1 * (1024 * 4096)) + (511 * (1024 * 131072)) The fragment allocation assigned to a directory is divided into records of 512 bytes each and grows in accordance with the allocation of these records. Direct The first eight addresses point directly to a single allocation of disk fragments. This method is used for files that are less than 32 KB in size.Student Notebook 131072) byte fragments. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Each disk fragment is 4 KB in size. .V2. 2003 Unit 8. These addresses point to disk fragments for each allocation. This method is used for files between 32KB and 4MB (1024 x 4KB) in size.0 Notes: Single indirect The i_rindirect field of the inode contains the address of an indirect block containing 1024 addresses.indirect) Indirect Page indir[0] indir[1023] (Logical volume block numbers) data block 0 data block 1023 Figure 8-7.0. Journaled File System 8-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Single Indirect File Size Between 32KB and 4MB Inode indirect (page index in . © Copyright IBM Corp. 2001. Single Indirect BE0070XS4.0. . rather than the default fragment size of 4096 bytes. Double Indirect BE0070XS4. the graphic still holds true.Student Notebook Double Indirect Inode Disk Addresses for File Size > 4MB Inode Indirect Root indir[0] Indirect Pages ind[0] indirect (page index in .The 512 addresses do not point to data but instead point to 1024 addresses that point to data blocks (512 x ( 1024 x 4KB) ) = 2GB. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. indir[511] are 32 x 4096 = 131072 bytes long.indirect) ind[1023] ind[0] ind[1023] (Logical volume block numbers) data block 0 data block 1023 data block 0 data block 1023 Figure 8-8. This method is used for files in the range from 4 MB to 2GB With large file support enabled. in this case all “data blocks” pointed to through indir[1] . . However.indirect) indir[511] (Pages indices in ..0 Notes: Double indirect The i_rindirect field of the inode points to a double indirect block that contains 512 addresses that point to indirect blocks. 8-18 Kernel Internals © Copyright IBM Corp. 2001. True or False? 4.0. The basic allocation unit in JFS is a disk block. 2. Journaled File System 8-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .0 Notes: © Copyright IBM Corp.3 Student Notebook Uempty Checkpoint 1. An allocation group contains __________ and __________.V2. 2001.0. Checkpoint BE0070XS4. JFS maps user data blocks and directory information into virtual memory. 2003 Unit 8. The root inode number of a filesystem is always 1. The last 128 bytes of an in core JFS inode is a copy of the disk inode. True or False? 5. True or False? Figure 8-9. True or False? 3. 0 Notes: 8-20 Kernel Internals © Copyright IBM Corp. 2001. thus relying on VMM to do the actual I/O operations. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Unit Summary BE0070XS4. . A JFS allocation group contains inodes and related data blocks. A JFS in core inode contains the disk inode data together with activity information such as open count and in core inode state information. The state information indicates whether the structure is active or available for re use.Student Notebook Unit Summary Principle components of the JFS are allocation groups. JFS accomplishes I/O by mapping all file system information into virtual memory. Figure 8-10. data blocks and indirect blocks. inodes. 3 Student Notebook Uempty Unit 9. 2003 Unit 9. you should be able to: • List the difference between the terms aggregate and fileset • Identify the various data structures that make up the JFS2 file system • Use the fsdb command to trace the various data structures that make up files and directories. References AIX Documentation: System Management Guide: Operating System and Devices © Copyright IBM Corp.0. Enhanced Journaled File System 9-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.V2. 2001. . Enhanced Journaled File System What This Unit Is About This unit is about the internal structures of the Enhanced Journaled File System (JFS2). What You Should Be Able to Do After completing this unit. How You Will Check Your Progress Accountability: • Exercises using your lab system.0. Identify the various data structures that make up the JFS2 filesystem. .0 Notes: 9-2 Kernel Internals © Copyright IBM Corp. 2001.Student Notebook Unit Objectives At the end of this lesson you should be able to: List the difference between the terms aggregate and fileset. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Use the fsdb command to trace the various data structures that make up files and directories. Unit Objectives BE0070XS4. Figure 9-1. 4096 Configurable block size 4 Petabytes Max. 2003 Unit 9. © Copyright IBM Corp.0. limited by disk space B+ tree Figure 9-2.0. is an extent based Journaled File System. file system size (supported) 1 Terabyte (16 Terabytes on AIX 5. Enhanced Journaled File System 9-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .2) Dynamic. files size (this is not the supported size!) Value 512 .2) Max.V2.0 Notes: Introduction The Enhanced Journaled File System (JFS2). Numbers BE0070XS4. file size (supported) Number if Inodes Directory Organization 1 Terabyte (16 Terabytes on AIX 5.3 Student Notebook Uempty Numbers Function Block Size Architectural max. It is the default file system for the 64-bit kernel of AIX 5L. 2001. The table above lists some general information about JFS2. Student Notebook Aggregate and Fileset There is exactly one aggregate per logical volume. The aggregate block size defines the smallest unit of space allocation supported on the aggregate. . 2001. Aggregate block size An aggregate has a fixed block size (number of bytes per block) that is defined at configuration time. The rules that define aggregates and filesets in JFS2 are listed above in the visual. The block size cannot be altered. and must be 9-4 Kernel Internals © Copyright IBM Corp. from the notion of a mountable file system sub-tree. The layout of a JFS2 aggregate is also described. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The meta-data has been designed to support multiple filesets. There may be multiple filesets per aggregate Currently only one fileset per aggregate is supported. and this feature may be introduced in a future release of AIX 5L Aggregate Block Size 512 bytes 1024 bytes 2048 bytes 4096 bytes Figure 9-3. Aggregate and Fileset BE0070XS4. Definitions JFS2 separates the notion of a disk space allocation pool. called a fileset. called an aggregate.0 Notes: Introduction The term aggregate is defined in this section. 0. © Copyright IBM Corp. 2003 Unit 9.2048 bytes . Legal aggregate block sizes are: .512 bytes .0.1024 bytes . Enhanced Journaled File System 9-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .4096 bytes. 2001.V2. which defines the smallest unit of I/O.3 Student Notebook Uempty no smaller than the physical block size (currently 512 bytes). Do not confuse aggregate block size with the logical volume block size. 0 Notes: Aggregate layout The diagram above and the table below details the layout of the aggregate. • Aggregate block size. 1KB (One Aggregate Block) Aggregate Block # Reserved for LVM 0 31 32 Inodes (1 6KB) Aggregate Inode Ta ble.--bl ah bl ah 81 92 a ggr inode #2 : block ma p o wner: perm: etc: size: root -rwx ----. The primary aggregate superblock (defined as a struct superblock) contains aggregate-wide information..Student Notebook Aggregate Note: Aggregate Block Size is 1K in this example. Aggregate BE0070XS4. agg r inode #1 : “s elf” owne r: per m: et c: siz e: ro ot -r wx--. ixd Section length[0]: 16 addr[0]: 44 length[1]: 0 addr[1]: 0 .... 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Part Reserved area Function The first 32 KB is not used by JFS2.. • Size of allocation groups. Primary aggregate superblock 9-6 Kernel Internals © Copyright IBM Corp.blah blah 1638 4 aggr in od e #1 6: fi le set 0 owner: perm: etc: size: root -rwx-----blah blah 12288 0 240 8 8192 10284 4 aggr in od e # 17: fi le se t 1 owner: perm: etc: size: root -rwx-----blah blah 8192 Working Map 0xf8008000 0x00000000 . The first block is used by the LVM.. . 2001. inode numbers shown Primary Ag gregate Superblock 0 2 3 4 5 6 7 8 9 10 12 14 1 6 18 20 22 24 2 6 2 8 30 11 1 3 15 17 19 2 1 23 25 27 29 31 Control Page IAG 1 Secondary Aggregate Superblock 32 36 40 44 60 1st extent of Aggregate Inode Allocation Map Control Section iagnum: 0 Persistent Map 0xf8008000 0x00000000 . offse t: 0 add r: 36 lengt h: 8 xad entries (8 total) o ffset : 0 addr : 64 l ength : 16 offset: addr: length: offset: addr: length: offset: 0 addr: 5992 length: 8 Figure 9-4. such as the: • Size of the aggregate. Describes the aggregate inode table. This allows the superblocks to be found without depending on any other information.0. Enhanced Journaled File System 9-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The fsck working space always exists at the end of the aggregate. Provides space for logging the meta-data changes of the aggregate. One bit is needed for every aggregate block. It contains allocation state information on the aggregate inodes as well as their on-disk location. Inodes will be described later. just the addressing structures used to find the data and the inode itself. Provides space for fsck to be able to track the aggregate block allocations. Describes the secondary aggregate inode table.V2. For a very large aggregate. Contains inodes that describe the aggregate-wide control structures. Contains replicated inodes from the aggregate inode table. Describes the control structures for allocating and freeing aggregate disk blocks within the aggregate. The block allocation map maps one-to-one with the aggregate disk blocks. This space is necessary.0. The space is described by the superblock.3 Student Notebook Uempty Part Function The secondary aggregate superblock is a direct copy of the primary aggregate superblock. The space is described by the superblock. Since the inodes in the aggregate inode table are critical for finding file system information they are replicated in the secondary aggregate inode table. 2001. The secondary aggregate superblock is used if the primary aggregate superblock is corrupted. Secondary aggregate superblock Aggregate inode table Secondary aggregate inode table Aggregate inode allocation map Secondary aggregate inode allocation map Block allocation map fsck working space In-line Log © Copyright IBM Corp. Both primary and secondary superblocks are located at a fixed locations. The in-line log always exist after the fsck working space. . there might not be enough memory to track this information in memory when fsck is run. 2003 Unit 9. The actual data for the inodes will not be repeated. the aggregate inode table itself may have to grow to accommodate additional fileset inodes.Student Notebook Aggregate inodes When the aggregate is initially created. JFS2 can easily find aggregate inode one. Note that as of AIX 5. 3. Each of these aggregate inodes describe certain aspects of the aggregate itself. The preceding graphic shows a fileset 17. These inodes describe the control structures that represent each fileset. as follows: Inode # 0 Reserved Called the “self” inode. 2. This inode is allocated but no data is saved to disk. 4 KB after the primary aggregate superblock. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.2 release there can only be one fileset. Reserved for future extensions. additional inode extents are allocated and de-allocated dynamically as needed. Therefore. and is not realizable at present. this inode describes the aggregate disk blocks comprising the aggregate inode map. This is included to show design potential. This is a circular representation. in that aggregate inode one is itself in the file that it describes. Description 1. namely. 2001. . The obvious circular representation problem is handled by forcing at least the first aggregate inode extent to appear at a well-known location. 4 . Describes the In-line Log when mounted. the first inode extent is allocated.15 16 - 9-8 Kernel Internals © Copyright IBM Corp. and from there it can find the rest of the aggregate inode table by following the B+–tree in inode one Describes the block allocation map. As additional filesets are added to the aggregate. Starting at aggregate inode 16 there is one inode per fileset (the fileset allocation map Inode). The allocation group size must always be a power of 2 multiple of the number of blocks described by one dmap page. (for example 1.Distribute unrelated data throughout the aggregate. Allocation groups are used for heuristics only.0.. JFS2 will attempt to: . dmap pages) Figure 9-5. 4. 2..3 Student Notebook Uempty Allocation Group The maximum number of allocation groups per aggregate is 128.Group disk blocks for related data and inodes close together. 8.0. © Copyright IBM Corp. . Enhanced Journaled File System 9-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Allocation Group BE0070XS4. 2003 Unit 9. 2001. .0 Notes: Introduction Allocation Groups (AG) divide the space on an aggregate into chunks. The minimum number of allocation group is 8192 aggregate blocks. . Allocation policies When locating data on the disk. Allocation groups allow JFS2 resource allocation policies to use well known methods for achieving good JFS2 I/O performance.V2. Partial allocation group An aggregate whose size is not a multiple of the allocation group size contains a partial allocation group that it is not fully covered by disk blocks. The allocation group size is stored in the aggregate superblock. This partial allocation group will be treated as a complete allocation group. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Allocation group sizes Allocation group sizes must be selected which yield allocation groups that are sufficiently large to provide for contiguous resource allocation over time. except we mark the non-existent disk blocks allocated in the Block Allocation Map. The rules for setting the allocation group size are shown in the visual on the previous page. 9-10 Kernel Internals © Copyright IBM Corp. . 2001. fileset inode #2: root directory owner: perm: etc: size: r oot rwx-----b lah blah 4 096 Con trol Sect ion iag num: 1 iag free: -1 Wo rking Map 0x ffffffff 0x ffffffff . id otdot:2 2 Figure 9-6.0.. Part Fileset inode table Function Contains inodes describing the fileset-wide control structures. A fileset is completely contained within a single aggregate. . The visual illustration above and table below details the layout of a fileset. Pe rsistent Map 0x ffffffff 0x ffffffff . A fileset inode allocation map which describes the Fileset Inode Table... Fileset inode allocation map © Copyright IBM Corp. Fileset BE0070XS4.V2. .. ix d Section le ngth[0]: 0 ad dr[0]: 0 le ngth[1]: 0 ad dr[1]: 0 .. 2003 Unit 9.. . ixd Secti on length[0] : 16 addr[0]: 248 length[1] : 0 addr[1]: 0 . Enhanced Journaled File System 9-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction A fileset is a set of files and directories that form an independently mountable sub-tree that is equivalent to a UNIX file system file hierarchy..3 Student Notebook Uempty Fileset Fileset Inode Table 0 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 Control Pag e IAG 1 9 11 13 15 17 19 21 23 25 27 29 31 IAG 24 8 264 10284 240 fileset #0: AG Free Inode List i nofree: e xtfree: n uminos: n umfree: 1 inofree: extfree: numinos: numfree: inofree: extfree: numinos: numfree: 1 1 3 2 2 8 -1 -1 0 0 -1 -1 0 0 2 44 Fileset Inode Allocation M ap : 2n d extent IAG Free List: 1st entry Fileset Inode Allocation M ap: 1st extent Control S ection iagnum: 0 Working M ap 0xf000000 0 0xfffffff f .. 2001. The Fileset Inode allocation map contains allocation state information on the fileset inodes.0. Persisten t Map 0xf000000 0 0xfffffff f . The Fileset Inode Table logically contains an array of inodes.. . as well as their on-disk location. Note that all JFS2 meta data structures (except for the superblock) are represented as “files. They also “contain” a B+–tree to record the allocation of extents. file type (regular or directory). which contains the expected object-specific information such as time stamps. Inodes 9-12 Kernel Internals © Copyright IBM Corp. . 2001.Student Notebook Part Function Every JFS2 object is represented by an inode. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.” By reusing the inode structure for this data. the data format (on-disk layout) becomes inherently extensible. Inode Allocation Map BE0070XS4.3 Student Notebook Uempty Inode Allocation Map 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 Fileset Inode # 0 1 2 3 4- Description reserved. directories. additional inode extents are allocated and de-allocated dynamically as needed. the first inode extent is allocated. The ACL file for the fileset. user files. When the fileset is initially created.V2. Figure 9-7. Fileset inodes from four onwards are used by ordinary fileset objects.0. additional fileset information that would not fit in the fileset allocation map inode in the aggregate inode table. The root directory inode for the fileset. Since the aggregate inode table is replicated. © Copyright IBM Corp. Inodes Every file and directory in a fileset is describe by an on-disk inode. there is also a secondary version of this inode. which points to the same data.0. and symbolic links. 2001. The inodes in a fileset are allocated as shown above in the visual. Enhanced Journaled File System 9-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 9. .0 Notes: Super inode Super inodes found in the aggregate inode table (#16 and greater) describe the fileset inode allocation map and other fileset information resides in the aggregate inode table. offset=0 len=3 addr=101 xad_flag. Extents BE0070XS4.0 Notes: Introduction Disk space in a JFS2 file system is allocated in a sequence of contiguous aggregate blocks called an extent. xad_reserved. xad_offset.h struct xad { uint8 uint16 uint40 uint24 uint40 }. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 9-14 Kernel Internals © Copyright IBM Corp. .Student Notebook Extents XADs for a file flag reserved /usr/include/j2/j2_xtree. File system disk blocks disk block 101 disk block 503 offset=0 len=4 addr=503 disk block 856 offset=0 len=2 addr=856 Figure 9-8. 2001. xad_length. xad_address. both the length and address are expressed in units of the aggregate block size. In an xad. and is the block offset from the beginning of the aggregate. A 40-bit field. containing the address of the first block of the extent. 2001. Details of the xad data structure are shown in the visual on the previous page. 2003 Unit 9.1 aggregate blocks. Extent allocation descriptor Extents are described in an xad structure (a 16 byte structure). A 24-bit field. containing the length of the extent in aggregate blocks.Are indexed in a B+-tree. and its address. Member xad_flag xad_reserved Description Flags set on this extent.V2. . See /usr/include/j2/j2_xtree. The address is in units of aggregate blocks.Are variable in size and can range from 1 to 224.h for a list of flags.1 aggregate blocks.Large extents may span multiple allocation groups. xad_offset xad_length xad_address © Copyright IBM Corp.3 Student Notebook Uempty Extent rules An extent: . The xad_offset. . . xad description The elements of the xad structure are described in this table. Enhanced Journaled File System 9-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. . The two main values describing an extent are its length. Reserved for future use. Extents are generally grouped together to form a larger group of disk blocks.0. describes the logical block offset this extent represents in the larger group. An extent can range in size from 1 to 224 .Is made up of a series contiguous aggregate blocks.Are wholly contained within a single aggregate . This allows for larger I/O transfer. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 9-16 Kernel Internals © Copyright IBM Corp. 2001. resulting in improved performance. the allocation policy for JFS2 tries to maximize contiguous allocation by allocating a minimum number of extents.Student Notebook Increasing an Allocation File system disk blocks Before flag reserved offset=0 len=100 addr=101 maximize contiguous allocation After flag reserved offset=0 len=200 addr=101 File system disk blocks 100 disk blocks 100 disk blocks 100 disk blocks 100 disk blocks 100 disk blocks flag reserved offset=0 len=100 addr=701 flag reserved offset=0 len=100 addr=701 flag reserved offset=100 len=100 addr=1001 100 disk blocks Figure 9-9.0 Notes: Introduction In general. . Increasing an Allocation BE0070XS4. keeping each extent as large and contiguous as possible. For example. We have a limited amount of memory available. Another case is restriction of the extent size. so we must ensure we will have enough room for the decompressed extent.V2. copy-on-write clone of a segment will cause a contiguous extent to be partitioned into a sequence of smaller contiguous extents. © Copyright IBM Corp. 2003 Unit 9.0.3 Student Notebook Uempty Exceptions In special cases.0. the extent size is restricted for compressed files. . 2001. Enhanced Journaled File System 9-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Fragmentation The user can configure a JFS2 aggregate with a small aggregate block size of 512 bytes to minimize internal fragmentation for aggregates with large numbers of small size files. since we must read the entire extent into memory and decompress it. The defragfs utility can be used to defragment a JFS2 file system. For example. this is not always possible to keep extent allocation contiguous. 2001.0 Notes: Introduction Objects in JFS2 are stored in groups of extents arranged in binary trees. these headers reside in the second inode quadrant and in 4KB blocks referenced by the inode. The concepts of binary trees are introduced in this section. 9-18 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Binary Tree of Extents BE0070XS4. A flag in the node header identifies the role of the node in the tree. As we will show in subsequent material. Trees Binary trees consists of nodes arranged in a tree structure. .Student Notebook Binary Tree of Extents Root node Header flags=BT_ROOT Internal node Header flags= BT_INTERNAL Leaf node Header flags= BT_LEAF Leaf node Header flags= BT_LEAF Leaf node Header flags= BT_LEAF Array of extent descriptors xad xad xad Array of extent descriptors xad xad xad Array of extent descriptors xad xad xad Figure 9-10. Each node contains a header describing the node. © Copyright IBM Corp.0. 2001. the most common operations. The entries are sorted by the offsets of the xad structures. 2003 Unit 9.0.Providing fast reading and writing of extents. An internal node points to two or more leaf nodes or other internal nodes.Providing fast search for reading a particular extent of a file. . Why B+-tree? B+–trees are used in JFS2. . Leaf nodes point to the extents containing the objects data.V2. The data being indexed depends upon the object. and help performance by: . each of which is an entry in a node of a B+–tree. The B+–tree is keyed by the offset of the xad structure of the data being described by the tree. B+-tree index There is one generic B+–tree index structure for all index objects in JFS2 (except for directories).Providing efficient append or insert of an extent in a file.Being efficient for traversal of an entire B+–tree. .3 Student Notebook Uempty Header flags This table describes the binary tree header flags: Flag BT_ROOT BT_LEAF BT_INTERNAL Description The root or top of the tree. . The bottom of a branch of a tree. Enhanced Journaled File System 9-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. user Id. created time and more. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. split into four 128 byte sections. Section Description This section describes the POSIX attributes of the JFS2 object including the inode and fileset number. access time. The inode holds the root header for the extent binary tree. 2001. 9-20 Kernel Internals . created. object type. 1. © Copyright IBM Corp.0 Notes: Overview Every file on a JFS2 file system is describe by an on-disk inode.Student Notebook Inodes Inode Layout Section 1 Section 2 Section 3 y y y y POSIX Attributes extended attributes block allocation maps Inode allocation maps headers describing the inode data In-line data or xad's extended attributes or more in-line data or additional xad's Section 4 Figure 9-11. The sections of the inode are described in this table. group Id. object size. modified time. Inodes BE0070XS4. File attribute data and block allocation maps are also kept in the inode. Inode layout The inode is a 512 byte structure. 3 Student Notebook Uempty Section Description This section contains several parts: • Descriptors for extended attributes. • Header pointing to the data (b+-tree root. in-line data) This section can contain one of the following: 3. xad structures or in-line data.V2. • In-line file data for very small files (up to 128 bytes).0.0. 2001. 4. implementation Currently section 4 is not used. Design vs. . • Block allocation maps. • Inode allocation maps. 2003 Unit 9. This section extends section 3 by providing additional storage for more attributes. Enhanced Journaled File System 9-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Currently section 3 is only used for extent information. 2. • The first eight xad structures describing the extents for this file. directory. © Copyright IBM Corp. The inline data function of JFS2 is not currently enabled. 2001. di_size.Student Notebook Structure The current definition of the on-disk inode structure is: struct dinode { /* * I. /* 8: inode number. di_otime. aka file serial number */ uint32 di_gen.__bmap /* 9-22 Kernel Internals © Copyright IBM Corp. */ ead_t union { uint8 di_ea. /* 4: stamp to show inode belongs to fileset */ uint32 di_rsv1. /* 4: */ pxd_t int64 int64 uint32 uint32 int32 uint32 j2time_t j2time_t j2time_t j2time_t di_ixpxd. extension area (128 bytes) * -----------------------------*/ /* * extended attributes for file system (96). /* * block allocation map */ struct { struct bmap *__bmap. /* 8: inode extent descriptor */ /* 8: size */ /* 8: number of blocks allocated */ /* 4: uid_t user id of owner */ /* 4: gid_t group id of owner */ /* 4: number of links to the object */ /* 4: mode_t attribute format and permission */ /* /* /* /* 16: 16: 16: 16: time time time time last data accessed */ last status changed */ last data modified */ created */ /* * II. di_nblocks. di_nlink. di_uid. /* 16: ea descriptor */ _data[80]. inode # of inode map file */ uint32 di_inostamp. di_ctime. /* incore bmap descriptor */ } _bmap._bmap. di_gid. . di_mode. base area (128 bytes) * -----------------------* * define generic/POSIX attributes */ ino64_t di_number. /* 4: fileset #. #define di_bmap _data2. di_atime. /* 4: inode generation number */ uint32 di_fileset. di_mtime. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. * N. Enhanced Journaled File System 9-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. or dtroot_t for directory._di_btroot #define di_xtroot _data2r. #define di_gengen _data2.3 Student Notebook Uempty * inode allocation map (fileset inode 1st half) */ struct { uint32 _gengen. /* 16: data extent descriptor */ } _xd._imap._imap._data /* * regular file or directory * * B+-tree root node/inline data area */ struct { uint8 _xad[128]. #define di_inlinedata _data3.0. must be on 8-byte boundary. #define di_dxd _data2r._di_parent /* * III. 2003 Unit 9._gengen #define di_ipimap2 _data2.V2. . /* replica */ struct inomap *__imap. type-dependent area (128 bytes) * -----------------------------------* * B+-tree root node xad array or inline data * */ union { uint8 _data[128]. int32 _di_btroot[8]. */ union { struct { int32 _di_rsrvd[4]._imap. * or data extent descriptor for inline data.__imap /* * B+-tree root header (32) * * B+-tree root node header._di_dxd #define di_btroot _data2r. /* 16: */ dxd_t _di_dxd. /* 8: idotdot in dtroot_t */ } _data2r. /* di_gen generator */ struct inode *__ipimap2.0._di_btroot #define di_parent _data2r. } _data2.B. /* incore imap control */ } _imap.__ipimap2 #define di_imap _data2. /* 32: xtpage_t or dtroot_t */ ino64_t _di_parent._xd._di_btroot #define di_dtroot _data2r. /* * device special file */ © Copyright IBM Corp. } _file. There is no need to allocate “ten times as many inodes as you will ever need. . which provides the following advantages: . #define di_inlineea _data4. } _symlink. type-dependent extension area (128 bytes) * ----------------------------------------* * user-defined attribute. 9-24 Kernel Internals © Copyright IBM Corp. typedef struct dinode dinode_t. all the blocks contained in an allocation group can be used for data. Otherwise stored like a regular file. which decouples the inode number from the location. file system space utilization is optimized.Allows placement of inode disk blocks at any disk address. or * inline data continuation. which makes it unnecessary to search the directory structure to update the inode numbers. This decoupling simplifies supporting aggregate and fileset reorganization (to enable shrinking the aggregate).File allocation for large files can consume multiple allocation groups and still be contiguous. The inodes can be moved and still retain the same number. thus. . This is especially important with the larger inode size of 512 bytes in JFS2. or * B+-tree root node continuation * */ union { uint8 _data[128]._data } _data4. #define di_rdev _rdev. /* * IV. as with file systems that contain a fixed number of inodes._specfile._fastsymlink } _data3. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. . }. With dynamic allocation. * * link is stored in inode if its length is less than * IDATASIZE. Static allocation forces a gap containing the initially allocated inodes in each allocation group. /* 8: dev_t device major and minor */ _data3. #define di_fastsymlink _data3._symlink._rdev /* * symbolic link. */ struct { uint8 _fastsymlink[128].Student Notebook struct { dev64_t } _specfile. Allocation policy JFS2 allocates inodes dynamically. Inode generation numbers Inode generation numbers are simply counters that will increment each time an inode is reused. (implicitly) require them. in JFS2.e.V2. With dynamic allocation. Although a fileset-wide generation counter will recycle faster than a per-inode generation counter. i. Inode initialization When a new inode extent is allocated. 2003 Unit 9. a simple calculation shows that the 32-bit value is still sufficient to meet NFS or DFS requirements. However. The static inode allocation practice of storing a per-inode generation counter will not work with dynamic inode allocation. . the space may be reclaimed for ordinary file data storage).0. replicating the B+–tree structures.3 Student Notebook Uempty Dynamic inode allocation causes a number of problems. such as NFS. © Copyright IBM Corp. their inode numbers and extent addresses are set.0. Information about the inode extent is also added to the inode allocation map. Inode extents Inodes are allocated dynamically by allocating inode extents that are simply a contiguous chunk of inodes on the disk. and the mode and link count fields are set to zero. Due to the overhead involved in replicating these structures we accept the risk of losing these maps.The inode mapping structures are critical to JFS2 integrity. the inodes in the extent are initialized. Therefore we must have a means of finding the inodes on disk. because when an inode becomes free its disk space may literally be reused for something other than an inode (e. allows us to find the maps. By definition. an inode extent occupies 16 KB on the disk. Network file system protocols. they form part of the file identifier manipulated by VNOP_FID() and VFS_VGET(). Enhanced Journaled File System 9-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. there is simply one inode generation counter that is incremented on every inode allocation (rather than one counter per inode that would be incremented when that inode is reused). Therefore. . Inode allocation map Dynamic inode allocation implies that there is no direct relationship between an inode number and the disk address of the inode.With static allocation. the geometry of the file system implicitly describes the layout of inodes on disk. a JFS2 inode extent contains 32 inodes. The inode allocation map provides this function.. 2001. including: . With a 512 byte inode size. separate mapping structures are required.g. . The header found in the second section of the inode points to the data that is stored in the third and fourth section of the inode. the data may be stored in the inode itself. 9-26 Kernel Internals © Copyright IBM Corp. This is called in-line storage. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Inline Data Inode Info Header for in-line data Figure 9-12.0 Notes: In-line data If a file contains small amounts of data. This design feature has not been implemented yet. Inline Data In-line data BE0070XS4. 2001. If there are 8 or fewer extents for the file. An inode containing 8 or less xad structures would look like the figure shown above. If the INLINEEA bit is set in the di_mode field of the inode.3 Student Notebook Uempty Binary Trees Inode Info B+-tree header offset: 0 addr: 68 length: 16 offset: 84 addr: 4096 length: 48 68 16KB Data In-line data 4096 48KB Data offset: 256 addr: 26624 length:48 26624 8KB Data Figure 9-13. Enhanced Journaled File System 9-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 9.0 Notes: Binary trees When more storage is needed than can be provided in-line the data must be placed in extents. an attempt is made to use the last quadrant of the inode for more xad structures. INLINEEA bit Once the 8 xad structures in the inode are filled. © Copyright IBM Corp. This design feature has not been implemented yet.V2. then the xad structures describing the extents are contained in the inode. The header in the inode now becomes the binary tree root header. then the last quadrant of the inode is available for 8 more xad structures. 2001.0. Binary Trees BE0070XS4. .0. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. The offset for this new xad structure contains the offset of the first entry in the leaf node. and that it contains the pure root of a B+-tree.Student Notebook More Extents ino de I node Info B+. 4 KB of disk space is allocated for a leaf node of the B+–tree. and the header is initialized to point to the 9th entry as the first free entry.0 Notes: More extents Once all of the available xad structures in the inode are used. which is logically an array of xad entries with a header.tree header offset: addr: length: offset: addr: length: 0 412 4 0 0 0 header 412 68 16KB Data 254 xad leaf node entries xad entries (8 total) 4096 48KB Data offset: 0 addr: 0 length: 0 26624 8KB Data Figure 9-14. and the inode header is updated to indicate that only one xad structure is now being used. . The first xad structure in the inode is updated to point to the newly allocated leaf node. More Extents BE0070XS4. the B+–tree must be split. The organization of the inode now looks like the figure above. The 8 xad entries are moved from the inode to the leaf node. 9-28 Kernel Internals © Copyright IBM Corp. and the second xad structure from the inode is set to point to this newly allocated node.0.0 Notes: Continuing to add extents As new extents are added to the file.3 Student Notebook Uempty Continuing to Add Extents inode Inode Info B+-tree header offset: addr: length: offset: addr: length: 0 412 4 750 560 4 header 16KB Data 412 68 254 xad leaf node entries xad entries (8 total) 4096 48KB Data offset: 0 addr: 0 length: 0 560 header 254 xad leaf node entries 26624 8KB Data Figure 9-15. 2001. © Copyright IBM Corp. Continuing to Add Extents BE0070XS4. Enhanced Journaled File System 9-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. they continue to be added to the leaf node in the necessary order. The node now looks like the figure shown above. . Once the node fills an additional 4 KB of disk space is allocated for another leaf node of the B+–tree. until the node fills.0. 2003 Unit 9.V2. An internal node looks exactly like a leaf node. 9-30 Kernel Internals © Copyright IBM Corp. this behavior continues until all 8 xad structures in the inode contain leaf node xad structures. which is used purely to route the searches of the tree. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Another Split BE0070XS4. the 8 xads of the leaf nodes are moved from the inode to the newly created internal node.0 Notes: Another split As extents are added to the inode. and the internal node header is initialized to point to the 9th entry as the first free entry. This split creates an internal node of the B+–tree. 2001. at which time another split of the B+–tree will occur.Student Notebook Another Split inode Inode Info B+-tree header offset: addr: length: offset: addr: length: 0 380 4 8340 212 4 header 380 header 16KB Data 412 68 254 xad internal node entries 254 xad leaf node entries xad entries (8 total) 4096 48KB Data offset: 0 addr: 0 length: 0 212 header header 560 254 xad internal node entries 254 xad leaf node entries 26624 8KB Data Figure 9-16. The root of the B+–tree is then updated by making the inode’s first xad structure point to the newly allocated internal node. and the header in the inode is updated to indicate that only 1 xad structure is now being used for the B+–tree. 4 KB of disk space is allocated for the internal node of the B+–tree. Enhanced Journaled File System 9-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .0.3 Student Notebook Uempty As extents continue to be added. 2001. the inode’s second xad structure is updated to point to the new internal node.V2. and these leaf nodes are added to the internal node. © Copyright IBM Corp.0. a second internal node is allocated. 2003 Unit 9. This behavior continues until all eight of the inode’s xad structures contain internal nodes. Once the first internal node is filled. additional leaf nodes are created to contain the xad structures for the extents. fsdb Utility BE0070XS4. alter. 2001. and debug a file system.Student Notebook fsdb Utility # fsdb /dev/lv00 Aggregate Block Size: 512 > > help Xpeek Commands a[lter] <block> <offset> <hex string> b[tree] <block> [<offset>] dir[ectory] <inode number> [<fileset>] d[isplay] [<block> [<offset> [<format> [<count>]]]] dm[ap] [<block number>] dt[ree] <inode number> [<fileset>] h[elp] [<command>] ia[g] [<IAG number>] [a | <fileset>] i[node] [<inode number>] [a | <fileset>] q[uit] su[perblock] [p | s] Figure 9-17. . Starting fsdb It is best to run fsdb against an unmounted file system. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Use the following syntax to start fsdb: fsdb <path to logical volume> For example: # fsdb /dev/lv00 Aggregate Block Size: 512 > 9-32 Kernel Internals © Copyright IBM Corp.0 Notes: Introduction The fsdb command enables you to examine. 2003 Unit 9.0. The commands available in fsdb are different depending on what file system type it is running against. 2001.0.3 Student Notebook Uempty Support file systems fsdb supports both the JFS and JFS2 file systems. Enhanced Journaled File System 9-33 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. © Copyright IBM Corp. Commands The commands available in fsdb can be viewed with the help command as shown in the visual.V2. . The following explains how to use fsdb with a JFS2 file system. 0 Notes: Turn to your lab workbook and complete exercise seven.Student Notebook Exercise Complete exercise seven Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Use the fsdb utility to examine a JFS2 file system. . Exercise BE0070XS4. Identify a file's inode number Identify extent descriptors Locate the data extents that hold the contents of a file Figure 9-18. 2001. 9-34 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 0. which indicate the files and sub-directories contained in the directory. © Copyright IBM Corp. Enhanced Journaled File System 9-35 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Directory entry Stored in an array.3 Student Notebook Uempty Directory inumber next namelen name[22] Figure 9-19. Member inumber Description Inode number.0 Notes: Introduction In addition to files. A directory is a journaled meta-data file in JFS2. Directory BE0070XS4. 2003 Unit 9. an inode can represent a directory.V2. The directory entry is a 32 byte structure and has the members shown here.0. 2001. and is composed of directory entries. the directory entries link the names of the objects in the directory to an inode number. /* 22: 2-byte aligned */ 9-36 Kernel Internals © Copyright IBM Corp. /* 8: 4-byte aligned */ leaf node entry head/only segment int8 uint8 #ifdef next.h. up to 22 characters. namlen. /* 1: */ /* 1: */ _J2_UNICODE UniChar name[11]. Directory entry structure definition Following is the structure definition for a directory entry. It is from /usr/include/j2/j2_dtree. /* (32) */ name[22]. 2001. /* 22: 2-byte aligned */ #else char #endif } ldtentry_t. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. File name.Student Notebook Member next namelen name[22] Description If more than 22 characters are needed. Length of the name. . /* * */ typedef struct { ino64_t inumber. additional entries are linked using the next pointer. h. As with files. Enhanced Journaled File System 9-37 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The number of free slots in the directory entry array. The last used slot in the directory entry slot array. Indicates if the node is an internal or leaf node. idotdot. /* /* /* /* /* /* /* /* /* 8: parent inode number */ 8: */ 1: */ 1: next free entry in stbl */ 1: free count */ 1: freelist header */ 4: */ 8: sorted entry index table */ (32) */ Figure 9-20. 2003 Unit 9. 2001.3 Student Notebook Uempty Directory Root Header typedef union { struct { ino64_t int64 uint8 int8 int8 int8 int32 int8 } header.0 Notes: Root header In order to improve the performance of locating a specific directory entry. and whether it is the root of the binary tree. the header section of a directory inode contains the binary tree root header. rsrvd1.0. Member idotdot flag nextindex freecnt Description Inode number of parent directory. nextindex. flag. Directory Root Header BE0070XS4. rsrvd2. a binary tree sorted by name is used. dtslot_t } dtroot_t. The root header is a 32 byte structure defined by dtroot_t in /usr/include/j2/j2_dtree.V2. stbl[8]. .0. slot[9]. © Copyright IBM Corp. Each header describes an eight element array of directory entries. freelist. freecnt. 9-38 Kernel Internals © Copyright IBM Corp. Leaf and internal node header When more than eight directory entries are needed a leaf or internal node is added.h. contained in /usr/include/j2/j2_dtree. The directory internal and leaf node headers are similar to the root node header. The page header is defined by a dtpage_t structure. . The header is stored in the first slot.Student Notebook Member freelist stbl[8] slot[9] Description The slot number of the head of the free list The indices to the directory entry slots that are currently in use. except that they may have up to 128 directory entries (corresponding to a 4096 byte leaf page). 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. The array of directory entries. The entries are sorted alphabetically by name. There are eight entries. This limits the amount of shifting necessary when directory entries are added or deleted. the directory entry table contains four files. © Copyright IBM Corp. since the array is much smaller than the entries themselves.V2. 2003 Unit 9. Example In the example show above. The entries are sorted alphabetically by name. A binary search can be used on this array to search for particular directory entries. Enhanced Journaled File System 9-39 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Directory Slot Array BE0070XS4.3 Student Notebook Uempty Directory Slot Array Directory Entry table 1 2 3 4 5 6 7 8 def abc xyz hij 2 1 4 STBL[8] 3 0 0 0 0 Figure 9-21.0 Notes: Directory slot array The directory slot array (stbl[]) is a sorted array of indices of the directory slots that are currently in use. 2001. .0. The stbl table contains the slot numbers of the entries ordering the entries alphabetically.0. JFS2 allocates a leaf node the same size as the aggregate block size. When all entries are free in a leaf page. The directory slot array will only have been big enough to reference enough slots for the smaller page. the directory tables must be increase in size. and the data from the old extent must be copied to the new extent. 2. 3. . these will be represented in the inode itself. Action Initial directory entries are stored in the directory inode in-line data area. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. First attempt to double the extent in place. double the current size. Instead.”) and parent (“. 5. When that initial leaf node becomes full and the leaf node is not yet 4 KB. Update the header to point to this array and add the slots for the old array to the free list. directories A directory does not contain specific entries for the self (“.Student Notebook . Every leaf node after the initial one will be allocated as 4 KB to start. Self is the directory’s own inode number. This table describes the steps used. and . When all the entries in the last leaf page are deleted. so a new slot array will have to be created. 4. and the parent inode number is held in the “idotdot” field in the header. the page will be removed from the B+–tree. If the leaf node again becomes full and is still not 4 KB repeat step 3. Step 1. When the in-line data area of the directory inode becomes full.. 2001. a new extent must be allocated. Growing directory size As the number of files in the directory grow. 9-40 Kernel Internals © Copyright IBM Corp.”) directories. Once the leaf node reaches 4 KB allocate a new leaf node.. if there is not room to do this. the directory will shrink back into the directory inode in-line data area. Use the slots from the beginning of the newly allocated space for the larger array and copy the old array data to the new location. 2 . all the inode information fits into the in-line data area. © Copyright IBM Corp. Small directories Initial directory entries are stored in the directory inode in-line data area.3 Student Notebook Uempty Small Directory Example # ls -ai 69651 .0.V2. Examine the example of a small directory. Small Directory Example BE0070XS4.. 2003 Unit 9. 2001.0. In the example shown above. . 0. 69652 foobar1 69653 foobar12 69654 foobar3 69655 longnamedfilewithover22charsinitsname fl ag: B T_R OOT B T_L EA F ne xti nd ex: 4 fr eec nt : 3 fr eel is t: 6 id otd ot : 2 st bl: { 1. 2 .3.0 Notes: Introduction This section demonstrates how the directory structures change over time.0 } 1 i nu mbe r: 69 652 n ex t: -1 n am ele n: 7 n am e: foo ba r1 i nu mbe r: 69 653 n ex t: -1 n am ele n: 8 n am e: foo ba r12 i nu mbe r: 69 654 n ex t: -1 n am ele n: 7 n am e: foo ba r2 i nu mbe r: 69 655 n ex t: 5 n am ele n: 37 n am e:l ong na med fi lew it hov er 2 n ex t: -1 c nt : 0 n am e: 2ch ar sin it sna me 2 3 4 5 Figure 9-22.0 . Note: the file with a long name has its name split across two slots. Enhanced Journaled File System 9-41 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 4. 69656 afile 69652 foobar1 69653 foobar2 69654 foobar3 69655 longnamedfilewithover22charsinitsname f la g: B T_ RO OT B T_ LE AF n ex ti nd ex : 5 f re ec nt : 2 f re el is t: 7 i do td ot : 2 s tb l: { 6. 9-42 Kernel Internals © Copyright IBM Corp. . 1. 0.Student Notebook Adding a File # ls -ai 69651 . alphabetically. 2.. 4. As this is now. 0.0 Notes: Adding a file An additional file called “afile” is created. 2 . the search table array (stbl[]) is re-organized. 3. 0} 1 in um be r: 6 96 52 ne xt : -1 na me le n: 7 na me : fo ob ar 1 in um be r: 6 96 53 ne xt : -1 na me le n: 8 na me : fo ob ar 12 in um be r: 6 96 54 ne xt : -1 na me le n: 7 na me : fo ob ar 2 in um be r: 6 96 55 ne xt : 5 na me le n: 3 7 na me :l on gn am ed fi le wi th ov er 2 ne xt : -1 cn t: 0 na me : 2c ha rs in it sn am e 6 in um be r: 6 96 56 ne xt : -1 na me le n: 5 na me : af il e 2 3 4 5 Figure 9-23. 2001. so that the entry in slot 6 is now in the first entry. the first file in the directory. Details for this file are added at the next free slot (slot 6). Adding a File BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 0 Notes: Adding a leaf node When the directory grows to the point where there are more entries than can be stored in the in-line data area of the inode.8} 1 xd. 2001.3.addr2: 52 next: -1 namelen: 0 name: file0 flag: BT_LEAF nextindex: 20 freecnt: 103 freelist: 25 maxslot: 128 stbl: {1. 8. © Copyright IBM Corp.7. Note: the internal node entry contains the name of the first file (in alphabetical order) for that leaf node.addr1: 0 xd. Enhanced Journaled File System 9-43 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.6.0. an internal node is added at the next free in-line data slot in the inode. . The in-line entries are moved to a leaf node.3 Student Notebook Uempty Adding a Leaf Node Block 52 flag: BT_ROOT BT_INTERNAL nextindex: 1 freecnt: 7 freelist: 2 idotdot: 2 stbl: {1. as illustrated above. Adding a Leaf Node BE0070XS4.13.V2. then JFS2 allocates a leaf node the same size as the aggregate block size.. 2003 Unit 9.len: 1 xd.14} 1 inumber: 5 next: -1 namelen: 5 name: file0 inumber: 6 next: -1 namelen: 5 name: file1 inumber: 15 next: -1 namelen: 6 name: file10 2 3 19 20 inumber: 23 next: -1 namelen: 6 name: file18 inumber: 24 next: -1 namelen: 6 name: file19 Figure 9-24.5.2.15.0..4. Once the leaf is full. which will contain the address of the next leaf node. .2. 1 na mel en : 0 na me: f ile 0 x d. 19 .Student Notebook Adding an Internal Node Bl ock 1 18 fla g: BT _R OOT B T_I NT ERN AL nex ti nde x: 4 fre ec nt: 4 fre el ist : 5 ido td ot: 2 stb l: {1 . two layers of internal nodes are required to reference all the files.2 . 2.a dd r1: 0 x d.18 ..1 na mel en : 8 na me: f ile 14 72 x d. 7 .ad dr 2: 52 ne xt: . add r1 : x d.ad dr 1: -1 xd . add r2 : n ex t: n am ele n: n am e: Bl ock 5 2 fl ag: BT _L EAF ne xti nde x: 64 fr eec nt: 5 9 fr eel ist : 21 ma xsl ot: 1 28 st bl: {1 .a dd r2: 11 8 n ext : -1 n ame le n: 0 n ame : fil e0 x d.ad dr 1: 0 xd . and the first in-line data slot updated with the address of the new internal node.15 .1 12 } 1 i nu mbe r: 5 n ex t: -1 n am ele n: 5 n am e: fil e0 i nu mbe r: 6 n ex t: -1 n am ele n: 5 n am e: fil e1 i nu mbe r: 15 n ex t: -1 n am ele n: 6 n am e: fil e1 0 2 2 2 3 3 1 26 4 xd .l en : 1 x d. len : 1 x d.a dd r1: 0 x d.ad dr 2: 14 73 ne xt: . 9-44 Kernel Internals © Copyright IBM Corp.l en : 1 x d.0 Notes: Adding an internal node Once all the in-line slots have been filled by internal nodes. a separate node block is allocated.le n: 0 xd .a dd r1: 0 x d. that the internal node entries in the inode contain the name of the alphabetical first entry referenced by each of the second level internal nodes.8} 1 x d. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.l en : 1 x d. Note: now.4. After many extra files have been added to the directory. .a dd r2: 26 09 n ext : -1 n ame le n: 8 n ame : fil e17 72 3 f lag : BT_ IN TER NA L n ext in dex : 64 f ree cn t: 59 f ree li st: 7 6 m axs lo t: 12 8 s tbl : {1. . 11 3.2 . 2001.l en : 1 x d. len : x d. the entries from the in-line data slots are moved to this new node.le n: 1 xd ..a dd r1: 0 x d.8} 1 xd .3 .. add r1 : 0 x d.a dd r2: 12 04 n ext : -1 n ame le n: 8 n ame : fil e48 45 x d. 6. Adding an Internal Node BE0070XS4.a dd r2: 19 91 n ext : -1 n ame le n: 9 n ame : fil e13 83 3 x d. . add r2 : 1 47 2 n ex t: -1 n am ele n: 8 n am e: fi le1 01 7 12 6 12 7 12 7 inu mb er: 1 005 7 nex t: -1 nam el en: 9 nam e: fi le 100 52 inu mb er: 1 004 1 nex t: -1 nam el en: 9 nam e: fi le 100 36 Figure 9-25. and each entry in these references the name of the alphabetically first entry in each leaf node.7 . and .0.0 Notes: © Copyright IBM Corp. True or False? The data contents of a file is stored in objects called _____. A JFS2 directory contains directory entries for the . . directories.. The number of inodes in a JFS2 file system is fixed. 2003 Unit 9. 2001.0.3 Student Notebook Uempty Checkpoint There is ____ aggregate per logical volume. An allocation group is at least ____aggregate blocks.V2. True or False? Figure 9-26. Enhanced Journaled File System 9-45 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Checkpoint BE0070XS4. A single extent can be up to ____ in size. 2001. . Exercise BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 9-46 Kernel Internals © Copyright IBM Corp.0 Notes: Turn to your lab workbook and complete exercise eight.Student Notebook Exercise Complete exercise eight Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Use fsdb to examine the structures of directories in a JFS2 file system Figure 9-27. Unit Summary BE0070XS4.0 Notes: © Copyright IBM Corp.0.3 Student Notebook Uempty Unit Summary Aggregate is a pool of space allocated to filesets A fileset is a mountable file system The contents of files and directories are stored in extents Extents are arranged in B+ trees for fast file and directory traversal Figure 9-28. .V2. 2003 Unit 9.0. Enhanced Journaled File System 9-47 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. 2001.Student Notebook 9-48 Kernel Internals © Copyright IBM Corp. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Volume 1 AIX Documentation: Technical Reference: Kernel and Subsystems. Volume 2 © Copyright IBM Corp. you should be able to • List the 3 uses for kernel extensions • Build a kernel extension from scratch • Compose an export file • Create an extended system call How You Will Check Your Progress Accountability: • Exercises using your lab system References AIX Documentation: Kernel Extensions and Device Support Programming Concepts AIX Documentation: Technical Reference: Kernel and Subsystems. What You Should Be Able to Do After completing this unit.V2. 2003 Unit 10. Kernel Extensions 10-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. Kernel Extensions What This Unit Is About This unit describes how the AIX 5L kernel is dynamically extended.0. .3 Student Notebook Uempty Unit 10. 2001. Student Notebook Unit Objectives At the end of this lesson you should be able to: List the 3 uses for kernel extensions Build a kernel extension from scratch Compose an export file Create an extended system call Figure 10-1. Unit Objectives BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. .0 Notes: 10-2 Kernel Internals © Copyright IBM Corp. 0 Notes: Introduction The AIX kernel is dynamically extensible and can be extended by adding additional routines called kernel extensions. A kernel extension could best be described as a dynamically loadable module that adds functionality to the kernel. and ease of system administration to AIX.V2.0. . Kernel extensions add extensibility. Kernel protection domain These modules are extensions to the kernel in the sense that they run within the protection domain of the kernel.3 Student Notebook Uempty Kernel Extensions Kernel extensions can include: Device drivers System calls Virtual file systems Kernel processes Other device driver management routines Kernel extensions run within the protection domain of the kernel Extensions can be loaded into the kernel during: system boot runtime Extensions can be removed at runtime Figure 10-2. User-level code can only access kernel extensions through the system call interface. 2003 Unit 10. configurability. © Copyright IBM Corp. 2001. Kernel Extensions BE0070XS4. Kernel Extensions 10-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Advantages Allowing kernel extensions to be loaded and unloaded allows a system administrator to customize a system for particular environments and applications. The option of loading and unloading kernel extensions at runtime increases system availability and ease of use. pageability. Extensions are loaded and removed from the running kernel using the sysconfig() system call. Rather than bundling all possible options into the kernel at compile time (and creating a large kernel). Such issues as execution environment. and serialization must be taken into account when writing extensions to the kernel. In addition. 2001.Student Notebook Loading extensions Extensions can be added at system boot or while the system is in operation. . 10-4 Kernel Internals © Copyright IBM Corp. development time is reduced since a new kernel does not have to be compiled and installed for each development cycle. kernel extensions allow maximum flexibility. Disadvantages Importing new code into the kernel allows the possibility of an unlimited number of runtime errors to be introduced into the system. path length. V2. 2003 Unit 10. 2001. Kernel Extensions 10-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. .0.0 Notes: Kernel Components The schematic drawing above illustrates the relationship to the kernel. © Copyright IBM Corp.0. Relationship With the Kernel Nucleus BE0070XS4.3 Student Notebook Uempty Relationship With the Kernel Nucleus Commands System Calls Kernel Protection Boundary System Call Interface Virtual File System Device Drivers Extended System Calls Private routines Extended Kernel Mode Experts Extended Kernel Services Nucleus Kernel Services Figure 10-1. Student Notebook Global Kernel Name Space /usr/lib/kernex. Global Kernel Name Space BE0070XS4. Name space The kernel contains many functions and storage locations that are represented by symbols. 2001.exp Global kernel Name space export Core kernel services (/unix) Extended system calls import/ export import export Other kernel extensions import Kernel Extensions Device drivers Extended kernel Services Figure 10-2. Some of these symbols are private to the parts of the kernel that use them. The set of symbols used by the kernel makes up the kernel’s name space. 10-6 Kernel Internals © Copyright IBM Corp. . Some of these symbols are made available for other parts of the kernel and kernel extensions to use. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction This section describes how symbol names are shared between the kernel and kernel extensions. Extensions can make symbols they define visible to other extensions by exporting these symbols.0. Kernel Extensions 10-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Exported symbols The kernel makes symbols available for kernel extensions by exporting them. 2003 Unit 10.V2. they are exported from the /unix binary. 2001. The linker uses the kernel export file to resolve the kernel symbols used by the kernel extension code. Export file The kernel export file has the following format: #!/unix * list of kernel exports devswadd devswchg devswdel devswqry devwrite e_assert_wait e_block_thread e_clear_wait System calls There is an additional file that lists the system calls that are exported from the kernel (/usr/lib/syscalls. Kernel exports file The purpose of the kernel exports file is to list the symbols exported by the kernel.exp. The kernel exports file is imported by the kernel extension when the linker command (ld) is run. In the case of the kernel exports file. Exports file format The first line of the kernel export file indicates the binary where the symbols are being exported from.0. If a kernel extension or other program wants to reference these symbols they must import them. © Copyright IBM Corp. The remainder of the file lists the symbols that are exported. The kernel export file is /usr/lib/kernex. .exp). 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. This system call is only available in the 64-bit kernel. syscall3264 syscall3264 syscall3264 syscall3264 syscall3264 Tag syscall syscall32 syscall64 syscall3264 Description This system call does not pass any arguments by reference (address).Student Notebook The format of the file syscalls. . 10-8 Kernel Internals © Copyright IBM Corp. This system call supports both 32-bit and 64-bit applications. 2001.exp file is similar to the format of the kernel exports file except for an additional tag for each system call. . Here is a fragment of the file syscalls. This descriptor indicates the ability of the system call to interact with 64-bit processes. absinterval access accessx acct adjtime .exp and a description of the tags. This system call is a 32-bit system call and passes 32-bit addresses. This default action can be changed by creating an export file for the extension. Why Export Symbols? BE0070XS4. which makes these symbols available for reference outside the kernel extension.0 Notes: Introduction Kernel extensions can export symbols that are defined by the extension. . 2003 Unit 10.0. Any symbols which are exported by a kernel extension are automatically added to the kernel global name space when the module is explicitly loaded.0. Kernel Extensions 10-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. The export file lists the symbols you want to exported from the kernel extension. © Copyright IBM Corp. Symbols are exported by creating an export file. All symbols within a kernel extension remain private by default. 2001. This means that other kernel extensions cannot use the routines and variables within the extension.V2. An exports file for a kernel extension is used as an import file by other kernel extensions that wish to use the symbols exported by the latter. The format of the exports file is identical to the format of the imports file.3 Student Notebook Uempty Why Export Symbols? To make symbols available for use by other extensions To share private symbols between extensions To define extended system calls to programs that will call them Figure 10-3. The filename specified should be where the file will be installed when the kernel extension is loaded into the kernel. For object files that reference each other's symbols. . each file should use the other's export file as an import file during link-edit. the first line of the export file should be: #!/usr/lib/drivers/pci/scsi_ddpin Extended system calls When a kernel extension creates a new system call. The export file for the object file providing the services should specify the directory path to the object file as the first line in the exports file. If the system call has parameters a different “tag” value such as syscall3264 must be used. 10-10 Kernel Internals © Copyright IBM Corp. 2001. an export file must be created containing the symbol name of the new system call. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Load-time binding is useful where several kernel extensions use common routines provided in a separate object file.Student Notebook Using private routines Kernel extensions can also consist of several separately link-edited object files that are bound at load time. Examples of these files are shown here: #!/unix sys_call_name syscall Note that the above will only work if “sys_call_name” has no parameters. For example. This was explained earlier. Kernel Extensions 10-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Kernel Libraries BE0070XS4. The C library for application programs is a shared object.0 Notes: Introduction Normal C applications are linked with the C library. kernel extensions should not be linked the normal C library.V2. © Copyright IBM Corp.0.a are only a very small subset of those provided in the normal C library. libc. which provides a set of useful programming routines. 2003 Unit 10. These are static libraries (ar format library with static. .o files).3 Student Notebook Uempty Kernel Libraries libcsys. and contain special kernel safe versions of some useful routines such as atoi() and strlen() that are normally found in the regular C library.a. Note that the routines provided by libcsys.a d_align newstack xdump d_roundup secs_to_date date_to_jul timeout date_to_secs timeoutcf untimeout Figure 10-4. For this reason. the kernel extension may link with the libraries libcsys.a. 2001. It is not possible to access this user-level library from within the kernel protection domain.a and libsys. Instead.0.a a641 164a memmove strchr strncat strspn atoi memccpy memset strcmp strncmp strstr bcmp memchr ovbcopy strcpy strncpy strtok bcopy memcmp remque strcspn strpbrk bzero memcpy strcat strlen strrchr libsys. a are available in the AIX online documentation.Student Notebook Kernel libraries Libraries available to kernel extensions are shown in the visual on the previous page. 10-12 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.a and libsys. . 2001. Reference Additional information on the libcsys. 3 Student Notebook Uempty Configuration Routines Kernel extension int module_entry (cmd. Value of cmd CFG_INIT CFG_TERM Description Initialize Terminate Figure 10-5. cmd. Device Driver int dd_entry (dev. These routines can have any name.0 Notes: Introduction Unlike a normal user-level C language application. uiop) dev_t dev. uiop) int cmd.V2.0. 2001. it is normally best to prepend the names of exported symbols with something that indicates the extension which defines the symbol.0. © Copyright IBM Corp. a kernel extension does not have a routine called main. Kernel Extensions 10-13 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. and are automatically exported to the global name space. int cmd. struct uio *uiop. . 2003 Unit 10. Instead it has a configuration routine and one or more entry points. In order to avoid conflicts in the kernel name space. Configuration Routines BE0070XS4. struct uio *uiop. the symbol nfs_config is the entry point routine for the NFS kernel extension. For example. When linking the extension the configuration routine is specified with the -e option of the ld command. 2001. The uio structure is used to pass arguments from the configuration method. The format of the configuration routine is below. 10-14 Kernel Internals © Copyright IBM Corp. These are routines that could be called as a result of a system call or other action that invokes the kernel extension. Entry points Kernel extensions typically define one or more entry points.Student Notebook Configuration routine An extension configuration routine is typically executed shortly after loading the extension. See later section on sysconfig() for details. . The value of cmd depends on the operation the configuration method is being requested to perform. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Kernel Extensions 10-15 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.3 Student Notebook Uempty Compiling and Linking Kernel Extensions Compile cc -q64 -c ext. kernel code should be compiled with either the cc or xlc commands. Compiling and Linking Kernel Extensions BE0070XS4.0. 2001. © Copyright IBM Corp. 2) Link the required object files to create the extension binary. In general.exp \ -bI: /usr/lib/kernex. 2003 Unit 10.0 Notes: Introduction Compiling and linking a kernel extension must be split into two phases: 1) Compile each source file to create an object file.c -o ext64.exp -lsys -lcsys Figure 10-6. The commands call the same compiler core with a different set of options.V2. Compiler command A number of different commands can be used to invoke the compiler on AIX.o -e init_routine -bE:extension. .o -D__64BIT_KERNEL -D_KERNEL -D_KERNSYS Link ld -b64 -o ext64 ext64.0. This value should always be used. This value is automatically defined by the kernel if the -q64 option is specified. The compiler automatically defines a conditional compile variable to indicate which platform the code is being compiled on. Compiling kernel extension or device driver code. Enable kernel symbols in header files. Additional values should be chosen appropriately. and are decided by the developer. .Student Notebook Conditional compiler values One of the main requirements of the compile stage is that the appropriate conditional compile values are used to select the correct code sections. This value should always be used for 64-bit kernel extensions and device drivers. 2001. Some conditional compile values will vary from extension to extension. Other conditional compile values should be used to ensure that the correct sections of system-provided header files are used for environment (32-bit or 64-bit kernel) for which the extension is being built. _KERNSYS _KERNEL __64BIT_KERNEL __64BIT__ 10-16 Kernel Internals © Copyright IBM Corp. This value should always be used. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Code is being compiled in 64-bit mode. Value _POWER_MP Meaning Code is being compiled for a multiprocessor machine. Code is being compiled for a 64-bit kernel. Produce a . Linking Once you have created all of the object files.V2. some are optional. output goes to stdout. Linker option -b64 -b32 -eLabel -lcsys -lsys -oName -bE:FileID -bI:FileID Meaning Generate a 64-bit executable Generate a 32-bit executable Set the entry point of the executable to Label. Generate optimized code. output goes to . 2001. Exports the external symbols listed in the file FileID. 1 is assumed. Compiler option -q64 -qlist -qsource -c -D<name>[=<def>] -M -O -S -v -qcpluscmt -qwarn64 Meaning Generate 64-bit object files. Allow C++ style comments // Enables checking for possible long-to-integer or pointer-to-integer truncation.0.a libraries with the kernel extension.s output file (assembler source) Displays language processing commands as they are invoked by the compiler. Generate information to be included in a “make” description file. Other compiler options may be used to generate additional information about the source files being compiled. Do not send files to the linkage editor. Names the output file Name. Link the libcsys. (-q32 is the default) Produce an object listing. Produce a source listing. use the linker (ld) to create the kernel extension binary.a and libsys.0. The general format of the linker command is: © Copyright IBM Corp. the -q64 option should be used.lst file. Kernel Extensions 10-17 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. In order to compile 64-bit code.3 Student Notebook Uempty Compiler options The default mode for the compiler is 32-bit. Define <name> as in #define directive. If <def> is not specified.lst file. Imports the symbols listed in FileID. 2003 Unit 10. and some are platform dependent. . output goes to . Some linker options will always be used when creating the binary. .Student Notebook ld -e entry_point [import files] [export files] \ -o output_file object1.o object2.o -lcsys -lsys The order of arguments is not important. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 10-18 Kernel Internals © Copyright IBM Corp. Kernel Extensions 10-19 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. if the kernel detects that the file is an ar format library.c -D_KERNEL -D_KERNSYS Link a 32-bit module file using the -b32 linker option.c -D_KERNEL -D_KERNSYS \ -D__64BIT_KERNEL 3 4 5 Build a 64-bit object file using the -b64 linker option. cc -q64 -o ext64. 2001.3 Student Notebook Uempty How to Build a Dual Binary Extension Step 1 2 Action Compile a 32-bit object file using the -q32 compiler option.0. ld -b32 -o ext32 ext32.0 Notes: Introduction Machines with 64-bit hardware can run either the 32-bit kernel or the 64-bit kernel. cc -q32 -o ext32.o -e ext_init \ -bI: /usr/lib/kernex. A kernel extension that supports both 32-bit and 64-bit kernels is packaged as an ar format archive library.exp -lcsys Create an archive of both 32. The library contains both the 32-bit and 64-bit binary versions of the kernel extension. a 64-bit kernel will extract the 64-bit binary from the library.V2.and 64-bit extensions ar -X32_64 -r -v ext ext32 ext64 Figure 10-7.o -c ext. How to Build a Dual Binary Extension BE0070XS4.o -e ext_init \ -bI: /usr/lib/kernex.0. When the extension is loaded. 2003 Unit 10. . For example. © Copyright IBM Corp. ld -b64 -o ext64 ext64.exp -lcsys Build a 64-bit object file from the same source file as step 1.o -c ext. it will load the appropriate binary for the type of kernel. A kernel extension must be of the same binary type as the kernel. Student Notebook Creating a dual binary extension The table/visual on the previous page describes the steps to building a dual binary kernel extension. 10-20 Kernel Internals © Copyright IBM Corp. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.a format. 2001. Note: The name of the library file does not need to be of the libnnn. even on systems running the 64-bit kernel. 2001.V2. 2003 Unit 10. The program is normally a 32-bit executable. © Copyright IBM Corp.0.0.3 Student Notebook Uempty Loading Extensions sysconfig() system call can be used to: Load kernel extensions Unload kernel extensions Invoke the extension's entry point Query the kernel to determine if a extension is loaded loadext() library routine can be used to: Load kernel extensions Unload kernel extensions Query the kernel to determine if an extension is loaded Figure 10-8. . Kernel Extensions 10-21 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: Introduction A user-level program called a Configuration Method is used to load a kernel extension into the kernel. Loading Extensions BE0070XS4. sysconfig() and loadtext() There are two routines available for loading the extension into the kernel as shown in the visual above. unloading and querying When loading. sizeof(cfg_load) ) Cmd Value SYS_KLOAD SYS_SINGLELOAD SYS_QUERYLOAD SYS_KUNLOAD Description Loads a kernel extension object file into kernel memory. 10-22 Kernel Internals © Copyright IBM Corp.Student Notebook sysconfig() . &cfg_load. caddr_t libpath. the caller specifies the kmid. .Loading and Unloading sysconfig ( Cmd. mid_t kmid. and the sysconfig routine returns the kmid. Unloads a previously loaded kernel object file. struct cfg_load { caddr_t path. unloading or querying. the sysconfig() subroutine is passed a pointer to a cfg_load structure (defined in <sys/sysconfig.Loading and Unloading BE0070XS4. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. }. 2001. and the path and libpath are ignored. Loads a kernel extension object file only if it is not already loaded. When unloading. The caller provides the path value. /* ptr to object module pathname */ /* ptr to a substitute libpath */ /* kernel module id (returned) */ Figure 10-9. Determines if a specified kernel object file is loaded.0 Notes: Loading.h>) and one of the commands shown in this table. sysconfig() . The libpath is optional. a pointer to a cfg_kmod structure and the SYS_CFGKMOD command is passed to sysconfig(). © Copyright IBM Corp.3 Student Notebook Uempty sysconfig() .V2. 2003 Unit 10. }. sizeof(cfg_kmod) ) struct cfg_kmod { mid_t kmid.0. the next step is to call the entry point or configuration routine. .0. 2001. caddr_t mdiptr.Configuration sysconfig(SYS_CFGKMOD. int mdilen. The cfg_kmod structure is used with the SYS_CFGKMOD command to call the entry point of a kernel extension. For all extensions other than device drivers. int cmd. &cfg_kmod.Configuration BE0070XS4.0 Notes: Calling the entry point Once the kernel extension has been loaded into the kernel. sysconfig() . /* /* /* /* module ID of module to call command parameter for module pointer to module dependent info length of module dependent info */ */ */ */ Figure 10-10. Kernel Extensions 10-23 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. /* }. module ID of device driver*/ device major/minor number*/ config command code for device */ pointer to DD structure*/ length of DD structure */ Figure 10-11. sizeof(cfg_dd) ) struct cfg_dd { mid_t kmid. . Values are defined in <sys/device.Student Notebook sysconfig() .0 Notes: Device driver entry point The cfg_dd structure is used with the SYS_CFGDD command to the sysconfig() routine to call the entry point of a device driver.Device Driver Configuration BE0070XS4. /* caddr_t ddsptr. Entry point options A number of commands can be passed to the entry point of a kernel extension in the cmd parameter of the cfg_dd or cfg_kmod structure passed to sysconfig()./* int ddslen. /* dev_t devno. &cfg_dd.h> as follows: Value Meaning CFG_INIT CFG_TERM CFG_QVPD Initialize the extension Terminate the extension Query of vital product data 10-24 Kernel Internals © Copyright IBM Corp. 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. /* int cmd. sysconfig() .Device Driver Configuration sysconfig(SYS_CFGDD. 2003 Unit 10. Returns a structure containing the current values of run-time system parameters found in the var structure. Checks the status of a device switch entry in the device switch table. Calls the specified module at its module entry point for configuration purposes.0.3 Student Notebook Uempty CFG_UCODE Value Meaning Download of microcode sysconfig() commands This table provides a complete list of commands for the sysconfig() system call: Cmd Value SYS_KLOAD SYS_SINGLELOAD SYS_QUERYLOAD SYS_KULOAD SYS_QDVSW SYS_CFGDD SYS_CFGKMOD SYS_GETPARMS SYS_SETPARMS Result Loads a kernel extension object file into kernel memory.V2. this flag can be bit-wise OR'ed with the cmd parameter (if the cmd parameter is SYS_KLOAD or SYS_SINGLELOAD). For device drivers. this indicates that the kernel extension does not export 64-bit system calls.0. Loads a kernel extension object file only if it is not already loaded. 2001. Calls the specified device driver configuration routine (module entry point). Kernel Extensions 10-25 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Determines if a specified kernel object file is loaded. Unloads a previously loaded kernel object file. but that all 32-bit system calls also work for 64-bit applications. Sets run-time system parameters from a caller-provided structure. SYS_64BIT © Copyright IBM Corp. this indicates that the device driver can be used by 64-bit applications. . For kernel extensions. When running on the 32-bit kernel. . 2001. The dd_name argument “pci/fred” would result in the loadext routine trying to load the file /usr/lib/drivers/pci/fred into the kernel. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. then it is concatenated to the string “/usr/lib/drivers/”. 10-26 Kernel Internals © Copyright IBM Corp.h> mid_t loadext (dd_name. or a “/”). Figure 10-12. dd_name The dd_name string specifies the pathname of the extension module to load. is often used to perform the task of loading the extension code into the kernel.a library./”. int load. query. it does not start with “. defined in the libcfg. load and unload of kernel extensions. The loadext() Routine BE0070XS4. PCI device drivers are normally stored in the /usr/lib/drivers/pci directory.Student Notebook The loadext() Routine The loadext() routine is defined as follows: #include <sys/types.0 Notes: Introduction The loadext() routine. It uses a boolean logic interface to perform the query. query) char *dd_name. If the dd_name string is not a relative or absolute path name (in other words. For example. “. . load./”. V2.0.0.3 Student Notebook Uempty load and query parameters The load and query parameters are either TRUE or FALSE, and indicate the action to be taken as follows: loadext(“pci/fred”, FALSE, TRUE); /* Query of pci/fred */ loadext(“pci/fred”, TRUE, FALSE); /* SYS_SINGLELOAD of pci/fred */ loadext(“pci/fred”, FALSE, FALSE); /* Unload pci/fred */ Multiple copies If you require multiple copies of a kernel extension to be loaded, you should use the sysconfig interface with the SYS_KLOAD command, since loadext uses SYS_SINGLELOAD, which will only load the extension if it is not already loaded. Calling entry points Even if using the loadext routine to load the kernel extension, you still need to use the sysconfig() routing to call the entry point. © Copyright IBM Corp. 2001, 2003 Unit 10. Kernel Extensions 10-27 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook System Calls User program main(){ . . sys_call(arg1, arg2,..... ) . . } User address space Kernel address space System call code in kernel sys_call( arg1, arg2,.....) { . . . } 1) Switch protection domain from user to kernel 2) Switch to the kernel stack. 3) Execute the system call code. Figure 10-13. System Calls BE0070XS4.0 Notes: Introduction A system call is a function called by user-process code that runs in the kernel protection domain. What is a system call? A system call: - Provides user access to kernel functions and resources - Runs with kernel-mode privileges - Protects the kernel from direct user mode access to the kernel domain 10-28 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Differences from a user-mode function From an external view the mechanism used to call a system call appears the same as calling a user-mode function. There are, however, several significant differences between a user-mode function and a system call. In a system call: - Execution mode is switched from user to kernel mode - Code and data are located in global kernel memory - Cannot use the shared user libraries - Cannot reference symbols outside of the kernel protection domain - System calls can’t be interrupted by signals (must poll for signals) - Can create kernel process to perform asynchronous processing © Copyright IBM Corp. 2001, 2003 Unit 10. Kernel Extensions 10-29 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Sample System Call - Export/Import File question.exp #!/unix question syscall Figure 10-14. Sample System Call - Export/Import File BE0070XS4.0 Notes: Introduction This section describes the creation of a very simple kernel extension that adds a new system call to the kernel. The extended system call created here is called question(). Export and import files When creating an extended system call, the function name of the system call must be exported by the kernel extension and imported by any program calling the system call. Shown above is the export and import file used for this example. Note: The “tag”, syscall, shown above works here because the question() function has no parameters. If it did have parameters we would need to use a “tag” such as syscall3264. 10-30 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Sample System Call - question.c /* question.c */ #include <stdio.h> #include <sys/device.h> question_init(int cmd, struct uio *uio) { switch(cmd) { case CFG_INIT:{ /* do init stuff here */ printf("question_init: command=CFG_INIT\n"); break; } case CFG_TERM:{ /* clean up */ printf("question_init: command=CFG_TERM\n"); break; } default: printf("question_init: command=%d\n",cmd); } return(0); } question() { return(42); /* return the answer to the user */ } Figure 10-15. Sample System Call - question.c BE0070XS4.0 Notes: Example extension This is the kernel extension code. The init routine question_init() is run when the extension is loaded. The function question() is the new system call. The code uses kernel printf() calls. The output from these calls will be displayed on /dev/console if the running kernel image has the kernel debugger loaded. © Copyright IBM Corp. 2001, 2003 Unit 10. Kernel Extensions 10-31 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook Sample System Call - Makefile question: question.c cc -q32 -D_KERNEL -D_KERNSYS -o question32.o \ -c question.c ld -b32 -o question32 question32.o -e question_init \ -bE:question.exp -bI:/usr/lib/kernex.imp cc -q64 -D_KERNEL -D_KERNSYS -D_64BIT_KERNEL \ -o question64.o -c question.c ld -b64 -o question64 question64.o -e question_init \ -bE:question.exp -bI:/usr/lib/kernex.imp rm -f question ar -X32_64 -r -v question question32 question64 Figure 10-16. Sample System Call - Makefile BE0070XS4.0 Notes: System call makefile This is the Makefile used to build the kernel extension. In this example both 32-bit and 64-bit objects are built. The two objects are archived (ar) into a single file. When loaded into the kernel, the object matching the kernel type will be extracted from the archive and loaded. 10-32 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. V2.0.0.3 Student Notebook Uempty Argument Passing 64-bit User Process: 32-bit User Process: 32-bit User Process: 64-bit User Process: sys_call(int * ) sys_call(int * ) sys_call(int * ) sys_call(int * ) User mode Kernel mode 32-bit pointers are zero extended Low-order 32 bits only are passed. 64-bit kernel 32-bit kernel sys_call(int * ) sys_call(int * ) Figure 10-17. Argument Passing BE0070XS4.0 Notes: Introduction System calls can accept up to 8 arguments. Often these arguments are 64-bits long or pointers to buffers in the user’s address space. Because AIX supports a mix of 32-bit and 64-bit environments, care must be taken when processing 64-bit arguments. 64-bit kernels When running a 64-bit kernel, pointer arguments passed from a 32-bit process will be zero extended. This case requires no special handling. 32-bit kernels In the 32-bit kernel, a kernel service that accepts a pointer as a parameter expects a 32-bit value. When dealing with a 64-bit user process however, things are different. Although the kernel expects (and indeed receives) 32-bit values as the arguments, the © Copyright IBM Corp. 2001, 2003 Unit 10. Kernel Extensions 10-33 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Student Notebook parameters in the user process itself are 64-bit. The system call handler copies the low-order 32-bits of all parameters onto the kernel stack it creates before entering the system call. The high-order 32-bits are stored elsewhere. A new kernel service called get64bitparm() is used to retrieve the stored high-order 32-bits and reconstruct the 64-bit value inside the kernel. get64bitparm() The get64bitparm() kernel service is defined in the header file <sys/remap.h> as follows: unsigned long long get64bitparm(unsigned long low32, int parmnum); The get64bitparm() kernel service is used to reconstruct a 64-bit long pointer that was passed (and truncated) from a 64-bit user process to the 32-bit kernel. The 64-bit system call handler stores the high order 32-bits of all system call arguments. Once the 64-bit value has been re-constructed, the kernel service may use it for whatever purpose it deems necessary. In the following material we demonstrate the use of this service in forming a 64-bit address which is then used to read parameter data from a 64-bit process into a 32-bit kernel extension. In this case the get64bitparm() call is used to obtain a user space address which is then accessed by the copyin64() kernel service. 10-34 Kernel Internals © Copyright IBM Corp. 2001, 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Kernel Extensions 10-35 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Kernel extensions reside in the kernel protection domain and cannot directly access user space memory. 2001. Overview User applications reside in the user protection domain and cannot directly access kernel memory. kernel_buffer Figure 10-18.0 Notes: Introduction Within the kernel. © Copyright IBM Corp.0. int count ){ copyin(buffer.buffer. 2003 Unit 10.3 Student Notebook Uempty User Memory Access sys_call( &user_buffer.&kernel_buffer. copyout(&kernel_buffer. .count). and from kernel space to user space.0. sizeof(user_buffer) ).h>. List of services The following services can be used to transfer data between user and kernel address space. user_buffer User address space Kernel address space copyout copyin sys_call( void * buffer. User Memory Access BE0070XS4. count). Prototypes are defined for the services in the header file <sys/uio.V2. a number of services can be used to copy data from user space to kernel space. copyout64(char * kaddr. int count) copyinstr64(unsigned long long uaddr. int val). suword64(unsigned long long uaddr. size_t count) . Both the user structure and the IS64U macro are defined in /usr/include/sys/user.Student Notebook Copy data from user to kernel space Use the following services to copy data from user to kernel space: . char val). void * kaddr. . int fuword(void *uaddr). . fubyte64(unsigned long long uaddr).copies count bytes of data copyout(void * kaddr. The macro evaluates to true if the calling process is 64-bit. unsigned long long uaddr. void * uaddr. uint *actual). size_t count) . 2001. copyin64(unsigned long long uaddr. Copy data from kernel to user space Use the following services to copy data from kernel to user space: . fuword64(unsigned long long uaddr). 32-bit kernels Additional services can be used by 32-bit kernels when dealing with a 64-bit user process.store a byte and word respectively subyte(void *uaddr. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. size_t *actual).fetch a byte and word respectively int fubyte(void *uaddr). int count) subyte64(unsigned long long uaddr.copies a character string (including the null character) copyinstr(void * uaddr. size_t max. int val). caddr_t kaddr. void * kaddr. uchar val). char * kaddr.copies count bytes of data copyin (void * uaddr. It checks the U_64bit member of the user structure described earlier. 10-36 Kernel Internals © Copyright IBM Corp. suword(void *uaddr.h. uint max. IS64U The macro IS64U can be used by system call code to determine if the calling process is 64-bit or 32-bit. } else #endif { /* this path is taken if 32-bit kernel & 32-bit process ** OR any size process if running in 64-bit kernel */ copyin(buf. © Copyright IBM Corp. .3 Student Notebook Uempty 64-bit argument code sample The following code sample shows the logic used in a kernel extension that can handle calls from 64-bit user applications when running in the 32-bit kernel. } else #endif { rc = copyout(localmem. int rc.0.V2.localmem. } .buf. lsize = get64bitparm( size. . #ifndef __64BIT_KERNEL /* 32-bit kernel logic */ unsigned long long lbuf.count). long size) { void *localmem = xmalloc(count. 2).kernel_heap). size = (long) lsize. int count.lbuf. 0). 2003 Unit 10. Kernel Extensions 10-37 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.2.count). /* body of kernel service */ #ifndef __64BIT_KERNEL /* 32-bit kernel logic */ if(is64u) { /* 32-bit kernel & caller is a 64-bit process */ rc = copyout64(localmem. char is64u = IS64U. if(is64u) { /* 32-bit kernel & caller is a 64-bit process */ lbuf = get64bitparm( (unsigned long) buf. } if (rc != 0 ) { .0.count).count). copyin64(lbuf. 2001.localmemm. int myservice(void * buf. . lsize. . . . Figure 10-19.Student Notebook Checkpoint Kernel extensions can be loaded at _____ _____ and during _______. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. A kernel extension can be compiled and linked like a regular user application. True or False? A kernel extension must supply a routine called main(). The ________ system call is used to invoke the entry point of a kernel extension. F_____ S______ and S______ C______. Checkpoint BE0070XS4. True or False? Kernel extensions are used mainly for D_____ D_____. 2001.0 Notes: 10-38 Kernel Internals © Copyright IBM Corp. link and load a kernel extension Write your own system call Write a kernel extension that creates kernel processes Create your own ps command Figure 10-20. . The consequences of invoking a kernel service with incorrect arguments include data corruption. or even causing the system to crash. In general.V2. This is in stark contrast to similar problems in a user-level application which normally would result in the application terminating because of a SIGSEGV signal. kernel services perform very little (or no) checking of arguments for error conditions. Kernel Extensions 10-39 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Unit 10.0. 2001. Exercise BE0070XS4.0. Turn to your lab workbook and complete exercise ten.0 Notes: Developing code for the kernel environment is very different compared with developing a user-level application.3 Student Notebook Uempty Exercise Complete exercise ten Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Compile. © Copyright IBM Corp. 2001. file systems and extended system calls Kernel extensions can be loaded at boot time or runtime. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Unit Summary Kernel extensions are used to implement device drivers. Unit Summary BE0070XS4.0 Notes: 10-40 Kernel Internals © Copyright IBM Corp. and can be unloaded at runtime Kernel extensions require special compile and link steps Kernel extensions need to match the binary type of the running kernel Kernel extension code must take into account that the kernel is pageable Figure 10-21. . . © Copyright IBM Corp. The AIX kernel is preemptable.0. Checkpoint Solutions Unit 1 Checkpoint Solutions 1. and only runs on 64-bit hardware. The 32-bit kernel supports 64-bit user applications when running on 64-bit hardware. 2003 Appendix A.V2. pageable and dynamically extendable. 2.3 Student Notebook Uempty Appendix A. 3. Checkpoint Solutions A-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. The kernel is the base program of the operating system. 2001. The 64-bit AIX kernel supports only 64-bit kernel extensions. 4. 5. The processor runs interrupt routines in kernel mode. The value of the dbg_avail kernel variable indicates how the debugger is loaded. 2001. The system dump image contains only selected areas of kernel memory. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.Student Notebook Unit 2 Checkpoint Solutions 1. kdb is used for system image analysis. True or False? False. A system dump image contains everything that was in the kernel at the time of the crash. A-2 Kernel Internals © Copyright IBM Corp. 4. KDB is used for live system debugging. 3. 2. The process table is an array of pvproc structures. . 5.0. A new thread is created by the thread create() system call. © Copyright IBM Corp.V2. AIX provides three programming models for user threads.3 Student Notebook Uempty Unit 3 Checkpoint Solutions 1. 2. 6. Checkpoint Solutions A-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. True or False? True. All process IDs (except pid 1) are even.0. 3. 4. 2003 Appendix A. A thread holding a lock may have its priority boosted. 2001. A thread table slot number is included in a thread ID. The shared library text segments are shared. 5. 3. . but the data segments are private. A segment can be up to 256MB in size. True or False? False. 6. A-4 Kernel Internals © Copyright IBM Corp. Shared library data segments can be shared between processes. 2. The 32-bit user address space layout is the same s the 32-bit kernel address space layout. A 32-bit effective address contains a 4-bit segment number. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2001. AIX divides physical memory into frames. The virtual memory manager provides each process with its own effective address space. True or False? False. 4.Student Notebook Unit 4 Checkpoint Solutions 1. 2001. © Copyright IBM Corp. . The system hardware maintains a table of recently referenced virtual to physical address translations. 2003 Appendix A. Checkpoint Solutions A-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. A page fault when interrupts are disabled will cause the system to crash.0. 4. Each working storage has an XPT. 5.0. The PSALLOC environment variable can be used to change the paging space policy of a process. The Software Page Frame Table contains information on all pages resident in physical memory.V2. 6. 2.3 Student Notebook Uempty Unit 5 Checkpoint Solutions 1. 3. A SIGDANGER signal is sent to every process when the free paging space drops below the warning threshold. A real address is not equivalent to a physical address in the partitioned environment. True or False? The statement is False. A-6 Kernel Internals © Copyright IBM Corp. 4) In a partitioned environment. 5) Any piece of code can make hypervisor calls. depending on the amount of memory allocated to the partition. a real address is the same as a physical address. 6) Which physical addresses in the system can a partition access? A partition can access the PMBs allocated to the partition. AIX 5.2 and Linux need 256MB. (and with hypervisor assistance) the partition's own page table. and the TCE windows for the allocated I/O slots. True or False? The statement is False. True or False? The statement is False.1 requires 256MB.Student Notebook Unit 6 Checkpoint Solutions 1) What processor features are required in a partitioned system? RMO. 2001. Only kernel code can make hypervisor calls. 1GB or 16GB. . AIX 5. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. RML and LPI registers are needed in a partitioned system 2) Memory is allocated to partitions in units of ____256___MB. 3) All partitions have the same real mode memory requirements. There is one gfs structure for each mounted file system. The kdb subcommand volgrp and the AIX command lsvg both reflect volume group information. Checkpoint Solutions A-7 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 2003 Appendix A. . True or False? False. The three kernel structures volgrp. lvol and pvol are used to track LVM volume group. © Copyright IBM Corp.0.V2. 2001. respectively. The kernel maintains a vfs structure and a vmount structure for each mounted file system.3 Student Notebook Uempty Unit 7 Checkpoint Solutions (1 of 2) Each user process contains a private File Descriptor Table. logical volume and physical volume data. There is one gfs structure for each file system type registered with the kernel.0. The reason for this is that gnode structures are of one format. These structures are of different formats. Why? The reason is that ls is giving us the root inode of the /usr filesystem. True or False? True. They are imbedded in the corresponding inode/specnode/rnode structure for the file in question. Each vnode for an open file points to a gnode structure. not the inode of the /usr directory in the /(root) filesystem. Every open file in a filesystem is represented by exactly one file structure. So. To obtain this directory inode we need to follow the vfs_mntdover pointer in the /usr filesystem vfs structure. .Student Notebook Unit 7 (continued) Checkpoint Solutions (2 of 2) There is one vmount/vfs structure pair for each mounted filesystem. There is one file structure (system file table entry) for each unique open() of a file. True or False? False. 2001. A-8 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. a given file may be represented by several file structures. The inode number given by ls -id/usr is shown as 2. This will point us to the vnode structure of directory /usr in the root filesystem. which contains the directory inode number. . This includes such items as open count and in-core inode state. The root inode number is always 2. The root inode number of a filesystem is always 1. True or False? False. An allocation group contains disk inodes and fragments. 2003 Appendix A. True or False? False.0. The last 128 bytes of an in core JFS inode is a copy of the disk inode.3 Student Notebook Uempty Unit 8 Checkpoint Solutions 1. 2. 5. The basic allocation unit is a fragment. 2001. Checkpoint Solutions A-9 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3. JFS itself does copy operations and relies on VMM to do the actual I/O operations. JFS maps user data blocks and directory information into virtual memory. The first part of an in core JFS inode contains data relevant only when the associated object is being referenced.V2. This is a reason for JFS I/O efficiency. © Copyright IBM Corp. True or False? True.0. True or False? True. The basic allocation unit in JFS is a disk block. 4. A single extent can be up to 224-1 in size. A JFS2 directory contains directory entries for the .. directories.Student Notebook Unit 9 Checkpoint Solutions There is one aggregate per logical volume. A-10 Kernel Internals © Copyright IBM Corp. The data contents of a file is stored in objects called extents. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.. An allocation group is at least 8192 aggregate blocks. . The number of inodes in a JFS2 file system is fixed. True or False? False. is contained in the inode of the directory. and . 2001. and . True or False? False. The information for . .V2. © Copyright IBM Corp. File Systems and System Calls. True or False? False.3 Student Notebook Uempty Unit 10 Checkpoint Solutions Kernel extensions can be loaded at system boot and during runtime. The sysconfig system call is used to invoke the entry point of a kernel extension. A kernel extension can be compiled and linked like a regular user application. Checkpoint Solutions A-11 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. A kernel extension must supply a routine called main(). True or False? False. 2003 Appendix A.0. 2001. Kernel extensions are used mainly for Device Drivers.0. Student Notebook A-12 Kernel Internals © Copyright IBM Corp. . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. KI Crash Dump What This Unit Is About This unit describes how to configure and perform system dumps on a system running a version of the AIX 5L operating system. What You Should Be Able to Do After completing this unit.0. you should be able to: • Configure a system to perform a system dump • Test the system dump configuration of a system • Validate a dump file How You Will Check Your Progress Accountability: • Exercises using your lab system References © Copyright IBM Corp. 2001.V2.3 Student Notebook Uempty Appendix B. 2003 Appendix B.0. KI Crash Dump B-1 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. . Student Notebook Unit Objectives At the end of this unit you should be able to: Configure a system to perform a system dump Test the system dump configuration of a system Validate a dump file Figure B-1. 2001.0 Notes: B-2 Kernel Internals © Copyright IBM Corp. . 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. Unit Objectives BE0070XS4. 0 Notes: System Dump Facility in AIX 5L What is crash dump? A system crash dump is a snapshot of the operating system state at the time of the crash or manually initiated dump.3 Student Notebook Uempty Crash Dumps What is a crash dump? When is a crash dump created? What is a crash dump used for? Figure B-2.0.V2. 2003 Appendix B. the system dump facility automatically copies selected areas of kernel data to the primary dump device. When a manually-initiated or unexpected system halt occurs. Crash Dumps BE0070XS4. such as unexpected or unrecoverable kernel mode exceptions. These areas include kernel memory as well as other areas registered in the Master Dump Table by kernel modules or kernel extensions. KI Crash Dump B-3 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. It can also be initiated by the system administrator when the system is hung. When is a crash dump created? An AIX 5L system will generate a system crash dump when encountering a severe system error. 2001.0. © Copyright IBM Corp. . . 2001. B-4 Kernel Internals © Copyright IBM Corp. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM. the system will be booted and returned to production. The dump is then typically submitted to IBM for analysis.Student Notebook What is a crash dump used for? The system dump facility provides a mechanism to capture sufficient information about the AIX 5L kernel for later expert analysis. Once the preserved image is written to disk. 2003 Appendix B.V2. . copycore is called by rc.0 Notes: System dump process Introduction The process of performing a system dump is illustrated in the chart. AIX 5L is booted and the memory image is moved to a permanent location in the /var/adm/ras directory. The process involves two stages. Process Flow BE0070XS4. In stage one. the contents of memory is copied to a temporary disk location.0. © Copyright IBM Corp.boot System Panics System is booted Stage 2 Memory dumper is run Memory is copied to disk location specified in SWservAt ODM object class Figure B-3. KI Crash Dump B-5 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0. 2001. In stage two.3 Student Notebook Uempty Process Flow AIX 5L in production Stage 1 copycore copies dump into /var/adm/ras. About This Exercise BE0070XS4. . 2001. 2003 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.0 Notes: B-6 Kernel Internals © Copyright IBM Corp.Student Notebook Exercise Complete exercise A Consists of theory and hands-on Ask questions at any time Activities are identified by a What you will do: Learn about the sysdumpdev command Configure your lab system to perform a system dump Test the crash dump configuration Verify you have obtained a successful system dump Figure B-4. V2.0 backpg Back page . ® .