MODEL BASED TEST GENERATION WHAT¶S A MODEL? A model: ´ Is an abstraction or simplified representation of the system from a particular perspective ´ Supports investigation, discovery, explanation, prediction, or construction ´ May be expressed as a description, table, graphical diagram, or quantitative mathematical model ´ Is not necessarily comprehensive 2 MODELS IN EVERYDAY LIFE 3 EXAMPLES OF MODELS ´ ´ ´ ´ ´ ´ ´ ´ ´ Flow Charts Data Flow Diagrams Entity- Relationship Diagrams State Diagrams Deployment Diagrams Class Diagrams Use Cases Activity Diagrams State Transition Tables 4 MODEL BASED TEST GENERATION Formal Model set of Generator Test Generation Directives Test Cases sequence of stimuli expected responses 5 MODELING Product Idea / Specification What product are you Modeling building? Validation Are you building the right product? Specification AsmL Refinement Verification Are you building the product right ? 6 Implementation C/C++/C# FSM: START STATE & ACTION Start state (Entry Point) Files Shown: 0 File Select: Not Selected Exit Point: False 7 start OTHER STATES 8 SIMPLE FSM MODEL (STATE MODEL) 9 WALK EVERY STATE-CHANGE Create Invert Selection Invert Selection Delete 10 TEST EVERY WALK < 5 STEPS A: Invert Selection B: Create; Delete C: Create; Invert Selection; Invert Selection; Delete 11 MODEL BASED TEST GENERATION TOOLS T-VEC(Test Vector Generation System) ´ TestMaster ´ UniTesk (Unified Testing and spec toolkit) ´ GOCHA Techbeans ´ ASML Etc«.. ´ 12 THE ASML MODELING TOOLKIT Abstract state machine Language is a powerful modeling language AsmL provides « « « « « Authoring via MS Word and XML Simulation and interoperability via COM and .NET Automatic test oracle Semi-automatic test case generation Integration with Microsoft·s test platform 13 ASML: GETTING STARTED, µSORT¶ EXAMPLE var A as Seq of Integer = [] Non-determinism Swap() choose i in indices(A), j in indices(A) where i < j and A(i) > A(j) A(i) := A(j) Parallelism A(j) := A(i) A = [2,3,1] A = [1,3,2] A = [2,1,3] A = [1,2,3] sort() step until fixpoint swap() 14 ASML: PREDEFINED DATA TYPES Sets menu = {"ham", "cheese", "blt"} Sequences random = [3, 6, 7, 1, 5] //as Seq of Integer //as Maps directory as Map of String to Integer = {³police´ -> 112 , ³info´ -> 118} 118} Tuples nameAndAge = (³John´,39) //(String,Integer) ³John´,39) Integer) 15 USING MODELS FOR TEST Model Generates User Info Provides expected results for Test Cases Are run by Pass Test Oracle No pass Provides actual results for Implementation 16 ASML-BASED TEST HARNESS Any client Side-by-side runs Discrepancies automatically flagged Test harness AsmL model Implementation under test 17 ASML TEST CASE GENERATION - 1 Solves two problems How to find ´interestingµ parameter values and combinations How to find ´interestingµ call sequences Approach Use information derived from the model to optimize the number of test cases 18 ASML TEST CASE GENERATION 2 Divide state space according to state predicates System ´coversµ the state predicates Driven by input generator (parameter selection) or Driven by enabledness (call sequences) Tester provides inputs, gives interesting state predicates, directs the search 19 THE DUAL ROLE OF ASM MODELS Testing harness AsmL Model M Call next action IUT Get state view Guided generation (Param Select with Feedback) Reduce FSM FSM tool Pass S Is S valid according to M? AsmL Model M view Test cases Fail: witness view 20 ASML TEST CASE GENERATION - 3 Generate only as many test cases as needed to have full coverage of the specification Generator Input Feedback Model Coverage criteria Branch: Every ³branch´ of the spec must be taken Modified condition: independent atomic conditions Off-by-One: off-by-one errors Off-byoff-by Hyperstate coverage: Every abstract state Hyperstate must be visited 21 ASML TEST (ASMLT) asmlT can be used to: 1. Find interesting sequences of method calls ¹ finds the shortest route in a generated FSM that uses every edge to traverse the states 2. Find interesting parameters for each method call Access Driven Filtering (ADF) ¹ you write a predicate (boolean function) to decide if a given parameter is interesting 3. Perform a conformance test Compare Model with Implementation 22 REFERENCES ´ Model-Based Test Generation Tools, Alan Hartman, AGEDIS CONSORTIUM http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf Finite State Model-Based Testing on a Shoestring, Harry Robinson http://www.model-based-testing.org/shoestring.htm Model Based Testing, L. Apfelbaum, et al, Software Quality Week Conference in May, 1997. http://www.model-based-testing.org/sqw97.pdf Abstract State Machine Language, Microsoft Research http://research.microsoft.com/fse/asml/ The AsmL Test Generator tool, Microsoft Research http://research.microsoft.com/fse/AsmL/doc/AsmLTester.html ´ ´ ´ ´ ´ Intelligent Test Automation, Harry Robinson http://www.model-based-testing.org/robinson.pdf Software Testing & Quality Engineering (STQE) magazine Sept/Oct 2000 23 Queries..?? 24