1CALIFONIA STATE UNIVERSITY MONTEREY BAY UCAT Training Website CAPSTONE PROPOSAL Submitted in partial satisfaction of requirements of the degree of MASTER OF SCIENCE in Instructional Science and Technology Bassel El Ahmadieh 7 December 2017 Capstone Approvals: (At least one advisor and capstone instructor should approve) ___________________________ _____________ Advisor Name Signature Date __________________________ _____________ Capstone Instructor Name Signature Date 2 Table of Contents Executive Summary……………………………………………………………………………4 Introduction and Background…….……………………………………………………………4 Problem description…...………………………………………………….……..………….5 Target Audience.…………………………………………………………………………….6 Setting…………. ………...…………………………………………………………….…...6 Literature Review.............................................................................................................…...7 Solution ……………………...….………………...………………………….…......................9 Goals…………….……………...……………...……………………………….…….…….9 Learning objectives ………………….……………….……………………………...…….10 Proposed solution………….…………………………….……………………………...….11 Learning Theories….………….………………...………...……………………………….11 Principles…………………………………………………….………………….………….12 Strategy and Role of the trainer…………………………………...……………………….12 Delivery system and Media decisions………….………………………………….……….13 Challenges……………………………………….….…………………………...…...……14 Method………………………………………………….…………………………….…….…14 Deliverables……………………………….………….…………………………….…. ….15 Resources…….………………………….……………………………………………....……16 Time and Cost ...………………………………………………….………………….…….16 Timeline………….....…………….…………………………………………………….…….17 Evaluation and Testing………………..………………………………………..….………….18 Formative Evaluation………………………...……………………………………………….18 3 Usability Testing………………………………….……………………………………….18 Pre- and Post-Testing………...………….……………………………………...…………21 Results…………….………...…………………………………………………...…………21 Summative evaluation………….………………………………………………………….22 Conclusion…………………...…………………………………………………...…………23 References…………………………………………………………………………………….25 Appendices……………………………………………………………….……...………….27 Appendix A………………………………………………………………………...…………27 Appendix B………………………………………………………………………...…………29 Appendix C………………………………………………………………………...…………31 Appendix D………………………………………………………………………...…………33 Appendix E………………………………………………………………………...…………35 Appendix F………………………………………………………………………...…………37 Release Form………………………………………………………………………...………39 4 Executive Summary UCAT Training Website trains UMA teachers into how to facilitate the integration of Universal Curriculum Assessment Tool (UCAT) into teaching Levantine dialect by developing and conducting professional development online sessions on infusing UCAT into the core areas of the academic curriculum. This provides insight on ways to increase teachers’ use of UCAT as it positively impacts students’ performance in the classroom. Using UCAT in the classroom keeps students alert and eager to learn. Plenty of studies show that technology has an impact on the instructional process since it stimulates students’ thinking and makes them engage in the learning process. UCAT helps students create, analyze and use critical thinking which makes learning more fun and meaningful to their daily life experience. Moreover, The UCAT Training Website consists of pre-test, modules, post-test and survey Module 1: UCAT Terminology; Module 2: Creating listening and FLO activities on UCAT; Module 3: Creating reading activities on UCAT; Module 4: Design a lesson on UCAT. Ten foreign language teachers who are teaching Levantine dialect, a spoken Arabic of Lebanon, Syria, Jordan, and Palestine, volunteered to take this training. The teachers worked on this training during their spare time at work. The foreign language teachers first took the pre-test in order to be tested on their information about UCAT terminology, UCAT features, development of listening, FLO and reading activities. These questions look the same as the final test in UCAT module. The foreign language teachers took the post-test after the training which is exactly the same as the pre-test. Teachers who took the final evaluation survey that was hosted on Google Forms provided effective feedback. The feedback from the foreign language teachers was collected and analyzed for the purpose of improving the UCAT Training Website. Introduction and Background Technology in the classroom assists in students’ autonomous learning and students perform inquiry-based instructional activities and a variety of other tasks (Roschelle & Pea, 2002). The presence of technology may help students excel more in the learning process and 5 enable teachers to reach all the students. Technology helps teachers create instructional strategies that address students’ various learning needs. For instance, teachers try to engage students by preparing visual representations for instruction such as PowerPoint presentations containing pictures, charts, graphs, maps or videos to assist the visual learner in grasping better the information. Thus, teachers can individualize instruction by choosing materials that benefit a variety of academic levels and learning preferences. Since technology plays a major role in bettering student learning experiences, improving learner productivity and enabling teachers to deliver the instruction effectively, I designed and developed Universal Curriculum Assessment Tool (UCAT) Training Website that trains teachers at the Defense Language Institute (DLIFLC) how to use UCAT to engage students in the learning process and motivate them to achieve better results. The UCAT Training Website includes instructional training into how to use UCAT in teaching DLIFLC students a second language. Moreover, the UCAT Training Website instructs teachers by showing tutorials on how to use this powerful tool to create, develop and design activities, tasks, quizzes, assignments, and lessons to increase students’ proficiency. Problem description The majority of teachers at our school are not using UCAT in the teaching process simply because they don’t know how to use this powerful tool. UCAT is an online curriculum delivery system specifically tailored to meet the needs of DLIFLC faculty and students. It allows foreign language teachers to develop and design online interactive activities that engage students. The format of UCAT activities include categorization, true or false, multiple choice, drag and drop, fill in the blanks, short answer, transcription and matching. Teachers have expressed their desire to learn about this valuable tool since this will assist in solving many training problems and aid in achieving their goals. Some teachers believe that lack of using technology in the classroom may lead to having less interested students who do not pay attention in the classroom and do not study at home. Moreover, the needs assessment indicates that UCAT is a good training tool, and if used wisely can assist in solving many training problems. This will ultimately increase student success rates. Teachers and students at Defense Language Institute (DLI) can use UCAT in the training process to increase Institute annual productivity. Nevertheless, the majority of teachers don’t benefit from this tool that may help students pass the Defense Language 6 Proficiency Test (DLPT), which is an exam that measures students’ second language skills in listening and reading. To learn more about this issue, I conducted an interview with the chairperson asking questions about the technology tools and software applications that the teachers use inside and outside the classroom. Moreover, I prepared a questionnaire to be answered by the teachers in our department for the purpose of studying and analyzing how the teachers use technology tools inside and outside the classroom to increase students’ proficiency. Target Audience The audience is adults whose ages range between 30 and 80 years old. The gender percentage is as follows: 55% females and 45% males. The audience is foreign language instructors teaching Levantine dialect, a version of Arabic spoken by people of Lebanon, Syria, Jordan, and Palestine, at DLIFLC. The audience is originally either from Lebanon, Syria, Jordan, Iraq or Palestine living in the United States of America as permanent residents or American citizens or holders of work visas. The data collected from the survey shows that the majority of the teachers at our school are interested in learning about UCAT. The teachers want to try UCAT because this will allow them to implement new technology in the classroom and to develop activities that promote critical thinking and autonomous learning. Employing UCAT will engage students in the learning process and help them excel in their learning to achieve good results at the final exams. The challenges that I might face when conducting training to the foreign language teachers are that some teachers still have difficulties in learning new technology tools and others feel that learning new software applications is wasting time because there is no time to implement these tools in the classroom. Setting DLIFLC’s goal in 2017-2018 is to get students pass the DLPT reading and listening exam at a level 2+. The DLIFLC administration’s goal for the Oral Proficiency Interview exam (OPI) is level 2. Solving the motivation problem and finding a means to improve the type and accessibility of instruction pave the road to more language training. To reach DLIFLC’s goals and to assist in solving training problems, DLIFLC administration 7 distributed in December of 2013 iPads and MacBooks to teachers and students to be used inside and outside the classroom. However, it is not enough to distribute iPads and MacBooks to teachers and students. It is essential to continuously train teachers on how to use these powerful tools in the training process. For instance, there are many benefits when implementing UCAT in the learning process since this platform acts as an effective instructional tool in the hand of teachers that promote students’ motivation, cooperation, communication, and organization. In addition, UCAT assists in students’ autonomous learning and students perform inquiry-based instructional activities and a variety of other tasks. With the presence of UCAT, students excel more in the learning process and teachers are able to reach all the students. Thus, teachers can individualize instruction by choosing materials that benefit a variety of academic levels and learning preferences. Furthermore, this can aid the teacher in delivering the information more easily and solving the problem faster. UCAT Training Website trains foreign language teachers in how to use and implement UCAT in teaching a second language. UCAT offers a wide variety of elements and possibilities for making the authentic materials more engaging and educational. UCAT can help increase class participation by giving students more confidence and helping them improve their listening, speaking, reading and writings skills. The UCAT Training Website contains five sections. The first module includes a module that introduces UCAT Terminology. The second module instructs foreign language teachers on how to use UCAT to create online listening and FLO activities for students. The third module demonstrates and trains foreign language teachers in how to use UCAT to develop and design reading activities on UCAT to improve students’ reading skills. The fourth module instructs teachers on how to design an online lesson on UCAT. There is an evaluative assessment at the end of each module that examines the trainees on the topic discussed by asking different types of questions to ensure that the learners understand the topic. The fifth section gives trainees the opportunity to evaluate the UCAT Training Website by providing feedback to continuously improve this project. Literature Review Using online software application in the classroom keeps students alert and eager to learn. Studies show that technology has an impact on the instructional process since it 8 stimulates the students’ thinking and makes them engage in the learning process. Many theorists have studied the relationship between education and technology; especially after the positive effect that computers and software have to our lives. The online delivery system helps teachers and students to perform better in the training process by motivating students to excel in their learning and studies (Biggs, 2011). Use of the online interaction solves many training problems that the teachers face in the classroom especially when it comes to time and quality. Utilizing computerized applications in education provides cognitive implications for reorganizing mental process and thus helps students in using their critical thinking by planning and solving problems (Pea & Kurland, 1984). Moreover, online interaction helps students to perform all language skills encouraging the collaborative learning methodology in which students share information together via the internet (Berge & Collins, 1995). The students work as a team to solve problems or to answer reading comprehension questions, which encourage computer collaborative learning through synchronous and asynchronous participation. The online delivery system acts as an effective instructional tool in the hand of teachers that promotes students’ motivation, cooperation, communication and organization (Sherry, 1996). Technology in the classroom assists in students’ autonomous learning and students perform inquiry-based instructional activities and a variety of other tasks (Hmelo- Silver, 2004). With the presence of online interaction, students excel more in the learning process (Palloff & Pratt, 1999). The online delivery system helps teachers to create teaching strategies that address students’ learning needs. For instance, teachers try to engage students by preparing interactive activities such as multiple choice, true or false, matching, and categorization. Thus, teachers can individualize instruction by choosing materials that benefit a variety of academic levels and learning preferences. Furthermore, the online delivery system can aid the teacher in delivering the information more easily and solving the problem faster (Mayer, 1998). Teachers using online interaction are able to create, design, and develop activities that address students’ various learning abilities and levels. In addition, the online delivery system provides the teachers with the option of adapting lectures, curriculum and activities to best meet the students learning needs (Harasim, 1995). The online delivery systems encourage the use of collaborative learning methodology inside and outside the classroom. “Classrooms should become classrooms 9 more characterized by computer-supported collaborative learning. Students will work towards shared understanding in groups. Students will build joint representations of their knowledge. To enhance understanding, students will point to, annotate, and use external representations in diverse sense-making and discourse practices” (Roschelle & Pea, 2002). Using online learning in training is a new trend in instructional technology in which it acts as effective instructional tools in the hand of teachers and learners that promote learners’ motivation, cooperation, communication, and organization. With the presence of online learning, learners excel more in the learning process and teachers are able to reach all the students (Clark & Mayer,2016). Technology tools make it easy for teachers to create teaching strategies that address learners’ various learning abilities and needs. For instance, teacher designs and develops different online activities depending on learners’ interests, needs, and cultural background by using various online software applications. Therefore, online training is more effective if used wisely with the implementation of effective teaching strategies that address various learners’ needs, abilities and interests. Solution Goals of the Project The purpose of the UCAT Training Website is to train foreign language teachers who teach Levantine dialect, a spoken language of Lebanon, Syria, Jordan, and Palestine, how to use UCAT in teaching the second language that assists in increasing student motivation and proficiency. This website instructs teachers how to develop level 2+ lessons and activities which assist in training students to pass the DLPT reading and listening exam at a level 2+. These levels are specified by the ILR scale, which has six levels ranging from zero (no functional ability) to five (full professional proficiency). Figure (1) describes ILR Levels 2, 2+, and 3 that specify what student should master in each skill to reach these levels. 10 Figure 1: ILR scale Table (Brau, Brooks, 2009) Learning Objectives The learning objectives of UCAT Training Module are as follows: • From memory, the foreign language teachers will be able to recognize UCAT terminology with 100% accuracy by answering multiple choice questions • Given a software simulation of UCAT, the foreign language teachers will be able to create a module on UCAT and share it with other colleagues with 100% accuracy. • Given a laptop and access to the internet, foreign language teachers will be able to design level 2+ listening and FLO activities using the Universal Curriculum Assessment Tool (UCAT) without making errors • Given a laptop and access to the internet, foreign language teachers will be able to design level 2+ reading activities with the Universal Curriculum Assessment Tool (UCAT) without making errors • Given a laptop and access to the internet, foreign language teachers will be able to design a level 2+ Lesson on Universal Curriculum Assessment Tool (UCAT) without making errors 11 Proposed Solution The UCAT Training Website uses a multitude of different teaching modalities that incorporate the various aspects of Captivate 9 during instructional training focusing on learners’ different learning needs, levels, and abilities. UCAT Training Website demonstrates step by step how to use this powerful tool inside and outside the classroom. The UCAT Training Website enables teachers to learn about UCAT terminology, features and activity components, and steps required for registering a UCAT account. Moreover, this Website instructs foreign language teachers into how to create and design online UCAT activities and lessons that target student needs to ensure the success of the students when taking the DLPT and OPI. The UCAT Training Website can provide both synchronous instructions, where students work together in the classroom and an asynchronous, self- study learning environment where they can learn at their own pace. The link to the UCAT Module was distributed to the foreign language teachers by sending it by email for teachers to take the training and to keep the link as a reference in case they need to review some instructions when needed. Learning Theories For the purposes of this training, three primary theories were applied. First is Behaviorist learning theory in which students acquire new behavior and achieve great learning outcome based on environmental conditions (Hoy & Miskel,1987). Students learn best when present in a suitable environment that allows the learners to work using their own learning strategies, regulate their behavior and get engaged in self-reflection. However, the external environment and learners’ own beliefs and values affect this process in a positive or negative way. That’s why the mission of the trainer is to create a learning environment, urge and encourage the learner to do better to achieve the goals, applaud students’ success, and support failing students in order to get better grades in upcoming tests. Second is the cognitive theory which focuses on the internal process and connections that take place during learning. In this theory, we need to focus on the what, how and why of learning which is the process of learning or gathering facts and information through hearing, seeing and reading. To have a good learning experience, students should be exposed to various ways, methodologies, and strategies of teaching. This can easily be done 12 when using technology in the classroom. For example, Captivate 9 can be used in various ways to explain how to use UCAT in teaching adults second language, such as audio, video, and presentations. Third is constructivist which explains how learners acquire knowledge and learn. This is the process of expressing ideas, solving problems, and producing information based on learner experience and back ground knowledge. For example, Captivate 9 provides a platform for students learning through UCAT to do the following: solve problems, answer questions that are based on real life experience. Principles The following principles are used in this project to ensure that the website includes interactive materials and design that engage foreign language teachers in the learning process. ❖ Coherence Principle ❖ Modality Principle ❖ Segmenting Principle ❖ Redundancy Principle ❖ Contiguity Principle ❖ Multimedia Principle Strategy and Role of the Trainer There are some challenges when training foreign language teachers how to use UCAT in teaching second languages especially when it comes to design activities, instructions, tasks, and assessments for the training module. Foreign language teachers have different cultural and educational backgrounds and they have diverse abilities, interests, and needs. The UCAT Training Website includes teaching strategies that keep learners motivated, engaged and interested in the material given. The UCAT Training Website is an interactive module that includes meaningful tasks that are related to learners’ life experience. There are three stages in task-based activity: pre-task, task, and post-task (Skehan,1996). In the pre-task stage, learners receive from the trainer necessary information 13 and knowledge to perform the task. Moreover, in the task stage learners receive the instructions from the trainer to plan, design, create and produce. The learners in the post- task stage present and discuss their product and get feedback. The best kind of learning practice involves the learner in expressing, interpreting and negotiating meaning. During production stage, learners experience the forms of the learning in a context that helps them to understand how to use the forms appropriately. Instructors encourage learners to recognize their own thinking process that leads to developing independent learner skills. Students proceed with a learning task, observing trainer performance continuously, and evaluating their own learning when the tasks are completed. For students to succeed in the learning process they need to know and value the strategies that assist in their success (Keller, 1987). Bridging is another teaching strategy that allows students to build on their previous knowledge and establishes a relationship between the learners and the curriculum. Contextualization is the strategy of getting to know vague concepts through direct learning. Modeling is when I model a learning concept by speaking and demonstrating it slowly and clearly. Delivery System and Media Decisions UCAT Training Website demonstrates step by step how to use UCAT and how to develop and design listening, FLO and Reading activities and lessons on UCAT. There will be engaging activities, assessments, and assignments for learners to work on and that to ensure the foreign language teachers understand the process of designing online activities and lessons employing UCAT. The delivery format for this instructional training is online. The UCAT training Website will be hosted on WIX, a leading cloud-based development platform. The lectures will be designed and developed using Captivate 9, a rapid responsive authoring tool that is used for creating e-learning content. The link to the UCAT Training Website was distributed to the foreign language teachers. 14 Figure 2: UCAT Training Website home page Challenges The primary challenges that this project faced included: 1- Adobe Captivate: Some problems have been encountered while working on captivate 9 to develop the modules. Here are some of the issues: The navigation becomes slow at the end of module 2; The play bar disappears in Module 1 and 2. 2- Move to another school: Department E is moving to another school which means that new meetings need to be conducted with the new administrative staff and hopefully the new stakeholders will agree to use this website to train foreign language teachers in how to use UCAT in teaching the second language. 3- UCAT update: New updates have been occurring to UCAT which needed to be reflected on this product. Methods The steps to finish this project followed ADDIE phases: analysis, design, develop, implement and evaluate. Analysis. This the first stage of the project which was done earlier by conducting learner, workplace, and content analysis to gather information needed to come up with an appropriate solution for the existing problem. 15 Design and Develop. UCAT training Website includes five sections. The first section includes the first module which was designed, developed, and tested. This module instructs foreign language teachers about UCAT terminology and different UCAT features. The second module instructs foreign language teachers about how to create Listening and FLO activities on UCAT. The third module instructs foreign language teachers about how to create reading activities on UCAT. The fourth module instructs teachers into how to create a lesson on UCAT. Implement and Evaluate. Ten foreign language teachers volunteered to test the UCAT Training Website. At the end of the training, the foreign language teachers evaluated the website by taking the survey on Google Forms. The results of the evaluation were taken into consideration to continuously improve the website. Deliverables The UCAT Training Module contains four modules and summative assessment: ❖ Module one includes instructions, practice, assignment, and evaluative assessment that examines the trainees on the topic discussed. The practice and assessment include questions about UCAT terminology and UCAT different features in the form of multiple choice questions, true or false and drag and drop. The assignment requires the learners to create a module on UCAT. Appendix C includes sample pictures from module 1. ❖ Module two consists of instructions, practice, assignment and evaluative assessment. The practice and assessment questions are in the form of drag and drop asking students about how to create listening and FLO activities on UCAT. The assignment instructs teachers to create level 2+ listening and FLO activities on UCAT that must be shared on UCAT with the trainer. Appendix D includes sample pictures from module 2. ❖ Module three consists of instructions, practice, assignment and evaluative assessment. The practice and assessment questions are in the form of drag and drop asking students about how to create reading activities on UCAT. The assignment instructs teachers to create level 2+ reading activities on UCAT that must be shared on UCAT with the trainer. Appendix E includes sample pictures from module 3. 16 ❖ Module four consists of instructions, practice, assignment and evaluative assessment. The practice and assessment questions are in the form of drag and drop asking students about how to create a lesson on UCAT. The assignment instructs teachers to create level 2+ lesson on UCAT that must be shared on UCAT with the trainer. The teachers can work on a team to create this lesson. The teachers present their lessons in the lab for trainer and teachers feedback. Appendix F includes sample pictures from module 4. ❖ Summative assessment includes a survey that collects teachers’ feedback about the training materials and if there is any suggestion to improve this website. After all the modules were completed, ten foreign language teachers volunteered to take the training, including pre-test and posttest in which usability test was administered. The results from the administered test were used to create the final report and the UCAT Training Website. Resources Timeline and Costs Developing this module took around three weeks to finish designing and producing the instructional modules. The time needed to work on this project was critical due to the researcher having a full-time job and family obligations. The best time to work on this project was during the evenings and weekends which was dedicated to designing and producing the modules. It is worth mentioning that this project costs nothing since there are no additional costs required to complete this project. Captivate 9 software was used to develop the modules which were hosted on Wix, a leading cloud-based development platform that makes it easy for everyone to create a beautiful, professional web presence, to create the Technology Training Website (Ji &Fu, 2015). I volunteered my hours to complete this project since it will benefit many foreign language teachers and students at DLIFLC. I asked my advisor, coworkers and classmates who work in the same teams as me to test this website for the purpose of receiving their constructive feedback to improve this product. I collected the feedback from Dr. Fischer who is my professor and advisor at the MIST program, DLIFLC team members who are expert in the subject and my MIST team who are 17 expert in the instructional technology field. Moreover, feedback was studied and analyzed to edit and justify some of the modules content. Timeline Report Task Name Duration Start Finish Resource Design 30 days 9/5/2017 10/5/2017 Content Design 15 days 9/5/2017 9/20/2017 MIST Student UCAT Training Website 3 days 9/5/2017 9/8/2017 MIST student, Wix look Lesson 1 7 days 9/5/2017 9/12/2017 MIST student, Captivate 9 Lesson 2 7 days 9/5/2017 9/12/2017 MIST student, Captivate 9 Lesson 3 7 days 9/5/2017 9/12/2017 MIST student, Captivate 9 Lesson 4 7 days 9/5/2017 9/12/2017 MIST student, Captivate 9 Formative evaluation ongoing 10-5-2017 MIST student Development 30 days 10/5//2017 11/5/2017 Website look 3 days 10/5/2017 10/8/2017 MIST student, WIX Lesson 1 5 days 10/5/2017 10/10/2017 MIST student, Captivate 9 Lesson 2 20 days 10/5/2017 10/25/2017 MIST student, Captivate 9 Lesson 3 20 days 10/5/2017 10/25/2017 MIST student, Captivate 9 Lesson 4 20 days 10/5/2017 10/25/2017 MIST student, Captivate 9 Summative assessment 5 days 10/25/2017 10/30/2017 MIST student, Google form 18 Implementation 10 days 11/6/2017 11/16/2017 Update content based on 10 days 11/6/2017 11/16/2017 MIST student Feedback Evaluation 50 days 9/5/2017 11/25/2017 Pre & post test 10 days 9/5/2017 9/10/2017 MIST student Usability test 4 days 9/5/2017 9/9/2017 MIST student Summative Evaluation Finalize Capstone 17 days 11/15/2017 12/02/2017 Capstone Summary 15 days 11/15/2017 12/02/2017 MIST student Capstone Presentation 15 days 11/15/2017 12/02/2017 MIST student Evaluation and Testing Formative Evaluation The online training website includes four modules. Each module includes a final assessment that tests students’ understanding. The first module assessment includes questions about UCAT terminology, setting up UCAT profile and different UCAT features. The second module assessment tests teachers’ ability to create listening and FLO activities on UCAT. The third module assessment tests teachers’ ability to create reading activities on UCAT. The fourth module assessment tests teachers’ ability to create a lesson on UCAT. These assessments will be conducted online. Usability Testing The usability test was conducted to make sure that the objectives of UCAT training Website were met and to assure that the navigation in the website and modules were functioning as expected. Doctor Donald Fischer, my advisor and professor at California State University Monterey Bay, and three other foreign language teachers took this training in which they detected some issues and they came up with recommendations and advice to improve the training. Table 1 sums these issues. 19 Table 1: Identifying issues and solutions Issue Is it corrected? Remarks Module 1: Navigation issues (learner cannot corrected pause or repeat audio) Module 1: add progress line Not corrected Unable to resolve Module 1: Spelling mistakes corrected Module 1: Mistakes in practice activities and corrected final exam Module 1: titles on each page don’t exist corrected Module 1: outdated screenshots corrected Module 1: change “quiz” into “module 1 final corrected exam” Module 1: closed captions font is too small corrected Module 1: add instructions to take module 2 corrected Module 1: The Pause and Continue Buttons uncorrected In progress need work in Module 1. I could not get them to function Lesson 2, Module 1 Info page the audio cuts off Uncorrected In progress in mid-sentence On the matching for Lesson 2, there are still Uncorrected In progress font differences On getting to the Module 1 final exam, I got a Uncorrected In progress blank page Explaining how to remove the first choice by Uncorrected In progress clicking on it 20 Module 2: Navigation issues (learner cannot Uncorrected In progress pause or repeat audio) On the Lesson 3, Module 2 True/False, add Uncorrected In progress feedback on why True is not correct The T/F question at the end of Lesson 5 I Uncorrected In progress should add where the + sign they are to click is located In the quiz on Activity 2, It is confusing to Uncorrected In progress learner how to select words Final exam for module 2 has many navigation Uncorrected I cannot resolve issues Module 3: Not allowing learners to redo Uncorrected In progress quizzes and tests. Module 4 practice includes information not Uncorrected In progress provided in the lecture and other issues Click on the button and could not get back to Uncorrected In progress where I was in Module 4. Module 4: Activity 1, T/F. “Rational” should Uncorrected In progress be “rationale” Module 4 Home Page: Lesson 8 tab is before Uncorrected In progress the Lesson 7 Tab. Spelling mistakes uncorrected In progress All the certificates have your “signature” that Uncorrected In progress overlaps the frame 21 Pre- and Post-testing The purpose of conducting pre-test is to test foreign language teachers on their information about UCAT terminology, UCAT features, development of listening, FLO and reading activities. These questions look the same as the final test in the UCAT module. The foreign language teachers took the post-test after the training which is exactly the same as the pre-test. The purpose of conducting such testing is to check if the learner knowledge and skills have improved and teachers are more confident on working on UCAT to produce interactive materials that can be used on their daily teaching. The post-test checks if the objective of the training is met and fulfilled by foreign language teachers. Appendix B includes the pre-test and posttest questions. Results The teachers answered the ten questions present in the pre-test and post-test. Each question was worth one point. The grades were uploaded from Google forms and copied to an excel sheet. There are two sets of grades shown as pre-test and post-test grades as we see in table 1 which also shows the grades distribution. The average of pre-test grades is 6.6 while the average of post-test grades is 9.3. The difference between pre-test and post-test is shown in table 2. The post-test grades are much better than the pre-test grades which indicates that the teachers experience the transfer of learning and they become more knowledgeable about UCAT. Pretest Post-test 4 8 8 10 9 10 9 10 4 8 4 8 6 9 7 10 7 10 8 10 Table 2 22 My hypothesis for this training is that foreign language teachers after taking UCAT training will score high on the posttest which means that the mean of the posttest would be greater than the mean of the pretest. However, the null hypothesis was that the grades would not change from pre-test to post-test and the mean of the pretest and posttest would be equal. I conducted the paired two sample t-test for dependent samples. The degree of freedom was 9 using one-tail results for a directional hypothesis. We accept the research hypothesis because the absolute value of the computed t-statistic, |7.36|, is greater than the t critical one tail value of 1.83. Therefore, the results support a conclusion that the tutorial contributed to higher posttest scores indicating an increase in the ability to apply UCAT to DLI learning situations. Table 3: t-Test: Paired Two Sample for Means Variable 1 Variable 2 Mean 6.6 9.3 Variance 4.044444444 0.9 Observations 10 10 Pearson Correlation 0.943456353 Hypothesized Mean Difference 0 df 9 t Stat -7.363636364 P(T<=t) one-tail 2.13247E-05 t Critical one-tail 1.833112933 P(T<=t) two-tail 4.26494E-05 t Critical two-tail 2.262157163 Summative Evaluation Knowledge check. There are two ways to test learner knowledge in the UCAT training website. First, learners need to score 80% or more in every module final assessment. Second, learners need to design a lesson on UCAT that includes the following parts: activation of Schemata, reading, listening, and task-based activity. It is worth mentioning that the learners have the opportunities to practice and interact with the 23 materials before conducting the assessments which are accompanied by effective immediate feedback. Survey. The questionnaire was designed in Google Forms to receive learners’ feedback about the training and is embedded in the website. The purpose of this questionnaire is to invest in the input to continuously improve the training module to fit learners’ needs and interests. The questionnaire includes eight questions that ask foreign language teachers about their opinion about the activities, assessment, and materials present in this module and ways to improve this product. The feedback from the foreign language teachers is collected and analyzed for the purpose of improving the UCAT Training Website. User Experience Survey Results. Seven out of ten teachers answered the survey questions. They strongly agreed that the objectives of the training were clearly defined and the content was organized and easy to follow and the time allotted for the training was sufficient. Four out of seven agreed that the course has practical value in their lives and the training experience will be useful in their work. Some teachers recommended that this training should include modules that instruct how to use the other UCAT activities like matching and hot spot activities. Others recommended that this training should include modules that instruct how to design level 1, 1+ and 2 activities and lessons on UCAT. Conclusion The results from the evaluative assessments indicated that the foreign language teachers did benefit from this training. The learners now have the ability to develop interactive activities and lessons on UCAT. Moreover, the foreign language teachers expressed that the instructions were easy to follow and clear. There are some issues in the modules that need to be fixed. However, there are some problems that I was unable to fix in the short term. UCAT Training Website is a unique product because it is the first website that is developed at our school level that instructs teachers how to use UCAT step by step. Many improvements will occur to this website in the future to make sure that it supports DLIFLC goals and learners’ objectives and needs. Some teachers face difficulties designing Level 2+ activities and lessons on UCAT which assist students in getting prepared for the DLPT. This website looks like an online manual for teachers that guides them into how to 24 develop interactive UCAT lessons to support the Levantine Dialect curriculum course. Many UCAT modules will be developed in the future to support DLIFLC mission and will be embedded in UCAT Training Website. Module 5: creating higher level activities on UCAT; Module 6: creating games on UCAT; Module 7: developing semester one activities on UCAT. 25 References Berge, Z. L., & Collins, M. P. (Eds.). (1995). Computer mediated communication and the Online classroom: distance learning. Cresskill: Hampton press. Biggs, J. B. (2011). Teaching for quality learning at university: What the student does. McGraw-Hill Education (UK). Brau Maria, Brooks Rachel “ILR 101”, Language Testing and assessment unit FBI, 2009 http://www.slideshare.net/jasdhillo/ilr-101 Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. John Wiley & Sons. Harasim, L. M. (1995). Learning networks: A field guide to teaching and learning online. MIT press. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational psychology review, 16(3), 235-266. Hoy, W. K., & Miskel, C. G. (1987). Theory research and practice (Vol. 278). Educational administration, New York: Random House. Ji, Y., & Fu, X. (2015). WEBSITE CONSTRUCTION FOR SMALL COSMETICS ENTERPRISES. Keller, J. M. (1987). Strategies for stimulating the motivation to learn. Performance Improvement, 26(8), 1-7. Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional science, 26(1-2), 49-63. Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace (Vol. 12). San Francisco: Jossey-Bass. 26 Pea, R. D., & Kurland, D. M. (1984). On the cognitive effects of learning computer programming. New ideas in psychology, 2(2), 137-168. Roschelle, J., & Pea, R. (2002). A walk on the WILD side: How wireless handhelds may change computer-supported collaborative learning. International Journal of Cognition and Technology, 1(1), 145-168. Sherry, L. (1996). Issues in distance learning. International journal of educational telecommunications, 1(4), 337-365. Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied linguistics, 17(1), 38-62. 27 Appendices Appendix A : Summative Evaluation Questions Rate the following statements 1- The objectives of the training were clearly defined ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 2- The content was organized and easy to follow ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 3- The course and the instructor meet your needs as a learner ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 4- The training experience will be useful in my work ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 5- The time allotted for the training was sufficient ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 6- This course has practical value in your life ○ Strongly disagree ○disagree ○undecided ○agree ○strongly agree 7- What improvements would you recommend for this course? 28 Any additional comments 29 Appendix B : Pre-test and Post-test Questions 1- UCAT webpage URL is * a) https:\\ucat.dlfilc.com b) http:\\ucat.dliflc.edu c) https:\\ucat.dliflc.edu d) http:\\ucat.dliflc.net 2-UCAT includes the following activities except * a) multiple choice b) short answer c) matching d) draw and paint 3-In the info page you can do the following except * a) add module objectives b) add module description c) select ILR level d) change password 4-In UCAT profile feature you can * a) change your profile picture b) add an activity c) preview the module d) share the module 5-In UCAT social center you can * a) change your profile picture b) delete an activity c) add module objectives d) add a comment 6-More than one person can work on one module * a) True b) False 7-You can access UCAT from anywhere * a) True b) False 8-you can share the module with the students by the way of * a) zip folder b) link 30 c) username and password d) accessing UCAT website 9-you can add to your module all of the following except * a) picture b) video c) audio d) shapes 10-You can design a lesson on UCAT * a) false b) true 31 Appendix C: Screen Shots of Module 1 32 33 Appendix D: Screen Shots of Module 2 34 35 Appendix E : Screen Shots of Module 3 36 37 Appendix F: Screen Shots of Module 4 38 39 Release Form