A Meta-Analytical Review of Empirical Mobile Usability Studies

Vol. 6, Issue 3, May 2011, pp.117-171 A Meta-Analytical Review of Empirical Mobile Usability Studies An earlier version of this paper was presented at the 2006 Americas Conference on Information Systems (AMCIS), Acapulco, Mexico. Constantinos K. Coursaris Assistant Professor Dep‘t of Telecommunication, Information Studies, and Media Usability/Accessibility Research and Consulting 424 CAS Michigan State University East Lansing, Michigan 48824 Twitter: @DrCoursaris [email protected] Dan J. Kim Associate Professor Computer Information Systems University of Houston Clear Lake 2700 Bay Area Boulevard Houston, Texas 77058 [email protected] Abstract In this paper we present an adapted usability evaluation framework to the context of a mobile computing environment. Using this framework, we conducted a qualitative meta-analytical review of more than 100 empirical mobile usability studies. The results of the qualitative review include (a) the contextual factors studied; (b) the core and peripheral usability dimensions measured; and (c) key findings in the form of a research agenda for future mobile usability research, including open and unstructured tasks are underutilized, interaction effects between interactivity and complexity warrant further investigation, increasing research on accessibility may improve the usability of products and services for often overlooked audiences, studying novel technology and environmental factors will deepen contextual mobile usability knowledge, understanding which hedonic factors impact the aesthetic appeal of a mobile device or service and in turn usability, and a high potential for neuroscience research in mobile usability. Numerous additional findings and takeaways for practitioners are also discussed. Keywords Human Computer Interaction, mobile, usability, efficiency, effectiveness, satisfaction, mobile device, wireless, context, meta-analysis, empirical Copyright © 2010-2011, Usability Professionals‘ Association and the authors. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. URL: http://www.usabilityprofessionals.org. 118 Introduction Mobile devices are becoming increasingly popular, having already reached over one billion mobile subscribers. A recent forecast by the UMTS forum (2005) estimated that the global number of subscribers will be between 1.7 to 2.6 billion for mobile voice and 600 to 800 million for mobile data. As consumers‘ technology fears and adoption costs are reduced, mobile devices are approaching ―mainstream‖ status around the developed world. Mobile devices propose increasing value to consumers found in ―anytime, anywhere, and customized‖ connectivity, communication, and data services. Although progress has been made in terms of technological innovations, there are obvious limitations and challenges for mobile device interfaces due to the characteristics of mobile devices (i.e., the size of small screens, low resolutions of the displays, non-traditional input methods, and navigational difficulties; Nah, Siau, & Sheng, 2005). Therefore, usability is a more important issue for mobile technology than for other areas, because many mobile applications remain difficult to use, lack flexibility, and lack robustness. Research Motivation and Objectives Usability has been the focus of discussion (Venkatesh, Ramesh, & Massey, 2003) and described by varying definitions (Nielsen, 1993; Nielsen & Levy, 1994; Shackel, 1991) in both academia and industry for a long time. Many of these definitions proposed that the central theme of usability is that people can employ a particular technology artifact with relative ease in order to achieve a particular goal within a specified context of use. The turn of this century marked an increased focus on mobile usability studies for research in the field of Human Computer Interaction (HCI). Although a considerable volume of research on general usability exists, due to the novelty of mobile technology relatively few studies have been conducted focusing on mobile usability. Even worse, only 41% of mobile usability papers are empirical1 in nature (Kjeldskov & Graham, 2003). Moreover, there is no qualitative study on the usability dimensions considered in such mobile studies. Thus, our research aims to fill this gap and in doing so will also provide a roadmap for future mobile usability studies that will be of value to this relatively young research area. Specifically, this study addresses the following research question: What are the key formation and evaluation dimensions of usability in mobile technology usability studies? To this end, this paper describes the qualitative review of more than 100 published empirical mobile usability studies. First, following a brief review of a usability evaluation framework in a non-mobile context, a framework of contextual usability for mobile computing2 is presented. Next, by using the proposed framework a qualitative review of empirical mobile usability studies is presented along with a discussion on the taxonomy used during the coding in this study. The results emerging from the comprehensive review of mobile usability studies are then presented, which include (a) the contextual factors studied, (b) the core usability dimensions defined and measured, (c) the peripheral usability dimensions explored, and (d) key findings in the form of a research agenda. Finally, this paper discusses the contributions and limitations of the research. Literature Review and a Mobile Usability Framework Usability studies have their roots as early as the 1970s in the work of ―software psychology.‖ Over time, the focus of this body of research has shifted and most recently centered on the relevance of context of use for usability. The concept of context of use, as it relates to usability, emerged out of the work of several scholars (Bevan & Macleod, 1994; Shami, Leshed, & Klein, 2005; Thomas & Macredie, 2002) who attempted to identify additional variables that may impact usability. Varied situational contexts will result in emerging usability factors, making traditional approaches to usability evaluation inappropriate. The significance of this area emerges from its importance in yielding a reasonable analysis during a usability study (Maguire, 1 Empirical studies deal with empirical evidence that is derived by means of observation, experiment, or experience. In this study, we further classified empirical evidence as survey, interview, observation, and device/server logs in either a lab, the field, or both settings, as well as focus groups. 2 Even though we mainly focus on mobile usability, our adapted framework can be used for usability studies in general. Journal of Usability Studies Vol. 6, Issue 3, May 2011 119 2001; Thimbleby, Cairns, & Jones, 2001). Furthermore, during the evolution of HCI mentioned above, the conceptualization of usability has varied extensively. The broad set of definitions and measurement models of usability complicate the generalizability of past studies at the level of the latent usability variable. Therefore, a usability study gains value when it is based on a standard definition and operationalization of usability. In the following section, we review a set of key approaches in evaluating usability as communicated in previous work. Approaches to Usability Evaluation Different approaches to usability evaluation have been proposed in different contexts such as websites (Agarwal & Venkatesh, 2002), digital libraries (Jeng, 2005), audiovisual consumer electronic products (Han, Yun, Kwahk, & Hong, 2001; Kwahk & Han, 2002), and many others. In the context of website usability, Agarwal and Venkatesh (2002) presented five categories (i.e., content, ease of use, promotion, made-for-the-medium, and emotion) and subcategories (i.e., relevance, media use, depth/breadth, structure, feedback, community, personalization, challenge, plot, etc.) of website usability evaluation components based on Microsoft Usability Guidelines (MUG ; see Keeker, 1997). They also discussed the development of an instrument that operationalizes the measurement of website usability. Recently, employing the MUG-based model, Venkatesh and Ramesh (2006) explored an examination of differences in factors important in designing websites for stationary devices (e.g., personal computers) versus websites for wireless mobile devices (e.g., cell phones and PDAs). In the context of digital libraries, Jeng (2005) proposed an evaluation model of usability for digital libraries on the basis of the usability definition of ISO 9241-11 (ISO, 2004). The model included four usability evaluation comports: effectiveness, efficiency, satisfaction, and learnability. The satisfaction of digital libraries was further evaluated by the areas of ease of use, organization of information, clear labeling, visual appearance, contents, and error corrections. In the context of audiovisual consumer electronic products (e.g., VCR, DVD players, etc.), Han et al. (2001; Kwahk & Han, 2002) suggested a usability evaluation framework that was similar to the subsequent work of Hassanein and Head (2003). The framework consisted of two layers: formation of usability and usability evaluation. The formation of usability layer had four contextual-components (i.e., product, user, user activity, and environment) that were well accepted as the principal components in a human-computer interaction upon which good system design depends (Kwahk & Han, 2002; Shackel, 1991). The usability evaluation layer was organized with three groups of variables: design variables (i.e., product interface features), context variables (i.e., evaluation context), and dependent variables (i.e., measures of usability). Interestingly, there is no usability evaluation framework that yet exists in the context of a mobile computing environment. We believe it is a critical omission and an important topic warranting investigation. The next section looks at the key formative factors of usability as explored in contextual mobile usability studies. From this review, we propose a contextual usability framework for a mobile computing environment. A Contextual Usability Framework for a Mobile Computing Environment The work of several scholars (Bevan & Macleod, 1994; Shami et al., 2005; Thomas & Macredie, 2002) who attempted to identify additional variables that may impact usability and subsequently adoption, led to the conceptual emergence of context of use (herein referred to as context) as it relates to usability, also referred to as contextual usability. Several frameworks encapsulating context have been proposed (Han et al., 2001; Lee & Benbasat, 2003; Sarker & Wells, 2003; Tarasewich, 2003; Yuan & Zheng, 2005). While there may be other usability frameworks that attempt to capture the essence of context, the models cited here provide a representative set of work in this area. From these we adapted the framework proposed by Han et al. (2001) because it offers considerable detail for each dimension they identified. On the basis of the discussion on approaches to usability evaluation and the framework proposed by Han et al. (2001) and Kwahk and Han (2002), we propose a contextual usability framework for a mobile computing environment. The framework is depicted in Figure 1 and contains three elements. First, the outer circle shows the four contextual factors (i.e., User, Technology, Task/Activity, and Environment) described earlier as impacting usability. Second, the inner circle shows the key usability dimensions (i.e., Effectiveness, Efficiency, Satisfaction, Learnability, Flexibility, Attitude, Operability, etc.). Third, the box on the top of contextual Journal of Usability Studies Vol. 6, Issue 3, May 2011 120 factors shows a list of consequences (i.e., improving systems integration, increasing adoption, retention, loyalty, and trust, etc.). Compared to the framework proposed by Han et al. (2001) and Kwahk and Han (2002), there are several advantages of the suggested mobile usability framework. Although the previous frameworks proposed by Han et al. (2001) and Kwahk and Han (2002) are comprehensive, they are difficult to follow due to formation and evaluation dimensions being merged into one diagram. Thus, the suggested framework depicted in Figure 1 represents a simple yet direct way to identify and address the various contextual mobile usability dimensions. In addition, with its central focus on usability, it offers specific guidance on the implementation of any interface/interaction project along with potential outcomes. In addition, two modifications are introduced in terms of nomenclature for mobile contextual usability. First, ―Technology‖ replaces ―Product,‖ as this term helps conceive the system that a user may interact with a greater set of components, instead of simply the device or application itself. One example of this is found in the case of mobile usability where the inclusion of the wireless network is likely in addition to the mobile device (i.e., the product) when studying usability of a mobile product or service. Because mobile usability is mainly related to mobile technology, which continually improves the limitations of mobile interfaces and its applications, the technological factor of a mobile usability framework is an important and unique component that needs to be taken care of. Second, ―Task/Activity‖ replaces ―Activity,‖ as the former term appears more commonly in usability literature when describing the nature of users‘ interaction with the technology. In addition, a list of consequences of usability was added to the framework as an output of usability evaluations. These four variables (i.e., user, task/activity, environment, technology) were used for the presentation of the qualitative review of previous empirical research3 that relates to the usability assessment of mobile applications and/or mobile devices. The benefit of using these variables for the literature review is found in both the structure it provides for the discussion to follow, as well as to help highlight any areas that are lacking investigation. 3 Since this study focuses on mobile usability, we only reviewed empirical studies on mobile usability. Journal of Usability Studies Vol. 6, Issue 3, May 2011 121 Consequences of Usability Improving systems integration, Increasing adoption, retention, loyalty, improving trust, etc. Context Ph ys y c ic a c al re ffi P u (A E t s t f-e l yc n u el s tex r c ho e udit vir / x se ics e/s tion con o so pe ry on U h nc ni l ci ri , v m p a al m is en a ri e o g i c r g or en ua t og xpe n/c olo so t ty l, o m e e/e pti ch ci pe co D g e sy al ) -lo c d co ca le Per al/p nd w tio n o iti io n, t o Kn o Usability Dimensions ns m E Effectiveness, Efficiency Satisfaction, Errors, Attitude, Learnability, Accessibility, Operability, Accuracy, Acceptability, Flexibility, Memorability, Ease of use, Usefulness, Utility, Playfulness e gy lo pe od no ty t m ch ice pu Te ev In D ce fa r te In . vs ) e m oal ) ) o d c u t o r g u re y o t it iv es e uc ct fin tcom str /A de u un sk er d o en / a T (us ine op n ef s ( pe e-d ion : O (pr ript m c lis sed es a d e o R Cl sk Ta Figure 1. A suggested mobile usability framework Next, we conducted a qualitative review of empirical mobile usability studies using the framework to demonstrate the validity of the framework. Using the outer and inner circle of the framework in Figure 1, we looked at the contextual factors as well as the key usability dimensions in collected mobile usability studies. Journal of Usability Studies Vol. 6, Issue 3, May 2011 6. and interpret selected sets of empirical studies (Glass. and (c) the time frame for included studies was from 2000 through 2010. setting the phone on silent mode. We continued with cross-referencing the references of the retrieved studies. Also. The meta-analytical review for this study began with the search for empirical mobile usability studies literature from the year 2000 through 2010. it appears that the human needs to be entered back in the HumanComputer Interaction investigations that focus on mobile usability. studies up to 2006 vs. Thus. followed by technology (46%). the user and the environment. while most statistics were not reported in that publication. and interactivity and complexity understudied The framework called for the identification of either closed or open tasks.. user (26%). because on-screen keyboards are now a standard of smartphone technology. i. (Note: distribution exceeds 100% as multiple areas may have been studied in a single study. as well as journals deemed relevant to the field of usability by the authors. where single-nation populations in studies are not included. as it would significantly reduce the reviewed articles. environment (14%). there is a lack of empirical research on the relevance of user characteristics and the impact of the environment on mobile usability. Issue 3. and environment characteristics (7%). and user characteristics (14%. technology (22%).) Hence.122 Methods Through systematic procedures of coding. Task characteristics: Open and unstructured tasks. the same analysis was performed on both samples (i. and examples would include checking the list of received calls. It is interesting to note that the proportion of studies that considered the environment doubled. the independent variables studied are described under each of the four contextual framework categories of Figure 1. Open tasks were used in 35% of studies. integrate. 2006).e. 2000). and part of this increased emphasis is a result of a number of recent studies that compared and contrasted different usability testing methods and environments. and key findings. a meta-analysis is an organized way to summarize.e. In doing so. and examples include interacting with a network of services using verbal or visual information. sample size. enabling the vibrating alert. The following sets of analysis pertain to the contextual factors studied among the 100 empirical mobile usability studies reviewed. it would be important to understand the optimal design of on-screen smartphone/mobile device keyboards according to target user groups and their characteristics. The above procedure resulted in the identification of 100 empirical mobile usability studies. and other tasks that have a predefined state or outcome. empirical mobile usability studies have been focused on investigating task characteristics (47%). many more articles in this study‘s larger sample appear to focus on tasks and related technologies far more frequently than on the other two dimensions. (b) the study was empirical in nature (see footnote 1 of the Literature Review and a Mobile Usability Framework section). recording. Overall. key usability dimensions measured. The review results are summarized in terms of the context defined in the study. Results of Analysis The literature review of empirical research on mobile usability performed appears in the Appendix. and computing. McGaw. Closed tasks were used most frequently (58%). For example. all 100 studies retrieved by the end of 2010. To this end. albeit one might consider them as cultural studies depending on the frame of reference). given the relative infancy of the mobile usability field. May 2011 . we used multiple databases to minimize the chance of omitting relevant studies. the distribution of research emphasis included research on task (56%). so as to observe scholarship trends in mobile usability between the two temporal reference points. keeping a pocket diary and filling in forms with each Journal of Usability Studies Vol.. Specific criteria were set for the selection of articles sought in this literature review: (a) a mobile technology was studied. 1981. Lipsey & Wilson. finding a ―Welcome Note‖ on a mobile website or a mobile app. An earlier analysis of the first 45 studies retrieved was presented at a conference in 2006 (Coursaris & Kim. research methodology used. & Smith. Hand searching of appropriate journals in this research included journals ranked among the top 10 in terms of perceived quality. By contrast with our earlier data set of 45 empirical studies published by 2006. A conscious decision was made to not limit the reviewed literature to peer-reviewed journal articles. Comparing these statistics with our 2006 sample. examining the role of technology in assisting users with visual impairment and memory loss respectively. whether the lack of support for Flash by iOS (available at the time this paper was written) significantly impacts the usability of mobile (iPhone/iPad) users.e. 6. Technology characteristics: Enabling technology beyond the interface is overlooked in mobile studies The most popular variable investigated in these studies pertaining to the technology used was the interface. novice samples (from 25% to 16%) to an examination of the impact of experience (from 9 to 16%) on the dependent constructs appears. an important question that arises is to what extent can such devices enhance a learner‘s experience. Examples of such limitations are found in the myriad of disabilities that can negatively impact a mobile user‘s experience or even prohibit the use of certain services. and other environment-specific factors present a significant opportunity for future research in mobile usability. the lack of research as it relates to technology beyond the interface continues. Cross-cultural studies did not emerge significantly during this period. No empirical mobile usability research studied the role of gender or age.. and services in the classroom or education environments at large. while extending previously validated research models and theories in the mobile domain. Hence. the outcome is user dependent). psychosocial. and yet are extremely underserved. Culture (3%) and job-specific roles (i. partly due to an emphasis on usability evaluation methods becoming more relevant and scholars‘ interest in comparing lab to field-based methods. humidity. This research design pattern is fairly consistent with our earlier analysis from 2006. a tablet PC. From these statistics it becomes apparent that research has been limited in both the range and frequency of user characteristics studied. by contrast. Environment characteristics: Area with greatest potential for future mobile usability research Eleven percent of studies explored factors as they relate to the environment. With the increasingly important role of mobile devices in academia. focusing on either novices (16%). Disability was only explored twice (i.. effects of task interactivity and task complexity on mobile usability were not investigated. complexity.123 use of the Internet. 8%) were also measured. The above distribution was quite similar to the 2006 sample. which is somewhat surprising considering the uptake of mobile devices around the world. Hence. Again. engineers. exploring the potential interaction effect between task interactivity and task complexity can help inform the design and use of mobile technology. These studies involved mobile phones (44%). and mobility was investigated in just 6% of studies. Hence. experts (13%). while convenient samples of students were utilized at similar rates. Issue 3. applications. and other tasks that do not have a pre-defined outcome (i. these frequencies exceed 100% because a few studies involved multiple devices. work-related context was investigated proportionately twice as much. temperature. and various interfaces (19%) including a desktop. or to what extent does network interoperability enhances a device‘s mobile usability would be of significant value particularly among the practitioner community. Hence. as well as social aspects. May 2011 . and others as they relate to mobile usability. Pocket PCs (5%). a mobile user being in physical proximity to other individuals) on the use of a Journal of Usability Studies Vol. Nine percent of reviewed studies did not report on tasks. and wearable or prototype devices. logging in to websites and rewriting web diaries that were first written on a pocket diary. a discman. Lighting and noise levels previously studied were joined by studies on sound. there is a relative lack of research involving open and unstructured tasks. and factors such as interactivity. For example. Also. again.e. User characteristics: A narrow focus on studied user dimensions is prevalent The most prominent user-related variable studied in empirical mobile usability research was (prior) experience. not reporting). or both (16%). physicians. a small shift away from convenient. Thus.. the same research gap exists surrounding open and unstructured tasks. little is known about the impact of colocation (i. This focus has shown an increase since the 2006 reported research incidence rate of 7%. where closed-open tasks were used 69% and 22% respectively (with 9%..e. For example. the same need and corresponding opportunities for user-centered empirical mobile usability studies still exists. acceleration. PDAs (38%). 2%). physical.e. there is limited research that contrasts self-reported data with device data. Chou. Knowledge dissemination outlets can both benefit and support the fueling of such research through special calls for related works. & Sears. & Butz. 2005. Chan. with only 13% of the studies being the case. & Cankar. 2002. 2003. Analysis of Mobile Usability Measurement Dimensions Because the focus of this study was on the usability dimensions measured in empirical mobile usability studies. Methodology characteristics: A call for neuroscience research in mobile usability The final set of analysis pertains to the experiment setup and methodology. 2007. Chittaro & Dal Cin. 2001. 2002. observation (5%). direct observation (7%). 2005. Issue 3. Table 1 presents a summary of these 31 measured usability dimensions. Jones. 2003. Chittaro & Dal Cin.124 mobile device (e. Kekäläinen. followed by field studies (21%). 2000. Pousman. Kankainen. Frequency of Usability Measures Used in the Reviewed Studies Original List of Measures Collapsed List Of Measures MEASURES SOURCES COUNT MEASURES UNIQUE COUNT % Efficiency Barnard. 2010. With the associated cost of the needed technology to employ related methods. Clarkson. design/card. multiple methodologies were identified in these studies. device data (33%). 2002. Kim. 2001. 2009. 2002. Kjeldskov & Graham. Kaikkonen. Table 1. 2001. Hence. Spence. Garcia-Molina. & Bias. May 2011 . Huang. which was also the trend found in our 2006 sample. & Chong. Chin & Salomaa. Such insight could further advance the contextual designs of mobile devices. and voice mail and web mail diaries. 2009. Kjeldskov. Costa. or other means. Kruger. Gupta & Sharma. including questionnaires (61%).g. 2002. Next. e. focus groups (7%). Buyukkoten. Duda. sorting/task analysis. Brewster. 2002. Schiel. and single studies leveraging a usability test/expert. Iachello. as well as Think Aloud Method (each at 2%). discussions (3%). & Aparicio. while 10% of studies involved both. & Thimbleby.. Specifically. Lyons. Yi. Koltringer & 41 Efficiency 61 33 Journal of Usability Studies Vol. 2009. Jacko.g. Clawson. However. collocated with familiar or unfamiliar individuals). Buchanan. 2006. & Starner. & Hess.. which types of applications are more likely to be used when alone vs. Frequencies of methodology used exceed 100% because most studies (45%) involved a multi-method approach. or interview data (4%). Brewster & Murray. we reorganized them in terms of usability dimensions. & Gupta. 2005. Bohnenberger. Jameson. there were no studies involving neuroscience. & Stasko. eye tracking and brain imaging. Fitchett & Cockburn. Lastly. whether through user-configured settings. Laboratory studies were conducted most often (47%). James & Reischel. sensors. Skov. & Werdenhoff. & Stage. 6. Butts & Cockburn. Bruijn. Moghazy. the area is prime for growth and novel contributions to the field. 2002. 2002. evaluation/participatory. & Paepcke. Kallio. Fithian. Alsio. something that has remained unchanged from the results of our 2006 sample. 2007. Goldstein. lab-tested mobile usability research was dominant. Silva. device data were most commonly triangulated with questionnaire (13%). an area that is of particular importance in mobile usability. 2005. 2000. Lindroth et al. 2001. Stage. Jones. Stickel.. Langan-Fox. Kjeldskov & Graham. Gupta & Sharma. Poupyrev. Kim et al. 2006. May 2011 . 2001. 2001. Koltringer & Grechenig. Nagata. Hinckley. & Blackwell. 2005. Olmsted. Waterson. 2008. & Thimbleby.. Landay. Fang. 2004. Hsu. Fithian et al. Fitchett & Cockburn. 2001. 2007. Pedersen. 2008. Lu. & Gupta. 2010. Dicke. & Ivanov. 2006. Huang. Wigdor. Scerbakov. 2002. 2003. 2000.125 Original List of Measures Collapsed List Of Measures COUNT MEASURES UNIQUE COUNT % Errors Andon. 2004. Jones. 2008. Pierce. & Xu. Shu. Danesh. Jones. Nagata. Wigdor & Balakrishnan. Kaikkonen et al. & Matthews 2002. 2010. Issue 3. 2006. Head. & Rekimoto. & Balakrishnan. & Yeh. Mao. Lau... Overgaard. Smith. Nielsen. 2003 Journal of Usability Studies Vol. Huang et al. Chae. Buchanan. Ross & Blasch. 2005. Ryan & Gonsalves. 2003.. Pousttchi & Thurnher. Sinclair. & Binshan. 2003 27 Effectiveness 49 27 Ease of Use Cheverst et al. Palen & Salzman. & Stenild. Platania-Phung. Sommerer. 2003. Massimi & Baecker. 2000. Shami et al. 2002. & Waycott. Brewster & Murray. Maruyama. 2003.. Milic-Frayling. 2006. 2006. & Thimbleby. Ryan & Gonsalves. Ooi. 2005. Kim. 2005... Kim. & Holzinger. & Hsu. Cheverst. Kaikkonen. 2009. & Billinghurst. Lee. Kim et al. Massimi & Baecker. Cyr. & Yaprak. & 26 Satisfaction 18 10 MEASURES SOURCES Grechenig. 2002. 2002. Langan-Fox et al.. 2008. & Rasmussen. James & Reischel. 2000. Khatri. Ross & Blasch. Friday. & Choi. & Booth. 2009. Mitchell. 2003. Kim et al. Licoppe & Heurtin. 2006. Srite. Brzezinski. Kober. MacKenzie. & Efstratiou. & Skepner. 2002. 2001. Momaya. 2010. Sodnik. Juola & Voegele 2004. Tomazic. Butts & Cockburn 2002. 2007. 2002. Buchanan. 2007. Chong. 2004. Darmawan. Inkpen. 2009. 2003. 2004. Chan. 2002. Thatcher. Seth. Nilsson. Ebner. Lehikoinen & Salminen. 2001. Massey. Li & Yeh. & Horvitz. Liang. Ervasti & Helaakoski. 2005. Lindroth. Davies. 6. Rodden.. 2010. 2002. 2002. . 2008 Journal of Usability Studies Vol. Ervasti & Helaakoski. Keeker. Palen & Salzman. & Vartiainen. Koltringer & Grechenig. 2004. Hummel et al. Costa et al. Pagani. Shami et al.. 2004. 2005. Kim et al. 2008 16 Learnability 8 4 Satisfaction Dahlberg & Öörni. Ruyter. Huang et al. 2006. 2010. 2007. 2007. 2001.. Issue 3. 2002. 2006. Duh et al.. 2006. 2005 15 Workload 7 4 Accuracy Barnard et al.. & Grill. 2009. 2002. Fithian et al.. Sodnik et al. Chittaro. Wu & Wang. Koivisto... 2002. Duh. 2005.. Palen & Salzman. Kaikkonen et al. 2004.. Li & Yeh. Pousttchi & Thurnher. Cyr et al. Olmsted. 2005. & Chen. Fang et al. Qiu. Kleijnen. Xu et al. Palen & Salzman. Ryan & Gonsalves.. Nielsen et al. 2010. Gimpel. Mao et al. Bohnenberger et al. 2005 8 Acceptability 3 2 MEASURES SOURCES Ramesh. 2001. Ebner et al. 2003... Liao... Wu & Wang. 2006. 2007. Huang et al.. 2006. 2006. 2004. 2007. 2005. 2002. Roto. Lindroth. Olmsted. 2009. Hess. Hummel. 2005. 2009.. 2003. 2007.. Kleijnen et al... Hsu et al... Juola & Voegele. Wu & Wang. 6. Brewster & Murray. 2006. Pousttchi & Thurnher. 2008. 2008. Zhang. Roto et al. 2005. 2005. 2004. 2007. Ryan & Gonsalves. 2004. 2003. 2006. 2006. Chong et al. Popescu. Pousttchi & Thurnher. May 2011 . & Hedman.. Dahlberg & Öörni. 2004. Ebner et al. Ryan & Gonsalves. 2008 17 Accessibility 15 8 Effectiveness Barnard et al. 2009.. Xu. 2010. 1997. 2005. 2005. Fithian et al. Ryan & Gonsalves. & Li.. 2005. & Wetzels.. 2006. 2006. 2002. Brewster. 2002. Burigat. 2003. 2005. 2000. Lindroth. Kim et al.. 2002. Liang et al. 2007. 2004.. Pagani. 2007. 2008.. Goldstein et al.. Chin & Salomaa. & Gabrielli. 2002. Shami et al. 2001. Tan. 2010. Clarkson et al. MacKenzie et al. Wigdor & Balakrishnan. Kim et al.126 Original List of Measures Collapsed List Of Measures COUNT MEASURES UNIQUE COUNT % Usefulness Bødker. & Huang. 2006. Thomas & Macredie. Kallinen. Nielsen et al. 2005 10 Enjoyment 4 2 Learnability Butts & Cockburn. Olmsted.. 2004. 2005.. 2002. 2006 4 Memorability 2 1 Enjoyment Cyr et al.. Kaikkonen et al. 2005 6 Aesthetics 4 2 Attitude Goldstein et al. Fang. 2007 3 Aesthetics Cyr et al. 2005. Kim et al. Zhong. Chin & Salomaa. 2009. Patel. Jones. Li & McQueen. 2006 2 Attractiveness Lin et al. Issue 3. Koivumäki. 2002. 2010. Kjeldskov & Graham. 2002. Marsden. Bødker et al. & Lee. 2005 2 Memorability Langan-Fox et al. 6. 2005. Price. & Wang. Kleijnen et al. Palen & Salzman. 2002. Strom. 2005.. Suzuki et al. Nielsen et al. 2008 7 Quality 3 2 Accessibility King & Mbogho. 2002. Sears. Dal Cin. Goldman. & Kesti. 2005. Wang. Juola & Voegele 2004. Zhang. 2009 6 Security 3 2 Reliability Andon. & Brzezinski. Jones.. 2009 3 Utility Duda et al. 2004 1 Functionality Pagani. Palen. Seth et al. 2005. Hummel. 2001 5 Utility 2 1 Problems Observed Kaikkonen.. Ristola... 2004.. 2004. 2005. Butts & Cockburn. 2007. 2001 2 Responsiveness Barnard et al. 2008. & Sears. Li & Yeh.127 Original List of Measures Collapsed List Of Measures MEASURES SOURCES COUNT MEASURES UNIQUE COUNT % Workload Barnard et al.. 2002. 2009.... 2003. 2004 1 Journal of Usability Studies Vol. Ebner et al. Lin. Hassanein & Head. 2002.. Kim et al. Kim. 2008.. Mao et al. & Cockburn. 2008. Juola & Voegele 2004 3 Flexibility 1 1 Quality Barnard. 2008 1 Availability Pagani. Khalifa & Cheng. Kaikkonen et al. 2003 1 Technicality Hummel et al. 2009. 2006... 2007 2 Content Kim.. Fang et al.. Pagani... 2007. 2002.. Lindroth et al. 2007 1 Flexibility Cheverst et al. 2000 1 Playfulness Fang et al.. 2007 3 Playfulness 1 1 Security Andon. 2004.... Chan. 2009. May 2011 . 2006.. 2003. Kleijnen et al. & Jacko. Yi.. Costa et al. 2007. Lv. 2007. 2003 2 Operability Chittaro. Jacko. 2001. 2010 4 Content 2 1 Acceptability Andon. Wu & Wang. 2006. Barnard et al. 2007. Kleijnen et al. 2004. Sodnik et al... 2005. Salzman & Youngs.. and integrity is a security dimension. attractiveness speaks to the broader concept of aesthetics. This broader interpretation of effectiveness may be extended to encompass the extent to which a system achieves its intended objective. (2001) on the classification of performance and image/impression dimensions with slight variations. and economical manner. Aesthetics (2%). as these factors have been set as the standard for more than a decade. 6. Hence. and learnability are most commonly measured in empirical mobile usability studies. (2001) defined satisfaction as ―the degree to which a product is giving contentment or making the user satisfied‖ p. Lastly. In addition. other measures found in studies that speak to this concept include reliability. ―ability to prevent the user from making mistakes and errors‖ p. accuracy. and problems observed measures found in this literature review with effectiveness (effectiveness offering a broader definition and operationalization). errors. Hence. Shackel defined attitude as the ―level of user satisfaction with the system‖ (2009. including Accessibility (8%).128 Original List of Measures Collapsed List Of Measures MEASURES SOURCES COUNT Interconnectivity Andon. 2004 1 Integrity Costa et al.147). responsiveness. this was just to clarify from the scope accessibility in the context of vulnerable/disabled users.. ―accuracy and completeness with which specified users achieved specified goals‖ p. The remaining measures identified in Table 1 reflect the peripheral dimensions measured in empirical mobile usability studies cited in the Appendix. Journal of Usability Studies Vol. 147) and (b) effectiveness (i. Hence. Han et al..e.e..147. Enjoyment (2%). The measure of errors was defined by Nielsen (1993) as the ―number of such actions made by users while performing some specified task‖ (p.32). With respect to the reviewed literature. All of these measures were defined in the work of Han et al. The above findings are arguably neither surprising nor favorable for the field. Acceptability (2%).e. Furthermore. the growing popularity of games and similarly engaging and hedonically oriented experiences in the use of mobile devices might suggest that both the factors studied and the definitions set forth for mobile usability may be revisited before too long. because the ―easier‖ a system is to use the less resources are consumed during the task. ease of use may be collapsed with efficiency. the publication would count these only once for the unique count). Han et al. Hence. effectiveness. p 341). the latter may also be collapsed with effectiveness. Workload (4%). accessibility had been studied in most cited studies as the degree to which a system was accessible. regardless of significant technology advances and use settings and scenarios—the usability scholar‘s lens has gone unchanged. if errors and effectiveness were found in the same study. functionality. and interconnectivity. effective. associated with the system. as opposed to error prevention. Issue 3. we collapsed the errors. Hence. its usefulness. attitude (as defined in these usability studies) may be collapsed into the single measure of satisfaction. and can be collapsed under accessibility.. so these can be grouped respectively.  Satisfaction: The degree to which a product is giving contentment or making the user satisfied. or simply put. or is hindering performance.  Effectiveness: Accuracy and completeness with which specified users achieved specified goals in a particular environment. However. Learnability (4%). ease of use. (2001) addressed errors through two measures: (a) error prevention (i. Similarly. It should be noted that the frequency count for each collapsed criterion is based on unique counts of a particular publication (i. Upon review of the measures‘ relative appearance in the reviewed literature the three core constructs for the measurement of usability appear to be the following:  Efficiency: Degree to which the product is enabling the tasks to be performed in a quick. the second order measure of efficiency often attempts to capture the first-order factor of ease of use. 2007 1 MEASURES UNIQUE COUNT % A preliminary inspection of Table 1 shows that the constructs of efficiency. May 2011 . satisfaction. This is supported conceptually. availability. mobile usability studies measured the error rate. The domain could arguably benefit by extending the defined core by considering a subset of the peripheral dimensions so as to allow for an even deeper understanding of mobile usability. mental models. difficult-to-use interface. limited input methods. First. three key findings emerge from the above data. is warranted. Utility (1%). The use of this standard would allow for consistency with other studies in the measurement of general usability (Brereton. effectiveness. This observation may come as a surprise. network connectivity reliability. After more than a decade‘s worth of research that centers on the standard usability measures articulated by ISO in 1998. Nielsen & Levy. 2007. This domain would benefit by having a further emphasis placed on the complexity of contextual usability and answering such research questions as those within and/or between each of the following areas:  Technology: Beyond the interface—how do mobile technology components beyond the interface (e. Further exploration of this construct. Playfulness (1%). Content (1%). even though there is support for the effect of such factors on performance and satisfaction (Coursaris.e. urgency. Swierenga. these usability dimensions are more important in mobile applications and technologies because of the inherent characteristics of mobile devices. low display resolutions.129 Quality (2%). given the growing popularity of accessibility research in less conventional (e. Third. & Watrall.  Task/Activity: Real world–real tasks—how do task complexity and task interactivity impact mobile usability? By considering these two dimensions and engaging in research involving open tasks in a field setting approximates real-world situations and results improve in their external generalizability. Beyond the benefit of a standard view of usability.. experience and efficacy). memory) impact the usability of mobile devices?  User: Study the human factors in HCI—what other user characteristics (e. 2005. aesthetic/hedonic constructs were studied in just 2% of empirical mobile usability studies. effectiveness. non-IS. These findings in turn call for a critical review of the current operationalization of usability as several dimensions are not captured in the international standard defined by ISO 9241 in 1998. and many others. any single peripheral usability dimension was measured in fewer than 8% of the studies reviewed. The results of the meta-analytical review of empirical research on mobile usability identified 31 usability-related measures. 2008). cognitive aptitude.. 1994).g. Issue 3. appears to be one of the most underserved research areas having been studied only twice in this set of 100 mobile usability studies reviewed.g. May 2011 . The results described earlier enhance our understanding of mobile usability research considerations and serve as the basis for a research agenda in this field. Second. wind) and their effects on mobile technology.. and Flexibility (1%). Nielsen & Levy. anywhere—how do conditions in the environment impact mobile usability? A higher rate of field studies and/or complex lab studies will enhance our understanding of such dynamic factors (e. physical ability) should be considered when studying mobile usability? More research is also needed on variables previously investigated (e. Security (2%). the three core dimensions of mobile usability measurements (i. our understanding of their inter-relationships is mature.g. However. in the context of vulnerable populations/disabled users.g.. and satisfaction) reflect the ISO 9241 standard making a strong case for its use in related future studies. and the increasing levels of legislative support and community interest. which are actually consistent with the standard diminutions of other general usability studies (Brereton. 2005. 2007. Hornbaek & Law. 1994). Recommendations and Conclusion To the best of our knowledge.. efficiency. accessibility. this research is the first analysis of the contextual factors and measurement dimensions investigated in the empirical body of knowledge of mobile usability studies published to-date by leveraging a proposed qualitative review framework for mobile usability.. Adding to the Journal of Usability Studies Vol. including small screens. 6. including its role with the remaining usability dimensions. Memorability (1%).g.  Environment: Usable anytime. Hornbaek & Law. and satisfaction. The main usability measures studied in mobile usability studies are efficiency. Moreover. nonpeer-reviewed) publication outlets. this could offer a unified view of empirical mobile usability studies. Beyond the benefit of a standard view of usability. the results of which expand both the scholarly body of knowledge. First. Second. first. non-peer-reviewed) publication outlets. Journal of Usability Studies Vol. In closing.  Focus beyond the interface—usability is an aggregate experience—when developing applications. this breakthrough meta-analytical research is the first attempt. Lipsey and Wilson (2000). the following measurement considerations are outlined for future research: (a) accessibility—increasing research in this area may improve the usability of products and services for often overlooked audiences. (b) hedonics—which factors impact the aesthetic appeal of a mobile device or service.. and how do they impact usability?. As with all research. Second. while being flexible to accommodate others. to offer a comprehensive view of usability dimensions found in empirical mobile usability studies. even though the meta-analysis of this study followed the procedures suggested by Glass et al.g. In turn.130 earlier research agenda. On the academic level. and Rosenthal (1991). Second. May 2011 . non-IS. The results of this study encourage practitioners to pay more attention to the key contextual factors and mobile usability measurement dimensions when they develop their mobile products and/or services. is warranted. an important opportunity for future research arises from the data in Table 1. Further exploration of this construct. (1981). to our knowledge. Practitioner’s Take Away The following are key points raised in this paper:  Consider the wide range of usability dimensions identified in this study when evaluating the usability of mobile interfaces and applications. it is hoped that the above findings and the suggested research agenda will stimulate further research in this domain. this study comes with the caveat of the following limitations. this observation may come as a surprise. even though the authors searched intensively for all possible research articles of empirical mobile usability studies. and the increasing levels of legislative support and community interest. but also have direct and tangible benefits for everyday users of mobile technology.e. this study summarizes the existing mobile usability research findings and organizes them based on a set of usability contextual factors and measurement dimensions using a comprehensive mobile usability framework. Accessibility appears to be one of the most underserved research areas. Although arguments were given. including its relationship with the remaining usability dimensions. We hope that the framework and the findings of this study will be used as the basis for continuing research that aims to enhance our understanding of mobile usability considerations and measurement.  Design mobile interfaces and applications that fit particular contextual settings. given the growing popularity of accessibility research in less conventional (e.. some subjective decisions were made when two mobile usability dimensions were collapsed into a single measure.. 6. this could be a limitation of a subset of the reported results. the mobile usability framework identified by this study can be used during a usability evaluation of mobile products and/or services. the case may be that relevant articles were omitted in this process.  Study the human factors in HCI. and (c) usability—what are the relationships between various usability measurement dimensions? Should usability be redefined to reflect additional utilitarian and/or hedonic dimensions? This study offers several contributions and implications for both researchers and practitioners. a meta-analysis of measured usability dimensions in a mobile setting). the identification of a common measurement metric with a review framework would support a future quantitative analysis of mobile usability studies at the construct level (i. Again. and identify cognitive factors and physical abilities that future mobile devices could be designed to accommodate. because the current mobile usability evaluation process is more of a ―fuzzy art‖ without a structured framework and there is a need for a more structured approach to evaluate mobile usability. First. Issue 3. This study also provides a couple of important implications for practitioners. Retrieved from http://www. noise) and their impact on mobile usability. C. Jameson.. & Venkatesh. International Journal of HumanComputer Studies.. (2005). Gimpel. D.. Yi. S.. Brereton. Raphael Vonthron. L. R.. Personal and Ubiquitous Computing. (2009). Italy.131  Consider the limitations of the laboratory and conduct research involving real (not simulated) and open tasks through field studies that will offer rich and relevant findings. G. May 2011 .. Mexico. 62(4). Acapulco. Usability measurement in context. (2002). S. R. T. S. (2002). A...shtml Barnard. Usability analysis of wireless tablet computing in an academic emergency department. A. 4. Liaoning. Dalian. Master of Biomedical Informatics. & Chong. M.. Bruijn. Burigat. 10-11. A. Y. 11(2). 132-145. L. 245-252. Presenting dynamic information on mobile computers. V. 188-205. A.. (2004). China. O. Personal and Ubiquitous Computing. Overcoming the lack of screen space on mobile computers. Bødker. J.  Explore the interplay among dynamic factors (e. 6.g. Don't neglect usability in the total cost of ownership.. (1994). & Gabrielli. J. & Macleod.ohsu. Location-aware shopping assistance: Evaluation of a decision-theoretic approach. 81-96. (2008). Yi. Bohnenberger. 6. & Murray. Chittaro. A. Capturing the effects of context on human performance in mobile computing systems. Kruger. (2000). An earlier version of this paper was presented in the 2006 Americas Conference on Information Systems (AMCIS). Information Systems Research. M. Appendix Appendix: Formations and Dimensions of Usability References Agarwal. Issue 3. S. (2005). The authors also thank the reviewers for their comments at the conference. Personal & Ubiquitous Computing.. S. 13(2). urgency. Brewster. Acknowledgements The authors greatly appreciate the detailed. A. 487-520. Journal of Usability Studies Vol. constructive feedback by the reviewers and particularly the Editor-In-Chief. Oregon Health & Science University. Assessing a firm's web presence: A heuristic evaluation procedure for the measurement of usability. Paper presented at the Mobile HCI 2002. 66(2). RSVP browser: Web browsing on small screen devices. Navigation techniques for small-screen devices: An evaluation on maps and web pages. (2002). Brewster. 13. (2007). Jacko. 168186. The authors would also like to thank Jieun Sung. J. Paper presented at the Eighth International Conference on Mobile Business. Dr. 6(4). & Hedman.. Oregon. Behavior and Information Technology.. N. L.. Jacko. Ming Liu. & Sears. Smart phones and their substitutes: Taskmedium fit and business models.. Pisa.edu/library/newbooklists/newbooks200406. Joe Dumas. R. S. An empirical comparison of use-in-motion evaluation scenarios for mobile computing devices. Portland. Communications of the ACM. J. & Butz. E. 209-212.. 47(7). Personal and Ubiquitous Computing. J. M. International Journal of Human-Computer Studies. 78-97. & Sears. and Tuan Le for their assistance in this study. Andon. Barnard. Bevan. Spence. A. (2002)... Journal of Electronic Commerce Research. A. (2001).. (2010). S.. C. & Brzezinski. Paper presented at the the 8th conference on Human-computer interaction with mobile devices and services. Lyons. USA. C. Paper presented at the Mobile HCI 2001.. D. An empirical investigation of color temperature and gender effects on web aesthetics. Italy. A study on the compatibility of ubiquitous learning (u-Learning) systems at university level. 129-149. El Paso. HI. C. A. J. K. Ooi. Usability for mobile commerce across multiple form factors. Geney: Designing a collaborative activity for the palm handheld computer.. An evaluation of mobile phone text input methods. A.. (2007)..-L. H. K. L. Chittaro. Coursaris.. J. C. 40th Annual Hawaii International Conference on System Sciences. & Binshan. Y.. T. (2009).. S. N. Lau. A. & Dal Cin.. Paper presented at the the 25th annual ACM international conference on Design of communication. Fang. J. Journal of Usability Studies. France. (2006). E.132 Butts. L.. O. Issue 3. Ebner. B. Universal Access in HumanComputer Interaction. & Booth. Costa. Evaluating interface design choices on WAP phones: Navigation and selection. M. J. & Hess. Shu. & Kim. Clarkson. A. J. Paper presented at the 2006 Americas Conference on Information Systems (AMCIS).. Swierenga. (2002). L. & Paepcke. A user study of mobile web services and applications from the 2008 Beijing Olympics. C.. In Proceedings of CHI2000. 950-963. Darmawan. 43(8). An empirical study of typing rates on mini-qwerty keyboards. Duh. USA. Personal and Ubiquitous Computing. & Chen. A. K. Chin.. F. Berlin: Springer-Verlag. & Öörni. H. 3(3). Scerbakov.. Duda. Paper presented at the Conference on Human Factors in Computing Systems. Cyr.. 34-43). P. 237-244. J.. Lille. N... Schiel. (2000). M. 6(4).. Usability—Nutzerfreundliches webdesign (pp. Understanding changes in consumer payment habits . & Aparicio. 55 . Stockholm. S. ACM International Conference Proceeding Series. The Hauge. & Ivanov. Davies. 3(3). (2002). K. Paper presented at the Intl. P. International Journal of Mobile Communications... CA. V.. & Holzinger.59 Buyukkoten. Adoption of 3G services among Malaysian consumers: An empirical analysis. Mobile usability: Usability evaluation for mobile device: a comparison of laboratory and field tests. Evaluating web usability using small display devices. Sweden. K. (2007).-h. WA. A qualitative review of empirical mobile usability studies. USA. K. Head. (2001). Information & Management. Seattle.. G. A. K. (2009). & Salomaa.Do mobile payments and electronic invoices attract consumers? Paper presented at the HICSS 2007.. 103117. & Cockburn. (2006). TX. S. Friday. Netherlands.. Acapulco. Coursaris. (2001). Seeing the whole in parts: Text summarization for web browsing on handheld devices. K... E. Mobile usability. Waikoloa.. World Wide Web Conf.. M. Mitchell. Chan. Torino. X. 173-199). N. Applications and Services (pp. Tan. Danesh. Chittaro.. H. Oregon. B. A. 8(2). Design aesthetics leading to m-loyalty in mobile commerce. M. Garcia-Molina. Inkpen. (2005). J.. C. Paper presented at the 20th ACM conference on Hypertext and hypermedia. Clawson. A. (2002). Portland. Paper presented at the CHI2001. Journal of Usability Studies Vol. May 2011 . P. L. 20... A.. Evaluating interface design choices on WAP phones: Singlechoice list Selection and navigation among cards. San Diego.. Stickel. Dahlberg. P. Cheverst. Chong. (2008). & Efstratiou. Developing a contextaware electronic tourist guide: Some issues and experiences. M. Silva. 187-199. (2006).. D. & Dal Cin. T. & Watrall. & Starner. (2002). Mexico. B. 6. (2008). CHI2001.. Australia. M. S. Fang.. The impact of product type on website adoption constructs.. T. Austria. Florence. E. Glass. Huang.. J. Paper presented at the the 2nd HCI in MIS Research Workshop. J. CA. Lu.. Personal and Ubiquitous Computing. G. 8796. Paper presented at the The 21st Annual Conference of the Australian Computer-Human Interaction Melbourne. Texas. H.. Customer loyalty and approach of service providers: An empirical study of mobile airtime service industry in India. R. S. D. London: International Standards Organization. Par 11: Guidance on usability. H. Evaluating reading and analysis tasks on mobile devices: A case study of tilt and flick scrolling. Mobile HCI 2003.. (2009). & Helaakoski. Buchanan. James. S.. (2009). Hornbaek. 47-56. Hsu. A. Sage Publications. 143-151. (2000). Issue 3. (2004). & Hsu. ISO.-F.. G.. J. & Sharma. Pierce. W. (2006).. S.. Sensing techniques for mobile interaction.. K. Pisa. California... K. H.. Kwahk. L. J. E. Paper presented at the the 5th International Symposium on Human-Computer Interaction with Mobile Devices and Services. Paper presented at the Mobile HCI 2002. X. The design and evaluation of a mobile location-aware handheld event planner. Case study of application-based mobile service acceptance and development in Finland. M. Journal of Usability Studies 1(2). C. USA. International Journal of Industrial Ergonomics. Fitchett. Iachello. 91-108. 30(4). May 2011 .. & Grill. M. Hess.. Linz.. Hassanein. The media equation does not always apply: People are not polite towards small computers. C. & Viller. Omega. Meta-analysis in social research.. Brzezinski J.-C. Moghazy. Dallas. H. (2003). Goldstein. Text input for mobile devices: Comparing model prediction to actual performance. J. K.. Environmental context sensing for usability evaluation in mobile HCI by means of small wireless sensor networks. A. S. S. & Bias. M. Paper presented at the UIST2000. B.. M.133 Ervasti. A. L.. Italy. L. Cross-channel mobile social software: An empirical study.. Ergonomic requirements for office work with visual display terminals. H. (2002). Paper presented at the Conference on Human Factors in Computing Systems.. 35(3). & Cockburn.. (2005). Italy. & Smith. G. (2007). San Diego. I.. (2007). 6.364.. 342 . H. Fithian. Hinckley. Paper presented at the CHI 2007 San Jose. Paper presented at the Tthe 6th International Conference on Advances in Mobile Computing and Multimedia. Alsio. K. Chou. Seattle. Empirical evaluation of a popular cellular phone's menu system: Theory meets practice. Paper presented at the the Sixth International Conference on Electronic Commerce Research (ICECR6). WA. Meta-analysis of correlations among usability measures. H. Yun. (2008). & Hong. Brereton. (2010). (2003). 6. Han. 28(3-4). M. & Stasko. R. (1981). Jones. D. (2002). M. A. K. C. Gupta. Pousman. (2003). Heyer. Sinclair. & Werdenhoff. (2001). M. Journal of Usability Studies Vol. Reischel. & Thimbleby. Chan. & Head.. G. & Xu. Services Marketing Quarterly. Udine. Hummel. USA. Jeng. USA. 24(2). P.. 243-259. Z.. & Law. A study of task characteristics and user intention to use handheld devices for mobile commerce. Usability of consumer electronic products. Adoption of the mobile Internet: An empirical study of multimedia message service (MMS). J. & Horvitz. 715-726. G. S. (2001). M.. McGaw. Italy.. International Journal of Information Technology and Management 9(3). What is usability in the context of the digital library and how can it be measured? Information Technology and Libraries. Sorting out searching on small screen devices.-C. Kallinen. M. 418-435. Kim. Y. K. Kim.. S. (2005). Ristola.. A. Information Systems Frontier. Decision Support Systems. (2002). D. H. H. 6... First time usability testing for Bluetooth-enabled devices. 8(1).. J. Ruyter. T. Y. Kim. 29-52. & Stage. A. O. J. An empirical study of use of contexts in the mobile Internet. Evaluating the usability and suitability of mobile tagging media in educational settings in a developing country. & Wetzels. M. Udine. An empirical study of the use contexts and usability problems in mobile Internet. & Graham. & Cockburn. emotional responses. Austria Kargin. Paper presented at the IADIS International Conference Mobile Learning 2009. S.com/enus/library/cc889361(v=office. Applications and Systems. A. & Voegele. & Lee. Journal of Retailing.. Paper presented at the The 35th Hawaii International Conference on System Sciences Kim. King. Kaikkonen.. Keeker. & Cheng. A review of mobile HCI research methods. Kaasinen. International Journal of Human-Computer Studies. Y. Journal of Usability Studies. & Kesti.. E.. (1997).. (2004). (2003). M. B. B. & Daim. An assessment of value creation in mobile service delivery and the moderating role of time consciousness.. Improving web-site usability and appeal: Guidelines compiled by MSN usability research. Focusing on the usability of information architecture. 1(1).aspx Khalifa.. Chae. Proceedings of the 35th Hawaii International Conference on System Sciences Kim. 33-46. Kekäläinen. H.. (2009). M. S. Value-based adoption of mobile Internet: An empirical investigation. Kjeldskov. 79(6). C. S. T. Italy. Kleijnen. Marsden.. The effects of background music on using a pocket computer in a cafeteria: Immersion. N. (2007). (2005). 53(1). Kim. Predicting consumer acceptance in mobile services: empirical evidence from an experimental end user environment... 53-70. (2003).. Koivumäki. C. 4(4). International Journal of Mobile Communications. (2007)... A.microsoft. A longitudinal study of usability in health care— Does time heal? International Journal of Medical Informatics. T. I. J. M. Juola. Chan. A.. Basoglu. 7(2). B. A. Paper presented at the Conference on Human Factors in Computing Systems Vienna. Spain.. Lee. Issue 3. and social richness of medium. 7(1). (2010). G. M. Jung. A. et al. Skov. 135-143. Communications of the ACM. Kaikkonen. 111-126. Y. An evaluation of integrated zooming and scrolling on small screens.. Usability testing of mobile applications: A comparison between laboratory and field testing. (2002). & Choi. 63(3).. Adoption of mobile commerce: Role of exposure. Kankainen. Kjeldskov.. (2009). J. International Journal of Mobile Communications. The University of Kansas.0 with multidisplay buttons. & Krogstie. K. & Cankar. International Journal of Services Sciences 2(1).. D. Usability problems in today’s mobile Internet portals. P. Lee. Kofod-Petersen. 43(1). S.. (2004). Paper presented at the The 5th International Mobile HCI 2003 conference. Patel. A.. Lee. 70-79. J.. Jones. (2005).. Personal and Ubiquitous Computing. & Gupta. (2010). (2006). & Mbogho. May 2011 .. (2010). 4-16.11). K. Kallio. Paper presented at the 2nd International Conference on Mobile Technology. (2005)... Kim. K. 271-303. An empirical investigation of attitude towards location-aware social network service. Gransæther. Journal of Usability Studies Vol.. Retrieved from http://msdn.134 Jones. Factors affecting the adoption of mobile services. A. H. J. 175-186. M. S. A. Park. 136-141. Barcelona. J... Mobile web 2. W. User needs for location-aware mobile services. 83(1). J.. W. 5. S. Goldman. Computers in Human Behavior. J. Kowatsch. 231257. Li. Kober. Sears. (2006). P. Applied Cognitive Psychology.. International Material Data System.. J. Thousand Oaks. 419-431. (2001). Applied Ergonomics.. K. (2003).. 65(9). R. & Cheng. H. 5(2). MacKenzie. E. (2007). C. Hwang. 673-684. Jones. Y.. Managing one‘s availability to telephone communication through mobile phones: A French case study of the development of dynamics of mobile phone use. (2009). (In Press). W. Austria. Huang. G. Mobile usability—rigour meets relevance when usability goes mobile. C. & Fleisch. (2001).. Maass.135 Koltringer. J. Practical meta-analysis.. S.. International Journal of Electronic Commerce. Athens. T. Langan-Fox. I.. D. The use of free and paid digital product reviews on mobile devices in in-store purchase situations.. (2001). Lipsey. Platania-Phung. K. A. (2006). & Rasmussen. 5. C. (2010). Diversified users' satisfaction with advanced mobile phone features. An empirical and theoretical evaluation of BinScroll: A rapid selection technique for alphanumeric lists. H. I. & Waycott. Increasing trust in mobile commerce through design aesthetics.. T. (2007).. J. (2001). Personal and Ubiquitous Computing. J. T. mental models and abilities on task and recall performance using a mobile phone network. 46(12). M. T. T. Kwahk. 33. International Journal of Human-Computer Studies. A methodology for evaluating the usability of audiovisual consumer electronic products. H. Universal Access in the Information Society. M. 6.. C. 587-634.. & Benbasat. 43(1). Y. (2001). Personal and Ubiquitous Computing. Li. & Wilson. Paper presented at the The 4th Mediterranean Conference on Information Systems. P. 1154-1169. (2002). & Salminen. Norway. & Salvendy. Nilsson. International Journal of HumanComputer Studies. H. I. & McQueen. 123-136. (2007). Methods to support human-centered design. R. 413-432. Lee. Y. S. S. Cheng.. R. H. Personal and Ubiquitous Computing. Decision Support Systems. 239-249. A framework for the study of customer interface design for mobile commerce. Adoption of mobile technology in business: A fit-viability model. & Yeh. Liang. M... Paper presented at the UIST 2001. 6. Kurniawan. S. 99-108. Smith. 95-110.. 16(4)... Vienna.. Exploring consumer adoption of mobile payments—A qualitative study. 20(9). 141-150.. We release them little by little: Maturation and gender identity as seen in the use of mobile telephony. & Grechenig. Mallat. H. W. P. 48-52. Effects of advance organizers. Price. International Journal of Human-Computer Interaction. International Journal of Mobile Communications 6(2). (2004). Lin. 55. C. Journal of Strategic Information Systems. (2002). Lee.. 26(4). An Empirical study of mobile commerce in insurance industry: Task-technology fit and individual differences. (2000). J. J. Older people and mobile phones: A multi-method investigation.. Comparing the immediate usability of Graffiti 2 and Virtual Keyboard. Lehikoinen. Greece. Issue 3. Ulvik. N. & Skepner. Y. & Han. Licoppe. C. D. M. How do people tap when walking? An empirical investigation of nomadic data entry. W. CA: Sage Publications. Paper presented at the IRIS24.. & Heurtin. LetterWise: Prefixbased disambiguation for mobile text input. May 2011 . Lindroth. Ling. Barriers to mobile commerce adoption: an analysis framework for a country-level perspective.. Ling. J. (2007). Journal of Usability Studies Vol. Maguire.. E. Paper presented at the Conference on Human Factors in Computing Systems. 107 (8). & Jacko. T. 1143-1165. 759-769. (2008). E.. & Yeh. 119-127. M. & Raulas. 46.. B. United Kingdom. K. Maruyama.. Massimi.. & Ramesh. Rosenthal. It's worth the hassle!: The added value of evaluating the usability of mobile systems in the field. P. H. 7. Paper presented at the 4th Nordic conference on Human-computer interaction. & Blackwell.. M. Usability study on the use of handheld devices to collect census data. A. Ross. 5. Mihailidis. Journal of Interactive Marketing. Journal of Usability Studies Vol. Personal and Ubiquitous Computing. B. Development of a wearable Computer Orientation System. Qiu.. 59-66). F.. L. Rodden. & Thurnher.. M. & Youngs. & Vartiainen. 9(2).. M. E. Kajalo. J. Measuring usability: Preference vs. & Blasch. (1994). Paper presented at the the 38th Annual Hawaii International Conference on System Sciences (HICSS'05). Discovery and integration of mobile communications in everyday life. 8(4). (2004). 49-63. Merisavo. M. 6.. (2003). and Turkey. & Levy.. 37(4). CA: Sage Publications. Pousttchi. B. (1993). May 2011 .. Khatri. & Yaprak. J. (2006). & Salzman. 4(2). S. Pedersen. Beyond the handset: Designing for wireless communications usability... M. Paper presented at the Human Factors and Ergonomics Society 47th Annual Meeting. Personal and Ubiquitous Computing. Arponen. H. L. E. (2005). Paris. R. M. International Journal of Mobile Communications. Vesanen. Nielsen. (2004). 66-75. J. & Sheng. The effectiveness of targeted mobile advertising in selling mobile services: An empirical study. Stage. (2006). S. F.. S.. M. D. (2008). Understanding effects and determinants of mobile support tools: A usability-centered field study on IT service technicians. (2003).). (2002). & Huang. A. R. M... Pagani. A. An empirical study of web interface design on small display devices. 85-90. New York: AP Professional. Communications of the ACM. M. Paper presented at the UIST2002. I.. QC. J. In A. Palen. Technology and aging—Selected Papers from the 2007 International Conference on Technology and Aging: Vol. L. Usability engineering. E. O. Siau.. (2002). (2005). Olmsted. S. & Baecker. Massey. V. K. Overgaard. Roto. Paper presented at the ACM CHI 2006. (2001). Paper presented at the ICMB '06. & Rekimoto. R. Palen. Normie (Eds.136 Mao.. Nielsen. Srite. J. M. Salzman. Kautz & L... From the web to the wireless web: Technology readiness and usability.. K. N.. (2004).S. A.. Paper presented at the Proceedings of the 17th Annual Conference on Human-Computer Interaction. France. Determinants of adoption of third generation mobile multimedia services. Zhang.. F.. Paper presented at the Professional Communication Conference. 21 (pp. B. performance. (2006).. The value of mobile applications: A utility company study. ACM Transactions on Human Computer Interaction. Ambient touch: Designing tactile interfaces for handheld devices... B. M. Paper presented at the IEEE/WIC/ACM International Conference on Web Intelligence (WI' 04). J. (2002). 6. 18(3). V. 109-122. M. 125-151. Thatcher. Multitasking and interruptions during mobile web tasks. Nagata. Nielsen. Nah. Issue 3. Sommerer. Minimap: A web page visualization method for mobile phones. Newbury Park. Journal of Global Information Technology Management. A research model for mobile phone service behaviors: Empirical validation in the U. Boger.. E. V. Bath. Canada. Milic-Frayling. C. (2006). Popescu. 48(2). K. J. K. An empirical study of seniors‘ perceptions of mobile phones as memory aids. J. Meta-analytic procedures for social research. (1991). Effective web searching on mobile devices.. & Stenild. Poupyrev. A. Koivisto. International Conference on Mobile Business. (2005). Montreal. Communications of the ACM. A. V. (2009).. & Matthews. Journal of Usability Studies Vol... C.137 Ryan. In Memoriam: Professor Brian Shackel 1927-2007). UK: UMTS Forum 2005. December.. (2003). Interacting with Computers. Shackel. G. Cairns. Tomazic. Usability – Context. M. (2010). 9(2). Ambient. T. 79(4). Mobile devices as props in daily role playing.. Tarasewich. D. 524–535). & Wells. P. Venkatesh. Understanding usability in mobile commerce. N. LNCS#3585. TiltText: Using tilt for text input to mobile phones. (1991). M.. Dicke. Communications of the ACM. Suzuki. (2005). B. 35-40. & Billinghurst. 69-73. P. Issue 3. 24-34. Communications of the ACM. S. (2003). Ramesh. A. Y. S.. Web and wireless site usability: Understanding differences and modeling use. Paper presented at the the 16th Annual ACM UIST Symposium on User Interface Software and Technology. Introduction to the new usability.. definition. In B. The effect of context and application type on mobile usability: An empirical study.). Alsosa. A. definition. V. V. A user study of auditory versus visual interfaces for use while driving. Thomas. France. A. Sodnik.. Leshed. J. framework.. 19-34. 46(12). Ubiquitous and Intelligent Interaction (pp. & Ramesh. (2001). & Gupta. Svanæsa. framework. UMTS-Forum. Human Factors for Informatics Usability (pp. Zhong. T. G.. Managing the customer perceived service quality for cellular mobile telephony: An empirical investigation.. Lv.. & Massey. W. J. Paper presented at the Mobile HCI 2001. Nakao. Z. Waterson. (2003). M. Magic mobile future 2010-2020 Report No 37. (2002). R. ACM Transactions on Computer-Human Interaction. Learning by playing.. Paper presented at the Twenty-eighth Australasian conference on Computer Science. Yee. design and evaluation. Vikalpa: The Journal for Decision Makers. A. Seth. H. CA: Springer Berlin/Heidelberg.. Empirical comparison of task completion time between mobile phone models with matched interaction sequences. O. Context of use evaluation of peripheral displays INTERACT 2005.. Zhang. (2005). Wang. 8(2). 33(1)... A. (2009. Minnesota. & Fukuzumi. Usability testing of mobile ICT for clinical settings: Methodological and practical challenges. P. 46(12). San Diego. Lille. May 2011 . (2008). S.. Momaya. & Dahla. 69-73. V. International Journal of Medical Informatics. (2002). & Macredie. Thimbleby. & Jones. ACM Transactions on Computer-Human Interaction. Y. Understanding mobile handheld device use and adoption. 114-122). S. Paper presented at the Conference on Human Factors in Computing Systems Minneapolis. (2006). D. R. 21 (56). International Journal of Human-Computer Studies. (2005). Cambridge: Cambridge University Press. MIS Quarterly. Usability-context. P... (2009). Shami. Venkatesh. L. Berlin/Heidelberg: Springer. 46(1246). Strom. USA Wigdor. S. In the lab and out in the wild: Remote web usability testing for mobile devices. 318-332. & Gonsalves. V.. Communications of the ACM. (2001). D. Game-based education system design and development (pp. C.. H. B. & Wang. Empirical research and design of Mlearning system for college English. (2008). 579-587. Bellotti. London... Sarker. 53-56. Asahi. K. 66(5). S. 6. S. N. S. & Klein. 57-60. Landay. 181-206. design and evaluation.. J.. 339-346. Shackel & S. 21-38). Designing mobile commerce applications. (2003). 30(1). Shackel. Richardson (Eds. Human-Computer Interaction.. Usability analysis with Markov models.. & Balakrishnan. DSS. S..-H. IJHCI. 6. & Zheng. Recently he has focused on trust in e-commerce. Australia. JGIM. 710-724. and virtual world. usability. His research interests are in multidisciplinary areas such as e-commerce. S.. His formal training consists of a B. and Media at Michigan State University and an expert in human-computer interaction.. J.E. His research work has been published in more than 89 papers in refereed journals and conference proceedings. and a PhD in Information Systems with a concentration on eBusiness and mCommerce. Paper presented at the International Conference on Mobile Business (ICMB 2005) Sidney. 719-729. Stationary work support to mobile work support: A theoretical framework. Information and Management 42 (5). Liao. Combining empirical experimentation and modeling techniques: A design research approach for personalized mobile advertising applications Decision Support Systems. IEEETPC. in Aerospace.A. What drives mobile commerce? An empirical evaluation of the revised technology acceptance model. information security. He has a second appointment with MSU‘s Usability/Accessibility Research and Consulting.-C. May 2011 . He leads a number of international projects in Japan. an MBA in eBusiness.. 44(3). Issue 3. Follow him @DrCoursaris. Yuan. wireless and mobile commerce. strategy and marketing. Greece. France. Information Studies. mcommerce. JOEUC. (2005). CACM. About the Authors Constantinos Coursaris Dan J. D. & Wang. and so on. Coursaris is an Assistant Professor with the Department of Telecommunication. (2008). JMIS. web usability. S. CAIS. including ISR. (2005). Q. Xu. Canada. Jordan. Y. W. Kim is an Associate Professor of Computer Information Systems at University of HoustonClear Lake (UHCL). & Li. J. IEEE IT Professional. and assurance. mobile and social media.138 Wu. Journal of Usability Studies Vol. Saudi Arabia. Kim Dr. Dr. and the U.Eng. Electronic Market. and frustration. Scrolls -Reading time was fastest on a treadmill in high light. Score. or free walking along a path. reliability. 2004 N/A Job: Physicians (attending and resident) N/A Tablet PCs Lab & Field -Focus group -Survey (9) Errors Weight. -Word search time was fastest on treadmill in both high & low light. May 2011 . effort. Effect of lighting differences. 2005 N/A Closed . -Response time was fastest walking in high light. -Subjective measures are more sensitive to changes in conditions then performance measures. walking on a treadmill. 6.. wireless infrastructure. Undergraduat e students Journal of Usability Studies Motion & light Vol. Issue 3. interconnectivi ty Weightacceptabi lity Barnard et al. performance.139 Appendix Formations and Dimensions of Usability Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Technology/ system/product Environment Andon. technical support.Users had to perform a set of tasks (reading comprehension. Workload. Palm m505 (PDA) Lab -Experiment (126) -Survey -Device data Reading & response time (therefore efficiency) effectiveness Salience. word search) while sitting. security. -Walking caused most mental demand. 2002 Novices Open Shopping PDA Field -Experiment (20) -Survey Effectiveness Efficiency Adoption PDA  less time. Assurance. Empathy.. Referent. Customer perceived network quality Responsiveness was the best predictor of service quality in cellular mobile context. customer perceived network quality. effectiveness Journal of Usability Studies Vol. Measurement instrument for SERVQUAL and functional quality Bødker et al. Issue 3. Context. empathy. effort.. sell shares Palm V (PDA) Lab -Experiment (12) -Device data Effectiveness Efficiency N/A Audio presentationeffi ciency. 6. SMS. 2000 N/A Job: Students Open . convenience. and Task medium fit Perceived quality differences between a new option and the referent impact the decision Context  Perceived Usefulness & Appropriateness of the medium Bohnenbe rger et al.Used the iPhone for 6 months (email. Perceived quality.. and frustration Brewster & Murray. Responsivenes s. 2007 Experts N/A Technology/ system/product Mobile services on mobile phones Environment N/A Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Interviews (20) -Survey (225) -Age: 18-50+ N/A Reliability.140 Usability Formation of usability studies User Task/activity Barnard et al. GPS. MP3 iPhone 3G with voice. Omnipresence.Search trade information. and tangibles. followed by reliability. Web. Convenience. assurance. May 2011 . Perceived appropriatenes s. 2009 N/A Culture: Denmark Open . SMS and data plan Field -Experiment -Surveys -Focus groups -Interviews Perceived usefulness 16 participants Prior/post use of ICT. Tangibles. cognitive effort. 2002 Burigat et al. more frustration and performance. small buttonmore workload. Journal of Usability Studies Vol. 2001 NoviceExperts (computer exp. less data entry Novices Closed (navigation to find the answers of questions) Mobile phone Lab -Experiment -Device data (30) Efficiency Steps. browser WAP is more efficient and significantly fewer steps than RSVP N/A Closed Navigated large maps & web pages on small screen & used spatial memory acquisition 624 Mhz Pocket PC Lab -Experiment -Interview -Device data Accuracy Driving performance.. button size. more data entry. 2002 Novices (students and staff) Bruijn et al. sound type. 6.141 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment Closed Entering a series of five digit codes using the numeric keypad Palm III (PDA) Lab Experiment (12.. 2008 Undergraduate or graduate students Butts & Cockburn. search tasks using different methods) Palm (PDA) and mobile phone Lab -Experiment (15) -Device data Efficiency Performance (user and system) Combination of keywords and single-sentence summaries provides sig.). 2002 Experts Closed (enter five sentences using each input method to send SMS) Mobile phone Lab -Experiment -Device data -Observation (8) Efficiency Error Learnability Acceptability Text entry interface Reliable differences in efficiency among different text entry interfaces. acceptability given to certain text entry interfaces over other forms Buyukkote n et al. 16) -Device data -Survey Effectiveness Efficiency The amount of data. level of concentration Closed (accomplish single-page info. no learnability difference. May 2011 .. Issue 3. workload Sonicallyenhanced buttons less workload. improvements in efficiency. Preferences -Grab and drag  Performance in a task involving little navigation -Double scroll bar & zoom enhanced navigation  Performance & user orientation in a task involving larger amounts of navigation studies User Task/activity Brewster. stock quotes.. and TLX) Chittaro & Dal Cin. searching and buying a book. effectiveness) Contextual factors (task type. 2009 N/A Closed . etc. WAP-enabled mobile phones. 2002 Novices Closed (search and selection) Mobile phone Lab -Experiment -Device data (40) Efficiency. & pocket PC Lab -Experiment -Device data (6) Cheverst et al. May 2011 . Flexibility Interface friendly (thus. depth of site structure.Two tasks (reading comprehension & word search) in high vs. 2002 NovicesExperts Open Checking & booking a flight. low light.Search and selection WAP phone Lab & Field -Experiment (40) -Survey Efficiency Perceived difficulty N/A Chittaro & Dal Cin. score. PDA. and TLX) Different lighting> all experimental measures (time. and TLX Word search task: Different motion > all experimental measures (time. Issue 3. scrolls. connection feedback and latency Strong relationship between ecommerce and mcommerce studies User Task/activity Chan et al. NASA TLX (subjective workload assessment) Reading comprehension task: Different motion -> reading time User motion (and light) Students Environment N/A Different lighting> response time. score. 2001 Novices Gender Open . 2000 Experts Job: Visitors N/A GUIDE prototype Field -Interview -Observation (60) Error.142 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Information overload. ease of use) N/A N/A Chin & Salomaa. walking on treadmill. walking along a path around a room) -Experiment (80) -Observation Completion time (thus. while seated or walking PDA -Lab -Light & user motion (sitting. efficiency) Score (thus. Operability Screen interface Sig. search.. differences in Efficiency and Operability among different screen interface Journal of Usability Studies Vol. 6. motion and lighting level). ease of use) .Usefulness (lack of real valueadding m-services. Reliability Using links is more efficient and effective as a navigation option than scroll. and ―folio.‖ Desktop QWERTY Vol. Touch Screen studies User Task/activity Chong et al. -Desktop Qwerty had faster speeds than Mini QWERTY. Lab -Experiment (8) -Device data Efficiency Effectiveness Integrity. unawareness. either mcommerce participants or facilitators Clarkson et al. fulfillment issue) Technology Self-efficacy Usability (learnability.Simplicity in use. 2005 Costa et al. 6. ease of use.. usefulness)  mcommerce service adoption -Purposeful sampling 10 participants Lab -Experiment (14) -Demographics -Survey -Device data WPM (therefore efficiency) Accuracy Comfort -Targus group typed faster & typing speeds of users improved over time.. search & scroll for answers. & write in text box PDA. folio. 2007 Novices N/A Job: Architecture Students Journal of Usability Studies Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Interview: Face to face or phone .Read questions. ease of access to relevant information or services.. search. -Users found the mini-QWERTY marginally comfortable & much less comfortable than full keyboard. thus. navigate Web via links. Issue 3. User friendliness (interface characteristics. May 2011 . 2010 N/A Culture: New Zealand Job: From 10 companies.143 Usability Formation of usability Technology/ system/product Environment N/A .Asked about their awareness of m-commerce services & adoption barriers N/A N/A Closed Complete 20 sessions in 11 days on Mini QWERTY & full QWERTY -10 phrases in 20 sessions -Typed with thumbs Dell (Dell Axim) and Targus (Palm m505) Brand Open . Culture: Chinese (30) or Canadian (30) in origin Technology/ system/product Mobile phone (Nokia 6600 device) Environment N/A Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Experiment (60) -Survey (60) -Interview (60) Design aesthetics Usefulness Ease of use Enjoyment. Issue 3. Mloyalty Design aesthetics  Perceived usefulness. ease of use. and enjoyment in mobile context Perceived ease of use  Perceived usefulness Perceived usefulness and enjoyment  Mloyalty (user willingness to revisit a site) No significant differences between Canadians and Chinese (living in Canada) Journal of Usability Studies Vol. 2006 Experts Open . complete survey. May 2011 . 6.Choose restaurant on cell phone using bookmarked site (CityGuide).144 Usability Formation of usability studies User Task/activity Cyr et al. and complete openended interview questions.. 6 seniors with MCI 5 normally aged seniors Danesh et al. Interactivity. Organizers (called ―memory books‖). impersonal nature of tech. May 2011 . Seniors designed their own mobile phone software for memory support and provided rationales for their design choices. Creating a routine of use. Proactive alarms. fear of changing routines/breaking phone.Age: 29 (av.and output (SIO). Ease of carrying. Consolidated information. Flexibility and revision. mobile PDA/phone (iMate K-JAM model) Seniors with mild cognitive impairment and/or memory loss Key usability dimensions/ constructs** Other variables Key findings Environment Research methodology* (sample size) Lab Geriatric hospital and research center -Experiment -Observations -Participatory design sessions N/A Portability.WAP services exploring WAP phones N/A -Experiment (36) (B2C service) .. radiation & health concerns. hardware designed inappropriately for seniors. Easy backups. -Barriers were poor conceptual design. Transference of data.Kept a paper organizer & used their new organizers (called ―memory books‖). use album. Issue 3. complexity. 2002 Experts Open . Communicatio n support -Mobile phones were one of the most feasible platforms for memory support technologies. Feeling of control In order of importance: Utility  Acceptance Usability  Acceptance SIO  Acceptance Feeling of control  Acceptance Speed  Acceptance Lower cost  Acceptance Gender (18 male.145 Usability Formation of usability Technology/ system/product studies User Task/activity Dahlberg & Öörni.Survey ..Observation . -Commercial phones targeted at seniors should support memory aids and ease of using. 6.) Speed (therefore efficiency) Acceptance (therefore acceptability) Utility Usability System in. 18 female) Journal of Usability Studies Vol. 2001 N/A Elementary Students N/A.Interview . 2007 N/A Open . drawing Palm (PDA) Lab -Experiment -Device data -Observation (14) Error N/A N/A Duda et al. Attitude. Intention to use mobile service PU -> ATT Culture: Singapore Environment 12 interviewees Culture: Turkey Ervasti & Helaakosk i. Utilization of context– specific information. 6.Mobile Services N/A N/A -Interview Ease of use Usefulness Cost. Ebner et al. PDA Lab & field -Survey (100) -Experiment -Observation Effectiveness Contextual awareness. image. quality and delivery time of content) -Usefulness & Ease of use is most important in m-service adoption. -Social influences are more important than user characteristics in terms of social aspects.. problems observed There were many more types and occurrences of usability problems found in the field than in the laboratory. Reference group.. Content (correctness. May 2011 Perceived usefulness Perceived ease of use -Service aspect (content and mobility) is more significant than social aspect.). PEOU -> ATT CON -> INT ATT -> INT INT -> USE . Enjoyment/ent ertainment. Social influence (external influence).146 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Duh et al. download Mora mobile application and use it Mora mobile service Field (Campus) -Experiment (two months) -Survey 52 participants Vol. 2009 NovicesExperts N/A . Barriers to use. 2006) N/A N/A Mobile phones. User characteristic. Perceived behavior control. Mobility. 2010 N/A Culture: Finland Journal of Usability Studies Closed Register via webpage. (innovativenes s. Issue 3. etc. visual attention. task hierarchy. hand manipulation & mobility. Adding a visit card & make an appointment PDAs or Smart phone Lab -Experiment -Survey (25) Attitude. moving 1 and moving 2 (differ by kind of task) -Observation Delay (thus. experience with stylus and PDAs & with IM & SMS writing Closed . view event details & attendee locations PDA/phone combination Field -Experiment -Survey (9) -Interview -Observation Ease of use..Evaluate characteristics of the devices . flick scrolling) 14 postgraduate students -Observation -Survey Task times (thus. Perceived security (PS) PU. Usefulness.Locate an individual & send a message. efficiency) Error rates Scrolling percentage preferred walking speeds. Preferences Tilt outperformed flick scrolling when stationary (faster task completion times & fewer errors). PS. Both performed similarly while moving. Learnability. stationary. alumni. Issue 3. Sound Acceleration Temperature Humidity -Experiment (3 runs: Sitting. studies User Task/activity Fang et al.Mobile commerce tasks N/A N/A . PP. tilt vs.. May 2011 . PP  Intention* *intention to perform a task Closed . students Research methodology* (sample size) Involved a text task & a grid task Fithian et al. caused by movement and this environmental setup. 2003 NovicesExperts: Age.Survey .Experiment (101) . Performance (therefore efficiency) Proximity between target and questioning source N/A Gupta & Sharma. 2009 N/A N/A Mobile smart phone (Qtek S200) -Lab Light. efficiency) error rate N/A User performance (in terms of delay and error rate) decreased. 2009 N/A Adults.Age: 20-50 Ease of use Playfulness Usefulness Perceived task complexity (PTC). Effectiveness. EOU Intention* positively Mostly PU. therefore efficiency) Appreciation Task Completion Time (-) Participant‘s Experience with Stylus and PDAs. 6. 2002 Novices Closed .. tilt scrolling iPod Touch Lab -Experiment (walking vs. and with IM and SMS writing Goldstein et al.147 Usability Formation of usability Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment N/A . Culture: Austria (with sensors to capture data) Journal of Usability Studies Vol. but users preferred and walked faster with flick scrolling. PS. Performance (note: task completion time. 2003 N/A Fitchett & Cockburn.Flick scrolling vs. Parents (6)..Visual tracking (simulate driving) Palm-sized devices (PDA) Lab -Experiment (7) -Device data -Age: 30-50 Errors Ease of use Sensing Techniques (ST). Recommendati on to others Skills. Students (7).Use of the mobile service (Smart Rotuaari) SmartRotuaari: mservice with wireless multiaccess network. Barriers include the mobile payment market. 2008 N/A N/A .148 Usability Formation of usability Technology/ system/product Environment N/A N/A Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Focus group -Interview N/A Relative advantages. Guidance and support. Impact of use situations M-Payment more important in presence of queues. Perceived risks and trust in mobile payment service providers. Middle-aged (9) Random sample in Finland. Young adults 1 (8). May 2011 User satisfaction. middleware. web portal with content provider interface (CPI). Costs. low adoption rates.. complex solutions. but students dominant group 192 Participants Vol. studies User Task/activity Heyer.Good Design  EOU . unexpected need for a payment.MPayment Hinckley et al. Perceived external resources (guidance and support offered). Usefulness -> Recommendation to others . Likelihood of future use. perceived risks and perceived incompatibility with large value purchases. Young adults 2(8). 6. time pressure. 2000 N/A Closed . Network externality. Brereton. Usefulness -> Likelihood of future use Culture: Finland Culture: Finland Journal of Usability Studies Teens (8). and lack of cash or loose change. Issue 3. & Viller. 2007 N/A Open . Usability . & collection of functional context-aware mobile multimedia services/PDA Field (field-office located on the Rotuaari Pedestrian area in Finland) -Experiment -Survey Usefulness Ease of use Satisfaction Perceived internal resources (skills). Compatibility. premium pricing.ST  EOU for certain tasks Hsu et al. Design. Complexity. find ―Welcome Note. set phone on silent mode Mobile phones (Nokia Series 40 Developer Platform 1..Check received calls. accessibility) -Cell phone‗s menu selection -Limited size display Users prefer a less extensive menu structure on a small screen device.149 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Huang et al. efficiency) Complexity Complexity  errors 11 Nokia users. 2006 Novicesexperts Closed ..Mobile Internet Mobile phone Field -Survey (online and participants recruited via ads in forums) Usefulness Perceived fee Perceived value Enjoyment Technicality -Value perception is a major determinant of mInternet adoption.‖ turn on vibrating alert. 2008 Experts Culture: Singapore Environment 161 participants James & Reischel 2001 NovicesExperts Journal of Usability Studies Closed . 8 non Nokia users Hummel et al. Issue 3. group 2(10)] Satisfaction Error Effectiveness (and success rate) Efficiency (and time) Number of attempts (thus. find wireless internet access. N/A . 6. -Mediating effect of perceived value on customer‘s benefit (usefulness and enjoyment) and sacrifice related beliefs (technicality and perceived fee)  Customer‘s adoption intention Errors Time (thus. May 2011 .0) Lab -Survey (19) -Experiment (19) -Interview (19) -Focus group [group 1(9).Text typing (multitap and T9) Mobile phone N/A -Experiment (20) -Observation -Age: 18-45 Vol. 3 tourist type . Buchanan. mobile phone N/A -Experiment (48) -Surveys -Observation (monitoring and recording) Satisfaction Errors Attitude Acceptability Make the device work (MTDW).Locate document .Frustration (F) .task for each scenario PDA Lab -Experiment (12) -Observation -Survey Errors Ease of use Time (therefore efficiency) .PDA interface . 2003 NovicesExperts: Men. & Thimbleby .Establish Bluetooth . Workload Decreased screen space decreased the impact of SDAZ 1-D navigation: normal interface is better than SDAZ 2-D navigation: supports a more accurate target location & longer task completion. 2-D vs. action timings.Experiment (55) ..Small SS errors . Issue 3. SDAZ) Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Experiment (12) -Survey Efficiency (and task duration. half satisfaction Bluetooth  Acceptability (favorable attitude to use) Kaasinen. experts) Closed . Undergraduate or post graduate students Jones.WAP interface .150 Usability Formation of usability Technology/ system/product studies User Task/activity Jones et al. youth w/ various backgrounds Closed .Group interviews . women.PDA interface  EOU . mobile phones…) Lab & Field . Intention of adoption Use  Satisfaction Satisfaction  Intention of Adoption MTDV  Errors. SDAZ requires less interface actions & less physical effort than the standard interface .Two sets of 24 tasks. Stand scrolling and speed dependent automatic zooming (SDAZ) tasks Standard desktop Computer and Compaq iPAQ Lab Environment Scroll vs.) accuracy User interface actions.WAP I.Screen size (SS) .3 scenarios . 2002) Novices Volunteers (University students.Small SS  TC . Zoom (1-D vs. 2005 N/A Closed . .Follow instructions using a GPS system Different GPS devices (PDA.Add contact entry Bluetooth devices.Small SS  F Juola & Voegele 2004 N/A Job: Undergraduat e students (engineering & psychology) Closed . < PDA I.Create calendar .Device data -Age: 14-66 N/A Location aware features Location aware features  Enhance mobile services Journal of Usability Studies Vol. 6. May 2011 . 2009 N/A Open .Order the mobile service following permissionbased SMS advertising (communication . & Daim. music. Effectiveness of m-advertising varies between customers with different content preferences (entertainment. May 2011 . or mixed) and service usage levels (heavy. positive/negati ve emotional response. Issue 3. surrounding noise. 2005 N/A Kaikkonen . information. information and entertainment) on demand by sending SMS a specified keyword to local short number N/A N/A -Survey: Control group (3047)–no SMS marketing. 2004 N/A Closed . Usage class. Average daily expenditure Permission-based mobile advertising  Increased sales of mobile services. treatment group (2453)–got SMS marketing N/A Content preference (entertainment . NovicesExperts (mobile Internet user) Open Navigation Mobile phone (Nokia Series 60 smart phones & occasionally others) N/A -Experiment -Survey -Usability tests (6) -Expert evaluations (12) Errors Problems observed Navigation in mobile portals Kallinen. medium. or light users). perceived social richness. mixed). information. time of use No music  Attention affected by SN Music  Time of use longer Music  Immersion Music  Positive Emotional Response Music  User Satisfaction Music  Perceived social richness Kargin. Basoglu.Read a story on a PDA. attention. 2005 Culture: Finland Journal of Usability Studies Vol. other problems listed/ observed The number of times errors/problems were observed in the two settings. studies User Task/activity Kaikkonen et al.10 tests Mobile phone Lab & field -ExperimentDevice data (40) -Lab study (20) -Field study (20) Errors Learnability Operability Navigation and operability problems.151 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment Closed . 6. with and without listening music PDA Field (cafeteria) Noisy Public -Experiment (30) -Device Data -Survey -Age:15-47 Satisfaction Immersion.. Keep pocket diary & fill in forms with each use of internet. Communicatio n. hand & leg.Comparing paper and econtent . Auditory distraction . Mobile Phone Model IM-1200 made by SK Telecom Field -Experiment (37) -Survey -Web Diaries -Server Log N/A Usability (content. representation difficulties.Survey . 6.Collecting and analyzing data . which was written in the pocket diary. had only one hand available for use. May 2011 .Trial (mostly). Hands availability.152 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Technology/ system/product Environment Khalifa & Cheng 2002 N/A -Undergrad/ Grad -Students in second (and third) degree N/A N/A N/A -Experiment (202) -Survey -Age: 18-47 Attitude Trial. Issue 3. Observation. representation ) Use contexts (goal & emotion. structure. Intention to adopt .Age: 15-40 Errors Satisfaction Ease of Use Goal (utilitarian/hed onic use). visual & audio distraction.AD  EOU Kim et al.. Culture: Korea Journal of Usability Studies Vol.Experiment (37) . Usability problems occurred most often. Log on to website & rewrite web diary. navigation. 2002 Experts (mobile Internet) Open . and users were alone in a quiet calm environment.Goal  EOU .Use in movement /static + Good Emotion  Satisfaction .Attitude  ITA behavioral control  ITA Kim et al. Use in movement/ static. then navigation issues. Perceived behavioral control.Subjective norms  Intention to adopt (ITA) . Emotion.Lack of appropriate content over internet  Errors . colocation & interaction) Mobile internet used primarily: when feeling joyful.. Exposure to mobile commerce.Web diaries Mobile Internet phone Field: -Noisy and Quiet -Visual cues -Public and alone . 2005 Experts Open . Subjective norms. and structure problems.Device data .HA  EOU . and Communication  Exposure to mobile commerce .UM  EOU . mobile use skills Mobile payment habit used currently.Focus groups: Six groups of MBA master thesis students -Field survey: 948 participants -Random sample -Age: 18=65 Ease of use Generic (efficiency. Compatibility (large applicability. Internet use skills. time) Benefits. experience. Education (elementary). education. Internet skills. 2010 Experts (mobile phone users) Journal of Usability Studies Open Exploratory browsing: Browse photos & videos & select three of them. Ease of use. traditional UI (tag-based structure + multidisplay button interface) and new UI (folder-based hierarchical structure + fixedbutton interface) User generated content m-service Lab Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Survey development: -Interviews: University students & bank employees . bill payments). 2007 N/A N/A – Asked about intention to change payment habits (shifting to electronic and mobile payments) Culture: Finland Kim et al. Payment habit specific (purchase. there were no statistical differences between mean scores for perceived ease of use of the two UIs. Vol. Profession (upper clerical) -> The acceptance of mobile payments -Experiment -Survey Perceived usefulness Perceived ease of use Satisfaction Perceived enjoyment Behavioral intention New UI enhanced exploratory browsing within mobile UGC services in terms of usefulness. satisfaction. and intention to use the system again. Availability of payment transaction information. Profession (entrepreneur) -> The acceptance of electronic invoices However. Trust (security). profession). Ease of use.153 Usability Formation of usability studies User Task/activity Kim et al. Issue 3. Independence of time & space (convenience). Social norm. 33 participants Electronic invoice habit used currently. May 2011 .. 6. Profession (upper clerical). Demo (age. enjoyment.. Technology/ system/product Environment N/A Field Mobile phone: basic. PDA tasktechnology fit. & agents‘ cognitive style didn‘t impact the cognitive fit of using PDA technology for insurance tasks. 2009 N/A Kjeldskov & Graham. 2003 Culture: Taiwan Journal of Usability Studies Vol. 6. EMS. computer selfefficacy  PDA cognitive fit -Age. position experience. and computer experience) -Different individual traits  different cognitive fit in using PDA -PDA: different degrees of support. education. age. computer experience. Impact on task performance.Self reported survey on performing three major types of insurance tasks with one question for each insurance task Random sample of agents from one insurance company in Taiwan N/A Closed PDA. position experience. mobile phones Lab & Field -Experiment -Device data -Survey -Observation (48) studies User Task/activity King & Mbogho. education.154 Usability Formation of usability Technology/ system/product Environment Research methodology* (sample size) PDA -Field -Survey (238) Insurance industry Job: Insurance agents N/A . May 2011 Key usability dimensions/ constructs** Other variables Key findings N/A Cognitive style. Computer selfefficacy. Errors Efficiency Situation (sitting or moving) Seating at a tableErrors Amount of physical activityWorkloa d . Issue 3. Demographic variables (gender. different TTF to different tasks -Gender. Prior to First study. but used system for 1 year -Age: 31 .54 Kleijnen et al. tangibles. reliability.‖ N/A Mobile airtime N/A -Interviews -Survey (Consumers and service providers) Satisfaction (service and service provider) Service quality. cost Vol. serious. and customer loyalty. 7 tasks.. Second test: Nurses had 15 months experience Desktop PC with hardware used at the hospital. more critical and serious problems -Experts experienced sig.155 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Kjeldskov et al. May 2011 . more cosmetic problems. 2007 Experts Culture: India Mobile phone users and service providers from the three cities of Lucknow.. 2010 NovicesExperts Open . frustrated user. responsiveness . Culture: Denmark Job: Female nurses with 2-31 yrs. or critical) -Experience  Effectiveness -Novices experienced sig. mobile phone number (unchanged) Reliable service quality and honesty costs (billing) were the two most important determinants of consumer satisfaction. empathy. -Experience did not  Efficiency (on complex tasks & critical problems with the EPR) -―Time does not heal usability problems. Issue 3. cost. which lead to customer loyalty. not understood by user). quality of service. Think Aloud. Kanpur. subjects attended a course on the IPJ system. Severity rating by evaluator (cosmetic. exp. Novices. Health care information system (EPR system)/ digital video Lab -Experiment (7) -Think Aloud -Observation Effectiveness Efficiency (completion time) Usability problems (prevented task solving. 6. Service brand image. First usability test: Novices. and Agra (northern India) Journal of Usability Studies Environment -Judgmental sampling (70) -Ages: 18-55+ Characteristics of service provider: honesty in billing. usability. Attitude towards citywide location-aware systems.Many (55%) would use similar citywide system & most thought it‘d be fun. utility and accessibility should be accounted together.Used RHUB (system that supported messaging. . . .156 Usability Formation of usability studies User Task/activity KofodPetersen. IM.Differences in the content and usage habits were across channels (mobile phone vs.Users with the most friends used their tags most often. computer).Primary reason for not using such a system was fear of losing privacy. . . May 2011 . 2010 N/A N/A Koivumäki et al. 2006 N/A Culture: Norway Journal of Usability Studies Open . & Krogstie. discussions.Two-thirds used terminals at Samfundet to locate their friends. user profiles and group management) Technology/ system/product Environment FindMyFriends system (developed & installed at the student society building. Gransæth er. Frequency of tag use.Registered users (2769) . Samfundet) allowed users to locate each other in different rooms within the building. Use of internal and external terminals (for logging in). 6. -Field Prototype: RHUB (built in Web.The shift to facilitating group messaging as well as socialization across media engendered specific kinds of use. Norway Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Experiment -Survey N/A Number of friends. Issue 3.Respondents to survey (207) -Experiment -Device data (108) -Interviews (15) -RHUB-delivered quizzes (102) -Informal conversation (4) -Content analysis (500 random messages from RHUB) Vol.To make a system useful. . N/A N/A . . SMS and email) N/A Biennial student festival in the society building in Trondheim. Privacy ..Most respondents used their tags. employees. Problems & benefits. staff. free vs. researchers & consultants Closed-Entered text phases and alphabet multiple times on different programs PDA (PalmOS) Input: Graffiti2 vs. Issue 3. Univ. paid). Maass. Maximal amount of the review's fee Product type  Adoption of product reviews Open Reviewed brochures and asked to make suggestions on preferred features. Desired & unwanted Features.157 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Koltringer & Grechenig . . Graffiti 2 preferred ( > intuitive). Kowatsch. Gender . 2004 N/A Students. May 2011 N/A Product review fee (-) Intention to prefer review enabled stores Intention to use product reviews  Intention to prefer review-enabled stores . Roles of phones. 2009 N/A N/A . Intention to prefer a review-enabled store. Virtual Keyboard Lab -Experiment (12) -Survey -Interview -Device data Error Accuracy Speed and task load (therefore efficiency) N/A Graffiti 2 input is slower and more error prone than with input on Virtual Keyboards. and safety features Kurniawan . . visual aids. 6.Subjects told they own a mobile device capable of identifying products with an RFID-reader & requesting product reviews after touching them with their device N/A N/A In-store shopping scenario on a picture -Survey (scenario based) N/A Intention to use (product reviews in general.Women focus on haptic aid and men on perceptual aid.Characteristics of age friendly phones: memory aids. In Press Culture: Germany N/A >60 year olds Journal of Usability Studies Environment Field 116 participants -Experiment (14) -Delphi Interviews -Focus group -Discussions -Online survey 100 respondents Vol. & Fleisch. N/A Usage patterns.Older people are passive users. haptic aids. 158 Usability Formation of usability Mobile phone Lab -Survey (94) -Focus group -Experiment: Text advance organizer (AO) group (32). Intention to use (INT) TC→MVAL (positive) N/A ―BinScroll‖. AOs‘ utility is highly conditional. User control (UC). Time consciousness (TC). procedural knowledge) Cognitive ability variables (verbal reasoning. Cheng. Risk (RISK). May 2011 Errors UC→MVAL (positive) RISK→MVAL (negative) CE→MVAL (negative) MVAL→INT (positive) RVAL→INT (negative) EVAL→INT (negative) TC x TO→MVAL (positive) RISK x TO→MVAL (positive) CE x TO→MVAL (positive) . & Cheng. Perceived value retail channel (RVAL). Cognitive efforts (CE).Before survey.. a technique to navigate and search for words on mobile devices LanganFox et al. mobile banking and brokerage were. associative memory) N/A The text AO had a facilitative effect for two of the three task performance variables. N/A . 2002 Key usability dimensions/ constructs** Technology/ system/product studies Culture: Netherlands Research methodology* (sample size) Closed . Issue 3. Value mchannel (MVAL). total error) Recall performance (thus. Perceived value electronic channel (EVAL).Search tasks -Random sample Computer Lab -Experiment (24) -Device data Vol. 6. 2007 N/A Journal of Usability Studies Key findings Open Interactions with a network of services using verbal or visual information Task/activity Novices (students/tea chers/enginee rs) Other variables Environment User Lehikoine n& Salminen. declarative knowledge. control group (31) -Observation Task performance (inefficiency. provided a short introduction on what mobile transaction services. Service compatibility (SC). graphic AO group (31). 2006 Novice Lee. proportion correct. memorability. and several examples of the possibilities involved with mobile services N/A -Field (Street) -Survey (375) N/A Time convenience (TC). W61H & W53H Licoppe & Heurtin. Customization. -Tactile key press sensation due to hardware differences between mobile phone models may impact usability.. Lab (for 20 people) -Survey -Interview (20) -Anonymous traffic database (1. but might not assess the viability properly. Design aesthetics Design aesthetics -> PU studies User Task/activity Li & McQueen. Open Information retrieval task Device: Google G1. IT infrastructure. effectiveness) N/A -Do not consider multiple mobile phone models with matched interaction sequences as equivalent to the same model. D (two applications). N905i and N905iμ Lab -Experiment (within group. Use of a mobile (UOM). 2010 N/A Culture: Taiwan Research methodology* (sample size) 200 participants Design aesthetics -> PEOU Design aesthetics -> Customization Design aesthetics -> m-trust PU -> m-trust PEOU -> m-trust Customization -> m-trust Liang et al.Ergonomics  UOM . Medical distributor Li & Yeh. 2008 N/A . 6. two mobile phone models for each group) -Observation Task completion time (thus. Task execution process (thus. Issue 3. Accessibility Joinability.Sociological reasons  UOM Group B‘s model: W61CA. Services: Three virtual mvendors with snapshots (buying digital cameras.Price  UOM . and booking a hotel) Lab -Experiment -Survey Perceived usefulness Perceived ease of use Trust. Sociological reasons . 2007 NovicesExperts Culture: Japan Closed CogTool (user interface evaluation tool) Based on key layout: Group A‘s model for CogTool. 2001 N/A Culture: France Journal of Usability Studies N/A N/A Vol.Four firms: Convenience store. -High fit does not guarantee system success. Technology (fit). Manufacturin g. and Organization (viability) -Organizations aware of tasktech fit importance in choosing m-apps.000) Ease of use. renting a car for travel from a rental car agency. May 2011 . A and B (one application) Performance Task. Insurance.159 Usability Formation of usability Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment N/A Interviewed about their mapps (each firm implemented mobile technology for > than 1 year) N/A N/A -Interview: C (three applications). efficiency). Economic. scheduling lessons. The free-bird-onthe-fly principle. studies User Task/activity Lin et al. no link to usability Journal of Usability Studies Vol. used by 15 students Nokia‘s N95. Must test in the field. used by 2 students.Use of text messages N/A N/A -Surveys (2007 youth) (1001 parents) -Interviews (12) N/A N/A Social research. Ling. 2001 N/A Culture . May 2011 . More satisfaction problems than efficiency and learnability. 6. interaction situations Users use device differently in different situations. using diaries.Adding a person to the address book.160 Usability Formation of usability Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment Closed .Six tasks to find specific information on the web-based Learning Management System Apple‘s iPod Touch Lab -Experiment -Thinking Aloud Method -Interviews 17 participants User experience (attractiveness. and ethnography. Issue 3. Lindroth et al. efficiency.. perspicuity. direct observation.Youth . 2007 NovicesExperts Culture: Austria Research methodology* (sample size) or iPhone.Parents N/A . creating a card PDA Lab -Experiment -Survey -Device data (12) Efficiency Errors Learnability Memorability Satisfaction Weather.. stimulation and novelty) Mobile internet design principles: The fat-manwalking-nonarrow-path principle. and the onehandedbandit-on-therun principle The implementation of intelligent pervasive learning environments demands holistic approaches of thinking. design and testing. 2001 NovicesExperts Closed . dependability. voice activated dialing. Visual tags that encode the URL for the photo album. Hwang. voice activated dialing & mobile internet received higher satisfaction scores. library collections accessed online & online photographic collection. learnability) Cost. Features (camera. Internet browsing. Internet browsing. and wireless connectivity) and rate satisfaction. color screen. Issue 3. Lab (simulated university library) -Experiment -Interview Ease of learning (thus. Terminal phase (TP) DP  high error rate (ER) MRAP  average ER TP  Low ER Learnability  ER Mallat. -More males own phones with cameras. 6.Text Typing PC Concepts KB5640 numeric keypad Lab .571) Satisfaction College Students N/A . 2001 N/A Students Closed .Observation . Mobile phone Mfgr. color screen. Education level The mobile tagging media in educational setting was easy to use and comments from participants showed their high interests about the tagging system.Data collection through computer Learnability Error Rate Discovery phase (DP). Ethnicity. voice activated dialing. May 2011 . wireless connectivity). Vol.Evaluate five design features (camera. age.Experiment (20) . & Salvendy. -Female Asians have a higher preference level on color screen. Motor reflex acquisition phase (MRAP). Academic major.. Culture: Caucasians & Asians in US Culture: Africa Journal of Usability Studies 20 participants (students and non-academic staff from university) High cost of camera phones and lack of local language support were barriers to adoption. Internet browsing. Preference level of features. and wireless connectivity. Gender. Mobile phone experience -Phones with color screens. C. 2007 Novices Closed Capture a 2D visual tag and try to use a visual tag application to navigate visual tag-reading systems (for accessing digital library content). 2006 N/A Users current mobile phone (8 different brands) Lab Survey (2. Availability & Experience  Satisfaction MacKenzie et al.161 Usability Formation of usability Technology/ system/product Key usability dimensions/ constructs** Other variables Key findings Environment Research methodology* (sample size) studies User Task/activity Ling. Nokia 6280 camera phone with readers for the visual tags.. made for the medium. Cognitive load -Obstacle course was not the same as the walking conditions that used a treadmill.Used stylus to tap on targets shown in various locations on the display) under 1 of the 4 mobility conditions PDA. and promotion. PEOU usefulness. ease of use.View websites on two devices & rate sites on content. Issue 3. May 2011 Ease of use Error rate Task completion time Target selection time (thus. Promotion Technology readiness  Importance placed on usability characteristics Mobility condition (seated and walking). HP Jonathan 568 PDA Closed . emotion.162 studies User Task/activity Technology/ system/product Environment Research methodology* (sample size) Mao et al. 2008 N/A Job: Students N/A Journal of Usability Studies Open . efficiency) Technology Readiness. intention to use For USA sample: PU adoption. personal innovativeness PEOU For the Turkish sample: All above and PEOU adoption. even after they reduced their walking speed. Efficacy ease of use. 2005 Massimi & Baecker. personal innovativenessu sefulness. Emotion. personal innovativenesse fficacy Massey et al. treadmill Lab -Experiment (76) -Survey (35) for Web Usability (41) for wireless Web Usability -Lab -Seated -Walking -Obstacle course -Experiment -Observation 64 Students Vol.. Technology readiness moderates relationship site type  site ratings . priceadoption. personal innovativeness . 6.. Content. efficacy. 2005 N/A N/A Mobile phone N/A -Survey (273） Usability Formation of usability Culture: Turkey and USA Key usability dimensions/ constructs** Other variables Key findings Ease of use Accessibility Usefulness Price. Made-for-themedium. -Error rates increased when the participants walked through the obstacle course. Willingness to utilize the madvertisements -The most important factor influencing personalization was the context factor. 2003 Experts Closed Responded to the phone call. Nagata. anticipation Efficiency Nielsen et al. desktop Lab -Experiment -Survey -Interview -Device data (8) -Age (25-54) Efficiency Errors Anticipation & origin (external & internal) Interruption (unexpected external & internal. intercom message. 2004 N/A Closed . content. etc.163 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Merisavo. 2006 N/A Closed Receive SMS mobile ads Prototype that provided personalized advertisements to mobile users N/A -Survey (135) -Experiment and survey (183). followed by user preference and content. 2006 Novice Closed Transmit data. expected external & internal) Sig. difference between the ODA and desktop groups: origin Efficiency. m-ad with personalization) N/A Personalization (user preference. Attitude toward the mobile advertisements .. Issue 3. or IM notification PDA.) Lab -Experiment -Observation (14) -Interview (4) -Survey Efficiency Ease of use Satisfaction Accuracy N/A Use of handheld devices to collect census data Culture: China Journal of Usability Studies Environment Vol. May 2011 . & Raulas.Collect data Handheld devices (PDA. (m-ad without personalization vs. register Mobile phone (Sony Ericsson T68i) Lab & field -Experiment (14) -Survey (14) -Focus group Efficiency Effectiveness Satisfaction Problems observed Comparison of a field-based and a lab-based usability evaluation of a mobile system Olmsted. -Personalized mobile ads were effective & can influence users' consumption behavior. 6. Kajalo. and context). Vesanen. Arponen. PDA. Weather Other Variables  EOU Other Variables  PU Other Variables  Errors Errors  Satisfaction. Building material. i-Pocket PC Field -Interview (56) -Focus groups (24 groups) -Phone surveys (1000) -Age: 21-28 -56 users (28 Italy. availability.Experiment (19) . May 2011 .164 Usability Formation of usability Technology/ system/product Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Pagani. 2002 Novices Open . functions. interest for service. preference. Call traffic. cost. bandwidth.Observation . Geographical terrain. and speed of use PU  Adoption PEOU  Adoption Price  Adoption Speed  Adoption Palen & Salzman. Phone antenna. ranking of service Usefulness most important  adoption. 28 USA) Ease of use Accessibility Mobility. price. followed by ease of use. 6. degree of service innovation. Attitude (underutilization) Culture: Italy and USA Journal of Usability Studies Environment Vol. Issue 3. privacy Motivation.Age: 16-75 Errors (software/ hardware) Ease of use Attitude Satisfaction Usefulness Network.Interviews .Phone calls .Explore the functionalities of the phone Wireless Telephone (N/A) Field Everyday life . hardware and software functionality. 2004 N/A N/A Mobile phone. Interior/ exterior call. accessibility. Jobrelated reason.Business reason  Adoption . Social.. presentation. Salzman. Special event. Freedom . conversion..Special event  Adoption .Device  Increased mobility.Web tasks PDA Lab -Experiment -Observation -Interviews (27) -Graduate students Ease of use Zooming.165 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment N/A . Issue 3. 2001 Novices Poupyrev et al. optimization Zooming  Ease of Use studies User Task/activity Palen. which personnel is most likely to benefit from mobile tool support. and what improvements on business metrics are to be expected Qiu et al.Security  Adoption . & Youngs.DeviceShare resource. Safety. semantic. 2004 NovicesExperts Open . safety/proximity . pocket PC Lab & field -Experiment (30) -Log files -Video capturing -Survey -Interviews Effectiveness Ease of use Time for solving a task (therefore efficiency). Security. 2006 Novices Closed Selecting info about customers previous problems -Search a location -Search a problem solution suggestion -Reading docs PDA. Net of safety /proximity.Social  Adoption .Job-related reason  Adoption . May 2011 Semantic Conversion  Ease of Use .Talk about their experience Mobile phones Lab -Interview Voice mail diaries -Calling behavior data (19) -Ages: 16-75 Accessibility Right price. Usefulness Use of context mobility Which tasks are suitable for mobile application support. Freedom N/A All male Closed . 6.Right price  Adoption . Business reason. 2002 Job: IT service technicians Journal of Usability Studies Vol.Safety  Adoption . Mobility.Scroll a text list Palm (PDA) Lab -Experiment -Survey Performance (therefore efficiency) N/A Tactile feedbackefficien cy Pousttchi & Thurnher. Learnability Efficiency (and time) Ease of use Context awareness Client-side processing & location context  Mobile devicebased application = PC-based objective performance and subjective usability measures studies User Task/activity Rodden et al. 2005 Research methodology* (sample size) Novices Culture: Australia Mobile web-based application  lowest quantitative performance Journal of Usability Studies Vol. 2003 Experts (computer background) Ross & Blasch. May 2011 . need to improve interfaces to increase performance (time/less veer) NovicesExperts Open .Cross three intersections Wearable device Field -Experiment -Interview -Device data (15) -Age: 62-80 Error Efficiency Hesitations.tasks on list on all four applications Mobile phone (smart phone) PC web based(HTML).166 Usability Formation of usability Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment Open & Closed . PC device based (AT). Nokia 6600 phone Lab & field -Lab study (8) -Field study (20) Longitudinal Usability Study (20) -Diary -Group discussion -Survey Ease of use Learnability Preference -Most preferred Minimap: easier to use. mobile device based (AWT) Lab -Survey (12) -Experiment (12) Errors Satisfaction. 6. background and cell phone internet use Ryan & Gonsalves .. mobile web based (XHTML).12 tasks (web browsing & search) Pocket PC Lab -Experiment -Survey (24) -Age: 20-42 Efficiency N/A Performance between tasks and interaction between browser and task N/A Severe visual impairment Closed . 2006 Divided into two groups of similar age. Confusion episodes. less hesitation. Preference interface veering performance.. pages looked more familiar -Minimap better on pages with big data tables or simple layouts -Neither browser suitable in a hurry -Familiar pages easier to browse then unfamiliar ones Closed . Issue 3.Used Minimap browser first and then switched to narrow layout browser after 8 days to complete given tasks. 2002 Roto et al. 2005 N/A Job: Medicaldental students. PUI based (ergonomic consideration. display style). and task-based (see left) -Framework for evaluating the usability of mobile phones to support task-based and interface-based usability evaluation. four sets of checklists.. Ergonomic support Critical level: LUI based (information architecture. 6. GUI based (icon. and an evaluation process PDA N/A -Experiment -Device data -Survey (43) Effectiveness Efficiency Satisfaction Form of assessing checklists (paper. echecklists) Research methodology* (sample size) Vol. -Hierarchical model of usability factors. a quantification method. cognitive burden of execution) Indicator level: Visual support of task goals. Support of cognitive interaction. font. PDA checklisteffectiv eness. stability of use. support of operation sequence. wording. function options).167 Usability Formation of usability studies User Task/activity Seth et al.Taking a picture and related operations Culture: Korean Job: Usability practitioners working for a mobile phone company in Korea Shami et al. contextual consideration). Issue 3. assessors Journal of Usability Studies Technology/ system/product Environment Key usability dimensions/ constructs** Other variables Key findings Two mobile phones (similar functionality) N/A -Experiment -Case study with 8 Practitioners -Card Sorting -Task Analysis -Meta-Review Task based (efficiency of procedure. 2008 Experts Closed .. PDA checklistsatisfac tion Evaluate the mobile phone using the usability tests Closed . Functional support of user needs. Support of efficient interaction.Clinical exam (paper vs. May 2011 . electronic) PDA checklistefficien cy. -Using auditory interfaces increase driving performance and perceived workload. camera Field -Interviews -Observation (7) Attitude Use. N/A N/A Mobile phone. 2009 Culture: Taiwan Journal of Usability Studies 207 respondents Vol. & voluntariness: varied effects for different categories of adopters. visibility. and users studies User Task/activity Sodnik et al. triability. Issue 3. walk/disc man. Workload -Auditory interfaces were effective to use in a mobile environment. May 2011 . 6. potential adopters. make call.168 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings Technology/ system/product Environment Closed . delete image. 2008 N/A Strom. change profile picture..Five tasks on a mobile phone in a car simulator (write text message. 2001 Suzuki et al. play song) Nokia 60 Series Lab -Experiment (18) -Survey -Interviews -Device data Effectiveness Efficiency (and task completion time) Satisfaction Driving performance. but weren‘t faster than visual interface. result demonstrability. Social attractiveness Use  Less social attitude NovicesExperts N/A MMS Field -Survey Ease of use Compatibility Triability Image Result demonstrabilit y Voluntariness Visibility Relative advantage -(Except for laggards) Relative advantage  MMS adoption -Compatibility also key in motivating the adopters and potential adopters -Ease of use. PDA. image.. 169 Usability Formation of usability Technology/ system/product studies User Task/activity Svanæsa. PDA. select and drag X-ray image to a terminal icon on the PDA vs. Digital Noldus video-recording solution with roofmounted/remote control/stationary/ wireless ―spy‖ cameras. PDA as a remote control to navigate menu on the bedside terminal). 2009 N/A Open . Social aspects of usability (private vs. Updating information timely. Rich referential content Vol. Curriculums‖ (from parts of the lessons from College English Intensive Reading of Shanghai Foreign Language Education Press) N/A Culture: China Environment Usability lab for m-ICT in medical setting Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Experiment -Usability testing Experiment 1: Combining handheld devices & patient terminals Experiment 2: Automatic identification of patients at point of care -Interview N/A Graphical user interface (GUI) usability.. Media types. Curriculum content.. Function preferences. which has a T-flash Card of 2G capacity (mobile device) Journal of Usability Studies Interview: Individual requirement. Learning fragment duration. 6. public. Issue 3. face to face dialogue). audio mixer. wireless mics. & implement mode System design of a college English m-learning system 20 freshmen and 20 sophomores in Software Engineering Device: PPC of Dopod CHT9000.g.Used every function of the system and record any confusion & problems encountered Software: ―Mobile Learning Center. Physical and bodily aspects of usability (screen size. PC Patient terminal Lab Wang et al. Contextual nature of usability Ergonomic aspects: social aspects & factors related to how well the system integrates with existing work practice  usability of m-ERP -Experiment -Survey -Interview N/A Survey: English learning tools. 2010 N/A Closed . & Dahla.Eight designs tested by a physician using a bedside terminal to show X-ray images to a patient (e. software for remote ―mirroring‖ of content on mobile devices. body movement and the use of hands). May 2011 . Alsosa. May 2011 . Reliability B.I. Actual use. to use MC Actual use usefulness & risk  BI to use Cost (-)  BI to use MC 26.8% were familiar with MC Compatibility.Engaged in online transactions via B2C Mobile Commerce (MC) for personal use N/A Lab -Experiment (310) -Survey -Interview Accuracy Ease of use Usefulness Perceived risk. error rates higher for TileText than for MultiTap Wu & Wang. but device-related issues are more difficult to capture. effects for the technique Different efficiency increase for different users.170 Usability Formation of usability Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings studies User Task/activity Technology/ system/product Environment Waterson et al. 6. Behavioral intention to use. most important effect on BI & 2nd most important effect on actual use Culture: Taiwan Journal of Usability Studies Vol. 2002 N/A Closed PDA Lab & Field -Experiment Lab (5) Field (5) -Observation -Device data -Survey (10) Errors N/A This testing technique can more easily gather many content related issues. 2005 N/A Open . Issue 3. 2003 N/A Closed Entered short phases of text Mobile phones Lab -Experiment -Device data (10) Efficiency Errors Text entry interface Sig. Compatibility. Cost. Wigdor & Balakrishn an.. Survey. and Nokia Maps application. usefulness. operability. 2008 N/A Open . usefulness) Number of times used. errors. applications used Olympics guide. Field study) ** Key usability dimensions: Effectiveness.171 Usability Formation of usability studies User Task/activity Xu et al. 158 participants Note: * Research methodology: How (Observation. Interview. English-ChineseEnglish phrasebook. Continuance intention Study suggested how usage patterns can be used to determine when to use the applications and how user activity and environment can be used to improve the applications as well as to develop personalized mobile Web applications. acceptability Journal of Usability Studies Vol. satisfaction. learnability. Environment Field Research methodology* (sample size) Key usability dimensions/ constructs** Other variables Key findings -Experiment -Device data -Survey Ease of use Helpful (thus. attitude. Sports Tracker. accuracy. flexibility. Photo sharing on Ovi.. memorability. Issue 3. Menu Reader. 6. Device data) and Where (Lab study. ease of use. Focus group. May 2011 . accessibility. efficiency.Used suite of mobile Web services and applications at Beijing Olympics Job: Guests of the Beijing Olympics Technology/ system/product Nokia N82 mobile phone.