Addressing Usability Problems

After completing the playtesting in the actual museum with real visitors (kids), we had to redesign the system to address the problems found. In this report I include a list of the most serious flaws identified and the way they have been tackled. I also include some design decisions that had to be taken based on feedback from instructors and peers. For a detailed list of these usability problems please see the previous post.

#1 How do I scan?

Neither the camera not the actual way of scanning was obvious to none of the participants in the playtesting. To address this issue we are using two ways: on the hardware level and on the software level. For the hardware the intention is to place the camera in an obvious place with a clear label "scan here", but also use some glass in a distance where children can place the tagged label on. In software level we have included an image depicting clearly the way to hold the code in front of the camera.

#2 When do I scan?

This was another frequently-occurring problem where children needed some prompt to start scanning. Again, a virtual helper will be available in the system that will prompt users whenever there is a need to scan a label. The mockup at the moment includes a mouse holding a label which says "scan now" (what more obvious!) Similar prompts appear throughout the game as a means to make always clear what is the current state of the system and provide meaningful feedback on the user's actions. The messages appear at the right of the screen and are not always visible so as not to distract the player's attentions while interacting or reading other information.

#3 Icons... which icons?

An important observation during playtesting which was also pointed out during the pin up sessions what that visual hints were not obvious. To remedy that problem we have made those small images animated, starting exactly at the point of the textual hint and traveling to their final destination (at the top of the image) while passing by the notes counter and increasing it by one. This ensures three things: a) that the meaning of each icon is clear, b) that the icon is visible at ll times and is not lost in the image, and c) that there is a clear connection between hints and notes taken. The drawback is that the icons do not appear in the context of the environment, but we consider this to be a trivial issue after reviewing the nature of the hints (few of them can actually make sense in the environment without the animal).

#4 Where are the labels?

Many kids could not find the tags placed near animal exhibits. This will be easily addressed when designing the labels so as they are big enough and standing at an obvious point. Also, there could be an indication of a code-enabled label in the front side, which will make those labels distinguishable from other labels that might already reside in the museum. A potential design could be like the one depicted here, where the label is standing on a properly cut cork with the animal image and name in front and a thumbnail of the QR code above the picture. This will make the labels recognizable from a distance with minimal effort. However, a full-scale code will exist on the back of the label to make to extremely easy to scan without demanding too much accuracy (the larger the code the easier for the software to recognize the encoded message).

Challenges

However, implementing these features is not without cost. Having a dedicated space for instructions consumes valuable space, but we think that this pays off by making the intentions of the system clear, which is fundamental for the user experience. The labels are fairly easy to be designed as an obvious part of the game, but then the question is if we are leading the player in choosing animals, instead of actually thinking of the animal to look for! Actually this was observed during the playtesting, when children were asking us where are the labels hidden. However, speaking with the client I was told that making the animals that are part of the game obvious is a requirement, despite my reluctance due to making the game fairly easy and with less educational value for the reason explained above.

The biggest challenge was the animated icons and how they would allow future expansion of the system by the museum. A special technique had to be devised in the software for animating those elements while also keeping them accessible on the hard disk. This allows someone to edit them or add new ones using, however, a template with dimension characteristics. There is no way for the system to reject of accept images based on their specifications, so extra care should be taken when updating the content (similarly to doing so with textual information).

Playtesting #1

In order to test the effectiveness of the design as an instructional tool but also its assumed engaging character I run a playtesting session in the museum.  The session took place on a Saturday morning (3/30 from 11:00 to 12:30am) when many kids are visiting the museum with their parents. This was a very good opportunity to test the system with real users in a fashion similar to the intended one. An initial prototype of the game was used which contained two habitats with one animal for each one of them (the bear for the mountains and the turtle for the lake). A couple of more QR coded tags have been placed around the space to test if players actually recognized the described animals or just used the first tag that came to their attention. There was no level or player avatar selection at the beginning.

Setup

The prototype was  set up to run on a laptop computer with an embedded camera which was also connected to a 32'' LCD display mounted on the wall. The laptop was placed on a small kid's table that was underneath and a few feet further from the large monitor. The prototype was developed in a 4:3 ratio (1024x768 pixels) and not the intended panoramic resolution of 1366x768 pixels; consequently it was displayed in a window in both widescreen displays. The system included some initial habitat information from wikipedia, a few clipart illustrations for the hints (but not all of them), and also voice-over only for the informative pop-up messages. A picture of the setup can be seen in the following figure.

The setup of the playtesting.

Procedure

 Children who approached the setup were asked if they were willing to play a game which was under development, having the chance to be part of its evolution. If parents were nearby their permission was requested as well. As soon as kids were seating on the table they were introduced to the game by a screen describing the game's scope. For younger children the explanation was provided verbally, since the narration was missing from the prototype. They then proceeded to the first habitat (by clicking a button) and then to the first hint (again, with a button press). After that they were prompted to find the item described by the game by looking around in the museum. Some encouragement was provided to children who were reluctant to do so.

After finding the QR coded label next to the item (no item was directly QR-coded in our test) they had to scan it and check the system's response. Additional scans were encouraged after an item has been correctly identified. After finishing with both habitats/animals children were asked if they liked the game and would like to see it installed in museum permanently. Also they were asked if they would be willing to fill in some of the information they have came across in the game, on a piece of paper with the animals pictures already printed on. Finally, they were thanked and asked for their name in case they wanted to be part of the game's credits for helping in its development.

Results - Observations

The prototype of Nature Explorer was used by 6 children in total (4 girls and one boy) during one and a half hour. The youngest one was 7 and the oldest 10 with a mean of 9 years. They spend in average around 10 mins playing the game, including the time for filling in the information. All of them wrote something on the paper that they recalled from playing the game; one girl decided to take the "report" with her while the rest wrote their name and left it behind. One 10-year-old boy wrote a comment on the back of the paper (can be seen below).

Evaluation was made in two ways: by asking them what they were doing or looking for during the game, and mainly through observation during their (inter)actions. The main comments from the playtesting are presented below, including suggestions for remedy whenever a problem is presented:

• The information introducing each habitat was probably too dense, which was expected since it was not appropriate for their age
• Most kids had problem identifying where the camera was; this should be made explicit by a physical pointer at the camera in the final physical exhibit
• Most of them held the tag very close to the camera; a sample image of how to hold the item should be provided during scanning interaction
• The tags were not easily identified in the museum, which is mainly due to the fact that they had no clue what to look for; an explanation image with sample QR tag should be presented at the beginning of the game; also labels should be larger
• Some kids did not notice the hints appearing in visual format on the reference image; image hints should be made more conspicuous either by flashing or animating
• The younger girl needed her sister's help to read the information; voice over for all text messages will resolve this
• Some kids tried to type the response; this was due to the text box being visible which is going to be eliminated in the final version
• Kids did not seem enthusiastic about playing the game and looked around reluctantly; I believe this is attributed to the fact that they were being observed by many people (i.e., parents, me, museum director) which made them feel like being examined
• Siblings playing together were observed to help each other, especially the older helping the younger by repeating the clue and prompting her to look around
• Most players were using the laptop's monitor positioned in front of them, while parents-observers were watching the large monitor encouraging their children to follow the prompts (an exception can be seen in the following image); in the final exhibit it would be preferable to have the camera positioned in front of the large display so that children can interact by scanning without blocking the view to other players/participants
• All kids recalled some type of the information received and wrote that on the report

A boy scanning a tag while standing in front of the laptop with his sister.

It seems that most kids enjoyed the experience and would like to see and play the game more often in the museum. One 9-year-old boy specifically, wrote the following comment on his report and also told the museum director that this was a cool game!

A 9-year-old boy's comment about the Nature Explorer prototype.

Overall, I believe the playtesting revealed some of the system's flaws that need to be addressed in the next design iteration. Besides the expected deficiencies due to the unfinished nature of the prototype other findings will be addressed with guiding tips throughout the interaction and context-sensitive help. Instructional and educational effectiveness was not assessed rigorously both due to the type of space and visit (some parents were eager to leave as soon as the game finished) but also due to the informal type of assessment (not IRB protocol was acquired). However, all kids were willing to note down some of the things they learned during the experience, like the girl in the following photo.

A girl writing some of the information she learned, overlooked by her brother.

Instructional Goals and Design

The main instructional objective of Nature Explorer (NE) is to provide some type of scaffolding for the young visitors of the Price House Nature Center to facilitate their learning in this informal education setting. This has been accommodated with the design of a scavenger hunt game incorporating an open-ended learning environment which leverages the physical space and the knowledge already existing in that space. The learners become explorers who have to seek the space of the museum and discover the items described by the clues that the game is giving them. Nature Explorer is designed in such a way to provide the information in the form of a narrative through a role playing game. The engagement evoked by the game and the playfulness of the content, are the main factors that justify the chosen design for the instructional objectives on this informal, free-choice learning setting.

Instructional Approach

As was mentioned in a previous post, Nature Explorer is trying to address all four phases of instructional design in the following ways:

• Present information: information is presented in three modalities (i.e., as images, text and audio) to accommodate for the diverse needs of the audience (children between 4 to 10 years old), either through hints or extra info (trivia) about the item/animal being requested by the system
• Guide the learner: different levels of hints are provided to guide the visitor/player as of which artifact in the museum the assignment refers to; habitat information is presented to help the player set his mind on a specific environment (the museum is divided in similar natural settings)
• Practice: speed is not desired for this relaxed type of informal learning; two different levels (intentionally not connected with age) and ability to play again with a random set of items might elicit retention and fluency, although this is not the focus of the system
• Assessment: although assessment cannot be ensured in such an informal environment, NE will provide a printout after the quest prompting visitors to fill in the missing information related to their accomplishments (things they have already learned)

Motivation

Since Nature Explorer is addressed to young children it is imperative that it is compelling to them. In order to cater for this need various techniques have been used to increase motivation and engagement. The following are the most important design decisions that were taken towards this direction:

• Provide a meaningful goal through story telling and role playing (fantasy)
• Assign the role of explorer to provide a context and identity
• Exploit physical space through assignments that demand exploration and discovery (curiosity)
• Enable collaborative game play through group avatar choice to encourage social play
• Provide performance feedback and score keeping (experience points)
• Vary the difficulty level to allow advancing through experience (challenge)
• Get report and make it public to increase sense of accomplishment
• Get a summary of your achievements (accomplishment)
• Involving the child emotionally in the story (achieved through the narrative)
• Include interesting audio and visual effects to enhance sensation

Overall, NE tries to harness all three constituents of the motivational theory according to Malone (1981) which appear with red italics in the list above: curiosity, challenge, and fantasy. Additionally, it is a major design consideration to make Nature Explorer a "cool" experience for the museum visitors; hence, I have tried to accommodate the four factors of the Wheel of Joy according to Holtzblatt (2011): identity, accomplishment, sociality, and sensation, noted with dark blue color in the list. The design elements used to satisfy both these approached will hopefully increase the motivational impact.

Additionally, the layout of the interface has followed some design guidelines to ensure that it is closer to children’s culture, and communicates information more effectively for these age group (4 to 10 years old). Some of the design decisions that support this claim by evoking emotions and increasing engagement are the following:

• Use of colorful cartoon imagery with matching hand-drawn interface elements (no straight, strict lines)
• Use of the golden proportion rule to divide the screen asymmetrically but meaningfully between the main content (text and images) and the map
• Use of a warm color palette as a means to both evoke emotions but also as a good match for a system related to nature (earth color tones)
• Use of a serif, decorative font with a playful character in a big enough size to be easily readable from a distance of 3-4 feet
• Use of animations and cross-fade transitions in a sequence from left to right and top to bottom to allow seamless processing of new information
• Use of narration for all textual content to accommodate for younger ages but also for groups where not everyone has equal access to the screen
• Use of motion tween and transition sound effects to provide aural feedback of visual changes
• Use of other sound effects and sound tracks in specific points to increase engagement and draw attention to things happening on screen (e.g., item appearing on the image)

These guidelines for effective information design were drawn by the book “White space is not your enemy” (Golombiski and Hagen, 2012) and my own multimedia developing experience.

Storyboarding

During storyboarding I have used some techniques of cinematography to communicate the scope of the system more effectively. This was quite appropriate in my design since it involves physical motion of the user and not just manipulation of a computer system. Thus, long shots have been used to represent the big picture of the system in the museum hall and the actions of the user in space. Close shots were mainly used to reveal actions of the system and interaction of the user with the interface. Close-ups were used to display details of the system, like the QR codes attached on museum artifacts. Finally, different perspectives were used to reveal a cinematographic approach of the user interacting with other people in space (e.g., last scene). For details see the full storyboard of the system.

Golombisky, K., & Hagen, R. (2010). White Space is Not Your Enemy: A Beginner's Guide to Communicating Visually through Graphic, Web and Multimedia Design. Focal Press.

Holtzblatt, K. (2011). What makes things cool?: intentional design for innovation. interactions, 18(6), 40-47.
Malone, Thomas W. (1981). Toward a Theory of Intrinsically Motivating Instruction. Cognitive Science: A Multidisciplinary Journal, 5(4), 333-369.
Greenberg, S., Carpendale, S., Marquardt, N., & Buxton, B. (2012). Sketching User Experiences: The Workbook: The Workbook. Morgan Kaufmann.