Playtesting #1

In order to test the effectiveness of the design as an instructional tool but also its assumed engaging character I run a playtesting session in the museum.  The session took place on a Saturday morning (3/30 from 11:00 to 12:30am) when many kids are visiting the museum with their parents. This was a very good opportunity to test the system with real users in a fashion similar to the intended one. An initial prototype of the game was used which contained two habitats with one animal for each one of them (the bear for the mountains and the turtle for the lake). A couple of more QR coded tags have been placed around the space to test if players actually recognized the described animals or just used the first tag that came to their attention. There was no level or player avatar selection at the beginning.

Setup

The prototype was  set up to run on a laptop computer with an embedded camera which was also connected to a 32'' LCD display mounted on the wall. The laptop was placed on a small kid's table that was underneath and a few feet further from the large monitor. The prototype was developed in a 4:3 ratio (1024x768 pixels) and not the intended panoramic resolution of 1366x768 pixels; consequently it was displayed in a window in both widescreen displays. The system included some initial habitat information from wikipedia, a few clipart illustrations for the hints (but not all of them), and also voice-over only for the informative pop-up messages. A picture of the setup can be seen in the following figure.

The setup of the playtesting.

Procedure

 Children who approached the setup were asked if they were willing to play a game which was under development, having the chance to be part of its evolution. If parents were nearby their permission was requested as well. As soon as kids were seating on the table they were introduced to the game by a screen describing the game's scope. For younger children the explanation was provided verbally, since the narration was missing from the prototype. They then proceeded to the first habitat (by clicking a button) and then to the first hint (again, with a button press). After that they were prompted to find the item described by the game by looking around in the museum. Some encouragement was provided to children who were reluctant to do so.

After finding the QR coded label next to the item (no item was directly QR-coded in our test) they had to scan it and check the system's response. Additional scans were encouraged after an item has been correctly identified. After finishing with both habitats/animals children were asked if they liked the game and would like to see it installed in museum permanently. Also they were asked if they would be willing to fill in some of the information they have came across in the game, on a piece of paper with the animals pictures already printed on. Finally, they were thanked and asked for their name in case they wanted to be part of the game's credits for helping in its development.

Results - Observations

The prototype of Nature Explorer was used by 6 children in total (4 girls and one boy) during one and a half hour. The youngest one was 7 and the oldest 10 with a mean of 9 years. They spend in average around 10 mins playing the game, including the time for filling in the information. All of them wrote something on the paper that they recalled from playing the game; one girl decided to take the "report" with her while the rest wrote their name and left it behind. One 10-year-old boy wrote a comment on the back of the paper (can be seen below).

Evaluation was made in two ways: by asking them what they were doing or looking for during the game, and mainly through observation during their (inter)actions. The main comments from the playtesting are presented below, including suggestions for remedy whenever a problem is presented:

• The information introducing each habitat was probably too dense, which was expected since it was not appropriate for their age
• Most kids had problem identifying where the camera was; this should be made explicit by a physical pointer at the camera in the final physical exhibit
• Most of them held the tag very close to the camera; a sample image of how to hold the item should be provided during scanning interaction
• The tags were not easily identified in the museum, which is mainly due to the fact that they had no clue what to look for; an explanation image with sample QR tag should be presented at the beginning of the game; also labels should be larger
• Some kids did not notice the hints appearing in visual format on the reference image; image hints should be made more conspicuous either by flashing or animating
• The younger girl needed her sister's help to read the information; voice over for all text messages will resolve this
• Some kids tried to type the response; this was due to the text box being visible which is going to be eliminated in the final version
• Kids did not seem enthusiastic about playing the game and looked around reluctantly; I believe this is attributed to the fact that they were being observed by many people (i.e., parents, me, museum director) which made them feel like being examined
• Siblings playing together were observed to help each other, especially the older helping the younger by repeating the clue and prompting her to look around
• Most players were using the laptop's monitor positioned in front of them, while parents-observers were watching the large monitor encouraging their children to follow the prompts (an exception can be seen in the following image); in the final exhibit it would be preferable to have the camera positioned in front of the large display so that children can interact by scanning without blocking the view to other players/participants
• All kids recalled some type of the information received and wrote that on the report

A boy scanning a tag while standing in front of the laptop with his sister.

It seems that most kids enjoyed the experience and would like to see and play the game more often in the museum. One 9-year-old boy specifically, wrote the following comment on his report and also told the museum director that this was a cool game!

A 9-year-old boy's comment about the Nature Explorer prototype.

Overall, I believe the playtesting revealed some of the system's flaws that need to be addressed in the next design iteration. Besides the expected deficiencies due to the unfinished nature of the prototype other findings will be addressed with guiding tips throughout the interaction and context-sensitive help. Instructional and educational effectiveness was not assessed rigorously both due to the type of space and visit (some parents were eager to leave as soon as the game finished) but also due to the informal type of assessment (not IRB protocol was acquired). However, all kids were willing to note down some of the things they learned during the experience, like the girl in the following photo.

A girl writing some of the information she learned, overlooked by her brother.