Lessons Learned

While the development of the Nature Explorer is reaching an end and I am getting ready for the final project presentation, it would be beneficial to look back and identify what this experience has taught me. Starting from the very first post at the beginning of February about ILMD definitions to the very last one concerning the collaborative effort of designing for educational games, I will try to identify the learning experiences that were gained and their applicability in other design situations.

ILMD understanding

My idea of interactive multimedia for learning has not changed much since the beginning of the project. Maybe to my extensive experience developing educational software has helped me have a pretty good idea about interactivity, multimedia, and learning and their interplay. Also, theories about interaction design from the human-computer interaction side have helped me a lot to comprehend the design and development process, especially the importance of iteration and rapid prototyping. Nature Explorer was just another case where these tools allowed me to be productive right away and get feedback from peers and testing at the early stages of design, informing my redesigned prototypes.

Problem or technology first?

In this next phase I came to realize the importance of problem identification as a priority in instructional technology. Coming from a technology background I was inclined to try and find useful applications from existing and/or novel technologies. Thus, it came natural to me to think in terms of cool technologies that could be exploited for learning purposes; the Kinect was one of the first things that came to mind. Although such technologies play an important motivational role in instructional design, they could very easily guide someone astray during the important problem identification phase. Through the followed process I learned to focus on identifying actual instructional challenges and opportunities for applying computer-assisted instruction. I do recognize though, that depending on the domain you are working for you may need to wear different hats (MAE's favorite expression) to address different problems.

We have a problem, and now what...?

My interest about informal learning led me, one way or another, to the  Price House Nature Center, a volunteer-run museum which was "waiting" for my assistance. The unique needs of the space imposed some design guidelines that came right to mind after my first meeting with the Associate Director; the system should be extremely easy to use, require no instructions, and should be  durable to withstand kids' abuse! Normally I would have started right away to test my ideas (which I almost did) first on paper and then with actual prototypes. The requirement to do some precedent study and actually try to align the client's needs with some solid learning objectives were extremely helpful in providing a foundation on which I could build my instructional approach. Overall, the application evolved to be an instructional tool of informal learning instead of a fun scavenger hunt game with some potential flares of learning!

Are all the phases relevant?

The fours phases of instruction, although implicitly integrated in most of my previous educational software endeavors, were useful in forcing me think in terms of specific learning goals to be met. Despite the fact that not all of them seemed pertinent for the informal type of learning of the Nature Explorer (it was hard to see the role of practice and assessment at the beginning), they started to fall in place as soon as I started working in depth on the desired learning outcomes. Those four stages provide a framework when thinking about the objectives of any type of educational software. Similarly, the role of motivation was equally important in leveraging all these design elements that are considered engaging by children (e.g., role play, physical motion, social play, free exploration, etc.) and producing a fun learning experience. This was imperative even more in the context of this space where curiosity, challenge, and fantasy can be fostered in a way to promote discovery learning... and this is what NE is trying to achieve.

(Play)testing makes you better!

The best way to get valuable feedback in interaction design is to test with actual users, and that what I did in the next stage of the project. This is even more the case when developing learning software, where the role of testing is double: identify usability problems but also the learning effectiveness of the tool under development. Since these two aspects are intertwined it is hard to identify the latter disconnected from the former; i.e., problems in using the software prevent users from using it effectively and thus achieving the desired learning outcomes. Playtesting allowed me to identify some serious design flaws (e.g., how to scan a code, or how to identify one in the museum space) that I could not have predicted and would have, most probably, hindered the learning process. If the time allowed I would have preferred to play test a second, more polished prototype and give emphasis to learning assessment this time.

Usability remedy

The results of the playtesting have eventually paid off by revealing the weaknesses of the system mainly in supporting children in their learning process. As a designer enacting multiple roles you are doomed to make assumptions about how the tool is going to be used; this is even more the case if you don't have the necessary support from other experts (for more details see my previous post). Usability problems that were identified helped me understand the points where users needed elaborate explanations and/or some scaffolding in order to build a mental model of the system, in order to be able to follow its progress smoothly. Additionally, feedback from the instructors and peers was another source of potential issues that I decided to address to prevent future interaction glitches.

Transferring knowledge

One of the most important aspects of every learning experience is how to transfer the acquired knowledge to other domains and experiences. Firstly, the most significant gain was the unique experience of working for such an informal space where there are seemingly not enough guidelines on what needs to be achieved. Learning goals were quite ill-defined and I had to create the instructional objectives based on my understanding on the space's needs.
This was an interesting design challenge which can be widely applied in other domains where clients think they know what they want, but you still have to elicit their actual needs. 

Secondly, due to the space's irregular visit schedule and the sensitivity of the audience's age it was not feasible to observe and discuss about "users' needs" in a meaningful way. Although normally this is the case in user study techniques like contextual inquiry where you have to be temporarily integrated in the environment in order to understand the current work practice, there are instances where this is not feasible (e.g., working with private and secure data) and you have to find workarounds. Working for such a young target group gave me the opportunity to devise alternative methods to understand their needs (e.g., the playtesting session). 

Finally, the lack of available (domain and content) experts or their inability to respond promptly to my requests forced me to take over some roles in order to facilitate development. This is often the case when designing in the real world, where clients might not always appreciate the importance of such experts and the designer has to bear some of this burden.

Overall, working for such an open-ended project, with loosely-defined needs, "inarticulate" users, and non-existent instructional objectives was a great opportunity for me to test my -instructional and interactive- design abilities. I had to impose some design guidelines and constraints based on my understanding of the problem and test my assumptions during an iterative, rapid prototyping process. The challenge might have been unique in this situation, but the resulting process was an invaluable tool for future design experiences.

Communicating Educational Game Design

Coordinating Tasks

Building educational games involves the collaboration of an interdisciplinary team consisting of the game designer(s), the graphic designer(s), the domain expert(s), and the pedagogy expert(s). For the purposes of the Nature Explorer I had to undertake some roles while also communicating design decisions with other members of the team to develop a final product that would be appropriate for the client but also effective for its learning objectives. This was no trivial task since different stakeholders have different priorities in designing an educational game; my main concern was to have a technologically perfect product that can withstand children use but also be entertaining to keep their interest, while other parties were more concerned of the content and its appropriateness for the target audience.

Although I did try to accommodate what I believed to be of interest to 10-year old kids, there was no way I could have done this as effectively as an educator. Also, finding the appropriate informative hints that would make sense to the knowledge level of potential young museum visitors was another task that I wanted to offload to someone considered a content expert. Unfortunately, the volunteered-based nature of the museum, acting as the client, had no resources at hand to provide this information promptly. Instead, they have assigned someone as a contact person to coordinate compiling the material (basically the text), also assuming it would be in the right tone and style. As far as I know, the "content experts" in this case were parents who were visiting the museum and often volunteered to help with its activities. This fact made it impossible to either have a timely response by everyone but also to get a consistent style in the content according to my guidelines.

As a consequence the client had to provide most of the content after realizing that the delegates were not as efficient as expected! Even in this case the content did not conform to the constraints and principles imposed by the game design. More specifically, some hints were too generic to provide any meaningful help to the players (e.g., I am a reptile). Also, the hints had to be of specific nature so that they could be easily sketched in a way that can communicate the illustrated message meaningfully (e.g., My young are called "kits"). Even more, a necessary constraint imposed by the required expandability of the system (i.e., the museum should be able to add more animals easily) the hints had to be the same between the two levels, which was not the case with most of the provided content. These conflicts necessitated my intervention in content-writing and a new cycle of edits and reviews by the museum person acting as domain expert.

At the same time this person was acting as the educator who had the capacity to format the content appropriately. She has also been giving some useful advice on the level of knowledge that kids are supposed to have at this age, which has also been affecting graphic design representation. As an example, I was instructed that the mountains should be more rounded instead of being pointy with snowed peaks, because this is how the Appalachian mountains are. Some other advice included the use of specific wording that was supposed to be keen to children; however, after further review and use of such wording in the prototype it was revealed that this was not such a common word in children's dictionary (e.g., 'tree cookie' to reveal a slice of tree trunk). Such incidents revealed the need to have a dedicated pedagogy expert who knows exactly the culture and cognitive abilities of the 4-10 years old children in the area.

Intercontinental Collaboration

During the decision and content negotiation process I had to work with the animator in order to produce some initial sketches for the prototype. Since he was located in Athens, Greece it was quite hard to coordinate our time schedule and work at the same time. Consequently, we had to arrange some online meetings where things were explained and clarified about graphic design needs. To avoid any miscommunication of intentions most of the content was drafted on the fly and had to be approved by me before proceeding to the final designs. His drawings were based on my guidelines both for the layout of the interface, but also as far as the habitats and hints specifics are concerned. For the former I had to setup an interface wireframe which he used as a guide to draw the sketches in the right proportions (see image below).

The wireframe used for sketching the interface elements.

The latter involved a much more complicated process, since more considerations had to be taken into account. Firstly, the comments about specific characteristics of the habitats had to be accommodated by the designer (e.g., comments like the one about the mountains' shape mentioned above). Also, it was necessary that the attributes of each habitat were drawn in such a way that there was adequate space to place the animal within context (e.g., have the squirrel appear on a tree branch). Another equally important concern was to draw the environment in such a perspective that all animals would appear in a natural size and still preserve the habitat's characteristics. This proved very challenging especially for the 'underground' where animals had to appear in holes and caves, while also maintaining a view of the surroundings. The following figure provides a draft which was a result of many previous attempts to tackle the problem; the skunk's home appears as transparent while the fox' cave has an outwards view.

The 'underground' habitat which hosts the fox and the skunk.

Secondly, animals had to be accurately designed without sacrificing their playful and cartoon character; this demanded thorough research about the animal and its more prominent characteristics which were also described in the hints. What was even harder was to draw specific hints that could be meaningful in guiding players to visually comprehend the hint, without giving away the actual animal. To avoid such incidents, I had to come up with an initial list describing the hint visualization that was then reviewed and negotiated with the animator. An excerpt from the table provided to the animator is displayed below for the owl and the skunk. The resulting sketch drafts, that represent those hints after our discussions, are shown in the following image.

Owl	I am bird living at night.	I like to eat insects and mice.	I have a wide face and really big eyes.	I am a wise bird!
	night sky (already have it)	a plate with a mouse and a fly	a wide female face with big eyes	a pile of books
Skunk	I have stripes in my fur.	I eat bugs, mice, birds, eggs, berries, and nuts.	I can spray liquid to blind my predators.	I use my smell to scare others away.
	a fur (coat) with B&W stripes	a plate with a mouse, eggs, and nuts	a spray in a dog's face	a really bad smell and someone with a disgusted face

Overall, the coordination of the different parties was a laborious and time consuming task that demanded timely coordination of content creation and provision. Unfortunately, time constraints did not allow a second playtesting for evaluating the effectiveness of these negotiated outcomes of educational game design in communicating the message to players[1]. However, we hope that the outcome will pay off by engaging the children in a fun and engaging exploration of the Nature Center!

---

[1] Winn & Heeter (2006). Resolving conflicts in educational game design through playtesting. Innovate 3(2).

Addressing Usability Problems

After completing the playtesting in the actual museum with real visitors (kids), we had to redesign the system to address the problems found. In this report I include a list of the most serious flaws identified and the way they have been tackled. I also include some design decisions that had to be taken based on feedback from instructors and peers. For a detailed list of these usability problems please see the previous post.

#1 How do I scan?

Neither the camera not the actual way of scanning was obvious to none of the participants in the playtesting. To address this issue we are using two ways: on the hardware level and on the software level. For the hardware the intention is to place the camera in an obvious place with a clear label "scan here", but also use some glass in a distance where children can place the tagged label on. In software level we have included an image depicting clearly the way to hold the code in front of the camera.

#2 When do I scan?

This was another frequently-occurring problem where children needed some prompt to start scanning. Again, a virtual helper will be available in the system that will prompt users whenever there is a need to scan a label. The mockup at the moment includes a mouse holding a label which says "scan now" (what more obvious!) Similar prompts appear throughout the game as a means to make always clear what is the current state of the system and provide meaningful feedback on the user's actions. The messages appear at the right of the screen and are not always visible so as not to distract the player's attentions while interacting or reading other information.

#3 Icons... which icons?

An important observation during playtesting which was also pointed out during the pin up sessions what that visual hints were not obvious. To remedy that problem we have made those small images animated, starting exactly at the point of the textual hint and traveling to their final destination (at the top of the image) while passing by the notes counter and increasing it by one. This ensures three things: a) that the meaning of each icon is clear, b) that the icon is visible at ll times and is not lost in the image, and c) that there is a clear connection between hints and notes taken. The drawback is that the icons do not appear in the context of the environment, but we consider this to be a trivial issue after reviewing the nature of the hints (few of them can actually make sense in the environment without the animal).

#4 Where are the labels?

Many kids could not find the tags placed near animal exhibits. This will be easily addressed when designing the labels so as they are big enough and standing at an obvious point. Also, there could be an indication of a code-enabled label in the front side, which will make those labels distinguishable from other labels that might already reside in the museum. A potential design could be like the one depicted here, where the label is standing on a properly cut cork with the animal image and name in front and a thumbnail of the QR code above the picture. This will make the labels recognizable from a distance with minimal effort. However, a full-scale code will exist on the back of the label to make to extremely easy to scan without demanding too much accuracy (the larger the code the easier for the software to recognize the encoded message).

Challenges

However, implementing these features is not without cost. Having a dedicated space for instructions consumes valuable space, but we think that this pays off by making the intentions of the system clear, which is fundamental for the user experience. The labels are fairly easy to be designed as an obvious part of the game, but then the question is if we are leading the player in choosing animals, instead of actually thinking of the animal to look for! Actually this was observed during the playtesting, when children were asking us where are the labels hidden. However, speaking with the client I was told that making the animals that are part of the game obvious is a requirement, despite my reluctance due to making the game fairly easy and with less educational value for the reason explained above.

The biggest challenge was the animated icons and how they would allow future expansion of the system by the museum. A special technique had to be devised in the software for animating those elements while also keeping them accessible on the hard disk. This allows someone to edit them or add new ones using, however, a template with dimension characteristics. There is no way for the system to reject of accept images based on their specifications, so extra care should be taken when updating the content (similarly to doing so with textual information).

Playtesting #1

In order to test the effectiveness of the design as an instructional tool but also its assumed engaging character I run a playtesting session in the museum.  The session took place on a Saturday morning (3/30 from 11:00 to 12:30am) when many kids are visiting the museum with their parents. This was a very good opportunity to test the system with real users in a fashion similar to the intended one. An initial prototype of the game was used which contained two habitats with one animal for each one of them (the bear for the mountains and the turtle for the lake). A couple of more QR coded tags have been placed around the space to test if players actually recognized the described animals or just used the first tag that came to their attention. There was no level or player avatar selection at the beginning.

Setup

The prototype was  set up to run on a laptop computer with an embedded camera which was also connected to a 32'' LCD display mounted on the wall. The laptop was placed on a small kid's table that was underneath and a few feet further from the large monitor. The prototype was developed in a 4:3 ratio (1024x768 pixels) and not the intended panoramic resolution of 1366x768 pixels; consequently it was displayed in a window in both widescreen displays. The system included some initial habitat information from wikipedia, a few clipart illustrations for the hints (but not all of them), and also voice-over only for the informative pop-up messages. A picture of the setup can be seen in the following figure.

The setup of the playtesting.

Procedure

 Children who approached the setup were asked if they were willing to play a game which was under development, having the chance to be part of its evolution. If parents were nearby their permission was requested as well. As soon as kids were seating on the table they were introduced to the game by a screen describing the game's scope. For younger children the explanation was provided verbally, since the narration was missing from the prototype. They then proceeded to the first habitat (by clicking a button) and then to the first hint (again, with a button press). After that they were prompted to find the item described by the game by looking around in the museum. Some encouragement was provided to children who were reluctant to do so.

After finding the QR coded label next to the item (no item was directly QR-coded in our test) they had to scan it and check the system's response. Additional scans were encouraged after an item has been correctly identified. After finishing with both habitats/animals children were asked if they liked the game and would like to see it installed in museum permanently. Also they were asked if they would be willing to fill in some of the information they have came across in the game, on a piece of paper with the animals pictures already printed on. Finally, they were thanked and asked for their name in case they wanted to be part of the game's credits for helping in its development.

Results - Observations

The prototype of Nature Explorer was used by 6 children in total (4 girls and one boy) during one and a half hour. The youngest one was 7 and the oldest 10 with a mean of 9 years. They spend in average around 10 mins playing the game, including the time for filling in the information. All of them wrote something on the paper that they recalled from playing the game; one girl decided to take the "report" with her while the rest wrote their name and left it behind. One 10-year-old boy wrote a comment on the back of the paper (can be seen below).

Evaluation was made in two ways: by asking them what they were doing or looking for during the game, and mainly through observation during their (inter)actions. The main comments from the playtesting are presented below, including suggestions for remedy whenever a problem is presented:

• The information introducing each habitat was probably too dense, which was expected since it was not appropriate for their age
• Most kids had problem identifying where the camera was; this should be made explicit by a physical pointer at the camera in the final physical exhibit
• Most of them held the tag very close to the camera; a sample image of how to hold the item should be provided during scanning interaction
• The tags were not easily identified in the museum, which is mainly due to the fact that they had no clue what to look for; an explanation image with sample QR tag should be presented at the beginning of the game; also labels should be larger
• Some kids did not notice the hints appearing in visual format on the reference image; image hints should be made more conspicuous either by flashing or animating
• The younger girl needed her sister's help to read the information; voice over for all text messages will resolve this
• Some kids tried to type the response; this was due to the text box being visible which is going to be eliminated in the final version
• Kids did not seem enthusiastic about playing the game and looked around reluctantly; I believe this is attributed to the fact that they were being observed by many people (i.e., parents, me, museum director) which made them feel like being examined
• Siblings playing together were observed to help each other, especially the older helping the younger by repeating the clue and prompting her to look around
• Most players were using the laptop's monitor positioned in front of them, while parents-observers were watching the large monitor encouraging their children to follow the prompts (an exception can be seen in the following image); in the final exhibit it would be preferable to have the camera positioned in front of the large display so that children can interact by scanning without blocking the view to other players/participants
• All kids recalled some type of the information received and wrote that on the report

A boy scanning a tag while standing in front of the laptop with his sister.

It seems that most kids enjoyed the experience and would like to see and play the game more often in the museum. One 9-year-old boy specifically, wrote the following comment on his report and also told the museum director that this was a cool game!

A 9-year-old boy's comment about the Nature Explorer prototype.

Overall, I believe the playtesting revealed some of the system's flaws that need to be addressed in the next design iteration. Besides the expected deficiencies due to the unfinished nature of the prototype other findings will be addressed with guiding tips throughout the interaction and context-sensitive help. Instructional and educational effectiveness was not assessed rigorously both due to the type of space and visit (some parents were eager to leave as soon as the game finished) but also due to the informal type of assessment (not IRB protocol was acquired). However, all kids were willing to note down some of the things they learned during the experience, like the girl in the following photo.

A girl writing some of the information she learned, overlooked by her brother.

Instructional Goals and Design

The main instructional objective of Nature Explorer (NE) is to provide some type of scaffolding for the young visitors of the Price House Nature Center to facilitate their learning in this informal education setting. This has been accommodated with the design of a scavenger hunt game incorporating an open-ended learning environment which leverages the physical space and the knowledge already existing in that space. The learners become explorers who have to seek the space of the museum and discover the items described by the clues that the game is giving them. Nature Explorer is designed in such a way to provide the information in the form of a narrative through a role playing game. The engagement evoked by the game and the playfulness of the content, are the main factors that justify the chosen design for the instructional objectives on this informal, free-choice learning setting.

Instructional Approach

As was mentioned in a previous post, Nature Explorer is trying to address all four phases of instructional design in the following ways:

• Present information: information is presented in three modalities (i.e., as images, text and audio) to accommodate for the diverse needs of the audience (children between 4 to 10 years old), either through hints or extra info (trivia) about the item/animal being requested by the system
• Guide the learner: different levels of hints are provided to guide the visitor/player as of which artifact in the museum the assignment refers to; habitat information is presented to help the player set his mind on a specific environment (the museum is divided in similar natural settings)
• Practice: speed is not desired for this relaxed type of informal learning; two different levels (intentionally not connected with age) and ability to play again with a random set of items might elicit retention and fluency, although this is not the focus of the system
• Assessment: although assessment cannot be ensured in such an informal environment, NE will provide a printout after the quest prompting visitors to fill in the missing information related to their accomplishments (things they have already learned)

Motivation

Since Nature Explorer is addressed to young children it is imperative that it is compelling to them. In order to cater for this need various techniques have been used to increase motivation and engagement. The following are the most important design decisions that were taken towards this direction:

• Provide a meaningful goal through story telling and role playing (fantasy)
• Assign the role of explorer to provide a context and identity
• Exploit physical space through assignments that demand exploration and discovery (curiosity)
• Enable collaborative game play through group avatar choice to encourage social play
• Provide performance feedback and score keeping (experience points)
• Vary the difficulty level to allow advancing through experience (challenge)
• Get report and make it public to increase sense of accomplishment
• Get a summary of your achievements (accomplishment)
• Involving the child emotionally in the story (achieved through the narrative)
• Include interesting audio and visual effects to enhance sensation

Overall, NE tries to harness all three constituents of the motivational theory according to Malone (1981) which appear with red italics in the list above: curiosity, challenge, and fantasy. Additionally, it is a major design consideration to make Nature Explorer a "cool" experience for the museum visitors; hence, I have tried to accommodate the four factors of the Wheel of Joy according to Holtzblatt (2011): identity, accomplishment, sociality, and sensation, noted with dark blue color in the list. The design elements used to satisfy both these approached will hopefully increase the motivational impact.

Additionally, the layout of the interface has followed some design guidelines to ensure that it is closer to children’s culture, and communicates information more effectively for these age group (4 to 10 years old). Some of the design decisions that support this claim by evoking emotions and increasing engagement are the following:

• Use of colorful cartoon imagery with matching hand-drawn interface elements (no straight, strict lines)
• Use of the golden proportion rule to divide the screen asymmetrically but meaningfully between the main content (text and images) and the map
• Use of a warm color palette as a means to both evoke emotions but also as a good match for a system related to nature (earth color tones)
• Use of a serif, decorative font with a playful character in a big enough size to be easily readable from a distance of 3-4 feet
• Use of animations and cross-fade transitions in a sequence from left to right and top to bottom to allow seamless processing of new information
• Use of narration for all textual content to accommodate for younger ages but also for groups where not everyone has equal access to the screen
• Use of motion tween and transition sound effects to provide aural feedback of visual changes
• Use of other sound effects and sound tracks in specific points to increase engagement and draw attention to things happening on screen (e.g., item appearing on the image)

These guidelines for effective information design were drawn by the book “White space is not your enemy” (Golombiski and Hagen, 2012) and my own multimedia developing experience.

Storyboarding

During storyboarding I have used some techniques of cinematography to communicate the scope of the system more effectively. This was quite appropriate in my design since it involves physical motion of the user and not just manipulation of a computer system. Thus, long shots have been used to represent the big picture of the system in the museum hall and the actions of the user in space. Close shots were mainly used to reveal actions of the system and interaction of the user with the interface. Close-ups were used to display details of the system, like the QR codes attached on museum artifacts. Finally, different perspectives were used to reveal a cinematographic approach of the user interacting with other people in space (e.g., last scene). For details see the full storyboard of the system.

Golombisky, K., & Hagen, R. (2010). White Space is Not Your Enemy: A Beginner's Guide to Communicating Visually through Graphic, Web and Multimedia Design. Focal Press.

Holtzblatt, K. (2011). What makes things cool?: intentional design for innovation. interactions, 18(6), 40-47.
Malone, Thomas W. (1981). Toward a Theory of Intrinsically Motivating Instruction. Cognitive Science: A Multidisciplinary Journal, 5(4), 333-369.
Greenberg, S., Carpendale, S., Marquardt, N., & Buxton, B. (2012). Sketching User Experiences: The Workbook: The Workbook. Morgan Kaufmann.