Background

This project was part of the Play ‘n’ Learn project of Sesame Workshop India (SWI) in India, supported by Qualcomm Wireless Reach project as donors since 2007. Sesame Workshop India’s association with the Qualcomm Wireless Reach program has always sought to bring modern communication technologies to make high quality educational content accessible to children with an emphasis on reaching the underserved. The larger aim of the project was to demonstrate the use of digital technology on children’s learning outcomes. At the time of this formative research, the project was in its phase III, which built on the work that had been done in the initial phase with the objective of taking learning on digital media to scale and looking at building a sustainable model of development.

Research Objectives

The objective of the formative research was to assess the responses of children to the existing Sesame Workshop games primarily vis-à-vis ‘Appeal’ and make selection related decisions about the games. This necessitated the need to assess the Comparative Appeal of the games. The games tested in the formative research were:

Age (4-6 years)

• Rollercoaster Shapes

• Path Waiter

• Hide and Seek Animals

• Bubble Colors

Age (6-8 years)

• Train Track Engineers

• Oscar’s Trash

• Hide and Seek Letters

• Bubble Numbers

These games were built on HTML-5. Since learning was the purpose of the games, testing them for comprehension was also part of the formative research. Another objective of the formative research was to understand the larger themes vis-à-vis Appeal (including ‘stickiness factors’), which could inform game design and development related decisions in the future.

Thus, specifically, the objectives of the study were:

1. To assess the games, primarily vis-à-vis Appeal and see what children liked about each game.

2. To compare the games with respect to appeal i.e. ‘Comparative Appeal’ and its possible reasons

3. To see which themes were liked the most across games

4. To assess the games with respect to Comprehension

Defining Appeal

‘Appeal’ is a construct that may mean different things to different people. In order to avoid problems of diverse interpretations of ‘Appeal’, we arrived at a minimum common understanding

of ‘Appeal’ as: Something that evokes the desire to remain engaged and participative. Here, ‘engagement’ implies attention to activity and concentration. This understanding of ‘Appeal’ was arrived at through:

• Literature review

• Structured conversations with the representatives of Sesame Workshop India (SWI) to elicit the meanings of ‘Appeal’ implicitly held by them

Design Research Process

• Structural Research Plan

• Literature review

• Conducting on-field activities

• Data Mining

• Synthesis

• Individual Game Appeal - identification of patterns and themes

• Comparative Game Appeal

• Arrive at Recommendations

Research Method

• 48 children participated in this study – 24 from each of the two age groups (4-6 years and 6-8 years), 9-10 children from each of the 5 SECs (A,B,C,D,E) and similar number of girls and boys (4-5) from each SEC (Socio- economic classification)

• Co-discovery method was used, where children played games in pairs

• Each pair comprised children from the same/similar SECs in order to avoid confounding of results due to mixing SECs and to enable comfortable interaction within a pair.

• Each pair played two games, where a maximum of five minutes per game was offered to each player in terms of taking active control of the mouse.

• Data was collected through observations, primarily by an observer but also by the facilitator, and questionnaires at the end of each game. Verbal responses of children were noted down.

• The average time taken by a pair for the entire session was 30 minutes.

• Each game was played by 12 players.

• Besides the SEC criterion, pairing was also based on including both same gender and mixed gender pairs and contingent issues such as how comfortable children were working with each other.

• Accommodating more than two games and their respective questionnaires would have pushed the session time beyond 30 minutes, which roughly corresponds to the upper limits of the concentration span of children in the age group 4-8 years, especially the 4-6 year group (Barendregt W. , 2006)

• This number was considered appropriate as 8-13 children have been found to detect 80% of issues in user testing (Barendregt & Bekker, 2003)

Data Collection

Co-discovery method had the advantage of being closer to children’s natural pattern of behavior and more enjoyable (Als, Jensen, & Skov, 2005).Children verbalized their thoughts through constructive interaction. Combining observed appeal with reported appeal (through questionnaires) had the advantage of comparing what children said with what they did. This was useful given the limitations of young children to read, verbalize, abstract their experience to ‘reasons’ (Markopoulos, 2002).The observations were made using verbatim accounts and low inference descriptors.

Table 1: Gender Distribution across SECs for each age group.

Table 2: Game-pair wise distribution of SECs and gender for age group 4-6 yrs.

Table 3: Game-wise distribution of gender and SECs A, B, C, D and E for age group 4-6 yrs. Please note that the total number of players are 24. The numbers in a column do not show unique players as each player played two games.

Table 4: Game-pair wise distribution of SECs and gender for age group 6-8 yrs.

Table 5: Game wise distribution of gender and SECs A, B, C, D and E for age group 6-8 yrs. Please note that the total number of players are 24. The numbers in a column do not show unique players as each player played two games.

Analysis

• Data from observer notes, facilitator notes and questionnaires was triangulated for each pair for validity

• Triangulation comprised checking if observer notes were consistent with facilitator notes whether what children were observed doing was consistent with what they said

• Data was then coded1 vis-à-vis Appeal, following the ladder of inference

• Appeal related patterns emerged for each game

• To compare the appeal of the games, the observed data for the games each pair played was compared with the responses of children when asked direct questions about game preferences

• Since there are no fixed categories of behavior that indicate ‘Appeal’, the definition of Appeal for this study was used to identify appeal related behaviors. Patterns related to Appeal emerged from clustering these behaviors

• Ladder of inference (Argyris, 1983) is a guiding concept by that helps in making valid interpretations by moving from low descriptor observations to infer meanings and finally draw conclusions

Data Analysis Process

Schematic depicting the data analysis process

Findings

Note: For the purpose of this portfolio, key findings of only 2 games per age groups 4-6 years and 6-8 years have been detailed. Design Research methodology remains the same for on-field observation, facilitation and data analysis of all the 8 games across age groups.

1. INFLUENCE OF MOUSE/TOUCHPAD BASED INTERACTION

• The ability to use mouse/touch pad effectively to achieve game goals affected the gaming experience of children. The game experience of those who could not use mouse/touchpad effectively was interrupted. On the other hand, for children who could pick up mouse usage, mastering the skill added positively to the challenge of some games.

  Thus, we categorized players into three groups based on their observed use of mouse/touchpad and reported familiarity with computers. These were –

• EFFECTIVE USE (EU)- Children who demonstrated effective use of mouse/touchpad to achieve game goals and also reported familiarity with computers.

• LEARNED EFFECTIVENESS (LE)- Children who learned how to use the mouse/touchpad effectively during the course of the games and did not report familiarity with computers.

• INEFFECTIVE USE (IE) -Children who could not use the mouse/touch pad effectively to achieve game goals and did not report familiarity with computers.

  The games were broadly categorized into two types, depending on their interactional requirements :

• COMPLEX - That required click-hold-drag / tap-press-drag action.

• SIMPLE- That required point and click/ move and tap action.

  For both age groups, the ability to use mouse effectively did not seem to have any relationship with SECs or gender. If at all, it seemed to be linked to age, as more children in age group 6-8 years, used mouse/touchpad effectively, without any struggle. Numerous studies (Bruckman, Bandlow, & Forte, 2002) confirm that children’s performances with mice and other input devices increase with age. In cases of conflict between reported familiarity and observed effectiveness of use of mouse/touch pad, the latter was given preference, since this data was richer (and more credible) than reported one (Hanna, Risden, & Alexander, 1997).

  Children have difficulty holding down the mouse button for extended periods and have difficulty performing a dragging motion (Bruckman, Bandlow, & Forte, 2002)

  2. APPEAL POINTS PER GAME (Age 4-6 Years)

  Note: Statements in blue, inverted quotes and italics are verbatim from the data and represent the essence of the appeal point. Children communicated in Hindi and their English translations are in bold. Appeal patterns were categorized into themes, comprehension, goal of the game, play engagement and human computer interaction

  a. PATH WAITER (PW)

• THEME OF FEED AND FEEDING : “Ladka.. kisi ka khana pahunchana tha. To madad kari” Liked the game because “The guy had to deliver someone’s food. I helped him.” Children seemed to like the theme of food. They found the goal of feeding someone meaningful and saw themselves as those who made that possible

• MAKING A PATH, SKIRTING OBSTACLES : “Seedha le ja chal chal” “Come on! Take it straight ahead!” Children liked the act of making a path. Most children who enjoyed this activity were those who picked up mouse usage (LU group). Their newly acquired skills to use mouse effectively seemed to have positively added to the challenge of making a path. The addition of obstacles with each level upped the challenge of skirting them and kept the children engaged

• CHARACTER ANIMATION : “tak tak tak” (onomatopoeically emulating Grover’s walk) “phir bolega that’s right!” laughs and says “tak tak tak”, onomatopoeically emulating Grover’s walk, “and then he’ll say, ‘that’s right!” amused children, perhaps because of the exaggerated nature of the character’s movement

• b. ROLLER COASTER SHAPES (RCS)

• ROLLERCOASTER : “Ab train chalegi! Dekha?!” “Now the train will move! Saw that?!” Children liked the rollercoaster that appeared at the end of the game. Some were fascinated by the theme of going for rides/ “trains”

• SHAPE RECOGNITION : “oye! ye kaunsi shape hai?” “Oi! which shape is that?” Identifying the shape appearing on the screen engaged children. Some took a cue from the voice over that announced the shape. Some associated these shapes with familiar objects like home, box

• TRACING : The act of tracing engaged children till it held some challenge i.e. using longer strokes for the EU and LE groups and using click and drag for the IE group.

• LOW REPLAY APPEAL, OBSERVER ENGAGEMENT : “Aur Nahi khelna. Bahut baar khel liya.” “I don’t want to play any more. Have played enough”. Children, especially the passive players, exhibited verbal and non verbal behavior that indicated low replay appeal and engagement. It seemed to be because tracing was not as engaging after children learned how to make smoother strokes.

3. COMPARATIVE GAME APPEAL

The ‘complex’ games that required click-hold drag interaction were Path Waiter (PW) and Roller Coaster Shapes (RCS). The ‘simple’ games that required point and click interaction were HSA and BC. The EUs and LEs can be regarded as fairly homogenous, since for both, mouse interaction did not become an impediment to experiencing goals of games.

A. FOR THE INEFFECTIVE USE (IU) GROUP- 5 out of 6 participants who played both simple (HSA, BC) and complex (PW,RCS) interaction games preferred the simple ones. Possible Reasons were:

• They were unable to effect the click-hold drag action required in the complex games

• The one child who played both ‘complex’ games (PW and RCS), preferred RCS as he could still manage to play the game with a point and click interaction, though it took much longer.

B. FOR THE EFFECTIVE USE (EU) AND LEARNED EFFECTIVENESS (LE) GROUPS - Between PW and any other game, PW was always preferred. Possible Reasons were:

• All players belonged to LE group. Drawing a path with newly acquired mouse skills may have positively added to the game challenge.

• Each level had different obstacles that added to game complexity and kept children engaged.

• Unlike RCS, the path to be drawn was not fixed, hence positively challenging

• The goal of feeding was relatively more meaningful than goals in other games

4. GAME APPEAL : SUMMARY

• Path Waiter (PW) was most appealing as its goal of feeding was more meaningful and the act of making a path (that was not fixed) sufficiently challenging, with varying complexity at each level.

• Hide and Seek Animals (HSA) was liked very much as the theme of animals appealed to children and they found its fantasy environment stimulating. There were some variations in each level

• Bubble Colors (BC) was liked for the act of bursting and continual performance indicators but some passive players were distracted. The game had little variation between levels.

• Rollercoaster Shapes RCS) was liked primarily for the rollercoaster/theme of rides but the act of tracing was relatively less engaging, especially with replays. Some passive players were also distracted. It had little variation between levels.

5. APPEAL POINTS PER GAME (Age 6-8 Years)

Note: Statements in blue, inverted quotes and italics are verbatim from the data and represent the essence of the appeal point. Children communicated in Hindi and their English translations are in bold. Appeal patterns were categorized into themes, comprehension, goal and recall of the game, play engagement and human computer interaction

a. Game - Oscar’s Trash (OT)

AIMING/ ESTIMATING PULL FORCE : “Ismein kam lagega. Kam kar! Seedha seedha!” “This will need less force. reduce it! Go Straight, straight!”

• Children were highly engaged in estimating pull force and would show reactions of accomplishment when the trash landed correctly.

• It challenged them appropriately.

• A few children found it difficult to use the mouse and perform the click-hold-drag action

MEANINGFUL GOAL : “Isme dustbin mein kachra phenkte hain aur kooda dustbin mein phenkna bahut accha hota hai” “We threw the garbage in the dustbin and that is a good thing to do”

• The goal of throwing trash in the can was part of their everyday experience and they associated it with “doing good”

• Some children created narratives around it as well, such as , “Kachre ko kooda dena hota hai. Woh bahaut bura hota hai, usmein badboo hoti hai. Isliye spring se dete hain” . “Oscar has to be given trash. Trash stinks, so we use a spring”

b. Game - Train Train Engineers (TTE)

FIGURING OUT THE TRACK : “Mazaa aa raha tha. Dimag lag raha tha. Priya saath mein bana rahi thi. Aise aise jod rahe the. Mazaa aa rahaa tha” “I had fun. I applied my mind. Priya was playing with me. We connected the tracks like this. It was fun.”

• A few found it difficult to use the mouse and perform click-hold-drag action to pull the tracks.

• The mental challenge of figuring out the track was one of the most engaging activities across games.

• Highest pair-engagement was observed for this, with children deliberating together

• Children would let out triumphant smiles/ sighs when they could finish the track

• They even reported that they enjoyed it because it “involved their minds”

MEANINGFUL GOAL/ AGENCY : “sabse accha ki meine use apne pita ke paas pahunchaya.Kyunki wooh sirf baitha hua tha, uske paas pattriyaan to thi nahi, to maine game mein accha kaam, usko apne pita ke paas pahunchaya.” “What I liked the most was that I helped him reach his father. Because his father was just sitting and had no tracks, so I did a good thing in the game, helped him reach his father.”

• Children could relate with going to meet someone in a train.

• They spoke of it as something made possible by them

6. COMPARATIVE APPEAL

Here, ‘Complex’ Games that required click-hold-drag interaction were OT and TTE . ‘Simple’ Games that required point and click interaction were HSL and BN. The EUs and LEs can be regarded as fairly homogenous, since for both, mouse interaction did not become an impediment to experiencing goals of games.

A. IN THE INEFFECTIVE USE CATEGORY

• All (3 participants) who played a combination of simple (HSL, BN) and complex (OT) games, preferred the simpler ones. Possible Reasons : They were unable to effect the click-hold-drag action required in the complex games

• One player played both complex games and preferred OT to TTE as in OT some movement could be effected with click. Not so in TTE, which had an additional cognitive challenge

B. IN THE EFFECTIVE USE AND LEARNT EFFECTIVENESS GROUPS

Between the 2 complex games OT and TTE, OT was always preferred (3 out 3 Participants). Possible Reasons :

• TTE had an additional cognitive challenge, which was engaging but higher compared to estimating force in OT

• The goal of throwing trash in the can was more meaningful

• The challenge of figuring tracks more age appropriate

• The goal of taking someone to meet someone more meaningful

7. GAME APPEAL : SUMMARY

Note: The ranking for players from EU and LE groups is taken as final in terms of Appeal as this group comprised the majority. Moreover, the problem of mouse skills can be categorized as more of a Usability rather than Appeal problem using the procedure suggested by (Barendregt, Bekker, & Speerstra, 2003)

• In the order of comparative appeal, Oscar’s Trash was most preferred because estimating force was an age-appropriate challenge and the goal of throwing trash into the can associated with “doing good”.

• Train Track Engineers was highly appealing because figuring tracks was a highly engaging mental challenge. Children could relate to the goal of going to meet someone in train.

• Hide and Seek Letters was liked for an engaging fantasy environment. Each level had a different set of letters and environment which made it engaging for the first time, post which there were signs of lessened engagement.

• Bubble Numbers was liked for the challenge of figuring out numbers, using performance feedback indicators, especially voice over. But it had low replay appeal and passive player engagement as it became less challenging and levels were all similar.

8. APPEAL POINTS ACROSS GAMES (Age 4-6 and 6-8 Years)

1. MEANINGFUL GOAL AND AGENCY

• A goal that was part of children’s lived experience, that they related to positively, was appealing (Eg. Path Waiter, Oscar’s Trash, Train Track Engineers)

• They approached such games with a sense of purpose and saw their contribution as important.

2. VOCAL FEEDBACK AND ACKNOWLEDGEMENT

• Children responded very well to voice over feedback, especially ones that acknowledged them with, “good job!” etc. (Eg. Bubble Colos, Bubble Numbers, Train Track Engineers)

• They often spoke of these as if a person was talking to them, especially in games where the voice over would offer help (Train Track Engineers, Oscar’s Trash). This made such feedback more relevant than, say, visual ones, possibly because children are acknowledged/helped verbally in their lived experience.

3. HIGH ENERGY / EXAGGERATION

• Exaggerated movements on screen, or high energy voice over elicited very favorable responses. For example, the voice over in Bubble Numbers was much more mimicked than in Bubble Colors. On examination, it was found to be more energetic, which a child pointed out as well.

9. MUPPET APPEAL

• It was difficult to infer why certain muppets were liked more than others. This is because very few children could articulate reasons for liking a muppet and simply pointed them out.

• The reasons stated varied from appearance, to what muppets did or said. Another possibility to investigate was the relationship between game and muppet preference. In this regard –

• In the age group 4-6 yrs, 13/24 children preferred a muppet different from the one in the game they liked, suggesting no relationship between muppet and game preference

• In the age group 6-8 yrs, 17/24 children chose the same muppet as in the game they preferred, suggesting a possible relationship between the two.

It is difficult to posit a reason for this, but one possible explanation could be that more 4-6 yrs children made their preferences impulsively in the moment the question was asked, whereas older children perhaps experienced games and their muppets as integrated wholes. This explanation is based on the characteristics of children in the age group 3-7 yrs outlined by Acuff and Reiher (1997) as mentioned in (Barendregt W. , 2006). According to them, children in this age group are rather impulsive and reactive, rather than reflective or logical.

10. COMPREHENSION

Comprehension was also judged based on all questionnaire responses as well as observed behavior and was triangulated just as appeal points were. Comprehension meant understanding of the goal of the game. Game-wise comprehension of children was categorized to various mouse/touchpad usage categories : EU + LE (Effective Use + Learned Effectiveness) and IU (Ineffective Use)

For both age groups,

• No patterns observed between gender, SECs and Comprehension.

• Comprehension seemed a function only of the ability to use mouse effectively

• The ability of children of age group 4-6 yrs to recognize color names from vocal instructions seemed questionable

Key Take Aways

• There was a relationship between game appeal and how effectively children could use the mouse/touch pad. Children who could not use these effectively preferred simpler games that required only point and click interaction.

• No relationship was found between the ability to use the mouse/touchpad effectively and SECs and gender. It seemed to be related to age, with more children in the 6-8 yr old group using the mouse effectively. The relationship between age and mouse dexterity is substantiated by studies.

Recommendations

To create Appeal-

• Themes should be part of children’s lived experience

• Goals should be meaningful where children relate to the situation and see their acts as making a useful contribution, for example helping someone.

• Involve energetic vocal feedback as children tend to associate it with ‘a person talking/ interacting’ with them, which is perhaps a natural form of feedback in their lived experience.

• Include some variation at different levels in games – either in difficulty level or in environment to engage curiosity.

To limit challenges-

Games for the younger age group (4-6 yrs) can be designed to be more platform neutral by involving less complex interactions where input devices are mouse/touchpad.