Interactive Prototype I: Testing Data

Here is the raw data from the testing sessions that were run for the Interactive Prototype I of Meow Meow Cat.  The data forms two parts, the observations recorded during the session and the questionnaire users filled out.

Testing session: 15 September (week 8 workshop A)

Users:

• 6 total
• 4 male, 2 female
• 1 not from class

Observations

User #1

• played three full games
• played two other games with errors (game over message at wrong point, I don't think the game reset in-between)
• wanted to be previous time when he played
• times: 24, 15, 13 seconds

User #2

• I reset game in between goes due to problems with previous session
• played three games
• found colours hard to see
• second go, mashed controls with fist (i.e. pressing all colours at once) to get better time
• third go randomly pressed keys to get better time
• times: 24, 17, 6 seconds

User #3

• played twice
• had problems telling red and purple colours apart
• times: 23, 19 seconds

User #4

• played twice
• asked "what feedback is there if you press the wrong colour?"
• times: 24, 22 seconds

User #5

• played once
• was unable to tell the difference between colours (blue and purple), had to guess
• suggested adding pattern to colours to help
• time: 31 seconds

User #6

• played four times
• tried to beat score, stopped playing when couldn't beat score (i.e. when there was no more challenge)
• commented that it was fun to play
• times: 17, 13, 9, 9

Observation overview

• Limitations of prototype included the modified keyboard; user had to re-learn keyboard input
• Survey took a long time, possibly too many questions
• Game played 15 times
• Playing as intended:
• Lowest time: 9 seconds
• Longest time: 24 seconds
• Playing in other manner:
• Lowest time: 6 seconds (by pressing random keys as quickly as possible)
• Longest time: 31 seconds (user who had to guess colours because they were too similar to differentiate)
• Average time:
• 17.73 seconds (all goes)
• 17.67 seconds (not including mashing keys, random selecting keys or guessing keys)
• no real difference between averages
• Problems with colours being too similar and making it hard to see
• could change colours
• could add pattern to colours
• Start/Stop/Reset button a little buggy - had to press in correct order to work
• All users who played multiple games improved on their previous time

Questionnaire results

Total responses: 6

All questions except the last open 'Any comments?' question was compulsory.

Q1. Describe what you did in this session

• I played the game three or four times.
• Played the game until I could not determine the difference between two of the colours. I then had to stop as I could not play the game.
• Used the colour keys to move the cat along the path of coloured dots
• I pressed the colours that came up on the screen.
• Make sure the colour of each direction key, and try to remember it. play game as fast as possible.
• Pressed the coloured dots in the correct order to advance the cat and try to beat my record time.

Q2. How would you describe the instructions for this testing session?
Scale '1: Clear - it was easy to understand what to do' to '5: Confusing - I did not understand what to do'

5 users: 1/5 on scale

1 users: 2/5 on scale

Q3. Describe the movement that happened during the game. 

• The cat stayed still and the colours moved.
• when I press the right key, the dot will move below the cat
• The cat moved from left to right across the game screen and the coloured dots disappeared
• the dots moved towards the cat/the cat moved along the dots. I'm not sure. I pressed the colour that matched the next dot on the screen.
• the cat moved forward (right) one dot at a time
• The colors shifted left each time I pressed them.

Q4. Apart from the cat and the dots, did you notice any other elements on the screen? 

• I noticed the time and the buttons at top, but during gameplay I didn't pay any attention to them - just concentrated at looking at the dot keys and the dots on screen
• not many, just the those functional buttons
• not after I started playing.
• There was a yellow background. There was a start, stop, reset button at the top. There was a timer up the top left of the screen.
• The time on the top of the screen.
• The Timer in the top left hand side of the game. The Buttons in the top right were also visible

Q5. If you noticed the timer, did it affect how you played the game?

3 options

3 users: Yes, it affected how I played
1 user: No, it did not affect how I played
2 users: No, I did not notice the timer

Q6. Describe how the timer affected how you played.  

• I didn't really look at the timer during the game - only after to see if I beat my previous game's time. perhaps there could be audio reminder every 5 or ten second intervals? Might be worth trying, hopefully it wouldn't be too annoying, but will add some element of pressure!
• I tried to beat my previous times.
• I didn't pay attention to it until the end of the game.
• I want to finish the game as soon as possible
• I did not notice it. So it did not affect the way I played
• Added stress, but also made me want to continue playing to see how fast I could complete the course

Q7. Which elements on the screen moved? 

• the dots are moving
• The dots moved and the timer counted.
• the dots
• The colours moved.
• The colors.
• Only the dots. It seems like the cat is moving but i am pretty sure it remains stationary

Q8. Any other comments?

• It was sometimes a little confusing which colour needed to be pressed as. If that cat was currently on green, I wanted to press green but I actually had to press blue which was the next colour in the sequence. I was able to learn to press the correct button after a few plays but it was not immediately obvious.
• add a key press check function, maybe, because user can press all direction, and the game will be finished very quick
• For some reason, I kept wanting to press the color under the cat rather than the color to the right of that color. I think it would be easier to press the color under that cat because it's not between to other colors, and it's more noticeable because it's under the cat. I think if I played this game more, I would more easily press the colors instinctively rather than having to look down to see where colors were. I think this will be easier in your next prototype because you won't have the correlated instinct of up/down/left/right keys. Overall, I really liked this game! Good work!
• The first comment is about colour. I could not tell the difference between two of the colours and as such I could not finished the game. Change the colours or add a pattern to make it easier to determine which is which. Secondly I would add a space between the colour I am on and the next colour. I was confused at times. If the colour I was on was a different size or shape I would automatically assume it was the colour I was on and look to the next number.
• It would be good if there was a sound when you hit the wrong colour, I found it hard to know if I had pressed the wrong button, or just not pressed anything.

Questionnaire overview

• Users understood that they were moving the cat along the coloured path even though it was the path that was actually moving
• Users found my instructions for the testing easy to understand (Q2) but I should have reversed the scale (so 5/5 was 'completely understand' not 1/5)
• When asked what they did, users understood the gameplay of the cat moving along the coloured path 
• When asked to describe the movement, some (3-4/6) thought the cat moved (what I intended to be perceived), some (2-3/6) thought the dots moved (what actually happened) but when asked what element/s moved, all users reported the dots moved.
• confusion about what moved might be clearer if the cat had movement, even if it didn't physically move across the screen e.g. 'jumped' up and down on the dots
• users knew that the only 'moving' element was the dots but about half perceived the cat moving (what I wanted to happen).
• Half the users (3/6) noticed the timer and used it to compete against themselves even though they were not told to, and they didn't know their times were being recorded.  One other user noticed the time but reported it did not affect how they played.
• Players liked the pressure of the timer to compete but didn't notice during gameplay, only at end of level.
• Give users some feedback when they press the wrong button
• maybe sound or text/image on screen
• Physically distance the inputs for the next prototype to make it impossible (or at least much harder) to randomly press inputs or press more than one at a time
• Colours (accessibility) needs to be fixed
• Game started with cat off the dots and when you pressed the first colour, dot moved underneath, then press next colour, etc. Once you had pressed a couple of colours, it was then unclear/easy to get mixed up which colour to press next.  Was it the colour under the cat of the colour next to (right of) the cat.
• could change size of dots once they are pressed (i.e. colours under the cat and to the left of)
• could add in a market (like a caret) underneath the dot you are to press next
• could animate the cat so it looks more like it is jumping to the next dot
• add more of a space between the dots I have pressed and the dots I haven't pressed yet