Week 11: Testing IP2 videos

As the in-class testing session had some problems, here are two videos (apologies for the vertical filming orientation) of the first testing session with the two formations.

Week 7: Reviewing video prototype testing

Question	Type	Traps
Q1. Do you understand the concept and gameplay? Scale '1: No, did not understand at all' to '5: Yes, understood the whole concept'	Quantitative	This was a compound question.  Understanding the concept and how the game works is two different things. Should have separated this question.
Q2. Are there any parts you did not understand?	Qualitative	Not specifically, it builds on the previous question which should have been separated.
Q3. Did the video format help with your understanding of the concept? Scale '1: No, it didn't help' to '5: Yes, it did help'	Quantitative	This was a bit leading.  Maybe "How did you find the video format of the prototype?"
Q4. What would you improve about the game?	Qualitative	No
Q5. Do you agree with this statement?  The concept is engaging. Scale '1: Strongly disagree' to '5: Strongly agree'	Quantitative	Could be leading.  Might have been better as a semantic differential e.g. "Rate the concept (engaging to boring)" or similar.
Q6. Was the audio of a quality that you could understand what was being said?	Qualitative	Probably a bit wordy "could you hear what was said" would have been simpler.
Q7. Were my instructions clear?	Qualitative	Leading question, could again be a semantic differential which would have made it quantitative as well. e.g."Rate the instructions (clear to confusing)"
Q8. Any other comments?	Qualitative	No

Video Prototype: Testing

Outcomes

Users were asked to view the video to provide feedback.  They could pause, rewind, scrub and re-watch.  Users were also encouraged to ask questions at any time.

For the most part users watched the video through.  One person watched twice and a few people stopped at the credits, but for the most part they watch in a linear fashion.

I measured user feedback through observation, in-person questions and an online questionnaire. The data from these measures can be found here.

The feedback I was after was about the concept (was it clear, easy to understand, fun, engaging) and the prototype format (video and audio quality).

Reflections

The video format for the prototype was a fairly quick way to get feedback about the concept without having to code anything.  It was more detailed and controlled than simply telling someone my idea and also allowed for engaging visuals.

I received a number of pieces of useful feedback on the concept through the video prototype. Things I hadn't considered (e.g. how do you start/stop) and extensions of existing ideas (e.g. power-ups).

Testing in class provided a number of willing participants but I know this is not indicative of testing prototypes with 'real' users.  The testing protocol I laid out worked well but I know I was much more informal with users than I have been in other usability testing situations or if they hadn't been my classmates with a similar background to developing the concept (i.e. we all knew it would be a game mash-up or a wearable).

Effectiveness

The protoype was very effective and achieved the goals I set out to assess whether the concept is entertaining and engaging, the gameplay is clear and logical, and to facilitate feedback on the concept.  All three of these goals were achieved.

Constraints

The constraint of the video format meant that users could not interact with the concept, only take in information.  This meant that I had to show how users would interact in the video protoype (i.e. show feet stepping on mat).  I had a number of bits of feedback to say this was clear so I think this constraint was able to be overcome.

The constraint of the testing session was that it was an artificial testing environment.  This meant that users were already prepared with background knowledge (negative) and very willing to give feedback (positive).

Implications

Changes to my concept

• need a start/stop/exit button (or method)
• walkthrough/game instructions might be helpful
• extra 'fun' could be added through power-ups (tuna) or traps (catnip)

Future prototypes

• Interactive Prototype 1  (IP1)
• I will add a start/stop/exit button on screen 
• verbal instructions
•  Further prototypes
• start/stop/exit could be implemented by another method e.g. jumping or double tap on mat??
• think about best way to display instructions, maybe a walkthrough/video/intro
• add in the extra bits (maybe IP3)

Future testing sessions

• Run testing in addition to class session
• Act more formal in-class session (i.e pretend they are real users who know nothing about the background) and print out testing script

Video Prototype: Testing Data

Here is the raw data from the testing sessions that were run for the video prototype of Meow Meow Cat.  The data forms two parts, the observations recorded during the session and the questionnaire users filled out.

Testing session: 19 August (week 4 workshop B)

Users:

• 7 total
• 5 male, 2 female

Observations

User #1

• nodded head as watching 
• bored at credits and stopped
• watched twice through
• second time turned up volume (video volume ok, room background noise was competing)
• comments
• easy to understand explanation
  this is what happens (screen animation) and then this is how you do it (floor mats)

User #2

• laughed at GIFs
• thought character could be anything (changeable) so did not see link between intro (cat GIFs) and the game
• clear on concept
• concentrated on viewing

User #3

• concept was clear
• loves cats so loves concept
• smiled a lot
• interface
• how do i exit?
• how to start/try again?

User #4

• no questions
• smiled, seemed happy at end of video

User #5

• laughed at cats
• had trouble understanding gameplay, said it might be English comprehension
• didn't initially understand how screen and floor-mat related to each other
• had trouble reading and filling out questionnaire
• would like a how-to-play screen

User #6

• lots of concentrating
• no questions about gameplay

User #7

• laughed at cat GIFs
• nodded when floor mat was shown (understanding?)

Observation overview

• GIFs/cat videos were well received, many laughed, lots of smiling throughout, showed engagement with idea
• 6/7 thought concept was easy to understand
• showing the floor mat after the interface screen seemed to be crucial in understanding how the gameplay worked
• instruction/walk-through screen might be helpful
• need to add start/stop button to begin play, allow players to exit and replay.

Questionnaire results

Total responses: 7

Less than 7 responses to an individual question are noted

Q1. Do you understand the concept and gameplay?

Scale '1: No, did not understand at all' to '5: Yes, understood the whole concept'

All - Yes, understood the whole concept (5/5 on scale)

Q2. Are there any parts you did not understand? (4 responses)

• no
• No .. it was all very clear.
• Are there multiple levels?
• Not really no. I guess more information about levels and difficult? But that's something that could always be added later.

Q3. Did the video format help with your understanding of the concept? 

Scale '1: No, it didn't help' to '5: Yes, it did help'

2 users: 4/5 on scale

5 users: 5/5 on scale

Q4. What would you improve about the game? (4 responses)

• Nothing, it seemed good to me.
• It would be interesting to have a two player game where you race your own meow meow cat against others. And there are 'traps' along the way that push your cat backwards. For example, land on catnip and your cat takes a nap for a while. Or gets distracted by a laser pointer.
• Based on the video, i have a clear understanding about how the mechanics work in the game, however, i might miss the part that explain the end for the game(like where to quit?)
• Power ups? Some sort of added fun mechanic to diversify the gameplay.

Q5. Do you agree with this statement?  The concept is engaging. 

Scale '1: Strongly disagree' to '5: Strongly agree'

1 user: 3/5

1 user: 4/5

5 users: 5/5

Q6. Was the audio of a quality that you could understand what was being said?  

• Yes, but maybe a little bit fast
• audio is clear and easy to understand.
• yes. It was very clear.
• Yep. Audio fade at the start was good. Only one part where it was talking about Cats and Experience not sure if it said Catsperzience or not.
• Yes I could understand everything.
• Yes!
• Absolutely

Q7. Were my instructions clear?  (6 responses)

• yes
• Crystal clear
• Yes!
• Yes instructions on how to play the game were very clear. Good Visual representation made it easy to understand.
• Yes
• Yes.

Q8. Any other comments?

• Really cute drawings, cute cat, cute sun, cute concept :)
• the game doesn't have many connection with cats. cats can be changed to anything animal
• None. It looks like a fun concept.
• Not really. Just interested to know about multiple levels.
• Maybe it better to display the game as an example
• That is a great game at all aspect
• Meow ^.^

Questionnaire overview

• Concept was easy to understand
• Video format was helpful
• sound levels were good
• game instructions were clear
• add in more info/details about levels
• add power-ups or traps for more fun
• need an exit button

Week 4: Editing

Editing the video wasn't as hard as I'd imagined but it definitely took a lot of time.  There were a few tricks that I picked up (like the only way to import a GIF into a video timeline was to convert it to a mp4 first), but mostly what I learned was around planning (or over-planning to be precise).

My script was good, my storyboard had a plan for all my scenes but what I didn't account for was planning for time.  Our video has to be between 1:30 and 3:00 minutes long and I didn't understand what that meant in terms of planning.  I had a succinct plan and I thought it would be about right for time.  I didn't time my script because sections of it were illustrative/visual only.  In the end, I fell a little short and had to go back and add in an extra scene to make up time.  I would have liked more shots of the game-play stop-animation but ran out of time to go back and shoot more of this (the set-up and filming time were quite long for this section).

Lessons learned:  Shoot more footage than you need.  If you only shoot exactly what you need - you may run short!

Video Prototype: Testing

The video prototype below is about a game mash-up called Meow Meow Cat.
Please follow the instructions below.

1. Watch the video.  You can stop, pause or rewatch at any time.
2. Complete a short questionnaire about this video

Video Prototype: Final Concept

Meow Meow Cat is a single-player, side-scrolling game that requires players to move in a continuous direction, from left to right, through a series of coloured dots.

Mock-up of game display

The player takes on the role of the cat, Meow Meow, whose aim is to reach a ray of sunshine to take a nap.

The game consists of two main items, the digital game interface and a physical floor mat with four large coloured dots.

Paper prototype of floor mat

Players traverse the dots on screen by stepping on the corresponding coloured dot on the floor mat.  The faster you move, the closer you get to the sunshine and complete the level. If you’re too slow, the sun disappears and the game is over. As the levels increase, the complexity of the coloured path and the speed also increase.

Meow Meow Cat is a game mash-up based on Lemmings and Twister.  It takes elements from Lemmings including: cute animal character, movement through levels, and it’s overarching goal, and combines it with elements from Twister including: coloured dots, floor mat interaction, and the ‘game’ choosing the colour order (and complexity).

Week 3: Filming

I had my script, storyboard, supplies and apps ready to go, now I just have to dive in and film!  I'm sure many people don't like filming themselves but I was definitely dreading it.  My script is broken down into four main parts for filming: person straight to camera introducing the concept, stop-animation of the game play using cardboard and whiteboard + voice over, footage of person interacting with the play mat + voice over, and intermittent bits of animation/high res graphics.

The voice over was easy enough, I used the blanket-over-the-head technique from Grumo Media.

The animation/graphics I have done some of and will complete as I edit the film together.

The stop-animation was interesting to put together.  I found I had to re-shoot the beginning sections a few times as I figured things out.  I started out with a lynda.com tutorial on this so was prepared with an app and a plan.  The first few takes I used a stop-animation app but it looked like it reduced the quality of the pictures and I thought it might compromise the end result.  So, I went back and re-set and took images again just using the camera app.  I then realised it wasn't set out quite how I wanted and third time lucky, re-set and re-shot.  I used a piece of cardboard with a line on it to measure the distance from the side of the cat to the edge of the screen.  I also drew lines on the whiteboard to line up the shot, one under the dots that I lined up the bottom of the photo with, the other I used to align the sun with as it moved.  Here is an image with all my guides.

The next section I filmed was talking straight to camera + the interactive mat.  I ended up with quite a set-up to film the introduction with books on stands to get the camera right and a toilet roll cut out to hold the camera in place.  I also had a large sheet of white cardboard set-up to help reflect light.  

I filmed at least a dozen takes, the first few individually, and then once I worked out exactly where to stand, where to look, when I closed (or rolled!) my eyes, I just kept it running and did it over and over again.

Next stop: editing.

Week 3: Video Prototype storyboard

In our workshop session this week we developed storyboards and presented them to the class for feedback. I was hesitant to get started as I was only about halfway through my script.  I had a rough idea of what I wanted to say and had refined some phrases, but it wasn't complete. In class, I was able to quickly sketch up a rough storyboard of what I wanted to appear in each scene.  Having a clearer picture of the different scenes, I was then able to look at my script development.   Jumping straight into storyboarding actually helped me finish my script as I had a better idea of what I wanted to say in the context of when I would say it.

After class, I was able to finish my script easily and much quicker than I think it would have taken me if I hadn't done the storyboard.  When approaching this prototype I had a clear idea of what steps I would take and the order I would take them.  What I learnt from this process was that jumping to the next step before finishing the last isn't always a bad thing and it's important to be flexible.  Sometimes going on to the next thing can help you finish the previous task.

Week 3: Video Prototype concept

After brainstorming a lot over the past week, I was struggling to come up with an idea that:

• i didn't hate
• felt original 
• wasn't just a tweak of an existing game.

I have worked an idea now into a beginning concept that I am finally happy enough to move forward with. As we have a workshop class tomorrow and the video prototype is due next week - I am relieved.  I have also tried to keep in mind how this idea will play out for other types of prototypes (e.g. physical prototype with the makey makey).  

Initial concept: game mash-up

Lemmings x Twister

Elements of Lemmings:

• cute animal
• move through levels 
• overarching goal (in lemmings = don't die)

Elements of Twister

• interact with coloured dots
• more colours = more complexity
• 'game' choose colour order

Game description

• Animal moves through levels trying to reach/chase something (it's pack, it's prey, whatever)
• To move along a level you must pass through/step on coloured circles
• As levels increase complexity of coloured path increases as does time
• To step on coloured circle you must press corresponding key (left, right, up, down corresponds with 4 different colours)
• The faster you move, the closer you get to what you're chasing (pacl)
• As levels increase, speed of pack also increases.

Animal ideas

• Honey badger chasing things to eat "Honey Badger Hurry"
• Wolverine chasing pack
• Cat chasing sunny spot

Week 3: Video critiques

In class this week we viewed other, Kickstarter-style videos and discussed their pros and cons. During this process, it became really clear what to do (and not do) for our own video prototypes.

Video 1: Brisbane Park finder

Communication of concept

First impression
It was average; it wasn't terrible but not a good example to follow.

Can you really understand the concept?
Kind of, I was a little confused.  It took a while to understand the concept.

What questions does it raise?
What was the problem they were solving?  It wasn't clear so didn't make a lot of sense.

What could they do better?
Write a script and vary the visuals, one person talking at the camera got boring after a while. They also used a bit of jargon and acronyms which made the concept harder to understand.

What do they do well?
The prototype look good.

Video Content & Approach

Is there a better way to show certain things?
It would have been good if the prototype matched the description of the audio and also showed the context of use.  Who would use this, when and where?

Quality of the video? Audio?
Video was average quality but audio was not good.  There was a lot of background (traffic?) noise and crows.

Video 1 take-home message: Explain problem and concept upfront, quickly and easily.  Tell people why they should be interested in your product.

Video 2: Google docs in plain English

Communication of concept

First impression
It was good

Can you really understand the concept?
Yes, the explanation went into a lot of detail and included in-use scenarios that I could imagine myself in.  There was a simple story supplemented by visuals.

What questions does it raise?
None

What could they do better?
It felt really long.  It was quite low-fi considering it was Google but the method of communication helped convey how easy it was to use (i.e. you don't need a lot of tech skills).

What do they do well?
Explained the story and scenario of use.  Outlined the problem at the start.

Video Content & Approach

Is there a better way to show certain things?
Could have had actual footage of Google Docs, shown that it is just like working in a word doc.

Quality of the video? Audio?
Great, audio very clear.

Video 2 take-home message: Tell a story, give a good scenario of use, animation doesn't need to be complicated.

Video 3: Pegasus

Communication of concept

First impression
I don't understand what is happening

Can you really understand the concept?
No.  I knew it was a game, and I assumed it was similar to checkers or chess, but I got no details.

What questions does it raise?
What is it exactly (online, IRL)? What does it do? What are the rules? How is the game played?

What could they do better?
There was no story or explanation.  It could have had a voice over with a script to tell the story of the game - what does it do and why would I be interested?

What do they do well?
Prototype was high fidelity.

Video Content & Approach

Quality of the video? Audio?
Music choice was distracting but reminded me a little of GoT.

Video 3 take-home message: Make sure you explain what it is and how you interact with it.  Tell the viewer why would they be interested.

Video 4: Gauss

Communication of concept

First impression
Good video, animation of burning the back of the retinas was a bit confronting.

Can you really understand the concept?
Yes, it was quite repetitive about the dangers of blue light.

What questions does it raise?
Am i slowly burning my retinas, am I going to go blind prematurely?

What could they do better?
Slightly too technical, some information only made sense when compared to other bits of information.

What do they do well?
Good mix of animation and other footage.  Showed interviews and product and process.

Video Content & Approach

Quality of the video? Audio?
Set the scene to appeal to target audience by showing process.  They also used a lot of blue light - subconsciously driving home the point.

Video 4 take-home message: Range of visuals used is good but don't use confronting images that make viewers shy away from product.

Week 2: Video editing in Photoshop

In week 2, we discussed video creation and editing ahead of our first prototype assessment: a video prototype.  We looked at different types of video (educational vs promotional) and the steps involved to create a video: script, storyboard, voiceovers, video, and music.  Part of the challenge of the video prototype assessment is shooting the video on a mobile phone.  To this end, we were reminded of the importance of shooting horizontally, not vertically!


However, most of the workshop time was spent of learning how to edit video in Photoshop.  We went through a tutorial that covered importing, splicing, transitions and adding audio.

I am quite experienced with Photoshop for image creation and manipulation, but the closest I have previously come to video editing was using Photoshop to create GIFs from video files of my cats.

It was good to learn something new and I am looking forward to practicing this skill over the next few weeks.