Usability: What to test with newbies?

[Simon]

Hi,

what parts of the Lix user interface should be tested with new users?

I do not want everybody to go out of their way now and do this kind of testing. If you ever have the chance to entice new users to play Lix, help them enjoy the game instead. Rather, it's my own duty to have random people try the game, and observe their reactions closely.

Here, I would like to ask you for ideas what to test for.

This is inspired by ccx's concerns about the user interface.

How to test

• It's best if your candidate hasn't played before, or at least not in a long time.
  
• Keep a clipboard, paper and pen ready for yourself. You'll have to take quick notes.
  
• Set up a fresh installation of Lix. Run the game.
  
• Have them take over the computer now. Interfere as little as possible. Maybe you'll have to get the program out of weird state for the next test. But during a test/question, there should be no interference at all.
  
• Give your candidate one of the tasks below. Clearly explain the goal of the task, but do not explain buttons/hotkeys/etc. This applies to questions by them, too: Do not answer questions about the user interface! We're trying to watch a new user without skewing their actions. Clarify the task itself as much as you want.
  
• Write down what they try first to accomplish the task. This feedback is most important to me. Maybe they don't hit keys/buttons immediately, then record what they examine first, i.e., where they look for a solution. If they don't get it right on first or second try, you can also write down what they try next.
  
• Don't mix tasks. If they don't seem able to get it right at all, it's OK to abandon a task. Make a clear note of this! This is also extremely important feedback.
  
• Pay more attention to their actions than to what they say. People are trigger-happy with rationalizations on whatever doesn't fit perfectly into their mental model. You're fine to write down their recommendations still.
  

What to test

If you have further ideas, suggest them!

• 1. Play the first tutorial level. (Remember you shall not explain any UI from the fresh installation's language/name entry menu onwards!)
  
• 2. Singleplayer: Pause during the game.
  
• 3. Singleplayer: Exit a level before it's finished.
  
• 4. Menu: Play a non-tutorial level, right after playing a tutorial level, with the singleplayer level still listing tutorial levels.
  
• 5. Menu: Start the editor on a given level.
  
• 6. Editor: Select an existing piece with the mouse, then move it using the keyboard.
  
• 7. Editor: Add a new piece, doesn't matter which one they find.
  
• 8. Editor: Add a new hatch to the level.
  
• 9. Editor: Make that newly added hatch the first one.
  
• 10. Editor: Exit without saving.
  
• 11. Multiplayer: Can they learn to assign skills quickly?
  
• 12. Multiplayer: Can they have a good feeling for what's going on in large maps?
  
• 13. Multiplayer: Do they notice the sound effect when other players are stealing their lix?
  
• 14. Multipalyer: Do they understand the wrapping behaviour of maps?
  
• 15. Multiplayer: Do they feel overwhelmed?
  
• ...?
  

-- Simon

[NaOH]

Multiplayer is also a place where users can get frustrated easily. This is harder to monitor in a controlled setting, though. This is also somewhere where good interface design and good gameplay can help the player.

Things to check for:
- can they learn to assign skills quickly?
- can they have a good feeling for what's going on in large maps?
- do they notice the sound effect when other players are stealing their lix?
- do they understand the wrapping behaviour of maps?
- do they feel overwhelmed?

[Simon]

Yes, multiplayer newbie testing has been neglected. We have been playing with those who stumble into IRC, but then some long-term player is always happy to explain the rules.

It's hard getting to watch in reallife one or even two new players playing a networked. I'll keep an eye out for opportunities.

We've talked about making a dedicated directory of easy multiplayer levels, that's still on the agenda. New players run the risk of choosing any map, then be overwhelmed by failing to build their own route in the first place.

-- Simon

[RubiX]

Heres just a thought I have, which doesn't really match what you are trying to get across (as you are wanting to see how new players do on their own without being helped)


This is more of internal help system one day possibly:
If there was a new tab in the menu called something like   "Lix Informational"   the menu will have rows of buttons the size of an ability button with a picture in each.
Clicking each picture would open up to a youtube video which explains just the one thing they are looking to get information on.

Page might look something like this setup: X's are picture boxes

    X          X           X          X
    X          X           X          X
    X          X           X          X
    X          X           X          X
    X          X           X          X

Unless A5 allows embedding a video to play in a small frame inside the game, so you would just click play to see an informational video on something.



Anyway, i'll see if I can find someone to test things out as your request when a chance may arise.

[Proxima]

A good task would be to get some players to play all the way through the tutorial levels, others to play the beginning of the community set without having played the tutorials. How easily does the player understand:

* What the basic goal of the game is
* That each level has a specific save requirement, and how this is displayed
* How to restart a failed level
* How to play the next level after completing a level
* How to use fast-forward
* How to use framestepping
* How to use savestates
* How to save and view replays

[chaos_defrost]

We've talked about making a dedicated directory of easy multiplayer levels, that's still on the agenda. New players run the risk of choosing any map, then be overwhelmed by failing to build their own route in the first place.

-- Simon

So, maps designed with new players in mind for multiplayer. Important things to consider:

1) New players are not as likely to know how overpowered blockers and the like can be. New players will have a lot harder time using other skills to turn or deflect Lix in maps where alternative methods are required. Skills that are usually avoided in levels made for expert players are needed for accessible levels.
2) Maps designed by and for expert players are fairly complicated. Simple maps for newer players are usually better. Balancing becomes less of an issue on these maps. Consider fewer wrappings, more open skillsets, etc.
3) Consider what levels might have been fun in multiplayer L1 in these designs. These levels were not made for experts, but for a couple friends playing together at home.
4) Difficulty ratings for multiplayer maps. It's a bit condescending for accessible maps to be called "Easy" as a rating and stuff. Multiplayer maps might be best rated in difficulty like single player levels. I propose a lot less flair in these ratings tiers' names (Something like Standard/Veteran/Pro/Extreme, where the easiest rating is considered the benchmark for play and not "baby mode"). In conjunction with 1), "repeat" levels can be made in harder skillsets without skills that can be very important crutches to a less experienced player but become really level-breaking to experts.

[geoo]

Just some belated reports from some testing I've done, or more precisely, from watching two guys play through most of the tutorial levels:

In order to pause the game, the user first tried Return, then Space. So I think the choice is actually more intuitive than P which might be just what lemmings users are used to. I can think of video players like Youtube and VLC that also use Space to pause the video.

Users (self-admittedly) don't feel like reading chunks of text. The tutorial levels shouldn't describe what the skills do. The first level should describe the aim of the game, the other levels should just introduce control features. Sample tutorial text (in a level where it is useful): "Press [Space] to pause the game." Save requirement should be introduced, and the other features of the panel.

The symbols on the nuke and the pause (paws) button aren't clear.

Some of the tutorial levels are too hard or it's not entirely clear what to do (Jumper unclear, miner/platformer too hard without pause. Blocker/builder has potential to mess up).

The game displays a Play button when the user first enters the level browser, pressing it doesn't do anything. A player might want to dive right into the game instead of being confused. At the same time, the player should also realize that the single player menu is a level browser (this wasn't clear in my test case). So by default, probably the first tutorial level should be selected. But then it's not clear to the user that this is a level browser, and how to navigate it...

Player also wished for a next level button at the end of level dialogue.

[ccexplore]

The game displays a Play button when the user first enters the level browser, pressing it doesn't do anything. A player might want to dive right into the game instead of being confused. At the same time, the player should also realize that the single player menu is a level browser (this wasn't clear in my test case). So by default, probably the first tutorial level should be selected. But then it's not clear to the user that this is a level browser, and how to navigate it...

It's actually a good thing that you didn't make it clear on the outset that it is a level browser.  Admittedly I probably don't play as many different games nowadays as most of you here do, but t feels to me that most games just don't feature a level browser designed quite like Lix's, so it is an excellent newbie usability test to find out how quickly they are able to figure it out with how much effort/trouble.

Buttons that don't seem to do anything are clearly bad design and it's good that the usability testing uncovered this.  Clearly those buttons need to be either displayed in disabled states in contexts where they won't do anything, or have them pop up modal messages when invoked to try to explain to the user they need to do something with the left side of the level browser UI.

I agree that while it is probably an easy way out to default to selecting the first tutorial level, in the long run that will likely just defer the issue of getting the user proficient with the level browsing interface.  Unless further testing shows that a more radical UI redesign is needed here, it is probably best to start with minor UI tweaks first, but still lead the user to have to navigate to levels, and see whether they are sufficient.

[ccexplore]

Buttons that don't seem to do anything are clearly bad design and it's good that the usability testing uncovered this.  Clearly those buttons need to be either displayed in disabled states in contexts where they won't do anything, or have them pop up modal messages when invoked to try to explain to the user they need to do something with the left side of the level browser UI.

Or even easier, let's just try hiding the Play and Edit buttons whenever there is no selected level.  Having buttons that hide and show dynamically may be questionable in some cases, but here it seems like it'd be perfect to help the user understand the proper steps to take--if those buttons aren't there at all when there are no selected levels, the user has no choice but to try the buttons on the left that goes into subdirectories.  Once they get to a place where there are actual levels listed in the middle (in place of those long explanatory texts they probably won't read ), hopefully they'll try clicking on a level, and then that causes a level to be finally selected, which causes the Play and Edit buttons to show up.  The human eye is somewhat sensitive to changes, so hopefully the sudden appearance of the Play button will be very discoverable in this context, making it clear what to click next.

[Yes, I'm aware currently the Edit button overloads as "create new level" when no level is selected.  Let's table that can of worms for another time after we get some usability data for level editing. ]

[Simon]

Yeah, hiding the play button is necessary. It's a little sad because the C++ browser classes need some fundamental change for this, while it's been implemented in the D port months ago.

Hiding the edit button for usability -- I'm considering to hide the button in the level root dir, and in the tutorial subdir that's one level away from root. I'm fine with hardcoding this. Or hide the edit button until at least one level has been started, even though I shun unlockables in games.

Other ideas by everybody to whom I haven't replied -- I've read everything so far. :-)

Bonus usability picture: Aleph-null-many exploders, another button with bad usability, but magnificent coolness factor.

-- Simon

[NaOH]

Hiding the edit button for usability -- I'm considering to hide the button in the level root dir, and in the tutorial subdir that's one level away from root. I'm fine with hardcoding this. Or hide the edit button until at least one level has been started, even though I shun unlockables in games.

An alternative that doesn't require hardcoding or unlockables -- have a lock.x.txt file that prevents editing in this directory. (The user can delete the lock in the filesystem manually, of course.) This also lets level sets be locked so you don't see the tempting edit button...

Bonus usability picture: Aleph-null-many exploders, another button with bad usability, but magnificent coolness factor.

This is amazing, how do I get this?

[Simon]

An alternative that doesn't require hardcoding or unlockables -- have a lock.x.txt file

Hmm, this is better than hardcoding. I still have two hunches against this. The weak one: The editor allows to verify that there are no hidden exits or traps inside solid terrain. This is a weak counter-argument, because our culture already weeds out offending levels before they make their way into the distributed level tree. Maybe this idea points to a design flaw of the game class instead, where trigger areas could be visualized on command.

The stronger hunch is that it's bad to activate features by deleting a file, instead of by checking an option for advanced features.

People learn the level editor by opening and modifying existing levels, by deleting/moving tiles. They don't create levels from scratch early in their learning curve. I would like to foster such learning by exploration, and keep levels editable. On first sight, this seems only possible by separating singleplayer and editor further than they are now.

Bonus usability picture: Aleph-null-many exploders, another button with bad usability, but magnificent coolness factor.

This is amazing, how do I get this?

It's yet-unpushed D code, where I have been toying with unicode symbols. The bitmap font used in the C++ port doesn't provide this many unicode chars. To get it in C++ Lix, you'd have to draw it yourself over the star glyph in data/font_big.I.tga.

This is more of internal help system one day possibly:
If there was a new tab in the menu called something like   "Lix Informational"   the menu will have rows of buttons the size of an ability button with a picture in each.

Videos may be appropriate for multiplayer techniques, hard singleplayer levels, or level editing. The design challenge is to make videos for simpler things unnecessary. Therefore, such links might not go into the game, but, e.g., on the homepage.

-- Simon

[NaOH]

The stronger hunch is that it's bad to activate features by deleting a file, instead of by checking an option for advanced features.

Perhaps one can disable the lock within the game with a checkbox somewhere in the level browser.

Edit: degarbled

[Simon]

Loose idea: Inside the editor, there is no way to make a directory. Where will new players save their levels? Even if they manage not to overwrite the existing file, which is easy to overwrite with one button that doesn't ask again, they would save in the same dir as the included levels.

Implement reasonable virtual file tree in the D port, for optional systemwide installation of the game. Allow mkdir from within the game.

Generate a blank dir for each new user, so they can save edited levels there? Then point editor save path there by default.

Gnah, so many things that were a problem from day 1, never caught.

-- Simon

[NaOH]

The lix filebrowser has some issues. A scrollbar (or at least a way of scrolling up) would be helpful. Knowing how many files are in a large directory would be nice too.

It should allow mkdir. Creating directories, organizing files into directories -- these are all things a file manager should do, even if it is a simple one.