Testers were easily able to accomplish the following tasks:
Gedit (text editor)
- Write a simple note. The scenario asked them to type in a short note (about two paragraphs) so I could observe them recovering from typos and errors.
- Basic editing. Replacing a single instance of a few words, and move a sentence from one paragraph to the other.
- Replacing text. This asked testers to replace all occurrences of two names with new spellings of the same names.
- Saving a file as a new copy of the file, with a new name.
Firefox (web browser)
- Search the web for a particular news site, and navigate to that site.
- Set that website as the browser's default startup page.
- Increase the font size.
- Create a new tab or window, and navigate to a particular URL.
- Save the page for off-line viewing.
- Download an image from the website.
- Make a bookmark to a website for later use, and give the bookmark a new name.
Nautilus (file manager)
- Create a new folder.
- Move a folder to a new location.
- Give a folder a new name.
- Delete a file.
So, what's common that made these scenarios go well for the testers? I can provide commentary, but let me first share a few of the testers' responses to the questions "Across all three programs, what went well? Can you suggest why this went well for you?":
- "Replace All" worked well to replace Jeff with Geoff, and Jenny with Ginny.
- Having right-click menus everywhere helped a lot.
- All programs sort of look the same.
- Firefox was "good" and straightforward, seemed easy to use.
- Right-click options were good.
- Having the "+" button in Firefox to make a new tab made it easy, and because it was next to the other tab made it easy to find.
- Highlighting was good, especially in the editor.
- Firefox is like other browsers.
In general, I see Familiarity as a theme. Testers indicated that the programs seemed to operate more or less like their counterparts in Windows, and remember that most (6 out of 7) testers used Windows as their primary desktop. Gedit wasn't too different from Windows Notepad, or even Word. Firefox is like other browsers. Nautilus is very similar to Windows Explorer. To some extent, these testers had been trained under Windows, so having functionality - and paths to that functionality - that was approximately equivalent to the Windows experience was an important part of their success.
Also, Consistency was a recurring theme in the feedback. Right-click menus worked in all programs, and the programs looked and acted the same.
Menus were also helpful. While some said that icons (such as in the toolbar) were helpful to them during the test, most testers did not actually use the toolbar in Gedit - except to use the Save button. Instead, they used the program's drop-down menus: File, Edit, View, Tools, etc. In fact, testers experienced problems when the menus did not present possible actions clearly.
And to extend the analysis slightly further, allow me to contrast with what didn't work well - Obviousness. When an action produced an obvious result, or clearly indicated success - such as saving a file, creating a folder, opening a new tab, etc. - the testers were able to quickly move through the scenarios. When an action did not produce obvious feedback, the testers became confused. (These problems were especially evident when trying to create a bookmark or shortcut in the Nautilus file manager, but the program did not provide feedback, did not indicate whether or not the bookmark had been created.)