RSS feed Email Subscribe

Archive - Usability Testing


An email to a market research client of mine.


“To my mind, a usability report is the beginning of a discussion between the business, technology and research. This discussion is an attempt to balance cost time and brand experience. In this case, it is not a score card or ranking, as most people would like it to be. Usability is subjective and difficult to measure. The art is is about making the best of all limitations and giving customers mostly what they want.”


All to often in presentations and workshops I ask: “Who has done usability?”

What does it mean when people raise their hand?

- Yes! I have been a participant

- Yes! I have facilitated a test

- Yes! I have watched a testing session

    My guess is that executives in companies, who engage usability consultants, have never been a particpant and may not know what it is like?


    We try and test all our clients now, as part of the test planning process.  This gives them an appreciation for what we do and more importantly, they experience their software just like a test participant will!

    What we find is that the client’s feedback on our usability testing scripts is more detailed than normal and our clients are more engaged with the process!


    I had a realisation the other day! Much of the stuff that is (I have) written on the web about UX and Usability is a bit high brow for most of our clients!

    I’ve renamed our Objective Digital blog “Understanding Usability” to make it really clear what clients will get on the blog. I’ve also added a swag of new OD FAQs. These are the sorts of questions I get asked everyday in sales calls and project scoping meetings. For example,

    • “How many people should I recruit in usability testing?”
    • “How do you do recruitment?”
    • “What is a heuristic?
    • “How does an eye tracker work?”

    The blog posts are contextually linked on our site as FAQs. Here’s an example on the usability testing page:

    I’ve also summarised all the FAQs in a new FAQs section.

    What do you think?


    I just found the ‘environmentally friendly’ bamboo memory stick that I was given at OZCHI 08 in my drawer.

    I’ve never used it because:

    1) It doesn’t look like a memory stick

    But you can hang it round your neck!

    2) It doesn’t fit

    Phast USB - Please take all your other USB cables out. Oh and your video cable too!

    Phat USB - Please take all your other USB cables out. Oh and your video cable too!

    Unless you use one of these…

    Environmentally friendly indeed! Use an extra cable please!

    A use for that cable at last!

    I have another memory stick that was designed by a moron.

    Phat memory stick?

    But it looks cool...

    Both ‘oh, so creative’! Lucky they were free…


    Writing a short report is one of the hardest things in my industry.  People often seem to think that the bigger the wad of PDF the better the usability report! But your clients don’t read that stuff!

    I was invited in to a client’s office to present a report with Rebecca today. They wanted to hear it from the ‘horse’s mouth’ and ask us some more detailed questions.  I’m also sure that they wanted us to help them digest the info quickly and easily.

    The report was short and to the point and not repetitive. It simply laid down the usability facts from the testing.  This meant that Rebecca could present straight from the Word document without having to waste time on a pretty PowerPoint preso.


    Creating a pithy report requires you to be very organised before you get into the prose.

    Rebecca and I used a mind map to debate and structure all of the content of the report. Then Rebecca simply used the framework we created to ‘fill in the blanks’ and focus on the insights required to inform and inspire our client.

    She also chose the right report template up front.  That meant she wasn’t distracted from writing good content by the reporting ‘process’.

    It’s interesting, if you use mind maps, collaborate with others and choose the right template first, you will get a much more successful deliverable than if you just started writing straight like crazy after the testing.

    When you get prepared your attention can be placed on the content, voice, quality, findings and solutions that need to be communicated, instead of stressing about how to put the report together in a meaningful way.

    Our client loved the report today, we chatted for an hour and they headed off to implement the changes we recommended immediately.

    Other reporting writing articles:


    Lately I’ve been seeing briefs for Rich Internet Apps (RIAs) that require a certain number of ‘wireframes’?

    RIAs are characterised by multiple interactions on one screen without a full screen refresh. One screen has many different states - Drop downs, fly outs, sliders, changing grids, the list goes on. They are just like a desktop app.

    I get worried because most people think a wireframe refers to a Visio or PowerPoint diagram that is a flat file and has limited simulation or clear description of how interactions work on the screen.  This is not sufficient as you need to create many very similar wireframes to show how the screen works. It can be done, as Steve Collins suggests, ‘You create lots and lots of wireframes, like a traditional cartoon animation, to show how all the screen states can change!’ A waste of time. 

    The night before last, when I was at Interesting South, the problem with traditional wireframes was hammered home.  I was chatting to Twitter guru and coder, Brad Kellett (@bck) about how requirements get communicated to him. He basically said, ‘I usually get some crappy wireframes that leave too many options open. I am often unclear of how a client wants to put together.’

    ‘So how do you get an RIA right?’, I ask. ‘ Oh, it ends up in lengthy face-to-face interaction, while I try and work out what they need!’. He even said that he’d love a library of the hand gestures clients use to interpret how they think the RIA should be designed!

    What he really needs is something much more detailed than just wireframes and interaction. The ‘wireframe’ has to realistically represent all the interactions that occur on one screen of an RIA! He needs a high res model of the interface.

    Hi-res and Lo-res wireframes

    Thanks to Steve Collins for putting this together for me.  Firstly, Lo-res wireframes are the same as the wireframes that I see referred to in briefs, but with an RIA they are just not enough to communicate to a developer what to do with them. They have a valid use:

    • laying out core page functionality
    • representing major navigation items
    • validating major components and structure. 

    Lo-res wireframes are critical to developing an understanding of potential application and site framework as well as building an understanding of major application interactions prior to developing hi-res wireframes. 

    Hi-res wireframes build upon the lo-res wireframes, they are used to:

    • instantiate and test the understanding of application and site flow
    • show to clients and potential end-users exactly what they are getting
    • describe every screen interaction in detail to application developers and graphic designers to aid in building the application.

    Hi-res wireframes can also be used as the foundation of a functional prototype for testers, designers and developers to test assumptions against.  Testing against hi-res wireframes and prototypes can provide significant cost savings early in the project, militating against expensive and time-consuming changes late in the project lifecycle.

    Axure have solved this problem with a fantastic tool that can be used to produce hi-res wireframes and is not much harder to use than Visio. You can even use it to run usability testing on the out put and it can be printed into a very clear spec document too.


    I’ve seen many [usability] reports with an inordinate amount of repetition. A consultant might identify that users have trouble finding things, using the navigation of a business application, and then proceed to mention the issue in the:

    1. Exec summary
    2. Findings
    3. Task specific data and
    4. Heuristic (criteria) evaluation section. 

    What a waste of time and how boring for the reader (and the author)!

    Instead the report should be planned properly so that there is a maximum of two repeats:

    1. In the report, and
    2. Exec summary.

    Our reports use lots of pictures and tables most of the time they don’t even have an exec summary!

    In this post I describe how I use mind maps to organise my thoughts and arrange an efficient hierarchy of information. This is the best way to make sure you don’t repeat yourself. And you don’t leave anything out!

    So far in this report writing series, you’ll be on the right track if you:

    1. use some spin,
    2. design it properly and
    3. minimise repetitiion.

    Clients are busy people and they never read long, text heavy [usability] reports. Design can be a useful tool to buy more time from them and make it easier for them to draw meaningful Nuggets from your documents.

    Following on from my post about spin in report writing, here’s 12 report design tips to engage your time poor readers:

    1. ‘Design’ your report template so that it captures your readers’ attention immediately and increases the likelihood of them reading your report. 
    2. Use a designer to create the template. Don’t make the design up yourself, if that’s not your specialty, it will look silly.
    3. Your report markets your brand, forever - Ask yourself, ‘Does my client feel proud of my report when they show their colleagues?’
    4. Be consistent with your branding and design across documents so they can recognise/find them.
    5. Be consistent with your branding and design within a document - That’s common sense really!
    6. Use images that catch the eye  - we know a picture tells 1000s of words.
    7. Put call outs (speech bubbles) on the images - they can be quick easy to read (if you say useful stuff).
    8. Highlight the important messages with formatting and headings - so they are scannable and easily recognised.
    9. Use a standard layout that highlights the priority findings on all pages - that way a people can choose the level of detail to read.
    10. Use tables with images and icons - they may the report look shorter and allow easy comparison and scanning. However, you can fit lots of info in there if your smart!
    11. If you have a design recommendation, don’t describe it, draw it! If you can’t draw then find an example and paste a screen shot in the report.
    12. If you are using wireframes make them look ‘designed’ with some logos, curves, colour and shading. Plain black and white wireframes look amateurish!

    i reckon we need to bring back the design into usability! It’s no longer about making things work, that’s a given. It’s about making them cool too!


    When my staff or I are reporting findings from website usability testing there is always the challenge of knowing the right ‘tone’ of the document.

    In the last month we’ve heard two interesting comments on projects for two different Agencies:

    Agency 1 - “Can you be more BRUTAL?”

    Agency 2 - “Can you make it all a bit more POSITIVE sounding?”

    Both agencies had different needs:

    Agency 1 was pitching to a new account and wanted to show the client that things weren’t right with the current site and it required more work.

    Agency 2 built the site and wanted to look good so that the client could see all the opportunities to improve the site.

    As usability consultants, I believe we need to objectively and accurately report what happened.  However, it can be done in a number of ways to meet the particular agency’s expectations.  For example:

    Negative spin Positive spin
    No spin
    Issues with your site Your current practice Findings
    Interactive tools are hidden Interactive tools are secondary to other information Interactive tools are hard to find
    Participants did not understand the navigation links Navigation links can be easily changed so that participants can understand them Participants had very different perceptions of the words in the navigation

    Understanding the impact of your writing style is critical to your project’s success.  It can completely change how the report is received.  Some practitioners would say, ‘just report what you saw with no spin’; however, there is an opportunity to write the report in such a way as to achieve the best outcomes.
    A good way to identify the ‘tone’ required is to find out what the agency is doing with the report and then check that they like your language at least twice during the reporting process.  As follows:

    1) Conduct the test

    2) Find out what the Agency is going to use the report for and plan it with them

    4) Debrief the Agency at draft stage, and see if the language is right for their needs

    5) Make amendments to the tone of the report if necessary.


    I was just watching Demogirl’s screencast about user friendly websites below.

    What is so compelling screencast is that it is really short and about really obvious usability issues.

    Often I head off blogging about stuff that is just too complex and will appeal to a small percentage of net users who are in the usability industry.

    If I want to promote usability to the broader population I need to Keep It Simple… Stupid!