communication for the real world

Measuring quality — The talk

This is the talk I gave with the webinar I delivered on Tuesday 22 March 2011, Measuring the Quality of your Documentation.

If you would like to have the accompanying slides and quality-measurement tool, download them here.

Factory man being efficient

Getting things done

Wherever you work, whether it’s in a calmly organized editorial group, alone facing down deadlines in a home office, or running from project to project like a blue-assed fly, you would like to know how you stack up, how you are doing.

But you don’t have time to stop and look around. You have deliverables. You need to have things move along so that you get things done. The result of getting things done, of course, is product. And product can be measured.

So, while you are getting things done, how can you measure the quality of what you do? How can you measure the progress you might be making to improve your quality?

Peer review

Annually, for years stretching back into the mid-1990s, I have taken part in peer review exercises called technical publications competitions. First it was curiosity that drove me to get involved in my local STC chapter competition. I wanted to learn, from people who considered their work excellent, what excellent looked like. What do other technical communicators do to get their message across? Are there particularly good ways of doing things?

Of course, not all the entries in a competition are as excellent as the entrant might believe them to be. For my part, and it was really painful at the time, I once entered a tech pubs competition and I didn’t win anything. But I got feedback and my work improved. My idea of excellent improved. I jumped back into work with renewed interest in doing my job better, renewed interest in quality.

Gradually, through my peer review work, I got to witness and respond to many different approaches. And my curiosity didn’t go away. I wanted to go deeper. What are better ways of getting information across? What does the reader need and — equally important —what is distracting? How can this work be done really well? How can this work be done better?

Making a tool

The peer-review forms we had to work with were vague and open ended. They are better now, but still I have never seen anything like what I envisage in my head, which closes the argument by creating a weight or optimal rating for each necessary element in the technical communication being reviewed. When you start to consider necessary elements you can look for concrete things, gauge how much they are needed, and look at how well that need is met.

Full disclosure: Someone I showed my tool to told me that JoAnn Hackos might have a similar sort of tool. To be honest, I was a bit bummed out by that. I thought my idea was such an original! I have not seen the JoAnn Hackos tool so feel free to look it up and let me know about it.

What’s in it for you?

Here’s what you can do with improved quality

  • Improve the user experience of your technical communication product and of the products you write for
  • Be localisation ready
  • Develop your skills
  • Build your team and develop their skills
  • Build knowledge and confidence

Here’s what you can do with the information you generate from using the tool

  • Self-audit
  • Build useful metrics
  • Have the power to address your company’s bottom line
  • Build knowledge and confidence

Here are some strong spin-off benefits of using the tool

  • Manage corporate pressure
  • Compete and improve your profile within your company
  • Manage expectations
  • Build knowledge and confidence

It sounds like a lot to expect from a simple tool. But I use it for all my projects and it really works.

What do you need to measure?

I created categories to break down and organize a technical document. Then I arranged the categories to cascade from the structural and superficial, which are relatively quick to process and evaluate, to the deeper material, which takes longer to process and needs more time to think about.

I came up with this:

  • Document structure
  • Reference and navigation
  • Graphics and other visual elements
  • Accuracy and grammar
  • Terminology
  • Consistency
  • Clarity
  • Task orientation
  • Completeness

We’ll use these categories today. But keep in mind that the beauty of this tool is that you can change your categories to suit your product.

Introducing the tool

Starting from the top, you can see that

  • The categories are laid out in the order I have described, from structural to more in depth as you get further down
  • Depending on what your document requires, you set a weight for each element in each category, with 1 as low and 5 as high — You might want higher or lower rates than I have used here, I change them for different types of documents
  • You can add elements to or remove elements from the categories — If you do that, be sure to check the Excel formulae, which are, by the way, very simple

When you are ready, take a technical communication or any piece of writing that you want to review and go through it, entering rates for each element.

Using your category score

Your category score is useful for finding out how things are going category by category. I have used the category scores to

  • Show a learning curve for individuals and the team
  • Give individuals solid feedback about what to do to improve their work
  • Provide a granular induction to new hires to help them understand the level of work that they can aspire to
  • Argue for initiatives such as different media for information delivery
  • Show exactly where and how the documentation is improving and what that means for internal and external clients and company revenue

Using your overall score

Your overall score is useful as a broader metric. I have used the overall scores to

  • Create a documentation team history
  • Create a project history — This is particularly useful over several editions of a document set
  • Show why I need more time for a project
  • Discuss professional development with my team and with my boss
  • Show documentation efficiency — Using this form is persuasive
  • Negotiate perks for my team
  • Show my worth to the company
  • Show in general where and how the documentation is improving overall and what that means for internal and external clients and company revenue

Now I can open the floor and un-mute the mics for a question and answer period.

Questions and answers

Some of these questions were developed by me, some were asked by webinar participants. All of the answers were developed by me.

Q: How long does it take for the tool to show its worth?
A: My experience is that it takes about six months for the tool to show useful results. To begin with, I apply it to any legacy documentation. By doing that I find that I can accumulate useful results more quickly — and I can show what is perhaps the most persuasive data: What difference you made when you started to work on the legacy documents.

Q: How can you use the tool to train a new hire or a junior member of the team?
A: Keeping your results from applying the tool to legacy documentation separate, ask your new hire or junior team member to apply the tool to legacy documents. Then go through the results and, working with the new hire or junior team member, compare them to your earlier results. In addition to doing the exercise, the form itself is a training tool. Anyone reading through it knows what to pay attention to and what to make an effort on.

Q: How do you set weights?
A: If you have access to users of your company’s products, use this opportunity to ask them specific questions about the documents they use, or ask your editor. As much as possible, set weights in line with the business value of the document.

Q: If you change a weight, do you have to re-do all the Excel formulae?
A: No, it’s set up so that you can change things easily. You have to check if the formulae still include all the rows if you add a row. To do that, you select the formula and click the function field. The rows that are included in the formula show up and you can see if anything needs to be added. Other than that, the tool works smoothly if you change rates, weights, or remove rows.

Q: What if you are collecting data on a document set and you’ve changed the rates one or more times during the process of collecting the data?
A: Try not to do that. But, I have done it in the past. To make the metric useful, I found the places I had changed, which were few, and took an average. The metric is still useful and you can still see where you are improving as well as the places you might need to pay more attention.

Q: How do you handle grade inflation over time?
A: As much as possible within your team or organization, make sure that the reviewer is a disinterested third party. An editor can be a good choice. If you notice grade inflation and you don’t have a way to stop it, you can discuss it with your team, and in a dramatic case, you can adjust the results.

Q: Can you use the tool to show the handicap imposed by extenuating circumstances that might influence the quality of your document?
A: Yes, you can create a category with the elements that historically influence document quality (lack of time, difficult access to the subject matter expert, review cycle blockage, management interference in grammar, etc.) and show that category score as handicap data that you can apply to the document. In addition, you can collect the handicap data over time to show handicap trends and how they influence quality.

Q: What do you do if your document is to be translated?
A: The tool addresses translation and localisation here:

  • Graphics are not to have embedded text because you then cannot translate the text
  • Any measurements in the document must be in either metric, imperial, or both
  • Agreed terminology must be used and used consistently

Of course, if you find categories that need more elements specific to translation and localisation, you can add them.

Q: Translation and localisation provide useful metrics on re-use of material. Can you collect that data with this tool?
A: Yes, you can add elements or even a whole category to address re-use of material. Try to align what you do in the tool with your translation and localisation team.

Thanks to all the participants, the organizers, our corporate sponsor and everyone who gave me advice as I developed the tool!


There are no comments yet...

Leave a Comment