Validation Shell Game

Validation is the most misunderstood part of assessment shopping. I have taken or reviewed hundreds of assessments. I have yet to find one that was not validated.

Problem #1 – The Terminology

There is a misconception about the term validation. Some of the commonly understood definitions of validation are:

  • determining whether a statement is true or false
  • confirming that a product or service meets a set of specifications or quality standards
  • giving legal force to something
  • giving official sanction or approval

None of these match the psychometric meaning of the term. Yet for years I have seen diligent and careful people who are shopping for assessments accept an instrument’s validation as some kind of official endorsement or approval. Validation simply means that the instrument measures what it claims to measure. There is no direct or implied, official or otherwise, endorsement of the product’s quality or effectiveness.

Problem #2 – Throw the Book at Them

Once upon a time, there were many seminars on selecting assessment tools. One of the key recommendations at these seminars was to ask for the technical manual or the validation manual. This is supposed to be a description of how the assessment instrument was developed. The manual normally explains the construct of the scales, the choice of item formats, the development of the items, the demographics of the sample population, and the statistical analyses of the data resulting from the study of that population.

In a novel, this is where the plot thickens. Few people can really understand the details in a typical technical manual. The psychometric terminology is foreign. The statistics are analyzed with specialized software that expresses the results in esoteric language with numerical data puzzling to a layperson. This works in favor of some of the more questionable assessment companies. They present “validation manuals” filled with statistics and bold statements about the instrument, knowing that it will be accepted as their bona fides by most buyers.

Some standouts are:
  • An actual hardback book filled with questionable statistics purporting to support the quality of an equally questionable instrument
  • A manual in which the development of the instrument was attributed to “an employee of the Cincinnati Public School System”
  • An integrity instrument validated using a prison population

There are instruments of the highest quality that also have a hardback manual. There are other instruments of equally high quality that have a brief summary of their validation on a few stapled pages. A better way for a layperson to evaluate an assessment is to use some of the other information on this site to verify obvious things like the item format or whether it is a normative instrument.

Problem #3 – Valid Does Not Mean It Will Work for You

The message of this site is for you to be a careful buyer. Just because an assessment is valid does not mean that it will do what you expect it to do or what the company promises it will do. “Valid” is not a Good Housekeeping Seal of Approval.