How to Measure the Validity of a Psychometric Assessment

Are your psychometric tests valid?

One of our most frequently asked questions is about the validity of our assessments. So, what exactly is psychometric validity, and how do you measure it?

There are two fundamental requirements for any workplace assessment, reliability and validity. A reliable test will produce a similar result every time. The validity, an essential part of a reliable psychometric test, helps to determine whether or not the test measures what we believe it to be measuring. Validity is not regulated by a single test or statistic but by a body of research. It is important to note that, just because a test has reliability, it does not mean that it has validity. Extended DISC® Assessments are one of the most highly validated and reliable workplace assessments. Extended DISC® International completes a validation study bi-annually to ensure the results are reliable.

Reliability

Reliability refers to the consistency of a test. It ensures the test produces the same or similar results every time it is taken. One of the most common measures of reliability is test-retest reliability. In this method, the test is evaluated by administering an assessment twice at two different points in time. This type of reliability assumes that there will be no change in the behaviour being measured. Another form of reliability is Cronbach's Alpha which also provides a measure of consistency. A general suggestion is that robust scientific instruments should have a Cronbach's Alpha of at least 0.7.

The reliability of Extended DISC® Assessments is tested using a mixture of test-retest and Cronbach's Alpha Methods. The assessments consistently achieve Cronbach's Alpha of at least 0.79 across each DISC Style (D=0.84, I=0.82, S=0.85 and C=0.79) which indicates that test-retest correlation is high. If your client or company is asking for proof that an assessment is consistent or reliable, Cronbach's Alpha is a great indication!

Important Note: Always seek out Psychometric Assessments that report on an individual's natural and adjusted style. Some variance is expected in Psychometric testing due to different conditions, i.e. current work environment and to take Behavioural Modification into account. Extended DISC® Assessments contain two profiles. Profile I reports on a person's current or adjusted style. We expect this to change based on their current environment. Whereas Profile II identifies the person's natural or unconscious Extended DISC style, we expect this to be more stable over a person's lifetime.

Validity 

Validity is defined as an assessment's ability to measure what it claims to measure. The validity of a Psychometric test depends heavily on the sample set of participants (including age, culture, language and gender) to ensure the results apply to a vast range of cultures and populations. The Extended DISC® Assessment 2019 Validation Study was one of the largest yet. It included a population of 886,943 participants from 77 different languages and across several countries.

The validity of Psychometric Assessments is also commonly measured through the methods of Construct Validity and Criterion Validity. Extended DISC® Assessments are validated using both of these methods.

Construct Validity

Construct validity demonstrates that a test measures a construct rather than a skill or ability. Intelligence, motivation and anxiety are all examples of constructs. They exist in the brain and are not directly observable. For example, you can tell someone is anxious if they are trembling, sweaty and restless, but you cannot directly observe anxiety. Construct validity is usually measured by comparing one test to another test that measures similar qualities to find a correlation. For example, a psychologist may have a client who is feeling fatigued, hopeless and has experienced a loss of appetite. These symptoms fit the description of depression. However, the psychologist cannot determine the severity of the anxiety without the Beck Depression Inventory (BDI), which measures the construct of depression. The Construct Validity of the BDI was measured by comparing the score of people who do not have depression with a group that does. The group that has depression scored higher on the BDI than the group without depression, so we can say the BDI has a high Construct Validity.

The Constructs measured in the Extended DISC® Assessment are the behavioural traits derived from the Jungian theory; Dominance, Inducement, Submission and Compliance. The process was conducted by comparing the expected responses to actual responses in all 77 languages. The consistency of these scores was very high in the results from the 2019 Extended DISC® Validation Study and is supported in other studies from previous years. Extended DISC® Assessments have one of the highest Construct Validity scores and continue to achieve these in the bi-annual Validation Studies.

Criterion Validity (Predictive Validity)

Criterion or predictive validity measures how well a test accurately predicts an outcome. For example, if a pre-employment test accurately predicts how well an employee will perform in the role, the test is said to have high criterion validity. Extended DISC® International conducts a Predictive Validity study on a bi-annual basis. In this study, candidates predict their behavioural style and then take the Extended DISC® Online Questionnaire. According to these results, the proportion of 'correct' answers between the individuals own evaluations, and the results of the Extended DISC tool is 83.7 %. The results show high predictive validity of the Extended DISC Tool.

Internal Report Validity of Extended DISC Assessments

Extended DISC® Assessments have an in-built reliability mechanism, where the online system will produce an Invalid Report if a clear answering pattern is not established. Some DISC tools never provide an invalid report. Therefore, a candidate can randomly select an answer or try to 'cheat' the system, and there will still be a result produced. One of the strengths of the Extended DISC® Assessment is that a result will not be generated if a DISC style cannot be clearly identified. Only 3.01% of all results are returned Invalid in New Zealand. Worldwide the overall figure is 4.13%. 

There are a number of reasons why Extended DISC® assessments are occasionally classified as unreliable and therefore invalid. Fortunately, Extended DISC® has the strictest internal rules for identifying results that do not carry the reliability required. Unlike other assessment tools, the system recognises inconsistent answers, classifies them as "Invalid" and does not produce a result. This ensures that the overall validity of the results remains reliable.  

Assessment tools based on DISC theory are becoming more and more recognised as a valid system of analysing human behaviour. The Validation process is an integral part of the development process of a Psychometric Assessment, and many strategies have evolved for validation. Reliability and validity characteristics of the Extended DISC® Assessment System are analysed by using various statistical methods. According to the results of the Validation Study, the Extended DISC® Online System has both high validity and reliability.

 

Population Statistics and Interesting Research Findings

1. The global DISC distribution