How to Measure the Validity of a Psychometric Assessment
Are your psychometric tests valid? This is one of our most frequently asked questions. So, what exactly is validity and how do you measure it?
There are two fundamental requirements for any instrument used for assessment testing, reliability and validity. A reliable test will produce the same or similar results every time. Validity is an important part of a reliable psychometric test as it helps to determine whether or not the test actually measures what we believe it to be measuring. Validity is not measured by a single test or statistic but by a body of research. It is important to note that, just because a test has reliability it does not mean that it has validity. Extended DISC® Assessments are one of the most highly validated workplace assessments. Extended DISC® International completes a validation study bi-annually to ensure the results are reliable.
Reliability refers to the consistency of a test and ensures the test produces the same or similar results every time it is taken. One of the most common measures of reliability is test-retest reliability. In this method, the test is evaluated by administering an assessment twice at two different points in time. This type of reliability assumes that there will be no change in the behaviour being measured. Another form of reliability is Cronbach’s Alpha which also provides a measure of consistency. A general suggestion is that solid scientific instruments should a have Cronbach’s Alpha of at least 0.7.
The reliability of Extended DISC® Assessments is tested using a mixture of test-retest and Cronbach’s Alpha Methods. The assessments consistently achieve Cronbach’s Alpha of at least 0.78 across each DISC Style which indicates that test-retest correlation is high. If your client or company is asking for proof that an assessment is consistent or reliable, Cronbach’s Alpha is a great indication!
Important Note: Always seek out Psychometric Assessments that report on an individual’s natural and adjusted style. Some variance is expected in Psychometric testing due to different conditions i.e. current work environment and to take Behavioural Modification into account. Extended DISC® Assessments contain two profiles. Profile I reports on a person’s current or adjusted style, we expect this to change based on their current environment. Whereas Profile II reports on the person’s natural or unconscious style, we expect this to be more stable over a person’s life-time.
Validity is defined as an assessment’s ability to measure what it claims to measure. Validity of a Psychometric test heavily depends on the sample set of participants, including age, culture, language, gender etc., to ensure the results can be applied to different cultures and populations. The Extended DISC® Assessment 2017 Validation Study was one of the largest yet and included a population of 487,560 participants from 68 different languages and across several countries.
Validity of Psychometric Assessments is also commonly measured through the methods of Construct Validity and Criterion Validity. Extended DISC® Assessments are validated using both of these methods.
Construct Validity demonstrates that a test measures a construct rather than a skill or ability. Intelligence, motivation and anxiety are all examples of constructs. They exist in the brain and are not directly observable. For example, you can tell someone is anxious if they are trembling, sweaty and restless but you cannot directly observe anxiety. Construct Validity is usually measured by comparing one test to another test that measure similar qualities to observe a correlation. For example, a psychologist may have a client who is feeling fatigued, hopeless and has experienced a loss of appetite. These symptoms fit the description of depression. However, the psychologist cannot determine the severity of the anxiety without the Beck Depression Inventory (BDI), which measures the construct of depression. The Construct Validity of the BDI was measured by comparing the score of people who do not have depression with a group that does. The group that has depression scored higher on the BDI than the group without depression, so we can say the BDI has a high Construct Validity.
The Constructs measured in the Extended DISC® Assessment are the behavioural traits derived from the Jungian theory; Dominance, Inducement, Submission and Compliance. The process was conducted by comparing the expected responses to actual responses in all 68 languages. The consistency of these scores was very high in the results from the 2017 Extended DISC® Validation Study and is supported in other studies from previous years. Extended DISC® Assessments have one of the highest Construct Validity scores and continue to achieve these in the bi-annual Validation Studies.
Criterion Validity (Predictive Validity)
Criterion validity measures how well a test accurately predicts an outcome. For example, if a pre-employment test accurately predicts how well an employee will perform in the role, the test is said to have high criterion validity. Criterion Validity is usually measured using a couple of methods including Predictive Validity. The method of Predictive Validity is used to test the validity of the Extended DISC® Assessment Suite. All the DISC Styles were compared separately. The results exemplified an 83.7% percent match of correct scores between the two measures. The results show high predictive validity of the Extended DISC Assessment Suite.
Internal Report Validity of Extended DISC Assessments
Extended DISC® Assessments have an in-built reliability mechanism, where the online system will produce an Invalid Report if a clear answering pattern is not established. Some DISC tools never produce an invalid report. Therefore, a candidate can randomly select answer or try to ‘cheat’ the system and there will still be a result produced. One of the strengths of the Extended DISC® Assessment is that a result will not be generated if a DISC style cannot be clearly identified. Only 2.96% of results in New Zealand come back in Invalid and only 2.62% of results are returned Invalid in Australia.
There are a number of reasons why Extended DISC® assessments are occasionally classified as unreliable and therefore invalid. Fortunately Extended DISC® has the strictest internal rules for identifying results that do not carry the reliability required. Unlike other assessment tools, the system recognises inconsistent answers, classifies them as “Invalid” and does not produce a result. This ensures that the overall validity of the results remain reliable.
Assessment tools based on DISC theory are becoming more and more recognised as a valid system of analysing human behaviour. The Validation process is an important part of the development process of a Psychometric Assessment and many strategies have evolved for validation. Reliability and validity characteristics of the Extended DISC® Assessment System are analysed by using various statistical methods. According to the results of the Validation Study, the Extended DISC® Online System has both high validity and reliability.
For more information regarding the Validity of Extended DISC® Assessments please contact us.