Written by Brad Winn, BCBA.
As behavior analysts, we rely upon the research published in peer-reviewed journals as the foundation for the programs we design for our clients. Our ethics code dictates this because the peer review process selects those which are valid, reliable, and accurate. But a reader may ask, what does it mean to ensure our methods meet a standard of validity, reliability, and accuracy?
When determining whether a strategy, IEP goal, or method of data collection is valid, we should ask ourselves, “What is our intent?” For example, when I train for my next marathon, I’ll set a goal to run faster than my last. Both races are the same distance and attaining my goal will be measured by the time it takes me to finish. If I achieve my goal (running a faster time), I can say my accomplishment is valid because I am using data (duration of time over the same distance) that is directly related to my goal.
The reliability of our data can be found in whether the results repeat themselves. The more repetitions, the more reliable the pattern. Often a behavior analyst selects a mastery criterion that requires repeated demonstration of mastery. This provides confidence that the outcome is reliable rather than a fluke. The reliability of baseline is critical to use as a contrast for the treatment effects, so we need a minimum of 3 data points to demonstrate it. The need for reliability also means it is generally best to wait until a baseline is stable before implementing. Why? Because a stable baseline is a reliable baseline: You’re seeing similar results repeating themselves. When an intervention yields a change, it will be more detectable. However, we need to be careful that we do not simply have reliable data which is not accurate
Accuracy is incredibly important when assessing your data collection, goals, or strategies. It’s also the trickiest to understand. Another good example relates to running. When I run marathons, they’re certified by strict standards to be 26.2 miles. This is considered the true value (they actually measured the distance from the starting line to the finish line with tape and a Jones counter). However, when I run the course, my GPS watch never shows 26.2 miles; it’s always slightly more or less. This is considered the observed value. Because the course is certified and considered the true value, I can safely say every time that my watch is not 100% accurate.
This graphic does a great job of simplifying complicated definitions and providing synonyms for each term. What are you intending to measure? Was the measurement repeated? What’s the true value you’re comparing it to? These three things will help you be a better practitioner and with your mastery of them, allow you to pass the exam!
If you need more information on this topic, applying this concept or identifying in scenario questions, check out our Beyond the Basics Workshops!
Pin