Self-control, conscientiousness, grit and other non-cognitive skills are important factors in student achievement.
Educators want to do more to develop and measure non-cognitive skills. But a new report from the Brookings Institution warns that self-reporting measures have several potential pitfalls. Besides the social desirability bias (“faking”), which leads students to answer in ways that make them more attractive to others and to themselves, the reference bias may also skew results.
A reference bias occurs when survey responses are influenced by differing standards of comparison (e.g. I am a hard worker). One child might consider a hard worker to be someone who does all of her homework well before bedtime and in addition, organizes and reviews all of her notes from the day’s classes. Another child might consider a hard worker someone who attempts to complete assignments even if they are not finished.
Children might rate themselves differently based on their school environment. If children attend a “no excuses” charter school that emphasizes building non-cognitive skills, they may rate themselves higher or differently than children who attend an open-enrollment public school.
The researcher says one survey captured behavioral and academic outcomes such as absenteeism and suspension that were associated with self-reported non-cognitive skills. For example, students who rated themselves in the bottom quartile for self-control were absent 2.9 more days than students in the top quartile and nearly 3 times more likely to be suspended.
“Paradoxically, however, the positive relationships between these self-reported measures of noncognitive skills and growth in academic achievement dissipate when the measures are aggregated to the school level,” the researcher writes. “In other words, schools in which the average student reports higher levels of conscientiousness, self-control, and grit do not exhibit higher test-score gains than do other schools.”
Researcher Martin West concludes that existing survey-based measures of non-cognitive skills may be more useful for making comparisons among students within the same educational environment than for gauging the effectiveness of schools, teachers or interventions developing those skills.
“The Limitations of Self-Report Measures of Non-cognitive Skills,” by Martin R. West, December 18, 2014 Paper The Brown Center Chalkboard Number 93.