EVAAS: Let's TOK About It

Note: This is a Blog Post written by Molly Crabb, Chris Bakolia, and myself. If you'd like to learn about EVAAS and math and things, you've come to the right (blog)spot! If, however, you're searching for something on Macbeth, it seems you've lost a few letters. Reinsert your "cbe" and go back to the last post. As always: Thanks for reading!

"Okay, okay, okay, so tell me about this EVAAS thing."


SAS EVAAS provides valuable diagnostic information about past practices and reports on students’ predicted success probabilities at numerous academic milestones from grades K-12. Effectively implemented, EVAAS allows educators to recognize progress and growth over time. It measures student progress objectively and accurately and allows for teachers to improve instruction. Teachers are able to understand the incoming academic preparedness of students before they enter the classroom and monitor student progress (from low achieving to high achieving) ensuring growth opportunities for all students. EVAAS helps teachers and principals improve student learning and provide an exceptional education to every child, thereby providing an equitable learning opportunity to all students.

 "I don't get it. Where's the math?"

EVAAS uses Value Added Modeling (VAM) as a means of measurement. VAM is a method of teacher evaluation that measures the teacher's contribution in a given year by comparing the current test scores of their students to the scores of those same students in previous school years, as well as to the scores of other students in the same grade.  Two relatively recent additions to EVAAS’ VAM use nonparametric analyses that make fewer assumptions. These two models are the standardized gain model (SGM) and the student growth percentile model (SGPM). Both models are regression models that calculate value-added by “averaging” regression residuals.  Of course with any regression model, there is going to be measurement error in the predictor variable(s). The result is that school and/or teacher effectiveness, as estimated by these models, tends to be correlated with school/classroom composition, with high-poverty/high-minority schools/classrooms being more likely to be evaluated as ineffective.

"Wait a second...so the math is ineffective? Math can't be ineffective!"

That's where you're wrong, dear reader.

A status of perfection is nearly impossible to gain in the field of mathematics. Even if a model, system, or equation is considered flawless, it is always assumed that natural human error will bleed into, and negatively affect experimental results. That’s the reason mathematicians and scientists include percent errors in their calculations. What are some possible flaws in the EVAAS value-added model, and how are they accounted for so as not to negatively influence ratings of teacher or school effectiveness?

One possible flaw of the EVAAS model, as Chris touched on above, is that it does not account for socio-economic factors which may affect students’ performance on test scores. Value added models which account for factors such as race and economic status do exist, but research has found that a thorough student test history diminishes the need for the inclusion of demographic data. Simply put: demographic data is unnecessary and, when utilized, does not affect predictions. However, it can have an effect on teachers, legislators, parents, and students subjected to value added modeling, who may be wary of utilizing such systems to measure effectiveness. Including demographic data can often increase the public’s confidence and comfort with value added modeling.

 

Oh yeah. They look confident in the education system alright.

Let’s go back to the issue of mindset for a moment. We often see math and science as wholly logical fields which produce factual conclusions based on observations. We see the results which stem from math and science as truth, unclouded by human emotion and opinion. However, this mindset is misleading; systems, like humans, can be biased, and EVAAS is no exception. A sort of “system bias” exists in the way EVAAS handles missing data.

A good predictor model includes LOTS of predictor variables. While this sounds like a great plan in theory, in practice it often falls short. The truth of the matter is, the more predictor variables you include, the more likely it is that students will miss one or more of the predictor tests. In order to be included in the EVAAS model of teacher effectiveness, a student must have data on all the predictors being utilized. Students with lower attendance rates (typically lower achieving students) will tend to miss more tests, and will therefore be excluded from data more often than their higher achieving peers. 

Betcha that kid has got all his predictor variables.

The system creators claim to account for this. They say that EVAAS assumes some scores are missing at random (meaning the available scores are a fair representation of those missing as well). This is not the case, however, as it has been discovered that lower distribution scores are missing from data more often than scores higher on the distribution.

A similar issue is that students who move into the district in later years cannot be included, as they may be missing predictor variables from elementary or middle school tests. This occurs in much smaller numbers and has less effect than the previous example.

Both of these examples contribute to errors in the evaluation of teacher effectiveness as the number of students per teacher is necessary for the evaluation. The loss of even a handful of students can have a big impact on the standard error of an estimate, particularly in elementary school settings where teachers typically teach only one class.

The real flaw with the EVAAS model lies not in the model itself, but in the education surrounding it. One of the reasons the state claims to use EVAAS is because it believes that the models student score predictions can help teachers identify students in danger of failing or performing below average and prepare interventions for them. SAS, the company that hosts the EVAAS model has web resources for teachers, which explain the EVAAS model and how it can be used for this purpose. How many teachers actually utilize these resources? Of those that do, how many of them truly understand EVAAS and the data it produces? SAS does have a good customer service system in place, but that method of education can only go so far. The state lacks the capacity to provide training at the teacher level; so, instead, they train district leadership and assume that districts communicate with principles who then communicate with teachers. This sort of “trickle down” education is a poorly designed system. How can teachers be expected to utilize data to create interventions and assist their students if they don’t understand the information? 

"How do people feel about EVAAS?"

The EVAAS method of assessing student and teacher performance is viewed by its creators and supporters as a useful tool for wide spread understanding of success in education. It uses as many test scores as possible to make an estimate of a student's future scores. The model holds every student to the same standard of improvement by not taking into account socio-economic variables. Creators claim that even if a student is affected by their social or economic situations, it would show in their previous scores as well, which would keep the system fair. The model is not the sole basis on which assessment is made but it is an additional tool to help schools assess their teachers. Schools have not yet implemented consequences based only on the EVAAS assessment.

The model is a completely quantitative tool that uses numbers from a standardized test to define a student's success. This is a problem for EVAAS critics. Many teachers believe that assessment of students and classes should be completely based on observation of teachers. It is impossible to evaluate how a teacher is doing based on scores. If a teacher has students who are unwilling to learn or are not motivated to do well (lurking variables) the results are still blamed on teacher performance. Differences in summer learning for each student are also not taken into account. In some cases, the curriculum the teacher is supposed to teach does not cover all that is on the standardized test, greatly impacting their performance rating.

"Well how do you feel about EVAAS?"

We believe that EVAAS is a good model for predicting student scores and measuring teacher and school effectiveness. However, there are issues that need to be resolved before it can be considered a great model! State legislators and EVAAS overseers should become more involved at the school level, educating teachers and principals about EVAAS and how they can use it to be more successful. The state should also implement uniform policies that dictate how they will deal with teachers and schools that are not meeting performance expectations. This will create standards for all schools and teachers to strive for, as well as regulating intervention and discipline, which will hopefully help all of North Carolina’s schools provide equal educations to their students. 

 

After reading our EVAAS overview, learning of the possible flaws in the EVAAS model, and hearing both the negative and positive perspectives on value added modeling, we hope you understand enough about value added modeling, and EVAAS, to form your own opinion.