Update 3: The Lost in the Numbers

The final weeks of my research have been spent going over a lot of statistics. Since I am an English major who has not really worked with hard data since coming to William & Mary, this is the stage that I was most concerned about going into the summer. Some of my fears turned out to be justified, as it was extremely intimidating to look at tables of SOL scores and try to draw usable conclusions from them. I ran into two main stumbling blocks during the process of breaking down that data: my own bias and how to compare the many charts I had to get usable data.

I was specifically looking at SOL scores for WJCC middle schools in the 2009-10 and 2010-11 school years to see how the redistricting that occurred in the spring of 2010 affected student outcomes. I then planned to use my conclusions from that process to predict how the new attendance zones would affect middle school students in the 2018-19 school year. The boundaries created during the 2010 process are the ones that WJCC had for its middle schools until the most recent redistricting, and they created schools that were less balanced when it came to percentage of students on free and reduced lunch than the 2018 zones. From my background research, I know that generally, economically disadvantaged students perform better in schools that are more economically integrated. That information made me more inclined to look for poor scores from the school that was less racially and economically diverse, and I had to put in some effort to be objective in looking at the numbers. Plus, the data is very finicky, as the changes from one year to the next are rather small and, in some cases, completely the opposite of what I expected based on the previous research I had done. I had to constantly remind myself that the numbers I was looking at came from the results of tests that real students, like the ones I had observed at the beginning of my research, had taken. Their individual experiences with teachers, testing, and their own abilities to learn were affecting the numbers I was looking at, and thus I was not always going to come away with clean conclusions.

There was also a wealth of data to look at for my project, as the Virginia Department of Education’s website gave me the ability to break down the SOL scores by school, grade, race, being economically advantaged/disadvantaged, and a few other factors. This made it difficult at first to discern which categories of results I should be comparing to get the more relevant conclusions to my project. I first chose to only study the seventh and eighth graders, since they were affected by redistricting, but beyond that there were still more choices to be made. The SOL results could be broken down by race, but race was not an overt factor in the school board’s recent deliberations, even though the comparison of subgroups had interesting results. I found that comparing advantaged and disadvantaged students was a more focused way to attack the data, with the racial comparisons serving as supplementary information that was more tied to the background research I did than to the school board deliberations I witnessed.

This struggle with data showed me that when assessing academic achievement, it is extremely difficult to come to solid conclusions based on something like test scores. Talking to some William & Mary professors has also shown me that there are many variables that could affect student achievement in addition to redistricting, which may jeopardize any conclusions I make using the SOL data. However, I still plan to include it in my final results, though I will rely on it less than I initially anticipated. One thing this research process has taught me is that when it comes to education studies, it’s easy to get lost in the numbers, but you have to remember where they come from!


  1. rckline says:

    This is such an awesome project, Rachel! It’s really obvious just how much thought and planning you’ve put in to every part of the project. It’s very important too, as this can and will have significant impacts on the educations and lives of thousands of kids and families. As someone going in to education, I’m very intrigued by observational and concrete differences between areas of social equality and inequality. Will you be combining both your observations and the hard data to form a final conclusion? Don’t let the data analysis get you down, I’m super excited to hear what you find out from this project!

  2. jcpsathas says:

    I too have struggled with my own bias when judging the information gathered in my research, even if my information is in a different medium (visual). For instance, I completely misinterpreted a bronze statue of an athlete as representing a confident, humble, and victorious individual before reading a very compelling argument by the former curator of the museum which houses the statue that argued the complete opposite. It’s very easy to be one-sided in your arguments, especially when you have a preemptive set of results you would like to obtain. I’m glad that you’ve analyzed your data for what it was, and considered the people behind the data. It’s exactly what I have to do in my project, that is analyzing vases and statues but considering perhaps what the artists and sculptors really had in mind based on what is actually given in the art. I appreciate you discussing these critical steps. Looking forward to seeing your final results.

  3. kpopham says:

    Very fascinating research! It seems the hiccups you are running into with data are challenging; but you also seem to be evaluating their relative significance very well and understanding how you will frame your research limitations in the final results. Incorporating data in my project, as well, has been a challenging component of my research, as it is at times very unpredictable what direction the numbers will lead you. It seems that the redistricting relationship you’re evaluating could be tied to other background identifications such as race and socio-economic status. Does redistricting, alone, change educational outcomes? Or does being redistricted from a low-income area to a marginally higher-income area influence scores as well? I understand SOLs are a difficult way to evaluate achievement; as a Virginia student, I remember feeling that way, as well. However, I think your tendency to remind yourself of the students behind the data is an important, humanistic way to view the research. It’s great that you are considering all of the environmental factors that are influencing student performance. Keep up the great work!

  4. Rick Stevenson says:

    Hey Rachel, I just finished reading through all your posts after coming across your abstract while browsing the blog. I was intrigued because I grew up in Williamsburg, going through wjcc schools, and I was switched middle schools between 7th and 8th grade because of the 2010 redistricting. I know by now your research is probably wrapped up for the most part, but I wanted to say how much everything you said and observed echoed my experience going through that system.

    The difference between School B and School A in your first update is representative of what I remember from each (I went from B to A). Everything about those places was night-and-day, from the way they were run to the energy of the students, all the way to the band uniforms. All the kids new it too, which through a kind of perceived hierarchy based on who went where. I remember kids talking about friends who got ‘stuck’ in School B after the redistricting. That’s how they saw it. My little brother just finished up his 8th grade year in wjcc and the mentality according to him is still the same. It’s always a seemed like disaster to me because of the way it enforces the socio-economic segregation you mention in your second update. Hopefully this next round of redistricting will help level the playing field, but right now there are zones that from 1st grade to graduation go through only the nicest, newest schools.

    Anyway, nice work on all of this. I’ll be really curious to see your results at the research symposium.

  5. Jo Weech says:

    Hi Rachel, your project sounds really thoroughly researched! I sympathize with the data struggles. As another humanities major, I fear the day when I’ll have to wade through statistics again. I think it’s impressive that you’re relying on numbers as well as personal observations. It’s clear how much work and planning you’ve put into your summer.

    How applicable do you think these redistricting struggles are to other counties in VA? Do you know if WJCC is particularly advanced in trying to better integrate its schools, or is this a movement happening in other districts across the state?

Speak Your Mind