What Does “Big Data” Mean for School Improvement?


Latest Story

Here are some additional charts based on Georgia’s Criterion Referenced Competency Tests (CRCT) for the period 2008 – 2014, and some specific data for 2014.

Using “big data” to analyze how students and teachers are doing doesn’t always tell the real story of what happens in classrooms.  Over reliance on “big data” can lead to unintended consequences, as it did in the cheating scandals in Atlanta and Washington, D.C.  “Big data” can also land in policy makers hands that don’t have the interests of teachers and students in mind, nor how education can be improved.   Perhaps the most contentious use of “big data” in education the use of Value-Added Models (VAM), that is, how much does a teacher “add” to students’ performance on achievement tests.

As you look at the data that I’ve provided in this post, keep in mind that all these numbers are based on Georgia’s CRCT achievement tests that are administered for about two weeks in the spring of each year.

The questions on the CRCT are multiple choice, and each test includes a sampling of what the students were expected to learn in each core subject: reading, English language arts, math, science, and social studies.

In most of the press releases use percentage graphs comparing last years’ results to 2014.  The press also loves to rank schools, so you will find lists upon list of schools ranked by scores on the CRCT.  You’ll find the top ten, and of course, the bottom ten.  But is this a helpful use of test scores?  How does publishing these lists improve the learning of the students that took the tests, or the improve their teacher’s ability to teach.

It doesn’t.

I’ve included graphs that show how students scored on the CRCT in the 8th grade for 2008 – 2014.  You can see that the trend in average is scores has been upward, and that the range for the scores has diminished since 2008.

Georgia: All Subjects, 2008 – 2014

Figure 1.  Average Score of Students on All Core Subjects, 2008 - 2014

Figure 1. Average Percentage of Students At or Above Proficient on CRCT on All Core Subjects, 2008 – 2014

 

Figure  Average Scale Scores for All Subjects, 8th Grade, 2008 - 2014

Figure 2. Average Scale Scores for All Subjects, 8th Grade, 2008 – 2014

Figure  Range for Average Scores for All Subjects, 8th Grade, 2008 - 2014

Figure 3. Range for Average Scores for All Subjects, 8th Grade, 2008 – 2014

Figure

Figure 4. Line Graph of Average Scores from 2008 – 2014 in the five core subject areas, and predicted linear growth lines.

All Subjects As a Whole in 2014

Another way to look at the CRCT scores is using a technique called a process chart.  Figures 5 and 6 are process charts of all subjects, and math & science, respectively for 8th grade in 20014.  This is a systems view of 8th grade performance in all the subject areas, and math & science specifically.

Figure 5 is a process chart for the average score computed from average scores in 8th grade reading, English language arts, math, science, and social studies.  Note an Upper Control Limit, and a Lower Control Limit.  Average scores that fall within this zone are normal, and are due to non-special causes.  However, if we find scores that outside these lines, we might conclude that some special cause is operating to make the difference so large.

For most districts, their average scores fall within the UCL and LCL.  But note there are 13 districts that scored above the UCL, and 11 districts below the LCL.  It would be worth doing a study for find out what is going on in these districts.  What are the special causes creating these wide variations?

Figure 1.

Figure 5. Georgia 2014 8th Grade Average CRCT Score All Core Subjects: Reading, English Language Arts, Math, Science, Social Studies

 Math and Science 2014

Figure 6 shows Georgia’s system for teaching 8th grade math and science.  Most of Georgia’s districts fall within expected limits.  But there are a few that score above the UCL, and it would be valuable to find out if there are special causes related to these variations.  We should also look at the those districts that scored below the LCL, and ask, what can be done to help improve the situation here.

Figure 1

Figure 6. Georgia 8th Grade 2014 Average Score of Math and Science

Well, what do you think? Is big data helpful in sorting out differences?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s