3 on BI Basics

School CIO published three articles recently on D3M and school analytics. Bravo!

First, Dr. Lane Mills offers us a bit of BI Basics at School CIO; it’s worth a read but remember it’s only an introduction. For example, Dr Mills introduces the subject of dashboards and performance indicators, but spends precious little time discussing them. Of course, the article is entitled “Basics.” If you’re curious to learn more about education performance indicators, After3 suggests a short hop over to NCES (National Center for Education Statistics) for an another overview, this time of education indicators.

Second, Keith Waters explains his experience with “Education Performance Management” in the St. Louis metro area, a concept in which educators use data and BI tools in the service of continuous school improvement and student learning (After3’s paraphrase). The concept includes several key disciplines: knowledge leverage, instructional differentiation, directional alignment, practice integration, situation analytics, and improvement diligence. Pique your interest

And third, Rick Whiting teases us with a case study from Lafourche Parish, La., where a middle school supervisor focused on the disciplinary reports filed last year by teachers and administrators. That supervisor was interested in finding insight into the root causes of behaviors like tardiness, fighting, vandalism, and more. The school system used SPSS’s data mining and test analysis software for the work. The School CIO article includes neither more detail about those root causes nor a link to the case study it cites. The original InformationWeek article doesn’t provide a link, either. After3 sincerely thanks Mr. Whiting for an intriguing brief on what Lafourche Parish is doing, but with equal sincerity wishes for more detail if we’re to reach “info-satiation.” It’s all a bit of a wash. We’ll run a search and see if we can come up with the link on our own.

Update

Well, SPSS has a white paper on Lafourche Parish, but it’s about student learning rather than the discipline study that Mr. Whiting’s article mentions.

K12 data domains

Ah yes, let’s talk data. Your data. The data. The data is where everything about your data analysis system sinks or swims. So it pays to have some idea of what kind of data to collect.

After3 acknowledges that our focus is continuous school improvement, rather than Accountability. Notice we didn’t use the formalized process, that is, “Continuous School Improvement,” while we did capitalize Accountability. By continuous school improvement, we simply mean some “well defined, repeatable process — actuated by human agency — by which an organization incorporates data-use into its daily culture and decision-making for the purposes of policy evaluation and instructional change.” In contrast, we understand Accountability as federal or state mandates which require districts and schools to achieve outcomes typically determined by some legislative or executive body.

We won’t bother to address the differences beyond these statements, for the time being. That’s not really the purpose of this post.

So. Given that we’re considering data within the framework of continuous school improvement (for the sake of brevity hereafter, CSI), what kind of data might you consider collecting?

Student data and test data are two obvious domains. Is there anything beyond this? If you’re implementing CSI, there is. The acknowledged thought-leader in the arena is a lady who has published books about K12 data and CSI since 1997: Dr. Victoria L. Bernhardt.

Take a gander at this illustration. In an invited monograph for the California Association for Supervision and Curriculum Development, “Multiple Measures,” Dr. Bernhardt suggests that you gather data within four “data domains.” Those domains include

• Demographics (enrollment, attendance, ethnicity, gender and the like),
• Student Learning (standardized tests, criterion-referenced tests, benchmarks, teacher observations and so on),
• School processes (school programs and ‘ways of doing business’), and finally
• Perceptions (survey results about how students, staff and parents feel about the learning environment, its mission, and the community values).

Study the illustration (you can find the entire article here). It’s not the only way to categorize the data you’ll need, but it is effective. And Dr. Bernhardt has certainly given a great deal of thought to what questions can be answered within each domain. That’s probably enough to consider for now.

Accountability vs Instruction

Dr. Scott McLeod, associate professor at the Department of Educational Leadership and Policy Studies at Iowa State University, rightly points out that NCLB’s accountability requirements and the goals of D3M are not synonymous. You can read his post here. His thoughts (cross-posted at eduwonkette) elicited this response from the eduwonkette herself.

Dr. McLeod is spot on when he lays out the problem of using your annual state test as a means of improving instruction. By now an old theme, such analyses are little better than what has elsewhere been called a “post-mortem.” That is to say, the analysis comes too late to be of any use. The students have moved on (we hope). Formative testing serves instructional staff as a means of measuring student progress far better than the yearly test. McLeod succinctly makes this point.

In our experience, however, school districts across the nation have yet to adopt some form of local formative testing en masse the way we would hope, despite the presence of fairly reliable test instruments like NWEA’s Measures of Academic Progress, Scantron’s Achievement Series or Performance Series, or any number of other instruments. Districts such as Iredell-Statesville (NC) have even created their own predictive formative tests, while others like Mesa County Valley (CO) have adopted MAP. Get thee a predictive benchmark, we say.

eduwonkette gently chides Dr. McLeod over this issue of testing for instruction, suggesting that there is a “dark side” to D3M. “Some schools are using benchmark tests and other newly available data to play the system and up their numbers,” she points out. eduwonkette is absolutely correct. “The practice of focusing on kids who are close to passing has been well-documented by now,” eduwonkette reminds us.

And indeed, we have been in schools in Texas, North Carolina and South Carolina where this practice is the accepted norm. Assigned to raise the achievement levels of an under-performing school, a Texas principal bluntly insisted that while he knew his school must serve to the learning needs of all students, if he knew precisely which 21 students scored just above and just below “the bubble” of acceptable performance, he would focus his teachers on intensive remediation for those 21 kids. If he could raise the performance of those 21 children and it rescued his school from receiving an “unacceptable” rating, he wouldn’t hesitate.

After3 wonders if this truly illustrates a “dark side” of using testing for instruction. If accountability were removed from the equation of education policy, would we not still use assessment as our primary means of identifying specific under-performing students as a way to target them for some form of intervention? It all seems quite pragmatic to us.

But we do issue one caveat: we expect the test results to be used to teach to the standards and for targeted intervention, not to teach simply to the test. To that much we can all agree.

In our experience, a far more fundamental problem plagues school districts: reliable data. Your data analysis means little if your data are not reliable. But more on data reliability some other time.