NCEA Methodology
for Identifying Higher-Performing Schools
General Approach
NCEA’s methodology analyzes each school’s performance with students in each tested grade, subject, and prior achievement group (for example, students in eighth-grade mathematics in 2005 who were below passing in seventh-grade mathematics the previous year). Each group’s performance is analyzed using a regression approach to identify how far “above predicted” or “below predicted” the group performed in a given grade and subject (the group’s regression residual) given the group’s average achievement score in the prior grade and the overall demographics of the school.
For elementary schools, prior achievement scores were only available for grades four and higher. Rather than creating a data analysis that would only apply to one-third of the grades in a typical K–5 elementary school, NCEA looked at scores of students from third grade and higher who had been continuously enrolled in the school for three years or more.[1] Prior scores were not used for the elementary school analysis, as that measure was not available for third-graders.
In all cases, three years of data from 2004 to 2006 for each school were used to identify consistently higher-performing schools.
The steps in the analysis were as follows:
Steps for Middle and High Schools
Step 1: Merge each student’s current year test
results with those of the same student from the previous year. Current year
math is merged with prior year math, reading with prior year reading, writing and
social studies with prior year reading, and science with prior year math. (Grade
11 science is matched with grade 10 science, and grade 11 social studies with
grade 10 social studies.)
Current year fall enrollment records were also used to identify whether the student was enrolled at the same school during the current school year. Students without prior year scores in the relevant subject, or who were not enrolled in the same school in the fall of the current year, were not included in the analysis.
Step 2: Divide students into groups based on their prior year performance in the subject in question. For example, students tested in mathematics in the current year in ABC Middle School, a school containing grades 6–8, were divided between those who passed TAKS mathematics in the previous year and those who failed the exam. Calculate the average current and prior-year test scores for each of these student groups. For example, separate averages would be calculated for the eighth-graders who passed and failed seventh-grade mathematics.
Step 3: For each grade, subject, year, and prior performance group, regress average scale scores on prior year average scale scores, the schoolwide percentages of low income, African American, and Asian students, school enrollment, a flag for whether the school was/was not considered a magnet school, the percentage of students that are in the prior performance group being analyzed, and the overall percentage of students tested in the subject in the grade in question. In addition, the percentage of English Language Learner (ELL) students was used in the middle school analysis. Weighted least squares regression was used with the weights based on the number of continuously enrolled tested students in each grade, subject, year, and student group. The standardized residuals for this analysis were saved and used as the performance measure for the grade, subject, year, and student group in question.
Under this system, nine separate regression residuals in mathematics would be calculated for the previously below-passing student group in a middle school containing grades 6 though 8: residuals for sixth-, seventh-, and eighth-grade mathematics in each of 2004, 2005, and 2006. Another nine residuals would be calculated for the previously passing and above student group, assuming there were at least ten students in that group in each grade and year. Six residuals would be calculated for each prior achievement group in a grades 7–8 middle school.
Step 4: For each school, subject, and prior achievement group, calculate a weighted average of the residuals across grades and years using the number of continuously enrolled tested students as weights. Calculate standard deviations of all grades’ and years’ residuals to measure the consistency of performance. Thus, for the previously below-passing group in a grades 6–8 middle school or a 9–12 high school in mathematics, the weighted average and standard deviation of the nine mathematics residuals are calculated.
Step 5: (Consistent performance requirement) Check to see if the weighted average and standard deviation of the residuals fall in the right region (shown in green in Figure 1) on a plot in which the average residual (representing performance and labeled the “Performance Score”) is shown on the y-axis and the standard deviation of the residuals (representing consistency of performance and labeled the “Variability Score”) is on the x-axis. The requirements for this region are:
1) The standard deviation of the residuals is below 1.5.
2) The weighted average of the residuals is above a line with 0.5 slope and the following intercept:
1.5 If the school has only one tested middle or high school grade
1.0 If the school has two tested middle or high school grades
0.5 If the school has three or more tested middle or high school grades
0.75 For all middle and high schools in writing, science, and social studies
Step 6: (Sufficient data requirement) Check that the school has no missing residuals in 2006, and no more than the following number of residuals missing in 2004 and 2005:
|
Number of possible residuals in 2004 and 2005 |
2–4 |
5–8 |
|
Number of residuals from 2004 and 2005 that may be missing |
0 |
1 |
Residuals could be missing because the school has missing demographic data or not enough tested students in the grade, subject, and student prior achievement group in question.
Step 7: (Achievement level requirements) Check to see whether the school meets the following achievement criteria in the subject in question:
For the previously at- or above-passing student group:
1. At least 15 percent of the students from this group must meet the NCEA College Readiness Benchmark (if the subject is reading, English/language arts, or mathematics) or the Commended standard (if the subject is writing, science, or social studies) for all grades and all three years. High school science was checked only for the 2006 Commended standard.
2. At least 75 percent of the students in this group must meet the state’s passing standard in all grades across all three years.
3. The school’s JFTK opportunity gap
(see www.just4kids.org) must be
–30 or above in every grade in 2006.
For the previously below-passing student group:
1. At least 45 percent of the students in this group must meet the state’s passing standard in all grades across all three years. (There is no requirement for college readiness or performance on the Commended standard.)
2. The school’s JFTK opportunity gap
(see www.just4kids.org) must be
–30 or above in every grade in 2006.
Step 8: (Adequate Yearly Progress [AYP] and percent tested requirements) Check to see whether the school met its AYP requirements in all three years of analysis. In years where AYP status is missing, at least 85 percent of the enrolled students had to be tested in all grades in the subject in question. For science, writing, and social studies, the school needed to test at least 85 percent of enrolled students in all grades in all three years.
A school meeting all of the requirements in Steps 5–8 is deemed to be consistently higher performing in the subject and student group in question (for example, in mathematics with students who passed the exam in the previous year).
Steps for Elementary Schools
Elementary schools were defined as any school whose highest grade was six or lower. For schools whose highest grade was seven or higher and whose lowest grade was five or below, the portion of the school ending in grade 5 was treated as an elementary school.
Step 1: Merge each student’s test results for each year in the analysis with the same student’s fall enrollment records from the same year and the two prior school years. Use this information to separate out test results of students who were continuously enrolled at the same school for at least three years (“continuously enrolled students”). Where the school’s grade span would not normally result in students having been at the school for three years (for example, fourth- and fifth-graders in a grades 4–6 school), select students who were enrolled as many years as the school’s grade span allows and in the same district for at least three years. As with the middle and high school analyses, three years of test scores are analyzed: 2004, 2005, and 2006.
Step 2: Calculate the average test scores for the continuously enrolled students in each grade and year. Scale scores of students tested in Spanish and English were averaged together in this analysis, following the statement by the Texas Education Agency (www.tea.state.tx.us/student.assessment/resources/techdig05/chapter2.pdf ) that a scale score of 2100 on the Spanish and English TAKS can be treated as comparable.
Step 3: For each grade, subject, and year, regress the average scale score of the continuously enrolled students on the school’s overall percentages of low income, ELL, African American, and Asian students; the school’s total enrollment; a flag for whether the school was/was not considered a magnet school; and the grade level percent of students tested in the subject. Weighted least squares regression was used with the weights based on the number of continuously enrolled tested students in each grade, subject, and year. The standardized residuals for this analysis were saved and used as the school’s performance measure for the grade, subject, and year in question.
Step 4: For each subject in each school, calculate a weighted average of the residuals across grades and years using the number of continuously enrolled tested students as weights. Calculate standard deviations of all grades’ and years’ residuals to measure the consistency of performance.
Step 5: (Consistent performance requirement) Check to see if the weighted average and standard deviation of the residuals fall in the right region (shown in green in Figure 1) on a plot in which the average residual (representing performance and labeled the “Performance Score”) is shown on the y-axis and the standard deviation of the residuals (representing consistency of performance and labeled the “Variability Score”) is on the x-axis. The requirements for this region:
1) The standard deviation of the residuals is below 1.5.
2) The weighted average of the residuals is above a line with 0.5 slope and the following intercept:
1.25 If the school has only one tested elementary grade
1.0 If the school has two tested elementary grades
0.75 If the school has three or four tested elementary grades
1.25 For all schools in writing and science
Step 6: (Sufficient data requirement) Check that the school has no missing residuals in 2006, and no more than the following number of residuals missing in 2004 and 2005:
|
Number of possible residuals in 2004 and 2005 |
2–4 |
5–8 |
|
Number of residuals from 2004 and 2005 that may be missing |
0 |
1 |
Residuals could be missing because the school has missing demographic data or not enough tested students in the grade, subject, and student prior achievement group in question.
Step 7: (Achievement level requirements) Check to see whether the school meets the following achievement criteria in the subject in question:
1. At least 15 percent of the continuously enrolled students in each tested grade must meet the NCEA College Readiness Benchmark (if the subject is reading or mathematics) or the Commended standard (if the subject is writing or science) in all three years.
2. At least 75 percent of the continuously enrolled students must meet the state’s passing standard in all tested grades across all three years.
3. The school’s JFTK opportunity gap
(see www.just4kids.org) must be –30
or above in each tested grade in 2006.
Step 8: (Adequate Yearly Progress [AYP] and percent tested requirements) Check to see whether the school met its AYP requirements in all three years of analysis. In years where AYP status is missing, at least 85 percent of the enrolled students had to be tested in all grades. For writing and science, the school needed to test at least 85 percent of enrolled students in all grades in all three years.
A school meeting all of the requirements in Steps 5–8 is deemed to be consistently higher performing in the subject in question.
Figure 1
Consistency Performance Requirement
