How does your school rate? Yvan Guillemette, policy analyst for the C.D. Howe Institute answered your questions on the unique school comparisons it created that take socio-economic data into account.
The Q&A is now over, but please send us your comments.
_______________________________
Q: Is there a similar study on high schools, or will you be conducting one in the future?
A: There is no similar study on high schools for the moment, and we have no current plans for one.
We do, however, have plans for a similar study of primary schools in Alberta and BC (and perhaps Quebec).
Q: Sorry, I don't understand the answer to my earlier question about how student postal codes were determined as a basis for this whole study.
Please clarify "a six-digit postal code" -- is that the student's home address or the school's address? What is a "census dissemination area?" When census data is used, does it actually relate to the individual student's circumstances, or does it relate to the population as a whole in that neighbourhood? (For example, in inner-city schools where the local school families are often poorer on average than the neighbourhood residents who don't have children.)
A: The six-digit postal codes used are the students' postal codes, from which we can tell in what census dissemination areas (DAs) they live. A DA is the smallest geographic unit available in the 2001 census. It includes 400 to 700 people. The school's community profile is then constructed by weighting the socio-economic variables of each DAs by the number of students at a given school that live in those DAs. Some of the socio-economic variables used relate to the entire population in the DA, others relate only to those with children.
Of course, ideally, one would like to be able to measure directly each individual student’s socio-economic characteristics and associate them with the student-level assessment results. Such data are not collected, however, because of concerns about privacy. Instead, one is able to measure only the average characteristics of households in the community from which a school draws its students -- and then only indirectly.
So, even though we do not know the socio-economic characteristics of individual students, the characteristics of the small census units in which students live can tell us much about them. And we can determine which small census units students live in because we know their postal codes.
Q: There's no rating for Thomas L. Wells Public School in Scarborough. It opened in 2005. Why didn't C.D. Howe include it?
A: To ensure fair representation, only schools where we have three years of data in all three assessments and 15 students or more per assessment per year are included in the rankings.
Since the school you mention opened less than three years ago, there is no evaluation for that school.
Q: In some schools there is a huge difference in rating between Grade 3 and Grade 6, for example, the Grade 3 percentile would be 20 but the Grade 6 is 80, or vice-versa.
What could be the reasons for such a discrepancy for the same school? In this case, how do you rate the overall performance of the school?
A: Such a large discrepancy between the Grade 3 score and the Grade 6 score is rare. When it happens, the most likely explanation is that the school is small, with perhaps only one or two classes in each grade, and that the teacher(s) in one grade is far better than the teacher(s) in the other grade. It could also be that for some reason the two grade are drawing students from different neighbourhoods, that is, from different socio-economic contexts.
Q: When defining postal code, what exactly is being used? (On page R1: "By linking student postal codes to census data on education, employment..." ). Is it the first three digits of the postal code?
I'm asking because I believe census data use the first three digits only. So where I live, this includes inner-city schools and schools that in the Star's terminology are "carriage trade." In other words, any attempt to define a neighbourhood using only the first three digits would be flawed.
A: A six-digit postal code is linked to either one census "dissemination area" or, in a smaller percentage of cases, to several dissemination areas. The "neighbourhood" is NOT a geographic concept in the sense of a physical area around a school. It is the actual dissemination areas in which the students at a given school live.
Q: What is the definition in the charts for the column "pass rate"? Achieving which levels - 3 and 4, or 2, 3 and 4? I'm not an expert but I believe 2 is considered a pass.
A: The pass rate is the percentage of students at a given school and grade that achieve at levels 3 or 4 according to EQAO Method 1.
Q: Which schools were compared against each other? Specifically, which schools was Bedford Park compared against?
A: Through the statistical methodology used, every school is effectively compared to every other school in the province. The methodology used is hard to fully grasp unless one is familiar with the statistical technique of regression analysis. To put it simply, a mathematical equation based on several socio-economic variables is estimated using the EQAO school results and the census variables for all schools in the province. This equation is then used to "predict" the score of each school given the particular values of the socio-economic variables for that school. If a school does better than this equation predicts, than it scores above the 50th percentile. If it does worse, then it scores below.
In this sense the comparison really uses all the information from all the schools in the province, the comparison is in no way restricted to a specific area or a specific socio-economic context.
Again, this methodology is explained in much more details in Signposts of Success, David Johnson's 2005 book available from the C.D. Howe Institute. It is explained more concisely with the help of Figure 1 in the e-brief available at http://www.cdhowe.org/pdf/ebrief_39.pdf.
Q: I live in Port Sydney and was looking at our local school's results. I noticed that the socio-economic factors was a factor of -21.4. How is this factor determined? Do they consider all residents or only those who have children in the school? Are seasonal residents included?
A: -21.4 is the adjusted pass rate for that school, that is, the pass rate from which we subtract the provincial average pass rate. It means that this school's pass rate is 21.4 percentage points below the provincial average (which is about 60). The adjusted pass rate has nothing to do with the socio-economic variables. The measures adjusted for the socio-economic factors are the percentiles.
Q: Why does David Johnson's formula take percentage of single detached homes as a socio-economic factor? My semi-detached home in Riverdale is worth many times more than many homes in other areas of the province. Does the high amount of semi-detached homes in Riverdale bring down our socio-economic rating? Are high density areas penalized or do they get a boost by his "crunching" formula?
A: The proportion of single detached homes in an area turns out to be a good indicator of wealth, stability and community involvement of the residents in that area. The higher the percentage of single detached homes in an area, the higher the socio-economic status of that area tends to be. Schools in an area with a higher percentage of single detached homes would be expected to do better, all else equal.
Q: Regarding alternative schools, are all students included in this testing? Also, for schools that have less the 16 students participating, why are the results not included and how do you get and accurate result for the past five years comparing with other schools?
A: The rankings use an average of three years of EQAO results over three assessments, but only if there are 15 students or more in all years/assessments. The reason this is done is to insure sufficient results for a fair and representative comparision. All students in public schools are included in the rankings if they meet the filters above. Alternative schools are included if they participate in the EQAO testing process.
Q: Please indicate the census data you have used in the study. What year is it taken from and what formula did the professor use to compensate for the factors considered?
A: The socio-economic data are from the 2001 Census. The detailed methodology included the regression equations used to construct the rankings are explained in Chapter 6 of David Johnson's book, Signposts of Success, available in print from the C.D. Howe Institute.
Q: Were private schools rated as well? If so, where can I find this information?
A: Only public schools that participate in the EQAO testing process were evaluated.
Q: How are the school ratings calculated? I know that EQAO is used as the primary source of data, are there any other assessments used? If EQAO is used, how do you adjust for the fact that the EQAO test has been changed recently, and therefore cannot be used to track students progress effectively? After all, the assessment cannot be used to assess if a school educates students.
For example, if a student moves into a school in Grade 3, and takes the EQAO test, it cannot be determined if the student improves. Of course the test will be taken again in Grade 6, but the test has changed since they last took it. Any comment?
A: Only EQAO test results are used, along with 2001 census data to adjust for the schools' socio-economic contexts. The EQAO data are not used to track a student's or a school's progress, they are used to evaluate what percentage of students in a grade meet EQAO standards as compared to other schools in similar socio-economic environments. To get a more representative evaluation, 3-year averages are used.
Q: Did the C.D. Howe Institute use Method 1 or Method 2 EQAO results? If Method 1 ... are you aware that there are many problems with this reporting? For example, Method 1 includes the results of Mildly Intellectually Disabled students. These students typically do not write EQAO and thus in Method 1 receive zeros. This skews the results of any school with an MID program.
Moreover, as a teacher I can tell you that our results almost always include scores of zero for children no longer even enrolled in our school at the time of EQAO testing. The ministry still has them listed as attending. Students absent, for example on a trip during testing (quite frequently happens nowadays in inner city schools - families travel during the school year) also count as zero and skew the results.
A: David Johnson's methodology uses EQAO "Method 1," because the process of exempting students from writing assessments creates more significant problems with Method 2.
His book explains the reasoning as follows (Chapter 5): "Some Ontario elementary school students are exempted from the Grade 3 and Grade 6 assessments undertaken by the Education Quality and Accountability Office (EQAO). Students can be exempted when, even with all possible accommodations, 'the student would be unable to participate productively and/or where the student’s participation would be harmful.' The overall exemption rate is quite low — falling between 4 and 7 percent — but it varies across years and across assessments. There is also some evidence that exemption rates vary across boards — that is, two schools with the same socio-economic characteristics but located in different boards can have systematically different exemption rates. Exemption rates are important because some users of school assessment results emphasize EQAO’s 'Method 2' presentation of results, which are open to manipulation. In this methodology, the key variable is not the percentage of all students at a school that achieves at Level 3 or Level 4 but the percentage that actually wrote the assessment at a school that achieves at those levels. Thus, a school or board that wanted to improve its apparent assessment results using Method 2 could do so simply by exempting more students who would not score at Level 3 or Level 4. In the logical extreme, 'perfect' Method 2 results could be obtained by exempting all students except those who would score at Level 3 or Level 4. Most school boards, EQAO itself, and most users of EQAO data do not make use of Method 2 results."
_______________________________
C.D. Howe is the first organization to calculate the socio-economic level of each grade school in the province, and then compare its test scores with schools of a similar background, rather than lumping them all into the same crude ranking. Their way is a being hailed as a much fairer way of determining if a school is doing a good job educating its students.
Designed by Wilfrid Laurier economics professor David Johnson, the ratings calculate each school’s socio-economic profile based on census data for that neighbourhood, then compares test scores among schools with similar demographics, from family income to parents’ education. Schools scoring a 50 are about average; those below 80 are doing a tremendous job and those scoring in the 30th percentile or lower should be cause for concern.
Students in French immersion schools have to write the provincial tests in English.
While both grades are more disadvantaged relative to English schools, my guess is that Grade 3 kids are at an even bigger disadvantage than Grade 6 kids.
Do you have any statistics that reflects this disadvantage?
(Editor's note: This was not a factor included in the C.D. Howe report. According to the EQAO, however, Grade 3 French immersion students have three choices - write the entire test in English; do the reading/writing portions in English, and write a French translation of the English math; or write a French translation of the English math only. Grade 6 immersion students write the entire test - reading, writing and math - in English.)
Posted by: Dodie | March 03, 2007 at 06:25 PM
It's great that your results have taken into account socio-economic factors, as this definitely does more to even the playing field. However, there are still other factors that need to be considered. For example, Crosby Heights in Richmond Hill scored in the 80th percentile for Grade 6, but only in the 27th percentile for Grade 3. Couldn't this be explained by the fact that Crosby offers the "Gifted" program in the area, and that program only starts in Grade 4. Beverley Acres scored very high, but this is a French Immersion school. French Immersion is basically an enriched program and children weaker in language skills often leave the program early on to return to their English schools. Thus in both these cases, (as in others) even your scores may not accurately reflect the quality of the teaching.
Posted by: B. Davis | March 02, 2007 at 06:42 PM
How would I find out what are my neighbourhood school characteristics? e.g. Burnhamthorpe PS and Nahani Way PS
Thank you
Editor's note: You can find the neighbourhood characteristics at
http://www.cdhowe.org/pdf/ebrief_39profilesupdate2007.pdf
Posted by: Chafei Mutassem | March 02, 2007 at 04:19 PM
Can I access a copy of the full report so I can explore the methodology used?
I would like to see the full list of critera that make up the 40% percent environmental impact factored in to your school rankings. Also, how did Dr. Johnson create a percentile measure for each factor (i.e. single parenthood or parental income) and if so how did he come up with this? Whereas it is now well recognized that environmental factors play a role in student success, I'm not sure that there is adequate research to allow these factors to be quantified.
(Editor's note: It sounds like you are looking for more detailed information than what's in the online report. Try contacting the C.D. Howe Institute for a copy of David Johnson's book, "Signposts of Success.")
Posted by: connie donner | March 02, 2007 at 11:30 AM
Why was the term "Pass Rate" used?
This implies those who don't achieve Level 3 have "failed".
Unless there is some new definition of "Pass/Fail" that I have missed.
Implying a student has "failed" is a harmful statement to make to the many children who didn't achieve Level 3, and their teachers. Is a 2 "bad"? (It's a C, including C-plus).
As a parent who has dedicated many hours to improving the quality of the school experience for all of the children at my school, I am disappointed in this project. Does it really add any value at all?
Posted by: Robert Miller | March 02, 2007 at 11:09 AM
What can parents do if a school is in a downward trend of not meeting board level scores i.e. low EQAO pass rates and extremely low percentile rating (6%)? What can be done to make schools more accountable to deliver and maintain acceptable/high standards of teaching and learning?
Posted by: Valerie | March 02, 2007 at 10:36 AM
Spend what it cost to run EQAO on healthy breakfast and lunch programs and the marks or more importantly the knowledge of most students, in economocally depressed areas, will improve.
Posted by: Peter Komlos | March 02, 2007 at 07:17 AM
Ms. Brown
What are the success rates of graduates for each high school ;
-% going unto university
-% going unto college
-% of exceptional students going to university
-% exceptional students going to college
Some Principal proudly announce these stats while others state they have no knowledge of such facts.
It is our understanding that 80% of all jobs today require a post secondary degree.
Success rates for all would be helpful.
Any enlighten you could provide is appreciated.
Thank you.
Posted by: K. Strachan | March 02, 2007 at 12:32 AM
Is there any follow up studies to see how these ratings translate to future success in secondary school and university achievement?
Thank you.
Posted by: Melody Kram | March 01, 2007 at 11:31 PM
It disappoints me that the sucess of our students is not being noted but rather, the perception of their overall failure.
The fact that the scores published include children in a variety of regionally placed classrooms (LI, Closed LD, Behavioural) is unfair to all. Despite the progress students differentiated learning classrooms are making, and the progress students in homeroom classes are making - all are judged by the public to be failures without knowing the big picture of their school or their situation.
If these types of results continue to be reported, as is the case with your published results and that of EQAO, what is the point of an IEP (Individual Education Plan) or differentiated learning?
But of course, if we give the picture - school by school with regard to what is really going on: children arriving late or having frequent absences, not getting breakfast, sleeping on couches in the living room,... - surprisingly, that wouldn't make a headline.
Posted by: Colette Bell | March 01, 2007 at 11:09 PM
I believe it would be interesting to look at the cost of EQAO. Isn't it possible that our children would benefit more by taking that money and investing it directly into the classrooms? We all know that children with low reading skills do not do well in other subjects...in some cases cannot even understand the questions let alone the answer. However, "low-readers" are sometimes not given extra help because of lack of school funds.
Maybe it is the appropriateness of curriculum that we should be looking at instead of test scores that are used in rankings of schools. It seems that our students are being taught solely to pass and get good grades on EQAO. Maybe we should look more closely at what types of teaching leads our children to WANT to learn because they are being taught age appropriate inspiring and attainable information. We should focus more on attainable goals that help all children to progress in a worthwhile manner. EQAO leaves no room for variables or individual struggles. Just visit a grade 4 class where a student has been left behind and given up...EQAO means little for such a child.
How can we believe in a test result that includes a child that has not written a test and who is given a zero which is then included in the classes average. Statistics can be very misleading and prove what we want them to prove.
What is EQAO and this curriculum really costing our children?
Posted by: L. Thomas | March 01, 2007 at 08:15 PM
Very informative study. Let's not forget brains create wealth. Wealth buys a good education and thus greater brain power for the future. Good teachers follow wealth and brain power thereby adding to the continuation of better educated children. Every society is similar. Unfortunately there are far too many parents in Canada who have limited brain power and unable to help their children get a better education. Teachers are not going to put out for children who fall behind because they get no support from the parents. We will always have this segment in society. This will only change when the Government pays serious attention to bright and willing students of poor families. They cannot be allowed to fall through the crack, but how many times have we heard that story.
Posted by: Daniel Stevenson | March 01, 2007 at 02:42 PM
I am a Literacy Lead teacher for one of the schools in this article. It was very disturbing to see how data can be so easily biased in its presentation. A very large factor that was completely disregarded was the existence of special classes within schools. Are you aware of gifted classes and "Community Classes" and their affect of the data for each school? The "standardized" testing results are greatly affected by the inclusion of data from these classes. Our school population includes many students who face considerable challenges, both physical and emotional. Some students are working long and hard to make essential gains, such as being able to blink effectively in order to communicate. These students, who come to us from a larger community are included in our school's EQAO data. As I consider the legitimacy of your results, I remember John MacEnroe's words, "You can't be serious!"
Ranking such as yours can be a very dangerous tool indeed.
Posted by: Nancy Newman | March 01, 2007 at 12:42 PM
How is the local community determined and defined for each school in providing the socio-economic data? If it is Statscan postal code data that is problematic as schools often draw students from beyond their immediate vicinity. Also, schools that are in close geographic proximity may have a very different socio-economic background of their students.
Posted by: William Hick | March 01, 2007 at 12:20 PM
I think this study is very innovative and gives a true picture of how schools should be assessed. Unfortunately, I cannot find the school that my daughter goes to on this list. The school is St. Julia (Dufferin-Peel Catholic Board) in Mississauga, Peel region. Can you please let me know if this is a miss or they did not participate in this study ?
Thank you
(Editor's note: Schools must have at least three years worth of testing data, and at least 15 students must have taken part each of those years, for a total of 45. New schools would also have been left out of the study for this reason.)
Posted by: Anu Sabapathy | March 01, 2007 at 12:16 PM
Were the French Catholic Durham School boards rated?
Editor's note: The French boards were also a part of the C.D. Howe report, go to http://www.cdhowe.org/pdf/ebrief_39scoresupdate2007.pdf
Posted by: Cherie | March 01, 2007 at 11:43 AM
Dear sir,
The variable "Official Language" - (percentage of the school community that speaks official language at HOME)needs to be defined. I live in Etobicoke North, community of Highfield JS. I can resonably predict that not more than 10% of the community speaks official language at home, as 64.5% suggested by your report. 64.5% of the people might understand an official language, but that doesnt necessarily mean the same number speaks the same at home. The language spoken at home has a great influence on the reading and vucablary skills of a child. I tutored few children in my apartment building for free last year. Their main problem was to understand the meaning of the question, although they were brilliant enough to solve the problem of their own.
Looking forward for your comments.
Thanking you
Yours truly,
Rajeshwar Harchand
Posted by: Rajeshwar Harchand | March 01, 2007 at 11:26 AM
Hi, Sir,
Where can I find the neignbourhood characteristics information for those schools?
Thanks.
Editor's note: You can find the neighbourhood characteristics used by C.D. Howe at: http://www.cdhowe.org/pdf/ebrief_39profilesupdate2007.pdf
Posted by: Lina Ren | March 01, 2007 at 11:20 AM
Dear C.D.Howe:
Your work is a great start in the right direction to correcting the flawed perceptions the provincial tests support. Please influence the minister of education to change the test methods.
Can you please do this!
We should be testing the improvement, not the raw value of test result. That's why tests should be held at both the start and the end of each school year being tested. This is the only way to fairly capture school performance as it relates to children. Testing less often does not work in our transient society, were both families and teachers are on the move. Start/End tests will give a much more accurate picture of school and teacher performance. That's what we want!
Thank you,
Please keep up the excellent work!!!
Daryl Diamond
Posted by: Daryl Diamond | March 01, 2007 at 11:11 AM
Hi there....did you do any tests on the French immersion schools in Toronto. Is the "quality" of the education different, superior? How would you test the Grade 3 and 6 students - in French?
We are currently struggling with whether to put our child into French immersion (available in our district school) or into the regular English class. However, since our district school happens to have quite a low rating, I'm wondering whether there is any way to compare the French students?
Thank-you - I look forward to your comments
Posted by: Natalia Dukszta | March 01, 2007 at 11:02 AM
Is it just possible that at least a part of the Warden School's success could be atrrtibuted to presence of the child care program that exists on site?
I think that the analysis could be enhanced by identifying the presence or absence of community/neighbourhood supports such as child care and other early learning and care programs.
Posted by: Petr Varmuza | March 01, 2007 at 10:49 AM
Do you look for any trends in test scores versus date on which the tests were marked to see if the marking is consistent across the multiple days of the marking period?
Posted by: Annie | March 01, 2007 at 10:39 AM
A school in my community that consistently ranks very high also hosts the gifted program and draws children from outside of the immediate community. How is this reflected in the ranking? While the school performs well in both Grade3 and Grade6 EQAO results, there is a marked improvement in the Grade6 results (gifted starts in Grade4). The immediate community cannot supply the enrollment to support the local school and has a high number of retirees (lowering the average income of the neighbourhood). The school population is about 50% immediate community and 50% bussed gifted students. Wouldn't this impact the accuracy of your reported rankings as 50% of the students are not in the local area so their household information would not be reflected in the adjusted rankings for the report?
Posted by: Mark | March 01, 2007 at 10:19 AM
Very interesting study.
You recommend that administrators take a look at schools with either a high percentile rank or a low one. Looking at your data for say the Toronto District School Board there are quite a lot of schools where the rankings move significantly between Grade 3 and Grade 6. That is, for some schools their percentile rank in Grade 3 is high and then in Grade 6 drops to a much lower rank or vice versa.
For example looking at your report at two schools in midtown Toronto: the ranking for Rosedale is 29th percentile in Grade 3 and then jumps to 60th percentile in Grade 6; while the nearby Jesse Ketchum is 78th percentile in Grade 3 and then drops to 31st percentile in Grade 6.
These results suggest that it would be worth looking at why this is happening. For instance, does your data show that lower income schools are the ones that tend to drop their ranks from Grade 3 to 6, while the higher income schools tend to improve from their ranks from Grade 3 to 6? Further analysis of your data would be fascinating to see and would provide insights on what are the most important factors contributing to a child's academic success.
Posted by: Natasha Cuddy | March 01, 2007 at 09:50 AM
Why does David Johnson's formula take percentage of single detached homes as a socio-economic factor. My semi-detached home in Riverdale is worth many times more than many homes in other areas of the province. Does the high amount of semi-detached homes in Riverdale bring down our socio-economic rating? Are high density areas penalized or do they get a boost by his 'crunching' forumla?
Posted by: Darryl Patrick | March 01, 2007 at 09:43 AM