Measuring Progress in the South Bronx

Clara Hemphill got to know District 7 schools in the South Bronx early in the Bloomberg administration. She returned there in 2009. Here is what she and our research team learned about New York's high-stakes school reform and its impact in a high-poverty neighborhood

For decades, PS 25 in the South Bronx was one of the lowest performing schools in the city. Designated as a bilingual school and led by a politically connected principal, PS 25 had classes that were conducted mostly in Spanish, with children learning English for less than one hour a day. When I first visited in 2004, I saw teachers make grammatical errors in both English and Spanish. Barely one-third of children were reading at grade level. Math scores were not much better.

When the city's Department of Education (DOE) appointed a new principal, Carmen Toledo, in 2008, she had her work cut out for her: Under the department's then-new accountability system, the school could have been closed within the year unless test scores went up. Toledo enlisted her entire staff to work on test prep, organizing Saturday classes and extending the school day from 3 p.m. to 5 p.m. She hired two English as a Second Language teachers to improve the skills of children who spoke only Spanish. The proportion of children reading on grade level soared from 33 percent in 2008 to an astonishing 69 percent in 2009. Math gains were even more dramatic.

Amazingly, the school rose from the first percentile—near the bottom of all city schools—to the 99th percentile—the very top—on its annual Progress Report, the DOE's statistical measure of a school's success. In other words, the school showed greater gains than all but a handful of schools in the city. The department's analysts gave PS 25 a grade of "A" in 2009, up from a "D" in 2008. And Toledo received a $25,000 bonus.

Instead of being satisfied, however, Toledo is worried. She knows that unusually high test scores one year are likely to be followed by average scores the next, and the city's accountability system punishes schools when test scores decline. Because of the way state standardized tests are constructed, scores fluctuate considerably from year to year, particularly in small schools like PS 25 and especially among children with low levels of achievement, testing experts say.

Toledo knows that her school still has serious challenges that she is only beginning to tackle. For one thing, several teachers have a shaky command of English. On my visit in December 2009, some classes had only a dozen children present, even though 20 were on the register. Children stared off into space. In some classes, transitions from one activity to another were slow, children squirmed and teachers struggled to get them to pay attention. It was hard not to wonder: Were the high test scores an anomaly, or the result of sustainable improvements?

"I'm scared," Toledo says in an interview in her office, seated beneath what she calls her data wall, with charts tracking the test scores of every child in the school. "We made such big gains. Maintaining it is a challenge."

Far-Reaching Experiment

Toledo and PS 25 are part of a giant, far-reaching experiment that began in 2002 when Mayor Michael Bloomberg wrested control of the city's schools from the discredited Board of Education and appointed Joel Klein, a former Justice Department trust-buster, as his schools chancellor. Appalled by the neglect and mismanagement that had allowed schools like PS 25 to languish for decades, Klein dismantled an ossified bureaucracy and turned traditional notions of school management upsidedown. He gutted district offices and fired hundreds of administrative staff with the goal of redirecting millions of dollars to individual school budgets and higher teacher salaries.

At first, Klein organized the city's schools into 10 large regions and directed regional superintendents to establish a core curriculum for reading and math, a common set of teaching methods for all schools. But in 2007, impatient with the pace of change, he charted a new course. He dismantled the 10 regions, abandoned the core curriculum and put in place a new administrative structure unlike any other in the country. In this new structure, he simultaneously centralized authority—using highstakes accountability measures monitored by department officials—and decentralized responsibility, treating each school as an autonomous entity under a principal's leadership rather than as part of a larger district or region. The name given to these reforms was Children First.

The idea, as Klein has said repeatedly, is not to create a great school system but to create a system of great schools.

This means Toledo and all of the city's 1,588 principals are freed from the dictates of a district office under Children First, which is colloquially known across the system as "empowerment." Principals may make decisions about everything from hiring teachers and allocating budget dollars to choosing which books children should read; as long as their test scores continue to rise their decisions will not be overruled.

Today, only tiny vestiges of the district offices remain. While each superintendent once had dozens of staff members, now each has only two: a secretary and a "district family advocate" charged with responding to parents' concerns. According to state law, the superintendents retain the power to hire, remove and evaluate principals, but Klein has interpreted this law narrowly. In fact, superintendents are not permitted to visit a school without the principal's permission. While some superintendents do this regularly, others visit as little as once a year. Superintendents appoint principals from a pool of candidates approved by the central DOE. Any decision to remove a principal is heavily influenced by statistical measures analyzed and interpreted by department officials headquartered downtown, in the former Tweed Courthouse next to City Hall.

Klein's restructuring has all but eliminated middle management and abolished day-to-day supervision of principals. His colleagues at Tweed say this method is working well. As signs of their policy success, they point to state test scores and graduation rates that have risen dramatically in recent years. Not everyone agrees, however. Critics say these indicators are inflated, based on easier tests in the elementary and middle schools and inflated grades in high school.

In an attempt to find out which view is closer to the truth, during the past year I revisited 12 schools in District 7 in the South Bronx that I had first come to know earlier in the decade. Along with our team at the Center for New York City Affairs, I also studied the Progress Reports and related testing data. In the district and citywide, we interviewed several hundred principals and other school administrators, teachers, school staff, parents, policy-makers and others during the fall of 2009 and the spring of 2010.

Here is what we found.

Klein's reforms have allowed some very talented principals to turn around failing schools or create new schools from scratch, to forge their own vision and assemble their own faculty without bureaucratic interference. These principals have succeeded in bringing order, discipline and solid teaching to some of the city's most troubled schools.

At the same time, the reforms have left inexperienced or ineffective principals to manage without much guidance or direction. Some principals receive high marks on the city's Progress Reports 18 even though their schools offer little more than a thin gruel of test prep. Meanwhile, the city's accountability system makes it nearly impossible to tell which schools offer children engaging instruction and a rich curriculum. This, in combination with the fact that there is no day-to-day oversight of principals and their schools, means it is often unclear which schools are struggling but moving in the right direction—and which are so distressed that dramatic action, such as the removal of a principal or the closing of a school, is in order.

The formula the city uses to judge elementary and middle schools rests heavily on the highly volatile results of two state tests, reading and math. Because of this volatility, a school's percentile ranking in the Progress Reports can move in just one year from the very bottom to the very top, as PS 25's did, and vice versa. The swings are particularly dramatic when a school adds or drops entire grades, as PS 25 did when it eliminated its eighth grade in 2009.

"The scores should not swing wildly that like that," says Howard T. Everson, a professor at the City University of New York Graduate Center and chairman of a committee that advises the state on testing. "There is obviously something amiss in the measurement."

The District 7 schools offer encouraging evidence that real progress has been made in one of the city's poorest neighborhoods. But they also offer evidence of another sort, revealing that the city's current accountability system doesn't accurately reflect every school's progress—and may well be hindering further gains.

Large Gains, Significant Problems

District 7, part of the poorest congressional district in the nation, was emblematic of everything wrong with the city's schools during the latter decades of the 20th century. One of 32 community school districts created in 1969 under a law designed to give local school boards control of elementary and middle schools, District 7 had a long history of hiring driven by patronage and nepotism. It also had some of the lowest-performing schools in the city. According to a 1996 report by the city's special commissioner of investigation, the district superintendent, Pedro Crespo, hired unqualified friends and relatives of school board members and approved expensive junkets and perks. In one instance, Crespo appointed a principal with a poor command of English who had failed eight licensing exams. Teachers and principals were pressured to buy and sell tickets for large parties organized to raise campaign funds for local politicians. School board meetings regularly erupted into shouting matches during which, for example, school board members were accused of stealing computers from the district office.

Reports of corruption and nepotism declined after a 1996 state law limited the powers of the city's community school boards and expanded those of the chancellor. Still, achievement in District 7 remained pitifully low. Although overt political influence declined, principals still paid homage to elected officials: In 2002, five District 7 principals made contributions to the re-election campaign of Carmen Arroyo, a longtime member of the state Assembly; in 2005, six principals did, according to financial disclosure reports filed with the state Board of Elections.

When I visited 30 schools in District 7 in the South Bronx as a reporter for the Insideschools.org website early in Mayor Bloomberg's first term, the schools, with a few noteworthy exceptions, were in a sorry state. I met principals who routinely called for an ambulance to take an out-of-control child to the nearest psychiatric emergency room because they didn't know what else to do. The middle schools were chaotic, with children wandering aimlessly in the hallways as teachers lectured to half-empty classrooms. Some of the elementary schools were sweet, warm places with kindhearted teachers doing their best—but the children didn't know how to read. While I saw pockets of good instruction, some parents complained to me that their children were taught mostly in Spanish for as many as five or six years, learning almost no English. Books and supplies were scarce.

Returning to a dozen of those District 7 schools recently, I found much has changed. Books and supplies are abundant. Most of the schools I visited were orderly, with children in classrooms rather than roaming the corridors. Instruction is mostly in English; and the bilingual classes that remain are designed either as "transitional," that is, temporary instruction in a native language, or "dual language," in which children become fluent (and literate) in both English and Spanish. Principals are now appointed from the applicant pool selected by Tweed, rather than by the district office. Some of these new principals have a wealth of talent and experience.

The principals, who, with some restrictions, may now hire teachers as they see fit (rather than having them assigned by the district office), say it's easier to recruit and retain staff largely because teacher salaries are substantially higher than they were before the Bloomberg-era increases. While student achievement, as measured by standardized tests, is still far below the state average, an analysis of fourth and eighth grade test scores by the Center for New York City Affairs found that the gap between District 7 and the rest of the state narrowed significantly from 2002 to 2009 in fourth and eighth grade math and fourth grade reading. A similar analysis of New York State's 64 largest school districts conducted by the city Department of Education shows that District 7's test scores started at the absolute bottom in 2002 and made some of the most dramatic gains of any large district in the state. In other words, these South Bronx schools appear to be making real progress relative to the rest of the state.

Click to enlarge

Yet for all these gains, significant problems remain. While some schools have a rich curriculum, others offer bare-bones instruction narrowly designed to help children pass standardized tests. Many of the newly hired principals have had minimal teaching experience and almost no administrative experience, and struggle mightily with basics like student discipline. While middle school attendance has improved, attendance in District 7 elementary and high schools has not improved significantly since 2002 and remains well below the citywide average. Little progress has been made in special education, officials say, and parents of children with disabilities told me their children continue to receive woefully inadequate help. And, while high school graduation rates have increased markedly, a number of principals openly acknowledge that their students have met only the bare minimum requirements for graduation and are poorly prepared for college.

Many principals say the chancellor's Children First structure has released them from the yoke of ineffectual supervisors and allowed them to do better work. At the same time, some say they feel isolated and long for a chance to share ideas with fellow principals who face similar struggles.

"In the old days, there was the tyranny of superintendents who treated their principals liked soldiers in an army," says John Garvey, the City University of New York's liaison to the public schools until his retirement in 2008. "Good riddance to that. But now, instead of shoving things down people's throats, schools are left to their own devices, for the most part." From 2007 to 2008, Garvey oversaw one of several nonprofit organizations charged with providing support to principals and schools, and he's had experience with the school system both before Klein and under Klein's various reorganizations.

"The system was in desperate need of a breakup," Garvey adds. "It was fossilized. It needed to change. But the present structure is unwieldy. The schools are scattered. People tend to be insular and isolated." Since principals became empowered, he says, there is no coherent way for them to share strategies for improving instruction with other schools nearby.

Wishing to Learn Locally

PS 25 is a welcoming, well-kept building constructed in 1897, with tall windows that let in ample light. More than 90 percent of its 361 students are poor enough to qualify for free lunch; nearly two-thirds are classified as English Language Learners; and 20 percent receive special education services.Homelessness and chronic health problems such as asthma contribute to poor attendance. In her first year as principal, Toledo succeeded in raising average daily attendance by cajoling parents not to take children out of school for extended vacations and sending staffers to visit the homes of children who were absent. She assigned teachers who had vague, non-classroom assignments to work directly with children. "I asked one teacher what he did and he said, 'I monitor the book room,''' she recalls. "I said, 'The book room doesn't need monitoring,' and told him to work with children instead." She brought in a new reading program designed especially for children learning English and those having particular trouble learning to read. She offered assessments of each child every six to eight weeks. Many teachers welcome her collaborative approach. "Everyone has really jumped into her vision and supports her," says fourth-grade teacher Nancy Pacheco. "It's not like we are left to wander on our own."

The school has also benefited from a decision, made long before Toledo arrived, to phase out the upper grades. PS 25 previously served children in pre-kindergarten through eighth grade. Her first year, it served only pre-k through fifth grade. The stunning increase in the percentage of children reading at grade level was due, in part, to the fact that the scores no longer included those of the older students, whose performance was particularly weak.

Toledo knows her school still has a long way to go. Some of the older teachers resist change. A number of them have taught mostly in Spanish for years, and it's a challenge to develop coherent instruction that balances English and Spanish. Toledo knows other schools—including one that is just a block away—that have managed to build an effective corps of teachers to work successfully with children who live in poverty and who don't speak English at home. She yearns to learn from these schools. "What are they doing that we aren't doing?" she asks.

Just a block west of PS 25 is another school, PS 5, which serves a similar population and which has solved many of the problems that bedevil Toledo. When I visited in mid-December, every classroom was alive with engaged children and imaginative teachers, even late in the school day when most kids begin to drag. While the children at PS 25 were slow to move from one activity to another, those at PS 5 were consistently attentive. Children's work covered the walls, including a colorful paper timeline with dates of inventions such as the zipper, the automobile and the Band-Aid, a concrete history lesson the children had made themselves. Principal Mary Padilla, who works closely with Teachers College at Columbia University to improve children's writing skills, limits test prep to half an hour a day and offers children a curriculum that includes frequent essay writing, music, art, science, ballroom dancing and trips to museums and the zoo. While PS 5's test scores are just about the same as PS 25's—and both schools received an "A" on their 2009 Progress Report—PS 5's scores reflect a steady improvement since Padilla became principal in 2001, rather than a large one-year increase.

There is no easy way for Toledo and her staff to share the knowledge that Padilla and her colleagues just one block away have gained over the past decade. That's because PS 25 and PS 5 are in two different networks, groups of schools that were created when Klein dismantled regional offices in 2007. Rather than relying on a district or a regional office for support and guidance, principals now join with like-minded colleagues to form these networks, which offer help with everything from payroll to training new teachers. Each network chooses its own leader, who acts more like a coach than a boss. The networks are not geographically based, and may have schools in three or even four boroughs. Toledo's network has schools from the Bronx, Manhattan and Queens, while PS 5's network is made up of a completely different group of schools in the Bronx and Manhattan, as well as several as far away as East New York in Brooklyn.

So, while Toledo's teachers have the opportunity to attend their network's workshops in Manhattan on topics such as helping children improve their behavior, they have no opportunity to learn from the teachers of PS 5. Toledo says that although her network leader is a great source of support for many issues, the network has no roots in the South Bronx. When problems occur in the neighborhood—when a girl from another school was shot in the head at dismissal time, or when her own assistant principal was robbed outside the school—there is no easy way to connect with other schools nearby to share their strategies and concerns. "I'm in an organization that is disconnected from the community," Toledo says.

For her part, Padilla says she misses the regular contact she had with other principals when they were part of the same district. "I miss the collegiality," she says. "I miss getting together and saying, 'I had a rotten day. How was your day?'"

Architect of Empowerment

Separating schools in a neighborhood from one another is not an unfortunate byproduct of Klein's reorganization. Rather, it is central to his notion of how schools should be organized—and how schools learn to improve.

Klein's reorganization relies heavily on the work of Eric Nadelstern, his top deputy and a key architect of the empowerment structure. Nadelstern says that improvement must come from within a school building, not from outside sources. It is important for school leaders to decide their own management structure, curriculum and strategies for tough issues such as improving attendance, discipline, parent involvement and services for students with special needs.

It is a controversial notion. Some critics say the approach is inefficient because it forces each school to reinvent the wheel. For example, if someone finds a particularly good way to teach English to new immigrants or algebra to ninth graders, there should be a mechanism to spread that knowledge, says Garvey, the former CUNY administrator. "The notion that every teacher should be developing his own curriculum over and over again is ridiculous," Garvey says.

But Nadelstern responds that this is precisely the point. He says each group of teachers must figure out anew what works for their particular students. "If you don't give people the opportunity to reinvent the wheel, they don't have the opportunity to improve the wheel," he says. "People have to invent it for themselves and then they own it. It's that ownership that inspires them to do their best work."

Nadelstern says breaking up the districts and allowing principals to form networks without regard to geography has allowed schools in poor neighborhoods to work more closely with schools in middleclass or wealthy neighborhoods. That is the case for some schools in District 7. Ramón González, principal of MS 223, is part of a network that includes middle schools on the Upper East Side of Manhattan, some of the highest performing schools in the city. González says he has benefited from regular meetings with their principals on issues at which those schools excel, such as improving the quality of students' writing. At the same time, he been able to contribute his expertise to the group, based on his years of working with children in the South Bronx. For example, he has effective techniques for encouraging students' good behavior, and is knowledgeable about special education. "Behavior management is my specialty," González says.

Nadelstern says each school can and should be different. If one school has a quasi-military structure, with strict rules and severe discipline, and another has a relaxed atmosphere where kids wear hats in class and call teachers by their first name, that's fine, as long as both schools get results. While some network leaders say the current structure offers no continuity from elementary to middle school and from middle school to high school, Nadelstern says that isn't a significant issue because all schools must meet the same state standards that outline what children need to learn.

Nadelstern dismisses "professional development" for teachers as a waste of time. His favorite anecdote, often repeated, is about a staff development session he attended on the deck of the aircraft carrier Intrepid: "You've got a thousand people on the deck of the Intrepid listening to some poor schmuck at the front of the room, thinking about what they're going to do on the weekend. That's professional development." Above all, he sees traditional school districts as bloated bureaucracies that impede
rather than foster learning. The Bronx high school superintendent had a staff of 120 people supporting just 20 schools when Nadelstern worked there in 2001—and most of those schools were failing miserably, he says.

Nadelstern first experimented with the notion of school empowerment when he was principal of the International High School at LaGuardia Community College in Queens, a small alternative high school that was particularly effective in teaching English to new immigrants and preparing them for college. Always a maverick, Nadelstern was one of the first principals to apply for his school to become a charter school after the state legislature passed a law permitting such schools in 1999. But Nadelstern quickly became disillusioned with charter status when promised budget enhancements did not appear. In 2001, he returned his school to the Board of Education, but under different circumstances: A new budgeting formula developed by Harold Levy, Klein's predecessor as schools chancellor, gave each high school principal a lump sum of $653,000 to hire assistant principals, deans, guidance counselors or teachers. This money gave Nadelstern flexibility he did not have before and was a bonanza for his small school. He was able to hire 13 additional teachers for his 430 pupils.

"It was the best educational experience I or my staff or my kids have had," says Nadelstern, who is now chief schools officer for the DOE. He formed the germ of an idea that would fundamentally change the school system. "The lesson is that the money is better used at the school level: hiring more staff, reducing teacher load, buying better supplies and materials and enriching the curriculum."

Several months later, he left International High School to become deputy superintendent of the Bronx high school division, where he proceeded to replace large, dysfunctional high schools with several dozen new small schools sharing space in existing buildings. In 2004, he left the Bronx for DOE headquarters in downtown Manhattan, where he persuaded Klein to let him start what came to be known as the "autonomy zone."

The idea of autonomy was simple: Principals who were confident they could manage their schools better without reporting to a district office could apply to be in the zone. They would receive extra money and, as long as they met certain metrics in terms of student achievement, they would be left alone. In the first year, there were 29 schools in the zone. In the second, there were 48. By the third year, there were 321. Nadelstern presented data that persuaded Klein that the autonomy zone was more effective than the regional structures he had created in his 2003 reorganization. In 2007 Klein agreed, in essence, to expand the autonomy zone to include every school in the city.

Under this new structure, Klein gave principals unprecedented power over hiring and budgets so long as they met certain statistical benchmarks. Under the new system, elementary and middle schools were graded "A" to "F", based mostly on children's progress on standardized tests in reading and math compared with a group of schools with similar demographics. High schools were graded on how many students passed their classes each year, how many passed state Regents exams, and how many graduated on time. Schools competed against one another for a grade. The principal and teachers of schools that demonstrated the most progress would get thousands of dollars in bonuses, while schools with "D"s, "F"s or a string of "C"s three years in a row were in danger of being closed.

The advantage of this system, Nadelstern says, is to transfer power and responsibility from the districts to the principals and their staff. Accountability measurements accomplish what a district superintendent could not, he asserts. Superintendents were "constantly trying to force people to behave differently. Then teachers would close the door and do whatever the hell they wanted," he says. "What we're saying is: That's likely to always be the case. Let's take advantage of it in ways that acknowledge it, and say they are in the best position to make the decisions."

For principals, the freedom is greater, but the stakes are also higher. "Our job, really, is to find the best school leaders that we can find, invest in their autonomy, and hold them accountable for the results," says Nadelstern. "If we think they can't do it, let's get rid of them and put somebody in there who can. And if they can't do it, let's close the school and start over again."

This strategy has shown some success, and resourceful principals have used their freedom to create effective schools. But in District 7, at least, there's been a downside as well: In some schools there has been a rapid turnover of leadership without significant improvements in performance.

The Sad History of PS 156

The building that housed PS 156, for example, has had five principals since Klein became chancellor in 2002. It still struggles with uneven discipline, poor attendance and low levels of academic achievement.

Built on top of Conrail rail yards in 1970, PS 156 has a sad history. When I first visited the brown brick building in 2003, teachers complained of rats coming into classrooms from the cavernous, subterranean area below the school. A wheelchair-accessible building, PS 156 had a large number of medically fragile children, a large special education population, and many children in foster care or living in temporary housing. Absenteeism among both children and teachers was high, and teachers struggled to maintain order. During my visit, Principal Maxine O'Connor called an ambulance to take an angry girl who had ripped paper from a bulletin board to the psychiatric emergency room at nearby Lincoln Hospital; the principal didn't know how else to control the girl's behavior.

O'Connor took early retirement in 2005 and the regional superintendent replaced her with James Lee, a teacher from San Francisco and a graduate of the Leadership Academy, the city's one-year program for aspiring principals. Lee worked hard to improve staff morale, student discipline and the quality of instruction. When I visited again in the fall of 2006, the school was cleaner and more orderly. "He has a vision. He really cares about the kids," a teacher told me at the time.

Teacher absenteeism—a sign of low morale—had declined. Children played on the playground after lunch; on my previous visit, they had watched videos in the auditorium during recess because there wasn't adequate staff to supervise the playground. These improvements weren't reflected in test scores, however. Educators say test scores often decline for two years after a new principal takes charge, as teachers adjust to a change in instructional style. PS 156 was no exception. Its test scores went down in the spring of 2007, and the DOE gave the school an "F'' on its Progress Report. In December 2007, it announced that the school would be closed the following June.

Five months later, however, the work of Lee and his staff finally began to bear fruit. The test scores released in the summer of 2008 were still low, but the school had made substantial gains from the previous year. Based on that progress, Lee received a $7,000 bonus and his staff shared $351,000. However, the decision to close the school had already been made.

The education department removed Principal Lee, reopened the school with a new name, "The Performance School," and hired another principal, David Scott Parker. He was a teacher from Manhattan's Upper West Side and a graduate of the Leadership Academy. This was his first administrative job.

At the same time, the department turned over part of the building to a new charter school, the Bronx Global Learning Institute for Girls, led by Celia Domenich, a principal with several decades' experience on Long Island as well as in the Bronx. Parker inherited most of the old school's 600 students in kindergarten through fifth grade, and half of its staff. (According to the teachers' contract, new schools must hire 50 percent of their staff from the faculty of the school that is closing. Parker was permitted to pick any candidates he wanted for the remaining half.) The charter school, on the other hand, was able to begin from scratch with 100 kindergarten and first-grade girls, chosen by lottery, and newly hired teachers.

Perhaps not surprisingly, the experienced principal with the tiny enrollment of very young girls and hand-picked teachers had greater success than the brand-new principal with a large enrollment of older children and teachers demoralized by the closing of their school and the removal of half of their colleagues.

Teachers say discipline deteriorated and children ran willy-nilly in the corridors of the Performance School that year. "It went from bad to worse," says a teacher. Parker left after six months, replaced in January 2009 by Principal Lourdes Estrella. This time, the DOE installed someone with significant experience as principal of another high-poverty elementary school in the Bronx, PS 62. Like her predecessors, she faced an enormous challenge. Just 30 percent of teachers said order and discipline were maintained at the school and three-quarters said students were often bullied and threatened, according to the 2009 Learning Environment Survey, the DOE's annual polling of teachers and parents. (The results of this survey count toward 15 percent of a Progress Report grade; test scores account for 85 percent.) Just 36 percent of fourth graders were reading at grade level in 2009—one of the worst rates in the city. The chancellor's decision to close the school in 2008 and remove its principal appeared to have exacerbated rather than solved problems, at least in the short term.

"Closing the school might have been a good idea, but don't do it unless you have a good backup plan," says a teacher. After years of turmoil, the Performance School faced many of the same challenges PS 156 had eight years before.

Similar Schools, Dramatically Different Results

Klein's vision of a system of great schools (rather than a great school system) rests on the notion that principals—not superintendents—are best equipped to decide what their schools need. In accordance with this vision, Klein has charged principals with choosing (and paying for) any outside support they feel they need, such as training for their teachers or help with special education.

In 2007, Klein created several new School Support Organizations—some within the department, others managed by nonprofit contractors—to provide many of the services previously offered by district offices. No longer bound by geography, principals were invited to choose a support organization based on their educational philosophy, such as the back-to-basics approach favored by E.D. Hirsch and his Core Knowledge curriculum, or the progressive methods pioneered by Teachers College at Columbia. Principals who wanted a lot of support could choose an expensive package. Principals who felt they could manage without much help could choose the bare-bones "empowerment" package. (See "Principal Power Deconstructed," for more details on the DOE's district changes.)

In a cost-saving move, this support structure was modified in the spring of 2010. Instead of choosing different packages of support with different philosophies and prices, all schools were placed in groups of the same size—with the same price—called Children First Networks. These 60 networks, each with about 25 schools, have a staff of about a dozen people apiece to help schools with everything from payroll problems to curriculum development. (The nonprofits charged with supporting schools retain their roles, at least for the duration of their contracts.)

Still, the theory of change remains the same: Principals, not district superintendents, make all significant decisions, and principals call for help only when they want it. Not surprisingly, this theory is controversial, particularly given the fact that many of the city's principals are inexperienced. Indeed, some 80 percent have been hired since Chancellor Klein was appointed in 2002.

"It doesn't make sense to give everyone the same level of freedom, autonomy and empowerment," says Bill Colavito, head of the nonprofit Center for Educational Innovation-Public Education Association (CEI-PEA), one of the nonprofit organizations under contract to the DOE to provide support to principals. "There are principals who don't deserve to be empowered because they haven't demonstrated that they can run a school effectively."

Many of the new network leaders come from outside the ranks of the Department of Education, reflecting Klein's desire to reinvigorate his administration with new blood from different disciplines. Some network leaders have experience in education policy or law, rather than teaching, which has raised some eyebrows among those who believe experience counts.

"You have principals with little experience and you have support people with little experience," says Hal Epstein, a retired principal and network leader with schools in three boroughs. While a network leader who has never been a principal may offer valuable advice on matters such as payroll and instruction, experience as a principal is more important for issues such as how to deal with the teachers union or how to engage parents, he says. "How do you offer support to a principal if you have never walked in his shoes?"

The job of network leaders can be frustrating. If a principal ignores their advice, there is nothing they can do because the principal can change networks if he or she is unsatisfied. Nonetheless, network leaders are evaluated on the progress their schools make. "I've had schools that were getting 'C's and 'D's that would ignore my advice, but I was still held responsible for their success," says Epstein. Another network leader, who asked not to be identified, recalls a principal who continuously canceled appointments. "In the olden days, I would have twisted her arm and made it happen," this leader says. "Now, I swallow hard. She is calling the shots." There is a high rate of turnover among the network leaders; some schools in District 7 have had four different leaders in as many years. However, perhaps not surprisingly, principals, by and large, seem happy with the arrangement.

In District 7, some principals welcome the support that good network leaders can provide while others keep them at arm's length. Principal Pablo LaSalle, who led PS 161 for 14 years until his retirement in February 2010, was satisfied with his school's performance and didn't see any reason to ask for help. About 52 percent of his pupils met state standards for reading in 2009. But, because he had a very needy population and because most of his pupils made what the DOE considers at least a year's progress—that is, his fourth and fifth graders scored at least as high on their standardized tests as they had the year before—his school received an "A" on its report card and was ranked in the 86th percentile overall.

As an experienced principal, he felt he didn't need much supervision. "I wasn't looking for any more headaches," he says. He chose to join the network led by the former superintendent of District 7, Elvira Barone. At the time, Barone was a part of the Empowerment Support Organization, originally set up by Nadelstern and designed for principals who sought only minimal support, cost and interference. Barone's network maintains those values in the current Children First Network structure. "It's pretty much telling you [that] you are on your own," LaSalle says. "No one comes into the building unless the principal asks. Who is the last one to call? It's probably me."

LaSalle has many strengths. Parents say he welcomed each child with a handshake outside the school every morning and always had a coffee pot and a plate of doughnuts for parent volunteers. His Progress Report shows the lowest-performing children made significant gains at his school. Still, there is much room for improvement. When I visited the school, transitions from one activity to another were slow, and a lot of time was taken up by tasks such as taking children to the bathroom as a class. Some kids had their heads on their desks. There were no class discussions in the half-dozen classes I visited; indeed, there was little talking except for teachers giving instructions. A number of teachers sat at their desks reading to themselves, while their pupils quietly filled out worksheets.

LaSalle adopted a scripted reading program called Reading First and teachers relied on fill-in-the-blank and multiple-choice questions. "The children spend a lot of time on test prep," LaSalle acknowledges, although he hastens to add that they also study art and dance.

Under the city's new school governance structure, it is up to the principal to ask for help. If a principal, like LaSalle, is happy with the progress his school is making and can keep his test scores at an adequate level, no one will intervene. His network was made up of 22 schools in 13 districts in Manhattan, the Bronx and Brooklyn, and the schools have little opportunity to work together. He says his network offered little professional development to his teachers and, even if it did, he's not sure he would have sent them to another borough to take part in training. "It's tough sending people all over the place," he says. Neither does he have regular contact with PS 5 and PS 25, just a block away. As far as LaSalle is concerned, PS 161 is doing fine the way it is.

Other principals seek out networks that promise to help teachers become ever more skillful in their craft. Half a mile west of PS 161 is PS 277, a stately 100-year-old stone building decorated with impressive Greek columns and a cupola. Principal Cheryl Tyler chose as her network leader Dan Feigelson, who had worked closely with Teachers College and who she believed could help her staff improve the way they taught writing. For Tyler, principal empowerment has meant the freedom to not only choose a network, but also the curriculum and teaching methods she thinks are best for her pupils. "No one knows better what this school needs than the people working in the building," she says. At the same time, she believes in working closely with people outside the building: The principals in her network, which includes schools on Manhattan's Upper East Side as well as schools in the South Bronx, visit one another regularly to share ideas and strategies. And Feigelson, the former principal of PS 6 on the Upper East Side, comes frequently to offer guidance.

On a recent visit, Feigelson spent the morning with PS 277's third grade teachers discussing how best to teach punctuation. Punctuation, Feigelson told the teachers, was not a mechanical exercise, but a creative, important part of a writer's craft. Feigelson had written a book on punctuation in which he interviewed writers such as Jimmy Breslin and Frank McCourt. The teachers talked about how punctuation can be used to modulate rhythm, how it divides and connects at the same time, how it serves as a sort of stage direction to a reader.

Back in her office, Tyler said this kind of discussion—which takes teachers' work seriously and transforms a simple grammar lesson into an intellectually stimulating conversation—is one of the reasons PS 277 is able to attract and retain highly trained teachers, many with master's degrees from Teachers College. Tyler, who came to the school in 2006, believes that investing in teachers' continuing development will pay off in the long run.

But, unless her test scores improve quickly, she is vulnerable under the city's accountability system. Students reading at grade level leapt from 25.6 percent in 2008 to 49.8 percent in 2009—yet despite these gains, her school was ranked in the bottom fifth of the city's schools.

Tyler is committed to offering her children at least an hour of reading and an hour of writing each day, as well as science, social studies and a chance to play outdoors at recess. She wants to teach her children to research topics that interest them. One group of fifth graders, for example, researched crime rates in the Bronx after a classmate's mother was shot dead just outside the school. Test prep—the daily drill in reading short passages and answering multiple choice questions that some schools offer—is not on her agenda. "Giving an hour a day of test prep is not ethical," she says.

Tyler's and LaSalle's schools are roughly comparable: Both have high poverty rates, both have large numbers of children who speak Spanish at home. At both schools, about half of the pupils read at grade level, and about three-quarters made what the DOE considered a year's progress in reading in 2009.

But because of the complicated way in which schools are judged, LaSalle's school was ranked in the 86th percentile, near the top of city schools, while Tyler's school was in the 20th percentile, near the bottom. Although she received an "A" on her 2009 Progress Report, a "C" or a "D" on future reports could put Tyler's job in jeopardy.

I asked Robert Tobias, a professor of education at New York University who was formerly in charge of testing and accountability at the old Board of Education, to help me understand why such similar schools received such different scores. He looked at both Progress Reports and pointed out that LaSalle's children scored better than Tyler's in math. He also noted that the schools were in different "peer groups"—groups of schools with similar demographics which are the basis for the department's comparisons.

The Progress Reports are designed to rank schools within their peer group,in order to compare similar schools to one another. Even though Tyler's school has a higher poverty rate, higher rates of student mobility, and more children living in homeless shelters than LaSalle's, the DOE considers LaSalle's population more challenging. That's because the DOE doesn't weigh mobility or the rate of homelessness when it assigns schools to peer groups. Rather, it considers the number of children in special education and those still learning English, and LaSalle's school has more of those than Tyler's. (Depending on their disability or the length of time these children have been in the country, some of them may not be required to take the state tests, which makes the "peer index" an imperfect reflection of the challenges a school faces.)

While LaSalle has a high rate of referrals to special education, Tyler tries to keep children who are struggling in mainstream classes. Many experts agree with her philosophy, saying children should be labeled and segregated only if their disabilities are severe. But the Progress Report may penalize Tyler for her reluctance to refer children to special education because it assigns her a "peer index" that suggests she has a less challenging population.

"Small differences in performance and peer index are resulting in huge differences in percentile rankings," Tobias says. "At the end of the day, it is all based on test data which is flawed."

Problems with the Progress Report

The Progress Reports by which elementary and middle schools principals are judged rests heavily on what testing experts agree is an unreliable measure of school progress: the year-to-year growth in children's scores on two state tests, English and math, given every year beginning in third grade.

The issue is not that the tests themselves are flawed: Most experts agree these multiple choice exams are a rough but useful gauge of whether each child is able to read and understand short passages and to complete math exercises the state has determined are appropriate for his or her grade level. Nadelstern says the tests are useful because they can accurately predict who is likely to finish high school and who will drop out. A child scoring a low Level 3 in eighth grade—that is, one who is just meeting state standards for his or her grade level—has a 55 percent chance of graduating on time with a Regents diploma. One who scores a low Level 2, or below grade level, has less than a 10 percent chance and one who scores Level 4, or above grade level, has a better than 90 percent chance.

However, the tests cover only a small portion of what the state says children should learn. For example, the state "learning standards" for English Language Arts say children should learn to use a library, select appropriate books, speak clearly, express opinions, and write and revise their work using multiple sources of information. Examples of meeting these standards include delivering a campaign speech, writing a letter to the editor, reciting a favorite poem, performing a dramatic reading or writing a research paper using sources such as interviews, databases, magazines and science texts.

These are the skills, many educators say, that prepare children for high school and college, yet none of these skills are measured by the state's elementary and middle school tests. Under the city's current accountability system, a school that focuses exclusively on boosting performance on standardized tests and ignores all the other voluminous state standards—for English and math as well as music, art, science, social studies and physical education—may receive the same grade on the city's Progress Report as a school that works diligently to meet all the state standards. In fact, schools—like Tyler's—that try to teach all curriculum areas may actually be penalized, some experts say.

"We are judging schools on the basis of an impoverished view of what students are learning," says Henry Braun, a Boston College professor and former official at the Educational Testing Service, which administers the SAT and several other standardized tests. "Even worse than that, if we have teachers who take the content standards seriously, they may even be disadvantaged because they are teaching stuff that isn't represented on the test and maybe the kids aren't getting enough drill and practice on the low-level stuff."

Another difficulty is more technical. The state tests are designed for one purpose: to determine whether children have reached a benchmark—called proficiency—or Level 3, for reading and math in each grade. For that reason, most of the questions on the exam are designed to distinguish children who are just below the Level 3 standard from those who are within it, and therefore considered proficient for their grade level. Children may receive scores of Level 1 if they are far below grade level, Level 2 if they are approaching grade level, Level 3 if they are proficient at their grade level standards or Level 4 if they exceed those standards. However, only a few questions are easy enough for a child whose score is Level 1 and only a few are hard enough for a child whose score is Level 4. That means a lucky guess or simple misunderstanding on just one or two questions can move the scores of a low-performing (or high-performing) child up or down significantly from one year to the next.

The city relies on these tests to show year-to-year progress, but because the tests are designed to show proficiency—not year-to-year growth—they are inadequate tools for the task. "The tests weren't designed for that purpose," says Everson, the state testing expert based at the CUNY Graduate Center.

"It's unreliable for the low-performing kids, and it's unreliable for the high-performing kids," says NYU's Tobias. "Kids at the top of the scale will tend to go down the next year and kids at the bottom will tend to go up."

In schools with lots of low-performing students, like those in the South Bronx, this means test scores for schools are extremely volatile, and may go up or down for reasons unrelated to the level of children's learning. The problem is compounded at small schools, like PS 25, because the fluctuation in scores of just a few students can represent a large percentage change.

To complicate matters further, the state tests have been almost identical from year to year, and past years' tests are posted on the state Education Department website. That leaves it open to what testing experts call "score inflation." Whether or not test questions get easier, and whether or not anyone cheats, scores on a test that is identical from year to year tend to increase because it is so predictable and so easy to prepare for, says Tobias.

Daniel Koretz, a professor at the Harvard School of Education, maintains that score inflation is common with high-stakes tests—that is, tests that have serious consequences for teachers or students. "Worse, this inflation is highly variable and unpredictable, so one cannot tell which school's score are inflated and which are legitimate," he writes in his 2008 book Measuring Up. Everson says score inflation may be more pronounced in high-poverty schools than in schools in middle-class neighborhoods because there is more pressure on them to increase test scores.

In New York City, test scores rose so rapidly in 2009 that 97 percent of elementary and middle schools and 75 percent of high schools were awarded "A"s or "B"s on their Progress Reports because they met benchmarks set by the city the year before. The DOE recognized that these scores were inflated, and vowed to mark schools on a curve in 2010, with the number of "A"s capped at 25 percent. For its part, the state vowed to rescale the tests, essentially making them harder. But a long-term solution—which would involve making the tests less predictable and including a wider range of the material and subjects children are expected to learn—remains elusive.


Tweed's Test Reform Group

The conventional wisdom about Tweed is that lawyers and businessmen rule the system and that all decisions are made according to a corporate, data-driven mindset. Indeed, the philosophies of the businessman mayor, Bloomberg, and his lawyer chancellor, Klein, are the driving force of the school reforms of the past eight years, and their policies have led scores of career educators to retire or leave.

Yet the top ranks of Tweed actually include a number of former principals of alternative high schools who, before they took their current jobs, were among the city's staunchest critics of standardized testing. Before Nadelstern joined the department's leadership, he was perhaps best known for his lawsuit to force the state to exempt a group of alternative high schools from the state Regents exams. Shael Polakow-Suranksy, deputy chancellor for performance and accountability and the official in charge of the Progress Reports, was the founding principal of Bronx International High School, a sister school to Nadelstern's International School in Queens. Two of Nadelstern's deputies, Vincent Brevetti and Anthony Conelli, were alternative high school principals as well. While each of these officials has embraced the city's new accountability measures, each is well aware of the limitations of standardized tests. "You can get big swings in the data," Polakow-Suranksy acknowledges.

Nadelstern says the standardized tests don't capture many of the important things children should be learning. "Can the kids write? Can they calculate? Can they create? Can they do scientific experiments? Can they paint a masterpiece? Can they write a poem? Can they put on a play? Those are all artifacts of what kids learn," he says. "The problem is, we're not smart enough to figure out how to use that as a way of evaluating how well the school is doing and then compare it with other schools. We've got to get smarter about that."

So why does the DOE continue to close schools and award bonuses based on the results of test scores that don't capture the full range of what children should be learning and that fluctuate substantially from year to year?

Nadelstern says a school's grade on the Progress Reports is a large factor in the decision to close a school, but not only the one. Officials at central confer with the superintendent and the network leader of a school that receives a "D", an "F" or three "C''s before they make a decision. They also read the Quality Review, the annual report written by a superintendent or consultant based on a visit to the school. "One of the things we weigh is how long the principal has been there and whether the school has the capacity to change," says Nadelstern. Still, he defends the department's decision to close schools based on their grades. "You wouldn't want your child attending one of the schools that got an 'F'," he says.

For 2010, the city decided to change the way it interprets test scores for elementary and middle schools in an attempt to limit the volatility. Because of this volatility, more than half of the elementary schools saw their percentile rankings swing up or down by a total of more than 50 percentage points within three years.

Click chart to enlarge

To limit these year-to-year fluctuations, the DOE is making technical changes in the way it grades schools using a new formula called a "growth percentile model" that takes into account the fact that it is significantly easier to move children from, say, Level 1 to Level 2 than from Level 3 to Level 4. (See "Building a Better Yardstick") "We won't completely solve the volatility issue but it's a very significant change," says Phil Vaccaro, head of testing for the DOE.

Significant issues remain: 85 percent of the Progress Report grades for elementary and middle school still rely on two state tests for English and math which the DOE acknowledges assesses only a small portion of what the state believes children should learn. The high school reports measure what proportion of children are passing their classes and their Regents exams and graduating on time. But they don't measure whether children have had the broad education that will prepare them adequately for college. "It's not a perfect system and we want it to be better," says Polakow-Suranksy.

At PS 277, Principal Tyler focuses on helping her pupils as best she can without paying too much attention to the details of the Progress Report, such as the "peer index" that compares her schools to others based on data such as how many children receive special education services. She takes particular pride in the progress of a 9-year-old boy who was so angry that he threw chairs when he first started at PS 277. Rather than assigning him to a segregated special education class, as many principals would have done, she matched him to a particularly kind male teacher who worked hard to engage him. Now, if the boy gets upset, he simply leaves the room and walks down the hall to talk things over with a counselor. He's coming to school almost every day, reading books and getting along well with the other children.

Tyler boasts that her pupils read books whenever they get a chance, even as they wait on the playground for school to begin. "You see them out at lineup with their noses in a book," she says. She is philosophical about the Progress Reports, which ranked her school in the 20th percentile in 2009 and which may well give her a "C" or a "D" in coming years unless test scores show quick and dramatic gains. "If they learn to really love books, which they do, and they learn to really love writing, which they do, I won't let the letter grade define us as a community," she says.

Nonetheless, it's hard not to be concerned. "We all have some trepidation about the future," she says.