By design, EQA reports offer an impartial and objective review of the management practices in the district as they impact classroom practice and student performance. These reports discuss the quality and effectiveness of the all aspects of school business including: leadership and governance, curriculum and instruction, student academic support services, access and participation, business and financial management.

Because the EQA team has no connection to any party within a district, candid interviews can be conducted at all levels within the district including: the superintendent, assistant/deputy superintendents, business manager, directors, principals, district-wide program coordinators, school committee, municipal officials, groups of teachers, the president of the local teachers’ association, and parents on school councils. This unprecedented access to district personnel allows examiners to collect extensive evidence (corroborated by personnel at various levels of the organization) and present it in a report.

The report can then be used by the district to assist it with its future improvement initiatives.

The legislation which created this process expects all districts and charter schools to be examined for their effectiveness and efficiency in utilizing local and public funds to promote student achievement (as measured by the MCAS tests). K-12 public education is the single most expensive financial undertaking in the Commonwealth amounting to more than $5 billion annually. There is great public interest in how these monies are spent and the results of those efforts.

In addition, a review of this nature is highly beneficial to a district. The interviews, discussions and the report generated are great tools that will help even a highly performing district improve further.

The Office of Educational Quality and Accountability was created by the Legislature in July 2000 to perform independent audits of school districts in Massachusetts. The agency uses a process to assess the effectiveness and efficiency of each school district’s administration in advancing student performance as measured by the MCAS tests.

In accordance with its enabling legislation, the EQA process is designed to do the following:

• provide a comprehensive evaluation of a school district’s performance;
• provide annual reports on student performance in districts selected for review;
• monitor statewide public education performance to inform policy decisions; and
• provide the public access to information that helps it hold districts and schools, including charter schools, accountable.

Participation in this review program is mandatory. There is no cost passed on to the district to conduct this review. These reviews are conducted by experienced, highly trained educators (typically former senior district administrators) currently working for EQA.

A NEASC review on the other hand, according to its website, “relies on a voluntary, peer review process... Each [review] relies on a 12- to 18-month self-study process undertaken by schools and colleges in regular review cycles. The goals are school effectiveness, improvement and public assurance. The primary focus of this process is secondary (9-12) schools. Unlike popular trends, this process does not involve ranking institutions, but rather, establishes a level of acceptable quality for all accredited institutions.” Participation in this review program is voluntary. There is a significant cost passed on to the district to conduct this review. These reviews are conducted by ad hoc groups of experienced educators currently working in member districts. Being ad hoc, there is little chance of report comparability, except by chance.

The CPR carried out by the Massachusetts Department of Elementary and Secondary Education (DESE) oversees local compliance with the federal education requirements of the Elementary and Secondary Education Act (ESEA) which assigns federal entitlements by title (Title I-XI). This is focused primarily on issues related to: special education, civil rights, Title I reading and math assistance programs, Safe and Drug-Free Schools, nutrition programs and services, etc. Primarily, CPR process is compliance driven and designed to ensure that school districts adhere to the laws and regulations that govern the program titles carried out within their educational programs. Participation in this review program is mandatory. There is no cost passed on to the district to conduct this review. Each school district is expected to file a response and when necessary a remediation plan with the DESE. These reviews are conducted by staff currently working in the Program Quality Assurance Services (PQA) group of the DESE.

Chart showing a comparison of EQA, NEASC, and CPR reviews

In the spring of each year districts are notified if they are going to receive an on-site review during the upcoming school year. EQA makes every effort to accommodate the needs of districts while scheduling its on-site reviews. EQA does take into account whether a district has been previously scheduled for a Coordinated Program Review or a NEASC accreditation visit. If a district is already scheduled for either of these types of review the superintendent will be given the option to have the EQA review rescheduled so as not to conflict with the existing review. The district may also opt to have the reviews take place closer together or at the same time as other reviews.

EQA is charged by the Legislature to review all districts regularly (EQA mandate). There are 328 operating school districts in the Commonwealth, excluding charter schools. While the statute requires at least 24 districts to be reviewed annually, EQA has the capacity to review approximately 40 to 50 per year. As a result, EQA expects to review every district at least once every six to seven years.

Data relevant to district performance are reviewed annually for all districts. This includes factors such as competency determination attainment rates, attendance rates, MCAS performance data, MCAS participation rates, and adequate yearly progress data. A further on-site review may be scheduled if particular anomalies are found in the data or if certain trends begin to emerge.

Depending on the state of its document and data archives, a district will have to spend an amount of time preparing for the on-site visit. The tasks involved in preparing for the review include:

• assigning a liaison to communicate with EQA;
• gathering documents specified in the Document Checklist;
• selecting appropriate staff to be interviewed by EQA examiners;
• working with the EQA coordinator to prepare the on-site interview schedule; and
• scheduling spaces and facilities for use by the visiting EQA team while on site.

You should direct your questions to:

EQA Operations Center
65 South Street
Suite 104
Hopkinton, MA 01748

Phone: (508) 435-5126
Fax: (508) 435-5249

A liaison will need to be assigned to communicate with EQA. This person should be a member of the district’s leadership team. The liaison should be someone who understands the district operations and is able to schedule people and facilities for interviews.

At EQA we understand that each district has its own set of unique circumstances and challenges. As a result, in most instances we are not looking for a particular practice or methodology. It is for this reason that EQA visiting teams are composed of veteran educators. Overall, the EQA process looks for data-driven practices, effective management, and improved achievement of all students.

Throughout the growth of EQA, the executive director has been focused on providing districts, the Legislature, and the public with the most accurate and relevant information possible regarding the operation and performance of schools. To accomplish this, a great deal of attention has been paid to the substance and format of the standards and indicators.

Over the EQA's history the standards and indicators have been modified almost annually. A close analysis of the various iterations of the standards shows that they are still focused on the same areas as they were initially. These are (leadership and governance, curriculum and instruction, student academic support services, access and participation, business and financial management) There is a clear line between the original standards and the ones that are in use today. The most significant changes have occurred in the organization of the information and not to the substantive areas being measured. EQA tries to model the behavior it looks for in districts – data-driven decision-making. As a result, changes have been made to best address the issues found in the Commonwealth's school districts.

Each summer all aspects of the EQA process are reviewed by core groups of the EQA examiners. They review and update the standards and indicators as necessary. Any changes are then forwarded to a wider group of examiners and staff for further critical review. Finally, any alterations are forwarded to EMAC for final review. Once approved, the revised standards and indicators are sent to districts scheduled for review.

Documents are provided to EQA to be reviewed prior to the on-site visit. This allows examiners to get a ‘snapshot’ of the district prior to arrival on site. The documents that are required are specified in the document checklist. This checklist should be interpreted broadly as EQA understands that in most cases no statewide standard exists for the organization of information or the naming of documents. The expectation is that the district will not need to create any new documents especially for the visit, with the exception of the five attachments provided with the checklist.

As specified on the document checklist, please compile the documents for the school year preceding the school year of the scheduled visit, except where noted otherwise, for review by EQA prior to the site visit. That is, if your review is scheduled for the 2007-08 school year, the relevant documents would be for the 2006-07 school year. Electronic submissions (via our website or on CD) are welcome, otherwise they should be submitted in file folders, labeled by the document number (see example). If a document is unavailable or does not exist just indicate this. In addition, you should either compile in advance, locate, or be prepared to provide similar documents for the 2004-2005 and 2005-2006 school years to the examiners during their site visit.

Documents should be mailed or shipped to:

EQA Operations Center
65 South Street
Suite 104
Hopkinton, MA 01748.

Click here if you have further questions about completing the attachments, or call us at (508) 435-5126.

The directions for completing each attachment can be found at the top of each sheet. They should be interpreted broadly as EQA understands that in most cases no statewide standard exists for the organization of information.

Here are some simple pointers to help complete each of the attachments:

Attachment A – Policy Checklist: In districts where the National School Board Association/National Educational Policy Network (NSBA/NEPN) classification system is used to organize policies, just provide all policies from each of the sections listed. In districts where an alternate system is used, either provide policies thought to be relevant to each of the areas on the checklist or just provide a copy of the entire policy manual.

Attachment B - School District Teacher Licensure Survey: To complete page one of this attachment, just fill in each of the boxes on the table provided. The questions should be fairly self explanatory and refer to staff employed during the current school year. Clarification can be found on the Department of Education website regarding “Highly Qualified” for paraprofessionals.

Sample of a completed form

Page two contains an optional survey which allows EQA examiners to more fully understand the state of licensure of district staff. Once again, just fill in each of the boxes on the table provided. The questions should be fairly self explanatory and refer to staff employed during the current school year. The final column asking for numbers of vacant positions should be just an estimate.

Sample of a completed form

Attachment C – Report of Instructional Staff Attendance: Teacher attendance information should be completed for each school in the district. Definitions are provided for each of the categories. FTE refers to Full Time Equivalent.

Sample of a completed form

Attachment D – Report of Student Learning Time: As indicated, core content areas refers to ELA, math, science, and history/social studies. The data input into the table should reflect the last completed school year, i.e. 2006-2007. Given that individual student schedules may vary widely due to instructional cycles, remediation, or electives, EQA asks that you input information for a ‘typical’ student. When calculating the average add up the number of hours in one instructional cycle (this may be a week, month, semester, etc.) and then divide the total by the number of days in that cycle.

Sample of a completed form

Attachment D1 – Secondary Level Student Learning Time: Use the same information as listed above for Attachment D.

Sample of a completed form

Attachment E – Facilities Inventory: To complete this attachment just fill in each of the boxes on the table provided. The column headings should be fairly self explanatory and refer to facilities used during the current school year.

Sample of a completed form

If you have further questions about completing the attachments call us at (508) 435-5126.

As specified on the Document Checklist please submit the required documents electronically or in file folders, labeled with the document number. If a document is unavailable or does not exist just indicate this. Please plan to get your documents to the EQA office no later than two weeks before the scheduled site visit.

Documents should be mailed, shipped, or dropped off in person to:

EQA Operations Center
65 South Street
Suite 104
Hopkinton, MA 01748.

If you have further questions feel free to call us at (508) 435-5126.

As part of trying to fully understand a district’s management and operations, EQA reviews documentation and data for the previous three school years. This means that currently district reviews cover the time period from September 2004 through June 2007. Greatest attention is given to the final year of the examination period, i.e. September 2006 through June 2007. EQA examiners will look at initiatives currently in action if the planning for them took place in the previous school year.

The EQA review process is looking for thoughtful management of all aspects of district operations. This can only be done by considering efforts over an extended time period as detailed above, and any changes in student performance results.

An EQA visiting team is typically composed of five to seven examiners. These examiners were selected based on their extensive district experiences. Prior to working with EQA, most examiners held senior positions in Massachusetts public schools such as superintendent of schools, assistant superintendent, business manager, director of curriculum, principal, etc. These examiners have varied backgrounds in urban, rural, regional, or vocational education. Once selected by EQA, examiners are given rigorous training and when ready are sent on reviews as an observer to see experienced examiners in action. Most examiners complete five to eight reviews per year, participate in a spring symposium, and receive additional professional development during the summer. Each of the visiting teams is individually constructed with the specific district in mind and typically has approximately 150 combined years of experience in public education. All districts are notified ahead of time as to the membership of the visiting team. The superintendent will also be provided with biographies of each of the examiners detailing his/her experience and background.

The EQA team will require some simple items to successfully complete the on-site review:

• a private workspace (such as a conference room or classroom);
• access to a phone, internet, copier, etc.;
• spaces for interviewing district personnel, which can be done in a conference room or a staff member’s office if space permits;
• access to staff as needed;
• access to documentation as needed; and
• access to all facilities and classrooms as needed.

The on-site visit typically lasts for four days, and during that time a large number of interviews, school visits, and class visits take place. As a result, it is important that the schedule is developed very carefully. The interview schedule is developed in a collaborative process between the liaison in the district and the coordinator at EQA. Initially the coordinator will review the process and discuss the factors that require specific attention (such as interviews with school committee members, town officials, teachers, etc.). The liaison will then use the schedule template to set up the specific on-site schedule. The template contains blocks for all of the interviews that the EQA team will need to conduct during its visit as well as explanatory notes to help you. REMEMBER - the template is just a guide, in most instances the blocks can be rearranged to suit the needs of your district. Once all blocks have been filled, you can submit the completed schedule to your EQA coordinator for review.

The EQA review process is primarily a survey of management practices within the district. As a result, the district should expect for EQA to meet with the majority of its administrators including the superintendent, assistant/deputy superintendents, business manager, directors, principals, and district-wide program coordinators. Additionally, interviews will take place with the school committee and municipal officials. Finally, EQA examiners will meet with groups of teachers and the president of the local teachers’ association, as well as parents on school councils.

Interviews will be conducted for each of the six standard areas listed in the 2007-2008 Standards and Indicators. For each interview the district should select two to three people who are most knowledgeable about the standard area in question. It is preferable to assign staff with strong working knowledge of the area in question and not just a senior administrator with overall responsibility. EQA strongly discourages interviews to take place where staff are assigned with their direct supervisor.

Once scheduled, interviewees should not be substituted unless cleared with the EQA coordinator to ensure adequate representation of roles and responsibilities.

Interviews are typically scheduled for 90 minutes in length. Each interview will be conducted by one of the EQA examiners. The interviewer will be accompanied by at least one other EQA examiner who will serve as a scribe. For each scheduled interview the district will have selected two to three people who are most knowledgeable about the standard area in question.

The interview will begin with a brief introduction, taking of attendance, and a statement of the objectives for the interview. The interviewer will then ask a series of focused questions. During the interview the examiner will be trying to determine what has been taking place in the district, what is its quality, for how long it has been in place, and how widespread the particular practice has been. As examiners concentrate on particular details they are ‘drilling down’ in order to fully understand what has taken place and be able to accurately report the information back. Interviewees should be aware that it is not a problem if they are unable to address a particular question or line of inquiry. They should feel free to suggest an alternate interview candidate or point the examiner towards another document that may contain the information required.

Interviewers will summarize facts as they go through the interview. This provides the interviewees a chance to confirm that the information is accurate.

To try to maximize efficiency, interviewers may deviate from the scope of the indicators associated with the standard listed on the schedule.

All schools’ and districts’ annual MCAS results are subjected to an intensive analysis aligned with the requirements and standards of DOE regulations and federal statutes from the ‘No Child Left Behind’ legislation of 2001.

An analysis is made of several sets of data for each district. This includes competency determination attainment rates, attendance rates, MCAS performance data, MCAS participation rates, and adequate yearly progress data. These data are disaggregated further to show performance of the district over time, of the district relative to state averages, for schools relative to each other, and for subgroups relative to each other over time.

The EQA review of the MCAS results is framed by five essential questions:

1. Achievement: To what extent is the MCAS performance of all the district’s students meeting or exceeding state proficiency standards?
2. Equity of Achievement: How does MCAS performance vary among the district’s student subgroups?
3. Improvement: How has MCAS performance for all students in the district changed over time?
4. Equity of Improvement: How has MCAS performance for the district’s student subgroups changed over time?
5. Opportunity and Access: Are all eligible students attending and participating in all required programs and assessments?

Data are provided to EQA directly from the Massachusetts Department of Elementary and Secondary Education. These are then analyzed by EQA, with additional statistical analyses conducted on EQA’s behalf by the Merrimack Education Center and Cape Ann Economics.

After the EQA team leaves the district each examiner writes a report on the particular standard area he/she was the lead on. This report details how the district performed relative to each of the indicators contained within the particular standard. Approximately one week later the team reconvenes for a corporate session where each examiner presents his/her report to the group. These draft reports are subjected to intense critique and scrutiny by the team. Edits are made based on the consensus opinion of the group.

The report is then forwarded to the EQA’s technical writing team, which subjects the report to further editing. A contextual background piece and data section are added, an executive summary is prepared, and the report is then forwarded to the executive director for final review. The report is then sent to the superintendent of schools for a review factual accuracy. Final edits are made and the report is sent to the EMAC. In some cases further edits may be made. Once EMAC has accepted the report it is released and copies are sent out in accordance with M.G.L. Chapter 15, section 55A.

The draft report is sent to the superintendent of schools. This is not yet a public document and should be treated as such. EQA leaves it to the superintendent’s discretion as to whoever else on the staff should read the draft. The draft should be reviewed for factual accuracy. For instance, if the report states that the high school enrollment is 590 when in fact it is 950, this should be corrected. The district response should contain any proposed edits and should be submitted to EQA in writing (as hard copy and electronically). The response should detail which standard and indicator is being addressed, quote the section that is thought to be in error, and correct the factual detail. The district is expected to enclose specific documentation to support its claims, and this should be referred to directly in the response. The district has ten working days from initially receiving the report to submit any responses to EQA.

Responses should be sent to:

Dr. John J. Aherne
Interim Executive Director
65 South Street, Suite 104
Hopkinton, MA 01748

Phone: (508) 435-5126
Fax: (508) 435-5249

Email: eqa.mass@state.ma.us

Once reports have been approved by EMAC, EQA’s board, they are placed on the EQA website in the Technical Reports section. These are organized alphabetically and show the year in which the districts were reviewed. In addition, printed copies sent to the superintendent of the district, the local legislators, and the state library.

A report is generated following the examination visit and after review for factual accuracy by the district. The report is then scheduled for review by EQA’s board, EMAC.

EMAC makes recommendations based on a review of the report and discussion at its meeting. The possible recommendations are: accept the report with special commendations for the district; accept the report with no further action; accept the report and recommend that a management letter be sent to the district to express concerns; accept the report and place the district on 'watch' status; accept the report with a recommendation that the state Board of Education declare the district underperforming; or reject the report and request modifications be made.

The EMAC will transmit its findings and any resultant recommendations to the Governor; the state Board of Education; the Attorney General; the President of the Senate; the Speaker of the House of Representatives; and the Clerk of the House of Representatives who will forward the same to the Joint Committee on Education, Arts, and the Humanities. The report is released and copies are distributed in accordance with M.G.L. Chapter 15, section 55A.

Dr. John J. Aherne
Interim Executive Director
65 South Street, Suite 104
Hopkinton, MA 01748

Phone: (508) 435-5126
Fax: (508) 435-5249

Email: eqa.mass@state.ma.us