Study Abstract

Systemic Research conducts educational program evaluation and assessment using a systems analysis approach. Based on the program mission statement, goals, objectives, and implementation plan, we identify the most effective performance measures and evaluation strategies to assess progress. We design and develop a key indicator database management system to collect both quantitative and qualitative data components according to the framework of the Government Performance Result Act. We provide technical assistance to our clients, as well as maintain a continuous open communication channel. For large-scale longitudinal studies, our multidisciplinary team conducts site visits, surveys for target populations, case studies and focus groups. Research reports, along with electronic publications and CD-ROM versions, are tailored and disseminated according to our client's specific needs.

The study team will develop a Key Indicator Data System (KIDS) to collect/compile annual core data from the baseline year using both quantitative (K-1) and qualitative templates (K-2). Our study team will conduct an annual KIDS workshop designed for USI core data managers and evaluators to enhance data integrity and share expertise. Based on collaborative research agreements, we will receive SAT-I, AP, and ACT test results (as part of K-1 data) directly from the Educational Testing Service, the College Board, and ACT, Inc., respectively, for the next three years.

Each year, the team will conduct four site visits including interviews with USI leaders, focus groups, and school visits, to further explore detailed implementation issues. In addition, a modified version of the Council of Chief State School Officers' Survey of the Enacted Curriculum will be administered to 80 selected math and science teachers within these four sites. Finally, we will gather and review information from national databases such as the Third International Mathematics and Science Survey (TIMSS), National Assessment of Educational Progress (NAEP), Goals 2000, Equity 2000, as well as the USI sites' annual reports and relevant documents.

The team has identified key research questions in eight areas mapped to the NSF drivers: (1) student outcomes, (2) curriculum and instruction, (3) assessment instruments, (4) professional development, (5) policy, (6) leadership, governance, and management, (7) convergence of resources, and (8) broad-based support for reform. We will use a variety of statistical analysis methods to validate the hypothesis that a well-implemented USI program has a positive impact on student outcomes. Further systemic analysis will explore the determinants of successful implementation of urban reform in mathematics, science, and technology education.

During the three-year study period of October 1998 to September 2001, we will produce a USI Annual Fact Book and USI Evaluative Study Report, both in print and in CD-ROM format, and a master database to be available on the world wide web. Beginning in Year 2 we will publish a series of newsletters and monographs detailing our research findings.

Our research team has expertise in systemic education reform, assessment, evaluation, academic research, and core data collection and analysis. A research team of three co-Principal Investigators, all of whom have been involved in the NSF systemic reform effort, will be led by Dr. Jason Kim (PI) of Systemic Research, Inc. An advisory committee with representation from other major national reform initiatives will provide further review and consultation in order to situate the proposed study in the national context.