Helping Schools Measure and Support Their Students’ College Readiness: The Central Office Perspective

At the heart of the College Readiness Indicator Systems (CRIS) project is the work of staff within large, urban districts to not only develop indicators, but also connect those indicators with interventions and supports for students. Evaluating those student-based interventions and supports then sends signals back to the district about its effectiveness in ensuring all students are college ready. A year and a half into the CRIS work, VUE guest editor Jacob Mishook corresponded with site liaisons from two districts and a partnership support organization that oversees seventy-six schools in New York City to better understand the indicators themselves, and also the connections being developed between those indicators and how they are being used in schools and classrooms. The three site liaisons are Jamie Alter from New Visions for Public Schools in New York City, Shane Hall from Dallas Independent School District, and Marcy Lauck from San Jose Unified School District.

How is your college readiness indicator system team working to use college readiness indicators to identify students who need supports?

Jamie Alter: Over the last five years, New Visions has partnered with DataCation to develop and roll out a comprehensive student information system in our seventy-five schools. This system captures students’ progress beginning in the ninth grade, through high school graduation. In its current iteration, DataCation enables each school to inform and support instruction, professional development, curriculum planning, accountability, and policy.

New Visions’ college readiness indicator system incorporates three research-based components of an effective student early warning system: clear benchmarks, easy-to-interpret and actionable data reports that synthesize data for different audiences, and real-time data reporting. New Visions’ data work is informed by a set of clear benchmarks that provide the framework for understanding student performance and monitoring student progress. For example, the College Readiness Metric combines credit accumulation, core subject credit accumulation, Regents exam scores, and semester sequence – clearly indicating where a student needs to be on the graduation and college readiness trajectory over the course of eight semesters. We have also developed a set of tools (based on a set of clear benchmarks) designed for school staff, parents, and students. The School Snapshot, the Ninth-Grade Tracker, and the College Readiness Tracker convey data related to students’ progress to graduation in a visually accessible, user-friendly format at different levels of aggregation. Functionality within the DataCation platform integrates data from different New York City Department of Education (NYCDOE) data systems so that the data tools are always up-to-date and relevant. DataCation also allows schools to move from aggregated school-level reports to specific, data-rich student-level reports. The availability of real-time data at different levels of aggregation permits staff to monitor improvements or setbacks in student achievement and begin to identify patterns across time and within cohorts.

Shane Hall: An early “quick win” pertains to the College Knowledge dimension.¹ Important measures for our district at the school level include completion/submission rates for the Free Application for Federal Student Aid (FAFSA) and ApplyTexas.

ApplyTexas is a common application by which a student can apply for admission to any public college or university in the State of Texas, as well as participating private institutions of higher education. To provide campus principals and counselors with information about their schools, we created a report on our MyData Portal system to which principals, teachers, and counselors can log in and see their schools’ ApplyTexas completion rates, as well as how they compare with other high schools in Dallas Independent School District (ISD). Counselors reported that seeing how their schools compare with others fosters competition to raise ApplyTexas completion rates.

With FAFSA completion, Dallas ISD created another type of report on MyData Portal. Like the ApplyTexas report, campus-level users can view FAFSA submission and completion rates for their school compared to others. However, this report contains an added feature in which the user (usually a counselor) can download a spreadsheet of all students that shows their FAFSA completion status, or filter the data to capture only certain students, such as those with incomplete FAFSAs or who have not begun their application. This feature assists counselors by identifying students who need intervention in starting or completing their FAFSA. Dallas’s participation in the national FAFSA pilot enables weekly updates to these data, helping ensure timely information for counselors to use.

Our CRIS team also has developed a new tool that we hope will better link indicators to intervention. This Excel-based tool combines academic achievement metrics with survey-based measures of key cognitive strategies to better assess students’ academic preparedness. We hope to conduct a Year 2 study, piloting this tool in a limited number of middle and high school campuses, demonstrating the data and helping campus teams identify students in need of additional support.

Marcy Lauck: The CRIS work specifically supports the San Jose Unified School District’s (SJUSD) commitment to data-driven decision making. Our plan is to build on existing, good systems to reinforce behaviors that apply more resources to our students in most need. This is driven primarily by the publication of our Key Performance Measures (KPMs) that will both guide and hold the organization accountable for results.²

From your position, how have you seen CRIS indicators used at the school level (school leadership, teaching, and counseling staff)?

Shane Hall: School-level usage of the indicators has been somewhat limited overall. However, usage varies across campuses, with some schools using them more than others. Preparing for a new state assessment system (the State of Texas Assessments of Academic Readiness, or STAAR) has dominated the time and attention of campuses, pushing other concerns to the background.

Jamie Alter: The student data management platform DataCation comprises five portals with an array of tools enabling educators, parents, and students to track student progress on college readiness indicators and use the data to make key decisions. For example, administrators and school leadership teams use real-time, school-level data on students’ credit and Regents gaps to think through scheduling and programming strategies, tutoring and course recovery or course acceleration interventions, and instructional changes. Department and grade teams or individual teachers utilize the data tools to quickly drill down to student achievement at the individual level, identify students at risk, determine appropriate interventions and support, and continuously monitor improvement in student performance. Counselors use data about college readiness indicators to gather and communicate information on student progress, ensure proper programming for students, monitor interim progress, and monitor progress to graduation.

Marcy Lauck: This has been the heart of CRIS work in San Jose. At all levels, indicators are tied to student supports, and as cycles of inquiry are refined, we are seeing encouraging improvements in student outcomes from improved attendance, better and more precise identification of student needs, increased participation and closing of opportunity gaps in Advanced Placement (AP) or International Baccalaureate (IB) coursework, and increased GPAs for students benefiting from targeted, subject-specific interventions.

What are some examples of connecting indicators to interventions for students?

Marcy Lauck: There are as many as there are indicators. At the high school level, students missing six or more days of school and with a D or F in any core course – an academic tenacity “cut point” – were assigned to teacher mentors. “AP Potential” students have been identified and recruited for participation in AP coursework. And high school students’ gaps in college knowledge were identified in the College Readiness Survey and are being addressed by the college counselor in ninth-grade courses.

Jamie Alter: New Visions College Engagement and Career Readiness (CECR) staff supports schools in connecting indicators to interventions for students. Specifically, they are providing technical assistance to schools’ College Access Teams in using data from the College Success Snapshot to set school-level college readiness goals and identify strategies and external resources to reach those goals. Data from the snapshot has helped schools examine their allocation of personnel and identify the need for formalized college guidance, such as dedicated programming to support students through the postsecondary transition or a college advisory course that allows students to engage in postsecondary research.

We are also assisting teachers in co-examining College Readiness Trackers with students and parents during one-on-one meetings, using the data to set goals and develop an academic plan (including key interventions to reach those goals) and co-monitoring their progress over time. Finally, we are supporting grade teams in identifying almost-on-track students, implementing targeted interventions to address areas such as their credit accumulation deficits, and tracking student progress to college readiness over time.

Shane Hall: As I mentioned earlier, we are enabling high school counselors to download a spreadsheet of student-level FAFSA data, filtering by FAFSA completion status if they wish. Armed with this information, counselors can intervene, for instance, to help students complete their FAFSA applications or get applications started. This summer, Dallas ISD Counseling Services is participating in a summer college access study that examines two types of interventions – personalized support from a school counselor and a series of text message reminders – designed to help students complete necessary pre-enrollment tasks, such as financial aid applications, placement testing, and, where applicable, on-campus housing arrangements.

What is needed to make CRIS data more useful for school-level staff?

Jamie Alter: New Visions is seeking to expand and refine visualization tools that provide school-level staff with access to key longitudinal data that can inform instructional and structural decisions and deepen capacity-building around data utilization by, for example, developing case studies about a school’s effective data-use practices and providing staff with micro-level data and reports on student skill attainment connected to the Common Core State Standards.

Shane Hall: Many Dallas schools face a struggle to meet state and federal accountability standards. The new STAAR system, a program of more rigorous state assessments, only increases this pressure. As a consequence, meeting accountability standards often dominates the work at the campus level. One of the most important things the CRIS team can do with the data and indicators is to present them in such a way that schools see that college readiness is linked to, not separate from, these accountability concerns. Improved success on STAAR exams boosts college readiness, and improved academic preparation and tenacity improves college readiness and boosts performance on these assessments.

Is your site tracking interventions and supports for students? In what ways? What else needs to happen to have a better sense of the effectiveness of the supports provided to “off-track” students?

Marcy Lauck: As our CRIS schools pilot indicator development and related interventions for their at-risk students, juggling the many resources and connections among those who are invested in students’ success remains a highly complex and time-consuming process. As we scale the work beyond our pilot schools, we will need to address the most effective ways to systemically track interventions to keep everyone informed about the progress of students they support. To this end, the San Jose Unified School District is developing Key Performance Measures cross-functional study teams of fifteen to twenty participants to create action plans that will result in system momentum towards success on any given measure. The team will be co-led by senior managers within the organization and will meet approximately once a month. The district is also exploring the use of OnTrackEDU’s social networking platform to create a common understanding of the needs, status, and interventions for at-risk students to make more efficient use of available support resources, optimize existing programs, and share efforts and outcomes with other schools working on the same challenges.

Jamie Alter: We have developed a “Learning Framework,” aligned with Common Core, the Partnership for Assessment of Readiness for College and Careers (PARCC) assessment consortium, and the new teacher evaluation system that defines best practices, systems, and strategies that should be present in all schools to produce growth in student outcomes.

Based on longitudinal analysis conducted by the University of Chicago, the Learning Framework identifies five key areas of focus that will support schools’ improvement of student achievement: Rigorous Ambitious Instruction; Leadership; Strong Professional Capacity; Student-Centered Learning Environment; and Parent and Community Engagement. New Visions has built out strategies and action items in support of these five areas. New Visions’ knowledge management staff will then use the framework, together with a detailed analysis of school performance metrics and student outcomes, to judge and improve the effectiveness of specific elements of programs and interventions. Together with increasingly sophisticated data analysis, it enables us to define the “black box” of classroom instruction, making transparent the effective school practices that can then be replicated and scaled.

What successes, opportunities, and challenges do you see for your district’s college readiness indicator system?

Marcy Lauck: Our teams have done a great job! But our next step and biggest challenge will be in scaling the CRIS efforts to guide the overall KPM cycles of inquiry at all forty-two schools, including the standardization of data collection and reporting the use of data and goal-setting based on those measures’ cut points, and the monitoring of implementation strategies for interventions and supports based on the data analysis.

Jamie Alter: Much of the data in DataCation provides insight into a student’s achievement at a single moment in time. New Visions is beginning to expand its thinking around student data to systematically look at the impact of strategies, teachers, or classes over time. Specifically, New Visions’ longitudinal data work helps the organization understand students’ needs, challenges, and progress over the course of their four years in school and how this intersects with school structures. New Visions’ “stock and flow maps” aggregate this longitudinal data at the school level to understand how aspects of a school or program contribute or impede students’ growth and ultimate success.³

As we examine these maps, patterns emerge that suggest the root causes of visible outcomes and provide insight into areas needing improvement in schools’ operations, staffing, and structure. We have also recently developed a College Success Snapshot that includes a range of college knowledge and postsecondary success indicators for graduates of New Visions’ Partnership Support Organization schools. The College Success Snapshot is connected to the bottom line issues for high schools: graduation, postsecondary readiness, and college enrollment, and points to successes and areas for improvement in high school academic curriculum, and guidance and college access support services for students. The goal of this tool, and the corresponding theory of change, is that access to key pieces of postsecondary data will drive school-level decision making to inform positive postsecondary outcomes – including college enrollment and completion – for students.

New Visions has created a host of tools and supports to measure and foster academic preparedness and college knowledge. However, we have not focused on developing and tracking a set of academic tenacity indicators. Our CRIS work seeks to deepen our understanding of student academic tenacity, particularly students’ ability to regulate their progress toward mastery on the Common Core State Standards.

Shane Hall: The strength of Dallas Independent School District’s CRIS is twofold: first, the early research we have done in college readiness and postsecondary enrollment and completion patterns provides us a picture of our graduates’ level of college preparedness, as well as their levels of enrollment in and graduation from college. Prior to receiving the CRIS subaward, we completed an eleven-year longitudinal study of college enrollment and completion by our graduates. We further developed a College Readiness Measurement Model, which included indicators on content knowledge, cognitive strategies, academic behaviors, and college skills and awareness. It identified indicators in which we already possessed the needed data, as well as metrics in which additional data were needed. In short, Dallas ISD put a lot of thought into college readiness indicators.

Our second strength lies in the wealth of data we have as a district that provide valuable insight into our students’ college readiness. Our MyData Portal contains extensive academic achievement data for measuring academic preparedness. We have bolstered these data with our College Readiness Survey, which includes measures related to key cognitive strategies, academic tenacity, and college knowledge. Dallas ISD also participates in the national FAFSA pilot, giving us ongoing access to FAFSA completion and submission details on our students. We successfully completed a three-year interlocal agreement for data sharing with the Dallas County Community College District, where many of our graduates enroll upon graduation from high school.

Dallas ISD has developed reliable indicators of college readiness and is working to develop others. However, the biggest challenge we face is connecting these indicators to the appropriate set of student supports and interventions that will help improve student college readiness. A second weakness is developing reliable setting-level indicators for academic tenacity. This dimension of college readiness remains the most elusive to measure, especially at a setting level. We are continuing our efforts to find better measures of tenacity, then find appropriate actions to foster student tenacity.

By the end of the CRIS-funded work, we hope to have a full set of indicators at the individual, setting, and district level for our College Readiness Measurement Model. We also hope that collaboration with campuses will help us identify appropriate interventions for different indicators. Support from district-level administration, which supervises principals, is essential. With a new superintendent, who has made college and career readiness a centerpiece of his improvement plan for Dallas ISD, we have an opportunity to maximize support for these indicators and related interventions.