Inside IES Research

In 2018, NCER funded a research network to build the evidence base for career and technical education (CTE). As with other research networks, the CTE Research Network comprises a set of research teams and a lead team, which is responsible for collaboration and dissemination among the research teams. On March 20, 2024, the Network held a final convening to present its findings to the field. In this blog, Network Lead Member Tara Smith, Network Director Kathy Hughes, and NCER Program Officer Corinne Alfeld reflect on the success of the final convening and share thoughts about future directions.

Insights From the Convening

An audience of CTE Research Network members, joined by educators, administrators, policymakers and other CTE stakeholders, gathered for a one-day convening to hear about the Network’s findings. Several aspects of the meeting contributed to its significance.

• The presentations highlighted an important focus of the Network – making research accessible to and useable for practitioners. The agenda included presentations from four Network member researchers and their district or state partners from New York City and North Carolina. Each presentation highlighted positive impacts of CTE participation, but more importantly, they demonstrated the value of translating research findings into action. Translation involves collaboration between researchers and education agency staff to develop joint research questions and discuss the implications of findings for improving programs to better serve students or to take an innovative practice and scale it to other pathways and schools.
• Brand-new and much-anticipated research was released at the convening. The Network lead announced a systematic review of all of the rigorous causal research on secondary-level CTE from the last 20 years. This is an exciting advancement for building the evidence base for CTE, which was the purpose of the Network. The meta-analysis found that CTE has statistically significant positive impacts on several high school outcomes, such as academic achievement, high school completion, employability skills, and college readiness. The review found no statistically significant negative impacts of CTE participation. The evidence points to the power of CTE to transform lives, although more research is needed. To guide future research, the review provided a “gap analysis” of where causal research is lacking, such as any impacts of high school CTE participation on academic achievement in college or attainment of a postsecondary degree.
• National CTE leaders and experts put the research findings into a policy context and broadcasted its importance. These speakers commented on the value of research for CTE advocacy on Capitol Hill, in states, and in informing decisions about how to target resources. Luke Rhine, the deputy assistant secretary of the Office of Career, Technical, and Adult Education (OCTAE) said, “The best policy is informed by practice [...] and the best practice is informed by research.” Kate Kreamer, the executive director of Advance CTE, emphasized the importance of research in dispelling myths, saying that “if the data are not there, that allows people to fill the gaps with their assumptions.” However, she noted, as research increasingly shows the effectiveness of CTE, we must also guard against CTE programs becoming selective, and thus limiting equitable access.

New Directions

In addition to filling the critical gaps identified by the Network lead’s review, other future research questions suggested by researchers, practitioners, and policymakers at the convening include:

• How can we factor in the varied contexts of CTE programs and the wide range of experiences of CTE students to understand which components of CTE really matter?  What does it look like when those are done well?  What does it take to do them well? Where is it happening?
• How can we learn more about why students decide to participate in CTE generally and in their chosen pathway? What are the key components of useful supports that schools can provide to help them make these decisions?
• How do we engage employers more deeply and actively in CTE programs and implement high quality work-based learning to ensure that students are acquiring skills and credentials that are valued in the labor market?
• What are evidence-based practices for supporting special student populations, such as students with disabilities, or English language learners?
• How can we harness state longitudinal data systems that link education and employment data to examine the long-term labor market outcomes of individuals from various backgrounds who participated in different career clusters or who had access to multiple CTE experiences?

While IES alone will not be able to fund all the needed research, state agencies, school districts, and even individual CTE programs can partner with researchers to study what works in their context and identify where more innovation and investment is needed. The work of the CTE Research Network has provided a good evidence base with which to start, and a good model for additional research that improves practice and policy. Fortunately, the CTE research field will continue to grow via the support of a new CTE Research Network – stay tuned for more information!

This blog was co-written by CTE Network Lead Member Tara Smith of Job for the Future, CTE Network Director Kathy Hughes of AIR, and NCER Program Officer Corinne Alfeld.

Questions can be addressed to Corinne.Alfeld@ed.gov.

With rapid technological advances, the U.S. labor market exhibits a growing need for more frequent and ongoing skill development. Community college noncredit career and technical education (CTE) programs that allow students to complete workforce training and earn credentials play an essential role in providing workers with the skills they need to compete for jobs in high-demand fields. Yet, there is a dearth of research on these programs because noncredit students are typically not included in state and national postsecondary datasets. In this guest blog for CTE Month, researchers Di Xu, Benjamin Castleman, and Betsy Tessler discuss their IES-funded exploration study in which they build on a long-standing research partnership with the Virginia Community College System and leverage a variety of data sources to investigate the Commonwealth’s FastForward programs. These programs are noncredit CTE programs designed to lead to an industry-recognized credential in one of several high-demand fields identified by the Virginia Workforce Board.

In response to the increasing demand for skilled workers in the Commonwealth, the Virginia General Assembly passed House Bill 66 in 2016 to establish the New Economy Workforce Credential Grant Program (WCG) with the goal of providing a pay-for-performance model for funding noncredit training. The WCG specifically funds FastForward programs that lead to an industry-recognized credential in a high-demand field in the Commonwealth. Under this model, funding is shared between the state, students, and training institutions based on student performance, with the goal of ensuring workforce training is affordable for Virginia residents. An important implication of WCG is that it led to systematic, statewide collection of student-level data on FastForward program enrollment, program completion, industry credential attainment, and labor market performance. Drawing on these unique data, coupled with interviews with key stakeholders, we generated findings on the characteristics of FastForward programs, as well as the academic and labor market outcomes of students enrolled in these programs. We describe the preliminary descriptive findings below.

FastForward programs enroll a substantially different segment of the population from credit-bearing programs and offer a vital alternative route to skill development and workforce opportunities, especially for demographic groups often underrepresented in traditional higher education. FastForward programs in Virginia enroll a substantially higher share of Black students, male students, and older students than short-duration, credit-bearing programs at community colleges that typically require one year or less to complete. Focus groups conducted with FastForward students at six colleges indicate that the students were a mix of workers sent by their employers to learn specific new skills and students who signed up for a FastForward program on their own. Among the latter group were older career changers and recent high school graduates, many of whom had no prior college experience and were primarily interested in landing their first job in their chosen field. Moreover, 61% of FastForward participants have neither prior nor subsequent enrollment in credit-bearing programs, highlighting the program’s unique role in broadening access to postsecondary education and career pathways.

FastForward programs offer an alternative path for students who are unsuccessful in credit-bearing programs. The vast majority of students (78%) enrolled in only one FastForward program, with the average enrollment duration of 1.5 quarters, which is notably shorter than most traditional credit-bearing programs. While 36% have prior credit-bearing enrollment, fewer than 20% of these students earned a degree or certificate from it, and less than 12% of FastForward enrollees transitioned to credit-bearing training afterward. Interviews with administrators and staff indicated that while some colleges facilitate noncredit-to-credit pathways by granting credit for prior learning, others prioritize employment-focused training and support over stackable academic pathways due to students’ primary interest in seeking employment post-training.

FastForward programs have a remarkable completion rate and are related to high industry credential attainment rates. Over 90% of students complete their program, with two-thirds of students obtaining industry credentials. Student focus groups echoed this success. They praised the FastForward program and colleges for addressing both their tuition and non-tuition needs. Many students noted that they had not envisioned themselves as college students and credited program staff, financial aid, and institutional support with helping them to be successful.

Earning an industry credential through FastForward on average increases quarterly earnings by approximately $1,000. In addition, industry credentials also increase the probability of being employed by 2.4 percentage points on average. We find substantial heterogeneity in economic return across different fields of study, where the fields of transportation (for example, commercial driver’s license) and precision production (for example, gas metal arc welding) seem to be associated with particularly pronounced earnings premiums. Within programs, we do not observe significant heterogeneity in economic returns across student subgroups.

What’s Next?

In view of the strong economic returns associated with earning an industry credential and the noticeable variation in credential attainment between training institutions and programs, our future exploration intends to unpack the sources of variation in program-institution credential attainment rates and to identify specific program-level factors that are within the control of an institution and which are associated with higher credential rates and lower equity gaps. Specifically, we will collect additional survey data from the top 10 most highly-enrolled programs at the Virginia Community College System (VCCS) that will provide more nuanced program-level information and identify which malleable program factors are predictive of higher credential attainment rates, better labor market outcomes, and smaller equity gaps associated with these outcomes.

Di Xu is an associate professor in the School of Education at UC, Irvine, and the faculty director of UCI’s Postsecondary Education Research & Implementation Institute.

Ben Castleman is the Newton and Rita Meyers Associate Professor in the Economics of Education at the University of Virginia.

Betsy Tessler is a senior associate at MDRC in the Economic Mobility, Housing, and Communities policy area.

Note: A team of researchers, including Kelli Bird, Sabrina Solanki, and Michael Cooper contributed jointly to the quantitative analyses of this project. The MDRC team, including Hannah Power, Kelsey Brown, and Mark van Dok, contributed to qualitative data collection and analysis. The research team is grateful to the Virginia Community College System (VCCS) for providing access to their high-quality data. Special thanks are extended to Catherine Finnegan and her team for their valuable guidance and support throughout our partnership.

This project was funded under the Postsecondary and Adult Education research topic; questions about it should be directed to program officer James Benson (James.Benson@ed.gov).

This blog was produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER program officer for the CTE research topic.

Cost analysis is a critical part of education research because it communicates what resources are needed for a particular program or intervention. Just telling education leaders how promising a program or practice can be does not tell the whole story; they need to know how much it will cost so that they can prioritize limited resources. Since 2015, cost analysis has been required for IES-funded Efficacy/Impact studies (and for Development Innovation studies as of 2019) and is included in the IES Standards for Excellence in Education Research.

In this guest blog for CTE Month, two members of the CTE Research Network’s cost analysis working group, David Stern, an advisor to the network, and Eric Brunner, a co-PI of one of the research teams, discuss how costs associated with CTE programs may differ from those of standard education and how to measure those costs.

Why is cost analysis different in Career & Technical Education (CTE) research?

Due to additional, non-standard components needed in some types of career training, CTE can cost much more than the education resources needed in regular classrooms. For instance, CTE classes often use specialized equipment—for example, hydraulic lifts in automotive mechanics, stoves and refrigerators in culinary arts, or medical equipment in health sciences—which costs significantly more than equipment in the standard classroom. Having specialized equipment for student use can also constrain class size to be smaller, resulting in higher cost-per-pupil.  High schools and community colleges may also build labs within existing buildings or construct separate buildings to house CTE programs with specialized equipment. These required facility expenses will need to be recognized in cost calculations.

CTE programs can also provide co-curricular experiences for students alongside classes in career-related subjects, such as work-based learning, career exploration activities, or integrated academic coursework. Schools are usually required to provide transportation for students to workplaces, college campuses for field trips, or regional career centers, which is another expense. Finally, the budget lines for recruiting and retaining teachers from some higher paying career areas and industries (such as nursing or business) may exceed those for average teacher salaries. All of these costs add up. To provide useful guidance for the field, CTE researchers should measure and report the cost of these features separately.

How is resource cost different from reported spending? 

There are also some hidden costs to account for in research on CTE. For example, suppose a school does not have a work-based learning (WBL) coordinator, so a CTE teacher is allowed one of their 5 periods each day to organize and oversee WBL, which may include field trips to companies, job shadowing experiences, internships, or a school-based enterprise. The expenditure report would show 20% of the teacher’s salary has been allocated for that purpose. In reality, however, a teacher may devote much more than 20% of their time to this. They may in fact be donating to the program by spending unpaid time or resources (such as transportation in their own vehicle to visit employer sites to coordinate learning plans) outside the workday. It is also possible that the teacher would spend less than 20% of their time on this. To obtain an accurate estimate of the amount of this resource cost at a particular school, a researcher would have to measure how much time the teacher actually spends on WBL.  This could be done as part of an interview or questionnaire.

Similarly, high school CTE programs are increasingly being developed as pathways that allow students to move smoothly to postsecondary education, such as via dual enrollment programs or directly to the labor market. Building and sustaining these pathways takes active collaboration between secondary and postsecondary educators and employers. However, the costs of these collaborations in terms of time and resources are unlikely to be found in a school expenditure report. Thus, an incremental cost analysis for CTE pathway programs must go beyond budgets and expenditure reports to interview or survey program administrators and staff about the resources or “ingredients” that programs require to operate. A recent example of a cost study of a CTE program can be found here.

Are there any resources for calculating CTE Costs?

In this blog, we have presented some examples of how the costs associated with CTE programs may differ from those of a standard education. To help CTE researchers conduct cost analysis, the CTE Research Network has developed a guide to measuring Incremental Costs in Career and Technical Education, which explains how to account for the particular kinds of resources used in CTE. The guide was developed by the working group on cost analysis supported by the CTE Research Network.

The Career and Technical Education (CTE) Research Network has supported several cross-network working groups comprised of members of network research teams and advisors working on issues of broad interest to CTE research. Another CTE Network working group developed an equity framework for CTE researchers, which was described in a blog for CTE month in February, 2023.

This blog was produced by Corinne Alfeld, NCER program officer for the CTE research topic and the CTE Research Network. Contact: Corinne.Alfeld@ed.gov.

IES funded the Expanding the Evidence Base for Career and Technical Education (CTE) Research Network (CTERN) in FY 2018 in order to increase the quality and rigor of CTE research, specifically by (1) coordinating IES-funded researchers studying CTE using causal designs and (2) training new researchers in causal methods to address CTE-related research questions. In this guest blog, the Network Lead’s PI, Katherine Hughes, and Training Lead, Jill Walston, from the American Institutes for Research (AIR), discuss the evolution of the institute across four years of training supported by the grant and what they learned about the components of effective training, in the hopes of sharing lessons learned for future IES-funded trainings.

About the Summer Training Institute

Each summer since 2020, CTERN has held summer training institute on causal research methods in CTE.  Across four summers, we had 81 trainees, including junior faculty, researchers in state or university research offices or institutes, doctoral students, and researchers in non-profit organizations. During the institutes, we had expert CTE researchers and national and state CTE leaders deliver presentations about CTE history, policies, theories, and recent research.

The major focus of the training was on research designs and statistical methods for conducting research that evaluates the causal impact of CTE policies and practices on student outcomes. The participants learned about conducting randomized controlled trials—considered the gold standard for causal research—as well as two quasi-experimental approaches, regression discontinuity and comparative interrupted time series designs. After presentations about the approaches, students worked with data in small groups to complete data analysis assignments designed to provide practical experience with the kinds of data and analyses common in CTE research. The small groups had dedicated time to meet with one of the instructors to discuss their analyses and interpret findings together. The combination of presentations and practical applications of data analysis with real data, and time in small groups for troubleshooting and discussion with CTE researchers, made for a rich experience that students found engaging and effective. The students received an IES certificate of course completion to mark their accomplishment.

Making Continuous Improvements Based on Lessons Learned

We had a continuous improvement mindset for our summer institute. After each week-long session was completed, the CTE research network director, training coordinator, and instructors met to review their perceptions of the training and most importantly the feedback students provided at the end of the week. We applied the lessons learned to make improvements to the agenda, communications, and student grouping approaches to the plans for the following summer.

Over the course of the four years of the summer institute training, we made a number of adjustments in response to feedback.

• We continued to offer the institute virtually. The institute was originally intended to be held in person; an earlier blog describes our necessary pivot to the online format. While we could have safely changed to an in-person institute in 2022 and 2023, feedback from our students showed that the virtual institute was more accessible to a geographically diverse group. Many trainees said they would not have even applied to the institute if they would have had to travel, even with a stipend to help cover those costs.
• We added more time for the students to get to know one another with virtual happy hours. Compared to in-person trainings, virtual trainings lack those natural opportunities for informal communications between students and with instructors that can foster engagement, trust, and joint purpose. While we couldn’t replicate in-person networking opportunities, we were able to improve the experience for the students by being intentional with informal gatherings.
• We expanded the time for the small groups to meet with their instructors. Students reported that this office hour time was very valuable for their understanding of the material and in interpreting the output of the analyses they ran. We extended this time to optimize opportunities for discussion and problem solving around their data analysis assignments.   
• We made improvements to the data assignment guidance documents. In the first year, students reported that they spent more time on figuring out initial tasks with the data which left less time for running analyses and interpreting their output. We modified our guidance documents that accompanied the assignments to spell out more explicitly some of the initial steps to shorten the time students spent on set-up and maximize their time doing the important work of coding for the analyses and examining output. We also provided links to resources about the statistical packages used by the students for those that needed time to brush up on their skills before the training began.
• We doubled down on efforts to stay connected with the trainees and supported ways to have them stay connected to each other. For example, we let them know when CTERN’s researchers are presenting at conferences and invite them to connect with us and each other at these conferences. We’re now organizing a LinkedIn group to try to develop a community for our training alumni.

Our summer training institutes were a great success. We look forward to continuing this opportunity for researchers into the future, with a new version to be offered in the summer of 2025 by the CTE Research Network 2.0.

Jill Walston, Ph.D., is a principal researcher at the American Institutes for Research with more than 20 years of experience conducting quantitative research, developing assessments and surveys, and providing technical support to researchers and practitioners to apply rigorous research and measurement practices. Dr. Walston is the lead for training initiatives for the IES-funded Career and Technical Education Research Network.

Katherine Hughes, Ph.D., is a principal researcher at the American Institutes for Research and the principal investigator and director of the CTE Research Network and CTE Research Network 2.0. Dr. Hughes’ work focuses on career and technical education in high schools and community colleges, college readiness, and the high school-to-college transition.

This blog was produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), a Program Officer in the National Center for Education Research (NCER).

February is Career and Technical Education (CTE) month! As part of our 20^th anniversary celebration, we want to highlight the great work our CTE Research Network (CTERN) continues to accomplish. This guest blog was written by James Kemple, Director of the Research Alliance for New York City Schools and the principal investigator (PI) of a CTERN research project that is examining CTE in New York City.

While “college and career readiness” are familiar buzzwords in K-12 education, it has often seemed like system leaders shout “college” and whisper “career.” During the last decade, as it has become clear that a high school diploma has limited value in the 21^st century labor market, career and technical education (CTE) has become a more prominent way to explicitly prepare students for both college and career. For the CTE field to evolve productively, valid and reliable evidence should inform policy and practice, for example by identifying conditions under which CTE may be more or less effective and for whom.

CTE and College and Career Readiness in New York City

One such project is our ongoing study of New York City’s CTE programs. The current phase of the study focuses on 37 CTE-dedicated high schools, which are structured to ensure that all enrolled students participate in a CTE Program of Study from 9th through 12th grade. These programs are organized around an industry-aligned theme (for example, construction, IT, health services, etc.) and offer a sequence of career-focused courses, work-based learning opportunities, and access to aligned college-level coursework. Our study uses an especially rigorous approach to compare the experiences and outcomes of nearly 19,000 NYC students who were assigned to a CTE-dedicated high school between 2013 and 2016 with those of similar students who also applied to CTE programs but were assigned to another high school during the same period.

When our research team looked at the overall impact of 37 CTE-dedicated high schools in NYC, we found that CTE students graduated from these high schools and enrolled in college at rates that were similar to their counterparts in non-CTE high schools. On average, therefore, being in a CTE high school did not steer students away from a college pathway.

Variations in CTE Programs

A much more interesting story emerged when we took a closer look at variation in student experiences and outcomes. In fact, some of the schools produced statistically significant reductions in immediate college enrollment, while others produced increases in the rate at which students enrolled in college.  Why might this be?

The study team identified two possible reasons. First, the schools in our sample differed based on the policy context in which they were created: 21 of the schools were established after 2008 as the NYCDOE undertook a major expansion of CTE in the midst of a larger overhaul of the city’s high schools that included closing persistently low-performing schools, opening new small schools in their place, and creating a universal high school admissions system that gave students access to schools across the city. In contrast to the 16 longstanding CTE high schools—some of which dated back to the early 1900s—these new high schools were smaller, with more thematically aligned sets of CTE programs, and non-selective admission processes. Most of the longstanding CTE schools used test scores, grades, or other performance measures as part of their admissions criteria.

Second, the schools in the study differed in terms of their intended career pathways—and the extent to which these career pathways require a post-secondary credential for entry-level jobs. Notably, nine of the newer (post-2008) high schools focused on career pathways that were likely to require a bachelor’s degree (referred to “college aligned”). CTE programs in the remaining 12 newer high schools and all of the CTE programs in the 16 longstanding high schools focused on either “workforce-aligned” career pathways—allowing students to enter the labor market directly after high school—or “mixed” pathways that require additional technical training or an associate degree for entry-level jobs. Interestingly, each of these groups of schools included a mix of CTE career themes. For example, some health- or technology-focused CTE programs reflected college-aligned pathways, while other programs with these themes reflected workforce-aligned pathways.

We found that the newer, smaller, less selective CTE schools with more tightly aligned career themes had positive effects on key outcomes—particularly those that were focused on college-intended career paths. These schools produced a substantial, positive, statistically significant impact on college enrollment rates. Students in these schools were nearly 10 percentage points more likely to enroll in a four-year college than those in the non-CTE comparison group.

By contrast, the larger, more selective CTE schools, with a range of work-aligned career pathways, were associated with null or negative effects on key outcomes. Notably, these schools actually reduced four-year college enrollment rates.

Applying Lessons Learned

The extraordinary diversity of NYC’s CTE landscape and its student population provides a unique opportunity to gather information about program implementation, quality, accessibility, and costs, and about how these factors influence CTE’s impacts on college and career readiness. A recent report from the project provides new insights into strategies for learning from variation in CTE programs and contexts, as well as particular policies and programming conditions that may enhance or limit college and career readiness.

Recent efforts to enhance CTE, including those underway in NYC, wisely focus on such key elements as rigorous and relevant CTE course sequences, robust work-based learning opportunities, and articulated partnerships with employers and post-secondary education institutions. The findings from this study point to additional conditions that are likely to interact with these curricular and co-curricular elements of CTE—such as providing students with smaller, more personalized learning environments; using inclusive (less selective) admissions policies; and aligning high school requirements with post-secondary options. It will be crucial for policymakers to attend to these conditions as they work to strengthen students’ pathways into college and careers.

Finally, it is important to note that we do not yet have all the information needed to fully discern the impact of NYC’s diverse CTE options. Data on employment and earnings will be crucial to understanding whether students in these schools opted to enter the workforce instead of, or prior to, enrolling in college—and how these decisions affected their longer-term trajectories.

This blog was produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), program officer, NCER.