Interventions to increase youth employment: An evidence and gap map

1.1. The evidence for youth employment interventions is unevenly distributed by geography and population sub‐groups, and much of it is of poor quality

There is considerable evidence on training‐based youth employment interventions across all outcomes included in the evidence and gap map (EGM), but scarce evidence in the ‘information services’ and ‘decent work policies’ categories of interventions. Much of the evidence is of low quality.

2.1. Intervention: All interventions that increase youth employment

All interventions or programmes or projects aimed at helping youth to find and sustain employment are the focus of this EGM. This EGM broadly categories the interventions into three domains: strengthening training and education systems, enhancing labour market and, transforming financial sector markets.

Strengthening training and education systems, includes the conventional education systems and various forms of training such as direct training often provided to the unemployed. The other types of training namely; up‐skilling and retraining/re‐skilling normally target youth who are already in employment, to ensure that their skill relevant to the changing demands of employers.

Enhancing labour markets category of interventions includes: decent work policies, support to employment services, and Information services. Decent work polices generally regulate the relationship between employers and employees in the employment environment through application of labour standards, regulations and accountability systems. Support to employment interventions are generally meant to help youth find jobs through provision of jobs via initiatives like programmes for overseas employment and public works programmes.

Transforming financial sector markets based interventions focus on entrepreneurship promotion and financing. This category of interventions tend to be popular for targeting disadvantaged youth especially those with low levels of education. As Datta et al. (2018) observe, labour market opportunities are significantly influenced by the reverence of skills for the existing job market. An intervention like self‐help or financing groups tends to be dominant among youth with less education and training. Self‐help groups are small groups that save a certain amount of money on weekly or monthly basis and issue loans to members out of their collective savings (Flynn, 2013).

The focus of this EGM is therefore broad, for example covering economic and welfare dimensions. So, outcomes of the various interventions go beyond the youth getting employment which is often the primary motivation of implementing youth employment programmes.

2.2. Scope of the youth employment EGM1

Geographically, this EGM is global in coverage, considering all countries regardless of their level of development. That means all world geographical regions and the World Bank country classification by income have been covered. The geographical regions are: Sub‐Saharan Africa, Latin America and Caribbean, East Asia and Pacific, Middle East and North Africa, South Asia, Europe and Central Asia, and North America. The World Bank country classification by income includes: low‐income countries, lower‐middle income, upper middle income, and high‐income countries (World Bank, 2020).

By population, the map covered all young women and men aged 15–35 years from all countries. The three broad intervention categories include in the EGM are: strengthening training and education systems, enhancing labour market and, transforming financial sector markets.

The outcomes fall into five categories: education and skills, entrepreneurship, employment, welfare and economic.

In terms of evidence, the map included impact evaluations of interventions aimed at increasing youth employment and systematic reviews of such single studies, published or made available between January 2000 and December 2019.

2.3. Why it is important to develop the EGM

EGMs guide policy makers, development partners and researchers to available evidence to inform programme design and implementation of development interventions. Decision‐makers and researchers often lack awareness about the extent of evidence base, so can maps help in increasing the discoverability, and thus use of that evidence for evidence‐informed decision making in international development policy and practice. The immediate potential application of the youth employment EGM is its contribution to the implementation of the Mastercard Foundation's strategy titled, ‘Africa Works—Mastercard Foundation Strategy 2018–2030’. The goal of the strategy is to enable 30 million youth in Africa to secure dignified and fulfilling employment by 2030. Sharing of evidence‐based knowledge and innovation with stakeholders is stated as one way through which the strategy can be implemented (Mastercard Foundation, 2019). So, the youth employment EGM is a useful resource in that regard, as it can guide policy makers, development partners and researchers to relevant available evidence on youth employment interventions.

For researchers, the youth employment EGM has identified research gaps for new primary research and new synthesis. This can inform strategic policy‐oriented approach in commissioning relevant and coordinated research programmes (White et al., 2020).

Before the production of this EGM, two pre‐existing evidence gap maps on youth employment‐related interventions were inadequate in a number of ways. For instance, each of those maps had a narrow scope (geographical, study publication period and, area/interventions and outcomes). The maps were limited to low‐ and middle‐income countries and publication period of 1990–2015. Moreover, development interventions are often best appreciated and understood in a contemporary context (Mawn et al., 2017). Further, both maps did not include economic outcomes while one did not include welfare outcomes. In addition, at least one of the maps suffered methodological limitation stemming, whereby the study search strategy and screening of studies were conducted by individuals rather in pairs for validation. This is stated to have led to some studies being missed. Never the less, the two pre‐existing maps provided a basis for the development of the current youth employment EGM, with a broader focus (geographical, study publication period and, area/interventions and outcomes). Methodological limitations were also avoided by better planning for sufficient time and human resources.

2.4. Objectives

The research question guiding the production of the youth employment EGM was stated as follows: What is the nature and extent of the evidence base of impact evaluations and systematic reviews on youth employment programmes in the world?

The primary objective was to catalogue impact evaluations and systematic reviews on youth employment interventions to enhance discoverability of evidence by decision makers, development patterners and researchers, so as to promote evidence‐based decision making in programming and delivery of youth employment initiatives. This EGM was considered a primary input into the implementation of Mastercard Foundation's strategy titled ‘Africa Works: Mastercard Foundation Strategy 2018–2030’, which pointed out sharing of evidence‐based knowledge and innovation with stakeholders as a key strategy to be used (Mastercard Foundation, 2019). The time frame for the development of the youth EGM ran from the last quarter of 2019 to December 2020.

The five secondary objectives were:

• (i)

  To construct a framework for the classification of youth employment effectiveness studies.

• (ii)

  To identify available evidence, and clusters of evidence on effectiveness studies (impact evaluations and systematic reviews of impact evaluations) on youth employment interventions.

• (iii)

  To create a map of youth employment effectiveness studies equipped with an appealing user‐friendly web‐based search content visualisation using interactive mapping software.

• (iv)

  To produce a narrative report of the youth employment EGM.

• (v)

  To disseminate the EGM to users to increase awareness to support evidence‐informed decision‐making across countries.

2.5. Existing EGMs on youth employment interventions

Before the production of this EGM, there were two evidence gap maps on youth employment. The two EGMs were reviewed to inform the development of the framework for this EGM. The descriptions provided below for each of the maps pointed out associated strengthens and limitations which were of scope and methodological nature.

The first evidence gap map was the ‘Youth and Transferable Skills evidence gap map’. The map included 98 studies and is accessible at https://gapmaps.3ieimpact.org/evidence-maps/youth-transferable-skills-evidence-gap-map (Rankin et al., 2015). The map included studies published or made available between 1990 and 2015. The included studies were searched from January to February of 2015. The map is restricted to low‐ and middle‐income countries. In terms of youth employment as a development area, the map has a narrow focus, covering only transferable skills interventions and associated outcomes. For, instance economic outcomes are not covered, yet these set of outcomes for example ‘cost effectiveness’ often have important bearing on the implementation of programmes. The map also suffered methodological problems due to time constraints. It was mentioned that the use of a single specialist to supervise and compile the search work as well as reliance on one person to screen studies on titles and abstracts, could have led to some studies being missed (Rankin et al., 2015). The map had an accompanying published narrative report which provided detailed information on areas such as methodology and results, which is a strength. In addition, this map used an extensive study search strategy covering 34 websites and 4 research registries.

The second map was the ‘Youth employment evidence gap map’, produced by International Labour Organization. The map included 113 studies and is available at: https://gapmaps.3ieimpact.org/evidencemaps/youth-employment-evidence-gap-map (International Labour Organization, 2018). The map was restricted to low‐ and middle‐income countries and included studies published or made available between 1990 and 2014 which were contained in a systematic review by (Kluve et al., 2017), titled ‘Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidised employment interventions’. The map did not include economic and welfare outcomes. In addition, a narrative report accompanying the EGM was not accessible which is an indication of its absence. Although, a narrative report is an optional product in the production of an EGM (Saran & White, 2018), the absence or lack of access to such a document denies users vital information.

The above maps provided a basis for the development of the current youth employment EGM, with a broader focus (geographical, study publication period and, area/interventions and outcomes).

3.1. Definition and purpose of EGMs

Saran and White (2018), define an EGM as ‘a systematic [visual] presentation of the availability of relevant evidence for a particular policy domain. The evidence is identified by a search following a prespecified, published search protocol. The map may be accompanied by a descriptive report to summarise the evidence for stakeholders such as researchers, research commissioners, policy makers, and practitioners’ (p. 11). It's important to note that EGMs summarise what evidence exists but not what the evidence says. For instance, an EGM catalogues studies in a particular policy domain in terms of outcomes and interventions but does not say the magnitude of outcomes reported by the studies.

EGMs are useful in many ways. First, they guide policy makers, development partners and researchers to relevant available evidence to inform the design and implementation of development interventions. Decision‐makers and researchers often lack awareness about the extent of evidence base, so maps help in increasing the discoverability, and thus use of that evidence for evidence‐informed decision making in international development policy and practice (White et al., 2020). Second, they create awareness among implementing agencies where relevant evidence for their interventions is lacking, so that they can act accordingly by collecting evidence for the intervention they are supporting. Finally, maps identify research gaps for new primary research and new synthesis. This can inform strategic policy‐oriented approach in commissioning relevant and coordinated research programmes (White et al., 2020).

3.2. Framework development and scope

Development of the framework is considered the first and most important part in the development of an evidence map (White et al., 2020). The framework was therefore the first activity undertaken in the production of this EGM, in the last quarter of 2019. The framework provided the structure or layout of the EGM and was a primary resource in the development of the search strategy, screening and coding tools. A typical framework for an effectiveness EGM refers to the matrix of interventions (in rows) and outcomes (in columns), developed through a review of existing maps on a related policy domain, policy literature and consultations with stakeholders (Rankin et al., 2015).

The development of the framework for this EGM was achieved through a consultative process involving authors of the map, Mastercard Foundation and stakeholders in the youth employment area. The consultative approach helped the capture of a wide range conceptual and contextual positions of Mastercard Foundation and stakeholders involved in youth employment programming and implementation. The steps followed are described below.

First, using a workshop approach in Uganda, the EGM authors constructed a draft framework by brainstorming and reviewing existing EGMs that included impact evaluations of interventions to improve youth labour market outcomes and systematic reviews of such single studies. Dr. Howard White who is an expert in development evaluation and Dr. Ekwaro A. Obuku, an expert in evidence synthesis, led this activity.

Second, the draft framework was shared with Mastercard foundation to capture their input. It's important to note that Mastercard Foundation (funder) was engaged all through the project life by ensuring that they reviewed study tools (study screening tool, coding sheet and, a dictionary of outcomes and interventions), that were developed by the EGM authors.

Finally, the EGM authors led by Dr. Saran Ashrita, a methods expert had a training workshop in Uganda which, incorporate Mastercard Foundation's feedback on the draft framework into the final framework. Additional activities that were undertaken in this stage include drafting of the coding sheet, definitions of interventions and outcomes to guide coding of studies. Training of people who coded studies using EPPI Reviewer 4, a web‐based software program for managing and analysing data in literature reviews, was also carried. The framework was also piloted with about 100 studies.

3.3. Stakeholder engagement

Meetings and workshops to engage stakeholders were planned to be conducted in Uganda, in the last quarter of 2019. The target stakeholders were relevant officials in Uganda from government ministries, departments, agencies, private sector agencies, civil society organisations, vocational training institutes, international development agencies as well as academia. Unfortunately, the COVID‐19 global lockdown constrained stakeholder engagement.

However, keeping in touch with Mastercard Foundation policy leads and the literature were informative in arriving at reasonable priority list of the interventions and outcomes for this EGM.

3.4. How youth employment interventions are supposed to work

This section provides theoretical pass‐ways in which youth employment interventions included in this EGM may put more youth into employment through both job seeking or entrepreneurship. The principal assumption of participating in youth employment interventions is that youth can acquire relevant skills for labour market and, support (financial, guidance and information access to facilitate employment. Participation in youth employment interventions is expected to address constraints to youth employment, some of which, Datta et al. (2018), identifies, as summarised in (Box 1).

The Theory of change (ToC) provided in (Supporting Information Annex 8; Figure 17) highlights some of the pathways though which the interventions are expected to lead to outcomes. This approach identifies connections between interventions and outcomes that are contained in the Youth Employment EGM which is the subject of this report. The interventions and outcomes of the EGM later revisited in the methods section in Tables 1 and 2, respectively, are grouped into categories and sub‐categories as well as examples. The three intervention categories are: strengthening training and education systems; enhancing labour markets and; transforming financial sector markets. The following paragraphs attempt to show the linkage between intervention sub‐categories and outcomes, with supporting literature.

First, the ‘strengthening training and education systems’ category of interventions equip youth with skills which are necessary for increasing employment opportunities in the labour market (United Nations, 2018). Actually globally, skills training is the most widely implemented set of labour market interventions for youth (Kluve et al., 2017). Training programmes often equip youth with skills required by employers. However, training and formal education interventions often do not reach marginalised youth especially young women, indigenous groups, youth with disabilities. This leaves such youth without the skills needed to realise their potential (United Nations, 2018). Skills attained through training can be categorised into technical skills, business skills and life or soft skills (Kluve et al., 2017). Technical skills are achieved by individuals attending training initiatives such as technical and vocational education and training (TVET) and, internship and apprenticeship. Business skills training is normally provided to increase entrepreneurial activities among youth (Kluve et al., 2017). In the case of life skills training, the objective is to strengthen trainees' self‐esteem and work habits to help them achieve the goals set by employers (Lippman et al., 2015). The ‘training and skills development interventions results chain’, diagnostic framework documented by Kluve et al. (2017) is a useful resource for mapping the relationship between training and education intervention and potential outcomes. Participation in training and education is expected to create mainly three categories of outcome: (i) employment outcomes (i.e., change in employment status, earning and salary); (ii) education and skills outcomes such as acquisition of technical and vocational skills.

Second, the ‘enhancing labour market’ category of interventions are grouped into three sub‐categories including: support to employment; enhancing labour market; and transforming financial sector market. Support to employment interventions are generally meant to help youth find jobs through provision of jobs via initiatives like programmes for overseas employment and public works programmes. Employment services also facilitate the youth in the process of finding jobs through career guidance and supporting employee mobility needs. Decent work polices generally regulate the relationship between employers and employees in the employment environment through application of labour standards, regulations and accountability systems. Another enhancer of labour markets, is provision of a wide range information about the labour market and associated services. Potential outcomes from this category of interventions tend to be largely associated with employment outcomes such as job quality and change in employment status.

Finally, ‘transforming financial sector’ category of interventions focus on entrepreneurship promotion and financing. The category of interventions can be considered the most popular for targeting disadvantaged youth especially those excluded from training and education programmes. Datta et al. (2018) note that labour market opportunities are significantly influenced by the reverence of skills for the existing job market. An intervention like self‐help or financing groups tends to be dominant among youth with less education and training. Self‐help groups are small groups that save a certain amount of money on weekly or monthly basis and issue loans to members out of their collective savings (Flynn, 2013). The category of interventions (transforming financial sector) tend to be mainly associated with welfare outcomes and employment outcomes than economic outcomes.

3.5. Dimensions

The primary dimensions of the map are interventions (in rows) and outcomes (in columns), presented in a matrix. There are three broad intervention categories, each with subcategories. The intervention categories include: Strengthening training and education systems; enhancing labour markets; and transforming financial sector markets. The outcomes are arranged in a typology of five categories: education and skills, entrepreneurship, employment, welfare, and economic. The interventions and outcomes are later described in detail in Tables 1 and 2, respectively.

The secondary dimensions of the map are the taskbar menu (Filters, About and View records), which help a user to navigate the map. A detailed description of the taskbar menu is provided later below (Figure 4).

3.6. Types of study design

Only studies with the following study designs were included in the EGM: Experimental designs, Nonexperimental matching designs, Regression‐based approaches and, Systematic reviews.

The selected study designs are the appropriate designs for estimating effectiveness of program interventions. The designs (experimental and nonexperimental) are for conducting impact evaluations of development interventions. Examples of the following non‐evaluation‐based study designs were excluded from the EGM: ethnography, case control and cross‐sectional. For instance, the focus of a cross‐sectional study is limited to data from a particular population with variables of interest, at a given point in time. Cross‐sectional studies are observational in nature and not causal implying that they are not applicable for determining the effect of an intervention.

• (a)

  Experimental designs

Experimental designs fall into two types, namely randomised controlled trials (RCTs) and natural experiments.

• (i)

  RCTs: A typical RCT design involves randomising study participants into two or groups (an experimental/treatment/intervention group and control group) in which the researcher introduces an intervention and measures its impact on the dependent variable at least two times namely pre‐ and posttest measurements (White & Sabarwal, 2014). A key weakness of an RCT study design is that it suffers from missing outcomes resulting from changes that occur postrandomisation of study participants. For instance, withdrawal of subjects from the study and noncompliance with established study protocols or guidelines would lead to missing outcomes (White & Sabarwal, 2014). Therefore, application of Intention‐to‐treat (ITT) analysis in RCTs attempts to address this problem by including in the analysis every subject who is randomised according to randomised treatment assignment and ignoring anything that happens after randomisation (Gupta, 2011). So, studies using ITT analysis have been included in the EGM under RCT study design.

• (ii)

  Natural experiments: Despite the lack of universally accepted definition of the term natural experiment, researchers are in agreement that a natural experiment happens where and/or when an intervention is implemented without the control of a researcher (Butler et al., 2018). Therefore, a natural experiment is an experiential study design in which clearly defined sub‐populations are exposed to the experimental and control conditions that are determined by nature or other factors outside the control of the researchers. The allocation into experimental and control conditions allows researchers to use natural or unplanned variation in exposure to draw inferences about causality.

For example, a policy development emphasising promotion of TVET in Uganda can be seen as a natural experiment. TVET is a school‐based intervention, which makes it a commonplace intervention and thus underpinned by equity considerations. Therefore, a randomised experiment/RCT design is often considered to be politically and ethically not feasible. In this case a natural experiment is a preferred evaluation study design.

• (b)

  Nonexperimental matching designs

Nonexperimental designs are used where random assignment is not feasible for various reasons. For instance, where the need for evaluation arises when the program is completed or on going. Nonexperimental designs can be generally categorised as quasi‐experimental and regression‐based approaches. Quasi‐experimental methods create comparison groups by statistical methods, rather than by random assignment. These methods include difference‐in‐differences (DiD), propensity score matching (PSM), regression discontinuity designs (RDD), and synthetic controls designs (White & Raitzer, 2017).

• (i)

  PSM: In PSM, the matching enables construction of an artificial comparison group with almost similar characteristics as the treatment group. The artificially created comparison group from untreated observations is matched to treatment observations from the untreated sample, based on observable characteristics. The treated units are matched to untreated units with a similar propensity score. The matching approach is considered adequate to attain an unbiased impact estimate (White & Sabarwal, 2014).

• (ii)

  DiD: In DiD approach, impact is estimated by comparing the changes in outcome over time between treatment and comparison groups. The method is also known as controlled before and after studies or ‘double difference’ method (White & Sabarwal, 2014).

• (iii)

  RDD: This is a popular approach used in econometrics due to situations that make randomisation unfeasible to determine causal effects of interventions by assigning a cut‐off or threshold above or below which an intervention is assigned. The threshold refers to the criterion that participants must met before being included in the intervention. The threshold is usually based on a continuous variable (White & Sabarwal, 2014). For instance, adults above or below a particular age for enroling in a training programme. RDD approach compares observations on either side of the threshold to estimate average treatment effects of an intervention. The major limitation of RDD is that its greatly affected by confounding variables. For instance, average treatment effects of a local sanitation intervention may be affected by a regional related intervention if they were implemented at the same time.

• (c)

  Regression‐based approaches

All approaches based on regression models not listed above are included here. These include (but are not restricted to):

• (i)

  Instrumental variable (IV): ‘A statistical technique for estimating causal relationships when an RCT is not feasible or when an intervention does not reach every participant/unit in an RCT’ (White & Sabarwal, 2014, p. i).

• (d)

  Systematic reviews

A systematic review summarises bodies of literature with summary statements of the findings of that literature. The term ‘systematic’ in systematic review refers to the research process that is intended to minimise the biases that may occur in a traditional literature review. Key characteristics of a systematic review include: a clear scope for the review; set of research questions; clear study inclusion and exclusion criteria; systematic search strategy used to identify the single studies that would meet the eligibility criteria; and results. Therefore, studies included in the EGM as systematic reviews had the above‐mentioned characteristics even if they did not use the term ‘systematic’ in the titles. Studies using the term ‘systematic’ but lack key features of a systematic review were excluded. For instance, scoping reviews or literature scans and Meta‐analysis of evidence were excluded on study design criterion.

3.7. Types of intervention/problem

The EGM has three broad intervention categories, each with subcategories. The intervention categories include: Strengthening training and education systems; enhancing labour markets; and transforming financial sector markets. Table 1 lists the intervention categories, subcategories as well as examples to aided study search and coding. The intervention category of ‘strengthening training and education systems’, here refers to a group of sub‐interventions covering improvements in training, upskilling and retraining or re‐skilling components of education and training systems. Specifically, up‐skilling is the process of teaching workers new skills. Retraining is a practice where employers may ensure that their workers learn new skills specially to avoid stagnant workforce. So here, some examples of interventions to be captured include TVET, business skills training and, internship and apprenticeship (Table 1).

A detailed compilation of definitions of the interventions are provided in Supporting Information Annex 1, with a reference list provided as Supporting Information Annex 3.

3.8. Types of population

The only criterion used for selecting the target population is age. So, the target population is all youth or young women and men aged 15–35 years from all countries. The diversity of varying national definitions of the term youth was acknowledged. For instance, while the United Nations (UN) defines youth as young women and men aged 15–24 years, in South Africa youth fall in the age group 14–35 years (South Africa, 2015), in Zimbabwe it's from 15 to 35 years of age (Zimbabwe. Ministry of Youth Development, 2000) and in Uganda youth are within the age bracket of 15–30 years (Uganda. Ministry of Gender Labour and Social Development, 2016). This EGM uses a more encompassing classification to identify youth as young women and young men aged between 15 and 35 years.

Population subgroups included: both females and males; youth with disabilities; youth in fragility, conflict and violence (FCV) contexts; youth from disadvantaged background (low‐income families or low education); criminal background; ethnic minorities; and humanitarian settings. The population sub‐groups permitted the identification of studies reporting evidence on equity.

3.9. Types of outcome

Table 2 contains outcome categories and subcategories. The outcomes are arranged in a typology of five categories: education and skills, entrepreneurship, employment, welfare, and economic. For instance, the education and skills outcomes refer to notable achievements in education, satisfaction with quality of education programs; transformative experiences, career readiness and performance.

The welfare outcomes are flagged to ensure that even welfare outcomes not directly associated with employment are captured. For instance, some welfare outcomes (criminal and delinquent behaviour as well as citizenship values) may happen as result of participating in an intervention, though such youth could still be unemployed. The protocol also provided for inclusion of adverse and unintended outcomes in the map. That was important to avoid one‐sided summaries of the evidence. An example of unintended employment outcomes include youth offending such as, increased rate of alcohol abuse due to income resulting from change in employment status. An example of an adverse outcome is accidents and disease resulting from employment hazards. Multiplier effects were also coded. These refer to effects not directly from the programme such as job displacement and youth spending earnings to improve local commerce. Job displacement in this case refers to the loss of jobs by current employees due to the recruitment of new workers for example with more relevant skills for the employer.

Definitions of outcomes are provided in Supporting Information Annex 2, with associated reference list provided as Supporting Information Annex 3.

3.10. Eligibility criteria

A review of 15 agencies by Saran and White (2018), established that the inclusion criteria for EGMs generally follows the PICOS approach of population, intervention, relevant comparison groups, outcomes and study design. Therefore, the study inclusion criteria for this EGM also used a similar approach. Table 3 shows a summary of PICOS. Additional criterion of study publication period was included. The document should have been published or made available between January 2000 and December 2019. The inclusion and exclusion criteria is illustrated in (Figure 1).

Unpublished studies were included in the EGM to reduce the effects of publication bias, which refers to the failure to publish a study on the basis of the strength of the study findings.

For studies with interventions that combined youth and non‐youth populations (older adults and children), majority (over 51%) of the study population had to be youth. In addition, studies containing only subsets of eligible interventions were included in the map only when revenant outcomes were reported.

For a single study with multiple reports, it was included in the EGM multiple times only if the study employed different eligible study designs or reported different outcomes. However, if the multiple reports of a single study were for example a working paper and journal article, only one with the most detailed information (interventions and outcomes) was included in the EGM. For instance, where a journal article presented partial outcomes, a working paper was included in the EGM.

3.11. Types of settings

The coding tool provided for the following types of settings for interventions: high school, tertiary education institutions, training centres, firms, and the option for others identified when coding studies.

3.12. Search methods and sources

The Protocol which is the basis for the production of the Youth Employment EGM is published in the Campbell Systematic Reviews database (Apunyo et al., 2021).

3.13. Data collection

3.14. Quality appraisal of studies/risk of bias

Critical appraisal of each study (impact evaluations and systematic reviews) was conducted independently by a pair of reviewers who again followed the same procedure used at screening and coding phases, to reconcile disagreements.

Impact evaluation studies were assessed using the ‘Quality assessment of Impact Evaluations’ tool developed by Dr. Howard White and Dr. Saran Ashrita. The tool is a checklist of seven items with additional guidance on rating items, expressed as: high confidence, medium confidence or low confidence. However, of the seven items only four (study design (potential confounders taken into account); level of attrition or losses to follow up2; definition of outcomes; and baseline balance3 reports), were the most critical for making decisions.

A Measurement Tool to Assess systematic Reviews (AMSTAR 2) was used to conduct critical appraisal of systematic reviews. AMSTAR has been developed to facilitate the development of high‐quality reviews by guiding the conduct and evaluation of reviews. The AMSTAR 2 checklist4 contains 16 items, each with concise sentence questions having supplementary guidance on selecting response options (expressed as: ‘yes’, partial yes and ‘no’). Overall, the AMSTAR 2 tool rates confidence in components of a systematic review as; High: no or one no‐critical weakness, Moderate: more than one noncritical weakness, Low: one critical flaw with or without noncritical weaknesses and, Critically low: more than one critical flaw with or without noncritical weaknesses (Shea et al., 2017).

3.15. Methods for mapping

EPPI Reviewer 4—a web based software program for managing and analysing data in literature reviews (EPPI Centre, n.d.) was used for screening and coding studies as well conducting analysis. Screening and coding was based on predefined codes extracted from the eligibility criteria and coding sheet provide in Supporting Information Annex 5. The study flow diagram is presented later in Figure 5. The analysis involved generating frequency tables, cross‐tabulated tables, graphs and charts.

EPPI Mapper was used to generate the EGM. EPPI‐Mapper is a tool for visualising ‘maps’ of research evidence (EPPI Centre, n.d.). The tool is used generate an evidence and gap as follows. After coding data in EPPI Reviewer, a JSON file form is exported and uploaded to EPPI Mapping tool to generate the EGM, via: http://eppimapper.digitalsolutionfoundry.co.za/.

3.16. Analysis and presentation

4.1. Description of studies

4.2. Synthesis of included studies

4.3. Intervention categories

The five intervention sub‐categories are support to employment; decent work policies; training; information and finance and entrepreneurship. Training (in full training, up‐skilling and retraining or reskilling) was by far the most researched area (n = 283). This was followed by support into employment (n = 182). Together these two interventions are assessed in over 75% of the included studies. There are far fewer studies about the effect of information services, decent work policies, entrepreneurship promotion and financing on youth employment (Figure 6). The Figure 7 provides detailed coverage of intervention sub‐categories for the ‘training, upskilling and re‐training intervention category’ while Figure 8 shows intervention sub‐categories for the ‘support to employment’ intervention category.

Under the training and skilling category, most of the evidence was about education, technical and vocational training (TVET), n = 193, followed by the life skills training (n = 105) and internship and apprenticeship (n = 81) intervention sub‐categories. There are few studies regarding Prior Learning Assessment and Recognition (PLAR), n = 1, training of trainers and teachers (n = 2), and training centre accreditation and certification (n = 1) (Figure 7).

In support to employment intervention category, employee mobility and placement (n = 77) and mentoring (n = 67) were the leading intervention sub‐categories whilst career guidance (n = 46) and wage subsidies (n = 51) studies had little difference. There was an obvious evidence gap in the area of public works programmes (n = 8) (Figure 8).

4.4. Outcome categories

There are five outcome domains reported in the map: education and skills, entrepreneurship, employment, welfare and economic. Employment is the most reported outcome (n = 345) followed by welfare (n = 121) and, education and skills (n = 97) (Figure 9).

Among the outcome sub‐categories for employment outcomes, evidence is concentrated on employment status (including employment duration (n = 322), earnings and salary (n = 197), and hours worked (n = 89) (Figure 10). Within the outcome category of education and skills, education completion and qualification, n = 36 (Figure 14); and business creation, n = 54, under entrepreneurship category (Supporting Information Annex 8: Figure 15). For the outcome category welfare, the three most studies outcomes are citizenship, value and social behaviour, n = 62, inclusion and empowerment, n = 59, and economic outcomes (not including earnings), n = 54. The most reported economic outcomes are cost–benefit, n = 41 (Supporting Information Annex 8: Figure 16).

The clear evidence gaps in the employment outcome domain literature were about: vacancies, actively seeking employment, employment expectation and appropriate employment (Figure 10). Multiplier and spill over effects, digital skills and job creation sub‐categories were the obvious gaps in the evidence around economic, education and skills and entrepreneurship major outcome domains (Supporting Information Annex 8: Figures 13–15).

4.5. Aggregate map of evidence gaps

Table 12 is the aggregate map that shows the number of studies in each intervention and outcome category.

Well‐evidenced areas with more than 75 studies are the employment, welfare, as well as education and skills outcomes of training interventions. This level also is employment outcomes of support to employment interventions.

Moderately evidenced areas, with between 25 and 75 studies, are largely identified for support to employment interventions across a broader range of outcomes, except employment. Training interventions also have moderate level of evidence for economic and entrepreneurship outcomes. Similarly, entrepreneurship promotion and financing interventions had moderate level of evidence for employment and welfare outcomes. The evidence base is weak (fewer than 25 studies) especially for information intervention category across all outcomes.

4.6. Risk of bias in included reviews

The results from the quality assessment of systematic reviews using the AMSTAR 2 tool and, impact evaluation studies basing on the ‘Quality assessment of Impact Evaluations’ tool are presented below. These study quality assessment tools are detailed earlier in (Section 3.7). A detailed risk of bias or study quality table containing a list of 399 studies is provided in Supporting Information Annex 7.

Overall, most systematic reviews (71.4% of 21) were of medium and high quality while 28.6% were graded as low quality by AMSTAR score (Table 13). Across the intervention sub‐categories and outcomes categories, evidence quality was mainly rated as ‘medium/moderate and high quality’ (Table 14). The reviews with a low quality rating had at least one critical flaw (Table 15). For instance, all the reviews rated as low quality did not explicitly state if the review methods were established before the conduct of the review and did not also justify any significant deviations from the protocol. The inability of review authors accounting for risk of bias (RoB) in individual studies when interpreting/discussing the results of the review, was the second most dominant critical flaw (Table 15). A detailed report of RoB assessment for systematic reviews is provided in Supporting Information Annex 6. The conduct of systematic therefore calls for improvements in a number areas such as proper design of methods and their publication for transparency, undertaking risk of bias analysis and use of comprehensive literature search strategies.

Looking at impact evaluations, at least 73.4% of these category studies were of low confidence quality rating (Table 13). In terms of Intervention sub‐categories and Outcomes categories, the bulk of the evidence was of low quality impact evaluations across the board (Table 14). These studies obtained a low‐quality rating on at least one of the following three items: (1) high attrition rate; (2) design (potential confounders not taken into account and; (3) weak baseline balance performance. However, attrition bias and low baseline balance rating were the two main reasons for the low‐quality rating of impact evaluations.

Results of attrition assessment presented in Figure 11, show that most (70.4%) impact evaluations scored a low rating because either attrition was not reported or the reported measures fell outside What Works Clearinghouse (WWC) Standards acceptable combined levels. Ex‐post factor studies were not rated on this attrition parameter.

Regarding the ‘baseline balance parameter’,5 still a big number of studies (43.4%) scored a low rating because they did not have baseline balance data or were reported with significant differences on more than five measures (Figure 12). It means that most studies with treatment and comparison groups did not have the same average characteristics of participants at baseline. Such imbalances in participants' characteristics such as in the levels of education often affect the impact of the interventions.

The ‘before versus after’ studies were not rated on this parameter as it is not applicable. The RCTs were automatically rated high on this parameter.

4.7. Discussion

4.8. Potential biases in the mapping process

4.9. Conclusions

This report provides a summary of findings of the Youth Employment EGM that focuses on interventions aimed at increasing youth employment in any country of the world. The map was funded by Mastercard Foundation and Youth Futures Foundation and, developed by The African Centre for Systematic Reviews and Knowledge Translation—Makerere University of College of Health Sciences in partnership with the Campbell Collaboration Secretariat. The map is an effectiveness map. Hence included studies are impact evaluations of interventions to improve youth labour market outcomes or systematic reviews of such studies. In total 399 studies are included (21 systematic reviews and 378 impact evaluations).

The map identifies trends in evidence notably the following:

• Most evidence is from North America and Europe and as such, high‐income countries have the largest share of evidence.

• The most common study designs are experimental.

• Most of the evidence is of low quality. This finding serves to alert researchers, practitioners and policy makers that more rigorous work is needed to inform youth employment interventions.

• Blending of interventions is practiced. While this could be an indication blended intervention could be offering better outcomes, this remains an area with a research gap.

Inspecting the number of studies in each cell of the map shows clusters of evidence and gaps, as follows:

• Well‐evidenced areas, with more than 75 studies, are the employment, welfare, as well as education and skills outcomes of training interventions. This level also is employment outcomes of support to employment interventions.

• Moderately evidenced areas, with between 25 and 75 studies, are largely identified for support to employment interventions across a broader range of outcomes, except employment. Training interventions also have moderate level of evidence for economic and entrepreneurship outcomes. Similarly, entrepreneurship promotion and financing interventions had moderate level of evidence for employment and welfare outcomes.

• The evidence base is weak (fewer than 25 studies), for information and decent work policies.

The map reveals areas with well‐populated evidence as well as those with gaps. Hence to a great extent the map enhances discoverability of evidence by stakeholders engaged in the promotion of youth employment interventions.