Social researchers are used to handling data from a many diverse sources. In the space of a project timeline, a member of our team can, for example, gain a full understanding of a schools’ funding system, the main issues around racial harassment in UK higher education institutions (HEIs), or be able to recite the Graduate Outcomes Questionnaire word by word (one of us can do it backwards). Over many years of work within the HE sector, IFF have become experts in all kinds of areas within higher education and the broader education sector.
Every once in a while, though, a project comes along that feels particularly exciting; a true venture into unknown territory. IFF’s work, in partnership with research partner Ecorys, on the evaluation of the British Council’s Creative Spark programme very much fits this bill.
Whilst HEI portfolio development, partnership building and recruitment strategy are key areas that IFF work within, the evaluation of the Creative Spark programme, involving HEIs in the UK partnering with counterparts in Georgia, Kazakhstan, Kyrgyzstan, Azerbaijan, Uzbekistan and Armenia and Ukraine engaged us in the investigation a new educational landscape.
The creation of Creative Spark began with a report conducted by IFF research on behalf of the British Council published in May 2018. The report looked at the potential for partnership growth between UK and the wider Europe HEIs in the field of ‘creative enterprise’, making recommendations on the formation of a higher education enterprise programme that could facilitate and structure these partnerships.
Fast forward to November 2019, and Elizabeth Shepherd and Laura Hilger from our HE team presented findings from this study, and the monitoring and evaluation approach at the launch of Creative Spark programme in Tbisili, Georgia.
The Creative Spark Programme
Creative Spark is a British Council-funded programme involving the seven Wider European countries mentioned above. It is ambitious both in terms of scope and intended impact, funding 38 partnerships between UK HEIs and Wider Europe HEIs in the first year; with an initial year one aim to reach over 10,000 students and young entrepreneurs via Train the Trainer sessions, curriculum development and wider university strategy development to improve local capacity.
In addition to its overall objectives, the Creative Spark programme emphasises the need for partners to reflect and act upon local social issues when considering the design of their activities, and selection of beneficiaries.
The programme represents therefore a truly a multi-disciplinary approach, focusing on enterprise education for students and young entrepreneurs, raising awareness of the creative industries on a national level, and facilitating beneficiaries to tackle the most prevalent and challenging local social issues in their entrepreneurial activities.
Monitoring and Evaluation for year one
Seven Countries. 38 Partnerships. 88 individual organisations. Two researchers conducting qualitative fieldwork visits (with a few days of Russian Duolingo to help).
The scale of the programme was certainly a key challenge in developing an evaluation approach in collaboration with our colleagues from Ecorys. In development of a theory of change and analytical framework, the following monitoring & evaluation (M&E) objectives formed the bed rock of our approach:
- Assessing effectiveness of Creative Spark and sustainability of its outcomes and impacts over time
- Demonstrating long-term outcomes and impacts on programme countries’ economic growth, regional stability and engagement with UK culture and trade, in line with the British Council’s M&E responsibilities to its funders
- Demonstrating how the programme supports social inclusion and gender equality
- Building evaluation capacity across the partnerships and the British Council
- Helping track programme / partnership progress in a way that is consistent across partnerships and countries
Whilst designing the evaluation tools it was important for us not only to consider cultural and language barriers, functionality and design of online surveys and suitable timing for data collection, but also to acknowledge that the monitoring and evaluation for years 3, 4 and 5 of the programme would be led by the British Council project teams in each country. It was therefore essential for us to design an M&E programme that could be useable by the British Council following IFF’s involvement.
Learnings from year one
Now this would be giving it away a little! Next week we’ll be presenting the key findings from year one of Monitoring and Evaluation of the Creative Spark Programme at Queens College Oxford as part of the year two launch. As testament to the strength of the programme, the British Council have announced 12 additional partnerships which will be launched at the conference. More to come on that next week.
In the meantime, we will leave you with some pictures of the qualitative fieldwork visits across the seven Wider Europe countries which give some idea as to the enthusiasm, passion and dedication that partners and beneficiaries had for Creative Spark.
For more information on IFF’s work on monitoring and evaluation for the Creative Spark programme, and research within the HE sector please contact Elizabeth on Elizabeth.Shepherd@iffresearch.com
…And if you ever find yourself in Bishkek, Kyrgyzstan waiting for a delayed flight, make sure you try the Laghman.
Author: Daniel Salamons, Research Manager
Contemporary discussion about encouraging access to Master’s degree study has long contemplated the potential role of government-offered student loans in driving take-up of postgraduate study. Introduced in 2016, the Master’s Degree Loan Scheme marks the first time that the Government has offered such a contribution to the costs of Master’s study, in an attempt to bring the student loan offer for postgraduate students somewhat in line with that offered to undergraduates.
Given the importance of this DfE initiative for postgraduate students, IFF are delighted to see that our independent evaluation of the Master’s Degree Loan has been published by the Department for Education. This report presents the findings of a survey of 2,002 Master’s students who started their study in 2016/17 and a survey of 79 providers of postgraduate education as well as analysis of historic secondary data. Findings from the quantitative study were also supplemented by qualitative interviews with Master’s students.
This study found that the English-domiciled Master’s population increased by a third between 2015/16 and 2016/17, suggesting at this early stage that the loan has been successful in increasing access to Master’s level education. This was accompanied by an increase in the proportion of students from a BAME background that made up the loan-eligible population of students studying at Master’s level, suggesting some success in widening participation as well.
Our research also uncovered evidence of the Loan leading to earlier access to the benefits enjoyed after students graduate, facilitating full-time rather than part-time study at postgraduate level for some students. This was not echoed, however, through any conclusive evidence that the Loan has reduced the average time gap that passes between undergraduate and postgraduate study.
From the perspective of Higher Education Institutions, our report also demonstrates that the Loan will help to provide more sustainable income for the sector, with institutions benefitting from increased student volumes in 2016/17. Half of institutions surveyed also reported that they believe the Loan will lead to increased revenue in the future.
On the same day that the report was published, Chris Skidmore MP, the Minister of State for Universities, Science, Research and Innovation, quoted our findings as part of his speech at the London School of Economics regarding governmental efforts to “secure the research talent of tomorrow”.
Across the UK’s higher education sector, competition to attract and retain students is fierce. To improve their competitiveness against their competitors and develop their portfolios to ensure student success in the labour market, universities are increasingly engaging with research and data to inform their decision-making and support the development of their degree portfolios and wider student offer. Our experience shows a growing number of requests from higher education institutions for primary research to support portfolio planning, often seeking information from current or prospective students and/or employers to answer these questions. While this view is indeed an important consideration, we argue that the existing data landscape provides universities with the vast majority of information needed for informed portfolio planning – with little primary research needed. By taking a ‘data re-sight’ approach, universities can themselves (or with expert guidance) mine existing data to take a data-driven, future-facing approach to portfolio strategy.
In order to undertake this type of informed and data-driven approach to portfolio planning, universities must blend internal knowledge of their strengths and weaknesses, university plans and overall capabilities with population projections, performance of the existing portfolio and graduate outcomes, and labour market projections to develop a comprehensive view of their current strengths and weaknesses vs. future labour market needs. This can be used to redevelop their portfolio and wider student offer to ensure the portfolio and student offer are designed to meet future skills needs and market gaps.
Whilst complex and often disparate, much of this data already exists within the public domain, with a range of potential sources of information that can be used to support the evaluation of each of these areas:
- Population Projections: which demonstrate how the population is expected to change and, from this, the volumes completing secondary education and therefore eligible to enter tertiary education over the coming years – as available from the Office of National Statistics, Department for Education or other Higher Education bodies like Universities UK, and from many local authorities who produce projections for the total population and for pupil/student demand.
- Portfolio Performance: from overall and subject-level league tables (like the Complete University Guide and the Guardian, as well as TEF ratings), course-based student satisfaction and outcomes from the National Student Survey (NSS), and data on graduate outcomes from the Destination of Leavers from Higher Education (DLHE and Longitudinal DLHE), going forward from the new Graduate Outcomes study, and from the new LEO data.
- Labour Market Projections: national-level projections have been produced by UKCES to 2024 (‘Working Futures’) and Nesta to 2030 (‘The Future of Skills’), whilst local authorities also create local-level projections that are worth consideration, e.g. Greater London Authority’s 2017 ‘Long Term Labour Market Projections’.
Combining data from these sources offers universities an indication of how they could leverage portfolio strengths to support future growth and/or address portfolio weaknesses where labour market projections show a positive picture. The table below is an example from our recent work, which compared sectors with strong labour market projections against existing portfolio performance (based on subject league tables and NSS results for all courses in that subject area) in order to create a SWOT analysis of the university’s portfolio to support future planning.
In this instance, the ‘re-sight’ exercise offered a clear picture of where the university’s portfolio was strong against its competitors and in the context of future labour market needs, giving a steer on where the portfolio could be leveraged to ensure growth and positive student outcomes. Areas of opportunity were identified based on subject areas with a strong outlook in the labour market, but where current performance was average and would need bolstering (especially in relation to student satisfaction and outcomes) in order to be competitive. Weak-performing courses were referenced as those where substantial investment would be needed to existing courses to make them truly competitive within the competitive landscape, despite their labour market strengths.
Using this method, primary research should be designed to build on these findings, mindfully supplementing existing data with additional insights. For example, recent data re-sight projects have included primary research components with stakeholder interviews to understand the internal perspective on the portfolio’s strengths, weaknesses and areas of opportunity; focus groups with students or local employers to develop their perspectives on what courses need to provide in order to better ensure competitive graduates that are workforce-ready; and desk research into the competitor landscape to assess portfolio gaps against what others offer.
Taking a data re-sight approach is a cost- and time-efficient means for universities to gather the necessary information needed to make informed choices when university portfolio planning or defining their portfolio strategy, supplementing this only where needed with primary research. By triangulating existing, publicly available information and taking into account future forecasts, this approach ensures focussed, clear results for universities to take forward as they work toward aligning their portfolio to meet future labour market and skills needs.
British Council Regional Director, Education and Society, Wider Europe region
Across Wider Europe, a geographic region that spans Russia and 14 neighboring post-Soviet states, there is a mix of active and frozen conflicts; countries with large youth populations and high unemployment rates, in some cases up to 50%. Alongside this, there is a perception within the region that young people, including graduates, lack the skills and resilience needed to gain employment and play a positive role within society. This is a real issue for governments in the region, their economies, prosperity and security.
Within the region, 16m students are enrolled in over 1,000 Higher Education institutions, very few of which feature in international rankings. The rest, by and large, vary in quality, struggle with international engagement and are not perceived as making significant contributions to the global knowledge economy. Graduate employment levels are below international standards, and there is a lack of robust data collection methodology for analysis at scale. Quality and relevance of teaching also varies, and reform is hindered by inefficiency and, sometimes, corruption. There are few prominent, female leaders. The 16,000 VET schools operate without strategic links to the higher education sector, and there is a general lack of alignment between education systems and labour market needs
The future success of these societies and economies will require young leaders to be inspired and connected, and for young people to have skills for employability and an ability to communicate, reach out and prosper both as individuals and within wider communities.
To support these goals, the British Council has designed and implemented the Creative Spark programme across seven countries in this region. The programme was designed to stimulate entrepreneurship and enterprise skills in the creative economy sector to students, via the development of university partnerships. In the 2018/19 academic year, 38 UK universities are sharing their experience of delivering enterprise into the teaching curriculum, incubating small businesses and running training courses with universities in the region. The second call for partnership grants is currently active, closing on 12 May 2019, and aims to increase the number of partnerships to 50.
The success of the first year of this international higher education partnership programme was built on the evidence-led approach taken at the outset of the programme design phase. The report ‘Driving Creative Enterprise: analysis of UK higher education and partnership potential for Wider Europe’, produced by IFF Research, was a critical part of the process. The report provided evidence of the innovation taking place throughout UK universities in delivering entrepreneurship, together with the latest data evidencing the growth of the creative economy sector.
Prompted by the high level of engagement between UK institutions and those in the region, the British Council are now designing a regional Higher Education and Employability programme which includes working with national education ministries and associated agencies to develop methodologies for measuring graduate outcomes. We are basing the programme design on the UK’s world-leading expertise in gathering data to evidence graduates’ post-study employment outcomes, and to support governments to improve the alignment of skills provision for young people with the requirements of the labour market. The provision of high quality, consistent data will contribute towards understanding and supporting the needs of young people, which in turn will help to increase levels of employment, and therefore create stronger cultural relations with the UK in the long term.
Our previous entry provided a broad overview of the importance of universities focussing on student outcomes. We build on this now by exploring a particular avenue of this theme, namely what employers look for when recruiting for graduate level (or professional) roles.
For the purposes of this blog we focus on results from the most recent Employer Skills Survey (ESS). This survey, undertaken by IFF on a biennial basis on behalf of the Department for Education is – we believe – the largest telephone survey in the UK, acquiring interviews with close to 90,000 employers. The survey offers a rich stream of data, covering recruitment, skills shortages, training, underutilisation of staff, and the monetary investment towards training. Our focus here is on skills shortages specifically: the recruitment difficulties employers face, and the skills lacking from applicants, particularly those applying to graduate level roles.
In the most recent ESS study, published in 2018, UK employers reported a total of 309,000 vacancies for positions at graduate level, i.e. managers, professionals and associate professionals. Interestingly, around half of these vacancies occurred in just two sectors: Business Services (which includes recruitment agencies, business support companies, law firms etc.), and Health and Social Work. The most common professions therefore where employers were citing vacancies at a graduate level were Nurses, HR managers, Sales executives and IT professionals: clear areas where universities can look to focus their employability interventions.
Around 25% (79,000) of these graduate-level vacancies were considered hard to fill due to a skills shortage within the labour market (remaining vacancies occurred due to non-skills related issues such as low numbers of applicants, competition from other employers, poor terms and conditions of the job etc.). We consistently find each time we run the Employer Skills Survey that the skills lacking in applicants relate as much to ‘human’ or ‘people’ skills as they do to technical ones. As the chart below highlights, the ability to manage their own time and prioritise tasks is considered to be the most common skill lacking among those applying to professional roles: indeed, 43% of all professional-level skills shortage vacancies were due to a lack of time management skills. Other common skills lacking include the ability to persuade or influence others, and managing or motivating other staff.
Due to the sheer volume of interviews acquired, the survey also allows us to explore regional and local skills needs, to enable universities whose graduates are typically less mobile focus their employment support according to local needs.
The ESS is a vast (and complex) data source, but when used appropriately can bring individual universities new insights into the recruitment needs of local and target employers. It is a relatively untapped source of data offering masses of potential to universities as they seek to help students progress into the workplace, and therefore improve their institution’s student outcomes.
Dr Anthony Mansfield
University of East London
At the heart of Higher Education (HE), universities strive to ensure students achieve their aspirations. For many prospective and current students, the baseline after their journey through HE is a meaningful and fulfilling career. Of course, efforts are routinely made to enhance student and graduate opportunities, however, in the ever-changing landscape of employment the question remains: how do HE providers ensure the programme portfolio and wider provision of support give students the best possible prospects?
The environment that universities operate within has laid out a clear mandate to meet the challenges of the twenty-first century. Three drivers outside of university fuel the desire for increased knowledge of the labour market: government, public perception, and the employer – all of which have an interconnected relationship.
University leadership has always been interested in league tables. Published rankings have measured employability amongst other metrics such as good honours, completion, and even research quality. Therefore, a focus on improving these metrics has always been present. Government emphasis on outcomes of education, nevertheless, has grown in part due to the availability of big datasets. The impetuous has been underscored in the formulation of the Teaching Excellence and Student Outcomes Framework (TEF) as well as the Longitudinal Education Outcomes (LEO) dataset. The former – up to this point – has focussed heavily on the rate of graduates in highly-skilled roles; by contrast, the latter has highlighted the earnings of graduates. Such publications and datasets are visible and impact the reputation of a university at a time when Higher Education providers compete in fierce conditions for undergraduate enrolments.
We cannot underestimate the influence of tuition fees in the renewed focus on outcomes. Students carry a heavy burden through taking out loans and leaving HE with up to fifty-thousand pounds worth of debt – a wellbeing as well as financial drain. The increasing levels of participation, furthermore, have seen regular column inches dedicated to challenging the significance of HE and at times diminishing the achievement of graduating. The perception has become that a degree is commonplace and not an accomplishment in itself, has permeated the labour market too.
The increasing number of graduates gives employers a highly-educated workforce, and they seek, therefore, to differentiate candidates. Other skills, competencies and experiences are desired in addition to a degree as society stands on the precipice of Industry 4.0. In the age of big data and increasing impact of automation, employers need a workforce that is resilient, adaptable, innovative and passionate.
None of the observations above is to say that the stress on outcomes is misplaced or incorrect; instead it has required universities to recalibrate their efforts in which they hone in on the later stage of the student journey. An undergraduate degree can instil the skills and traits that an employer needs; however, the prevailing belief is that graduates need more and consequently the university needs to provide students with guidance, training and significant job opportunities.
Rightly, senior leaders, academics, and service staff are working to understand the labour market to prepare their students for life after graduation. To provide answers to these challenges there needs to be a strategic approach in understanding the environment outside of the institution’s walls. In any organisation, leaders need to have information to allow them to appreciate the broader context and confidently make decisions in the best interest of the students. For many institutions, such contextual insight is gleaned through a strategic horizon scanning function.
The horizon scanning function – often an activity led by a Strategic Planning unit– allows a university to see emerging trends that are not clear today. The process enables the assessment of potential opportunities and threats for effective planning and portfolio management. Understanding the labour market over the next ten years is not a light undertaking; gaining a sense of the market for just 2019 is not enough either. The decisions that will be made take time to implement and bare fruit; therefore, the labour market needs to be evaluated for at least the next ten years.
For a university, initiatives that promote highly-skilled employment are not enough. The programme portfolio needs to be critically reviewed to establish if it provides students with what they need for skilled jobs. Consequently, a deep dive of the labour market through investigation of large quantitative datasets, policy reports and even qualitative interviews is crucial from sources such as government, professional bodies, policy groups, and confederations. We should not forget the importance of the qualitative approach supporting the use of big data as projections based solely on datasets can be misleading in predicting future trends in the labour market – a poignant consideration as we come closer to the integration of further automation.
Although conclusions drawn from these datasets focus on the student journey from the perspective of outcomes they do also inform the front end. Universities can use this data to review their portfolio and align it to growing sectors in the economy which will meet the demand of prospective students and employers. As a student enters the university, an institution can provide assurance and opportunities outside of the classroom by using labour market data to set up initiatives (accreditations, knowledge exchange partnerships, apprenticeships or even internships) and develop programmes that allow students to develop their key competencies.
In the end, the focus on the outcome has caused a renewed appraisal of the student journey in its entirety which cannot be considered a bad thing.
This week, it will be 20 years since I started working at IFF Research, so I thought it was a good time for a quick spot of reminiscing, and to reflect on some of the work I’ve done and the impact it has had.
For a large chunk of this time I’ve been heavily involved in the development of the Employer Skills Survey (ESS) series (and its predecessors). As I’ve discussed before, and as my colleague Andrew highlighted only last week, ESS is one of the largest, if not the largest, employer skills surveys in the world, and is an envied source of insight that provides the gold standard for labour market information.
ESS has always had a significant impact on national level policy. It directly informs UK immigration policy through its use by the Migration Advisory Committee to help decide which occupations should be placed on Shortage Occupation List. And it is a key source used to measure the impact that various policies have on employers and their staff (e.g. the Apprenticeship levy and wider funding reform, T-levels). It is also widely used by Local Enterprise Partnerships to help develop regional skills strategies.
Over the last decade or so, I’ve been doing more and more work within Higher Education (HE), and now head up IFF’s large and rapidly growing HE offer. We conduct some of the highest profile studies for the likes of DfE, HESA and OfS, and work with individual institutions such as University of Portsmouth, University of East London, University of London and Open University.
The landscape of higher education is constantly evolving, but over the last decade there has been a period of fundamental change – tuition fees, alternative providers, lifting of the student cap on student numbers, TEF, proliferation of league tables, Brexit etc., etc., etc.. One change that I have had first-hand experience of, since IFF first undertook the Longitudinal DLHE survey for HESA in 2005, is the increased scrutiny placed on demonstrating evidence of successful progression of graduates into the labour market.
However, despite the critical importance of graduate outcomes to the sector, it still surprises me how little is known about ESS and how little it is used in institutional planning and portfolio development. To some extent, this could be blamed on the fact that ESS has not always been publicised as much as it could or should have been, and other less reliable surveys with much smaller sample sizes often get more attention in the press (take CBI’s Education and Skills Survey, for example). However, there is also a lack of widespread understanding of other powerful data sources that would help providers better understand their local labour market. Working Futures is another example. Working Futures projects the future size and shape of the labour market by anticipating changing skills needs – powerful insight for providers looking to refine their provision to best meet employer needs.
In the Advance HE conference next week, my colleague Liz will be delivering the keynote speech investigating the process of aligning higher education and the labour market, and will provide a practical framework for doing so. I hope that, through IFF’s involvement in events like this, over the next few years we will get more opportunities to help people understand the value that we see in this sort of labour market data, and to see it have a greater impact on the HE sector.
Latest News & Blogs
Get in touch
5th Floor, St Magnus House
3 Lower Thames Street
0207 250 3035