Social researchers are used to handling data from a many diverse sources. In the space of a project timeline, a member of our team can, for example, gain a full understanding of a schools’ funding system, the main issues around racial harassment in UK higher education institutions (HEIs), or be able to recite the Graduate Outcomes Questionnaire word by word (one of us can do it backwards). Over many years of work within the HE sector, IFF have become experts in all kinds of areas within higher education and the broader education sector.
Every once in a while, though, a project comes along that feels particularly exciting; a true venture into unknown territory. IFF’s work, in partnership with research partner Ecorys, on the evaluation of the British Council’s Creative Spark programme very much fits this bill.
Whilst HEI portfolio development, partnership building and recruitment strategy are key areas that IFF work within, the evaluation of the Creative Spark programme, involving HEIs in the UK partnering with counterparts in Georgia, Kazakhstan, Kyrgyzstan, Azerbaijan, Uzbekistan and Armenia and Ukraine engaged us in the investigation a new educational landscape.
The creation of Creative Spark began with a report conducted by IFF research on behalf of the British Council published in May 2018. The report looked at the potential for partnership growth between UK and the wider Europe HEIs in the field of ‘creative enterprise’, making recommendations on the formation of a higher education enterprise programme that could facilitate and structure these partnerships.
Fast forward to November 2019, and Elizabeth Shepherd and Laura Hilger from our HE team presented findings from this study, and the monitoring and evaluation approach at the launch of Creative Spark programme in Tbisili, Georgia.
The Creative Spark Programme
Creative Spark is a British Council-funded programme involving the seven Wider European countries mentioned above. It is ambitious both in terms of scope and intended impact, funding 38 partnerships between UK HEIs and Wider Europe HEIs in the first year; with an initial year one aim to reach over 10,000 students and young entrepreneurs via Train the Trainer sessions, curriculum development and wider university strategy development to improve local capacity.
In addition to its overall objectives, the Creative Spark programme emphasises the need for partners to reflect and act upon local social issues when considering the design of their activities, and selection of beneficiaries.
The programme represents therefore a truly a multi-disciplinary approach, focusing on enterprise education for students and young entrepreneurs, raising awareness of the creative industries on a national level, and facilitating beneficiaries to tackle the most prevalent and challenging local social issues in their entrepreneurial activities.
Monitoring and Evaluation for year one
Seven Countries. 38 Partnerships. 88 individual organisations. Two researchers conducting qualitative fieldwork visits (with a few days of Russian Duolingo to help).
The scale of the programme was certainly a key challenge in developing an evaluation approach in collaboration with our colleagues from Ecorys. In development of a theory of change and analytical framework, the following monitoring & evaluation (M&E) objectives formed the bed rock of our approach:
- Assessing effectiveness of Creative Spark and sustainability of its outcomes and impacts over time
- Demonstrating long-term outcomes and impacts on programme countries’ economic growth, regional stability and engagement with UK culture and trade, in line with the British Council’s M&E responsibilities to its funders
- Demonstrating how the programme supports social inclusion and gender equality
- Building evaluation capacity across the partnerships and the British Council
- Helping track programme / partnership progress in a way that is consistent across partnerships and countries
Whilst designing the evaluation tools it was important for us not only to consider cultural and language barriers, functionality and design of online surveys and suitable timing for data collection, but also to acknowledge that the monitoring and evaluation for years 3, 4 and 5 of the programme would be led by the British Council project teams in each country. It was therefore essential for us to design an M&E programme that could be useable by the British Council following IFF’s involvement.
Learnings from year one
Now this would be giving it away a little! Next week we’ll be presenting the key findings from year one of Monitoring and Evaluation of the Creative Spark Programme at Queens College Oxford as part of the year two launch. As testament to the strength of the programme, the British Council have announced 12 additional partnerships which will be launched at the conference. More to come on that next week.
In the meantime, we will leave you with some pictures of the qualitative fieldwork visits across the seven Wider Europe countries which give some idea as to the enthusiasm, passion and dedication that partners and beneficiaries had for Creative Spark.
For more information on IFF’s work on monitoring and evaluation for the Creative Spark programme, and research within the HE sector please contact Elizabeth on Elizabeth.Shepherd@iffresearch.com
…And if you ever find yourself in Bishkek, Kyrgyzstan waiting for a delayed flight, make sure you try the Laghman.
Author: Daniel Salamons, Research Manager
Contemporary discussion about encouraging access to Master’s degree study has long contemplated the potential role of government-offered student loans in driving take-up of postgraduate study. Introduced in 2016, the Master’s Degree Loan Scheme marks the first time that the Government has offered such a contribution to the costs of Master’s study, in an attempt to bring the student loan offer for postgraduate students somewhat in line with that offered to undergraduates.
Given the importance of this DfE initiative for postgraduate students, IFF are delighted to see that our independent evaluation of the Master’s Degree Loan has been published by the Department for Education. This report presents the findings of a survey of 2,002 Master’s students who started their study in 2016/17 and a survey of 79 providers of postgraduate education as well as analysis of historic secondary data. Findings from the quantitative study were also supplemented by qualitative interviews with Master’s students.
This study found that the English-domiciled Master’s population increased by a third between 2015/16 and 2016/17, suggesting at this early stage that the loan has been successful in increasing access to Master’s level education. This was accompanied by an increase in the proportion of students from a BAME background that made up the loan-eligible population of students studying at Master’s level, suggesting some success in widening participation as well.
Our research also uncovered evidence of the Loan leading to earlier access to the benefits enjoyed after students graduate, facilitating full-time rather than part-time study at postgraduate level for some students. This was not echoed, however, through any conclusive evidence that the Loan has reduced the average time gap that passes between undergraduate and postgraduate study.
From the perspective of Higher Education Institutions, our report also demonstrates that the Loan will help to provide more sustainable income for the sector, with institutions benefitting from increased student volumes in 2016/17. Half of institutions surveyed also reported that they believe the Loan will lead to increased revenue in the future.
On the same day that the report was published, Chris Skidmore MP, the Minister of State for Universities, Science, Research and Innovation, quoted our findings as part of his speech at the London School of Economics regarding governmental efforts to “secure the research talent of tomorrow”.
Across the UK’s higher education sector, competition to attract and retain students is fierce. To improve their competitiveness against their competitors and develop their portfolios to ensure student success in the labour market, universities are increasingly engaging with research and data to inform their decision-making and support the development of their degree portfolios and wider student offer. Our experience shows a growing number of requests from higher education institutions for primary research to support portfolio planning, often seeking information from current or prospective students and/or employers to answer these questions. While this view is indeed an important consideration, we argue that the existing data landscape provides universities with the vast majority of information needed for informed portfolio planning – with little primary research needed. By taking a ‘data re-sight’ approach, universities can themselves (or with expert guidance) mine existing data to take a data-driven, future-facing approach to portfolio strategy.
In order to undertake this type of informed and data-driven approach to portfolio planning, universities must blend internal knowledge of their strengths and weaknesses, university plans and overall capabilities with population projections, performance of the existing portfolio and graduate outcomes, and labour market projections to develop a comprehensive view of their current strengths and weaknesses vs. future labour market needs. This can be used to redevelop their portfolio and wider student offer to ensure the portfolio and student offer are designed to meet future skills needs and market gaps.
Whilst complex and often disparate, much of this data already exists within the public domain, with a range of potential sources of information that can be used to support the evaluation of each of these areas:
- Population Projections: which demonstrate how the population is expected to change and, from this, the volumes completing secondary education and therefore eligible to enter tertiary education over the coming years – as available from the Office of National Statistics, Department for Education or other Higher Education bodies like Universities UK, and from many local authorities who produce projections for the total population and for pupil/student demand.
- Portfolio Performance: from overall and subject-level league tables (like the Complete University Guide and the Guardian, as well as TEF ratings), course-based student satisfaction and outcomes from the National Student Survey (NSS), and data on graduate outcomes from the Destination of Leavers from Higher Education (DLHE and Longitudinal DLHE), going forward from the new Graduate Outcomes study, and from the new LEO data.
- Labour Market Projections: national-level projections have been produced by UKCES to 2024 (‘Working Futures’) and Nesta to 2030 (‘The Future of Skills’), whilst local authorities also create local-level projections that are worth consideration, e.g. Greater London Authority’s 2017 ‘Long Term Labour Market Projections’.
Combining data from these sources offers universities an indication of how they could leverage portfolio strengths to support future growth and/or address portfolio weaknesses where labour market projections show a positive picture. The table below is an example from our recent work, which compared sectors with strong labour market projections against existing portfolio performance (based on subject league tables and NSS results for all courses in that subject area) in order to create a SWOT analysis of the university’s portfolio to support future planning.
In this instance, the ‘re-sight’ exercise offered a clear picture of where the university’s portfolio was strong against its competitors and in the context of future labour market needs, giving a steer on where the portfolio could be leveraged to ensure growth and positive student outcomes. Areas of opportunity were identified based on subject areas with a strong outlook in the labour market, but where current performance was average and would need bolstering (especially in relation to student satisfaction and outcomes) in order to be competitive. Weak-performing courses were referenced as those where substantial investment would be needed to existing courses to make them truly competitive within the competitive landscape, despite their labour market strengths.
Using this method, primary research should be designed to build on these findings, mindfully supplementing existing data with additional insights. For example, recent data re-sight projects have included primary research components with stakeholder interviews to understand the internal perspective on the portfolio’s strengths, weaknesses and areas of opportunity; focus groups with students or local employers to develop their perspectives on what courses need to provide in order to better ensure competitive graduates that are workforce-ready; and desk research into the competitor landscape to assess portfolio gaps against what others offer.
Taking a data re-sight approach is a cost- and time-efficient means for universities to gather the necessary information needed to make informed choices when university portfolio planning or defining their portfolio strategy, supplementing this only where needed with primary research. By triangulating existing, publicly available information and taking into account future forecasts, this approach ensures focussed, clear results for universities to take forward as they work toward aligning their portfolio to meet future labour market and skills needs.
Latest News & Blogs
Get in touch
5th Floor, St Magnus House
3 Lower Thames Street
0207 250 3035