I was recently part of a panel session looking at Challenges for quality assurance in the wake of the pandemic – ensuring quality in digital learning, academic integrity in remote assessment, and emerging best practice
I gave a five minute presentation from my personal perspective.
We knew what we were doing.
We knew how to assure quality and academic integrity.
We thought everything was going to be fine….
There were early signs of the impact of Covid-19 back in December 2019 and January 2020.
Well that didn’t go as planned and was challenging to move online.
Moving assessment online is a journey and one for which we had no map and no real idea of where we wanted to get to.
We thought we could do it?
We thought technology was the solution.
It wasn’t.
The thing is translating practice online doesn’t really work. You lose the nuances of what made the in-person experience so great and you don’t take the advantage of the affordances of what digital can bring.
Can we be surprised when the quality of the experience suffers
Moving assessment online is not only possible, but can be done in a way which maintains quality and integrity.
How can you do this when the requirements in place from university exam board, PSRBs and professional accreditation are based on an model which expects a physical in-person assessment.
You need to transform and reimagine assessment.
You can not transform assessment on your own.
However this takes time.
It takes a certain level of digital capability.
It requires an insight into what we mean by assessment and what we are trying to assess.
At Jisc we are here to help the higher education sector through the use of digital to reimagine teaching, learning and assessment and to reframe the student experience.
I started as Head of Higher Education at Jisc on the 1st March 2019. So I have done two years (and a bit), had three line managers, a changing role and, oh yes, a global pandemic.
On Monday I had an excellent conversation with Isabel Lucas from Cumbria about the HEDG meeting I was presenting at, at the end of the week.
I have been sharing internally (and externally) the draft of the Jisc HE Strategy.
As higher education institutions plan for what will happen as we move slowly towards more students being on campus, there is continuing chatter about the form that teaching and learning will take. This includes how best to deliver it and how to communicate what this might look like. In all of this discussion, there has been a proliferation of words like “remote learning”, “digital learning”, and “hybrid learning” – and these terms have largely been taken for granted in respect to their pedagogical nuance. But if the preferred solution to the problems created by the pandemic in the first semester was “blended learning”, as we tumble through a second semester it would appear that the HE sector is beginning to settle on its next term of preference – “online learning”.
We do seem to spend a lot of time discussing what we should call what we do. The article makes the point that this does matter. I disagree slightly what matters is not what it is called, but whatever it is called, we have an agreed and shared understanding of what is means for you, for me and the students. We change the term, but this doesn’t necessarily mean that we change our understanding. I recall having this discussion about the use of the term hybridthat I used in an article to mean responsive and agile, whilst someone else was using the term to describe a mixed approach. Words are important, but shared understanding actually allows us to move forward.
Wednesday I joined a panel at the Westminster Education Forum to deliver a session on the future use of technology in assessment.
“The future for England’s exam system – building on best practice from the 2020 series, the role of technology and ensuring qualifications equip young people with the skills to succeed post-18”
I only had five minutes, so not a huge amount of time to reflect on the challenges and possibilities. To think a year ago I would have had to travel to London by train, find the venue and then join the panel in-person. Today, I just switched on the webcam and there I was, did my presentation and then answered a few questions. I didn’t use slides, as there wasn’t always a need to use slides in these kinds of panel sessions and at an in-person event I wouldn’t have used slides.
Of course at a physical in-person event they would have provided lunch, which would then give delegates an opportunity to come and chat about what I had been talking about. That didn’t happen this time. I would say that though using Twitter as a digital back channel at physical in-person events does sometimes work, but people have to be using the Twitter. At edtech events I find a fair few people are , at other kinds of events, not so much.
Lecturers are doing all they can during the pandemic to support the myriad different ways in which students learn
It’s not as though the physical campuses are closed, they are open for those courses which require a practical element.
Then again schools are not closed, they are open for the children of key workers, as well as vulnerable children, and staff are working with them and delivering remote teaching to the children at home.
Yes the experience is variable across the country, even across a school, but to keep saying they are closed, doesn’t really tell the whole story.
On Thursday I had a really good discussion with a university about digital strategy. How important it is aligned to the main university strategy, but also how it enables that strategy and other strategies as well. If you are in charge of a strategy, how does it enable others, and how do others enable yours?
At the end of the week I was involved in the HEDG meeting and did a presentation on Jisc’s Learning and Teaching Reimagined programme and where Jisc is going next. It was good meeting and the presentation seemed to hit the spot.
I had a planning meeting about a session we’re doing with Advance HE on digital leadership which looks like it will be a really good session.
Looked at the presentation I am doing next week at Digifest on the future of digital leadership, what it is and where we are potentially going.
Strange things sometimes happen in universities and we’ve reported plenty of them here over the years. From hauntings and strange happenings to animal action and of course true crime events on campus. But this event which recently caught my eye is one of the oddest I’ve noticed lately. It all happened just over half a century ago at Keele University. Those were turbulent times as the world transitioned out of the end of the heady 60s era into a very different decade.
My top tweet this week was this one.
I’m tired of hearing that universities are closed – it simply isn’t true | The Independent https://t.co/Z5h0QU98Ji
Well no snow again… everyone else seems to have had it.
Got bogged down into internal systems and processes this week, something which I have been avoiding in my new role, well I say new role, I have been doing this job for nearly two years now. In my old role I would have to deal with contracts and suppliers, not so much as Head of HE. However with working with universities on digital strategy, digital leadership and blended learning, means I need to immerse myself back into these systems.
Did a fair amount of work on future visions this week.
I think this is an interesting concept for future online events. The concept is to record some keynote videos that can be made available for online conferences. Reminded me of how some have used TED talks in the past. However there is a difference between YouTube or Netflix and an online conference. How do you add value to an event so that it is more than just streamed video? How do you facilitate social interactions, networking, discussion and also how do you encourage this?
Had an interesting and useful discussion on assessment this week with colleagues from the UK and Australia. The key challenges the HE sector have faced over the pandemic include: maintaining academic standards and quality as assessment is transformed, student wellbeing and engagement in regard to assessment, the skills and capabilities of staff to assess online, and how to transform at scale and at pace. There are still issues with assessing within in vocational and practical qualifications, There are challenges with PSRBs and Professional Certification.
These are the main challenges and pain points that arose from our research back in April and more recently, it is not an exclusive list and is potentially going to change as universities move again through the assessment process and learn new lessons on what they can and can not do.
Maintaining the academic standard and quality as required by internal and external regulations, as they translate and convert existing practice into online modes.
Ensuring staff have the necessary digital skills and capabilities to successfully deliver online assessment, across the assessment lifecycle. Each step of the lifecycle will require different skills to deliver.
Transform multiple modes of assessment to online versions at scale and at pace. Many universities have experience of designing and delivering online assessment, however they will not have done this at scale or transformed at the pace required.
Maintain student engagement through the next few weeks and through the assessment process, as they continue to socially isolate and study remotely.
Ensure student wellbeing during a time of crisis remotely and consider the impact of online assessment on wellbeing as an extra pressure and source of stress.
What technologies are out there that could be used to design, deliver and support online assessment? Which technologies should we be using?
What are other universities doing with online assessment? What best practice is out there? Who is doing it well? How do we compare?
The US election continues to dominate Twitter though seeing less of it on the mainstream news. Saw a number of people on Twitter claiming to have won the election!
Five years ago this week myself and Lawrie were delivering the second residential of the pilot for the Jisc Digital Leaders Programme at the Holland House Hotel in the heart of Bristol. We had spent four days delivering that week. We also had some great cakes and pastries.
Even the coffee was nice. We learnt a lot from the process and spent the next few months iterating the programme, dropping and adding stuff based on the feedback we had from the pilot delegates.
Less than a year later we delivered the programme to paying delegates in Loughborough, again we reviewed what we did and adapted the programme again, before delivering to groups in Manchester, Belfast and Leicester.
I had a fair few meetings this week on a range of topics, including learning and teaching, the Data Matters conference, consultancy, pipelines, and public affairs.
AI software designed to monitor students via webcam as they take their tests – to detect any attempts at cheating – sometimes fails to identify the students due to their skin color.
I am not surprised, in my work on the Intelligent Campus, when we did some research into facial recognition, there was quite a bit of coverage about how it only really worked with white males. Can we be surprised then when used for exam invigilation that it fails on the same issue?
Women with darker skin are more than twice as likely to be told their photos fail UK passport rules when they submit them online than lighter-skinned men, according to a BBC investigation. One black student said she was wrongly told her mouth looked open each time she uploaded five different photos to the government website.
There is a question here about removing the systemic bias we find in AI and algorthims being used in education (as well as the wider society). A deeper question is how does that bias get there in the first place?
Across the week we saw more universities report large covid-19 infections in their student populations.
Sheffield Hallam has seen over 370 cases of Covid since the beginning of term and the University of Sheffield has seen 589 cases. The local area has also seen a dramatic increase in the number of people testing positive.
Another 1,600 students have tested positive for coronavirus at Newcastle’s two universities. Newcastle University says 1,003 students and 12 members of staff have tested positive for Covid-19 in the past week. That’s up from the 94 students reported last Friday. There have also been 619 new cases among students at Northumbria University, compared with 770 last week. That means nearly 2,500 students and staff have tested positive since returning to studies.
More than 400 students and eight staff members at the University of Nottingham have tested positive for Covid-19. The university said the figures would be “higher than other universities” because it was running its own asymptomatic testing programme.
Almost 400 students and staff at Queen’s University in Belfast, Northern Ireland, are self-isolating after more than 160 people tested positive for Covid-19. A university spokesperson said the safety and wellbeing of staff and students was the university’s first priority.
One result of this is a lot of universities are moving back to online teaching.
This week, the University of Sheffield and Sheffield Hallam University both said they will move more learning online. The University of Sheffield said all teaching will move online from Friday until 18 October. Sheffield Hallam said it will increase the proportion of online teaching, but keep some on-campus.
Both universities (Newcastle and Northumbria) said they had extensive plans in place to support students. Earlier today they said they would move most of their teaching online in response to the outbreaks.
The two main universities in Manchester are teaching online until “at least” the end of the month after a coronavirus outbreak among students. Manchester Metropolitan University (MMU) and the University of Manchester (UM) said it was a “collaborative decision” with public health bosses and “won’t impact” on teaching quality. It comes after 1,700 students were told to self-isolate at MMU on 26 September.
I took the plunge and ordered full fibre from BT and if it goes to plan I will be getting 900Mb/s down and 110Mb/s up from the new all fibre connection. This will be much faster than my current 32Mb/s FTTC connection and so much faster than the ADSL connection I had between 2012 and 2017 which rarely went above 1Mb/s.
My top tweet this week was this one.
On this day and at this time eleven years ago I was buying a coffee from the this wonderful Piaggio van outside the Handheld Learning 2009 Conference. It was great coffee and it was a great conference. #coffeehttps://t.co/bz1jjC8ZVXpic.twitter.com/cr7tAVXjbq
Shorter week due to a Bank Holiday in England, the weather wasn’t up to much.
I wrote a piece about the reality of robots. The premise of the article was that:
When we mention robots we often think of the rabbit robots and Peppa robot that we have seen at events. As a result when we talk about robots and education, we think of robots standing at the front of a class teaching. However the impact that robotics will have on learning and teaching will come from the work being undertaken with the robots being used in manufacturing and logistics.
The draft of the article was based on conversations and some research I had done over the last few years. This was an attempt to draw those things together, as well as move the discussion about robots in education away from toy robots which are great for teaching robotics, but how robots could and may impact the future of learning and teaching.
I remember in one job when we bought a Peppa robot, in the support of teaching robotics. One of my learning technologists asked if the team could get one. We then had a (too) long discussion on why would be need a robot and how it would enhance learning and teaching in subjects other than robotics? The end consensus was more that it was cool. This was a real example of the tech getting in the way of the pedagogy.
It’s September, so schools and colleges are back this week, operating in a totally different way to what they were doing just six months ago.
At my children’s secondary school, the students will now remain in the same room throughout the day and it will be the teachers who move from room to room. Each child will have a designated desk which they will sit in each day for at least the first term, if not the rest of the academic year. It won’t be like this at colleges and universities, but restrictions will still need to be in place to mitigate the risk of infection.
There has been quite a bit of discussion online and in the press about people returning to the workplace. Sometimes the talk is of returning to work. Hello? Hello? Some of have never stopping working, we have been working from home! The main crunch of the issue appears to be the impact of people not commuting to the workplace and the impact this is having on the economy of the city centre and the businesses that are there.
Personally I think that if we can use this opportunity to move the work landscape from one where large portions of the population scramble to get to a single location via train or driving to one where people work locally (not necessarily from home) then this could have a really positive impact on local economies, as well as flattening the skewed markets that the commute to the office working culture can have on house prices, transport, pollution and so on.
Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI. I wrote a blog post about this article, my tweet and the responses to it.
There was a new publication from Jisc that may be of interest to those looking at digital learning, Digital learning rebooted.
This report highlights a range of responses from UK universities, ranging from trailblazing efforts at University of Northampton with its embedded ‘active blended learning’ approach, to innovation at Coventry University which is transforming each module in partnership with learning experience platform Aula. The University of Leeds, with its use of student buddies, and University of Lincoln’s long-standing co-creation work are notable for their supportive student-staff approaches. University of York, however, focused on simplicity in the short term and redesign longer-term. The University of the West of Scotland is also focusing on developing a community-based hybrid learning approach for the new year.
I am going teach, was a blog post I wrote about the nature of teaching in this new landscape.
The Office for Students are reviewing the challenges the sector faced during the Covid-19 pandemic and are calling for evidence.
This call for evidence is seeking a wide breadth of sector input and experience to understand the challenges faced, and lessons learned from remote teaching and learning delivery since the start of the coronavirus (COVID-19) pandemic in March 2020.
The OfS are looking to see what worked and what has not worked. What will work in the future and what about the student experience in all of this.
As many teachers and learners have discovered recently, Zoom fatigue, that that needs to be accounted for when designing curriculums. “You need to design an effective online curriculum or blended curriculum that takes advantage of the technology and opportunities it offers, but likewise doesn’t just bombard people with screentime that actually results in a negative impact on their wellbeing,” says Clay.
I also mentioned connectivity.
“As soon as you took away the kind of connectivity and resources you find on campus, it became a real challenge to be able to connect and stay connected,” says James Clay, head of higher education and student experience at Jisc.
During the coronavirus (COVID-19) lockdown, 52 per cent of students said their learning was impacted by slow or unreliable internet connection, with 8 per cent ‘severely’ affected.
The survey also found the lack of a quiet study space was also impacting on the student experience.
71 per cent reported lack of access to a quiet study space, with 22 per cent ‘severely’ impacted
Friday was full of meetings, which made for a busy day.
My top tweet this week was this one.
These students figured out their tests were graded by AI — and the easy way to cheat – The Verge https://t.co/caeLgVPQK0
The student in the article was undertaking an online test, and it wasn’t multiple choice, but short form answers to questions.
…he’d received his grade less than a second after submitting his answers. A teacher couldn’t have read his response in that time, Simmons knew — her son was being graded by an algorithm.
What the parent found was that by using a mix of keywords, or “word salad”, the system would mark the answer as correct. So the student could “cheat” the system!
The article itself was stemmed from a Twitter thread.
Algorithm update. He cracked it: Two full sentences, followed by a word salad of all possibly applicable keywords. 100% on every assignment. Students on @EdgenuityInc, there's your ticket. He went from an F to an A+ without learning a thing.
I posted the link to the article to the Twitter (as I often do with links) and it generated quite a response. Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI.
There was quite a bit of feedback that this wasn’t cheating, but actually providing an answer to the question that the AI would mark as correct.
I'd be tempted to see this as students learning how to successfully complete assignments. Good students learn what you're testing. If what you're testing is not what you really want them to learn, they won't learn what you want them to. This is a problem with human marking too. https://t.co/PZ6j7qB3YJ
I would agree, this was being called AI, but it was more of a system which matched keywords from answers given by students to a list provided by a teacher. The system wasn’t analysing what and how an answer was written, it was a text matching process.
A flawed approach to testing, which resulted in students been able to “game” the system to get 100%.
The lesson here is, for anyone looking at automated online assessment, is if there is a way in which the system can be manipulated, then it probably will be.
A shorter week as I was on leave for a couple of days.
Over the weekend I published the thinking and an expanded textual version of my presentation to the University of Hertfordshire, where I talked about the possibilities of technology, and the ethical, privacy and legal aspects of said technology.
Monday saw my end of year review meeting. These weeknotes have been useful in remembering what I have been doing and where. Blogging not just weeknotes, but also about events I have attended or presented at also helps in preparing for these kinds of things. Even if you don’t publish them as I do, maintaining some kind of record over the year helps with preparation for reviews.
I am working with colleagues on the Learning and Teaching Reimagined project. We are looking at undertaking various activities, as well as publishing some definitive guides for leaders in relevant areas.
One of the key aspects, which we are ensuring is recognising that though the technological challenges and issues do need to be addressed and resolved, one of the core issues is looking at the pedagogy in using technology to deliver learning and teaching, remotely and online. As demonstrated with my series on translation, it is often easier to translate existing physical face to face practice into online version, but this loses the nuances of that physical delivery, whilst ignoring the affordances that online and digital can provide.
As students sit their exams during the pandemic, universities have turned to digital proctoring services. They range from human monitoring via webcams to remote access software enabling the takeover of a student’s browser. Others use artificial intelligence (AI) to flag body language and background noise that might point to cheating.
In my work on assessment I did research and look at digital proctoring. Most universities realised that the technology, despite the protestations of the companies involved, was unfair and could negatively impact on wellbeing. There were also concerns about the validity of such proctoring. Universities have also recognised that not every student was in a space, have the connection or the right kind of device to enable them to participate in said remote exams.
XKCD have a wonderful perspective of some of the key issues of the day and this diagram looking at the risks of Covid-19 along with risks of non-Covid-19 activities did raise a smile in me.
I found this opinion article on the Guardian on facial recognition interesting and relevant to some of the work and research I have been doing on assessment, specifically remote assessment during the lockdown and plans for the future.
As students sit their exams during the pandemic, universities have turned to digital proctoring services. They range from human monitoring via webcams to remote access software enabling the takeover of a student’s browser. Others use artificial intelligence (AI) to flag body language and background noise that might point to cheating.
In my work on assessment I did research and look at digital proctoring. Most universities realised that the technology, despite the protestations of the companies involved, was unfair and could negatively impact on wellbeing. There were also concerns about the validity of such proctoring. Universities have also recognised that not every student was in a space, have the connection or the right kind of device to enable them to participate in said remote exams.
However, professional bodies, such as the Bar Standards Board in the article, have decided to use digital proctoring for their professional exams, and their chosen technology uses face-matching technology.
The Guardian article author, Meg Foulkes, rightly expresses her concerns about the biased nature of said technologies and is concerned that they are been used without sufficient safeguards in place, such as stricter regulation and ethical standards, for instance.
The article specifically mentions the concern of many over the bias that these technologies have.
Of most concern is the racialised bias that face-matching and facial recognition technologies exhibit.
This article reminds me of the discussion I had a few weeks back in my presentation to the University of Hertfordshire, where I talked about the possibilities of technology, but I said, first consider the ethical, privacy and legal aspects of said technology before blindly implementing it with students. This applies not just to universities, but also the professional bodies that they work and collaborate with.
I have been listening, writing and talking about how universities are planning for September. There is so much uncertainly about what the landscape will be like then, so working out what and how to design an effective student experience is challenging.
Courses will not be the same as they were and won’t be the same as they are now.
Universities are reflecting on their plans in light of the current lockdown, the easing of the lockdown, social distancing as well as guidance from the regulator.
Students applying for university places in England must be told with “absolute clarity” how courses will be taught – before they make choices for the autumn, says Nicola Dandridge of the Office for Students.
This has implications for future planning and announcements of what universities will be doing in the Autumn. They will probably need to start publishing in June their plans. Some have done this already.
In what I suspected was to be the start of a trend, the University of Manchester decided to keep lectures online for the autumn.
The University of Manchester has confirmed it will keep all of its lectures online for at least one semester when the next academic year starts. In an email to students Professor McMahon, vice-president for teaching, learning and students, confirmed the university’s undergraduate teaching year would begin in late September “with little change to our start dates”, but it would “provide our lectures and some other aspects of learning online”.
The whole student experience is not going online though as the article continues.
However, students would be asked to return physically to campus in the autumn as Manchester was “keen to continue with other face-to-face activities, such as small group teaching and tutorials, as safely and as early as we can”, added Professor McMahon.
“Given that it is likely that social distancing will continue to be required, the university has decided there will be no face-to-face lectures during the next academic year. Lectures will continue to be made available online and it may be possible to host smaller teaching groups in person, as long as this conforms to social distancing requirements.
There was a similar announcement from the University of Bolton.
The University will teach our excellent Undergraduate and Postgraduate programmes on campus from the start of the new academic year in September 2020 and also support your learning using a range of dynamic virtual learning tools.
Though very similar pronouncements, reading this Twitter thread:
The Cambridge story is everywhere this morning – but I feel that the University of Bolton announcement will end up being the important one. https://t.co/qaNIjH32yx
It's been assumed that a September start isn't possible – Bolton has made it possible with technical measures.
— @dkernohan@bluesky.social (@dkernohan) May 20, 2020
Most are thinking that Bolton and Cambridge are doing the same thing, but just spun it differently.
So how can universities plan their courses and curriculum in an uncertain future?
We see and hear plans for online courses, non-online courses, blended courses and other types of courses.
A phrase I had been using in my conversations and discussions is hybrid courses. This is less hybrid as in combining online and physical courses into a single course, that’s more a blended approach. My view was that hybrid was much more about analogous to how hybrid vehicles function.
There is a petrol engine in the hybrid car, but the car can run on electric power when needed. On longer journeys the petrol engine takes over, but on shorter (slower) trips the car uses electric power. Which power is used is dependent on the environment and situation the car is in.
With a hybrid course, some sessions are physical face to face sessions. There are live online sessions and there are asynchronous online sessions. In addition there could be asynchronous offline sessions as well. You may not want to be online all the time!
Some sessions could be easily switched from one format to another. So if there is a change in lockdown restrictions (tightening or easing) then sessions can move to or from online or a physical location.
These hybrid responsive courses will allow universities to easily clarify with prospective students about their experience and how they potentially could change as restrictions are either lifted or enforced. It helps staff plan their teaching and assessments to take into account the environment and changes to the situation.
There are hybrid variations across cars, some can be topped up by plugging in, whilst others just rely on charging form the petrol engine.
There could be a similar story with variations on hybrid courses. Some could have more online elements, whilst others reflecting the nature of that subject could have more physical face to face aspects.
There are of course still petrol cars and fully electric cars, but there is a whole spectrum of hybrid vehicles and it’s the same with hybrid courses.
You could translate your courses into online versions. You could transform them into courses which take advantage of the affordances of online. However the delivery of teaching is just one aspect of the overall student experience and thinking about that and reflecting on how your course and learning design will take into account the realities of an uncertain future, means you need to build that into the design of modules and courses. A hybrid model that is responsive and can adapt is one way which this could be done.
So it was interesting to see another person, Simon Thomson from University of Liverpool Centre for Innovation in Education (CIE) has been using it as well.
We’re using term hybrid. In the same that hybrid car has a mechanical engine and electric motor & you use each depending on journey you are taking you’ll potentially have an on campus or online experience which will reflect your learning journey (but maximise benefits of both)
“None of us know what’s going to be happening in the Autumn”, said OfS CEO Nicola Dandridge to the Commons Education Committee, who nevertheless added – in the same breath – that “we are requiring that universities are as clear as they can be to students so that students when they accept an offer from a university know in broad terms what they’ll be getting”. Via WonkHE
It’s an uncertain future and one that means courses will need to reflect that uncertainty. Designing hybrid courses which reflect the possibilities of that future, but are responsive enough to respond to changes are probably one way of ensuring that the student experience is meeting the demands of students in a challenging landscape.
news and views on e-learning, TEL and learning stuff in general…
