Category Archives: assessment

I haz fibre soon! – Weeknote #84 – 9th October 2020

I had a fair few meetings this week on a range of topics, including learning and teaching, the Data Matters conference, consultancy, pipelines, and public affairs.

This story from The Register was not really surprising, Unis turn to webcam-watching AI to invigilate students taking exams. Of course, it struggles with people of color.

AI software designed to monitor students via webcam as they take their tests – to detect any attempts at cheating – sometimes fails to identify the students due to their skin color.

I am not surprised, in my work on the Intelligent Campus, when we did some research into facial recognition, there was quite a bit of coverage about how it only really worked with white males. Can we be surprised then when used for exam invigilation that it fails on the same issue?

In a similar story, UK passport photo checker shows bias against dark-skinned women.

Women with darker skin are more than twice as likely to be told their photos fail UK passport rules when they submit them online than lighter-skinned men, according to a BBC investigation. One black student said she was wrongly told her mouth looked open each time she uploaded five different photos to the government website.

There is a question here about removing the systemic bias we find in AI and algorthims being used in education (as well as the wider society). A deeper question is how does that bias get there in the first place?

Across the week we saw more universities report large covid-19 infections in their student populations.

Sheffield Hallam has seen over 370 cases of Covid since the beginning of term and the University of Sheffield has seen 589 cases. The local area has also seen a dramatic increase in the number of people testing positive.

 Another 1,600 students have tested positive for coronavirus at Newcastle’s two universities. Newcastle University says 1,003 students and 12 members of staff have tested positive for Covid-19 in the past week. That’s up from the 94 students reported last Friday. There have also been 619 new cases among students at Northumbria University, compared with 770 last week. That means nearly 2,500 students and staff have tested positive since returning to studies.

More than 400 students and eight staff members at the University of Nottingham have tested positive for Covid-19. The university said the figures would be “higher than other universities” because it was running its own asymptomatic testing programme.

Almost 400 students and staff at Queen’s University in Belfast, Northern Ireland, are self-isolating after more than 160 people tested positive for Covid-19. A university spokesperson said the safety and wellbeing of staff and students was the university’s first priority.

One result of this is a lot of universities are moving back to online teaching.

This week, the University of Sheffield and Sheffield Hallam University both said they will move more learning online. The University of Sheffield said all teaching will move online from Friday until 18 October. Sheffield Hallam said it will increase the proportion of online teaching, but keep some on-campus.

Both universities (Newcastle and Northumbria) said they had extensive plans in place to support students. Earlier today they said they would move most of their teaching online in response to the outbreaks.

The two main universities in Manchester are teaching online until “at least” the end of the month after a coronavirus outbreak among students. Manchester Metropolitan University (MMU) and the University of Manchester (UM) said it was a “collaborative decision” with public health bosses and “won’t impact” on teaching quality. It comes after 1,700 students were told to self-isolate at MMU on 26 September.

fibre
Image by Chaitawat Pawapoowadon from Pixabay

I took the plunge and ordered full fibre from BT and if it goes to plan I will be getting 900Mb/s down and 110Mb/s up from the new all fibre connection. This will be much faster than my current 32Mb/s FTTC connection and so much faster than the ADSL connection I had between 2012 and 2017 which rarely went above 1Mb/s.

My top tweet this week was this one.

I am working, I may be at home, but I really am working – Weeknote #79 – 4th September 2020

Shorter week due to a Bank Holiday in England, the weather wasn’t up to much.

I wrote a piece about the reality of robots. The premise of the article was that:

When we mention robots we often think of the rabbit robots and Peppa robot that we have seen at events. As a result when we talk about robots and education, we think of robots standing at the front of a class teaching. However the impact that robotics will have on learning and teaching will come from the work being undertaken with the robots being used in manufacturing and logistics.

The draft of the article was based on conversations and some research I had done over the last few years. This was an attempt to draw those things together, as well as move the discussion about robots in education away from toy robots which are great for teaching robotics, but how robots could and may impact the future of learning and teaching.

I remember in one job when we bought a Peppa robot, in the support of teaching robotics. One of my learning technologists asked if the team could get one. We then had a (too) long discussion on why would be need a robot and how it would enhance learning and teaching in subjects other than robotics? The end consensus was more that it was cool. This was a real example of the tech getting in the way of the pedagogy.

Peppa

It’s September, so schools and colleges are back this week, operating in a totally different way to what they were doing just six months ago.

At my children’s secondary school, the students will now remain in the same room throughout the day and it will be the teachers who move from room to room. Each child will have a designated desk which they will sit in each day for at least the first term, if not the rest of the academic year. It won’t be like this at colleges and universities, but restrictions will still need to be in place to mitigate the risk of infection.

There has been quite a bit of discussion online and in the press about people returning to the workplace. Sometimes the talk is of returning to work. Hello? Hello? Some of have never stopping working, we have been working from home! The main crunch of the issue appears to be the impact of people not commuting to the workplace and the impact this is having on the economy of the city centre and the businesses that are there.

Personally I think that if we can use this opportunity to move the work landscape from one where large portions of the population scramble to get to a single location via train or driving to one where people work locally (not necessarily from home) then this could have a really positive impact on local economies, as well as flattening the skewed markets that the commute to the office working culture can have on house prices, transport, pollution and so on.

I wrote more thoughts on this on my tech and productivity blog.

video chat
Photo by Dylan Ferreira on Unsplash

I read an article on The Verge this week which sparked my interest.

These students figured out their tests were graded by AI — and the easy way to cheat

I posted the link to the article to the Twitter (as I often do with links) and it generated quite a response.

Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI. I wrote a blog post about this article, my tweet and the responses to it.

There was a new publication from Jisc that may be of interest to those looking at digital learning, Digital learning rebooted.

This report highlights a range of responses from UK universities, ranging from trailblazing efforts at University of Northampton with its embedded ‘active blended learning’ approach, to innovation at Coventry University which is transforming each module in partnership with learning experience platform Aula. The University of Leeds, with its use of student buddies, and University of Lincoln’s long-standing co-creation work are notable for their supportive student-staff approaches. University of York, however, focused on simplicity in the short term and redesign longer-term. The University of the West of Scotland is also focusing on developing a community-based hybrid learning approach for the new year.

I am going teach, was a blog post I wrote about the nature of teaching in this new landscape.

The Office for Students are reviewing the challenges the sector faced during the Covid-19 pandemic and are calling for evidence.

This call for evidence is seeking a wide breadth of sector input and experience to understand the challenges faced, and lessons learned from remote teaching and learning delivery since the start of the coronavirus (COVID-19) pandemic in March 2020.

The OfS are looking to see what worked and what has not worked. What will work in the future and what about the student experience in all of this.

I was quoted a few times in this article, How digital transformation in education will help all children.

As many teachers and learners have discovered recently, Zoom fatigue, that that needs to be accounted for when designing curriculums. “You need to design an effective online curriculum or blended curriculum that takes advantage of the technology and opportunities it offers, but likewise doesn’t just bombard people with screentime that actually results in a negative impact on their wellbeing,” says Clay.

I also mentioned connectivity.

“As soon as you took away the kind of connectivity and resources you find on campus, it became a real challenge to be able to connect and stay connected,” says James Clay, head of higher education and student experience at Jisc.

This was something that was echoed in a recent survey on digital poverty from the OfS.

During the coronavirus (COVID-19) lockdown, 52 per cent of students said their learning was impacted by slow or unreliable internet connection, with 8 per cent ‘severely’ affected.

The survey also found the lack of a quiet study space was also impacting on the student experience.

71 per cent reported lack of access to a quiet study space, with 22 per cent ‘severely’ impacted

Friday was full of meetings, which made for a busy day.

My top tweet this week was this one.

It wasn’t cheating!

using a laptop
Image by Free-Photos from Pixabay

I read an article on The Verge this morning which sparked my interest.

These students figured out their tests were graded by AI — and the easy way to cheat

The student in the article was undertaking an online test, and it wasn’t multiple choice, but short form answers to questions.

…he’d received his grade less than a second after submitting his answers. A teacher couldn’t have read his response in that time, Simmons knew — her son was being graded by an algorithm.

What the parent found was that by using a mix of keywords, or “word salad”, the system would mark the answer as correct. So the student could “cheat” the system!

The article itself was stemmed from a Twitter thread.

I posted the link to the article to the Twitter (as I often do with links) and it generated quite a response. Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI.

There was quite a bit of feedback that this wasn’t cheating, but actually providing an answer to the question that the AI would mark as correct.

Others felt that this wasn’t AI.

I would agree, this was being called AI, but it was more of a system which matched keywords from answers given by students to a list provided by a teacher. The system wasn’t analysing what and how an answer was written, it was a text matching process.

A flawed approach to testing, which resulted in students been able to “game” the system to get 100%.

The lesson here is, for anyone looking at automated online assessment, is if there is a way in which the system can be manipulated, then it probably will be.

Shorter – Weeknote #73 – 24th July 2020

A shorter week as I was on leave for a couple of days.

Over the weekend I published the thinking and an expanded textual version of my  presentation to the University of Hertfordshire, where I talked about the possibilities of technology, and the ethical, privacy and legal aspects of said technology.

Monday saw my end of year review meeting. These weeknotes have been useful in remembering what I have been doing and where. Blogging not just weeknotes, but also about events I have attended or presented at also helps in preparing for these kinds of things. Even if you don’t publish them as I do, maintaining some kind of record over the year helps with preparation for reviews.

tree trunk
Image by Picography from Pixabay

I am working with colleagues on the Learning and Teaching Reimagined project. We are looking at undertaking various activities, as well as publishing some definitive guides for leaders in relevant areas.

One of the key aspects, which we are ensuring is recognising that though the technological challenges and issues do need to be addressed and resolved, one of the core issues is looking at the pedagogy in using technology to deliver learning and teaching, remotely and online. As demonstrated with my series on translation, it is often easier to translate existing physical face to face practice into online version, but this loses the nuances of that physical delivery, whilst ignoring the affordances that online and digital can provide.

exam
Image by F1 Digitals from Pixabay

I found this opinion article on the Guardian on facial recognition interesting and relevant.

As students sit their exams during the pandemic, universities have turned to digital proctoring services. They range from human monitoring via webcams to remote access software enabling the takeover of a student’s browser. Others use artificial intelligence (AI) to flag body language and background noise that might point to cheating.

In my work on assessment I did research and look at digital proctoring. Most universities realised that the technology, despite the protestations of the companies involved, was unfair and could negatively impact on wellbeing. There were also concerns about the validity of such proctoring. Universities have also recognised that not every student was in a space, have the connection or the right kind of device to enable them to participate in said remote exams.

I wrote up my thoughts in this blog post.

Someone shared this excellent XKCD comic on the Twitter.

XKCD have a wonderful perspective of some of the key issues of the day and this diagram looking at the risks of Covid-19 along with risks of non-Covid-19 activities did raise a smile in me.

My top tweet this week was this one.

Consider the ethical issues first!

exam
Image by F1 Digitals from Pixabay

I found this opinion article on the Guardian on facial recognition interesting and relevant to some of the work and research I have been doing on assessment, specifically remote assessment during the lockdown and plans for the future.

As students sit their exams during the pandemic, universities have turned to digital proctoring services. They range from human monitoring via webcams to remote access software enabling the takeover of a student’s browser. Others use artificial intelligence (AI) to flag body language and background noise that might point to cheating.

In my work on assessment I did research and look at digital proctoring. Most universities realised that the technology, despite the protestations of the companies involved, was unfair and could negatively impact on wellbeing. There were also concerns about the validity of such proctoring. Universities have also recognised that not every student was in a space, have the connection or the right kind of device to enable them to participate in said remote exams.

However, professional bodies, such as the Bar Standards Board in the article, have decided to use digital proctoring for their professional exams, and their chosen technology uses face-matching technology.

The Guardian article author, Meg Foulkes, rightly expresses her concerns about the biased nature of said technologies and is concerned that they are been used without sufficient safeguards in place, such as stricter regulation and ethical standards, for instance.

The article specifically mentions the concern of many over the bias that these technologies have.

Of most concern is the racialised bias that face-matching and facial recognition technologies exhibit.

This article reminds me of the discussion I had a few weeks back in my presentation to the University of Hertfordshire, where I talked about the possibilities of technology, but I said, first consider the ethical, privacy and legal aspects of said technology before blindly implementing it with students. This applies not just to universities, but also the professional bodies that they work and collaborate with.

Hybrid

Chimera
Image by Dean Moriarty from Pixabay

I have been listening, writing and talking about how universities are planning for September. There is so much uncertainly about what the landscape will be like then, so working out what and how to design an effective student experience is challenging.

Courses will not be the same as they were and won’t be the same as they are now.

Universities are reflecting on their plans in light of the current lockdown, the easing of the lockdown, social distancing as well as guidance from the regulator.

Students applying for university places in England must be told with “absolute clarity” how courses will be taught – before they make choices for the autumn, says Nicola Dandridge of the Office for Students. 

This has implications for future planning and announcements of what universities will be doing in the Autumn. They will probably need to start publishing in June their plans. Some have done this already.

In what I suspected was to be the start of a trend, the University of Manchester decided to keep lectures online for the autumn.

The University of Manchester has confirmed it will keep all of its lectures online for at least one semester when the next academic year starts. In an email to students Professor McMahon, vice-president for teaching, learning and students, confirmed the university’s undergraduate teaching year would begin in late September “with little change to our start dates”, but it would “provide our lectures and some other aspects of learning online”.

The whole student experience is not going online though as the article continues.

However, students would be asked to return physically to campus in the autumn as Manchester was “keen to continue with other face-to-face activities, such as small group teaching and tutorials, as safely and as early as we can”, added Professor McMahon.

The following week, the Student University Paper at Cambridge and then many others reported, such as the BBC – All lectures to be online-only until summer of 2021.

“Given that it is likely that social distancing will continue to be required, the university has decided there will be no face-to-face lectures during the next academic year. Lectures will continue to be made available online and it may be possible to host smaller teaching groups in person, as long as this conforms to social distancing requirements. 

There was a similar announcement from the University of Bolton.

The University will teach our excellent Undergraduate and Postgraduate programmes on campus from the start of the new academic year in September 2020 and also support your learning using a range of dynamic virtual learning tools.

Though very similar pronouncements, reading this Twitter thread:

Most are thinking that Bolton and Cambridge are doing the same thing, but just spun it differently.

So how can universities plan their courses and curriculum in an uncertain future? 

We see and hear plans for online courses, non-online courses, blended courses and other types of courses.

A phrase I had been using in my conversations and discussions is hybrid courses. This is less hybrid as in combining online and physical courses into a single course, that’s more a blended approach. My view was that hybrid was much more about analogous to how hybrid vehicles function.

hybrid engine
Image by Davgood Kirshot from Pixabay

There is a petrol engine in the hybrid car, but the car can run on electric power when needed. On longer journeys the petrol engine takes over, but on shorter (slower) trips the car uses electric power. Which power is used is dependent on the environment and situation the car is in.

With a hybrid course, some sessions are physical face to face sessions. There are live online sessions and there are asynchronous online sessions. In addition there could be asynchronous offline sessions as well. You may not want to be online all the time!

Some sessions could be easily switched from one format to another. So if there is a change in lockdown restrictions (tightening or easing) then sessions can move to or from online or a physical location.

These hybrid responsive courses will allow universities to easily clarify with prospective students about their experience and how they potentially could change as restrictions are either lifted or enforced. It helps staff plan their teaching and assessments to take into account the environment and changes to the situation.

There are hybrid variations across cars, some can be topped up by plugging in, whilst others just rely on charging form the petrol engine.

There could be a similar story with variations on hybrid courses. Some could have more online elements, whilst others reflecting the nature of that subject could have more physical face to face aspects.

There are of course still petrol cars and fully electric cars, but there is a whole spectrum of hybrid vehicles and it’s the same with hybrid courses.

You could translate your courses into online versions. You could transform them into courses which take advantage of the affordances of online. However the delivery of teaching is just one aspect of the overall student experience and thinking about that and reflecting on how your course and learning design will take into account the realities of an uncertain future, means you need to build that into the design of modules and courses. A hybrid model that is responsive and can adapt is one way which this could be done.

So it was interesting to see another person, Simon Thomson from University of Liverpool Centre for Innovation in Education (CIE) has been using it as well.

“None of us know what’s going to be happening in the Autumn”, said OfS CEO Nicola Dandridge to the Commons Education Committee, who nevertheless added – in the same breath – that “we are requiring that universities are as clear as they can be to students so that students when they accept an offer from a university know in broad terms what they’ll be getting”. Via WonkHE

It’s an uncertain future and one that means courses will need to reflect that uncertainty. Designing hybrid courses which reflect the possibilities of that future, but are responsive enough to respond to changes are probably one way of ensuring that the student experience is meeting the demands of students in a challenging landscape.

Things are going to be different – Weeknote #60 – 24th April 2020

Having moved down into assessment over the last few weeks, I am now looking at teaching online and student wellbeing (and engagement).

We know that the move to teaching online was very much done quickly and rapidly, with little time for planning. Platforms needed to be scaled up to widespread use and most academics moved to translate their existing practice into remote delivery. This wasn’t online teaching, this was teaching delivered remotely during a time of crisis.

The Easter break gave a bit of breathing room, but even then there wasn’t much time for planning and preparation, so even now much of the teaching will be a response to the lockdown rather than  a well thought out planned online course.

Thinking further ahead though, with the potential restrictions continuing, institutions will need to plan a responsive curriculum model that takes into account possible lockdown, restrictions, as well as some kind of normality.

I was involved in a meeting discussing the content needs of Further Education, though my role is Higher Education, I am working on some responses to Covid-19 and content for teachers is one of those areas. What content do teachers need? Do they in fact need good online content? Who will provide that content? How will do the quality assurance? Do we even need quality assurance? And where does this content live? Continue reading Things are going to be different – Weeknote #60 – 24th April 2020

Challenging – Weeknote #59 – 17th April 2020

A shorter week this week, as we had Easter Monday, of course with the lockdown I didn’t go anywhere on the long weekend and I am not going off to the office this week… this could result in an easy blurring of home and work. Though I generally am less strict during the working week, I kept the entire weekend free of work to have a decent break away. Though reading the news some relevant stuff does bubble to the surface such as this report from The Guardian on the potential drop in income universities will suffer as foreign students drop out.

Some universities are already expecting to lose more than £100m as foreign students cancel their studies, with warnings that the impact of coronavirus will be “like a tsunami hitting the sector”.

Several organisations are now planning for a 80-100% reduction in their foreign student numbers this year, with prestigious names said to be among those most affected. The sector is already making a plea to the government for a cash injection amounting to billions of pounds to help it through the crisis, as it is hit by a drop in international student numbers, accommodation deals and conference income.

There are also concerns that many domestic students will defer their entry for a year, if the first term in September will be online as it is now. I am wondering how many prospective students will defer their entry for a year as they wouldn’t want to miss the physical aspect of attending university. Similarly will there be first and second year undergraduates who will take a gap year.

We are starting to see the potential impact of this, as some universities are looking at delaying the start of term, whilst others are planning for online delivery.

Durham University is proposing online-only degrees as part of a radical restructuring process, the Palatinate reveals.

Confidential documents seen by Palatinate show that the University is planning “a radical restructure” of the Durham curriculum in order to permanently put online resources at the core of its educational offer, in response to the Covid-19 crisis and other ongoing changes in both national and international Higher Education.

The proposals seek to “invert Durham’s traditional educational model”, which revolves around residential study, replacing it with one that puts “online resources at the core enabling us to provide education at a distance.”

We don’t know for sure when the lockdown will be lifted, or what measures will stay in place, or even if they are lifted, whether they may come back down again, if there is a second or even a third wave of coronavirus infections. So it makes sense for universities to plan for online and remote delivery, something which is not the emergency models they have had to put in place right now.

It looks like the TEF has been indefinitely postponed.

I spent a lot of time this week discussing assessment and the challenges moving to online assessment. Of course assessment is an integral part of teaching and learning, so things you do for assessment need to be incorporated into the teaching and learning.

There are many challenges that the higher education sector faces with regard to assessment, but through my research, interviews and a landscape study, these challenges emerged as key to the sector.

Student engagement – Maintain student engagement through the next few weeks and through the assessment process, as they continue to socially isolate and study remotely.

Translating and transforming to online assessment at scale and at pace – Transform multiple modes of assessment to online versions at scale and at pace. Many universities have experience of designing and delivering online assessment, however they will not have done this at scale or transformed at the pace required.

Ensuring academic standards and quality when moving to online assessment – Maintaining the academic standard and quality as required by internal and external regulations, as they translate and convert existing practice into online modes.

Online assessment dashboards and benchmarking. – What are other universities doing with online assessment? What best practice is out there? Which universities are doing it well? How do universities compare? What do universities know about themselves?

My top tweet this week was this one.

Did I predict the virus? – Weeknote #58 – 10th April 2020

This week I have spent a lot of time looking at assessment, but also reflecting on the Plymouth e-learning conference where ten years ago I chaired a debate about closing the physical campus in times of crisis and disruption.

It could be floods, high winds (remember 1987), flu or similar viral infections, transport strikes, fuel crisis, anything…

Bunker

I was supposed to be on leave this week, we were heading off to London for a few days, as we had tickets for the Only Fools and Horses musical at the Royal Haymarket. I had bought tickets for my wife as a Christmas present and it was something we were all looking forward to. Then all this lockdown happened and the theatre cancelled all the performances as required by the Government.

I did consider keeping my leave, but with leading a taskforce, it was apparent that I might not have the time to take some (and where would I go).

I saw on Monday that the OfS on Friday had published new guidance on academic standards and quality.

There is recognition that in these challenging times that maintaining quality will be difficult. Continue reading Did I predict the virus? – Weeknote #58 – 10th April 2020

Asssessing – Weeknote #50 – 14th February 2020

Monday I was off to Bristol, for a late afternoon meeting. It was nice to be back in the office and see the changes and improvements since I was last there a week or so back. It is a nice place to work.

Monday saw the publication of Jisc’s report on assessment.

This report is the result of an experts meeting exploring assessment in universities and colleges and how technology could be used to help address some of the problems and opportunities.

This report was widely reported in the press across the UK.

Assessment is a challenge for many institutions, often resulting in attempts to fix it, but sometimes I think we need to dig deeper and re-imagine assessment as a whole.

Having discussed the coronavirus in last week’s weeknote, the situation has been escalated and the Department of Health has described the coronavirus as a “serious and imminent threat” to public health.

It comes as the government announced new powers to keep people in quarantine to stop the spread of the virus.

In order to do this the Department of Health has described the coronavirus as a “serious and imminent threat” to public health.

The overall risk level to the UK remains “moderate”.

Wednesday I was at the 18th Jisc Learning Analytics Community Event at Newman University in Birmingham. There were various talks and discussions and overall it was an interesting day.

I published a blog post about the ALT Learning Spaces SIG that happened last month.

Could we build a treehouse?

The post was liked by people.

Thursday I was in our Bristol office working on a document with colleagues. I had quite a few conversations about the Education 4.0 roadmap I am working on and how the sector needs to start thinking and preparing for both the challenges, but also the opportunities that there is with this potential view of the future.

Friday I was on leave for my son’s graduation.

My top tweet this week was this one.