Category Archives: assessment

Once more to London – Weeknote #128 – 13th August 2021

Well after a week off work, it was back to work. As with a couple of weeks ago I spent the best part of the week working in our London office. It was also a shorter week as I was on leave on the Friday. London was not very busy, but I was expecting that following my previous time in London. The office was even less busy, I was the only person in the office. It was obvious that many staff were still working from home. I don’t mind working from home. There were a few articles about the shift back to office working. Whitehall was looking to remove the London weighting for staff according to this article in the Guardian.

Whitehall officials have held high-level talks about taking away a salary boost awarded to London-based civil servants amid efforts to encourage workers back to the office.

Whilst on the BBC website an article asked: Should I be working from home or going back to the office?

People in England are no longer being asked to work from home. Instead the Prime Minister Boris Johnson is recommending a “gradual return to work”. However, in the rest of the UK, people are still being advised to keep working at home where possible.

I had quite a bit of flexibility on where I worked before the pandemic, so there is less pressure to return to the office. I have been working in the office though, I like the change in scenery and routine. After eighteen months being forced to work from home, I like the option of choice. Also during the summer holidays it makes more sense for me, when working, to be away from home.

First job was to clear that inbox full of e-mail, which to be honest didn’t take too long.

We had a team wide call on Monday, which was interesting. There will be changes in the team from September (a new manager) and we have a new CEO from mid-September.

We had an interesting meeting about the evaluation of Connect More, which people seemed to enjoy and got a lot out of.

Wednesday I had an interesting review and discussion about assessment, and what Jisc can do to support the sector to transform assessment. Despite the opportunities of digital in regard to assessment, many of the issues relating to the digital transformation of assessment, are much more about the transformation of assessment, with digital just being a catalyst. What is the purpose of assessment for example.

On Thursday I had a catchup with our (newish) HR contact. In my role I have no line management responsibilities (though plenty of matrix management responsibilities). We had a great discussion and chat about how we can make that matrix management more effective and efficient going forwards. So, we can move people from one area of Jisc to another to work on projects more easily. Something that a company like Apple do quite often.

I had a useful and interesting meeting with a university talking about our Powering HE document and the possible opportunities and challenges that universities will face over the next few years.

My top tweet this week was this one.

Unknown Unknowns and Unknown Knowns – Weeknote #117 – 28th May 2021

Monday I was focusing on one of the projects we are working on with an university looking at various scope areas and how technology and digital can make a difference. I was reminded of the NSA quote of cylinders of excellence when it comes to silo working. The concept of excellent departments, but not an excellent university came to mind, but also about the inefficiencies of silos working in isolation and not thinking about the impact of their development and change on the rest of the university.

At the end of the day we were discussing assessment. What is happening with assessment in higher education now and what changes made as a result of Covid-19 are now in place, but also the wider issues of assessment as well.

old television
Image by Free-Photos from Pixabay

Cancelled my subscription to Britbox, in the main despite there being stuff I would like to watch, I wasn’t finding the time to watch it.

Tuesday saw me back to our office in Portwall Lane for an in-person meeting with my line manager, our first meeting in-person since August last year. It was actually nice to be both in the office and in an in-person meeting.

Something that keeps coming to my attention is the future of teaching, especially the concept of dual mode or hybrid teaching. What are peoples’ experiences of “dual-mode”, “muti-mode”, hybrid teaching? What has the student feedback being like? Something I have been reflecting on this week.

Students prioritise a return to face to face teaching from September 2021, so says the findings from a report published by the UPP Foundation.

Students want universities to prioritise a return to in person teaching and are missing face-to-face interaction around their wider student experience.

This is something which isn’t too surprising and is also something that has come out of our recent research into the student experience. Though digging deeper for us, it was more the in-person interaction students were missing and less the teaching.

Wednesday afternoon myself and Isabel Lucas of HEDG and the University of Cumbria hosted a share shop, facilitated by Advance HE, on how universities can support students transitioning in HE. We looked at both new students and returning students.

In the session, How can we best support learners coming in to HE from a Covid context? We discussed the transition arrangements others have put into place for the next academic year.

In September, third year students returning to HE will not have had a normal year in higher education and it is likely that their third year will not be like it was before.

We discussed a range of issues, focusing on the known knowns and the known unknowns. More difficult to discuss the unknown knowns and the unknown unknowns!

We are aiming to share the findings from the shareshop in June.

Thursday was a light day in terms of meetings, but got even lighter, as one meeting was cancelled five minutes before it was due to start, with the other meeting, two people who had accepted were in fact on leave, so in the end the meeting lasted only five minutes.

The future of the office keeps getting discussed, with those who own offices explaining why going back to the office is so important and those who don’t explaining why it isn’t. For me a lot is about the kind of work you do, I don’t do the same thing everyday, so there isn’t a single kind of space I need all the time. Before Covid, sometimes I would be working alone, sometimes I would be in meetings, sometimes we would be collaborating and sometimes I didn’t know, so it was useful to have other people around to bounce ideas off and chat over coffee.

My top tweet this week was this one.

Powering up – Weeknote #108 – 26th March 2021

I realised that I have been walking and exercising less during the last few weeks, now the children are back in school, so this week I made a determined effort to increase the amount of walking I do.

Like last week, I have spent a lot of the week interviewing staff and students as part of a project we’re doing at Jisc. We have been talking to them about their thoughts and perspectives on digital learning. As with a lot of these kinds of interviews there are some interesting individual insights, however the real insight comes from analysing all the interviews and seeing what trends are in there. I also spent time planning a similar, but different project.

I attended a roundtable on a digital vision for Scotland and facilitated a breakout room reflecting on the vision.

If you have watched a 60 minute TV programme, you will realise few if any have a talking head for 60 minutes. Few of us have the time or the skills to create a 60 minute documentary style programme to replace the lecture, and where would you go to film it? So if you change the monologue to a conversation then you can create something which is more engaging for the viewer (the student) and hopefully a better learning experience.

In a meeting this week with staff from a university I was discussing this issue and their response was, what about comedy stand-up? That’s a monologue. That got me thinking and reflecting, so I wrote a blog post about needing a tray.

Lego Star Wars
Image by 501stCommanderMax from Pixabay

Continue reading Powering up – Weeknote #108 – 26th March 2021

Challenges for quality assurance in the wake of the pandemic

I was recently part of a panel session looking at Challenges for quality assurance in the wake of the pandemic – ensuring quality in digital learning, academic integrity in remote assessment, and emerging best practice

I gave a five minute presentation from my personal perspective.

We knew what we were doing.

We knew how to assure quality and academic integrity.

We thought everything was going to be fine….

There were early signs of the impact of Covid-19 back in December 2019 and January 2020.

We thought everything was going to be fine….

A year ago, everything changed.

We went into a national lockdown and all universities needed to start doing things remotely, at pace and at scale.

We quickly translated our practices to online versions.

Our lectures became Zoom calls.

Our seminars became Teams conversations.

Office hours became 24/7 e-mail.

However we merely translated, we didn’t transform, we didn’t think differently.

As for exams and assessment….

Well that didn’t go as planned and was challenging to move online.

Moving assessment online is a journey and one for which we had no map and no real idea of where we wanted to get to.

We thought we could do it?

We thought technology was the solution.

It wasn’t.

The thing is translating practice online doesn’t really work. You lose the nuances of what made the in-person experience so great and you don’t take the advantage of the affordances of what digital can bring.

Can we be surprised when the quality of the experience suffers

Moving exams online doesn’t work.

Moving assessment online is not only possible, but can be done in a way which maintains quality and integrity.

How can you do this when the requirements in place from university exam board, PSRBs and professional accreditation are based on an model which expects a physical in-person assessment.

You need to transform and reimagine assessment.

You can not transform assessment on your own.

However this takes time.

It takes a certain level of digital capability.

It requires an insight into what we mean by assessment and what we are trying to assess.

At Jisc we are here to help the higher education sector through the use of digital to reimagine teaching, learning and assessment and to reframe the student experience.

A shared understanding – Weeknote #105 – 5th March 2021

cogs
Image by Pavlofox from Pixabay

I started as Head of Higher Education at Jisc on the 1st March 2019. So I have done two years (and a bit), had three line managers, a changing role and, oh yes, a global pandemic.

On Monday I had an excellent conversation with Isabel Lucas from Cumbria about the HEDG meeting I was presenting at, at the end of the week.

I have been sharing internally (and externally) the draft of the Jisc HE Strategy.

Image by Free-Photos from Pixabay

Another post on language, this time from Wonkhe: Why what we mean by “online learning” matters.

As higher education institutions plan for what will happen as we move slowly towards more students being on campus, there is continuing chatter about the form that teaching and learning will take. This includes how best to deliver it and how to communicate what this might look like. In all of this discussion, there has been a proliferation of words like “remote learning”, “digital learning”, and “hybrid learning” – and these terms have largely been taken for granted in respect to their pedagogical nuance. But if the preferred solution to the problems created by the pandemic in the first semester was “blended learning”, as we tumble through a second semester it would appear that the HE sector is beginning to settle on its next term of preference – “online learning”.

We do seem to spend a lot of time discussing what we should call what we do. The article makes the point that this does matter. I disagree slightly what matters is not what it is called, but whatever it is called, we have an agreed and shared understanding of what is means for you, for me and the students. We change the term, but this doesn’t necessarily mean that we change our understanding. I recall having this discussion about the use of the term hybridthat I used in an article to mean responsive and agile, whilst someone else was using the term to describe a mixed approach. Words are important, but shared understanding actually allows us to move forward.

Image by Pexels from Pixabay

Wednesday I joined a panel at the Westminster Education Forum to deliver a session on the future use of technology in assessment.

“The future for England’s exam system – building on best practice from the 2020 series, the role of technology and ensuring qualifications equip young people with the skills to succeed post-18”

I only had five minutes, so not a huge amount of time to reflect on the challenges and possibilities. To think a year ago I would have had to travel to London by train, find the venue and then join the panel in-person. Today, I just switched on the webcam and there I was, did my presentation and then answered a few questions. I didn’t use slides, as there wasn’t always a need to use slides in these kinds of panel sessions and at an in-person event I wouldn’t have used slides.

Of course at a physical in-person event they would have provided lunch, which would then give delegates an opportunity to come and chat about what I had been talking about. That didn’t happen this time. I would say that though using Twitter as a digital back channel at physical in-person events does sometimes work, but people have to be using the Twitter. At edtech events I find a fair few people are , at other kinds of events, not so much.

campus
Image by 小亭 江 from Pixabay

I liked this post from the Independent: I’m tired of hearing that universities are closed – it simply isn’t true

Lecturers are doing all they can during the pandemic to support the myriad different ways in which students learn

It’s not as though the physical campuses are closed, they are open for those courses which require a practical element.

Then again schools are not closed, they are open for the children of key workers, as well as vulnerable children, and staff are working with them and delivering remote teaching to the children at home.

Yes the experience is variable across the country, even across a school, but to keep saying they are closed, doesn’t really tell the whole story.

Lens
Image by Rudy and Peter Skitterians from Pixabay

On Thursday I had a really good discussion with a university about digital strategy. How important it is aligned to the main university strategy, but also how it enables that strategy and other strategies as well. If you are in charge of a strategy, how does it enable others, and how do others enable yours?

At the end of the week I was involved in the HEDG meeting and did a presentation on Jisc’s Learning and Teaching Reimagined programme and where Jisc is going next. It was good meeting and the presentation seemed to hit the spot.

I had a planning meeting about a session we’re doing with Advance HE on digital leadership which looks like it will be a really good session.

Looked at the presentation I am doing next week at Digifest on the future of digital leadership, what it is and where we are potentially going.

Favourite HE story of the week was Campus capers, 1970 style from Wonkhe.

Strange things sometimes happen in universities and we’ve reported plenty of them here over the years. From hauntings and strange happenings to animal action and of course true crime events on campus. But this event which recently caught my eye is one of the oddest I’ve noticed lately. It all happened just over half a century ago at Keele University. Those were turbulent times as the world transitioned out of the end of the heady 60s era into a very different decade.

My top tweet this week was this one.

No snow – Weeknote #102 – 12th February 2021

Well no snow again… everyone else seems to have had it.

Got bogged down into internal systems and processes this week, something which I have been avoiding in my new role, well I say new role, I have been doing this job for nearly two years now. In my old role I would have to deal with contracts and suppliers, not so much as Head of HE. However with working with universities on digital strategy, digital leadership and blended learning, means I need to immerse myself back into these systems.

Did a fair amount of work on future visions this week.

I think this is an interesting concept for future online events. The concept is to record some keynote videos that can be made available for online conferences. Reminded me of how some have used TED talks in the past. However there is a difference between YouTube or Netflix and an online conference.  How do you add value to an event so that it is more than just streamed video? How do you facilitate social interactions, networking, discussion and also how do you encourage this?

Had an interesting and useful discussion on assessment this week with colleagues from the UK and Australia. The key challenges the HE sector have faced over the pandemic include: maintaining academic standards and quality as assessment is transformed, student wellbeing and engagement in regard to assessment, the skills and capabilities of staff to assess online, and how to transform at scale and at pace. There are still issues with assessing within in vocational and practical qualifications, There are challenges with PSRBs and Professional Certification.

These are the main challenges and pain points that arose from our research back in April and more recently, it is not an exclusive list and is potentially going to change as universities move again through the assessment process and learn new lessons on what they can and can not do.

  • Maintaining the academic standard and quality as required by internal and external regulations, as they translate and convert existing practice into online modes.
  • Ensuring staff have the necessary digital skills and capabilities to successfully deliver online assessment, across the assessment lifecycle. Each step of the lifecycle will require different skills to deliver.
  • Transform multiple modes of assessment to online versions at scale and at pace. Many universities have experience of designing and delivering online assessment, however they will not have done this at scale or transformed at the pace required.
  • Maintain student engagement through the next few weeks and through the assessment process, as they continue to socially isolate and study remotely.
  • Ensure student wellbeing during a time of crisis remotely and consider the impact of online assessment on wellbeing as an extra pressure and source of stress.
  • What technologies are out there that could be used to design, deliver and support online assessment? Which technologies should we be using?
  • What are other universities doing with online assessment? What best practice is out there? Who is doing it well? How do we compare?

My top tweet this week was this one.

I WON THE ELECTION – Weeknote #90 – 20th November 2020

Official sources called this election differently

The US election continues to dominate Twitter though seeing less of it on the mainstream news. Saw a number of people on Twitter claiming to have won the election!

Five years ago this week myself and Lawrie were delivering the second residential of the pilot for the Jisc Digital Leaders Programme at the Holland House Hotel in the heart of Bristol. We had spent four days delivering that week. We also had some great cakes and pastries.

Even the coffee was nice. We learnt a lot from the process and spent the next few months iterating the programme, dropping and adding stuff based on the feedback we had from the pilot delegates.

Less than a year later we delivered the programme to paying delegates in Loughborough, again we reviewed what we did and adapted the programme again, before delivering to groups in Manchester, Belfast and Leicester.

Continue reading I WON THE ELECTION – Weeknote #90 – 20th November 2020

I haz fibre soon! – Weeknote #84 – 9th October 2020

I had a fair few meetings this week on a range of topics, including learning and teaching, the Data Matters conference, consultancy, pipelines, and public affairs.

This story from The Register was not really surprising, Unis turn to webcam-watching AI to invigilate students taking exams. Of course, it struggles with people of color.

AI software designed to monitor students via webcam as they take their tests – to detect any attempts at cheating – sometimes fails to identify the students due to their skin color.

I am not surprised, in my work on the Intelligent Campus, when we did some research into facial recognition, there was quite a bit of coverage about how it only really worked with white males. Can we be surprised then when used for exam invigilation that it fails on the same issue?

In a similar story, UK passport photo checker shows bias against dark-skinned women.

Women with darker skin are more than twice as likely to be told their photos fail UK passport rules when they submit them online than lighter-skinned men, according to a BBC investigation. One black student said she was wrongly told her mouth looked open each time she uploaded five different photos to the government website.

There is a question here about removing the systemic bias we find in AI and algorthims being used in education (as well as the wider society). A deeper question is how does that bias get there in the first place?

Across the week we saw more universities report large covid-19 infections in their student populations.

Sheffield Hallam has seen over 370 cases of Covid since the beginning of term and the University of Sheffield has seen 589 cases. The local area has also seen a dramatic increase in the number of people testing positive.

 Another 1,600 students have tested positive for coronavirus at Newcastle’s two universities. Newcastle University says 1,003 students and 12 members of staff have tested positive for Covid-19 in the past week. That’s up from the 94 students reported last Friday. There have also been 619 new cases among students at Northumbria University, compared with 770 last week. That means nearly 2,500 students and staff have tested positive since returning to studies.

More than 400 students and eight staff members at the University of Nottingham have tested positive for Covid-19. The university said the figures would be “higher than other universities” because it was running its own asymptomatic testing programme.

Almost 400 students and staff at Queen’s University in Belfast, Northern Ireland, are self-isolating after more than 160 people tested positive for Covid-19. A university spokesperson said the safety and wellbeing of staff and students was the university’s first priority.

One result of this is a lot of universities are moving back to online teaching.

This week, the University of Sheffield and Sheffield Hallam University both said they will move more learning online. The University of Sheffield said all teaching will move online from Friday until 18 October. Sheffield Hallam said it will increase the proportion of online teaching, but keep some on-campus.

Both universities (Newcastle and Northumbria) said they had extensive plans in place to support students. Earlier today they said they would move most of their teaching online in response to the outbreaks.

The two main universities in Manchester are teaching online until “at least” the end of the month after a coronavirus outbreak among students. Manchester Metropolitan University (MMU) and the University of Manchester (UM) said it was a “collaborative decision” with public health bosses and “won’t impact” on teaching quality. It comes after 1,700 students were told to self-isolate at MMU on 26 September.

fibre
Image by Chaitawat Pawapoowadon from Pixabay

I took the plunge and ordered full fibre from BT and if it goes to plan I will be getting 900Mb/s down and 110Mb/s up from the new all fibre connection. This will be much faster than my current 32Mb/s FTTC connection and so much faster than the ADSL connection I had between 2012 and 2017 which rarely went above 1Mb/s.

My top tweet this week was this one.

I am working, I may be at home, but I really am working – Weeknote #79 – 4th September 2020

Shorter week due to a Bank Holiday in England, the weather wasn’t up to much.

I wrote a piece about the reality of robots. The premise of the article was that:

When we mention robots we often think of the rabbit robots and Peppa robot that we have seen at events. As a result when we talk about robots and education, we think of robots standing at the front of a class teaching. However the impact that robotics will have on learning and teaching will come from the work being undertaken with the robots being used in manufacturing and logistics.

The draft of the article was based on conversations and some research I had done over the last few years. This was an attempt to draw those things together, as well as move the discussion about robots in education away from toy robots which are great for teaching robotics, but how robots could and may impact the future of learning and teaching.

I remember in one job when we bought a Peppa robot, in the support of teaching robotics. One of my learning technologists asked if the team could get one. We then had a (too) long discussion on why would be need a robot and how it would enhance learning and teaching in subjects other than robotics? The end consensus was more that it was cool. This was a real example of the tech getting in the way of the pedagogy.

Peppa

It’s September, so schools and colleges are back this week, operating in a totally different way to what they were doing just six months ago.

At my children’s secondary school, the students will now remain in the same room throughout the day and it will be the teachers who move from room to room. Each child will have a designated desk which they will sit in each day for at least the first term, if not the rest of the academic year. It won’t be like this at colleges and universities, but restrictions will still need to be in place to mitigate the risk of infection.

There has been quite a bit of discussion online and in the press about people returning to the workplace. Sometimes the talk is of returning to work. Hello? Hello? Some of have never stopping working, we have been working from home! The main crunch of the issue appears to be the impact of people not commuting to the workplace and the impact this is having on the economy of the city centre and the businesses that are there.

Personally I think that if we can use this opportunity to move the work landscape from one where large portions of the population scramble to get to a single location via train or driving to one where people work locally (not necessarily from home) then this could have a really positive impact on local economies, as well as flattening the skewed markets that the commute to the office working culture can have on house prices, transport, pollution and so on.

I wrote more thoughts on this on my tech and productivity blog.

video chat
Photo by Dylan Ferreira on Unsplash

I read an article on The Verge this week which sparked my interest.

These students figured out their tests were graded by AI — and the easy way to cheat

I posted the link to the article to the Twitter (as I often do with links) and it generated quite a response.

Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI. I wrote a blog post about this article, my tweet and the responses to it.

There was a new publication from Jisc that may be of interest to those looking at digital learning, Digital learning rebooted.

This report highlights a range of responses from UK universities, ranging from trailblazing efforts at University of Northampton with its embedded ‘active blended learning’ approach, to innovation at Coventry University which is transforming each module in partnership with learning experience platform Aula. The University of Leeds, with its use of student buddies, and University of Lincoln’s long-standing co-creation work are notable for their supportive student-staff approaches. University of York, however, focused on simplicity in the short term and redesign longer-term. The University of the West of Scotland is also focusing on developing a community-based hybrid learning approach for the new year.

I am going teach, was a blog post I wrote about the nature of teaching in this new landscape.

The Office for Students are reviewing the challenges the sector faced during the Covid-19 pandemic and are calling for evidence.

This call for evidence is seeking a wide breadth of sector input and experience to understand the challenges faced, and lessons learned from remote teaching and learning delivery since the start of the coronavirus (COVID-19) pandemic in March 2020.

The OfS are looking to see what worked and what has not worked. What will work in the future and what about the student experience in all of this.

I was quoted a few times in this article, How digital transformation in education will help all children.

As many teachers and learners have discovered recently, Zoom fatigue, that that needs to be accounted for when designing curriculums. “You need to design an effective online curriculum or blended curriculum that takes advantage of the technology and opportunities it offers, but likewise doesn’t just bombard people with screentime that actually results in a negative impact on their wellbeing,” says Clay.

I also mentioned connectivity.

“As soon as you took away the kind of connectivity and resources you find on campus, it became a real challenge to be able to connect and stay connected,” says James Clay, head of higher education and student experience at Jisc.

This was something that was echoed in a recent survey on digital poverty from the OfS.

During the coronavirus (COVID-19) lockdown, 52 per cent of students said their learning was impacted by slow or unreliable internet connection, with 8 per cent ‘severely’ affected.

The survey also found the lack of a quiet study space was also impacting on the student experience.

71 per cent reported lack of access to a quiet study space, with 22 per cent ‘severely’ impacted

Friday was full of meetings, which made for a busy day.

My top tweet this week was this one.

It wasn’t cheating!

using a laptop
Image by Free-Photos from Pixabay

I read an article on The Verge this morning which sparked my interest.

These students figured out their tests were graded by AI — and the easy way to cheat

The student in the article was undertaking an online test, and it wasn’t multiple choice, but short form answers to questions.

…he’d received his grade less than a second after submitting his answers. A teacher couldn’t have read his response in that time, Simmons knew — her son was being graded by an algorithm.

What the parent found was that by using a mix of keywords, or “word salad”, the system would mark the answer as correct. So the student could “cheat” the system!

The article itself was stemmed from a Twitter thread.

I posted the link to the article to the Twitter (as I often do with links) and it generated quite a response. Didn’t go viral or cause a Twitterstorm, but the article got people thinking about the nature of assessment and marking, with the involvement of AI.

There was quite a bit of feedback that this wasn’t cheating, but actually providing an answer to the question that the AI would mark as correct.

Others felt that this wasn’t AI.

I would agree, this was being called AI, but it was more of a system which matched keywords from answers given by students to a list provided by a teacher. The system wasn’t analysing what and how an answer was written, it was a text matching process.

A flawed approach to testing, which resulted in students been able to “game” the system to get 100%.

The lesson here is, for anyone looking at automated online assessment, is if there is a way in which the system can be manipulated, then it probably will be.