Monday I was off to Bristol, for a late afternoon meeting. It was nice to be back in the office and see the changes and improvements since I was last there a week or so back. It is a nice place to work.
This report is the result of an experts meeting exploring assessment in universities and colleges and how technology could be used to help address some of the problems and opportunities.
This report was widely reported in the press across the UK.
Assessment is a challenge for many institutions, often resulting in attempts to fix it, but sometimes I think we need to dig deeper and re-imagine assessment as a whole.
Having discussed the coronavirus in last week’s weeknote, the situation has been escalated and the Department of Health has described the coronavirus as a “serious and imminent threat” to public health.
It comes as the government announced new powers to keep people in quarantine to stop the spread of the virus.
In order to do this the Department of Health has described the coronavirus as a “serious and imminent threat” to public health.
The overall risk level to the UK remains “moderate”.
Wednesday I was at the 18th Jisc Learning Analytics Community Event at Newman University in Birmingham. There were various talks and discussions and overall it was an interesting day.
I published a blog post about the ALT Learning Spaces SIG that happened last month.
Really enjoyed reading @jamesclay's ideas for creating new learning spaces. Thinking beyond the classroom is crucial these days but unfortunately not a priority in most universities. We really need to reflect more on how students learn. https://t.co/CjeqXBea59
Thursday I was in our Bristol office working on a document with colleagues. I had quite a few conversations about the Education 4.0 roadmap I am working on and how the sector needs to start thinking and preparing for both the challenges, but also the opportunities that there is with this potential view of the future.
You can tell winter is coming, but I did enjoy having an extra hour on Sunday. I watched this video on Sunday morning about how university students in Europe and the US are paying Kenyans to do their academic work for them.
The global market for academic writing is estimated to be worth $1bn (£770m) annually.
Findings from the largest dataset gathered to date on contract cheating indicate that there are three influencing factors: speaking a language other than English (LOTE) at home, the perception that there are ‘lots of opportunities to cheat’, and dissatisfaction with the teaching and learning environment (Bretag & Harper et al., 2018).
These influencing factors could be mitigated, could we assess in the learner’s native language? Culd we improve satisfaction with the overall teaching and learning environment? Often easier said than done.
This contract cheating or collusion is a major headache for universities in the UK, but I wonder if the answer isn’t about creating systems or processes that can identify when cheating or collusion is taking place, but ensuring that assessment is designed in a way that means there is no incentive to chat, collude or pay someone else to undertake the assessment.
A surge in anxiety and stress is sweeping UK campuses. What is troubling students, and is it the universities’ job to fix it?
We know that there is a student mental health crisis and the reasons for this aren’t necessarily clear. We know there has been increase in the demand for mental health services at universities. The article notes that there has been research into the causes of this, but lays the blame for the crisis on the way in which universities are managed and run, leading to students not being in control of what they do and saddled with debt.
On the 2nd October 2009 I was at the ULCC Event, The Future of Technology in Education.
Little did I know the impact that this presentation would have on me, my future career and education in general.
I felt a little intimidated to be invited to talk at the event, we wouldn’t have called it imposter syndrome back then, but I did wonder if I was the right person to talk at such an interesting conference. It certainly had a TED talk feel to it. I must thank Frank Steiner and Tim Bush from ULCC for their support and help and inviting me to talk at this FOTE and future FOTE events.
2009 was quite a year for me, I had won the ALT Learning Technologist of the Year award that year. It was also the year of “The VLE is Dead” debate at the ALT Conference.
The event took place at the Royal Geographical Society in Kensington, which I remember wasn’t the easiest place to get to via the underground. Knowing London better now I think I would probably have just walked across Hyde Park from Paddington to get there. From about 2001 I started going to London a lot for work, well a few times a year, which was considerably more than when I was a lecturer in Bristol. I use to go to London, arrive at Paddington, take the underground, pop up somewhere, go to a meeting or an event, before popping back down into the underground on my way home. These days I visit London a lot more and have spent a lot more time walking around London, so have a much better grasp of the geography of the place. I remember being quite impressed with the place, and that you could see the nearby Albert Hall.
I spent a fair bit of time putting my presentation together, in the end it comprised 82 slides… and I only had twenty minutes to deliver my talk. A challenge that took some doing.
My presentation was entitled The future of learning…
The aim of my presentation was to discuss how learning would and could change with the affordances of technological change.
So what of my predictions?
Well we know predicting the future is hard and generally most people get it wrong.
You will no doubt not be surprised that I got a lot of things wrong…
One thing I feel I did get right was that mobile was going to be big and important. I said how I felt mobile was the future. The audience did have a range of mobile devices themselves, but most phones were nothing more than phones that could do SMS and the Snake game. There were a few smartphones out there, but if my experience was to go by, they were clunky and difficult to use. We had the iPhone, but it hadn’t quite had the impact that it has had by today.
We didn’t have the iPad, that would arrive the following year. So no surprise that in my talk at FOTE I didn’t mention tablets
My talk actually started off talking about the past, how we are still impacted and embedded by the past, which makes change challenging and difficult.
I then talked about the present and some of the issues and problems that technology was causing in classrooms and lecture theatres. PAT testing was a real concern for many back then, don’t hear much about it these days in relation to BYOD or learner devices.
One of the challenges I saw back then was how academics and educationalists wanted to categorise learning, so we had e-learning, m-learning, mobile learning, online learning, digital learning, etc….
I said that I thought categorising learning and putting it into different boxes was restricting and that really we should focus on learning and blur the boxes, blur the boundaries.
It was fine to talk about the “boxes” at conferences and in papers, but experience has shown that categorising learning into boxes caused confusion for teachers and academics, who rightly focussed on the word before the learning as a problem to be solved and then found it challenging.
However back then I said, and I still stand by this today, is that learners and academics need to understand the potential of technology and digital to better understand the affordances and opportunities that it can provide for learning. You don’t need to be ab le to do the technology, but you do need to know what it can do.
I also brought in scepticism about technological advances, something I would draw upon in future talks and presentations.
Video (and film) had been used for learning for years, but people were sceptical and convinced that video (ie lecture capture) would stop traditional learning activities. However we know that television didn’t destroy radio, we know that radio didn’t kill newspaper, books didn’t replace folk stories. When we have a new technological development, often the result is a negative impact on existing technologies, but often the result is affordances about the potential of the new technology, enabling access that otherwise wouldn’t be possible.
I also talked about the potential of video on mobile devices. Video cameras were getting smaller and cheaper, the quality was getting better as well. You could buy video cameras which could record HD video, even if it was a challenge to capture and edit it on standard computers of the time. This was before the concept of streaming became mainstream. I showed a Sanyo Xacti camera which was waterproof and dropped it in a jug of water. These cameras could be used in dirty and dusty environments and the washed under the tap!
Mobile phone video has become so much better now. I am still impressed that my iPhone can record 4K video… If only we could get people to record video in landscape!
GPS was usually an option on devices back then, today it is more prevalent in the devices we buy. I saw this as an opportunity, the concept of geo-location based learning was something that felt quite magical at the time. Your device knows where you are, so personalises the learning based on your location. What I missed was how location tracking and would become a very big issue for people.
There was a bit of a backlash against e-Books back in 2009, as people felt that they weren’t as good as “real” books. For me they weren’t a replacement for books, they enabled different ways of reading. For many e-Books and e-book readers enabled a new way to access books and content, that otherwise would mean they wouldn’t have access. I presented on the future of reading at #FOTE10 the following year. I became a bit of an expert on e-books as as result. I presented on e-books at many different events and conferences, as well as writing a chapter in a book, and finally a book on Preparing for Effective Adoption and Use of Ebooks in Education in 2012.
Today e-books are part and parcel off education with easier access to books by students from academic libraries. As I did predict, we didn’t see the end of physical books, we still have bookstores and people still buy physical books.
Back then in 2009 connectivity was either slightly haphazard, or expensive, or both. We had 3G, but it wasn’t widespread, it would be another three years before we saw 4G.
WiFi was there, but it didn’t always work and network congestion would often cause the WiFi to fail. This happened with frequent regularity at events and conferences I attended back then, as delegates killed the WiFi with too many connections.
In the future I felt connectivity wouldn’t just be important, it would be critical for the future of learning.
Today we have really good (and cheap) mobile data, 4G is more available and 5G is starting to appear. Ubiquitous WiFi is certainly there compared to ten years ago, Eduroam has made it easier for people in education to connect when travelling, but WiFi is easily found in most places. This has allowed users to do so much more when travelling and moving about, or just when drinking coffee. I certainly notice how many people are streaming video, having video chat, doing so much more, because they had the connection and the bandwidth to do so.
Mobile often means battery power, and access to charging. Everyone remembers how their Nokia phone would last days on a single charge, today, most people seem to complain how their smartphone battery doesn’t last the day. Batteries may not seem to have got better, they have, just that we demand more power for our complex devices. We have seen significant improvements in battery technology, but we have seen a huge increase in our demand for power on our devices. Streaming video requires more power than reading an e-mail. One thing that has taken time to filter through was the importance of the ability to charge devices. Since 2009 we have seen trains and buses adding power sockets, and USB ports for charging as well. Hotels have added similar sockets. Some lecture theatres now have plug sockets as well.
In my 2009 presentation I talked about the technological penknife.
This is one thing I got very wrong, I thought that the idea that a device that did everything meant it did everything badly. A penknife has multiple tools, but most of them aren’t very good doing the stuff they are designed to do. People would prefer to have specialist devices for specific activities. Why would you have rubbish video from a phone, when you could have a decent HD video camera? Why would you use the rubbish microphone on a device, when a specialist recording device would do it so much better? Well that didn’t happen, in reality we have seen devices become so much better that we don’t need to have multiple devices. We have the penknife, but it’s a really good penknife, really good at everything.
I then went on to talk about change and the importance of managing change. I talked about how change can be a series of small steps, but noted the importance of missing steps, endless steps and steps that trip you up.
These slides were really where I started to understand strategy and writing strategies much more. This certainly helped me in future roles and influenced heavily the design of certain aspects of the Jisc Digital Leaders Programme in which I was part of the research and development team led by Lawrie Phipps.
I talked about activity, technology should never be about the technology, it needed to be about how it could enhance or improve activities. Or where the affordances created new opportunities for different activities. We still have a perception that we shouldn’t talk about technology first, though sometimes I think we should.
Technology allow for flexibility, flexible curriculum, flexible approaches to delivery, flexible learning. I think we have made a little progress here, but so much more is possible these days. The technology enables flexibility, but that doesn’t mean it will just happen, there is so much more that needs to happen to enable flexibility.
Back then I felt sharing was important, not just sharing content (as in open) but also sharing ideas, concepts and approaches. Not that this didn’t happen, but it was difficult to do so. Today it is much easier to share than it was back then, so much so, I think we have forgotten about the time when this didn’t happen.
I talked about the importance of working collaboratively. Since the talk online tools have made it so much easier to collaborate. Collaboration across institutions (and countries) is so much easier these days. Tools such as Slack enable groups to talk and work together.
I talked about innovation, celebrating ideas. Innovation doesn’t always mean better, it means different or new. Following on from that I talked about experimentation and encouraging it within our institutions.
If you want innovation, then it needs to be embedded into the strategy, rewarded and not penalised when things go wrong. It needs to be done in collaboration with learners not done to them. I think we are seeing much more innovation and collaboration these days, and the student voice is helping to inform developments and ideas.
I said we need to re-think assessment, technology was going to have an impact. I think it has, but not in the way we thought it would. We try and use technology to “fix’ assessment today, rather than re-imagine how we assess.
I talked about culture and how culture can enable change, but also frustrate it. Culture is about what and who we are, it’s the sum of the people within an organisation. This was something we covered years later in the Jisc Digital Leaders Programme.
I have always seen technology as a solution to a problem. Technology in itself is not the problem needing to be solved. This was something that I wrote about in 2018.
I finished the presentation about talking about the future and how the future was about the learner, the student. It was about how they wanted to learn, where they wanted to learn, what they wanted to learn and with whom they wanted to learn. Why did we need to think about the future, it was because we needed to think about the learners, then, now and in the future.
So did I predict the future?
It certainly though had a huge impact on my future, some of which I have outlined above. As a result of this talk I was invited to speak at a range of events and conferences on the future of learning and a range of mobile learning events. I spoke the following year at FOTE 10 about the future of reading, which resulted in me doing much more in the e-book space.
So there is also a video of me (looking much younger) presenting, if you want to watch what happened…
We know that change isn’t easy, if it was then all we would need to do would be buy a book on the subject and just do it.
When it comes to the embedding of digital technologies into teaching, learning and assessment I have spent over twenty years undertaking this kind of activity at a range of organisations and across different levels.
Going back to when I was a Business Studies and Economics teacher at what was then Brunel College (now City of Bristol College) I kind of fell into the use of technology to support teaching and learning. I was an ILT (or TEL) Champion before even the phrase existed. Going back a little further I was never the kind of techno geek or computer nerd many of my peers appear to be when comparing histories. I didn’t do Computer Science at school. I didn’t own a computer, I didn’t have a BBC Micro, nor the Commodore 64, ZX Spectrum or anything like that. I did have a bike though!
At University in the late 1980s we had a VAX system and it was really that kind of got me interested in technology, but it was as a tool to solve problems. I discovered I could use this thing called electronic mail to send letters to a friend at another university instead of using the post! This was quite illuminating, until I got flamed by the administrator at the other university, for not using the correct format for my e-mail… Most of the time however the use of the computers was in many ways pointless as my examinations required me to hand write essays, so why would I use a word processor, having said that I did get introduced to Word Perfect 4.2 and did think that this was better than a typewriter.
After university on a business enterprise course I was introduced to spreadsheets that I used for creating balance sheets and cashflow forecasts. For me that was probably the eye opener that got me into technology, more so than anything I had seen before, well does that make me a boring person?
By the time I was working at City of Bristol College I was using my own PC at home to create presentations, photocopying onto clear acetates as initially we didn’t have a digital projector, and we were still using OHPs. When the college did buy a projector (we had one for the whole college) it was a real effort to use it, it was the size of a small suitcase and we also had to lug the screen around as well. Due to lack of processing power, I would often bring in my own PC box, as the laptop couldn’t cope with the strain of my presentations. My PC also had a Matrox Rainbow Runner video card which I used to show full screen video. There was no internet and certainly no wireless network. My what we take for granted today, looks at his phone which can stream HD and 4K video to a projector using 4G connectivity, things do change. Things did improve and we started to see more technology in classrooms.
One outcome from all this was that as I was seen as something of an innovator in this area I was asked to support and train staff, not just from my faculty, but also other areas of the college. One clear memory of this was the impact, often I would train individuals who would then go off and do their own thing (or not). Sometimes I would train all the staff in a faculty and this is where I would often see not only the most resistance, but also the biggest impact. Where a faculty set expectations about how technology would be used, you would see the greatest impact. One faculty I taught how to use Powerpoint to (probably badly) many of the staff were quit resistant or complained they couldn’t do this technology thing, there weren’t enough PCs, not all classrooms had PCs and projectors, and so on… remember this was 1998 or 1999. The head of faculty though had made it clear that not only were all staff to do the training, and create presentation materials, but that all the presentations would be stored and shared centrally. No presentations stored on floppy disks (we didn’t have USB sticks back then) being used by individuals only.
What was a transformative moment for me was the understanding that showcasing, cascading and piloting really didn’t have the transformative impact that senior managers hoped for. Generally the main impact was that enthusiasts would become more enthusiastic and those more reluctant, would either not do anything, or just pay lip service to any initiative. What really caused institutional change was effective strategy and leadership and clarity about what was going to be done, what was expected from staff and what they needed to do and by when.
This did stick with me over the years I moved into positions where my role was to embed technology into teaching and learning. Though I often used the cascade model for staff development, but knew that this was not the ideal model for systemic holistic change across an organisation. It worked well on some individuals, but it was not transformative.
In a similar vein the use of other people’s research and running pilots was interesting and useful, but did not result in institutional change, it could inform other activities, but the idea that the best way for mainstream transformation was to run a pilot was something that I found never worked and never had the impact that others thought it would.
What I really tried to do was transform the entire institution. I would use tools such as cheeses and models, but one key aspect was culture change. Changing the culture was often about hearts and minds, but also challenging the myths and misconceptions about technology and using learning technology with learners. I would use pilots and research to inform this process.
I also knew that if something didn’t work, then to try again, but this time do it differently. Don’t keep trying to do the same thing again and again.
I know that this isn’t easy, if it was easy then we would all have done it!
One thing that came out of this was the understanding that we often make assumptions about staff capabilities and their ability to know how to embed technology and the potential of what technology can do. Just because a member of staff can has been given the training in how to use the tool or service, it doesn’t mean they know how best to use that tool or service to enhance teaching and learning, and for what function or process of the learning activity the tool would support or enhance.
This week on ocTEL we’re looking at assessment. As part of my thinking I reflected on the use of quizzes in Moodle.
Designing Moodle quizzes is much more than just been able to use the quiz tool from a technical perspective. There is a real art to crafting questions so that they not only allow the learner to test their understanding, but also require a higher level of thinking.
If we look at the following multiple choice questions, the format of which is one of many different types available on Moodle, it provides the structure and the practitioner provides the question and the answers:
Which is these is a mammal?
This question does not test understanding, most students would be able to guess the answer or would not find it challenging. Within Bloom’s Taxonomy this is testing knowledge only, the bottom layer of the triangle.
In terms of feedback, you can design Moodle quizzes to provide feedback on questions. So you can explain why their answer is wrong or right and where to look for further information or support.
Onto a similar question:
What is the capital of Australia?
If we look at this question if you didn’t know the answer then you would need to do some research. However as with the previous question within Bloom’s Taxonomy this is testing knowledge only, the bottom layer of the triangle. It’s more challenging than the first question, but if you didn’t know it already then a quick Google search and you have the right answer.
So what about this question:
Which of these is the odd one out?
The “problem” with this question is that there is no single right answer. The answer needs an explanation, and it’s the explanation that demonstrates understanding of the question, not the answer.
If we look at Bloom’s Taxonomy it is possible with this question to go all the way to the top.
However Moodle will struggle with assessing a question with no “right” answer and certainly would not be able to assess the explanation.
You could provide generic feedback on why there is no “right” answer, but that may not be useful for all learners. Feedback needs to be personalised to be really effective. Students generally don’t appreciate generic feedback.
This doesn’t mean that Moodle quizzes aren’t an useful tool for checking learning, but its limitations in assessing higher order thinking needs to be considered when designing assessment.
Submitting assignments online through the VLE is a great way for learners to ensure a) that the work submitted is logged, b) it’s unlikely to be lost, c) they can, once graded, refer to the grade and feedback when they want to.
The problem with submitting assignments through the VLE is when the assignment asks for the students to submit a physical thing as their assessed piece of work.
If a learner creates a sculpture for example, or makes a lasagna, welds copper tubing, these things can not be submitted electronically. Submitting a photograph is of course possible, but why? It’s an extra step that isn’t strictly necessary.
Many learners will often do a presentation for an assessment, others will do a live performance or demonstrate a technique. Now yes you could video these “performances”, but video files can be quite big and you not want to upload these to the VLE (especially if you use tape over an SD card).
However it is still useful to grade these offline activities on the VLE to inform the learners how they did and provide feedback. Of course generally that is done anyway, so why put it on the VLE?
One reason is consistency, if other (written) assignments are been uploaded to the VLE then a learner will probably want to at some point know the grades for all their assessed work regardless whether it was a written essay or a live presentation.
Another reason is that one of the key factors that aids retention is ensuring learners know in terms of their assessment what they have done, what they are doing and what they need to do. If they can find all this information in one place rather than using different systems for different assignments using differing forms of assessment then it will make life better for the learner. If feedback and grading on written work is on the VLE, it makes sense then to use the VLE for all assessment whether it be written or another activity.
The key message I wanted to get across to the participants was that just because they had assessed the way they had always done, this didn’t mean that was the only way they could assess learners. Often we assess the way that we do, we do it because we have always done it that way. There are now new tools and technologies that allow us to enhance and enrich assessment. and make it more engaging and effective for learners.
Sometimes we need to think differently, especially if the current methods of assessment are not doing what we need them to do.
An interesting presentation from a keynote by that Steve Wheeler, Assessment in the Digital Age:Fair Measures?
Assessment, which in FE is heavily dictated to by the exam boards is always challenging to change to make “fit for purpose” and I do wonder if we assess because we need to assess rather than actually use it for something useful?
news and views on e-learning, TEL and learning stuff in general…