Show me the evidence…

I think this line is really interesting from a recent discussion on the ALT Members mailing list.

…in particular to share these with academics when they ask for the evidence to show technology can make a difference.

Often when demonstrating the potential of TEL and learning technologies to academics, the issue of evidence of impact often arises.

You will have a conversation which focuses on the technology and then the academic or teacher asks for evidence of the impact of that technology.

From my experience when an academic asks for the evidence, then the problem is not the lack of evidence, but actually something else.

Yes there are academics who will respond positively when shown the “evidence”, however experience has taught me that even when that happens then there is then another reason/problem/lack of evidence that means that the academic will still not start to use technology to “make a difference”.

When an academic asks “for the evidence to show technology can make a difference” the problem is not the lack of evidence, but one of resistance to change, fear, culture, rhetoric and motivation.

You really need to solve those issues, rather than find the “evidence”, as even if you find the evidence, you will then get further responses such as, wouldn’t work with my students, not appropriate for my subject, it wouldn’t work here, it’s not quite the same, not transferable…. etc…

Despite years of “evidence” published in a range of journals, can studies from Jisc and others, you will find that what ever evidence you “provide” it won’t be good enough, to justify that academic to start embedding that technology into their practice.

As stated before, when someone asks for the “evidence” more often then not this is a stalling tactic so that they don’t have the invest the time, energy and resources into using that technology.

Sometimes it can be “fear” as they really don’t have the capabilities to use technology and lack the basic ICT confidence to actually use various learning technologies, and as a result rather then fess up their lack of skills, they ask for the “evidence”, again to delay things.

Just turn it around, when you ask those academics who do use technology then, you find that the “evidence” generally plays little or no part in their decisions to make effective use of technology.

So what solutions are there to solve this issue? Well we need to think about the actual problems.

A lot of people do like things to remain as they are, they like their patterns of work, they like to do what they’ve always done. This is sometimes called resistance to change, but I think it’s less resistance to change, and more sticking to what I know. I know what works, it works for me, and anything else would require effort. This strikes me more about culture, a culture where improvement, efficiency and effectiveness are seen as not important and the status quo is rarely challenged.

Unless an organisation is focused strategically and operationally in improvement, widening participation, becoming more efficient, then it is hard to get people to think about changing their practice.

When it comes to embedding learning technologies we often talking about changing the culture of an organisation. This can be hard, but doesn’t necessarily have to be slow. I am reminded of a conversation with Lawrie Phipps though in which he said we have to remember that academics often like the current culture, it’s why they work in that place and in that job. So don’t be surprised when you are met with resistance!

Creating a culture which reflects experimentation, builds curiosity and rewards innovation, isn’t easy, but also isn’t impossible. There are various ways in which this can be done, but one lesson I have learnt in making this happen, is that the process needs to be holstic and the whole organisation needs to embrace that need to change the culture. What I have found that you need to identify the key stakeholders in the organisation, the ones who actually have the power to make change happen. I found in one college I worked in that the real “power” wasn’t with the Senior Leadership Team (who often had the same frustrations I had when it came to change) but the Heads of Faculty, the managers who led and managed the curriculum leaders. They had the power to make things happen, but they didn’t always realise they held that power.

Getting the rhetoric right, but also understood across the organisation is critical for success in embedding learning technologies. Often messages are “broadcast” across an organisation, but staff don’t really understand what is meant by them and many staff don’t think it applies to them. Getting a shared understanding what is required from a key strategic objective is challenging. I have done this exercise a few times and it works quite well, pick a phrase from your strategic objectives and ask a room of staff or managers what it means and to write it down individually. You find that everyone usually had a different understanding of what it means. A couple of examples to try include buzz phrases such as “the digital university” and “embrace technology”.

Finally looking at what motivates people to use technology to improve teaching, learning and assessment.

When I was teaching, I would often experiment with technology to see if it made a difference, if it did, I adopted it, if it didn’t I stopped using it. The impact on the learners was minimal, as I didn’t continue to use technology that didn’t make a difference or was even having a negative impact. What I also did was I applied the same process and logic to all my teaching. So when I created games to demonstrate various economic processes, if they made a difference I used them again, if they didn’t then I would ask the learners how they would change or improve them. When I gave out a reading list of books, I would ask the learners for their feedback and, those that didn’t make a difference or had no positive impact, then they would be removed from the list! I was personally motivated, but we know you can’t just make that happen.

When I was managing a team I ensured that any experimentation or innovation was part of their annual objectives and created SMART actions that would ensure they would be “motivated” to do this. Again you need to identify the key stakeholders in the organisation, the ones who actually have the power to make this happen.

So when someone asks you to show them the evidence what do you do?

16 thoughts on “Show me the evidence…”

  1. Yes yes yes – create a culture of innovation and reward experimentation – and play – and playful learning.
    Where people ask for the evidence – *perhaps* they are expressing doubt… or perhaps TEL is being imposed in a managerialist top down way – where the tech is prescribed and the play is proscribed… and the culture is surveillance and control?

  2. I think you are spot on, Sandra, with your observation about “a managerialist top down… culture [of] surveillance and control”. (I hope you don’t mind my elision of your final sentence:-)

    As I wrote in the ALT discussion that sparked James’ blog post:

    “… very often what lies behind such problems [in the implementation of TEL] is the same cultural hang-up – a perceived need for the institution to exercise top-down command and control over how technology is used within it. Those in charge don’t seem to realise that what technology enhanced learning is all about is a loosening of control.
    … we need a completely new institutional and cultural framework in order for the educational enhancements of technology to flourish.”

    1. Absolutely Terry!
      I just spotted the sort of replies to Sue’s query that divide us one from the other (hah – they don’t have to justify lectures or books – it’s fear – people are resistant).
      I think Sue was hoping to discover what unites us.
      If people are asking that question – there is a reason…
      I love your argument for a loosening of control.
      The trouble is that for so many, the digital age has in fact ushered in the age of senior management taking ever more control over how we teach – how we assess – how we feedback…

  3. “When an academic asks “for the evidence to show technology can make a difference” the problem is not the lack of evidence, but one of resistance to change, fear, culture, rhetoric and motivation.”
    Three assumptions (at least) in here. The first is the slightly patronising anecdote about the academic in question; the second is their assumed response to tech; the third is that the culture and its changes are positive. We can apply that last to changes in culture in any field and sensible, objective people will rightly ask what evidence accompanies such assertions of change, so that they too can comment on (or analyse) the potential impact of such change. Innovation and experimentation may well be encouraged, but speaking as an FE teacher I wonder how much time I can give to both support innovation and student progress. The second is not something to leave to whimsical chance, while I tinker with tech. Hence the call for robust research that enables purposeful use.

    1. These aren’t assumptions, but evidence based on experience of working with hundreds of staff across colleges and universities over the last twenty years. I am not saying that research and evidence aren’t important, the key message is that when some (therefore not all) academics ask to see the “evidence” we need to realise that the “lack of evidence” isn’t always and necessarily the issue at hand and we may want to delve deeper into the real reasons behind the “evidence” question.

      Thank you for reading and commenting.

      1. I ‘ve said as much in the past and in the original e-mail thread I questioned what the call for ‘evidence’ looks like. It’s forever been the case with learning technologies that they are difficult variables to isolate and measure in research. I sit on the fence with their regard – clearly they have their place in certain contexts. My own research found positives and negatives in user experiences, but while those kinds of value judgments are important it seems obvious that the uses must be guided by design based on research evidence of effective pedagogical strategy, focuses on critical realism and avoids hype. For example, Mitra’s SOLE environment is hype around self-determination that we cannot risk with students. However, Garrison et al’s Community of Inquiry model allows for more strategic support of community learning. As we recede from these meta-theories, we can find more examples of very specific pedagogical support mechanisms, so that the teaching and learning comes to the forefront and the technology is actually in the background – then we can stop with the hype and focus on the activity.

  4. But where is the student voice in all of this? To my mind, that’s the real position to consider. What is it that they are capable of adopting, in what format, when, where and why?

  5. I have also found the request for the evidence as a tactic to show their superiority, asking really for you to show your worth in their field, in their realm, in their sphere where evidence and published research is key. Even when you are able to support the request and show them evidence they will, as you say, find another excuse to dismiss it – it’s not the right quality of journal, not recent enough, etc.
    There’s always an excuse. The trick is to build a relationship with the academics so they have more trust in you and your work, and hopefully this will remove the barrier of fear from their arguments.

  6. This is brilliant reading. I just have one thought though: While it makes absolute sense that the request for evidence can often (although, importantly, not always) be a ‘stalling tactic’ that has its origins in broader cultural attitudes – is there a possible question about what kind of ‘evidence’ academics are looking for? The reason I ask is that there seems to be an assumption sometimes that the requested evidence of effectiveness relates to student outcomes: The impact is seen in terms of student results or performance. Now – and I want to try and say this without in any way sounding critical of academics – sometimes that is not the impact they need. Increasingly, the job of being a lecturer or academic is one of accepting, on a regular basis, new responsibilities and expectations. During my years as a lecturer, I would have loved to have the time to experiment with embedding technology just because of its impact in the effectiveness of teaching and learning. But I rarely did. Increasingly then, the ‘evidence’ I was looking for was to demonstrate that technology could balance the increasing workload at one end, by reducing it somewhere else. For example, if there was evidence to show that using technology to improve student writing could reduce marking time by, say, 30% – would there be quite so much resistance?

    1. I had a similar conversation yesterday with a colleague. Technology can bring so much to the table, it’s not just about student outcomes. Technology could for example widen participation, increase recruitment, reduce workloads. Reflecting on my own practice, I started using technology as a teacher in the main as it saved me time an wasteful duplication. A simple example was when I put all my resources on my homemade VLE, then I spent less time at the photocopier, copying resources for students who missed a session or had a lost a resource. I recall my line manager wondering if I was actually working as my photocopying bill was virtually non-existent compared to my colleagues.

    2. Beautifully put, Jon Tulloch. The discussion must inhabit the lived lives of academics and student academics in the hostile education and HE contexts we inhabit… The more we have to engage in sophistry, the further away we get from the fierce, lively and messy business that is education. Most academics I knew would have loved the time and space to play with the tech and see what it could do. It was specifically proscribed – so people did it at home – their self-development was de facto transgressive. SMT imposed strategies – but they did not enable them – and definitely did not enable them creatively. I only finally got to play with Twitter and blogging and FaceBook for learning the first time they deleted me…

  7. I completely agree that culture change, and people, are at the centre of this issue, rather than the technology or the research itself.

    I would add, and I think it links to Jon Tulloch’s point, that e-Learning-types probably have the hardest job in convincing their colleagues of any change to culture or process. Academics are ‘trained’ to question everything, in the same way that the general public may not be (see Brexit, Trump, etc).

    I get a nagging feeling that the evolution of the culture of e-learning is Kuhnian in nature. By that I don’t mean that the paradigm shift only occurs once the old lecturers die off, but only on a much faster cycle determined by the prevailing technology. Unfortunately, this shift is probably tied to external (read top-down) changes to their working environments rather than an intrinsic desire to change practice.

    I certainly don’t wish the above to sound like a them-and-us argument, because I see no distinction between an academic and e-learning – the only difference is our ‘audience’.

    Obviously, what I have just said could be a complete load of tosh, so feel free to pick holes 🙂

  8. “From my experience when an academic asks for the evidence, then the problem is not the lack of evidence, but actually something else.”

    I agree, and you are right to suggest that the ‘something else’ is “resistance to change, fear, rhetoric and motivation.” I think that the same issues lie behind the other enduring objection that academics raise to adopting digital technologies: “I haven’t got time”.

    As far as the question about evidence is concerned, my usual response is to point out that technology on its own neither improves nor diminishes teaching and learning, or outcomes. Teaching practices, study skills, and digital literacies do that, and digital technologies can play a part. Any educational setting is a complex ecology, and trying to isolate one variable from that complexity is at the very least difficult, and probably not very helpful. To me, asking whether digital technology ‘enhances’ teaching and learning is like asking whether having a curriculum or using classrooms enhances teaching and learning. We need to move away from seeing technology as being separate from the social practices of teaching and learning. It is part of the context, not an add-on.

    Glynis Cousin (2005) suggests that the ascent of new media forms and the decline of others is inevitably experienced as a loss, leading to a yearning for the safety of the past (‘resistance to change’, ‘fear’, etc.). However, rather than trying to soothe the furrowed brows of academics by assuring them that everything will be just fine, and providing evidence that technology will ‘enhance’ what they have always done, we should say that change is difficult, but inevitable and requires new ways of doing things. Then find ways to support that change. This all leads to your point that embedding technology needs to be a holistic process of cultural change – “Creating a culture which reflects experimentation, builds curiosity…” – this is something that academics can surely buy into, after all, it’s what they do for a living isn’t it?

    Cousin, G. (2005). Learning from Cyberspace. In R. Land & S. Bayne (Eds.), Education in Cyberspace (pp. 117–129). Routledge Falmer.

  9. There is another issue here about evidence, and I apologize for being late to the discussion #travelling. I know Glynis (reference above) and Ray Land. (And that book is on my shelf and well read) I would not say, at the time of their writing that book, that they were embedded in the culture of TEL, or actively using it in their practice. They were writing from the perspective of staff and educational developers. So some academics would look at that evidence as being both outside of their community from an academic perspective, but also and a key component of change in this community, they were seen as being outside of their subject area. (The LTSN subject network was one of the most intelligent ideas ever to be deployed in UK HE – #obvs imho). Always remember that for different disciplines the nature of evidence will also be very different. Finally, let’s look at the title. Education in Cyberspace. It is never going to leap off the shelf and resonate with the majority of UK academics who’s main role is lecturing undergrads. With apologies to Glynis and Ray who I have great respect for.

Leave a Reply