<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=115389302216927&amp;ev=PageView&amp;noscript=1">
LET'S TALK

Evaluating Training Effectivenes

    mega-menu-graphic

    Storyline Scheduled Public Courses

    Andrew Jackson

    Andrew Jackson

    Hi, I’m Andrew Jackson — co-founder of Pacific Blue Solutions and founder of Pacific Blue AI. I’ve spent almost 20 years helping L&D teams design learning that actually changes what people do at work. Alongside my weekly writing on Learning Re-Framed, this Learning Academy blog is where I share practical, evidence-based ideas for improving learning design and performance support in a changing, AI-enabled world.

    Recent posts by Andrew Jackson

    2 min read

    How Instructional Designers Can Manage Out of Control SMEs

    By Andrew Jackson on Fri, Feb 24,2012

    We can all feel our pulse quicken, our emotions rise when we get chance to talk or write about a topic that engages us totally.

    And we usually know lots about this topic. We can frequently talk about it for hours without getting bored. We can tell anyone willing to listen about its every last detail. In that sense, we are all subject matter experts (SMEs) in something.

    As instructional designers, when we have a talkative SME in front of us and limited time to get the information we want, it's worth remembering how our own passion for a particular subject matter can allow us to get carried away.

    So aside from being more empathetic to a talkative SME, is there anything else we can do to make our time with them more productive? I think there are four areas to consider when gathering content from SMEs. By the way, the greater the quantity of content you need to gather, the more you are likely to want to formalise the approaches below.

    Ownership
    Before any information gathering even happens, you need to take ownership of the process. This may involve becoming more assertive than normal: be quite specific about how you want the process to unfold, including the number of meetings you'll need, how long each meeting should be and how much time you'll need between meetings for reviewing and feedback.

    Planning
    Tempting as it might be to go into your early meetings knowing nothing, better to do research to familiarise yourself with the subject matter area. Spend time creating a basic project plan. Clearly define your and their roles in the whole process. Formally identify the risks of not getting the required information in a timely fashion and communicate this to the project sponsor.

    Connecting
    Your initial research can pay dividends once you start interacting with your SME. Exhibiting some knowledge of his/her topic can help build rapport and, more important, establish your credibility. Earn trust by emphasising the confidentiality of your information gathering sessions and the promise of a review of content before making it more widely available.

    As the content gathering progresses, aim to establish points of shared interest both within the subject matter area and outside. Most people appreciate a little interest in their life outside work.

    Focusing
    Set an agenda in advance of the meeting clearly stating goals and expectations.
    During your content gathering sessions, regularly paraphrase, clarify and summarise what you have covered; use closed questioning techniques if your SME has a tendency to go off on tangents. After the session, collate the content into a structured document you can share with your SME for review and feedback.


    It's easy to dismiss some of the subject matter experts we deal with in our professional capacity as out of control windbags who want to bore us and our learners with every last detail of their knowledge.

    That may be true. But let's not forget, given the right topic and the opportunity, many of us can happily do the same.

    So with a bit empathy and some detailed preparation and work before, during and after your content gathering, the analysis phase of your project need not be an out of control nightmare.

    If out of control SMEs are your current nightmare, check out our the Analysis and Planning modules in our instructional design programme for help on dealing with this problem.
    Topics: Instructional Design Course Design e-learning
    1 min read

    Does Compliance E-Learning Have to Be Boring?

    By Andrew Jackson on Tue, Feb 14,2012

     At a conference a year or so ago, I noticed a seminar that drew a good crowd was entitled, "Who says e-learning compliance training has to be boring?". Well not me for sure.

    Perhaps, I'm a bit naive, but even now (after many years in the world of learning) it shocks me that some people can shrug their shoulders and say., "well this material is pretty dry and boring, so we'll just have to accept that the way we deliver it is dry and boring". To me that's a bit like the designers at a car company saying, "well it's a bit difficult to design a really comfortable car seat, so we'll just fit the car with uncomfortable wooden benches and the passengers will have to lump it."

    Perhaps the acceptance of poor quality compliance training is linked to the box ticking mentality that often accompanies the dreaded 'c' word. We have to do the training  -  even though nobody wants to, so let's just collectively hold our noses and all be bored together - designers, trainers and delegates. Oh yes, and let's make it even worse, by delivering it as the most boring, sleep-inducing piece of e-learning you have ever seen.

    From this point of view, you'd think that compliance in a particular job role or organisation wasn't something anybody really needed to know or do. Yet it most definitely is.

    So rather than treating it as a dry abstract topic, why not relate it back to the context or contexts in which learners need to be compliant? Why not provide the learners with challenging, life-like scenarios and activities that require them to think about what they actually need to do to be compliant. How about some intrinsic, contextual feedback that vividly demonstrates the consequences of not being compliant or trying to cut corners.

    If you find yourself nodding your head in agreement but you are feeling a bit unsure about the effectiveness of your e-learning in general and your compliance e-learning in particular, take a couple of minutes to complete our E-Learning Impact Scorecard which can help you benchmark what you are doing against some of our recommended best practices. 
    Topics: Instructional Design Course Design e-learning
    2 min read

    Evaluating Training 2: Wear the Red Pants with Pride

    By Andrew Jackson on Fri, May 27,2011

    Last time, I shared Jim Kirkpatrick's story of 'red pants (trousers) syndrome' to illustrate how difficult it can be to get people to change the way they do things if they are unsupported after a training event.

    The Kirkpatrick four levels are all about minimising outbreaks of 'red trousers syndrome'. They encourage you to start at the end of the learning process, identify the results you want to achieve and figure out what kind of learning needs to take place to make that happen.

    Key to all this is not taking the 'sheep dip' approach to learning. In other words, the 'figuring out' of what you need has to take account of the fact that traditional approaches to designing learning are not necessarily the most effective.

    This is borne out by some astonishing results Jim shared with us. They are from long-term research carried out by Rob Brinkerhoff, comparing the benefits of a fairly traditional approach to training (emphasis on a one-off event) with a more collaborative approach (more balance between a training event and follow up activities). Here's a summary of the results.

    In a traditional approach to training design, 90% of the time is spent on design and development of the training event and only 10% on pre and post development activity. In this approach, typically the following happens to learners:
    • 15% do not try the new skills
    • 70% try to implement the learning but fail
    • 15% achieve and sustain the new learning
    In a more collaborative approach, the training designers work very closely with the client and 25% of time is devoted to pre-training prep and 50% to post-training follow-up. (Note: only 25% of the time is devoted to the training event itself). In this approach, typically the following happens to learners:
    • 5% do not try the new skills
    • 10% try to implement the learning but fail
    • 85% achieve and sustain the new learning
    This is one of the most compelling pieces of research-based evidence I have seen for a long time. It has made me realise that here at Pacific Blue we should make much greater efforts than we currently do to encourage, you, our clients to engage in this kind of collaborative approach.

    There's no question this is a more complex approach. It involves the co-operation of colleagues and managers who may not be taking part in the training event. But look at the results.

    If you are struggling with a 'red pants' problem of your own and are not getting effective application of learning after learners attend a training event, check out our learning impact strategy scorecard to help you do a bit of benchmarking around that. It only takes a couple of minutes to complete and it's entirely free.
    Topics: Measurement and evaluation
    2 min read

    Evaluating Training 1: Would You Wear the Red Pants?

    By Andrew Jackson on Fri, May 27,2011

    I had a great day recently at the Training Zone Live event. The highlight of the day for me was Jim Kirkpatrick's session on his (and his Dad's) four levels for evaluating the effectiveness of training.

    Jim was feeling a little jet-lagged, having just flown in from Australia the day before - but he ran an inspiring session, nevertheless.

    At one point, he explained to us that his Australian audience had introduced him to 'red pants syndrome' (that's pants in the American sense, by the way, so 'red trousers syndrome' for us Brits).

    So 'red trousers syndrome' is where you go on a training course and learn to do something in a particular way, then go back on the job and start implementing what you've learned. Only to discover that no one else much is bothering.

    In other words, it's a bit like wearing a pair of red trousers to work everyday, when everyone else wears black ones. You come in on the first day, feeling pretty pleased with your new look. But you quickly realise people are staring at your new trousers. Maybe over time they start to comment negatively on your appearance. Perhaps they even start avoiding you.

    In that situation, how long are you going to hold out? How long will it be before you start wearing black trousers, too?

    A nice metaphor to highlight the big problem that exists in many organisations. The one where the training happens, everyone feels enthused, but within a relatively short time they all go back to doing things in the same old way.

    Jim is pretty clear on what the consequences of not addressing this problem will be: training departments as we know them will eventually become obsolete.

    But as Jim explained, if you start at the end and identify the results you want to achieve and work back to work out exactly what you need to achieve those results, you can greatly minimise an outbreak of 'red trousers syndrome'.

    Next time, I'd like to share the results of some long-term research carried out by one of Jim's colleagues. This shows how avoiding 'sheep dip' training can have a massive impact on changing behaviours and embedding learning.
     
    If you are struggling with a 'red pants' problem of your own and are not getting effective application of learning after learners attend a training event check out our learning impact scorecard to help you do a bit of benchmarking around that. It only takes a couple of minutes to complete and it's entirely free.
    Topics: Measurement and evaluation