Monthly Archives: May 2013

Importance of question design #octel

One of my colleagues told me they were planning a session on question design for students who had opted to create eLearning materials as part of their course.  As we were looking at assessment and feedback on #octel this week, I asked them if they could send me their session.

Roberta was kind enough to record a quick video going over her session, explaining both what good question design involves and how she planned to deliver the session to the students.

Roberta Perli – Question Design

Does weighing a pig make it smarter? #octel

This week for #octel we need to”list and critique up to three different assessment approaches available on Technology Enhanced Learning courses”

The 3 I am going to use are

  • Online tests with automatic feedback
  • Online Essay submission with online marking
  • Online peer assessment

Online tests with automatic feedback – done right these can be very useful, make sure students get the feedback as soon as possible.  I talked a lot about these in my last post so I wont go into more detail here.  One thing I missed was the importance of designing your questions well – I hope to feature a guest post from my teams assessment expert which covers how to design questions (post should appear by the end of the week).  Also as the technology gets better the feedback can become more detailed and personalised (see this post about the automatic generation of personalized reporting)

Online Essay submission with online marking – at the really simple level if you stick to the same form as you would with paper (ie a student submits an essay, you write comments on it) then online has 2 advantages,

  • Its more convenient for students to submit, they can “hand it in” from anywhere with an internet connection rather than physically going to an office
  • Students (and staff) can look back at earlier essays done by the student and see feedback etc rather than searching through old paperwork for previous essays and feeback

However there are a number of advantages to online marking, if you use technology to be creative, this old case study from my departments website shows how use of audio and video feedback helps students.

Online Peer Assessment – you can do this offline but the paper shuffling is problematic and it can take much longer as students will have to physically hand over work.  You do need to be careful online as well because having some students not do the inital essay can have repercussions further along the process.  However it will ensure the students engage with the learning process giving them multiple passes over the material

  • First pass – Writing the essay
  • Second pass – reading the rubric/marking scheme the lecturer provides
  • Third to Fifth pass – reading, considering and marking other students work (I like to ask them to mark 3 essays)
  • Sixth pass – reading marks and comments from other students
  • Seventh pass – reading what the lecturer thinks of the marking they did
  • Eighth pass – reading the final marks of the lecturer on their work

This compares very well with the 2 passes done in a traditional “write essay/get marks” assessment.

Pig 001

So what about the smart pig?  Well its a reference to the quote “Weighing a pig doesn’t make it heavier” which is often trotted out when someone wants to criticise testing and say it doesn’t help learning.

Now it is true that a lot of students get tested to within an inch of their lives, and sometimes it seems that the test are simply so that someone can make league tables of schools, check the teacher is doing their job etc.  I will avoid going off into a rant about this and instead simply say – when you are testing your students, make sure they are learning from the experience, not just doing a test to get a mark.

Teaching using a VLE #4 Multiple choice tests (no seriously!)

One of my pet hates is the boring shiny store bought compliance e-learning, you know the kind – A page with a picture and 2 chunks of text and a next button, every few pages a handful of multiple choice questions that a monkey could answer to “check the learning”

Because of things like this I am not a big fan of the computer marked test (sometimes disparagingly called a “multiple guess test”) because it is something that is so easy to do badly.

I was going to cover computer tests later in the series but someone emailed me some fantastic research (Susanne Voelkel. Combining the formative with the summative: the development of a two-stage online test to encourage engagement and provide personal feedback in large classes. (2013) Research in Learning Technology 21: 19153) on the effect of tests so I am going to cover it now.

Feedback – The 2 key things about VLE/computer tests is that they A) mark themselves and B) Give instant feedback.  The thing is that while A) will save you time you really really need to spend a lot of time on B) because when students have just got something wrong (or right) they are receptive to immediate feedback and you need to make sure that it is good and it deals with the mistake that they made – sadly yes this does mean you need to provide individual feedback for each wrong answer.

Motivation –  as you can see from the research above, motivating students to take the test is key 

  1. If you give them good tests with decent feedback students grades will improve, but not all students will take the tests (and thus we assume miss out on the benefits)
  2. If you make the test compulsory (ie counts towards the final mark) all students will take the test but you cant make the results available instantly, students don’t seem to get the same benefit from feedback if it is delayed.
  3. If you set a pre-test with feedback and students need to get a certain score before they can take the test that counts towards the final mark, then grades will improve and all students will take the tests. (most VLE’s will allow you to set this up – ask your friendly Learning Technologist how to do it)

Sadly again number 3 will take the most effort on the part of academics – but if you are going to do it, then you should do it properly.

Don’t over test your students – If there are lots of questions then students will not pay as much attention to individual question feedback, similarly if there is a big test every week students will lose interest.

FoundX v2

Use the diagram to find X

Either have a couple of big tests and use something else to know how your students are doing on the rest of the course OR have weekly tests of 3 or 4 questions and a couple of big tests on top.

If I was doing it I would have 1 or 2 big tests and the weekly questions I would ask in class using the flipped classroom method of Eric Mazur (watch this you tube for an idea of the kind of effect it can have in class)

Finally dont add questions that require a human to mark them in your VLE/Computer tests, because A) this will mean you spend time marking them and B) the feedback will not be instant and so will not be as effective.

Finding resources, making resources? – I don’t have time! #octel

Your standard academic is overworked and time poor, they would like to provide high quality additional resources to help their students study but if finding them is a problem they may not have time, quality checking takes more time.  Creating new resources yourself can take time – loads of time if you have to learn a new tool to do it.

So how do we solve this?

Last week on the #octel mooc was all about resources, finding ones that other people have made and making them yourself.  I was really busy last week and so had literally no evenings free and so I didn’t even found time for the “Try to find one resource you can use” activity.

However I found this post by Jim Pettiward “Ramblings of an ocTEL junkie (Week 4)” and that and my thoughts about not having time myself kicked off this post.

How do I find it? The first and possibly the largest problem is finding something on the topic you want, you need to know A) Places to look for things and B) how to search.  For #octel we were given a number of sites and repositories to search, the most comprehensive evaluation I found was this post “iTunesU – you need to learn how to search it…” (which despite the title had searched everything!).

However even when you know where and how to search it still takes times so…

Get others to do the work – either by having a good PLE (as Helen Crump suggests in this interesting post) or by getting your students to do it! (its not hard – ask your students to all go away and find one good resource/explanation/reading on the topic.  They all add the links to a google doc then get them to evaluate which are the best 5 and why, you can then check 5 items and you will both have some good resources and an idea of if your students have a good understanding – next year you can use the resources)

How do I use it? Even when you have found something its not always clear how it was intended to be used, now its not that much of a problem for a experienced educator to weave it into their session or materials but it would be nice if people started adding a “here is how I used it” file when they put stuff online (TED Ed is a brilliant example of how this can be done).

Well its good but its not quite what I want (this can be particularly true for Big OER – for a discussion of Big Vs little OER try this from Wane Barry).  Sometimes it doesnt quite fit or covers something you dont want to cover or worse it has something that is wrong.  So you go to tweak it and you find they have only uploaded the finished product not the working files (for a simple idea of this imagine a document that you wanted to change a single picture in.  If its a pdf you cant edit it so you have to copy and paste all the words into a word doc, then save each picture and then import them into the word doc and then change the picture. If its a word doc (the working files) you simply change the picture and export to pdf).

So ideally people should upload the working files and finished product but even then you run into the problem of the tools they have used and how difficult it can be to learn a new tool set.  So you need to download the tool, learn to use it, then make your change.

At this stage some staff have reached the “Right I will make it myself” stage.

Danger Educators

Interestingly I was looking for CC image of someone making a wheel, after 10mins of searching I gave up and made this instead. Sometimes its quicker to make your own

More than a powerpoint takes skill/training – if we are assuming that the lecturer wants to do something more than a powerpoint then they need to learn new tools or system, for some this is not a problem (and I would advocate all staff at least having some creation skills).  However for some things its just not worth the time it would take to get staff upskilled.

For example a few years back a colleague of mine went out and shot a short video for a member of staff, gave the footage a quick edit and put it onto the VLE for them.  It took him all afternoon, I said he will be back every time he wants a video – could you have trained him instead?  My colleague responded, If I had trained him we would still be discussing lighting and framing the shot, I would spend 2 days training him and he wouldn’t then do another video for a year at which point I would have to train him again.

For simple things, train the staff.  For more complicated source an expert.

eFactory – at City of Bristol College, just before I left for another job they set up a team called the eFactory, the idea being that academic staff could pop in a discuss what they wanted created with a Learning Technologist, who would then mock it up, get it approved by the academic then create it and add to the VLE.  They have created a lot of materials from the highbrow to the sock puppet level.

So I guess if your institution sets up one of these then your problems are over yes?  Well not quite, when I checked back with my old colleagues they said they still have problems getting academic staff to engage with them, because the academics still dont have time!

Time, Money and Moocs – so we come back to the original problem, in this time of information abundance it still takes time and effort to find or create and then assemble materials into a coherent structure (course, lesson?).  If the course has run before then the chances are some materials already exist which means, if staff dont have the time, there is a danger that the existing materials (a tired old powerpoint?) are all the students will get.

Worse – if the course has already successfully run then the institution that provides the course will be very unlikely to understand the need for paid academic time to rework/reassemble materials.  So there is unlikely to be any money available to buy staff time to do this or even to buy staff time to spend with Learning Technologists who will then do the work.

However there is a new type of course emerging – the MOOC.  A lot of institutions are trying these out and because these are “open” everyone can see them, so institutions want them to be good and are prepared to spend money.  A number of the Universities rushing to create a MOOC have set aside the budget to buy staff time to build/run them and sometimes time for Learning Technologists as well.

So maybe institutions will start to comprehend that improving/updating a course takes time and money.  Maybe that time/money will actually be forthcoming if/when students start wondering – “If the free course is fantastic, why is my paid course just powerpoints?”

(*Important note – I am not saying that existing University education is not fantastic, large sections of it are, what I am saying is that there is potential to be better – in some cases much much better)

#octel Active learning session


Ok so I am going to borrow/rework an idea from someone I did some teacher training with a few years back (cheers Rob Hawkins!).
In the original, Rob told his A level students “you work for a university and have just landed a big research grant to look into stem cells” the students had to go off research, plan and record on their phones a 5 minute radio interview explaining to the public what their research was and what they hope to achieve/find out. When the students had finished they bluetooth’d the recordings to Robs phone. Oh and if you ask a bunch of A level students to work in groups of 4 you can pretty much guarantee that they have at least 1 smartphone.

So lets try to do this in an online environment.
I am assuming all your students know of the common webspace to go to (vle, mooc home page, class website etc). I am also going to use google docs and hangouts but you could use wikis and other conferencing software (skype)

My question will be research into the effect of neonicotinoids on bees (it has been in the news recently so is topical and fits my “env chem” background). Obviously you should choose something from your curriculum if you decide to try this.

So I will put up a bit of preliminary reading on the question and some initial links to news stories.
You need some way to split your students up into groups of 4, if control is important to you then most vles allow you to create groups and assign students to them.
I am going to be a bit more flexible and have a place where students can post that they are starting a group and comment that they would like to join. This will allow students to form their own groups.

The first task for the students will be to research the subject and to share any useful links on a google doc shared with the group.
Next they will have to plan out/script the 5 minute “interview”. Again this should be done on the google doc but it would also be good if the group used a google hangout so they can do some face2face planning.

The final stage will be to use a published hangout to do the interview and record it to youtube

The finished youtube video should then be posted on the course site OR sent to the lecturer. I will get my students to post it so they can see the work of the other groups.

There we are one active learning experience and I even have anecdotal evidence it works (thanks Rob)

What does my learning look like? #octel

So for #octel this week our first task is

What is learning? (30 minutes)

Think about the last time you learned something. Describe what you learned? How did you go about learning it? What strategies did you use? ”

I am going to look at 2 very different bits of learning that have happened to me recently.

First yesterday I attended a taster training on collaborative web tools (and to be fair it was pitched at people who know a lots less about the web than I do).  The trainer did a lot of talking and describing tools and I learnt that you can share edit rights on prezi.

This is a ” I: Know that” level of learning, someone told me something and I remember it (although I expect if I hadn’t written it down in the blog I would have forgotten in a few weeks because I don’t use prezi because I find it too fussy)

The other bit of learning was a month or so ago when I was taking part in another mooc (#EDCMooc) and we were looking at what it meant to be a human (see an earlier post of mine), I learnt how to create my own definition of what is a human.

This is at the top end of the scale “IV: Other”, I watched a number of short videos about aliens, robots etc to give some context, saw Steve Fuller give a short talk about defining humanity and read a number of articles on the subject.  I then read and commented on some of the blog posts of some of the other students before writing and posting my own blog (and responding to the comments)

It is this second piece of learning that I am going to use for

“Activity 3.1: Theories of active learning

Select a learning theory and write a short discussion piece on how you would relate it to

  • how you learn;
  • your current practice;
  • the design of Technology Enhanced Learning activities. “

The learning theory I will use is Constructivism.

Now when I think about learning I tend to think about “thought forms”, there is something to know/learn/comprehend and I gradually build up a understanding of it in my brain.

Asparagin 8418

I envisage this as assembling a bunch of connections in my neurons a bit like the molecular models we use in chemistry to show the structure of molecules (see on the left but also imagine the electricity of my thoughts running through it).

Each time I think about it or engage with more learning on the subject the thought pulses through the structure and adds bits or reinforces bits or gloms the whole lot onto another thought structure if I make a connection with another bit of prior learning.

When I am working with others I am building my “thought form” and helping them to build theirs (as they are helping me to build mine).  They dont actually reach into my head and make connections but as they explain their thinking I can consider it and use it to build connections in my head.  2 people that collaborate on learning the same thing will have different “thought forms” in their heads, an experienced lecturer that is giving a talk (turning her “thought forms” into words) will have a very different “thought form” to the ones that end up being created in the minds of her listeners.

In my current practice I tend to get my students to do things/work things out themselves rather than demo-ing them.  This gives them more time to construct their knowledge themselves, I also tell them to ask people around them if they get stuck as this helps both them and the person they ask.  I also do a lot of group discussion and pair and share activities because this gives several passes over a topic (discuss it, tell others, do it?) .

When designing Technology Enhanced Learning activities make sure you give students several opportunities to engage – this post on my #octel Intro listed the number of interactions on a mooc vs traditional class.  The thing with this is, that by using the other students and what they say and think, you are moving into the Connectivist theory where your learning is in knowing who has the information and where it is (the social structure is the learning?).  However some have claimed that this is not strictly different enough from Constructivsim and other theories to stand on its own as a theory so in true Connectivist style I will link to a member of my PLN who has a much more erudite take on this than I do.