Does weighing a pig make it smarter? #octel

This week for #octel we need to”list and critique up to three different assessment approaches available on Technology Enhanced Learning courses”

The 3 I am going to use are

• Online tests with automatic feedback
• Online Essay submission with online marking
• Online peer assessment

Online tests with automatic feedback – done right these can be very useful, make sure students get the feedback as soon as possible.  I talked a lot about these in my last post so I wont go into more detail here.  One thing I missed was the importance of designing your questions well – I hope to feature a guest post from my teams assessment expert which covers how to design questions (post should appear by the end of the week).  Also as the technology gets better the feedback can become more detailed and personalised (see this post about the automatic generation of personalized reporting)

Online Essay submission with online marking – at the really simple level if you stick to the same form as you would with paper (ie a student submits an essay, you write comments on it) then online has 2 advantages,

• Its more convenient for students to submit, they can “hand it in” from anywhere with an internet connection rather than physically going to an office
• Students (and staff) can look back at earlier essays done by the student and see feedback etc rather than searching through old paperwork for previous essays and feeback

However there are a number of advantages to online marking, if you use technology to be creative, this old case study from my departments website shows how use of audio and video feedback helps students.

Online Peer Assessment – you can do this offline but the paper shuffling is problematic and it can take much longer as students will have to physically hand over work.  You do need to be careful online as well because having some students not do the inital essay can have repercussions further along the process.  However it will ensure the students engage with the learning process giving them multiple passes over the material

• First pass – Writing the essay
• Second pass – reading the rubric/marking scheme the lecturer provides
• Third to Fifth pass – reading, considering and marking other students work (I like to ask them to mark 3 essays)
• Sixth pass – reading marks and comments from other students
• Seventh pass – reading what the lecturer thinks of the marking they did
• Eighth pass – reading the final marks of the lecturer on their work

This compares very well with the 2 passes done in a traditional “write essay/get marks” assessment.

So what about the smart pig?  Well its a reference to the quote “Weighing a pig doesn’t make it heavier” which is often trotted out when someone wants to criticise testing and say it doesn’t help learning.

Now it is true that a lot of students get tested to within an inch of their lives, and sometimes it seems that the test are simply so that someone can make league tables of schools, check the teacher is doing their job etc.  I will avoid going off into a rant about this and instead simply say – when you are testing your students, make sure they are learning from the experience, not just doing a test to get a mark.

Teaching using a VLE #4 Multiple choice tests (no seriously!)

One of my pet hates is the boring shiny store bought compliance e-learning, you know the kind – A page with a picture and 2 chunks of text and a next button, every few pages a handful of multiple choice questions that a monkey could answer to “check the learning”

Because of things like this I am not a big fan of the computer marked test (sometimes disparagingly called a “multiple guess test”) because it is something that is so easy to do badly.

I was going to cover computer tests later in the series but someone emailed me some fantastic research (Susanne Voelkel. Combining the formative with the summative: the development of a two-stage online test to encourage engagement and provide personal feedback in large classes. (2013) Research in Learning Technology 21: 19153) on the effect of tests so I am going to cover it now.

Feedback – The 2 key things about VLE/computer tests is that they A) mark themselves and B) Give instant feedback.  The thing is that while A) will save you time you really really need to spend a lot of time on B) because when students have just got something wrong (or right) they are receptive to immediate feedback and you need to make sure that it is good and it deals with the mistake that they made – sadly yes this does mean you need to provide individual feedback for each wrong answer.

Motivation –  as you can see from the research above, motivating students to take the test is key 

1. If you give them good tests with decent feedback students grades will improve, but not all students will take the tests (and thus we assume miss out on the benefits)
2. If you make the test compulsory (ie counts towards the final mark) all students will take the test but you cant make the results available instantly, students don’t seem to get the same benefit from feedback if it is delayed.
3. If you set a pre-test with feedback and students need to get a certain score before they can take the test that counts towards the final mark, then grades will improve and all students will take the tests. (most VLE’s will allow you to set this up – ask your friendly Learning Technologist how to do it)

Sadly again number 3 will take the most effort on the part of academics – but if you are going to do it, then you should do it properly.

Don’t over test your students – If there are lots of questions then students will not pay as much attention to individual question feedback, similarly if there is a big test every week students will lose interest.

Use the diagram to find X

Either have a couple of big tests and use something else to know how your students are doing on the rest of the course OR have weekly tests of 3 or 4 questions and a couple of big tests on top.

If I was doing it I would have 1 or 2 big tests and the weekly questions I would ask in class using the flipped classroom method of Eric Mazur (watch this you tube for an idea of the kind of effect it can have in class)

Finally dont add questions that require a human to mark them in your VLE/Computer tests, because A) this will mean you spend time marking them and B) the feedback will not be instant and so will not be as effective.