Bob van Vliet

An incomplete list of lectures’ functions

Since the switch to online education 18 months ago, it’s become clear that on-campus lectures used to serve a whole range of functions – not all of which can be served by online classes. We designed around that for our online and half-half courses.

Now that we’re expecting to be able to get back to fully on-campus for all our courses again, if we want to, people are asking: do we want to?

I’m not so sure. At least not for large scale lectures – as I wrote in my Delta column at the start of this year.

In any case, it seems good to explicitly think about what physical lectures do, so that we can think critically about how we want to achieve those things going forward.

OK, here goes:

One-way, teacher -> students stuff:
- Information delivery. Similar to what books do. The part many feel that Gutenberg made obsolete and the thing that the ‘flipped classroom’ transfers to asynchronous reading/watching by students themselves. Multiple categories:
  - Content
  - Logistics / Practical matters
- Motivation, by an enthusiastic teacher/speaker, by showing cool examples, by physical demonstrations, etc.
Teacher <-> Students (interaction is obviously low in traditional lecture halls)
- Quick checks on understanding (i.e. polls and multiple choice questions)
- Opportunity for asking questions/discussion, both during and before or after the lecture. Becomes more scary and more difficult as student numbers increase.
Students -> Teacher
- Signal perceived value of the lectures by showing up or not.
- Motivation for the teacher. Speaking to a hall of students is something to get out of bed for. It’s why many like ‘hybrid’, because at least they have some students there to see and make them feel like they’re doing it for actual people.
Students <-> Students
- Social gathering, seeing and interacting with peers and friends, during the lecture, before, and after.
- Discussing course content and materials, helping each other out, bot in terms of figuring out difficult topics, and answering each other’s questions about logistics: When is this due again? Wait, what is the assignment, exactly?
For students individually
- Structure to the day, a reason to get out of bed in the morning. Knowing a teacher is coming to a room, and knowing other students are going, is more social pressure/motivation than an online session nobody will see you at and that you can always watch the recording of.
- Being seen, feeling like you exist to others.
- Feedback on how well you’re doing, in the form of seeing other peope either understanding everything or struggling as much as you, or more.
For the teacher individually (I feel this category gets ignored a lot, as is Students -> Teacher, see above)
- Feeling cool. You’re on stage, giving a show, after all. Or at least, you’re delivering profound and valuable knowledge to the unknowing. Good for the ego.
- Provides a ready made size for chunking your content. Far too large, but still. It’s a template. That’s nice.

Critical Pedagogy and Engineering Design

So I’ve been reading a lot of Critical Pedagogy lately – Paulo Freire’s Pedagogy of the Oppressed, bell hooks’ Teaching to Transgress, and, currently, Jesse Stommel and Sean Michael Morris’ An Urgency of Teachers. I find myself both in strong agreement, nodding along and thinking ‘Yes!’, and at the same time with strong doubts about whether and how to translate it to engineering education, especially with first year’s BSc courses.

The question of how Critical Pedagogy applies to STEM fields has been addressed, but still: almost all the examples and proposals from its main proponents and practitioners appear to relate to liberal education humanities classes.

Design education, whether in engineering design, architecture, or other design disciplines, is a little bit in-between outright mechanics and maths classes and full-on humanities education. On the one hand, it is a matter of thinking critically about the world and your values and goals in relation to it, it empowers, it is already sometimes a liberating experience, I believe. But on the other hand – again, especially at the lower levels, in introductory projects – it very much has the feel of ‘training’ and ‘instruction’ as opposed to true education that is interactive and egalitarian from the outset.

As a design teacher, you do act from a position of authority, the authority of expertise. You have a skill, a set of abilities, that your students don’t yet have. They came to your faculty because they want to learn how to do what you do. And for that to happen, they need to submit to your instructions. First they need to do without understanding, before being able to look back critically and understand why it is you had them do certain things (Cf. Donald Schön).

Now that I’m writing this, I realize that student numbers make a big difference here. In a studio of ~25 students, it’s actually not so difficult to be truly responsive, to interrogate students’ ideas and ideals together as a group. With ~750 first year’s students, in 14 clusters of 8 groups of 6 or 7 students, together with a small army of coaches and student assistants, it’s a whole different story. There, you’re practically forced to put up a sort of obstacle course for the students to run through, egged on and managed by strict deadlines, and then to respond only in a much more limited way, only to selected work by a limited subset of students.

Perhaps the main obstacle to transforming my pedagogy, then, is simply the raw numbers? That would be ironic, as it’s exactly that massive quality of more and more classes that contributes to students learning to just do what’s required, to listen, and to adapt to how things are, instead of developing their own critical awareness.

Ranking, Evaluating, and Liking

I’ve been thinking and reading about ‘ungrading’.

I first encountered the argument against numerical grading of students’ performance in Sanjoy Mahajan’s course Teaching College-Level Science and Engineering, which links to Alfie Kohn’s The Case Against Grades. The term ‘ungrading’ was introduced by Jesse Stommel, who comes at the idea informed by the much broader notion of ‘critical pedagogy’. In a recent online presentation, Stommel mentioned the work of Peter Elbow, which led me to Elbow’s essay Ranking, evaluating, and liking: Sorting out three forms of judgment.

Now, I’ve read many arguments against grading — that it decreases intrinsic motivation, that grades are not effective feedback, that they do not express how much was learned, etc. — but Elbow states the case in a way I found striking:

Differences between student work are multi-dimensional.
Grades are one-dimensional.
Therefore, grades are mostly meaningless.
Peter Elbow, paraphrased.

When you put it like that, it’s so obvious! Of course grades feel like bullshit.

In the first part of his essay, Elbow argues that we should rank as little as possible. In the second part, he argues that we should try, instead, to evaluate — to provide feedback on multiple criteria. And he argues that teaching should include ‘evaluation-free-zones’, where students are free to follow their own judgement without worrying about what the teacher wants to see. I think Elbow is correct here, but this second part of the essay was hardly surprising, novel, or uncommonly insightful to me.

But then on to the last part, the part on liking. Elbow writes:

It’s not improvement that leads to liking, but rather liking that leads to improvement.
Elbow, P. (1993). Ranking, evaluating, and liking: Sorting out three forms of judgment. College English, 55(2), 187-206.

I found his discussion of the need for teachers to like their students’ work in order to be able to give good feedback spot-on. He’s talking about teaching and evaluating writing assignments, but the same goes for design projects, in my experience:

If I like a piece, I don’t have to pussyfoot around with my criticism. It’s when I don’t like their writing that I find myself tiptoeing: trying to soften my criticism, trying to find something nice to say–and usually sounding fake, often unclear. I see the same thing with my own writing. If I like it, I can criticize it better. I have faith that there’ll still be something good left, even if I train my full critical guns on it.

This!

I find it easy and natural to be excited and enthusiastic about student work. And I’ve always known that this was a big part of being able to teach well. But I had never quite put it together with my equally great eagerness to critique, to point out problems and possible improvements.

Good teachers see what is only potentially good, they get a kick out of mere possibility–and they encourage it. When I manage to do this, I teach well.

Yes!

Although I think I might need to make this more explicit to students, and to become better at pointing out what exactly I see that is potentially wonderful.

A Feedback Checklist

Using a ‘cover sheet’ is a simple and effective tool to enhance feedback on student work, this study claims.

Giving teachers a form to fill out instead of just having them make notes in the margines of the work, makes sure feedback is not limited to comments on what was handed in (feed-back), but includes an appraisal of how it matches up to the final criteria or end-goals (feed-up) and practical advice on how to proceed (feed-forward).

Sadly, the study didn’t test using only this form against the more traditional way of giving feedback. The experimental condition was adding this form to notes on the work, while the control was just those notes alone.

So yeah, extra feedback is better. No shit, Sherlock.

Nonetheless, a form like this seems like a good idea to use as a form of checklist. Because I do think i often forget one or two out of the three kinds of feedback.

(Via ScienceGuide)

An Online Lecture Platform Actually Designed for Online Lectures

Or: Why and How I Built My Own Online Lecture System

Current options for large scale online lectures are not designed for lectures.

Zoom, Teams, and similar platforms are enterprise software. They’re built for business meetings. Lectures are not business meetings. Especially when they are for large groups, they involve a very different, and very particular relationship between ‘presenter’ and ‘attendees’.

Let Them Make the Thing First

When introducing design methods, perhaps it would be a good idea to focus instruction purely on the product at first. What table, matrix, or whatever should they make? What criteria should that thing adhere to? Explanation as to why those rules apply, and what role the method can and/or cannot play in a design process seem to fall mostly on deaf ears at first contact.

Make the thing. Do it again in a different course. Reflect on the process of making and using it. Learn the theoretical considerations then instead of beforehand. Concrete application before abstract explanation. The other way ’round feels logical to teachers, but may simply not be very effective. You haven’t created a ‘time to tell’ yet (c.f. Daniel Schwartz)

Onderwijsontwikkeling: Programma van Eisen

Mijn eerste jaar als docent bij het ontwerpproject in het eerste kwartaal van het eerste jaar van Werktuigbouwkunde richtte ik me bij het programma van eisen vooral op het uitleggen van toetsbaarheid: operationalisatie en toegankelijkheid. Maar op het tentamen bleek dat studenten dat compleet verkeerd begrepen hadden. In plaats van dat ze keken naar of een criterium objectief meetbaar was (bijv. ‘zo licht mogelijk’), hadden ze vooral opgepikt dat er een harde grens moest zijn (bijv. ‘lichter dan de vorige versie’). Als er wél een grens was, maar géén objectief meetbare grootheid, dan rekenden ze het criterium goed (bijv. ‘makkelijker te gebruiken dan het product van de concurrent’). Vice versa hadden studenten begrepen dat criteria zonder harde grens, maar wel met een goed meetbare variable, niet goed waren (bijv. ‘zo goedkoop mogelijk’).

Het volgende jaar heb ik daarom niet alleen aandacht besteed aan toetsbaarheid, maar ook aan verschillende typen criteria. Het belangrijkst daarbij was het onderscheid tussen functionele eisen (‘Wat MOET het doen?’) en prestatiecriteria (‘Wanneer doet het dat GOED?’). Maar ook randvoorwaarden en specificaties had ik onderdeel van de stof gemaakt. Dit ging beter, maar het verschil tussen die 4 typen criteria kreeg ik maar lastig overgebracht.

Simple Overhead Draw-and-Talk Videos Are a Good Idea

Fiorella & Mayer (2016) conducted a series of experiments that show the effect of seeing diagrams being drawn vs. showing and/or pointing at already-drawn static diagrams in (short) video lectures. The paper appears to be a summary of a PhD project.

Seeing a diagram being drawn improves learning compared with instruction that uses a static, complete diagram, even if the instructor points at parts of it during their explanation. This is probably because the combination of drawing and talking naturally applies the multimedia learning principles of signalling, temporal contiguity, and segmenting.

Digital Khan-style videos where you see the lines appearing without the instructor’s hand were less effective than real life videos where you actually see the instructor that’s doing drawing. Seeing only the instructor’s hand seems to be slightly better than seeing their (upper) body.

From the conclusion:

Overall, this research suggests that observing the instructor draw diagrams promotes learning in part because it takes advantage of basic principles of multimedia learning, and that the presence of the instructor’s hand during drawing may provide an important social cue that motivates learners to make sense of the material.

In other words: making simple overhead draw-and-talk videos is a good idea.

—

Fiorella, L., & Mayer, R. E. (2016). Effects of observing the instructor draw diagrams on learning from multimedia messages. Journal of Educational Psychology, 108(4), 528.

https://sci-hub.se/https://doi.org/10.1037/edu0000065

Design Reports vs. Design Papers

One of the things I find difficult in design education is the difference between teaching our students the skill of doing design – coming up with and developing products, machines, and other plans – and teaching them the logic of how to argue for the results of that work – presenting, justifying, and giving reasons for their proposals.

We teach our students (some version of) the design process, and then we ask them to write a report that presents that process and their design. There is a tension in that combination. In this set-up it seems logical to show how your process ‘led to’ your design. Showing your (cleaned up, idealized) process is treated as the justification or support for the final design. But the quality of your process is not necessarily evidence for the quality of your design. Vice versa, with this approach it doesn’t make sense to present all your discarded ideas and other dead ends, or to show all seven and a half earlier versions of what became the final design. That would create a report that’s just as messy and chaotic as the average design process.

A ‘design report’ in this fashion tries to serve two functions: to provide evidence of learning activities, and to provide evidence for the final design’s quality. Those two sometimes conflict. At the very least they’re not the same and trying to do both in one document compromises the effect of both.

Perhaps, therefore, it would be good to make an explicit distinction between a ‘report’ and a ‘paper’? A report reports – it tells your teachers what happened. A paper presents – it describes a problem, shows evidence, and argues a proposal to a audience of peers.

If you want to see whether undergraduate students are learning the right skills and methods, ask them for a report. If you want graduate students to produce something similar to an academic paper, leave the reporting out of it.

Designing Effective Multimedia for Physics Education

Muller, D. A. (2008). Designing effective multimedia for physics education. Sydney: University of Sydney.

https://www.sydney.edu.au/science/physics/pdfs/research/super/PhD(Muller).pdf

–

Derek Muller (creator of ‘Veritasium’ on YouTube), in his PhD thesis, shows that science education videos need to start with students’ misconceptions to be effective. A straightforward exposition can be worse than no instruction at all, because students do not change their mistaken views but do become more confident that they know how something works.