Category: English

Weeknotes 2024 week 4

Tried

  • Experimented with a new way to make feedback videos responding to student work. Instead of actively pausing/unpausing the recording, I filmed myself reading (and responding) to the work, and then used Recut to cut out all the silences. Now it’s a more accurate recording of my initial response upon reading. It wasn’t as smooth as I’d hoped, but I think I’ll try it again. I just have to be a little more performative (and consistently loud) when voicing reactions. And I could also combine both features, for example to pause and collect my thoughts for final remarks/evaluation at the end.

Made

  • I printed the text of Ingrid Robeyn’s 2019 Van Hasselt lecture as a little booklet to gift to my faculty’s dean. Reminded me how fun it is to make simple little booklets/zines. I did a series of those a while ago. Perhaps I should do more of that again. Easy way to be creative.

Notes / Other stuff

  • Today was my last meeting as a teacher in the first year ME design project courses (because I’ll start to work part-time at Architecture starting February 1st). I wonder whether I’ll miss it. Probably. But it also felt like a good moment to step away.

Weeknotes 2024 week 3

Published

  • Delta column ‘Free of Expression’
  • YouTube video ‘Why you SHOULD be critical when brainstorming’. Made this one in a single day. Morning spent setting up and recording, edited the whole thing in the afternoon. Mostly an excuse to try my new teleprompter. Next time I think I should spend more time polishing the script, and to be more conscious of the tempo/rhythm of the video. It goes from hasty to slow and back again a few times now.

Learned

  • Discovered BlackMagic’s Resolve lessons and followed ‘Introduction to Editing’. So many little things I would never have figured out on my own.

Notes / Other stuff

  • Submitted Comenius grant application this week. Reasonably happy with the project proposal, in the end. We’ll see.
  • Thursday was my very last afternoon coaching first year Mechanical Engineering students! I wonder whether I’ll miss it. And looking back I’m struck by how difficult students seem to be finding things this year. In previous years, we got much more in-depth with modelling (even Python scripts) than this or last year. Way more tinkering and way less engineering that I remember from earlier years. Then again, the end-results were always dodgy so perhaps I’m misremembering… In any case, I find myself thinking I would change so much if I were the boss of these courses (do less better). Which means it’s good I won’t be doing it anymore, as I’m not in that position.
  • So yeah, I think I might try to do weeknotes weekly-ish for a while and see what they bring, inspired by Russel Davies‘ book ‘DO Interesting’ that I read but never did anything with before.

The Format of Design Project Reports Leads Students to Develop Misconceptions

This is an unstructured, thinking-out-loud exploration of pretty much the same point as I made in Design Reports vs. Design Papers.

I think there may be a fundamental problem in the way we ask students to produce design reports that is making it unclear what the lessons are, exactly.

Design reports in education serve two separate functions: presenting/justifying the design proposal and showing that the student(s) did the work. This combination leads students to develop misconceptions, I believe. Because the reason we want to see some things in their reports (evidence of their process) is that we want to check whether they did and learned from applying the methods we ask them to practice. And the reason for wanting other things/properties in their reports (a consistent and coherent argument with only the evidence relevant to that argument) is another one: because that is what is necessary for a convincing outcome.

We judge things like a morphological chart (especially in earlier projects) based on criteria relevant to how the student is developing their approach/process. But those criteria are not quite the same as the ones relevant for a judgment as to how convincing the overal result of final claims are.

Another way to phrase this difference might be the difference between efficiency and effectiveness. We want students to develop an efficient and effective process, but the value of this is instrumental. In the end, only the effectiveness counts when we’re judging design proposals.

This tension or difference also becomes apparent when we compare student design reports with published papers reporting the results of design work. In a paper or presentation to critical peers, it is not a relevant question whether you wasted time or not. The only thing that counts is the final design, what claims you make about it, and what evidence you have for those claims. Much is left out that we do ask students to show in their reports. And this is a difference in kind, not just a difference in level, depth, detail, or quality.

This difference also highlights the contrast between design as an academic discipline and design as professional practice. In industry, efficiency, risk management, effective use of time and resources are important. Satisficing strategies are often appropriate. Academic values are different. There, understanding, logical consistency, accuracy, and other goals are more important. Aiming at ‘complete’ exploration and mapping of options is more important in this context. And tolerance for leaving certain practical matters in the design for later and focussing on a core, novel working principle first is far higher.

Three Ways of Justifying Design Features

Yesterday, in a discussion with a student on how to structure their design report, I found myself constructing a little typology of three types of justification for design decisions, each with their own rhetorical structure and form of presentation.

First, a particular feature of a design can be selected from alternatives developed in parallel. We do this at the overall level with concepts, usually three of them. These alternatives do not follow from the other, but are developed independent of each other, they are explorations of different approaches, and each represent a different set of trade-offs. Sometimes, these are developed in a sequence, one after the other but they are sufficiently independent of each other that they could have been developed in parallel, as three alternative answers to the same design problem, and so that each option can be evaluated using the same set of criteria. You can also do this at the level of details. Alternative ways to construct the frame, for instance, or different options for a hub assembly. In a report, you’d present these options side-by-side, with an argument for why one of them is the better choice.

Second, design features or geometries can be the endpoint of a single-track, iterative exploration or evolution. In this case you also have a number of alternatives that were considered, but they are not equivalent, and could not have been developed independently, in parallel. Instead, they form a sequence, where an evaluation of the strengths and weaknesses of each iteration forms the argument for the next one. The criteria used to get from one step to the next might differ from the considerations that led to the step after that. In a report, you can present the main stages of such an evolution, arranged chronologically, together with an explanation of the dimensions, features, or phenomena that turned out to be the most relevent, and how they shaped (and justify) the final form and properties of the part or construction.

Third, design features can also be the outcome of calculations that determine their correct or optimal value. Such design decisions may also have gone through iterations, or have been considered next to alternatives, but that history is no longer relevant for arguing the final outcome. Such decisions (a gear ratio, the length of a lever, the thickness of a beam) are best and most clearly justified by presenting a mathematical model, or formula, incorporation particular assumptions, constraints, and safety margins, leading to a single correct or optimal value.

Deschooling Society

I’m reading Ivan Illich’s Deschooling Society. Illich argues against the institution of compulsory schooling. Quite persuasively.

Schooling teaches that learning is the result of instruction. And thus, Illich argues, the mere existence of schools creates an ever increasing demand for schools. Society and students come to believe that (important) learning only happens through professional instruction. Even after the age at which attending school is no longer legally obligatory, it continues to be what you need to do if you have a desire to learn. The process of education (according to a curriculum set by others) becomes the important thing, not what a person actually learns.

The “convivial” alternative is “an educational network or web for the autonomous assembly of resources under the personal control of each learner”. It should be made illegal for employers to ask about someone’s educational history. People should be judged only on their knowledge and ability, not on how that was acquired. The process of completing an educational programme does not actually guarantee that a student has learned what they were supposed to learn, precisely because the process is compulsory, predefined, and imposed from without.

The idea of students seeking out their own education is an attractive one. And not new, of course. This is how the schools of ancient Greece worked, how students came to sages like Confucius, or the master-apprentice model. I also agree with Illich that it is an interesting question how modern technological tools and cognitive science could support a contemporary version of that model.

In fact, this is a little bit how certain YouTube channels, newsletters, and other online communities work. They gather a community of people interested to learn about a topic, and can engage in activities that look a lot like coursework, but in a free and voluntary way. That model is hampered by its commercial underpinning, although funding through Patreon and other subscriptions is highly similar to the direct payment of professors by students in early European universities.

What Illich (and those educational YouTube communities) miss, however, is the fact that students do often need to do stuff of which they can only see the value after they’ve done it a few times. In my own field – design education – it really takes (being forced through) a few projects before you can effectively start to reflect on the process and the required skills to build expertise. It may be difficult to learn anything meaningful when you’re only there because you have to be, but that does not invalidated the concept of a predesigned course and teacher-directed instruction (?).

I find this idea of deschooling and putting learning under “the personal control of the learner” interesting. But education also sometimes requires committing to a programme or submitting to instruction for an extended period – voluntarily entering into a series of obligations. It is a good question, however, how long you can productively sustain such a suspension of disinterest – and when you can expect it from aspiring students in the first place.

Teaching as gift-giving

It’s easy – and common, I think – to think of education as the teacher giving a gift to their students. You have something valuable – knowledge, or a skill – and you give (something of) that to your students. Or you try to, in any case. You give back. Or you hand on what you yourself received when you were a student, or a novice.

This looks like a very positive view of education. And I don’t know whether I think that there is always something wrong with it. But it is not how I want to view and do education.

In Freire’s terms, education as gift-giving is humanitarian, not humanist. In terms of Sadie’s grandmother (in Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin), it’s charity, not friendship.

Even though the image of giving a gift is positive and full of good intentions, it is hierarchical. Authoritarian, even. It’s a variation on the ‘banking’ model of education.

Is education an act of friendship? Of love? Perhaps, in a general fellow-feeling-with-other-humans kind of way. We could say it’s an act of fellowship. Or solidarity? In any case, education should be a collaboration, a dialogue, an encounter. For this, it needs to not be hierarchical. Or as non-hierarchical as you can manage.

How do you teach other things than the climate crisis in a time of climate crisis?

In April, I’ll teach a new round of the course ‘Designing Medical Technology’, an introductory design course for students in the BSc programme ‘Clinical Technology’. I’ve been making an effort in past years to collect assignments focused on improving healthcare in the Global South and to emphasize the downsides of high-tech high-energy healthcare systems.

I still feel conflicted about the course as it is. The goal is to teach basic design skills and have students experience a design process from problem definition through exploration and development to validation. But should you even teach other things than the climate crisis, in a time of climate crisis? If so, how?

One way to say ‘Yes’ to that is when you teach a skill that is valuable or necessary even to deal with the crises we’re in. And designing seems to qualify for that.

OK, so if we accept that it’s perfectly sensible to still teach design in the current situation, how should we teach that now? Because it’s certainly not a given that students are going to use their design abilities to deal with the climate and social justice crises. In fact, I feel there is quite a big risk that the opposite will happen if they exercise those skills within the current extactive and destructive system.

Design problems that lend themselves to introductory courses often call for a new or improved product. When you focus on functionality, reliability, or usablitity, the results (if designed by first-time novice designers) are likely more resource and energy intensive instead of less. Electronics are added; extra materials are used; whole new product categories are invented.

So why not focus on sustainability instead? Wouldn’t that be an easy fix? I’m not so sure. This is where the conflict is, for me. Because function, construction, and use can be straightforwardly and realistically explored and experimented with by naïeve designers (through sketching and modelling). And that experimental, explorative way of thinking (by doing and making) is at the core of learning to design.

I’m afraid that the (abstract) systems thinking required for going after more sustainable systems is both too complex and too difficult to make accessible in an understaffed, too short, introductory course. At the very least, my own toolkit of exercises, prompts, and instructions has developed around experimenting with the more basic industrial design domains. And therefore those do not lend themselves well for a systems-critical approach. Industrial design starts from the assumption that the ‘solution’ to ‘the problem’ is going to be an industrially produced, commercial product. Something that a company can market and turn a profit on. Something that does something new, or outperforms current products. Industrial design tends to lead to more.

But what we need, of course, is to start doing less. Use less energy. Waste less. Dump less. Rely on complex supply chains and cheap exploitative labor less. Fewer electronics. Fewer products. Coming to terms with the fact that our dreams about improved medical devices are often actually impossible as soon as we admit the rest of the world, and the future of our world, into the system boundary of ‘the problem’.

Ungrading with friends

Or: What to do in a complex, massive enrollment undergraduate project course that you have limited control over after you’ve seen the light?

My recent experience with ungrading left me convinced. It led to such great discussions and – above all – a trusting, and more safe environment for me and my students (an experience that I wrote about here). I never want to go back to grading. This september, however, I have to go back. The course where I took an ungrading approach is relatively small (90 students), and I teach it pretty much by myself. In other words, I have complete freedom to teach it as I see fit. At the start of the new academic year, I teach a truly massive course together with a dozen or so colleagues. I am not the responsible (lead) teacher for the course. The teacher who is, is not on board for ungrading. And even grading less or differently is difficult, because the teaching team is not fully known yet and I cannot organize – for instance – a series of meetings or workshops where we discuss our assessment practice.

We’ve just started preparing for the next year, so I want to try to get some of my thoughts, questions, and ideas down on paper (in pixels?).

About the course

The course in question is WB1641 ‘Mechanical Engineering Design Project’. It runs in the first quarter of the first year of the BSc programme Mechanical Engineering.

The course has two parts. The first is a lecture series with weekly exercises and a final exam about basic mechanical principles and analysis methods. The second part consists of a design project and a bunch of not-always-too-connected ‘individual skills’ like sketching exercises, an intro to Python coding, workshop safety, etc.

The design project is ‘my’ part. I set the assigment(s) and give weekly lectures on design methods and giving feedback on intermediate work. Those lectures are for all 800+ students together. Students do the project in 128 teams of 6 or 7. Each ‘cluster’ of 8 groups has a ‘project coach’ that they meet with every week. Some coaches have multiple clusters, so there’s roughly a dozen of them. There is large variation in their backgrounds, age, teaching experience, and in how well I know them.

In terms of the topic/content: students design little mechanical contraptions (a chain of marble machines). The project is usually quite fun. This video from the final day livestream gives a sense of the topic and complexity. (All lectures from 2021/2022 are on my YouTube channel, but they’re in Dutch.)

Asssment: Current situation

The group project is graded by the project coaches. I never introduced detailed rubrics for this, but I did make an assessment/feedback form that asks for the evaluation of 7 broad aspects/criteria on a 4-point scale (not present/insufficient, weak, good, excellent). The final grade is not calculated from these, but is given holistically. Or rather, that’s the idea. In previous years, a number of the project couches really wanted to grade on a more precise scale (asking for the return of a 5-point scale with a mid-point and/or checking the form inbetween two boxes), and/or making excel sheets that turn it into a calculation nonetheless.

A table with criteria to the left, 4 columns for insufficient/absent, weak, good, and excellent, and a column for remarks/feedback. At the bottom there is a single box for an overall grade.
Assessment form for the design project

Goals for next year

So, given all this, I don’t think I can be too ambitious. But I also don’t want to do nothing. I think I have three goals. I’d like to:

  1. Build in a bit more student agency and (at the very least) reflection for all students.
  2. Organize what freedom I can for myself to do more and more radical things in my section without too much discord with the others (and without too much extra work, I’m already swamped with work in the first quarter).
  3. Come up with a structure/way of working/assessment instructions that can support/will stimulate discussion amongst the teaching team about assessment and grading during the course. If I can’t practice ungrading as much as I’d want to, at least I can try to get people thinking.

Reflections of a first-time ungrader

I just ran my first ungraded course. Obviously, I did it all wrong. Nonetheless, it was wonderful. I’ve never had better conversations with students about what they take away from a course.

In this post, I want to write down some first rough thoughts before the experience fades, also in preparation for the Ungrading as Emancipation track at MYFest.

About the course

KT2700 ‘Designing Medical Technology’ is an introductory design course in the 2nd year of the ‘Clinical Technology’ BSc programme at Delft University of Technology. It’s a group project, with 90 students in 18 groups of 5. There’s clinical and technical experts that come by for one-time meetings with each group, but I mostly teach this course by myself. The course runs for 7 weeks, with three 4-hour afternoons of class time each week.

Students choose a problem to work on, do problem analysis, develop concepts, practice drawing, and conduct a simple empirical validation of their design.

My approach to ungrading

As this was my first real course-wide ungrading attempt, I wanted to play it semi-safe and use a common, tried and tested model. In essence, this meant I replaced my assessment of the final report with a self-assessment by each group, that we discussed in a final 20 minute meeting. I did what Jesse Stommel does and told my students that although I reserve the right to change a grade, I fully intended to follow their proposals. (And in the end, I really did not change anyone’s self-determined grade.)

This course already contained a number of rounds of peer feedback on intermediate results. This year I added a kick-off workshop in which I asked students what they hoped to get out of the course, and what their personal challenges and goals were. I also added a (very light-weight and informal) mid-course reflection that asked students to think about how the course was going and what they were learning.

So, what did I learn?

1. First, nothing changed

I expected revolution from day one. I was super excited to blow my students’ minds in the introductory lecture with how they’d determine their own grades. Education as the practice of freedom! Equity! Rainbows and unicorns!

Yeah, not so much. The first few weeks, nothing changed. The course felt very similar to previous years. At week 5 or so, I was feeling decidedly disappointed.

Looking back, my expectations were pretty naïeve, of course. These students are so used to just doing what they’re told, getting the grade, and moving on. It was foolish to think that students can go from paint-by-numbers to full autonomy in a few short weeks.

I also realized that teacher-assigned grades are not the only thing that determines the overall character of a course. (No shit, Sherlock.) The structure I set up, with frequent peer-review assignments and deadlines, including review questions that gave the impression that there were ‘right’ or ‘correct’ ways of doing things, worked against a culture of freedom, exploration, and self-direction. I also have a lot to learn in how I respond to student questions that assume or look for a single right answer. I’m not yet so good at responding to those in a way that ignores or challenges those assumptions.

In other words, students just did the work as usual. And they still asked a lot of questions in the category ‘How do you want us to do it?’ instead of feeling free to decide for themselves. They asked to be told what to do and they did what they were told. Just like in previous years.

2. Then, everything changed

Ungrading really is transformative. Even though the students were behaving as usual in those first weeks of the course, for me my relationship with them felt completely different from the get-go. And the way the course ended transformed the experience for students as well.

Throughout the course, I felt a lot better about the way I was coaching students. Previously, I was pushing them to take risks, knowing that it might lead to output that I had to rank lower than it could have been ranked if they took the safe approach. That felt massively unfair. As betrayal, almost. And that feeling was completely gone. Even if the students were still a bit too focused on doing it right, and according to my standards, I was much more free and able to direct their attention to what they were actually learning, and to prioritize learning experiences over polished results in how I advised them.

And then I sat down to read the group self-assessments and individual reflections. Oh, boy. So much good stuff. Responding to a very general prompt, many students pointed out the exact things I would list as the core lessons for this course. And many students described learning experiences that started with apprehension, worry, or expecting not to be interested, moving through initial surprises and a growing sense of skill and comprehension, to outright enthusiasm, pride, and consequences for how they saw their future after graduating from the programme.

And then, I sat down to talk with them about all this. Those two afternoons of final meetings were some of the best conversations I’ve ever had with students. A lot of students mentioned how they rarely think about what they’d learned so effectively either. “Usually, when you hand in a report you hope to never have to see it again,” one student said. “But the way we discussed our end-results with other groups this time, and looking back at what we delivered together, was really fun and valuable.”

I’ve had final meetings after the end of a project in other courses. But those were mostly me explaining and justifying a grade. Those did not feel like conversations between equals. But these were real, open discussions.

3. Self-grading is still grading

In those final meetings I had with student groups about their reflection and self-assessment, many students explained their grade in comparative terms. They were still ranking. A few also asked me – after I’d written down their self-determined grade – whether it matched what I would have given them. They wanted to know wheter the grade was ‘correct’. I told them I have no clue what a ‘correct’ grade is anymore.

Most students didn”t really know how to translate their assessment of what they’d learned and accomplished (or not) into a single number. At the time, I felt a bit guilty for not giving them more guidance. Later I realized that this was the whole reason for not wanting to grade them in the first place: it’s impossible to distill a multidimensional set of qualitative judgements into one unidimensional number.

I don’t know yet whether my conclusion from this is that next time I want to go with a pass/fail model, or that I should accept this cognitive dissonance because the attempt to come up with the ‘right’ grade does seem to serve as an effective means of stimulating evaluative reflection and in-depth comparison with others’ work.

Side-note: the Dunning-Kruger effect

In general the level of grades was half a point higher than in previous years (the average rose from 7.8 to 8.3). That effect was less strong than I’d expected. And in any case, I really don’t care. The students did great work and learned lots. Who cares about the numbers?

But I also got the strong sense that there had been a reversal at the top end. Many of the self-given 8/10s would have been 9/10s if I had done the grading, and most 9/10s would have been 8/10s.

The Dunning–Kruger effect is a cognitive bias whereby people with low ability at a task overestimate their ability. Some researchers also include in their definition the opposite effect for high performers: their tendency to underestimate their skills.

Wikipedia

So yeah. That seems to be relevant for self-grading . At least in the context of projects, where it’s not a question of self-assessing correctness on something like exam questions, but an overall, holistic judgement of open-ended work.

4. Self assessment in groups can be problematic

One of the bigger ‘Oh, crap!’-moments in this course was when I realized that I was putting students in a difficult situation with self-grading as a group. I don’t think I ever read anything ungrading in group projects, where students have to come up with a grade proposal together as a team. In groups that didn’t get along so well, this led to issues. And even in groups that worked well together, it could lead to friction.

In one case, this led (almost) to a tragic situation with a student who’d had some issues with the group work because of his autism, but also learned a ton by working through those issues (and taught me something in the process, which I wrote about elsewhere), that still wanted to grade himself lower than the other group members because he had ‘contributed less’. Luckily I didn’t need to step in. Another member of his team would’t let him be that unfair to himself. (Students are awesome.)

5. It’s the reflection, stupid!

Earlier, I wrote:

We want our students to learn to evaluate their own work critically. The best way to do this is by practising evaluating their own work. But it is difficult to have students do this honestly when it is ultimately our evaluation that counts.

Me, in a column on ungrading for the Delta, TU Delft’s ‘journalistic platform’, informed by the work of D Royce Sadler.

When we grade our students, we take one of the most important acts of learning away from our students by doing it for them: practicing judgement. Even in its most basic, mistake-laden form (i.e. what I did in this course), ungrading makes students actually responsible for their own learning by putting the final say in their hands.

If, whatever happens during a course, in the end it’s our decisions that count, then what reason do students have to really think about the quality of their work? Why reflect seriously if the outcome of that reflection doesn’t matter, and it’s someone else’s judgement and only that other’s judgement that matters? If someone else is going to tell you what the conclusion of your own reflection should have been?

Ungrading breaks that power structure. It breaks that adversarial relationship. It emancipates. Education as the practice of freedom. Rainbows and unicorns, after all.