Bob van Vliet

Simple Overhead Draw-and-Talk Videos Are a Good Idea

Fiorella & Mayer (2016) conducted a series of experiments that show the effect of seeing diagrams being drawn vs. showing and/or pointing at already-drawn static diagrams in (short) video lectures. The paper appears to be a summary of a PhD project.

Seeing a diagram being drawn improves learning compared with instruction that uses a static, complete diagram, even if the instructor points at parts of it during their explanation. This is probably because the combination of drawing and talking naturally applies the multimedia learning principles of signalling, temporal contiguity, and segmenting.

Digital Khan-style videos where you see the lines appearing without the instructor’s hand were less effective than real life videos where you actually see the instructor that’s doing drawing. Seeing only the instructor’s hand seems to be slightly better than seeing their (upper) body.

From the conclusion:

Overall, this research suggests that observing the instructor draw diagrams promotes learning in part because it takes advantage of basic principles of multimedia learning, and that the presence of the instructor’s hand during drawing may provide an important social cue that motivates learners to make sense of the material.

In other words: making simple overhead draw-and-talk videos is a good idea.

—

Fiorella, L., & Mayer, R. E. (2016). Effects of observing the instructor draw diagrams on learning from multimedia messages. Journal of Educational Psychology, 108(4), 528.

https://sci-hub.se/https://doi.org/10.1037/edu0000065

Design Reports vs. Design Papers

One of the things I find difficult in design education is the difference between teaching our students the skill of doing design – coming up with and developing products, machines, and other plans – and teaching them the logic of how to argue for the results of that work – presenting, justifying, and giving reasons for their proposals.

We teach our students (some version of) the design process, and then we ask them to write a report that presents that process and their design. There is a tension in that combination. In this set-up it seems logical to show how your process ‘led to’ your design. Showing your (cleaned up, idealized) process is treated as the justification or support for the final design. But the quality of your process is not necessarily evidence for the quality of your design. Vice versa, with this approach it doesn’t make sense to present all your discarded ideas and other dead ends, or to show all seven and a half earlier versions of what became the final design. That would create a report that’s just as messy and chaotic as the average design process.

A ‘design report’ in this fashion tries to serve two functions: to provide evidence of learning activities, and to provide evidence for the final design’s quality. Those two sometimes conflict. At the very least they’re not the same and trying to do both in one document compromises the effect of both.

Perhaps, therefore, it would be good to make an explicit distinction between a ‘report’ and a ‘paper’? A report reports – it tells your teachers what happened. A paper presents – it describes a problem, shows evidence, and argues a proposal to a audience of peers.

If you want to see whether undergraduate students are learning the right skills and methods, ask them for a report. If you want graduate students to produce something similar to an academic paper, leave the reporting out of it.

Designing Effective Multimedia for Physics Education

Muller, D. A. (2008). Designing effective multimedia for physics education. Sydney: University of Sydney.

https://www.sydney.edu.au/science/physics/pdfs/research/super/PhD(Muller).pdf

–

Derek Muller (creator of ‘Veritasium’ on YouTube), in his PhD thesis, shows that science education videos need to start with students’ misconceptions to be effective. A straightforward exposition can be worse than no instruction at all, because students do not change their mistaken views but do become more confident that they know how something works.

Effective Educational Videos: Principles and Guidelines for Maximizing Student Learning from Video Content

Brame, C. J. (2016). Effective educational videos: Principles and guidelines for maximizing student learning from video content. CBE—Life Sciences Education, 15(4), es6.

https://doi.org/10.1187/cbe.16-03-0125

–

A clear, short, and highly readable review paper that discusses research on educational video and provides practical advice on optimizing three aspects of using video in education effectively:

Managing cognitive load:
- Use signaling to highlight important information.
- Use segmenting to chunk information.
- Use weeding to eliminate extraneous information.
- Match modality by using auditory and visual channels to convey complementary information.
Maximizing student engagement:
- Keep each video brief.
- Use conversational language.
- Speak relatively quickly and with enthusiasm.
- Create and/or package videos to emphasize relevance to the course in which they are used.
Promoting active learning:
- Package video with interactive questions.
- Use interactive features that give students control.
- Use guiding questions.
- Make video part of a larger homework assignment.

Or, more concisely (quote from the conclusion):

Keep videos brief and targeted on learning goals.

Use audio and visual elements to convey appropriate parts of an explanation; consider how to make these elements complementary rather than redundant.

Use signaling to highlight important ideas or concepts.

Use a conversational, enthusiastic style to enhance engagement.

Embed videos in a context of active learning by using guiding questions, interactive elements, or associated homework assignments.

Understanding in-video dropouts and interaction peaks in online lecture videos

Kim, J., Guo, P. J., Seaton, D. T., Mitros, P., Gajos, K. Z., & Miller, R. C. (2014, March). Understanding in-video dropouts and interaction peaks inonline lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 31-40).

https://doi.org/10.1145/2556325.2566237

–

Empirical study of drop-out rates and viewing statistics in a large number of edX course videos.

Peaks in viewing numbers – indicating rewinding and re-watching – occur around visual transitions. Students go back to a slide that’s suddenly gone and use the transitions as visual ‘bookmarks’ to re-watch or rewind to a particular explanation or section of the video.

Therefore, avoid taking away diagrams and other visual aides too soon and/or abruptly. Include clear visual anchors (e.g. title cards) at the start of sections, provide timestamps, or cut up longer videos into shorter ones.

Recording Quick Feedback Videos

As an alternative to written feedback, I make simple videos. Once you have it set up, this actually takes less time than responding with text. It’s more fun, and research suggests it’s also more effective:

Students found screencast technologies to be helpful to their learning and their interpretation of positive affect from their teachers by facilitating personal connections, creating transparency about the teacher’s evaluative process and identity, revealing the teacher’s feelings, providing visual affirmation, and establishing a conversational tone.

Cost-Benefit Considerations in Design Arguments

In architecture, it may be perfectly acceptable to present predictions that are based purely on theoretical ideas about how people will behave and feel in response to a proposed building. Human behavior is so complex, and buildings so large, that such claims can be utterly impractical to test or otherwise validate. We have little choice but to trust the architect’s expertise, or accept an argument by analogy.

An engineer presenting a design for a novel surgical device, however, is expected to present a prototype that has been tested on simulated or even actual tissues, in addition to a theoretical model that predicts and explains its behavior. The physics of metal devices have been reliably modeled, and it is perfectly feasible to produce one-off prototypes and set up empirical experiments to validate these predictions with a reasonable investment of resources.

In design disicplines, we expect or do not expect certain types of evidence based on the possiblity and cost of supplying them. Engineering arguments are subject to cost/benefit considerations, similar to the designs themselves that the arguments are about.

Diminishing Validity of Concept Selection as an Argument down the Line

The detailed development, implementation, and operation of a design usually represents a significant investment. This makes it a good idea to first explore a number of possible approaches before committing to a single concept.

But concept selection is a strategic choice. The decision comes down to a judgement of which concept looks most promising, not to a determination of which one is certain to have the best possible performance. And at the end of a completed design project, you can never be certain that a choice to go with a different concept would not, in fact, have led to a better outcome. It is just that at the time, this concept looked best, and that therefore it was the one selected for further investment of development resources. Who knows what would have happened if those same resources had been invested into a different concept?

Soft Spots in Design Arguments

A design is always presented as a means to achieve a goal of some kind, in a certain situation or context. To argue that the proposed design will actually do this requires a bit of a detour, however. First of all, goals are usually complex, ambiguous, and ill-defined. They need to be made operational in a set of objectively testable criteria (functional requirements, performance criteria, and constraints). Secondly, it is not obvious from the plans for an artefact how that thing will do its work, precisely. Its behavior needs to be predicted. Predicted behavior can be evaluated in terms of the operational criteria. This is the claim that designers can actually establish. It serves as a proxy for the actual motivation behind the design, the expectation that the design will actually achieve its goals in the real world.

The translation of a complex goal into an unambiguous, operational set of criteria is not straightforward. Different people can legitimately interpret the same goal differently. The argument for a design proposal needs to establish, therefore, that this translation is a good one. Does it capture all the relevant aspects? Is anything lost in the definitions and quantifications employed? Is it possible to formally meet these criteria, while clearly failing to achieve the actual goal?

Predicting the behavior and performance of the proposed system can look like the straightforward, rational, objective part of a design project. But this is not straightforward either. To predict something’s behavior, we need to model it. Models are always simplified, partial and idealized representations. Abstract models can be validated through controlled tests with a prototype, but tests also only pick out parts of the actual operation of a system, and prototypes are, like abstract models, partial, idealized representations. In fact, they often introduce properties that the actually proposed design would not have. Here as well, the argument relies heavily on judgements of definition, translation, and interpretation.

Discovery and Justification in Design Proposals

What is the logic of design proposals? What argument is or needs to be made when you present a design? What is it that a design proposal does and what criteria must it meet to perform this function?

Engineering can be contrasted with science in that it is not only descriptive, but also prescriptive. The goal of a scientific paper is to describe and explain the world as it is. An engineer’s design prescribes or at least proposes what should be done or changed in the world: ‘if you have a certain goal, then here is a plan to achieve it’.

This makes a design proposal, in rhetorical terms, an argument about policy. Much of it may be concerned with facts and causation, in the end it is a question of means, ends, and value. Such an argument is always relative. The proposal can be compared to existing options, alternative proposals, and to leaving the situation unchanged. And while scientific claims aim at universality, designs are always context-dependent, appropriate to a specific time and place.

If this is the argument we need to make, how do we argue it?