Developing a framework to re-design writing assignment assessment for the era of Large Language Models (LLMs)

[Audio] Hi, my name is Amy Hsiao. I am going to tell you about our article: Developing a framework to re-design writing assignment assessment for the era of Large Language Models. Many teachers in higher education are concerned about whether students misuse these chatbots to ghostwrite their academic assignments and whether assessment can still effectively test what it intends to assess. This article aims to help teachers address these concerns..

[Audio] We were under high time pressure to solve a real educational challenge. Therefore, we adopted Van Aken and Berends' Design Science Research (DSR) approach. This means we developed a solution to the problem, that is the framework, and then we used cases to show teachers how to apply this framework in the workshops. We also explored the effectiveness of our framework..

[Audio] The framework consists of six design dimensions. The most important step for teachers to consider is the course context and learning goals..

[Audio] Although tools based on Large Language Models (LLMs) have the potential to support student learning, teachers should think about how using these tools in their courses aligns with the overall learning outcomes of their programs. For introductory courses, using these tools could potentially make it more challenging for students to acquire fundamental knowledge and skills..

[Audio] When including critical thinking in the course's learning goals, it's essential to consider the students' developmental stage in their critical thinking abilities. For example, first-year Bachelor's students are at the beginning stage of critical thinking and need to focus on acquiring information literacy skills. On the other hand, third-year Bachelor's students are more advanced critical thinkers and should be able to carry out complex problem-solving tasks. Once this distinction is made, it becomes more straightforward for teachers to create suitable writing assignments for assessing critical thinking..

[Audio] Once the course learning goals and writing assignment types are determined, the next step is to define appropriate grading criteria..

[Audio] These criteria must align with the targeted critical thinking skills. At the same time, they should encourage students to critically evaluate the weaknesses and limitations of current tools involving Large Language Models..

[Audio] To enhance teachers' professional development, we conducted three workshops to explain the design dimensions of the framework and address teachers' learning questions. We used cases that were relevant to the participants' respective disciplines to demonstrate how this framework can be applied to redesign their writing assignments. Our observations indicate that the participating teachers responded positively to this framework. In the future studies, we will assess the effectiveness of teachers' implementation of this framework in their self-redesign process..