Innovating Pedagogy 2024 part 4

39 Intelligent textbooks Challenges, barriers, limitations While intelligent textbooks are a reality, there is much work to be done to bring them into the educational landscape. A major challenge is their integration in the current teaching practice in ways that are reflective of possible biases and meet existing learning objectives and curricula. For example, students should become aware of concerns related to textbooks such as the limited number of languages available, constraining the development of intelligent textbooks in under-resourced languages, and biases in the human data used to train models behind textbooks. An example is AI bias toward the writing of non-native speakers (NNS) of English. Because intelligent textbooks often focus on having readers generate knowledge, ideas produced by NNS may receive differential feedback from that of native speakers. That feedback may be biased or less accurate and provide lower scores to NNS. Furthermore, student privacy is an issue if intelligent textbooks rely on industrial LLMs like ChatGPT, where student data is processed and shared by commercial entities. Another concern is the amount of data needed to train LLMs to provide accurate feedback to users. As an example, the summary feedback models in iTELL were trained with over 15,000 summaries from around 50 source texts that had been hand-scored by experts. The constructed response model in iTELL was trained with over 25,000 questions and answers from around 450 sources. More specialised intelligent textbooks will require specific datasets to train models, which may be cost-prohibitive. Context-specific adaptation may also prove difficult in some areas. For instance, current LLMs are not efficient at processing maths word problems (maths problems described in words) or interpreting figures, graphs, and tables. This makes adapting a mathematics textbook into an intelligent framework difficult. remembering improves when ideas are generated by a person’s mind rather than simply reading about them Another important issue is the cost of developing intelligent textbooks and their maintenance. These costs include for example, developing material (unless the material is open-source), constructing an interface, training AI models, and integrating content. The cost incurred can be substantial and will likely require support from commercial enterprises. This probably means that many intelligent textbook frameworks will be proprietary, making analysis of the systems difficult, including the reliability of their educational features and the validation of their AI interactions and learning potential. Conclusions The potential to make reading material interactive and enhance learning through the use of strategies known to increase text comprehension and skill development will lead many organisations to adopt intelligent textbooks. As the use of AI is changing how people work, intelligent textbooks that integrate AI will become important tools to support learning at school as well as learning beyond schooling. Learners will need to develop new skills – reskill or upskill – throughout their professional lives when they complete formal education. Intelligent textbooks can play a key role in making education and training more accessible, affordable, efficient, and adaptable..

40 Innovating Pedagogy 2024 References 1. A short description of the history of intelligent books in the AI magazine: Brusilovsky, P., Sosnovsky, S., and Thaker, K. (2022) ‘The Return of Intelligent Textbooks’, AI Magazine, 43, pp. 337–340. Available at: https://doi. org/10.1002/aaai.12061 (Accessed 27 June 2024). 2. A paper presenting the potential for interactivity in digital texts: Clinton-Lisell, V., Seipel, B., Gilpin, S., and Litzinger, C. (2021) ‘Interactive features of E-texts’ effects on learning: a systematic review and meta-analysis’, Interactive Learning Environments, pp. 1–16. 3. An overview of the effect COVID had on the adoption of digital texts from a consultancy firm: Seaman, J. E., and Seaman, J. (2020) ‘Digital Texts in the Time of COVID: Educational Resources in U.S. Higher Education’, Bay View Analytics. Available at: https://eric.ed.gov/?q=higher+education+chang es+in+the+u.s.&ff1=dtysince_2019&id=ED616838 (Accessed 27 June 2024). 4. A study introducing generation effects: Slamecka, N. J., and Graf, P. (1978) ‘The generation effect: Delineation of a phenomenon’, Journal of Experimental Psychology: Human Learning and Memory, 4(6), pp. 592–604. Available at: https://doi.org/10.1037/0278-7393.4.6.592 (Accessed 27 June 2024). 5. A study discussing generation effects in learning: Chen, O., Kalyuga, S., and Sweller, J. (2015) ‘The worked example effect, the generation effect, and element interactivity’, Journal of Educational Psychology, 107(3), pp. 689–704. Available at: https://doi.org/10.1037/edu0000018 (Accessed 27 June 2024). 6. A paper that compares generation to simple reading: McCurdy, M. P., Viechtbauer, W., Sklenar, A. M., Frankenstein, A. N., and Leshikar, E. D. (2020) ‘Theories of the generation effect and the impact of generation constraint: A meta-analytic review’, Psychonomic Bulletin & Review, 27(6), pp. 1139–1165. Available at: https://doi.org/10.3758/s13423-020- 01762-3 (Accessed 27 June 2024). 7. An article introducing iTELL and summarisation interactivity: Morris, W., Crossley, S., Holmes, L., Ou, Chaohua, Dascalu, M., and McNamara, D. (2024) ‘Formative Feedback on Student-Authored Summaries in Intelligent Textbooks using Large Language Models’, Journal of Artificial Intelligence in Education. Available at: https://doi.org/10.1007/s40593-024- 00395-0 (Accessed 27 June 2024). 8. A case study of using intelligent textbooks with university students in the US: Sun, Q., Norman, T. J., and Abdourazakou, Y. (2018) ‘Perceived value of interactive digital textbook and adaptive learning: Implications on student learning effectiveness’, Journal of Education for Business, 93(7), pp. 323–331. 9. A case study examining student perspectives using an intelligent textbook in the US: Morris, W., Choi, J., Holmes, L., Gupta, V., and Crossley, S. A. (in press). Automatic Question Generation and Constructed Response Scoring in Intelligent Texts. Proceedings of the 17th International Conference on Educational Data Mining (EDM). Atlanta, GA. 10. An experimental study comparing student achievements when using an intelligent textbook: Crossley, S. A., Choi, J. S., Morris, W., Holmes, L., Joyner, D. and Gupta, V. (in press). Using Intelligent Texts in A Computer Science Classroom: Findings from an iTELL Deployment. Proceedings of the 17th International Conference on Educational Data Mining (EDM). Atlanta, GA. Resources • A short video about iTELL: Intelligent Textbooks for Enhanced Lifelong Learning produced by the AI institute for adult learning and online education: iTELL Demo. Available at: https://www.youtube.com/ watch?v=YZXVQjSDZtI (Accessed 27 June 2024). • A short video of an example of how intelligent textbooks have been developed and used at the Rice University in Houston, Texas: Revolutionizing learning with free “intelligent” textbooks. Available at: https://www.youtube.com/ watch?v=XqTiS8VmYQg (Accessed 27 June 2024)..

41 Assessments through extended reality Assessments through extended reality Harnessing immersion to demonstrate and develop skills Introduction Simulation-based learning replicates aspects of the real world, requiring learners to take actions and making the consequences of these actions visible. This can allow practical skills to be developed and demonstrated without real-world constraints or risks. Simulation is already an important part of training in areas such as healthcare and aviation. Extended reality (XR) – encompassing techniques such as virtual reality (VR) and augmented reality (AR) – increases opportunities for immersion, which is understood as the combination of an increased sense of presence (the feeling of ‘being there’) and of agency (having control over our actions and being able to manipulate the environment). In diverse industries, companies such as Hilton (hospitality), DHL (logistics), Volkswagen (automobile manufacture) and Rolls Royce (aerospace) are making use of XR simulations as part of workplace training. As greater immersion enhances the sense of agency, immersive simulations are particularly suited to testing and gaining feedback on procedural knowledge – how to do a particular task well, such as flying a plane, responding to an emergency or performing surgery.1 Assessment is key to learning, and assessing procedural competences is a core goal of many learning activities. It is increasingly recognised that XR could fulfil the growing desire for authentic assessments that test what someone can do and how well they can do it. This offers a radical alternative to traditional written assignments2. It can address difficulties with other ways of assessing performance in the real world, such as a lack of repeatability or availability of a particular scenario on demand, or the need for a specific physical environment, equipment or people, which are all limited resources. Extended reality creates new forms of interactive activities which generate rich data on procedural knowledge and learners’ abilities.

42 Innovating Pedagogy 2024 Simulation provides important potential for assessment of competences where there may be serious consequences for real-world failure, or situations which are difficult to find or create for the purposes of assessment. In a simulation, formative and summative assessment can occur without these consequences, but emphasising the related learning points, testing skill levels and decision-making abilities. The sense of presence that comes from increased immersion can allow assessment of whether someone can perform in a facsimile of a real situation, where environmental or affective elements, such as navigating a complex physical space, time pressure, or managing interpersonal interactions can be made more realistic. For example, activities to assess and develop soft skills in hospitality can be undertaken in a realistic virtual simulation of the actual hotel environment, complete with guests that present challenging situations which need to be responded to by working with colleagues. Where is this useful? Examples of assessments through XR show reasons to use immersive approaches and provide insights into appropriate design. Medical teaching has been an early adopter of VR and AR, with demonstrations of this being used to assess proficiency and differentiate levels of skill in some procedures. Researchers have shown for example that VR can be used to distinguish novice and expert skill in neurosurgeons by capturing data on how they complete complex motor skills-based tasks in a virtual environment3. There has been substantial interest in VR-based training for health and safety. It is recognised, for example, that many construction workers are not able to identify and manage hazards, with potentially serious consequences. VR offers the opportunity to practise and assess these skills away from the dangers of a real-world construction site but with the benefits of feeling as if you are present in a real setting3. Simulations can capture data on whether a person can correctly identify hazards in a simulacrum (representation) of a workplace, such as a chemical laboratory or factory, and if the person can act appropriately to resolve or avoid these hazards. Other examples involve capturing and assessing data on the operation of complicated and potentially dangerous machinery, such as cranes or lathes4. A whole range of further vocational subjects, where a simulation of the work environment can be produced including relevant objects, actors and decision points, offer opportunities for students to be assessed on their ability to respond to situations. The Sheffield College in the UK chose Animal Care, Catering, and Carpentry and Joinery courses as subjects in which to trial VR-based assessments. In each case, a virtual resource was built based around an actual environment using 360 degree photos. For example, in the animal care resource, students virtually navigate around the room at the college that contains animal enclosures and a walk-in fridge area, interact with enclosures and other objects, and answer questions to test their knowledge5. assessing procedural competences is a core goal of many learning activities By reducing reliance on reading and writing, some assessments in XR can also be designed to be completed regardless of the language proficiency of the student. The Augmented Assessment project (see Resources) is exploring how these assessments can be more inclusive of migrants studying in schools when compared to assessments with written questions and answers. For example, the students’ mathematical or scientific understanding can be demonstrated through interaction with virtual objects. Designing assessments with XR While further research and development is needed, some initial principles for the design of assessment using XR could include: • Use multiple assessment methods and data sources to understand different dimensions of learning and performance: Many forms of analytical data can be identified, logged and analysed to assess skills and procedural knowledge through.

43 Assessments through extended reality activities in XR. Features such as the time taken, amount of work completed successfully, or adherence to modelled behaviours can be logged and analysed. Decisions made during the simulation, and how the person communicates with others, might also be captured and assessed. This can be combined with observations or review by assessors to provide richer understanding and reflection. External to the data collected through the simulation, quizzes can test understanding and post-simulation activities can encourage reflection and identify ways to improve. It can also be valuable to test knowledge at a later date, well after the initial simulation activity, to check for retention4. Through a combination of these data sources the assessment can offer rich understanding that can be hard to gather through other means. • Start with formative or low stakes assessments: Learners have expressed more positive views of VR in low stakes or formative assessment, where the novelty and challenge supports new forms of feedback but cannot impact negatively on their grades5. If the assessment is summative and carries weight for passing the course, then it will be particularly important to offer inductions and chances to develop experience in advance3. • Align the assessment purpose and measures: Research suggests that using existing VR-based games to assess competencies such as emotional intelligence or working at heights is problematic7. In any novel assessment it is important to think carefully about the alignment of the assessment tasks and measures with expected learning outcomes. Barriers and potential solutions for equitable assessment VR experiences using headsets and other hardware can create a strong sense of immersion, but they raise barriers for some people and pose risks for new inequities to emerge when using them in assessment: • People who are more susceptible to motion sickness and with less experience of VR are more likely to suffer from ‘VR sickness’, which could result in reduced performance or aversion to taking part. However, VR sickness occurs when motions in the virtual space do not match physical movement, such as moving over a large virtual distance by pressing a joystick direction6. Activities in smaller spaces where the user can move naturally are less likely to cause symptoms. • Some report other forms of discomfort such as eye strain or headaches from extended use of headsets1. Ensuring proper fitting, comfortable equipment, and limiting the length of time a person is expected to wear a headset is therefore recommended. • The costs of headsets have reduced substantially in recent years with models available for a similar price to a smartphones or laptop. But as an additional device, cost is still a potential barrier and only a small minority own their own headset. • A minimum clear space of 2 by 2 metres is suggested for each user. Limited space may reduce performance and create risks where people are unaware of their surroundings. Standard computer or mobile XR experiences also exist, so in some cases it may not be necessary to use specialist hardware to take part in an XR-based assessment. These could also be offered as an alternative assessment method where barriers do exist. Conclusions Assessments through XR can address the challenge of achieving more authentic assessments by simulating situations that are difficult to achieve reliably or safely in the real world. It is attractive to think that assessments can be based on detailed analytical data and allow in-depth analysis, reflection, and potentially repetition. There remain uncertainties about the extent to which headset-based VR will become mainstream and can support equitable assessment, but where VR is available for learning it will become increasingly important to think about how to embed assessment effectively..

44 Innovating Pedagogy 2024 References 1. A widely cited framework for immersive learning: Makransky, G., and Petersen, G. B. (2021) ‘The cognitive affective model of immersive learning (CAMIL): A theoretical research-based model of learning in immersive virtual reality’, Educational Psychology Review, 33(3), pp. 937–958. 2. A paper exploring frameworks for assessments in VR and AR, along with an evaluation of an example in laboratory safety which uses multiple types of data to assess students: Udeozor, C., Chan, P., Russo Abegão, F. and Glassey, J. (2023) ‘Game-based assessment framework for virtual reality, augmented reality and digital game-based learning’, International Journal of Educational Technology in Higher Education, 20(1), pp. 36. 3. A review of research on VR training across subject areas: Abich IV, J., Parker, J., Murphy, J.S. and Eudy, M. (2021) ‘A review of the evidence for training effectiveness with virtual reality technology’, Virtual Reality, 25(4), pp. 919–933. 4. A review of research on VR training in health and safety applications which identifies different ways in which learning was assessed: Toyoda, R., Russo-Abegão, F., and Glassey, J. (2022) ‘VR-based health and safety training in various high-risk engineering industries: a literature review’, International Journal of Educational Technology in Higher Education, 19(1), pp. 42. 5. Report on pilot studies of VR in summative assessments in vocational learning: The Sheffield College, NCFE. (2023) An investigation into the use of virtual reality on assessment. Available at: https://www.ncfe.org. uk/media/nq3afg0n/final-report-the-sheffield- college.pdf (Accessed 24 June 2024). 6. Paper reviewing research on the causes of VR sickness: Chang, E., Kim, H.T. and Yoo, B. (2020) ‘Virtual reality sickness: a review of causes and measurements’, International Journal of Human– Computer Interaction, 36(17), pp. 1658–1682. 7. Study exploring the use of VR games to assess competencies with mixed results, suggesting next steps for valid XR assessments: Sanchez, D.R., Weiner, E. and Van Zelderen, A. (2022) ‘Virtual reality assessments (VRAs): exploring the reliability and validity of evaluations in VR’, International Journal of Selection and Assessment, 30(1), pp. 103–125. Resources • Summary of current trends around virtual reality and the metaverse for training employees: What does virtual reality and the metaverse mean for training? Blog post by Scott Likens and Andrea Mower from PwC. Available at: https://www.pwc.com/us/en/tech- effect/emerging-tech/virtual-reality-study.html (Accessed 24 June 2024). • A list of examples of how companies use virtual reality in training, describing features such as how learning is assessed and feedback given: Available at: https://www.vrowl.io/the-22-best- examples-of-how-companies-use-virtual- reality-for-training/ (Accessed 24 June 2024). • Augmented Assessment: Resources from a European Union funded project exploring how augmented reality assessments could be beneficial for migrants: Available at: https://augmented-assessment.eu/ (Accessed 24 June 2024). • Drawing on a UK-based survey with responses from across 110 educational institutions, Jisc’s Extended reality in learning and teaching report 2023/24 describes the growth in applications of XR in tertiary education but also outlines current challenges experienced by educators: Available at: https://repository.jisc.ac.uk/9534/1/ extended-reality-in-learning-and-teaching- report-2023-24.pdf (Accessed 24 June 2024). • A set of standards and guidance from the Association for Simulated Practice in Healthcare (ASPiH), guided by evidence on simulation-based learning and emphasising the appropriate use of assessment within this: Available at: https://aspih.org.uk/standards-2/ (Accessed 24 June 2024)..

45 Immersive language and culture Immersive language and culture Using games to step back in time for authentic learning experiences Introduction Students often benefit from opportunities to develop and apply their learning in authentic settings. Language students can progress quickly when they are immersed in a setting where everyone is using their target language. Field trips can also support lessons about historical periods and diverse cultures. However, providing authentic experience of an ancient language or culture is challenging. Immersive language and culture provides game-based opportunities for extended investigation of and roleplay within historic scenarios. This can be done through video-game-based experiences or through structured sessions of live-action role play. Authentic learning Immersive language and culture draws on the strengths of authentic learning. This is an approach to learning-by-doing that prompts students to find solutions to complex problems in multi-disciplinary settings. Students who are immersed in an authentic learning activity can be supported to develop the judgment to distinguish between reliable and unreliable information, the ability to recognise patterns in unfamiliar contexts, the flexibility to develop innovative solutions and to work across cultural boundaries1. Some features of authentic learning include sustained investigations, the challenge of drawing on multiple sources and perspectives, and opportunities to seek out multiple interpretations and work towards a variety of outcomes. This can be achieved in real-world settings but, in the case of past languages and cultures, students can benefit from the use of digital immersive video game environments, or an approach known as Reacting to the Past. Video games and immersive learning As access to digital technology has become more widespread, virtual language learning has become increasingly popular. Propelled by the need to teach in online environments due to COVID lockdowns, virtual learning has become increasingly mainstream2. Language-learning software and apps, from Rosetta Stone and Duolingo to the software included with textbooks, have become an increasingly common part of learning a language. Many language-learning programs do not attempt to provide an authentic cultural environment, but function more like flashcards for the learner. The use of commercial entertainment industry video games to learn languages has provided an informal way (now sometimes formalised in language classrooms) of connecting sensory perceptions with unfamiliar languages, and these opportunities can be provided simply by changing the language in which the commercial game is installed. However, this technique is not possible for ‘dead’ and endangered languages such as Latin or medieval French. Furthermore, to achieve the goal of language learning within a cultural context, the immersive environment must mirror the world of the target language and culture. For example, a modern kitchen table in a US-based game set in the 21st century would not be an appropriate table to include in a medieval court environment. Likewise, the vocabulary used in an off-the-shelf video game like Assassin’s Creed applies only to a specific set of conquest and conflict situations, leaving many gaps for those who will be working with ancient and medieval literature – for example, vocabulary relating to rich religious experiences and interpersonal relationships..

46 Innovating Pedagogy 2024 Game scholars have uncovered and established the powerful links between learning and play3. Using off-the-shelf games can be effective, but these games usually lack linguistic and cultural input that would make them authentic learning environments. In other words, playing a popular game like Fortnite in English may teach some new vocabulary, but it offers little awareness of any anglophone culture. Much more valuable3 are instructor- (or student-) produced games as well as game play systems designed specifically to promote learning the target language and culture. These have been shown to be effective – for example, students who played a 3D-immersive game related to an essay topic subsequently crafted richer second- language narratives than those who did not use those environments4,5. Brendan’s Voyage is an example of an innovative use of video game design in support of learning and teaching dead languages and assessing the outcomes. The game, an adaptation of the medieval text The Voyage of Saint Brendan the Abbot teaches players to speak, read and write the Anglo-Norman dialect of medieval French. The entire experience is infused with architecture, art and literature appropriate to 12th-century Europe, meaning players also learn about the culture, politics and aesthetics of the period. For instance, a player is coached by a magical codex (an ancient manuscript text in book form) through the language skills needed to interact with a merchant. The codex can be consulted at any point to help the player converse in Anglo-Norman with the merchant. Using a game to provide learners with an immersive experience offers a variety of advantages. Game-based learning can help students to bypass the anxiety and lack of self-confidence that can be experienced when starting to learn a language. Any mistakes can be attributed to the in-game character rather than the student. The game also provides opportunities to communicate in medieval French, rather than being limited to the study of grammar and translation. Interacting in the game world emphasises how the language relates to the activity, context, and culture in which it was developed. The codex can be consulted at any point to help the player converse in Anglo-Norman with the merchant in the game “Brendan’s Voyage” by Lynn Ramey and Jacob Abel..

47 Immersive language and culture Immersive language and culture draws on the strengths of authentic learning A similar approach is applied in Operation LAPIS5, a two-year interactive adventure in which Latin learners develop skills in speaking, reading, and translating as they role-play as Romans. The game environment is used to foster engagement – it also provides a way of supporting a gradual shift from instructor-led learning to self-study, with students using the CODEX (a set of web- based language resources) as a resource to support their decisions about in-game interactions and how to proceed with their adventure. As they do this, the teacher can provide feedback and guidance on areas such as language use, problem-solving techniques and quality of background research. Unlike Brendan’s Voyage, Operation LAPIS is not set in a single game environment but uses a range of online tools and resources. However, an immersive language and culture approach can also be adopted without the use of technology, using a form of live-action role-playing game known as Reacting to the Past. Reacting to the Past Reacting to the Past6 consists of complex games in which students are assigned roles to explore an historic topic. Class sessions are run by learners, with educators advising and guiding them, as well as assessing both oral and written work. The intention is to draw students into the past, to encourage them to engage with big ideas, and to develop their intellectual and academic skills. For example, a class of students might take on the roles of countries’ representatives at the 1919 Paris Peace Conference, working to develop a treaty that establishes the terms of peace after the World War. This requires them to carry out research to establish what their countries’ visions and priorities are, and then negotiate with others in topical sub-committees to recommend a course of action of the leaders who will determine the final treaty. In another class, a group of students might take on roles in ancient Athens, in a time following military defeat and rebellion. The newly restored democracy is unstable, and students will debate topics such a democracy, oligarchy, imperialism, women’s rights and immigration – debates informed by the use of classical texts including Plato’s Republic. The Reacting Consortium currently has more than 75 games on offer, covering subjects including politics, religion, economics and STEM. Each of these comes with a student gamebook, which describes the historical context, explains the game premise, introduces the central debates and sets out the rules. An instructor manual includes handouts and gives instructions on running the game – how many students it is designed for and how long it will last. Role sheets are supplied for the students, so they can find out what their individual goals are and which strategies they can use. There are also companion texts, including readings from primary sources. Some role-playing games – such as Monumental Consequence (in which students consider whether art is ever worth dying for) – can be played in less than an hour and could provide an introduction to Reacting for both students and educators. Others can play out over weeks or months. Athens 403BCE, described above, is one of the larger scale games and can run for as many as 15 sessions with roles for up to 50 students. Versailles 1919, the Paris Peace Conference game, takes around eight sessions and can run with as few as ten students. Challenges, barriers, limitations A recent study7 looked at the experience of running a Reacting game three times with classes of undergraduates and graduates. The students found some aspects challenging, particularly giving oral presentations, and engaging in debates. They also found it difficult to understand and summarise research and to critique arguments. Overall, though, they found the experience valuable. They reported that it not only helped them to strengthen skills and knowledge but also to identify areas where they had previously been weak. They also felt it prompted them to learn the material better than they would have done in other circumstances, and they believed that they would retain it better because the experience was so memorable..

48 Innovating Pedagogy 2024 Almost a decade ago, Julie Sykes and Jonathon Reinhardt anticipated an explosion of purpose-built games for language learning in authentic contexts, noting that it is ‘critical that researchers and practitioners evaluate and, in many cases, participate in the creation of these games’8. However, immersive language- learning games created with researchers have not yet become mainstream or widely available. Perhaps due to the complexities of game production, studies have focused on games produced for entertainment (most commonly, multi-player role-playing games) that are co-opted into a classroom setting to be played in a target language. Conclusions Immersive and game-based learning methods provide a rich, engaging way for students to learn languages and understand historical and cultural contexts. By integrating digital game environments and live-action role-playing, these educational techniques allow students to dive deep into historical and cultural settings, enhancing their learning experience and retention. Games like Brendan’s Voyage and Operation LAPIS exemplify innovative approaches that not only teach language skills but also imbue students with a deeper appreciation of historical cultures and practices. Despite challenges such as the complexity of creating authentic game-based learning tools, the potential for immersive educational experiences is vast, promising a more dynamic and effective approach to learning languages and exploring cultures. As virtual reality and augmented reality technologies advance, immersive learning environments could become dramatically enhanced, offering an even deeper level of immersion and interaction. Immersive pedagogies are likely to become widely adopted across the curriculum..

49 Immersive language and culture References 1. A paper that explains and exemplifies authentic learning: Lombardi, M.M. and Oblinger, D.G. (2007) ‘Authentic learning for the 21st century: An overview’, Educause Learning Initiative, 1(2007), pp. 1–12. Available at: https://library.educause. edu/resources/2007/1/authentic-learning-for- the-21st-century-an-overview (Accessed 27 June 2024). 2. A web page setting out some consequences of the COVID-19 pandemic: Li, C. and Lalani, F. (2020) ‘The COVID-19 pandemic has changed education forever. This is how’, World Economic Forum, April 29, 2020. Available at: https://www.weforum.org/agenda/2020/04/ coronavirus-education-global-covid19-online- digital-learning/ (Accessed 27 June 2024). 3. A book about video games in education: Gee, J.P. (2003) What video games have to teach us about learning and literacy. Palgrave Macmillan, New York. 4. A journal article reporting using an interactive fiction game to teach German vocabulary, reading, and culture to university students: Neville, D.O., Shelton, B.E., and McInnis, B. (2009) ‘Cybertext redux: using digital game-based learning to teach L2 vocabulary, reading, and culture’, Computer Assisted Language Learning, 22(5), pp.409–424. 5. A magazine article describing the Operation LAPIS game and how it is used in teaching: Slota, S.T., Ballestrini, K., and Pearsall, M. (2013) ‘Learning through Operation LAPIS: A game-based approach to the language classroom’, The Language Educator, October, pp.36–38. 6. A journal article discussing the pedagogy of Reacting to the Past: Lamontagne, K. (2020) ‘When the Past is the Classroom: Merging ‘Reacting to the Past’ and Experiential Education’, Impact: The Journal of the Center for Interdisciplinary Teaching & Learning, 9(2), pp.9–17. 7. A journal article reflecting on the value of immersive, experiential active learning: Richardson, D.S., Bledsoe, R.S., and Manning, K. (2023) ‘Pushing Active Learning to the Extreme: Is It Worth It?’, College Teaching, 71(1), pp. 9–17. 8. A book chapter on digital games in language teaching and learning: Reinhardt, J., and Sykes, J.M. (2012) ‘Conceptualizing digital game-mediated L2 learning and pedagogy: Game-enhanced and game-based research and practice’, in Digital games in language learning and teaching (pp. 32–49). London: Palgrave Macmillan UK. Resources • A short video introduction to the Brendan’s Voyage video game, designed to teach Anglo- Norman language and culture: Brendan’s Voyage. Available at: https://www. youtube.com/watch?v=-3pVVCzQfP8&ab_ channel=LR (Accessed 27 June 2024). • Resources on Reacting to the Past: Reacting to the Past. Available at: https://reacting.barnard.edu/ – browse the games, sign up for an event (they tend to be in the US), join the community and/or sign up for the newsletter (Accessed 27 June 2024). Reacting Consortium. Resources from a consortium of institutions and instructors ‘united by the belief that imagination, inquiry, and engagement are essential features of teaching and learning’. Available at: https://reactingconsortium.org/ (Accessed 27 June 2024). • An article by Nicolas Trépanier about teaching history with video games, from the Newsmagazine of the American Historical Association: The Assassin’s Perspective: Teaching History with Video Games. May 1, 2014. Available at: https://www.historians.org/ perspectives-article/the-assassins-perspective- teaching-history-with-video-games-may-2014/ (Accessed 27 June 2024). • Barnard and Columbia undergraduates play Defining a Nation: India on the Eve of Independence, 1945 at Barnard College (5-minute video): Reacting to the past: The student perspective. Available at: https://www.youtube.com/watch?v=_ U6L9ERzw0U&ab_channel=ReactingtothePast (Accessed 27 June 2024). • Students—with costumes and trumpet in hand— assume roles in the Athenian Assembly to debate issues as Athens reconstructs after the Peloponnesian War (2-minute video). Reacting to the Past FYS142 Class Visit. Available at: https://www.youtube. com/watch?v=clO0uvCgwlM&ab_ channel=SmithCollege (Accessed 27 June 2024)..

50 Innovating Pedagogy 2024 Exploring scientific models from the inside Rich embodied experiences supported by extended reality and AI Introduction The value of hands-on activities in science, manipulatives in maths (concrete objects showing concepts), and embodied empathy in literature (experiencing a character’s world) clearly demonstrates how real- world experiences can enhance learning. An embodied learning approach seeks to expand the repertoire of resources students use to learn from their experiences in the world by incorporating how they move their bodies in space and how they interact with the physical environment around them. This approach is particularly powerful for understanding scientific models, allowing students to experience them first-hand ‘from the inside.’ Pedagogically, such learning is more robust when followed by reflective activities, helping to anchor abstract concepts in tangible experiences. However, the complexity of embodied exploration can hinder effective reflection. Teachers need useful digital data traces to help students focus on key moments of insight and difficulty, facilitating a deeper understanding of the phenomenon being modelled. Two emerging technical trends are set to enhance student learning of models in classrooms. The first is extended reality (XR) technologies, which include virtual reality (VR), augmented reality (AR), and mixed reality (MR). These technologies blend physical surroundings with digital elements, transforming interactions with technology and learning methods. The second trend is the significant growth in artificial intelligence (AI) capabilities, which can now process extremely large sets of data from multiple modalities. The merging of XR and AI holds promise for evolving and refining educational strategies and tools for model learning by increasing the responsiveness and adaptability of XR-based learning environments and supporting AI- enhanced interpretation of the full spectrum of human sensory information available in these environments. With guidance and support from teachers, such digitally enhanced experiences can help students move from direct experiences to conceptual understanding of scientific and other models that might otherwise remain ungrounded or misunderstood. Immersion in models through extended reality A common theme among XR environments is immersing students in a digitally enhanced experience that can be used as a springboard for more powerful understandings. VR strives for sensory immersion to help users suspend disbelief and engage in the digital world as if it were real. AR overlays digital information onto the physical world, enhancing real-world experiences with interactive and contextual digital elements. MR combines the physical and digital worlds, supporting interactive actions and sensory immersion in dynamic ways. • In MR, physical movements and interactions with real objects are mirrored by their digital counterparts in a simulated environment. For example, to learn about the complex relationships between bees and flowers in pollination, students can move their bodies around in the physical space of a classroom while seeing virtual bees moving where they move in a virtual garden.1 This integration is visually achieved by overlaying digital elements onto a live video feed that captures real-world phenomena. These virtual representations can be viewed on devices ranging from handheld smartphones to large-scale projections on walls and screens. these environments are novel because they provide playful first- hand explorations of scientific models.

51 Exploring scientific models from the inside • In more extensive MR setups, the system tracks user movement within a physical space to display digital information in the context of a scenario being enacted and to enable interactions with virtual entities positioned in specific real-world locations. For example, in the pollination scenario, students can see where different kinds of flowers are placed in specific locations in the virtual garden and move their bee to interact with them by moving their body in physical space. Importantly, the system is tracking and responsive to multiple bees’ actions at once, allowing students to experience the complex dynamics of the system and engage in teamwork strategies to optimise pollination and maximise the health of the garden. Room- sized space-based MR systems have been successful at enabling students to learn about various types of models from the inside, including biological systems such as animal foraging 2, physical science concepts such as states of matter3, and geological concepts related to celestial entities found in outer space4. Pedagogically, these environments are novel because they provide playful first- hand explorations of scientific models. For example, the STEP (Science through Technology Enhanced Play) project5 has students pretend to become parts of a scientific system they want to study. Digging deeper into the case of pollination, children learn by becoming bees and exploring a digital flower patch together. In one implementation, they enter a gymnasium and see a video feed of themselves walking around the room, but everywhere they go, a bee in the virtual space follows them (based on an array of sensors placed around the room). In certain locations around the room, there are virtual flowers that only appear when students are next to them. The students-as-bees collect nectar, which can vary in quantity and quality. The bees bring this nectar back to the hive and begin to discuss how best to organise to collect the most nectar before winter comes and it is all gone. Eventually, they discover that depending on their foraging activity, certain flowers get pollinated, or not, and therefore live or die. The most recent version of the STEP environment is programmable by students. They can choose to add new agents (e.g., birds as predators) or explore edge-case interactions (e.g., program the bees to all go to the ‘best’ flower or to distribute themselves to all the flowers). Moving back and forth between being an agent within a model and reflecting on/programming this model balances the ‘diving in’ and ‘stepping out’ that Edith Ackerman describes as powerful for constructing deep understanding of experiences6. Particularly powerful in this case is the way that taking the role of modeler allows students to tell a story about a place that is personally and culturally meaningful. For instance, students in urban areas may explore the effects on the flower-bee system when new building developments are created with and without green rooftops. Such a student- led approach to modelling is a powerful new form of ‘computational inquiry’ (often learning through simulations) that allows students to control, and see the effects of changing the ‘rules of the game’, rather than simply learning a constrained set of rules that have been pre-programmed into a system. Data visualisation then serves as a critical tool for teachers to recognise and interpret key moments of discovery and/or confusion among students and use these for reflective discussions that can lead to further experimentation with the model. AI-processing of data from multiple modalities Embodied learning data, which captures student interactions in a three-dimensional space and time, presents a complex challenge to represent and interpret due to its multimodal, dynamic nature. Unlike traditional educational data, which often focuses primarily on language or digital interactions, embodied learning data encompasses a wide range of both verbal and non-verbal information and movements through space. Strategic sensors placed in XR embodied learning environments collect vast amounts of data in multiple modalities, including streaming video, sound (commonly speech), positioning data, and system logs from the modelling environment..

52 Innovating Pedagogy 2024 This data represents aspects of learners’ movements, attention focus, emotional states, and digital interactions, which are essential for a comprehensive understanding of students’ learning and problem-solving activities. However, managing and analysing this diverse data is complex and requires sophisticated computational tools and approaches. Interaction Analysis (IA) is a key method used by educational researchers to extract insights from video data of embodied and other learning interactions7. IA examines human interactions in context, revealing the subtle dynamics necessary to understand collaborative, embodied learning environments. However, IA is labour- intensive and requires significant human effort. An interdisciplinary team of learning sciences and computer science researchers at Vanderbilt and Indiana universities has been exploiting recent advances in artificial intelligence (AI) and multimodal learning analytics (MMLA) to augment – but not supplant – human analysis and help support the generation of useful insights for students, teachers, and researchers. These technologies are being used to help provide a deeper understanding of learner engagement in embodied modelling activities, enabling teachers to gain a richer grasp of their students’ learning process and offer personalised feedback and support for reflection. For example, a timeline visualisation has been developed that integrates information across multiple data modalities to bring together information about what learners said (from the audio stream), where they were looking (from the video stream), what they were doing (from the system data logs), and how they were feeling (again based on information extracted from the video stream). This timeline can enhance teachers’ understanding of how students interact with each other and their environment over time, allowing them to facilitate better reflective discussions of the students’ embodied modelling experiences. To effectively analyse cases of embodied model learning, AI-based algorithms document the evolving states in XR environments, tracking and interpreting student engagement with scientific processes both individually and in groups. The classroom teacher actively guides students, helping them relate their activities to the modelled phenomena. The AI-driven timeline visualisation combines system log data on movements and interactions, gaze data to understand attention shifts, and video-derived emotional expressions. This comprehensive visualisation format provides clear, compelling insights for teachers and researchers, highlighting key moments of insight and difficulty. For example, the timeline may show a student’s period of confusion or expression of frustration when they did not get the attention of other students to help them complete a task, offering teachers valuable information to guide further interactions. The visualisation can also serve as a tool for reviewing, recognising and interpreting key moments of discovery among students, which can be linked to their evolving understanding of the scientific content being studied. Teachers use these insights to facilitate meaningful discussions, supporting reflection and advancing the students’ learning processes. Embodied Model Learning in the Classroom with Timeline supporting Interactivity Analyses. This includes elements such as transcription of each individuals’ contributions, gaze, and emotions..