\n
Dr. Gary Munnelly<\/strong> Technical Lead, Beyond 2022 Trinity College Dublin, Ireland<\/p>\n<\/div><\/div>\n\n\n\nGary is a research engineer working on Beyond 2022. He completed his PhD in Computer Science in 2019 under the supervision of Prof. S\u00e9amus Lawless. His thesis focused on entity linking for text-based cultural heritage collections. He is primarily interested in modelling and structuring cultural heritage information so that computers can better support humanities researchers at all levels of expertise. This includes research into linked data, and information extraction. Within Beyond 2022 he is currently focused on developing tools and methods to support historians in reconstructing the archive. In addition to his work on Beyond 2022, Gary is also involved in Provenance, an EU funded project which aims to identify and label instances of fake news online.<\/p>\n\n\n\n
\n\n\n\nJoint<\/strong> Keynote #2:<\/strong> “Multimodality and Hypertext: Theoretical and empirical considerations”<\/em><\/p>\n\n\n\nAbstract:<\/strong><\/p>\n\n\n\nOne important contribution commonly ascribed to hypertext is the ability to combine different forms of expression, and so be considered ‘multimodal’ (or, at least, ‘multimedial’). On closer analysis, however, theorizing just what this entails has remained limited. Similarly to the situation that long held concerning ‘written’ texts, it is too easily assumed that different modalities, sometimes labelled with terms such as ‘text’ or ‘image’, combine ‘naturally’ and so users should be able to follow such combinations with relative ease. Research on literacy, particularly with respect to contemporary media configurations, has shown this assumption to be false.<\/p>\n\n\n\n
Constructing coherent interpretations of combinations of modalities can be far from straightforward, even when supported by good interface design; with poor design, which from the perspective of displayed ‘documents’ is unfortunately rather common, finding intended interpretations can present significant challenges. Now, when translated to the even more complex medial environment of hypertext, these potential problems are magnified considerably. Moreover, traditional considerations of where the ‘boundaries’ of hypertext might lie are now being redrawn as hypertext and the increasingly ‘hyper’-connected medial world become increasingly permeable. The entire multimodal world of social media and participatory digital cultures might then be considered from a hypertext perspective, but research on hypertext itself lacks conceptual tools with the power necessary to engage with that world. Simple ‘extensions’ of traditional notions of hypertext are likely to prove insufficient for a full-blown account of multimodality.<\/p>\n\n\n\n
In this talk we address these concerns from the perspective of current developments in multimodality studies, where the starting point is communication as such, regardless of the expressive forms that are used for that communication and whether communication is mediated computationally, via interlinked artefacts and pathways, or by cross-linked practices of digital and non-digital use. In short, current medial practices demand that hypertext be seen not simply as, for example, a shift from page-based documents to video, but as a further computationally supported environment for the development and deployment of core multimodal theoretical constructs such as semiotic modes, media and genres. We introduce these concepts and show several practical examples of processing from ongoing projects with a variety of media.<\/p>\n\n\n\n
References:<\/em><\/strong><\/p>\n\n\n\nBateman, John A. (2017). Triangulating transmediality: a multimodal semiotic framework relating media, modes and genres. Discourse, Context & Media 20: 160-174.<\/em><\/p>\n\n\n\nBateman, John A. \/ Wildfeuer, Janina \/ Hiippala, Tuomo (2017). Multimodality \u2013 Foundations, Research and Analysis. A Problem-Oriented Introduction. Berlin: de Gruyter Mouton.<\/em><\/p>\n\n\n\nHiippala, Tuomo \/ Bateman, John A. (2021) Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora. Digital Scholarship in the Humanities. DOI: 10.1093\/llc\/fqab063<\/em><\/p>\n\n\n\nSpeakers:<\/strong><\/p>\n\n\n\n <\/figure>\n
Prof. Dr. John Bateman<\/strong> Professor for Applied Linguistics Bremen University, Germany<\/p>\n<\/div><\/div>\n\n\n\n<\/p>\n\n\n\n
Over the past 15 years Bateman’s research has focused increasingly on communication that combines diverse forms of expression, including verbal language, images, moving images, image sequences, sound and movement, in all cases applying and extending models of textuality generalised from linguistics and semiotics. 2017 saw the publication of a broad transdisciplinary textbook on the theory and practice of multimodality research as a field of study in its own right (with co-authors Janina Wildfeuer and Tuomo Hiippala). His current work includes expanding notions of semiotics to combine discourse and embodiment (in the context of Human-Robot-Interaction in the \u2018Everyday Activity Science and Engineering\u2019 SFB of the university), broadening notions of textuality and narrative for diverse media (for example in the university’s interdisciplinary and collaborative research platform \u2018World of Contradictions\u2019 Lab on multimodal narrative), applying principles of formal ontology to the modelling of semantics for language and other modalities, and pursuing more robust empirical connections between data and theory for diverse media. He studied linguistics and computer studies at Lancaster University and obtained his PhD at Edinburgh University in Artificial Intelligence within the \u2018Epistemics\u2019 postgraduate programme combining AI, linguistics, psychology, and philosophy. After working and researching on computational linguistics, multilingual and multimodal document generation, and formal ontology at the universities of Kyoto, Southern California, Saarbr\u00fccken, and Stirling and the Gesellschaft f\u00fcr Mathematik und Datenverarbeitung (GMD) in Darmstadt, he joined Bremen University in 1999.<\/p>\n\n\n\n
<\/figure>