Rationale
Our excursion through Bremen was intended to raise questions and prompt reflection around multimodality’s relationship with the digital, whilst making the case for its compatibility with work in the field of mobile learning, for instance as a way of investigating our urban surroundings. In the words that follow we briefly offer a rationale for the different components and and reference points for the activity that took place yesterday evening, the 20th September 2017.
Mobile learning in Bremen
This activity combined some of the principle interests of multimodality with theoretical and methodological work in mobile learning. Mobile learning is “the private and public processes of coming to know through exploration and conversation across multiple contexts, amongst people and interactive technologies” (Sharpes, 2007). It is this movement through multiple contexts that the mobility of mobile learning emerges. As Sharples et al suggest, “we learn across time, by revisiting knowledge that was gained earlier in a different context, and more broadly, through ideas and strategies gained in early years…we move from topic to topic, managing a range of personal learning projects, rather than following a single curriculum” (2007). In this definition, the mobility in mobile learning can be both material, in terms of learning artifacts (media, text, and other material elements emerging from the learning process) and cognitive as it involves knowledge sharing and practice sharing across contexts.
There is a symmetry between the evolution of the mobile phone and the growing critical interest in multimodality, both of which have a close relationship with technological innovation. While the mobile phone initially served the purpose of language-based communication, subsequent generations of smartphones have come to reflect our increasingly visually-mediated world. According to the visual culture theorist Nicholas Mirzoeff, the trillion-plus photos that are taken annually on mobile devices can be seen as a collective response to a world 'too enormous to see but vital to imagine' (2015: 12). Mirzoeff's position usefully points towards the complex relationship we enjoy with the digital: technologies help to shape - but are also shaped by - society. Venturing beyond the visual, the factory settings of the smartphone also enable us to make sound recordings (reflected in some of the data gathered during yesterday evening's excursion), to track our movement, to measure the distance we have travelled (and the corresponding number of calories burned) and to construct and consume other material far beyond the spoken conversation and the sms message. This in turn has opened new avenues for research according to the sonic studies scholar Jonathan Sterne (2012) who sees the audio recording potential of the 5.3 billion mobile devices in the world as presenting new opportunities for working critically with sound. In the moments before setting off on our excursion through Bremen we suggested that in the hands of the multimodal researcher the smartphone supports the consumption and communication of a varied range of digital data, even if the constitution of the images, sounds and other phenomena we record themselves benefit from critique.
Multimodality in the city
For this purpose of this activity we drew on multimodality’s interest in the full range of resources that have the potential to convey meaning, alongside the belief that the particular orchestration of these resources influences how meaning is conveyed or interpreted. And of course, the importance and interpretation we attach to the different resources is shaped by our own interests and histories. To apply these conceptual principles to the performance of an urban walk, as we explore the city the way that we make sense of our surroundings is influenced by the manner in which the aural, visual, sensory and other meaning-making phenomena that we encounter come together in concert and collision. In yesterday's activity the city was our text. As we explain below, in some respects our activity follows a similar methodological path to sensory walking activities around sound (see for instance Nilsen's acoustic flanerie in Berlin from 2014) and smell (including fascinating ongoing work by Kate McLean). Where we deviate from these and similar approaches however is that, rather than looking to foreground the meaning-making capacities of the aural, olfactory or other of the senses, with multimodality as our conceptual guide, we are open to a broad and shifting arrange of phenomena and how they come together in the moment. If we accept multimodality’s belief that the representation of meaning always depends on an orchestration of meaning-making material, then our interpretation of the city is shaped by what is simultaneously seen, heard, felt, tasted and so on.
Meaning through walking
Our excursion through Bremen can be seen in light of the growing critical and pedagogic interest in walking. While the practice of traversing the environment by foot is firmly established across a range of disciplines - take for example the exploratory outdoor pursuits of students in architecture, anthropology, environmental science and beyond - for the most part it has tended be seen as playing a functional, supporting role to the acquisition of knowledge. More recently however, work by the likes of Ingold (2004) and Edensor (2000) have proposed walking as being to central to the practice of meaning-making rather than merely a mode of transit between sites of data collection or study. At the same time, walking increasingly represents a complementary method to those with a primary interest in investigating for instance the aural (e.g Nilsen 2014), and olfactory (e.g. Springgay 2011) character of our urban environment. As a method our excursion drew on multimodality’s oopenness to the full range of meaning-making phenomena, combined with the unrehearsed yet inquisitive pursuit of the dérive.’
Dérive and disposition
As we devised and delivered earlier versions of this excursion undertaken in Bremen, we came to use ‘multimodal dérive as shorthand for the activity. The dérive was proposed by Guy Debord and the Situationist International (1967) as ‘A mode of experimental behaviour linked to the conditions of urban society: a technique of transient passage through varied ambiances.’ Our activity through Bremen shared with dérive the notion of walking-with-purpose yet without a predetermined path, where ‘one or more persons during a certain period drop their relations, their work and leisure activities and all other usual motives for movement and action, and let themselves be drawn to the attractions of the terrain and the encounters they find there.’ (Bridger, 2015, p 230). At the same time, the theoretical and methodological interest of our walk (explained above) lacked the political commitment of the excursions undertaken by the Debord and his Situationist colleagues. The performance of the dérive embodied the same spirit of inquiry associated with Walter Benjamin’s flâneur, itself a reconceptualisation of Baudelaire’s earlier literary depiction of the gentleman stroller, into an investigator of the urban experience. At the same time, through its association with leisure, flanerie felt out-of-step with our own excursion through Bremen, which was tied to the equivalent period of ‘conference presentation performed in the street’. In the absence of a previously agreed route, we followed a path through the city that was guided, at least partly, by embodied disposition, or what Bourdieu describes as habitus (1977), an evolving personality structure of the individual, a composite set of schemata, sensibilities, tastes, and dispositions. For the purposes of this activity, habitus was repurposed as disposition. In mobile learning, it is expressed as the ability, even expectation, of the individual to interact across contexts as disposition, rather than as explicitly purposeful. Individuals interact because they are disposed to, rather than always in response to pressing need or predefined purpose. As we made our way through Bremen, our pace and direction was shaped by the way that our of our own feelings, meaning schemes, intuitions and physical traits came together with the range of meaning-making phenomena we encountered.
Walking in a group
An important feature of our excursion through Bremen was it being undertaken within groups. As Ingold and Vergunst argue, ‘walking is a profoundly social activity: that in their timings, rhythms, and inflections, the feet respond as much as does the voice to the presence and activity of others’ (2008: 1). At different points in the exercise we were prompted (via Telegram) to share our gathered data and to discuss our experiences with other members of the group. This was intended to have the effect of demonstrating how we differently experience and interpret our urban surroundings within a single setting, drawing attention to the relational and multi-narrative character of the city (Hannerz, 2001: 206). With an attention to the conceptual interests of multimodality meanwhile, these moments of group reflection sought to highlight how we differently privilege material in our sense-making practices. More generally, in the absence of a predetermined route through the city, direction and pace required to be negotiated by each of the three groups, presenting us with impromptu sites of learning. Faced with a range of possible paths, we might find discord or harmony as we express our dislike or desire for a path that traverses the pedestrianised shopping precinct, the public park or the tree-lined pavement. These same insights would not be apparent by an excursion undertaken in isolation, even if at one point during our walk we were asked to resist the natural instinct towards discussing our experiences.
Continuing without conversation
Part-way through the excursion we were invited (via a Telegram prompt) to continue our walk without conversation and instead with the purpose of individually taking account of the full range of meaning-making phenomena that we would encounter. While this approach might seem to contradict the function of a group walk, this short activity has a precedent in the practice of sound walks of Drever (2013) amongst others, and as well as in ear cleaning games of Schafer (1977) in his influential work around soundscapes. The premise of this part of our excursion was that, in the absence of spoken conversation (although not silence) we would each be better able to focus on the broad range of meaning-carrying phenomena that help us to make sense of our surroundings. While Schafer and others working in acoustic ecology and similar fields might remove conversation as a way of sharpening their attention to the sonic character of their environment, reflecting our interest in multimodality, we instead used these moments without spoken dialogue as a way of drawing attention to the breadth of meaning-carrying material we encounter even across a short period of time. Therefore rather than foregrounding the aural realm, by walking without conversation we sought to emphasise the richly multimodal character of the street. Further, by inviting participants to share their experience after a few minutes our intention was to demonstrate how even with a single place and across a short period of time, our interest was differently drawn towards different meaning-carrying phenomena.
Your mobile data
While mobile digital technologies have opened up opportunities for working with data in ways that were previously the preserve of those with a reasonably high level of technical sophistication or technological ownership, a number of prompts during our Bremen excursion encouraged participants to reflect critically on the nature of the data they were able to collect. This included an invitation to discuss the ability of their mobile devices to record the full range of meaning-making phenomena they had encountered during the preceding part of the excursion. If the smartphone enables the gathering of aural, visual and various location-based representations of the city, as yet they are less well equipped to adequately gather and reproduce the smell of street food (sausages and onions to offer particular examples from last night's walk!), the coldness of air or the hardness of the pavements we encountered during an evening walk though Bremen's Old Town. In their discussion of the compatibility between multimodality and ethnography Dicks et al (2006) and Kress (2011) have pointed towards the varying ability of the researcher to satisfactorily collect or classify the broad range of meaning-carrying material that we experience, for instance as we move through the city. Whilst recognising that Powell (2010) argues for synaesthesia's ability to evoke different sensory experiences through image-based content, our excursion sought to consider (from the perspective of the multimodal researcher) how the smartphone varies in its ability to record and reproduce different forms of meaning-making phenomena.
Algorithmic culture and code
If smartphones are better equipped to take account of the visual and aural than other forms of meaning-carrying material, we should be wary of assuming they produce an exact or innocent record of what was experienced in the street. Mirzoeff unsettles our tendency to refer to 'digital photographs' by drawing attention to the sophisticated sensors that shape what is constructed and displayed on our screens, going as far as to suggest that these digital realisations are sufficiently distant from the traditions of photography that they instead need to be described as 'images'. In a similar vein the acoustician Cox (2014) is amongst those who point to the ways that aural representations of our surroundings are subject to the devices we use for recording and reproduction. What makes the workings of our smartphones significant is the ways that the data we collect is subject to complex code concealed inside our smartphones and possibly beyond our control and understanding. Considered from the perspective of Knox's critical work around algorithmic culture and posthumanism (2016), where digital activity is a complex assemblage of human, technological, commercial and other interests and influences, we might become less confident describing how we use our smartphones to 'capture' what is seen and heard. Perhaps this in turn discourages us from seeing the smartphone as a 'tool' that we use in our research, and instead as having some shared agency with the researcher. In light of multimodality's relationship with technology, combined with the increasingly digital nature of our lived world, there is value in challenging the dualistic binary between technicist and instrumentalist assumptions around technology (Bayne 2014) in favour of a more nuanced understanding of our complex and shifting relationship with the digital. The final Telegram prompt in our activity reflected this interest, as it asked participants to consider how their experience had been affected by the influence of algorithm and code. A critical departure point to end our multimodal excursion through Bremen.
Mobile learning in Bremen
This activity combined some of the principle interests of multimodality with theoretical and methodological work in mobile learning. Mobile learning is “the private and public processes of coming to know through exploration and conversation across multiple contexts, amongst people and interactive technologies” (Sharpes, 2007). It is this movement through multiple contexts that the mobility of mobile learning emerges. As Sharples et al suggest, “we learn across time, by revisiting knowledge that was gained earlier in a different context, and more broadly, through ideas and strategies gained in early years…we move from topic to topic, managing a range of personal learning projects, rather than following a single curriculum” (2007). In this definition, the mobility in mobile learning can be both material, in terms of learning artifacts (media, text, and other material elements emerging from the learning process) and cognitive as it involves knowledge sharing and practice sharing across contexts.
There is a symmetry between the evolution of the mobile phone and the growing critical interest in multimodality, both of which have a close relationship with technological innovation. While the mobile phone initially served the purpose of language-based communication, subsequent generations of smartphones have come to reflect our increasingly visually-mediated world. According to the visual culture theorist Nicholas Mirzoeff, the trillion-plus photos that are taken annually on mobile devices can be seen as a collective response to a world 'too enormous to see but vital to imagine' (2015: 12). Mirzoeff's position usefully points towards the complex relationship we enjoy with the digital: technologies help to shape - but are also shaped by - society. Venturing beyond the visual, the factory settings of the smartphone also enable us to make sound recordings (reflected in some of the data gathered during yesterday evening's excursion), to track our movement, to measure the distance we have travelled (and the corresponding number of calories burned) and to construct and consume other material far beyond the spoken conversation and the sms message. This in turn has opened new avenues for research according to the sonic studies scholar Jonathan Sterne (2012) who sees the audio recording potential of the 5.3 billion mobile devices in the world as presenting new opportunities for working critically with sound. In the moments before setting off on our excursion through Bremen we suggested that in the hands of the multimodal researcher the smartphone supports the consumption and communication of a varied range of digital data, even if the constitution of the images, sounds and other phenomena we record themselves benefit from critique.
Multimodality in the city
For this purpose of this activity we drew on multimodality’s interest in the full range of resources that have the potential to convey meaning, alongside the belief that the particular orchestration of these resources influences how meaning is conveyed or interpreted. And of course, the importance and interpretation we attach to the different resources is shaped by our own interests and histories. To apply these conceptual principles to the performance of an urban walk, as we explore the city the way that we make sense of our surroundings is influenced by the manner in which the aural, visual, sensory and other meaning-making phenomena that we encounter come together in concert and collision. In yesterday's activity the city was our text. As we explain below, in some respects our activity follows a similar methodological path to sensory walking activities around sound (see for instance Nilsen's acoustic flanerie in Berlin from 2014) and smell (including fascinating ongoing work by Kate McLean). Where we deviate from these and similar approaches however is that, rather than looking to foreground the meaning-making capacities of the aural, olfactory or other of the senses, with multimodality as our conceptual guide, we are open to a broad and shifting arrange of phenomena and how they come together in the moment. If we accept multimodality’s belief that the representation of meaning always depends on an orchestration of meaning-making material, then our interpretation of the city is shaped by what is simultaneously seen, heard, felt, tasted and so on.
Meaning through walking
Our excursion through Bremen can be seen in light of the growing critical and pedagogic interest in walking. While the practice of traversing the environment by foot is firmly established across a range of disciplines - take for example the exploratory outdoor pursuits of students in architecture, anthropology, environmental science and beyond - for the most part it has tended be seen as playing a functional, supporting role to the acquisition of knowledge. More recently however, work by the likes of Ingold (2004) and Edensor (2000) have proposed walking as being to central to the practice of meaning-making rather than merely a mode of transit between sites of data collection or study. At the same time, walking increasingly represents a complementary method to those with a primary interest in investigating for instance the aural (e.g Nilsen 2014), and olfactory (e.g. Springgay 2011) character of our urban environment. As a method our excursion drew on multimodality’s oopenness to the full range of meaning-making phenomena, combined with the unrehearsed yet inquisitive pursuit of the dérive.’
Dérive and disposition
As we devised and delivered earlier versions of this excursion undertaken in Bremen, we came to use ‘multimodal dérive as shorthand for the activity. The dérive was proposed by Guy Debord and the Situationist International (1967) as ‘A mode of experimental behaviour linked to the conditions of urban society: a technique of transient passage through varied ambiances.’ Our activity through Bremen shared with dérive the notion of walking-with-purpose yet without a predetermined path, where ‘one or more persons during a certain period drop their relations, their work and leisure activities and all other usual motives for movement and action, and let themselves be drawn to the attractions of the terrain and the encounters they find there.’ (Bridger, 2015, p 230). At the same time, the theoretical and methodological interest of our walk (explained above) lacked the political commitment of the excursions undertaken by the Debord and his Situationist colleagues. The performance of the dérive embodied the same spirit of inquiry associated with Walter Benjamin’s flâneur, itself a reconceptualisation of Baudelaire’s earlier literary depiction of the gentleman stroller, into an investigator of the urban experience. At the same time, through its association with leisure, flanerie felt out-of-step with our own excursion through Bremen, which was tied to the equivalent period of ‘conference presentation performed in the street’. In the absence of a previously agreed route, we followed a path through the city that was guided, at least partly, by embodied disposition, or what Bourdieu describes as habitus (1977), an evolving personality structure of the individual, a composite set of schemata, sensibilities, tastes, and dispositions. For the purposes of this activity, habitus was repurposed as disposition. In mobile learning, it is expressed as the ability, even expectation, of the individual to interact across contexts as disposition, rather than as explicitly purposeful. Individuals interact because they are disposed to, rather than always in response to pressing need or predefined purpose. As we made our way through Bremen, our pace and direction was shaped by the way that our of our own feelings, meaning schemes, intuitions and physical traits came together with the range of meaning-making phenomena we encountered.
Walking in a group
An important feature of our excursion through Bremen was it being undertaken within groups. As Ingold and Vergunst argue, ‘walking is a profoundly social activity: that in their timings, rhythms, and inflections, the feet respond as much as does the voice to the presence and activity of others’ (2008: 1). At different points in the exercise we were prompted (via Telegram) to share our gathered data and to discuss our experiences with other members of the group. This was intended to have the effect of demonstrating how we differently experience and interpret our urban surroundings within a single setting, drawing attention to the relational and multi-narrative character of the city (Hannerz, 2001: 206). With an attention to the conceptual interests of multimodality meanwhile, these moments of group reflection sought to highlight how we differently privilege material in our sense-making practices. More generally, in the absence of a predetermined route through the city, direction and pace required to be negotiated by each of the three groups, presenting us with impromptu sites of learning. Faced with a range of possible paths, we might find discord or harmony as we express our dislike or desire for a path that traverses the pedestrianised shopping precinct, the public park or the tree-lined pavement. These same insights would not be apparent by an excursion undertaken in isolation, even if at one point during our walk we were asked to resist the natural instinct towards discussing our experiences.
Continuing without conversation
Part-way through the excursion we were invited (via a Telegram prompt) to continue our walk without conversation and instead with the purpose of individually taking account of the full range of meaning-making phenomena that we would encounter. While this approach might seem to contradict the function of a group walk, this short activity has a precedent in the practice of sound walks of Drever (2013) amongst others, and as well as in ear cleaning games of Schafer (1977) in his influential work around soundscapes. The premise of this part of our excursion was that, in the absence of spoken conversation (although not silence) we would each be better able to focus on the broad range of meaning-carrying phenomena that help us to make sense of our surroundings. While Schafer and others working in acoustic ecology and similar fields might remove conversation as a way of sharpening their attention to the sonic character of their environment, reflecting our interest in multimodality, we instead used these moments without spoken dialogue as a way of drawing attention to the breadth of meaning-carrying material we encounter even across a short period of time. Therefore rather than foregrounding the aural realm, by walking without conversation we sought to emphasise the richly multimodal character of the street. Further, by inviting participants to share their experience after a few minutes our intention was to demonstrate how even with a single place and across a short period of time, our interest was differently drawn towards different meaning-carrying phenomena.
Your mobile data
While mobile digital technologies have opened up opportunities for working with data in ways that were previously the preserve of those with a reasonably high level of technical sophistication or technological ownership, a number of prompts during our Bremen excursion encouraged participants to reflect critically on the nature of the data they were able to collect. This included an invitation to discuss the ability of their mobile devices to record the full range of meaning-making phenomena they had encountered during the preceding part of the excursion. If the smartphone enables the gathering of aural, visual and various location-based representations of the city, as yet they are less well equipped to adequately gather and reproduce the smell of street food (sausages and onions to offer particular examples from last night's walk!), the coldness of air or the hardness of the pavements we encountered during an evening walk though Bremen's Old Town. In their discussion of the compatibility between multimodality and ethnography Dicks et al (2006) and Kress (2011) have pointed towards the varying ability of the researcher to satisfactorily collect or classify the broad range of meaning-carrying material that we experience, for instance as we move through the city. Whilst recognising that Powell (2010) argues for synaesthesia's ability to evoke different sensory experiences through image-based content, our excursion sought to consider (from the perspective of the multimodal researcher) how the smartphone varies in its ability to record and reproduce different forms of meaning-making phenomena.
Algorithmic culture and code
If smartphones are better equipped to take account of the visual and aural than other forms of meaning-carrying material, we should be wary of assuming they produce an exact or innocent record of what was experienced in the street. Mirzoeff unsettles our tendency to refer to 'digital photographs' by drawing attention to the sophisticated sensors that shape what is constructed and displayed on our screens, going as far as to suggest that these digital realisations are sufficiently distant from the traditions of photography that they instead need to be described as 'images'. In a similar vein the acoustician Cox (2014) is amongst those who point to the ways that aural representations of our surroundings are subject to the devices we use for recording and reproduction. What makes the workings of our smartphones significant is the ways that the data we collect is subject to complex code concealed inside our smartphones and possibly beyond our control and understanding. Considered from the perspective of Knox's critical work around algorithmic culture and posthumanism (2016), where digital activity is a complex assemblage of human, technological, commercial and other interests and influences, we might become less confident describing how we use our smartphones to 'capture' what is seen and heard. Perhaps this in turn discourages us from seeing the smartphone as a 'tool' that we use in our research, and instead as having some shared agency with the researcher. In light of multimodality's relationship with technology, combined with the increasingly digital nature of our lived world, there is value in challenging the dualistic binary between technicist and instrumentalist assumptions around technology (Bayne 2014) in favour of a more nuanced understanding of our complex and shifting relationship with the digital. The final Telegram prompt in our activity reflected this interest, as it asked participants to consider how their experience had been affected by the influence of algorithm and code. A critical departure point to end our multimodal excursion through Bremen.