Introduction

In many species, individuals rely on multiple sensory channels for communication, which can also be deployed simultaneously, resulting in what is commonly referred to as “multi-modal communication”. A recent upsurge in interest in multi-modal communication paved the way for the realization that studying communicative channels such as vocalizations and visual displays in a vacuum fails to capture the full complexity of a communicative system, and since then, multi-modal communication has been described in groups as diverse as insects, anurans, birds, reptiles and mammals (Higham and Hebets 2013). For example, in ants (Aphaenogaster albisetosus) and spiders (Cupiennius salei), signals involved in recruitment and mate attraction may involve a pheromone release in conjunction with a vibrational element (Rovner and Barth 1981; Hölldobler 1999), while in African elephants (Loxodonta africana), “greeting” behaviours may comprise a visual and a tactile component (Allen et al. 2021). Whether multi-modal signals are employed in the communication systems of non-human primates has received growing attention, given the valuable insight such data can also provide regarding the evolutionary origins of human communication and language (Slocombe et al. 2011; Liebal et al. 2022). The term “multi-modal” has, however, been used differently in previous primate communication studies, in some cases denoting multiple signaling channels (e.g. facial expressions vs gestures; Liebal et al. 2014; Wilke et al. 2017), while in others denoting multiple sensory modalities (e.g. acoustic vs. visual modality; Fröhlich and van Schaik 2018; Singletary and Tecot 2020). Here, we define a multi-modal signal as one that is received in at least two sensory modalities (Partan and Marler 2005; Fröhlich et al. 2019).

Previous research on multi-modal communication in non-human primates has shown that acoustic and visual signal components are often tightly linked (Liebal et al. 2014; Fröhlich et al. 2019; Aychet et al. 2021). For example, in monkeys, receivers are adept at recognizing the correspondence between facial expressions and vocalizations (Ghazanfar 2013) and integrating this composite information for crucial tasks such as individual recognition (Habbershon et al. 2023). Furthermore, great apes are known to augment their vocalizations with specific visual gestures. Vocalisations combined with a gesture are more likely to elicit a response from receivers compared to unimodal vocal signals (Wilke et al. 2017), and combining vocal and gestural signals may also serve to disambiguate or refine meaning, akin to the function of so-called ‘extralinguistic cues’ as semantic devices in language (Holle and Gunter 2007; Hobaiter et al. 2017; Wilke et al. 2017). For example, in bonobos, the “contest hoot” vocalization can be combined with a threatening “stomp” gesture during agonistic challenges, or with a playful “wrist shake” in friendly play (Genty et al. 2014). In chimpanzees, mothers interacting with infants often combine the “soft hoo” vocalization, which is given in multiple contexts, with an “arm reach” or “present back” gesture, when they want to invite the infant to climb onto their back (Fröhlich et al. 2016).

These examples and more have additionally provided valuable insight into the proximate and ultimate mechanisms underlying multi-modal communication. At the proximate level, multi-modal signals can be described as either “fixed”, whereby the distinct components which make up the signal are necessarily emitted together (e.g. the croaking of a frog is necessarily accompanied by the visually salient inflation of the throat sac), or “free”, when signal components can also occur separately, such as the independently produced visual bowing and vocal cooing in male pigeon courtship displays (Partan et al. 2005). Furthermore, distinct components can be combined into a multi-modal signal for different functions (Partan and Marler 1999; Fröhlich and van Schaik 2018). Combinations can serve to increase the reliability of message content (“redundancy hypothesis”) or the likelihood of signal transmission across a variety of environmental conditions (“efficacy-backup hypothesis” Hebets and Papaj 2005). Combinations may also elicit enhanced responses compared to those observed when either component occurs alone, such as the increased anti-predator response observed in squirrels when tail flagging and alarm calling behaviour are combined (Partan et al. 2010). In addition, non-redundant multi-modal signalling may provoke an entirely new response in receivers, which has been termed “emergence” (Partan and Marler 2005). This function is well-attested in humans and is arguably partly responsible for the enormous generativity of our communication system (Scherer 2013).

Regarding the ultimate function of multi-modal communication, it may be that more sophisticated multi-modal signaling is associated with increased social demands, in line with previous work indicating an evolutionary relationship between communicative complexity and social complexity (Freeberg et al. 2012). Complexity in multi-modal communication could be manifested via abundant multi-modal combination types, many signal components within a combination or a diversity of functions of multi-modal signals and indeed evidence from chimpanzees suggests that signal combinations represent an adaptation for navigating challenging social situations (Leroux et al. 2022). Further, signaling behaviour in chimpanzees is known to vary based on demographic factors (e.g. age, rank; Tomasello et al. 1994; Hobaiter et al. 2017), denoting a high degree of flexibility that can be tailored to individual social needs. However, investigating either the functional mechanisms or the evolutionary drivers of multi-modal combinations requires first a comprehensive and systematic documentation of multimodal repertoires (Aychet et al. 2021). Focusing on non-human species with an extensive vocal and visual repertoire and an attested ability to produce “free” multi-modal signals would be particularly promising. Chimpanzees, one of our two closest living relatives, represent precisely such a system.

Chimpanzees, like humans, have complex social lives: they reside in groups of ~ 20–200 individuals, forming strong and durable relationships with relatives as well as non-kin (Rosati et al. 2020). Likely as a way to navigate this complex social environment, chimpanzees are also equipped with a rich system of communication comprising signals in several modalities (Parr and Waller 2006; Slocombe and Zuberbühler 2010; Hobaiter and Byrne 2011). In line with a wealth of literature addressing the notion of communicative signals in non-humans, we use the term “signal” to mean a vocal or visual behaviour that has evolved specifically for the purpose of communication, as opposed to behaviours designed to fulfil other mechanical or social functions (Seyfarth and Cheney 2003; Genty et al. 2009; Hobaiter and Byrne 2011; Laidre and Johnstone 2013; Grund et al. 2024). The chimpanzee’s repertoire of vocal signals consists of approximately 13 different call types (Slocombe and Zuberbühler 2010). The repertoire is commonly described as graded (Slocombe and Zuberbühler 2010), meaning that there is acoustic variation within a single category, as well as a degree of overlap in acoustic features also between certain categories. Chimpanzee vocal production is constrained to a limited range of sounds compared to human vocal production (Seyfarth and Cheney 2010; Fitch et al. 2016). By contrast, visual signal production in chimpanzees is highly flexible and the repertoire comprises at least 66 visual gesture types (Hobaiter and Byrne 2011) and 9 facial expressions (Parr and Waller 2006). Although some gestures may also be received in the tactile modality (e.g. slap, poke) and important information relating to identity and reproductive state may be received in the olfactory modality (Liebal and Oña 2018), we focus here on the core modalities which have been most thoroughly investigated and which are empirically most tractable, namely vocal and visual communication (Liebal et al. 2014). Previous work has shown that both wild and captive chimpanzees combine vocal and gestural signals (Taglialatela et al. 2015; Hobaiter et al. 2017) and a repertoire of combinations of vocalizations, gestures and facial expressions has been described (Wilke et al. 2017). However, since vocalizations may co-occur with other visual components simply by chance, differentiating random from non-random multi-modal combinations is paramount.

Importantly, in chimpanzees as well as humans, vocal signals, facial expressions and manual gestures are also complemented by an equally broad array of bodily behaviours, such as posture (e.g. sitting vs. standing), head and body orientation, (towards vs. away from the recipient), or movements and actions (e.g. walking, running, playing, eating). In comparison to signals, these behaviours might be rather described as cues, i.e. behaviours that have not evolved for a communicative purpose, and serve direct mechanical functions other than communication, yet may nonetheless carry communicative value (Searcy and Nowicki 2010; Roberts et al. 2014). In humans, such non-verbal cues are known to play an important role in augmenting the meaning of an utterance, constituting a vital component of the communicative interaction (Müller et al. 2013). For example, body posture can inform the receiver regarding the believability of what is being said (Müller et al. 2013). However, these cues are rarely, if ever considered in discussions of primate multi-modal communication. Given the importance of such cues in human communication and in light of the evidence that primates integrate a wide range of information sources when responding to a signal (Seyfarth et al. 2017), we regarded all visual information offered by the signaler during vocal production as integral to a comprehensive analysis of chimpanzee multi-modal communication.

In this study, we systematically investigated the multi-modal communicative behaviour of wild chimpanzees. Firstly, taking a bottom-up, inductive approach we built a vocal-visual repertoire by focusing on all non-vocal behaviour (NVB) offered by the signaler during vocal production (body postures, orientations, behaviours, gestures or facial expressions). Secondly, we quantified the non-random nature of identified vocal-visual combinations using collocation analysis, a method borrowed from computational linguistics (Bosshard et al. 2022). The use of quantitative models and theoretical concepts from linguistics has become increasingly advocated by primatologists studying communication. For example, previous work has highlighted the validity of applying conversation-analytic tools, along with numerous linguistic laws, to describe gestural and vocal communicative behaviour in apes (e.g. Fröhlich 2017; Heesen et al. 2019; Watson et al. 2020; Safryghin et al. 2022). Here, building on work from Leroux et al. (2022), we investigated chimpanzee signals using collocation analysis. This method quantifies whether signal combinations are non-random, defined here as the co-occurrence of signal components with a frequency that is statistically above chance. In contrast to observational data and field experiments that focus on receiver responses to gauge if a single combination produced is communicatively relevant (e.g. Hobaiter et al. 2017; Wilke et al. 2017; Leroux et al. 2023), collocation analysis provides an overview of combinatoriality across the whole repertoire and therefore can help in identifying candidate, potentially meaningful combinations in a species. In line with related approaches such as network analyses (see Aychet et al. 2021), our study therefore represents an additional novel data-driven quantification of non-random multi-modal combinations, illustrating which vocalizations and non-vocal behaviours are systematically associated at frequencies above chance in a non-human primate.

In addition to establishing a repertoire of non-random vocal-visual combinations, we aimed to further examine variation in multi-modal combinations within the population. Previous research has highlighted substantial inter-individual flexibility in primate signal production across demographic classes (Tomasello et al. 1994; Luef and Pika 2019) and social contexts (Clark 1993; Bouchard and Zuberbühler 2022) and as a function of audience presence (Schel et al. 2013; Hobaiter et al. 2017). In particular, these studies have implicated demographic factors such as age, sex and rank in driving variation in both gestural and vocal behaviour. For example, female chimpanzees are known to produce a higher rate of call combinations than their male counterparts (Leroux et al. 2022), while highest-ranking males were shown to be the most prolific gesture producers (Hobaiter et al. 2017). However, in contrast, Wilke et al. (2017) found no effects of age, rank or sex on the rate of multi-modal signal production in chimpanzees. Given the mixed evidence for demographic factors influencing the rate of unimodal and multi-modal signal production, we explored whether demographic factors affected the number of visual components individuals combined with vocal signals. We also examined low-level explanations (i.e. call duration and call type) for variation in number of NVBs produced alongside a call. Longer vocalization bouts might provide more opportunity for movements, changes in body posture or gestures to be produced. Finally, as call types produced in evolutionarily urgent events such as agonistic encounters may be associated with more NVBs to reduce ambiguity, we tested whether NVB production was influenced by call type.

Methods

Study site and data collection

The study was conducted on wild chimpanzees from the Kanyawara community in Kibale national park, Uganda (Thompson et al. 2020). The population consists of ~ 60 individuals inhabiting a home range of ~ 15km2. The Kanyawara community has been the object of long-term study since 1987 and is entirely habituated. The data used in this study were collected between February-May 2013, and between June 2014 and March 2015 (Wilke et al. 2017). These data consist of video-audio recordings collected within the chimpanzee home range, between 0800 and 1900 h. The equipment included a hand-held camcorder (Panasonic HDC-SD90), and an external microphone (Sennheiser MKE 400).

The individuals observed in this study were 13 females and 14 males, between 10 and 48 years of age. Individuals were recorded from a distance of at least 7 m while engaged in their natural behaviour. Focal animal sampling was employed (Altmann 1974), involving 15 min of continuous video observation of one single animal, with the aim of capturing a clear and complete view of the animal and all its behaviours, including communication. Focal animals were only sampled once a day. Initially focal subjects were chosen on the basis of visibility and ease of pursuit to ensure high-quality recordings. Later in the study period, priority to certain subjects was given in order to homogenize the total focal time across individuals. Thirty-one hours of video/audio data were used in this study. It was not possible to record data blind because our study involved focal animals in the field.

Data extraction: the vocal-visual combinations

Subsequent data extraction was carried out on the video/audio recordings using Noldus Observer XT 10 events logging software (http://www.noldus.com/animal-behaviour-research). The annotation of video/audio footage was centered around events of vocal production (N = 297). For each of these events, the researcher coded information on both the vocal as well as the visual components of signal production.

Vocalizations were classified according to the call types described in existing chimpanzee repertoires and specific empirical studies (Slocombe and Zuberbühler 2010; Crockford et al. 2018). Of the ~ 13 call types described in the repertoires, this study focused on the seven most commonly produced: grunt, soft hoo, pant bark, pant grunt, pant hoot, scream and whimper. The minimum number of occurrences necessary for a call to be included in the analyses was 5. In the case of the calls “grunt” and “soft hoo”, the existing literature describes different call subtypes, whereby “soft hoo” can be divided into “travel hoo”, “rest hoo” and “alarm hoo” (Crockford et al. 2018), while “grunt” can refer to “rough grunt” or “general grunt” (Slocombe and Zuberbühler 2010). Here however, all respective subtypes were lumped into the broad categories of “soft hoo” and “grunt”. Rough grunts and general grunts were collapsed given that our sample only included low-frequency rough grunts, which are acoustically similar to general grunts. High-pitched rough grunts and rare call types did not occur in the available video-audio footage with sufficient frequency to be included in this study. Additional call types that were not observed at least 5 times and therefore not included in the study were the following: bark, waa bark, pant, cough, wraa, laughter, squeak. The number of events observed for the seven call types included ranged from 5 to 98. Chimpanzee vocalizations are often produced in bouts. A bout was defined as a sequence of the same call type with pauses shorter than 10s between the individual acoustic elements. A bout was considered terminated when followed by 10s of silence or by the production of a different call type. Bouts constituted single data points. The duration of vocal bouts ranged between 1 and 62 s.

In association with each vocal event, between 1 and 8 non-vocal behaviours (NVBs) were recorded. NVBs were only annotated if the behaviour occurred during the vocalization or vocal bout. A total of 31 different NVB types were recorded in this study. Table 1 provides the full list of NVBs annotated in this study, as well as a description of the behavioural criteria used to assign each NVB type. The list of NVBs illustrated in Table 1 includes behaviours such as feed, groom and rest. Whilst previous research has used the concurrent behaviour of the vocalizing individual to determine the broader behavioural context in which communication is produced (e.g. Wilke et al. 2017), these behaviours also potentially provide conspecific receivers with valuable visual information in the same manner as a body posture such as “sit”, or a gaze orientation such as “look towards”. Therefore, we include such behaviours as NVBs. NVBs included in this list represent an attempt to illustrate the observable variation in visual behaviour, and the level of granularity takes into account the risks of an over-representation of NVBs, general feasibility in coding, and complying with inter-observer reliability. Additional measures taken to maximally standardize the annotation procedure can be found in the ESM.

Table 1 Full list of NVBs (non-vocal behaviour) annotated in this study with corresponding behavioural description used to assign NVBs. The term “specific individual” refers to the individual who is closest to the signaler
Full size table

Data extraction: demographic context of the vocalization

In addition to describing vocal signals and accompanying NVBs, demographic data were annotated for each event. Specifically, identity and sex of the individual were noted and each individual’s age in years was calculated based on the long-term data which includes birth dates for all IDs (Thompson et al. 2020). Next, dominance ranks were calculated using an Elo-rating method (Muller et al. 2021; Wilke et al. 2022) based on the long-term data on aggressive interactions and submissive pant grunt vocalizations (de Vries et al. 2006). Rank scores were calculated every 3 months and ranged between 1 and 24.

Inter-observer reliability

To ensure videos were coded reliably, a second independent researcher coded 11% of the events (i.e. 34 events out of 297) and extracted both (i) the call type (at least one call for each call type was present in the subset) and (ii) non-vocal behaviours (at least one instance of each NVB type was coded in the subset). We calculated a Cohen’s kappa value of 0.82 and 0.88 for vocalisation type and NVB type respectively, indicating excellent levels of agreement in both cases (Fleiss 1981).

Collocation analysis

To generate a vocal-visual combination repertoire based on the communicative events observed, we implemented a collocation analysis in R (R Development Core Team 2009). This method, originating in the field of linguistics and recently adapted to the study of animal communication, estimates the relative attraction between communicative units, based on how frequently they co-occur in the dataset (Bosshard et al. 2022). We specifically implemented a Multiple Distinctive Collocation Analysis (MDCA; Bosshard et al. 2022) given that this method has been shown to account for datasets that are small and skewed (Gries and Stefanowitsch 2004; Hilpert 2006; Gries 2014). This feature was especially relevant in our analysis given that certain call types occurred infrequently (pant bark N = 7 and whimper N = 5), which was therefore explicitly accounted for via implementing the MDCA method. Using MDCA, we examined the co-occurrence of a particular vocal signal with a specific visual component. For example, if “grunt” + “arm reach” co-occur, collocation analysis compares the frequency of “grunt + arm reach” with the frequency of all other vocal-visual combinations which contain either “grunt” or “arm reach”. In multiple distinctive collocation analysis, the association between units is calculated using binomial probability values on each possible two-unit combination. Specifically, the binomial probability mass function renders probability values for each unit with each other unit, and the log-transformed results provide a numerical estimate of how exclusively units combine with one another. Values above 1.3 represent co-occurrence at above-chance level, while values below − 1.3 represent significant repulsion between collocates.

A feature of the communicative events included in this dataset is that one vocal signal commonly co-occurs with more than one NVB simultaneously. For example, a “grunt” vocalization may co-occur with a “sit” posture, a “scratch self” gesture and a “look towards” movement within the same event. Consequently, our analysis aimed to investigate not only the above-chance occurrence of vocalizations and NVBs individually, but also the association between a given call and multiple NVBs at once. Therefore, a modified collocation analysis was designed to test the association between one call and up to four concomitant NVBs. A threshold of 4 was chosen as 93% of events exhibited between 1 and 4 NVBs. In order to test associations between vocalizations and NVBs at all levels of combination, each event where > 1 NVB occurred was entered into the dataset multiple times; first with each NVB individually, and then with all possible combinations of two, three and four NVBs given the NVBs present in that event. When such combinations were entered into the data table, this was done while maintaining the two-column structural requirement of collocation analyses. As shown in Table 2, the event “grunt + sit + scratch self + look towards” generated 7 separate rows of data to account for each unique combination within the communicative event. Given that what is important in MDCA is the frequency of specific combinations and not the frequency at which any given element appears within the sample, the inclusion of several entries per event does not lead to an inflation of collocation strengths. Instead, the inclusion of multiple entries per event results in a greater extent of recombination, which has been shown to subsequently decrease collocation strength (Bosshard et al. 2022). As such, we regard our MDCA results to be a conservative estimate of the combinations of vocal signals alongside non-vocal behaviours.

Table 2 Illustration of procedure for entering each communication event into a suitable dataset for implementing the multiple-NVBs collocation analysis
Full size table

Statistical analyses: demographic and call-related drivers of NVB production

To examine variation in the number of NVBs produced alongside vocalizations as a function of demographic variation and call characteristics (i.e. call type and call duration), we performed a generalized linear mixed model (GLMM) with a negative binomial error structure and log link function using the glmmTMB function, glmmTMB package in R. We modeled the number of NVBs produced per event as a numerical integer response variable. As demographic predictors, we fitted age (years) as a second-order polynomial, sex as a binary categorical variable (M/F) and rank as a numerical integer. As call-related predictors, we fitted call type as a 7-level categorical variable, and duration of call bout (seconds) as a numerical predictor. Given that the effect of call type and duration may not be independent, an interaction term was fitted between these predictors. Individual identity was fitted as a random factor to account for multiple events from single individuals.

We first compared the full model including all predictors and random effects with a null model which was identical in structure minus the predictors, for which we report a likelihood ratio test (chi-squared statistic and p-value). We ascertained the relative contribution of each variable to the model by comparing the full model to a reduced model lacking each individual predictor in turn. We then report chi-squared values of likelihood ratio tests regarding the effect of each individual predictor, as well as p-values using a 95% significance threshold.

Model assumptions were checked using the DHARMa package in R. The model was not found to exhibit overdispersion (nonparametric dispersion test P = 0.74), no outliers were detected (P = 0.4) and visual inspection of the Q-Q plots confirmed normality (K-S test: P = 0.77).

Results

Vocal-visual repertoire via collocation analysis

Following collocation analyses, 108 combinations of one vocal signal and between 1 and 4 NVBs were found to co-occur significantly more frequently than expected by chance (all p-values < 0.05). The number of significant combinations varied between call types: for example, four combinations were documented for the “pant bark” call, six for the “scream”, 11 for the “whimper”, 16 for the “soft hoo”, 22 for the “pant grunt”, 24 for the “pant hoot” and 25 combinations for the “grunt” call. Of the 31 NVB types present in the raw data, 21 featured in significant combinations with vocal signals. Eighteen out of these 21 NVB types (i.e. 86%) were recombined productively across multiple call types. The full set of significant combinations which constitute the vocal-visual repertoire is presented in Tables 3 and 4.

Table 3 List of 31 single NVBs (non-vocal behaviour) and 7 call types included in this analysis. Colour codes denote strength of attraction/repulsion between NVBs and each call type: darkest green = strongest attraction, darkest red = strongest repulsion. All values above 1.3 represent co-occurrence at above-chance level with 95% confidence interval, while values below − 1.3 represent significant repulsion between components
Full size table
Table 4 All combinations of vocal and visual components that were found to co-occur more frequently than expected by chance
Full size table

Demographic and call-related drivers of NVB production

Our GLMM analysis indicated that the full model, including all predictors, explained significantly more variation in the response variable compared to a null model (χ216 = 38.96, p = 0.001). Likelihood ratio tests revealed that there was no significant main effect of age (χ22 = 1.39, p = 0.497), sex (χ21 = 1.25, p = 0.262) or rank (χ21 = 1.29, p = 0.254) on the number of NVBs produced per vocalization. However, there was a significant interaction between call type and duration (χ26 = 19.68, p = 0.003), such that the effect of duration on the number of NVBs differed between call types. Longer call duration was associated with more NVBs in “pant grunt”, “pant hoot” and “soft hoo” calls, while no such effect was observed in the other call types. Overall, the “pant grunt” call was produced in association with the most NVBs while the “scream” was associated with the fewest, as shown in Fig. 1.

Fig. 1
Fig. 1
Full size image

Raw data illustrating variation in the number of NVBs (non-vocal behaviours) produced at significantly above chance frequencies with the different call types analysed in this study. Boxes indicate the inter quartile range (IQR), the central line depicts the median and the whiskers extend to the maximum and minimum values excluding outliers. Circles represent outliers, and crosses represent means for each call type

Discussion

By systematically observing naturally occurring communication events, and through implementing frequency-based analyses, we show that chimpanzees combine their vocal signals in non-random ways with a wide range of body movements, behaviours, postures, gestures and facial expressions, collectively referred to here as non-vocal behaviours (NVBs). More than 100 such combinations of vocal and visual components occur more frequently than expected by chance, indicating a strikingly diverse repertoire of vocal-visual combinations. Some NVBs are used productively across multiple call types, yet each call type is associated with its own set of single and combined NVBs. Further, variation in the number of NVBs produced per vocalization is not predicted by demographic variables such as age, sex or rank.

It has not escaped our attention that within the ~ 100 combinations found to occur most frequently, approximately two thirds involve visual behaviours that need not necessarily be interpreted as signals, i.e. evolved for the purpose of communication, but rather as visual cues (see ESM for a separation of combinations containing visual signals and those containing only visual cues). Whilst signals and cues are traditionally considered separately, we argue in favor of casting the net wide and taking both together, asserting that the traditional stark dichotomy between signals and cues may benefit from being represented more as a graded continuum. For instance, in our dataset, bodily movements such as standing up, retracting or extending the body, whilst superficially cue-like, could be deployed for communication purposes, signaling willingness to initiate travel, or submission. Thus, we maintain that not only signals, but all visual information offered by the vocalizer constitutes an integral component of the multi-modal communicative interaction. Ultimately, carefully designed experiments using captive animals and video playbacks are needed to understand the relative importance of cues or signals being combined with vocalizations (Waller et al. 2016; Whitehouse et al. 2016).

Given the findings of the collocation analysis, it appears that sub-adult and adult chimpanzees have access to a highly diversified repertoire of combined visual and vocal components. Although the constrained vocal repertoire of chimpanzees (Seyfarth and Cheney 2010) might suggest a limited capacity for information transfer, the free and productive use of accompanying NVBs instead reveals a high potential for refining the meaning of the limited range of available calls. This finding was enabled by the application of methods developed in the field of linguistics, further adding to the growing body of work which highlights the importance of inter-disciplinary integration between linguistic concepts and biological research (Semple et al. 2022).

A fundamental implication of this investigation is that unimodal approaches to primate communication, which analyze vocal or visual components separately, result in a drastically oversimplified picture of flexibility in signal production. A multi-modal approach is therefore crucial to accurately represent the communicative abilities and real-life communicative exchanges of non-human primates (Slocombe et al. 2011; Liebal et al. 2022), as well as other species outside the primate taxon. For instance, many non-primate examples of communication traditionally regarded as unimodal could be revisited using the broad inductive approach outlined here as a means to identify non-random cues that may enhance receiver reception of signals or otherwise modify signal meaning.

Chimpanzee social life is characterized by a wide variety of interactions, each of which is typically mediated by communication. Thus, it is likely that the suite of combined vocal and visual components identified here plays a key role in supporting the demands of a chimpanzee’s daily social life (Freeberg et al. 2012; Bouchet et al. 2013). Now that we have identified a suite of systematically re-occurring vocal-visual combinations, further investigation may shed light on the communicative function of combining vocal and visual components, especially with regard to the function of “emergence” whereby new meaning is created (Partan and Marler 2005). Importantly, we have identified potentially meaningful vocal-visual combinations using a data-driven, bottom-up approach rather than by selecting combinations a priori. Consequently, we may now begin to appreciate the full diversity of possible nuances that can be generated by combining a given vocalization with many different non-vocal behaviours, as in the case of the “grunt”, which was associated with the longest suite of NVBs in our study. Vocalizations described as “grunt” are common across many primate species (Cheney and Seyfarth 1982; Gamba and Giacoma 2007; Salmi et al. 2013; Katsu 2016; Silk et al. 2018) and are involved in a variety of interaction types including reconciliation (Cheney et al. 1995), travel initiation (Sperber et al. 2017) and more (Rendall et al. 1999; Whitham et al. 2007). Within chimpanzees, “grunt” is a call type that can be used in many different contexts (Slocombe and Zuberbühler 2010). Therefore, this vocalization represents an appropriate model for a signal whose meaning may be disambiguated by integrating the signaler’s body posture, gaze, movements and gestures. For example, our results show that “grunt” is commonly combined with “sit” or “rest”, but also with “stand”, “walk” or “climb”. Such substantial variation in body postures is likely to correlate with distinct behavioural contexts of emission and thus engender contrasting outcomes on behalf of receivers. Similarly, if we compare the “grunt + look towards + sit + turn body towards” combination reported in our collocation results with the alternative “grunt + look away + stand + walk”, it is reasonable to assume that the respective chimpanzee receivers in these circumstances might exhibit divergent behavioural responses. As such, simply by inspecting the variation in NVBs associated with each call type, we may already begin to infer the communicative importance of combining vocal and visual elements in chimpanzees.

It is unknown whether chimpanzee signalers voluntarily combine vocal signals with all of the NVBs reported in this study. Nonetheless, chimpanzee receivers likely rely on the integration of all the vocal and visual information available in order to guide their own adaptive behavioural response (Seyfarth and Cheney 2003). Confirming this hypothesis requires further investigation into how NVBs are perceived by receivers and their potential role in the disambiguation of meaning. One fruitful method to address meaning empirically involves a systematic analysis of behavioural reactions to signals as a function of signal type (Hobaiter and Byrne 2014), showing, for example, that a chimpanzee grunt vocalization combined with a gesture is more likely to exhibit a response than if the grunt is produced on its own (Wilke et al. 2017). Furthermore, recent developments which combine insights from linguistics and animal behaviour offer new theoretical frameworks and empirical toolkits for addressing the meaning of signal components in nonhumans (Berthet et al. 2022). Berthet and colleagues illustrate how semantics and pragmatics, key concepts in linguistics which relate to meaning interpretation, can be adapted to primatological research to achieve a more detailed picture of signal meaning in primates. This valuable approach could be applied to the wide range of vocalization and NVB combinations highlighted in this study, offering critical insights into the meaning of chimpanzee vocal-visual combinations (see also Wheeler and Fischer 2012). A complementary avenue of investigation is to infer which cues are most salient to recipients for meaning disambiguation, using measures of attentional bias during video playbacks. The application of eye-tracking technology in captive great apes, for example, has enjoyed a recent surge of advances, bringing this goal confidently within reach (Kano et al. 2017).

Our study also investigated variation in the number of NVBs produced per vocalization as a function of individual demographic attributes such as age, sex and rank. However, males and females did not differ in the number of NVBs produced, nor was the observed variation explained by age or rank. These results broadly support Wilke et al.’s (2017) finding that demographic factors did not influence rate of multimodal signal production but are in contrast to Leroux et al.’s (2022) finding that females and low-ranking chimpanzees produce more vocal combinations. A possible implication of this result is that unlike vocal sequences comprising two acoustic items, combining components from different modalities may predominantly serve a more general function such as that of redundancy or efficacy-backup (Hebets and Papaj 2005), which is crucial irrespective of demographic status. Replicating this work in other communities of chimpanzees would prove useful for establishing the universality of this finding. Indeed, it remains possible that a population which experiences different ecological or social pressures, may display more pronounced demographic patterns in NVB production than those observed here. In contrast to demographic factors, we found a significant effect of call duration on number of NVBs produced, but this effect was conditional on call type, such that longer vocalization events were associated with a greater number of NVBs in some call types but not in others. Interestingly, a positive effect of duration was observed most strongly in pant hoot, soft hoo and pant grunt vocalisations, not in screams or pant barks which are associated with more evolutionarily urgent events, such as agonistic encounters or threatening interactions with dominant individuals. Given the urgency of these contexts, it is possible that the NVBs associated with screams and pant barks tend to be produced within the first few seconds of the vocalization bout, while NVB production in more relaxed and nuanced contexts can be more temporally protracted.

In conclusion, our findings reveal a hitherto unappreciated diversity of vocal-visual combinations in the communication of wild chimpanzees. Follow-up behavioural observations and experimental work are key to unpacking the function and meaning of such combinations. Nonetheless, the extent and variety of non-random vocal-visual combinations described here broadens our understanding of the potential combinatorial information available to receivers in our closest-living relative. Furthermore, ~ 90% of the visual components of communicative exchanges observed in this study were shown to be produced in association with multiple call types. In line with previous work, this is suggestive that multi-modal events represent combinatorial structures, of which vocal and visual components constitute the building-blocks (Davila-Ross et al. 2015). By virtue of our phylogenetic proximity to chimpanzees, the range of vocal-visual combinations presented here also informs our understanding of the communicative behaviour of our hominin ancestors, corroborating convergent evidence that multi-modal communication has an ancient history in our primate lineage (Evans et al. 2005; Kulahci et al. 2014) and supporting the notion that complex multi-modal signaling may have played a role in scaffolding language evolution (Wheeler and Fischer 2012; Seyfarth and Cheney 2014, 2017).