Skip to main content
Springer Nature Link
Log in

Multimodal semantic communications

  • Review Article
  • Published:
View saved research

From Nature Reviews Electrical Engineering

View current issue Sign up to alerts

Abstract

Multimodal semantic communication focuses on transmitting meaning of multimodal data rather than exact bits or symbols. By exploiting semantic correlations and complementarities across modalities, it supports reliable transmission under extremely poor conditions and severely limited bandwidth. For instance, it remains effective under 0-dB signal-to-noise ratio and with only a few kilohertz of available spectrum. These properties make multimodal semantic communication attractive for applications that involve massive multimodal data and tight spectrum constraints, such as immersive communication and satellite networks. In this Review, we clarify the distinction between multimedia communication and multimodal communication, revealing a shift from independent data processing towards joint semantic understanding across modalities. Here, semantic theories provide quantitative metrics for semantic communication and core methods in multimodal semantic communication allow practical approaches to process and transmit multimodal semantics. Deep learning-based representation, sampling, coding, modulation and resource allocation methods enable efficient and reliable multimodal semantic communication systems. Finally, we outline a unified, green and compatible multimodal semantic communication to support intelligent connectivity for humans and agents.

Key points

  • Multimodal semantic communication differs from multimedia communication by exploiting cross-modal correlation and complementarity rather than independent bitstreams.

  • Multimodal semantic methods have enabled reliable transmission under extreme conditions, including operation at 0-dB signal-to-noise ratio and audiovisual delivery below 1 kbps.

  • Deep learning supports the full multimodal semantic pipeline, from representation and semantic sampling to joint semantic-channel coding, modulation and semantic-aware resource allocation.

  • Immersive communication, satellite communication, vehicular networks and Internet of Things (IoT) show that multimodal semantic communication can reduce traffic from 40× compression in IoT to 60% lower overhead in satellite links while preserving task performance.

  • Future multimodal semantic communication is evolving toward token communication, where unified token representations may support efficient and more compatible multimodal systems and provide a communication framework for future agents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1: A typical multimodal semantic communication system.
Fig. 2: Development of semantic feature fusion mechanisms.
Fig. 3: The architecture and key technologies of a multimodal semantic communication system.
Fig. 4: Applications of LAMs.
Fig. 5: From multimodal semantic communication to token communication.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  1. Wang, C.-X. et al. On the road to 6G: visions, requirements, key technologies, and testbeds. IEEE Commun. Surv. Tutor. 25, 905–974 (2023).

    Article  Google Scholar 

  2. Haykin, S. Communication Systems (Wiley, 2000).

  3. Clemm, A., Vega, M. T., Ravuri, H. K., Wauters, T. & De Turck, F. Toward truly immersive holographic-type communication: challenges and solutions. IEEE Commun. Mag. 58, 93–99 (2020).

    Article  Google Scholar 

  4. Saad, W., Bennis, M. & Chen, M. A vision of 6 G wireless systems: applications, trends, technologies, and open research problems. IEEE Netw. 34, 134–142 (2019).

    Article  Google Scholar 

  5. Li, Z., Chitre, M. & Stojanovic, M. Underwater acoustic communications. Nat. Rev. Electr. Eng. 2, 83–95 (2025).

    Article  Google Scholar 

  6. Molisch, A. F. Wireless Communications: From Fundamentals to Beyond 5G (Wiley, 2022).

  7. Qin, Z. et al. AI empowered wireless communications: from bits to semantics. Proc. IEEE 112, 621–652 (2024).

    Article  Google Scholar 

  8. Gündüz, D. et al. Beyond transmitting bits: context, semantics, and task-oriented communications. IEEE J. Sel. Areas Commun. 41, 5–41 (2022). Introduces the history and basic concepts of semantic communication clearly, and provides a good summary of the key techniques in unimodal semantic communication.

    Article  Google Scholar 

  9. Wolf, L. C., Griwodz, C. & Steinmetz, R. Multimedia communication. Proc. IEEE 85, 1915–1933 (2002). Established an early framework for multimedia communication by defining it as the integrated manipulation of discrete and continuous media.

    Article  Google Scholar 

  10. Misra, S., Mali, G. & Mondal, A. Distributed topology management for wireless multimedia sensor networks: exploiting connectivity and cooperation. Int. J. Commun. Syst. 28, 1367–1386 (2015).

    Article  Google Scholar 

  11. Chiariglione, L. MPEG and multimedia communications. IEEE Trans. Circuits Syst. Video Technol. 7, 5–18 (1997).

    Article  Google Scholar 

  12. Zhang, Q., Zhu, W. & Zhang, Y.-Q. Resource allocation for multimedia streaming over the Internet. IEEE Trans. Multimed. 3, 339–355 (2001).

    Article  Google Scholar 

  13. Ehsan, S. & Hamdaoui, B. A survey on energy-efficient routing techniques with QoS assurances for wireless multimedia sensor networks. IEEE Commun. Surv. Tutor. 14, 265–278 (2011).

    Article  Google Scholar 

  14. Huang, Y. et al. Toward holographic video communications: a promising AI-driven solution. IEEE Commun. Mag. 60, 82–88 (2022).

    Article  Google Scholar 

  15. Akyildiz, I. F. & Guo, H. Holographic-type communication: a new challenge for the next decade. ITU J. Future Evol. Technol. 3, 421–442 (2022).

    Article  Google Scholar 

  16. Tian, Y., Ying, J., Qin, Z., Jin, Y. & Tao, X. Synchronous multi-modal semantic communication system with packet-level coding. IEEE Trans. Wirel. Commun. 24, 3684–3697 (2025). Builds an ultralow-bit rate video communication system through multimodal communication methods with text, speech and visual modalities.

    Article  Google Scholar 

  17. Tong, H. et al. Multimodal semantic communication for generative audio-driven video conferencing. IEEE Wirel. Commun. Lett. 14, 93–97 (2025).

    Article  Google Scholar 

  18. Tandon, P. et al. Txt2vid: ultra-low bitrate compression of talking-head videos via text. IEEE J. Sel. Areas Commun. 41, 107–118 (2022).

    Article  Google Scholar 

  19. Xie, H., Qin, Z., Tao, X. & Letaief, K. B. Task-oriented multi-user semantic communications. IEEE J. Sel. Areas Commun. 40, 2584–2597 (2022).

    Article  Google Scholar 

  20. Guo, J., Chen, W., Sun, Y., Xu, J. & Ai, B. VideoQA-SC: adaptive semantic communication for video question answering. IEEE J. Sel. Areas Commun. 43, 2462–2477 (2025).

    Article  Google Scholar 

  21. Gu, J., Zhang, X., Cui, Q. & Tao, X. Semantic communication for multi-modal data transmission. In International Conference on Wireless Communications and Signal Processing (WCSP) https://doi.org/10.1109/WCSP58612.2023.10404655 (IEEE, 2023).

  22. Hosonuma, E. et al. Image generative semantic communication with multi-modal similarity estimation for resource-limited networks. IEICE Trans. Commun. E108-B, 260–273 (2025).

    Article  Google Scholar 

  23. Du, H. et al. Generative artificial intelligence-aided joint training-free secure semantic communications via multi-modal prompts. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP48485.2024.10447237 (IEEE, 2024).

  24. Zhang, G. et al. A unified multi-task semantic communication system for multimodal data. IEEE Trans. Commun. 72, 4101–4116 (2024).

    Article  Google Scholar 

  25. Lu, J. et al. Generative artificial intelligence-enhanced multimodal semantic communication in internet of vehicles: System design and methodologies. IEEE Veh. Technol. Mag. 20, 71–82 (2025).

    Article  Google Scholar 

  26. Xie, S. et al. Semantic communication for efficient point cloud transmission. In Proc. IEEE Global Communications Conference (GLOBECOM) https://doi.org/10.1109/GLOBECOM52923.2024.10901573 (IEEE, 2024).

  27. Feng, W. & Shi, Z. Multi-modal semantic communication with point cloud diffusion. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP49660.2025.10888756 (IEEE, 2025).

  28. Zhu, Y. et al. Hashing-based multi-modal semantic communication. In Proc. 2024 IEEE Wireless Communications Network Conference (WCNC) https://doi.org/10.1109/WCNC57260.2024.10570632 (IEEE, 2024)

  29. Zhu, Z., Zhang, R., Cheng, X. & Yang, L. Synesthesia of machines-enabled multi-task semantic communication system. IEEE Trans. Mob. Comput. 25, 4632–4647 (2026).

    Article  Google Scholar 

  30. Niu, X., Tan, L., Wu, J., Yuan, W. & Quek, T. Q. S. Multimodal-oriented interactive joint source channel coding for lightweight semantic communication. IEEE Trans. Veh. Technol. 74, 16516–16520 (2025).

    Article  Google Scholar 

  31. Gao, J., Li, P., Chen, Z. & Zhang, J. A survey on deep learning for multimodal data fusion. Neural Comput. 32, 829–864 (2020).

    Article  MathSciNet  Google Scholar 

  32. Xu, P., Zhu, X. & Clifton, D. A. Multimodal learning with transformers: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 45, 12113–12132 (2023).

    Article  Google Scholar 

  33. Yao, L. et al. You only transmit once: unified generation and comprehension for efficient semantic communication. IEEE J. Sel. Areas Commun. 44, 3349–3364 (2026).

    Article  Google Scholar 

  34. Zhang, Y. et al. Multimodal LLM integrated semantic communications for 6G immersive experiences. IEEE Commun. Mag. 64, 68–75 (2026).

    Article  Google Scholar 

  35. Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423 (1948). Laid the foundation of modern digital communication by introducing entropy and channel capacity as fundamental limits for source coding and reliable transmission.

    Article  MathSciNet  Google Scholar 

  36. Carnap, R. & Bar-Hillel, Y. An outline of a theory of semantic information. RLE Tech. Rep. 247, 1–49 (1952).

    Google Scholar 

  37. Bao, J. et al. Towards a theory of semantic communication. In Proc. 2011 IEEE Network Science https://doi.org/10.1109/NSW.2011.6004632 (IEEE, 2011).

  38. Liu, X., Jia, W., Liu, W. & Pedrycz, W. AFSSE: an interpretable classifier with axiomatic fuzzy set and semantic entropy. IEEE Trans. Fuzzy Syst. 28, 2825–2840 (2019).

    Article  Google Scholar 

  39. Chattopadhyay, A., Haeffele, B. D., Geman, D. & Vidal, R. Quantifying task complexity through generalized information measures. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=vcKVhY7AZqK (ICLR, 2021).

  40. Soatto, S. & Chiuso, A. Visual representations: defining properties and deep approximations. In Proc. International Conference on Learning Representations (ICLR, 2016).

  41. Achille, A. & Soatto, S. Emergence of invariance and disentanglement in deep representations. J. Mach. Learn. Res. 19, 1–34 (2018).

    MathSciNet  Google Scholar 

  42. Xin, G. & Fan, P. EXK-SC: a semantic communication model based on information framework expansion and knowledge collision. Entropy 24, 1–14 (2022).

    Article  MathSciNet  Google Scholar 

  43. Choi, J., Loke, S. W. & Park, J. A unified approach to semantic information and communication based on probabilistic. IEEE Access. 10, 129806–129822 (2022).

    Article  Google Scholar 

  44. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630 (2024). Proposes semantic entropy as a meaning-level uncertainty measure for detecting large language model confabulations, helping assess when model outputs are likely to be unreliable.

    Article  Google Scholar 

  45. Nikitin, A., Kossen, J., Gal, Y. & Marttinen, P. Kernel language entropy: fine-grained uncertainty quantification for LLMs from semantic similarities. In Proc. Advanced Neural Information Processing Systems https://doi.org/10.52202/079017-0283 (NeurIPS, 2024).

  46. Aichberger, L., Schweighofer, K., Ielanskyi, M. & Hochreiter, S. Improving uncertainty estimation through semantically diverse language generation. In Proc. International Conference on Learning Representation https://openreview.net/forum?id=HSi4VetQLj (ICLR, 2025).

  47. Niu, K. & Zhang, P. The Mathematical Theory of Semantic Communication (Springer, 2025).

  48. Yates, R. D. et al. Age of information: an introduction and survey. IEEE J. Sel. Areas Commun. 39, 1183–1210 (2021).

    Article  Google Scholar 

  49. Sagduyu, Y. E., Ulukus, S. & Yener, A. Age of information in deep learning-driven task-oriented communications. In Proc. IEEE Conference on Computer Communication Workshops (INFOCOM Workshops) https://doi.org/10.1109/INFOCOMWKSHPS57453.2023.10226176 (IEEE, 2023).

  50. Kosta, A., Pappas, N., Ephremides, A. & Angelakis, V. The cost of delay in status updates and their value: non-linear ageing. IEEE Trans. Commun. 68, 4905–4918 (2020).

    Article  Google Scholar 

  51. Wang, Z., Badiu, M.-A. & Coon, J. P. A framework for characterizing the value of information in hidden Markov models. IEEE Trans. Inf. Theory 68, 5203–5216 (2022).

    Article  MathSciNet  Google Scholar 

  52. Maatouk, A., Assaad, M. & Ephremides, A. The age of incorrect information: an enabler of semantics-empowered communication. IEEE Trans. Wirel. Commun. 22, 2621–2635 (2022). Proposes the age of incorrect information as a semantic-aware metric that goes beyond AoI and error-based formulations.

    Article  Google Scholar 

  53. Ameer, I., Bölücü, N., Sidorov, G. & Can, B. Emotion classification in texts over graph neural networks: semantic representation is better than syntactic. IEEE Access. 11, 56921–56934 (2023).

    Article  Google Scholar 

  54. Liu, C. et al. Explainable semantic communication for text tasks. IEEE Internet Things J. 11, 39820–39833 (2024).

    Article  Google Scholar 

  55. Zhou, F. et al. Cognitive semantic communication systems driven by knowledge graph: principle, implementation, and performance evaluation. IEEE Trans. Commun. 72, 193–208 (2023).

    Article  Google Scholar 

  56. Hello, N., Di Lorenzo, P. & Strinati, E. C. Semantic communication enhanced by knowledge graph representation learning. In Proc. IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) https://doi.org/10.1109/SPAWC60668.2024.10694291 (IEEE, 2024).

  57. Wu, W. et al. Multi-modal knowledge graph enhanced StyleGAN-based cognitive semantic communications for image transmissions. In Proc. IEEE International Conference on Communication Workshops (ICC Workshops) https://doi.org/10.1109/ICCWorkshops59551.2024.10615975 (IEEE, 2024).

  58. Xing, C., Lv, J., Luo, T. & Zhang, Z. Representation and fusion based on knowledge graph in multi modal semantic communication. IEEE Wirel. Commun. Lett. 13, 1344–1348 (2024).

    Article  Google Scholar 

  59. Zhao, Z. et al. A joint communication and computation design for semantic wireless communication with probability graph. J. Frankl. Inst. 361, 1–16 (2024).

    Article  MathSciNet  Google Scholar 

  60. Sun, S., Qin, Z., Xie, H. & Tao, X. Task-oriented scene graph-based semantic communications with adaptive channel coding. IEEE Trans. Wirel. Commun. 23, 17070–17083 (2024).

    Article  Google Scholar 

  61. Yang, M. et al. SG2SC: a generative semantic communication framework for scene understanding-oriented image transmission. In Proc. IEEE International Conference on Acoustic Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP48485.2024.10446594 (IEEE 2024).

  62. Li, C. et al. VR + HD: video semantic reconstruction from spatio-temporal scene graphs. IEEE J. Sel. Top. Signal Process. 17, 935–948 (2023).

    Article  Google Scholar 

  63. Du, Q., Duan, Y., Yang, Q., Tao, X. & Debbah, M. Object–attribute–relation representation based video semantic communication. IEEE J. Sel. Areas Commun. 43, 2446–2461 (2025).

    Article  Google Scholar 

  64. Uysal, E. et al. Semantic communications in networked systems: a data significance perspective. IEEE Netw. 36, 233–240 (2022).

    Article  Google Scholar 

  65. Zhang, G., Zhou, K., Cai, Y., Hu, Q. & Yu, G. Toward compatible semantic communication: a perspective on digital coding and modulation. IEEE Commun. Mag. 63, 156–162 (2025).

    Article  Google Scholar 

  66. Chen, B. & Zhang, J. Practical compact deep compressed sensing. IEEE Trans. Pattern Anal. Mach. Intell. 47, 1610–1626 (2025).

    Article  Google Scholar 

  67. Zhang, B., Qin, Z. & Li, G. Y. Semantic-aware image compressed sensing. In Proc. IEEE International Workshop on Machine Learning for Signal Processing (MLSP) https://doi.org/10.1109/MLSP55844.2023.10285864 (IEEE, 2023).

  68. Yu, K., He, Q., Wu, G. & Qin, Z. Partial sampling-based semantic communications. IEEE Trans. Commun. 73, 9059–9070 (2025).

    Article  Google Scholar 

  69. Zhang, B., Qin, Z. & Li, G. Y. Compression ratio learning and semantic communications for video imaging. IEEE J. Sel. Top. Signal. Process. 18, 312–324 (2024).

    Article  Google Scholar 

  70. Gong, M., Wang, S., Bi, S., Wu, Y. & Qian, L. Digital semantic communications: an alternating multi-phase training strategy with mask attack. IEEE Trans. Wirel. Commun. 25, 4452–4466 (2026).

    Article  Google Scholar 

  71. Oh, Y., Park, J., Choi, J., Park, J. & Jeon, Y.-S. Blind training for channel-adaptive digital semantic communications. IEEE Trans. Commun. 73, 11274–11290 (2025).

    Article  Google Scholar 

  72. Guo, L., Chen, W., Sun, Y. & Ai, B. Digital-SC: digital semantic communication with adaptive network split and learned non-linear quantization. IEEE Trans. Cogn. Commun. Netw. 11, 2499–2511 (2024).

    Article  Google Scholar 

  73. Xie, S. et al. Robust information bottleneck for task-oriented communication with digital modulation. IEEE J. Sel. Areas Commun. 41, 2577–2591 (2023).

    Article  Google Scholar 

  74. Huh, Y., Seo, H. & Choi, W. Universal joint source-channel coding for modulation-agnostic semantic communication. IEEE J. Sel. Areas Commun. 43, 2560–2574 (2025).

    Article  Google Scholar 

  75. Huang, J., Yuan, K., Huang, C. & Huang, K. D2-JSCC: digital deep joint source-channel coding for semantic communications. IEEE J. Sel. Areas Commun. 43, 1246–1261 (2025).

    Article  Google Scholar 

  76. Tung, T. Y., Kurka, D. B., Jankowski, M. & Gunduz, D. DeepJSCC-Q: constellation constrained deep joint source-channel coding. IEEE J. Sel. Areas Inf. Theory. 3, 720–731 (2022).

    Article  Google Scholar 

  77. Yang, P., Zhang, G. & Cai, Y. Digital wireless image transmission via distribution matching. In Proc. IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) https://doi.org/10.1109/PIMRC59610.2024.10817327 (IEEE, 2024).

  78. Zhang, G., Yang, P., Cai, Y., Hu, Q. & Yu, G. From analog to digital: Multi-order digital joint coding–modulation for semantic communication. IEEE Trans. Commun. 73, 4257–4271 (2025). Designs a digital semantic communication method with adaptive rate, modulation order and channel awareness.

    Article  Google Scholar 

  79. Jang, E., Gu, S. & Poole, B. Categorical reparameterization with Gumbel–softmax. In Proc. International Conference on Learning Representation https://openreview.net/forum?id=rkE3y85ee (ICLR, 2017).

  80. Bo, Y., Duan, Y., Shao, S. & Tao, M. Joint coding–modulation for digital semantic communications via variational autoencoder. IEEE Trans. Commun. 72, 5626–5640 (2024).

    Article  Google Scholar 

  81. Forney, G. D., Gallager, R., Lang, G., Longstaff, F. & Qureshi, S. Efficient modulation for band-limited channels. IEEE J. Sel. Areas Commun. 2, 632–647 (1984).

    Article  Google Scholar 

  82. Qin, Z. et al. Ten challenges in semantic communications. China Commun. 22, 24–43 (2025).

    Article  Google Scholar 

  83. Fu, Y., Wang, X. & Fang, F. Multi-objective multi-dimensional resource allocation for categorized QoS provisioning in beyond 5G and 6G radio access networks. IEEE Trans. Commun. 72, 1790–1803 (2024).

    Article  Google Scholar 

  84. Yan, L. et al. QoE-based semantic-aware resource allocation for multi-task networks. IEEE Trans. Wirel. Commun. 23, 11958–11971 (2024).

    Article  Google Scholar 

  85. Chen, F., Wang, X., Fan, X. & Huang, L. QoE-oriented hybrid semantic and bit communications under mismatched knowledge. IEEE Trans. Commun. 73, 6036–6051 (2025).

    Article  Google Scholar 

  86. Shao, Y., Cao, Q. & Gündüz, D. A theory of semantic communication. IEEE Trans. Mob. Comput. 23, 12211–12228 (2024).

    Article  Google Scholar 

  87. Xie, H., Qin, Z., Li, G. Y. & Juang, B.-H. Deep learning enabled semantic communication systems. IEEE Trans. Signal Process. 69, 2663–2675 (2021).

    Article  MathSciNet  Google Scholar 

  88. Zhang, R., Isola, P., Efros, A. A., Shechtman, E. & Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00068 (IEEE, 2018).

  89. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proc. Advances in Neural Information Processing Systems 30, 6626–6637 (NeurIPS, 2017).

  90. Zhao, J. et al. Resource allocation for semantic-aware wireless communication networks with imperfect CSI. IEEE Trans. Wirel. Commun. 24, 9840–9853 (2025).

    Article  Google Scholar 

  91. Du, H. et al. Attention-aware resource allocation and QoE analysis for metaverse xURLLC services. IEEE J. Sel. Areas Commun. 41, 2158–2175 (2023).

    Article  Google Scholar 

  92. Li, Y. et al. QoE-aware latency optimization in semantic transmission empowered edge-assisted AR. IEEE Trans. Veh. Technol. 74, 14432–14444 (2025).

    Article  Google Scholar 

  93. Jiang, Y., Kang, J., Ge, X., Niyato, D. & Xiong, Z. QoE analysis and resource allocation for wireless metaverse services. IEEE Trans. Commun. 71, 4735–4750 (2023).

    Article  Google Scholar 

  94. Liu, J. et al. Optimizing resource allocation for multi-modal semantic communication in mobile AIGC networks: a diffusion-based game approach. IEEE Trans. Cogn. Commun. Netw. 11, 3346–3360 (2025).

    Article  Google Scholar 

  95. Zheng, M. et al. Reliable semantic communication with QoE driven resource scheduling for UAV-assisted MEC. IEEE Trans. Veh. Technol. 74, 11484–11489 (2025).

    Article  Google Scholar 

  96. Hu, H. et al. Resource allocation for multi-modal semantic communication in UAV collaborative networks. IEEE Trans. Commun. 73, 7599–7616 (2025).

    Article  Google Scholar 

  97. Minani, F. et al. Resource allocation for semantic communication in NTN downlinks. IEEE Trans. Cogn. Commun. Netw. 12, 524–544 (2025).

    Article  Google Scholar 

  98. Naveed, H. et al. A comprehensive overview of large language models. ACM Trans. Intell. Syst. Technol. 16, 1–72 (2025).

    Article  Google Scholar 

  99. Liang, C. et al. Generative AI-driven semantic communication networks: architecture, technologies, and applications. IEEE Trans. Cogn. Commun. Netw. 11, 27–47 (2025).

    Article  Google Scholar 

  100. Fei, N. et al. Towards artificial general intelligence via a multimodal foundation model. Nat. Commun. 13, 1–13 (2022).

    Article  Google Scholar 

  101. Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems 30, 5998–6008 (NeurIPS, 2017).

  102. Zhou, H.-Y. et al. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat. Biomed. Eng. 7, 743–755 (2023).

    Article  Google Scholar 

  103. Yang, W. et al. Rethinking generative semantic communication for multi-user systems with large language models. IEEE Wirel. Commun. 32, 170–178 (2025).

    Article  Google Scholar 

  104. Jiang, P. et al. Semantic communications using foundation models: design approaches and open issues. IEEE Wirel. Commun. 31, 76–84 (2024).

    Article  Google Scholar 

  105. Liu, B., Liu, X., Gao, S., Cheng, X. & Yang, L. LLM4CP: adapting large language models for channel prediction. J. Commun. Inform. Networks. 9, 113–125 (2024).

    Article  Google Scholar 

  106. Shao, J. et al. WirelessLLM: empowering large language models towards wireless intelligence. J. Commun. Inform. Networks. 9, 99–112 (2024).

    Article  Google Scholar 

  107. Lin, B. et al. Video-LLaVA: learning united visual representation by alignment before projection. In Proc. Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/2024.emnlp-main.342 (EMNLP, 2024).

  108. Li, J., Li, D., Savarese, S. & Hoi, S. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. International Conference on Machine Learning https://openreview.net/forum?id=KU9UojoX7U (ICML, 2023).

  109. Yin, H. et al. Generative video semantic communication via multimodal semantic fusion with large models. IEEE Trans. Veh. Technol. 75, 1701–1706 (2025).

    Article  Google Scholar 

  110. Zhao, Y., Yue, Y., Hou, S., Cheng, B. & Huang, Y. LaMoSC: large language model-driven semantic communication system for visual transmission. IEEE Trans. Cogn. Commun. Netw. 10, 2005–2018 (2024).

    Article  Google Scholar 

  111. Jiang, F. et al. Large AI model empowered multimodal semantic communications. IEEE Commun. Mag. 63, 76–82 (2025).

    Article  Google Scholar 

  112. Jiang, F. et al. Large generative model-assisted talking-face semantic communication system. IEEE J. Sel. Areas Commun. 43, 4152–4165 (2025).

    Article  Google Scholar 

  113. Tian, Y. et al. Large speech model enabled semantic communication. IEEE Trans. Mob. Comput. 25, 11600–11614 (2026).

    Article  Google Scholar 

  114. Du, B., Du, H., Niyato, D. & Li, R. Task-oriented semantic communication in large multimodal models based vehicle networks. IEEE Trans. Mob. Comput. 24, 9822–9836 (2025). Designs a multimodal cross-attention mechanism to capture user-attended regions in images and allocate communication resources accordingly.

    Article  Google Scholar 

  115. Liu, C., Guo, C., Yang, Y., Chen, M. & Quek, T. Q. Lightweight task-oriented semantic communication empowered by large-scale AI models. IEEE Trans. Veh. Technol. 74, 14823–14827 (2025).

    Article  Google Scholar 

  116. Valenzise, G., Martin, A., Zerman, E. & Ozcinar, C. Immersive Video Technologies (Academic Press, 2022).

  117. Qi, C. R., Yi, L., Su, H. & Guibas, L. J. PointNet++: deep hierarchical feature learning on point sets in a metric space. In Proc. Advances in Neural Information Processing Systems 30, 5099–5108 (NeurIPS, 2017).

  118. Zhao, H., Jiang, L., Jia, J., Torr, P. H. S. & Koltun, V. Point transformer. In Proc. International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.01595 (IEEE, 2021).

  119. Bian, C., Shao, Y. & Gündüz, D. Over-the-air learning-based geometry point cloud transmission. IEEE J. Sel. Areas Commun. 43, 4042–4058 (2026). Demonstrates that semantic transmission of point clouds can achieve superior or comparable performance to conventional separate coding schemes, while avoiding cliff and levelling effects.

    Article  Google Scholar 

  120. Ying, J., Qin, Z., Feng, Y., Wang, L. & Tao, X. Joint semantic-channel coding and modulation for token communications. IEEE Trans. Wirel. Commun. 25, 8179–8193 (2026).

    Article  Google Scholar 

  121. Yang, D. & Qin, Z. Generalizable 3D Gaussian splatting enabled immersive video communications with semantic coding. In Proc. 3rd ACM Workshop on Mobile Immersive Computing and Networked Systems (ImmerCom) https://doi.org/10.1145/3742889.3768350 (ACM, 2025).

  122. Tang, H. et al. HGC-Avatar: Hierarchical Gaussian compression for streamable dynamic 3D avatars. In Proc. 33rd ACM International Conference on Multimedia (MM) https://doi.org/10.1145/3746027.3755317 (ACM, 2025).

  123. Kerbl, B., Kopanas, G., Leimkühler, T. & Drettakis, G. 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42, 1–14 (2023).

    Article  Google Scholar 

  124. Kodheli, O. et al. Satellite communications in the new space era: a survey and future challenges. IEEE Commun. Surv. Tutor. 23, 70–109 (2021).

    Article  Google Scholar 

  125. Yin, Y., Liu, S., Wen, D., Wu, Y. & Shi, Y. Joint source and channel coding for multi-modal satellite-to-ground semantic communications. In Proc. 2025 IEEE Wireless Communications and Networking Conference (WCNC) https://doi.org/10.1109/WCNC61545.2025.10978250 (IEEE, 2025).

  126. Chen, W. et al. Free space optical semantic communication for satellite remote sensing image transmission. IEEE Trans. Commun. 73, 9168–9183 (2025).

    Article  Google Scholar 

  127. Nguyen, L. X. et al. Semantic communication enabled 6G-NTN framework: a novel denoising and gateway hop integration mechanism. IEEE Trans. Wirel. Commun. 24, 10149–10165 (2025).

    Article  Google Scholar 

  128. Hussein, N. H., Yaw, C. T., Koh, S. P., Tiong, S. K. & Chong, K. H. A comprehensive survey on vehicular networking: communications, applications, challenges, and upcoming research directions. IEEE Access. 10, 86127–86180 (2022).

    Article  Google Scholar 

  129. Wan, Z. et al. A semantic communication approach for multiscene target detection in intelligent vehicle networks. IEEE Internet Things J. 11, 35877–35890 (2024).

    Article  Google Scholar 

  130. Yuan, Y. et al. Learning-based task-centric multi-user semantic communication solution for vehicle networks. IEEE Trans. Veh. Technol. 74, 9328–9342 (2025).

    Article  Google Scholar 

  131. Yang, W., Chi, X., Zhao, L., Xiong, Z. & Jiang, W. Task-driven semantic-aware green cooperative transmission strategy for vehicular networks. IEEE Trans. Commun. 71, 5783–5798 (2023).

    Article  Google Scholar 

  132. Aouedi, O. et al. A survey on intelligent Internet of Things: applications, security, privacy, and future directions. IEEE Commun. Surv. Tutor. 27, 1238–1292 (2025).

    Article  Google Scholar 

  133. Xie, H. & Qin, Z. A lite distributed semantic communication system for Internet of Things. IEEE J. Sel. Areas Commun. 39, 142–153 (2021).

    Article  Google Scholar 

  134. Tan, W., Wang, Y., Liu, L., Wang, X. & Ding, T. Adaptive federated deep learning-based semantic communication in the social Internet of Things. IEEE Internet Things J. 12, 34921–34930 (2025).

    Article  Google Scholar 

  135. Zeng, W. et al. A secure and efficient distributed semantic communication system for heterogeneous Internet of Things. IEEE Trans. Mob. Comput. 25, 10370–10384 (2026).

    Article  Google Scholar 

  136. Shao, K. et al. A survey of token compression for efficient multimodal large language models. Trans. Mach. Learn. Res. 1–42 (2026).

  137. Qiao, L. et al. Token communications: a large model-driven framework for cross-modal context-aware semantic communications. IEEE Wirel. Commun. 32, 80–88 (2025). Proposes the concept of token communication and demonstrates its advantages in multimodal transmission.

    Article  Google Scholar 

  138. Lee, S., Park, J., Choi, J. & Park, H. Low-complexity semantic packet aggregation for token communication via lookahead search. IEEE Trans. Commun. 74, 9639–9654 (2026).

    Article  Google Scholar 

  139. Chen, W. et al. AI agent access (A3) network: an embodied, communication-aware multi-agent framework for 6G coverage. IEEE Trans. Commun. 74, 5333–5349 (2025).

    Google Scholar 

  140. Jiang, F. et al. From large AI models to agentic AI: a tutorial on future intelligent communications. IEEE J. Sel. Areas Commun. 44, 3507–3540 (2026).

    Article  Google Scholar 

  141. Lyons, J. Linguistic Semantics: An Introduction (Cambridge Univ. Press, 1995).

  142. Patterson, K., Nestor, P. J. & Rogers, T. T. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat. Rev. Neurosci. 8, 976–987 (2007).

    Article  Google Scholar 

  143. Weaver, W. Recent contributions to the mathematical theory of communication. ETC Rev. Gen. Semant. 10, 261–281 (1953).

    Google Scholar 

  144. Sullivan, G. J., Ohm, J.-R., Han, W.-J. & Wiegand, T. Overview of the high efficiency video coding (HEVC) Standard. IEEE Trans. Circuits Syst. Video Technol. 22, 1649–1668 (2012).

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Key Research and Development Program of China under grant 2023YFB2904300, in part by the National Natural Science Foundation of China (NSFC) under grant 62293484, and in part by the Beijing Natural Science Foundation under grant JR25020. The authors would also like to thank G. Ye Li for his valuable discussions on the concepts and key technical aspects of this work, as well as for his helpful comments and revisions to the manuscript.

Author information

Author notes

  1. These authors contributed equally: Zhijin Qin, Jingkai Ying.

Authors and Affiliations

  1. Department of Electronic Engineering, Tsinghua University, Beijing, China

    Zhijin Qin, Jingkai Ying & Xiaoming Tao

  2. State Key Laboratory of Space Network and Communications, Beijing, China

    Zhijin Qin, Jingkai Ying & Xiaoming Tao

  3. Wireless Technology Labs, Huawei Technologies, Ottawa, Ontario, Canada

    Wen Tong

Authors
  1. Zhijin Qin
    View author publications

    Search author on:PubMed Google Scholar

  2. Jingkai Ying
    View author publications

    Search author on:PubMed Google Scholar

  3. Xiaoming Tao
    View author publications

    Search author on:PubMed Google Scholar

  4. Wen Tong
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Z.Q. and J.Y. researched data and references for the article, wrote the article and contributed equally to this work. X.T. and W.T. contributed substantially to discussions of the content, carefully reviewed the manuscript and provided valuable comments for revision. All authors reviewed and edited the manuscript before submission.

Corresponding author

Correspondence to Zhijin Qin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Electrical Engineering thanks Feibo Jiang, Xuewen Luo and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

6G scenarios: https://www.itu.int/md/R19-WP5D-230612-TD-0905/en

Glossary

Age of information

(AoI). A measure of information freshness defined as the time elapsed since the generation of the most recently received update.

Channel bandwidth ratio

A normalized measure of communication resources relative to source data size, used to indicate how much channel bandwidth or channel use is available for transmission.

Cliff effects

A sharp degradation in communication performance that occurs when channel quality falls below a critical threshold.

Degree of confirmation

A measure of the extent to which a proposition is supported by available evidence, used in semantic theory to quantify the confirmation of semantic statements.

Energy efficiency

A measure of how much communication utility or data rate is achieved per unit of energy consumption.

Extended reality

An umbrella term that includes virtual reality, augmented reality and mixed reality.

Gumbel–Softmax

A differentiable approximation to discrete sampling that enables gradient-based training for models with categorical outputs.

Holographic communication

A communication mode that delivers immersive 3D representations of people or scenes with strong spatial realism.

Latent features

Abstract representations learned by deep learning models that capture important patterns or characteristics of input data in a latent space.

Neural codec

A codec designed using neural networks for encoding and decoding data.

Semantic entropy

A metric for measuring the amount of semantic information.

Shannon–Nyquist sampling theorem

A theorem stating that a band-limited signal can be reconstructed without loss when sampled at least twice its highest frequency.

Spectrum efficiency

The amount of information transmitted per unit bandwidth, typically measured in bits per second per hertz.

Sum-rate

The total achievable transmission rate obtained by summing the rates of multiple users or links in a communication system.

Value of information

A measure of the usefulness of received information that considers the impact of information freshness and source relevance.

Vector quantization

A data quantization technique that transforms high-dimensional vector into a small, discrete set of representative vectors called codewords.

Visual question answering

A task in which a model answers natural-language questions about visual input, such as an image or video.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, Z., Ying, J., Tao, X. et al. Multimodal semantic communications. Nat Rev Electr Eng (2026). https://doi.org/10.1038/s44287-026-00336-0

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1038/s44287-026-00336-0

  • Springer Nature Limited

Profiles

  1. Zhijin Qin View author profile