Results 331 to 340 of about 1,322,987 (378)

Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design

International Conference on Machine Learning
Combining discrete and continuous data is an important capability for generative models. We present Discrete Flow Models (DFMs), a new flow-based model of discrete data that provides the missing link in enabling flow-based generative models to be applied
Andrew Campbell, Jason Yim, R. Barzilay, Tom Rainforth, T. Jaakkola   +4 more
semanticscholar   +1 more source

BLINK: Multimodal Large Language Models Can See but Not Perceive

European Conference on Computer Vision
We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna   +9 more
semanticscholar   +1 more source

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

Annual Meeting of the Association for Computational Linguistics
This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step ...
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig   +13 more
semanticscholar   +1 more source

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

Computer Vision and Pattern Recognition
We introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, C. Ruan, Ping Luo   +10 more
semanticscholar   +1 more source

Multimodality

AbstractThis chapter considers multimodality in Casey O’Callaghan’s strict sense of that term: a perceptual representation’s object is represented neither by a single sense modality nor merely as a collection of features each of which is represented by a single modality. By definition, multimodal representation cannot be simply a case of layering.
Denise Newfield, Sarah Crinall
openaire   +2 more sources

HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

arXiv.org
The rapid development of multimodal large language models (MLLMs), such as GPT-4V, has led to significant advancements. However, these models still face challenges in medical multimodal capabilities due to limitations in the quantity and quality of ...
Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, Benyou Wang   +11 more
semanticscholar   +1 more source

Hallucination of Multimodal Large Language Models: A Survey

arXiv.org
This survey presents a comprehensive analysis of the phenomenon of hallucination in multimodal large language models (MLLMs), also known as Large Vision-Language Models (LVLMs), which have demonstrated significant advancements and remarkable abilities in
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, Mike Zheng Shou   +6 more
semanticscholar   +1 more source

Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

arXiv.org
Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning.
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, Jifeng Dai   +10 more
semanticscholar   +1 more source

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

IEEE Transactions on Geoscience and Remote Sensing
Accurate semantic segmentation of remote sensing data plays a crucial role in the success of geoscience research and applications. Recently, multimodal fusion-based segmentation models have attracted much attention due to their outstanding performance as
Xianping Ma, Xiaokang Zhang, M. Pun, Ming Liu   +3 more
semanticscholar   +1 more source

Multimodal Interfaces

Artificial Intelligence Review, 1996
Alex Waibel, Minh Tue Vo, Paul Duchnowski, Stefan Manke   +3 more
openaire   +1 more source