Results 251 to 260 of about 152,555 (310)

Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning

arXiv.org
Large reasoning models (LRMs) have demonstrated impressive capabilities in complex problem-solving, yet their internal reasoning mechanisms remain poorly understood.
Chen Qian, Dongrui Liu, Hao Wen, Zhengbiao Bai, Yong Liu, Jing Shao   +5 more
semanticscholar   +1 more source

NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

arXiv.org
Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-intensive diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quantization loss ...
NextStep Team Chunrui Han, Guopeng Li, Jingwei Wu, Quan Sun, Yan Cai, Yuang Peng, Zheng Ge, Deyu Zhou, Haomiao Tang, Hongyu Zhou, Kenkun Liu, Ailin Huang, Bin Wang, Changxing Miao, Deshan Sun, En Yu, Fukun Yin, Gang Yu, Hao Nie, Haoran Lv, Hanpeng Hu, Jia Wang, Jian Zhou, Jian‐Yuan Sun, Kaijun Tan, Kang An, Kangheng Lin, Liang Zhao, Mei Chen, Peng Xing, Rui Wang, Shiyu Liu, Shu-Tao Xia, Tian You, Wei Ji, Xianfang Zeng, Xin-pu Han, Xuelin Zhang, Yana Wei, Yanming Xu, Yimin Jiang, Yingming Wang, Yu Zhou, Yu Han, Zi Meng, Binxing Jiao, Daxin Jiang, Xiangyu Zhang, Yibo Zhu   +48 more
semanticscholar   +1 more source

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

International Conference on Machine Learning
Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around
Yiran Ding, L. Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang   +7 more
semanticscholar   +1 more source

Externalism and token-token identity

Philosophia, 1995
Selon la theorie de l'identite de pure forme (token-token) de l'esprit et du corps, les signes evenementiels de l'esprit, compris comme des occurrences donnees concretement et particulierement, sont identiques aux signes evenementiels physiques, compris de facon semblable. Dans le cadre de cette theorie, l'A.
openaire   +1 more source

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

arXiv.org
Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLMs), with recent innovations such as Group Relative Policy Optimization (GRPO) demonstrating exceptional effectiveness.
Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, Yunjian Xu   +6 more
semanticscholar   +1 more source

Discrete Audio Tokens: More Than a Survey!

Trans. Mach. Learn. Res.
Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, as well as competitive performance across diverse downstream tasks ...
Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, R. Marxer, B. Ramabhadran, Benjamin Elizalde, Loren Lugosch, Jinyu Li, Cem Subakan, Phil Woodland, Minje Kim, Hung-yi Lee, Shinji Watanabe, Yossi Adi, M. Ravanelli   +20 more
semanticscholar   +1 more source

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

arXiv.org
Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics.
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, Ying Shan   +8 more
semanticscholar   +1 more source

Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning

arXiv.org
We completely discard the conventional spatial prior in image representation and introduce a novel discrete visual tokenizer: Self-consistency Tokenizer (Selftok).
Bo Wang, Zhongqi Yue, Fengda Zhang, Shuo Chen, L. Bi, Junzhe Zhang, Xue Song, K. Chan, Jiachun Pan, Weijia Wu, Min Zhou, Wang Lin, Kaihang Pan, Saining Zhang, Liyu Jia, Wentao Hu, Wei Zhao, Hanwang Zhang   +17 more
semanticscholar   +1 more source

Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference

AAAI Conference on Artificial Intelligence
Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation.
Zhihang Lin, Mingbao Lin, Luxi Lin, Rongrong Ji   +3 more
semanticscholar   +1 more source

Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference

Conference on Machine Learning and Systems
Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath   +5 more
semanticscholar   +1 more source