Results 31 to 40 of about 391,322 (253)

Grouped self-attention mechanism for a memory-efficient Transformer

CoRR, 2022
10 pages, 3 figures, under review as a conference paper at ICLR ...
Bumjun Jung, Yusuke Mukuta, Tatsuya Harada   +2 more
openaire   +2 more sources

Time Series Imputation Based on Self-Attention Mechanism [PDF]

Jisuanji gongcheng
As core data for maritime traffic, ship trajectory data can be used for trajectory prediction, early warning, and other tasks with pronounced temporal characteristics.
XU Lei, ZENG Yan, YUAN Junfeng, YUE Lupeng, YIN Yuyu, ZHANG Jilin, XUE Meiting, HAN Meng
doaj   +1 more source

Parallel Scheduling Self-attention Mechanism: Generalization and Optimization

CoRR, 2020
Over the past few years, self-attention is shining in the field of deep learning, especially in the domain of natural language processing(NLP). Its impressive effectiveness, along with ubiquitous implementations, have aroused our interest in efficiently scheduling the data-flow of corresponding computations onto architectures with many computing units ...
Mingfei Yu, Masahiro Fujita
openaire   +2 more sources

Self-attention mechanism-based CSI eigenvector feedback for massive MIMO

Dianxin kexue, 2023
Massive multiple-input multiple-output (MIMO) system can provide satisfying gain of spectrum efficiency for 5G and future wireless communication systems.In frequency-division duplex (FDD) mode, downlink channel state information (CSI) needs to be ...
Bei YANG, Xin LIANG, Hang YIN, Zheng JIANG, Xiaoming SHE   +4 more
doaj   +2 more sources

Multimodal Fusion Method Based on Self-Attention Mechanism

Wireless Communications and Mobile Computing, 2020
Multimodal fusion is one of the popular research directions of multimodal research, and it is also an emerging research field of artificial intelligence. Multimodal fusion is aimed at taking advantage of the complementarity of heterogeneous data and providing reliable classification for the model.
Hu Zhu, Ze Wang, Yu Shi, Yingying Hua, Guoxia Xu, Lizhen Deng   +5 more
openaire   +1 more source

Fine-tune BERT with Sparse Self-Attention Mechanism [PDF]

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019
In this paper, we develop a novel Sparse Self-Attention Fine-tuning model (referred as SSAF) which integrates sparsity into self-attention mechanism to enhance the fine-tuning performance of BERT. In particular, sparsity is introduced into the self-attention by replacing softmax function with a controllable sparse transformation when fine-tuning with ...
Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang   +3 more
openaire   +1 more source

Heart Sound Classification Using Bi⁃LSTM and Self⁃attention Mechanism

Shuju Caiji Yu Chuli
Heart sound auscultation is an effective diagnostic method for early screening of heart disease. In order to improve the performance of abnormal heart sound detection， this paper proposes a heart sound classification algorithm based on bi-directional ...
LU Guanming, LI Qijian, LU Junhe, QI Jirong, ZHAO Yuhang, WANG Yang, WEI Jinsheng   +6 more
doaj   +1 more source

A Novel Clothing Attribute Representation Network-Based Self-Attention Mechanism

IEEE Access, 2020
As highly increasing of on-line fashions retail industry, automatic recognition and representation of clothing items have huge potentials. With the help of deep learning methods, many clothing attribute representation models have been proposed.
Yutong Chun, Chuansheng Wang, Mingke He
doaj   +1 more source

Self‐attention mechanism for distributed compressive sensing

Electronics Letters, 2022
Abstract This letter addresses the joint sparse recovery problem with multiple measurement vectors in compressive sensing (CS). In this letter, it is assumed that the sparse vectors are related to each other but the specific dependencies are unknown.
Feng Shu, Linghua Zhang, Yin Ding, Qin Cheng, Xu Wang   +4 more
openaire   +1 more source

Mechanics of Next Token Prediction with Self-Attention

CoRR
Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revolutionary advances in natural language processing. Underlying this success is the self-attention mechanism.
Yingcong Li, Yixiao Huang 0004, Muhammed Emrullah Ildiz, Ankit Singh Rawat, Samet Oymak   +4 more
openaire   +3 more sources