The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Untuk banyak bidang dalam kehidupan sebenar, ramalan siri masa adalah penting. Kajian terbaru menunjukkan Transformer mempunyai kelebihan tertentu apabila menangani masalah sedemikian, terutamanya apabila menangani masalah input masa jujukan panjang dan ramalan masa jujukan panjang. Untuk meningkatkan kecekapan dan kestabilan tempatan Transformer, kajian ini menggabungkan Transformer dan CNN dengan struktur yang berbeza. Walau bagaimanapun, model rangkaian ramalan siri masa sebelumnya berdasarkan Transformer tidak boleh menggunakan sepenuhnya CNN dan ia tidak digunakan dalam gabungan yang lebih baik daripada kedua-duanya. Sebagai tindak balas kepada masalah ini dalam ramalan siri masa, kami mencadangkan algoritma ramalan siri masa berdasarkan Transformer konvolusi. (1) Mekanisme perhatian ES: Menggabungkan perhatian luaran dengan mekanisme perhatian diri tradisional melalui rangkaian dua cawangan, kos pengiraan mekanisme perhatian diri dikurangkan, dan ketepatan ramalan yang lebih tinggi diperolehi. (2) Blok dipertingkatkan kekerapan: Blok Dipertingkatkan Frekuensi ditambah di hadapan modul ESAttention, yang boleh menangkap struktur penting dalam siri masa melalui pemetaan domain frekuensi. (3) Konvolusi diluaskan sebab: Modul mekanisme perhatian diri disambungkan dengan menggantikan lapisan lilitan piawai tradisional dengan lapisan lilitan diluaskan sebab, supaya ia memperoleh medan penerimaan pertumbuhan secara eksponen tanpa meningkatkan penggunaan pengiraan. (4) Gabungan ciri berbilang lapisan: Output modul mekanisme perhatian kendiri yang berbeza diekstrak, dan lapisan konvolusi digunakan untuk melaraskan saiz peta ciri untuk gabungan. Maklumat ciri yang lebih halus diperolehi pada kos pengiraan yang boleh diabaikan. Eksperimen pada set data dunia sebenar menunjukkan bahawa struktur model ramalan rangkaian siri masa yang dicadangkan dalam kertas kerja ini boleh meningkatkan prestasi ramalan masa nyata model Transformer terkini yang terkini, dan kos pengiraan dan ingatan jauh lebih rendah. Berbanding dengan algoritma sebelumnya, algoritma yang dicadangkan telah mencapai peningkatan prestasi yang lebih baik dalam kedua-dua keberkesanan dan ketepatan ramalan.
Na WANG
Nanjing University of Aeronautics and Astronautics,Nanjing Audit University Jinshen College
Xianglian ZHAO
Nanjing University of Aeronautics and Astronautics
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Salinan
Na WANG, Xianglian ZHAO, "Time Series Forecasting Based on Convolution Transformer" in IEICE TRANSACTIONS on Information,
vol. E106-D, no. 5, pp. 976-985, May 2023, doi: 10.1587/transinf.2022EDP7136.
Abstract: For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2022EDP7136/_p
Salinan
@ARTICLE{e106-d_5_976,
author={Na WANG, Xianglian ZHAO, },
journal={IEICE TRANSACTIONS on Information},
title={Time Series Forecasting Based on Convolution Transformer},
year={2023},
volume={E106-D},
number={5},
pages={976-985},
abstract={For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.},
keywords={},
doi={10.1587/transinf.2022EDP7136},
ISSN={1745-1361},
month={May},}
Salinan
TY - JOUR
TI - Time Series Forecasting Based on Convolution Transformer
T2 - IEICE TRANSACTIONS on Information
SP - 976
EP - 985
AU - Na WANG
AU - Xianglian ZHAO
PY - 2023
DO - 10.1587/transinf.2022EDP7136
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E106-D
IS - 5
JA - IEICE TRANSACTIONS on Information
Y1 - May 2023
AB - For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.
ER -