The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Log sistem berguna untuk memahami status dan mengesan kerosakan dalam rangkaian berskala besar. Walau bagaimanapun, disebabkan kepelbagaian dan jumlah log ini, analisis log memerlukan banyak masa dan usaha. Dalam makalah ini, kami mencadangkan kaedah pengesanan anomali peristiwa log untuk rangkaian berskala besar tanpa pra-pemprosesan dan pengekstrakan ciri. Idea utama adalah untuk membenamkan sejumlah besar data yang pelbagai ke dalam keadaan tersembunyi dengan menggunakan pembolehubah terpendam. Kami menilai kaedah kami dengan log sistem selama 12 bulan yang diperoleh daripada rangkaian akademik seluruh negara di Jepun. Melalui perbandingan dengan pengesanan pecah univariat Kleinberg dan analisis multivariat tradisional (iaitu, PCA), kami menunjukkan bahawa kaedah yang dicadangkan kami mencapai 14.5% penarikan semula lebih tinggi dan ketepatan 3% lebih tinggi daripada PCA. Kajian kes menunjukkan anomali yang dikesan adalah maklumat yang berkesan untuk menyelesaikan masalah kerosakan sistem rangkaian.
Kazuki OTOMO
the University of Tokyo
Satoru KOBAYASHI
National Institute of Informatics
Kensuke FUKUDA
National Institute of Informatics,Sokendai
Hiroshi ESAKI
the University of Tokyo
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Salinan
Kazuki OTOMO, Satoru KOBAYASHI, Kensuke FUKUDA, Hiroshi ESAKI, "Latent Variable Based Anomaly Detection in Network System Logs" in IEICE TRANSACTIONS on Information,
vol. E102-D, no. 9, pp. 1644-1652, September 2019, doi: 10.1587/transinf.2018OFP0007.
Abstract: System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2018OFP0007/_p
Salinan
@ARTICLE{e102-d_9_1644,
author={Kazuki OTOMO, Satoru KOBAYASHI, Kensuke FUKUDA, Hiroshi ESAKI, },
journal={IEICE TRANSACTIONS on Information},
title={Latent Variable Based Anomaly Detection in Network System Logs},
year={2019},
volume={E102-D},
number={9},
pages={1644-1652},
abstract={System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.},
keywords={},
doi={10.1587/transinf.2018OFP0007},
ISSN={1745-1361},
month={September},}
Salinan
TY - JOUR
TI - Latent Variable Based Anomaly Detection in Network System Logs
T2 - IEICE TRANSACTIONS on Information
SP - 1644
EP - 1652
AU - Kazuki OTOMO
AU - Satoru KOBAYASHI
AU - Kensuke FUKUDA
AU - Hiroshi ESAKI
PY - 2019
DO - 10.1587/transinf.2018OFP0007
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E102-D
IS - 9
JA - IEICE TRANSACTIONS on Information
Y1 - September 2019
AB - System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.
ER -