The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Rangkaian gred pembawa terkini menggunakan banyak elemen rangkaian (suis, penghala) dan pelayan untuk pelbagai perkhidmatan berasaskan rangkaian (cth, video atas permintaan, permainan dalam talian) yang menuntut kualiti yang lebih tinggi dan kebolehpercayaan yang lebih baik. Data log rangkaian yang dijana daripada elemen ini, seperti syslog penghala, adalah sumber yang kaya untuk mengesan tanda-tanda kegagalan kritikal untuk mengekalkan kualiti perkhidmatan dengan cepat. Walau bagaimanapun, data log mengandungi sejumlah besar mesej teks yang ditulis dalam format tidak berstruktur dan mengandungi pelbagai jenis acara rangkaian (cth, log masuk pengendali, pautan ke bawah); oleh itu, mesej log yang benar-benar penting untuk operasi rangkaian sukar dicari secara automatik. Kami mencadangkan sistem pengesanan kegagalan proaktif untuk rangkaian berskala besar. Ia secara automatik mencari corak abnormal mesej log daripada sejumlah besar data tanpa memerlukan pengetahuan sebelumnya tentang format data yang digunakan dan boleh mengesan kegagalan kritikal sebelum ia berlaku. Untuk mengendalikan mesej log tidak berstruktur, sistem mempunyai pengekstrakan templat log dalam talian bahagian untuk mengekstrak format mesej log secara automatik. Selepas pengekstrakan templat, sistem mengaitkan kegagalan kritikal dengan data log yang muncul di hadapannya berdasarkan pembelajaran mesin yang diselia. Dengan mengaitkan setiap mesej log dengan templat log, kita boleh mencirikan corak penjanaan mesej log, seperti pecah, bukan hanya kata kunci dalam mesej log (cth ERROR, FAIL). Kami menggunakan data log sebenar yang dikumpul daripada rangkaian pengeluaran yang besar untuk mengesahkan sistem kami dan menilai sistem dalam mengesan tanda-tanda kegagalan sebenar peralatan rangkaian melalui kajian kes.
Tatsuaki KIMURA
NTT Corporation
Akio WATANABE
NTT Corporation
Tsuyoshi TOYONO
NTT Corporation
Keisuke ISHIBASHI
NTT Corporation
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Salinan
Tatsuaki KIMURA, Akio WATANABE, Tsuyoshi TOYONO, Keisuke ISHIBASHI, "Proactive Failure Detection Learning Generation Patterns of Large-Scale Network Logs" in IEICE TRANSACTIONS on Communications,
vol. E102-B, no. 2, pp. 306-316, February 2019, doi: 10.1587/transcom.2018EBP3103.
Abstract: Recent carrier-grade networks use many network elements (switches, routers) and servers for various network-based services (e.g., video on demand, online gaming) that demand higher quality and better reliability. Network log data generated from these elements, such as router syslogs, are rich sources for quickly detecting the signs of critical failures to maintain service quality. However, log data contain a large number of text messages written in an unstructured format and contain various types of network events (e.g., operator's login, link down); thus, genuinely important log messages for network operation are difficult to find automatically. We propose a proactive failure-detection system for large-scale networks. It automatically finds abnormal patterns of log messages from a massive amount of data without requiring previous knowledge of data formats used and can detect critical failures before they occur. To handle unstructured log messages, the system has an online log-template-extraction part for automatically extracting the format of a log message. After template extraction, the system associates critical failures with the log data that appeared before them on the basis of supervised machine learning. By associating each log message with a log template, we can characterize the generation patterns of log messages, such as burstiness, not just the keywords in log messages (e.g. ERROR, FAIL). We used real log data collected from a large production network to validate our system and evaluated the system in detecting signs of actual failures of network equipment through a case study.
URL: https://global.ieice.org/en_transactions/communications/10.1587/transcom.2018EBP3103/_p
Salinan
@ARTICLE{e102-b_2_306,
author={Tatsuaki KIMURA, Akio WATANABE, Tsuyoshi TOYONO, Keisuke ISHIBASHI, },
journal={IEICE TRANSACTIONS on Communications},
title={Proactive Failure Detection Learning Generation Patterns of Large-Scale Network Logs},
year={2019},
volume={E102-B},
number={2},
pages={306-316},
abstract={Recent carrier-grade networks use many network elements (switches, routers) and servers for various network-based services (e.g., video on demand, online gaming) that demand higher quality and better reliability. Network log data generated from these elements, such as router syslogs, are rich sources for quickly detecting the signs of critical failures to maintain service quality. However, log data contain a large number of text messages written in an unstructured format and contain various types of network events (e.g., operator's login, link down); thus, genuinely important log messages for network operation are difficult to find automatically. We propose a proactive failure-detection system for large-scale networks. It automatically finds abnormal patterns of log messages from a massive amount of data without requiring previous knowledge of data formats used and can detect critical failures before they occur. To handle unstructured log messages, the system has an online log-template-extraction part for automatically extracting the format of a log message. After template extraction, the system associates critical failures with the log data that appeared before them on the basis of supervised machine learning. By associating each log message with a log template, we can characterize the generation patterns of log messages, such as burstiness, not just the keywords in log messages (e.g. ERROR, FAIL). We used real log data collected from a large production network to validate our system and evaluated the system in detecting signs of actual failures of network equipment through a case study.},
keywords={},
doi={10.1587/transcom.2018EBP3103},
ISSN={1745-1345},
month={February},}
Salinan
TY - JOUR
TI - Proactive Failure Detection Learning Generation Patterns of Large-Scale Network Logs
T2 - IEICE TRANSACTIONS on Communications
SP - 306
EP - 316
AU - Tatsuaki KIMURA
AU - Akio WATANABE
AU - Tsuyoshi TOYONO
AU - Keisuke ISHIBASHI
PY - 2019
DO - 10.1587/transcom.2018EBP3103
JO - IEICE TRANSACTIONS on Communications
SN - 1745-1345
VL - E102-B
IS - 2
JA - IEICE TRANSACTIONS on Communications
Y1 - February 2019
AB - Recent carrier-grade networks use many network elements (switches, routers) and servers for various network-based services (e.g., video on demand, online gaming) that demand higher quality and better reliability. Network log data generated from these elements, such as router syslogs, are rich sources for quickly detecting the signs of critical failures to maintain service quality. However, log data contain a large number of text messages written in an unstructured format and contain various types of network events (e.g., operator's login, link down); thus, genuinely important log messages for network operation are difficult to find automatically. We propose a proactive failure-detection system for large-scale networks. It automatically finds abnormal patterns of log messages from a massive amount of data without requiring previous knowledge of data formats used and can detect critical failures before they occur. To handle unstructured log messages, the system has an online log-template-extraction part for automatically extracting the format of a log message. After template extraction, the system associates critical failures with the log data that appeared before them on the basis of supervised machine learning. By associating each log message with a log template, we can characterize the generation patterns of log messages, such as burstiness, not just the keywords in log messages (e.g. ERROR, FAIL). We used real log data collected from a large production network to validate our system and evaluated the system in detecting signs of actual failures of network equipment through a case study.
ER -