The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Menguruskan sistem SaaS memerlukan pentadbir memantau dan menganalisis pelbagai jenis data log yang dikumpul daripada pelbagai komponen seperti aplikasi dan sumber IT. Sistem pemantauan bersepadu, didayakan dengan stor data yang mampu menyimpan dan pemprosesan berasaskan pertanyaan bagi data separa berstruktur (cth, NOSQL - beberapa pangkalan data dokumen tertentu), ialah penyelesaian yang menjanjikan yang boleh menyimpan dan menanyakan sebarang jenis data log dengan satu set bersatu anak tetingkap pengurusan. Walau bagaimanapun, disebabkan oleh peningkatan skala sistem SaaS dan hayat perkhidmatannya yang panjang, sistem pemantauan bersepadu telah menghadapi masalah dalam masa tindak balas analisis log dan penggunaan storan untuk log. Dalam kerja ini, kami menyelesaikan masalah dengan membangunkan kaedah pengurusan log yang cekap untuk sistem SaaS. Pemerhatian empirikal kami adalah bahawa masalah adalah terutamanya berasal daripada pemprosesan log yang tidak selektif bagi stor data, sedangkan perlu ada heterogen dalam data log yang boleh kami manfaatkan untuk pengurusan log yang cekap. Berdasarkan pemerhatian ini, kami mula-mula mengesahkan cerapan ini dengan menyiasat corak penggunaan data log secara kuantitatif dengan set data sebenar sejarah akses log yang diperoleh daripada sistem SaaS yang menyediakan perkhidmatan kepada puluhan ribu pengguna perusahaan sepanjang lebih daripada 1.5 tahun . Kami menunjukkan bahawa terdapat heterogeniti dalam tempoh pengekalan log yang diperlukan, masa tindak balas analisis log, dan jumlah data, dan heterogeniti bergantung pada kategori data log dan senario analisisnya. Berbekalkan bukti kepelbagaian dalam data log dan corak penggunaan yang ditemui daripada penyiasatan, kami mereka bentuk metodologi pengurusan data log yang sedar konteks, ciri utamanya adalah untuk pra-cache secara spekulatif hasil analisis log dan mengarkib secara proaktif data log, bergantung pada kategori data log dan senario analisis. Penilaian dengan pelaksanaan prototaip menunjukkan kaedah yang dicadangkan mengurangkan masa tindak balas sebanyak 47% berbanding kaedah konvensional dan penggunaan storan sebanyak lebih kurang 40% berbanding data log asal.
Tatsuya SATO
Hitachi Ltd.
Yosuke HIMURA
Hitachi Ltd.
Yoshiko YASUDA
Hitachi Ltd.
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Salinan
Tatsuya SATO, Yosuke HIMURA, Yoshiko YASUDA, "Evidence-Based Context-Aware Log Data Management for Integrated Monitoring System" in IEICE TRANSACTIONS on Communications,
vol. E101-B, no. 9, pp. 1997-2006, September 2018, doi: 10.1587/transcom.2017EBP3396.
Abstract: Managing SaaS systems requires administrators to monitor and analyze diverse types of log data collected from a variety of components such as applications and IT resources. Integrated monitoring systems, enabled with datastore capable of storing and query-based processing of semi-structured data (e.g., NOSQL - some specific document database), is a promising solution that can store and query any type of log data with a single unified set of management panes. However, due to the increasing scale of SaaS systems and their long service lives, integrated monitoring systems have faced the problems in response times of log analysis and storage consumption for logs. In this present work, we solve the problems by developing an efficient log management method for SaaS systems. Our empirical observation is that the problems are primarily derived from the unselective log processing of datastore, whereas there should be heterogeneities in log data that we can take advantage of for efficient log management. Based on this observation, we first confirm this insight by investigating the usage patterns of log data in a quantitative manner with an actual dataset of log access histories obtained from a SaaS system serving tens of thousands of enterprise users over the course of more than 1.5 years. We show that there are heterogeneities in required retention period of logs, response time of log analysis, and amount of data, and the heterogeneities depend on log data category and its analysis scenario. Armed with the evidence of the heterogeneities in log data and the usage patterns found from the investigation, we design a methodology of context-aware log data management, key features of which are to speculatively pre-cache the result of log analysis and to proactively archive log data, depending on log data category and analysis scenario. Evaluation with a prototype implementation shows that the proposed method reduces the response time by 47% compared to a conventional method and the storage consumption by approximately 40% compared to the original log data.
URL: https://global.ieice.org/en_transactions/communications/10.1587/transcom.2017EBP3396/_p
Salinan
@ARTICLE{e101-b_9_1997,
author={Tatsuya SATO, Yosuke HIMURA, Yoshiko YASUDA, },
journal={IEICE TRANSACTIONS on Communications},
title={Evidence-Based Context-Aware Log Data Management for Integrated Monitoring System},
year={2018},
volume={E101-B},
number={9},
pages={1997-2006},
abstract={Managing SaaS systems requires administrators to monitor and analyze diverse types of log data collected from a variety of components such as applications and IT resources. Integrated monitoring systems, enabled with datastore capable of storing and query-based processing of semi-structured data (e.g., NOSQL - some specific document database), is a promising solution that can store and query any type of log data with a single unified set of management panes. However, due to the increasing scale of SaaS systems and their long service lives, integrated monitoring systems have faced the problems in response times of log analysis and storage consumption for logs. In this present work, we solve the problems by developing an efficient log management method for SaaS systems. Our empirical observation is that the problems are primarily derived from the unselective log processing of datastore, whereas there should be heterogeneities in log data that we can take advantage of for efficient log management. Based on this observation, we first confirm this insight by investigating the usage patterns of log data in a quantitative manner with an actual dataset of log access histories obtained from a SaaS system serving tens of thousands of enterprise users over the course of more than 1.5 years. We show that there are heterogeneities in required retention period of logs, response time of log analysis, and amount of data, and the heterogeneities depend on log data category and its analysis scenario. Armed with the evidence of the heterogeneities in log data and the usage patterns found from the investigation, we design a methodology of context-aware log data management, key features of which are to speculatively pre-cache the result of log analysis and to proactively archive log data, depending on log data category and analysis scenario. Evaluation with a prototype implementation shows that the proposed method reduces the response time by 47% compared to a conventional method and the storage consumption by approximately 40% compared to the original log data.},
keywords={},
doi={10.1587/transcom.2017EBP3396},
ISSN={1745-1345},
month={September},}
Salinan
TY - JOUR
TI - Evidence-Based Context-Aware Log Data Management for Integrated Monitoring System
T2 - IEICE TRANSACTIONS on Communications
SP - 1997
EP - 2006
AU - Tatsuya SATO
AU - Yosuke HIMURA
AU - Yoshiko YASUDA
PY - 2018
DO - 10.1587/transcom.2017EBP3396
JO - IEICE TRANSACTIONS on Communications
SN - 1745-1345
VL - E101-B
IS - 9
JA - IEICE TRANSACTIONS on Communications
Y1 - September 2018
AB - Managing SaaS systems requires administrators to monitor and analyze diverse types of log data collected from a variety of components such as applications and IT resources. Integrated monitoring systems, enabled with datastore capable of storing and query-based processing of semi-structured data (e.g., NOSQL - some specific document database), is a promising solution that can store and query any type of log data with a single unified set of management panes. However, due to the increasing scale of SaaS systems and their long service lives, integrated monitoring systems have faced the problems in response times of log analysis and storage consumption for logs. In this present work, we solve the problems by developing an efficient log management method for SaaS systems. Our empirical observation is that the problems are primarily derived from the unselective log processing of datastore, whereas there should be heterogeneities in log data that we can take advantage of for efficient log management. Based on this observation, we first confirm this insight by investigating the usage patterns of log data in a quantitative manner with an actual dataset of log access histories obtained from a SaaS system serving tens of thousands of enterprise users over the course of more than 1.5 years. We show that there are heterogeneities in required retention period of logs, response time of log analysis, and amount of data, and the heterogeneities depend on log data category and its analysis scenario. Armed with the evidence of the heterogeneities in log data and the usage patterns found from the investigation, we design a methodology of context-aware log data management, key features of which are to speculatively pre-cache the result of log analysis and to proactively archive log data, depending on log data category and analysis scenario. Evaluation with a prototype implementation shows that the proposed method reduces the response time by 47% compared to a conventional method and the storage consumption by approximately 40% compared to the original log data.
ER -