Normalization of Unstructured Log Data into Streams of Structured Event Objects

Investor logo

Warning

This publication doesn't include Faculty of Sports Studies. It includes Institute of Computer Science. Official publication website can be found on muni.cz.
Authors

TOVARŇÁK Daniel PITNER Tomáš

Year of publication 2019
Type Article in Proceedings
Conference 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM)
MU Faculty or unit

Institute of Computer Science

Citation
Web
Keywords log management; logging; data integration; normalization; stream processing; monitoring
Attached files
Description Monitoring plays a crucial role in the operation of any sizeable distributed IT infrastructure. Whether it is a university network or cloud datacenter, monitoring information is continuously used in a wide spectrum of ways ranging from mission-critical jobs, e.g. accounting or incident handling, to equally important development-related tasks, e.g. debugging or fault-detection. Whilst pursuing a novel vision of new-generation event-driven monitoring systems, we have identified that a particularly rich source of monitoring information, computer logs, is also one of the most problematic in terms of automated processing. Log data are predominantly generated in an ad-hoc manner using a variety of incompatible formats with the most important pieces of information, i.e. log messages, in the form of unstructured strings. This clashes with our long-term goal of designing a system enabling its users to transparently define real-time continuous queries over homogeneous streams of properly defined monitoring event objects with explicitly described structure. Our goal is to bridge this gap by normalizing the poorly structured log data into streams of structured event objects. The combined challenge of this goal is structuring the log data, whilst considering the high velocity with which they are generated in modern IT infrastructures. This paper summarizes the contributions of a dissertation thesis "Normalization of Unstructured Log Data into Streams of Structured Event Objects" dealing with the matter at hand in detail.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.

More info