Product Overview: Syslog-ng
Syslog-ng is a highly advanced and versatile system logging application designed to collect, process, and store log messages from a wide range of sources. Developed by Balázs Scheidler in 1998 and now maintained by One Identity, syslog-ng has evolved to become a robust and reliable solution for centralized log management.
Key Functions
- Log Collection: Syslog-ng collects log messages from various sources, including system logs, application logs, and files. It supports multiple log formats such as RFC3164, RFC5424, JSON, and Journald, and can collect logs from diverse platforms including Linux, Unix, BSD, and Solaris.
- Log Processing: The application processes log messages through advanced filtering, parsing, and rewriting capabilities. It can classify, normalize, and structure log messages using built-in parsers like PatternDB for unstructured logs, and specific parsers for CSV, JSON, and key=value formats. This processing enables the enrichment of logs by adding data from external lookup files or correlating logs based on common fields.
- Log Storage and Transfer: Syslog-ng ensures secure and reliable log transfer and storage. It supports TCP and TLS encryption, which provides a more reliable and secure log transfer compared to traditional UDP-based syslog. The application also uses local disk buffering, client-side failover, and application layer acknowledgement to prevent message loss. Logs are stored in encrypted, compressed, indexed, and timestamped binary files, ensuring the integrity and confidentiality of the data.
Key Features
- Advanced Filtering and Classification: Syslog-ng allows for complex filtering using regular expressions and boolean operators, enabling the selective forwarding of important log messages to various destinations. It can classify log messages into predefined categories, such as user login, application crash, or file transfer events.
- Scalability and Performance: The application is highly scalable and can handle a large volume of log messages. A single syslog-ng server can collect over half a million log messages per second from thousands of log sources and process over 24 GB of raw logs per hour on standard server hardware. It supports a client-relay configuration to collect logs from tens of thousands of log sources.
- Flexibility and Customization: Syslog-ng offers a flexible configuration language that allows users to construct complex log processing systems. It supports dynamic creation of directories, files, and database tables using macros. The configuration file is clean and well-structured, making it easier to maintain and reuse sources, destinations, or filters.
- Integration and Compatibility: The application can integrate with various enterprise monitoring solutions such as IBM Tivoli Netcool, Riemann, Redis, or Graphite. It also supports natively collecting and processing log messages from SQL databases and can run on multiple operating systems and architectures.
Security and Compliance
- Encryption and Authentication: Syslog-ng uses Transport Layer Security (TLS) protocol for encrypted communication and mutual authentication of hosts and servers using X.509 certificates. This ensures that logs cannot be tampered with, preserving the digital chain of custody.
- Compliance and Forensics: The application supports secure storage of log messages in encrypted files and allows for timestamping from external authorities, which is crucial for compliance and forensic purposes.
In summary, syslog-ng is a powerful and flexible log management solution that offers advanced features for collecting, processing, and securely storing log messages. Its scalability, performance, and customization capabilities make it an ideal choice for creating centralized and trusted logging solutions in large IT environments.