NXLog

Last updated
NXLog
Developer(s) NXLog Ltd.
Initial releaseOctober 2011;
12 years ago
 (2011-10)
Stable release
NXLog EE v6.2 / Dec 4, 2023
Repository
Written in C
Operating system Windows, macOS, Amazon Linux, CentOS, RHEL, Ubuntu, Debian, AIX, FreeBSD, OpenBSD Oracle Solaris
Type Log management, Cybersecurity, SIEM
License NPL
Website nxlog.co

NXLog is a multi-platform log management solution that allows to collect logs from various sources, filter log events, transform log data and route (forward) it to different destinations. [1] It's available both as a free-of-charge NXLog Community Edition and as a commercial NXLog Enterprise Edition with enhanced capabilities, including agent management. [2]

Contents

NXLog is used as an integration component of many security products, like SIEM solutions, including Google Chronicle, [3] Microsoft Azure Sentinel, [4] Securonix, [5] LogPoint; [6] XDR/EDR solutions, including Rapid7 InsightIDR/OPS, [7] Vectra Platform, [8] Trellix XDR; [9] and MSSP/MDR solutions (e.g. from Arctic Wolf [10] and AT&T Cybersecurity [11] ). It’s also known as integration component for application performance monitoring (APM) and observability platforms like Datadog, [12] Graylog, [13] Coralogix, [14] SolarWinds Loggly, [15] IBM Log Analysis, [16] Mezmo (former LogDNA), [17] as well as a log management component for industrial control systems (SCADA/ICS), including power automation solutions from Siemens. [18]

Being able to run as a local log collection agent and as a network log aggregator, [19] NXLog supports a wide list of operating systems including Microsoft Windows, Apple macOS, Amazon Linux, Ubuntu, RedHat Linux, CentOS, Debian, SLES, IBM AIX, Oracle Solaris, FreeBSD and OpenBSD. [20]

The source code for NXLog Community Edition is public and available as a GitLab project. [21]

Overview

NXLog can be installed on many operating systems and it is enabled to operate in a heterogeneous environment, collecting event logs from thousands of different sources in many formats. NXLog can accept event logs from TCP, UDP, [22] file, database and various other sources in different formats such as syslog, windows event log, etc. [23] It supports SSL/TLS encryption to ensure data security in transit.

In concept NXLog is similar to syslog-ng or Rsyslog but it is not limited to UNIX and syslog only. NXLog can handle different log sources and formats, [24] so it can be used to implement a secured, centralized, [25] scalable logging system.

It can perform log rewrite, correlation, alerting, and pattern matching, it can execute scheduled jobs, and can perform log rotation. It was designed to be able to fully utilize modern multi-core CPU systems. Its multi-threaded architecture enables input, log processing and output tasks to be executed in parallel. Using an I/O layer it is capable of handling thousands of simultaneous client connections and process log volumes above the 100,000 EPS range.

NXLog does not drop any log messages unless instructed to. It can process input sources in a prioritized order, meaning that a higher priority source will be always processed before others. This can further help avoiding UDP message loss for example. In case of network congestion or other log transmission problems, NXLog can buffer messages on the disk or in memory. Using loadable modules it supports different input sources and log formats, not only limited to syslog but windows event log, audit logs, and custom binary application logs.

With NXLog it is possible to use custom loadable modules similarly to the Apache Web server. In addition to the online log processing mode, it can be used to process logs in batch mode in an offline fashion. NXLog's configuration language, with an Apache style configuration file syntax, enables it to rewrite logs, send alerts or execute any external script based on the specified criteria.

History

Back in 2009 Botond Botyanszki, founder and CEO of NXLog Ltd. used a modified version of msyslog to suit his needs, but when he found a requirement to implement a high performance, scalable, centralized log management solution, there was no such modern logging solution available. There were some alternatives to msyslog with some nice features (e.g. Rsyslog, syslog-ng, etc.), but none of them fit requirements. Most of these were still single threaded, syslog oriented, without native support for MS Windows, and came with an ambiguous configuration syntax and so on.

He decided to develop the tool from scratch, instead of hacking something else. Thus, NXLog was born in 2009 and was a closed source product in the beginning, heavily used in several production deployments. The source code of NXLOG Community Edition was released in November 2011, and has been freely available since.

Design

Most log processing solutions are built around the same concept. The input is read from a source, then the log messages are processed. Finally output is written or sent to a sink in other terminology.

When an event occurs in an application or a device, depending on its configuration, a log message is emitted. This is usually referred to as an "event log" or "log message". These log messages can have different formats and can be transmitted over different protocols depending on the actual implementation.

There is one thing common in all event log messages. All contain important data such as user names, IP addresses, application names, etc. This way an event can be represented as a list of key-value pairs which we call a "field". The name of the field is the key and the field data is the value. In another terminology this meta-data is sometimes referred to as event property or message tag.

The following example illustrates a syslog message:

<30>Nov 21 11:40:27 log4ensics sshd[26459]: Accepted publickey for log4ensics from 192.168.1.1 port 41193 ssh2 

The fields extracted from this message are as follows:

AuthMethod publickey SourceIPAddress 192.168.1.1 AccountName log4ensics SyslogFacility DAEMON SyslogSeverity INFO Severity INFO EventTime 2009-11-21 11:40:27.0 Hostname log4ensics ProcessID 26459 SourceName sshd Message Accepted publickey for log4ensics from 192.168.1.1 port 41193 ssh2 

NXLog has a special field$, raw_event. This field is handled by the transport (UDP, TCP, File, etc.) modules to read input into and write output from it. This field is also used later to parse the log message into further fields by various functions, procedures and modules.

Architecture

By utilizing loadable modules, the plugin architecture of NXLog allows it to read data from any kind of input, parse and convert the format of the messages, and then send it to any kind of output. Different input, processor and output modules can be used at the same time to cover all the requirements of the logging environment. The following figure illustrates the flow of log messages using this architecture.

NXLog architecture Nxlog architecture.png
NXLog architecture

The core of NXLog is responsible for parsing the configuration file, monitoring files and sockets, and managing internal events. It has an event based architecture, all modules can dispatch events to the core. The NXLog core will take care of the event and will optionally pass it to a module for processing. NXLog is a multi-threaded application, the main thread is responsible for monitoring files and sockets. These are added to the core by the different input and output modules. There is a dedicated thread handling internal events. It sleeps until the next event is to be processed then wakes up and dispatches the event to a worker thread. NXLog implements a worker thread-pool model. Worker threads receive an event which must be processed immediately. This way the NXLog core can centrally control all events and the order of their execution making prioritized processing possible. Modules which handle sockets or files are written to use non-blocking I/O in order to ensure that the worker threads never block. The files and sockets monitored by the main thread also dispatch events which are then delegated to the workers. Each event belonging to the same module is executed in sequential order, not concurrently. This ensures that message order is kept and prevents concurrency issues in modules. Yet the modules (worker threads) run concurrently, thus the global log processing flow is greatly parallelized.

When an input module receives data, it creates an internal representation of the log message which is basically a structure containing the raw event data and any optional fields. This log message is then pushed to the queue of the next module in the route and an internal event is generated to signal the availability of the data. The next module after the input module in a route, can be either a processor module or an output module. Actually an input or output module can also process data through built-in code or using the NXLog language execution framework. The only difference is that processor modules are run in another worker thread, thus parallelizing log processing even more. Considering that processor modules can also be chained, this can efficiently distribute work among multiple CPUs or CPU cores in the system.

Distributions

Releases

License

NXLog Community Edition is licensed under the NXLOG PUBLIC LICENSE v1.0. [27]

Related Research Articles

<span class="mw-page-title-main">CUPS</span> Computer printing system

CUPS is a modular printing system for Unix-like computer operating systems which allows a computer to act as a print server. A computer running CUPS is a host that can accept print jobs from client computers, process them, and send them to the appropriate printer.

Btrieve is a transactional database software product. It is based on Indexed Sequential Access Method (ISAM), which is a way of storing data for fast retrieval. There have been several versions of the product for DOS, Linux, older versions of Microsoft Windows, 32-bit IBM OS/2 and for Novell NetWare.

<span class="mw-page-title-main">Log4j</span> Java-based logging software

Apache Log4j is a Java-based logging utility originally written by Ceki Gülcü. It is part of the Apache Logging Services, a project of the Apache Software Foundation. Log4j is one of several Java logging frameworks.

<span class="mw-page-title-main">Event Viewer</span> Component of Microsofts Windows NT operating system

Event Viewer is a component of Microsoft's Windows NT operating system that lets administrators and users view the event logs on a local or remote machine. Applications and operating-system components can use this centralized log service to report events that have taken place, such as a failure to start a component or to complete an action. In Windows Vista, Microsoft overhauled the event system.

BlackBerry Enterprise Server designates the middleware software package that is part of the BlackBerry wireless platform supplied by BlackBerry Limited. The software plus service connects to messaging and collaboration software on enterprise networks to redirect emails and synchronize contacts and calendaring information between servers, desktop workstations, as well as mobile devices. Some third-party connectors exist, including Scalix, Zarafa, Zimbra, and the Google Apps BES Connector, although these are not supported by BlackBerry Limited. As of June 2018, BlackBerry Enterprise Server has been renamed to BlackBerry Unified Endpoint Manager (UEM).

logparser

logparser is a flexible command line utility that was initially written by Gabriele Giuseppini, a Microsoft employee, to automate tests for IIS logging. It was intended for use with the Windows operating system, and was included with the IIS 6.0 Resource Kit Tools. The default behavior of logparser works like a "data processing pipeline", by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.

Log management is the process for generating, transmitting, storing, accessing, and disposing of log data. A log data is composed of entries (records), and each entry contains information related to a specific event that occur within an organization’s computing assets, including physical and virtual platforms, networks, services, and cloud environments.

Windows Vista has many significant new features compared with previous Microsoft Windows versions, covering most aspects of the operating system.

Windows Vista contains a range of new technologies and features that are intended to help network administrators and power users better manage their systems. Notable changes include a complete replacement of both the Windows Setup and the Windows startup processes, completely rewritten deployment mechanisms, new diagnostic and health monitoring tools such as random access memory diagnostic program, support for per-application Remote Desktop sessions, a completely new Task Scheduler, and a range of new Group Policy settings covering many of the features new to Windows Vista. Subsystem for UNIX Applications, which provides a POSIX-compatible environment is also introduced.

In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software. A message or log entry is recorded for each such event. These log messages can then be used to monitor and understand the operation of the system, to debug problems, or during an audit. Logging is particularly important in multi-user software, to have a central overview of the operation of the system.

syslog-ng is a free and open-source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. As of today, syslog-ng is developed by Balabit IT Security Ltd. It has three editions with a common codebase. The first is called syslog-ng Open Source Edition (OSE) with the license LGPL. The second is called Premium Edition (PE) and has additional plugins (modules) under a proprietary license. The third is called Storebox (SSB), which comes as an appliance with a Web-based UI as well as additional features including ultra-fast-text search, unified search, content-based alerting and a premier tier support.

Prelude SIEM is a Security information and event management (SIEM).

Microsoft SQL Server is a proprietary relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network. Microsoft markets at least a dozen different editions of Microsoft SQL Server, aimed at different audiences and for workloads ranging from small single-machine applications to large Internet-facing applications with many concurrent users.

<span class="mw-page-title-main">PowerShell</span> Cross-platform command-line interface and scripting language for system and network administration

PowerShell is a task automation and configuration management program from Microsoft, consisting of a command-line shell and the associated scripting language. Initially a Windows component only, known as Windows PowerShell, it was made open-source and cross-platform on August 18, 2016, with the introduction of PowerShell Core. The former is built on the .NET Framework, the latter on .NET.

<span class="mw-page-title-main">Snare (software)</span>

Snare is a collection of software tools that collect audit log data from a variety of operating systems and applications to facilitate centralised log analysis. Enterprise Agents are available for Linux, macOS, Windows, Solaris, Microsoft SQL Server, a variety of browsers, and more. Snare Enterprise Epilog for Windows facilitates the central collection and processing of Windows text-based log files such as ISA/IIS. Snare Enterprise Epilog for Unix provides a method to collect any text based log files on the Linux and Solaris operating systems. Opensource Agents are available for Irix and AIX.

Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, queued operations to handle offline outputs, support for different module outputs, flexible configuration options and adds features such as using TCP for transport.

The Log Management Knowledge Base is a free database of detailed descriptions on over 20,000 event logs generated by Windows systems, syslog devices and applications. Provided as a free service to the IT community by Prism Microsystems, the aim of the Knowledge Base is to help IT personnel make sense of the large amounts of cryptic and arcane log data generated by network systems and IT infrastructures.

<span class="mw-page-title-main">Shinken (software)</span> Network monitoring software

Shinken is an open source computer system and network monitoring software application compatible with Nagios. It watches hosts and services, gathers performance data and alerts users when error conditions occur and again when the conditions clear.

Checkmk is a software developed in Python and C++ for IT Infrastructure monitoring. It is used for the monitoring of servers, applications, networks, cloud infrastructures, containers, storage, databases and environment sensors.

<span class="mw-page-title-main">Octopussy (software)</span> Log analysis software

Octopussy, also known as 8Pussy, is a free and open-source computer-software which monitors systems, by constantly analyzing the syslog data they generate and transmit to such a central Octopussy server. Therefore, software like Octopussy plays an important role in maintaining an information security management system within ISO/IEC 27001-compliant environments.

References

  1. "nxlog - A multi-platform universal log collector and forwarder". Windows Remix. Retrieved 30 April 2021.
  2. Enterprise Edition vs. Community Edition, NXLog
  3. Collect Microsoft Windows Event data  |  Chronicle  |  Google Cloud
  4. NXLog DNS Logs connector for Microsoft Sentinel | Microsoft Learn
  5. NXLog Management - Securonix, Feb 21, 2024
  6. LogPoint Agent Collector powered by NXLog — LogPoint Agent release/5.2.5 documentation, Feb 21, 2024
  7. NXLog | InsightIDR Documentation (rapid7.com), Feb 21, 2024
  8. Windows Event Log Ingestion | vectra.ai
  9. Windows logging with NXLog (trellix.com)
  10. NXLog Integration - Arctic Wolf Docs, Feb 21, 2024
  11. NXLog CE for Windows Hosts (att.com)
  12. NXLog Integration Guide (datadoghq.com)
  13. Ingest Windows Eventlog, Graylog 5.2, Feb 21, 2024
  14. Smooth NXLog Integration Process | Coralogix
  15. Centralizing Windows Logs - The Ultimate Guide To Logging (loggly.com), 2023
  16. Logging from Windows Server systems | IBM Cloud Docs
  17. NXLog Integration | Mezmo (LogDNA), Feb21, 2023
  18. SICAM / SIPROTEC System Hardening for Substation Automation and Protection, v1.50, Jan 2024, Siemens
  19. Centralized Logging | NXLog Docs, 2024
  20. Supported platforms | NXLog Docs, as of Feb 21, 2024
  21. NXLog's GitLab | nxlog-public / nxlog-ce
  22. Messier, Ric (2017). Network forensics. Indianapolis, IN: WILEY. p. 219. ISBN   9781119328285 . Retrieved 30 April 2021.
  23. "Using NXLog to enhance Azure Sentinel's ingestion capabilities". 2 February 2021.
  24. Messier, Ric (2017). Network forensics. Indianapolis, IN: Wiley. p. 256. ISBN   9781119328285 . Retrieved 30 April 2021.
  25. "Centralizing Windows Logs".
  26. "NXLog Community Version".
  27. https://nxlog.co/nxlog-public-license NXLOG PUBLIC LICENSE v1.0