Tracing (software)

Tracing in software engineering refers to the process of capturing and recording information about the execution of a software program. This information is typically used by programmers for debugging purposes, and additionally, depending on the type and detail of information contained in a trace log, by experienced system administrators or technical-support personnel and by software monitoring tools to diagnose common problems with software. ^[1] Tracing is a cross-cutting concern.

There is not always a clear distinction between tracing and other forms of logging , except that the term tracing is almost never applied to logging that is a functional requirement of a program (therefore excluding logging of data from an external source, such as data acquisition in a high-energy physics experiment, and write-ahead logging). Logs that record program usage (such as a server log) or operating-system events primarily of interest to a system administrator (see for example Event Viewer ) fall into a terminological gray area.

Tracing is primarily used for anomaly detection, fault analysis, debugging or diagnostic purposes in distributed software systems, such as microservices or serverless functions. ^[2]

Software tracing

Software tracing is a tool for developers to gather information for debugging. This information is used both during development cycles and post-release. Unlike event logging, software tracing usually does not have the concept of a "class" of event or an "event code". Other reasons why event-logging solutions based on event codes are inappropriate for software tracing include:

• Because software tracing is low-level, there are often many more types of messages that would need to be defined, many of which would only be used at one place in the code. The event-code paradigm introduces significant development overhead for these "one-shot" messages.
• The types of messages that are logged are often less stable through the development cycle than for event logging.
• Because the tracing output is intended to be consumed by the developer, the messages don't need to be localized. Keeping tracing messages separate from other resources that need to be localized (such as event messages) is therefore important.
• There are messages that should never be seen.
• Tracing messages should be kept in the code, because they can add to the readability of the code. This is not always possible or feasible with event-logging solutions.

Tools

OpenTelemetry is a CNCF open source project that provides comprehensive support for distributed tracing. ^[3] Some vendors including Datadog, New Relic, Splunk also offer tracing SaaS services. ^[4]

Google and Meta have developed their own tracing frameworks namely Dapper and Canopy. ^[2]

Application-specific tracing

• Tracing with GNU Debugger's trace command ^[5]
• Linux C/C++ application tracing with cwrap
• Linux application tracing with UST – part of the same project as LTTng
• Windows software trace preprocessor (aka WPP)
• Instruction set simulation

System-specific tracing

In operating systems, tracing can be used in situations (such as booting) where some of the technologies used to provide event logging may not be available.

Linux offers system-level and user-level tracing capabilities with kernel markers and LTTng. ftrace also supports Linux kernel tracing. syslog is another tool in various operating systems for logging and tracing system messages.

FreeBSD and SmartOS employ DTrace for tracing for the kernel and the userland.

In embedded software, tracing also requires special techniques for efficient instrumentation and logging and low CPU overhead. ^[6]

Techniques

Trace generation and collection

Trace generation of method calls can be done with source code instrumentation, runtime information collection, or under debugger control. ^[7] Tracing macros, aspect-oriented programming and related instrumentation techniques can be employed.

Libraries used in source code send data to an agent or directly to the collection component. ^[4]

Trace analysis

To model execution trees, ISVis converts a rooted tree into a directed acyclic graph while Jinsight utilizes the call frame principle to gather and represent cumulative information about traces. ^[7]

The primary visualization method is the swimlane view, which is exemplified by tools like Jaeger and often includes annotations and key-value attributes. Despite its widespread use, this design lacks rigorous justification and users frequently face challenges like missing features and confusing navigation. Alternatives to swimlane views exist, like Jaeger’s service dependency view or SkyWalking’s List, Tree, and Table views. Aggregate visualizations are also used for analyzing large volumes of traces, with systems like Canopy offering queryable metrics and Jaeger providing trace comparison features. ^[8]

Event logging

Event logging provides system administrators with information useful for diagnostics and auditing. The different classes of events that will be logged, as well as what details will appear in the event messages, are often considered early in the development cycle. Many event logging technologies allow or even require each class of event to be assigned a unique "code", which is used by the event logging software or a separate viewer (e.g., Event Viewer) to format and output a human-readable message. This facilitates localization and allows system administrators to more easily obtain information on problems that occur.

Because event logging is used to log high-level information (often failure information), performance of the logging implementation is often less important.

A special concern, preventing duplicate events from being recorded "too often" is taken care of through event throttling.

Difficulties in making a clear distinction between event logging and software tracing arise from the fact that some of the same technologies are used for both, and further because many of the criteria that distinguish between the two are continuous rather than discrete. The following table lists some important, but by no means precise or universal, distinctions that are used by developers to select technologies for each purpose, and that guide the separate development of new technologies in each area:

Event logging	Software tracing
Consumed primarily by system administrators	Consumed primarily by developers
Logs "high level" information (e.g. failed installation of a program)	Logs "low level" information (e.g. a thrown exception)
Must not be too "noisy" (containing many duplicate events or information that is not helpful for its intended audience)	Can be noisy
A standards-based output format is often desirable, sometimes even required	Few limitations on output format
Event log messages are often localized	Localization is rarely a concern
Addition of new types of events, as well as new event messages, need not be agile	Addition of new tracing messages must be agile

Challenges and limitations

Enabling or disabling tracing during runtime often necessitates the inclusion of extra data in the binary. This can lead to performance degradation, even when tracing is not active.

If tracing is enabled or disabled at compile-time, collecting trace data from a client's system hinges on their willingness and capability to install a version of the software specifically enabled for tracing, and subsequently replicate the issue.

Tracing in software typically demands high standards of robustness, not only in the accuracy and reliability of the trace output but also in ensuring that the process being traced remains uninterrupted.

Given its low-level nature, tracing can generate a large volume of messages. To mitigate performance issues, it's often necessary to have the option to deactivate software tracing, either at the time of compilation or during run-time.

Security and privacy

In proprietary software, tracing data may include sensitive information about the product's source code.

See also

• Branch trace
• Debugging
• Instrumentation (computer programming)
• Logging
• Observability (software)

References

1. "The Tracing Book". Archived from the original on 2009-02-24.
2. Li, Bowen; Peng, Xin; Xiang, Qilin; Wang, Hanzhang; Xie, Tao; Sun, Jun; Liu, Xuanzhe (2022). "Enjoy your observability: an industrial survey of microservice tracing and analysis". Empirical Software Engineering. 27 (1): 25. doi:10.1007/s10664-021-10063-9. ISSN   1382-3256. PMC   8629732 . PMID   34867075.
3. Mandel, Maya (2023-06-07). "Council Post: Distributed Tracing: The Key To Microservices Observability". Forbes. Retrieved 2024-01-12.
4. Janes, Andrea; Li, Xiaozhou; Lenarduzzi, Valentina (2023). "Open tracing tools: Overview and critical comparison". Journal of Systems and Software. 204. Elsevier BV: 111793. arXiv: 2207.06875 . doi:10.1016/j.jss.2023.111793. ISSN   0164-1212.
5. "Tracepoints (Debugging with GDB)". sourceware.org. Retrieved 2022-06-24.
6. Kraft, Johan; Wall, Anders; Kienle, Holger (2010), "Trace Recording for Embedded Systems: Lessons Learned from Five Industrial Projects", Runtime Verification, Springer Berlin Heidelberg, pp. 315–329, doi:10.1007/978-3-642-16612-9_24, ISBN   9783642166112
7. Mertz, Jhonny; Nunes, Ingrid (2019). On the Practical Feasibility of Software Monitoring: a Framework for Low-Impact Execution Tracing. CASCON '04: Proceedings of the 2004 conference of the Centre for Advanced Studies on Collaborative research. IEEE. pp. 169–180. doi:10.1109/SEAMS.2019.00030. ISBN   978-1-7281-3368-3.
8. "A Qualitative Interview Study of Distributed Tracing Visualisation: A Characterisation of Challenges and Opportunities". IEEE Xplore. 2023-02-01. Retrieved 2024-01-12.