Logging solution

[AgainPsychoX]

Is there any suggested logging solution to be used with Arduino, or rather this specific core?

In simple projects I just use few Serial.prints without much thinking, sometimes with conditional macros. In some project I have singular macro/function like dprintf or something to encapsulate it and iron the thing out (snprintf to buffer, then Serial.print).

In ESP32 projects I use ESP-IDF (SDK) built-in macros/functions ESP_LOG*, and it nicely support multiple logging levels etc. There are few sad things, like being unable to easily set log level per-file, but in general it feels firm and nice. AFIK the Arduino-Pico core supports RP2040 C SDK, maybe there is some mini logging framework I missed?

On one of my larger scale Arduino-based projects I used my own simple logging system built with macros, also with multiple warning levels, set from separate file. While I guess I could use it for my current project as well, I wanted to ask here first in case there is something a bit more "official" or anything worth considering.

[egnor]

I miss the ESP32 chatty-by-default logs also.

There are some debug-logging macros enabled by defining DEBUG_RP2040_PORT=Serial (or the UART of your choice) as mentioned in the Arduino IDE and platformio guides.

Once enabled you can use things like DEBUGV("hello logging %d", 123);. It's VERY basic, even compared to the relatively basic facilities in ESP32 Arduino (and some other Arduino cores), but it's a print statement you can turn on and off.

There are a number of independent logging libraries for Arduino

• Arduino-logging-library, forked and improved as ArduinoLog, itself again forked
• arduino-logger aka FlexLogger
• Arduino-logging-library
• dbg-trace
• DebugLog
• EasyLogger
• JBLogger
• log4arduino
• MycilaLogger
• Log
• Logger
• roo_logging
• vavagorodok_ArduinoStreamLogger

I probably missed some! I have no idea which of those is any good, actually maintained, and/or compatible with the RP2040 and this particular core. Please report back if you find something especially likeable. It would be nice to have a logging facility that isn't tied to a processor-specific core.

Judging by the questionable metrics of Github stars and forks, probably ArduinoLog, arduino-logger, and DebugLog would seem like they have the most momentum?

[AgainPsychoX]

I found some time to go though the list and note things:

_{Note: Scroll the table to the right for more columns, you can use middle mouse button and move cursor a bit right.}

Library	Popularity	Release optimization1	Levels	Variables	Flash strings support2	Other features	License	Notes
Arduino-logging-library	41 stars, 118 forks, activity 12 years ago	No	4 levels (Error, Info, Debug, Verbose)	Simple custom printf-like formatting	No		MIT	Wastes a memory for baud rate & log level in the logger structure, doesn't use flash memory for message strings, using the class twice causes problems, generally not good.
ArduinoLog	159 stars, 74 forks, activity last year	Yes	5 levels (Error, Info, Warn, Debug, Verbose)	Simple custom printf-like formatting	Yes	Fixed memory allocation; Can create logger to specific log output which implements Arduino Print; Can change logging level cutoff on runtime	MIT	Looks good enough, tho I wish release optimization1 could be more granular instead all-or-nothing logs.
arduino-logger aka FlexLogger	41 stars, 10 forks, activity 5 months ago	Yes, granular3	5 levels (Critical, Error, Warning, Info, Debug)	printf-based formatting	No	Multiple logging implementations (Serial, SD, Circular buffer) and flushing management	MIT	Doesn't work ESP family microcontrollers, due to printf from external library (but there is pull request to address this issue)
dbg-trace	4 stars, 1 fork, activity 5 years ago	No	?	?	No	?	MIT	No examples, no README; feels overcomplicated, too old to worth digging into it and understanding it I guess.
DebugLog	42 stars, 10 forks, activity 3 months ago	Yes	6 levels (Error, Warn, Info, Debug, Trace + Asserts)	No nice formatting; Variadic arguments passing to Arduino Stream print, bit by bit	Partial4	Output to console and/or file; Support for some C++ standard containers (std::vector, std::deque, std::map); Preamble control (incl. file & line), but sadly it's not flash string;	MIT	No formatting for variables (like if you want HEX or padding). Basically (advanced) wrapper on Arduino Print. Overall interesting, but idk...
EasyLogger	3 stars, 2 forks, activity 2 years ago	Yes, granular3	8 levels (Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug)	No nice formatting; Variables via chaining C++-like streams, built on Arduino Print	Partial4	Can be configured to output to any Arduino Print; Run-time per-service/module filtering	Apache 2.0	Again, basically wrapper on Arduino Print, no fancy formatting.
JBLogger	Fresh, 6 months old project	No	5 levels (Error, Warning, Info, Debug, Trace)	printf-based formatting	Yes	Some (run-time only) control over behaviour of logging; Nice dumping tools for HEX/BIN/ASCII	MIT	Interesting, but no way of freeing the program size during release builds is sad. Also, IMO minimal memory waste for _showModuleName flag when _moduleName can be check for nullptr.
log4arduino	7 stars, 3 forks, activity 2 years ago	Yes	None, acts like single one ("because YAGNI")	printf-based formatting	Yes	Can be configured to output to any Arduino Print; Always logging time (in milliseconds) and free memory (no opt-out; a bit waste)	MIT	Feels simple, but unfinished, not polished; but maybe okay for someone.
MycilaLogger	1 stars, 0 forks, activity 3 weeks ago	No	4 levels (Error, Warn, Info, Debug; can configure more)	printf-based formatting	No	No Arduino support, but ESP32	MIT	No Arduino support, but ESP32; causes dynamic memory allocations; no flash optimizations (well, you don't need those on ESP32); Not for applicable in this discussion I guess.
Log	4 stars, 2 forks, activity 7 years ago	Yes	None	printf-based formatting	No	None.	MIT	Very simple snprintf to Arduino Print wrapper; can cause memory allocations.
Logger	11 stars, 6 forks, activity 6 years ago	No	5 levels (Fatal, Error, Warning, Notice, Verbose)	No nice formatting; try playing around with Arduino Strings?	Partial4	Allow customization of logging function (defaults to serial)	Not specified	Can cause memory allocations; Since F() macro is String compatible I guess you can use it as memory optimization, but you can't use variables in that case. Overall, not very impressive library.
roo_logging	Fresh, 5 months old project	Yes, granular3	4 (5?) levels (Fatal, Error, Warning, Info + a bit weird Debug, see docs)	No nice formatting; Variables via chaining C++-like streams, built on Arduino Print	Partial4	Log counters and limiters facilities; Configurable fatal error action handler; Colors; Stacktrace with demangling;	MIT	Requires other dependencies, see the author other repositories - a lot of related stuff; Quite a program size footprint (6kB?); Can cause memory allocations; Since F() macro is String compatible I guess you can use it as memory optimization; Overall, maybe useful for ESP32 (incl. other libraries by the same author), but I doubt I will ever use it outside the "ecosystem" the author builds.
vavagorodok_ArduinoStreamLogger	2 stars, 0 forks, activity 6 months ago	Yes, granular3	5 levels (Error, Warning, Info, Debug, Trace)	Uses standard library input/output streams (like std::cout or printf)	No		MIT	Requires user to implement _write used by C++ standard library as output function (no default); No flash string support; Using standard library is another layer of indirection and buffers, which can worsen the performance, introduce memory moving, extra allocations etc.
	thisforcesthetablewide			scrolltorightformoredetails		aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa		aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

• It's for my personal use, but if I missed anything important (or anything update) please let me know.
• TODO: maybe do performance tests some day? maybe not even libraries itself but approaches:
  • test overhead of using standard library (see Notes in vavagorodok_ArduinoStreamLogger row in the table)
  • test different SERIAL_TX_BUFFER_SIZE values, default is 16 or 64 (1kB+ RAM)

Footnotes

1. By "release optimization" I mean whenever the library can be easily configured so irrelevant logging code can be removed from the binary when compiling for release. ↩ ↩²

2. By "flash strings support" I mean ability for the library to use/read messages from flash/program memory instead of RAM. Read about PROGMEM attribute, FSTR, F macros. On some platforms its pointless (like on RP Pico or ESPs) but on smaller AVRs (incl. official Arduinos) its a must sometimes. By the way, there is interesting article/question on Arduino StackExchange about those. ↩

3. By "granular" in "release optimization" I mean you can decide log level cut-off instead of all-or-nothing situation. ↩ ↩² ↩³ ↩⁴

4. By "partial flash string support" I mean you can use F() macro, but library either: wastes RAM internally anyways OR format from flash string is not supported OR due to lack of proper formatting you need to use multiple F() macros, which can have degraded performance if used on multiple short strings. ↩ ↩² ↩³ ↩⁴

[egnor]

I ended up rolling my own because, you know, 15 competing standards. (I did try to use a couple of the top ones, first. Really didn't enjoy the limited pseudo-printf.) I have Weird Opinions about logging though.

[AgainPsychoX]

Some of my thoughts I wrote down, would be waste to not share it. Of course, described things are HIGHLY SUBJECTIVE, and not something I would actually actively pursue; maybe one day I will make some better logging solution for myself; for now I'm good enough with the ones I mentioned in my original post above. Search and wondering around was fun tho. Maybe I will revisit the problem and come up with some better logging solution myself one day.

What I want from logging library?

Note: By "ability" I mean there is option, not forced behaviour, to do something.

• Logging levels
  • IMO (subjective) 5 levels is enough: Error, Warning, Notice/Info, Debug, Trace/Verbose. Sometimes maybe Fatal could be needed, but it's should be so rare you can handle it yourself, or just use Error.
  • Error = module/certain functionality is dead; i.e. thermometer not responding, constantly giving obviously invalid value.
  • Warning = functionality failed, but it's be recoverable/expected sometimes; i.e. cloud logger couldn't upload log/samples, maybe internet down, can retry later.
  • Notice/Info (default) = nothing bad happen, but worth taking notice; i.e. default password being set, you might want to change it.
  • Debug = anything worth knowing "to be sure"; i.e. current configuration read from EEPROM, some internal operations or even conditions hit inside the modules (if not too fast to avoid spam).
  • Trace/Verbose = madness & spam; i.e. dumping readings from the sensors as they go, reporting all web server traffic, etc.
• Release optimization, with granular control.
  • In other words: Ability to limit logging on compile-time to reduce program size footprint, with set level cutoff.
  • Extra: a way to get/check current lowest logging level (on compile-time), to remove bigger chunks of (usually testing) code that supports/prepares the logging.
• Modules, and release optimization granularity per-module level cutoff.
• Ability to using flash strings for both messages and formats.
• Ability to redirect logs on compile-time (as output function or some object with interface, like Arduino Print).
• Ability to include file name & line number.
• Ability to include timestamp (even date-time, or at least (and/or as fallback) number of millis/macros-seconds from boot).

Other features nice to have

• Colors (using ASCII characters).
• Clean between few platforms:
  • Arduino - Serial.prints are standard; stdout not configured by default.
  • ESP-IDF - ESP_LOGx (x being one of E,W,I,D,V) is standard; configurable with vprintf-like function; also stdout is used.
  • PicoSDK - stdout is used.
    Warning: not usable by default if using Arduino core. See print (stdio) linker errors. #1433 (comment)
• Little extra program size overhead compared to using standard framework logging ways.
• Ability to set logging level in run-time (of course, respecting the compile-time limit).
• Memory dumping as nicely formatted HEX/BIN/ASCII table
• Configurable default logging format/preamble (in compile-time),
  • Extra: ability to include evaluated expressions in the preamble, like timestamps, free memory, buffer usage, etc.
• Ability to limit, count & group logs generated at some points, to avoid spam.
• Very limited dynamic allocations (use stack or static buffer when possible); preferably controlled on compile-time.
• Very limited boilerplate code required.

Other features, which might be too over-engineered

• Logging from ISRs (Interrupt-Service-Routines) or other critical sections, which must be fast, which might require extra buffering with some rules.
• Stacktrace, maybe even with demangling the symbols (but it would require debug symbols be present).
• Somewhat standardized logging outputs (as examples):
  • Serial console (default)
  • Circular buffer and manual flushing to other output,
  • File (SD card or something, using file system library)
  • Remote logging (via HTTP request; maybe with Excel Google Apps script, or simple local Python/NodeJS script)
  • Web panel (for WiFi enabled microcontrollers)
• Barely readable (or even binary) output with magic numbers/codes with variables aside instead full messages, with special Serial console filter/script to decode the messages.