1.24.0

This release adds an example recipe for a systemd service that depends on
memfaultd. Additionally it changes the query frequency of eMMC lifetime
metrics.

Added

• Added an example service that depends on memfaultd. This is useful for when
  you want to run some code after memfaultd has finished starting.

Changed

• Changed the read interval for eMMC lifetime values to only happen once an hour
  regardless of configuration. The lifetime values will change very
  infrequently. Reading them at a lower frequency can help lower CPU utilization
  and reduce eMMC controller wakeups without sacrificing any resolution. This
  can improve battery life for battery powered devices.

1.23.0

This release adds support for Python based stacktraces, as well as a few minor
changes. Additionally we've made some changes to flash lifetime metrics, and how
they're collected.

Added

• Add generic custom trace support
  • This allows you to send text based backtraces to memfaultd.
  • They can be sent via the memfaultctl save-trace command
  • They can also be sent via /v1/trace/save local endpoint
• Added Python stack trace support
  • Python stack traces utilize the above custom backtrace support
  • By integrating pyfault you can send
    backtraces directly from a crashing python process.
• Add new virtual memory metrics from /proc/vmstat. These are represented as
  the following keys:
  • memory/vm/swaps_in_per_second
  • memory/vm/swaps_out_per_second
  • memory/vm/pages_in_per_second
  • memory/vm/pages_out_per_second
• Added OUI (Organizationally Unique Identifier) collection for network device
  identification and troubleshooting

Changed

• Changed flash lifetime metrics to use pct_remaining format instead of
  previous representation, providing clearer visibility into remaining device
  lifespan

Fixed

• Fixed MMC lifetime reading where memfaultd would attempt to read lifetime
  values from older MMC devices that don't support this feature
• Fixed handling of invalid individual lifetime values - they no longer cause
  the entire lifetime collection to fail
• Fixed overly verbose device info logging that was cluttering log output
• Fixed build configuration for macOS by removing
  --compress-debug-sections=zlib from cargo config that was causing
  compilation issues
• Fixed unit test reliability by properly mocking boottime_ms values

1.22.0

Added

• The Lifetime Estimation B value for monitored eMMCs is now reported as
  diskstats/<disk>/lifetime_b_pct
• A warning is now emitted when there is an issue parsing the output from
  memfault-device-info in memfault-core-handler. Previously, this would
  cause coredump collection to fail silently.
• Built-in CPU, memory, and thermal metrics can now be disabled individually
  with the new metrics.system_metric_collection.{cpu,memory,thermal}.enable
  boolean config.
• memfaultd's .cargo/config.toml now sets --compress-debug-sections to
  zlib and --force-unwind-tables to yes by default. These settings enable
  building efficient debug sections and binaries whose coredumps can be fully
  processed. We strongly recommend using them with any Rust programs you intend
  to monitor with Memfault.

Changed

• collectd-based metrics that overlap with built-in metrics will now be
  selectively filtered out if both are enabled based on whether the overlapping
  built-in metric group itself is enabled. This allows for using a mix of
  collectd-based metrics and built-in metrics more easily.

Fixed

• A bug in which the values for the MemfaultSdkMetric_os_name and
  MemfaultSdkMetric_os_version built-in Attributes were reversed.

1.21.1

This is a patch release that fixes an overflow issue with the recently added
diskstats/<disk>/bytes_written metric.

Fixed

• Fixed an issue with the diskstats/<disk>/bytes_written metric where it would
  overflow when the diskstats counter wrapped. This would cause very large
  readings to be sent. Whenever there is a potential overflow now the reading
  will be discarded.

1.21.0

This release adds several new metrics for tracking flash wear, as well as a new
syslog log source. It also includes a config flag to enable legacy statsd metric
names, and a data retention check to the upload process. For more information on
the new metrics, check out the
docs.

Added

• Added metrics for tracking flash wear for mmc block devices like eMMC and SD
  cards
  • diskstats/<disk_name>/bytes_written
  • diskstats/<disk_name>/lifetime_pct
  • diskstats/<disk_name>/manufacturer_id
  • diskstats/<disk_name>/name
• Added syslog as a logs source. This adds support for messages in both RFC5424
  and RFC3164 format sent over UDP. This is enabled with the syslog feature
  which defaults to enabled.
• Added a config flag to enable legacy statsd metric names. This will generate
  metrics that follow the format generated by the collectd statsd server. This
  is a helpful option for people looking to remove collectd as a dependency,
  but keep the legacy format for metric keys. The config value can be found in
  metrics->statsd_server->legacy_key_names.
• Added time-based data filtering: When upgrading multiple devices' visibility
  levels, the Memfault backend now tells memfaultd to only upload data from
  the time of the change forward. This prevents unexpected large data uploads
  from device history.
• Added interface/<interface>/total_bytes/{tx,rx} metrics. These allow us to
  track the total bytes sent over an interface over the course of an hour.

Changed

• Added intelligent traffic distribution: Device initial connections now include
  a random delay (up to 25% of the configured upload interval). This prevents
  traffic spikes when many devices connect simultaneously, ensuring smoother
  backend performance.
• ram and loop disks are no longer tracked with diskstats in auto mode.
  These are virtual disks, and provide little value when tracking metrics.

1.20.0

This release adds support for log filtering as well as a few bug fixes. Check
out the logging docs for more
information on log filtering!

Added

• Added support for log message filtering. This feature gives you the ability to
  exclude log lines that are spammy or contain potentially sensitive information
  from being uploaded to Memfault.
• Added built-in logs-to-metric rules that increment counter metrics when a
  systemd_restarts or oomkill event is detected. These are widely applicable
  to most Linux distros, and represent a good base for metric collection

Fixed

• Fixed a problem where large HRT files could take a long time to write. This
  was due to the file writer not being buffered.
• Fixed a race condition that caused the journald cursor to not be written.
  This would lead to the journald log collector capturing duplicate log lines
  if memfaultd is restarted.
• Fixed a problem where daily heartbeats were being saved when they were not
  enabled.

Removed

• Combined the regex feature with the logging feature to simplify feature
  management as log filtering and matching based on regexes becomes a key part
  of our logging offering.

1.19.0

This release adds memfaultctl upload as a command to allow
users to force memfaultd to upload data without forcing a
write and sync of in-progress data.

Added

• memfaultctl upload, which tells memfaultd to upload
  any data that has been written to the staging direcoty.

1.18.1

This release adds mickledore to the list of compatible
Yocto versions for meta-memfault.

Added

• mickledore to LAYERSERIES_COMPAT_memfault in meta-memfault's
  layer.conf

1.18.0

Added

• mar.mar_entry_max_count, which allows for setting the max number of MAR
  entries in your tmp directory. This will default to 1000.
• logs.max_buffered_lines, which sets the max number of lines that will be
  kept in memory before lines are dropped. Note that this is the same as
  fluent-bit.max_buffered_lines, but will apply to both the fluent-bit log
  source as well as the journald log source. If you have previously configured
  fluent-bit.max_buffered_lines that value will be used instead of the general
  config.
• Added the kind field to the MAR POST body. This is a field that is used by
  Memfault to differentiate the source of a MAR entry.
• Added logs.extra_attributes option to configure extra log attributes to keep
  in the log file (similar to
  fluent-bit.extra_attributes
  which still works but will be deprecated)

Changed

• The LogCollector has gone through a refactor to make the concurrency methods
  more consistent with the rest of the code base. This has also allowed us to
  make the following change.
• Moved recovery of logs in the logs directory into the LogCollector thread.
  This prevents a case where memfaultd could be slow to start when there is a
  large number of MAR entries on disk, and depending on systemd configuration,
  could be considered crashed and restarted before booting completely.

Fixed

• When passing a null to disable a feature there was a case where the config
  merging logic would fail. This would result in a case where a subset of
  configs were not possible. Fixed this logic to allow these valid use cases.

1.17.0

This release introduces the new memfaultctl write-metrics command
as well as some important bugfixes.

Added

• memfaultctl write-metrics, which allows users to write
  metrics to memfaultd from a script or shell by specifying
  them as arguments to the command in the form KEY=VALUE.
• 2 new built-in system metrics under the new diskstats
  metric category. These metrics follow the naming pattern
  diskstats/<device name>/reads_per_second and
  diskstats/<device name>/writes_per_second where
  <device name> is the name of a device listed in
  /proc/diskstats. The list of devices monitored
  can be configured with the metrics.system_metric_collection.diskstats
  configuration field.

Changed

• The log message emitted when the MAR cleaner
  encounters an invalid MAR entry in the MAR
  staging area has been lowered from WARN level
  to DEBUG level.
• A log message emitted when the configured
  high_resolution_telemetry.max_samples_per_minute
  rate limit is violated has been lowered from WARN
  level to DEBUG. This is to avoid repeatedly
  logging the same message at a high frequency
  when the system is sending more readings than
  permitted.

Fixed

• All deltas calculated based on a current and previous
  counter from procfs now take potential overflow into account.
• Fixed a sequencing issue with the cleaning of the MAR directory
  concerning an edge case where logs would fail to recover on
  devices that are near their disk space or inode quotas.
  This would result in stranded log files that would not
  get deleted or uploaded to Memfault until there is enough
  space to recover them.