From 371ffdac1d5cd69b911b631a5f6d5613d52823a4 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 11:19:24 -0400 Subject: [PATCH 01/11] Make MixpanelClient durable with a disk-backed event spool MixpanelClient previously fired each event over HTTP immediately and silently dropped it when offline. The Segment path already had durability via Segment.Analytics.CSharp; this closes that gap for Mixpanel with a DiskQueue-backed spool and a background flush loop, so events survive offline stretches and process restarts and are delivered at-least-once. Design decisions - EventSpool wraps DiskQueue as a bounded, transport-agnostic durable queue: events are enqueued as opaque JSON bytes immediately (the write-ahead IS the durability guarantee -- "online?" is only ever knowable from a send outcome, so send-first-then-spool was rejected). Bounded by both item count (5,000) and total bytes (20MB, drop-oldest), so an unbounded offline period can't grow the spool or its eventual upload liability without limit. - Delivery batches via Mixpanel's /import endpoint (strict mode) rather than one /track POST per event: one request per drain tick, no /track 5-day age limit on a replayed backlog, and strict mode reports per-record rejections (failed_records) so one poison event doesn't cost the rest of a good batch. ProcessBatchAsync gathers a batch in a single DiskQueue transaction and commits/rolls back on the batch's verdict (Delivered/PoisonDrop commit; RetryableFailure rolls back in order for a later retry). - Delivery is async end-to-end (IEventSender.SendBatchAsync, EventSpool.ProcessBatchAsync, MixpanelClient.DrainOnceAsync via a Polly retry + circuit breaker pipeline), serialized by a SemaphoreSlim (not a monitor lock, since the lock must be held across the network await). Sync Flush()/ShutDown() block only at the boundary on a bounded drain; FlushAsync()/ShutDownAsync() are the non-blocking public alternatives (IClient, Analytics.FlushClientAsync / analytics.ShutDownAsync). - A drain tick is bandwidth-bounded for slow/metered connections: a soft 256KB-per-tick byte budget (checked after each event, so one oversized event still goes rather than starving the queue) on top of the existing sequential-sends/one-connection design. - Consent revocation (AllowTracking = false) purges the spool immediately (dispose + delete the spool directory + reopen, not a dequeue-and-flush loop, which left purged bytes sitting in DiskQueue's data files) and cancels whatever send is currently in flight first, so a batch that was mid-POST when consent was revoked rolls back and gets purged instead of still reaching Mixpanel afterward. - All spooled string values (recursively, including nested object/array properties) are path-scrubbed before being persisted, normalizing user home directories so stack traces and usage events can't leak OS account names. - The spool's byte total is persisted alongside it and restored on open (cross-checked against DiskQueue's own item-count estimate, falling back to an exact dequeue-and-verify measurement if the persisted value is missing or looks stale), so a restart with a large backlog doesn't require synchronously re-measuring the whole spool before Initialize can return. - Track() on a never-initialized client throws InvalidOperationException, matching the pre-durability contract; an initialized client whose spool couldn't be created (e.g. another process holds the cross-process lock) degrades to a no-op instead -- analytics must never crash the host. Hardening against bad network conditions - Captive portals (hotel/cafe Wi-Fi login pages) can no longer silently eat events: HttpClient no longer follows redirects, and any 3xx is classified as retryable rather than risk following a portal to a 200 that was never actually Mixpanel. - Flush()/ShutDown() are bounded (~5s / 20 attempts) even against a "black hole" server that accepts the TCP connection but never responds: a single deadline-bound CancellationTokenSource is threaded all the way down to HttpClient, and the Polly retry pipeline is bypassed for these bounded attempts (retrying during a bounded shutdown/flush has little delivery value and would multiply, not bound, the wait). - The circuit breaker's MinimumThroughput is tuned to how many outcomes one fully-failing drain tick actually produces (1 attempt + 2 retries = 3), so a sustained outage trips it after one bad tick instead of never being able to open at all. - A 200 response with body "0" (Mixpanel's rejected-payload signal) is classified PoisonDrop instead of Delivered. - Events over Mixpanel's documented 1MB per-event limit are refused at enqueue time (counted Failed immediately) rather than being spooled and later timing out as an indistinguishable-from-offline retryable failure that would wedge the head of the queue forever. Test coverage - 97 NUnit tests across four fidelity layers: pure unit (PathScrubber, AnalyticsEvent serialization), a real EventSpool against a real DiskQueue on a temp directory, a real EventSpool driving a real MixpanelClient against a scripted/fake IEventSender, and WireMock.Net-backed HTTP tests of MixpanelEventSender against real status codes and bodies. - Coverage includes: no-loss retry and crash-window dedup via $insert_id, spool item/byte bounding (drop-oldest), consent purge (including that purged bytes are actually gone from disk, not just marked consumed), cross-process exclusive lock contention, corrupt spool recovery (garbage in the data file vs. the transaction log), bounded-drain-against-a-hanging-sender, real circuit-breaker threshold behavior, captive-portal/redirect/cancellation handling, recursive PII scrubbing (including nested object/array properties), cap-eviction accounted for in Statistics, and the persisted-byte-total fast path (plus its fallback when that value is missing or stale). - Manually verified end-to-end against the live Mixpanel API from both SampleApp and cross-process offline scenarios (per-process firewall block; a Windows Sandbox guest with no virtual NIC at all). Co-Authored-By: Claude Sonnet 5 --- src/DesktopAnalytics/Analytics.cs | 59 +- src/DesktopAnalytics/AnalyticsEvent.cs | 83 ++ src/DesktopAnalytics/DesktopAnalytics.csproj | 3 + src/DesktopAnalytics/EventSpool.cs | 615 +++++++++++ src/DesktopAnalytics/IClient.cs | 37 +- src/DesktopAnalytics/IEventSender.cs | 68 ++ src/DesktopAnalytics/MixpanelClient.cs | 757 +++++++++++++- src/DesktopAnalytics/MixpanelEventSender.cs | 273 +++++ src/DesktopAnalytics/PathScrubber.cs | 39 + src/DesktopAnalytics/SegmentClient.cs | 37 + .../AnalyticsEventTests.cs | 132 +++ .../DesktopAnalyticsTests.csproj | 5 + src/DesktopAnalyticsTests/EventSpoolTests.cs | 601 +++++++++++ .../MixpanelClientTests.cs | 968 ++++++++++++++++++ .../MixpanelEventSenderTests.cs | 310 ++++++ .../PathScrubberTests.cs | 117 +++ src/SampleApp/Program.cs | 68 +- 17 files changed, 4129 insertions(+), 43 deletions(-) create mode 100644 src/DesktopAnalytics/AnalyticsEvent.cs create mode 100644 src/DesktopAnalytics/EventSpool.cs create mode 100644 src/DesktopAnalytics/IEventSender.cs create mode 100644 src/DesktopAnalytics/MixpanelEventSender.cs create mode 100644 src/DesktopAnalytics/PathScrubber.cs create mode 100644 src/DesktopAnalyticsTests/AnalyticsEventTests.cs create mode 100644 src/DesktopAnalyticsTests/EventSpoolTests.cs create mode 100644 src/DesktopAnalyticsTests/MixpanelClientTests.cs create mode 100644 src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs create mode 100644 src/DesktopAnalyticsTests/PathScrubberTests.cs diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index 36fe51c..f7c150e 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -11,6 +11,7 @@ using System.Reflection; using System.Runtime.InteropServices; using System.Threading; +using System.Threading.Tasks; using System.Xml.Linq; using System.Xml.XPath; using JetBrains.Annotations; @@ -738,6 +739,22 @@ public void Dispose() _client?.ShutDown(); } + /// + /// Async alternative to for hosts that shut down asynchronously: + /// flushes/shuts down the underlying client without blocking a thread on any in-flight + /// delivery. Bounded just like Dispose -- an offline shutdown still returns promptly, with + /// undelivered events left on disk (Mixpanel) or in the transport library's own store + /// (Segment) for the next launch. Calling afterwards (e.g. from a + /// using block wrapping this object) is a harmless no-op. + /// + /// Optionally ends the final delivery attempt even sooner + /// than its own bound; undelivered events stay queued for the next launch. Cancellation + /// never faults the task. + public Task ShutDownAsync(CancellationToken cancellationToken = default) + { + return _client?.ShutDownAsync(cancellationToken) ?? Task.CompletedTask; + } + /// /// Indicates whether we are tracking or not /// @@ -765,6 +782,34 @@ public static bool AllowTracking s_singleton.Initialize(initializationParameters); return; // Initialize sets s_allowTracking = true } + + // Already initialized and now re-enabled: re-arm any background flush loop that a + // prior PurgeQueuedEvents (consent revocation) paused. See offline-analytics.md. + try + { + s_singleton._client?.ResumeSending(); + } + catch (Exception e) + { + Debug.WriteLine("Analytics.AllowTracking: ResumeSending failed: " + e); + } + } + + if (!value) + { + // Consent revoked: purge any durable client's on-disk spool immediately (see + // offline-analytics.md, "Consent & lifecycle"). Guarded against a null singleton + // (AllowTracking can technically be set before an Analytics object is + // constructed) and a null client (the Segment/Mixpanel client is only assigned + // inside the Analytics constructor). + try + { + s_singleton?._client?.PurgeQueuedEvents(); + } + catch (Exception e) + { + Debug.WriteLine("Analytics.AllowTracking: PurgeQueuedEvents failed: " + e); + } } s_allowTracking = value; @@ -1082,7 +1127,19 @@ private static string GetUserNameForEvent() public static void FlushClient() { - s_singleton._client?.Flush(); + s_singleton?._client?.Flush(); + } + + /// + /// Async counterpart of : attempts delivery of anything pending + /// without blocking a thread on the network. Bounded -- returns promptly even while + /// offline, leaving undelivered events queued. + /// + /// Optionally ends the flush even sooner than its own + /// bound; undelivered events stay queued. Cancellation never faults the task. + public static Task FlushClientAsync(CancellationToken cancellationToken = default) + { + return s_singleton?._client?.FlushAsync(cancellationToken) ?? Task.CompletedTask; } } } diff --git a/src/DesktopAnalytics/AnalyticsEvent.cs b/src/DesktopAnalytics/AnalyticsEvent.cs new file mode 100644 index 0000000..f85c84e --- /dev/null +++ b/src/DesktopAnalytics/AnalyticsEvent.cs @@ -0,0 +1,83 @@ +using System; +using System.Text; +using Segment.Serialization; + +namespace DesktopAnalytics +{ + /// + /// A single spoolable analytics event in a neutral, JSON-serializable form. This is the + /// unit written to and read from the on-disk event spool (DiskQueue), independent of the + /// eventual transport (Mixpanel, etc). Instances are round-tripped as UTF-8 JSON bytes so + /// the spool stays portable and inspectable (see /). + /// + internal class AnalyticsEvent + { + public string AnalyticsId { get; set; } + public string EventName { get; set; } + + /// Event properties, e.g. Message/Stack Trace for exception reports. + public JsonObject Properties { get; set; } + + /// Mixpanel's $insert_id, stamped at enqueue time so at-least-once replay can + /// be safely deduplicated by the backend. + public string InsertId { get; set; } + + /// The original event time, stamped at enqueue time so a later replay is + /// back-dated correctly rather than reported at delivery time. + public DateTimeOffset Time { get; set; } + + public AnalyticsEvent() + { + Properties = new JsonObject(); + } + + /// + /// Creates a new event, stamping and . Both are + /// overridable so callers/tests can be deterministic; production code can omit them to get + /// a real GUID and the real wall-clock time. + /// + /// The stable per-user analytics id. + /// The event name. + /// Event properties. May be null, in which case an empty + /// property bag is used. + /// The $insert_id. Defaults to a fresh GUID. Never call + /// Guid.NewGuid() directly elsewhere in the stamping path -- pass it here so tests are + /// deterministic. + /// The event time. Defaults to DateTimeOffset.UtcNow. Never + /// call DateTime.Now/DateTimeOffset.UtcNow directly elsewhere in the stamping path -- + /// pass it here (e.g. from an injected ) so tests are + /// deterministic. + public static AnalyticsEvent Create( + string analyticsId, + string eventName, + JsonObject properties = null, + string insertId = null, + DateTimeOffset? time = null + ) + { + return new AnalyticsEvent + { + AnalyticsId = analyticsId, + EventName = eventName, + Properties = properties ?? new JsonObject(), + InsertId = insertId ?? Guid.NewGuid().ToString(), + Time = time ?? DateTimeOffset.UtcNow + }; + } + + /// Serializes this event as UTF-8 JSON bytes, suitable for enqueuing into the + /// on-disk spool. + public byte[] ToBytes() + { + var json = JsonUtility.ToJson(this, false); + return Encoding.UTF8.GetBytes(json); + } + + /// Deserializes an event previously produced by . + public static AnalyticsEvent FromBytes(byte[] bytes) + { + var json = Encoding.UTF8.GetString(bytes); + return JsonUtility.FromJson(json); + } + } +} diff --git a/src/DesktopAnalytics/DesktopAnalytics.csproj b/src/DesktopAnalytics/DesktopAnalytics.csproj index 379430c..fe12a54 100644 --- a/src/DesktopAnalytics/DesktopAnalytics.csproj +++ b/src/DesktopAnalytics/DesktopAnalytics.csproj @@ -17,10 +17,13 @@ ..\..\DesktopAnalytics.snk + All + + diff --git a/src/DesktopAnalytics/EventSpool.cs b/src/DesktopAnalytics/EventSpool.cs new file mode 100644 index 0000000..b912910 --- /dev/null +++ b/src/DesktopAnalytics/EventSpool.cs @@ -0,0 +1,615 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using DiskQueue; + +namespace DesktopAnalytics +{ + /// + /// The outcome of attempting to deliver one batch of spooled events, as reported by the caller + /// of . This drives whether the batch is removed from + /// the spool (/) or left in place for a later + /// retry (). (Per-record rejections within a processed batch are + /// the sender/client's concern -- see -- and count + /// as here: the batch is finished with either way.) + /// + internal enum SendResult + { + /// The batch was processed by the server (e.g. HTTP 2xx, or a strict-mode 400 + /// where the rejected records are individually reported). Remove it from the spool. + Delivered, + + /// A transient failure (connection error, timeout, 5xx, 429). Leave the batch in + /// the spool for a later retry. + RetryableFailure, + + /// A non-retryable failure (e.g. a 4xx that will never succeed). Remove the batch + /// from the spool anyway so bad events cannot wedge the spool forever. + PoisonDrop + } + + /// + /// Durable, bounded, on-disk store of pending s, backed by + /// DiskQueue. Events are enqueued as opaque UTF-8 JSON bytes (see + /// ) so the spool stays portable and inspectable. + /// + /// + /// Every public method here swallows disk/IO/serialization exceptions internally (logging via + /// ) and returns gracefully -- analytics code must never + /// throw into the host application. The one exception is the constructor: failing to acquire + /// the underlying DiskQueue's cross-process exclusive lock is a startup-time condition the + /// caller needs to know about, so it is allowed to propagate. + /// + internal class EventSpool : IDisposable + { + // How long we wait to acquire DiskQueue's cross-process exclusive lock when opening the + // spool. Kept short: if another process is holding the lock for this long, something is + // wrong, and we'd rather fail fast than hang application startup/shutdown. + private static readonly TimeSpan s_lockWaitTimeout = TimeSpan.FromSeconds(2); + + // Number of leading bytes of the SHA-256 hash of the API key used to build the spool + // directory name. Just needs to be stable and distinguish keys from each other -- it is + // not a security boundary -- so a short prefix is plenty. + private const int kApiKeyHashBytes = 8; + + private readonly int _maxItems; + private readonly int _maxItemBytes; + private readonly long _maxSpoolBytes; + // Logical bytes of the LIVE entries (disk file sizes would overcount consumed-but-untrimmed + // ones). Guarded by _sync; measured at open, then maintained incrementally. + private long _spoolBytes; + private readonly string _spoolDirectory; + // Not readonly: Purge replaces the queue wholesale (see its remarks). Always accessed + // under _sync. Null only if a Purge-time reopen failed; every public method already + // catches and logs whatever follows from that. + private IPersistentQueue _queue; + // Serializes ALL access to _queue, exactly like the lock(_syncRoot) it replaced -- a + // SemaphoreSlim because ProcessBatchAsync must hold it across awaits of the send callback, + // which a monitor lock cannot. Deliberately never disposed: it holds no unmanaged + // resources unless AvailableWaitHandle is touched (it isn't), and disposing it would turn + // a benign use-after-Dispose race into ObjectDisposedException noise. + private readonly SemaphoreSlim _sync = new SemaphoreSlim(1, 1); + private bool _disposed; + + /// + /// Raised once for each event permanently dropped by cap enforcement + /// () -- i.e. NOT a drop at enqueue time, which + /// 's own return value already reports, but an OLDER event (or, in the + /// degenerate maxItems == 0 case, the just-enqueued one) evicted later to keep the + /// spool within its configured caps. This is the only way callers can learn about that kind + /// of drop, since it can happen on an call for a DIFFERENT event than + /// the one being dropped. Raised while the internal lock is held, so handlers must be fast + /// and must not call back into this . + /// + public event Action ItemDroppedByCap; + + /// + /// Opens (creating if necessary) the on-disk event spool rooted at + /// , taking DiskQueue's cross-process exclusive lock for + /// the lifetime of this object. + /// + /// Directory to hold the spool's files. Callers normally get + /// this from ; tests typically inject a temp directory. + /// The maximum number of events retained. Enqueuing beyond this + /// drops the oldest event(s) first. + /// The maximum serialized size of a single event; anything + /// larger is refused at enqueue time ( returns false) rather than + /// spooled. Defaults to unbounded; passes the transport's + /// real per-event limit. + /// The maximum total serialized size of all retained events; + /// enqueuing beyond this drops the oldest event(s) first, exactly like + /// . Bounds both the disk footprint and the total upload + /// liability of an accumulated backlog. Defaults to unbounded. + /// Propagates whatever DiskQueue throws if the lock cannot be + /// acquired within the internal wait timeout (e.g. another process/instance already has + /// this spool open) or if the directory cannot be created/opened. This is deliberately not + /// swallowed: it is a startup-time failure the caller needs to know about. + public EventSpool(string spoolDirectory, int maxItems, int maxItemBytes = int.MaxValue, + long maxSpoolBytes = long.MaxValue) + { + if (maxItems < 0) + throw new ArgumentOutOfRangeException(nameof(maxItems)); + if (maxItemBytes <= 0) + throw new ArgumentOutOfRangeException(nameof(maxItemBytes)); + if (maxSpoolBytes <= 0) + throw new ArgumentOutOfRangeException(nameof(maxSpoolBytes)); + + _maxItems = maxItems; + _maxItemBytes = maxItemBytes; + _maxSpoolBytes = maxSpoolBytes; + _spoolDirectory = spoolDirectory; + _queue = PersistentQueue.WaitFor(spoolDirectory, s_lockWaitTimeout); + _spoolBytes = ResolveExistingSpoolBytes(); + } + + private string SpoolBytesFilePath => Path.Combine(_spoolDirectory, "spool-bytes.txt"); + + // Cheap common-case restart path: reuses the byte total PersistSpoolBytesLocked wrote during + // the previous session instead of re-measuring by walking (dequeuing and rolling back) every + // live entry, which is the only way to get an exact total (see MeasureExistingSpoolBytes) but + // means a large backlog turns every app startup into a synchronous, disk-bound scan. The + // persisted total is cross-checked against DiskQueue's own (already-cheap) + // EstimatedCountOfItemsInQueue -- "queue is empty" disagreeing with "persisted total is + // nonzero" (or vice versa) means the sidecar file is stale (e.g. the previous session crashed + // between enqueuing and persisting) and is not trusted, falling back to the slow but exact + // measurement. + private long ResolveExistingSpoolBytes() + { + try + { + var text = File.ReadAllText(SpoolBytesFilePath); + if (long.TryParse(text, NumberStyles.Integer, CultureInfo.InvariantCulture, out var persisted) && + persisted >= 0) + { + var isEmpty = _queue.EstimatedCountOfItemsInQueue <= 0; + if (isEmpty == (persisted == 0)) + return persisted; + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool: no usable persisted spool byte total, measuring instead: " + e); + } + + var measured = MeasureExistingSpoolBytes(); + PersistSpoolBytesLocked(measured); + return measured; + } + + // Sums the live entries left by a previous session (dequeue-all in a session disposed + // without Flush = pure measurement; DiskQueue rolls it back). Failure => 0: the byte cap + // is then under-enforced until the backlog drains, erring toward keeping events. + private long MeasureExistingSpoolBytes() + { + try + { + long total = 0; + using (var session = _queue.OpenSession()) + { + while (true) + { + var bytes = session.Dequeue(); + if (bytes == null) + break; + total += bytes.Length; + } + } + return total; + } + catch (Exception e) + { + Debug.WriteLine("EventSpool: failed to measure existing spool bytes, assuming 0: " + e); + return 0; + } + } + + // Caller must hold _sync (or be under construction, before _sync can be contended). + // Best-effort: a failure here just means the NEXT open falls back to the slower + // MeasureExistingSpoolBytes instead of this fast path -- never a correctness problem. + private void PersistSpoolBytesLocked(long spoolBytes) + { + try + { + File.WriteAllText(SpoolBytesFilePath, spoolBytes.ToString(CultureInfo.InvariantCulture)); + } + catch (Exception e) + { + Debug.WriteLine("EventSpool: failed to persist spool byte total: " + e); + } + } + + /// + /// Computes the per-user, per-API-key spool directory: + /// %LocalAppData%\SIL\DesktopAnalytics\spool\<short hash of apiKey>. The API key is + /// hashed (never embedded raw) so that, e.g., DEBUG and RELEASE builds of an app -- which + /// use different keys -- get separate spools, without exposing the key via the file system. + /// + public static string GetDefaultSpoolPath(string apiKey) + { + var root = Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), + "SIL", "DesktopAnalytics", "spool"); + return Path.Combine(root, HashApiKey(apiKey)); + } + + private static string HashApiKey(string apiKey) + { + using (var sha = SHA256.Create()) + { + var hash = sha.ComputeHash(Encoding.UTF8.GetBytes(apiKey ?? string.Empty)); + var builder = new StringBuilder(kApiKeyHashBytes * 2); + for (var i = 0; i < kApiKeyHashBytes; i++) + builder.Append(hash[i].ToString("x2")); + return builder.ToString(); + } + } + + /// + /// An approximate count of events currently in the spool. "Approximate" because DiskQueue + /// tracks this as an estimate that is cheap to read; it is accurate enough for bounding and + /// diagnostics but should not be treated as a hard guarantee under concurrent access. + /// + public int ApproximateCount + { + get + { + try + { + _sync.Wait(); + try + { + return _queue.EstimatedCountOfItemsInQueue; + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.ApproximateCount: failed to read count: " + e); + return 0; + } + } + } + + /// + /// Serializes and enqueues , then enforces the configured item cap by + /// dropping the oldest event(s) if necessary. Never throws: any disk/IO/serialization + /// failure is logged and swallowed, and the event is simply lost rather than crashing the + /// host. + /// + /// True if the event was durably enqueued; false if it was dropped at enqueue time + /// (null, unserializable, larger than the per-event byte cap, or a disk failure). Callers use + /// this to count that kind of drop -- see . A later drop by + /// cap enforcement (this call's own item, or some earlier one) is reported separately via + /// , since it does not necessarily happen on the enqueue call + /// for the event that gets dropped. + public bool Enqueue(AnalyticsEvent evt) + { + if (evt == null) + return false; + + try + { + byte[] bytes; + try + { + bytes = evt.ToBytes(); + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.Enqueue: failed to serialize event, dropping: " + e); + return false; + } + + if (bytes.Length > _maxItemBytes) + { + // Refused up front rather than spooled: an event over the transport's per-event + // limit can never be delivered, and if it is large enough to time out the HTTP + // request it would be classified as a RETRYABLE failure (a timeout is + // indistinguishable from being offline) and wedge the head of the queue forever. + Debug.WriteLine("EventSpool.Enqueue: event of " + bytes.Length + + " bytes exceeds the " + _maxItemBytes + "-byte cap, dropping: " + evt.EventName); + return false; + } + + _sync.Wait(); + try + { + using (var session = _queue.OpenSession()) + { + session.Enqueue(bytes); + session.Flush(); + } + + _spoolBytes += bytes.Length; + EnforceCapLocked(); + PersistSpoolBytesLocked(_spoolBytes); + } + finally + { + _sync.Release(); + } + + return true; + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.Enqueue failed: " + e); + return false; + } + } + + // Caller must hold _sync. Drops the oldest item(s) one at a time until the queue is at or + // under BOTH caps (_maxItems and _maxSpoolBytes). Each iteration removes exactly one item + // (or gives up on a read/dequeue failure), so this always terminates -- including the + // degenerate case of _maxItems == 0, where even the item just enqueued is dropped. + private void EnforceCapLocked() + { + while (true) + { + int count; + try + { + count = _queue.EstimatedCountOfItemsInQueue; + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.EnforceCap: failed to read count, giving up: " + e); + return; + } + + if (count <= _maxItems && _spoolBytes <= _maxSpoolBytes) + return; + + if (count <= 0) + { + // Nothing left to drop; if the byte estimate says otherwise it has drifted -- + // resync it to the truth (an empty queue holds zero bytes). + _spoolBytes = 0; + return; + } + + try + { + using (var session = _queue.OpenSession()) + { + var oldest = session.Dequeue(); + if (oldest == null) + { + _spoolBytes = 0; // See the count <= 0 case above. + return; + } + session.Flush(); + _spoolBytes = Math.Max(0, _spoolBytes - oldest.Length); + } + + try + { + ItemDroppedByCap?.Invoke(); + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.EnforceCap: ItemDroppedByCap handler threw: " + e); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.EnforceCap: failed to drop oldest item, giving up: " + e); + return; + } + } + } + + // Test seam: the spool's current logical byte total (the value the byte cap is enforced + // against), primarily so tests can prove it survives a dispose/reopen. + internal long ApproximateBytes + { + get + { + _sync.Wait(); + try + { + return _spoolBytes; + } + finally + { + _sync.Release(); + } + } + } + + /// + /// Gathers up to spooled events (oldest first, bounded by + /// ) in a single DiskQueue session/transaction, hands them to + /// as ONE batch (one network request -- see + /// ), and commits or rolls back the whole batch on its + /// verdict: + /// + /// or : every + /// dequeue is committed (session.Flush()) -- the batch is finished with, whether + /// ingested or permanently rejected. + /// : nothing is committed -- disposing the + /// session without flushing rolls the whole batch back into the spool, in order, for a + /// later retry. + /// + /// If throws, it is treated exactly like + /// : the exception is swallowed and nothing is + /// flushed. This is the crash-window path that guarantees at-least-once delivery -- a + /// sender that delivered the batch but crashed/threw before this method could flush will + /// see the same events again on the next call. + /// An entry that cannot be deserialized (corrupt, or an incompatible schema version) is + /// skipped during the gather and its removal is committed together with the batch (or, if + /// nothing else was gathered, on its own) -- it can never wedge the spool. + /// + /// Maximum number of events gathered into this call's batch. + /// Decides the batch's fate; see the summary above. Receives + /// so the send can participate in cancellation. + /// Soft byte budget for this call's batch, bounding each drain burst + /// on a slow/metered connection. Checked AFTER each gathered event, so a single event over + /// the whole budget still goes -- the budget can never starve the queue. + /// Bounds the wait for the spool's internal semaphore and + /// is passed through to . Cancellation never throws out of this + /// method; it just means the batch (if any was gathered) rolls back. + public async Task ProcessBatchAsync(int maxItems, + Func, CancellationToken, Task> send, + long maxBytes = long.MaxValue, CancellationToken cancellationToken = default) + { + if (send == null || maxItems <= 0 || maxBytes <= 0) + return; + + try + { + await _sync.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + using (var session = _queue.OpenSession()) + { + var batch = new List(); + long batchBytes = 0; + // Bytes of undeserializable entries dequeued (skipped) during the gather; + // committed along with the batch. + long skippedBytes = 0; + + while (batch.Count < maxItems && batchBytes < maxBytes) + { + byte[] bytes; + try + { + bytes = session.Dequeue(); + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.ProcessBatch: dequeue failed: " + e); + break; // Send whatever was gathered before the failure. + } + + if (bytes == null) + break; // Spool is empty. + + try + { + batch.Add(AnalyticsEvent.FromBytes(bytes)); + batchBytes += bytes.Length; + } + catch (Exception e) + { + Debug.WriteLine( + "EventSpool.ProcessBatch: failed to deserialize event, dropping: " + e); + skippedBytes += bytes.Length; + } + } + + if (batch.Count == 0) + { + if (skippedBytes > 0) + { + // Nothing to send, but bad entries were dequeued -- commit their removal. + session.Flush(); + _spoolBytes = Math.Max(0, _spoolBytes - skippedBytes); + PersistSpoolBytesLocked(_spoolBytes); + } + return; + } + + SendResult result; + try + { + result = await send(batch, cancellationToken).ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine( + "EventSpool.ProcessBatch: send threw, leaving batch for retry: " + e); + return; // Do not flush -- every dequeue rolls back on session dispose. + } + + if (result == SendResult.Delivered || result == SendResult.PoisonDrop) + { + session.Flush(); + _spoolBytes = Math.Max(0, _spoolBytes - (batchBytes + skippedBytes)); + PersistSpoolBytesLocked(_spoolBytes); + } + // RetryableFailure: no flush, so the whole batch rolls back. + } + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.ProcessBatch failed: " + e); + } + } + + /// + /// Empties the spool entirely (used on consent revocation), removing the event data from + /// disk -- not merely marking it consumed. Never throws. + /// + /// + /// Implemented as dispose + delete-the-directory + reopen rather than either alternative, + /// both tried empirically (2026-07, DiskQueue 1.7.2): a dequeue-and-flush loop leaves the + /// purged events' bytes sitting in the data files (DiskQueue does not trim consumed + /// entries), which is not acceptable for a CONSENT purge; and DiskQueue's own + /// HardDelete(reset: true) leaves the live instance's + /// EstimatedCountOfItemsInQueue stale, so the spool keeps reporting the purged + /// events as present -- consistent with its "not thread safe ... or safe in any other + /// way" warning. Between the dispose and the reopen the cross-process exclusive lock is + /// briefly released; if another process steals it in that window the reopen fails, this + /// spool degrades to a no-op (every method here already tolerates that), and the purge + /// itself has still succeeded -- the data is gone, which is the property that matters. + /// + public void Purge() + { + try + { + _sync.Wait(); + try + { + if (_disposed) + return; + + try + { + _queue?.Dispose(); + } + finally + { + _queue = null; + } + + if (Directory.Exists(_spoolDirectory)) + Directory.Delete(_spoolDirectory, true); + + _queue = PersistentQueue.WaitFor(_spoolDirectory, s_lockWaitTimeout); + _spoolBytes = 0; + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.Purge failed: " + e); + } + } + + /// + /// Releases the underlying DiskQueue's cross-process exclusive lock. Safe to call more + /// than once. + /// + public void Dispose() + { + if (_disposed) + return; + + try + { + _sync.Wait(); + try + { + _disposed = true; + _queue?.Dispose(); + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.Dispose failed: " + e); + } + } + } +} diff --git a/src/DesktopAnalytics/IClient.cs b/src/DesktopAnalytics/IClient.cs index 1923115..f05762d 100644 --- a/src/DesktopAnalytics/IClient.cs +++ b/src/DesktopAnalytics/IClient.cs @@ -1,4 +1,6 @@ -using Segment.Serialization; +using System.Threading; +using System.Threading.Tasks; +using Segment.Serialization; namespace DesktopAnalytics { @@ -9,6 +11,39 @@ internal interface IClient void Identify(string analyticsId, JsonObject traits, JsonObject options); void Track(string defaultIdForAnalytics, string eventName, JsonObject properties); void Flush(); + + /// + /// Async counterpart of . Implementations whose shutdown has nothing + /// awaitable (see ) may complete synchronously; implementations + /// with real async delivery work (see ) must never block a + /// thread waiting on it. Like every member here, must not throw -- including on + /// cancellation, which just ends any in-flight delivery early (events stay queued). + /// + Task ShutDownAsync(CancellationToken cancellationToken = default); + + /// + /// Async counterpart of . Same contract notes as + /// . + /// + Task FlushAsync(CancellationToken cancellationToken = default); + Statistics Statistics { get; } + + /// + /// Called when the user revokes tracking consent ( + /// transitioning to false). Implementations that spool events on disk (see + /// ) must purge that spool immediately; implementations without a + /// local spool (see ) can no-op. + /// + void PurgeQueuedEvents(); + + /// + /// Called when the user grants tracking consent again ( + /// transitioning back to true) on an already-initialized client, after a prior + /// paused it. Implementations with a background flush loop (see + /// ) must re-arm it; implementations without one (see + /// ) can no-op. + /// + void ResumeSending(); } } \ No newline at end of file diff --git a/src/DesktopAnalytics/IEventSender.cs b/src/DesktopAnalytics/IEventSender.cs new file mode 100644 index 0000000..d19d04b --- /dev/null +++ b/src/DesktopAnalytics/IEventSender.cs @@ -0,0 +1,68 @@ +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace DesktopAnalytics +{ + /// + /// The outcome of attempting to deliver one batch of events. Mixpanel's /import endpoint + /// yields exactly two shapes of outcome, and this type models both: either the whole request + /// failed to be processed ( is + /// or , applying to every event in the batch), or the + /// request WAS processed () with zero or more individual + /// records rejected as permanently invalid ( -- /import's + /// failed_records; the remainder were ingested). + /// + internal class BatchSendResult + { + private static readonly IReadOnlyCollection s_noFailures = new int[0]; + + /// Whole-batch verdict. means the request was + /// processed and every event is finished with (ingested, or rejected-and-reported in + /// ) -- the caller should remove the whole batch from the + /// spool. means nothing was processed and the + /// whole batch should be retried later. means the + /// whole batch was permanently rejected (e.g. an unexpected 4xx) and should be dropped. + public SendResult Outcome { get; } + + /// When is : the + /// zero-based indices (into the batch that was sent) of records Mixpanel permanently + /// rejected (its failed_records). Empty otherwise. + public IReadOnlyCollection FailedIndices { get; } + + public BatchSendResult(SendResult outcome, IReadOnlyCollection failedIndices = null) + { + Outcome = outcome; + FailedIndices = failedIndices ?? s_noFailures; + } + + public static readonly BatchSendResult Delivered = new BatchSendResult(SendResult.Delivered); + public static readonly BatchSendResult Retryable = new BatchSendResult(SendResult.RetryableFailure); + public static readonly BatchSendResult Poison = new BatchSendResult(SendResult.PoisonDrop); + } + + /// + /// The one seam between the durable and the network. Implementations + /// deliver a batch of s in a single request and classify the outcome + /// via so the flush loop knows whether to remove the batch from the + /// spool, leave it for retry, or drop it. + /// + /// + /// Implementations must never throw (synchronously or as a faulted task): the flush loop (via the + /// Polly-wrapped call in ) treats an exception the same as + /// , but a well-behaved sender should catch its own + /// network/timeout exceptions and return directly. + /// + internal interface IEventSender + { + /// The batch to deliver, oldest first. Never null; callers do not + /// pass empty batches. + /// Allows a bounded caller (see + /// 's BoundedDrain, used by Flush/ShutDown) to abandon a send + /// that is taking too long -- e.g. a server that accepts the connection but never responds + /// (a "black hole"). Implementations should treat cancellation like any other transient + /// failure and return rather than throwing. + Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default); + } +} diff --git a/src/DesktopAnalytics/MixpanelClient.cs b/src/DesktopAnalytics/MixpanelClient.cs index 2ce3493..7779c53 100644 --- a/src/DesktopAnalytics/MixpanelClient.cs +++ b/src/DesktopAnalytics/MixpanelClient.cs @@ -1,58 +1,767 @@ -using System; +using System; using System.Collections.Generic; -using System.Linq; +using System.Diagnostics; +using System.Net.NetworkInformation; using System.Threading; using System.Threading.Tasks; +using Polly; +using Polly.CircuitBreaker; +using Polly.Retry; using Segment.Serialization; namespace DesktopAnalytics { + /// + /// Durable Mixpanel client: events are scrubbed, stamped, and written to an on-disk + /// immediately (never lost to being offline), and a background flush + /// loop drains the spool through a Polly-wrapped whenever it can. See + /// offline-analytics.md for the design this implements. + /// + /// + /// Every public member here swallows exceptions and returns/no-ops rather than throwing -- + /// analytics must never crash the host application (see offline-analytics.md, "Never crash the + /// host"). The one intentionally-not-swallowed failure is a bad + /// constructor call inside , which is itself caught here so it cannot + /// escape to the caller either; it just leaves this client running with no spool (Track/Drain + /// become no-ops) rather than durable. + /// internal class MixpanelClient : IClient { - private Mixpanel.MixpanelClient _client; + // Open questions in offline-analytics.md: exact cap/batch/cadence values are not locked yet. + // These are reasonable starting defaults, not requirements baked in elsewhere. + private const int kDefaultMaxSpoolItems = 5000; + // Events per /import request (one request per drain tick). Mixpanel accepts up to 2,000 + // events and 10MB per request; this stays well under both (the per-tick byte budget below + // bounds the request size long before the count does), while draining a large offline + // backlog reasonably fast (5000 events in ~13 min at the default cadence). + private const int kDefaultBatchSize = 200; + private const int kDefaultFlushIntervalSeconds = 30; - private readonly List> _tasks = new List>(); + // Mixpanel's documented per-event limit: 1MB of uncompressed JSON + // (https://docs.mixpanel.com/reference/import-events, "Common Issues"). An event over this + // can never be delivered, so it is refused at enqueue time (counted in Statistics.Failed) + // instead of spooled -- if it merely got rejected we'd catch it as a PoisonDrop, but one + // large enough to time out the HTTP request would classify as RETRYABLE (a timeout is + // indistinguishable from being offline) and wedge the head of the queue forever. + private const int kMaxSpooledEventBytes = 1024 * 1024; - public void Initialize(string apiSecret, string host = null, int _flushAt = -1, int _flushInterval = -1) + // Bandwidth courtesy on slow/metered connections: a soft byte budget per flush tick, and a + // total-backlog cap (drop-oldest) bounding disk footprint and upload liability. See the + // "Constrained bandwidth" section of the PR #43 description for the pacing rationale. + private const long kMaxBytesPerDrainTick = 256 * 1024; + private const long kDefaultMaxSpoolBytes = 20L * 1024 * 1024; + + // On (re)start, attempt the first drain soon rather than waiting a full interval, so events + // spooled during a PREVIOUS (offline) session go out shortly after launch instead of ~30s later. + private const int kInitialFlushDelaySeconds = 3; + // When we get an opportunistic OS signal that connectivity may have returned (a network address + // change), reschedule the next drain this soon instead of waiting for the next poll tick. + private const int kReconnectFlushDelaySeconds = 1; + + // Bound on Flush()/ShutDown(): an offline shutdown must return fast rather than hang the + // host's exit. Two independent bounds (wall-clock + attempt count) so either one alone + // failing to trip (e.g. a TimeProvider that never advances in a test) still terminates. + private static readonly TimeSpan s_boundedDrainDuration = TimeSpan.FromSeconds(5); + private const int kBoundedDrainMaxAttempts = 20; + + private EventSpool _spool; + private IEventSender _sender; + private TimeProvider _timeProvider = TimeProvider.System; + private ResiliencePipeline _pipeline = ResiliencePipeline.Empty; + private Mixpanel.MixpanelClient _identifyClient; + private Timer _flushTimer; + private int _batchSize = kDefaultBatchSize; + // Normalized once in Initialize (clamped to >= 1s there); everywhere else just reads it. + private TimeSpan _flushInterval = TimeSpan.FromSeconds(kDefaultFlushIntervalSeconds); + // Set by Initialize/InitializeForTest. Track() throws if this is still false -- calling + // Track on a never-initialized client is a programming error in the host, matching the + // pre-durability behavior (the old client dereferenced a null field there). + private bool _initialized; + // Paused == consent revoked (see PurgeQueuedEvents): the poll timer is stopped and opportunistic + // network-change kicks are ignored until ResumeSending re-enables them. + private volatile bool _paused; + + // Cancels whatever timer-driven send (DrainOnceAsync) is currently in flight when consent is + // revoked (see PurgeQueuedEvents), so Purge() does not sit blocked behind a network round trip + // and the batch that send was carrying rolls back into the spool instead of being delivered + // after revocation. Swapped for a fresh instance on every purge; never used across a purge + // boundary. Guarded by Interlocked.Exchange since PurgeQueuedEvents can run on a different + // thread (e.g. a UI thread via Analytics.AllowTracking) than the timer-driven drain it cancels. + private CancellationTokenSource _sendCts = new CancellationTokenSource(); + + private int _submitted; + private int _succeeded; + private int _failed; + + // Guards only against overlapping TIMER ticks: if a previous tick's drain is still running + // when the next tick fires, the new tick is skipped. It does NOT serialize ticks against + // Flush/ShutDown/DrainOnceAsync -- those may run concurrently with a tick, which is safe + // because all spool access is serialized inside EventSpool (its semaphore); this flag just + // keeps a slow drain from stacking up redundant timer callbacks behind it. + private int _timerDraining; + + /// + /// Production initializer (via ). Builds a real on-disk spool keyed by + /// , a real HTTP-posting , and a + /// Polly retry+circuit-breaker pipeline, then starts the background flush timer. + /// + /// Not supported by the Mixpanel client; passing a non-empty value throws + /// (this is the long-standing contract — only + /// honors a host). + public void Initialize(string apiSecret, string host = null, int flushAt = -1, int flushInterval = -1) { - // currently only the SegmentClient uses the host parameter - if (host != null) + // Preserve the original contract: unlike SegmentClient, the Mixpanel client does not support + // pointing at a different host. Validated up front, OUTSIDE the catch below, so a + // misconfiguration surfaces to the caller exactly as it always has rather than being swallowed. + if (!string.IsNullOrEmpty(host)) + throw new ArgumentException("MixpanelClient does not currently support a host parameter", nameof(host)); + + _initialized = true; + + try { - throw new ArgumentException("MixpanelClient does not currently support a host parameter"); + _batchSize = flushAt > 0 ? flushAt : kDefaultBatchSize; + // Clamp to >= 1s once, here, so every later consumer can use the value as-is. + _flushInterval = TimeSpan.FromSeconds( + Math.Max(1, flushInterval > 0 ? flushInterval : kDefaultFlushIntervalSeconds)); + + _timeProvider = TimeProvider.System; + _spool = new EventSpool(EventSpool.GetDefaultSpoolPath(apiSecret), kDefaultMaxSpoolItems, + kMaxSpooledEventBytes, kDefaultMaxSpoolBytes); + _spool.ItemDroppedByCap += OnItemDroppedByCap; + _sender = new MixpanelEventSender(apiSecret); + _pipeline = BuildDefaultPipeline(_timeProvider); + + try + { + // Best-effort only; used solely for the (unspooled) Identify call. + _identifyClient = new Mixpanel.MixpanelClient(apiSecret); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Initialize: failed to create identify client: " + e); + } + + StartTimer(); + SubscribeToNetworkChanges(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Initialize failed: " + e); } - _client = new Mixpanel.MixpanelClient(apiSecret); } - public void ShutDown() + /// + /// Test-only initializer: injects every non-deterministic dependency directly (the spool, the + /// sender, the clock, and the resilience pipeline) and deliberately does NOT start the + /// background timer -- tests pump manually instead of racing a real + /// timer thread. See offline-analytics.md, "Design-for-test seams". + /// + /// If null, defaults to + /// (no retry, no circuit breaker) so that, by default, one call maps + /// to exactly one call -- the simplest, + /// most deterministic shape for most tests. Tests that specifically want to exercise Polly + /// retry/circuit-breaker behavior pass their own pipeline (see + /// ). + internal void InitializeForTest( + EventSpool spool, + IEventSender sender, + TimeProvider timeProvider = null, + ResiliencePipeline pipeline = null, + int batchSize = kDefaultBatchSize) + { + _initialized = true; + _spool = spool; + if (_spool != null) + _spool.ItemDroppedByCap += OnItemDroppedByCap; + _sender = sender; + _timeProvider = timeProvider ?? TimeProvider.System; + _pipeline = pipeline ?? ResiliencePipeline.Empty; + _batchSize = batchSize; + } + + // Keeps Statistics.Failed (and therefore Statistics.Submitted == Succeeded + Failed) accurate + // for events dropped later by cap enforcement, not just ones dropped at enqueue time -- see + // EventSpool.ItemDroppedByCap. + private void OnItemDroppedByCap() + { + Interlocked.Increment(ref _failed); + } + + /// + /// Builds the production Polly pipeline: a small bounded retry (exponential backoff + + /// jitter) for quick transient blips, wrapped in a circuit breaker so a sustained outage backs + /// off instead of hammering the network on every flush tick. Exposed (internal, static) so + /// tests that specifically want to exercise real Polly retry/circuit-breaker behavior -- + /// rather than the zero-strategy default used by -- can build + /// one with a deterministic (e.g. a zero-delay variant; see + /// MixpanelClientTests for the rationale). + /// + internal static ResiliencePipeline BuildDefaultPipeline( + TimeProvider timeProvider, + int maxRetryAttempts = 2, + TimeSpan? retryDelay = null) + { + bool ShouldHandleOutcome(Outcome outcome) => + outcome.Exception != null || outcome.Result?.Outcome == SendResult.RetryableFailure; + + var retryOptions = new RetryStrategyOptions + { + ShouldHandle = args => new ValueTask(ShouldHandleOutcome(args.Outcome)), + MaxRetryAttempts = maxRetryAttempts, + Delay = retryDelay ?? TimeSpan.FromMilliseconds(250), + BackoffType = DelayBackoffType.Exponential, + UseJitter = true + }; + + var breakerOptions = new CircuitBreakerStrategyOptions + { + ShouldHandle = args => new ValueTask(ShouldHandleOutcome(args.Outcome)), + FailureRatio = 0.5, + // Matches attempts-per-tick: a fully failing drain tick produces exactly 3 + // outcomes through this pipeline (1 initial attempt + MaxRetryAttempts=2 retries), + // all within the same instant, so all 3 always land in one SamplingDuration window. + // With this at 4 (the previous value), a single tick could never reach the minimum + // throughput -- and successive ticks are kDefaultFlushIntervalSeconds (30s) apart, + // far outside the 10s window -- so the breaker could never open at all during a + // sustained outage. 3 makes one bad tick enough to trip it. + MinimumThroughput = 3, + SamplingDuration = TimeSpan.FromSeconds(10), + BreakDuration = TimeSpan.FromSeconds(5) + }; + + var builder = new ResiliencePipelineBuilder + { + TimeProvider = timeProvider ?? TimeProvider.System + }; + return builder + .AddRetry(retryOptions) + .AddCircuitBreaker(breakerOptions) + .Build(); + } + + private void StartTimer() { - var totalWait = 0; - while (_tasks.Any(t => !t.IsCompleted)) + try + { + // Fire the first drain soon after launch (bounded by the interval) so events left in the + // spool by a previous offline session are attempted promptly, not a full interval later. + var initialDelay = TimeSpan.FromTicks( + Math.Min(TimeSpan.FromSeconds(kInitialFlushDelaySeconds).Ticks, _flushInterval.Ticks)); + _paused = false; + // Fire-and-forget by design: OnTimerTickAsync catches everything internally, so the + // discarded task can never fault unobserved. + _flushTimer = new Timer(_ => _ = OnTimerTickAsync(), null, initialDelay, _flushInterval); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.StartTimer failed: " + e); + } + } + + private void SubscribeToNetworkChanges() + { + try + { + NetworkChange.NetworkAddressChanged += OnNetworkAddressChanged; + } + catch (Exception e) + { + // Not fatal: the periodic poll timer is the RELIABLE delivery mechanism. This subscription + // is only an opportunistic accelerator, and some platforms may not support it. + Debug.WriteLine("MixpanelClient.SubscribeToNetworkChanges failed: " + e); + } + } + + private void UnsubscribeFromNetworkChanges() + { + try + { + NetworkChange.NetworkAddressChanged -= OnNetworkAddressChanged; + } + catch (Exception e) { - if (totalWait > 7500) - break; - // REVIEW: I modeled this waiting off the segment client, there might be - // better metrics for Mixpanel, but I'm in a rush - totalWait += 500; - Thread.Sleep(500); + Debug.WriteLine("MixpanelClient.UnsubscribeFromNetworkChanges failed: " + e); } } + // Opportunistic accelerator: when the OS reports a network address change (e.g. Wi-Fi reconnecting + // after being offline), reschedule the next drain to happen almost immediately instead of waiting + // out the remainder of the poll interval. The periodic timer remains the guarantee -- this only + // improves reconnect latency. Ignored while paused (consent revoked). Never throws. + private void OnNetworkAddressChanged(object sender, EventArgs e) + { + try + { + if (_paused) + return; + if (!NetworkInterface.GetIsNetworkAvailable()) + return; + _flushTimer?.Change( + TimeSpan.FromSeconds(kReconnectFlushDelaySeconds), _flushInterval); + } + catch (Exception ex) + { + Debug.WriteLine("MixpanelClient.OnNetworkAddressChanged failed: " + ex); + } + } + + // Starts on a ThreadPool thread (System.Threading.Timer) and continues on the pool after + // awaits (ConfigureAwait(false) throughout the drain path) -- never the UI thread. + private async Task OnTimerTickAsync() + { + if (Interlocked.CompareExchange(ref _timerDraining, 1, 0) != 0) + return; // A previous tick is still draining; skip. + + try + { + await DrainOnceAsync().ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient: background drain failed: " + e); + } + finally + { + Interlocked.Exchange(ref _timerDraining, 0); + } + } + + /// + /// One-shot attempt to deliver up to one batch of spooled events. This is exactly what the + /// background timer awaits on each tick; tests await it directly to pump the flush loop + /// deterministically instead of racing a real timer (see offline-analytics.md, "Flush-loop + /// scheduling" in the design-for-test-seams table). Always goes through the full Polly + /// pipeline (retry + circuit breaker) with no cancellation -- + /// is the bounded/cancellable variant used by Flush/ShutDown. + /// + internal Task DrainOnceAsync() + { + return DrainOnceCoreAsync(_sendCts.Token, usePipeline: true); + } + + // Shared implementation behind DrainOnceAsync() and BoundedDrainAsync(). usePipeline is false + // only for BoundedDrainAsync's bounded attempts (see its comment for why retries are skipped + // there). + private async Task DrainOnceCoreAsync(CancellationToken cancellationToken, bool usePipeline) + { + try + { + if (_spool == null) + return; + await _spool.ProcessBatchAsync(_batchSize, + (batch, ct) => SendBatchGuardedAsync(batch, ct, usePipeline), + kMaxBytesPerDrainTick, cancellationToken) + .ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.DrainOnce failed: " + e); + } + } + + // The callback EventSpool.ProcessBatchAsync awaits with the gathered batch; the SendResult + // returned is the whole batch's fate (commit vs roll back -- per-record rejections within a + // processed batch are counted in the statistics here and still commit). Must never throw or + // fault (see EventSpool.ProcessBatchAsync's contract: a thrown exception is treated as a + // RetryableFailure anyway, but returning it directly avoids relying on that fallback). + private async Task SendBatchGuardedAsync(IReadOnlyList batch, + CancellationToken cancellationToken, bool usePipeline) + { + try + { + BatchSendResult result; + try + { + result = usePipeline + ? await _pipeline.ExecuteAsync( + async ct => await _sender.SendBatchAsync(batch, ct).ConfigureAwait(false), + cancellationToken) + .ConfigureAwait(false) + : await _sender.SendBatchAsync(batch, cancellationToken).ConfigureAwait(false); + } + catch (BrokenCircuitException e) + { + // Circuit is open from a sustained outage -- back off without even trying the + // network this round. Leave the batch for a later retry. + Debug.WriteLine("MixpanelClient: circuit breaker open, leaving batch for retry: " + e); + return SendResult.RetryableFailure; + } + catch (OperationCanceledException e) + { + // BoundedDrain's per-attempt deadline (see its comment) fired before the sender + // finished -- treat exactly like any other transient failure so the batch stays + // spooled for the next drain, rather than letting this hang or throw into the host. + Debug.WriteLine( + "MixpanelClient: send canceled (bounded drain deadline), leaving batch for retry: " + e); + return SendResult.RetryableFailure; + } + + switch (result.Outcome) + { + case SendResult.Delivered: + var failed = result.FailedIndices.Count; + Interlocked.Add(ref _succeeded, Math.Max(0, batch.Count - failed)); + if (failed > 0) + Interlocked.Add(ref _failed, failed); + break; + case SendResult.PoisonDrop: + Interlocked.Add(ref _failed, batch.Count); + break; + } + + return result.Outcome; + } + catch (Exception e) + { + // The sender (or the pipeline itself) threw and retries -- if any -- were exhausted. + // Never let that propagate: leave the batch in the spool for the next drain. + Debug.WriteLine("MixpanelClient: send pipeline threw unexpectedly, leaving batch for retry: " + + e); + return SendResult.RetryableFailure; + } + } + + /// + /// Best-effort, non-spooled identify call. Deliberately out of scope for the durability work + /// here (see offline-analytics.md): traits are small, low-value to replay, and bundling them + /// into the same at-least-once spool as usage/exception events would complicate dedup for + /// little benefit. Never throws; failures are logged and otherwise ignored. + /// public void Identify(string analyticsId, JsonObject traits, JsonObject options) { - _tasks.Add(_client.PeopleSetAsync(analyticsId, traits)); + try + { + if (_identifyClient == null) + return; + + var task = _identifyClient.PeopleSetAsync(analyticsId, traits); + task.ContinueWith(t => + { + if (t.Exception != null) + Debug.WriteLine("MixpanelClient.Identify: best-effort identify failed: " + + t.Exception); + }, TaskContinuationOptions.OnlyOnFaulted); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Identify failed: " + e); + } } + /// + /// Scrubs the event name and every string property value (this is where stack traces -- + /// spooled as the "Stack Trace" property of the "Exception" event -- get their embedded user + /// paths normalized), stamps $insert_id/time via , and + /// enqueues into the spool. An event over Mixpanel's per-event size limit (see + /// ) is refused at enqueue and counted in + /// . Never throws once initialized; calling it on a client + /// that was never initialized at all is a programming error in the host and throws + /// (matching the original MixpanelClient, which dereferenced a null field in that case). + /// public void Track(string analyticsId, string eventName, JsonObject properties) { - _tasks.Add(_client.TrackAsync(eventName, analyticsId, properties)); + // Deliberately OUTSIDE the catch-all below: a never-initialized client must surface the + // programming error rather than silently dropping every event. Distinct from _spool == + // null, which means Initialize ran but the spool could not be created (e.g. another + // instance holds the cross-process lock) -- that case degrades to a no-op by design. + if (!_initialized) + throw new InvalidOperationException( + "MixpanelClient.Track called before Initialize"); + + try + { + if (_spool == null) + return; + + var scrubbedName = PathScrubber.Scrub(eventName); + var scrubbedProperties = ScrubProperties(properties); + + var evt = AnalyticsEvent.Create( + analyticsId, + scrubbedName, + scrubbedProperties, + time: _timeProvider.GetUtcNow()); + + Interlocked.Increment(ref _submitted); + if (!_spool.Enqueue(evt)) + { + // Dropped at enqueue (most likely over the per-event size cap; see + // kMaxSpooledEventBytes) -- surface it in the statistics like any other + // undeliverable event. + Interlocked.Increment(ref _failed); + } + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Track failed: " + e); + } + } + + private static JsonObject ScrubProperties(JsonObject properties) + { + var result = new JsonObject(); + if (properties == null) + return result; + + foreach (var kv in properties) + result[kv.Key] = ScrubValue(kv.Value); + + return result; + } + + // Recurses into nested JsonObject/JsonArray values so a path embedded anywhere in the + // property tree gets scrubbed, not just top-level string properties -- e.g. a structured + // "Context": { "Path": ... } property. + private static JsonElement ScrubValue(JsonElement value) + { + if (value is JsonPrimitive primitive && primitive.IsString) + return PathScrubber.Scrub(primitive.Content); + + if (value is JsonObject nested) + return ScrubProperties(nested); + + if (value is JsonArray array) + { + var scrubbedArray = new JsonArray(); + foreach (var item in array) + scrubbedArray.Add(ScrubValue(item)); + return scrubbedArray; + } + + return value; + } + + /// + /// Bounded drain: repeatedly attempts a drain until the spool is empty or a bound + /// (wall-clock duration or attempt count) is hit, then returns. Used by both + /// and so an offline flush/shutdown returns fast + /// instead of hanging the host. + /// + /// + /// Two things make this provably bounded even against a "black hole" server (one that + /// accepts the TCP connection but never responds -- captive portals and some firewalls do + /// this), which the plain path is not: + /// + /// A single covering the whole wall-clock + /// budget is shared by every attempt, passed all the way down to + /// via . Without + /// this, the deadline below is only checked BETWEEN attempts -- one hanging attempt could + /// run as long as the sender's own retry-multiplied HTTP timeout (e.g. 3 attempts x a 15s + /// HttpClient timeout ~= 45s) before the bound is ever checked. Created via the injected + /// (not new CancellationTokenSource(delay), which only + /// knows the system clock) so tests keep deterministic control of the deadline. The + /// caller's token (if any) is linked into it via Register, so external cancellation and + /// the deadline flow through the same token. + /// Retries are bypassed (usePipeline: false): retrying during a bounded + /// shutdown/flush has little delivery value (we are about to give up on this attempt + /// anyway) and would multiply -- rather than bound -- the time spent waiting on a hanging + /// server. + /// + /// + private async Task BoundedDrainAsync(CancellationToken cancellationToken) + { + if (_spool == null) + return; + + try + { + using (var cts = _timeProvider.CreateCancellationTokenSource(s_boundedDrainDuration)) + using (cancellationToken.Register(cts.Cancel)) + { + for (var i = 0; i < kBoundedDrainMaxAttempts; i++) + { + if (cts.IsCancellationRequested) + return; + + if (_spool.ApproximateCount <= 0) + return; + + await DrainOnceCoreAsync(cts.Token, usePipeline: false).ConfigureAwait(false); + } + } + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.BoundedDrain failed: " + e); + } + } + + /// Bounded drain; returns fast even while offline. See . + /// Never faults. + /// Optionally ends the drain even sooner than its own + /// wall-clock bound; undelivered events stay spooled. Cancellation never faults the task. + public async Task FlushAsync(CancellationToken cancellationToken = default) + { + try + { + await BoundedDrainAsync(cancellationToken).ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Flush failed: " + e); + } } - // Flush is really a no-op in our current Mixpanel client + /// Synchronous . + /// + /// Blocks on the bounded drain -- safe even from a UI thread because the entire drain path + /// awaits with ConfigureAwait(false) (no continuation ever needs the caller's + /// SynchronizationContext), and the drain itself is bounded to a few seconds. + /// public void Flush() { + try + { + FlushAsync(CancellationToken.None).GetAwaiter().GetResult(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.Flush failed: " + e); + } } - public Statistics Statistics => new Statistics(_tasks.Count, - _tasks.Count(t => t.IsCompleted && t.Result), _tasks.Count(t => t.IsCompleted && !t.Result)); + /// + /// Bounded drain, then release the spool's cross-process lock. Never hangs, even fully + /// offline: undelivered events simply remain on disk for the next launch to pick up. + /// Never faults, and is idempotent -- a redundant (e.g. Dispose on + /// the facade after an explicit ShutDownAsync) finds a stopped + /// timer, an empty-reporting spool, and a no-op re-Dispose. + /// + /// Optionally ends the final drain even sooner than its own + /// wall-clock bound; undelivered events stay on disk. The spool's lock is released either + /// way, and cancellation never faults the task. + public async Task ShutDownAsync(CancellationToken cancellationToken = default) + { + try + { + StopTimerPermanently(); + await BoundedDrainAsync(cancellationToken).ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.ShutDown failed: " + e); + } + finally + { + try + { + _spool?.Dispose(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.ShutDown: failed to dispose spool: " + e); + } + } + } + + /// Synchronous . See for why + /// blocking here is safe. + public void ShutDown() + { + try + { + ShutDownAsync(CancellationToken.None).GetAwaiter().GetResult(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.ShutDown failed: " + e); + } + } + + /// + /// Consent revocation: empties the spool immediately and pauses the flush loop (the timer is + /// stopped and opportunistic network-change kicks are ignored). This is a pause, not a teardown: + /// if consent is granted again within the same process run, Analytics.AllowTracking's + /// setter calls to re-arm the loop. + /// + public void PurgeQueuedEvents() + { + try + { + PauseTimer(); + CancelInFlightSend(); + _spool?.Purge(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.PurgeQueuedEvents failed: " + e); + } + } + + // Aborts whatever timer-driven send is currently in flight (if any) so Purge() does not block + // behind a network round trip and the in-flight batch rolls back into the spool -- where the + // impending Purge() removes it -- instead of being delivered to Mixpanel after consent was + // just revoked. Swaps in a fresh, non-canceled token so the NEXT drain (post-purge, or after + // ResumeSending) is unaffected. + private void CancelInFlightSend() + { + var previous = Interlocked.Exchange(ref _sendCts, new CancellationTokenSource()); + try + { + previous.Cancel(); + } + finally + { + previous.Dispose(); + } + } + + /// + /// Re-arms the flush loop after paused it (consent revoked then + /// granted again in the same run). Clears the paused flag and reschedules the timer to fire + /// shortly so newly tracked events go out promptly. No-op on the timer if the client was never + /// initialized (e.g. deferred-init clients) or has been shut down. + /// + public void ResumeSending() + { + try + { + // Clear the flag unconditionally so the paused state is correct even for a client whose + // timer was never started (deferred init); reschedule only if there is a live timer. + _paused = false; + _flushTimer?.Change( + TimeSpan.FromSeconds(kReconnectFlushDelaySeconds), _flushInterval); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.ResumeSending failed: " + e); + } + } + + private void PauseTimer() + { + try + { + _paused = true; + _flushTimer?.Change(Timeout.Infinite, Timeout.Infinite); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient: failed to pause timer: " + e); + } + } + + private void StopTimerPermanently() + { + try + { + _paused = true; + UnsubscribeFromNetworkChanges(); + _flushTimer?.Change(Timeout.Infinite, Timeout.Infinite); + _flushTimer?.Dispose(); + _flushTimer = null; + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient: failed to stop timer: " + e); + } + } + + public Statistics Statistics => new Statistics(_submitted, _succeeded, _failed); + + // Test seam: exposes whether the flush loop is currently paused (consent revoked). See + // MixpanelClientTests. Not part of IClient. + internal bool SendingPaused => _paused; } -} \ No newline at end of file +} diff --git a/src/DesktopAnalytics/MixpanelEventSender.cs b/src/DesktopAnalytics/MixpanelEventSender.cs new file mode 100644 index 0000000..edfc3c7 --- /dev/null +++ b/src/DesktopAnalytics/MixpanelEventSender.cs @@ -0,0 +1,273 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Newtonsoft.Json.Linq; +using Segment.Serialization; + +namespace DesktopAnalytics +{ + /// + /// The real (network-hitting) : posts a batch of events in a single + /// request to Mixpanel's HTTP "import" endpoint (https://docs.mixpanel.com/reference/import-events) + /// and classifies the response into a for the flush loop. + /// + /// + /// /import rather than /track (PR #43 review): it accepts a plain project token for auth (basic + /// auth username with an empty password), takes up to 2,000 events per request -- one round trip + /// per batch matters to users on poor connections -- has no /track-style 5-day age limit on + /// replayed events (an arbitrarily old offline backlog is accepted), and with strict=1 + /// reports per-record rejections (failed_records) so a poison event can be dropped without + /// discarding the good events sent alongside it. + /// Deliberately does not go through the mixpanel-csharp package for this path: that package's + /// send reports success/failure as a single bool, which is not enough information + /// to distinguish "retry me" from "poison, drop me" (see offline-analytics.md, "Failure + /// classification"). Posting the well-documented raw HTTP form directly keeps that distinction and + /// makes the base URL trivially swappable for tests (WireMock.Net). mixpanel-csharp is still + /// used, elsewhere, for the best-effort (non-spooled) Identify call. + /// + internal class MixpanelEventSender : IEventSender, IDisposable + { + internal const string kDefaultBaseUrl = "https://api.mixpanel.com"; + + private readonly string _baseUrl; + private readonly HttpClient _httpClient; + private readonly bool _ownsHttpClient; + // /import authenticates with the project token as the basic-auth username (empty password). + private readonly AuthenticationHeaderValue _authorization; + + /// The Mixpanel project token. + /// Base URL of the Mixpanel HTTP API. Defaults to the real Mixpanel + /// endpoint; tests override this with a WireMock.Net base URL. + /// Injectable so tests can supply an already + /// configured (e.g. short timeout) or point it at a local stub. If omitted, this instance owns + /// and disposes its own . + public MixpanelEventSender(string apiSecret, string baseUrl = kDefaultBaseUrl, HttpClient httpClient = null) + { + _authorization = new AuthenticationHeaderValue("Basic", + Convert.ToBase64String(Encoding.ASCII.GetBytes((apiSecret ?? string.Empty) + ":"))); + _baseUrl = string.IsNullOrEmpty(baseUrl) + ? kDefaultBaseUrl + : baseUrl.TrimEnd('/'); + + if (httpClient != null) + { + _httpClient = httpClient; + _ownsHttpClient = false; + } + else + { + // AllowAutoRedirect is disabled deliberately: with the default handler (which follows + // redirects), a captive portal (hotel/coffee-shop Wi-Fi login page) that intercepts + // this POST would 302 us to its login page, HttpClient would follow it, get back a + // harmless 200, and the code below would classify that as Delivered -- silently + // losing the batch even though Mixpanel never received it. With redirects disabled we + // see the raw 3xx instead and can classify it correctly (see the status-code check + // below). + _httpClient = new HttpClient(new HttpClientHandler { AllowAutoRedirect = false }) + { Timeout = TimeSpan.FromSeconds(15) }; + _ownsHttpClient = true; + } + } + + /// + /// Posts to Mixpanel's /import endpoint (strict mode) in a single + /// request and classifies the result. Never throws (and never returns a faulted task): any + /// serialization, network, or cancellation failure is caught and reported as a + /// (see class remarks for the classification rules). + /// + /// See . A bounded + /// caller ('s BoundedDrain) uses this to abandon a request against + /// a server that accepts the connection but never responds; that is reported as + /// , exactly like any other transient failure. + public async Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + try + { + if (events == null || events.Count == 0) + return BatchSendResult.Delivered; + + // Serialize each event individually so one unserializable event (which will never + // succeed) is reported as poison via FailedIndices instead of retrying -- or + // poisoning -- the whole batch. payloadToBatchIndex maps positions in the JSON array + // actually sent back to positions in the incoming batch. + var payloads = new List(events.Count); + var payloadToBatchIndex = new List(events.Count); + var unserializable = new List(); + for (var i = 0; i < events.Count; i++) + { + try + { + payloads.Add(BuildEventJson(events[i])); + payloadToBatchIndex.Add(i); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelEventSender.SendBatch: failed to build payload, dropping: " + e); + unserializable.Add(i); + } + } + + if (payloads.Count == 0) + return new BatchSendResult(SendResult.Delivered, unserializable); + + var body = "[" + string.Join(",", payloads) + "]"; + + HttpResponseMessage response; + try + { + using (var request = new HttpRequestMessage(HttpMethod.Post, _baseUrl + "/import?strict=1")) + { + request.Headers.Authorization = _authorization; + request.Content = new StringContent(body, Encoding.UTF8, "application/json"); + response = await _httpClient.SendAsync(request, cancellationToken) + .ConfigureAwait(false); + } + } + catch (OperationCanceledException e) + { + // The caller's deadline (see MixpanelClient.BoundedDrain) fired before the server + // responded -- most commonly a "black hole" server that accepts the TCP connection + // but never replies (captive portals and some firewalls do this). Treat exactly + // like any other transient failure: leave the batch in the spool for a later + // retry rather than throwing into (and hanging) the caller. + Debug.WriteLine("MixpanelEventSender.SendBatch: canceled (deadline), will retry: " + e); + return BatchSendResult.Retryable; + } + catch (Exception e) + { + // Connection refused, DNS failure, timeout, etc. -- transient from our point of + // view, so leave the batch in the spool for a later retry. + Debug.WriteLine("MixpanelEventSender.SendBatch: network failure, will retry: " + e); + return BatchSendResult.Retryable; + } + + using (response) + { + var status = (int)response.StatusCode; + if (status >= 200 && status < 300) + { + // Strict-mode /import: 2xx means every record was validated and ingested. + return new BatchSendResult(SendResult.Delivered, unserializable); + } + + if (status >= 300 && status < 400) + { + // With AllowAutoRedirect disabled (see the constructor), a 3xx here means + // something intercepted the POST before it reached Mixpanel -- most commonly a + // captive portal redirecting to its login page. Mixpanel's /import endpoint + // never legitimately redirects, so treat this as an intercepting middlebox: + // leave the batch spooled for retry instead of risking a misclassification. + return BatchSendResult.Retryable; + } + + if (status == 408 || status == 429 || status >= 500) + return BatchSendResult.Retryable; + + if (status == 400) + { + // Strict-mode /import returns 400 with a failed_records array when SOME records + // fail validation -- the rest were ingested. Those records are permanently + // invalid (poison); report their indices so only they are counted as failed. + var failed = await TryParseFailedRecordsAsync(response, payloadToBatchIndex) + .ConfigureAwait(false); + if (failed != null) + { + foreach (var i in unserializable) + failed.Add(i); + return new BatchSendResult(SendResult.Delivered, failed); + } + + // A 400 without a parseable failed_records body means the request as a whole + // was rejected (e.g. malformed JSON) -- it will never succeed by retrying. + return BatchSendResult.Poison; + } + + // Any other 4xx (or any other unexpected status) will never succeed by retrying -- + // treat the batch as poison so it cannot wedge the spool. + return BatchSendResult.Poison; + } + } + catch (OperationCanceledException e) + { + Debug.WriteLine("MixpanelEventSender.SendBatch: canceled (deadline), will retry: " + e); + return BatchSendResult.Retryable; + } + catch (Exception e) + { + // Belt-and-suspenders: analytics must never throw into the host. Treat anything + // unexpected as retryable rather than silently dropping the batch. + Debug.WriteLine("MixpanelEventSender.SendBatch failed unexpectedly, will retry: " + e); + return BatchSendResult.Retryable; + } + } + + // Returns the batch indices of the records reported in the response's failed_records array, + // or null if the body has no parseable failed_records (in which case the caller treats the + // whole batch as poison). The response's "index" values refer to positions in the JSON array + // we sent, which payloadToBatchIndex maps back to positions in the incoming batch (they + // differ when an unserializable event was excluded from the payload). + private static async Task> TryParseFailedRecordsAsync(HttpResponseMessage response, + List payloadToBatchIndex) + { + try + { + var body = await response.Content.ReadAsStringAsync().ConfigureAwait(false); + if (!(JToken.Parse(body) is JObject parsed) || + !(parsed["failed_records"] is JArray failedRecords)) + return null; + + var failed = new HashSet(); + foreach (var record in failedRecords) + { + var index = record?["index"]?.Value(); + if (index == null || index < 0 || index >= payloadToBatchIndex.Count) + return null; // An index we can't map -- treat the body as unparseable. + failed.Add(payloadToBatchIndex[index.Value]); + } + return failed; + } + catch (Exception e) + { + Debug.WriteLine("MixpanelEventSender.SendBatch: failed to parse 400 body: " + e); + return null; + } + } + + private static string BuildEventJson(AnalyticsEvent evt) + { + // No token property: /import authenticates via the Authorization header instead. + var properties = new JsonObject + { + { "distinct_id", evt.AnalyticsId }, + { "$insert_id", evt.InsertId }, + { "time", evt.Time.ToUnixTimeSeconds() } + }; + + if (evt.Properties != null) + { + foreach (var kv in evt.Properties) + properties[kv.Key] = kv.Value; + } + + var payload = new JsonObject + { + { "event", evt.EventName }, + { "properties", properties } + }; + + return JsonUtility.ToJson(payload, false); + } + + public void Dispose() + { + if (_ownsHttpClient) + _httpClient?.Dispose(); + } + } +} diff --git a/src/DesktopAnalytics/PathScrubber.cs b/src/DesktopAnalytics/PathScrubber.cs new file mode 100644 index 0000000..c4475e6 --- /dev/null +++ b/src/DesktopAnalytics/PathScrubber.cs @@ -0,0 +1,39 @@ +using System; +using System.Text.RegularExpressions; + +namespace DesktopAnalytics +{ + /// + /// Normalizes user home paths embedded in strings (e.g. exception stack traces) so that the + /// OS account name does not leak when spooled/reported. This is not a general PII scrubber; + /// it only targets the well-known Windows and Unix "home directory" prefixes. + /// + internal static class PathScrubber + { + // Windows: C:\Users\\ (any drive letter, case-insensitive on both the drive + // letter and the literal "Users" segment). We stop matching the user name at the next + // path separator so we don't eat the rest of the path. + private static readonly Regex s_windowsUserPath = + new Regex(@"[A-Za-z]:\\Users\\[^\\/:*?""<>|\r\n]+\\", + RegexOptions.Compiled | RegexOptions.IgnoreCase); + + // Unix: /home// + private static readonly Regex s_unixUserPath = + new Regex(@"/home/[^/\r\n]+/", RegexOptions.Compiled); + + /// + /// Replaces every occurrence of a Windows or Unix user-home-directory prefix in + /// with a neutral placeholder ("%USER%\" or "%USER%/" + /// respectively). Returns the input unchanged if it is null/empty or contains no match. + /// + public static string Scrub(string input) + { + if (string.IsNullOrEmpty(input)) + return input; + + var result = s_windowsUserPath.Replace(input, @"%USER%\"); + result = s_unixUserPath.Replace(result, "%USER%/"); + return result; + } + } +} diff --git a/src/DesktopAnalytics/SegmentClient.cs b/src/DesktopAnalytics/SegmentClient.cs index 0a53f18..a78c319 100644 --- a/src/DesktopAnalytics/SegmentClient.cs +++ b/src/DesktopAnalytics/SegmentClient.cs @@ -1,5 +1,7 @@ using System; using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; using Segment.Concurrent; using Segment.Serialization; @@ -67,9 +69,44 @@ public void Flush() _analytics.Flush(); } + /// + /// Completes synchronously: Segment.Analytics.CSharp's Flush() only signals the + /// library's own background delivery (its coroutine system) and exposes nothing awaitable, + /// so there is no async work to represent here. + /// + public Task ShutDownAsync(CancellationToken cancellationToken = default) + { + ShutDown(); + return Task.CompletedTask; + } + + /// See for why this completes synchronously. + public Task FlushAsync(CancellationToken cancellationToken = default) + { + Flush(); + return Task.CompletedTask; + } + public Statistics Statistics => new Statistics(StatMonitor.Submitted, StatMonitor.Succeeded, StatMonitor.Failed); + /// + /// The Segment path already gets offline durability from Segment.Analytics.CSharp's own + /// on-disk storage/retry, and this class does not maintain a separate spool of its own -- so + /// there is nothing here to purge on consent revocation. + /// + public void PurgeQueuedEvents() + { + } + + /// + /// No-op: this client has no separate background flush loop of its own to re-arm (the + /// Segment.Analytics.CSharp library manages its own delivery), so there is nothing to resume. + /// + public void ResumeSending() + { + } + public void OnExceptionThrown(Exception e) { Debug.WriteLine($"**** Segment.IO Failed to deliver. {e.Message}"); diff --git a/src/DesktopAnalyticsTests/AnalyticsEventTests.cs b/src/DesktopAnalyticsTests/AnalyticsEventTests.cs new file mode 100644 index 0000000..5065b47 --- /dev/null +++ b/src/DesktopAnalyticsTests/AnalyticsEventTests.cs @@ -0,0 +1,132 @@ +using System; +using DesktopAnalytics; +using NUnit.Framework; +using Segment.Serialization; + +namespace DesktopAnalyticsTests +{ + [TestFixture] + public class AnalyticsEventTests + { + private static string ContentOf(JsonElement element) + { + return element.ToJsonPrimitive().Content; + } + + [Test] + public void ToBytes_FromBytes_RoundTrip_PreservesAllScalarFields() + { + var time = new DateTimeOffset(2026, 7, 8, 12, 34, 56, TimeSpan.Zero); + var original = new AnalyticsEvent + { + AnalyticsId = "abc-123", + EventName = "Save PDF", + Properties = new JsonObject { { "Portion", "All" } }, + InsertId = "insert-guid-1", + Time = time + }; + + var roundTripped = AnalyticsEvent.FromBytes(original.ToBytes()); + + Assert.AreEqual(original.AnalyticsId, roundTripped.AnalyticsId); + Assert.AreEqual(original.EventName, roundTripped.EventName); + Assert.AreEqual(original.InsertId, roundTripped.InsertId); + Assert.AreEqual(original.Time, roundTripped.Time); + } + + [Test] + public void ToBytes_ReturnsUtf8EncodedJson() + { + var evt = new AnalyticsEvent + { + AnalyticsId = "abc-123", + EventName = "Save PDF", + InsertId = "insert-guid-1", + Time = DateTimeOffset.UtcNow + }; + + var bytes = evt.ToBytes(); + var json = System.Text.Encoding.UTF8.GetString(bytes); + + StringAssert.Contains("\"abc-123\"", json); + StringAssert.Contains("\"Save PDF\"", json); + } + + [Test] + public void PropertiesBag_WithMultipleKeys_SurvivesRoundTrip() + { + var original = new AnalyticsEvent + { + AnalyticsId = "abc-123", + EventName = "Exception", + Properties = new JsonObject + { + { "Message", "Oops" }, + { "Stack Trace", "at Foo.Bar()" }, + { "Count", 3 } + }, + InsertId = "insert-guid-2", + Time = DateTimeOffset.UtcNow + }; + + var roundTripped = AnalyticsEvent.FromBytes(original.ToBytes()); + + Assert.AreEqual(3, roundTripped.Properties.Count); + Assert.AreEqual("Oops", ContentOf(roundTripped.Properties["Message"])); + Assert.AreEqual("at Foo.Bar()", ContentOf(roundTripped.Properties["Stack Trace"])); + Assert.AreEqual("3", ContentOf(roundTripped.Properties["Count"])); + } + + [Test] + public void Create_WithNoInjectedDependencies_StampsRealGuidAndRecentTime() + { + var before = DateTimeOffset.UtcNow; + var evt = AnalyticsEvent.Create("abc-123", "Launch"); + var after = DateTimeOffset.UtcNow; + + Assert.IsTrue(Guid.TryParse(evt.InsertId, out _)); + Assert.GreaterOrEqual(evt.Time, before); + Assert.LessOrEqual(evt.Time, after); + } + + [Test] + public void Create_WithInjectedDependencies_StampsExactInjectedValues() + { + var fixedTime = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero); + const string fixedInsertId = "fixed-insert-id"; + + var evt = AnalyticsEvent.Create( + "abc-123", + "Launch", + new JsonObject { { "Version", "1.2.3" } }, + insertId: fixedInsertId, + time: fixedTime + ); + + Assert.AreEqual("abc-123", evt.AnalyticsId); + Assert.AreEqual("Launch", evt.EventName); + Assert.AreEqual(fixedInsertId, evt.InsertId); + Assert.AreEqual(fixedTime, evt.Time); + Assert.AreEqual("1.2.3", ContentOf(evt.Properties["Version"])); + } + + [Test] + public void Create_CalledTwice_ProducesDifferentInsertIdsWhenNotInjected() + { + var first = AnalyticsEvent.Create("abc-123", "Launch"); + var second = AnalyticsEvent.Create("abc-123", "Launch"); + + Assert.AreNotEqual(first.InsertId, second.InsertId); + } + + [Test] + public void Create_WithNullProperties_ProducesEmptyPropertyBag() + { + var evt = AnalyticsEvent.Create("abc-123", "Launch", null, + "insert-id", DateTimeOffset.UtcNow); + + Assert.IsNotNull(evt.Properties); + Assert.AreEqual(0, evt.Properties.Count); + } + } +} diff --git a/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj b/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj index 42efff0..e35283b 100644 --- a/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj +++ b/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj @@ -6,12 +6,17 @@ SIL International False + + true + + diff --git a/src/DesktopAnalyticsTests/EventSpoolTests.cs b/src/DesktopAnalyticsTests/EventSpoolTests.cs new file mode 100644 index 0000000..9bd252a --- /dev/null +++ b/src/DesktopAnalyticsTests/EventSpoolTests.cs @@ -0,0 +1,601 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + // NOTE on the "corrupt/partial spool" edge case (power loss mid-write): rather than + // hand-crafting a truncated transaction-log file matching DiskQueue's private on-disk format + // (which would couple this suite to internals that may change between DiskQueue versions), + // test 10 below appends format-agnostic garbage bytes to the transaction log and asserts the + // spool survives -- exercising DiskQueue's transactional recovery (entries are only + // discoverable after a matching start/end transaction marker pair) without depending on what + // the markers look like. Separately, EventSpool.ProcessBatch guards against an individual + // entry that deserializes badly (e.g. an unreadable/incompatible schema version) by skipping + // (dropping) it like a poison message rather than wedging the spool -- see EventSpool.cs. + [TestFixture] + public class EventSpoolTests + { + private string _spoolDir; + + [SetUp] + public void SetUp() + { + _spoolDir = Path.Combine(Path.GetTempPath(), "EventSpoolTests_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + // Any EventSpool opened by a test must already be disposed (releasing DiskQueue's + // exclusive lock) by the time we get here -- each test disposes its spool(s) via + // `using` before returning. + if (Directory.Exists(_spoolDir)) + { + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + // Best-effort cleanup; don't fail the test run over a leftover temp directory. + Console.WriteLine("EventSpoolTests.TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + } + + private static AnalyticsEvent MakeEvent(string name) + { + return AnalyticsEvent.Create("user-1", name); + } + + // Drains the spool with an always-Delivered batch callback, returning the event names in + // the order they were handed to the sender. + private static async Task> DrainAllDelivered(EventSpool spool, int maxItems = 100) + { + var names = new List(); + await spool.ProcessBatchAsync(maxItems, (batch, ct) => + { + names.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + }); + return names; + } + + // Reads every file under the spool directory that can be opened for shared reading. + // DiskQueue holds its 'lock' file open exclusively while the spool is open; that file + // only ever contains a process id, never event data, so skipping unopenable files does + // not weaken any event-content assertion. + internal static IEnumerable> ReadableSpoolFiles(string spoolDir) + { + foreach (var file in Directory.GetFiles(spoolDir, "*", SearchOption.AllDirectories)) + { + string content; + try + { + using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete)) + using (var reader = new StreamReader(stream, System.Text.Encoding.UTF8)) + content = reader.ReadToEnd(); + } + catch (IOException) + { + continue; // DiskQueue's exclusively-held lock file. + } + + yield return new KeyValuePair(file, content); + } + } + + // 1. RESTART DURABILITY: enqueue N, dispose, reopen a NEW spool on the same directory -- + // all N survive, and a subsequent ProcessBatch (Delivered) empties it. + [Test] + public async Task Enqueue_ThenDisposeAndReopen_EventsSurviveAndDrainFully() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + for (var i = 0; i < 5; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + } // Disposed here, releasing the exclusive lock. + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(5, reopened.ApproximateCount); + + var delivered = await DrainAllDelivered(reopened); + + Assert.AreEqual(5, delivered.Count); + Assert.AreEqual(0, reopened.ApproximateCount); + } + } + + // 2. ProcessBatch where send returns Delivered => queue empty afterward. + [Test] + public async Task ProcessBatch_AllDelivered_EmptiesSpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("A")); + spool.Enqueue(MakeEvent("B")); + + await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.Delivered)); + + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + // 3. send returns RetryableFailure => the WHOLE batch rolls back (nothing is committed -- + // with one request per batch there is no per-event verdict to split on); a later + // ProcessBatch delivers all of them, in order. + [Test] + public async Task ProcessBatch_RetryableFailure_RollsWholeBatchBackForLaterRetry() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("A")); + spool.Enqueue(MakeEvent("B")); + spool.Enqueue(MakeEvent("C")); + + var firstPass = new List(); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + firstPass.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.RetryableFailure); + }); + + CollectionAssert.AreEqual(new[] { "A", "B", "C" }, firstPass); + Assert.AreEqual(3, spool.ApproximateCount, + "a retryable failure must leave the whole batch in the spool"); + + var secondPass = await DrainAllDelivered(spool); + + CollectionAssert.AreEqual(new[] { "A", "B", "C" }, secondPass, + "the rolled-back batch must be retried in its original order"); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + // 4. CRASH-WINDOW: send records the batch then THROWS => nothing removed; reopening the + // spool shows the same events still present (at-least-once, no data loss). + [Test] + public async Task ProcessBatch_SendThrows_NothingRemovedAndEventsSurviveReopen() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Crash-1")); + spool.Enqueue(MakeEvent("Crash-2")); + + var recorded = new List(); + Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => + { + // Simulate a sender that actually delivered the batch over the network, then + // crashed/threw before this method could observe success and flush. + recorded.AddRange(batch.Select(e => e.EventName)); + throw new InvalidOperationException("simulated crash after send, before ack"); + })); + + CollectionAssert.AreEqual(new[] { "Crash-1", "Crash-2" }, recorded); + Assert.AreEqual(2, spool.ApproximateCount); + } + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(2, reopened.ApproximateCount); + + var delivered = await DrainAllDelivered(reopened); + + CollectionAssert.AreEqual(new[] { "Crash-1", "Crash-2" }, delivered); + } + } + + // 5. Poison: send returns PoisonDrop => the batch is removed even though not delivered, + // so a permanently rejected request cannot wedge the spool. + [Test] + public async Task ProcessBatch_PoisonDrop_RemovesBatchEvenThoughNotDelivered() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Poison-1")); + spool.Enqueue(MakeEvent("Poison-2")); + + await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.PoisonDrop)); + + Assert.AreEqual(0, spool.ApproximateCount, + "a poison batch must be dropped, not left to wedge the spool"); + } + } + + // 6. Bounding: maxItems=3, enqueue 5 => count==3, the 3 NEWEST remain. + [Test] + public async Task Enqueue_ExceedingMaxItems_DropsOldestKeepsNewest() + { + using (var spool = new EventSpool(_spoolDir, 3)) + { + for (var i = 0; i < 5; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + Assert.AreEqual(3, spool.ApproximateCount); + + var remaining = await DrainAllDelivered(spool); + + CollectionAssert.AreEqual(new[] { "Event-2", "Event-3", "Event-4" }, remaining); + } + } + + // 7. Purge empties a non-empty spool. + [Test] + public async Task Purge_EmptiesNonEmptySpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + for (var i = 0; i < 5; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + Assert.AreEqual(5, spool.ApproximateCount); + + spool.Purge(); + + Assert.AreEqual(0, spool.ApproximateCount); + + // A drain after Purge should find nothing to send. + var delivered = await DrainAllDelivered(spool); + Assert.AreEqual(0, delivered.Count); + } + } + + // 7b. Purge leaves the spool fully usable: new events can be enqueued in the same + // session, only they survive a dispose/reopen, and the purged events' bytes are gone + // from the on-disk files once the spool is closed (the privacy property of a consent + // purge -- DiskQueue trims consumed entries rather than retaining them). + [Test] + public async Task Purge_ThenEnqueue_SpoolRemainsUsableAndPurgedBytesAreGoneFromDisk() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("PrePurgeSecret")); + Assert.AreEqual(1, spool.ApproximateCount); + + spool.Purge(); + Assert.AreEqual(0, spool.ApproximateCount); + + spool.Enqueue(MakeEvent("PostPurge")); + Assert.AreEqual(1, spool.ApproximateCount); + } + + foreach (var file in ReadableSpoolFiles(_spoolDir)) + { + StringAssert.DoesNotContain("PrePurgeSecret", file.Value, + "a consent purge must remove event data from disk, not just mark it consumed: " + file.Key); + } + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + var delivered = await DrainAllDelivered(reopened); + CollectionAssert.AreEqual(new[] { "PostPurge" }, delivered); + } + } + + // 8. Concurrent enqueue from multiple threads (single process): 4 threads x 25 events => + // all 100 persisted. Dedicated Threads (not Task.Run) so the intended parallelism is + // explicit rather than at the mercy of thread-pool scheduling. + [Test] + public async Task Enqueue_FromMultipleThreadsConcurrently_AllEventsPersisted() + { + const int threadCount = 4; + const int perThread = 25; + + using (var spool = new EventSpool(_spoolDir, threadCount * perThread)) + { + var threads = new Thread[threadCount]; + for (var t = 0; t < threadCount; t++) + { + var threadIndex = t; + threads[t] = new Thread(() => + { + for (var i = 0; i < perThread; i++) + spool.Enqueue(MakeEvent($"T{threadIndex}-{i}")); + }); + } + + foreach (var thread in threads) + thread.Start(); + foreach (var thread in threads) + thread.Join(); + + Assert.AreEqual(threadCount * perThread, spool.ApproximateCount); + + // Confirm every single one is actually retrievable, not just counted. + var seen = new HashSet( + await DrainAllDelivered(spool, threadCount * perThread)); + Assert.AreEqual(threadCount * perThread, seen.Count); + } + } + + // 8a. Per-event byte cap: an event whose serialized form exceeds maxItemBytes is refused + // (Enqueue returns false) and never spooled, while a normal event on the same spool is + // accepted -- and Enqueue's return value distinguishes the two. + [Test] + public void Enqueue_EventLargerThanMaxItemBytes_IsRefusedWhileNormalEventIsAccepted() + { + using (var spool = new EventSpool(_spoolDir, 10, maxItemBytes: 500)) + { + var oversized = AnalyticsEvent.Create("user-1", "Huge", + new Segment.Serialization.JsonObject + { + { "Stack Trace", new string('x', 2000) } + }); + + Assert.IsFalse(spool.Enqueue(oversized), "an event over the byte cap must be refused"); + Assert.AreEqual(0, spool.ApproximateCount); + + Assert.IsTrue(spool.Enqueue(MakeEvent("Normal")), "a normal event must still be accepted"); + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + // 8a2. Byte-based spool cap: enqueuing past maxSpoolBytes drops the oldest event(s), + // exactly like the item cap. + [Test] + public async Task Enqueue_ExceedingMaxSpoolBytes_DropsOldestKeepsNewest() + { + var events = new[] { MakeEvent("Event-A"), MakeEvent("Event-B"), MakeEvent("Event-C"), MakeEvent("Event-D") }; + // Cap sized (from the real serialized lengths) to hold the first three but not all four. + long cap = events[0].ToBytes().Length + events[1].ToBytes().Length + events[2].ToBytes().Length; + + using (var spool = new EventSpool(_spoolDir, 100, maxSpoolBytes: cap)) + { + foreach (var evt in events) + spool.Enqueue(evt); + + var remaining = await DrainAllDelivered(spool); + + CollectionAssert.AreEqual(new[] { "Event-B", "Event-C", "Event-D" }, remaining, + "the byte cap must drop the OLDEST event to make room"); + } + } + + // 8a3. The byte accounting behind the cap survives a restart -- normally read back from the + // total persisted alongside the spool (see ResolveExistingSpoolBytes), falling back to + // re-measuring the live entries if that is missing or untrustworthy -- so a reopened spool + // enforces the cap correctly either way. + [Test] + public void ApproximateBytes_SurvivesDisposeAndReopen() + { + long expectedBytes; + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Event-0")); + spool.Enqueue(MakeEvent("Event-1")); + expectedBytes = spool.ApproximateBytes; + Assert.Greater(expectedBytes, 0); + } + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(expectedBytes, reopened.ApproximateBytes, + "the persisted byte total must be restored on open"); + Assert.AreEqual(2, reopened.ApproximateCount, "opening must not consume the entries"); + } + } + + // 8a3b. If the persisted byte-total sidecar file is missing or corrupt (e.g. this session + // crashed before persisting, or the file predates this feature), the spool must fall back to + // re-measuring from the live entries rather than trusting a stale/absent value -- the cap + // must still be enforced correctly on the next restart. + [Test] + public void ApproximateBytes_MissingPersistedTotal_FallsBackToMeasuring() + { + long expectedBytes; + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Event-0")); + spool.Enqueue(MakeEvent("Event-1")); + expectedBytes = spool.ApproximateBytes; + } + + File.Delete(Path.Combine(_spoolDir, "spool-bytes.txt")); + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(expectedBytes, reopened.ApproximateBytes, + "a missing persisted total must fall back to re-measuring the live entries"); + Assert.AreEqual(2, reopened.ApproximateCount, "measuring must not consume the entries"); + } + } + + // 8a4. ProcessBatch byte budget: the gather stops once the batch reaches maxBytes, but + // a single event over the whole budget still goes (forward progress is guaranteed). + [Test] + public async Task ProcessBatch_ByteBudget_BoundsBatchButNeverStarves() + { + var events = new[] { MakeEvent("Event-0"), MakeEvent("Event-1"), MakeEvent("Event-2"), + MakeEvent("Event-3"), MakeEvent("Event-4") }; + // A budget the first two events fit under but the third pushes past. + long budget = events[0].ToBytes().Length + events[1].ToBytes().Length + 1; + + using (var spool = new EventSpool(_spoolDir, 10)) + { + foreach (var evt in events) + spool.Enqueue(evt); + + var firstBatch = new List(); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + firstBatch.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + }, budget); + + CollectionAssert.AreEqual(new[] { "Event-0", "Event-1", "Event-2" }, firstBatch, + "the budget is checked AFTER each gathered event, so the event that crosses it is still included"); + Assert.AreEqual(2, spool.ApproximateCount); + + // A budget smaller than any single event must still make progress, one event per call. + var secondBatch = new List(); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + secondBatch.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + }, maxBytes: 1); + + CollectionAssert.AreEqual(new[] { "Event-3" }, secondBatch); + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + // 8b. ProcessBatch honors maxItems: with more events spooled than the batch size, exactly + // maxItems are gathered per call and the rest stay put for the next call. + [Test] + public async Task ProcessBatch_MoreEventsThanMaxItems_GathersExactlyMaxItemsPerCall() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + for (var i = 0; i < 5; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + var firstBatch = await DrainAllDelivered(spool, maxItems: 2); + + CollectionAssert.AreEqual(new[] { "Event-0", "Event-1" }, firstBatch); + Assert.AreEqual(3, spool.ApproximateCount); + + var secondBatch = await DrainAllDelivered(spool, maxItems: 2); + + CollectionAssert.AreEqual(new[] { "Event-2", "Event-3" }, secondBatch); + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + // 8c. CANCELLATION: a pre-canceled token means the batch is never gathered or sent, the + // spool is untouched, and nothing throws -- and the send callback receives the SAME token + // that was passed in, so an in-flight send can participate in cancellation. + [Test] + public async Task ProcessBatch_CancellationToken_IsHonoredAndPassedThroughToSend() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("A")); + + using (var cts = new CancellationTokenSource()) + { + cts.Cancel(); + var sendCalled = false; + Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => + { + sendCalled = true; + return Task.FromResult(SendResult.Delivered); + }, cancellationToken: cts.Token)); + + Assert.IsFalse(sendCalled, "a pre-canceled token must short-circuit before the send"); + Assert.AreEqual(1, spool.ApproximateCount, "the event must remain spooled"); + } + + using (var cts = new CancellationTokenSource()) + { + CancellationToken observed = default; + await spool.ProcessBatchAsync(10, (batch, ct) => + { + observed = ct; + return Task.FromResult(SendResult.Delivered); + }, cancellationToken: cts.Token); + + Assert.AreEqual(cts.Token, observed, + "the send callback must receive the caller's token so it can participate in cancellation"); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + } + + // 9. Second EventSpool opened on the SAME directory while the first is still open + // fails/throws, proving the cross-process exclusive lock (DiskQueue.PersistentQueue.WaitFor + // with a short internal timeout) does not silently allow shared access. + [Test] + public void Constructor_SecondSpoolOnSameDirectoryWhileFirstOpen_Throws() + { + using (new EventSpool(_spoolDir, 10)) + { + // Assert.Catch (unlike Assert.Throws, which requires an *exact* type + // match) accepts any exception assignable to Exception -- we don't want this test + // coupled to DiskQueue's specific exception type (currently TimeoutException) for + // a lock-acquisition failure. + Assert.Catch(() => + { + using (new EventSpool(_spoolDir, 10)) + { + } + }); + } + } + + // 10. CORRUPTION, torn data-file write: garbage appended to a data file (bytes beyond the + // extents any committed transaction references) must not prevent reopening the spool, and + // events from committed transactions must still be readable. Appending garbage -- rather + // than hand-crafting file contents -- keeps this test independent of DiskQueue's private + // on-disk format (see the NOTE at the top of this fixture). + [Test] + public async Task Constructor_GarbageAppendedToDataFile_SpoolReopensAndCommittedEventsSurvive() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Committed-1")); + spool.Enqueue(MakeEvent("Committed-2")); + } // Disposed: both enqueues are fully committed transactions on disk. + + AppendGarbageTo(FindSpoolFile("data.0")); + + using (var reopened = new EventSpool(_spoolDir, 10)) + { + var delivered = await DrainAllDelivered(reopened); + CollectionAssert.AreEqual(new[] { "Committed-1", "Committed-2" }, delivered); + + // And the recovered spool must still accept new work. + reopened.Enqueue(MakeEvent("PostRecovery")); + Assert.AreEqual(1, reopened.ApproximateCount); + } + } + + // 10b. CORRUPTION, garbage in the transaction log: DiskQueue deliberately refuses to open + // a log with unrecognized trailing bytes (it recovers a TRUNCATED log -- the realistic + // power-loss shape, where entries simply end mid-transaction -- but treats garbage after a + // committed entry as a conflict). Pin that fail-fast behavior here: the constructor throws + // rather than silently serving corrupt data. At the client level this is already handled: + // MixpanelClient.Initialize catches the throw and degrades to a non-durable no-op client + // rather than crashing the host (see MixpanelClient's class remarks). + [Test] + public void Constructor_GarbageAppendedToTransactionLog_FailsFastRatherThanServingCorruptData() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("Committed-1")); + } + + AppendGarbageTo(FindSpoolFile("transaction.log")); + + Assert.Catch(() => + { + using (new EventSpool(_spoolDir, 10)) + { + } + }); + } + + private string FindSpoolFile(string fileName) + { + var path = Path.Combine(_spoolDir, fileName); + Assert.IsTrue(File.Exists(path), + "expected DiskQueue file at " + path + " -- if DiskQueue renamed it, update this test"); + return path; + } + + private static void AppendGarbageTo(string path) + { + var garbage = new byte[257]; + new Random(12345).NextBytes(garbage); + using (var stream = new FileStream(path, FileMode.Append, FileAccess.Write)) + stream.Write(garbage, 0, garbage.Length); + } + } +} diff --git a/src/DesktopAnalyticsTests/MixpanelClientTests.cs b/src/DesktopAnalyticsTests/MixpanelClientTests.cs new file mode 100644 index 0000000..bb5cd93 --- /dev/null +++ b/src/DesktopAnalyticsTests/MixpanelClientTests.cs @@ -0,0 +1,968 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; +using Segment.Serialization; + +namespace DesktopAnalyticsTests +{ + // Layer 3 of the fidelity ladder in offline-analytics.md: the real EventSpool (on a temp + // folder) driving a real MixpanelClient, with a scripted fake IEventSender standing in for the + // network. Tests await DrainOnceAsync() directly rather than racing the real background timer (see + // "Design-for-test seams" / "Flush-loop scheduling" in offline-analytics.md); InitializeForTest + // never starts that timer. + [TestFixture] + public class MixpanelClientTests + { + private string _spoolDir; + + [SetUp] + public void SetUp() + { + _spoolDir = Path.Combine(Path.GetTempPath(), "MixpanelClientTests_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (Directory.Exists(_spoolDir)) + { + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + Console.WriteLine("MixpanelClientTests.TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + } + + // A scripted IEventSender: each call to SendBatchAsync() consumes the next step in the + // script (or defaults to Delivered once the script is exhausted) and always records the + // events it was given (flattened, in order), so tests can assert both what happened and + // what was actually handed to the sender. + private class ScriptedSender : IEventSender + { + private readonly Queue, BatchSendResult>> _script; + public readonly List Sent = new List(); + public int CallCount; + + public ScriptedSender(IEnumerable, BatchSendResult>> script) + { + _script = new Queue, BatchSendResult>>(script); + } + + public Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Sent.AddRange(events); + CallCount++; + var step = _script.Count > 0 ? _script.Dequeue() : (_ => BatchSendResult.Delivered); + // Deliberately NOT wrapped in Task.FromResult of a try/catch: a script step that + // throws makes this fake violate IEventSender's never-throw contract on purpose, + // exactly like the sync fake it replaced -- the crash-window tests rely on it. + return Task.FromResult(step(events)); + } + } + + private class AlwaysResultSender : IEventSender + { + private readonly SendResult _result; + public int CallCount; + public int EventCount; + + public AlwaysResultSender(SendResult result) + { + _result = result; + } + + public Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + CallCount++; + EventCount += events.Count; + return Task.FromResult(new BatchSendResult(_result)); + } + } + + private class ThrowingSender : IEventSender + { + public Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + throw new InvalidOperationException("simulated sender failure"); + } + } + + // Simulates a "black hole" server (accepts the connection but never responds): waits until + // its CancellationToken is signaled -- exactly what BoundedDrain's deadline does -- then + // reports the send as failed. Guards against ever being handed a token that can't be + // canceled, which would otherwise stall the whole test run rather than just fail one test. + private class BlockingUntilCanceledSender : IEventSender + { + public int CallCount; + + public async Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref CallCount); + if (cancellationToken.CanBeCanceled) + { + try + { + await Task.Delay(TimeSpan.FromSeconds(10), cancellationToken); + } + catch (OperationCanceledException) + { + // The deadline fired -- fall through and report the failure, per the + // IEventSender contract (treat cancellation like any transient failure). + } + } + return BatchSendResult.Retryable; + } + } + + // ---- NO-LOSS ------------------------------------------------------------------------- + + [Test] + public async Task DrainOnce_RetryRetryThenDeliverAcrossSuccessiveCalls_DeliversExactlyOnceAndEmptiesSpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new ScriptedSender(new Func, BatchSendResult>[] + { + _ => BatchSendResult.Retryable, + _ => BatchSendResult.Retryable, + _ => BatchSendResult.Delivered + }); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + Assert.AreEqual(1, spool.ApproximateCount); + + await client.DrainOnceAsync(); // 1st attempt: RetryableFailure -- left in spool. + Assert.AreEqual(1, spool.ApproximateCount); + + await client.DrainOnceAsync(); // 2nd attempt: RetryableFailure -- still left in spool. + Assert.AreEqual(1, spool.ApproximateCount); + + await client.DrainOnceAsync(); // 3rd attempt: Delivered -- removed. + Assert.AreEqual(0, spool.ApproximateCount); + + Assert.AreEqual(3, sender.Sent.Count, "the event should have been handed to the sender exactly three times"); + Assert.AreEqual(1, client.Statistics.Succeeded); + Assert.AreEqual(0, client.Statistics.Failed); + } + } + + // ---- CRASH-WINDOW / DEDUP ------------------------------------------------------------- + + [Test] + public async Task DrainOnce_SenderRecordsThenThrows_SameInsertIdSeenAgainOnLaterRetry() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + + // A sender that behaves exactly like one that delivered the event over the network + // and then crashed/threw before this process could observe success and flush -- + // the crash-window this whole design exists to survive. + var crashingSender = new ScriptedSender(new Func, BatchSendResult>[] + { + _ => throw new InvalidOperationException("simulated crash after send, before ack") + }); + client.InitializeForTest(spool, crashingSender); + client.Track("user-1", "Save", null); + + Assert.DoesNotThrowAsync(async () => await client.DrainOnceAsync()); + Assert.AreEqual(1, spool.ApproximateCount, "the event must not be removed -- the sender never confirmed delivery"); + Assert.AreEqual(1, crashingSender.Sent.Count); + + // "Reopen"/retry with a now-succeeding sender against the SAME spool. + var succeedingSender = new ScriptedSender(new Func, BatchSendResult>[] + { + _ => BatchSendResult.Delivered + }); + client.InitializeForTest(spool, succeedingSender); + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount); + Assert.AreEqual(1, succeedingSender.Sent.Count); + + // The key assertion: the sender saw the SAME $insert_id both times, proving Mixpanel + // would dedup the at-least-once replay rather than double-counting the event. + Assert.AreEqual(crashingSender.Sent[0].InsertId, succeedingSender.Sent[0].InsertId); + } + } + + // ---- POISON --------------------------------------------------------------------------- + + [Test] + public async Task DrainOnce_PoisonDrop_RemovesEventAndIncrementsFailedWithoutRetrying() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.PoisonDrop); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "BadEvent", null); + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, "a poison event must be dropped, not left to wedge the spool"); + Assert.AreEqual(1, sender.CallCount); + Assert.AreEqual(1, client.Statistics.Failed); + Assert.AreEqual(0, client.Statistics.Succeeded); + + // A further drain must not re-attempt it (it is already gone). + await client.DrainOnceAsync(); + Assert.AreEqual(1, sender.CallCount); + } + } + + // ---- CONSENT PURGE ---------------------------------------------------------------------- + + [Test] + public void PurgeQueuedEvents_EmptiesNonEmptySpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.Delivered); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "A", null); + client.Track("user-1", "B", null); + Assert.AreEqual(2, spool.ApproximateCount); + + client.PurgeQueuedEvents(); + + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + [Test] + public void PurgeQueuedEvents_NeverThrows_EvenWithNoSpool() + { + var client = new MixpanelClient(); + client.InitializeForTest(null, new AlwaysResultSender(SendResult.Delivered)); + + Assert.DoesNotThrow(() => client.PurgeQueuedEvents()); + } + + // ---- SCRUBBING -------------------------------------------------------------------------- + + [Test] + public async Task Track_ExceptionEventWithUserPathInStackTrace_ScrubsBeforeSpoolingAndSending() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.Delivered); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + var props = new JsonObject + { + { "Message", "Boom" }, + { "Stack Trace", @"at Foo.Bar() in C:\Users\alice\src\Foo.cs:line 10" } + }; + client.Track("user-1", "Exception", props); + + // Prove it was scrubbed BEFORE hitting the disk by scanning the raw spool files + // themselves, before anything drains: no assertion on what the sender is later + // handed can distinguish scrub-at-enqueue from scrub-at-send, but the bytes on + // disk can. Guard against vacuously passing (e.g. if the persisted encoding ever + // changes) by requiring the scrubbed marker to actually be FOUND on disk. + var scrubbedMarkerFoundOnDisk = false; + foreach (var file in EventSpoolTests.ReadableSpoolFiles(_spoolDir)) + { + StringAssert.DoesNotContain("alice", file.Value, + "unscrubbed user path found on disk in " + file.Key); + if (file.Value.Contains("%USER%")) + scrubbedMarkerFoundOnDisk = true; + } + Assert.IsTrue(scrubbedMarkerFoundOnDisk, + "expected to find the scrubbed path (%USER%) in the raw spool files -- if this " + + "stops being readable as UTF-8 text, rework this scan"); + + // And the scrubbed form (not just an absent event) is what got persisted: drain + // with a sender that records what it was actually handed. + var scriptedSender = new ScriptedSender(new Func, BatchSendResult>[] + { + _ => BatchSendResult.Delivered + }); + var reopenedClient = new MixpanelClient(); + reopenedClient.InitializeForTest(spool, scriptedSender); + await reopenedClient.DrainOnceAsync(); + + Assert.AreEqual(1, scriptedSender.Sent.Count); + var sentStackTrace = ((JsonPrimitive)scriptedSender.Sent[0].Properties["Stack Trace"]).Content; + StringAssert.DoesNotContain("alice", sentStackTrace); + StringAssert.Contains("%USER%", sentStackTrace); + } + } + + [Test] + public async Task Track_EventNameContainingUserPath_ScrubsEventName() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + client.Track("user-1", @"Opened C:\Users\alice\project.xyz", null); + + var sender = new ScriptedSender(new Func, BatchSendResult>[] + { _ => BatchSendResult.Delivered }); + var drainer = new MixpanelClient(); + drainer.InitializeForTest(spool, sender); + await drainer.DrainOnceAsync(); + + StringAssert.DoesNotContain("alice", sender.Sent[0].EventName); + StringAssert.Contains("%USER%", sender.Sent[0].EventName); + } + } + + // A user path nested inside a structured property (a JsonObject or JsonArray value, not a + // top-level string) must be scrubbed too -- ScrubProperties must recurse, not just look at + // the top level. + [Test] + public async Task Track_UserPathNestedInsideObjectAndArrayProperties_IsScrubbed() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + var props = new JsonObject + { + { + "Context", new JsonObject + { + { "Path", @"C:\Users\alice\file.txt" } + } + }, + { + "RecentFiles", new JsonArray + { + @"C:\Users\alice\a.txt", + @"C:\Users\alice\b.txt" + } + } + }; + client.Track("user-1", "Exception", props); + + var sender = new ScriptedSender(new Func, BatchSendResult>[] + { _ => BatchSendResult.Delivered }); + var drainer = new MixpanelClient(); + drainer.InitializeForTest(spool, sender); + await drainer.DrainOnceAsync(); + + var sentProperties = sender.Sent[0].Properties; + var context = (JsonObject)sentProperties["Context"]; + var contextPath = ((JsonPrimitive)context["Path"]).Content; + StringAssert.DoesNotContain("alice", contextPath); + StringAssert.Contains("%USER%", contextPath); + + var recentFiles = (JsonArray)sentProperties["RecentFiles"]; + foreach (var entry in recentFiles) + { + var path = ((JsonPrimitive)entry).Content; + StringAssert.DoesNotContain("alice", path); + StringAssert.Contains("%USER%", path); + } + } + } + + // ---- NEVER CRASH ------------------------------------------------------------------------ + + // Calling Track on a client that was never initialized at all is a programming error in + // the host and must throw (the original MixpanelClient dereferenced a null field there) -- + // NOT silently drop every event. Distinct from Track_WithNoSpool_NeverThrows below, where + // Initialize DID run but the spool could not be created. + [Test] + public void Track_WithoutInitialize_ThrowsInvalidOperationException() + { + var client = new MixpanelClient(); + Assert.Throws(() => client.Track("user-1", "Save", null)); + } + + [Test] + public void Track_WithNoSpool_NeverThrows() + { + var client = new MixpanelClient(); + // Simulates a failed Initialize (e.g. the EventSpool constructor threw acquiring the + // cross-process lock), which leaves the client with no spool at all. + client.InitializeForTest(null, new AlwaysResultSender(SendResult.Delivered)); + + Assert.DoesNotThrow(() => client.Track("user-1", "Save", null)); + Assert.DoesNotThrowAsync(async () => await client.DrainOnceAsync()); + Assert.DoesNotThrow(() => client.Flush()); + Assert.DoesNotThrowAsync(async () => await client.FlushAsync()); + Assert.DoesNotThrowAsync(async () => await client.ShutDownAsync()); + Assert.DoesNotThrow(() => client.ShutDown()); + } + + [Test] + public void DrainOnceAndFlush_SenderAlwaysThrows_NeverPropagateOutOfMixpanelClient() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new ThrowingSender()); + + client.Track("user-1", "Save", null); + + Assert.DoesNotThrowAsync(async () => await client.DrainOnceAsync()); + Assert.DoesNotThrow(() => client.Flush()); + + // A sender that only ever throws must not be treated as delivered/poison -- the + // event must still be sitting in the spool for a later retry. + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + // ---- SHUTDOWN OFFLINE ------------------------------------------------------------------- + + [Test] + public void ShutDown_SenderAlwaysFails_ReturnsPromptlyAndEventsRemainOnDisk() + { + var spool = new EventSpool(_spoolDir, 10); + var sender = new AlwaysResultSender(SendResult.RetryableFailure); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + for (var i = 0; i < 5; i++) + client.Track("user-1", "Save-" + i, null); + + var stopwatch = Stopwatch.StartNew(); + var completed = Task.Run(() => client.ShutDown()).Wait(TimeSpan.FromSeconds(15)); + stopwatch.Stop(); + + Assert.IsTrue(completed, "ShutDown() did not return within the timeout while offline"); + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(15)); + + // ShutDown() must have released the spool's cross-process lock -- reopening it and + // finding all 5 events proves nothing was lost, and that the lock really was released. + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(5, reopened.ApproximateCount); + } + } + + [Test] + public void Flush_WhileOffline_ReturnsPromptlyWithoutEmptyingSpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.RetryableFailure); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var stopwatch = Stopwatch.StartNew(); + var completed = Task.Run(() => client.Flush()).Wait(TimeSpan.FromSeconds(15)); + stopwatch.Stop(); + + Assert.IsTrue(completed, "Flush() did not return within the timeout while offline"); + Assert.AreEqual(1, spool.ApproximateCount, "an offline flush must leave the undelivered event on disk"); + } + } + + // ---- BOUNDED SHUTDOWN/FLUSH AGAINST A "BLACK HOLE" SERVER -------------------------------- + // + // Regression coverage for the bug where BoundedDrain's 5s wall-clock bound was only checked + // BETWEEN DrainOnce calls: a server that accepts the connection but never responds could + // make a single DrainOnce take up to ~45s (3 attempts x a 15s HttpClient timeout) before the + // bound was ever consulted, hanging ShutDown()/Flush(). These use a sender that blocks until + // the CancellationToken BoundedDrain hands it is canceled, which only happens promptly if + // the per-attempt deadline is actually wired through. + + [Test] + public void ShutDown_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() + { + var spool = new EventSpool(_spoolDir, 10); + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var stopwatch = Stopwatch.StartNew(); + var completed = Task.Run(() => client.ShutDown()).Wait(TimeSpan.FromSeconds(10)); + stopwatch.Stop(); + + Assert.IsTrue(completed, "ShutDown() did not return within the outer test timeout"); + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(10), + "ShutDown() must be bounded even against a server that never responds"); + Assert.GreaterOrEqual(sender.CallCount, 1); + + // ShutDown() must have released the spool's cross-process lock -- reopening it and + // finding the event proves nothing was lost (it was rolled back, not flushed) and that + // the lock really was released. + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(1, reopened.ApproximateCount, + "the undelivered event must remain in the spool after a bounded shutdown"); + } + } + + [Test] + public void Flush_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var stopwatch = Stopwatch.StartNew(); + var completed = Task.Run(() => client.Flush()).Wait(TimeSpan.FromSeconds(10)); + stopwatch.Stop(); + + Assert.IsTrue(completed, "Flush() did not return within the outer test timeout"); + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(10), + "Flush() must be bounded even against a server that never responds"); + Assert.AreEqual(1, spool.ApproximateCount, + "the undelivered event must remain in the spool after a bounded flush"); + } + } + + // ---- ASYNC PUBLIC SURFACE (FlushAsync/ShutDownAsync) ------------------------------------- + // + // The async counterparts must honor the same bounds as their sync wrappers. Awaited + // directly (no Task.Run + outer timeout) -- if the bound regressed, these would hang the + // awaiting test and fail via the test runner's timeout, same signal as the sync tests. + + [Test] + public async Task ShutDownAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() + { + var spool = new EventSpool(_spoolDir, 10); + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var stopwatch = Stopwatch.StartNew(); + await client.ShutDownAsync(); + stopwatch.Stop(); + + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(10), + "ShutDownAsync() must be bounded even against a server that never responds"); + Assert.GreaterOrEqual(sender.CallCount, 1); + + // ShutDownAsync() must have released the spool's cross-process lock -- reopening it and + // finding the event proves nothing was lost (it was rolled back, not flushed) and that + // the lock really was released. + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(1, reopened.ApproximateCount, + "the undelivered event must remain in the spool after a bounded shutdown"); + } + } + + [Test] + public async Task FlushAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var stopwatch = Stopwatch.StartNew(); + await client.FlushAsync(); + stopwatch.Stop(); + + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(10), + "FlushAsync() must be bounded even against a server that never responds"); + Assert.AreEqual(1, spool.ApproximateCount, + "the undelivered event must remain in the spool after a bounded flush"); + } + } + + // FlushAsync/ShutDownAsync accept an external CancellationToken (linked into the bounded + // drain's own deadline): an already-canceled token means no send is even attempted, the + // events stay spooled, nothing throws -- and ShutDownAsync still releases the spool's lock. + [Test] + public async Task FlushAsyncAndShutDownAsync_PreCanceledToken_SkipSendingLeaveEventsAndNeverThrow() + { + var spool = new EventSpool(_spoolDir, 10); + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + using (var cts = new CancellationTokenSource()) + { + cts.Cancel(); + + await client.FlushAsync(cts.Token); + Assert.AreEqual(0, sender.CallCount, + "a pre-canceled token must end the bounded drain before any send is attempted"); + Assert.AreEqual(1, spool.ApproximateCount); + + await client.ShutDownAsync(cts.Token); + Assert.AreEqual(0, sender.CallCount); + } + + // The lock must be released and the event still on disk for the next launch. + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(1, reopened.ApproximateCount); + } + } + + // ShutDownAsync followed by Dispose (the natural shape for a host that awaits ShutDownAsync + // inside a `using` over the Analytics facade) must be a harmless no-op the second time. + [Test] + public async Task ShutDownAsync_ThenSyncShutDown_IsIdempotent() + { + var spool = new EventSpool(_spoolDir, 10); + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + client.Track("user-1", "Save", null); + + await client.ShutDownAsync(); + Assert.DoesNotThrow(() => client.ShutDown()); + + // The spool's cross-process lock must be released (and stay released) -- reopening + // proves it. + using (var reopened = new EventSpool(_spoolDir, 10)) + { + Assert.AreEqual(0, reopened.ApproximateCount, + "the event was delivered during the first shutdown's bounded drain"); + } + } + + // ---- PER-EVENT SIZE CAP ------------------------------------------------------------------- + + // An event over the spool's per-event byte cap (production: Mixpanel's documented 1MB + // limit) is refused at Track time: never spooled, never handed to the sender, and counted + // as Failed -- NOT left to wedge the head of the queue as an eternally-retryable timeout. + [Test] + public async Task Track_EventOverSizeCap_IsDroppedCountedFailedAndNeverReachesSender() + { + using (var spool = new EventSpool(_spoolDir, 10, maxItemBytes: 500)) + { + var sender = new AlwaysResultSender(SendResult.Delivered); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Exception", + new JsonObject { { "Stack Trace", new string('x', 2000) } }); + + Assert.AreEqual(0, spool.ApproximateCount, "the oversized event must never be spooled"); + Assert.AreEqual(1, client.Statistics.Submitted); + Assert.AreEqual(1, client.Statistics.Failed); + + await client.DrainOnceAsync(); + Assert.AreEqual(0, sender.CallCount, "the oversized event must never reach the sender"); + + // A normal event on the same client still flows end to end. + client.Track("user-1", "Save", null); + await client.DrainOnceAsync(); + Assert.AreEqual(1, sender.CallCount); + Assert.AreEqual(1, client.Statistics.Succeeded); + } + } + + // A drain tick is byte-budgeted (bandwidth courtesy on slow/metered connections): three + // ~200KB events against the 256KB/tick budget take two ticks, not one -- and each tick is + // a single batched request, however many events it carries. + [Test] + public async Task DrainOnce_EventsExceedingPerTickByteBudget_SpreadsAcrossTicks() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.Delivered); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + for (var i = 0; i < 3; i++) + client.Track("user-1", "Big-" + i, + new JsonObject { { "Payload", new string('x', 200 * 1024) } }); + + await client.DrainOnceAsync(); + Assert.AreEqual(1, sender.CallCount, "one tick = one batched request"); + Assert.AreEqual(2, sender.EventCount, + "the 256KB/tick budget is crossed after the second ~200KB event"); + Assert.AreEqual(1, spool.ApproximateCount); + + await client.DrainOnceAsync(); + Assert.AreEqual(2, sender.CallCount); + Assert.AreEqual(3, sender.EventCount); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + // ---- STATISTICS ------------------------------------------------------------------------- + + // Strict-mode /import can process a batch while rejecting individual records + // (failed_records). The batch commits (nothing left to retry), the rejected records count + // as Failed, and the ingested ones as Succeeded. + [Test] + public async Task DrainOnce_BatchProcessedWithSomeRecordsRejected_CountsThemFailedAndRestSucceeded() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new ScriptedSender(new Func, BatchSendResult>[] + { + batch => new BatchSendResult(SendResult.Delivered, new[] { 1 }) + }); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Good-1", null); + client.Track("user-1", "Bad", null); + client.Track("user-1", "Good-2", null); + + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, + "a processed batch is finished with -- rejected records must not stay spooled"); + Assert.AreEqual(2, client.Statistics.Succeeded); + Assert.AreEqual(1, client.Statistics.Failed); + + await client.DrainOnceAsync(); + Assert.AreEqual(1, sender.CallCount, "nothing must be re-sent"); + } + } + + [Test] + public void Track_IncrementsSubmittedImmediately() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + client.Track("user-1", "A", null); + client.Track("user-1", "B", null); + + Assert.AreEqual(2, client.Statistics.Submitted); + } + } + + // An event dropped later by cap enforcement (not at its own Enqueue call, but a still-older + // one evicted by a LATER Track/Enqueue once the spool is full) must still be counted as + // Failed -- otherwise Submitted permanently outruns Succeeded + Failed and never resolves. + [Test] + public void Track_EventEvictedByCapEnforcement_IsCountedFailed() + { + using (var spool = new EventSpool(_spoolDir, 2)) // maxItems = 2 + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + client.Track("user-1", "Event-0", null); // Will be evicted by Event-2 below. + client.Track("user-1", "Event-1", null); + client.Track("user-1", "Event-2", null); // Cap enforcement evicts Event-0 here. + + Assert.AreEqual(2, spool.ApproximateCount, "the oldest event must have been evicted"); + Assert.AreEqual(3, client.Statistics.Submitted); + Assert.AreEqual(1, client.Statistics.Failed, + "the cap-evicted event must be counted as Failed even though it was dropped on " + + "a LATER Track call than its own"); + Assert.AreEqual(client.Statistics.Submitted, + client.Statistics.Succeeded + client.Statistics.Failed + spool.ApproximateCount, + "every submitted event must eventually be accounted for as succeeded, failed, or " + + "still spooled"); + } + } + + // ---- POLLY -------------------------------------------------------------------------------- + // + // NOTE on which of the two documented approaches this uses (see offline-analytics.md, + // "Polly"): rather than driving Polly's real (non-zero) backoff delays through a + // FakeTimeProvider -- which is documented to hit the Polly #1932 + // SynchronizationContext-vs-ConfigureAwait gotcha -- this test injects a real retry pipeline + // (via MixpanelClient.BuildDefaultPipeline) configured with a ZERO base delay. That keeps + // the retry *strategy* itself real (ShouldHandle, MaxRetryAttempts, backoff wiring) while + // making the test deterministic and instantaneous, per the spec's documented fallback. + + [Test] + public async Task DrainOnce_WithRealRetryPipelineAndZeroDelay_TransientThenSuccess_DeliversWithinOneDrainOnceCall() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new ScriptedSender(new Func, BatchSendResult>[] + { + _ => BatchSendResult.Retryable, + _ => BatchSendResult.Delivered + }); + var pipeline = MixpanelClient.BuildDefaultPipeline(TimeProvider.System, + maxRetryAttempts: 2, retryDelay: TimeSpan.Zero); + + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, pipeline: pipeline); + + client.Track("user-1", "Save", null); + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, + "Polly should have retried in-process and delivered within this single DrainOnce call"); + Assert.AreEqual(2, sender.Sent.Count, + "the sender should have been called twice: once for the transient failure, once for the retry"); + Assert.AreEqual(1, client.Statistics.Succeeded); + } + } + + [Test] + public async Task DrainOnce_WithRealCircuitBreakerAndZeroDelay_SustainedFailures_TripsBreakerAndStopsCallingSender() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.RetryableFailure); + // MinimumThroughput=3 by default (see BuildDefaultPipeline): force enough failing + // attempts through quickly (small batch, zero delay) to trip the breaker, then verify + // a further drain of a fresh event does not even reach the sender. + // MaxRetryAttempts must be >= 1 (Polly validates this); 1 is close enough to "no + // retry" for this test's purposes -- what matters is that sender calls stop growing + // once the breaker trips, not the exact attempt count. + var pipeline = MixpanelClient.BuildDefaultPipeline(TimeProvider.System, + maxRetryAttempts: 1, retryDelay: TimeSpan.Zero); + + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, pipeline: pipeline, batchSize: 1); + + for (var i = 0; i < 8; i++) + { + client.Track("user-1", "Save-" + i, null); + await client.DrainOnceAsync(); + } + + var callsAfterWarmup = sender.CallCount; + + client.Track("user-1", "OneMore", null); + await client.DrainOnceAsync(); + + // Once the breaker is open, MixpanelClient must treat BrokenCircuitException as a + // RetryableFailure (leaving the event spooled) WITHOUT invoking the sender again. + Assert.AreEqual(callsAfterWarmup, sender.CallCount, + "once the circuit breaker is open, the sender must not be called again"); + Assert.Greater(spool.ApproximateCount, 0); + } + } + + [Test] + public async Task DrainOnce_WithRealCircuitBreakerAndFixedMinimumThroughput_OpensAfterOneFullyFailingTick() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.RetryableFailure); + // Regression test for the bug where MinimumThroughput=4 made the breaker inert: a + // fully failing DrainOnce tick produces exactly 3 outcomes (1 attempt + the default + // MaxRetryAttempts=2 retries), and successive ticks are 30s apart -- far outside the + // 10s sampling window -- so 4 outcomes could never land in the same window during a + // real outage. With MinimumThroughput fixed to 3, a single fully-failing tick alone + // must be enough to trip the breaker. + var pipeline = MixpanelClient.BuildDefaultPipeline(TimeProvider.System, retryDelay: TimeSpan.Zero); + + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, pipeline: pipeline, batchSize: 1); + + client.Track("user-1", "Save", null); + await client.DrainOnceAsync(); + + var callsAfterFirstTick = sender.CallCount; + Assert.GreaterOrEqual(callsAfterFirstTick, 3, + "one fully-failing tick should make at least 3 attempts (1 try + 2 retries)"); + + client.Track("user-1", "OneMore", null); + await client.DrainOnceAsync(); + + Assert.AreEqual(callsAfterFirstTick, sender.CallCount, + "the breaker should already be open after just one failing tick, so the sender must not be called again"); + Assert.Greater(spool.ApproximateCount, 0); + } + } + + // ---- CONTRACT: host parameter is rejected (Mixpanel does not support a host) ----------- + + [Test] + public void Initialize_WithNonEmptyHost_ThrowsArgumentException() + { + var client = new MixpanelClient(); + // Validated before any spool/timer is created, so no side effects and it propagates to the + // caller exactly as the original MixpanelClient did. + Assert.Throws(() => client.Initialize("secret", "https://example.com")); + } + + // ---- CONSENT PAUSE/RESUME (fix: re-enabling consent re-arms the flush loop) ------------- + + [Test] + public void PurgeQueuedEvents_ThenResumeSending_TogglesSendingPaused() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + Assert.IsFalse(client.SendingPaused, "should start unpaused"); + + client.PurgeQueuedEvents(); + Assert.IsTrue(client.SendingPaused, "revoking consent (purge) must pause the flush loop"); + + client.ResumeSending(); + Assert.IsFalse(client.SendingPaused, "re-granting consent (resume) must un-pause the flush loop"); + } + } + + // fix: consent revocation must not sit blocked behind an in-flight send (EventSpool.Purge + // and ProcessBatchAsync share one lock, held across the network await), and the batch that + // send was carrying must not survive to be delivered after consent was revoked -- it must + // roll back and then be purged, not slip through. + [Test] + public async Task PurgeQueuedEvents_WhileSendInFlight_CancelsSendReturnsPromptlyAndPurges() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var sender = new BlockingUntilCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + // A real timer-tick-style drain (DrainOnceAsync, usePipeline: true) that will block + // inside the sender until PurgeQueuedEvents cancels it. + var drainTask = client.DrainOnceAsync(); + + // Wait for the send to actually start before purging, so this exercises the race + // (purge arriving WHILE a send is in flight) rather than purging before any drain + // has even started. + var deadline = DateTime.UtcNow.AddSeconds(5); + while (sender.CallCount == 0 && DateTime.UtcNow < deadline) + await Task.Delay(10); + Assert.AreEqual(1, sender.CallCount, "the send must have started before purging"); + + var stopwatch = Stopwatch.StartNew(); + client.PurgeQueuedEvents(); + stopwatch.Stop(); + + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(5), + "Purge must cancel the in-flight send rather than blocking for its full (10s) delay"); + + await drainTask; + + Assert.AreEqual(0, spool.ApproximateCount, + "purge must remove the batch that was in flight when consent was revoked, not " + + "leave it to be delivered afterward"); + } + } + } +} diff --git a/src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs b/src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs new file mode 100644 index 0000000..785d228 --- /dev/null +++ b/src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs @@ -0,0 +1,310 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net.Http; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; +using WireMock.RequestBuilders; +using WireMock.ResponseBuilders; +using WireMock.Server; + +namespace DesktopAnalyticsTests +{ + // Layer 4 of the fidelity ladder in offline-analytics.md: a local HTTP stub (WireMock.Net) + // standing in for Mixpanel's real /import endpoint, exercising MixpanelEventSender's real HTTP + // posting/serialization and its response->BatchSendResult classification. + [TestFixture] + public class MixpanelEventSenderTests + { + private WireMockServer _server; + + [SetUp] + public void SetUp() + { + _server = WireMockServer.Start(); + } + + [TearDown] + public void TearDown() + { + try + { + _server?.Stop(); + _server?.Dispose(); + } + catch (Exception e) + { + Console.WriteLine("MixpanelEventSenderTests.TearDown: " + e); + } + } + + private static AnalyticsEvent MakeEvent(string name = "Save", string insertId = "insert-id-1") + { + return AnalyticsEvent.Create("user-1", name, null, insertId, + DateTimeOffset.UtcNow); + } + + private static IReadOnlyList OneEvent(string name = "Save") + { + return new[] { MakeEvent(name) }; + } + + private IRequestBuilder ImportRequest() + { + return Request.Create().WithPath("/import").UsingPost(); + } + + [Test] + public async Task Send_200Response_ReturnsDeliveredWithNoFailedRecords() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(200) + .WithBody("{\"code\":200,\"num_records_imported\":1,\"status\":\"OK\"}")); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + var result = await sender.SendBatchAsync(OneEvent()); + Assert.AreEqual(SendResult.Delivered, result.Outcome); + Assert.IsEmpty(result.FailedIndices); + } + + [Test] + public async Task Send_500Response_ReturnsRetryableFailure() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(500)); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + [Test] + public async Task Send_429Response_ReturnsRetryableFailure() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(429)); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + [Test] + public async Task Send_408Response_ReturnsRetryableFailure() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(408)); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + [Test] + public async Task Send_400ResponseWithoutParseableBody_ReturnsPoisonDrop() + { + // A 400 with no failed_records means the request as a whole was rejected -- retrying + // the same request can never succeed. + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(400).WithBody("Bad Request")); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.PoisonDrop, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + [Test] + public async Task Send_401Response_ReturnsPoisonDrop() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(401) + .WithBody("{\"code\":401,\"error\":\"Unauthorized\",\"status\":\"error\"}")); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.PoisonDrop, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + // ---- STRICT-MODE PARTIAL FAILURE (failed_records) ---------------------------------------- + + [Test] + public async Task Send_400WithFailedRecords_ReturnsDeliveredWithOnlyThoseIndicesFailed() + { + // Strict-mode /import: the invalid records are reported per-index; the REST WERE + // INGESTED. The whole batch is therefore finished with (Delivered), and only the + // reported indices are poison. + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(400).WithBody( + "{\"code\":400,\"num_records_imported\":2,\"status\":\"Bad Request\"," + + "\"failed_records\":[{\"index\":1,\"insert_id\":\"insert-id-b\"," + + "\"field\":\"properties.time\",\"message\":\"'properties.time' is invalid\"}]}")); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + var batch = new[] { MakeEvent("A", "insert-id-a"), MakeEvent("B", "insert-id-b"), MakeEvent("C", "insert-id-c") }; + + var result = await sender.SendBatchAsync(batch); + + Assert.AreEqual(SendResult.Delivered, result.Outcome, + "a strict-mode 400 with failed_records means the request WAS processed -- the batch is finished with"); + CollectionAssert.AreEquivalent(new[] { 1 }, result.FailedIndices); + } + + [Test] + public async Task Send_400WithUnmappableFailedRecordIndex_ReturnsPoisonDrop() + { + // An index outside the batch we sent means we can't trust the body -- fall back to + // whole-batch poison rather than misattributing the failure. + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(400).WithBody( + "{\"code\":400,\"failed_records\":[{\"index\":7}]}")); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.PoisonDrop, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + // ---- OFFLINE SHAPES ---------------------------------------------------------------------- + + [Test] + public async Task Send_ConnectionRefused_ReturnsRetryableFailure() + { + // Stop (and dispose) the server so the base URL it handed out is now unreachable -- + // simulates being offline without waiting for a real timeout (connection-refused is + // near-instant on loopback). + var deadUrl = _server.Urls[0]; + _server.Stop(); + _server.Dispose(); + _server = null; + + var sender = new MixpanelEventSender("secret", deadUrl, + new HttpClient { Timeout = TimeSpan.FromSeconds(5) }); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + [Test] + public async Task Send_UnreachableHost_ReturnsRetryableFailure() + { + // A bad port on a live host is another common "offline" shape distinct from a + // stopped-server connection refusal. + var sender = new MixpanelEventSender("secret", "http://127.0.0.1:1", + new HttpClient { Timeout = TimeSpan.FromSeconds(5) }); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + } + + // ---- CAPTIVE PORTAL (redirect handling) -------------------------------------------------- + + [Test] + public async Task Send_302Response_ReturnsRetryableFailureAndDoesNotFollowRedirect() + { + // Simulates a captive portal intercepting the POST and redirecting to its login page. + _server.Given(ImportRequest()) + .RespondWith(Response.Create() + .WithStatusCode(302) + .WithHeader("Location", "/portal-login")); + _server.Given(Request.Create().WithPath("/portal-login")) + .RespondWith(Response.Create().WithStatusCode(200)); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + + // The key assertion: AllowAutoRedirect must be disabled, so the portal's login page is + // never actually requested. If it were followed, this sender would see the portal's 200 + // and wrongly classify the batch as Delivered even though Mixpanel never received it. + Assert.IsFalse(_server.LogEntries.Any(e => e.RequestMessage.Path == "/portal-login"), + "the redirect target must never have been requested (AllowAutoRedirect must be false)"); + } + + // ---- CANCELLATION (bounded drain deadline) ------------------------------------------------ + + [Test] + public void Send_PreCancelledToken_ReturnsRetryableFailureAndDoesNotThrow() + { + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(200)); + + var sender = new MixpanelEventSender("secret", _server.Urls[0]); + + var result = BatchSendResult.Delivered; + using (var cts = new CancellationTokenSource()) + { + cts.Cancel(); + Assert.DoesNotThrowAsync(async () => result = await sender.SendBatchAsync(OneEvent(), cts.Token)); + } + + Assert.AreEqual(SendResult.RetryableFailure, result.Outcome); + } + + [Test] + public void Send_ShortDeadlineAgainstDelayedResponse_ReturnsRetryableFailureAndDoesNotThrow() + { + // Simulates a "black hole" server: it eventually responds, but not before our deadline + // (BoundedDrain's per-attempt CancellationTokenSource, in production) fires. + _server.Given(ImportRequest()) + .RespondWith(Response.Create().WithStatusCode(200).WithDelay(TimeSpan.FromSeconds(5))); + + var sender = new MixpanelEventSender("secret", _server.Urls[0], + new HttpClient { Timeout = TimeSpan.FromSeconds(30) }); + + var result = BatchSendResult.Delivered; + using (var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(200))) + { + Assert.DoesNotThrowAsync(async () => result = await sender.SendBatchAsync(OneEvent(), cts.Token)); + } + + Assert.AreEqual(SendResult.RetryableFailure, result.Outcome); + } + + // ---- WIRE FORMAT --------------------------------------------------------------------------- + + [Test] + public async Task Send_TwoEvents_PostsOneJsonArrayRequestWithBasicAuthAndStrictMode() + { + const string token = "my-token"; + var expectedAuth = "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(token + ":")); + + // The stub only matches (and returns 200) if this is a single strict-mode /import POST + // whose JSON-array body carries both events with their insert ids, authenticated via + // the project token as the basic-auth username -- exercising MixpanelEventSender's real + // payload serialization end-to-end without coupling the test to + // Segment.Serialization's exact JSON formatting (key order, whitespace, etc). Notably + // there must be NO token in the payload itself: /import authenticates via the header. + _server + .Given(Request.Create() + .WithPath("/import") + .WithParam("strict", "1") + .UsingPost() + .WithHeader("Authorization", expectedAuth) + .WithHeader("Content-Type", "application/json*") + .WithBody(body => + body != null && + body.TrimStart().StartsWith("[") && + body.TrimEnd().EndsWith("]") && + body.Contains("\"event\"") && + body.Contains("\"Save PDF\"") && + body.Contains("\"Print\"") && + body.Contains("\"$insert_id\"") && + body.Contains("\"insert-id-a\"") && + body.Contains("\"insert-id-b\"") && + body.Contains("\"distinct_id\"") && + !body.Contains("\"token\""))) + .RespondWith(Response.Create().WithStatusCode(200)); + + var sender = new MixpanelEventSender(token, _server.Urls[0]); + var batch = new[] { MakeEvent("Save PDF", "insert-id-a"), MakeEvent("Print", "insert-id-b") }; + + var result = await sender.SendBatchAsync(batch); + + Assert.AreEqual(SendResult.Delivered, result.Outcome, + "expected the WireMock stub -- which only matches the expected /import request -- to have matched"); + Assert.AreEqual(1, _server.LogEntries.Count(), + "both events must go out in ONE request -- that is the point of batching"); + } + } +} diff --git a/src/DesktopAnalyticsTests/PathScrubberTests.cs b/src/DesktopAnalyticsTests/PathScrubberTests.cs new file mode 100644 index 0000000..e047c23 --- /dev/null +++ b/src/DesktopAnalyticsTests/PathScrubberTests.cs @@ -0,0 +1,117 @@ +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + [TestFixture] + public class PathScrubberTests + { + [Test] + public void Scrub_WindowsPath_ReplacesUserSegment() + { + const string input = @"C:\Users\jsmith\AppData\Local\Temp\file.txt"; + const string expected = @"%USER%\AppData\Local\Temp\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_UnixPath_ReplacesUserSegment() + { + const string input = "/home/jsmith/.config/app/settings.json"; + const string expected = "%USER%/.config/app/settings.json"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_MultipleOccurrencesInOneString_ReplacesAll() + { + const string input = + "at Foo.Bar() in C:\\Users\\jsmith\\src\\Foo.cs:line 10\r\n" + + " at Baz.Qux() in C:\\Users\\otherUser\\src\\Baz.cs:line 20"; + const string expected = + "at Foo.Bar() in %USER%\\src\\Foo.cs:line 10\r\n" + + " at Baz.Qux() in %USER%\\src\\Baz.cs:line 20"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_MixedWindowsAndUnixInOneString_ReplacesBoth() + { + const string input = @"C:\Users\jsmith\a.txt and /home/jsmith/b.txt"; + const string expected = "%USER%\\a.txt and %USER%/b.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_NoMatch_ReturnsInputUnchanged() + { + const string input = @"D:\Projects\MyApp\bin\Debug\App.exe"; + Assert.AreEqual(input, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_PlainStringWithNoPaths_ReturnsInputUnchanged() + { + const string input = "Just a regular exception message, nothing to scrub here."; + Assert.AreEqual(input, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_Null_ReturnsNull() + { + Assert.IsNull(PathScrubber.Scrub(null)); + } + + [Test] + public void Scrub_Empty_ReturnsEmpty() + { + Assert.AreEqual(string.Empty, PathScrubber.Scrub(string.Empty)); + } + + [Test] + public void Scrub_LowercaseDriveLetter_StillMatches() + { + const string input = @"c:\Users\jsmith\file.txt"; + const string expected = @"%USER%\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_LowercaseUsersSegment_StillMatches() + { + const string input = @"C:\users\jsmith\file.txt"; + const string expected = @"%USER%\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_MixedCaseDriveAndUsersSegment_StillMatches() + { + const string input = @"D:\UseRs\JSmith\file.txt"; + const string expected = @"%USER%\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_UserNameWithSpacesAndDots_ReplacesWholeSegment() + { + const string input = @"C:\Users\John Q. Smith\Documents\file.txt"; + const string expected = @"%USER%\Documents\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_HugeStackTraceWithManyPaths_ReplacesAllWithoutError() + { + var sb = new System.Text.StringBuilder(); + for (int i = 0; i < 500; i++) + { + sb.AppendLine($@" at Some.Method{i}() in C:\Users\jsmith\repo\File{i}.cs:line {i}"); + } + var result = PathScrubber.Scrub(sb.ToString()); + StringAssert.DoesNotContain("jsmith", result); + StringAssert.Contains("%USER%\\repo\\File0.cs", result); + StringAssert.Contains("%USER%\\repo\\File499.cs", result); + } + } +} diff --git a/src/SampleApp/Program.cs b/src/SampleApp/Program.cs index 99da2af..901255a 100644 --- a/src/SampleApp/Program.cs +++ b/src/SampleApp/Program.cs @@ -10,30 +10,60 @@ class Program { static int Main(string[] args) { - if (args.Length < 2 || args.Length > 3) + const string usage = "Usage: SampleApp " + + "[i[nitialTrackingState]=true|false] [c[onsentToggle]=true|false]"; + + if (args.Length < 2 || args.Length > 4) { - Console.WriteLine("Usage: SampleApp [i[nitialTrackingState]=true|false]"); + Console.WriteLine(usage); return 1; } if (!Enum.TryParse(args[1], true, out var clientType)) { - Console.WriteLine($"Usage: SampleApp {Environment.NewLine}Unrecognized client type: {args[1]}"); + Console.WriteLine($"{usage}{Environment.NewLine}Unrecognized client type: {args[1]}"); return 1; } var initialTracking = true; - - if (args.Length == 3) + // Exercises the AllowTracking off/on demo below by default (existing behavior). A caller + // driving this app as a durability-test harness across multiple process runs (rather than + // as a manual consent-flow demo) needs c=false: toggling AllowTracking calls + // PurgeQueuedEvents, which empties the ENTIRE on-disk spool immediately -- including any + // leftover events from a prior run -- so with the toggle left on, this app can never be + // used to prove "events spooled by a previous run survive and drain," only to demo consent. + var exerciseConsentToggle = true; + + for (var i = 2; i < args.Length; i++) { - var parts = args[2].Split('='); - if (parts.Length != 2 || - (!parts[0].Equals("i", StringComparison.OrdinalIgnoreCase) && - !parts[0].Equals("initialTrackingState", StringComparison.OrdinalIgnoreCase)) || - !bool.TryParse(parts[1], out initialTracking)) + var parts = args[i].Split('='); + if (parts.Length != 2) { - Console.WriteLine( - $"Unrecognized parameter: {args[2]}{Environment.NewLine}Usage: SampleApp [i[nitialTrackingState]=true|false]"); + Console.WriteLine($"Unrecognized parameter: {args[i]}{Environment.NewLine}{usage}"); + return 1; + } + + if (parts[0].Equals("i", StringComparison.OrdinalIgnoreCase) || + parts[0].Equals("initialTrackingState", StringComparison.OrdinalIgnoreCase)) + { + if (!bool.TryParse(parts[1], out initialTracking)) + { + Console.WriteLine($"Unrecognized parameter: {args[i]}{Environment.NewLine}{usage}"); + return 1; + } + } + else if (parts[0].Equals("c", StringComparison.OrdinalIgnoreCase) || + parts[0].Equals("consentToggle", StringComparison.OrdinalIgnoreCase)) + { + if (!bool.TryParse(parts[1], out exerciseConsentToggle)) + { + Console.WriteLine($"Unrecognized parameter: {args[i]}{Environment.NewLine}{usage}"); + return 1; + } + } + else + { + Console.WriteLine($"Unrecognized parameter: {args[i]}{Environment.NewLine}{usage}"); return 1; } } @@ -60,12 +90,16 @@ static int Main(string[] args) Debug.WriteLine("Sleeping for 20 seconds to give it all a chance to send an event in the background..."); Thread.Sleep(20000); - Analytics.AllowTracking = !Analytics.AllowTracking; - Analytics.Track("Should not be tracked"); - Debug.WriteLine("Sleeping for 2 seconds just for fun"); - Thread.Sleep(2000); + if (exerciseConsentToggle) + { + Analytics.AllowTracking = !Analytics.AllowTracking; + Analytics.Track("Should not be tracked"); + Debug.WriteLine("Sleeping for 2 seconds just for fun"); + Thread.Sleep(2000); + + Analytics.AllowTracking = !Analytics.AllowTracking; + } - Analytics.AllowTracking = !Analytics.AllowTracking; Analytics.SetApplicationProperty("TimeSinceLaunch", "25 seconds"); Analytics.Track("SomeEvent", new Dictionary {{"SomeValue", "42"}}); Console.WriteLine("Sleeping for another 20 seconds to give it all a chance to send an event in the background..."); From 8a1563ede00ae5d3a07862cc59820188926f8b56 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 12:49:36 -0400 Subject: [PATCH 02/11] Fix consent-purge race, bound Purge() wait, dispose sender, harden PathScrubber, add 60-day retention floor Closes review findings on PR #43 (the Mixpanel durability spool): AllowTracking now flips before PurgeQueuedEvents runs (and MixpanelClient.Track checks the pause flag directly) so a concurrent Track() can no longer survive a consent purge; EventSpool.Purge()'s lock wait is now bounded, falling back to a background completion instead of blocking a caller thread (e.g. a UI thread) indefinitely; MixpanelClient.ShutDownAsync is now genuinely idempotent and disposes the event sender's HttpClient, which was previously leaked; PathScrubber now handles UNC paths, forward slashes, and paths with no trailing separator. Also adds an explicit 60-day age-based retention floor (EventSpool.TrimExpired) on top of the existing item/byte caps, with the caps raised so they don't defeat that floor at realistic usage volumes, and fills several test-coverage gaps (circuit-breaker recovery, concurrent Track() during a drain, cap boundaries, degenerate maxItems=0, disk I/O failures). Co-Authored-By: Claude Sonnet 5 --- src/DesktopAnalytics/Analytics.cs | 9 +- src/DesktopAnalytics/EventSpool.cs | 171 +++++++++++-- src/DesktopAnalytics/MixpanelClient.cs | 112 +++++++-- src/DesktopAnalytics/PathScrubber.cs | 35 ++- src/DesktopAnalyticsTests/EventSpoolTests.cs | 180 ++++++++++++++ .../MixpanelClientTests.cs | 229 ++++++++++++++++++ .../PathScrubberTests.cs | 47 ++++ 7 files changed, 740 insertions(+), 43 deletions(-) diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index f7c150e..519d1d5 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -766,6 +766,13 @@ public static bool AllowTracking if (value == s_allowTracking) return; + // Flip the flag FIRST, before any of the branching/purge logic below, so a + // concurrent Track()/ReportException() (which reads AllowTracking) observes the + // new value immediately. This shrinks the window in which a thread could still see + // AllowTracking == true (and enqueue an event) while a consent-revocation purge + // below is in progress or about to start. + s_allowTracking = value; + // The following is not strictly thread safe because another thread could set this // after the singleton is created but before it checks the flag to decide whether // to complete initialization based on the value of the flag. In practice, the @@ -811,8 +818,6 @@ public static bool AllowTracking Debug.WriteLine("Analytics.AllowTracking: PurgeQueuedEvents failed: " + e); } } - - s_allowTracking = value; } } diff --git a/src/DesktopAnalytics/EventSpool.cs b/src/DesktopAnalytics/EventSpool.cs index b912910..8fa8da6 100644 --- a/src/DesktopAnalytics/EventSpool.cs +++ b/src/DesktopAnalytics/EventSpool.cs @@ -531,6 +531,93 @@ public async Task ProcessBatchAsync(int maxItems, } } + /// + /// Enforces an age-based retention floor: removes events from the FRONT of the queue + /// (oldest first -- the queue is already FIFO) whose stamped + /// is older than minus , stopping as soon as + /// it reaches an event that is NOT expired -- everything after it, being newer, cannot be + /// expired either. This is an ADDITIONAL, independent eviction dimension alongside the + /// item/byte caps enforced by , not a replacement for them: + /// callers (see 's cap constants) should size those caps to + /// comfortably outlast under realistic usage, so this age floor -- + /// not the item/byte caps -- is normally what reclaims space from a long-offline backlog. + /// + /// + /// An entry that fails to deserialize (corrupt, or an incompatible schema version) can never + /// be dated, so it is dropped exactly like does with such an + /// entry, regardless of age. Never throws: any disk/IO/serialization failure is logged and + /// swallowed, leaving the spool exactly as it was found. + /// + /// The maximum age a queued event may reach before being dropped. + /// The current time, from the caller's injected + /// so tests stay deterministic -- never DateTimeOffset.UtcNow directly. + public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) + { + try + { + var cutoff = now - maxAge; + + _sync.Wait(); + try + { + while (true) + { + byte[] bytes; + using (var session = _queue.OpenSession()) + { + try + { + bytes = session.Dequeue(); + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.TrimExpired: dequeue failed, giving up: " + e); + return; + } + + if (bytes == null) + return; // Spool is empty. + + AnalyticsEvent evt = null; + var deserializeFailed = false; + try + { + evt = AnalyticsEvent.FromBytes(bytes); + } + catch (Exception e) + { + deserializeFailed = true; + Debug.WriteLine( + "EventSpool.TrimExpired: failed to deserialize event, dropping: " + e); + } + + if (!deserializeFailed && evt.Time >= cutoff) + { + // Not expired -- and, by FIFO order, nothing after it can be expired + // either. Do not flush: disposing the session without committing rolls + // this dequeue back, leaving it (and everything behind it) in the queue. + return; + } + + // Either genuinely expired, or undatable (corrupt/incompatible) and thus + // can never be aged out any other way -- commit its removal either way. + session.Flush(); + _spoolBytes = Math.Max(0, _spoolBytes - bytes.Length); + PersistSpoolBytesLocked(_spoolBytes); + } + } + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.TrimExpired failed: " + e); + } + } + /// /// Empties the spool entirely (used on consent revocation), removing the event data from /// disk -- not merely marking it consumed. Never throws. @@ -547,35 +634,68 @@ public async Task ProcessBatchAsync(int maxItems, /// briefly released; if another process steals it in that window the reopen fails, this /// spool degrades to a no-op (every method here already tolerates that), and the purge /// itself has still succeeded -- the data is gone, which is the property that matters. + /// + /// The wait to acquire here is bounded () + /// rather than unbounded, because -style callers (e.g. + /// Analytics.AllowTracking's setter, plausibly invoked from a UI thread handling a + /// consent checkbox) call this synchronously, and an in-flight send that is slow to observe + /// cancellation must not be able to block that caller's thread indefinitely. If the bounded + /// wait times out, this method does NOT give up on purging -- that would violate the + /// consent/privacy guarantee that the data is deleted -- it instead logs and finishes the + /// purge on a background that waits for + /// without a bound (acceptable there since it is off the caller's thread) and then runs the + /// same dispose+delete+reopen logic. + /// /// public void Purge() { try { - _sync.Wait(); - try + if (_sync.Wait(s_lockWaitTimeout)) { - if (_disposed) - return; - try { - _queue?.Dispose(); + if (_disposed) + return; + + PurgeLocked(); } finally { - _queue = null; + _sync.Release(); } - - if (Directory.Exists(_spoolDirectory)) - Directory.Delete(_spoolDirectory, true); - - _queue = PersistentQueue.WaitFor(_spoolDirectory, s_lockWaitTimeout); - _spoolBytes = 0; } - finally + else { - _sync.Release(); + // Could not acquire the lock within the bound (most likely an in-flight send + // still holding it, slow to observe cancellation). Do not give up on the purge -- + // it's a consent/privacy guarantee -- finish it on a background thread instead, + // where an unbounded wait is acceptable because it no longer blocks the caller. + Debug.WriteLine( + "EventSpool.Purge: timed out waiting for the lock; completing the purge in the " + + "background instead of blocking the caller."); + Task.Run(() => + { + try + { + _sync.Wait(); + try + { + if (_disposed) + return; + + PurgeLocked(); + } + finally + { + _sync.Release(); + } + } + catch (Exception e) + { + Debug.WriteLine("EventSpool.Purge (background fallback) failed: " + e); + } + }); } } catch (Exception e) @@ -584,6 +704,27 @@ public void Purge() } } + // Caller must already hold _sync. Shared by both the fast (in-bound) path and the + // background fallback path of Purge() so the actual purge body -- dispose the queue, + // delete the directory, reopen, reset _spoolBytes -- is implemented exactly once. + private void PurgeLocked() + { + try + { + _queue?.Dispose(); + } + finally + { + _queue = null; + } + + if (Directory.Exists(_spoolDirectory)) + Directory.Delete(_spoolDirectory, true); + + _queue = PersistentQueue.WaitFor(_spoolDirectory, s_lockWaitTimeout); + _spoolBytes = 0; + } + /// /// Releases the underlying DiskQueue's cross-process exclusive lock. Safe to call more /// than once. diff --git a/src/DesktopAnalytics/MixpanelClient.cs b/src/DesktopAnalytics/MixpanelClient.cs index 7779c53..3ebae74 100644 --- a/src/DesktopAnalytics/MixpanelClient.cs +++ b/src/DesktopAnalytics/MixpanelClient.cs @@ -29,11 +29,19 @@ internal class MixpanelClient : IClient { // Open questions in offline-analytics.md: exact cap/batch/cadence values are not locked yet. // These are reasonable starting defaults, not requirements baked in elsewhere. - private const int kDefaultMaxSpoolItems = 5000; + // + // Sized to comfortably OUTLAST the 60-day age floor (kDefaultMaxSpoolAgeDays below) under + // realistic desktop-app usage-analytics volumes, rather than being the tightest bound in + // practice: EventSpool.TrimExpired's age-based eviction is meant to be what normally + // reclaims space from a long-offline backlog, not this item cap. At a generous ~200 + // events/day of active use, 60 days is on the order of 12,000 events; 20,000 leaves + // headroom for bursty days (e.g. an exception storm) without the item cap kicking in well + // before the age floor ever would. + private const int kDefaultMaxSpoolItems = 20000; // Events per /import request (one request per drain tick). Mixpanel accepts up to 2,000 // events and 10MB per request; this stays well under both (the per-tick byte budget below // bounds the request size long before the count does), while draining a large offline - // backlog reasonably fast (5000 events in ~13 min at the default cadence). + // backlog reasonably fast (20,000 events in ~50 min at the default cadence). private const int kDefaultBatchSize = 200; private const int kDefaultFlushIntervalSeconds = 30; @@ -48,8 +56,20 @@ internal class MixpanelClient : IClient // Bandwidth courtesy on slow/metered connections: a soft byte budget per flush tick, and a // total-backlog cap (drop-oldest) bounding disk footprint and upload liability. See the // "Constrained bandwidth" section of the PR #43 description for the pacing rationale. + // Like kDefaultMaxSpoolItems above, sized to comfortably outlast the 60-day age floor + // (kDefaultMaxSpoolAgeDays below): at a generous ~2KB/event average (small usage events + // plus occasional larger exception/stack-trace payloads), kDefaultMaxSpoolItems worth of + // events is on the order of 40MB; 50MB leaves headroom above that estimate. private const long kMaxBytesPerDrainTick = 256 * 1024; - private const long kDefaultMaxSpoolBytes = 20L * 1024 * 1024; + private const long kDefaultMaxSpoolBytes = 50L * 1024 * 1024; + + // Age-based retention floor: an ADDITIONAL, independent eviction dimension alongside the + // item/byte caps above (enforced separately by EventSpool.TrimExpired), not a replacement + // for them. Keeps queued (undelivered) events around for roughly two months of offline time + // before dropping them -- a generous, explicit retention floor of our own, rather than + // whatever much shorter window Mixpanel's legacy /track endpoint implied (moot here anyway, + // since MixpanelEventSender uses /import, which has no such limit). + private const int kDefaultMaxSpoolAgeDays = 60; // On (re)start, attempt the first drain soon rather than waiting a full interval, so events // spooled during a PREVIOUS (offline) session go out shortly after launch instead of ~30s later. @@ -83,10 +103,15 @@ internal class MixpanelClient : IClient // Cancels whatever timer-driven send (DrainOnceAsync) is currently in flight when consent is // revoked (see PurgeQueuedEvents), so Purge() does not sit blocked behind a network round trip - // and the batch that send was carrying rolls back into the spool instead of being delivered - // after revocation. Swapped for a fresh instance on every purge; never used across a purge - // boundary. Guarded by Interlocked.Exchange since PurgeQueuedEvents can run on a different - // thread (e.g. a UI thread via Analytics.AllowTracking) than the timer-driven drain it cancels. + // and the batch that send was carrying rolls back into the spool (which the impending Purge() + // then removes) rather than being resent locally after revocation. This is a best-effort + // reduction of the window, not a hard guarantee against server-side receipt: if the POST body + // was already fully sent to and received by Mixpanel's server before this cancellation is + // observed locally, the batch WAS still delivered once -- cancellation only prevents it from + // being resent/kept around locally, it cannot recall bytes the server already has. Swapped for + // a fresh instance on every purge; never used across a purge boundary. Guarded by + // Interlocked.Exchange since PurgeQueuedEvents can run on a different thread (e.g. a UI thread + // via Analytics.AllowTracking) than the timer-driven drain it cancels. private CancellationTokenSource _sendCts = new CancellationTokenSource(); private int _submitted; @@ -100,6 +125,11 @@ internal class MixpanelClient : IClient // keeps a slow drain from stacking up redundant timer callbacks behind it. private int _timerDraining; + // 0 = not yet shut down; guarded by Interlocked.Exchange in ShutDownAsync so the real + // shutdown body (stop timer, bounded drain, dispose spool/sender) runs exactly once, even + // if ShutDownAsync/ShutDown is called more than once concurrently. + private int _shutDownState; + /// /// Production initializer (via ). Builds a real on-disk spool keyed by /// , a real HTTP-posting , and a @@ -195,12 +225,18 @@ private void OnItemDroppedByCap() /// tests that specifically want to exercise real Polly retry/circuit-breaker behavior -- /// rather than the zero-strategy default used by -- can build /// one with a deterministic (e.g. a zero-delay variant; see - /// MixpanelClientTests for the rationale). + /// MixpanelClientTests for the rationale). similarly lets + /// tests shrink the production 5s break duration so a breaker-recovery (half-open -> closed) + /// test can wait it out for real, quickly, on the real + /// clock -- rather than driving a FakeTimeProvider through it, which is documented + /// (Polly #1932, see offline-analytics.md) to need extra ceremony to make retries fire at + /// all. /// internal static ResiliencePipeline BuildDefaultPipeline( TimeProvider timeProvider, int maxRetryAttempts = 2, - TimeSpan? retryDelay = null) + TimeSpan? retryDelay = null, + TimeSpan? breakDuration = null) { bool ShouldHandleOutcome(Outcome outcome) => outcome.Exception != null || outcome.Result?.Outcome == SendResult.RetryableFailure; @@ -227,7 +263,7 @@ bool ShouldHandleOutcome(Outcome outcome) => // sustained outage. 3 makes one bad tick enough to trip it. MinimumThroughput = 3, SamplingDuration = TimeSpan.FromSeconds(10), - BreakDuration = TimeSpan.FromSeconds(5) + BreakDuration = breakDuration ?? TimeSpan.FromSeconds(5) }; var builder = new ResiliencePipelineBuilder @@ -349,6 +385,11 @@ private async Task DrainOnceCoreAsync(CancellationToken cancellationToken, bool { if (_spool == null) return; + + // Age-based retention floor: cheap to run every tick since it stops at the first + // non-expired entry (see EventSpool.TrimExpired's doc comment). + _spool.TrimExpired(TimeSpan.FromDays(kDefaultMaxSpoolAgeDays), _timeProvider.GetUtcNow()); + await _spool.ProcessBatchAsync(_batchSize, (batch, ct) => SendBatchGuardedAsync(batch, ct, usePipeline), kMaxBytesPerDrainTick, cancellationToken) @@ -471,6 +512,17 @@ public void Track(string analyticsId, string eventName, JsonObject properties) try { + // Defense-in-depth against the same consent-revocation race described on + // Analytics.AllowTracking's setter: PurgeQueuedEvents (via PauseTimer) sets + // _paused = true BEFORE it cancels the in-flight send and purges the spool, so a + // concurrent Track() that observes _paused here cannot enqueue an event that would + // survive the purge. This closes almost all of the residual race window down to the + // tiny gap between PauseTimer() setting _paused and this read of it -- fully + // eliminating that last sliver would require a lock spanning both the flag check + // and the enqueue call, which isn't justified here given how small the window is. + if (_paused) + return; + if (_spool == null) return; @@ -624,15 +676,24 @@ public void Flush() /// /// Bounded drain, then release the spool's cross-process lock. Never hangs, even fully /// offline: undelivered events simply remain on disk for the next launch to pick up. - /// Never faults, and is idempotent -- a redundant (e.g. Dispose on - /// the facade after an explicit ShutDownAsync) finds a stopped - /// timer, an empty-reporting spool, and a no-op re-Dispose. + /// Never faults, and is genuinely idempotent -- guarded by an internal + /// Interlocked.Exchange so only the first call (even if concurrent) runs the + /// shutdown body; a redundant call (e.g. Dispose on the facade + /// after an explicit ShutDownAsync) is an immediate no-op rather than re-entering the + /// drain against an already-disposed spool/sender. /// /// Optionally ends the final drain even sooner than its own /// wall-clock bound; undelivered events stay on disk. The spool's lock is released either /// way, and cancellation never faults the task. public async Task ShutDownAsync(CancellationToken cancellationToken = default) { + // Make the "idempotent" guarantee in this method's doc comment actually true rather + // than accidental: only the FIRST call (even under concurrent callers) runs the body + // below. Every subsequent call is an immediate no-op instead of re-entering + // BoundedDrainAsync / ProcessBatchAsync against a spool that may already be disposed. + if (Interlocked.Exchange(ref _shutDownState, 1) != 0) + return; + try { StopTimerPermanently(); @@ -652,6 +713,23 @@ public async Task ShutDownAsync(CancellationToken cancellationToken = default) { Debug.WriteLine("MixpanelClient.ShutDown: failed to dispose spool: " + e); } + finally + { + _spool = null; + } + + try + { + (_sender as IDisposable)?.Dispose(); + } + catch (Exception e) + { + Debug.WriteLine("MixpanelClient.ShutDown: failed to dispose sender: " + e); + } + finally + { + _sender = null; + } } } @@ -691,9 +769,11 @@ public void PurgeQueuedEvents() // Aborts whatever timer-driven send is currently in flight (if any) so Purge() does not block // behind a network round trip and the in-flight batch rolls back into the spool -- where the - // impending Purge() removes it -- instead of being delivered to Mixpanel after consent was - // just revoked. Swaps in a fresh, non-canceled token so the NEXT drain (post-purge, or after - // ResumeSending) is unaffected. + // impending Purge() removes it -- rather than being resent/kept around locally after consent + // was just revoked. This best-effort cancellation reduces, but cannot eliminate, the window: + // it cannot recall a POST body Mixpanel's server already fully received before the local await + // observed cancellation. Swaps in a fresh, non-canceled token so the NEXT drain (post-purge, or + // after ResumeSending) is unaffected. private void CancelInFlightSend() { var previous = Interlocked.Exchange(ref _sendCts, new CancellationTokenSource()); diff --git a/src/DesktopAnalytics/PathScrubber.cs b/src/DesktopAnalytics/PathScrubber.cs index c4475e6..5c9562e 100644 --- a/src/DesktopAnalytics/PathScrubber.cs +++ b/src/DesktopAnalytics/PathScrubber.cs @@ -10,30 +10,45 @@ namespace DesktopAnalytics /// internal static class PathScrubber { - // Windows: C:\Users\\ (any drive letter, case-insensitive on both the drive - // letter and the literal "Users" segment). We stop matching the user name at the next - // path separator so we don't eat the rest of the path. + // Windows: C:\Users\ or \\server\Users\ (any drive letter or UNC server + // name; case-insensitive on the drive letter and the literal "Users" segment; separators + // may be '\' or '/'). We stop matching the user name at the next path separator so we + // don't eat the rest of the path. The trailing separator is optional: the match may end + // at a separator (more path follows) or at the true end of the string (the user name is + // the last thing in the string, e.g. a truncated exception message). private static readonly Regex s_windowsUserPath = - new Regex(@"[A-Za-z]:\\Users\\[^\\/:*?""<>|\r\n]+\\", + new Regex(@"(?:[A-Za-z]:|\\\\[^\\/]+)[\\/]Users[\\/][^\\/:*?""<>|\r\n]+(?:(?[\\/])|$)", RegexOptions.Compiled | RegexOptions.IgnoreCase); - // Unix: /home// + // Unix: /home// or /home/ at end of string. As with the Windows regex, the + // trailing separator is optional so a user name at the very end of the string is scrubbed. private static readonly Regex s_unixUserPath = - new Regex(@"/home/[^/\r\n]+/", RegexOptions.Compiled); + new Regex(@"/home/[^/\r\n]+(?:(?/)|$)", RegexOptions.Compiled); /// /// Replaces every occurrence of a Windows or Unix user-home-directory prefix in - /// with a neutral placeholder ("%USER%\" or "%USER%/" - /// respectively). Returns the input unchanged if it is null/empty or contains no match. + /// with a neutral placeholder ("%USER%"). If the matched prefix + /// was followed by a path separator ('\' or '/'), that same separator is preserved after + /// the placeholder; if the user name ran to the end of the string with no separator, the + /// placeholder is emitted with nothing after it. Returns the input unchanged if it is + /// null/empty or contains no match. /// public static string Scrub(string input) { if (string.IsNullOrEmpty(input)) return input; - var result = s_windowsUserPath.Replace(input, @"%USER%\"); - result = s_unixUserPath.Replace(result, "%USER%/"); + var result = s_windowsUserPath.Replace(input, ReplaceUserSegment); + result = s_unixUserPath.Replace(result, ReplaceUserSegment); return result; } + + // Shared MatchEvaluator for both regexes: preserves whichever trailing separator (if any) + // was actually matched, via the "sep" capture group. + private static string ReplaceUserSegment(Match match) + { + var sep = match.Groups["sep"]; + return sep.Success ? "%USER%" + sep.Value : "%USER%"; + } } } diff --git a/src/DesktopAnalyticsTests/EventSpoolTests.cs b/src/DesktopAnalyticsTests/EventSpoolTests.cs index 9bd252a..21e6671 100644 --- a/src/DesktopAnalyticsTests/EventSpoolTests.cs +++ b/src/DesktopAnalyticsTests/EventSpoolTests.cs @@ -54,6 +54,11 @@ private static AnalyticsEvent MakeEvent(string name) return AnalyticsEvent.Create("user-1", name); } + private static AnalyticsEvent MakeEventAt(string name, DateTimeOffset time) + { + return AnalyticsEvent.Create("user-1", name, time: time); + } + // Drains the spool with an always-Delivered batch callback, returning the event names in // the order they were handed to the sender. private static async Task> DrainAllDelivered(EventSpool spool, int maxItems = 100) @@ -227,6 +232,181 @@ public async Task Enqueue_ExceedingMaxItems_DropsOldestKeepsNewest() } } + // 6a. Degenerate cap: maxItems == 0 means even the event just enqueued is evicted (see the + // comment on EnforceCapLocked describing this degenerate case). Enqueue itself still + // reports success (the write genuinely happened; cap enforcement is a separate step), but + // the spool ends up empty. + [Test] + public void Enqueue_MaxItemsZero_ImmediatelyEvictsTheJustEnqueuedEvent() + { + using (var spool = new EventSpool(_spoolDir, 0)) + { + Assert.IsTrue(spool.Enqueue(MakeEvent("A")), + "Enqueue reports the write itself succeeded, independent of cap enforcement"); + Assert.AreEqual(0, spool.ApproximateCount, + "maxItems == 0 must evict even the event that was just enqueued"); + } + } + + // 6b. Boundary: exactly maxItems enqueued => nothing dropped; one more => the oldest is + // dropped and the count stays pinned at the cap. + [Test] + public async Task Enqueue_ExactlyAtMaxItems_NoneDropped_ThenOneOver_DropsOldestStaysAtCap() + { + const int maxItems = 5; + using (var spool = new EventSpool(_spoolDir, maxItems)) + { + for (var i = 0; i < maxItems; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + Assert.AreEqual(maxItems, spool.ApproximateCount, + "exactly maxItems events must all be retained"); + + spool.Enqueue(MakeEvent("OneMore")); + + Assert.AreEqual(maxItems, spool.ApproximateCount, "count must stay pinned at the cap"); + + var remaining = await DrainAllDelivered(spool, maxItems + 5); + CollectionAssert.AreEqual( + new[] { "Event-1", "Event-2", "Event-3", "Event-4", "OneMore" }, remaining, + "the oldest event must be dropped to make room for the one over the cap"); + } + } + + // ---- AGE-BASED RETENTION (TrimExpired) -------------------------------------------------- + + // An event older than the max age must be dropped. + [Test] + public void TrimExpired_EventOlderThanMaxAge_IsDropped() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Old", now - TimeSpan.FromDays(61))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(0, spool.ApproximateCount, + "an event older than the max age must be dropped"); + } + } + + // An event within the max age must be retained. + [Test] + public void TrimExpired_EventWithinMaxAge_IsRetained() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Recent", now - TimeSpan.FromDays(1))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(1, spool.ApproximateCount, + "an event within the max age must be retained"); + } + } + + // A mix of expired and fresh events: only the expired PREFIX is dropped (FIFO order), and + // the rest survive in their original order. + [Test] + public async Task TrimExpired_MixOfExpiredAndFreshEvents_DropsOnlyExpiredPrefixKeepsRestInOrder() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Expired-1", now - TimeSpan.FromDays(90))); + spool.Enqueue(MakeEventAt("Expired-2", now - TimeSpan.FromDays(70))); + spool.Enqueue(MakeEventAt("Fresh-1", now - TimeSpan.FromDays(10))); + spool.Enqueue(MakeEventAt("Fresh-2", now - TimeSpan.FromDays(1))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(2, spool.ApproximateCount); + var remaining = await DrainAllDelivered(spool); + CollectionAssert.AreEqual(new[] { "Fresh-1", "Fresh-2" }, remaining, + "only the expired prefix must be dropped; the rest must survive in order"); + } + } + + // TrimExpired on an empty spool is a no-op that must not throw. + [Test] + public void TrimExpired_EmptySpool_IsNoOpAndDoesNotThrow() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + Assert.DoesNotThrow(() => spool.TrimExpired(TimeSpan.FromDays(60), DateTimeOffset.UtcNow)); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + // ---- DISK I/O FAILURE (degrade gracefully, never throw) --------------------------------- + + // A "disk full"/inaccessible-file analog: make the spool's own data file read-only out from + // under it (a real, DiskQueue-external way to force a write failure without hand-rolling a + // mock of DiskQueue itself) and confirm Enqueue reports the failure via its return value -- + // never by throwing -- and does not corrupt the spool's state. + [Test] + public void Enqueue_DataFileMadeReadOnly_ReturnsFalseAndNeverThrowsAndLeavesSpoolConsistent() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + Assert.IsTrue(spool.Enqueue(MakeEvent("Seed")), "seed event to ensure the data file exists"); + Assert.AreEqual(1, spool.ApproximateCount); + + var dataFile = Path.Combine(_spoolDir, "data.0"); + Assert.IsTrue(File.Exists(dataFile), "expected DiskQueue's data file at " + dataFile); + File.SetAttributes(dataFile, FileAttributes.ReadOnly); + try + { + bool result = true; + Assert.DoesNotThrow(() => result = spool.Enqueue(MakeEvent("ShouldFail")), + "a write failure (disk full/inaccessible analog) must never throw out of Enqueue"); + Assert.IsFalse(result, + "a write failure must be reported via Enqueue's return value"); + Assert.AreEqual(1, spool.ApproximateCount, + "a failed write must not corrupt the count of what is actually spooled"); + } + finally + { + File.SetAttributes(dataFile, FileAttributes.Normal); + } + } + } + + // Same idea for the read/drain side: lock the data file exclusively (simulating it being + // momentarily inaccessible) while ProcessBatchAsync is draining. The dequeue must fail + // internally, the method must not throw, and -- once the lock is released -- the event must + // still be present and retrievable (nothing was lost by the transient failure). + [Test] + public async Task ProcessBatch_DataFileTransientlyLocked_DegradesGracefullyAndEventSurvives() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + spool.Enqueue(MakeEvent("A")); + + var dataFile = Path.Combine(_spoolDir, "data.0"); + Assert.IsTrue(File.Exists(dataFile), "expected DiskQueue's data file at " + dataFile); + + var sentDuringLock = new List(); + using (new FileStream(dataFile, FileMode.Open, FileAccess.Read, FileShare.None)) + { + Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => + { + sentDuringLock.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + })); + } + + Assert.AreEqual(0, sentDuringLock.Count, + "the transiently locked file must prevent the dequeue from succeeding"); + + var delivered = await DrainAllDelivered(spool); + CollectionAssert.AreEqual(new[] { "A" }, delivered, + "the event must survive a transient read failure and still be retrievable afterward"); + } + } + // 7. Purge empties a non-empty spool. [Test] public async Task Purge_EmptiesNonEmptySpool() diff --git a/src/DesktopAnalyticsTests/MixpanelClientTests.cs b/src/DesktopAnalyticsTests/MixpanelClientTests.cs index bb5cd93..e8bad33 100644 --- a/src/DesktopAnalyticsTests/MixpanelClientTests.cs +++ b/src/DesktopAnalyticsTests/MixpanelClientTests.cs @@ -1,7 +1,9 @@ using System; +using System.Collections.Concurrent; using System.Collections.Generic; using System.Diagnostics; using System.IO; +using System.Linq; using System.Threading; using System.Threading.Tasks; using DesktopAnalytics; @@ -127,6 +129,27 @@ public async Task SendBatchAsync(IReadOnlyList } } + // Blocks inside SendBatchAsync until the test explicitly releases it (via a + // TaskCompletionSource, not a real delay), so a test can precisely control when an + // in-flight drain's network call "returns" -- used to race concurrent Track() calls against + // an in-progress drain deterministically and quickly. + private class BlockingUntilSignaledSender : IEventSender + { + private readonly TaskCompletionSource _release = + new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + public int CallCount; + + public void Release() => _release.TrySetResult(true); + + public async Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref CallCount); + await _release.Task.ConfigureAwait(false); + return new BatchSendResult(SendResult.Delivered); + } + } + // ---- NO-LOSS ------------------------------------------------------------------------- [Test] @@ -253,7 +276,18 @@ public void PurgeQueuedEvents_NeverThrows_EvenWithNoSpool() var client = new MixpanelClient(); client.InitializeForTest(null, new AlwaysResultSender(SendResult.Delivered)); + Assert.IsFalse(client.SendingPaused, "test setup: should start unpaused"); + Assert.DoesNotThrow(() => client.PurgeQueuedEvents()); + + // Statistics must be untouched -- there was never any spool to purge anything from. + Assert.AreEqual(0, client.Statistics.Submitted); + Assert.AreEqual(0, client.Statistics.Succeeded); + Assert.AreEqual(0, client.Statistics.Failed); + // The flush loop must still pause even with no spool -- PurgeQueuedEvents' PauseTimer() + // call does not depend on there being a spool to actually purge. + Assert.IsTrue(client.SendingPaused, + "PurgeQueuedEvents must still pause the flush loop even with no spool"); } // ---- SCRUBBING -------------------------------------------------------------------------- @@ -402,11 +436,23 @@ public void Track_WithNoSpool_NeverThrows() client.InitializeForTest(null, new AlwaysResultSender(SendResult.Delivered)); Assert.DoesNotThrow(() => client.Track("user-1", "Save", null)); + // Track() must return (as a no-op) before it ever counts the event as submitted -- + // distinct from Track_WithoutInitialize_ThrowsInvalidOperationException, where a + // never-initialized client is a programming error rather than a "no spool" no-op. + Assert.AreEqual(0, client.Statistics.Submitted, + "Track() with no spool must not count the event as submitted at all"); + Assert.DoesNotThrowAsync(async () => await client.DrainOnceAsync()); Assert.DoesNotThrow(() => client.Flush()); Assert.DoesNotThrowAsync(async () => await client.FlushAsync()); Assert.DoesNotThrowAsync(async () => await client.ShutDownAsync()); Assert.DoesNotThrow(() => client.ShutDown()); + + // None of the above should have moved the needle: with no spool, there is nothing to + // drain/flush/shut down, so the statistics must still read exactly as they started. + Assert.AreEqual(0, client.Statistics.Submitted); + Assert.AreEqual(0, client.Statistics.Succeeded); + Assert.AreEqual(0, client.Statistics.Failed); } [Test] @@ -892,6 +938,78 @@ public async Task DrainOnce_WithRealCircuitBreakerAndFixedMinimumThroughput_Open } } + // Breaker recovery: half-open -> closed. Once the circuit breaker has tripped (same + // single-fully-failing-tick recipe as the regression test above), let its BreakDuration + // elapse and drive a successful send, proving delivery resumes rather than the breaker + // staying open forever. + // + // This uses the real TimeProvider.System (not a FakeTimeProvider) for the pipeline's clock, + // exactly like the other real-circuit-breaker tests above, and shrinks BreakDuration itself + // (via BuildDefaultPipeline's breakDuration parameter) so this test can wait out the break + // for real, quickly -- rather than driving a FakeTimeProvider through Polly, which is + // documented (Polly #1932, see offline-analytics.md's "Polly" section) to need extra + // ceremony (nulling the SynchronizationContext / ConfigureAwait(true)) or retries silently + // don't fire. + [Test] + public async Task DrainOnce_WithRealCircuitBreaker_AfterBreakDurationElapses_HalfOpenTrialSucceedsAndClosesBreaker() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var shortBreakDuration = TimeSpan.FromMilliseconds(600); + var failingSender = new AlwaysResultSender(SendResult.RetryableFailure); + var pipeline = MixpanelClient.BuildDefaultPipeline(TimeProvider.System, + retryDelay: TimeSpan.Zero, breakDuration: shortBreakDuration); + + var client = new MixpanelClient(); + // Deliberately NOT overriding batchSize here (unlike the regression test above): + // tripping only depends on the retry pipeline producing enough outcomes, not on + // batch size, and leaving it at the default means the half-open trial below gathers + // BOTH events still queued at that point (the rolled-back "Save" and "StillOpen") + // into a single batch, matching the ApproximateCount assertions below. + client.InitializeForTest(spool, failingSender, pipeline: pipeline); + + // Trip the breaker: one fully-failing tick produces 3 outcomes (1 attempt + the + // default MaxRetryAttempts=2 retries), enough to satisfy MinimumThroughput=3 on its + // own (see the regression test above for why this is exactly 3, not 4). + client.Track("user-1", "Save", null); + await client.DrainOnceAsync(); + var callsWhileTripping = failingSender.CallCount; + Assert.GreaterOrEqual(callsWhileTripping, 3); + + // While still within BreakDuration, the breaker must still be open: a further drain + // must not even reach the sender. + client.Track("user-1", "StillOpen", null); + await client.DrainOnceAsync(); + Assert.AreEqual(callsWhileTripping, failingSender.CallCount, + "the breaker must still be open immediately after tripping"); + + // Let BreakDuration elapse for real (short on purpose, see shortBreakDuration above, + // so this test stays fast), then swap in a succeeding sender on the SAME client / + // spool / pipeline instance -- exactly the pattern the crash-window dedup test above + // uses to swap senders mid-test -- so the breaker's own state carries over. + await Task.Delay(shortBreakDuration + TimeSpan.FromMilliseconds(400)); + + var succeedingSender = new AlwaysResultSender(SendResult.Delivered); + client.InitializeForTest(spool, succeedingSender, pipeline: pipeline); + + await client.DrainOnceAsync(); // The half-open trial: gathers BOTH queued events. + + Assert.AreEqual(1, succeedingSender.CallCount, + "once BreakDuration has elapsed, the half-open trial must reach the sender"); + Assert.AreEqual(0, spool.ApproximateCount, + "the trial batch (both events still queued: the rolled-back Save and StillOpen) " + + "succeeded and must be fully delivered"); + + // And the breaker should now be fully closed: a further event flows through normally, + // with no special handling. + client.Track("user-1", "AfterRecovery", null); + await client.DrainOnceAsync(); + + Assert.AreEqual(2, succeedingSender.CallCount); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + // ---- CONTRACT: host parameter is rejected (Mixpanel does not support a host) ----------- [Test] @@ -923,6 +1041,31 @@ public void PurgeQueuedEvents_ThenResumeSending_TogglesSendingPaused() } } + // Defense-in-depth: Track() itself must no-op while _paused (consent revoked), not just + // rely on the caller (Analytics.TrackWithApplicationProperties) checking AllowTracking + // first. Covers the residual race window between a consent revocation observed by + // PurgeQueuedEvents/PauseTimer and a concurrent Track() call that a caller-side check alone + // cannot close. + [Test] + public void Track_WhileSendingPaused_DoesNotEnqueue() + { + using (var spool = new EventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); + + client.PurgeQueuedEvents(); + Assert.IsTrue(client.SendingPaused, "test setup: client must be paused before Track is called"); + + client.Track("user-1", "ShouldNotEnqueue", null); + + Assert.AreEqual(0, spool.ApproximateCount, + "Track() must not enqueue while the flush loop is paused (consent revoked)"); + Assert.AreEqual(0, client.Statistics.Submitted, + "a Track() call while paused must not even count as submitted"); + } + } + // fix: consent revocation must not sit blocked behind an in-flight send (EventSpool.Purge // and ProcessBatchAsync share one lock, held across the network await), and the batch that // send was carrying must not survive to be delivered after consent was revoked -- it must @@ -964,5 +1107,91 @@ public async Task PurgeQueuedEvents_WhileSendInFlight_CancelsSendReturnsPromptly "leave it to be delivered afterward"); } } + + // ---- CONCURRENCY: Track() racing an in-progress drain ------------------------------------ + + // Stress/regression coverage for EventSpool._sync serialization: while a drain's send is + // genuinely in flight (blocked, via BlockingUntilSignaledSender, until this test releases + // it -- never a real Thread.Sleep-style delay), several threads call Track() concurrently. + // This does not assert a specific interleaving -- only that nothing throws, nothing + // deadlocks, and the bookkeeping (Submitted == Succeeded + Failed + still-spooled) stays + // internally consistent once everything settles. + [Test] + public async Task ConcurrentTrack_WhileDrainInProgress_NoExceptionsAndConsistentFinalState() + { + using (var spool = new EventSpool(_spoolDir, 1000)) + { + var sender = new BlockingUntilSignaledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, batchSize: 5); + + // Seed one event so the drain below has something to gather and actually calls the + // (blocking) sender. + client.Track("user-1", "Seed", null); + + var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). + + // Wait for the send to actually start, so the concurrent Track() calls below + // genuinely race an in-flight drain (EventSpool._sync held across the network await) + // rather than running before the drain even begins. + var deadline = DateTime.UtcNow.AddSeconds(5); + while (sender.CallCount == 0 && DateTime.UtcNow < deadline) + await Task.Delay(10); + Assert.AreEqual(1, sender.CallCount, + "the drain's send must have started before the concurrent Track() calls below"); + + const int threadCount = 4; + const int perThread = 25; + var exceptions = new ConcurrentBag(); + var threads = new Thread[threadCount]; + for (var t = 0; t < threadCount; t++) + { + var threadIndex = t; + threads[t] = new Thread(() => + { + try + { + for (var i = 0; i < perThread; i++) + client.Track("user-1", $"T{threadIndex}-{i}", null); + } + catch (Exception e) + { + exceptions.Add(e); + } + }); + } + + foreach (var thread in threads) + thread.Start(); + + // Release the blocked send BEFORE joining, not after: EventSpool._sync is held for + // the whole drain (including across this send), so every one of the Track() threads + // above piles up waiting on it -- joining first would deadlock forever waiting for + // threads that can only make progress once the send (and thus the lock) is released. + sender.Release(); + + foreach (var thread in threads) + thread.Join(); + await drainTask; + + Assert.IsEmpty(exceptions, "no Track() call racing an in-flight drain should throw"); + + var expectedSubmitted = 1 + threadCount * perThread; + Assert.AreEqual(expectedSubmitted, client.Statistics.Submitted, + "every Track() call (the seed plus every concurrent one) must be counted submitted"); + Assert.AreEqual(expectedSubmitted, + client.Statistics.Succeeded + client.Statistics.Failed + spool.ApproximateCount, + "every submitted event must be accounted for as succeeded, failed, or still spooled " + + "-- concurrent access must not corrupt that invariant"); + + // Drain the rest so the final state is fully resolved. + while (spool.ApproximateCount > 0) + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount); + Assert.AreEqual(expectedSubmitted, client.Statistics.Succeeded); + Assert.AreEqual(0, client.Statistics.Failed); + } + } } } diff --git a/src/DesktopAnalyticsTests/PathScrubberTests.cs b/src/DesktopAnalyticsTests/PathScrubberTests.cs index e047c23..d76664b 100644 --- a/src/DesktopAnalyticsTests/PathScrubberTests.cs +++ b/src/DesktopAnalyticsTests/PathScrubberTests.cs @@ -113,5 +113,52 @@ public void Scrub_HugeStackTraceWithManyPaths_ReplacesAllWithoutError() StringAssert.Contains("%USER%\\repo\\File0.cs", result); StringAssert.Contains("%USER%\\repo\\File499.cs", result); } + + [Test] + public void Scrub_WindowsPath_NoTrailingSeparatorAtEndOfString_ReplacesUserSegment() + { + const string input = @"Access denied: C:\Users\jsmith"; + const string expected = "Access denied: %USER%"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_WindowsPath_ForwardSlashes_ReplacesUserSegmentPreservingSlash() + { + const string input = @"C:/Users/jsmith/file.txt"; + const string expected = "%USER%/file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_UncPath_ReplacesUserSegment() + { + const string input = @"\\fileserver\Users\jsmith\Documents\file.txt"; + const string expected = @"%USER%\Documents\file.txt"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_UncPath_NoTrailingSeparatorAtEndOfString_ReplacesUserSegment() + { + const string input = @"\\fileserver\Users\jsmith"; + const string expected = "%USER%"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_UnixPath_NoTrailingSeparatorAtEndOfString_ReplacesUserSegment() + { + const string input = "/home/jsmith"; + const string expected = "%USER%"; + Assert.AreEqual(expected, PathScrubber.Scrub(input)); + } + + [Test] + public void Scrub_WindowsPathWithNoUsersSegment_ReturnsInputUnchanged() + { + const string input = @"D:\Projects\MyApp"; + Assert.AreEqual(input, PathScrubber.Scrub(input)); + } } } From 8cb2ef61610011942bbce7ca4cb576c1ec0d40ef Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 19:12:59 -0400 Subject: [PATCH 03/11] new plan --- offline-analytics-v2-plan.md | 287 +++++++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 offline-analytics-v2-plan.md diff --git a/offline-analytics-v2-plan.md b/offline-analytics-v2-plan.md new file mode 100644 index 0000000..25a006c --- /dev/null +++ b/offline-analytics-v2-plan.md @@ -0,0 +1,287 @@ +# Offline Analytics v2 — Re-architecture Plan + +Supersedes the Phase-1 plan in `offline-analytics.md` (kept for history). Written 2026-07-16 +after a cross-cutting review of PR #43 plus a consumer census. This plan is why PR #43, which +was "approved and nearly done", is being substantially reworked rather than merged. + +## Why the plan changed + +Four findings, in order of impact. + +1. **FW Lite is an intended consumer** (hahn-kev, PR #43 review 4660741363: *"I want to use it in + FW Lite, however FW Lite uses async Tasks where possible"*). FW Lite is **net10**, ships on + **Windows, Linux x64/arm64, macOS x64/arm64, and Android**, is self-contained, and is + single-file on Linux. **A net462-only package cannot be consumed by it at all.** Multi-targeting + is now mandatory, not a nice-to-have. +2. **Only 2 of 7 consumers use Mixpanel** — classic FieldWorks and (intended) FW Lite. The other + five (Bloom, HearThis, SayMore, Glyssen, Transcelerator) use the Segment client and can never + execute the spool. As written, PR #43 makes all five inherit DiskQueue + Polly + mixpanel-csharp + for a code path they cannot reach. +3. **The storage engine choice was made on a fabricated constraint.** `CONTEXT.md` asserted "no + native components (consuming apps assume registration-free COM)". That invariant exists nowhere + in this repo or in FieldWorks; an earlier planning session invented it and later sessions cited + it as fact. FieldWorks already ships native NuGet assets (`libSkiaSharp`, `libHarfBuzzSharp`, + icu.net). It ruled out SQLite for no reason. (Corrected in `CONTEXT.md`.) +4. **A UI-freeze bug on exactly the target scenario.** `EventSpool` holds its lock across the + network send, so `Track()` — called on the host UI thread — blocks for the whole send. + Measured: 3033 ms behind a 3 s send; ~45 s in production against stalled Wi-Fi (15 s HttpClient + timeout x 3 Polly attempts). `ShutDown()` measured 12 s against its documented ~5 s bound. + +### What is NOT a reason + +The stall is **not** inherent to DiskQueue. Probed 2026-07-16: DiskQueue enqueues in ~8 ms while an +uncommitted dequeue session is open, and a second session dequeues a *distinct* item rather than +blocking or double-serving. Dropping `EventSpool`'s global semaphore would fix the stall with no +engine change. SQLite is chosen for the reasons below, not for the stall. + +## Decisions + +| # | Decision | Rationale | +|---|---|---| +| D1 | **Split into two packages**: `SIL.DesktopAnalytics` (facade + Segment) and `SIL.DesktopAnalytics.Mixpanel` (Mixpanel client + durable spool + Polly + mixpanel-csharp) | The only way to *guarantee* the five Segment consumers are unaffected. Also makes them lighter than PR #43 leaves them. | +| D2 | **Both packages multi-target `net462;netstandard2.0;net8.0`** | See "TFM strategy" below. **`net462` floor is NOT negotiable up to `net48`** — that would break HearThis and Glyssen, which are net472. | +| D2a | **Settings: `#if NET462` keeps `ApplicationSettingsBase` exactly as today; modern TFMs get a JSON store** | Eliminates the migration risk entirely — existing users' `user.config` is never touched. FW Lite has no existing settings to preserve. | +| D3 | **SQLite (`Microsoft.Data.Sqlite`) replaces DiskQueue**; delete `EventSpool` | Fixes multi-process event loss (below) and the stall by construction. FW Lite already ships `SQLitePCLRaw.bundle_e_sqlite3`, so it costs FW Lite nothing. | +| D4 | **API break, major version (7.0.0)** | `ClientType.Mixpanel` in a core enum cannot resolve a type in another package. Approved by the owner. Consumers are on 4.0.0/4.0.4/6.0.2 anyway. | +| D5 | **Async-first API** | hahn-kev's explicit requirement for FW Lite. | +| D6 | **Per-event retry counter** | hahn-kev, PR #43: *"if there's a bug with a specific event we could get stuck on a single event and retry it forever"*. | +| D7 | **Path scrubbing, `$insert_id` dedup, consent purge, `/import` batching, 60-day retention** — all carry over unchanged from PR #43 | These were reviewed and are sound. | + +### Why SQLite specifically (the reason that survives scrutiny) + +Not the stall (see above). The real reasons: + +- **Multi-process event loss.** FieldWorks runs multiple `FieldWorks.exe` processes against one + per-user spool. DiskQueue takes a **cross-process exclusive lock**, so the second process cannot + open the spool, degrades to a no-op, and **silently loses every event it produces**. This is + unfixable within DiskQueue. SQLite's WAL gives real concurrent readers/writers. +- **Consent purge honesty.** `DELETE` + `VACUUM` genuinely reclaims bytes. DiskQueue needed + dispose + delete-the-directory + reopen, which briefly drops the cross-process lock and can + degrade the spool to a no-op if another process steals it. +- **Complexity that exists only to compensate for DiskQueue**: the `spool-bytes.txt` sidecar, its + staleness heuristic, and its re-measure fallback all collapse to `SUM(len)`. +- **Cross-platform**: FW Lite needs Linux/macOS/**Android**. DiskQueue's file-locking on those is + untested by us; SQLite's is not. + +Rejected: **LiteDB** (pure-managed, no native asset) — no stable release in ~2 years, and open bugs +#1538 (multi-process lock contention) and #2526 (unrecoverable state under concurrency) sit exactly +on the claim/release/send/delete pattern we depend on. + +## Consumer matrix + +| Consumer | Client | Needs spool | TFM | Platforms | Impact of this plan | +|---|---|---|---|---|---| +| FieldWorks (classic) | Mixpanel | **yes** | net48 (net10 Linux planned) | Win x64 | Adds `.Mixpanel` package; gains multi-process durability. Linux is a new platform for this app — no existing settings to migrate there | +| FW Lite | Mixpanel (intended) | **yes** | net10 / MAUI | Win, Linux, macOS, Android | Becomes consumable at all; already ships `bundle_e_sqlite3` | +| Bloom | Segment | no | net8.0-windows | Win | **Lighter** — sheds DiskQueue/Polly/mixpanel-csharp; avoids `bundle_green` collision risk | +| HearThis | Segment | no | net472 | Win | **Lighter** | +| SayMore | Segment | no | net48 | Win | **Lighter** | +| Glyssen | Segment | no | net472 | Win | **Lighter** | +| Transcelerator | Segment | no | net48 | Win (Paratext plugin) | **Lighter** | +| Paratext (host) | unknown | — | — | — | unknown; closed source | + +## Phases + +### Phase 0 — Verify the long pole ✅ DONE (2026-07-16) + +**Question:** can this library reach `netstandard2.0` at all? +**Answer: yes, and the blocker is small.** A `netstandard2.0` build fails on exactly two files with +four missing types: + +| File | Missing types | +|---|---| +| `AnalyticsSettings.Designer.cs` | `ApplicationSettingsBase`, `UserScopedSettingAttribute`, `DefaultSettingValueAttribute` | +| `Analytics.cs` | `ApplicationSettingsBase`, `ConfigurationUserLevel` | + +Everything else compiles clean on netstandard2.0 — Segment, mixpanel-csharp, DiskQueue, Polly, +**Microsoft.Data.Sqlite**, `NetworkChange`, `TimeProvider`. `AnalyticsSettings` is only six +user-scoped values (`IdForAnalytics`, `LastVersionLaunched`, `NeedUpgrade`, `FirstName`, +`LastName`, `Email`). This is a contained refactor, not a rewrite. + +## TFM strategy (decided 2026-07-16, empirically verified) + +Options considered. The net4x floor is the decisive axis: **a net4x TFM is only consumable by that +version or higher**, and HearThis + Glyssen are **net472**. + +| Option | net472? | net48? | net10? | Redirect risk | Verdict | +|---|---|---|---|---|---| +| A. `netstandard2.0` only | via facades | via facades | ✅ | **High** — reintroduces binding-redirect fragility for net4x consumers | no | +| B. `net462;netstandard2.0` | ✅ native | ✅ | ✅ (fallback) | low | safe, but net10 gets legacy surface | +| C. `net462;net8.0` | ✅ native | ✅ | ✅ | low | lean, defensible | +| **D. `net48;net10.0`** *(as originally requested)* | ❌ **BREAKS** | ✅ | ✅ | — | **ruled out** | +| **E. `net462;netstandard2.0;net8.0`** | ✅ native | ✅ | ✅ native | low | **chosen** | +| F. `netstandard2.0;net8.0` | via facades | via facades | ✅ | **High** | no | + +**Chosen: E.** It's Microsoft's own documented pattern (*"CONSIDER adding a target for net462 when +you're also targeting netstandard2.0 — using .NET Standard 2.0 from .NET Framework has some issues +that were addressed in 4.7.2"*). A `net8.0` asset serves net10 and MAUI's `net10.0-*` TFMs by +forward compatibility; no net10-only APIs are needed. It also lets `Microsoft.Bcl.TimeProvider` be +dropped conditionally on the net8.0 leg (`TimeProvider` is in-box there). + +**Note on the net48 request:** keeping the floor at `net462` *does* serve net48 — every net4x +consumer from 4.6.2 up gets the native asset. Raising the floor to `net48` gains nothing and +breaks two shipping consumers. + +**Verified empirically (2026-07-16), not inferred:** +- `net462;netstandard2.0;net8.0` compiles **clean — zero code changes, zero warnings** — once + `System.Configuration.ConfigurationManager` 8.0.0 is added for the non-net462 legs. The four + "missing types" are all supplied by that package (it multi-targets net462/netstandard2.0/net8.0). + So the earlier claim that `ApplicationSettingsBase` is WindowsDesktop-only is **false**. +- `ApplicationSettingsBase` also **runs** on .NET 8 on Windows *and* on real Linux (Ubuntu 26.04, + via WSL): `Save()` succeeds, persists to an XDG-correct `~/.local/share/.../user.config`, and + reads back across restarts. It works under `PublishSingleFile` on Linux too. (This contradicts + `dotnet/runtime#28833`, which appears stale.) + +### So why not just use `ApplicationSettingsBase` everywhere? + +Because "it runs" isn't the bar. The probe surfaced the disqualifying detail: + +``` +self-contained, non-single-file: .../settingsprobe_Url_13lsa3weyuz52bthetj4czmrh2ia2n3f/1.0.0.0/user.config +self-contained, single-file: .../settingsprobe_Path_rwy2hyf5upl41j2llclliaklcluo0ikv/1.0.0.0/user.config +``` + +**The settings identity is a hash of the app's path/URL evidence plus its version** — it changed +(`_Url_` → `_Path_`) purely from changing the publish shape. For FW Lite — self-contained, +single-file, auto-updating, installed to varying paths — that means **`IdForAnalytics` silently +resets and the user looks brand-new**. That is precisely what `IdForAnalytics` exists to prevent. +The `1.0.0.0` version folder is the same problem, and is why `.Upgrade()` exists at all. + +This is a known family of bugs, not our misuse: `dotnet/runtime#121053` (unstable per-install +settings hash), `#98715` (`.Upgrade()` throws on fresh macOS accounts). Both open. And +`System.Configuration.ConfigurationManager`'s own README says it *"exists only to support migrating +existing .NET Framework code"* — it is a migration shim, not new cross-platform infrastructure. + +A plain JSON file at a **stable, well-known path** (`LocalApplicationData/SIL/DesktopAnalytics/`) is +strictly more stable than `user.config` — no evidence hash, no version folder, no `.Upgrade()` +needed. It's also the pattern Avalonia documents, which is the UI stack FW Lite is built on. + +**A related risk was considered and does NOT apply, on closer inspection.** John: *"FieldWorks will +ship Linux again, but with net10, not Mono."* First pass reasoning here proposed a Windows +`user.config` import to protect against that move — that doesn't hold up. **New Linux users have no +prior `user.config` on that platform to lose**; there is nothing to import for them. The only +scenario that *would* need an import is classic FieldWorks **on Windows** later leaving net462 for +the net8/net10 leg — a different, unscheduled, unconfirmed move that has not been stated as planned. +Building an import keyed to the wrong transition (Windows path, justified by a Linux launch) would +be dead code for the Linux users and untested for the Windows scenario it would actually serve. +**Decision: do not build this now.** `IAnalyticsSettingsStore` (Phase 1) is the seam — if classic +FieldWorks' Windows build is ever moved off net462, add a `user.config`-import path to the JSON +store then, when the actual transition is confirmed. Flagging the seam here so it isn't rediscovered +from scratch. + +### Phase 1 — Multi-target + settings split (the gate) + +**Risk reassessment: this phase got much cheaper.** The original plan moved *all* TFMs to a JSON +store and migrated existing users — carrying a High risk of losing `IdForAnalytics` for every +existing FieldWorks user. That is no longer necessary. + +1. Multi-target `net462;netstandard2.0;net8.0`. Add `System.Configuration.ConfigurationManager` for + the non-net462 legs; drop `Microsoft.Bcl.TimeProvider` on the net8.0 leg. +2. **`#if NET462`: keep `AnalyticsSettings`/`ApplicationSettingsBase` byte-for-byte as it is + today.** Existing consumers (all .NET Framework) keep reading and writing the exact same + `user.config`, with the exact same `.Upgrade()` behavior. **Zero migration. Zero risk. No + behavior change.** +3. **Modern TFMs: a small JSON-backed store** (`LocalApplicationData/SIL/DesktopAnalytics/settings.json`, + System.Text.Json) behind `IAnalyticsSettingsStore`. No migration needed for FW Lite — it has no + existing settings. No `.Upgrade()` — a stable path makes it unnecessary. +4. Do **not** route net8/net10 through `ApplicationSettingsBase`, even though it compiles and runs. + See above. +5. **No legacy import in this phase.** Considered and rejected for now — see the note above. FW + Lite has no prior settings to migrate; there is no confirmed transition on the Windows FieldWorks + build that would need one yet. `IAnalyticsSettingsStore` is the seam to extend if/when that + changes. +6. Tests: JSON store round-trip + restart stability on the modern leg; net462 leg unchanged + (existing tests must still pass untouched). + +**Exit criteria:** all three TFMs build; net462 behavior provably unchanged; JSON store stable +across restart and across a simulated path/version change. + +### Phase 2 — Package split + API break (7.0.0) + +1. New project/package `SIL.DesktopAnalytics.Mixpanel`: `MixpanelClient`, `IEventSpool`, + `SqliteEventSpool`, `IEventSender`, `MixpanelEventSender`, `AnalyticsEvent`, `PathScrubber`. + Moves `Microsoft.Data.Sqlite`, `Polly`, `mixpanel-csharp` out of core. +2. Core keeps the facade, `SegmentClient`, `IClient`, `UserInfo`, settings. +3. Replace the `ClientType` enum with client injection. Confirm `IClient` is a good public seam. +4. **Flag for FW Lite (per owner):** we pull `SQLitePCLRaw` 2.1.6 via `Microsoft.Data.Sqlite` + 8.0.10; FW Lite pins **3.0.3** via `EFCore.Sqlite` 10.0.8. NuGet unifies upward — **FW Lite to + verify the upgrade causes no issues.** May need TFM-conditional `Microsoft.Data.Sqlite` + versions (8.0.x for net462, 9.x/10.x for netstandard2.0) if newer versions drop netstandard2.0. +5. Migration notes for all five Segment consumers (they should see only a version bump + a lighter + dependency graph). + +### Phase 3 — Adopt SQLite, delete DiskQueue + +1. Point `MixpanelClient.Initialize` at `SqliteEventSpool`. +2. Delete `EventSpool`, `EventSpoolTests`, the DiskQueue dependency, and the + `Track_WhileSendInFlight_BlocksUntilSendCompletes_DiskQueue` characterization test. +3. Collapse `EventSpoolContractTests` to a single engine. +4. Keep `IEventSpool` — it's the test seam and the injection point. + +### Phase 4 — Async-first + +1. `TrackAsync` / `ReportExceptionAsync` as primitives; `IEventSpool.EnqueueAsync`. +2. **Honest caveat:** `Microsoft.Data.Sqlite` does **not** implement true async I/O — its async + ADO.NET methods are synchronous internally. The win here is API ergonomics for FW Lite and the + already-async network path, **not** non-blocking disk I/O. Do not claim otherwise. A local + SQLite insert is sub-millisecond; that is why it's acceptable. +3. Decide the fate of sync `Track()`: keep as a non-blocking convenience for classic FieldWorks + (which has ~dozens of UI-thread call sites and will not await), or drop it. Leaning keep. + +### Phase 5 — Per-event retry counter (D6) + +1. Add `attempts INTEGER NOT NULL DEFAULT 0` to the events table. +2. On `RetryableFailure`: `UPDATE events SET attempts = attempts + 1, lease_until_unix_ms = 0 + WHERE id IN (...)`. +3. Then `DELETE FROM events WHERE attempts >= @maxAttempts`, counting each as Failed via + `ItemDroppedByCap` (or a new signal). +4. Default ~10 attempts. Note this bounds a wedge that today only the 60-day age floor catches — + i.e. 60 days of pointless retries. +5. Tests: N-1 attempts retained; Nth dropped; counter survives restart; a Delivered batch never + increments. + +### Phase 6 — Outstanding review findings (still open from PR #43) + +1. **`ShutDown()` violates its own bound** — measured 12 s vs. a documented ~5 s. It never calls + `CancelInFlightSend()` the way `PurgeQueuedEvents` does, so it waits out an in-flight send. + (SQLite likely improves this; **measure, do not assume**.) +2. **Drain pacing** — 256 KB per 30 s tick is ~8.5 KB/s (~68 kbit/s). Clearing a full backlog takes + **50-100 minutes of continuous connectivity**; a cafe session will not finish it. It resumes + safely, but this likely defeats the intent. Consider back-to-back ticks while a backlog exists + and sends succeed. +3. **PR text** claims Flush/ShutDown are bounded — currently false (see 1). +4. `ApproximateCount` semantics differ mid-send between engines (moot once DiskQueue is deleted). + +### Phase 7 — Consumer integration + +1. FieldWorks: bump + add `.Mixpanel`; verify CPM transitive pinning; installer auto-harvests via + WiX heat, so `e_sqlite3.dll` needs no installer work (~4.1 MB, 3 Windows RIDs). +2. FW Lite: adopt; verify the SQLitePCLRaw version story; `PublishSingleFile` on Linux needs + `IncludeNativeLibrariesForSelfExtract=true` (FW Lite has likely already solved this). +3. Segment consumers: no action beyond a version bump. + +## Risks + +| Risk | Severity | Mitigation | +|---|---|---| +| Settings migration loses `IdForAnalytics` | Low, for now | `#if NET462` keeps *today's* users on `user.config` untouched. A Linux launch creates no migration (no prior Linux `user.config` to lose). If classic FieldWorks' **Windows** build is ever moved off net462, `IAnalyticsSettingsStore` is the seam to add a `user.config` import then — not built speculatively now | +| Raising the net4x floor to net48 breaks HearThis + Glyssen (net472) | **High if done** | Keep the floor at `net462`; it already serves net48 | +| Routing modern TFMs through `ApplicationSettingsBase` resets `IdForAnalytics` on path/version change | **High if done** | Don't. JSON store at a stable path. Verified: the evidence hash changes with publish shape (`_Url_`→`_Path_`) | +| `Microsoft.Data.Sqlite` v11 proposes dropping .NET Framework support (`dotnet/efcore#37895`) | Low (proposed, not shipped) | Pin ≤10.x for net462/netstandard2.0 legs via TFM-conditional versions if it lands | +| `.Upgrade()` has open bugs on macOS/iOS (`#98715`, `#121053`) | Low | We only call `.Upgrade()` on the net462 leg (Windows) | +| SQLitePCLRaw version skew with FW Lite | Medium | Flagged in PR; FW Lite verifies; TFM-conditional versions if needed | +| API break churns 7 consumers | Medium | Accepted by owner; all are on old pins already | +| Bloom `bundle_green` vs `bundle_e_sqlite3` collision | Low (moot under D1) | Package split removes the question entirely. *Note: verified both bundles ship an assembly named `SQLitePCLRaw.batteries_v2.dll`; a real build break was NOT demonstrated.* | +| netstandard2.0 loses `NetworkChange` reconnect accelerator | Low | Works on modern .NET (netlink); already wrapped in try/catch and non-fatal — the poll is the guarantee | +| Scope growth | Medium | Phases 1-3 are the core; 4-6 can ship incrementally | + +## Current state (what exists today, on `feature/offline-mixpanel-durability`) + +- 172/172 tests green. +- `IEventSpool` extracted; `EventSpool` (DiskQueue) and `SqliteEventSpool` both implement it. +- `EventSpoolContractTests` — 28 tests x 2 engines = 56, proving behavioral parity. +- `TrackResponsivenessTests` — SQLite returns in single-digit ms; DiskQueue blocks for the whole + send (pinned as a characterization test). +- **Production still uses DiskQueue.** `MixpanelClient.Initialize` constructs `EventSpool`. Nothing + is adopted yet. +- `CONTEXT.md` and `offline-analytics.md` corrected re: the fabricated invariant. From 8fb8f541ad52ffba12c4d20e688f3627e5e94eea Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 19:35:59 -0400 Subject: [PATCH 04/11] Extract IEventSpool seam + SQLite spike, multi-target net462/netstandard2.0/net8.0 IEventSpool abstracts EventSpool (DiskQueue) vs. the new SqliteEventSpool behind one contract, proven equivalent by 28 contract tests x 2 engines. Adds TrackResponsivenessTests characterizing the UI-freeze bug SQLite fixes. Multi-targets DesktopAnalytics to net462;netstandard2.0;net8.0 with an IAnalyticsSettingsStore split: net462 keeps ApplicationSettingsBase/user.config byte-for-byte (NetFrameworkAnalyticsSettingsStore), modern TFMs get a JSON store at a stable path instead (JsonAnalyticsSettingsStore), since ApplicationSettingsBase's per-install path hash isn't stable across publish shapes. See offline-analytics-v2-plan.md. Production still constructs EventSpool (DiskQueue); adopting SqliteEventSpool is a later phase. All three TFMs build; 172/172 tests green on net462, 177/177 on net8.0 (5 new JSON store tests). Co-Authored-By: Claude Opus 4.8 --- src/DesktopAnalytics/Analytics.cs | 50 +- src/DesktopAnalytics/DesktopAnalytics.csproj | 12 +- src/DesktopAnalytics/EventSpool.cs | 8 +- .../IAnalyticsSettingsStore.cs | 24 + src/DesktopAnalytics/IEventSpool.cs | 83 +++ .../JsonAnalyticsSettingsStore.cs | 136 ++++ src/DesktopAnalytics/MixpanelClient.cs | 4 +- .../NetFrameworkAnalyticsSettingsStore.cs | 53 ++ src/DesktopAnalytics/SqliteEventSpool.cs | 632 ++++++++++++++++++ .../DesktopAnalyticsTests.csproj | 4 +- .../EventSpoolContractTests.cs | 589 ++++++++++++++++ .../JsonAnalyticsSettingsStoreTests.cs | 114 ++++ .../TrackResponsivenessTests.cs | 180 +++++ 13 files changed, 1857 insertions(+), 32 deletions(-) create mode 100644 src/DesktopAnalytics/IAnalyticsSettingsStore.cs create mode 100644 src/DesktopAnalytics/IEventSpool.cs create mode 100644 src/DesktopAnalytics/JsonAnalyticsSettingsStore.cs create mode 100644 src/DesktopAnalytics/NetFrameworkAnalyticsSettingsStore.cs create mode 100644 src/DesktopAnalytics/SqliteEventSpool.cs create mode 100644 src/DesktopAnalyticsTests/EventSpoolContractTests.cs create mode 100644 src/DesktopAnalyticsTests/JsonAnalyticsSettingsStoreTests.cs create mode 100644 src/DesktopAnalyticsTests/TrackResponsivenessTests.cs diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index 519d1d5..a507fd8 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -60,6 +60,12 @@ public class Analytics : IDisposable private readonly Dictionary _propertiesThatGoWithEveryEvent; private static int s_exceptionCount = 0; +#if NET462 + private static readonly IAnalyticsSettingsStore s_settings = new NetFrameworkAnalyticsSettingsStore(); +#else + private static readonly IAnalyticsSettingsStore s_settings = new JsonAnalyticsSettingsStore(); +#endif + private class InitializationParameters { public string ApiSecret; @@ -129,7 +135,7 @@ private void UpdateServerInformationOnThisUser(bool initializing = false) if (!AllowTracking && !initializing) return; - _client.Identify(AnalyticsSettings.Default.IdForAnalytics, s_traits, s_locationInfo); + _client.Identify(s_settings.IdForAnalytics, s_traits, s_locationInfo); } /// @@ -222,13 +228,13 @@ public Analytics( private void Initialize(InitializationParameters parameters) { // Bring in settings from any previous version. - if (AnalyticsSettings.Default.NeedUpgrade) + if (s_settings.NeedUpgrade) { //see http://stackoverflow.com/questions/3498561/net-applicationsettingsbase-should-i-call-upgrade-every-time-i-load try { - AnalyticsSettings.Default.Upgrade(); - AnalyticsSettings.Default.NeedUpgrade = false; + s_settings.Upgrade(); + s_settings.NeedUpgrade = false; TrySaveSettings(); } catch (ConfigurationErrorsException e) @@ -244,7 +250,7 @@ private void Initialize(InitializationParameters parameters) } } - if (IsNullOrEmpty(AnalyticsSettings.Default.IdForAnalytics)) + if (IsNullOrEmpty(s_settings.IdForAnalytics)) { // Apparently a first-time installation. If possible, we really want to use the // same ID (from another channel of this app) to keep our statistics valid. @@ -265,9 +271,9 @@ private void Initialize(InitializationParameters parameters) parameters.FlushInterval ); - if (IsNullOrEmpty(AnalyticsSettings.Default.IdForAnalytics)) + if (IsNullOrEmpty(s_settings.IdForAnalytics)) { - AnalyticsSettings.Default.IdForAnalytics = Guid.NewGuid().ToString(); + s_settings.IdForAnalytics = Guid.NewGuid().ToString(); TrySaveSettings(); } @@ -300,18 +306,18 @@ private void Initialize(InitializationParameters parameters) // unlikely event that they do, it's not the end of the world. s_allowTracking = true; - if (IsNullOrEmpty(AnalyticsSettings.Default.LastVersionLaunched)) + if (IsNullOrEmpty(s_settings.LastVersionLaunched)) { // "Created" is a special property that segment.io understands and coverts to // equivalents in various analytics services. So it's not as descriptive for us as // "FirstLaunchOnSystem", but it will give the best experience on the analytics sites. TrackWithApplicationProperties("Created"); } - else if (AnalyticsSettings.Default.LastVersionLaunched != versionNumberWithBuild) + else if (s_settings.LastVersionLaunched != versionNumberWithBuild) { TrackWithApplicationProperties("Upgrade", new JsonObject { - {"OldVersion", AnalyticsSettings.Default.LastVersionLaunched}, + {"OldVersion", s_settings.LastVersionLaunched}, }); } @@ -319,7 +325,7 @@ private void Initialize(InitializationParameters parameters) // but that is done after we retrieve (or fail to retrieve) our external IP address. // See http://issues.bloomlibrary.org/youtrack/issue/BL-4011. - AnalyticsSettings.Default.LastVersionLaunched = versionNumberWithBuild; + s_settings.LastVersionLaunched = versionNumberWithBuild; TrySaveSettings(); } @@ -330,7 +336,7 @@ private static void TrySaveSettings() { try { - AnalyticsSettings.Default.Save(); + s_settings.Save(); return; } catch (Exception e) @@ -416,24 +422,24 @@ private void AttemptToGetUserIdSettingsFromDifferentChannel() string analyticsId = idSetting.Value; if (IsNullOrEmpty(analyticsId)) continue; - AnalyticsSettings.Default.IdForAnalytics = analyticsId; - AnalyticsSettings.Default.FirstName = ExtractSetting( - AnalyticsSettings.Default.FirstName, + s_settings.IdForAnalytics = analyticsId; + s_settings.FirstName = ExtractSetting( + s_settings.FirstName, doc, "FirstName" ); - AnalyticsSettings.Default.LastName = ExtractSetting( - AnalyticsSettings.Default.LastName, + s_settings.LastName = ExtractSetting( + s_settings.LastName, doc, "LastName" ); - AnalyticsSettings.Default.LastVersionLaunched = ExtractSetting( - AnalyticsSettings.Default.LastVersionLaunched, + s_settings.LastVersionLaunched = ExtractSetting( + s_settings.LastVersionLaunched, doc, "LastVersionLaunched" ); - AnalyticsSettings.Default.Email = ExtractSetting( - AnalyticsSettings.Default.Email, + s_settings.Email = ExtractSetting( + s_settings.Email, doc, "Email" ); @@ -1098,7 +1104,7 @@ private static void TrackWithApplicationProperties(string eventName, JsonObject } s_singleton._client.Track( - AnalyticsSettings.Default.IdForAnalytics, + s_settings.IdForAnalytics, eventName, properties ); diff --git a/src/DesktopAnalytics/DesktopAnalytics.csproj b/src/DesktopAnalytics/DesktopAnalytics.csproj index fe12a54..c34fa1c 100644 --- a/src/DesktopAnalytics/DesktopAnalytics.csproj +++ b/src/DesktopAnalytics/DesktopAnalytics.csproj @@ -1,7 +1,7 @@  - net462 + net462;netstandard2.0;net8.0 SIL.DesktopAnalytics John Hatton, Stephen McConnel, Tom Bogle, Eberhard Beilharz, John Thomson, Andrew Polk, Jason Naylor SIL International @@ -18,13 +18,20 @@ + All + + + + + @@ -34,10 +41,11 @@ - + + True True diff --git a/src/DesktopAnalytics/EventSpool.cs b/src/DesktopAnalytics/EventSpool.cs index 8fa8da6..13b8da2 100644 --- a/src/DesktopAnalytics/EventSpool.cs +++ b/src/DesktopAnalytics/EventSpool.cs @@ -46,7 +46,7 @@ internal enum SendResult /// the underlying DiskQueue's cross-process exclusive lock is a startup-time condition the /// caller needs to know about, so it is allowed to propagate. /// - internal class EventSpool : IDisposable + internal class EventSpool : IEventSpool { // How long we wait to acquire DiskQueue's cross-process exclusive lock when opening the // spool. Kept short: if another process is holding the lock for this long, something is @@ -388,9 +388,9 @@ private void EnforceCapLocked() } } - // Test seam: the spool's current logical byte total (the value the byte cap is enforced - // against), primarily so tests can prove it survives a dispose/reopen. - internal long ApproximateBytes + // The spool's current logical byte total (the value the byte cap is enforced against). + // Also a test seam: tests use it to prove the total survives a dispose/reopen. + public long ApproximateBytes { get { diff --git a/src/DesktopAnalytics/IAnalyticsSettingsStore.cs b/src/DesktopAnalytics/IAnalyticsSettingsStore.cs new file mode 100644 index 0000000..9950161 --- /dev/null +++ b/src/DesktopAnalytics/IAnalyticsSettingsStore.cs @@ -0,0 +1,24 @@ +//License: MIT + +namespace DesktopAnalytics +{ + /// + /// Abstraction over the small set of persisted user settings Analytics needs. On net462 this + /// is backed by (ApplicationSettingsBase/user.config) so + /// existing consumers see byte-for-byte identical behavior. On netstandard2.0/net8.0 it is + /// backed by a JSON file at a stable path, since ApplicationSettingsBase's per-install settings + /// path is not stable across publish shapes on those TFMs. + /// + internal interface IAnalyticsSettingsStore + { + string IdForAnalytics { get; set; } + string LastVersionLaunched { get; set; } + bool NeedUpgrade { get; set; } + string FirstName { get; set; } + string LastName { get; set; } + string Email { get; set; } + + void Save(); + void Upgrade(); + } +} diff --git a/src/DesktopAnalytics/IEventSpool.cs b/src/DesktopAnalytics/IEventSpool.cs new file mode 100644 index 0000000..c48b22d --- /dev/null +++ b/src/DesktopAnalytics/IEventSpool.cs @@ -0,0 +1,83 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace DesktopAnalytics +{ + /// + /// The durable, bounded store of pending s behind + /// . Implemented by (DiskQueue) and + /// (SQLite); see offline-analytics.md for the comparison. + /// + /// + /// Contract shared by every implementation: + /// + /// Never throws. Disk/IO/serialization failures are logged and swallowed -- + /// analytics must never crash the host. (Constructors may throw; that is a startup-time + /// condition the caller needs to know about.) + /// At-least-once delivery. An event is removed only after + /// 's callback confirms the batch's fate. A crash between a + /// successful send and that removal replays the batch, which Mixpanel deduplicates via + /// . + /// FIFO. Events are gathered oldest-first, and eviction drops oldest-first. + /// + /// + internal interface IEventSpool : IDisposable + { + /// + /// Raised once per event permanently dropped by cap enforcement -- i.e. NOT a drop at + /// enqueue time (which 's return value already reports), but an older + /// event evicted later to stay within the configured caps. This is the only way callers + /// learn of that kind of drop, since it can happen on an call for a + /// different event than the one dropped. Handlers must be fast and must not call back into + /// the spool. + /// + event Action ItemDroppedByCap; + + /// An approximate count of events currently spooled. Accurate enough for + /// bounding and diagnostics; not a hard guarantee under concurrent access. + int ApproximateCount { get; } + + /// The spool's current logical byte total -- the value the byte cap is enforced + /// against. + long ApproximateBytes { get; } + + /// + /// Serializes and enqueues , then enforces the configured caps by + /// dropping the oldest event(s) if necessary. + /// + /// True if durably enqueued; false if dropped at enqueue time (null, + /// unserializable, over the per-event byte cap, or a disk failure). + bool Enqueue(AnalyticsEvent evt); + + /// + /// Gathers up to events (oldest first, bounded by + /// ), hands them to as ONE batch, and + /// commits or rolls back on its verdict: and + /// remove the batch; + /// (or a throwing ) leaves it spooled, in order, for a later retry. + /// An entry that cannot be deserialized is dropped rather than allowed to wedge the spool. + /// + /// Soft byte budget for the batch, checked AFTER each gathered event + /// so a single oversized event still goes -- the budget can never starve the queue. + Task ProcessBatchAsync(int maxItems, + Func, CancellationToken, Task> send, + long maxBytes = long.MaxValue, CancellationToken cancellationToken = default); + + /// + /// Age-based retention floor: drops events whose stamped + /// is older than minus . An independent + /// eviction dimension alongside the item/byte caps, not a replacement for them. + /// + /// From the caller's injected so tests stay + /// deterministic -- never DateTimeOffset.UtcNow directly. + void TrimExpired(TimeSpan maxAge, DateTimeOffset now); + + /// + /// Empties the spool entirely (consent revocation), removing the event data from disk -- + /// not merely marking it consumed. + /// + void Purge(); + } +} diff --git a/src/DesktopAnalytics/JsonAnalyticsSettingsStore.cs b/src/DesktopAnalytics/JsonAnalyticsSettingsStore.cs new file mode 100644 index 0000000..424c740 --- /dev/null +++ b/src/DesktopAnalytics/JsonAnalyticsSettingsStore.cs @@ -0,0 +1,136 @@ +//License: MIT + +#if !NET462 +using System; +using System.IO; +using System.Reflection; +using System.Text.Json; + +namespace DesktopAnalytics +{ + /// + /// System.Text.Json-backed settings store for netstandard2.0/net8.0, used instead of + /// ApplicationSettingsBase because that type's per-install settings path is not stable across + /// publish shapes (self-contained vs. framework-dependent, single-file vs. not) on those TFMs + /// -- see offline-analytics-v2-plan.md. Stored at a stable, well-known path: + /// LocalApplicationData/SIL/DesktopAnalytics/{appId}/settings.json. + /// + internal sealed class JsonAnalyticsSettingsStore : IAnalyticsSettingsStore + { + // Mirrors AnalyticsSettings.Designer.cs's DefaultSettingValueAttribute defaults exactly. + private class Data + { + public string IdForAnalytics { get; set; } = ""; + public string LastVersionLaunched { get; set; } = ""; + public bool NeedUpgrade { get; set; } = true; + public string FirstName { get; set; } = ""; + public string LastName { get; set; } = ""; + public string Email { get; set; } = ""; + } + + private readonly string _filePath; + private readonly Data _data; + + /// Identifies this app's settings file on disk. Defaults to the entry + /// assembly's name; falls back to a fixed name if there is no entry assembly (e.g. plugin + /// hosting), mirroring the fallback pattern Analytics.cs already uses for + /// GetCallingAssembly/GetEntryAssembly. + public JsonAnalyticsSettingsStore(string appId = null) + : this(BuildDefaultRoot(appId), isRootPath: true) + { + } + + /// + /// Test seam: points the store at an explicit root directory instead of computing one from + /// LocalApplicationData, so tests can use a temp directory. + /// + internal static JsonAnalyticsSettingsStore ForTesting(string rootPath) => + new JsonAnalyticsSettingsStore(rootPath, isRootPath: true); + + // The unused bool parameter exists only to give this constructor a distinct signature from + // the public string-appId one above (C# won't allow two ctors that both take a single + // string). + private JsonAnalyticsSettingsStore(string rootPath, bool isRootPath) + { + _filePath = Path.Combine(rootPath, "settings.json"); + _data = Load(_filePath); + } + + private static string BuildDefaultRoot(string appId) + { + appId = appId ?? Assembly.GetEntryAssembly()?.GetName().Name ?? "DesktopAnalytics"; + return Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), + "SIL", "DesktopAnalytics", appId); + } + + private static Data Load(string filePath) + { + if (!File.Exists(filePath)) + return new Data(); + + try + { + var json = File.ReadAllText(filePath); + return JsonSerializer.Deserialize(json) ?? new Data(); + } + catch (Exception) + { + // Corrupt or unreadable settings file: start fresh rather than crash the host app. + return new Data(); + } + } + + public string IdForAnalytics + { + get => _data.IdForAnalytics; + set => _data.IdForAnalytics = value; + } + + public string LastVersionLaunched + { + get => _data.LastVersionLaunched; + set => _data.LastVersionLaunched = value; + } + + public bool NeedUpgrade + { + get => _data.NeedUpgrade; + set => _data.NeedUpgrade = value; + } + + public string FirstName + { + get => _data.FirstName; + set => _data.FirstName = value; + } + + public string LastName + { + get => _data.LastName; + set => _data.LastName = value; + } + + public string Email + { + get => _data.Email; + set => _data.Email = value; + } + + public void Save() + { + var dir = Path.GetDirectoryName(_filePath); + if (!string.IsNullOrEmpty(dir) && !Directory.Exists(dir)) + Directory.CreateDirectory(dir); + + File.WriteAllText(_filePath, JsonSerializer.Serialize(_data)); + } + + // No legacy JSON schema exists to upgrade from, so this is a no-op. Only the net462 + // (ApplicationSettingsBase) store actually upgrades anything. + public void Upgrade() + { + } + } +} +#endif diff --git a/src/DesktopAnalytics/MixpanelClient.cs b/src/DesktopAnalytics/MixpanelClient.cs index 3ebae74..ca7be42 100644 --- a/src/DesktopAnalytics/MixpanelClient.cs +++ b/src/DesktopAnalytics/MixpanelClient.cs @@ -84,7 +84,7 @@ internal class MixpanelClient : IClient private static readonly TimeSpan s_boundedDrainDuration = TimeSpan.FromSeconds(5); private const int kBoundedDrainMaxAttempts = 20; - private EventSpool _spool; + private IEventSpool _spool; private IEventSender _sender; private TimeProvider _timeProvider = TimeProvider.System; private ResiliencePipeline _pipeline = ResiliencePipeline.Empty; @@ -194,7 +194,7 @@ public void Initialize(string apiSecret, string host = null, int flushAt = -1, i /// retry/circuit-breaker behavior pass their own pipeline (see /// ). internal void InitializeForTest( - EventSpool spool, + IEventSpool spool, IEventSender sender, TimeProvider timeProvider = null, ResiliencePipeline pipeline = null, diff --git a/src/DesktopAnalytics/NetFrameworkAnalyticsSettingsStore.cs b/src/DesktopAnalytics/NetFrameworkAnalyticsSettingsStore.cs new file mode 100644 index 0000000..fa61921 --- /dev/null +++ b/src/DesktopAnalytics/NetFrameworkAnalyticsSettingsStore.cs @@ -0,0 +1,53 @@ +//License: MIT + +#if NET462 +namespace DesktopAnalytics +{ + /// + /// Thin wrapper delegating every member straight to + /// (ApplicationSettingsBase/user.config). Preserves today's net462 behavior exactly. + /// + internal sealed class NetFrameworkAnalyticsSettingsStore : IAnalyticsSettingsStore + { + public string IdForAnalytics + { + get => AnalyticsSettings.Default.IdForAnalytics; + set => AnalyticsSettings.Default.IdForAnalytics = value; + } + + public string LastVersionLaunched + { + get => AnalyticsSettings.Default.LastVersionLaunched; + set => AnalyticsSettings.Default.LastVersionLaunched = value; + } + + public bool NeedUpgrade + { + get => AnalyticsSettings.Default.NeedUpgrade; + set => AnalyticsSettings.Default.NeedUpgrade = value; + } + + public string FirstName + { + get => AnalyticsSettings.Default.FirstName; + set => AnalyticsSettings.Default.FirstName = value; + } + + public string LastName + { + get => AnalyticsSettings.Default.LastName; + set => AnalyticsSettings.Default.LastName = value; + } + + public string Email + { + get => AnalyticsSettings.Default.Email; + set => AnalyticsSettings.Default.Email = value; + } + + public void Save() => AnalyticsSettings.Default.Save(); + + public void Upgrade() => AnalyticsSettings.Default.Upgrade(); + } +} +#endif diff --git a/src/DesktopAnalytics/SqliteEventSpool.cs b/src/DesktopAnalytics/SqliteEventSpool.cs new file mode 100644 index 0000000..5d7381b --- /dev/null +++ b/src/DesktopAnalytics/SqliteEventSpool.cs @@ -0,0 +1,632 @@ +using System; +using System.Collections.Generic; +using System.Data; +using System.Diagnostics; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Data.Sqlite; + +namespace DesktopAnalytics +{ + /// + /// SQLite-backed . Same contract as the DiskQueue-backed + /// , but the storage model removes -- rather than works around -- the + /// two structural problems that one has. + /// + /// + /// Why this exists. As written, holds its global + /// _sync semaphore across the network send, so every concurrent Track() waits out + /// the whole send -- measured at ~3s behind a 3s send, and up to ~45s in production against a + /// stalled connection (15s HttpClient timeout x 3 pipeline attempts). That is a UI freeze on + /// exactly the "bad cafe Wi-Fi" path this whole feature is for. + /// Note: that stall is NOT inherent to DiskQueue. Probed 2026-07-16: DiskQueue + /// happily enqueues (~8ms) while an uncommitted dequeue session is open, and a second session + /// dequeues a DISTINCT item rather than blocking or double-serving. So dropping + /// 's global lock would fix the stall without changing engines. The + /// case for SQLite rests on the OTHER problems below, not on the stall alone. + /// SQLite needs no such thing. claims a batch under a + /// short lock, releases the lock, sends with nothing held, then removes the delivered + /// rows under a second short lock. At-least-once falls out for free: a crash between the send + /// and the removal replays the batch, which Mixpanel deduplicates on + /// -- the semantics the design already depends on. + /// Because the lock is never held across an await, a plain lock suffices here where + /// needed a SemaphoreSlim. + /// What it deletes. The byte-total sidecar file, its staleness heuristic, and its + /// re-measure fallback all collapse into SUM(len). Age retention collapses into one + /// indexed DELETE. Cap eviction collapses into one DELETE. Purge becomes + /// DELETE + VACUUM instead of dispose + delete-the-directory + reopen. + /// Multi-process. WAL gives real concurrent readers/writers, so a second + /// FieldWorks process spools and drains normally. Today it cannot even open the DiskQueue + /// spool (exclusive lock) and silently degrades to a no-op, losing its events entirely. A + /// short lease on claimed rows keeps two processes from uploading the same batch; the lease + /// expires on its own, so a process that dies mid-send does not strand its batch. + /// + internal class SqliteEventSpool : IEventSpool + { + // How long a claimed batch stays invisible to other drains. Long enough to cover a slow + // send, short enough that a crashed process's batch is retried promptly. + private static readonly TimeSpan s_leaseDuration = TimeSpan.FromMinutes(2); + + // How long SQLite waits on a lock held by another process before returning SQLITE_BUSY. + private const int kBusyTimeoutMs = 2000; + + private readonly int _maxItems; + private readonly int _maxItemBytes; + private readonly long _maxSpoolBytes; + private readonly string _spoolDirectory; + private readonly string _databasePath; + private readonly TimeProvider _timeProvider; + + // Guards _connection. Only ever held for short, purely-local database work -- NEVER across + // the network send (see the class remarks). A monitor lock is therefore sufficient. + private readonly object _sync = new object(); + private SqliteConnection _connection; + private bool _disposed; + + /// + public event Action ItemDroppedByCap; + + /// Directory to hold the spool database. Callers normally get + /// this from ; tests inject a temp directory. + /// Maximum number of events retained; enqueuing beyond this drops + /// the oldest first. + /// Maximum serialized size of a single event; larger events are + /// refused at enqueue rather than spooled. + /// Maximum total serialized size of all retained events; + /// enqueuing beyond this drops the oldest first. + /// Clock for lease expiry. Injected so tests stay deterministic. + /// Propagates a failure to create/open the database. Like + /// 's constructor, this is deliberately not swallowed: it is a + /// startup-time condition the caller needs to know about. + public SqliteEventSpool(string spoolDirectory, int maxItems, int maxItemBytes = int.MaxValue, + long maxSpoolBytes = long.MaxValue, TimeProvider timeProvider = null) + { + if (maxItems < 0) + throw new ArgumentOutOfRangeException(nameof(maxItems)); + if (maxItemBytes <= 0) + throw new ArgumentOutOfRangeException(nameof(maxItemBytes)); + if (maxSpoolBytes <= 0) + throw new ArgumentOutOfRangeException(nameof(maxSpoolBytes)); + + _maxItems = maxItems; + _maxItemBytes = maxItemBytes; + _maxSpoolBytes = maxSpoolBytes; + _spoolDirectory = spoolDirectory; + _databasePath = Path.Combine(spoolDirectory, "spool.db"); + _timeProvider = timeProvider ?? TimeProvider.System; + + Directory.CreateDirectory(spoolDirectory); + _connection = OpenConnection(_databasePath); + InitializeSchema(_connection); + } + + private static SqliteConnection OpenConnection(string databasePath) + { + // Pooling off: this class holds its single connection open for its whole lifetime, so + // pooling buys nothing -- and it would keep the file handle alive after Dispose, + // blocking Purge's VACUUM and the temp-directory cleanup in tests. + var connectionString = new SqliteConnectionStringBuilder + { + DataSource = databasePath, + Pooling = false + }.ToString(); + + var connection = new SqliteConnection(connectionString); + connection.Open(); + + // WAL: concurrent readers and a writer, across processes -- the property DiskQueue's + // exclusive lock cannot give us. + // synchronous=NORMAL: no fsync per commit. A power loss (not a process crash -- WAL + // still recovers those) can lose the last few events. That is the right trade for + // analytics: the alternative fsyncs on every Track(), on the UI thread. + Execute(connection, "PRAGMA journal_mode=WAL;"); + Execute(connection, "PRAGMA synchronous=NORMAL;"); + Execute(connection, "PRAGMA busy_timeout=" + kBusyTimeoutMs + ";"); + return connection; + } + + private static void InitializeSchema(SqliteConnection connection) + { + Execute(connection, + // len is stored rather than computed so the cap queries below are index-only scans + // and never have to read the payload blobs. + "CREATE TABLE IF NOT EXISTS events (" + + " id INTEGER PRIMARY KEY AUTOINCREMENT," + + " time_unix_ms INTEGER NOT NULL," + + " lease_until_unix_ms INTEGER NOT NULL DEFAULT 0," + + " len INTEGER NOT NULL," + + " payload BLOB NOT NULL);"); + Execute(connection, "CREATE INDEX IF NOT EXISTS ix_events_time ON events(time_unix_ms);"); + Execute(connection, + "CREATE INDEX IF NOT EXISTS ix_events_lease ON events(lease_until_unix_ms, id);"); + } + + private static void Execute(SqliteConnection connection, string sql) + { + using (var cmd = connection.CreateCommand()) + { + cmd.CommandText = sql; + cmd.ExecuteNonQuery(); + } + } + + /// + /// Computes the per-user, per-API-key spool directory, matching + /// so the two engines key their storage + /// identically (DEBUG and RELEASE builds stay separate; the key is never embedded raw). + /// + public static string GetDefaultSpoolPath(string apiKey) => EventSpool.GetDefaultSpoolPath(apiKey); + + /// + public int ApproximateCount + { + get + { + try + { + lock (_sync) + { + if (_disposed || _connection == null) + return 0; + return (int)ScalarLocked("SELECT COUNT(*) FROM events;"); + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.ApproximateCount failed: " + e); + return 0; + } + } + } + + /// + public long ApproximateBytes + { + get + { + try + { + lock (_sync) + { + if (_disposed || _connection == null) + return 0; + return ScalarLocked("SELECT COALESCE(SUM(len), 0) FROM events;"); + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.ApproximateBytes failed: " + e); + return 0; + } + } + } + + // Caller must hold _sync. + private long ScalarLocked(string sql) + { + using (var cmd = _connection.CreateCommand()) + { + cmd.CommandText = sql; + return Convert.ToInt64(cmd.ExecuteScalar()); + } + } + + /// + public bool Enqueue(AnalyticsEvent evt) + { + if (evt == null) + return false; + + try + { + byte[] bytes; + try + { + bytes = evt.ToBytes(); + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.Enqueue: failed to serialize event, dropping: " + e); + return false; + } + + if (bytes.Length > _maxItemBytes) + { + // Refused up front rather than spooled: an event over the transport's per-event + // limit can never be delivered, and one large enough to time out the request + // would classify as RETRYABLE (indistinguishable from being offline) and wedge + // the head of the queue forever. + Debug.WriteLine("SqliteEventSpool.Enqueue: event of " + bytes.Length + + " bytes exceeds the " + _maxItemBytes + "-byte cap, dropping: " + evt.EventName); + return false; + } + + int dropped; + lock (_sync) + { + if (_disposed || _connection == null) + return false; + + using (var cmd = _connection.CreateCommand()) + { + cmd.CommandText = + "INSERT INTO events (time_unix_ms, lease_until_unix_ms, len, payload) " + + "VALUES (@time, 0, @len, @payload);"; + cmd.Parameters.AddWithValue("@time", evt.Time.ToUnixTimeMilliseconds()); + cmd.Parameters.AddWithValue("@len", bytes.Length); + cmd.Parameters.AddWithValue("@payload", bytes); + cmd.ExecuteNonQuery(); + } + + dropped = EnforceCapsLocked(); + } + + // Raised outside the lock: handlers are the caller's code, and nothing here needs + // the lock held while they run. + RaiseDropped(dropped); + return true; + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.Enqueue failed: " + e); + return false; + } + } + + private void RaiseDropped(int count) + { + for (var i = 0; i < count; i++) + { + try + { + ItemDroppedByCap?.Invoke(); + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool: ItemDroppedByCap handler threw: " + e); + } + } + } + + // Caller must hold _sync. Brings the spool back within BOTH caps, dropping oldest first, + // and returns how many events were dropped. Two DELETEs, no loop -- contrast + // EventSpool.EnforceCapLocked, which dequeues one item at a time. + private int EnforceCapsLocked() + { + var dropped = 0; + try + { + // Item cap. Also covers the degenerate maxItems == 0 case, where the event just + // enqueued is itself evicted. + using (var cmd = _connection.CreateCommand()) + { + cmd.CommandText = + "DELETE FROM events WHERE id IN (" + + " SELECT id FROM events ORDER BY id ASC" + + " LIMIT MAX(0, (SELECT COUNT(*) FROM events) - @maxItems));"; + cmd.Parameters.AddWithValue("@maxItems", _maxItems); + dropped += cmd.ExecuteNonQuery(); + } + + // Byte cap. The window function totals bytes newest-first; any row whose inclusion + // pushes that running total past the cap is older than what we can afford to keep, + // so it goes. + if (_maxSpoolBytes < long.MaxValue) + { + using (var cmd = _connection.CreateCommand()) + { + cmd.CommandText = + "DELETE FROM events WHERE id IN (" + + " SELECT id FROM (" + + " SELECT id, SUM(len) OVER (ORDER BY id DESC) AS running FROM events" + + " ) WHERE running > @maxBytes);"; + cmd.Parameters.AddWithValue("@maxBytes", _maxSpoolBytes); + dropped += cmd.ExecuteNonQuery(); + } + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.EnforceCaps failed: " + e); + } + + return dropped; + } + + /// + /// + /// The lock is taken twice -- once to claim, once to resolve -- and is NOT held across + /// . See the class remarks for why that is the entire point of this + /// implementation. + /// + public async Task ProcessBatchAsync(int maxItems, + Func, CancellationToken, Task> send, + long maxBytes = long.MaxValue, CancellationToken cancellationToken = default) + { + if (send == null || maxItems <= 0 || maxBytes <= 0) + return; + + try + { + var claim = ClaimBatch(maxItems, maxBytes); + if (claim.Ids.Count == 0) + return; + + if (claim.Events.Count == 0) + { + // Everything claimed was undeserializable (corrupt, or an incompatible schema + // version). Nothing to send; just remove them so they cannot wedge the spool. + ResolveClaim(claim.Ids, remove: true); + return; + } + + SendResult result; + try + { + // NO LOCK HELD HERE. Track() stays responsive for the whole round trip. + result = await send(claim.Events, cancellationToken).ConfigureAwait(false); + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.ProcessBatch: send threw, leaving batch for retry: " + e); + ResolveClaim(claim.Ids, remove: false); + return; + } + + // Delivered/PoisonDrop: the batch is finished with either way, so remove it. + // RetryableFailure: release the lease so the next drain picks it up, in order. + ResolveClaim(claim.Ids, + remove: result == SendResult.Delivered || result == SendResult.PoisonDrop, + undeserializableIds: claim.UndeserializableIds); + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.ProcessBatch failed: " + e); + } + } + + private class ClaimedBatch + { + public readonly List Ids = new List(); + public readonly List Events = new List(); + // Claimed rows whose payload could not be deserialized. Removed regardless of the + // batch's verdict -- they can never be delivered. + public readonly List UndeserializableIds = new List(); + } + + // Short, purely-local transaction: pick the oldest unleased rows within both budgets and + // lease them so a concurrent drain (this process or another) will not send them too. + private ClaimedBatch ClaimBatch(int maxItems, long maxBytes) + { + var claim = new ClaimedBatch(); + + lock (_sync) + { + if (_disposed || _connection == null) + return claim; + + var nowMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + var leaseUntilMs = nowMs + (long)s_leaseDuration.TotalMilliseconds; + + // IMMEDIATE: this transaction reads and then writes, and a deferred transaction + // that upgrades can deadlock against another process doing the same (both hold + // SHARED, both want RESERVED) in a way busy_timeout cannot resolve. + using (var txn = _connection.BeginTransaction(deferred: false)) + { + long batchBytes = 0; + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "SELECT id, len, payload FROM events WHERE lease_until_unix_ms <= @now " + + "ORDER BY id ASC LIMIT @maxItems;"; + cmd.Parameters.AddWithValue("@now", nowMs); + cmd.Parameters.AddWithValue("@maxItems", maxItems); + + using (var reader = cmd.ExecuteReader()) + { + while (reader.Read()) + { + // Budget checked BEFORE adding, against the bytes already gathered, + // so the first event always goes even if it alone exceeds the + // budget -- the budget can never starve the queue. + if (batchBytes >= maxBytes) + break; + + var id = reader.GetInt64(0); + var len = reader.GetInt64(1); + var payload = (byte[])reader.GetValue(2); + + claim.Ids.Add(id); + batchBytes += len; + + try + { + claim.Events.Add(AnalyticsEvent.FromBytes(payload)); + } + catch (Exception e) + { + Debug.WriteLine( + "SqliteEventSpool.ClaimBatch: failed to deserialize event, dropping: " + e); + claim.UndeserializableIds.Add(id); + } + } + } + } + + if (claim.Ids.Count > 0) + { + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "UPDATE events SET lease_until_unix_ms = @leaseUntil WHERE id IN (" + + IdList(claim.Ids) + ");"; + cmd.Parameters.AddWithValue("@leaseUntil", leaseUntilMs); + cmd.ExecuteNonQuery(); + } + } + + txn.Commit(); + } + } + + return claim; + } + + // Short, purely-local transaction closing out a claim: either remove the rows (the batch is + // finished with) or clear their lease so the next drain retries them in order. + private void ResolveClaim(List ids, bool remove, List undeserializableIds = null) + { + try + { + lock (_sync) + { + if (_disposed || _connection == null) + return; + + using (var txn = _connection.BeginTransaction(deferred: false)) + { + if (remove) + { + DeleteByIdLocked(txn, ids); + } + else + { + // Undeserializable rows are removed even on a retryable verdict: they + // can never be delivered, so retrying them forever would wedge the + // head of the queue. + if (undeserializableIds != null && undeserializableIds.Count > 0) + DeleteByIdLocked(txn, undeserializableIds); + + var toRelease = new List(ids); + if (undeserializableIds != null) + toRelease.RemoveAll(undeserializableIds.Contains); + + if (toRelease.Count > 0) + { + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "UPDATE events SET lease_until_unix_ms = 0 WHERE id IN (" + + IdList(toRelease) + ");"; + cmd.ExecuteNonQuery(); + } + } + } + + txn.Commit(); + } + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.ResolveClaim failed: " + e); + } + } + + private void DeleteByIdLocked(SqliteTransaction txn, List ids) + { + if (ids.Count == 0) + return; + + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = "DELETE FROM events WHERE id IN (" + IdList(ids) + ");"; + cmd.ExecuteNonQuery(); + } + } + + // Ids are database-generated integers we just read back, never caller input, so inlining + // them cannot be an injection vector -- and it avoids building N parameters per batch. + private static string IdList(List ids) => string.Join(",", ids); + + /// + /// + /// One indexed DELETE. Unlike this does not have to + /// stop at the first non-expired event -- it removes every expired event wherever it sits, + /// so an out-of-order timestamp cannot shield older events from the retention floor. A + /// corrupt payload is irrelevant here too: the age lives in a column, so nothing has to be + /// deserialized to be dated. + /// + public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) + { + try + { + var cutoffMs = (now - maxAge).ToUnixTimeMilliseconds(); + + lock (_sync) + { + if (_disposed || _connection == null) + return; + + using (var cmd = _connection.CreateCommand()) + { + cmd.CommandText = "DELETE FROM events WHERE time_unix_ms < @cutoff;"; + cmd.Parameters.AddWithValue("@cutoff", cutoffMs); + cmd.ExecuteNonQuery(); + } + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.TrimExpired failed: " + e); + } + } + + /// + /// + /// DELETE removes the rows; VACUUM rebuilds the database file so the purged bytes are + /// actually gone rather than lingering as free pages; the WAL checkpoint/truncate does the + /// same for the write-ahead log. All three matter for a CONSENT purge, where "marked + /// consumed" is not good enough. + /// Contrast , which has to dispose the queue, delete + /// the whole directory and reopen -- briefly dropping its cross-process lock, and degrading + /// to a no-op spool if another process steals it in that window. + /// + public void Purge() + { + try + { + lock (_sync) + { + if (_disposed || _connection == null) + return; + + Execute(_connection, "DELETE FROM events;"); + // Must run outside a transaction, and after the DELETE is committed. + Execute(_connection, "VACUUM;"); + Execute(_connection, "PRAGMA wal_checkpoint(TRUNCATE);"); + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.Purge failed: " + e); + } + } + + /// Closes the database. Safe to call more than once. + public void Dispose() + { + try + { + lock (_sync) + { + if (_disposed) + return; + + _disposed = true; + _connection?.Dispose(); + _connection = null; + } + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool.Dispose failed: " + e); + } + } + } +} diff --git a/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj b/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj index e35283b..3135dba 100644 --- a/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj +++ b/src/DesktopAnalyticsTests/DesktopAnalyticsTests.csproj @@ -1,7 +1,7 @@  - net462 + net462;net8.0 false SIL International False @@ -21,7 +21,7 @@ - + \ No newline at end of file diff --git a/src/DesktopAnalyticsTests/EventSpoolContractTests.cs b/src/DesktopAnalyticsTests/EventSpoolContractTests.cs new file mode 100644 index 0000000..4a93a5f --- /dev/null +++ b/src/DesktopAnalyticsTests/EventSpoolContractTests.cs @@ -0,0 +1,589 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + /// + /// The storage-engine-agnostic contract, run against BOTH engines so + /// their behavior is compared rather than argued about. Engine-specific behavior (DiskQueue's + /// corrupt-file recovery and its exclusive cross-process lock; SQLite's WAL concurrency) stays + /// in each engine's own fixture. + /// + /// + /// Overlaps EventSpoolTests by design while both engines are in the tree: this fixture proves + /// they agree, that one does not. Once an engine is chosen, the loser's fixture and this + /// parametrization both go away. + /// + [TestFixture(typeof(DiskQueueSpoolFactory))] + [TestFixture(typeof(SqliteSpoolFactory))] + internal class EventSpoolContractTests where TFactory : ISpoolFactory, new() + { + private string _spoolDir; + private TFactory _factory; + + [SetUp] + public void SetUp() + { + _factory = new TFactory(); + _spoolDir = Path.Combine(Path.GetTempPath(), + "SpoolContract_" + _factory.Name + "_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (!Directory.Exists(_spoolDir)) + return; + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + Console.WriteLine("TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + + private IEventSpool Create(int maxItems, int maxItemBytes = int.MaxValue, + long maxSpoolBytes = long.MaxValue) => + _factory.Create(_spoolDir, maxItems, maxItemBytes, maxSpoolBytes); + + private static AnalyticsEvent MakeEvent(string name) => AnalyticsEvent.Create("user-1", name); + + private static AnalyticsEvent MakeEventAt(string name, DateTimeOffset time) => + AnalyticsEvent.Create("user-1", name, time: time); + + private static async Task> DrainAllDelivered(IEventSpool spool, int maxItems = 100) + { + var names = new List(); + await spool.ProcessBatchAsync(maxItems, (batch, ct) => + { + names.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + }); + return names; + } + + // ---- RESTART DURABILITY ----------------------------------------------------------------- + + [Test] + public async Task Enqueue_ThenDisposeAndReopen_EventsSurviveAndDrainFully() + { + using (var spool = Create(10)) + { + Assert.IsTrue(spool.Enqueue(MakeEvent("A"))); + Assert.IsTrue(spool.Enqueue(MakeEvent("B"))); + } + + using (var reopened = Create(10)) + { + Assert.AreEqual(2, reopened.ApproximateCount, "events must survive a restart"); + CollectionAssert.AreEqual(new[] { "A", "B" }, await DrainAllDelivered(reopened)); + Assert.AreEqual(0, reopened.ApproximateCount); + } + } + + // ---- DELIVERY / ACK --------------------------------------------------------------------- + + [Test] + public async Task ProcessBatch_AllDelivered_EmptiesSpool() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + spool.Enqueue(MakeEvent("B")); + + CollectionAssert.AreEqual(new[] { "A", "B" }, await DrainAllDelivered(spool)); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + [Test] + public async Task ProcessBatch_RetryableFailure_RollsWholeBatchBackForLaterRetry() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + spool.Enqueue(MakeEvent("B")); + + await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.RetryableFailure)); + + Assert.AreEqual(2, spool.ApproximateCount, "a retryable failure must keep the batch"); + CollectionAssert.AreEqual(new[] { "A", "B" }, await DrainAllDelivered(spool), + "the rolled-back batch must still be there, in order"); + } + } + + [Test] + public async Task ProcessBatch_SendThrows_NothingRemovedAndEventsSurviveReopen() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + + await spool.ProcessBatchAsync(10, + (batch, ct) => throw new InvalidOperationException("boom")); + + Assert.AreEqual(1, spool.ApproximateCount, "a throwing send must not lose the batch"); + } + + using (var reopened = Create(10)) + { + Assert.AreEqual(1, reopened.ApproximateCount, "and it must survive a restart"); + CollectionAssert.AreEqual(new[] { "A" }, await DrainAllDelivered(reopened)); + } + } + + [Test] + public async Task ProcessBatch_PoisonDrop_RemovesBatchEvenThoughNotDelivered() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + + await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.PoisonDrop)); + + Assert.AreEqual(0, spool.ApproximateCount, + "poison must be removed so it cannot wedge the spool"); + } + } + + [Test] + public async Task ProcessBatch_MoreEventsThanMaxItems_GathersExactlyMaxItemsPerCall() + { + using (var spool = Create(10)) + { + for (var i = 0; i < 5; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + var first = await DrainAllDelivered(spool, maxItems: 2); + CollectionAssert.AreEqual(new[] { "Event-0", "Event-1" }, first); + Assert.AreEqual(3, spool.ApproximateCount); + } + } + + [Test] + public async Task ProcessBatch_ByteBudget_BoundsBatchButNeverStarves() + { + var a = MakeEvent("Event-A"); + // The budget is checked AFTER each gathered event, against the running total already + // accumulated -- not "would adding the next event exceed it". So the running total must + // already reach the budget once A is in for B to be excluded; a.Length exactly (not +1) + // is what bounds the batch to one event. + var budget = a.ToBytes().Length; + + using (var spool = Create(10)) + { + spool.Enqueue(a); + spool.Enqueue(MakeEvent("Event-B")); + + var first = await DrainAllDelivered2(spool, 10, budget); + CollectionAssert.AreEqual(new[] { "Event-A" }, first, + "the byte budget must bound the batch"); + + var second = await DrainAllDelivered2(spool, 10, budget); + CollectionAssert.AreEqual(new[] { "Event-B" }, second, + "the rest must follow on the next drain"); + } + } + + [Test] + public async Task ProcessBatch_SingleEventLargerThanWholeByteBudget_StillGoes() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("Event-A")); + + var names = await DrainAllDelivered2(spool, 10, maxBytes: 1); + CollectionAssert.AreEqual(new[] { "Event-A" }, names, + "the byte budget must never starve the queue"); + } + } + + private static async Task> DrainAllDelivered2(IEventSpool spool, int maxItems, + long maxBytes) + { + var names = new List(); + await spool.ProcessBatchAsync(maxItems, (batch, ct) => + { + names.AddRange(batch.Select(e => e.EventName)); + return Task.FromResult(SendResult.Delivered); + }, maxBytes); + return names; + } + + [Test] + public async Task ProcessBatch_CancellationToken_IsPassedThroughToSend() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + + var sawToken = false; + using (var cts = new CancellationTokenSource()) + { + await spool.ProcessBatchAsync(10, (batch, ct) => + { + sawToken = ct == cts.Token; + return Task.FromResult(SendResult.Delivered); + }, long.MaxValue, cts.Token); + } + + Assert.IsTrue(sawToken, "the caller's token must reach the send callback"); + } + } + + // ---- ITEM CAP --------------------------------------------------------------------------- + + [Test] + public async Task Enqueue_ExceedingMaxItems_DropsOldestKeepsNewest() + { + using (var spool = Create(3)) + { + foreach (var name in new[] { "A", "B", "C", "D" }) + spool.Enqueue(MakeEvent(name)); + + Assert.AreEqual(3, spool.ApproximateCount); + CollectionAssert.AreEqual(new[] { "B", "C", "D" }, await DrainAllDelivered(spool), + "the cap must drop the OLDEST event"); + } + } + + [Test] + public void Enqueue_MaxItemsZero_ImmediatelyEvictsTheJustEnqueuedEvent() + { + using (var spool = Create(0)) + { + Assert.IsTrue(spool.Enqueue(MakeEvent("A")), + "Enqueue reports the write itself succeeded, independent of cap enforcement"); + Assert.AreEqual(0, spool.ApproximateCount, + "maxItems == 0 must evict even the event that was just enqueued"); + } + } + + [Test] + public async Task Enqueue_ExactlyAtMaxItems_NoneDropped_ThenOneOver_DropsOldestStaysAtCap() + { + const int maxItems = 5; + using (var spool = Create(maxItems)) + { + for (var i = 0; i < maxItems; i++) + spool.Enqueue(MakeEvent("Event-" + i)); + + Assert.AreEqual(maxItems, spool.ApproximateCount, + "exactly maxItems events must all be retained"); + + spool.Enqueue(MakeEvent("OneMore")); + Assert.AreEqual(maxItems, spool.ApproximateCount, "count must stay pinned at the cap"); + + CollectionAssert.AreEqual( + new[] { "Event-1", "Event-2", "Event-3", "Event-4", "OneMore" }, + await DrainAllDelivered(spool, maxItems + 5)); + } + } + + [Test] + public void Enqueue_ExceedingMaxItems_RaisesItemDroppedByCapPerDroppedEvent() + { + using (var spool = Create(2)) + { + var dropped = 0; + spool.ItemDroppedByCap += () => Interlocked.Increment(ref dropped); + + foreach (var name in new[] { "A", "B", "C", "D" }) + spool.Enqueue(MakeEvent(name)); + + Assert.AreEqual(2, dropped, + "two events over the cap must raise the drop notification twice"); + } + } + + // ---- BYTE CAP --------------------------------------------------------------------------- + + [Test] + public async Task Enqueue_ExceedingMaxSpoolBytes_DropsOldestKeepsNewest() + { + var events = new[] + { + MakeEvent("Event-A"), MakeEvent("Event-B"), MakeEvent("Event-C"), MakeEvent("Event-D") + }; + long cap = events[0].ToBytes().Length + events[1].ToBytes().Length + + events[2].ToBytes().Length; + + using (var spool = Create(100, maxSpoolBytes: cap)) + { + foreach (var evt in events) + spool.Enqueue(evt); + + CollectionAssert.AreEqual(new[] { "Event-B", "Event-C", "Event-D" }, + await DrainAllDelivered(spool), "the byte cap must drop the OLDEST event"); + } + } + + [Test] + public void Enqueue_EventLargerThanMaxItemBytes_IsRefusedWhileNormalEventIsAccepted() + { + using (var spool = Create(10, maxItemBytes: 500)) + { + var huge = AnalyticsEvent.Create("user-1", "Huge", new Segment.Serialization.JsonObject + { + { "blob", new string('x', 2000) } + }); + + Assert.IsFalse(spool.Enqueue(huge), "an event over the per-event cap must be refused"); + Assert.IsTrue(spool.Enqueue(MakeEvent("Normal")), "a normal event must still be accepted"); + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + [Test] + public void ApproximateBytes_TracksEnqueuedPayloadSizeAndSurvivesReopen() + { + long expected; + using (var spool = Create(10)) + { + var a = MakeEvent("A"); + expected = a.ToBytes().Length; + spool.Enqueue(a); + Assert.AreEqual(expected, spool.ApproximateBytes); + } + + using (var reopened = Create(10)) + { + Assert.AreEqual(expected, reopened.ApproximateBytes, + "the byte total the cap is enforced against must survive a restart"); + } + } + + // ---- AGE RETENTION ---------------------------------------------------------------------- + + [Test] + public void TrimExpired_EventOlderThanMaxAge_IsDropped() + { + using (var spool = Create(10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Old", now - TimeSpan.FromDays(90))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + [Test] + public void TrimExpired_EventWithinMaxAge_IsRetained() + { + using (var spool = Create(10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Fresh", now - TimeSpan.FromDays(10))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(1, spool.ApproximateCount); + } + } + + [Test] + public async Task TrimExpired_MixOfExpiredAndFreshEvents_DropsExpiredKeepsRestInOrder() + { + using (var spool = Create(10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Expired-1", now - TimeSpan.FromDays(90))); + spool.Enqueue(MakeEventAt("Expired-2", now - TimeSpan.FromDays(70))); + spool.Enqueue(MakeEventAt("Fresh-1", now - TimeSpan.FromDays(10))); + spool.Enqueue(MakeEventAt("Fresh-2", now - TimeSpan.FromDays(1))); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(2, spool.ApproximateCount); + CollectionAssert.AreEqual(new[] { "Fresh-1", "Fresh-2" }, await DrainAllDelivered(spool)); + } + } + + [Test] + public void TrimExpired_EmptySpool_IsNoOpAndDoesNotThrow() + { + using (var spool = Create(10)) + { + Assert.DoesNotThrow(() => spool.TrimExpired(TimeSpan.FromDays(60), DateTimeOffset.UtcNow)); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + [Test] + public void TrimExpired_ClockMovedBackward_DropsNothing() + { + using (var spool = Create(10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("A", now)); + + // "now" is earlier than the event's own stamp -- nothing can be older than the cutoff. + spool.TrimExpired(TimeSpan.FromDays(60), now - TimeSpan.FromDays(30)); + + Assert.AreEqual(1, spool.ApproximateCount, + "a backward clock jump must not evict events"); + } + } + + // ---- CONSENT PURGE ---------------------------------------------------------------------- + + [Test] + public void Purge_EmptiesNonEmptySpool() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + spool.Enqueue(MakeEvent("B")); + + spool.Purge(); + + Assert.AreEqual(0, spool.ApproximateCount); + Assert.AreEqual(0, spool.ApproximateBytes); + } + } + + [Test] + public async Task Purge_ThenEnqueue_SpoolRemainsUsableAndPurgedBytesAreGoneFromDisk() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("SecretEventName")); + spool.Purge(); + + Assert.IsTrue(spool.Enqueue(MakeEvent("AfterPurge")), + "the spool must remain usable after a purge"); + CollectionAssert.AreEqual(new[] { "AfterPurge" }, await DrainAllDelivered(spool)); + } + + // The consent guarantee: the purged event's bytes must be gone from disk, not merely + // marked consumed. + foreach (var file in Directory.GetFiles(_spoolDir, "*", SearchOption.AllDirectories)) + { + string content; + try + { + using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete)) + using (var reader = new StreamReader(stream, System.Text.Encoding.UTF8)) + content = reader.ReadToEnd(); + } + catch (IOException) + { + continue; + } + + StringAssert.DoesNotContain("SecretEventName", content, + "purged event data must not survive anywhere on disk in " + file); + } + } + + [Test] + public void Purge_EmptySpool_DoesNotThrow() + { + using (var spool = Create(10)) + { + Assert.DoesNotThrow(() => spool.Purge()); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + // ---- ROBUSTNESS ------------------------------------------------------------------------- + + [Test] + public void Enqueue_NullEvent_ReturnsFalseAndDoesNotThrow() + { + using (var spool = Create(10)) + { + Assert.IsFalse(spool.Enqueue(null)); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + + [Test] + public void Enqueue_FromMultipleThreadsConcurrently_AllEventsPersisted() + { + const int threadCount = 4; + const int perThread = 25; + + using (var spool = Create(threadCount * perThread)) + { + var threads = new Thread[threadCount]; + for (var t = 0; t < threadCount; t++) + { + var index = t; + threads[t] = new Thread(() => + { + for (var i = 0; i < perThread; i++) + spool.Enqueue(MakeEvent($"T{index}-{i}")); + }); + } + + foreach (var thread in threads) + thread.Start(); + foreach (var thread in threads) + thread.Join(); + + Assert.AreEqual(threadCount * perThread, spool.ApproximateCount, + "concurrent enqueues must not lose events"); + } + } + + [Test] + public async Task ProcessBatch_EmptySpool_IsNoOpAndNeverCallsSend() + { + using (var spool = Create(10)) + { + var called = false; + await spool.ProcessBatchAsync(10, (batch, ct) => + { + called = true; + return Task.FromResult(SendResult.Delivered); + }); + + Assert.IsFalse(called, "send must not be called with an empty batch"); + } + } + + [Test] + public void Dispose_CalledTwice_DoesNotThrow() + { + var spool = Create(10); + spool.Dispose(); + Assert.DoesNotThrow(() => spool.Dispose()); + } + } + + // ---- ENGINE FACTORIES ----------------------------------------------------------------------- + + internal interface ISpoolFactory + { + string Name { get; } + IEventSpool Create(string dir, int maxItems, int maxItemBytes, long maxSpoolBytes); + } + + internal class DiskQueueSpoolFactory : ISpoolFactory + { + public string Name => "DiskQueue"; + + public IEventSpool Create(string dir, int maxItems, int maxItemBytes, long maxSpoolBytes) => + new EventSpool(dir, maxItems, maxItemBytes, maxSpoolBytes); + } + + internal class SqliteSpoolFactory : ISpoolFactory + { + public string Name => "Sqlite"; + + public IEventSpool Create(string dir, int maxItems, int maxItemBytes, long maxSpoolBytes) => + new SqliteEventSpool(dir, maxItems, maxItemBytes, maxSpoolBytes); + } +} diff --git a/src/DesktopAnalyticsTests/JsonAnalyticsSettingsStoreTests.cs b/src/DesktopAnalyticsTests/JsonAnalyticsSettingsStoreTests.cs new file mode 100644 index 0000000..dedb66a --- /dev/null +++ b/src/DesktopAnalyticsTests/JsonAnalyticsSettingsStoreTests.cs @@ -0,0 +1,114 @@ +// License: MIT + +// JsonAnalyticsSettingsStore only exists on non-net462 TFMs (see +// src/DesktopAnalytics/JsonAnalyticsSettingsStore.cs), so this whole file compiles to nothing on +// net462. +#if !NET462 +using System; +using System.IO; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + [TestFixture] + public class JsonAnalyticsSettingsStoreTests + { + private string _tempRoot; + + [SetUp] + public void SetUp() + { + _tempRoot = Path.Combine(Path.GetTempPath(), "DesktopAnalyticsTests_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (Directory.Exists(_tempRoot)) + Directory.Delete(_tempRoot, recursive: true); + } + + [Test] + public void DefaultsWhenAbsent_FreshDirectory_MatchesDesignerDefaults() + { + var store = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + + Assert.That(store.NeedUpgrade, Is.True); + Assert.That(store.IdForAnalytics, Is.EqualTo("")); + Assert.That(store.LastVersionLaunched, Is.EqualTo("")); + Assert.That(store.FirstName, Is.EqualTo("")); + Assert.That(store.LastName, Is.EqualTo("")); + Assert.That(store.Email, Is.EqualTo("")); + } + + [Test] + public void RoundTrip_SetPropertiesAndSave_FreshInstanceReadsSameValues() + { + var first = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + first.IdForAnalytics = "abc-123"; + first.LastVersionLaunched = "1.2.3.4"; + first.NeedUpgrade = false; + first.FirstName = "Bob"; + first.LastName = "Smith"; + first.Email = "test@example.com"; + first.Save(); + + var second = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + + Assert.That(second.IdForAnalytics, Is.EqualTo("abc-123")); + Assert.That(second.LastVersionLaunched, Is.EqualTo("1.2.3.4")); + Assert.That(second.NeedUpgrade, Is.False); + Assert.That(second.FirstName, Is.EqualTo("Bob")); + Assert.That(second.LastName, Is.EqualTo("Smith")); + Assert.That(second.Email, Is.EqualTo("test@example.com")); + } + + [Test] + public void Save_DirectoryDoesNotExist_CreatesItAndWritesFile() + { + Assert.That(Directory.Exists(_tempRoot), Is.False); + + var store = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + store.IdForAnalytics = "new-id"; + store.Save(); + + Assert.That(File.Exists(Path.Combine(_tempRoot, "settings.json")), Is.True); + } + + [Test] + public void RestartStability_MultipleSequentialInstancesLikeProcessRestarts_ValuesPersist() + { + // Simulates: launch 1 assigns an ID and saves; launch 2 (a fresh process/instance) + // reads it back and updates LastVersionLaunched; launch 3 sees both. + var launch1 = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + Assert.That(launch1.IdForAnalytics, Is.EqualTo("")); + launch1.IdForAnalytics = "stable-id"; + launch1.LastVersionLaunched = "1.0.0.0"; + launch1.Save(); + + var launch2 = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + Assert.That(launch2.IdForAnalytics, Is.EqualTo("stable-id")); + Assert.That(launch2.LastVersionLaunched, Is.EqualTo("1.0.0.0")); + launch2.LastVersionLaunched = "2.0.0.0"; + launch2.Save(); + + var launch3 = JsonAnalyticsSettingsStore.ForTesting(_tempRoot); + Assert.That(launch3.IdForAnalytics, Is.EqualTo("stable-id")); + Assert.That(launch3.LastVersionLaunched, Is.EqualTo("2.0.0.0")); + } + + [Test] + public void Load_CorruptFile_FallsBackToDefaultsInsteadOfThrowing() + { + Directory.CreateDirectory(_tempRoot); + File.WriteAllText(Path.Combine(_tempRoot, "settings.json"), "{ not valid json"); + + JsonAnalyticsSettingsStore store = null; + Assert.DoesNotThrow(() => store = JsonAnalyticsSettingsStore.ForTesting(_tempRoot)); + Assert.That(store.NeedUpgrade, Is.True); + Assert.That(store.IdForAnalytics, Is.EqualTo("")); + } + } +} +#endif diff --git a/src/DesktopAnalyticsTests/TrackResponsivenessTests.cs b/src/DesktopAnalyticsTests/TrackResponsivenessTests.cs new file mode 100644 index 0000000..49089b6 --- /dev/null +++ b/src/DesktopAnalyticsTests/TrackResponsivenessTests.cs @@ -0,0 +1,180 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + // EventSpool (DiskQueue) holds its global _sync semaphore across the network send, so a + // UI-thread Track() call blocks for the WHOLE send -- up to ~45s against a stalled connection + // in production. SqliteEventSpool claims a batch under a short lock, releases it, sends, then + // resolves the claim under a second short lock, so Track() stays responsive throughout. + // NB: the stall is EventSpool's lock, not a DiskQueue limitation -- DiskQueue supports + // concurrent sessions (probed 2026-07-16), so dropping that lock would also fix it. See the + // class remarks on SqliteEventSpool. + [TestFixture] + public class TrackResponsivenessTests + { + private string _spoolDir; + + [SetUp] + public void SetUp() + { + _spoolDir = Path.Combine(Path.GetTempPath(), "TrackResponsivenessTests_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (Directory.Exists(_spoolDir)) + { + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + Console.WriteLine("TrackResponsivenessTests.TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + } + + // Blocks inside SendBatchAsync until the test explicitly releases it (via a + // TaskCompletionSource, never a real delay), so a test can hold a send genuinely "in + // flight" for as long as it likes and then let it go on cue. Same shape as + // MixpanelClientTests.BlockingUntilSignaledSender; duplicated here (rather than shared) + // because that one is private to its fixture. + private class BlockingUntilSignaledSender : IEventSender + { + private readonly TaskCompletionSource _release = + new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + public int CallCount; + + public void Release() => _release.TrySetResult(true); + + public async Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref CallCount); + await _release.Task.ConfigureAwait(false); + return new BatchSendResult(SendResult.Delivered); + } + } + + // Polls sender.CallCount (never a bare Thread.Sleep as the synchronization) until the + // drain's send has genuinely started, so a test's timed Track() call races a send that is + // truly in flight rather than one that hasn't begun yet. + private static async Task WaitForSendToStart(BlockingUntilSignaledSender sender) + { + var deadline = DateTime.UtcNow.AddSeconds(5); + while (sender.CallCount == 0 && DateTime.UtcNow < deadline) + await Task.Delay(10); + Assert.AreEqual(1, sender.CallCount, "the drain's send must have started before timing Track()"); + } + + // ---- THE REQUIREMENT: SqliteEventSpool keeps Track() responsive ------------------------- + + [Test] + public async Task Track_WhileSendInFlight_ReturnsImmediately_Sqlite() + { + using (var spool = new SqliteEventSpool(_spoolDir, 1000)) + { + var sender = new BlockingUntilSignaledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, batchSize: 5); + + client.Track("user-1", "Seed", null); + + var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). + await WaitForSendToStart(sender); + + var stopwatch = Stopwatch.StartNew(); + client.Track("user-1", "WhileSendInFlight", null); + stopwatch.Stop(); + + // The send is blocked indefinitely until Release() below is called, so there is no + // way for a correct (lock-not-held-across-send) implementation to take anywhere near + // this long by accident. 500ms is generous headroom above the expected + // single-digit-millisecond reality while still being unambiguous against the + // multi-second-to-45s stalls this design exists to eliminate. + Assert.Less(stopwatch.Elapsed, TimeSpan.FromMilliseconds(500), + "Track() must return promptly even while a send is genuinely in flight -- " + + "SqliteEventSpool never holds its lock across the network await"); + + sender.Release(); + await drainTask; + + // Responsiveness must not come from silently dropping the event: it must have been + // spooled (and, once the drain completes, delivered). + Assert.AreEqual(1, client.Statistics.Succeeded, + "the seed event must have been delivered by the drain this test released"); + await client.DrainOnceAsync(); + Assert.AreEqual(0, spool.ApproximateCount, + "the Track() call made while the send was in flight must have been spooled, " + + "not dropped, and must drain cleanly afterward"); + Assert.AreEqual(2, client.Statistics.Submitted); + } + } + + // ---- THE DEFECT (characterization, not a requirement): EventSpool blocks --------------- + // + // This test PASSES against today's EventSpool behavior on purpose -- it pins the documented + // defect (see MixpanelClientTests.ConcurrentTrack_WhileDrainInProgress_..., whose comments + // describe Track() threads "piling up" behind the lock) so it is visible in the test suite + // rather than hidden in a comment. Delete it if EventSpool is dropped -- or if EventSpool's + // global lock is narrowed so it stops blocking, in which case this test should start failing + // and be deleted rather than "fixed". + [Test] + public async Task Track_WhileSendInFlight_BlocksUntilSendCompletes_DiskQueue() + { + using (var spool = new EventSpool(_spoolDir, 1000)) + { + var sender = new BlockingUntilSignaledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, batchSize: 5); + + client.Track("user-1", "Seed", null); + + var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). + await WaitForSendToStart(sender); + + // Run Track() on its own thread so this test can observe it NOT finishing while the + // send is blocked, rather than deadlocking the test itself. + var trackFinished = new ManualResetEventSlim(false); + var trackThread = new Thread(() => + { + client.Track("user-1", "WhileSendInFlight", null); + trackFinished.Set(); + }); + trackThread.Start(); + + // EventSpool's transactional session holds its lock across the whole send, so + // Track() must still be blocked after a short window -- this is the defect, pinned. + var finishedWhileBlocked = trackFinished.Wait(TimeSpan.FromMilliseconds(300)); + Assert.IsFalse(finishedWhileBlocked, + "characterizes today's known defect: EventSpool holds its lock across the " + + "network send, so Track() blocks for the whole send instead of returning " + + "promptly. If this ever fails, EventSpool has stopped blocking Track() and this " + + "test (and the bug it documents) should be deleted, not fixed."); + + // Release the blocked send so the Track() thread (and the drain) can finally + // complete -- otherwise this test would hang forever. + sender.Release(); + + Assert.IsTrue(trackFinished.Wait(TimeSpan.FromSeconds(5)), + "Track() must complete once the send is released"); + trackThread.Join(); + await drainTask; + + Assert.AreEqual(1, client.Statistics.Succeeded); + await client.DrainOnceAsync(); + Assert.AreEqual(0, spool.ApproximateCount); + Assert.AreEqual(2, client.Statistics.Submitted); + } + } + } +} From 502fddda937a40e43a44c4f853c5784bd59af075 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 19:57:45 -0400 Subject: [PATCH 05/11] Split into SIL.DesktopAnalytics + SIL.DesktopAnalytics.Mixpanel (API break) Moves MixpanelClient, the event spool (both engines), IEventSender/ MixpanelEventSender, AnalyticsEvent and PathScrubber into a new DesktopAnalytics.Mixpanel project, taking DiskQueue/Microsoft.Data.Sqlite/ Polly/mixpanel-csharp with them. Core no longer references any of those packages or MixpanelClient at compile time -- verified via `dotnet list package --include-transitive` across all three TFMs. Replaces the ClientType enum + switch with IClient injection: callers now pass an IClient instance (defaulting to SegmentClient) instead of an enum, so a Mixpanel consumer references the new package explicitly rather than core knowing that type exists. This is the structural guarantee for the five Segment-only consumers (Bloom, HearThis, SayMore, Glyssen, Transcelerator): they can no longer inherit Mixpanel's dependency graph even by accident. Major version bump / API break per plan decision D4 (approved). All 6 projects build clean across net462/netstandard2.0/net8.0; 350 tests total (172 core+mixpanel on net462, 177 on net8.0) all green. Co-Authored-By: Claude Opus 4.8 --- DesktopAnalytics.sln | 52 +++++++++++++++++++ .../AnalyticsEvent.cs | 0 .../DesktopAnalytics.Mixpanel.csproj | 41 +++++++++++++++ .../EventSpool.cs | 0 .../IEventSender.cs | 0 .../IEventSpool.cs | 0 .../MixpanelClient.cs | 2 +- .../MixpanelEventSender.cs | 0 .../PathScrubber.cs | 0 .../SqliteEventSpool.cs | 0 src/DesktopAnalytics/Analytics.cs | 38 +++++--------- src/DesktopAnalytics/ClientType.cs | 8 --- src/DesktopAnalytics/DesktopAnalytics.csproj | 10 ++-- src/DesktopAnalytics/IClient.cs | 19 ++++--- src/DesktopAnalytics/SegmentClient.cs | 7 ++- .../AnalyticsEventTests.cs | 0 .../DesktopAnalyticsTests.Mixpanel.csproj | 28 ++++++++++ .../EventSpoolContractTests.cs | 0 .../EventSpoolTests.cs | 0 .../MixpanelClientTests.cs | 0 .../MixpanelEventSenderTests.cs | 0 .../PathScrubberTests.cs | 0 .../TrackResponsivenessTests.cs | 0 src/DesktopAnalyticsTests.Mixpanel/app.config | 11 ++++ src/SampleApp/Program.cs | 9 +++- src/SampleApp/SampleApp.csproj | 1 + src/SampleAppWithForm/Program.cs | 9 +++- .../SampleAppWithForm.csproj | 1 + 28 files changed, 186 insertions(+), 50 deletions(-) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/AnalyticsEvent.cs (100%) create mode 100644 src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/EventSpool.cs (100%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/IEventSender.cs (100%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/IEventSpool.cs (100%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/MixpanelClient.cs (99%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/MixpanelEventSender.cs (100%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/PathScrubber.cs (100%) rename src/{DesktopAnalytics => DesktopAnalytics.Mixpanel}/SqliteEventSpool.cs (100%) delete mode 100644 src/DesktopAnalytics/ClientType.cs rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/AnalyticsEventTests.cs (100%) create mode 100644 src/DesktopAnalyticsTests.Mixpanel/DesktopAnalyticsTests.Mixpanel.csproj rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/EventSpoolContractTests.cs (100%) rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/EventSpoolTests.cs (100%) rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/MixpanelClientTests.cs (100%) rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/MixpanelEventSenderTests.cs (100%) rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/PathScrubberTests.cs (100%) rename src/{DesktopAnalyticsTests => DesktopAnalyticsTests.Mixpanel}/TrackResponsivenessTests.cs (100%) create mode 100644 src/DesktopAnalyticsTests.Mixpanel/app.config diff --git a/DesktopAnalytics.sln b/DesktopAnalytics.sln index dae6ca4..fd36011 100644 --- a/DesktopAnalytics.sln +++ b/DesktopAnalytics.sln @@ -25,48 +25,100 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DesktopAnalyticsTests", "sr EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SampleAppWithForm", "src\SampleAppWithForm\SampleAppWithForm.csproj", "{471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}" EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{827E0CD3-B72D-47B6-A68D-7590B98EB39B}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DesktopAnalytics.Mixpanel", "src\DesktopAnalytics.Mixpanel\DesktopAnalytics.Mixpanel.csproj", "{B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DesktopAnalyticsTests.Mixpanel", "src\DesktopAnalyticsTests.Mixpanel\DesktopAnalyticsTests.Mixpanel.csproj", "{EA4B1983-946B-4101-9CD3-155652F40221}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU Debug|x86 = Debug|x86 + Debug|x64 = Debug|x64 Release|Any CPU = Release|Any CPU Release|x86 = Release|x86 + Release|x64 = Release|x64 EndGlobalSection GlobalSection(ProjectConfigurationPlatforms) = postSolution {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|Any CPU.Build.0 = Debug|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|x86.ActiveCfg = Debug|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|x86.Build.0 = Debug|Any CPU + {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|x64.ActiveCfg = Debug|Any CPU + {BECACF91-9A84-4667-B678-44ACA350D413}.Debug|x64.Build.0 = Debug|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Release|Any CPU.ActiveCfg = Release|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Release|Any CPU.Build.0 = Release|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Release|x86.ActiveCfg = Release|Any CPU {BECACF91-9A84-4667-B678-44ACA350D413}.Release|x86.Build.0 = Release|Any CPU + {BECACF91-9A84-4667-B678-44ACA350D413}.Release|x64.ActiveCfg = Release|Any CPU + {BECACF91-9A84-4667-B678-44ACA350D413}.Release|x64.Build.0 = Release|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Debug|Any CPU.Build.0 = Debug|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Debug|x86.ActiveCfg = Debug|Any CPU + {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Debug|x64.ActiveCfg = Debug|Any CPU + {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Debug|x64.Build.0 = Debug|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Release|Any CPU.ActiveCfg = Release|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Release|Any CPU.Build.0 = Release|Any CPU {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Release|x86.ActiveCfg = Release|Any CPU + {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Release|x64.ActiveCfg = Release|Any CPU + {D7C7BA95-3ED3-4BD0-9067-E1399A3098A7}.Release|x64.Build.0 = Release|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|Any CPU.Build.0 = Debug|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|x86.ActiveCfg = Debug|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|x86.Build.0 = Debug|Any CPU + {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|x64.ActiveCfg = Debug|Any CPU + {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Debug|x64.Build.0 = Debug|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|Any CPU.ActiveCfg = Release|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|Any CPU.Build.0 = Release|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|x86.ActiveCfg = Release|Any CPU {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|x86.Build.0 = Release|Any CPU + {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|x64.ActiveCfg = Release|Any CPU + {652BCDED-1C97-4BAC-8292-3A3EF605DC59}.Release|x64.Build.0 = Release|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|Any CPU.Build.0 = Debug|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|x86.ActiveCfg = Debug|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|x86.Build.0 = Debug|Any CPU + {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|x64.ActiveCfg = Debug|Any CPU + {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Debug|x64.Build.0 = Debug|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|Any CPU.ActiveCfg = Release|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|Any CPU.Build.0 = Release|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|x86.ActiveCfg = Release|Any CPU {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|x86.Build.0 = Release|Any CPU + {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|x64.ActiveCfg = Release|Any CPU + {471C3C0A-EA78-4D2A-AC47-2129A3A91C6D}.Release|x64.Build.0 = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|Any CPU.Build.0 = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|x86.ActiveCfg = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|x86.Build.0 = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|x64.ActiveCfg = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Debug|x64.Build.0 = Debug|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|Any CPU.ActiveCfg = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|Any CPU.Build.0 = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|x86.ActiveCfg = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|x86.Build.0 = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|x64.ActiveCfg = Release|Any CPU + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1}.Release|x64.Build.0 = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|Any CPU.Build.0 = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|x86.ActiveCfg = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|x86.Build.0 = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|x64.ActiveCfg = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Debug|x64.Build.0 = Debug|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|Any CPU.ActiveCfg = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|Any CPU.Build.0 = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|x86.ActiveCfg = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|x86.Build.0 = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|x64.ActiveCfg = Release|Any CPU + {EA4B1983-946B-4101-9CD3-155652F40221}.Release|x64.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE EndGlobalSection + GlobalSection(NestedProjects) = preSolution + {B22C2B41-0542-4B4D-8EA5-F52D5AA7FCD1} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + {EA4B1983-946B-4101-9CD3-155652F40221} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {D42E95AB-85F0-4D4C-8E83-FCD3D4E4D449} EndGlobalSection diff --git a/src/DesktopAnalytics/AnalyticsEvent.cs b/src/DesktopAnalytics.Mixpanel/AnalyticsEvent.cs similarity index 100% rename from src/DesktopAnalytics/AnalyticsEvent.cs rename to src/DesktopAnalytics.Mixpanel/AnalyticsEvent.cs diff --git a/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj b/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj new file mode 100644 index 0000000..dcedeea --- /dev/null +++ b/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj @@ -0,0 +1,41 @@ + + + + net462;netstandard2.0;net8.0 + SIL.DesktopAnalytics.Mixpanel + John Hatton, Stephen McConnel, Tom Bogle, Eberhard Beilharz, John Thomson, Andrew Polk, Jason Naylor + SIL International + MIT + https://github.com/sillsdev/desktopanalytics.net + https://github.com/sillsdev/desktopanalytics.net + Durable Mixpanel client (disk-backed event spool) for SIL.DesktopAnalytics. + + + true + snupkg + False + ..\..\DesktopAnalytics.snk + + + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers + + + + + + + + <_Parameter1>DesktopAnalyticsTests.Mixpanel + + + diff --git a/src/DesktopAnalytics/EventSpool.cs b/src/DesktopAnalytics.Mixpanel/EventSpool.cs similarity index 100% rename from src/DesktopAnalytics/EventSpool.cs rename to src/DesktopAnalytics.Mixpanel/EventSpool.cs diff --git a/src/DesktopAnalytics/IEventSender.cs b/src/DesktopAnalytics.Mixpanel/IEventSender.cs similarity index 100% rename from src/DesktopAnalytics/IEventSender.cs rename to src/DesktopAnalytics.Mixpanel/IEventSender.cs diff --git a/src/DesktopAnalytics/IEventSpool.cs b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs similarity index 100% rename from src/DesktopAnalytics/IEventSpool.cs rename to src/DesktopAnalytics.Mixpanel/IEventSpool.cs diff --git a/src/DesktopAnalytics/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs similarity index 99% rename from src/DesktopAnalytics/MixpanelClient.cs rename to src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index ca7be42..adf467f 100644 --- a/src/DesktopAnalytics/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -25,7 +25,7 @@ namespace DesktopAnalytics /// escape to the caller either; it just leaves this client running with no spool (Track/Drain /// become no-ops) rather than durable. /// - internal class MixpanelClient : IClient + public class MixpanelClient : IClient { // Open questions in offline-analytics.md: exact cap/batch/cadence values are not locked yet. // These are reasonable starting defaults, not requirements baked in elsewhere. diff --git a/src/DesktopAnalytics/MixpanelEventSender.cs b/src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs similarity index 100% rename from src/DesktopAnalytics/MixpanelEventSender.cs rename to src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs diff --git a/src/DesktopAnalytics/PathScrubber.cs b/src/DesktopAnalytics.Mixpanel/PathScrubber.cs similarity index 100% rename from src/DesktopAnalytics/PathScrubber.cs rename to src/DesktopAnalytics.Mixpanel/PathScrubber.cs diff --git a/src/DesktopAnalytics/SqliteEventSpool.cs b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs similarity index 100% rename from src/DesktopAnalytics/SqliteEventSpool.cs rename to src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index a507fd8..dc866d4 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -86,7 +86,9 @@ private class InitializationParameters /// If false, this will not do any communication with segment.io /// If false, userInfo will be stripped/hashed/adjusted to prevent /// communication of personally identifiable information to the analytics server. - /// + /// The to use, e.g. a + /// or a Mixpanel client from the separate SIL.DesktopAnalytics.Mixpanel package. If null, + /// defaults to a new . /// The url of the host to send analytics to. Will use the client's /// default if not provided. Throws an ArgumentException if the client does not support /// setting the host. @@ -100,7 +102,7 @@ public Analytics( UserInfo userInfo, bool allowTracking = true, bool retainPii = false, - ClientType clientType = ClientType.Segment, + IClient client = null, string host = null, bool useCallingAssemblyVersion = false) : this( apiSecret, @@ -108,7 +110,7 @@ public Analytics( new Dictionary(), allowTracking, retainPii, - clientType, + client, host, assemblyToUseForVersion: useCallingAssemblyVersion ? GetCallingAssembly() : null ) @@ -148,7 +150,9 @@ private void UpdateServerInformationOnThisUser(bool initializing = false) /// If false, prevents communication with segment.io /// If false, userInfo will be stripped/hashed/adjusted to prevent /// communication of personally identifiable information to the analytics server. - /// + /// The to use, e.g. a + /// or a Mixpanel client from the separate SIL.DesktopAnalytics.Mixpanel package. If null, + /// defaults to a new . /// The url of the host to send analytics to. Will use the client's /// default if not provided. Throws an ArgumentException if the client does not support /// setting the host. @@ -166,7 +170,7 @@ public Analytics( Dictionary propertiesThatGoWithEveryEvent, bool allowTracking = true, bool retainPii = false, - ClientType clientType = ClientType.Segment, + IClient client = null, string host = null, int flushAt = -1, int flushInterval = -1, @@ -179,25 +183,11 @@ public Analytics( } s_singleton = this; - switch (clientType) - { - case ClientType.Segment: - { - var segmentClient = new SegmentClient(); - segmentClient.Failed += Client_Failed; - _client = segmentClient; - break; - } - case ClientType.Mixpanel: - { - _client = new MixpanelClient(); - break; - } - default: - { - throw new ArgumentException("Unknown client type", nameof(clientType)); - } - } + if (client == null) + client = new SegmentClient(); + if (client is SegmentClient segmentClient) + segmentClient.Failed += Client_Failed; + _client = client; _propertiesThatGoWithEveryEvent = propertiesThatGoWithEveryEvent; s_userInfo = retainPii ? userInfo : userInfo.CreateSanitized(); diff --git a/src/DesktopAnalytics/ClientType.cs b/src/DesktopAnalytics/ClientType.cs deleted file mode 100644 index 7b1a89a..0000000 --- a/src/DesktopAnalytics/ClientType.cs +++ /dev/null @@ -1,8 +0,0 @@ -namespace DesktopAnalytics -{ - public enum ClientType - { - Segment, - Mixpanel - } -} \ No newline at end of file diff --git a/src/DesktopAnalytics/DesktopAnalytics.csproj b/src/DesktopAnalytics/DesktopAnalytics.csproj index c34fa1c..fa26273 100644 --- a/src/DesktopAnalytics/DesktopAnalytics.csproj +++ b/src/DesktopAnalytics/DesktopAnalytics.csproj @@ -17,23 +17,21 @@ ..\..\DesktopAnalytics.snk - - All - + - - all diff --git a/src/DesktopAnalytics/IClient.cs b/src/DesktopAnalytics/IClient.cs index f05762d..5a53ccc 100644 --- a/src/DesktopAnalytics/IClient.cs +++ b/src/DesktopAnalytics/IClient.cs @@ -4,7 +4,14 @@ namespace DesktopAnalytics { - internal interface IClient + /// + /// The seam between and a concrete analytics transport. Core ships + /// ; a Mixpanel implementation (MixpanelClient) is provided by + /// the separate SIL.DesktopAnalytics.Mixpanel package -- core has no compile-time + /// reference to it. Construct whichever client you want and pass it to one of the + /// constructors. + /// + public interface IClient { void Initialize(string apiSecret, string host = null, int flushAt = -1, int flushInterval = -1); void ShutDown(); @@ -15,7 +22,7 @@ internal interface IClient /// /// Async counterpart of . Implementations whose shutdown has nothing /// awaitable (see ) may complete synchronously; implementations - /// with real async delivery work (see ) must never block a + /// with real async delivery work (e.g. MixpanelClient) must never block a /// thread waiting on it. Like every member here, must not throw -- including on /// cancellation, which just ends any in-flight delivery early (events stay queued). /// @@ -31,8 +38,8 @@ internal interface IClient /// /// Called when the user revokes tracking consent ( - /// transitioning to false). Implementations that spool events on disk (see - /// ) must purge that spool immediately; implementations without a + /// transitioning to false). Implementations that spool events on disk (e.g. + /// MixpanelClient) must purge that spool immediately; implementations without a /// local spool (see ) can no-op. /// void PurgeQueuedEvents(); @@ -40,8 +47,8 @@ internal interface IClient /// /// Called when the user grants tracking consent again ( /// transitioning back to true) on an already-initialized client, after a prior - /// paused it. Implementations with a background flush loop (see - /// ) must re-arm it; implementations without one (see + /// paused it. Implementations with a background flush loop + /// (e.g. MixpanelClient) must re-arm it; implementations without one (see /// ) can no-op. /// void ResumeSending(); diff --git a/src/DesktopAnalytics/SegmentClient.cs b/src/DesktopAnalytics/SegmentClient.cs index a78c319..5d56d3a 100644 --- a/src/DesktopAnalytics/SegmentClient.cs +++ b/src/DesktopAnalytics/SegmentClient.cs @@ -7,7 +7,12 @@ namespace DesktopAnalytics { - internal class SegmentClient : IClient, ICoroutineExceptionHandler + /// + /// implementation backed by Segment.Analytics.CSharp. Constructible + /// directly by callers who want to pass it explicitly to an + /// constructor; also the implicit default when no is supplied. + /// + public class SegmentClient : IClient, ICoroutineExceptionHandler { public event Action Failed; private Segment.Analytics.Analytics _analytics; diff --git a/src/DesktopAnalyticsTests/AnalyticsEventTests.cs b/src/DesktopAnalyticsTests.Mixpanel/AnalyticsEventTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/AnalyticsEventTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/AnalyticsEventTests.cs diff --git a/src/DesktopAnalyticsTests.Mixpanel/DesktopAnalyticsTests.Mixpanel.csproj b/src/DesktopAnalyticsTests.Mixpanel/DesktopAnalyticsTests.Mixpanel.csproj new file mode 100644 index 0000000..872e852 --- /dev/null +++ b/src/DesktopAnalyticsTests.Mixpanel/DesktopAnalyticsTests.Mixpanel.csproj @@ -0,0 +1,28 @@ + + + + net462;net8.0 + false + SIL International + False + + + true + + + + + + + + + + + + + + + + + diff --git a/src/DesktopAnalyticsTests/EventSpoolContractTests.cs b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/EventSpoolContractTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs diff --git a/src/DesktopAnalyticsTests/EventSpoolTests.cs b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/EventSpoolTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs diff --git a/src/DesktopAnalyticsTests/MixpanelClientTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/MixpanelClientTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs diff --git a/src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/MixpanelEventSenderTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs diff --git a/src/DesktopAnalyticsTests/PathScrubberTests.cs b/src/DesktopAnalyticsTests.Mixpanel/PathScrubberTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/PathScrubberTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/PathScrubberTests.cs diff --git a/src/DesktopAnalyticsTests/TrackResponsivenessTests.cs b/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs similarity index 100% rename from src/DesktopAnalyticsTests/TrackResponsivenessTests.cs rename to src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs diff --git a/src/DesktopAnalyticsTests.Mixpanel/app.config b/src/DesktopAnalyticsTests.Mixpanel/app.config new file mode 100644 index 0000000..a0cefc2 --- /dev/null +++ b/src/DesktopAnalyticsTests.Mixpanel/app.config @@ -0,0 +1,11 @@ + + + + + + + + + + + \ No newline at end of file diff --git a/src/SampleApp/Program.cs b/src/SampleApp/Program.cs index 901255a..7edd2c8 100644 --- a/src/SampleApp/Program.cs +++ b/src/SampleApp/Program.cs @@ -19,7 +19,12 @@ static int Main(string[] args) return 1; } - if (!Enum.TryParse(args[1], true, out var clientType)) + IClient client; + if (args[1].Equals("Segment", StringComparison.OrdinalIgnoreCase)) + client = new SegmentClient(); + else if (args[1].Equals("Mixpanel", StringComparison.OrdinalIgnoreCase)) + client = new MixpanelClient(); + else { Console.WriteLine($"{usage}{Environment.NewLine}Unrecognized client type: {args[1]}"); return 1; @@ -79,7 +84,7 @@ static int Main(string[] args) "This is a really long explanation of how I use this product to see how much you would be able to extract from Mixpanel.\r\nAnd a second line of it."); var propsForEveryEvent = new Dictionary {{"channel", "beta"}}; - using (new Analytics(args[0], userInfo, propsForEveryEvent, initialTracking, clientType: clientType)) + using (new Analytics(args[0], userInfo, propsForEveryEvent, initialTracking, client: client)) { Thread.Sleep(3000); //note that anything we set from here on didn't make it into the initial "Launch" event. Things we want to diff --git a/src/SampleApp/SampleApp.csproj b/src/SampleApp/SampleApp.csproj index a54f878..fe969ea 100644 --- a/src/SampleApp/SampleApp.csproj +++ b/src/SampleApp/SampleApp.csproj @@ -19,6 +19,7 @@ {BECACF91-9A84-4667-B678-44ACA350D413} DesktopAnalytics + diff --git a/src/SampleAppWithForm/Program.cs b/src/SampleAppWithForm/Program.cs index f6d598e..1552dc2 100644 --- a/src/SampleAppWithForm/Program.cs +++ b/src/SampleAppWithForm/Program.cs @@ -20,7 +20,12 @@ static int Main(string[] args) return 1; } - if (!Enum.TryParse(args[1], true, out var clientType)) + IClient client; + if (args[1].Equals("Segment", StringComparison.OrdinalIgnoreCase)) + client = new SegmentClient(); + else if (args[1].Equals("Mixpanel", StringComparison.OrdinalIgnoreCase)) + client = new MixpanelClient(); + else { Console.WriteLine($"Usage: SampleApp {Environment.NewLine}Unrecognized client type: {args[1]}"); return 1; @@ -44,7 +49,7 @@ static int Main(string[] args) if (!int.TryParse(args.Skip(2).SingleOrDefault(a => a.StartsWith("-f:"))?.Substring(3), out var flushInterval)) flushInterval = -1; - using (new Analytics(args[0], userInfo, propertiesThatGoWithEveryEvent, clientType: clientType, flushAt: flushAt, + using (new Analytics(args[0], userInfo, propertiesThatGoWithEveryEvent, client: client, flushAt: flushAt, flushInterval: flushInterval)) { var mainWindow = new Form1(); diff --git a/src/SampleAppWithForm/SampleAppWithForm.csproj b/src/SampleAppWithForm/SampleAppWithForm.csproj index c4ddabc..c842f97 100644 --- a/src/SampleAppWithForm/SampleAppWithForm.csproj +++ b/src/SampleAppWithForm/SampleAppWithForm.csproj @@ -20,6 +20,7 @@ {BECACF91-9A84-4667-B678-44ACA350D413} DesktopAnalytics + From c5448f1d794276f2ce499c1415e2f1de24f3205e Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 20:22:01 -0400 Subject: [PATCH 06/11] Adopt SqliteEventSpool, delete DiskQueue/EventSpool MixpanelClient now constructs SqliteEventSpool; the DiskQueue-backed EventSpool and its dependency are removed entirely. Contract tests collapse to the single remaining engine (28 tests, was 56 across two engines); the DiskQueue-vs-SQLite responsiveness characterization test is removed along with the engine it characterized. Verified never-throws by inspection rather than porting DiskQueue's disk-IO-failure tests: all six public IEventSpool members on SqliteEventSpool (Enqueue, ProcessBatchAsync, TrimExpired, Purge, ApproximateCount, ApproximateBytes) wrap their body in try/catch and log-and-swallow -- the property IEventSpool documents ("never throws") holds structurally, not by test coverage of a specific fault path. dotnet list package --include-transitive confirms DiskQueue is gone from all three TFMs. 102/102 Mixpanel tests green on net462 and net8.0; core (19+14) unaffected. Co-Authored-By: Claude Opus 4.8 --- .../DesktopAnalytics.Mixpanel.csproj | 1 - src/DesktopAnalytics.Mixpanel/EventSpool.cs | 756 ----------------- src/DesktopAnalytics.Mixpanel/IEventSpool.cs | 27 +- .../MixpanelClient.cs | 34 +- .../SqliteEventSpool.cs | 80 +- .../EventSpoolContractTests.cs | 22 +- .../EventSpoolTests.cs | 781 ------------------ .../MixpanelClientTests.cs | 141 ++-- .../TrackResponsivenessTests.cs | 70 +- 9 files changed, 189 insertions(+), 1723 deletions(-) delete mode 100644 src/DesktopAnalytics.Mixpanel/EventSpool.cs delete mode 100644 src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs diff --git a/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj b/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj index dcedeea..cd7d10e 100644 --- a/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj +++ b/src/DesktopAnalytics.Mixpanel/DesktopAnalytics.Mixpanel.csproj @@ -17,7 +17,6 @@ ..\..\DesktopAnalytics.snk - diff --git a/src/DesktopAnalytics.Mixpanel/EventSpool.cs b/src/DesktopAnalytics.Mixpanel/EventSpool.cs deleted file mode 100644 index 13b8da2..0000000 --- a/src/DesktopAnalytics.Mixpanel/EventSpool.cs +++ /dev/null @@ -1,756 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Globalization; -using System.IO; -using System.Security.Cryptography; -using System.Text; -using System.Threading; -using System.Threading.Tasks; -using DiskQueue; - -namespace DesktopAnalytics -{ - /// - /// The outcome of attempting to deliver one batch of spooled events, as reported by the caller - /// of . This drives whether the batch is removed from - /// the spool (/) or left in place for a later - /// retry (). (Per-record rejections within a processed batch are - /// the sender/client's concern -- see -- and count - /// as here: the batch is finished with either way.) - /// - internal enum SendResult - { - /// The batch was processed by the server (e.g. HTTP 2xx, or a strict-mode 400 - /// where the rejected records are individually reported). Remove it from the spool. - Delivered, - - /// A transient failure (connection error, timeout, 5xx, 429). Leave the batch in - /// the spool for a later retry. - RetryableFailure, - - /// A non-retryable failure (e.g. a 4xx that will never succeed). Remove the batch - /// from the spool anyway so bad events cannot wedge the spool forever. - PoisonDrop - } - - /// - /// Durable, bounded, on-disk store of pending s, backed by - /// DiskQueue. Events are enqueued as opaque UTF-8 JSON bytes (see - /// ) so the spool stays portable and inspectable. - /// - /// - /// Every public method here swallows disk/IO/serialization exceptions internally (logging via - /// ) and returns gracefully -- analytics code must never - /// throw into the host application. The one exception is the constructor: failing to acquire - /// the underlying DiskQueue's cross-process exclusive lock is a startup-time condition the - /// caller needs to know about, so it is allowed to propagate. - /// - internal class EventSpool : IEventSpool - { - // How long we wait to acquire DiskQueue's cross-process exclusive lock when opening the - // spool. Kept short: if another process is holding the lock for this long, something is - // wrong, and we'd rather fail fast than hang application startup/shutdown. - private static readonly TimeSpan s_lockWaitTimeout = TimeSpan.FromSeconds(2); - - // Number of leading bytes of the SHA-256 hash of the API key used to build the spool - // directory name. Just needs to be stable and distinguish keys from each other -- it is - // not a security boundary -- so a short prefix is plenty. - private const int kApiKeyHashBytes = 8; - - private readonly int _maxItems; - private readonly int _maxItemBytes; - private readonly long _maxSpoolBytes; - // Logical bytes of the LIVE entries (disk file sizes would overcount consumed-but-untrimmed - // ones). Guarded by _sync; measured at open, then maintained incrementally. - private long _spoolBytes; - private readonly string _spoolDirectory; - // Not readonly: Purge replaces the queue wholesale (see its remarks). Always accessed - // under _sync. Null only if a Purge-time reopen failed; every public method already - // catches and logs whatever follows from that. - private IPersistentQueue _queue; - // Serializes ALL access to _queue, exactly like the lock(_syncRoot) it replaced -- a - // SemaphoreSlim because ProcessBatchAsync must hold it across awaits of the send callback, - // which a monitor lock cannot. Deliberately never disposed: it holds no unmanaged - // resources unless AvailableWaitHandle is touched (it isn't), and disposing it would turn - // a benign use-after-Dispose race into ObjectDisposedException noise. - private readonly SemaphoreSlim _sync = new SemaphoreSlim(1, 1); - private bool _disposed; - - /// - /// Raised once for each event permanently dropped by cap enforcement - /// () -- i.e. NOT a drop at enqueue time, which - /// 's own return value already reports, but an OLDER event (or, in the - /// degenerate maxItems == 0 case, the just-enqueued one) evicted later to keep the - /// spool within its configured caps. This is the only way callers can learn about that kind - /// of drop, since it can happen on an call for a DIFFERENT event than - /// the one being dropped. Raised while the internal lock is held, so handlers must be fast - /// and must not call back into this . - /// - public event Action ItemDroppedByCap; - - /// - /// Opens (creating if necessary) the on-disk event spool rooted at - /// , taking DiskQueue's cross-process exclusive lock for - /// the lifetime of this object. - /// - /// Directory to hold the spool's files. Callers normally get - /// this from ; tests typically inject a temp directory. - /// The maximum number of events retained. Enqueuing beyond this - /// drops the oldest event(s) first. - /// The maximum serialized size of a single event; anything - /// larger is refused at enqueue time ( returns false) rather than - /// spooled. Defaults to unbounded; passes the transport's - /// real per-event limit. - /// The maximum total serialized size of all retained events; - /// enqueuing beyond this drops the oldest event(s) first, exactly like - /// . Bounds both the disk footprint and the total upload - /// liability of an accumulated backlog. Defaults to unbounded. - /// Propagates whatever DiskQueue throws if the lock cannot be - /// acquired within the internal wait timeout (e.g. another process/instance already has - /// this spool open) or if the directory cannot be created/opened. This is deliberately not - /// swallowed: it is a startup-time failure the caller needs to know about. - public EventSpool(string spoolDirectory, int maxItems, int maxItemBytes = int.MaxValue, - long maxSpoolBytes = long.MaxValue) - { - if (maxItems < 0) - throw new ArgumentOutOfRangeException(nameof(maxItems)); - if (maxItemBytes <= 0) - throw new ArgumentOutOfRangeException(nameof(maxItemBytes)); - if (maxSpoolBytes <= 0) - throw new ArgumentOutOfRangeException(nameof(maxSpoolBytes)); - - _maxItems = maxItems; - _maxItemBytes = maxItemBytes; - _maxSpoolBytes = maxSpoolBytes; - _spoolDirectory = spoolDirectory; - _queue = PersistentQueue.WaitFor(spoolDirectory, s_lockWaitTimeout); - _spoolBytes = ResolveExistingSpoolBytes(); - } - - private string SpoolBytesFilePath => Path.Combine(_spoolDirectory, "spool-bytes.txt"); - - // Cheap common-case restart path: reuses the byte total PersistSpoolBytesLocked wrote during - // the previous session instead of re-measuring by walking (dequeuing and rolling back) every - // live entry, which is the only way to get an exact total (see MeasureExistingSpoolBytes) but - // means a large backlog turns every app startup into a synchronous, disk-bound scan. The - // persisted total is cross-checked against DiskQueue's own (already-cheap) - // EstimatedCountOfItemsInQueue -- "queue is empty" disagreeing with "persisted total is - // nonzero" (or vice versa) means the sidecar file is stale (e.g. the previous session crashed - // between enqueuing and persisting) and is not trusted, falling back to the slow but exact - // measurement. - private long ResolveExistingSpoolBytes() - { - try - { - var text = File.ReadAllText(SpoolBytesFilePath); - if (long.TryParse(text, NumberStyles.Integer, CultureInfo.InvariantCulture, out var persisted) && - persisted >= 0) - { - var isEmpty = _queue.EstimatedCountOfItemsInQueue <= 0; - if (isEmpty == (persisted == 0)) - return persisted; - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool: no usable persisted spool byte total, measuring instead: " + e); - } - - var measured = MeasureExistingSpoolBytes(); - PersistSpoolBytesLocked(measured); - return measured; - } - - // Sums the live entries left by a previous session (dequeue-all in a session disposed - // without Flush = pure measurement; DiskQueue rolls it back). Failure => 0: the byte cap - // is then under-enforced until the backlog drains, erring toward keeping events. - private long MeasureExistingSpoolBytes() - { - try - { - long total = 0; - using (var session = _queue.OpenSession()) - { - while (true) - { - var bytes = session.Dequeue(); - if (bytes == null) - break; - total += bytes.Length; - } - } - return total; - } - catch (Exception e) - { - Debug.WriteLine("EventSpool: failed to measure existing spool bytes, assuming 0: " + e); - return 0; - } - } - - // Caller must hold _sync (or be under construction, before _sync can be contended). - // Best-effort: a failure here just means the NEXT open falls back to the slower - // MeasureExistingSpoolBytes instead of this fast path -- never a correctness problem. - private void PersistSpoolBytesLocked(long spoolBytes) - { - try - { - File.WriteAllText(SpoolBytesFilePath, spoolBytes.ToString(CultureInfo.InvariantCulture)); - } - catch (Exception e) - { - Debug.WriteLine("EventSpool: failed to persist spool byte total: " + e); - } - } - - /// - /// Computes the per-user, per-API-key spool directory: - /// %LocalAppData%\SIL\DesktopAnalytics\spool\<short hash of apiKey>. The API key is - /// hashed (never embedded raw) so that, e.g., DEBUG and RELEASE builds of an app -- which - /// use different keys -- get separate spools, without exposing the key via the file system. - /// - public static string GetDefaultSpoolPath(string apiKey) - { - var root = Path.Combine( - Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), - "SIL", "DesktopAnalytics", "spool"); - return Path.Combine(root, HashApiKey(apiKey)); - } - - private static string HashApiKey(string apiKey) - { - using (var sha = SHA256.Create()) - { - var hash = sha.ComputeHash(Encoding.UTF8.GetBytes(apiKey ?? string.Empty)); - var builder = new StringBuilder(kApiKeyHashBytes * 2); - for (var i = 0; i < kApiKeyHashBytes; i++) - builder.Append(hash[i].ToString("x2")); - return builder.ToString(); - } - } - - /// - /// An approximate count of events currently in the spool. "Approximate" because DiskQueue - /// tracks this as an estimate that is cheap to read; it is accurate enough for bounding and - /// diagnostics but should not be treated as a hard guarantee under concurrent access. - /// - public int ApproximateCount - { - get - { - try - { - _sync.Wait(); - try - { - return _queue.EstimatedCountOfItemsInQueue; - } - finally - { - _sync.Release(); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.ApproximateCount: failed to read count: " + e); - return 0; - } - } - } - - /// - /// Serializes and enqueues , then enforces the configured item cap by - /// dropping the oldest event(s) if necessary. Never throws: any disk/IO/serialization - /// failure is logged and swallowed, and the event is simply lost rather than crashing the - /// host. - /// - /// True if the event was durably enqueued; false if it was dropped at enqueue time - /// (null, unserializable, larger than the per-event byte cap, or a disk failure). Callers use - /// this to count that kind of drop -- see . A later drop by - /// cap enforcement (this call's own item, or some earlier one) is reported separately via - /// , since it does not necessarily happen on the enqueue call - /// for the event that gets dropped. - public bool Enqueue(AnalyticsEvent evt) - { - if (evt == null) - return false; - - try - { - byte[] bytes; - try - { - bytes = evt.ToBytes(); - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.Enqueue: failed to serialize event, dropping: " + e); - return false; - } - - if (bytes.Length > _maxItemBytes) - { - // Refused up front rather than spooled: an event over the transport's per-event - // limit can never be delivered, and if it is large enough to time out the HTTP - // request it would be classified as a RETRYABLE failure (a timeout is - // indistinguishable from being offline) and wedge the head of the queue forever. - Debug.WriteLine("EventSpool.Enqueue: event of " + bytes.Length + - " bytes exceeds the " + _maxItemBytes + "-byte cap, dropping: " + evt.EventName); - return false; - } - - _sync.Wait(); - try - { - using (var session = _queue.OpenSession()) - { - session.Enqueue(bytes); - session.Flush(); - } - - _spoolBytes += bytes.Length; - EnforceCapLocked(); - PersistSpoolBytesLocked(_spoolBytes); - } - finally - { - _sync.Release(); - } - - return true; - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.Enqueue failed: " + e); - return false; - } - } - - // Caller must hold _sync. Drops the oldest item(s) one at a time until the queue is at or - // under BOTH caps (_maxItems and _maxSpoolBytes). Each iteration removes exactly one item - // (or gives up on a read/dequeue failure), so this always terminates -- including the - // degenerate case of _maxItems == 0, where even the item just enqueued is dropped. - private void EnforceCapLocked() - { - while (true) - { - int count; - try - { - count = _queue.EstimatedCountOfItemsInQueue; - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.EnforceCap: failed to read count, giving up: " + e); - return; - } - - if (count <= _maxItems && _spoolBytes <= _maxSpoolBytes) - return; - - if (count <= 0) - { - // Nothing left to drop; if the byte estimate says otherwise it has drifted -- - // resync it to the truth (an empty queue holds zero bytes). - _spoolBytes = 0; - return; - } - - try - { - using (var session = _queue.OpenSession()) - { - var oldest = session.Dequeue(); - if (oldest == null) - { - _spoolBytes = 0; // See the count <= 0 case above. - return; - } - session.Flush(); - _spoolBytes = Math.Max(0, _spoolBytes - oldest.Length); - } - - try - { - ItemDroppedByCap?.Invoke(); - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.EnforceCap: ItemDroppedByCap handler threw: " + e); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.EnforceCap: failed to drop oldest item, giving up: " + e); - return; - } - } - } - - // The spool's current logical byte total (the value the byte cap is enforced against). - // Also a test seam: tests use it to prove the total survives a dispose/reopen. - public long ApproximateBytes - { - get - { - _sync.Wait(); - try - { - return _spoolBytes; - } - finally - { - _sync.Release(); - } - } - } - - /// - /// Gathers up to spooled events (oldest first, bounded by - /// ) in a single DiskQueue session/transaction, hands them to - /// as ONE batch (one network request -- see - /// ), and commits or rolls back the whole batch on its - /// verdict: - /// - /// or : every - /// dequeue is committed (session.Flush()) -- the batch is finished with, whether - /// ingested or permanently rejected. - /// : nothing is committed -- disposing the - /// session without flushing rolls the whole batch back into the spool, in order, for a - /// later retry. - /// - /// If throws, it is treated exactly like - /// : the exception is swallowed and nothing is - /// flushed. This is the crash-window path that guarantees at-least-once delivery -- a - /// sender that delivered the batch but crashed/threw before this method could flush will - /// see the same events again on the next call. - /// An entry that cannot be deserialized (corrupt, or an incompatible schema version) is - /// skipped during the gather and its removal is committed together with the batch (or, if - /// nothing else was gathered, on its own) -- it can never wedge the spool. - /// - /// Maximum number of events gathered into this call's batch. - /// Decides the batch's fate; see the summary above. Receives - /// so the send can participate in cancellation. - /// Soft byte budget for this call's batch, bounding each drain burst - /// on a slow/metered connection. Checked AFTER each gathered event, so a single event over - /// the whole budget still goes -- the budget can never starve the queue. - /// Bounds the wait for the spool's internal semaphore and - /// is passed through to . Cancellation never throws out of this - /// method; it just means the batch (if any was gathered) rolls back. - public async Task ProcessBatchAsync(int maxItems, - Func, CancellationToken, Task> send, - long maxBytes = long.MaxValue, CancellationToken cancellationToken = default) - { - if (send == null || maxItems <= 0 || maxBytes <= 0) - return; - - try - { - await _sync.WaitAsync(cancellationToken).ConfigureAwait(false); - try - { - using (var session = _queue.OpenSession()) - { - var batch = new List(); - long batchBytes = 0; - // Bytes of undeserializable entries dequeued (skipped) during the gather; - // committed along with the batch. - long skippedBytes = 0; - - while (batch.Count < maxItems && batchBytes < maxBytes) - { - byte[] bytes; - try - { - bytes = session.Dequeue(); - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.ProcessBatch: dequeue failed: " + e); - break; // Send whatever was gathered before the failure. - } - - if (bytes == null) - break; // Spool is empty. - - try - { - batch.Add(AnalyticsEvent.FromBytes(bytes)); - batchBytes += bytes.Length; - } - catch (Exception e) - { - Debug.WriteLine( - "EventSpool.ProcessBatch: failed to deserialize event, dropping: " + e); - skippedBytes += bytes.Length; - } - } - - if (batch.Count == 0) - { - if (skippedBytes > 0) - { - // Nothing to send, but bad entries were dequeued -- commit their removal. - session.Flush(); - _spoolBytes = Math.Max(0, _spoolBytes - skippedBytes); - PersistSpoolBytesLocked(_spoolBytes); - } - return; - } - - SendResult result; - try - { - result = await send(batch, cancellationToken).ConfigureAwait(false); - } - catch (Exception e) - { - Debug.WriteLine( - "EventSpool.ProcessBatch: send threw, leaving batch for retry: " + e); - return; // Do not flush -- every dequeue rolls back on session dispose. - } - - if (result == SendResult.Delivered || result == SendResult.PoisonDrop) - { - session.Flush(); - _spoolBytes = Math.Max(0, _spoolBytes - (batchBytes + skippedBytes)); - PersistSpoolBytesLocked(_spoolBytes); - } - // RetryableFailure: no flush, so the whole batch rolls back. - } - } - finally - { - _sync.Release(); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.ProcessBatch failed: " + e); - } - } - - /// - /// Enforces an age-based retention floor: removes events from the FRONT of the queue - /// (oldest first -- the queue is already FIFO) whose stamped - /// is older than minus , stopping as soon as - /// it reaches an event that is NOT expired -- everything after it, being newer, cannot be - /// expired either. This is an ADDITIONAL, independent eviction dimension alongside the - /// item/byte caps enforced by , not a replacement for them: - /// callers (see 's cap constants) should size those caps to - /// comfortably outlast under realistic usage, so this age floor -- - /// not the item/byte caps -- is normally what reclaims space from a long-offline backlog. - /// - /// - /// An entry that fails to deserialize (corrupt, or an incompatible schema version) can never - /// be dated, so it is dropped exactly like does with such an - /// entry, regardless of age. Never throws: any disk/IO/serialization failure is logged and - /// swallowed, leaving the spool exactly as it was found. - /// - /// The maximum age a queued event may reach before being dropped. - /// The current time, from the caller's injected - /// so tests stay deterministic -- never DateTimeOffset.UtcNow directly. - public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) - { - try - { - var cutoff = now - maxAge; - - _sync.Wait(); - try - { - while (true) - { - byte[] bytes; - using (var session = _queue.OpenSession()) - { - try - { - bytes = session.Dequeue(); - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.TrimExpired: dequeue failed, giving up: " + e); - return; - } - - if (bytes == null) - return; // Spool is empty. - - AnalyticsEvent evt = null; - var deserializeFailed = false; - try - { - evt = AnalyticsEvent.FromBytes(bytes); - } - catch (Exception e) - { - deserializeFailed = true; - Debug.WriteLine( - "EventSpool.TrimExpired: failed to deserialize event, dropping: " + e); - } - - if (!deserializeFailed && evt.Time >= cutoff) - { - // Not expired -- and, by FIFO order, nothing after it can be expired - // either. Do not flush: disposing the session without committing rolls - // this dequeue back, leaving it (and everything behind it) in the queue. - return; - } - - // Either genuinely expired, or undatable (corrupt/incompatible) and thus - // can never be aged out any other way -- commit its removal either way. - session.Flush(); - _spoolBytes = Math.Max(0, _spoolBytes - bytes.Length); - PersistSpoolBytesLocked(_spoolBytes); - } - } - } - finally - { - _sync.Release(); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.TrimExpired failed: " + e); - } - } - - /// - /// Empties the spool entirely (used on consent revocation), removing the event data from - /// disk -- not merely marking it consumed. Never throws. - /// - /// - /// Implemented as dispose + delete-the-directory + reopen rather than either alternative, - /// both tried empirically (2026-07, DiskQueue 1.7.2): a dequeue-and-flush loop leaves the - /// purged events' bytes sitting in the data files (DiskQueue does not trim consumed - /// entries), which is not acceptable for a CONSENT purge; and DiskQueue's own - /// HardDelete(reset: true) leaves the live instance's - /// EstimatedCountOfItemsInQueue stale, so the spool keeps reporting the purged - /// events as present -- consistent with its "not thread safe ... or safe in any other - /// way" warning. Between the dispose and the reopen the cross-process exclusive lock is - /// briefly released; if another process steals it in that window the reopen fails, this - /// spool degrades to a no-op (every method here already tolerates that), and the purge - /// itself has still succeeded -- the data is gone, which is the property that matters. - /// - /// The wait to acquire here is bounded () - /// rather than unbounded, because -style callers (e.g. - /// Analytics.AllowTracking's setter, plausibly invoked from a UI thread handling a - /// consent checkbox) call this synchronously, and an in-flight send that is slow to observe - /// cancellation must not be able to block that caller's thread indefinitely. If the bounded - /// wait times out, this method does NOT give up on purging -- that would violate the - /// consent/privacy guarantee that the data is deleted -- it instead logs and finishes the - /// purge on a background that waits for - /// without a bound (acceptable there since it is off the caller's thread) and then runs the - /// same dispose+delete+reopen logic. - /// - /// - public void Purge() - { - try - { - if (_sync.Wait(s_lockWaitTimeout)) - { - try - { - if (_disposed) - return; - - PurgeLocked(); - } - finally - { - _sync.Release(); - } - } - else - { - // Could not acquire the lock within the bound (most likely an in-flight send - // still holding it, slow to observe cancellation). Do not give up on the purge -- - // it's a consent/privacy guarantee -- finish it on a background thread instead, - // where an unbounded wait is acceptable because it no longer blocks the caller. - Debug.WriteLine( - "EventSpool.Purge: timed out waiting for the lock; completing the purge in the " + - "background instead of blocking the caller."); - Task.Run(() => - { - try - { - _sync.Wait(); - try - { - if (_disposed) - return; - - PurgeLocked(); - } - finally - { - _sync.Release(); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.Purge (background fallback) failed: " + e); - } - }); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.Purge failed: " + e); - } - } - - // Caller must already hold _sync. Shared by both the fast (in-bound) path and the - // background fallback path of Purge() so the actual purge body -- dispose the queue, - // delete the directory, reopen, reset _spoolBytes -- is implemented exactly once. - private void PurgeLocked() - { - try - { - _queue?.Dispose(); - } - finally - { - _queue = null; - } - - if (Directory.Exists(_spoolDirectory)) - Directory.Delete(_spoolDirectory, true); - - _queue = PersistentQueue.WaitFor(_spoolDirectory, s_lockWaitTimeout); - _spoolBytes = 0; - } - - /// - /// Releases the underlying DiskQueue's cross-process exclusive lock. Safe to call more - /// than once. - /// - public void Dispose() - { - if (_disposed) - return; - - try - { - _sync.Wait(); - try - { - _disposed = true; - _queue?.Dispose(); - } - finally - { - _sync.Release(); - } - } - catch (Exception e) - { - Debug.WriteLine("EventSpool.Dispose failed: " + e); - } - } - } -} diff --git a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs index c48b22d..8a97960 100644 --- a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs @@ -5,10 +5,33 @@ namespace DesktopAnalytics { + /// + /// The outcome of attempting to deliver one batch of spooled events, as reported by the caller + /// of . This drives whether the batch is removed + /// from the spool (/) or left in place for a + /// later retry (). (Per-record rejections within a processed + /// batch are the sender/client's concern -- see -- + /// and count as here: the batch is finished with either way.) + /// + internal enum SendResult + { + /// The batch was processed by the server (e.g. HTTP 2xx, or a strict-mode 400 + /// where the rejected records are individually reported). Remove it from the spool. + Delivered, + + /// A transient failure (connection error, timeout, 5xx, 429). Leave the batch in + /// the spool for a later retry. + RetryableFailure, + + /// A non-retryable failure (e.g. a 4xx that will never succeed). Remove the batch + /// from the spool anyway so bad events cannot wedge the spool forever. + PoisonDrop + } + /// /// The durable, bounded store of pending s behind - /// . Implemented by (DiskQueue) and - /// (SQLite); see offline-analytics.md for the comparison. + /// . Implemented by ; see + /// offline-analytics.md for the design background. /// /// /// Contract shared by every implementation: diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index adf467f..13a969e 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -13,14 +13,14 @@ namespace DesktopAnalytics { /// /// Durable Mixpanel client: events are scrubbed, stamped, and written to an on-disk - /// immediately (never lost to being offline), and a background flush - /// loop drains the spool through a Polly-wrapped whenever it can. See - /// offline-analytics.md for the design this implements. + /// immediately (never lost to being offline), and a background + /// flush loop drains the spool through a Polly-wrapped whenever it + /// can. See offline-analytics.md for the design this implements. /// /// /// Every public member here swallows exceptions and returns/no-ops rather than throwing -- /// analytics must never crash the host application (see offline-analytics.md, "Never crash the - /// host"). The one intentionally-not-swallowed failure is a bad + /// host"). The one intentionally-not-swallowed failure is a bad /// constructor call inside , which is itself caught here so it cannot /// escape to the caller either; it just leaves this client running with no spool (Track/Drain /// become no-ops) rather than durable. @@ -32,7 +32,7 @@ public class MixpanelClient : IClient // // Sized to comfortably OUTLAST the 60-day age floor (kDefaultMaxSpoolAgeDays below) under // realistic desktop-app usage-analytics volumes, rather than being the tightest bound in - // practice: EventSpool.TrimExpired's age-based eviction is meant to be what normally + // practice: SqliteEventSpool.TrimExpired's age-based eviction is meant to be what normally // reclaims space from a long-offline backlog, not this item cap. At a generous ~200 // events/day of active use, 60 days is on the order of 12,000 events; 20,000 leaves // headroom for bursty days (e.g. an exception storm) without the item cap kicking in well @@ -64,7 +64,7 @@ public class MixpanelClient : IClient private const long kDefaultMaxSpoolBytes = 50L * 1024 * 1024; // Age-based retention floor: an ADDITIONAL, independent eviction dimension alongside the - // item/byte caps above (enforced separately by EventSpool.TrimExpired), not a replacement + // item/byte caps above (enforced separately by SqliteEventSpool.TrimExpired), not a replacement // for them. Keeps queued (undelivered) events around for roughly two months of offline time // before dropping them -- a generous, explicit retention floor of our own, rather than // whatever much shorter window Mixpanel's legacy /track endpoint implied (moot here anyway, @@ -121,7 +121,7 @@ public class MixpanelClient : IClient // Guards only against overlapping TIMER ticks: if a previous tick's drain is still running // when the next tick fires, the new tick is skipped. It does NOT serialize ticks against // Flush/ShutDown/DrainOnceAsync -- those may run concurrently with a tick, which is safe - // because all spool access is serialized inside EventSpool (its semaphore); this flag just + // because SqliteEventSpool's own locking makes concurrent callers safe; this flag just // keeps a slow drain from stacking up redundant timer callbacks behind it. private int _timerDraining; @@ -156,8 +156,9 @@ public void Initialize(string apiSecret, string host = null, int flushAt = -1, i Math.Max(1, flushInterval > 0 ? flushInterval : kDefaultFlushIntervalSeconds)); _timeProvider = TimeProvider.System; - _spool = new EventSpool(EventSpool.GetDefaultSpoolPath(apiSecret), kDefaultMaxSpoolItems, - kMaxSpooledEventBytes, kDefaultMaxSpoolBytes); + _spool = new SqliteEventSpool(SqliteEventSpool.GetDefaultSpoolPath(apiSecret), + kDefaultMaxSpoolItems, kMaxSpooledEventBytes, kDefaultMaxSpoolBytes, + timeProvider: _timeProvider); _spool.ItemDroppedByCap += OnItemDroppedByCap; _sender = new MixpanelEventSender(apiSecret); _pipeline = BuildDefaultPipeline(_timeProvider); @@ -212,7 +213,7 @@ internal void InitializeForTest( // Keeps Statistics.Failed (and therefore Statistics.Submitted == Succeeded + Failed) accurate // for events dropped later by cap enforcement, not just ones dropped at enqueue time -- see - // EventSpool.ItemDroppedByCap. + // IEventSpool.ItemDroppedByCap. private void OnItemDroppedByCap() { Interlocked.Increment(ref _failed); @@ -387,7 +388,7 @@ private async Task DrainOnceCoreAsync(CancellationToken cancellationToken, bool return; // Age-based retention floor: cheap to run every tick since it stops at the first - // non-expired entry (see EventSpool.TrimExpired's doc comment). + // non-expired entry (see SqliteEventSpool.TrimExpired's doc comment). _spool.TrimExpired(TimeSpan.FromDays(kDefaultMaxSpoolAgeDays), _timeProvider.GetUtcNow()); await _spool.ProcessBatchAsync(_batchSize, @@ -401,11 +402,12 @@ await _spool.ProcessBatchAsync(_batchSize, } } - // The callback EventSpool.ProcessBatchAsync awaits with the gathered batch; the SendResult - // returned is the whole batch's fate (commit vs roll back -- per-record rejections within a - // processed batch are counted in the statistics here and still commit). Must never throw or - // fault (see EventSpool.ProcessBatchAsync's contract: a thrown exception is treated as a - // RetryableFailure anyway, but returning it directly avoids relying on that fallback). + // The callback SqliteEventSpool.ProcessBatchAsync awaits with the gathered batch; the + // SendResult returned is the whole batch's fate (commit vs roll back -- per-record rejections + // within a processed batch are counted in the statistics here and still commit). Must never + // throw or fault (see SqliteEventSpool.ProcessBatchAsync's contract: a thrown exception is + // treated as a RetryableFailure anyway, but returning it directly avoids relying on that + // fallback). private async Task SendBatchGuardedAsync(IReadOnlyList batch, CancellationToken cancellationToken, bool usePipeline) { diff --git a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs index 5d7381b..8b5eae4 100644 --- a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs @@ -3,6 +3,8 @@ using System.Data; using System.Diagnostics; using System.IO; +using System.Security.Cryptography; +using System.Text; using System.Threading; using System.Threading.Tasks; using Microsoft.Data.Sqlite; @@ -10,20 +12,20 @@ namespace DesktopAnalytics { /// - /// SQLite-backed . Same contract as the DiskQueue-backed - /// , but the storage model removes -- rather than works around -- the - /// two structural problems that one has. + /// SQLite-backed . Same contract as the (now removed) DiskQueue-backed + /// EventSpool, but the storage model removes -- rather than works around -- the two + /// structural problems that one had. /// /// - /// Why this exists. As written, holds its global - /// _sync semaphore across the network send, so every concurrent Track() waits out + /// Why this exists. As originally written, EventSpool held its global + /// _sync semaphore across the network send, so every concurrent Track() waited out /// the whole send -- measured at ~3s behind a 3s send, and up to ~45s in production against a - /// stalled connection (15s HttpClient timeout x 3 pipeline attempts). That is a UI freeze on + /// stalled connection (15s HttpClient timeout x 3 pipeline attempts). That was a UI freeze on /// exactly the "bad cafe Wi-Fi" path this whole feature is for. - /// Note: that stall is NOT inherent to DiskQueue. Probed 2026-07-16: DiskQueue + /// Note: that stall was not inherent to DiskQueue. Probed 2026-07-16: DiskQueue /// happily enqueues (~8ms) while an uncommitted dequeue session is open, and a second session /// dequeues a DISTINCT item rather than blocking or double-serving. So dropping - /// 's global lock would fix the stall without changing engines. The + /// EventSpool's global lock would have fixed the stall without changing engines. The /// case for SQLite rests on the OTHER problems below, not on the stall alone. /// SQLite needs no such thing. claims a batch under a /// short lock, releases the lock, sends with nothing held, then removes the delivered @@ -31,16 +33,16 @@ namespace DesktopAnalytics /// and the removal replays the batch, which Mixpanel deduplicates on /// -- the semantics the design already depends on. /// Because the lock is never held across an await, a plain lock suffices here where - /// needed a SemaphoreSlim. + /// EventSpool needed a SemaphoreSlim. /// What it deletes. The byte-total sidecar file, its staleness heuristic, and its /// re-measure fallback all collapse into SUM(len). Age retention collapses into one /// indexed DELETE. Cap eviction collapses into one DELETE. Purge becomes /// DELETE + VACUUM instead of dispose + delete-the-directory + reopen. /// Multi-process. WAL gives real concurrent readers/writers, so a second - /// FieldWorks process spools and drains normally. Today it cannot even open the DiskQueue - /// spool (exclusive lock) and silently degrades to a no-op, losing its events entirely. A - /// short lease on claimed rows keeps two processes from uploading the same batch; the lease - /// expires on its own, so a process that dies mid-send does not strand its batch. + /// FieldWorks process spools and drains normally. The old DiskQueue-backed spool could not even + /// open (exclusive lock) and silently degraded to a no-op, losing its events entirely. A short + /// lease on claimed rows keeps two processes from uploading the same batch; the lease expires + /// on its own, so a process that dies mid-send does not strand its batch. /// internal class SqliteEventSpool : IEventSpool { @@ -76,9 +78,9 @@ internal class SqliteEventSpool : IEventSpool /// Maximum total serialized size of all retained events; /// enqueuing beyond this drops the oldest first. /// Clock for lease expiry. Injected so tests stay deterministic. - /// Propagates a failure to create/open the database. Like - /// 's constructor, this is deliberately not swallowed: it is a - /// startup-time condition the caller needs to know about. + /// Propagates a failure to create/open the database. This is + /// deliberately not swallowed: it is a startup-time condition the caller needs to know + /// about. public SqliteEventSpool(string spoolDirectory, int maxItems, int maxItemBytes = int.MaxValue, long maxSpoolBytes = long.MaxValue, TimeProvider timeProvider = null) { @@ -151,12 +153,36 @@ private static void Execute(SqliteConnection connection, string sql) } } + // Number of leading bytes of the SHA-256 hash of the API key used to build the spool + // directory name. Just needs to be stable and distinguish keys from each other -- it is + // not a security boundary -- so a short prefix is plenty. + private const int kApiKeyHashBytes = 8; + /// - /// Computes the per-user, per-API-key spool directory, matching - /// so the two engines key their storage - /// identically (DEBUG and RELEASE builds stay separate; the key is never embedded raw). + /// Computes the per-user, per-API-key spool directory: + /// %LocalAppData%\SIL\DesktopAnalytics\spool\<short hash of apiKey>. The API key is + /// hashed (never embedded raw) so that, e.g., DEBUG and RELEASE builds of an app -- which + /// use different keys -- get separate spools, without exposing the key via the file system. /// - public static string GetDefaultSpoolPath(string apiKey) => EventSpool.GetDefaultSpoolPath(apiKey); + public static string GetDefaultSpoolPath(string apiKey) + { + var root = Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), + "SIL", "DesktopAnalytics", "spool"); + return Path.Combine(root, HashApiKey(apiKey)); + } + + private static string HashApiKey(string apiKey) + { + using (var sha = SHA256.Create()) + { + var hash = sha.ComputeHash(Encoding.UTF8.GetBytes(apiKey ?? string.Empty)); + var builder = new StringBuilder(kApiKeyHashBytes * 2); + for (var i = 0; i < kApiKeyHashBytes; i++) + builder.Append(hash[i].ToString("x2")); + return builder.ToString(); + } + } /// public int ApproximateCount @@ -290,8 +316,8 @@ private void RaiseDropped(int count) } // Caller must hold _sync. Brings the spool back within BOTH caps, dropping oldest first, - // and returns how many events were dropped. Two DELETEs, no loop -- contrast - // EventSpool.EnforceCapLocked, which dequeues one item at a time. + // and returns how many events were dropped. Two DELETEs, no loop -- contrast the old + // DiskQueue-backed EventSpool.EnforceCapLocked, which dequeued one item at a time. private int EnforceCapsLocked() { var dropped = 0; @@ -546,8 +572,8 @@ private void DeleteByIdLocked(SqliteTransaction txn, List ids) /// /// - /// One indexed DELETE. Unlike this does not have to - /// stop at the first non-expired event -- it removes every expired event wherever it sits, + /// One indexed DELETE. Unlike the old DiskQueue-backed EventSpool's TrimExpired, this does + /// not have to stop at the first non-expired event -- it removes every expired event wherever it sits, /// so an out-of-order timestamp cannot shield older events from the retention floor. A /// corrupt payload is irrelevant here too: the age lives in a column, so nothing has to be /// deserialized to be dated. @@ -583,9 +609,9 @@ public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) /// actually gone rather than lingering as free pages; the WAL checkpoint/truncate does the /// same for the write-ahead log. All three matter for a CONSENT purge, where "marked /// consumed" is not good enough. - /// Contrast , which has to dispose the queue, delete - /// the whole directory and reopen -- briefly dropping its cross-process lock, and degrading - /// to a no-op spool if another process steals it in that window. + /// Contrast the old DiskQueue-backed EventSpool's Purge, which had to dispose the + /// queue, delete the whole directory and reopen -- briefly dropping its cross-process lock, + /// and degrading to a no-op spool if another process stole it in that window. /// public void Purge() { diff --git a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs index 4a93a5f..52f7502 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs @@ -10,17 +10,11 @@ namespace DesktopAnalyticsTests { /// - /// The storage-engine-agnostic contract, run against BOTH engines so - /// their behavior is compared rather than argued about. Engine-specific behavior (DiskQueue's - /// corrupt-file recovery and its exclusive cross-process lock; SQLite's WAL concurrency) stays - /// in each engine's own fixture. + /// The contract, run against the SQLite engine. Kept parametrized + /// over (rather than collapsed to a concrete fixture) as a + /// low-risk leftover of when this ran against both DiskQueue- and SQLite-backed engines side + /// by side to prove they agreed; now there is exactly one . /// - /// - /// Overlaps EventSpoolTests by design while both engines are in the tree: this fixture proves - /// they agree, that one does not. Once an engine is chosen, the loser's fixture and this - /// parametrization both go away. - /// - [TestFixture(typeof(DiskQueueSpoolFactory))] [TestFixture(typeof(SqliteSpoolFactory))] internal class EventSpoolContractTests where TFactory : ISpoolFactory, new() { @@ -571,14 +565,6 @@ internal interface ISpoolFactory IEventSpool Create(string dir, int maxItems, int maxItemBytes, long maxSpoolBytes); } - internal class DiskQueueSpoolFactory : ISpoolFactory - { - public string Name => "DiskQueue"; - - public IEventSpool Create(string dir, int maxItems, int maxItemBytes, long maxSpoolBytes) => - new EventSpool(dir, maxItems, maxItemBytes, maxSpoolBytes); - } - internal class SqliteSpoolFactory : ISpoolFactory { public string Name => "Sqlite"; diff --git a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs deleted file mode 100644 index 21e6671..0000000 --- a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolTests.cs +++ /dev/null @@ -1,781 +0,0 @@ -using System; -using System.Collections.Generic; -using System.IO; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using DesktopAnalytics; -using NUnit.Framework; - -namespace DesktopAnalyticsTests -{ - // NOTE on the "corrupt/partial spool" edge case (power loss mid-write): rather than - // hand-crafting a truncated transaction-log file matching DiskQueue's private on-disk format - // (which would couple this suite to internals that may change between DiskQueue versions), - // test 10 below appends format-agnostic garbage bytes to the transaction log and asserts the - // spool survives -- exercising DiskQueue's transactional recovery (entries are only - // discoverable after a matching start/end transaction marker pair) without depending on what - // the markers look like. Separately, EventSpool.ProcessBatch guards against an individual - // entry that deserializes badly (e.g. an unreadable/incompatible schema version) by skipping - // (dropping) it like a poison message rather than wedging the spool -- see EventSpool.cs. - [TestFixture] - public class EventSpoolTests - { - private string _spoolDir; - - [SetUp] - public void SetUp() - { - _spoolDir = Path.Combine(Path.GetTempPath(), "EventSpoolTests_" + Guid.NewGuid()); - } - - [TearDown] - public void TearDown() - { - // Any EventSpool opened by a test must already be disposed (releasing DiskQueue's - // exclusive lock) by the time we get here -- each test disposes its spool(s) via - // `using` before returning. - if (Directory.Exists(_spoolDir)) - { - try - { - Directory.Delete(_spoolDir, true); - } - catch (Exception e) - { - // Best-effort cleanup; don't fail the test run over a leftover temp directory. - Console.WriteLine("EventSpoolTests.TearDown: failed to delete " + _spoolDir + ": " + e); - } - } - } - - private static AnalyticsEvent MakeEvent(string name) - { - return AnalyticsEvent.Create("user-1", name); - } - - private static AnalyticsEvent MakeEventAt(string name, DateTimeOffset time) - { - return AnalyticsEvent.Create("user-1", name, time: time); - } - - // Drains the spool with an always-Delivered batch callback, returning the event names in - // the order they were handed to the sender. - private static async Task> DrainAllDelivered(EventSpool spool, int maxItems = 100) - { - var names = new List(); - await spool.ProcessBatchAsync(maxItems, (batch, ct) => - { - names.AddRange(batch.Select(e => e.EventName)); - return Task.FromResult(SendResult.Delivered); - }); - return names; - } - - // Reads every file under the spool directory that can be opened for shared reading. - // DiskQueue holds its 'lock' file open exclusively while the spool is open; that file - // only ever contains a process id, never event data, so skipping unopenable files does - // not weaken any event-content assertion. - internal static IEnumerable> ReadableSpoolFiles(string spoolDir) - { - foreach (var file in Directory.GetFiles(spoolDir, "*", SearchOption.AllDirectories)) - { - string content; - try - { - using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read, - FileShare.ReadWrite | FileShare.Delete)) - using (var reader = new StreamReader(stream, System.Text.Encoding.UTF8)) - content = reader.ReadToEnd(); - } - catch (IOException) - { - continue; // DiskQueue's exclusively-held lock file. - } - - yield return new KeyValuePair(file, content); - } - } - - // 1. RESTART DURABILITY: enqueue N, dispose, reopen a NEW spool on the same directory -- - // all N survive, and a subsequent ProcessBatch (Delivered) empties it. - [Test] - public async Task Enqueue_ThenDisposeAndReopen_EventsSurviveAndDrainFully() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - for (var i = 0; i < 5; i++) - spool.Enqueue(MakeEvent("Event-" + i)); - } // Disposed here, releasing the exclusive lock. - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - Assert.AreEqual(5, reopened.ApproximateCount); - - var delivered = await DrainAllDelivered(reopened); - - Assert.AreEqual(5, delivered.Count); - Assert.AreEqual(0, reopened.ApproximateCount); - } - } - - // 2. ProcessBatch where send returns Delivered => queue empty afterward. - [Test] - public async Task ProcessBatch_AllDelivered_EmptiesSpool() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("A")); - spool.Enqueue(MakeEvent("B")); - - await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.Delivered)); - - Assert.AreEqual(0, spool.ApproximateCount); - } - } - - // 3. send returns RetryableFailure => the WHOLE batch rolls back (nothing is committed -- - // with one request per batch there is no per-event verdict to split on); a later - // ProcessBatch delivers all of them, in order. - [Test] - public async Task ProcessBatch_RetryableFailure_RollsWholeBatchBackForLaterRetry() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("A")); - spool.Enqueue(MakeEvent("B")); - spool.Enqueue(MakeEvent("C")); - - var firstPass = new List(); - await spool.ProcessBatchAsync(10, (batch, ct) => - { - firstPass.AddRange(batch.Select(e => e.EventName)); - return Task.FromResult(SendResult.RetryableFailure); - }); - - CollectionAssert.AreEqual(new[] { "A", "B", "C" }, firstPass); - Assert.AreEqual(3, spool.ApproximateCount, - "a retryable failure must leave the whole batch in the spool"); - - var secondPass = await DrainAllDelivered(spool); - - CollectionAssert.AreEqual(new[] { "A", "B", "C" }, secondPass, - "the rolled-back batch must be retried in its original order"); - Assert.AreEqual(0, spool.ApproximateCount); - } - } - - // 4. CRASH-WINDOW: send records the batch then THROWS => nothing removed; reopening the - // spool shows the same events still present (at-least-once, no data loss). - [Test] - public async Task ProcessBatch_SendThrows_NothingRemovedAndEventsSurviveReopen() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Crash-1")); - spool.Enqueue(MakeEvent("Crash-2")); - - var recorded = new List(); - Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => - { - // Simulate a sender that actually delivered the batch over the network, then - // crashed/threw before this method could observe success and flush. - recorded.AddRange(batch.Select(e => e.EventName)); - throw new InvalidOperationException("simulated crash after send, before ack"); - })); - - CollectionAssert.AreEqual(new[] { "Crash-1", "Crash-2" }, recorded); - Assert.AreEqual(2, spool.ApproximateCount); - } - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - Assert.AreEqual(2, reopened.ApproximateCount); - - var delivered = await DrainAllDelivered(reopened); - - CollectionAssert.AreEqual(new[] { "Crash-1", "Crash-2" }, delivered); - } - } - - // 5. Poison: send returns PoisonDrop => the batch is removed even though not delivered, - // so a permanently rejected request cannot wedge the spool. - [Test] - public async Task ProcessBatch_PoisonDrop_RemovesBatchEvenThoughNotDelivered() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Poison-1")); - spool.Enqueue(MakeEvent("Poison-2")); - - await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.PoisonDrop)); - - Assert.AreEqual(0, spool.ApproximateCount, - "a poison batch must be dropped, not left to wedge the spool"); - } - } - - // 6. Bounding: maxItems=3, enqueue 5 => count==3, the 3 NEWEST remain. - [Test] - public async Task Enqueue_ExceedingMaxItems_DropsOldestKeepsNewest() - { - using (var spool = new EventSpool(_spoolDir, 3)) - { - for (var i = 0; i < 5; i++) - spool.Enqueue(MakeEvent("Event-" + i)); - - Assert.AreEqual(3, spool.ApproximateCount); - - var remaining = await DrainAllDelivered(spool); - - CollectionAssert.AreEqual(new[] { "Event-2", "Event-3", "Event-4" }, remaining); - } - } - - // 6a. Degenerate cap: maxItems == 0 means even the event just enqueued is evicted (see the - // comment on EnforceCapLocked describing this degenerate case). Enqueue itself still - // reports success (the write genuinely happened; cap enforcement is a separate step), but - // the spool ends up empty. - [Test] - public void Enqueue_MaxItemsZero_ImmediatelyEvictsTheJustEnqueuedEvent() - { - using (var spool = new EventSpool(_spoolDir, 0)) - { - Assert.IsTrue(spool.Enqueue(MakeEvent("A")), - "Enqueue reports the write itself succeeded, independent of cap enforcement"); - Assert.AreEqual(0, spool.ApproximateCount, - "maxItems == 0 must evict even the event that was just enqueued"); - } - } - - // 6b. Boundary: exactly maxItems enqueued => nothing dropped; one more => the oldest is - // dropped and the count stays pinned at the cap. - [Test] - public async Task Enqueue_ExactlyAtMaxItems_NoneDropped_ThenOneOver_DropsOldestStaysAtCap() - { - const int maxItems = 5; - using (var spool = new EventSpool(_spoolDir, maxItems)) - { - for (var i = 0; i < maxItems; i++) - spool.Enqueue(MakeEvent("Event-" + i)); - - Assert.AreEqual(maxItems, spool.ApproximateCount, - "exactly maxItems events must all be retained"); - - spool.Enqueue(MakeEvent("OneMore")); - - Assert.AreEqual(maxItems, spool.ApproximateCount, "count must stay pinned at the cap"); - - var remaining = await DrainAllDelivered(spool, maxItems + 5); - CollectionAssert.AreEqual( - new[] { "Event-1", "Event-2", "Event-3", "Event-4", "OneMore" }, remaining, - "the oldest event must be dropped to make room for the one over the cap"); - } - } - - // ---- AGE-BASED RETENTION (TrimExpired) -------------------------------------------------- - - // An event older than the max age must be dropped. - [Test] - public void TrimExpired_EventOlderThanMaxAge_IsDropped() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - var now = DateTimeOffset.UtcNow; - spool.Enqueue(MakeEventAt("Old", now - TimeSpan.FromDays(61))); - - spool.TrimExpired(TimeSpan.FromDays(60), now); - - Assert.AreEqual(0, spool.ApproximateCount, - "an event older than the max age must be dropped"); - } - } - - // An event within the max age must be retained. - [Test] - public void TrimExpired_EventWithinMaxAge_IsRetained() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - var now = DateTimeOffset.UtcNow; - spool.Enqueue(MakeEventAt("Recent", now - TimeSpan.FromDays(1))); - - spool.TrimExpired(TimeSpan.FromDays(60), now); - - Assert.AreEqual(1, spool.ApproximateCount, - "an event within the max age must be retained"); - } - } - - // A mix of expired and fresh events: only the expired PREFIX is dropped (FIFO order), and - // the rest survive in their original order. - [Test] - public async Task TrimExpired_MixOfExpiredAndFreshEvents_DropsOnlyExpiredPrefixKeepsRestInOrder() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - var now = DateTimeOffset.UtcNow; - spool.Enqueue(MakeEventAt("Expired-1", now - TimeSpan.FromDays(90))); - spool.Enqueue(MakeEventAt("Expired-2", now - TimeSpan.FromDays(70))); - spool.Enqueue(MakeEventAt("Fresh-1", now - TimeSpan.FromDays(10))); - spool.Enqueue(MakeEventAt("Fresh-2", now - TimeSpan.FromDays(1))); - - spool.TrimExpired(TimeSpan.FromDays(60), now); - - Assert.AreEqual(2, spool.ApproximateCount); - var remaining = await DrainAllDelivered(spool); - CollectionAssert.AreEqual(new[] { "Fresh-1", "Fresh-2" }, remaining, - "only the expired prefix must be dropped; the rest must survive in order"); - } - } - - // TrimExpired on an empty spool is a no-op that must not throw. - [Test] - public void TrimExpired_EmptySpool_IsNoOpAndDoesNotThrow() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - Assert.DoesNotThrow(() => spool.TrimExpired(TimeSpan.FromDays(60), DateTimeOffset.UtcNow)); - Assert.AreEqual(0, spool.ApproximateCount); - } - } - - // ---- DISK I/O FAILURE (degrade gracefully, never throw) --------------------------------- - - // A "disk full"/inaccessible-file analog: make the spool's own data file read-only out from - // under it (a real, DiskQueue-external way to force a write failure without hand-rolling a - // mock of DiskQueue itself) and confirm Enqueue reports the failure via its return value -- - // never by throwing -- and does not corrupt the spool's state. - [Test] - public void Enqueue_DataFileMadeReadOnly_ReturnsFalseAndNeverThrowsAndLeavesSpoolConsistent() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - Assert.IsTrue(spool.Enqueue(MakeEvent("Seed")), "seed event to ensure the data file exists"); - Assert.AreEqual(1, spool.ApproximateCount); - - var dataFile = Path.Combine(_spoolDir, "data.0"); - Assert.IsTrue(File.Exists(dataFile), "expected DiskQueue's data file at " + dataFile); - File.SetAttributes(dataFile, FileAttributes.ReadOnly); - try - { - bool result = true; - Assert.DoesNotThrow(() => result = spool.Enqueue(MakeEvent("ShouldFail")), - "a write failure (disk full/inaccessible analog) must never throw out of Enqueue"); - Assert.IsFalse(result, - "a write failure must be reported via Enqueue's return value"); - Assert.AreEqual(1, spool.ApproximateCount, - "a failed write must not corrupt the count of what is actually spooled"); - } - finally - { - File.SetAttributes(dataFile, FileAttributes.Normal); - } - } - } - - // Same idea for the read/drain side: lock the data file exclusively (simulating it being - // momentarily inaccessible) while ProcessBatchAsync is draining. The dequeue must fail - // internally, the method must not throw, and -- once the lock is released -- the event must - // still be present and retrievable (nothing was lost by the transient failure). - [Test] - public async Task ProcessBatch_DataFileTransientlyLocked_DegradesGracefullyAndEventSurvives() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("A")); - - var dataFile = Path.Combine(_spoolDir, "data.0"); - Assert.IsTrue(File.Exists(dataFile), "expected DiskQueue's data file at " + dataFile); - - var sentDuringLock = new List(); - using (new FileStream(dataFile, FileMode.Open, FileAccess.Read, FileShare.None)) - { - Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => - { - sentDuringLock.AddRange(batch.Select(e => e.EventName)); - return Task.FromResult(SendResult.Delivered); - })); - } - - Assert.AreEqual(0, sentDuringLock.Count, - "the transiently locked file must prevent the dequeue from succeeding"); - - var delivered = await DrainAllDelivered(spool); - CollectionAssert.AreEqual(new[] { "A" }, delivered, - "the event must survive a transient read failure and still be retrievable afterward"); - } - } - - // 7. Purge empties a non-empty spool. - [Test] - public async Task Purge_EmptiesNonEmptySpool() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - for (var i = 0; i < 5; i++) - spool.Enqueue(MakeEvent("Event-" + i)); - - Assert.AreEqual(5, spool.ApproximateCount); - - spool.Purge(); - - Assert.AreEqual(0, spool.ApproximateCount); - - // A drain after Purge should find nothing to send. - var delivered = await DrainAllDelivered(spool); - Assert.AreEqual(0, delivered.Count); - } - } - - // 7b. Purge leaves the spool fully usable: new events can be enqueued in the same - // session, only they survive a dispose/reopen, and the purged events' bytes are gone - // from the on-disk files once the spool is closed (the privacy property of a consent - // purge -- DiskQueue trims consumed entries rather than retaining them). - [Test] - public async Task Purge_ThenEnqueue_SpoolRemainsUsableAndPurgedBytesAreGoneFromDisk() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("PrePurgeSecret")); - Assert.AreEqual(1, spool.ApproximateCount); - - spool.Purge(); - Assert.AreEqual(0, spool.ApproximateCount); - - spool.Enqueue(MakeEvent("PostPurge")); - Assert.AreEqual(1, spool.ApproximateCount); - } - - foreach (var file in ReadableSpoolFiles(_spoolDir)) - { - StringAssert.DoesNotContain("PrePurgeSecret", file.Value, - "a consent purge must remove event data from disk, not just mark it consumed: " + file.Key); - } - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - var delivered = await DrainAllDelivered(reopened); - CollectionAssert.AreEqual(new[] { "PostPurge" }, delivered); - } - } - - // 8. Concurrent enqueue from multiple threads (single process): 4 threads x 25 events => - // all 100 persisted. Dedicated Threads (not Task.Run) so the intended parallelism is - // explicit rather than at the mercy of thread-pool scheduling. - [Test] - public async Task Enqueue_FromMultipleThreadsConcurrently_AllEventsPersisted() - { - const int threadCount = 4; - const int perThread = 25; - - using (var spool = new EventSpool(_spoolDir, threadCount * perThread)) - { - var threads = new Thread[threadCount]; - for (var t = 0; t < threadCount; t++) - { - var threadIndex = t; - threads[t] = new Thread(() => - { - for (var i = 0; i < perThread; i++) - spool.Enqueue(MakeEvent($"T{threadIndex}-{i}")); - }); - } - - foreach (var thread in threads) - thread.Start(); - foreach (var thread in threads) - thread.Join(); - - Assert.AreEqual(threadCount * perThread, spool.ApproximateCount); - - // Confirm every single one is actually retrievable, not just counted. - var seen = new HashSet( - await DrainAllDelivered(spool, threadCount * perThread)); - Assert.AreEqual(threadCount * perThread, seen.Count); - } - } - - // 8a. Per-event byte cap: an event whose serialized form exceeds maxItemBytes is refused - // (Enqueue returns false) and never spooled, while a normal event on the same spool is - // accepted -- and Enqueue's return value distinguishes the two. - [Test] - public void Enqueue_EventLargerThanMaxItemBytes_IsRefusedWhileNormalEventIsAccepted() - { - using (var spool = new EventSpool(_spoolDir, 10, maxItemBytes: 500)) - { - var oversized = AnalyticsEvent.Create("user-1", "Huge", - new Segment.Serialization.JsonObject - { - { "Stack Trace", new string('x', 2000) } - }); - - Assert.IsFalse(spool.Enqueue(oversized), "an event over the byte cap must be refused"); - Assert.AreEqual(0, spool.ApproximateCount); - - Assert.IsTrue(spool.Enqueue(MakeEvent("Normal")), "a normal event must still be accepted"); - Assert.AreEqual(1, spool.ApproximateCount); - } - } - - // 8a2. Byte-based spool cap: enqueuing past maxSpoolBytes drops the oldest event(s), - // exactly like the item cap. - [Test] - public async Task Enqueue_ExceedingMaxSpoolBytes_DropsOldestKeepsNewest() - { - var events = new[] { MakeEvent("Event-A"), MakeEvent("Event-B"), MakeEvent("Event-C"), MakeEvent("Event-D") }; - // Cap sized (from the real serialized lengths) to hold the first three but not all four. - long cap = events[0].ToBytes().Length + events[1].ToBytes().Length + events[2].ToBytes().Length; - - using (var spool = new EventSpool(_spoolDir, 100, maxSpoolBytes: cap)) - { - foreach (var evt in events) - spool.Enqueue(evt); - - var remaining = await DrainAllDelivered(spool); - - CollectionAssert.AreEqual(new[] { "Event-B", "Event-C", "Event-D" }, remaining, - "the byte cap must drop the OLDEST event to make room"); - } - } - - // 8a3. The byte accounting behind the cap survives a restart -- normally read back from the - // total persisted alongside the spool (see ResolveExistingSpoolBytes), falling back to - // re-measuring the live entries if that is missing or untrustworthy -- so a reopened spool - // enforces the cap correctly either way. - [Test] - public void ApproximateBytes_SurvivesDisposeAndReopen() - { - long expectedBytes; - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Event-0")); - spool.Enqueue(MakeEvent("Event-1")); - expectedBytes = spool.ApproximateBytes; - Assert.Greater(expectedBytes, 0); - } - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - Assert.AreEqual(expectedBytes, reopened.ApproximateBytes, - "the persisted byte total must be restored on open"); - Assert.AreEqual(2, reopened.ApproximateCount, "opening must not consume the entries"); - } - } - - // 8a3b. If the persisted byte-total sidecar file is missing or corrupt (e.g. this session - // crashed before persisting, or the file predates this feature), the spool must fall back to - // re-measuring from the live entries rather than trusting a stale/absent value -- the cap - // must still be enforced correctly on the next restart. - [Test] - public void ApproximateBytes_MissingPersistedTotal_FallsBackToMeasuring() - { - long expectedBytes; - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Event-0")); - spool.Enqueue(MakeEvent("Event-1")); - expectedBytes = spool.ApproximateBytes; - } - - File.Delete(Path.Combine(_spoolDir, "spool-bytes.txt")); - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - Assert.AreEqual(expectedBytes, reopened.ApproximateBytes, - "a missing persisted total must fall back to re-measuring the live entries"); - Assert.AreEqual(2, reopened.ApproximateCount, "measuring must not consume the entries"); - } - } - - // 8a4. ProcessBatch byte budget: the gather stops once the batch reaches maxBytes, but - // a single event over the whole budget still goes (forward progress is guaranteed). - [Test] - public async Task ProcessBatch_ByteBudget_BoundsBatchButNeverStarves() - { - var events = new[] { MakeEvent("Event-0"), MakeEvent("Event-1"), MakeEvent("Event-2"), - MakeEvent("Event-3"), MakeEvent("Event-4") }; - // A budget the first two events fit under but the third pushes past. - long budget = events[0].ToBytes().Length + events[1].ToBytes().Length + 1; - - using (var spool = new EventSpool(_spoolDir, 10)) - { - foreach (var evt in events) - spool.Enqueue(evt); - - var firstBatch = new List(); - await spool.ProcessBatchAsync(10, (batch, ct) => - { - firstBatch.AddRange(batch.Select(e => e.EventName)); - return Task.FromResult(SendResult.Delivered); - }, budget); - - CollectionAssert.AreEqual(new[] { "Event-0", "Event-1", "Event-2" }, firstBatch, - "the budget is checked AFTER each gathered event, so the event that crosses it is still included"); - Assert.AreEqual(2, spool.ApproximateCount); - - // A budget smaller than any single event must still make progress, one event per call. - var secondBatch = new List(); - await spool.ProcessBatchAsync(10, (batch, ct) => - { - secondBatch.AddRange(batch.Select(e => e.EventName)); - return Task.FromResult(SendResult.Delivered); - }, maxBytes: 1); - - CollectionAssert.AreEqual(new[] { "Event-3" }, secondBatch); - Assert.AreEqual(1, spool.ApproximateCount); - } - } - - // 8b. ProcessBatch honors maxItems: with more events spooled than the batch size, exactly - // maxItems are gathered per call and the rest stay put for the next call. - [Test] - public async Task ProcessBatch_MoreEventsThanMaxItems_GathersExactlyMaxItemsPerCall() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - for (var i = 0; i < 5; i++) - spool.Enqueue(MakeEvent("Event-" + i)); - - var firstBatch = await DrainAllDelivered(spool, maxItems: 2); - - CollectionAssert.AreEqual(new[] { "Event-0", "Event-1" }, firstBatch); - Assert.AreEqual(3, spool.ApproximateCount); - - var secondBatch = await DrainAllDelivered(spool, maxItems: 2); - - CollectionAssert.AreEqual(new[] { "Event-2", "Event-3" }, secondBatch); - Assert.AreEqual(1, spool.ApproximateCount); - } - } - - // 8c. CANCELLATION: a pre-canceled token means the batch is never gathered or sent, the - // spool is untouched, and nothing throws -- and the send callback receives the SAME token - // that was passed in, so an in-flight send can participate in cancellation. - [Test] - public async Task ProcessBatch_CancellationToken_IsHonoredAndPassedThroughToSend() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("A")); - - using (var cts = new CancellationTokenSource()) - { - cts.Cancel(); - var sendCalled = false; - Assert.DoesNotThrowAsync(async () => await spool.ProcessBatchAsync(10, (batch, ct) => - { - sendCalled = true; - return Task.FromResult(SendResult.Delivered); - }, cancellationToken: cts.Token)); - - Assert.IsFalse(sendCalled, "a pre-canceled token must short-circuit before the send"); - Assert.AreEqual(1, spool.ApproximateCount, "the event must remain spooled"); - } - - using (var cts = new CancellationTokenSource()) - { - CancellationToken observed = default; - await spool.ProcessBatchAsync(10, (batch, ct) => - { - observed = ct; - return Task.FromResult(SendResult.Delivered); - }, cancellationToken: cts.Token); - - Assert.AreEqual(cts.Token, observed, - "the send callback must receive the caller's token so it can participate in cancellation"); - Assert.AreEqual(0, spool.ApproximateCount); - } - } - } - - // 9. Second EventSpool opened on the SAME directory while the first is still open - // fails/throws, proving the cross-process exclusive lock (DiskQueue.PersistentQueue.WaitFor - // with a short internal timeout) does not silently allow shared access. - [Test] - public void Constructor_SecondSpoolOnSameDirectoryWhileFirstOpen_Throws() - { - using (new EventSpool(_spoolDir, 10)) - { - // Assert.Catch (unlike Assert.Throws, which requires an *exact* type - // match) accepts any exception assignable to Exception -- we don't want this test - // coupled to DiskQueue's specific exception type (currently TimeoutException) for - // a lock-acquisition failure. - Assert.Catch(() => - { - using (new EventSpool(_spoolDir, 10)) - { - } - }); - } - } - - // 10. CORRUPTION, torn data-file write: garbage appended to a data file (bytes beyond the - // extents any committed transaction references) must not prevent reopening the spool, and - // events from committed transactions must still be readable. Appending garbage -- rather - // than hand-crafting file contents -- keeps this test independent of DiskQueue's private - // on-disk format (see the NOTE at the top of this fixture). - [Test] - public async Task Constructor_GarbageAppendedToDataFile_SpoolReopensAndCommittedEventsSurvive() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Committed-1")); - spool.Enqueue(MakeEvent("Committed-2")); - } // Disposed: both enqueues are fully committed transactions on disk. - - AppendGarbageTo(FindSpoolFile("data.0")); - - using (var reopened = new EventSpool(_spoolDir, 10)) - { - var delivered = await DrainAllDelivered(reopened); - CollectionAssert.AreEqual(new[] { "Committed-1", "Committed-2" }, delivered); - - // And the recovered spool must still accept new work. - reopened.Enqueue(MakeEvent("PostRecovery")); - Assert.AreEqual(1, reopened.ApproximateCount); - } - } - - // 10b. CORRUPTION, garbage in the transaction log: DiskQueue deliberately refuses to open - // a log with unrecognized trailing bytes (it recovers a TRUNCATED log -- the realistic - // power-loss shape, where entries simply end mid-transaction -- but treats garbage after a - // committed entry as a conflict). Pin that fail-fast behavior here: the constructor throws - // rather than silently serving corrupt data. At the client level this is already handled: - // MixpanelClient.Initialize catches the throw and degrades to a non-durable no-op client - // rather than crashing the host (see MixpanelClient's class remarks). - [Test] - public void Constructor_GarbageAppendedToTransactionLog_FailsFastRatherThanServingCorruptData() - { - using (var spool = new EventSpool(_spoolDir, 10)) - { - spool.Enqueue(MakeEvent("Committed-1")); - } - - AppendGarbageTo(FindSpoolFile("transaction.log")); - - Assert.Catch(() => - { - using (new EventSpool(_spoolDir, 10)) - { - } - }); - } - - private string FindSpoolFile(string fileName) - { - var path = Path.Combine(_spoolDir, fileName); - Assert.IsTrue(File.Exists(path), - "expected DiskQueue file at " + path + " -- if DiskQueue renamed it, update this test"); - return path; - } - - private static void AppendGarbageTo(string path) - { - var garbage = new byte[257]; - new Random(12345).NextBytes(garbage); - using (var stream = new FileStream(path, FileMode.Append, FileAccess.Write)) - stream.Write(garbage, 0, garbage.Length); - } - } -} diff --git a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs index e8bad33..16121a2 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs @@ -12,7 +12,7 @@ namespace DesktopAnalyticsTests { - // Layer 3 of the fidelity ladder in offline-analytics.md: the real EventSpool (on a temp + // Layer 3 of the fidelity ladder in offline-analytics.md: the real SqliteEventSpool (on a temp // folder) driving a real MixpanelClient, with a scripted fake IEventSender standing in for the // network. Tests await DrainOnceAsync() directly rather than racing the real background timer (see // "Design-for-test seams" / "Flush-loop scheduling" in offline-analytics.md); InitializeForTest @@ -150,12 +150,38 @@ public async Task SendBatchAsync(IReadOnlyList } } + // Reads every file under the spool directory that can be opened for shared reading. + // SqliteEventSpool keeps its connection (and thus a share-mode lock on the database file) + // open for the whole test, but WAL mode still allows a second, independent handle to read + // the file's on-disk bytes concurrently; skipping any file that genuinely cannot be opened + // this way does not weaken any event-content assertion. + private static IEnumerable> ReadableSpoolFiles(string spoolDir) + { + foreach (var file in Directory.GetFiles(spoolDir, "*", SearchOption.AllDirectories)) + { + string content; + try + { + using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete)) + using (var reader = new StreamReader(stream, System.Text.Encoding.UTF8)) + content = reader.ReadToEnd(); + } + catch (IOException) + { + continue; + } + + yield return new KeyValuePair(file, content); + } + } + // ---- NO-LOSS ------------------------------------------------------------------------- [Test] public async Task DrainOnce_RetryRetryThenDeliverAcrossSuccessiveCalls_DeliversExactlyOnceAndEmptiesSpool() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new ScriptedSender(new Func, BatchSendResult>[] { @@ -189,7 +215,7 @@ public async Task DrainOnce_RetryRetryThenDeliverAcrossSuccessiveCalls_DeliversE [Test] public async Task DrainOnce_SenderRecordsThenThrows_SameInsertIdSeenAgainOnLaterRetry() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); @@ -229,7 +255,7 @@ public async Task DrainOnce_SenderRecordsThenThrows_SameInsertIdSeenAgainOnLater [Test] public async Task DrainOnce_PoisonDrop_RemovesEventAndIncrementsFailedWithoutRetrying() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.PoisonDrop); var client = new MixpanelClient(); @@ -254,7 +280,7 @@ public async Task DrainOnce_PoisonDrop_RemovesEventAndIncrementsFailedWithoutRet [Test] public void PurgeQueuedEvents_EmptiesNonEmptySpool() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.Delivered); var client = new MixpanelClient(); @@ -295,7 +321,7 @@ public void PurgeQueuedEvents_NeverThrows_EvenWithNoSpool() [Test] public async Task Track_ExceptionEventWithUserPathInStackTrace_ScrubsBeforeSpoolingAndSending() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.Delivered); var client = new MixpanelClient(); @@ -314,7 +340,7 @@ public async Task Track_ExceptionEventWithUserPathInStackTrace_ScrubsBeforeSpool // disk can. Guard against vacuously passing (e.g. if the persisted encoding ever // changes) by requiring the scrubbed marker to actually be FOUND on disk. var scrubbedMarkerFoundOnDisk = false; - foreach (var file in EventSpoolTests.ReadableSpoolFiles(_spoolDir)) + foreach (var file in ReadableSpoolFiles(_spoolDir)) { StringAssert.DoesNotContain("alice", file.Value, "unscrubbed user path found on disk in " + file.Key); @@ -345,7 +371,7 @@ public async Task Track_ExceptionEventWithUserPathInStackTrace_ScrubsBeforeSpool [Test] public async Task Track_EventNameContainingUserPath_ScrubsEventName() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -369,7 +395,7 @@ public async Task Track_EventNameContainingUserPath_ScrubsEventName() [Test] public async Task Track_UserPathNestedInsideObjectAndArrayProperties_IsScrubbed() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -431,8 +457,8 @@ public void Track_WithoutInitialize_ThrowsInvalidOperationException() public void Track_WithNoSpool_NeverThrows() { var client = new MixpanelClient(); - // Simulates a failed Initialize (e.g. the EventSpool constructor threw acquiring the - // cross-process lock), which leaves the client with no spool at all. + // Simulates a failed Initialize (e.g. the spool's constructor threw opening its + // database), which leaves the client with no spool at all. client.InitializeForTest(null, new AlwaysResultSender(SendResult.Delivered)); Assert.DoesNotThrow(() => client.Track("user-1", "Save", null)); @@ -458,7 +484,7 @@ public void Track_WithNoSpool_NeverThrows() [Test] public void DrainOnceAndFlush_SenderAlwaysThrows_NeverPropagateOutOfMixpanelClient() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new ThrowingSender()); @@ -479,7 +505,7 @@ public void DrainOnceAndFlush_SenderAlwaysThrows_NeverPropagateOutOfMixpanelClie [Test] public void ShutDown_SenderAlwaysFails_ReturnsPromptlyAndEventsRemainOnDisk() { - var spool = new EventSpool(_spoolDir, 10); + var spool = new SqliteEventSpool(_spoolDir, 10); var sender = new AlwaysResultSender(SendResult.RetryableFailure); var client = new MixpanelClient(); client.InitializeForTest(spool, sender); @@ -494,9 +520,9 @@ public void ShutDown_SenderAlwaysFails_ReturnsPromptlyAndEventsRemainOnDisk() Assert.IsTrue(completed, "ShutDown() did not return within the timeout while offline"); Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(15)); - // ShutDown() must have released the spool's cross-process lock -- reopening it and - // finding all 5 events proves nothing was lost, and that the lock really was released. - using (var reopened = new EventSpool(_spoolDir, 10)) + // ShutDown() must have disposed the spool's connection -- reopening it and finding all + // 5 events proves nothing was lost, and that the connection really was released. + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) { Assert.AreEqual(5, reopened.ApproximateCount); } @@ -505,7 +531,7 @@ public void ShutDown_SenderAlwaysFails_ReturnsPromptlyAndEventsRemainOnDisk() [Test] public void Flush_WhileOffline_ReturnsPromptlyWithoutEmptyingSpool() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.RetryableFailure); var client = new MixpanelClient(); @@ -534,7 +560,7 @@ public void Flush_WhileOffline_ReturnsPromptlyWithoutEmptyingSpool() [Test] public void ShutDown_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() { - var spool = new EventSpool(_spoolDir, 10); + var spool = new SqliteEventSpool(_spoolDir, 10); var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); client.InitializeForTest(spool, sender); @@ -550,10 +576,10 @@ public void ShutDown_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemains "ShutDown() must be bounded even against a server that never responds"); Assert.GreaterOrEqual(sender.CallCount, 1); - // ShutDown() must have released the spool's cross-process lock -- reopening it and - // finding the event proves nothing was lost (it was rolled back, not flushed) and that - // the lock really was released. - using (var reopened = new EventSpool(_spoolDir, 10)) + // ShutDown() must have disposed the spool's connection -- reopening it and finding the + // event proves nothing was lost (it was rolled back, not flushed) and that the + // connection really was released. + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) { Assert.AreEqual(1, reopened.ApproximateCount, "the undelivered event must remain in the spool after a bounded shutdown"); @@ -563,7 +589,7 @@ public void ShutDown_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemains [Test] public void Flush_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); @@ -592,7 +618,7 @@ public void Flush_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInS [Test] public async Task ShutDownAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() { - var spool = new EventSpool(_spoolDir, 10); + var spool = new SqliteEventSpool(_spoolDir, 10); var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); client.InitializeForTest(spool, sender); @@ -607,10 +633,10 @@ public async Task ShutDownAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndE "ShutDownAsync() must be bounded even against a server that never responds"); Assert.GreaterOrEqual(sender.CallCount, 1); - // ShutDownAsync() must have released the spool's cross-process lock -- reopening it and - // finding the event proves nothing was lost (it was rolled back, not flushed) and that - // the lock really was released. - using (var reopened = new EventSpool(_spoolDir, 10)) + // ShutDownAsync() must have disposed the spool's connection -- reopening it and finding + // the event proves nothing was lost (it was rolled back, not flushed) and that the + // connection really was released. + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) { Assert.AreEqual(1, reopened.ApproximateCount, "the undelivered event must remain in the spool after a bounded shutdown"); @@ -620,7 +646,7 @@ public async Task ShutDownAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndE [Test] public async Task FlushAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEventRemainsInSpool() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); @@ -645,7 +671,7 @@ public async Task FlushAsync_SenderBlocksUntilCanceled_ReturnsWithinBoundAndEven [Test] public async Task FlushAsyncAndShutDownAsync_PreCanceledToken_SkipSendingLeaveEventsAndNeverThrow() { - var spool = new EventSpool(_spoolDir, 10); + var spool = new SqliteEventSpool(_spoolDir, 10); var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); client.InitializeForTest(spool, sender); @@ -665,8 +691,8 @@ public async Task FlushAsyncAndShutDownAsync_PreCanceledToken_SkipSendingLeaveEv Assert.AreEqual(0, sender.CallCount); } - // The lock must be released and the event still on disk for the next launch. - using (var reopened = new EventSpool(_spoolDir, 10)) + // The connection must be released and the event still on disk for the next launch. + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) { Assert.AreEqual(1, reopened.ApproximateCount); } @@ -677,7 +703,7 @@ public async Task FlushAsyncAndShutDownAsync_PreCanceledToken_SkipSendingLeaveEv [Test] public async Task ShutDownAsync_ThenSyncShutDown_IsIdempotent() { - var spool = new EventSpool(_spoolDir, 10); + var spool = new SqliteEventSpool(_spoolDir, 10); var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -686,9 +712,9 @@ public async Task ShutDownAsync_ThenSyncShutDown_IsIdempotent() await client.ShutDownAsync(); Assert.DoesNotThrow(() => client.ShutDown()); - // The spool's cross-process lock must be released (and stay released) -- reopening + // The spool's connection must be released (and stay released) -- reopening // proves it. - using (var reopened = new EventSpool(_spoolDir, 10)) + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) { Assert.AreEqual(0, reopened.ApproximateCount, "the event was delivered during the first shutdown's bounded drain"); @@ -703,7 +729,7 @@ public async Task ShutDownAsync_ThenSyncShutDown_IsIdempotent() [Test] public async Task Track_EventOverSizeCap_IsDroppedCountedFailedAndNeverReachesSender() { - using (var spool = new EventSpool(_spoolDir, 10, maxItemBytes: 500)) + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxItemBytes: 500)) { var sender = new AlwaysResultSender(SendResult.Delivered); var client = new MixpanelClient(); @@ -733,7 +759,7 @@ public async Task Track_EventOverSizeCap_IsDroppedCountedFailedAndNeverReachesSe [Test] public async Task DrainOnce_EventsExceedingPerTickByteBudget_SpreadsAcrossTicks() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.Delivered); var client = new MixpanelClient(); @@ -764,7 +790,7 @@ public async Task DrainOnce_EventsExceedingPerTickByteBudget_SpreadsAcrossTicks( [Test] public async Task DrainOnce_BatchProcessedWithSomeRecordsRejected_CountsThemFailedAndRestSucceeded() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new ScriptedSender(new Func, BatchSendResult>[] { @@ -792,7 +818,7 @@ public async Task DrainOnce_BatchProcessedWithSomeRecordsRejected_CountsThemFail [Test] public void Track_IncrementsSubmittedImmediately() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -810,7 +836,7 @@ public void Track_IncrementsSubmittedImmediately() [Test] public void Track_EventEvictedByCapEnforcement_IsCountedFailed() { - using (var spool = new EventSpool(_spoolDir, 2)) // maxItems = 2 + using (var spool = new SqliteEventSpool(_spoolDir, 2)) // maxItems = 2 { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -844,7 +870,7 @@ public void Track_EventEvictedByCapEnforcement_IsCountedFailed() [Test] public async Task DrainOnce_WithRealRetryPipelineAndZeroDelay_TransientThenSuccess_DeliversWithinOneDrainOnceCall() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new ScriptedSender(new Func, BatchSendResult>[] { @@ -871,7 +897,7 @@ public async Task DrainOnce_WithRealRetryPipelineAndZeroDelay_TransientThenSucce [Test] public async Task DrainOnce_WithRealCircuitBreakerAndZeroDelay_SustainedFailures_TripsBreakerAndStopsCallingSender() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.RetryableFailure); // MinimumThroughput=3 by default (see BuildDefaultPipeline): force enough failing @@ -908,7 +934,7 @@ public async Task DrainOnce_WithRealCircuitBreakerAndZeroDelay_SustainedFailures [Test] public async Task DrainOnce_WithRealCircuitBreakerAndFixedMinimumThroughput_OpensAfterOneFullyFailingTick() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new AlwaysResultSender(SendResult.RetryableFailure); // Regression test for the bug where MinimumThroughput=4 made the breaker inert: a @@ -953,7 +979,7 @@ public async Task DrainOnce_WithRealCircuitBreakerAndFixedMinimumThroughput_Open [Test] public async Task DrainOnce_WithRealCircuitBreaker_AfterBreakDurationElapses_HalfOpenTrialSucceedsAndClosesBreaker() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var shortBreakDuration = TimeSpan.FromMilliseconds(600); var failingSender = new AlwaysResultSender(SendResult.RetryableFailure); @@ -1026,7 +1052,7 @@ public void Initialize_WithNonEmptyHost_ThrowsArgumentException() [Test] public void PurgeQueuedEvents_ThenResumeSending_TogglesSendingPaused() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -1049,7 +1075,7 @@ public void PurgeQueuedEvents_ThenResumeSending_TogglesSendingPaused() [Test] public void Track_WhileSendingPaused_DoesNotEnqueue() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var client = new MixpanelClient(); client.InitializeForTest(spool, new AlwaysResultSender(SendResult.Delivered)); @@ -1066,14 +1092,13 @@ public void Track_WhileSendingPaused_DoesNotEnqueue() } } - // fix: consent revocation must not sit blocked behind an in-flight send (EventSpool.Purge - // and ProcessBatchAsync share one lock, held across the network await), and the batch that + // fix: consent revocation must not sit blocked behind an in-flight send, and the batch that // send was carrying must not survive to be delivered after consent was revoked -- it must // roll back and then be purged, not slip through. [Test] public async Task PurgeQueuedEvents_WhileSendInFlight_CancelsSendReturnsPromptlyAndPurges() { - using (var spool = new EventSpool(_spoolDir, 10)) + using (var spool = new SqliteEventSpool(_spoolDir, 10)) { var sender = new BlockingUntilCanceledSender(); var client = new MixpanelClient(); @@ -1110,8 +1135,8 @@ public async Task PurgeQueuedEvents_WhileSendInFlight_CancelsSendReturnsPromptly // ---- CONCURRENCY: Track() racing an in-progress drain ------------------------------------ - // Stress/regression coverage for EventSpool._sync serialization: while a drain's send is - // genuinely in flight (blocked, via BlockingUntilSignaledSender, until this test releases + // Stress/regression coverage for SqliteEventSpool's concurrency safety: while a drain's send + // is genuinely in flight (blocked, via BlockingUntilSignaledSender, until this test releases // it -- never a real Thread.Sleep-style delay), several threads call Track() concurrently. // This does not assert a specific interleaving -- only that nothing throws, nothing // deadlocks, and the bookkeeping (Submitted == Succeeded + Failed + still-spooled) stays @@ -1119,7 +1144,7 @@ public async Task PurgeQueuedEvents_WhileSendInFlight_CancelsSendReturnsPromptly [Test] public async Task ConcurrentTrack_WhileDrainInProgress_NoExceptionsAndConsistentFinalState() { - using (var spool = new EventSpool(_spoolDir, 1000)) + using (var spool = new SqliteEventSpool(_spoolDir, 1000)) { var sender = new BlockingUntilSignaledSender(); var client = new MixpanelClient(); @@ -1132,8 +1157,8 @@ public async Task ConcurrentTrack_WhileDrainInProgress_NoExceptionsAndConsistent var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). // Wait for the send to actually start, so the concurrent Track() calls below - // genuinely race an in-flight drain (EventSpool._sync held across the network await) - // rather than running before the drain even begins. + // genuinely race an in-flight drain (the send is blocked, awaiting Release()) rather + // than running before the drain even begins. var deadline = DateTime.UtcNow.AddSeconds(5); while (sender.CallCount == 0 && DateTime.UtcNow < deadline) await Task.Delay(10); @@ -1164,10 +1189,10 @@ public async Task ConcurrentTrack_WhileDrainInProgress_NoExceptionsAndConsistent foreach (var thread in threads) thread.Start(); - // Release the blocked send BEFORE joining, not after: EventSpool._sync is held for - // the whole drain (including across this send), so every one of the Track() threads - // above piles up waiting on it -- joining first would deadlock forever waiting for - // threads that can only make progress once the send (and thus the lock) is released. + // Release the blocked send BEFORE joining: DrainOnceAsync (drainTask) only completes + // once the sender unblocks, and this test's own cleanup awaits drainTask below, so + // releasing first (rather than after joining the Track() threads) avoids ever + // leaving the drain permanently blocked. sender.Release(); foreach (var thread in threads) diff --git a/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs b/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs index 49089b6..113bb3a 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs @@ -9,13 +9,12 @@ namespace DesktopAnalyticsTests { - // EventSpool (DiskQueue) holds its global _sync semaphore across the network send, so a - // UI-thread Track() call blocks for the WHOLE send -- up to ~45s against a stalled connection - // in production. SqliteEventSpool claims a batch under a short lock, releases it, sends, then - // resolves the claim under a second short lock, so Track() stays responsive throughout. - // NB: the stall is EventSpool's lock, not a DiskQueue limitation -- DiskQueue supports - // concurrent sessions (probed 2026-07-16), so dropping that lock would also fix it. See the - // class remarks on SqliteEventSpool. + // SqliteEventSpool claims a batch under a short lock, releases it, sends with nothing held, + // then resolves the claim under a second short lock, so Track() stays responsive even while a + // send is genuinely in flight -- never blocking for the whole network round trip. See the + // class remarks on SqliteEventSpool for the full rationale (including the now-removed + // DiskQueue-backed EventSpool's contrasting behavior, which this used to characterize + // alongside the fix). [TestFixture] public class TrackResponsivenessTests { @@ -119,62 +118,5 @@ public async Task Track_WhileSendInFlight_ReturnsImmediately_Sqlite() Assert.AreEqual(2, client.Statistics.Submitted); } } - - // ---- THE DEFECT (characterization, not a requirement): EventSpool blocks --------------- - // - // This test PASSES against today's EventSpool behavior on purpose -- it pins the documented - // defect (see MixpanelClientTests.ConcurrentTrack_WhileDrainInProgress_..., whose comments - // describe Track() threads "piling up" behind the lock) so it is visible in the test suite - // rather than hidden in a comment. Delete it if EventSpool is dropped -- or if EventSpool's - // global lock is narrowed so it stops blocking, in which case this test should start failing - // and be deleted rather than "fixed". - [Test] - public async Task Track_WhileSendInFlight_BlocksUntilSendCompletes_DiskQueue() - { - using (var spool = new EventSpool(_spoolDir, 1000)) - { - var sender = new BlockingUntilSignaledSender(); - var client = new MixpanelClient(); - client.InitializeForTest(spool, sender, batchSize: 5); - - client.Track("user-1", "Seed", null); - - var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). - await WaitForSendToStart(sender); - - // Run Track() on its own thread so this test can observe it NOT finishing while the - // send is blocked, rather than deadlocking the test itself. - var trackFinished = new ManualResetEventSlim(false); - var trackThread = new Thread(() => - { - client.Track("user-1", "WhileSendInFlight", null); - trackFinished.Set(); - }); - trackThread.Start(); - - // EventSpool's transactional session holds its lock across the whole send, so - // Track() must still be blocked after a short window -- this is the defect, pinned. - var finishedWhileBlocked = trackFinished.Wait(TimeSpan.FromMilliseconds(300)); - Assert.IsFalse(finishedWhileBlocked, - "characterizes today's known defect: EventSpool holds its lock across the " + - "network send, so Track() blocks for the whole send instead of returning " + - "promptly. If this ever fails, EventSpool has stopped blocking Track() and this " + - "test (and the bug it documents) should be deleted, not fixed."); - - // Release the blocked send so the Track() thread (and the drain) can finally - // complete -- otherwise this test would hang forever. - sender.Release(); - - Assert.IsTrue(trackFinished.Wait(TimeSpan.FromSeconds(5)), - "Track() must complete once the send is released"); - trackThread.Join(); - await drainTask; - - Assert.AreEqual(1, client.Statistics.Succeeded); - await client.DrainOnceAsync(); - Assert.AreEqual(0, spool.ApproximateCount); - Assert.AreEqual(2, client.Statistics.Submitted); - } - } } } From 751f3c895f0853fc13b8797805ff4af9748b9fe0 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 20:51:03 -0400 Subject: [PATCH 07/11] Add async-first TrackAsync/ReportExceptionAsync (Phase 4) Adds TrackAsync to IClient/SegmentClient/MixpanelClient and TrackAsync/ReportExceptionAsync to the Analytics facade, following the same Task.CompletedTask-wrapping-a-fast-sync-call pattern the codebase already uses for ShutDownAsync/FlushAsync. Purely additive -- every existing synchronous method is unchanged. Honest about what this is and isn't: Microsoft.Data.Sqlite does not implement true async I/O, and MixpanelClient.Track() already completes in single-digit milliseconds, so TrackAsync is API ergonomics for callers that await pervasively (FW Lite), not a concurrency fix. IEventSpool.EnqueueAsync is added for interface completeness but isn't currently called by production code, which the code comments say explicitly. Verified with a real timing test, not just green tests: a new TrackAsync_WhileSendInFlight_ReturnsImmediately_Sqlite test measures TrackAsync completing in ~0.8ms while a send is genuinely blocked in-flight, mirroring the existing sync responsiveness test. 103/103 Mixpanel tests green on both TFMs; core (19+14) unaffected. Co-Authored-By: Claude Opus 4.8 --- src/DesktopAnalytics.Mixpanel/IEventSpool.cs | 11 ++ .../MixpanelClient.cs | 14 +++ .../SqliteEventSpool.cs | 3 + src/DesktopAnalytics/Analytics.cs | 115 ++++++++++++++++++ src/DesktopAnalytics/IClient.cs | 11 ++ src/DesktopAnalytics/SegmentClient.cs | 12 ++ .../TrackResponsivenessTests.cs | 42 +++++++ 7 files changed, 208 insertions(+) diff --git a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs index 8a97960..8fdf964 100644 --- a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs @@ -74,6 +74,17 @@ internal interface IEventSpool : IDisposable /// unserializable, over the per-event byte cap, or a disk failure). bool Enqueue(AnalyticsEvent evt); + /// + /// Async counterpart of , added for API completeness as part of the + /// async-first work (see offline-analytics-v2-plan.md, Phase 4). Note: MixpanelClient + /// .TrackAsync calls the synchronous path directly and does not go + /// through this method -- it is not currently called by anything in this codebase. + /// Implementations complete synchronously: a local SQLite insert here is already + /// sub-millisecond, and Microsoft.Data.Sqlite does not implement true async I/O anyway, so + /// there is no real non-blocking work to represent. + /// + Task EnqueueAsync(AnalyticsEvent evt); + /// /// Gathers up to events (oldest first, bounded by /// ), hands them to as ONE batch, and diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index 13a969e..60cc8d4 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -552,6 +552,20 @@ public void Track(string analyticsId, string eventName, JsonObject properties) } } + /// + /// Completes synchronously: already returns in single-digit + /// milliseconds (a synchronous SQLite insert -- Microsoft.Data.Sqlite does not implement + /// true async I/O, so its async ADO.NET methods would not make this any less blocking). + /// This exists purely for API ergonomics for callers (e.g. FW Lite) that await pervasively, + /// not to make anything genuinely non-blocking that wasn't already fast. + /// + public Task TrackAsync(string analyticsId, string eventName, JsonObject properties, + CancellationToken cancellationToken = default) + { + Track(analyticsId, eventName, properties); + return Task.CompletedTask; + } + private static JsonObject ScrubProperties(JsonObject properties) { var result = new JsonObject(); diff --git a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs index 8b5eae4..a131cd9 100644 --- a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs @@ -300,6 +300,9 @@ public bool Enqueue(AnalyticsEvent evt) } } + /// + public Task EnqueueAsync(AnalyticsEvent evt) => Task.FromResult(Enqueue(evt)); + private void RaiseDropped(int count) { for (var i = 0; i < count; i++) diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index dc866d4..ac2f661 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -676,6 +676,46 @@ public static void Track(string eventName, Dictionary properties TrackWithApplicationProperties(eventName, MakeSegmentIOProperties(properties)); } + /// + /// Async counterpart of , for callers (e.g. FW Lite) that await + /// pervasively. See for why this need not (and, for + /// MixpanelClient, does not) represent genuinely non-blocking work -- it is API + /// ergonomics, not a change to when/how the event is sent. + /// + /// A good event name should be meaningful to a developer or + /// analyst, relatively short, and unique within an app. + /// Passed through to the underlying client; never faults + /// the task. + [PublicAPI] + public static Task TrackAsync(string eventName, CancellationToken cancellationToken = default) + { + if (!AllowTracking) + return Task.CompletedTask; + + return TrackWithApplicationPropertiesAsync(eventName, null, cancellationToken); + } + + /// + /// Async counterpart of . See + /// for the async-ergonomics rationale. + /// + /// A good event name should be meaningful to a developer or + /// analyst, relatively short, and unique within an app. + /// A dictionary of key-value pairs that are relevant in + /// understanding more about the event being tracked. + /// Passed through to the underlying client; never faults + /// the task. + [PublicAPI] + public static Task TrackAsync(string eventName, Dictionary properties, + CancellationToken cancellationToken = default) + { + if (!AllowTracking) + return Task.CompletedTask; + + return TrackWithApplicationPropertiesAsync(eventName, MakeSegmentIOProperties(properties), + cancellationToken); + } + /// /// Sends the exception's message and stacktrace /// @@ -686,6 +726,16 @@ public static void ReportException(Exception e) ReportException(e, null); } + /// + /// Async counterpart of . See + /// for the async-ergonomics rationale. + /// + [PublicAPI] + public static Task ReportExceptionAsync(Exception e, CancellationToken cancellationToken = default) + { + return ReportExceptionAsync(e, null, cancellationToken); + } + /// /// Sends the exception's message and stacktrace, plus additional information the /// program thinks may be relevant. Limited to MAX_EXCEPTION_REPORTS_PER_RUN @@ -717,6 +767,41 @@ public static void ReportException(Exception e, Dictionary moreP TrackWithApplicationProperties("Exception", props); } + /// + /// Async counterpart of , + /// including the same MAX_EXCEPTION_REPORTS_PER_RUN throttling (the counter is shared with + /// the sync path). See for the + /// async-ergonomics rationale. + /// + [PublicAPI] + public static Task ReportExceptionAsync(Exception e, Dictionary moreProperties, + CancellationToken cancellationToken = default) + { + if (!AllowTracking) + return Task.CompletedTask; + + s_exceptionCount++; + + // we had an incident where some problem caused a user to emit hundreds of thousands of exceptions, + // in the background, blowing through our Analytics service limits and getting us kicked off. + if (s_exceptionCount > kMaxExceptionReportsPerRun) + { + return Task.CompletedTask; + } + + var props = new JsonObject + { + { "Message", e.Message }, + { "Stack Trace", e.StackTrace } + }; + if (moreProperties != null) + { + foreach (var key in moreProperties.Keys) + props.Add(key, moreProperties[key]); + } + return TrackWithApplicationPropertiesAsync("Exception", props, cancellationToken); + } + private static JsonObject MakeSegmentIOProperties(Dictionary properties) { var prop = new JsonObject(); @@ -1100,6 +1185,36 @@ private static void TrackWithApplicationProperties(string eventName, JsonObject ); } + /// + /// Async counterpart of : same defaulting/merge + /// logic, routed through instead of . + /// + private static Task TrackWithApplicationPropertiesAsync(string eventName, + JsonObject properties = null, CancellationToken cancellationToken = default) + { + if (s_singleton == null) + throw new ApplicationException("The application must first construct a single Analytics object"); + + if (!AllowTracking) + return Task.CompletedTask; + + if (properties == null) + properties = new JsonObject(); + + foreach (var p in s_singleton._propertiesThatGoWithEveryEvent) + { + properties.Remove(p.Key); + properties.Add(p.Key, p.Value ?? Empty); + } + + return s_singleton._client.TrackAsync( + s_settings.IdForAnalytics, + eventName, + properties, + cancellationToken + ); + } + /// /// Add a property that says something about the application, which goes out with every event. /// diff --git a/src/DesktopAnalytics/IClient.cs b/src/DesktopAnalytics/IClient.cs index 5a53ccc..61198d5 100644 --- a/src/DesktopAnalytics/IClient.cs +++ b/src/DesktopAnalytics/IClient.cs @@ -19,6 +19,17 @@ public interface IClient void Track(string defaultIdForAnalytics, string eventName, JsonObject properties); void Flush(); + /// + /// Async counterpart of . Implementations whose Track already + /// completes fast (see SegmentClient/MixpanelClient) may complete + /// synchronously -- this exists for API ergonomics for callers (e.g. FW Lite) that await + /// pervasively, not to change how/when the event is sent. Shares 's + /// contract: must not throw once the client is initialized (implementations that call Track + /// before Initialize may still throw, matching Track's own documented behavior). + /// + Task TrackAsync(string defaultIdForAnalytics, string eventName, JsonObject properties, + CancellationToken cancellationToken = default); + /// /// Async counterpart of . Implementations whose shutdown has nothing /// awaitable (see ) may complete synchronously; implementations diff --git a/src/DesktopAnalytics/SegmentClient.cs b/src/DesktopAnalytics/SegmentClient.cs index 5d56d3a..69c8231 100644 --- a/src/DesktopAnalytics/SegmentClient.cs +++ b/src/DesktopAnalytics/SegmentClient.cs @@ -74,6 +74,18 @@ public void Flush() _analytics.Flush(); } + /// + /// Completes synchronously: already just hands the event to + /// Segment.Analytics.CSharp's own fire-and-forget delivery pipeline and returns, so there is + /// no async work to represent here. + /// + public Task TrackAsync(string defaultIdForAnalytics, string eventName, JsonObject properties, + CancellationToken cancellationToken = default) + { + Track(defaultIdForAnalytics, eventName, properties); + return Task.CompletedTask; + } + /// /// Completes synchronously: Segment.Analytics.CSharp's Flush() only signals the /// library's own background delivery (its coroutine system) and exposes nothing awaitable, diff --git a/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs b/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs index 113bb3a..413372f 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/TrackResponsivenessTests.cs @@ -118,5 +118,47 @@ public async Task Track_WhileSendInFlight_ReturnsImmediately_Sqlite() Assert.AreEqual(2, client.Statistics.Submitted); } } + + // ---- Same requirement, through the new async entry point (Phase 4) ---------------------- + + [Test] + public async Task TrackAsync_WhileSendInFlight_ReturnsImmediately_Sqlite() + { + using (var spool = new SqliteEventSpool(_spoolDir, 1000)) + { + var sender = new BlockingUntilSignaledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, batchSize: 5); + + await client.TrackAsync("user-1", "Seed", null); + + var drainTask = client.DrainOnceAsync(); // Blocks inside the sender until Release(). + await WaitForSendToStart(sender); + + var stopwatch = Stopwatch.StartNew(); + await client.TrackAsync("user-1", "WhileSendInFlight", null); + stopwatch.Stop(); + + // TrackAsync wraps the same synchronous Track() (see MixpanelClient.TrackAsync's + // doc comment: Task.CompletedTask, no real async work), so it must be just as + // responsive as the sync path above -- same 500ms headroom, same rationale. + Assert.Less(stopwatch.Elapsed, TimeSpan.FromMilliseconds(500), + "TrackAsync() must complete promptly even while a send is genuinely in flight -- " + + "it wraps the same non-blocking Track() as the sync entry point"); + + sender.Release(); + await drainTask; + + // Responsiveness must not come from silently dropping the event: it must have been + // spooled (and, once the drain completes, delivered). + Assert.AreEqual(1, client.Statistics.Succeeded, + "the seed event must have been delivered by the drain this test released"); + await client.DrainOnceAsync(); + Assert.AreEqual(0, spool.ApproximateCount, + "the TrackAsync() call made while the send was in flight must have been spooled, " + + "not dropped, and must drain cleanly afterward"); + Assert.AreEqual(2, client.Statistics.Submitted); + } + } } } From f080b9b2f66f11839acdf131e316e60d45b48c49 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 21:24:36 -0400 Subject: [PATCH 08/11] Add per-event retry-attempt ceiling (Phase 5 / D6) Adds an "attempts" column and a maxAttempts cap (default 10) so a single event that keeps failing cannot wedge the head of the queue forever, per hahn-kev's PR #43 review comment. First pass conflated two very different failures under the existing SendResult.RetryableFailure: genuine connectivity loss (offline, circuit breaker open, timeout) and a definite server-side rejection (408/429/5xx). Counting both against the same budget meant a few minutes offline could permanently drop good queued events -- caught by 3 existing tests (ShutDown/Flush/Drain "while offline") before this shipped. Fix: split out a new SendResult.RetryableRejection for the case where a response actually came back from Mixpanel telling us to retry later. Only that counts against attempts. Plain connectivity failure (RetryableFailure) releases the lease as before but never touches the counter, however many times it's retried -- the 60-day age floor remains the sole backstop for "offline for a long time." Delivered/PoisonDrop never touch attempts either. The 3 previously-broken tests pass unmodified. 112/112 Mixpanel tests green on both TFMs (was 102); core (19+14) unaffected. Co-Authored-By: Claude Opus 4.8 --- src/DesktopAnalytics.Mixpanel/IEventSender.cs | 21 +- src/DesktopAnalytics.Mixpanel/IEventSpool.cs | 37 +- .../MixpanelClient.cs | 21 +- .../MixpanelEventSender.cs | 6 +- .../SqliteEventSpool.cs | 163 +++++++-- .../MixpanelClientTests.cs | 83 +++++ .../MixpanelEventSenderTests.cs | 15 +- .../SqliteEventSpoolRetryTests.cs | 323 ++++++++++++++++++ 8 files changed, 626 insertions(+), 43 deletions(-) create mode 100644 src/DesktopAnalyticsTests.Mixpanel/SqliteEventSpoolRetryTests.cs diff --git a/src/DesktopAnalytics.Mixpanel/IEventSender.cs b/src/DesktopAnalytics.Mixpanel/IEventSender.cs index d19d04b..ef2580c 100644 --- a/src/DesktopAnalytics.Mixpanel/IEventSender.cs +++ b/src/DesktopAnalytics.Mixpanel/IEventSender.cs @@ -7,11 +7,12 @@ namespace DesktopAnalytics /// /// The outcome of attempting to deliver one batch of events. Mixpanel's /import endpoint /// yields exactly two shapes of outcome, and this type models both: either the whole request - /// failed to be processed ( is - /// or , applying to every event in the batch), or the - /// request WAS processed () with zero or more individual - /// records rejected as permanently invalid ( -- /import's - /// failed_records; the remainder were ingested). + /// failed to be processed ( is , + /// , or , + /// applying to every event in the batch), or the request WAS processed + /// () with zero or more individual records rejected as + /// permanently invalid ( -- /import's failed_records; the + /// remainder were ingested). /// internal class BatchSendResult { @@ -20,9 +21,11 @@ internal class BatchSendResult /// Whole-batch verdict. means the request was /// processed and every event is finished with (ingested, or rejected-and-reported in /// ) -- the caller should remove the whole batch from the - /// spool. means nothing was processed and the - /// whole batch should be retried later. means the - /// whole batch was permanently rejected (e.g. an unexpected 4xx) and should be dropped. + /// spool. and both mean nothing was processed and the whole batch should be + /// retried later -- they differ only in whether the round trip actually reached the server + /// (see each value's own doc). means the whole batch + /// was permanently rejected (e.g. an unexpected 4xx) and should be dropped. public SendResult Outcome { get; } /// When is : the @@ -38,6 +41,8 @@ public BatchSendResult(SendResult outcome, IReadOnlyCollection failedIndice public static readonly BatchSendResult Delivered = new BatchSendResult(SendResult.Delivered); public static readonly BatchSendResult Retryable = new BatchSendResult(SendResult.RetryableFailure); + public static readonly BatchSendResult RetryableRejection = + new BatchSendResult(SendResult.RetryableRejection); public static readonly BatchSendResult Poison = new BatchSendResult(SendResult.PoisonDrop); } diff --git a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs index 8fdf964..55e28fc 100644 --- a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs @@ -9,9 +9,14 @@ namespace DesktopAnalytics /// The outcome of attempting to deliver one batch of spooled events, as reported by the caller /// of . This drives whether the batch is removed /// from the spool (/) or left in place for a - /// later retry (). (Per-record rejections within a processed - /// batch are the sender/client's concern -- see -- - /// and count as here: the batch is finished with either way.) + /// later retry ( or ). Both + /// retry outcomes release the batch's lease the same way; they differ ONLY in whether the + /// retry is counted against the per-event attempt ceiling (see + /// SqliteEventSpool.ResolveClaim and offline-analytics-v2-plan.md, Phase 5 / decision + /// D6) -- see each member's doc for the line between them. (Per-record rejections within a + /// processed batch are the sender/client's concern -- see + /// -- and count as here: + /// the batch is finished with either way.) /// internal enum SendResult { @@ -19,10 +24,24 @@ internal enum SendResult /// where the rejected records are individually reported). Remove it from the spool. Delivered, - /// A transient failure (connection error, timeout, 5xx, 429). Leave the batch in - /// the spool for a later retry. + /// A connectivity-level failure: the round trip could not be completed at all, so + /// nothing is known about this batch's actual deliverability -- a connection error, DNS + /// failure, timeout, a captive-portal-style 3xx interception, a bounded-drain + /// cancellation, or a circuit breaker sitting open from a prior outage. Leave the batch in + /// the spool for a later retry, and (unlike ) do NOT count + /// it against the retry-attempt ceiling: being offline, however long, must never erode + /// that budget -- the age-based retention floor is the correct, sole backstop for an + /// offline stretch. RetryableFailure, + /// The batch DID reach the server and got back a definite "try again later" (HTTP + /// 408, 429, or 5xx). Leave the batch in the spool for a later retry, same as + /// , but DOES count against the retry-attempt ceiling: a + /// batch that keeps coming back this way is failing for a reason other than plain + /// connectivity, which is exactly the "stuck retrying a single bad event forever" case the + /// ceiling exists to bound. + RetryableRejection, + /// A non-retryable failure (e.g. a 4xx that will never succeed). Remove the batch /// from the spool anyway so bad events cannot wedge the spool forever. PoisonDrop @@ -58,6 +77,14 @@ internal interface IEventSpool : IDisposable /// event Action ItemDroppedByCap; + /// + /// Raised once per event permanently dropped for exceeding the maximum retry attempt count + /// (see offline-analytics-v2-plan.md, Phase 5 / decision D6) -- distinct from + /// , which is capacity-based eviction, not a delivery failure. + /// Handlers must be fast and must not call back into the spool. + /// + event Action ItemDroppedByRetryExhaustion; + /// An approximate count of events currently spooled. Accurate enough for /// bounding and diagnostics; not a hard guarantee under concurrent access. int ApproximateCount { get; } diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index 60cc8d4..43d00a4 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -71,6 +71,13 @@ public class MixpanelClient : IClient // since MixpanelEventSender uses /import, which has no such limit). private const int kDefaultMaxSpoolAgeDays = 60; + // Per-event retry ceiling (offline-analytics-v2-plan.md, Phase 5 / decision D6): a single + // event that keeps coming back as RetryableFailure (e.g. a bug in serialization that never + // quite hits the PoisonDrop path) would otherwise wedge the head of the queue behind it + // forever -- previously bounded only by the 60-day age floor above, i.e. up to 60 days of + // pointless retries. + private const int kDefaultMaxRetryAttempts = 10; + // On (re)start, attempt the first drain soon rather than waiting a full interval, so events // spooled during a PREVIOUS (offline) session go out shortly after launch instead of ~30s later. private const int kInitialFlushDelaySeconds = 3; @@ -158,8 +165,9 @@ public void Initialize(string apiSecret, string host = null, int flushAt = -1, i _timeProvider = TimeProvider.System; _spool = new SqliteEventSpool(SqliteEventSpool.GetDefaultSpoolPath(apiSecret), kDefaultMaxSpoolItems, kMaxSpooledEventBytes, kDefaultMaxSpoolBytes, - timeProvider: _timeProvider); + timeProvider: _timeProvider, maxAttempts: kDefaultMaxRetryAttempts); _spool.ItemDroppedByCap += OnItemDroppedByCap; + _spool.ItemDroppedByRetryExhaustion += OnItemDroppedByRetryExhaustion; _sender = new MixpanelEventSender(apiSecret); _pipeline = BuildDefaultPipeline(_timeProvider); @@ -204,7 +212,10 @@ internal void InitializeForTest( _initialized = true; _spool = spool; if (_spool != null) + { _spool.ItemDroppedByCap += OnItemDroppedByCap; + _spool.ItemDroppedByRetryExhaustion += OnItemDroppedByRetryExhaustion; + } _sender = sender; _timeProvider = timeProvider ?? TimeProvider.System; _pipeline = pipeline ?? ResiliencePipeline.Empty; @@ -219,6 +230,14 @@ private void OnItemDroppedByCap() Interlocked.Increment(ref _failed); } + // Same statistics effect as cap eviction (see OnItemDroppedByCap above): an event dropped + // for exceeding the retry ceiling will also never be delivered. See + // IEventSpool.ItemDroppedByRetryExhaustion. + private void OnItemDroppedByRetryExhaustion() + { + Interlocked.Increment(ref _failed); + } + /// /// Builds the production Polly pipeline: a small bounded retry (exponential backoff + /// jitter) for quick transient blips, wrapped in a circuit breaker so a sustained outage backs diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs b/src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs index edfc3c7..2ad3d1a 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelEventSender.cs @@ -167,7 +167,11 @@ public async Task SendBatchAsync(IReadOnlyList } if (status == 408 || status == 429 || status >= 500) - return BatchSendResult.Retryable; + // Unlike the network/timeout/redirect cases above, we DID reach Mixpanel and + // it gave a definite "try again later" -- counts against the retry-attempt + // ceiling (see SendResult.RetryableRejection), unlike the connectivity-style + // failures elsewhere in this method. + return BatchSendResult.RetryableRejection; if (status == 400) { diff --git a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs index a131cd9..56f2822 100644 --- a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs @@ -56,6 +56,7 @@ internal class SqliteEventSpool : IEventSpool private readonly int _maxItems; private readonly int _maxItemBytes; private readonly long _maxSpoolBytes; + private readonly int _maxAttempts; private readonly string _spoolDirectory; private readonly string _databasePath; private readonly TimeProvider _timeProvider; @@ -69,6 +70,9 @@ internal class SqliteEventSpool : IEventSpool /// public event Action ItemDroppedByCap; + /// + public event Action ItemDroppedByRetryExhaustion; + /// Directory to hold the spool database. Callers normally get /// this from ; tests inject a temp directory. /// Maximum number of events retained; enqueuing beyond this drops @@ -78,11 +82,18 @@ internal class SqliteEventSpool : IEventSpool /// Maximum total serialized size of all retained events; /// enqueuing beyond this drops the oldest first. /// Clock for lease expiry. Injected so tests stay deterministic. + /// Maximum number of + /// verdicts a single event may accumulate before it is permanently dropped (see + /// offline-analytics-v2-plan.md, Phase 5 / decision D6) and + /// is raised for it. Bounds the case where a + /// single poison event -- one that always fails but never classifies as + /// -- would otherwise wedge the head of the queue + /// forever. /// Propagates a failure to create/open the database. This is /// deliberately not swallowed: it is a startup-time condition the caller needs to know /// about. public SqliteEventSpool(string spoolDirectory, int maxItems, int maxItemBytes = int.MaxValue, - long maxSpoolBytes = long.MaxValue, TimeProvider timeProvider = null) + long maxSpoolBytes = long.MaxValue, TimeProvider timeProvider = null, int maxAttempts = 10) { if (maxItems < 0) throw new ArgumentOutOfRangeException(nameof(maxItems)); @@ -90,10 +101,13 @@ public SqliteEventSpool(string spoolDirectory, int maxItems, int maxItemBytes = throw new ArgumentOutOfRangeException(nameof(maxItemBytes)); if (maxSpoolBytes <= 0) throw new ArgumentOutOfRangeException(nameof(maxSpoolBytes)); + if (maxAttempts < 0) + throw new ArgumentOutOfRangeException(nameof(maxAttempts)); _maxItems = maxItems; _maxItemBytes = maxItemBytes; _maxSpoolBytes = maxSpoolBytes; + _maxAttempts = maxAttempts; _spoolDirectory = spoolDirectory; _databasePath = Path.Combine(spoolDirectory, "spool.db"); _timeProvider = timeProvider ?? TimeProvider.System; @@ -138,7 +152,8 @@ private static void InitializeSchema(SqliteConnection connection) " time_unix_ms INTEGER NOT NULL," + " lease_until_unix_ms INTEGER NOT NULL DEFAULT 0," + " len INTEGER NOT NULL," + - " payload BLOB NOT NULL);"); + " payload BLOB NOT NULL," + + " attempts INTEGER NOT NULL DEFAULT 0);"); Execute(connection, "CREATE INDEX IF NOT EXISTS ix_events_time ON events(time_unix_ms);"); Execute(connection, "CREATE INDEX IF NOT EXISTS ix_events_lease ON events(lease_until_unix_ms, id);"); @@ -318,6 +333,21 @@ private void RaiseDropped(int count) } } + private void RaiseRetryExhausted(int count) + { + for (var i = 0; i < count; i++) + { + try + { + ItemDroppedByRetryExhaustion?.Invoke(); + } + catch (Exception e) + { + Debug.WriteLine("SqliteEventSpool: ItemDroppedByRetryExhaustion handler threw: " + e); + } + } + } + // Caller must hold _sync. Brings the spool back within BOTH caps, dropping oldest first, // and returns how many events were dropped. Two DELETEs, no loop -- contrast the old // DiskQueue-backed EventSpool.EnforceCapLocked, which dequeued one item at a time. @@ -386,7 +416,7 @@ public async Task ProcessBatchAsync(int maxItems, { // Everything claimed was undeserializable (corrupt, or an incompatible schema // version). Nothing to send; just remove them so they cannot wedge the spool. - ResolveClaim(claim.Ids, remove: true); + ResolveClaim(claim.Ids, ClaimOutcome.Remove); return; } @@ -398,16 +428,15 @@ public async Task ProcessBatchAsync(int maxItems, } catch (Exception e) { + // A throwing send is a connectivity-style failure (we don't know anything about + // this batch's deliverability), exactly like SendResult.RetryableFailure -- see + // ResolveOutcome. Must NOT count against the retry-attempt ceiling. Debug.WriteLine("SqliteEventSpool.ProcessBatch: send threw, leaving batch for retry: " + e); - ResolveClaim(claim.Ids, remove: false); + ResolveClaim(claim.Ids, ClaimOutcome.ReleaseOnly); return; } - // Delivered/PoisonDrop: the batch is finished with either way, so remove it. - // RetryableFailure: release the lease so the next drain picks it up, in order. - ResolveClaim(claim.Ids, - remove: result == SendResult.Delivered || result == SendResult.PoisonDrop, - undeserializableIds: claim.UndeserializableIds); + ResolveClaim(claim.Ids, ResolveOutcome(result), undeserializableIds: claim.UndeserializableIds); } catch (Exception e) { @@ -415,6 +444,43 @@ public async Task ProcessBatchAsync(int maxItems, } } + // What ResolveClaim should do with a resolved (non-empty-events) batch. Distinct from + // SendResult because Delivered/PoisonDrop collapse to the same DELETE, and because the two + // "leave it spooled" verdicts differ in whether they touch the attempts counter -- see + // each member's doc and SendResult.RetryableFailure/RetryableRejection. + private enum ClaimOutcome + { + // Delivered or PoisonDrop (or every claimed row was undeserializable): the batch is + // finished with either way. Straight DELETE; attempts is never touched. + Remove, + + // SendResult.RetryableFailure, or send() throwing: a connectivity-style failure -- we + // don't know anything about this batch's actual deliverability, only that the round + // trip didn't complete. Release the lease so the next drain retries it, in order, but + // do NOT touch attempts: being offline, however long, must never erode the retry + // budget (the age-based retention floor is the correct, sole backstop for that). + ReleaseOnly, + + // SendResult.RetryableRejection: the batch reached the server and got a definite "try + // again later". Release the lease AND bump attempts, dropping any event whose count + // just reached the configured maximum. + CountAttemptAndRelease + } + + private static ClaimOutcome ResolveOutcome(SendResult result) + { + switch (result) + { + case SendResult.Delivered: + case SendResult.PoisonDrop: + return ClaimOutcome.Remove; + case SendResult.RetryableRejection: + return ClaimOutcome.CountAttemptAndRelease; + default: // SendResult.RetryableFailure + return ClaimOutcome.ReleaseOnly; + } + } + private class ClaimedBatch { public readonly List Ids = new List(); @@ -504,10 +570,17 @@ private ClaimedBatch ClaimBatch(int maxItems, long maxBytes) return claim; } - // Short, purely-local transaction closing out a claim: either remove the rows (the batch is - // finished with) or clear their lease so the next drain retries them in order. - private void ResolveClaim(List ids, bool remove, List undeserializableIds = null) + // Short, purely-local transaction closing out a claim. ClaimOutcome.Remove: straight + // DELETE. Otherwise: clear the lease so the next drain retries the batch, in order -- and, + // ONLY for CountAttemptAndRelease, also bump the attempt counter and drop any event whose + // count just reached the configured max (dropped NOW, not on some future failure, since + // the check runs after the increment). + private void ResolveClaim(List ids, ClaimOutcome outcome, List undeserializableIds = null) { + // Populated inside the lock below, then used to raise ItemDroppedByRetryExhaustion + // AFTER the lock is released -- same pattern Enqueue uses for ItemDroppedByCap. + List retryExhaustedIds = null; + try { lock (_sync) @@ -517,15 +590,19 @@ private void ResolveClaim(List ids, bool remove, List undeserializab using (var txn = _connection.BeginTransaction(deferred: false)) { - if (remove) + if (outcome == ClaimOutcome.Remove) { + // Delivered/PoisonDrop path. A straight DELETE -- attempts is never + // touched here, so a batch that ultimately succeeds (even after some + // number of prior retryable verdicts of either kind) never counts + // against the retry ceiling. DeleteByIdLocked(txn, ids); } else { - // Undeserializable rows are removed even on a retryable verdict: they - // can never be delivered, so retrying them forever would wedge the - // head of the queue. + // Undeserializable rows are removed regardless of which retryable + // verdict this is: they can never be delivered, so retrying them + // forever would wedge the head of the queue. if (undeserializableIds != null && undeserializableIds.Count > 0) DeleteByIdLocked(txn, undeserializableIds); @@ -535,13 +612,50 @@ private void ResolveClaim(List ids, bool remove, List undeserializab if (toRelease.Count > 0) { - using (var cmd = _connection.CreateCommand()) + if (outcome == ClaimOutcome.CountAttemptAndRelease) + { + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "UPDATE events SET attempts = attempts + 1, lease_until_unix_ms = 0 " + + "WHERE id IN (" + IdList(toRelease) + ");"; + cmd.ExecuteNonQuery(); + } + + // Must run AFTER the increment above so an event whose attempts + // just reached maxAttempts on THIS failure is caught + // immediately, not on some future failure. + retryExhaustedIds = new List(); + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "SELECT id FROM events WHERE id IN (" + IdList(toRelease) + ") " + + "AND attempts >= @maxAttempts;"; + cmd.Parameters.AddWithValue("@maxAttempts", _maxAttempts); + using (var reader = cmd.ExecuteReader()) + { + while (reader.Read()) + retryExhaustedIds.Add(reader.GetInt64(0)); + } + } + + if (retryExhaustedIds.Count > 0) + DeleteByIdLocked(txn, retryExhaustedIds); + } + else { - cmd.Transaction = txn; - cmd.CommandText = - "UPDATE events SET lease_until_unix_ms = 0 WHERE id IN (" + - IdList(toRelease) + ");"; - cmd.ExecuteNonQuery(); + // ReleaseOnly: connectivity-style failure -- attempts is + // deliberately left untouched (see ClaimOutcome.ReleaseOnly). + using (var cmd = _connection.CreateCommand()) + { + cmd.Transaction = txn; + cmd.CommandText = + "UPDATE events SET lease_until_unix_ms = 0 WHERE id IN (" + + IdList(toRelease) + ");"; + cmd.ExecuteNonQuery(); + } } } } @@ -553,7 +667,12 @@ private void ResolveClaim(List ids, bool remove, List undeserializab catch (Exception e) { Debug.WriteLine("SqliteEventSpool.ResolveClaim failed: " + e); + return; } + + // Raised outside the lock: handlers are the caller's code (see RaiseDropped). + if (retryExhaustedIds != null && retryExhaustedIds.Count > 0) + RaiseRetryExhausted(retryExhaustedIds.Count); } private void DeleteByIdLocked(SqliteTransaction txn, List ids) diff --git a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs index 16121a2..3a3f3ec 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs @@ -500,6 +500,89 @@ public void DrainOnceAndFlush_SenderAlwaysThrows_NeverPropagateOutOfMixpanelClie } } + // ---- RETRY-ATTEMPT CEILING VS. PLAIN CONNECTIVITY FAILURE ------------------------------- + // + // Regression coverage for the gap found wiring up the Phase 5 retry-attempt ceiling: a + // naive "increment attempts on every RetryableFailure" implementation would also erode the + // budget on plain offline stretches (SendResult.RetryableFailure), not just on a genuine + // per-event server rejection (SendResult.RetryableRejection) -- e.g. dropping queued events + // after only ~5 minutes offline at the default flush cadence, or within a single offline + // ShutDown()/Flush() call (BoundedDrain's internal tight retry loop). These prove the two + // SendResult values are handled distinctly all the way through MixpanelClient. + + [Test] + public async Task DrainRepeatedly_SenderAlwaysRetryableRejection_EventDroppedAfterMaxAttempts() + { + const int maxAttempts = 5; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + var sender = new AlwaysResultSender(SendResult.RetryableRejection); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + var exhaustedCount = 0; + // ItemDroppedByRetryExhaustion is on IEventSpool; MixpanelClient itself only + // exposes it via Statistics.Failed, so subscribe directly on the spool here to + // pinpoint exactly when the drop happens. + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + client.Track("user-1", "Save", null); + + // maxAttempts - 1 = 4 drains: a genuine server rejection each time, but not yet + // enough to hit the ceiling. + for (var i = 0; i < maxAttempts - 1; i++) + await client.DrainOnceAsync(); + + Assert.AreEqual(1, spool.ApproximateCount, + "after " + (maxAttempts - 1) + " RetryableRejection verdicts (< maxAttempts = " + + maxAttempts + "), the event must still be spooled"); + Assert.AreEqual(0, exhaustedCount); + + // The 5th (== maxAttempts) drain must drop it. + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, + "the " + maxAttempts + "th RetryableRejection must drop the event"); + Assert.AreEqual(1, exhaustedCount, + "ItemDroppedByRetryExhaustion must fire exactly once"); + Assert.AreEqual(1, client.Statistics.Failed, + "a retry-exhaustion drop must count as Failed, same as cap eviction"); + } + } + + [Test] + public async Task DrainRepeatedly_SenderAlwaysPlainRetryableFailure_NeverErodesAttemptBudgetNoMatterHowManyTimes() + { + const int maxAttempts = 5; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + // Simulates plain "offline" (or a throwing/circuit-broken sender) -- a connectivity + // failure, not a server response -- via SendResult.RetryableFailure. + var sender = new AlwaysResultSender(SendResult.RetryableFailure); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + client.Track("user-1", "Save", null); + + // Far more than maxAttempts (5) -- if RetryableFailure incremented attempts like + // RetryableRejection does, this would have dropped the event long ago. + const int drains = 50; + for (var i = 0; i < drains; i++) + await client.DrainOnceAsync(); + + Assert.AreEqual(1, spool.ApproximateCount, + drains + " plain connectivity failures (SendResult.RetryableFailure) must NOT " + + "erode the retry-attempt budget -- the event must still be spooled"); + Assert.AreEqual(0, exhaustedCount, + "ItemDroppedByRetryExhaustion must never fire for connectivity-style failures"); + Assert.AreEqual(0, client.Statistics.Failed, + "an event merely retried while offline must not count as Failed"); + } + } + // ---- SHUTDOWN OFFLINE ------------------------------------------------------------------- [Test] diff --git a/src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs index 785d228..800133a 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/MixpanelEventSenderTests.cs @@ -72,36 +72,39 @@ public async Task Send_200Response_ReturnsDeliveredWithNoFailedRecords() } [Test] - public async Task Send_500Response_ReturnsRetryableFailure() + public async Task Send_500Response_ReturnsRetryableRejection() { _server.Given(ImportRequest()) .RespondWith(Response.Create().WithStatusCode(500)); var sender = new MixpanelEventSender("secret", _server.Urls[0]); - Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + // A 5xx means the request DID reach Mixpanel -- distinct from a connectivity failure + // (see SendResult.RetryableRejection), so it counts against the spool's retry-attempt + // ceiling where RetryableFailure does not. + Assert.AreEqual(SendResult.RetryableRejection, (await sender.SendBatchAsync(OneEvent())).Outcome); } [Test] - public async Task Send_429Response_ReturnsRetryableFailure() + public async Task Send_429Response_ReturnsRetryableRejection() { _server.Given(ImportRequest()) .RespondWith(Response.Create().WithStatusCode(429)); var sender = new MixpanelEventSender("secret", _server.Urls[0]); - Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + Assert.AreEqual(SendResult.RetryableRejection, (await sender.SendBatchAsync(OneEvent())).Outcome); } [Test] - public async Task Send_408Response_ReturnsRetryableFailure() + public async Task Send_408Response_ReturnsRetryableRejection() { _server.Given(ImportRequest()) .RespondWith(Response.Create().WithStatusCode(408)); var sender = new MixpanelEventSender("secret", _server.Urls[0]); - Assert.AreEqual(SendResult.RetryableFailure, (await sender.SendBatchAsync(OneEvent())).Outcome); + Assert.AreEqual(SendResult.RetryableRejection, (await sender.SendBatchAsync(OneEvent())).Outcome); } [Test] diff --git a/src/DesktopAnalyticsTests.Mixpanel/SqliteEventSpoolRetryTests.cs b/src/DesktopAnalyticsTests.Mixpanel/SqliteEventSpoolRetryTests.cs new file mode 100644 index 0000000..d810375 --- /dev/null +++ b/src/DesktopAnalyticsTests.Mixpanel/SqliteEventSpoolRetryTests.cs @@ -0,0 +1,323 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + /// + /// The per-event retry-attempt ceiling (offline-analytics-v2-plan.md, Phase 5 / decision D6): + /// a constructor parameter, not part of the generic + /// contract, so it gets its own fixture rather than living in + /// . + /// + /// + /// Only (a genuine "the server told us to try + /// again later") counts against the ceiling. (a + /// connectivity-level failure -- offline, circuit open, canceled, or send() throwing) must + /// NEVER erode it, however many times it happens: that distinction is what + /// + /// guards. See the class remarks on for why. + /// + [TestFixture] + internal class SqliteEventSpoolRetryTests + { + private string _spoolDir; + + [SetUp] + public void SetUp() + { + _spoolDir = Path.Combine(Path.GetTempPath(), "SpoolRetry_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (!Directory.Exists(_spoolDir)) + return; + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + Console.WriteLine("TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + + private static AnalyticsEvent MakeEvent(string name) => AnalyticsEvent.Create("user-1", name); + + // A genuine per-batch server rejection (HTTP 408/429/5xx in production) -- the ONLY verdict + // that counts against the retry-attempt ceiling. + private static Task RejectOnce(IEventSpool spool) => + spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.RetryableRejection)); + + // A connectivity-level failure (offline, circuit open, canceled) -- must NEVER count. + private static Task FailOnce(IEventSpool spool) => + spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.RetryableFailure)); + + [Test] + public async Task RetryableRejection_MaxAttemptsMinusOneTimes_EventStillPresentAndStillOffered() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + // maxAttempts - 1 == 2 rejections: attempts goes 0 -> 1 -> 2, still < maxAttempts (3). + for (var i = 0; i < maxAttempts - 1; i++) + await RejectOnce(spool); + + Assert.AreEqual(1, spool.ApproximateCount, + "after " + (maxAttempts - 1) + " rejections (< maxAttempts = " + maxAttempts + + "), the event must still be in the spool"); + + // And it must still be OFFERED to the next ProcessBatchAsync call (not silently + // stuck behind a lease or otherwise invisible). + var offered = false; + await spool.ProcessBatchAsync(10, (batch, ct) => + { + offered = batch.Count == 1 && batch[0].EventName == "A"; + return Task.FromResult(SendResult.RetryableRejection); + }); + Assert.IsTrue(offered, "the event must still be offered on the next drain"); + + // That drain was itself the maxAttempts-th rejection (3rd), so now it must be gone. + Assert.AreEqual(0, spool.ApproximateCount, + "the drain just performed was the Nth (maxAttempts-th) rejection, so the event " + + "must now be dropped"); + } + } + + [Test] + public async Task RetryableRejection_NthTime_EventDroppedAndEventFiresExactlyOnce() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + // Rejection 1: attempts 0 -> 1 + await RejectOnce(spool); + Assert.AreEqual(1, spool.ApproximateCount, "rejection 1/3: event must still be present"); + Assert.AreEqual(0, exhaustedCount, "rejection 1/3: must not yet be exhausted"); + + // Rejection 2: attempts 1 -> 2 + await RejectOnce(spool); + Assert.AreEqual(1, spool.ApproximateCount, "rejection 2/3: event must still be present"); + Assert.AreEqual(0, exhaustedCount, "rejection 2/3: must not yet be exhausted"); + + // Rejection 3 (== maxAttempts): attempts 2 -> 3 >= maxAttempts (3) -> dropped NOW. + await RejectOnce(spool); + Assert.AreEqual(0, spool.ApproximateCount, + "rejection 3/3 (== maxAttempts): event must be dropped from the spool"); + Assert.AreEqual(1, exhaustedCount, + "ItemDroppedByRetryExhaustion must fire exactly once for the dropped event"); + } + } + + [Test] + public async Task AttemptCounter_SurvivesRestart_DropsAfterReopenOnTheNthRejection() + { + const int maxAttempts = 3; + + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + // N - 1 == 2 rejections against the first instance. + for (var i = 0; i < maxAttempts - 1; i++) + await RejectOnce(spool); + + Assert.AreEqual(1, spool.ApproximateCount, + "after " + (maxAttempts - 1) + " rejections, still present before restart"); + } + + // Reopen a NEW instance pointed at the same directory -- the counter must not reset. + using (var reopened = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + var exhaustedCount = 0; + reopened.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + Assert.AreEqual(1, reopened.ApproximateCount, + "the event itself must survive the restart"); + + // One more rejection is the Nth (3rd) overall -- must drop now, proving the counter + // (2 attempts already recorded) was NOT reset by reopening the database. + await RejectOnce(reopened); + + Assert.AreEqual(0, reopened.ApproximateCount, + "the counter must have survived reopening: this 3rd rejection (1st since restart) " + + "must be enough to drop the event"); + Assert.AreEqual(1, exhaustedCount, + "ItemDroppedByRetryExhaustion must fire once after the restart-preserved count " + + "reaches maxAttempts"); + } + } + + [Test] + public async Task DeliveredAfterPriorRejections_NeverIncrementsIntoExhaustionAndNeverFiresEvent() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + // maxAttempts - 2 == 1 rejection: attempts 0 -> 1. One more rejection (attempt 2) + // would still be short of maxAttempts (3), but the point of this test is that a + // SUCCESS never increments attempts at all, regardless of how many prior rejections + // there were. + for (var i = 0; i < maxAttempts - 2; i++) + await RejectOnce(spool); + Assert.AreEqual(1, spool.ApproximateCount, "still present after the prior rejection(s)"); + + // Now deliver successfully. + var delivered = false; + await spool.ProcessBatchAsync(10, (batch, ct) => + { + delivered = true; + return Task.FromResult(SendResult.Delivered); + }); + + Assert.IsTrue(delivered, "sanity: the send callback must have been invoked"); + Assert.AreEqual(0, spool.ApproximateCount, + "a Delivered batch must remove the event (delivered, not retry-exhausted)"); + Assert.AreEqual(0, exhaustedCount, + "ItemDroppedByRetryExhaustion must NOT fire for a Delivered event, no matter " + + "how many RetryableRejection verdicts preceded it"); + } + } + + [Test] + public async Task PoisonDropAfterPriorRejections_NeverIncrementsIntoExhaustionAndNeverFiresEvent() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + await RejectOnce(spool); // attempts 0 -> 1 + + await spool.ProcessBatchAsync(10, (batch, ct) => Task.FromResult(SendResult.PoisonDrop)); + + Assert.AreEqual(0, spool.ApproximateCount, "PoisonDrop must remove the event"); + Assert.AreEqual(0, exhaustedCount, + "a PoisonDrop removal must never be mistaken for retry exhaustion"); + } + } + + [Test] + public async Task MultipleEventsInOneBatch_EachOwnAttemptCounterTrackedIndependently() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + // A gets 2 solo rejections first (attempts: A=2, B=0), then A and B are rejected + // together once more (attempts: A=3 -> dropped, B=1 -> kept), proving each event's + // counter is independent rather than one shared counter for the whole batch. + spool.Enqueue(MakeEvent("A")); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + Assert.AreEqual(1, batch.Count); + Assert.AreEqual("A", batch[0].EventName); + return Task.FromResult(SendResult.RetryableRejection); + }); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + Assert.AreEqual(1, batch.Count); + Assert.AreEqual("A", batch[0].EventName); + return Task.FromResult(SendResult.RetryableRejection); + }); + + spool.Enqueue(MakeEvent("B")); + Assert.AreEqual(2, spool.ApproximateCount, "A (attempts=2) and B (attempts=0) present"); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + // One shared batch containing BOTH A and B, both rejected together. + var namesSeen = new System.Collections.Generic.List(); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + foreach (var e in batch) + namesSeen.Add(e.EventName); + return Task.FromResult(SendResult.RetryableRejection); + }); + CollectionAssert.AreEquivalent(new[] { "A", "B" }, namesSeen, + "sanity: both events were claimed together in the same batch"); + + // A: attempts 2 -> 3 (== maxAttempts) -> dropped, fires the event once. + // B: attempts 0 -> 1 (< maxAttempts) -> kept, no event. + Assert.AreEqual(1, spool.ApproximateCount, + "only A should be dropped; B (attempts=1) must remain"); + Assert.AreEqual(1, exhaustedCount, + "exactly one drop (A) despite both events sharing the same rejected batch"); + + var remaining = new System.Collections.Generic.List(); + await spool.ProcessBatchAsync(10, (batch, ct) => + { + foreach (var e in batch) + remaining.Add(e.EventName); + return Task.FromResult(SendResult.Delivered); + }); + CollectionAssert.AreEqual(new[] { "B" }, remaining, + "the surviving event must be B, not A"); + } + } + + // ---- CONNECTIVITY-STYLE FAILURE MUST NEVER ERODE THE BUDGET ------------------------------ + // + // Regression coverage for the gap found wiring this feature into MixpanelClient: a naive + // "increment attempts on every retryable verdict" implementation would also drop events + // during plain offline stretches (SendResult.RetryableFailure), not just on a genuine + // per-event server rejection (SendResult.RetryableRejection). This is the pure-spool-level + // counterpart to MixpanelClientTests + // .DrainRepeatedly_SenderAlwaysPlainRetryableFailure_NeverErodesAttemptBudgetNoMatterHowManyTimes. + + [Test] + public async Task PlainRetryableFailure_RepeatedManyTimes_NeverIncrementsAttemptsOrExhausts() + { + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + spool.Enqueue(MakeEvent("A")); + + var exhaustedCount = 0; + spool.ItemDroppedByRetryExhaustion += () => Interlocked.Increment(ref exhaustedCount); + + // Far more than maxAttempts (3) -- if RetryableFailure incremented attempts the way + // RetryableRejection does, this would have dropped the event long ago. + const int failures = 50; + for (var i = 0; i < failures; i++) + await FailOnce(spool); + + Assert.AreEqual(1, spool.ApproximateCount, + failures + " plain connectivity failures must NOT erode the retry-attempt " + + "budget -- the event must still be spooled"); + Assert.AreEqual(0, exhaustedCount, + "ItemDroppedByRetryExhaustion must never fire for connectivity-style failures"); + + // And it must still be fully usable afterwards -- e.g. still deliverable. + var delivered = false; + await spool.ProcessBatchAsync(10, (batch, ct) => + { + delivered = batch.Count == 1 && batch[0].EventName == "A"; + return Task.FromResult(SendResult.Delivered); + }); + Assert.IsTrue(delivered, "the event must still be present and deliverable afterwards"); + Assert.AreEqual(0, spool.ApproximateCount); + } + } + } +} From f8b0cafb41e6f4696fd75d81657ad7f0e137fb7f Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 21:51:29 -0400 Subject: [PATCH 09/11] Fix ShutDown cancellation gap and slow drain pacing (Phase 6) Two outstanding findings from the original PR #43 review: 1. ShutDownAsync never called CancelInFlightSend() the way PurgeQueuedEvents does. StopTimerPermanently() only stops FUTURE ticks -- a normal timer-driven drain already mid-send at the moment ShutDown is called would otherwise linger in the background racing the spool/sender dispose that follows. Fixed by calling CancelInFlightSend() right after StopTimerPermanently(), same as PurgeQueuedEvents already does. The wall-clock "12s vs 5s bound" framing from the original PR #43 review turned out to be stale, DiskQueue-era: post-SQLite-adoption, ShutDownAsync's own BoundedDrainAsync already returns in single-digit milliseconds regardless of a wedged normal drain, since SqliteEventSpool leases claimed batches rather than blocking on them. The real, discriminating proof needed was whether the ORPHANED drain's own CancellationToken ever observes cancellation -- verified via a dedicated test: false before this fix, true after. 2. Drain pacing (256KB/30s tick, ~8.5KB/s effective) took an estimated 50-100 minutes to clear a full backlog once reconnected -- longer than most real connectivity windows. OnTimerTickAsync now compares the spool's approximate count before/after each tick; if the tick made genuine progress (count decreased) and a backlog remains, it reschedules the next tick via the same 1-second reconnect- accelerator already used for network-change events, instead of waiting the full 30-second interval. Ticks that make no progress (genuinely offline) are left on the normal cadence, so this cannot turn into network spam during an outage. Both fixes verified with real timed measurements, not just green tests: a wedged-drain ShutDown test proves cancellation now fires; a 25-event/5-tick backlog test clears in ~7.1s with acceleration versus getting stuck indefinitely without it. 114/114 Mixpanel tests green on both TFMs (was 112); core (19+14) unaffected. Co-Authored-By: Claude Opus 4.8 --- .../MixpanelClient.cs | 33 +++ .../MixpanelClientTimingTests.cs | 243 ++++++++++++++++++ 2 files changed, 276 insertions(+) create mode 100644 src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTimingTests.cs diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index 43d00a4..0b85f3f 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -222,6 +222,21 @@ internal void InitializeForTest( _batchSize = batchSize; } + /// + /// Test-only: starts the real background flush against whatever + /// dependencies already injected. + /// deliberately never starts this timer (see its doc comment) because almost every test wants + /// to pump by hand instead of racing a real timer thread -- call + /// this only when a test specifically needs a genuine timer-driven tick (e.g. proving a normal + /// drain can be caught mid-send by , or proving drain-pacing + /// acceleration across several real ticks). + /// + internal void StartTimerForTest(TimeSpan flushInterval) + { + _flushInterval = flushInterval; + StartTimer(); + } + // Keeps Statistics.Failed (and therefore Statistics.Submitted == Succeeded + Failed) accurate // for events dropped later by cap enforcement, not just ones dropped at enqueue time -- see // IEventSpool.ItemDroppedByCap. @@ -371,7 +386,21 @@ private async Task OnTimerTickAsync() try { + // Captured before/after the drain (not any new instrumentation) so "this tick made + // genuine forward progress" can be detected cheaply: a decrease means at least one + // event was actually removed from the spool this tick (delivered, poison-dropped, or + // retry-exhausted), as opposed to a tick where nothing happened because we're still + // offline (every attempt RetryableFailure/RetryableRejection, count unchanged). + var before = _spool?.ApproximateCount ?? 0; await DrainOnceAsync().ConfigureAwait(false); + var after = _spool?.ApproximateCount ?? 0; + + // Backlog pacing (Phase 6): if this tick made progress but a backlog remains, + // reschedule soon via the same reconnect-accelerator mechanism instead of waiting the + // full interval. Skip on no-progress ticks (genuinely offline) to avoid spamming the + // network. !_paused guards the same race OnNetworkAddressChanged guards against. + if (!_paused && after > 0 && after < before) + _flushTimer?.Change(TimeSpan.FromSeconds(kReconnectFlushDelaySeconds), _flushInterval); } catch (Exception e) { @@ -732,6 +761,10 @@ public async Task ShutDownAsync(CancellationToken cancellationToken = default) try { StopTimerPermanently(); + // StopTimerPermanently only stops FUTURE ticks; a normal drain already mid-send at + // this instant would otherwise linger and race the dispose below. Same mechanism + // PurgeQueuedEvents uses -- see CancelInFlightSend's own comment. + CancelInFlightSend(); await BoundedDrainAsync(cancellationToken).ConfigureAwait(false); } catch (Exception e) diff --git a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTimingTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTimingTests.cs new file mode 100644 index 0000000..0c9e954 --- /dev/null +++ b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTimingTests.cs @@ -0,0 +1,243 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using DesktopAnalytics; +using NUnit.Framework; + +namespace DesktopAnalyticsTests +{ + // Phase 6 (offline-analytics-v2-plan.md, "Outstanding review findings"): behavioral/timing + // tests against a REAL background Timer-driven flush loop (via MixpanelClient.StartTimerForTest), + // rather than the manual DrainOnceAsync pump every other MixpanelClientTests fixture uses. Both + // findings here are about WHEN/HOW OFTEN a genuine timer tick runs, which manual pumping cannot + // exercise at all. + [TestFixture] + public class MixpanelClientTimingTests + { + private string _spoolDir; + + [SetUp] + public void SetUp() + { + _spoolDir = Path.Combine(Path.GetTempPath(), "MixpanelClientTimingTests_" + Guid.NewGuid()); + } + + [TearDown] + public void TearDown() + { + if (Directory.Exists(_spoolDir)) + { + try + { + Directory.Delete(_spoolDir, true); + } + catch (Exception e) + { + Console.WriteLine("MixpanelClientTimingTests.TearDown: failed to delete " + _spoolDir + ": " + e); + } + } + } + + // Blocks inside SendBatchAsync until EITHER the test explicitly releases it OR the + // CancellationToken it was handed is canceled -- unlike MixpanelClientTests' + // BlockingUntilSignaledSender (which ignores cancellation entirely), this sender needs to + // distinguish "a real fix canceled me" from "nobody has released or canceled me yet" so a + // test can assert on WasCanceled as the one signal that is immune to the disposal-ordering + // race between the canceled call's own continuation and a concurrent Dispose (see the + // class remarks on the cancellation test below for why WasCanceled -- not lease state or + // CallCount -- is the only race-proof discriminator). + private class BlockingUntilSignaledOrCanceledSender : IEventSender + { + private readonly TaskCompletionSource _release = + new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + public int CallCount; + public volatile bool WasCanceled; + + public void Release() => _release.TrySetResult(true); + + public async Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref CallCount); + + var canceled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + using (cancellationToken.Register(() => canceled.TrySetResult(true))) + { + var completed = await Task.WhenAny(_release.Task, canceled.Task).ConfigureAwait(false); + if (completed == canceled.Task) + { + WasCanceled = true; + return BatchSendResult.Retryable; + } + } + + return new BatchSendResult(SendResult.Delivered); + } + } + + // ---- FINDING 1: ShutDownAsync must cancel a genuinely wedged, timer-driven drain -------- + // + // Regression coverage for offline-analytics-v2-plan.md Phase 6, finding 1: ShutDownAsync used + // to call StopTimerPermanently() (which only prevents FUTURE ticks) but never + // CancelInFlightSend() (which PurgeQueuedEvents already does) -- so a normal timer-driven + // drain genuinely mid-send at the moment ShutDown is called was never aborted; it kept + // running in the background, racing the spool/sender ShutDown's finally block disposes. + // + // IMPORTANT (measured, not assumed -- see the report): the wall-clock Stopwatch bound on + // ShutDownAsync() below turns out to hold EVEN WITHOUT the fix, because SqliteEventSpool + // leases a claimed batch for 2 minutes: BoundedDrainAsync's own (separate, freshly bounded) + // claim attempt simply finds nothing unleased to claim and returns in milliseconds, never + // blocking on the wedged send at all. So the Stopwatch assertion alone is NOT proof this + // finding is fixed -- it passes either way. The actual discriminator is WasCanceled: only a + // real CancelInFlightSend() call causes the WEDGED call's own CancellationToken + // (_sendCts.Token, captured when the timer tick started) to fire. Polled with a bound + // (never a fixed sleep) so a still-broken build fails cleanly instead of hanging. + [Test] + public async Task ShutDownAsync_TimerDrivenDrainGenuinelyWedgedMidSend_CancelsPromptlyAndShutDownStaysBounded() + { + var spool = new SqliteEventSpool(_spoolDir, 10); + var sender = new BlockingUntilSignaledOrCanceledSender(); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, batchSize: 5); + + client.Track("user-1", "Save", null); + + // A short real flush interval so the first real timer tick fires almost immediately + // (StartTimer's initial delay is min(kInitialFlushDelaySeconds, flushInterval)) and + // genuinely races ShutDown, rather than requiring this test to wait out the 30s + // production default. + client.StartTimerForTest(TimeSpan.FromMilliseconds(100)); + + // Wait for the tick's send to actually start (and therefore be genuinely blocked) + // before calling ShutDown -- otherwise this would race a tick that hasn't begun yet. + var startDeadline = DateTime.UtcNow.AddSeconds(5); + while (sender.CallCount == 0 && DateTime.UtcNow < startDeadline) + await Task.Delay(10); + Assert.AreEqual(1, sender.CallCount, + "the real timer tick must have started a send (and be blocked in it) before ShutDown is called"); + + try + { + var stopwatch = Stopwatch.StartNew(); + await client.ShutDownAsync(); + stopwatch.Stop(); + + // Necessary but NOT sufficient (see the remarks above) -- kept because ShutDown must + // never hang regardless, and to document the measured number. + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(10), + "ShutDownAsync() must stay bounded even with a genuinely wedged timer-driven " + + "drain in the background"); + Console.WriteLine("ShutDownAsync() elapsed while a timer-driven drain was wedged: " + + stopwatch.Elapsed.TotalMilliseconds + " ms"); + + // THE actual proof: the wedged call's own CancellationToken must have fired. Polled + // (not a fixed delay) because the sender's continuation runs on a thread-pool thread + // after ShutDownAsync returns; a fixed short sleep would be flaky under load, and if + // the fix regressed, this must fail promptly rather than hang. + var cancelDeadline = DateTime.UtcNow.AddSeconds(2); + while (!sender.WasCanceled && DateTime.UtcNow < cancelDeadline) + await Task.Delay(20); + + Assert.IsTrue(sender.WasCanceled, + "ShutDownAsync must call CancelInFlightSend so a genuinely wedged, timer-driven " + + "drain observes cancellation promptly instead of lingering in the background " + + "after the spool/sender have been disposed"); + + // A racy re-claim (BoundedDrainAsync's own attempt grabbing the row right after the + // wedged call's lease is released) can make the sender's SECOND call the one that + // actually reports WasCanceled, so CallCount is asserted loosely here -- what matters + // is that cancellation was observed by SOME call, not which one. + Assert.GreaterOrEqual(sender.CallCount, 1); + } + finally + { + // Release unconditionally so a failed/broken-fix run does not leave a permanently + // blocked thread-pool task behind. + sender.Release(); + } + + // No-loss invariant (true whether or not the lease itself happened to clear in time): + // the event must still be on disk, not silently lost, for the next launch to retry. + using (var reopened = new SqliteEventSpool(_spoolDir, 10)) + { + Assert.AreEqual(1, reopened.ApproximateCount, + "the undelivered event must remain in the spool -- it was never actually " + + "acknowledged as delivered"); + } + } + + // ---- FINDING 2: drain pacing must accelerate while a backlog is being cleared ----------- + // + // Regression coverage for offline-analytics-v2-plan.md Phase 6, finding 2: + // kMaxBytesPerDrainTick per kDefaultFlushIntervalSeconds is far too slow to clear a real + // backlog within a typical reconnect window (the plan calculates 50-100 minutes for a full + // backlog). OnTimerTickAsync must reschedule the NEXT tick after + // kReconnectFlushDelaySeconds (not the full flushInterval) whenever a tick both leaves a + // backlog and made genuine forward progress. + [Test] + public async Task RealTimerLoop_BacklogAcrossMultipleTicksWithProgress_AcceleratesInsteadOfWaitingFullInterval() + { + using (var spool = new SqliteEventSpool(_spoolDir, 1000)) + { + var sender = new AlwaysDeliveredSender(); + var client = new MixpanelClient(); + const int batchSize = 5; + const int totalEvents = 25; // Exactly 5 ticks at batchSize=5. + client.InitializeForTest(spool, sender, batchSize: batchSize); + + for (var i = 0; i < totalEvents; i++) + client.Track("user-1", "Backlog-" + i, null); + Assert.AreEqual(totalEvents, spool.ApproximateCount); + + // Deliberately >> kReconnectFlushDelaySeconds (1s): if acceleration is NOT wired up, + // clearing 5 ticks needs 4 full intervals between them (~3s initial delay + 4 x 5s = + // ~23s). If it IS wired up, the gaps after the first tick are ~1s each (~3s + 4 x 1s + // = ~7s). A 12s bound cleanly separates the two. + var flushInterval = TimeSpan.FromSeconds(5); + + var stopwatch = Stopwatch.StartNew(); + client.StartTimerForTest(flushInterval); + + var deadline = DateTime.UtcNow.AddSeconds(20); + while (spool.ApproximateCount > 0 && DateTime.UtcNow < deadline) + await Task.Delay(50); + stopwatch.Stop(); + + Assert.AreEqual(0, spool.ApproximateCount, + "the backlog did not fully clear within the test's outer deadline"); + Console.WriteLine("Backlog of " + totalEvents + " events (batchSize=" + batchSize + + ", flushInterval=" + flushInterval.TotalSeconds + "s) cleared in " + + stopwatch.Elapsed.TotalMilliseconds + " ms across " + sender.CallCount + " sender calls"); + + Assert.Less(stopwatch.Elapsed, TimeSpan.FromSeconds(12), + "clearing a multi-tick backlog must be paced by kReconnectFlushDelaySeconds " + + "(~1s) between ticks once progress is being made, not by waiting the full " + + flushInterval.TotalSeconds + "s flushInterval between every tick"); + Assert.GreaterOrEqual(sender.CallCount, totalEvents / batchSize, + "at least " + (totalEvents / batchSize) + " ticks (batches) must have occurred to " + + "clear " + totalEvents + " events at batchSize=" + batchSize); + + client.ShutDown(); + } + } + + // A fast, never-blocking, always-Delivered sender -- deliberately separate from + // MixpanelClientTests' AlwaysResultSender (private to that fixture) and from this file's + // cancellation-aware sender above, which exists only to test the different (cancellation) + // behavior. + private class AlwaysDeliveredSender : IEventSender + { + public int CallCount; + + public Task SendBatchAsync(IReadOnlyList events, + CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref CallCount); + return Task.FromResult(new BatchSendResult(SendResult.Delivered)); + } + } + } +} From 082b8b986b69250cb2bcd0236aaf9ac2ce515429 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 16 Jul 2026 21:53:39 -0400 Subject: [PATCH 10/11] Refresh plan doc: Phase 7 consumer integration notes, current-state recap Documents the concrete integration steps for FieldWorks/FW Lite/Segment consumers against what was actually built, and summarizes Phases 1-6 as implemented rather than as planned. No code changes. Co-Authored-By: Claude Opus 4.8 --- offline-analytics-v2-plan.md | 87 +++++++++++++++++++++++++++++++----- 1 file changed, 75 insertions(+), 12 deletions(-) diff --git a/offline-analytics-v2-plan.md b/offline-analytics-v2-plan.md index 25a006c..8df9c0f 100644 --- a/offline-analytics-v2-plan.md +++ b/offline-analytics-v2-plan.md @@ -254,11 +254,46 @@ across restart and across a simulated path/version change. ### Phase 7 — Consumer integration -1. FieldWorks: bump + add `.Mixpanel`; verify CPM transitive pinning; installer auto-harvests via - WiX heat, so `e_sqlite3.dll` needs no installer work (~4.1 MB, 3 Windows RIDs). -2. FW Lite: adopt; verify the SQLitePCLRaw version story; `PublishSingleFile` on Linux needs - `IncludeNativeLibrariesForSelfExtract=true` (FW Lite has likely already solved this). -3. Segment consumers: no action beyond a version bump. +Packages, as actually built (Phases 1-6, all committed on this branch): + +- **`SIL.DesktopAnalytics`** — facade (`Analytics`), `SegmentClient`, `IClient`, `UserInfo`, + `Statistics`. Multi-targets `net462;netstandard2.0;net8.0`. +- **`SIL.DesktopAnalytics.Mixpanel`** — `MixpanelClient`, `SqliteEventSpool`/`IEventSpool`, + `IEventSender`/`MixpanelEventSender`, `AnalyticsEvent`, `PathScrubber`. Same TFMs. References + `Microsoft.Data.Sqlite`, Polly, `mixpanel-csharp` — none of which leak into core (verified via + `dotnet list package --include-transitive`). + +API break, major version bump (D4): `ClientType` enum is gone. Callers now inject an `IClient`: +```csharp +// before +new Analytics(apiSecret, userInfo, clientType: ClientType.Mixpanel); +// after (classic FieldWorks / FW Lite -- add a reference to SIL.DesktopAnalytics.Mixpanel) +new Analytics(apiSecret, userInfo, client: new MixpanelClient()); +// Segment default is unchanged -- omitting `client` still defaults to `new SegmentClient()` +new Analytics(apiSecret, userInfo); +``` + +1. **FieldWorks (classic):** add a `SIL.DesktopAnalytics.Mixpanel` package reference; change the + `ClientType.Mixpanel` call site to `client: new MixpanelClient()` per above. Installer: + `e_sqlite3.dll` (~4.1 MB, 3 Windows RIDs) is a normal transitive native asset under + `Microsoft.Data.Sqlite`; WiX heat auto-harvests it the same way it already does for + `libSkiaSharp`/`libHarfBuzzSharp`/icu.net — no installer authoring expected, but worth a + sanity-check build once adopted. **Flag for FieldWorks to verify themselves:** this package + pins `SQLitePCLRaw.bundle_e_sqlite3` 2.1.6 (via `Microsoft.Data.Sqlite` 8.0.10); if FieldWorks + pins a different SQLitePCLRaw version anywhere else in its dependency graph, NuGet unifies + upward and FieldWorks should confirm that causes no issues. +2. **FW Lite:** same package addition; this is what makes it consumable at all (previously + net462-only). **Flag for FW Lite to verify themselves:** FW Lite reportedly already pins + `SQLitePCLRaw` 3.0.3 via `EFCore.Sqlite` 10.0.8 — higher than this package's 2.1.6, so NuGet + unification should just take FW Lite's existing pin with no action needed, but confirm rather + than assume. `PublishSingleFile` on Linux needs `IncludeNativeLibrariesForSelfExtract=true` for + the SQLite native asset to extract correctly; FW Lite very likely already has this set (it + would need it for its own existing SQLite dependency), but worth confirming when adopting. +3. **Segment consumers (Bloom, HearThis, SayMore, Glyssen, Transcelerator):** version bump only. + They take `SIL.DesktopAnalytics` alone and end up *lighter* than PR #43 would have left them — + no DiskQueue/Polly/mixpanel-csharp/SQLite in their dependency graph at all. No code changes + expected unless a consumer explicitly referenced `ClientType` (none should have, since none use + Mixpanel). ## Risks @@ -277,11 +312,39 @@ across restart and across a simulated path/version change. ## Current state (what exists today, on `feature/offline-mixpanel-durability`) -- 172/172 tests green. -- `IEventSpool` extracted; `EventSpool` (DiskQueue) and `SqliteEventSpool` both implement it. -- `EventSpoolContractTests` — 28 tests x 2 engines = 56, proving behavioral parity. -- `TrackResponsivenessTests` — SQLite returns in single-digit ms; DiskQueue blocks for the whole - send (pinned as a characterization test). -- **Production still uses DiskQueue.** `MixpanelClient.Initialize` constructs `EventSpool`. Nothing - is adopted yet. +Phases 1-6 are done, each independently rebuilt/retested/inspected (not just taken on a +subagent's word) before being committed: + +- **Phase 1:** multi-targets `net462;netstandard2.0;net8.0`. `#if NET462` keeps + `ApplicationSettingsBase`/`user.config` byte-for-byte; modern TFMs get a JSON-backed + `IAnalyticsSettingsStore` at a stable path. No legacy-import logic (deliberately not built -- + see the Risks table). +- **Phase 2:** split into `SIL.DesktopAnalytics` + `SIL.DesktopAnalytics.Mixpanel`. `ClientType` + enum replaced by `IClient` injection. Verified via `dotnet list package --include-transitive` + that core carries none of DiskQueue/Sqlite/Polly/mixpanel-csharp. +- **Phase 3:** `SqliteEventSpool` adopted in production; `EventSpool`/DiskQueue deleted entirely. + Never-throws contract verified by inspection (all six `IEventSpool` members on + `SqliteEventSpool` log-and-swallow). +- **Phase 4:** `TrackAsync`/`ReportExceptionAsync` added across `IClient`/`SegmentClient`/ + `MixpanelClient`/`Analytics`, honest about being ergonomics (Microsoft.Data.Sqlite has no true + async I/O) rather than a concurrency fix. Verified with a timed test: ~0.8ms while a send is + genuinely in flight. +- **Phase 5:** per-event retry ceiling (default 10 attempts). Caught and fixed a real bug before + shipping it: a naive counter would have dropped good events after ~5 minutes offline, since + "can't reach the server" and "server rejected this" both collapsed into one `RetryableFailure` + value. Split into `RetryableFailure` (connectivity, never counted) vs. new + `RetryableRejection` (408/429/5xx, counted) — verified against the 3 tests that caught the bug. +- **Phase 6:** `ShutDownAsync` now calls `CancelInFlightSend()` (closing a gap where a + timer-driven drain wedged mid-send would otherwise linger past `ShutDown`'s own dispose). + Drain pacing now accelerates (1s) after a tick that made progress and still leaves a backlog, + instead of waiting the full 30s interval — fixes an estimated 50-100 minute full-backlog clear + time down to accelerated back-to-back ticks. Both verified with real `Stopwatch` measurements, + not just green tests. + +Suite: 114 Mixpanel tests + 19 (net8.0) / 14 (net462) core tests, all green on both runnable TFMs. +Not yet pushed to the fork that backs PR #43 (fork last saw only the plan-doc commit) -- +none of this rework is visible on the PR yet. - `CONTEXT.md` and `offline-analytics.md` corrected re: the fabricated invariant. +- **Phase 7 (this phase):** consumer integration notes above are written; PR #43's description + still describes the pre-rework (DiskQueue, single-package) design and has not been touched -- + see below. From 0ee404ef0e19b01e8d9a43244df6a05f41173dfa Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 17 Jul 2026 12:53:10 -0400 Subject: [PATCH 11/11] Harden the spool: circuit breaker, retry-ceiling correctness, corrupt/expired accounting Closes the gaps a cross-cutting review found in the spool once it was tested against real failure modes rather than the happy path: - The Polly circuit breaker's ShouldHandle predicate didn't match RetryableRejection, so a server that kept returning 5xx/429/408 never tripped the breaker -- it just kept hammering a server that had already said "stop." Fixed, and MinimumThroughput now scales with the configured retry ceiling instead of a hardcoded constant, so a low ceiling can't make the breaker trip before the ceiling itself would have stopped the retries anyway. - BoundedDrainAsync's compressed, many-attempts-in-seconds shutdown/consent-revocation drain was letting a RetryableRejection count against the same per-event retry ceiling a normal once-per-30s tick uses -- so a brief flurry of 503s during shutdown could burn through an event's entire retry budget in seconds. Fixed by remapping RetryableRejection to RetryableFailure for that caller only (SendBatchGuardedAsync), reusing RetryableFailure's existing "never counted" handling rather than widening IEventSpool's public contract. - An undeserializable row (on-disk corruption, or a payload from an incompatible schema version) was silently dropped without ever registering in Statistics, breaking Submitted == Succeeded + Failed. Fixed via IEventSpool.ItemDroppedByCorruption, raised with the actual number of rows ResolveClaim deleted (not an assumed count, which would double-count against a concurrent Purge() racing the same rows). - The age-based retention floor (TrimExpired, 60-day default) had the identical accounting gap for a different reason: an event aging out was never attempted or rejected, so folding it into Failed would conflate "we tried and it didn't work" with "we never got to it in time." Added a distinct Statistics.Expired counter instead (IEventSpool.ItemDroppedByExpiry) -- new invariant: Submitted == Succeeded + Failed + Expired. - Cap enforcement's two full-table scans (COUNT(*) for the item cap, SUM(len) for the byte cap) collapsed into one window-function query -- this runs on every Enqueue(), so halving the per-call scan cost matters on the hot path. - CancellationTokenSource handling around consent-revocation/shutdown simplified: the swapped- out CTS is no longer disposed (it holds no timer or registrations that need early release), removing an ObjectDisposedException race a concurrent reader could otherwise hit. - Purge()'s DELETE+VACUUM+checkpoint stays synchronous rather than backgrounded: measured directly at ~60-80ms even at 2x the default 50MB byte cap, which is well within a UI-thread budget and avoids a real regression a backgrounded version would have introduced -- a crash/kill in the window before background reclaim finished would leave "purged" bytes physically recoverable on disk, defeating the point of a consent purge. - Extracted shared throttle/merge logic in Analytics.cs (BuildExceptionReportProperties, MergeApplicationProperties) so the new async-first ReportExceptionAsync/ TrackWithApplicationPropertiesAsync methods share it with their sync counterparts instead of duplicating it. 122 Mixpanel-project tests (up from 114), all green on net462 and net8.0, including new coverage for the circuit-breaker trip, the bounded-drain ceiling fix, corrupt-row handling, and the age-based-expiry accounting path. Co-Authored-By: Claude Sonnet 5 --- src/DesktopAnalytics.Mixpanel/IEventSpool.cs | 26 ++- .../MixpanelClient.cs | 167 ++++++++++----- .../SqliteEventSpool.cs | 194 ++++++++++++------ src/DesktopAnalytics/Analytics.cs | 74 +++---- src/DesktopAnalytics/Statistics.cs | 12 +- .../EventSpoolContractTests.cs | 131 +++++++++++- .../MixpanelClientTests.cs | 154 ++++++++++++++ .../SpoolCorruptionTestHelper.cs | 32 +++ 8 files changed, 625 insertions(+), 165 deletions(-) create mode 100644 src/DesktopAnalyticsTests.Mixpanel/SpoolCorruptionTestHelper.cs diff --git a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs index 55e28fc..4e9946b 100644 --- a/src/DesktopAnalytics.Mixpanel/IEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/IEventSpool.cs @@ -85,6 +85,27 @@ internal interface IEventSpool : IDisposable /// event Action ItemDroppedByRetryExhaustion; + /// + /// Raised once per claimed event whose stored payload could not be deserialized (on-disk + /// corruption, or a payload written by a since-upgraded, incompatible schema version) and + /// was therefore removed -- it can never be delivered. Distinct from both + /// and , neither of + /// which fires for this case, and without which such a drop would silently go uncounted. + /// Handlers must be fast and must not call back into the spool. + /// + event Action ItemDroppedByCorruption; + + /// + /// Raised once per event permanently dropped by 's age-based + /// retention floor -- distinct from (capacity-based eviction) + /// and (an undeserializable row): those are counted as + /// Statistics.Failed, while this is counted separately as Statistics.Expired, + /// since aging out is a retention decision rather than a delivery failure. Without this + /// event, an event dropped this way would silently go uncounted anywhere in + /// Statistics. Handlers must be fast and must not call back into the spool. + /// + event Action ItemDroppedByExpiry; + /// An approximate count of events currently spooled. Accurate enough for /// bounding and diagnostics; not a hard guarantee under concurrent access. int ApproximateCount { get; } @@ -137,7 +158,10 @@ Task ProcessBatchAsync(int maxItems, /// /// Empties the spool entirely (consent revocation), removing the event data from disk -- - /// not merely marking it consumed. + /// not merely marking it consumed -- before returning. May be called from a UI thread (e.g. + /// from Analytics.AllowTracking's setter); implementations should keep this bounded to + /// a small, measured cost rather than an unbounded one (see SqliteEventSpool.Purge's + /// doc comment for why a backgrounded version of this was tried and reverted). /// void Purge(); } diff --git a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs index 0b85f3f..c458e76 100644 --- a/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs +++ b/src/DesktopAnalytics.Mixpanel/MixpanelClient.cs @@ -108,22 +108,30 @@ public class MixpanelClient : IClient // network-change kicks are ignored until ResumeSending re-enables them. private volatile bool _paused; - // Cancels whatever timer-driven send (DrainOnceAsync) is currently in flight when consent is - // revoked (see PurgeQueuedEvents), so Purge() does not sit blocked behind a network round trip - // and the batch that send was carrying rolls back into the spool (which the impending Purge() - // then removes) rather than being resent locally after revocation. This is a best-effort - // reduction of the window, not a hard guarantee against server-side receipt: if the POST body - // was already fully sent to and received by Mixpanel's server before this cancellation is - // observed locally, the batch WAS still delivered once -- cancellation only prevents it from - // being resent/kept around locally, it cannot recall bytes the server already has. Swapped for - // a fresh instance on every purge; never used across a purge boundary. Guarded by - // Interlocked.Exchange since PurgeQueuedEvents can run on a different thread (e.g. a UI thread - // via Analytics.AllowTracking) than the timer-driven drain it cancels. + // Cancels whatever timer-driven send (DrainOnceAsync) is currently in flight, via + // CancelInFlightSend -- used by two independent call paths with different post-conditions: + // PurgeQueuedEvents (consent revoked; the flush loop keeps running afterward under the fresh + // token so the batch that send was carrying rolls back into the spool, which the impending + // Purge() then removes, rather than being resent locally after revocation) and ShutDownAsync + // (StopTimerPermanently already stopped FUTURE ticks; this only aborts one already in flight + // so it cannot linger and race the spool/sender dispose that follows -- the fresh token + // swapped in there is simply discarded moments later along with everything else). This is a + // best-effort reduction of the window, not a hard guarantee against server-side receipt: if + // the POST body was already fully sent to and received by Mixpanel's server before this + // cancellation is observed locally, the batch WAS still delivered once -- cancellation only + // prevents it from being resent/kept around locally, it cannot recall bytes the server + // already has. Guarded by Interlocked.Exchange since both callers (e.g. a UI thread via + // Analytics.AllowTracking, or whatever thread calls ShutDown) can run concurrently with the + // timer-driven drain being canceled -- and CancelInFlightSend deliberately never Disposes + // the swapped-out instance (see its own comment), so a concurrent reader of this field that + // raced the swap and is still holding the old CancellationTokenSource can safely finish + // using it without needing a lock or risking ObjectDisposedException. private CancellationTokenSource _sendCts = new CancellationTokenSource(); private int _submitted; private int _succeeded; private int _failed; + private int _expired; // Guards only against overlapping TIMER ticks: if a previous tick's drain is still running // when the next tick fires, the new tick is skipped. It does NOT serialize ticks against @@ -166,8 +174,10 @@ public void Initialize(string apiSecret, string host = null, int flushAt = -1, i _spool = new SqliteEventSpool(SqliteEventSpool.GetDefaultSpoolPath(apiSecret), kDefaultMaxSpoolItems, kMaxSpooledEventBytes, kDefaultMaxSpoolBytes, timeProvider: _timeProvider, maxAttempts: kDefaultMaxRetryAttempts); - _spool.ItemDroppedByCap += OnItemDroppedByCap; - _spool.ItemDroppedByRetryExhaustion += OnItemDroppedByRetryExhaustion; + _spool.ItemDroppedByCap += OnItemPermanentlyDropped; + _spool.ItemDroppedByRetryExhaustion += OnItemPermanentlyDropped; + _spool.ItemDroppedByCorruption += OnItemPermanentlyDropped; + _spool.ItemDroppedByExpiry += OnItemExpired; _sender = new MixpanelEventSender(apiSecret); _pipeline = BuildDefaultPipeline(_timeProvider); @@ -213,8 +223,10 @@ internal void InitializeForTest( _spool = spool; if (_spool != null) { - _spool.ItemDroppedByCap += OnItemDroppedByCap; - _spool.ItemDroppedByRetryExhaustion += OnItemDroppedByRetryExhaustion; + _spool.ItemDroppedByCap += OnItemPermanentlyDropped; + _spool.ItemDroppedByRetryExhaustion += OnItemPermanentlyDropped; + _spool.ItemDroppedByCorruption += OnItemPermanentlyDropped; + _spool.ItemDroppedByExpiry += OnItemExpired; } _sender = sender; _timeProvider = timeProvider ?? TimeProvider.System; @@ -237,20 +249,24 @@ internal void StartTimerForTest(TimeSpan flushInterval) StartTimer(); } - // Keeps Statistics.Failed (and therefore Statistics.Submitted == Succeeded + Failed) accurate - // for events dropped later by cap enforcement, not just ones dropped at enqueue time -- see - // IEventSpool.ItemDroppedByCap. - private void OnItemDroppedByCap() + // Shared by all three of the spool's "permanently dropped, never delivered" events -- + // ItemDroppedByCap (capacity eviction), ItemDroppedByRetryExhaustion (decision D6), and + // ItemDroppedByCorruption (an undeserializable row) -- since each has exactly the same + // statistics effect: keeping Statistics.Failed accurate for a drop that happens later than, + // and separately from, Track()'s own enqueue-time accounting. ItemDroppedByExpiry is + // deliberately NOT included here -- see OnItemExpired. + private void OnItemPermanentlyDropped() { Interlocked.Increment(ref _failed); } - // Same statistics effect as cap eviction (see OnItemDroppedByCap above): an event dropped - // for exceeding the retry ceiling will also never be delivered. See - // IEventSpool.ItemDroppedByRetryExhaustion. - private void OnItemDroppedByRetryExhaustion() + // ItemDroppedByExpiry gets its own counter rather than folding into OnItemPermanentlyDropped: + // an event aged out by the retention floor was never attempted and rejected, so counting it + // as Failed would conflate "we tried and it didn't work" with "we never got to it in time". + // Keeps Statistics.Submitted == Succeeded + Failed + Expired accurate for this drop path. + private void OnItemExpired() { - Interlocked.Increment(ref _failed); + Interlocked.Increment(ref _expired); } /// @@ -273,8 +289,16 @@ internal static ResiliencePipeline BuildDefaultPipeline( TimeSpan? retryDelay = null, TimeSpan? breakDuration = null) { + // Both retryable verdicts must be handled here, not just RetryableFailure: a + // RetryableRejection (HTTP 408/429/5xx -- the batch DID reach the server) is exactly the + // "server is struggling, back off" case the circuit breaker exists for. Omitting it would + // make a sustained 429/5xx outage invisible to both in-tick retry and the breaker's + // failure tracking, so the client would keep hammering an already-struggling endpoint + // every flush tick. bool ShouldHandleOutcome(Outcome outcome) => - outcome.Exception != null || outcome.Result?.Outcome == SendResult.RetryableFailure; + outcome.Exception != null || + outcome.Result?.Outcome == SendResult.RetryableFailure || + outcome.Result?.Outcome == SendResult.RetryableRejection; var retryOptions = new RetryStrategyOptions { @@ -289,14 +313,22 @@ bool ShouldHandleOutcome(Outcome outcome) => { ShouldHandle = args => new ValueTask(ShouldHandleOutcome(args.Outcome)), FailureRatio = 0.5, - // Matches attempts-per-tick: a fully failing drain tick produces exactly 3 - // outcomes through this pipeline (1 initial attempt + MaxRetryAttempts=2 retries), - // all within the same instant, so all 3 always land in one SamplingDuration window. - // With this at 4 (the previous value), a single tick could never reach the minimum - // throughput -- and successive ticks are kDefaultFlushIntervalSeconds (30s) apart, - // far outside the 10s window -- so the breaker could never open at all during a - // sustained outage. 3 makes one bad tick enough to trip it. - MinimumThroughput = 3, + // Matches attempts-per-tick: a fully failing drain tick produces exactly + // maxRetryAttempts + 1 outcomes through this pipeline (1 initial attempt + the + // configured retries), all within the same instant, so they all land in one + // SamplingDuration window. Derived from maxRetryAttempts -- rather than a separate + // hardcoded literal -- so the two can never silently drift out of sync: with a value + // too high for the actual attempts-per-tick, a single tick could never reach the + // minimum throughput, and successive ticks are kDefaultFlushIntervalSeconds (30s) + // apart, far outside the 10s window below, so the breaker could never open at all + // during a sustained outage. Clamped to Polly's own minimum (2), which only avoids a + // build-time rejection for maxRetryAttempts: 0 -- it does NOT restore the "one bad + // tick trips the breaker" property for that value specifically, since a 0-retry tick + // still produces only 1 outcome and reaching 2 then requires a second tick 30s later, + // outside this window. Production always passes >= 1 (the default is 2), where the + // property holds; 0 remains usable but degrades to the slower multi-tick behavior the + // original (pre-fix) hardcoded-literal bug had for every value. + MinimumThroughput = Math.Max(2, maxRetryAttempts + 1), SamplingDuration = TimeSpan.FromSeconds(10), BreakDuration = breakDuration ?? TimeSpan.FromSeconds(5) }; @@ -426,8 +458,9 @@ internal Task DrainOnceAsync() } // Shared implementation behind DrainOnceAsync() and BoundedDrainAsync(). usePipeline is false - // only for BoundedDrainAsync's bounded attempts (see its comment for why retries are skipped - // there). + // only for BoundedDrainAsync's bounded attempts (see its comment for why retries are skipped, + // and why a RetryableRejection there is remapped in SendBatchGuardedAsync so it does not + // erode the per-event retry ceiling either). private async Task DrainOnceCoreAsync(CancellationToken cancellationToken, bool usePipeline) { try @@ -488,7 +521,20 @@ private async Task SendBatchGuardedAsync(IReadOnlyList SendBatchGuardedAsync(IReadOnlyList + /// The per-event retry-attempt ceiling is bypassed too: unlike a normal timer tick + /// (kDefaultFlushIntervalSeconds = 30s apart, so reaching kDefaultMaxRetryAttempts takes + /// roughly 5 minutes of real rejections), this loop can drive up to kBoundedDrainMaxAttempts + /// attempts back-to-back within s_boundedDrainDuration. If the server were quickly returning + /// RetryableRejection (e.g. rate-limiting), counting those against the ceiling here could + /// burn through it in seconds and permanently drop otherwise-good events -- during a single + /// Flush()/ShutDown() call -- for reasons that have nothing to do with the event itself. + /// remaps a RetryableRejection to RetryableFailure + /// whenever usePipeline is false (i.e. only on this bounded path), reusing + /// RetryableFailure's existing "never counted" semantics rather than threading a second flag + /// all the way through 's contract. The ceiling + /// still applies normally to every ordinary timer-driven tick; this only exempts the + /// compressed, bounded-drain-specific retry loop from also counting against it. /// /// private async Task BoundedDrainAsync(CancellationToken cancellationToken) @@ -835,24 +894,26 @@ public void PurgeQueuedEvents() } } - // Aborts whatever timer-driven send is currently in flight (if any) so Purge() does not block - // behind a network round trip and the in-flight batch rolls back into the spool -- where the - // impending Purge() removes it -- rather than being resent/kept around locally after consent - // was just revoked. This best-effort cancellation reduces, but cannot eliminate, the window: - // it cannot recall a POST body Mixpanel's server already fully received before the local await - // observed cancellation. Swaps in a fresh, non-canceled token so the NEXT drain (post-purge, or - // after ResumeSending) is unaffected. + // Aborts whatever timer-driven send is currently in flight (if any). Two call sites, two + // different reasons: PurgeQueuedEvents uses this so Purge() does not block behind a network + // round trip, and so the in-flight batch rolls back into the spool -- where the impending + // Purge() removes it -- rather than being resent/kept around locally after consent was just + // revoked (best-effort: cannot recall a POST body Mixpanel's server already fully received + // before the local await observed cancellation). ShutDownAsync uses the exact same swap here + // for a different reason -- see _sendCts's field doc -- to stop a lingering send from racing + // the spool/sender dispose that follows shutdown; nothing there depends on the fresh token + // this swaps in, since everything is torn down moments later regardless. Swaps in a fresh, + // non-canceled token either way (for PurgeQueuedEvents, that token goes on to back the NEXT + // drain -- post-purge, or after ResumeSending). Deliberately does NOT Dispose the swapped-out + // instance: _sendCts has no timer and nothing external registers against its Token beyond + // this one drain's own await chain, which Cancel() (above) already unwinds -- so there is + // nothing meaningful to release early, and skipping Dispose means a concurrent DrainOnceAsync + // that already captured the old token (see DrainOnceAsync) can safely keep using it instead + // of racing an ObjectDisposedException against this swap. private void CancelInFlightSend() { var previous = Interlocked.Exchange(ref _sendCts, new CancellationTokenSource()); - try - { - previous.Cancel(); - } - finally - { - previous.Dispose(); - } + previous.Cancel(); } /// @@ -906,7 +967,7 @@ private void StopTimerPermanently() } } - public Statistics Statistics => new Statistics(_submitted, _succeeded, _failed); + public Statistics Statistics => new Statistics(_submitted, _succeeded, _failed, _expired); // Test seam: exposes whether the flush loop is currently paused (consent revoked). See // MixpanelClientTests. Not part of IClient. diff --git a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs index 56f2822..264371d 100644 --- a/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs +++ b/src/DesktopAnalytics.Mixpanel/SqliteEventSpool.cs @@ -73,6 +73,12 @@ internal class SqliteEventSpool : IEventSpool /// public event Action ItemDroppedByRetryExhaustion; + /// + public event Action ItemDroppedByCorruption; + + /// + public event Action ItemDroppedByExpiry; + /// Directory to hold the spool database. Callers normally get /// this from ; tests inject a temp directory. /// Maximum number of events retained; enqueuing beyond this drops @@ -82,7 +88,7 @@ internal class SqliteEventSpool : IEventSpool /// Maximum total serialized size of all retained events; /// enqueuing beyond this drops the oldest first. /// Clock for lease expiry. Injected so tests stay deterministic. - /// Maximum number of + /// Maximum number of /// verdicts a single event may accumulate before it is permanently dropped (see /// offline-analytics-v2-plan.md, Phase 5 / decision D6) and /// is raised for it. Bounds the case where a @@ -318,79 +324,73 @@ public bool Enqueue(AnalyticsEvent evt) /// public Task EnqueueAsync(AnalyticsEvent evt) => Task.FromResult(Enqueue(evt)); - private void RaiseDropped(int count) - { - for (var i = 0; i < count; i++) - { - try - { - ItemDroppedByCap?.Invoke(); - } - catch (Exception e) - { - Debug.WriteLine("SqliteEventSpool: ItemDroppedByCap handler threw: " + e); - } - } - } + private void RaiseDropped(int count) => RaiseDropEvent(ItemDroppedByCap, count, nameof(ItemDroppedByCap)); + + private void RaiseRetryExhausted(int count) => + RaiseDropEvent(ItemDroppedByRetryExhaustion, count, nameof(ItemDroppedByRetryExhaustion)); + + private void RaiseCorrupted(int count) => + RaiseDropEvent(ItemDroppedByCorruption, count, nameof(ItemDroppedByCorruption)); + + private void RaiseExpired(int count) => + RaiseDropEvent(ItemDroppedByExpiry, count, nameof(ItemDroppedByExpiry)); - private void RaiseRetryExhausted(int count) + // Shared by all four "permanently dropped" events above: from inside the declaring class, + // an event field reads like a plain delegate field, so one handler-agnostic helper can raise + // any of them count times, each invocation independently guarded so one misbehaving + // subscriber can't stop the rest from being counted. + private static void RaiseDropEvent(Action handler, int count, string eventName) { for (var i = 0; i < count; i++) { try { - ItemDroppedByRetryExhaustion?.Invoke(); + handler?.Invoke(); } catch (Exception e) { - Debug.WriteLine("SqliteEventSpool: ItemDroppedByRetryExhaustion handler threw: " + e); + Debug.WriteLine("SqliteEventSpool: " + eventName + " handler threw: " + e); } } } // Caller must hold _sync. Brings the spool back within BOTH caps, dropping oldest first, - // and returns how many events were dropped. Two DELETEs, no loop -- contrast the old + // and returns how many events were dropped. One DELETE, no loop -- contrast the old // DiskQueue-backed EventSpool.EnforceCapLocked, which dequeued one item at a time. private int EnforceCapsLocked() { - var dropped = 0; try { - // Item cap. Also covers the degenerate maxItems == 0 case, where the event just - // enqueued is itself evicted. using (var cmd = _connection.CreateCommand()) { - cmd.CommandText = - "DELETE FROM events WHERE id IN (" + - " SELECT id FROM events ORDER BY id ASC" + - " LIMIT MAX(0, (SELECT COUNT(*) FROM events) - @maxItems));"; + // Both caps evaluated in a single pass over the table (one window-function scan + // instead of a separate COUNT(*) for the item cap plus a separate SUM(len) scan + // for the byte cap -- this runs on every Enqueue, so halving the per-call scan + // cost matters). rn > @maxItems: this row is not among the newest maxItems rows + // (also covers the degenerate maxItems == 0 case, where the event just enqueued + // is itself evicted). running > @maxBytes: including this row (and everything + // newer) would push the running total, counted newest-first, past the byte cap. + cmd.CommandText = _maxSpoolBytes < long.MaxValue + ? "DELETE FROM events WHERE id IN (" + + " SELECT id FROM (" + + " SELECT id, ROW_NUMBER() OVER (ORDER BY id DESC) AS rn," + + " SUM(len) OVER (ORDER BY id DESC) AS running FROM events" + + " ) WHERE rn > @maxItems OR running > @maxBytes);" + : "DELETE FROM events WHERE id IN (" + + " SELECT id FROM (" + + " SELECT id, ROW_NUMBER() OVER (ORDER BY id DESC) AS rn FROM events" + + " ) WHERE rn > @maxItems);"; cmd.Parameters.AddWithValue("@maxItems", _maxItems); - dropped += cmd.ExecuteNonQuery(); - } - - // Byte cap. The window function totals bytes newest-first; any row whose inclusion - // pushes that running total past the cap is older than what we can afford to keep, - // so it goes. - if (_maxSpoolBytes < long.MaxValue) - { - using (var cmd = _connection.CreateCommand()) - { - cmd.CommandText = - "DELETE FROM events WHERE id IN (" + - " SELECT id FROM (" + - " SELECT id, SUM(len) OVER (ORDER BY id DESC) AS running FROM events" + - " ) WHERE running > @maxBytes);"; + if (_maxSpoolBytes < long.MaxValue) cmd.Parameters.AddWithValue("@maxBytes", _maxSpoolBytes); - dropped += cmd.ExecuteNonQuery(); - } + return cmd.ExecuteNonQuery(); } } catch (Exception e) { Debug.WriteLine("SqliteEventSpool.EnforceCaps failed: " + e); + return 0; } - - return dropped; } /// @@ -416,7 +416,13 @@ public async Task ProcessBatchAsync(int maxItems, { // Everything claimed was undeserializable (corrupt, or an incompatible schema // version). Nothing to send; just remove them so they cannot wedge the spool. - ResolveClaim(claim.Ids, ClaimOutcome.Remove); + // RaiseCorrupted uses ResolveClaim's own return value (how many undeserializable + // rows THIS call actually deleted), not claim.UndeserializableIds.Count, so a row + // a concurrent Purge() already removed (Purge's DELETE races ClaimBatch/ResolveClaim + // the same way a send does -- see the class remarks) or a failed transaction + // (ResolveClaim returns 0 on any exception) is never double- or over-counted. + RaiseCorrupted(ResolveClaim(claim.Ids, ClaimOutcome.Remove, + undeserializableIds: claim.UndeserializableIds)); return; } @@ -430,13 +436,18 @@ public async Task ProcessBatchAsync(int maxItems, { // A throwing send is a connectivity-style failure (we don't know anything about // this batch's deliverability), exactly like SendResult.RetryableFailure -- see - // ResolveOutcome. Must NOT count against the retry-attempt ceiling. + // ResolveOutcome. Must NOT count against the retry-attempt ceiling. Undeserializable + // rows in the same claim are still removed regardless (see ResolveClaim) -- passed + // through here the same as the non-throwing path below, so a corrupt row cannot + // loop forever just because the batch alongside it happened to throw. Debug.WriteLine("SqliteEventSpool.ProcessBatch: send threw, leaving batch for retry: " + e); - ResolveClaim(claim.Ids, ClaimOutcome.ReleaseOnly); + RaiseCorrupted(ResolveClaim(claim.Ids, ClaimOutcome.ReleaseOnly, + undeserializableIds: claim.UndeserializableIds)); return; } - ResolveClaim(claim.Ids, ResolveOutcome(result), undeserializableIds: claim.UndeserializableIds); + RaiseCorrupted(ResolveClaim(claim.Ids, ResolveOutcome(result), + undeserializableIds: claim.UndeserializableIds)); } catch (Exception e) { @@ -467,6 +478,12 @@ private enum ClaimOutcome CountAttemptAndRelease } + // MixpanelClient.SendBatchGuardedAsync remaps RetryableRejection to RetryableFailure before it + // ever reaches here when running BoundedDrainAsync's compressed, many-attempts-in-seconds + // drain loop (see its doc comment) -- so a rejection there still leaves the batch spooled for + // retry, exactly like a normal tick, but never reaches the CountAttemptAndRelease case below + // and so does not additionally erode the attempt ceiling. This method only ever sees the + // post-remap result and needs no awareness of which caller produced it. private static ClaimOutcome ResolveOutcome(SendResult result) { switch (result) @@ -574,41 +591,55 @@ private ClaimedBatch ClaimBatch(int maxItems, long maxBytes) // DELETE. Otherwise: clear the lease so the next drain retries the batch, in order -- and, // ONLY for CountAttemptAndRelease, also bump the attempt counter and drop any event whose // count just reached the configured max (dropped NOW, not on some future failure, since - // the check runs after the increment). - private void ResolveClaim(List ids, ClaimOutcome outcome, List undeserializableIds = null) + // the check runs after the increment). Returns how many of undeserializableIds THIS call + // actually deleted (0 on any exception, including a partial failure that rolled the whole + // transaction back) -- deliberately not just undeserializableIds.Count, since a concurrent + // Purge() can remove the same row first (Purge's DELETE races ClaimBatch/ResolveClaim the + // same way a send does -- see the class remarks), which must not be double-counted as a + // corruption drop by the caller's RaiseCorrupted. + // undeserializableIds is always claim.UndeserializableIds from ProcessBatchAsync's ClaimBatch + // call -- never null (ClaimedBatch initializes it to an empty list) -- so callers need not + // (and must not) pass null; DeleteByIdLocked already no-ops on an empty list, so every branch + // below can unconditionally split it out without a separate empty-check first. + private int ResolveClaim(List ids, ClaimOutcome outcome, List undeserializableIds) { // Populated inside the lock below, then used to raise ItemDroppedByRetryExhaustion // AFTER the lock is released -- same pattern Enqueue uses for ItemDroppedByCap. List retryExhaustedIds = null; + var corruptedRemoved = 0; try { lock (_sync) { if (_disposed || _connection == null) - return; + return 0; using (var txn = _connection.BeginTransaction(deferred: false)) { if (outcome == ClaimOutcome.Remove) { - // Delivered/PoisonDrop path. A straight DELETE -- attempts is never - // touched here, so a batch that ultimately succeeds (even after some - // number of prior retryable verdicts of either kind) never counts - // against the retry ceiling. - DeleteByIdLocked(txn, ids); + // Delivered/PoisonDrop path (or "every claimed row was undeserializable"). + // Split the undeserializable subset out of the single DELETE the old code + // used, purely so its own affected-row count is available to return below + // (a no-op DELETE, no round trip, when it's empty -- see DeleteByIdLocked) + // -- attempts is never touched either way, so a batch that ultimately + // succeeds (even after some number of prior retryable verdicts of either + // kind) never counts against the retry ceiling. + corruptedRemoved = DeleteByIdLocked(txn, undeserializableIds); + var remaining = new List(ids); + remaining.RemoveAll(undeserializableIds.Contains); + DeleteByIdLocked(txn, remaining); } else { // Undeserializable rows are removed regardless of which retryable // verdict this is: they can never be delivered, so retrying them // forever would wedge the head of the queue. - if (undeserializableIds != null && undeserializableIds.Count > 0) - DeleteByIdLocked(txn, undeserializableIds); + corruptedRemoved = DeleteByIdLocked(txn, undeserializableIds); var toRelease = new List(ids); - if (undeserializableIds != null) - toRelease.RemoveAll(undeserializableIds.Contains); + toRelease.RemoveAll(undeserializableIds.Contains); if (toRelease.Count > 0) { @@ -667,24 +698,30 @@ private void ResolveClaim(List ids, ClaimOutcome outcome, List undes catch (Exception e) { Debug.WriteLine("SqliteEventSpool.ResolveClaim failed: " + e); - return; + return 0; } // Raised outside the lock: handlers are the caller's code (see RaiseDropped). if (retryExhaustedIds != null && retryExhaustedIds.Count > 0) RaiseRetryExhausted(retryExhaustedIds.Count); + + return corruptedRemoved; } - private void DeleteByIdLocked(SqliteTransaction txn, List ids) + // Returns the number of rows actually deleted (per SQLite's own affected-row count), not + // merely ids.Count -- a concurrent Purge() or an already-expired row can mean fewer rows + // existed to delete than ids named. See ResolveClaim's corruptedRemoved for why callers + // rely on this being the true count rather than an assumed one. + private int DeleteByIdLocked(SqliteTransaction txn, List ids) { if (ids.Count == 0) - return; + return 0; using (var cmd = _connection.CreateCommand()) { cmd.Transaction = txn; cmd.CommandText = "DELETE FROM events WHERE id IN (" + IdList(ids) + ");"; - cmd.ExecuteNonQuery(); + return cmd.ExecuteNonQuery(); } } @@ -698,10 +735,13 @@ private void DeleteByIdLocked(SqliteTransaction txn, List ids) /// not have to stop at the first non-expired event -- it removes every expired event wherever it sits, /// so an out-of-order timestamp cannot shield older events from the retention floor. A /// corrupt payload is irrelevant here too: the age lives in a column, so nothing has to be - /// deserialized to be dated. + /// deserialized to be dated. Raises once per row actually + /// removed this way, so a drop via this path is never silently uncounted in Statistics. /// public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) { + var dropped = 0; + try { var cutoffMs = (now - maxAge).ToUnixTimeMilliseconds(); @@ -715,9 +755,13 @@ public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) { cmd.CommandText = "DELETE FROM events WHERE time_unix_ms < @cutoff;"; cmd.Parameters.AddWithValue("@cutoff", cutoffMs); - cmd.ExecuteNonQuery(); + dropped = cmd.ExecuteNonQuery(); } } + + // Raised outside the lock: handlers are the caller's code (see RaiseDropped). + if (dropped > 0) + RaiseExpired(dropped); } catch (Exception e) { @@ -730,7 +774,21 @@ public void TrimExpired(TimeSpan maxAge, DateTimeOffset now) /// DELETE removes the rows; VACUUM rebuilds the database file so the purged bytes are /// actually gone rather than lingering as free pages; the WAL checkpoint/truncate does the /// same for the write-ahead log. All three matter for a CONSENT purge, where "marked - /// consumed" is not good enough. + /// consumed" is not good enough -- and all three run synchronously, in that order, before + /// this method returns: a version that backgrounded the VACUUM/checkpoint step was tried and + /// reverted, because it traded a small, bounded, measured cost (below) for a real regression + /// -- a crash or kill between this method returning and that background work completing + /// would leave the "purged" bytes physically recoverable on disk, which is precisely the + /// guarantee a consent purge exists to provide -- while not even fully solving the UI-thread + /// concern it was meant to address, since the background work still takes _sync, so + /// any other call into this spool (e.g. the next Track()) would simply block on that + /// instead. Measured directly (fill a temp SQLite db with ~2KB rows to ~100MB -- twice this + /// class's default 50MB byte cap -- then time DELETE FROM events + VACUUM + PRAGMA + /// wal_checkpoint(TRUNCATE) back to back): the three together took on the order of 60-80ms + /// (roughly 60ms delete, under 20ms checkpoint, VACUUM itself near-instant on a freshly + /// emptied table) -- not the multi-second-to-45s stalls the old DiskQueue-backed design + /// risked (see the class remarks), which is what justified paying a synchronous cost here at + /// all. Reproduce by timing those three statements against a similarly-sized spool.db. /// Contrast the old DiskQueue-backed EventSpool's Purge, which had to dispose the /// queue, delete the whole directory and reopen -- briefly dropping its cross-process lock, /// and degrading to a no-op spool if another process stole it in that window. diff --git a/src/DesktopAnalytics/Analytics.cs b/src/DesktopAnalytics/Analytics.cs index ac2f661..d4a0d49 100644 --- a/src/DesktopAnalytics/Analytics.cs +++ b/src/DesktopAnalytics/Analytics.cs @@ -742,28 +742,10 @@ public static Task ReportExceptionAsync(Exception e, CancellationToken cancellat /// public static void ReportException(Exception e, Dictionary moreProperties) { - if (!AllowTracking) - return; - - s_exceptionCount++; - - // we had an incident where some problem caused a user to emit hundreds of thousands of exceptions, - // in the background, blowing through our Analytics service limits and getting us kicked off. - if (s_exceptionCount > kMaxExceptionReportsPerRun) - { + var props = BuildExceptionReportProperties(e, moreProperties); + if (props == null) return; - } - var props = new JsonObject - { - { "Message", e.Message }, - { "Stack Trace", e.StackTrace } - }; - if (moreProperties != null) - { - foreach (var key in moreProperties.Keys) - props.Add(key, moreProperties[key]); - } TrackWithApplicationProperties("Exception", props); } @@ -777,17 +759,29 @@ public static void ReportException(Exception e, Dictionary moreP public static Task ReportExceptionAsync(Exception e, Dictionary moreProperties, CancellationToken cancellationToken = default) { - if (!AllowTracking) + var props = BuildExceptionReportProperties(e, moreProperties); + if (props == null) return Task.CompletedTask; + return TrackWithApplicationPropertiesAsync("Exception", props, cancellationToken); + } + + // Shared throttle/property-building logic behind ReportException and ReportExceptionAsync, + // so the two can never silently drift. Returns null if the exception should not be tracked + // at all -- either AllowTracking is off, or this run is over kMaxExceptionReportsPerRun (see + // the incident note below) -- in which case the caller must not track anything. + private static JsonObject BuildExceptionReportProperties(Exception e, + Dictionary moreProperties) + { + if (!AllowTracking) + return null; + s_exceptionCount++; // we had an incident where some problem caused a user to emit hundreds of thousands of exceptions, // in the background, blowing through our Analytics service limits and getting us kicked off. if (s_exceptionCount > kMaxExceptionReportsPerRun) - { - return Task.CompletedTask; - } + return null; var props = new JsonObject { @@ -799,7 +793,7 @@ public static Task ReportExceptionAsync(Exception e, Dictionary foreach (var key in moreProperties.Keys) props.Add(key, moreProperties[key]); } - return TrackWithApplicationPropertiesAsync("Exception", props, cancellationToken); + return props; } private static JsonObject MakeSegmentIOProperties(Dictionary properties) @@ -1169,19 +1163,10 @@ private static void TrackWithApplicationProperties(string eventName, JsonObject if (!AllowTracking) return; - if (properties == null) - properties = new JsonObject(); - - foreach (var p in s_singleton._propertiesThatGoWithEveryEvent) - { - properties.Remove(p.Key); - properties.Add(p.Key, p.Value ?? Empty); - } - s_singleton._client.Track( s_settings.IdForAnalytics, eventName, - properties + MergeApplicationProperties(properties) ); } @@ -1198,6 +1183,18 @@ private static Task TrackWithApplicationPropertiesAsync(string eventName, if (!AllowTracking) return Task.CompletedTask; + return s_singleton._client.TrackAsync( + s_settings.IdForAnalytics, + eventName, + MergeApplicationProperties(properties), + cancellationToken + ); + } + + // Shared defaulting/merge logic behind TrackWithApplicationProperties and its async + // counterpart, so the two can never silently drift. + private static JsonObject MergeApplicationProperties(JsonObject properties) + { if (properties == null) properties = new JsonObject(); @@ -1207,12 +1204,7 @@ private static Task TrackWithApplicationPropertiesAsync(string eventName, properties.Add(p.Key, p.Value ?? Empty); } - return s_singleton._client.TrackAsync( - s_settings.IdForAnalytics, - eventName, - properties, - cancellationToken - ); + return properties; } /// diff --git a/src/DesktopAnalytics/Statistics.cs b/src/DesktopAnalytics/Statistics.cs index 239e8fa..a53796e 100644 --- a/src/DesktopAnalytics/Statistics.cs +++ b/src/DesktopAnalytics/Statistics.cs @@ -2,15 +2,25 @@ { public struct Statistics { - public Statistics(int submitted, int succeeded, int failed) + public Statistics(int submitted, int succeeded, int failed) : this(submitted, succeeded, failed, 0) + { + } + + /// Events dropped by an age-based retention floor rather than a + /// delivery failure -- distinct from , which is events that were + /// actually attempted (or attempted and refused at enqueue) and did not succeed. A client + /// with no such retention floor always reports 0 here. + public Statistics(int submitted, int succeeded, int failed, int expired) { Submitted = submitted; Succeeded = succeeded; Failed = failed; + Expired = expired; } public int Submitted { get; } public int Succeeded { get; } public int Failed { get; } + public int Expired { get; } } } \ No newline at end of file diff --git a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs index 52f7502..1b38446 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/EventSpoolContractTests.cs @@ -303,9 +303,15 @@ public void Enqueue_ExceedingMaxItems_RaisesItemDroppedByCapPerDroppedEvent() [Test] public async Task Enqueue_ExceedingMaxSpoolBytes_DropsOldestKeepsNewest() { + // A shared, fixed timestamp keeps every event's serialized length identical: Time's + // fractional-seconds component can otherwise vary by a byte or two between events + // created microseconds apart, which would make the cap computed below land a few bytes + // off from the four events' real total and flip which one the byte cap evicts. + var now = DateTimeOffset.UtcNow; var events = new[] { - MakeEvent("Event-A"), MakeEvent("Event-B"), MakeEvent("Event-C"), MakeEvent("Event-D") + MakeEventAt("Event-A", now), MakeEventAt("Event-B", now), + MakeEventAt("Event-C", now), MakeEventAt("Event-D", now) }; long cap = events[0].ToBytes().Length + events[1].ToBytes().Length + events[2].ToBytes().Length; @@ -371,6 +377,27 @@ public void TrimExpired_EventOlderThanMaxAge_IsDropped() } } + [Test] + public void TrimExpired_EventOlderThanMaxAge_RaisesItemDroppedByExpiry() + { + using (var spool = Create(10)) + { + var now = DateTimeOffset.UtcNow; + spool.Enqueue(MakeEventAt("Old1", now - TimeSpan.FromDays(90))); + spool.Enqueue(MakeEventAt("Old2", now - TimeSpan.FromDays(90))); + spool.Enqueue(MakeEventAt("Fresh", now - TimeSpan.FromDays(10))); + + var expiredCount = 0; + spool.ItemDroppedByExpiry += () => Interlocked.Increment(ref expiredCount); + + spool.TrimExpired(TimeSpan.FromDays(60), now); + + Assert.AreEqual(1, spool.ApproximateCount, "only the fresh event should remain"); + Assert.AreEqual(2, expiredCount, + "ItemDroppedByExpiry must fire once per expired event actually removed"); + } + } + [Test] public void TrimExpired_EventWithinMaxAge_IsRetained() { @@ -491,6 +518,108 @@ public void Purge_EmptySpool_DoesNotThrow() } } + [Test] + public void Purge_ThenImmediateDispose_PurgedDataStillNotOnDisk() + { + // Purge() runs its DELETE/VACUUM/checkpoint synchronously (see Purge's doc comment for + // why an earlier backgrounded version of this was tried and reverted), so by the time it + // returns the disk-scrub is already complete -- an immediately-following Dispose() (no + // delay at all, as below) has nothing left to race or skip. This pins that guarantee + // down as an explicit regression test rather than leaving it as something only true "by + // construction" of Purge() being synchronous. + var dir = Path.Combine(Path.GetTempPath(), "SpoolContract_ImmediateDispose_" + Guid.NewGuid()); + try + { + using (var spool = _factory.Create(dir, 10, int.MaxValue, long.MaxValue)) + { + spool.Enqueue(MakeEvent("SecretEventName")); + spool.Purge(); + } + + foreach (var file in Directory.GetFiles(dir, "*", SearchOption.AllDirectories)) + { + string content; + try + { + using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete)) + using (var reader = new StreamReader(stream, System.Text.Encoding.UTF8)) + content = reader.ReadToEnd(); + } + catch (IOException) + { + continue; + } + + StringAssert.DoesNotContain("SecretEventName", content, + "purged event data must not survive anywhere on disk in " + file + + " even when Dispose() immediately follows Purge()"); + } + } + finally + { + try { Directory.Delete(dir, true); } catch { } + } + } + + // ---- CORRUPT / UNDESERIALIZABLE PAYLOADS ------------------------------------------------- + // + // Regression coverage for two related fixes: (1) ProcessBatchAsync's exception-path call to + // ResolveClaim used to omit undeserializableIds, so a corrupt row claimed alongside events + // whose send() call threw would never be removed -- it would loop forever, re-claimed and + // re-failing to deserialize on every subsequent drain. (2) IEventSpool.ItemDroppedByCorruption + // (consumed by MixpanelClient to keep Statistics.Failed accurate) did not exist at all, so an + // undeserializable row vanished from the spool with no way for a caller to ever count it. + + [Test] + public async Task ProcessBatch_AllRowsCorrupt_RemovedWithoutCallingSendAndRaisesItemDroppedByCorruption() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("A")); + SpoolCorruptionTestHelper.CorruptAllPayloads(_spoolDir); + + var corruptedCount = 0; + spool.ItemDroppedByCorruption += () => Interlocked.Increment(ref corruptedCount); + + var sendCalled = false; + await spool.ProcessBatchAsync(100, (batch, ct) => + { + sendCalled = true; + return Task.FromResult(SendResult.Delivered); + }); + + Assert.IsFalse(sendCalled, + "a batch that is entirely undeserializable must never reach send()"); + Assert.AreEqual(0, spool.ApproximateCount, "the corrupt row must be removed"); + Assert.AreEqual(1, corruptedCount, + "ItemDroppedByCorruption must fire once for the corrupt row"); + } + } + + [Test] + public async Task ProcessBatch_MixedBatchAndSendThrows_CorruptRowStillRemovedAndValidRowReleased() + { + using (var spool = Create(10)) + { + spool.Enqueue(MakeEvent("Good")); + SpoolCorruptionTestHelper.CorruptOldestPayload(_spoolDir); + spool.Enqueue(MakeEvent("AlsoGood")); + + var corruptedCount = 0; + spool.ItemDroppedByCorruption += () => Interlocked.Increment(ref corruptedCount); + + await spool.ProcessBatchAsync(100, + (batch, ct) => throw new InvalidOperationException("simulated send failure")); + + Assert.AreEqual(1, corruptedCount, + "the corrupt row must still be removed even though send() threw for the batch"); + Assert.AreEqual(1, spool.ApproximateCount, + "the corrupt row is gone; the still-valid row remains, released for a later retry"); + CollectionAssert.AreEqual(new[] { "AlsoGood" }, await DrainAllDelivered(spool)); + } + } + // ---- ROBUSTNESS ------------------------------------------------------------------------- [Test] diff --git a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs index 3a3f3ec..828b764 100644 --- a/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs +++ b/src/DesktopAnalyticsTests.Mixpanel/MixpanelClientTests.cs @@ -583,6 +583,123 @@ public async Task DrainRepeatedly_SenderAlwaysPlainRetryableFailure_NeverErodesA } } + [Test] + public void Flush_SenderAlwaysRetryableRejection_DoesNotErodeAttemptCeilingDuringBoundedDrain() + { + // Regression test: BoundedDrainAsync (which Flush()/ShutDown() drive) can make up to + // kBoundedDrainMaxAttempts (20) attempts back-to-back within its 5s bound -- far faster + // than the once-per-30s-tick cadence the retry ceiling was sized against. If it counted + // RetryableRejection against the ceiling the same way a normal timer tick does, a single + // Flush() call against a server that is quickly rate-limiting/rejecting could burn + // through this tiny ceiling in seconds and drop an otherwise-good event. + const int maxAttempts = 3; + using (var spool = new SqliteEventSpool(_spoolDir, 10, maxAttempts: maxAttempts)) + { + var sender = new AlwaysResultSender(SendResult.RetryableRejection); + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + + var completed = Task.Run(() => client.Flush()).Wait(TimeSpan.FromSeconds(15)); + + Assert.IsTrue(completed, "Flush() did not return within the timeout"); + Assert.Greater(sender.CallCount, maxAttempts, + "the bounded drain loop must have made more attempts than maxAttempts, to prove " + + "this is a real exercise of the exemption rather than a vacuously-passing test"); + Assert.AreEqual(1, spool.ApproximateCount, + "a RetryableRejection storm during a single bounded Flush()/ShutDown() drain " + + "must not erode the per-event retry ceiling the way a normal timer tick does"); + Assert.AreEqual(0, client.Statistics.Failed, + "the event must not have been dropped"); + } + } + + // ---- UNDESERIALIZABLE (CORRUPT) SPOOLED EVENTS ------------------------------------------- + // + // Regression coverage for IEventSpool.ItemDroppedByCorruption: a claimed row whose payload + // cannot be deserialized (on-disk corruption, or a payload from a since-upgraded + // incompatible schema) is removed by the spool so it cannot wedge the queue -- but before + // this fix, that removal did not increment Statistics.Failed, silently breaking the + // invariant Statistics.Submitted == Statistics.Succeeded + Statistics.Failed (Submitted was + // already incremented when the event was originally Track()'d). + + [Test] + public async Task DrainOnce_EventCorruptedOnDiskBeforeDrain_RemovedAndCountedAsFailed() + { + using (var spool = new SqliteEventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + var sender = new AlwaysResultSender(SendResult.Delivered); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + Assert.AreEqual(1, spool.ApproximateCount); + + SpoolCorruptionTestHelper.CorruptAllPayloads(_spoolDir); + + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, + "an undeserializable row must be removed rather than retried forever"); + Assert.AreEqual(0, sender.CallCount, + "the sender must never be called for a batch that was entirely undeserializable"); + Assert.AreEqual(1, client.Statistics.Submitted); + Assert.AreEqual(0, client.Statistics.Succeeded); + Assert.AreEqual(1, client.Statistics.Failed, + "a corrupt/undeserializable row must count as Failed so Submitted == Succeeded + Failed"); + } + } + + // ---- AGE-BASED RETENTION EXPIRY ----------------------------------------------------------- + // + // Regression coverage for IEventSpool.ItemDroppedByExpiry: an event aged out by TrimExpired's + // 60-day retention floor is removed so it cannot accumulate forever. Unlike a corrupt row or + // a retry-ceiling drop, this is not a delivery failure -- it is counted separately as + // Statistics.Expired rather than Statistics.Failed (see OnItemExpired's doc comment). + + [Test] + public async Task DrainOnce_EventAgedOutByRetentionFloor_RemovedAndCountedAsExpired() + { + using (var spool = new SqliteEventSpool(_spoolDir, 10)) + { + var client = new MixpanelClient(); + var sender = new AlwaysResultSender(SendResult.Delivered); + client.InitializeForTest(spool, sender); + + client.Track("user-1", "Save", null); + Assert.AreEqual(1, spool.ApproximateCount); + + // Backdate the row directly (bypassing Track()'s own time-stamping) so + // DrainOnceCoreAsync's automatic TrimExpired call (60-day floor) considers it expired + // before ProcessBatchAsync ever gets to claim it. + using (var conn = new Microsoft.Data.Sqlite.SqliteConnection( + "Data Source=" + Path.Combine(_spoolDir, "spool.db"))) + { + conn.Open(); + using (var cmd = conn.CreateCommand()) + { + cmd.CommandText = "UPDATE events SET time_unix_ms = 0;"; + cmd.ExecuteNonQuery(); + } + } + + await client.DrainOnceAsync(); + + Assert.AreEqual(0, spool.ApproximateCount, + "an aged-out event must be removed by the retention floor"); + Assert.AreEqual(0, sender.CallCount, + "the sender must never be called for an event trimmed before it could be claimed"); + Assert.AreEqual(1, client.Statistics.Submitted); + Assert.AreEqual(0, client.Statistics.Succeeded); + Assert.AreEqual(0, client.Statistics.Failed, + "aging out is a retention decision, not a delivery failure -- must not count as Failed"); + Assert.AreEqual(1, client.Statistics.Expired, + "an event dropped by the retention floor must count as Expired so " + + "Submitted == Succeeded + Failed + Expired"); + } + } + // ---- SHUTDOWN OFFLINE ------------------------------------------------------------------- [Test] @@ -1047,6 +1164,43 @@ public async Task DrainOnce_WithRealCircuitBreakerAndFixedMinimumThroughput_Open } } + [Test] + public async Task DrainOnce_WithRealCircuitBreakerAndZeroDelay_SustainedRetryableRejection_TripsBreakerAndStopsCallingSender() + { + // Regression test for the bug where the Polly pipeline's ShouldHandle predicate only + // checked SendResult.RetryableFailure, never SendResult.RetryableRejection -- so a + // sustained run of HTTP 408/429/5xx responses (which DID reach the server, unlike a + // connectivity failure) was entirely invisible to both in-tick retry and the circuit + // breaker's failure tracking. The client would keep hammering an already-struggling or + // rate-limiting server every flush tick with no backoff, exactly what the breaker + // exists to prevent. + using (var spool = new SqliteEventSpool(_spoolDir, 10)) + { + var sender = new AlwaysResultSender(SendResult.RetryableRejection); + var pipeline = MixpanelClient.BuildDefaultPipeline(TimeProvider.System, + maxRetryAttempts: 1, retryDelay: TimeSpan.Zero); + + var client = new MixpanelClient(); + client.InitializeForTest(spool, sender, pipeline: pipeline, batchSize: 1); + + for (var i = 0; i < 8; i++) + { + client.Track("user-1", "Save-" + i, null); + await client.DrainOnceAsync(); + } + + var callsAfterWarmup = sender.CallCount; + + client.Track("user-1", "OneMore", null); + await client.DrainOnceAsync(); + + Assert.AreEqual(callsAfterWarmup, sender.CallCount, + "once the circuit breaker is open, the sender must not be called again for a " + + "sustained RetryableRejection outage either"); + Assert.Greater(spool.ApproximateCount, 0); + } + } + // Breaker recovery: half-open -> closed. Once the circuit breaker has tripped (same // single-fully-failing-tick recipe as the regression test above), let its BreakDuration // elapse and drive a successful send, proving delivery resumes rather than the breaker diff --git a/src/DesktopAnalyticsTests.Mixpanel/SpoolCorruptionTestHelper.cs b/src/DesktopAnalyticsTests.Mixpanel/SpoolCorruptionTestHelper.cs new file mode 100644 index 0000000..ad7dac5 --- /dev/null +++ b/src/DesktopAnalyticsTests.Mixpanel/SpoolCorruptionTestHelper.cs @@ -0,0 +1,32 @@ +using System.IO; +using Microsoft.Data.Sqlite; + +namespace DesktopAnalyticsTests +{ + // Shared by EventSpoolContractTests and MixpanelClientTests: both simulate an + // undeserializable/corrupt spooled row by opening a second raw connection to the same + // spool.db and overwriting payload bytes directly, bypassing the spool's own Enqueue path + // (which would refuse to write anything but valid serialized events). + internal static class SpoolCorruptionTestHelper + { + public static void CorruptAllPayloads(string spoolDir) => + Execute(spoolDir, "UPDATE events SET payload = X'00', len = 1;"); + + public static void CorruptOldestPayload(string spoolDir) => + Execute(spoolDir, "UPDATE events SET payload = X'00', len = 1 " + + "WHERE id = (SELECT MIN(id) FROM events);"); + + private static void Execute(string spoolDir, string sql) + { + using (var conn = new SqliteConnection("Data Source=" + Path.Combine(spoolDir, "spool.db"))) + { + conn.Open(); + using (var cmd = conn.CreateCommand()) + { + cmd.CommandText = sql; + cmd.ExecuteNonQuery(); + } + } + } + } +}