I suspect that using sync.Map instead of the map + rwmutex should get some better performance. If want even better performance we can get a specialized fork of sync.Map that doesn't operate on interface types, but swarm types directly, negating need for interface type conversions.
This probably requires implementing a benchmark to properly analyze if this will lead to a performance improvement