diff --git a/docs/README.md b/docs/README.md index 18b778b93..d1e5287e7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,7 @@ _Atomic Data is a specification for sharing, modifying and modeling graph data._ View it on [docs.atomicdata.dev](https://docs.atomicdata.dev). -If you're looking for **implementations of Atomic Data**, check out [atomic-server](https://github.com/atomicdata-dev/atomic-server) (server + cli + lib written in Rust) and [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) (react / typescript). +If you're looking for **implementations of Atomic Data**, check out [atomic-server](https://github.com/atomicdata-dev/atomic-server) (server + cli + lib + web app, written in Rust / TypeScript). ## About this repo diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index cd2184026..bb354ef71 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -11,10 +11,14 @@ - [AtomicServer](atomic-server.md) - [When (not) to use it](atomicserver/when-to-use.md) + - [Local-first](atomicserver/local-first.md) - [Installation](atomicserver/installation.md) - [Using the GUI](atomicserver/gui.md) - [Tables](atomicserver/gui/tables.md) + - [Sync & pairing](atomicserver/gui/sync-and-pairing.md) + - [Presence & collaboration](atomicserver/gui/presence.md) - [AI and Atomic Assistant](atomicserver/gui/ai-and-atomic-assistant.md) + - [Headless CMS, drafts & forks](atomicserver/cms.md) - [Plugins](plugins.md) - [Creating Plugins](plugins/creating-plugins.md) - [Custom Views](plugins/custom-views.md) @@ -83,6 +87,7 @@ - [Commits (writing data)](commits/intro.md) - [Concepts](commits/concepts.md) - [Compared to](commits/compare.md) + - [Forks (proposed edits)](commits/suggestions.md) - [WebSockets](websockets.md) - [Endpoints](endpoints.md) - [Collections, filtering, sorting](schema/collections.md) diff --git a/docs/src/atomic-data-overview.md b/docs/src/atomic-data-overview.md index 97eece9c1..cc0393316 100644 --- a/docs/src/atomic-data-overview.md +++ b/docs/src/atomic-data-overview.md @@ -6,6 +6,7 @@ ![Venn diagram showing Atomic Data is the combination of JSON, RDF and Type-Safety](assets/venn.svg) Atomic Data uses links to connect pieces of data, and therefore makes it easier to connect datasets to each other - even when these datasets exist on separate machines. +Resources are addressed with portable [`did:ad` identifiers](did.md), edited with signed [CRDT commits](commits/intro.md), and can live [entirely local-first](atomicserver/local-first.md) before syncing to peers. ## AtomicServer @@ -37,11 +38,11 @@ Atomic Data Extended is a set of extra modules (on top of Atomic Data Core) that ## Tools & libraries -- Browser app [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) ([demo on atomicdata.dev](https://atomicdata.dev)) -- Build a react app using [typescript & react libraries](https://github.com/atomicdata-dev/atomic-data-browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) +- Web app (data-browser) in the [`atomic-server` monorepo](https://github.com/atomicdata-dev/atomic-server/tree/master/browser) ([demo on atomicdata.dev](https://atomicdata.dev)) +- Typescript libraries: [`@tomic/lib`](js.md), [`@tomic/react`](usecases/react.md), [`@tomic/svelte`](svelte.md) - Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-server) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server`) -- Discover the command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) -- Use the Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) +- Command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) +- Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) ## Get involved @@ -50,12 +51,13 @@ Make sure to [join our Discord](https://discord.gg/a72Rv2P) if you'd like to dis ## Status Keep in mind that none of the Atomic Data projects has reached a v1, which means that breaking changes can happen. +The 0.41 line is the local-first / `did:ad` release β€” see the [roadmap](roadmap.md) and [changelog](https://github.com/atomicdata-dev/atomic-server/blob/develop/CHANGELOG.md). ## Reading these docs This is written mostly as a book, so reading it in the order of the Table of Contents will probably give you the best experience. That being said, feel free to jump around - links are often used to refer to earlier discussed concepts. -If you encounter any issues while reading, please leave an [issue on Github](https://github.com/ontola/atomic-data/issues). +If you encounter any issues while reading, please leave an [issue on Github](https://github.com/atomicdata-dev/atomic-server/issues). Use the arrows on the side / bottom to go to the next page. {{#include SUMMARY.md}} diff --git a/docs/src/atomic-server.md b/docs/src/atomic-server.md index 764d8c78c..49407a6be 100644 --- a/docs/src/atomic-server.md +++ b/docs/src/atomic-server.md @@ -8,7 +8,7 @@ AtomicServer is a real-time headless CMS, graph database server for storing and It's free, open source (MIT license), and has a ton of features: -- 🏠 **Local-first**: create and edit data with no server at all. Resources are addressed by [`did:ad` identifiers](https://docs.atomicdata.dev/did) and resolve peer-to-peer over the Mainline DHT, so an identity is a keypair you hold rather than an account on someone else's machine. Edits are signed CRDT commits that merge when you reconnect. +- 🏠 **Local-first**: create and edit data with no server at all. Resources are addressed by [`did:ad` identifiers](https://docs.atomicdata.dev/did) β€” an identity is a keypair you hold rather than an account on someone else's machine. Peer discovery uses pkarr; devices sync over [WebSocket and Iroh](https://docs.atomicdata.dev/websockets). Edits are signed CRDT commits that merge when you reconnect. See [Local-first](atomicserver/local-first.md). - πŸ”’ **Encrypted at rest, per agent**: each agent's in-browser database is encrypted with XChaCha20-Poly1305, under a key wrapped by that agent's own private key. Signing out leaves the cache in place but unreadable to the next session β€” no wipe required. - πŸ”‘ **Passkey-backed recovery**: a WebAuthn passkey wraps the backup of your agent secret (Argon2id + AES-GCM), so onboarding hands you nothing to write down, and a lost device doesn't have to mean a lost account. - πŸš€ **Fast** (less than 1ms median response time on my laptop), powered by [actix-web](https://github.com/actix/actix-web) and [redb](https://github.com/cberner/redb) @@ -18,12 +18,14 @@ It's free, open source (MIT license), and has a ton of features: - βš™οΈ **Restful API**, with [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html) responses. - πŸ”Ž **Full-text search** with fuzzy search and various operators, often <3ms responses. Powered by [tantivy](https://github.com/quickwit-inc/tantivy). - ✨ **AI** with [MCP](https://modelcontextprotocol.io/) support, use any model via OpenRouter or host your own with Ollama. -- πŸ—„οΈ **Tables**, with strict schema validation, keyboard support, copy / paste support. Similar to Airtable. +- πŸ—„οΈ **Tables**, with strict schema validation, kanban / calendar / timer views, filters, templates, computed columns, aggregates, keyboard and copy/paste support. Similar to Airtable. See [Tables](atomicserver/gui/tables.md). - πŸ“„ **Documents**, collaborative, rich text, similar to Google Docs / Notion. - πŸ’¬ **Group chat**, performant and flexible message channels with attachments, search and replies. +- πŸ‘οΈ **Presence**: see who is viewing a resource, follow someone through the app, live cursors while co-editing. See [Presence](atomicserver/gui/presence.md). +- πŸ“° **Headless CMS**: drafts by folder location, forks for proposed edits, website templates, and [i18n](schema/translations.md). See [CMS](atomicserver/cms.md). - πŸ“‚ **File management**: Upload, download and preview attachments. - πŸ’Ύ **Event-sourced versioning** / history powered by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) -- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). +- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). Pair devices with a QR code β€” see [Sync & pairing](atomicserver/gui/sync-and-pairing.md). - 🧰 **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). - πŸ“– **Pagination, sorting and filtering** queries using [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). - πŸ” **Authorization** (read / write permissions) and Hierarchical structures powered by [Atomic Hierarchy](https://docs.atomicdata.dev/hierarchy.html) diff --git a/docs/src/atomicserver/cms.md b/docs/src/atomicserver/cms.md new file mode 100644 index 000000000..ac557d898 --- /dev/null +++ b/docs/src/atomicserver/cms.md @@ -0,0 +1,86 @@ +# Headless CMS, drafts, and forks + +AtomicServer is a local-first headless CMS: model content with ontologies, edit +it in the GUI, and render it from any front-end that can read Atomic Data +([Astro guide](../astro-guide/1-index.md), Next.js / SvelteKit website templates, +or your own `@tomic/react` / `@tomic/svelte` app). + +Two generic capabilities make publishing work. Neither is Website-specific. + +## Visibility is location + +A resource is public because it lives somewhere public β€” not because of a +`status` field that can drift from reality. + +On a drive that is not blanket-public you typically have: + +- a **public folder** with `read` granted to the public agent; children inherit + public read through the [hierarchy](../hierarchy.md); +- a private **Drafts** folder (no public grant) for unpublished new content. + +**Publishing** a draft is moving it to a public parent. **Archiving** is moving +it somewhere non-public. No extra class or workflow engine required. + +The drive keeps a well-known Drafts folder (created on first use). New content +you are not ready to show the world goes there. + +## Forks: propose a change + +A **Fork** is a proposed edit to an *existing* resource. + +| Property | Meaning | +| --- | --- | +| `Fork` (class) | Marker: this resource proposes a change to another | +| `originalSubject` | The resource being changed | +| `forkBase` / `forkVersion` | Snapshot of the original at fork time (for merge) | + +Creating a fork copies the resource into a private Forks folder, sets +`originalSubject`, and lets you edit freely. The original is untouched until you +**merge**. Merge squashes the fork onto the original (properties + Loro body +where applicable). While a fork is open, the original can show that pending +forks exist. + +Forks are ordinary resources: they have their own subject, history, and rights. +They carry the content class alongside `Fork` (`isA: [BlogPost, Fork]`), so the +normal editors and site previews render them. + +This replaces the older "Suggestions / Inbox HTTP POST" idea. Cross-agent +"suggest an edit to someone else's drive" can reuse the same Fork mechanism once +authorization for that path is fully productized; today the GUI flow is +optimized for staging your own edits and merging them. + +## Websites and front-ends + +- **Website templates** in the app scaffold a site (pages, nav, locales) as + Atomic resources you can edit in the GUI. +- **Headless front-ends** fetch JSON-AD over HTTP or sync a drive locally, then + render with your stack. See [Build a portfolio with Astro](../astro-guide/1-index.md) + and [`@tomic/template`](../create-template/atomic-template.md). +- **Edit from the webpage** β€” guest edit / local clone flows let a visitor + propose changes without write access to the live drive (forks / local-only + drives); merge happens when an editor accepts them. + +## Internationalization + +Multilingual sites use the two mechanisms in +[Translations & Localization](../schema/translations.md): + +- **Document-level** β€” one resource per language (`language` + `translationOf`), + so each locale has its own path, history, and publish state (a translation in + Drafts is simply unpublished). +- **Field-level** β€” [`LocalizedText`](../schema/datatypes.md#localizedtext) for + short strings inside shared structure (nav labels, feature cards). + +Declare `defaultLanguage` (and optionally `languages`) on the website or drive. + +## Local-first CMS + +Editors can work offline: drafts and forks live in the local database until +sync. See [Local-first](local-first.md) and [Sync & pairing](gui/sync-and-pairing.md). + +## Related + +- [Use case: Headless CMS](../usecases/headless-cms.md) (why / comparison) +- [Tables](gui/tables.md) for structured content collections +- [Hierarchy and authorization](../hierarchy.md) +- [Commits](../commits/intro.md) diff --git a/docs/src/atomicserver/gui.md b/docs/src/atomicserver/gui.md index 73d4a103f..e3d44f3b0 100644 --- a/docs/src/atomicserver/gui.md +++ b/docs/src/atomicserver/gui.md @@ -20,12 +20,12 @@ At the start these will be: ## Creating an agent To create data in AtomicServer you'll need an agent. An agent is like a user account, it signs the changes (commits) you make to data so that others can verify that you made them. -Agents are identified by a DID derived from their public key (`did:ad:{publicKey}`), so they can be used on any AtomicServer without needing to be registered first. +Agents are identified by a DID derived from their public key (`did:ad:agent:{publicKey}`), so they can be used on any AtomicServer without needing to be registered first. To get started, you can use the [demo invite](https://atomicdata.dev/invites/1) on atomicdata.dev, or the `/setup` invite on your own server. Click the "Accept as new user" button. The app will generate a key pair and your Agent will be created. -Navigate to the User Settings page to find your agent secret. This secret is what you use to login, so keep it somewhere safe, like in a password manager. If you lose it you won't be able to recover your account. +Navigate to the User Settings page to find your agent secret. You can also set up **passkey-backed recovery** so a lost device does not mean a lost account. Keep a backup of the secret or passkey enrollment somewhere safe. ### Setting up the root Agent Next, we'll set up the root Agent that has write access to the Drive. @@ -49,3 +49,12 @@ Now that everything is up and running you can start creating some resources. To create a new resource, click the + button in the sidebar. You will be presented with a list of resource types to choose from like Tables, Folders, Documents etc. You can also create your own types by using ontologies. + +## Next in the GUI + +- [Tables](gui/tables.md) β€” views, filters, templates, aggregates +- [Sync & pairing](gui/sync-and-pairing.md) β€” connect devices and always-on peers +- [Presence & collaboration](gui/presence.md) β€” who is here, follow-me, meetings +- [Headless CMS, drafts & forks](cms.md) β€” publishing and proposed edits +- [Local-first](local-first.md) β€” how offline storage works +- [AI and Atomic Assistant](gui/ai-and-atomic-assistant.md) diff --git a/docs/src/atomicserver/gui/presence.md b/docs/src/atomicserver/gui/presence.md new file mode 100644 index 000000000..a4c144486 --- /dev/null +++ b/docs/src/atomicserver/gui/presence.md @@ -0,0 +1,55 @@ +# Presence & collaboration + +AtomicServer shows who else is in a workspace with you β€” without writing that +information into your data. Presence is ephemeral: it rides a drive-scoped relay +with a short TTL and disappears when someone leaves. + +## What you see + +- **Avatars / facepile** in the navbar for people viewing the current resource +- **Sidebar dots** on resources others have open +- **Document cursors** while co-editing rich text (colored per agent) +- **Canvas pointers** on collaborative canvases +- **Table cell rings** on the cell another person has selected +- **"Typing…"** in chat rooms and comments +- **Follow-me** β€” follow someone as they navigate; they can lead a tour through + the app +- **Meetings** β€” a meeting resource with agenda, notes, and live presence for + participants (optional peer-to-peer audio/video in the meeting panel) + +## How it works + +Each drive has a presence channel. Every session announces a small entry: + +- which agent you are +- which resource you are viewing +- optional follow target +- optional view-specific data (selected table cell, canvas coordinates, …) + +Entries expire if the heartbeat stops (~30s TTL). The browser and other clients +subscribe over the [WebSocket protocol](../../websockets.md) (`EPHEMERAL` / +presence frames). Document text cursors use a dedicated Loro ephemeral channel +because cursor positions must track the CRDT oplog; canvas and tables attach +lightweight payloads to the drive presence entry instead. + +Presence is **not** a Commit. It is never stored in the database and does not +appear in history. + +## Follow-me and meetings + +From the presence UI you can follow another session: your view tracks theirs as +they open resources. Meetings are first-class resources with agenda/notes and a +prepare-then-start flow; participants show up through the same presence layer. + +## Multi-device and P2P + +Same-agent presence across your own devices works through the always-on device +(or hub) that relays the drive channel. Direct multi-user presence over Iroh +without a hub is on the roadmap; today collaboration between people typically +shares a workspace on a server or desktop that both can reach. + +## Related + +- [Sync & pairing](sync-and-pairing.md) +- [WebSocket protocol](../../websockets.md) (`EPHEMERAL` tag) +- [Documents and real-time editing](../gui.md) diff --git a/docs/src/atomicserver/gui/sync-and-pairing.md b/docs/src/atomicserver/gui/sync-and-pairing.md new file mode 100644 index 000000000..565e9fb9a --- /dev/null +++ b/docs/src/atomicserver/gui/sync-and-pairing.md @@ -0,0 +1,74 @@ +# Sync & pairing + +AtomicServer treats every machine that holds your data as a **device**. A home +server or hosted instance is just an always-on device. Sync is how workspaces +converge between them. + +For the wire format, see the [WebSocket / sync protocol](../../websockets.md). +This page is the product guide. + +## Concepts + +| Say | Meaning | +| --- | --- | +| **Workspace** | What the schema calls a Drive β€” the unit you sync | +| **Device** | A machine running Atomic (phone, desktop, server) | +| **Always-on device** | A device that never sleeps (AtomicServer, desktop app left running) | +| **Pairing code** | A scannable code that tells one device how to reach another | +| **Sync** | Make two devices hold the same workspace data | + +Rights still decide what each side may read or write, on every transport. Pairing +only solves routing β€” it does not grant access. + +## Browser vs native devices + +- A **browser tab** stores data locally (see [Local-first](../local-first.md)) but + is not an Iroh node. It syncs through an always-on device over WebSocket. +- **Desktop (Tauri), mobile, and AtomicServer** are nodes. They can pair with each + other over Iroh using a pairing code, and also speak WebSocket. + +Two browser tabs with no always-on device between them cannot reach each other. +Say that plainly in onboarding copy when it applies. + +## Pairing two devices + +1. On the device that already has the workspace, open **Sync** (or Devices) and + show a pairing code / QR. +2. On the new device, scan or paste the code. +3. The devices connect (Iroh over a relay if needed), authenticate as agents, and + run a drive sync: version vectors β†’ diffs β†’ Loro snapshots β†’ any missing + file blobs. + +The code is **routing only**. It never carries your agent secret. Getting the +same identity onto a second device is a separate step (passkey restore, or +pasting the agent secret). + +After the first sync, reconnects need no scan β€” known peers are remembered. + +## Connecting a browser to an always-on device + +In the browser, open Sync and add the address of your AtomicServer (for example +`http://localhost:9885` or `https://atomicdata.dev`). Authenticate with your +agent. The browser then: + +- fetches and subscribes to workspaces you can read, +- drains its offline outbox as commits, +- receives live [presence](presence.md) and updates over the WebSocket. + +To move a workspace that was created only in the browser onto a server, use the +promote / push action on the Sync screen so the always-on device receives it. + +## What syncs + +- Resource state as Loro CRDT snapshots and deltas +- Signed [Commits](../../commits/intro.md) (authorship and history) +- File **blobs** by BLAKE3 content hash (`BLOB_REQUEST` / `BLOB_RESPONSE`) + +Ephemeral [presence](presence.md) (cursors, "who is viewing") is relayed live +and is **not** persisted or synced as history. + +## Related + +- [Local-first architecture](../local-first.md) +- [`did:ad:node:` identifiers](../../did.md#node-identifiers) +- [Protocol reference](../../websockets.md) diff --git a/docs/src/atomicserver/gui/tables.md b/docs/src/atomicserver/gui/tables.md index 2468d52f0..44bec70d3 100644 --- a/docs/src/atomicserver/gui/tables.md +++ b/docs/src/atomicserver/gui/tables.md @@ -1,31 +1,111 @@ # Tables -Tables are a way to create and group large amounts of structured data. +Tables are how you create and work with structured data in AtomicServer. +Each table is a folder of resources that share one [Class](../../schema/classes.md); +the class's properties become columns. Every cell is type-safe β€” a number column +cannot hold text. ![Table](../../assets/ui-guide/gui-tables-example.avif) -Tables consist of rows of resources that share the same parent and class. -The properties of that class are represented as columns in the table. -This means that each column is type-safe, a number column can not contain text data for example. +Under the hood a table is ordinary Atomic Data: rows are resources, columns are +properties, views are configuration resources. That is why the same table works +offline (the browser runs the same query engine through WASM), why an assistant +can build one for you, and why a "mini-app" is just a saved configuration rather +than a custom renderer. ## Creating a table -To create a table, click the "+" button in the sidebar or a folder and select "Table". -A dialog will appear prompting you to enter a name. -This name will be used as the title of the table as well as the name for the underlying class of the rows. -This new class will already have a `name` property. Using the `name` property as titles on your resources is a best practice as it helps with compatibility between other tools and makes your resources findable by AtomicServer's search functionality. -If you do not want to use the `name` property, you can remove it by clicking on the three dots in the column header and selecting "Remove". +Click the **+** button in the sidebar or a folder and choose **Table**. -While creating a new table you can also choose to use an existing class by selecting "Use existing class" in the dialog and selecting the desired class from the dropdown. +1. Pick a **template** (Issue Tracker, CRM, Expenses, Time tracker, …) or start + from **Blank**. +2. Give the table a name. That name becomes both the table title and the name of + the row class. Using a `name` column is recommended β€” it keeps resources + findable by search and compatible with other tools. +3. Optionally **use an existing class** instead of creating a new one. -Classes created by tables are automatically added to the default ontology of the drive. Same goes for the columns of the table. -If you chose to use an existing class, any columns created will be added to the ontology containing that class. +Classes and columns created by tables are added to the drive's default ontology. -## Features +## Views -- **Rearange columns**: You can drag and drop columns to rearrange them. -- **Resize columns**: You can resize columns by dragging the edges of the column header. -- **Sort rows**: Click on a column header to sort the rows by that column. -- **Fast keyboard navigation**: Use the arrow keys to navigate the table with hotkeys similar to Excel. -- **Copy & paste multiple cells**: You can copy and paste multiple cells by selecting them and using `Ctrl/Cmd + C` and `Ctrl/Cmd + V`. Pasting also works across different tables and even different applications that support HTML Table data (Most spreadsheet applications). -- **Export data to CSV**: You can export the data of a table to a CSV file by clicking the "Export" button in the top right. +A table can have many **views**. Each view is a different way of looking at the +same rows β€” filters, sort, column order, and renderer are all per-view. + +| Kind | What it shows | +| --- | --- | +| **Table** | Classic grid. Editable cells, sortable columns, aggregates. | +| **Kanban** | Columns grouped by a select property. Drag cards between columns. | +| **Calendar** | Rows placed on a date or datetime property. | +| **Timer** | Start/stop controls against datetime columns, with a live duration. | + +Create a view from the view switcher on the table page. Change a view's kind +from the view settings (table ↔ kanban ↔ calendar ↔ timer) without losing the +underlying data. + +## Filters + +Each view can AND together multiple `(property, operator, value)` filters. +Operators depend on the column datatype (equals, contains, greater/less than, +is empty, …). Filtered aggregates and totals describe the filtered set, not the +whole table. + +Filters are live: when a row is edited so it no longer matches, it leaves the +view immediately (including offline, in the local store). + +## Computed columns, totals, and actions + +- **Computed (derived) columns** β€” values calculated from other columns, e.g. + elapsed duration between two timestamps, days since a date, or quantity Γ— + price. They are configuration, not a separate datatype, so they appear in any + view that includes them. +- **Totals / aggregates** β€” sum, count, average, min, max over every matching + row (not just the page on screen), with optional breakdowns by a column + (exact value, day, or month). Computed by the store; identical online and + offline. +- **Row actions** β€” one-tap buttons on a row that apply a closed set of patches + (`setNow`, `setValue`, `toggle`, `increment`). Each press is an ordinary + [Commit](../../commits/intro.md). +- **Quick-add** β€” a create button that stamps presets onto a new row in one + press (for example: start a timer entry with the current time). + +## Templates + +The New Table dialog offers ready-made starting points. Examples: + +- **Issue Tracker** β€” Status / Assignee / Priority with a kanban board +- **Project tasks** β€” board + calendar + estimated-hours total +- **Time tracker** β€” start/stop timer grouped by project +- **Expenses** β€” receipts, monthly sum, category breakdown +- **Deals (CRM)** β€” pipeline board, deal value, days since contact +- Plus grocery list, plant care, workout log, and more + +Templates are data and configuration only (columns, views, computed columns, +totals). They do not ship a custom renderer β€” anything a template builds, you +can build by hand or ask the [Atomic Assistant](ai-and-atomic-assistant.md) to +build via `create_table` / `list_table_templates`. + +## Localization + +Columns with the [`LocalizedText`](../../schema/datatypes.md#localizedtext) +datatype store a map of language tag β†’ string in one cell. The table editor +shows a language switcher and can split a LocalizedText property into one +column per language. See [Translations & Localization](../../schema/translations.md). + +## Collaboration + +Other people viewing the same table appear as colored cell rings on the cell +they have selected, and as avatars in the navbar. See +[Presence & collaboration](presence.md). + +## Editing features + +- **Rearrange columns** β€” drag column headers (order is saved per view, + including computed and timer columns). +- **Resize columns** β€” drag the edges of the column header. +- **Sort rows** β€” click a column header. +- **Keyboard navigation** β€” arrow keys, Excel-like hotkeys. +- **Copy & paste** β€” multi-cell selection with `Ctrl/Cmd + C` / `V`. Pasting + works across tables and apps that support HTML table data. +- **Export to CSV** β€” Export button in the top right. +- **Works offline** β€” create, edit, filter, and total rows with no server; sync + when you reconnect. See [Local-first](../local-first.md). diff --git a/docs/src/atomicserver/installation.md b/docs/src/atomicserver/installation.md index f9a9d1e15..9f8cd9add 100644 --- a/docs/src/atomicserver/installation.md +++ b/docs/src/atomicserver/installation.md @@ -72,7 +72,7 @@ sudo apt-get install -y build-essential pkg-config libssl-dev --fix-missing - You can configure the server by passing arguments (see `atomic-server --help`), or by setting ENV variables. - The server loads the `.env` from the current path by default. Create a `.env` file from the default template in your current directory with `atomic-server generate-dotenv` -- After running the server, check the logs and take note of the `Agent Subject` and `Private key`. You should use these in the [`atomic-cli`](https://crates.io/crates/atomic-cli) and [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) clients for authorization. +- After running the server, check the logs and take note of the `Agent Subject` and `Private key`. You should use these in the [`atomic-cli`](https://crates.io/crates/atomic-cli) and [data-browser](https://github.com/atomicdata-dev/atomic-server/tree/master/browser) clients for authorization. - A directory is made: `~/.config/atomic`, which stores your newly created Agent keys, the HTTPS certificates other configuration. Depending on your OS, the actual data is stored in different locations. See use the `show-config` command to find out where, if you need the files. - Visit `http://localhost:9883/setup` to **register your first (admin) user**. You can use an existing Agent, or create a new one. Note that if you create a `localhost` agent, it cannot be used on the web (since, well, it's local). More info and steps in [getting started with the GUI](gui.md). diff --git a/docs/src/atomicserver/local-first.md b/docs/src/atomicserver/local-first.md new file mode 100644 index 000000000..11afe457d --- /dev/null +++ b/docs/src/atomicserver/local-first.md @@ -0,0 +1,65 @@ +# Local-first + +Atomic Data is **local-first**: you can create and edit data with no server in +reach. A server, when you use one, is an always-on peer you sync with β€” not the +source of truth. + +## What that means in practice + +- **Your identity is a keypair.** An [Agent](../agents.md) is identified by + `did:ad:agent:{publicKey}`. There is no account on a host that can lock you + out; losing the secret (or its [passkey backup](#encryption-and-recovery)) is + what loses the identity. +- **Resources have portable addresses.** Subjects use the [`did:ad`](../did.md) + scheme, so moving data between devices does not rename it. +- **Every write is a signed CRDT commit.** Edits are [Atomic Commits](../commits/intro.md) + backed by [Loro](https://loro.dev) documents. Concurrent edits merge instead of + overwriting; history and authorship stay verifiable. +- **The browser runs the real database.** Through WASM, the web app uses the + same Rust storage and query engine as AtomicServer (`atomic_lib`), persisted in + the Origin Private File System (OPFS). An offline table computes the same + totals as an online one because it is the same code. + +## Where data lives + +| Client | Storage | Notes | +| --- | --- | --- | +| Browser (data-browser) | Per-agent encrypted OPFS database via WASM | Works fully offline; outbox drains commits when a peer is reachable | +| Desktop / mobile (Tauri) | Embedded `atomic_lib` + local files | Can pair with other devices over Iroh | +| Flutter canvas app | `atomic_lib` via flutter_rust_bridge | Same local-first store and peer sync | +| AtomicServer | redb on disk | An always-on device; HTTP/WS plus the same sync protocol | + +A browser tab is **not** a peer node by itself: it reads and writes through an +always-on device (your desktop app, a home server, or a hosted instance) when it +needs to reach other machines. Creating a workspace in the browser alone still +works locally; getting it onto a phone requires a device that can pair. See +[Sync & pairing](gui/sync-and-pairing.md). + +## Encryption and recovery + +Each agent's in-browser database is encrypted at rest (XChaCha20-Poly1305), under +a key wrapped by that agent's private key. Signing out leaves the cache in place +but unreadable to the next session. + +Account backup is passkey-first: a WebAuthn passkey wraps a backup of the agent +secret (Argon2id + AES-GCM), so onboarding does not require writing a secret +down. You can still export the agent secret manually from User Settings. + +## Syncing + +When two devices share rights on a workspace, they converge with the +[sync protocol](../websockets.md): + +1. **WebSocket** β€” browser ↔ always-on device (subscribe, live commits, bulk sync). +2. **Iroh (QUIC)** β€” device ↔ device after scanning a pairing code. + +Discovery of *which* node holds a drive uses pkarr (and Iroh's own relay +discovery). Trust always comes from commit signatures and +[hierarchy](../hierarchy.md) rights β€” never from who serves the bytes. + +## Related + +- [`did:ad` identifiers](../did.md) +- [WebSocket / sync protocol](../websockets.md) +- [Sync & pairing (GUI)](gui/sync-and-pairing.md) +- [Commits](../commits/intro.md) diff --git a/docs/src/atomicserver/when-to-use.md b/docs/src/atomicserver/when-to-use.md index 0e2749a4a..e3a423354 100644 --- a/docs/src/atomicserver/when-to-use.md +++ b/docs/src/atomicserver/when-to-use.md @@ -3,11 +3,12 @@ ## When should you use AtomicServer - You want a **lightweight, fast, realtime** and easy to use **headless CMS** with live updates, editors, modelling capabilities and an intuitive API -- You want **realtime** updates and collaboration functionality +- You want **local-first** apps that work offline and sync across devices ([Local-first](local-first.md)) +- You want **realtime** updates, presence, and collaboration functionality - You want **high performance**: AtomicServer is incredibly fast and can handle thousands of requests per second. - You want **standalone app**: no need for any external applications or dependencies (like a database / nginx). - You want **versioning** or **full-text search**. -- You want to build a webapplication, and like working with using [React](https://github.com/atomicdata-dev/atomic-data-browser) or [Svelte](https://github.com/atomicdata-dev/atomic-svelte). +- You want to build a webapplication, and like working with [React](https://github.com/atomicdata-dev/atomic-server/tree/master/browser) or [Svelte](https://github.com/atomicdata-dev/atomic-svelte). - You want to make (high-value) **datasets as easily accessible as possible** - You want to specify and share a **common vocabulary** / ontology / schema for some specific domain or dataset. Example classes [here](https://atomicdata.dev/classes). - You want to use and **share linked data**, but don't want to deal with most of [the complexities of RDF](https://docs.atomicdata.dev/interoperability/rdf.html), SPARQL, Triple Stores, Named Graphs and Blank Nodes. @@ -15,10 +16,10 @@ ## When _not_ to use AtomicServer -- High-throughput **numerical data / numerical analysis**. AtomicServer does not have aggregate queries. +- High-throughput **numerical analysis / OLAP**. Table views support sum/count/avg/min/max aggregates over filtered rows, but AtomicServer is not a warehouse for heavy analytical workloads. - If you need **high stability**, look further (for now). This is beta sofware and can change. -- You're dealing with **very sensitive / private data**. The built-in authorization mechanisms are relatively new and not rigorously tested. The database itself is not encrypted. -- **Complex query requirements**. We have queries with filters and features for path traversal, but it may fall short. Check out NEO4j, Apache Jena or maybe TerminusDB. +- You're dealing with **very sensitive / private data** and need a battle-hardened security audit. Authorization and at-rest encryption exist, but the threat model is still evolving β€” see the encryption planning notes in the repo. +- **Complex query requirements**. We have queries with filters, multi-property constraints, path traversal, and aggregates, but it may fall short of a dedicated graph query language. Check out NEO4j, Apache Jena or maybe TerminusDB. ## Up next diff --git a/docs/src/authentication.md b/docs/src/authentication.md index f14d13353..9fad1f6ab 100644 --- a/docs/src/authentication.md +++ b/docs/src/authentication.md @@ -41,9 +41,9 @@ Here's what a JSON-AD Authentication Resource looks like for a WebSocket: ```json { - "https://atomicdata.dev/properties/auth/agent": "http://example.com/agents/N32zQnZHoj1LbTaWI5CkA4eT2AaJNBPhWcNriBgy6CE=", + "https://atomicdata.dev/properties/auth/agent": "did:ad:agent:N32zQnZHoj1LbTaWI5CkA4eT2AaJNBPhWcNriBgy6CE", "https://atomicdata.dev/properties/auth/requestedSubject": "wss://example.com/ws", - "https://atomicdata.dev/properties/auth/publicKey": "N32zQnZHoj1LbTaWI5CkA4eT2AaJNBPhWcNriBgy6CE=", + "https://atomicdata.dev/properties/auth/publicKey": "N32zQnZHoj1LbTaWI5CkA4eT2AaJNBPhWcNriBgy6CE", "https://atomicdata.dev/properties/auth/timestamp": 1661757470002, "https://atomicdata.dev/properties/auth/signature": "19Ce38zFu0E37kXWn8xGEAaeRyeP6EK0S2bt03s36gRrWxLiBbuyxX3LU9qg68pvZTzY3/P3Pgxr6VrOEvYAAQ==" } diff --git a/docs/src/commits/intro.md b/docs/src/commits/intro.md index b9ee58cbb..34a60aaf3 100644 --- a/docs/src/commits/intro.md +++ b/docs/src/commits/intro.md @@ -1,11 +1,14 @@ {{#title Atomic Commits - Event standard for Atomic Data}} # Atomic Commits -_Disclaimer: Work in progress, prone to change._ - Atomic Commits is a specification for communicating _state changes_ (events / transactions / patches / deltas / mutations) of [Atomic Data](../core/concepts.md). It is the part of Atomic Data that is concerned with writing, editing, removing and updating information. +Today every write is a signed commit whose payload is a **Loro CRDT update** +(`loroUpdate`). The older `set` / `push` / `remove` commit fields are deprecated +and rejected by current servers. See [Concepts](concepts.md) for the live shape +of a commit. + ## Design goals - **Event sourced**: Store and standardize _changes_, as well as the _current_ state. This enables versioning, history playback, undo, audit logs, and more. diff --git a/docs/src/commits/suggestions.md b/docs/src/commits/suggestions.md index 0dc81f6b5..848edf5ad 100644 --- a/docs/src/commits/suggestions.md +++ b/docs/src/commits/suggestions.md @@ -1,44 +1,60 @@ -# Atomic Suggestions +# Atomic Suggestions and Forks -Atomic Suggestions is a proposed standard that enables decentralized collaboration on resources. -It's basically Git for linked data. -Practically, it should enable right-clicking on any piece of Atomic Data on the web, and suggesting an edit to the owner. +Atomic Data supports proposing a change to an existing resource without editing +it in place. The shipped mechanism is a **Fork**. -## Design goals +> Historical note: an earlier draft of this page described HTTP POST "Suggestions" +> to an Owner Inbox. That design was never the product path and is superseded by +> Forks below. See also the CMS guide: [Drafts and forks](../atomicserver/cms.md). -- **Asynchronous collaboration**: Various users can work on the same thing at the same time. -- **Branching & merging**: Issues that result from async changes (merge conflicts) can be resolved. +## Design goals -## Concepts +- **Asynchronous collaboration**: Several people can prepare changes without + blocking the live resource. +- **Safe staging**: The original stays readable and stable until a deliberate + merge. +- **Ordinary resources**: A proposal is itself Atomic Data β€” with a subject, + history, and rights β€” not a side-channel patch format. -### Fork +## Fork -Forking is the first step to making a suggestion. -Forking refers to: +Forking: -1. copying some resource -1. changing the subject URL to some URL that you control -1. adding a reference to the original URL using the `atomic:originalSubject` Property. +1. Copies the resource (or creates a resource that carries the same content + class). +2. Places it under the drive's private Forks folder (not publicly readable by + default). +3. Sets `originalSubject` to the resource being changed. +4. Records merge baselines (`forkBase` / `forkVersion`) so property and Loro-body + merges are well-defined. -The newly created copy with the different URL is a _Fork_. -Since the Fork is a resource that you own (see [Ownership](ownership.md)), you can make changes to is. +The copy is a **Fork** (`isA` includes the `Fork` class). Because you (or your +staging folder) control it, you edit it with normal [Commits](intro.md). The +live resource is unchanged until merge. -Whenever you make changes, the app making the changes _should_ keep track of them as Atomic Commits. -These Commits make it easier to apply (small) changes to (large) resources, even when multiple people are working on the same thing at the same time. +### Merge -### Suggestion +Merging applies the fork onto `originalSubject` and retires the fork. Conflict +handling uses the stored baseline plus Loro CRDT merge for collaborative +document bodies. -When you've forked some resource and made some changes, you can Suggest these changes to the original owner. -This is done by sending an HTTP POST request containing the Commits to the Owner URL. +### Rights -A Suggestion is a (set of?) Commit(s?) that is proposed to be appended to some Ledger. -The important difference between a Suggestion and a Commit, is that a Commit has been verified, signed and approved by the Controller. +A fork is authorized like any other resource under the hierarchy. Staging your +own edit requires write access to create the fork; accepting a merge requires +write access on the original. Cross-agent "suggest from outside" can use the +same `originalSubject` shape once the productized invite/authorization path for +that case is enabled β€” the data model does not need a second proposal type. -### Controller +## Drafts (not forks) -The actor (person / organization) that is in control of a specific Resource and its Commits. +**Draft** means unpublished *new* content: a resource that lives in a private +Drafts folder. Publishing is moving it to a public parent. Drafts do not use +`originalSubject`; they are not proposals against an existing subject. See +[Headless CMS, drafts, and forks](../atomicserver/cms.md). -### Inbox +## Related -An Inbox represents a resource that contains incoming Suggestions. -It's similar to an e-mail inbox. +- [Commits](intro.md) +- [Hierarchy](../hierarchy.md) +- [Ownership](ownership.md) (historical companion notes) diff --git a/docs/src/did.md b/docs/src/did.md index 5a0680161..5360881cc 100644 --- a/docs/src/did.md +++ b/docs/src/did.md @@ -1,16 +1,16 @@ {{#title Decentralized Identifiers (DIDs) in Atomic Data }} # Decentralized Identifiers -_status: work in progress_ +_Identity forms below are stable and used in production. Discovery transports keep evolving β€” see [Resolution](#resolution)._ -Atomic Data is moving from HTTP URLs to Decentralized Identifiers (DIDs) as the primary way to address resources. -This makes resources portable, self-authenticating, and resolvable over both the internet and local mesh networks. +Atomic Data uses Decentralized Identifiers (DIDs) as the primary way to address resources. +This makes resources portable, self-authenticating, and resolvable without tying identity to a hostname. ## Design goals - **Self-sovereign**: Identifiers don't depend on any server or domain name. You generate a keypair, and you have an identity. - **Portable**: Resources can move between servers without changing their identifier. -- **Multi-transport**: The same identifier can be resolved over the internet (Mainline DHT) or local mesh networks (Reticulum). +- **Multi-transport**: The same identifier can be discovered and synced over WebSocket, Iroh (QUIC), and (planned) mesh stacks such as Reticulum. - **Verifiable**: Trust comes from [Commit](commits/intro.md) signatures, not from who hosts the data. - **Replicatable**: Any node can replicate and serve a Drive without holding the Drive's private key. @@ -70,7 +70,7 @@ For most operations, agents don't need to be "resolved" at all: - **Displaying profile info** (name, avatar): Drives cache agent metadata when agents interact with them (e.g. accepting an [Invite](invitations.md), making a [Commit](commits/intro.md)). The drive you're connected to typically already has it. If a client encounters an unknown agent, it can show the truncated public key as a fallback. -More sophisticated resolution (e.g. using [Mainline DHT](#3-mainline-dht-internet) or [Reticulum](#2-reticulum-mesh-resolution) announces) can be layered on later without changing the DID format. +Peer discovery for the drives an agent touches uses [pkarr](#3-pkarr--iroh-internet) (and can grow additional transports later) without changing the DID format. ### Commit identifiers @@ -119,7 +119,7 @@ Like resources and commits, blob DIDs accept a routing hint pointing at a Drive did:ad:blob:{blake3}?drive=did:ad:{drive_genesis} ``` -A client looks up peers for the Drive via Mainline DHT or Reticulum, then asks any of them for the blob. Over the v2 sync protocol, blobs travel as raw 32-byte hashes inside `BLOB_REQUEST`/`BLOB_RESPONSE` frames β€” the DID is for *identity*, the bytes on the wire are the underlying hash. (This parallels commits: the DID is `did:ad:commit:{sig}`, but the wire never re-prepends the prefix.) +A client looks up peers for the Drive (pkarr / known peers / configured server), then asks any of them for the blob. Over the v2 sync protocol, blobs travel as raw 32-byte hashes inside `BLOB_REQUEST`/`BLOB_RESPONSE` frames β€” the DID is for *identity*, the bytes on the wire are the underlying hash. (This parallels commits: the DID is `did:ad:commit:{sig}`, but the wire never re-prepends the prefix.) The HTTP form `/download/files/{blake3}` is a deployment-specific alias for `did:ad:blob:{blake3}` and remains supported for browsers and existing tooling. @@ -171,21 +171,13 @@ did:ad:4f7ba2...910?drive=did:ad:7e6a9d...038 A Drive is a first-class resource identified by its own `did:ad` identifier. -When a Drive is used as a routing hint (the `?drive=` parameter), network nodes derive an **internal discovery hash** for lookups on decentralized networks (Mainline DHT or Reticulum). This hash is never stored as an explicit property; it is derived on-the-fly when needed for discovery. +When a Drive is used as a routing hint (the `?drive=` parameter), nodes may derive an **internal discovery key** from the Drive's DID string for announce/lookup on a given transport. That key is not stored as a resource property; it is computed when publishing or resolving peers. -The formula for the discovery hash is: -```text -discovery_hash = HASH(drive_did_string) -``` - -The specific hash algorithm depends on the transport protocol: -- **Mainline DHT**: Uses `SHA1(drive_did_string)` to produce a 20-byte ID. -- **Reticulum**: Uses `truncated_SHA256(drive_did_string)` to produce a 16-byte destination. +What matters for applications: -This ensures: - **Consistency**: Everything is a `did:ad` identifier. - **Portability**: The identifier depends only on the Drive's genesis state, not its location. -- **Protocol Independence**: The same DID can be mapped to different binary formats required by different networks. +- **Protocol independence**: The same DID can be mapped to different discovery backends (pkarr today; additional mesh or DHT backends later). ### Drive replication @@ -194,47 +186,36 @@ Trust comes from [Commit signatures](commits/intro.md), not from who serves the 1. The Drive owner creates resources and signs [Commits](commits/intro.md) with their Agent key. 2. A replica node syncs the data and verifies every Commit signature. -3. The replica announces itself as a peer for this Drive (on Mainline DHT, Reticulum, or both) using the discovery hash derived from the Drive's DID string. -4. Clients fetching data derive the same hash from the `?drive=` hint and look up peers. +3. The replica announces itself as a peer for this Drive (pkarr today; additional transports later). +4. Clients discover peers from that announce, a pairing code, or a configured server address. 5. Clients fetch data and verify Commit signatures themselves β€” they don't need to trust the serving node. ## Resolution Resolving a `did:ad` URL means finding a network node that holds the requested Drive and resource. -Multiple resolution strategies can be tried in order: +Strategies are tried from local to remote: ### 1. Local cache -If the resource has been fetched before, serve it from the local store. - -### 2. Reticulum mesh resolution - -[Reticulum](https://reticulum.network/) is a mesh networking stack that works over any medium β€” radio, LoRa, serial, TCP, UDP, and more. -Its addressing model is a natural fit for `did:ad`: +If the resource is already in the local store (browser OPFS, device redb, etc.), serve it from there. This is the [local-first](atomicserver/local-first.md) path. -- Reticulum destinations are 16-byte hashes. -- To reach a Drive on a Reticulum mesh, a client sends a **path request** for the 16-byte destination derived from the Drive's DID string. Any Transport Node that has seen an announce for that destination can route the request. -- The Drive node (or any replica) announces its destination on the mesh, making it reachable within minutes even on slow, multi-hop networks. +### 2. Direct connection / known peers -This means two Atomic Server nodes on a Reticulum mesh (e.g. over LoRa radio) can exchange and resolve resources **without any internet access**, using the exact same `did:ad` identifiers they would use online. +If the node's address is already known β€” Sync settings, a previous session, or a [pairing code](atomicserver/gui/sync-and-pairing.md) β€” connect over WebSocket or Iroh and fetch or sync the resource. -### 3. Mainline DHT (internet) +### 3. pkarr + Iroh (internet) -[Mainline DHT](https://en.wikipedia.org/wiki/Mainline_DHT) is the BitTorrent distributed hash table β€” a decentralized network with millions of active nodes. -It provides a way for any node to announce that it hosts a given Drive, and for clients to discover those nodes: +Production discovery uses **[pkarr](https://pkarr.org/)** to publish and resolve which [Node](#node-identifiers) holds a Drive, then **[Iroh](https://iroh.computer)** (QUIC, with relay fallback) to sync. The flow: -1. A node hosting a Drive calls `announce_peer(SHA1(drive_did_string))` on the Mainline DHT. -2. A client resolving a Drive calls `get_peers(SHA1(drive_did_string))` and receives a list of IP:port pairs. -3. The client connects to any discovered peer and requests the resource using the original DID. -4. Commit signatures are verified client-side. +1. A node hosting a Drive publishes its NodeID via pkarr (keyed from the Drive identity). +2. Another device resolves that NodeID, connects over Iroh, and runs the [sync protocol](websockets.md). +3. Commit signatures are verified client-side. -No special signing keys (BEP44) are needed at the DHT layer. -The DHT is a pure _discovery_ mechanism β€” all trust and authenticity comes from the Commit signatures in the data itself. -Any node β€” the original or a replica β€” can announce itself as a peer. +pkarr is a pure _discovery_ mechanism β€” authenticity still comes only from Commit signatures. Any replica that holds the data can announce itself. -### 4. Direct connection +### 4. Reticulum mesh (planned) -If the node's IP or domain is already known (e.g. from configuration or a previous session), connect directly. +[Reticulum](https://reticulum.network/) is a mesh networking stack (radio, LoRa, serial, TCP, and more). Carrying the same sync protocol over Reticulum is a design goal so two nodes could exchange `did:ad` resources without internet access. It is **not** required for current deployments; see the internal `planning/reticulum-sync.md` notes. ## HTTP Discovery @@ -257,7 +238,7 @@ The three variants map to different resolution strategies: | `Subject` variant | Format | Use case | |---|---|---| | `Internal` | `internal:/path` | Local resources on this server. Resolved to an absolute URL using the server's origin for serialization. | -| `Did` | `did:ad:...` | Agents (by public key), Commits (by signature), Blobs (by BLAKE3 hash), Nodes (as routing identities), and Resources in Drives (by genesis commit signature). Routing hints (`?drive=did:ad:...`) are used for peer discovery via Reticulum or Mainline DHT. | +| `Did` | `did:ad:...` | Agents (by public key), Commits (by signature), Blobs (by BLAKE3 hash), Nodes (as routing identities), and Resources in Drives (by genesis commit signature). Routing hints (`?drive=did:ad:...`) help peer discovery (pkarr / known peers). | | `External` | `https://...` | Resources on other servers. Resolved via HTTP. Used for backward compatibility and external linked data. | When serializing to [JSON-AD](core/json-ad.md), `Internal` subjects are resolved to absolute URLs using the server's configured origin. @@ -268,7 +249,8 @@ When serializing to [JSON-AD](core/json-ad.md), `Internal` subjects are resolved | | `did:ad` | `did:web` | `did:dht` | `did:key` | |---|---|---|---|---| | **Decentralized** | βœ… No server dependency | ❌ Depends on DNS | βœ… Mainline DHT | βœ… Self-contained | -| **Mesh-capable** | βœ… Native Reticulum | ❌ | ❌ | βœ… But no routing | +| **Discovery today** | βœ… pkarr + Iroh / WS | DNS | Mainline DHT | N/A | +| **Mesh-capable** | 🚧 Reticulum planned | ❌ | ❌ | βœ… But no routing | | **Updatable** | βœ… Drive can move | βœ… Update DNS | βœ… Mutable records | ❌ Static | | **Replicatable** | βœ… Any node can serve | ❌ Single server | ❌ Key holder only | N/A | | **Trust model** | Commit signatures | TLS + DNS | BEP44 signatures | Key-based | diff --git a/docs/src/extended-table.md b/docs/src/extended-table.md index 4a57309dc..10e3150e9 100644 --- a/docs/src/extended-table.md +++ b/docs/src/extended-table.md @@ -1,9 +1,14 @@ -- [Commits](commits/intro.md) communicate state changes. These Commits are signed using cryptographic keys, which ensures that every change can be audited. Commits are also used to construct a history of versions. -- [Agents](agents.md) are Users that enable [authentication](authentication.md). They are Resources with their own Public and Private keys, which they use to identify themselves. +- [Commits](commits/intro.md) communicate state changes. These Commits are signed using cryptographic keys, which ensures that every change can be audited. Commits are also used to construct a history of versions. Live writes carry [Loro](https://loro.dev) CRDT updates. +- [Agents](agents.md) are Users that enable [authentication](authentication.md). They are Resources identified by `did:ad:agent:{publicKey}`. +- [Decentralized Identifiers (DIDs)](did.md): portable `did:ad` subjects for resources, agents, commits, blobs, and nodes. - [Collections](schema/collections.md): querying, filtering, sorting and pagination. - [Paths](core/paths.md): traverse graphs. - [Hierarchies](hierarchy.md) used for authorization and keeping data organized. Similar to folder structures on file-systems. - [Invites](invitations.md): create new users and provide them with rights. -- [WebSockets](websockets.md): real-time updates. +- [WebSockets / sync protocol](websockets.md): real-time updates, drive sync, and peer transport (also over Iroh). - [Endpoints](endpoints.md): provide machine-readable descriptions of web services. -- [Files](files.md): upload, download and metadata for files. +- [Files](files.md): upload, download and metadata for files (content-addressed `did:ad:blob:`). +- [Translations](schema/translations.md): document-level and field-level (`LocalizedText`) localization. +- [Local-first](atomicserver/local-first.md): offline stores, encryption, and sync without a central source of truth. +- [Presence](atomicserver/gui/presence.md): ephemeral cursors, avatars, and follow-me (not persisted as commits). +- [CMS, drafts & forks](atomicserver/cms.md): publishing by location, and Fork-based proposed edits. diff --git a/docs/src/get-started.md b/docs/src/get-started.md index cc0cf3424..9005506c7 100644 --- a/docs/src/get-started.md +++ b/docs/src/get-started.md @@ -12,7 +12,7 @@ There's a couple of levels at which you can start working with Atomic Data (from ## Play with the demo - Open [the Invite](https://atomicdata.dev/invites/1) on `atomicdata.dev` -- Press `Accept`. The front-end app will generate a cryptographic key pair. Your public key determines your Agent identity (`did:ad:{publicKey}`), and the server grants you rights to the shared resource. +- Press `Accept`. The front-end app will generate a cryptographic key pair. Your public key determines your Agent identity (`did:ad:agent:{publicKey}`), and the server grants you rights to the shared resource. - You're now signed in! You can edit the document on your screen. - Edit your Agent by going to [user settings](https://atomicdata.dev/app/agent) - Copy your `secret`, and save it somewhere safe. You can use this to sign in on a different machine. @@ -32,9 +32,9 @@ There's a couple of levels at which you can start working with Atomic Data (from -- Browser app [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) ([demo on atomicdata.dev](https://atomicdata.dev)) -- Build a react app using [typescript & react libraries](https://github.com/atomicdata-dev/atomic-data-browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) -- Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-data-browser) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server`) +- Web app in the [`atomic-server` monorepo](https://github.com/atomicdata-dev/atomic-server/tree/master/browser) ([demo on atomicdata.dev](https://atomicdata.dev)) +- Build a react app using [`@tomic/lib` and `@tomic/react`](https://github.com/atomicdata-dev/atomic-server/tree/master/browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) +- Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-server) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server`) - Discover the command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) - Use the Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) diff --git a/docs/src/headless-cms.md b/docs/src/headless-cms.md deleted file mode 100644 index 0af7c1f87..000000000 --- a/docs/src/headless-cms.md +++ /dev/null @@ -1 +0,0 @@ -# Using as a Headless CMS diff --git a/docs/src/interoperability/solid.md b/docs/src/interoperability/solid.md index 274cd9b69..e438abe1f 100644 --- a/docs/src/interoperability/solid.md +++ b/docs/src/interoperability/solid.md @@ -127,9 +127,9 @@ I believe that as of today (february 2022), Atomic-Server has quite a few advant - **Dynamic schema validation** / type checking using [Atomic Schema](https://docs.atomicdata.dev/schema/intro.html), combining the best of RDF, JSON and type safety. - **Fast** (1ms responses on my laptop) -- **Lightweight** (8MB download, no runtime dependencies) +- **Lightweight** (~70MB download, no runtime dependencies) - **HTTPS + HTTP2 support** with Built-in LetsEncrypt handshake. -- **Browser GUI included** powered by [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser). Features dynamic forms, tables, authentication, theming and more. Easy to use! +- **Browser GUI included** powered by the data-browser in [atomic-server](https://github.com/atomicdata-dev/atomic-server). Features dynamic forms, tables, authentication, theming and more. Easy to use! - **Event-sourced versioning** / history powered by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) - **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/serialization.html#json-ad), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). - **Full-text search** with fuzzy search and various operators, often <3ms responses. diff --git a/docs/src/interoperability/sql.md b/docs/src/interoperability/sql.md index dc582d15f..96f490e33 100644 --- a/docs/src/interoperability/sql.md +++ b/docs/src/interoperability/sql.md @@ -69,7 +69,7 @@ If you want to store arbitrary Atomic Data in a SQL database, you might be best ### Is Atomic Data NOSQL or SQL? Generally, Atomic Data apps do not use SQL - so they are NOSQL. -Atomic-server, for example, internally uses a key-value store (sled) for persistence. +Atomic-server, for example, internally uses a key-value store (redb) for persistence. Like most NOSQL systems, Atomic Data does not limit data entries to a specific table shape, so you can add any property that you like to a resource. However, unlike most NOSQL systems, Atomic Data _does_ perform validations on each value. diff --git a/docs/src/plugins.md b/docs/src/plugins.md index 3729ddffd..20f0dbd89 100644 --- a/docs/src/plugins.md +++ b/docs/src/plugins.md @@ -45,7 +45,7 @@ Handlers receive JSON-AD payloads that describe the Resource or Commit they shou ### Installing a WASM class extender 1. Build a component that targets `wasm32-wasip2`. Use `wit-bindgen` or `cargo component` to satisfy the interface defined in `lib/wit/class-extender.wit`. -2. Copy the resulting `.wasm` file into the `wasm-class-extenders/` directory inside your Atomic data directory (next to the sled store). +2. Copy the resulting `.wasm` file into the `wasm-class-extenders/` directory inside your Atomic data directory (next to the redb store). 3. Restart `atomic-server` (or recreate the `Db`) so it scans the folder and instantiates your component. All `.wasm` files in that folder are loaded on startup. Errors are logged but do not prevent the server from running, making it safe to iterate on plugins. diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md index 07cd0cd65..2123094cb 100644 --- a/docs/src/roadmap.md +++ b/docs/src/roadmap.md @@ -40,11 +40,12 @@ This should help you understand how and where you may be able to contribute. - **[@tomic/svelte](https://github.com/atomicdata-dev/atomic-svelte)** (2022-12). Library for integrating Atomic Data with Svelte(Kit). - **[Atomic Tables](https://github.com/atomicdata-dev/atomic-data-browser/issues/25)** (2023-09). A powerful table editor with keyboard / copy / paste / sort support that makes it easier to model and edit data. - **Ontology Editor** (2023-10). Easily create & edit Classes, Properties and Ontologies. -- **Local-First & did:ad Schema** (2026-06). Transitioned to a Local-First architecture using the `did:ad` schema. Instead of relying on a hosted HTTP origin, resources resolve over Mainline DHT. Agents are decentralized, relying solely on an Ed25519 private key. +- **Local-First & did:ad Schema** (2026-06). Transitioned to a Local-First architecture using the `did:ad` schema. Agents are decentralized keypairs; resources get portable DIDs. Peer discovery uses pkarr; devices sync over WebSocket and Iroh. - **Collaborative Sync with Loro CRDT** (2026-06). Integrated Loro CRDTs for collaborative real-time sync across devices, making documents conflict-free. - **Drafts and Suggestions** (2026-07). Added CMS publishing, drafts, and user suggestions as a clean, fork-based squash-merge mechanism. - **Meetings & Follow-Me Tours** (2026-07). Shipped purpose-built meeting workspaces, collaborative live meeting notes, and follower follow-along live tours. - **P2P pairing & Zero-scan auto-sync** (2026-07). Built routing-only deep-link QR pairing and SaaS-assisted zero-scan device synchronization. +- **Tables as apps** (2026-07). Kanban, calendar, and timer views; filters; computed columns; aggregates; templates. ## Where we're at diff --git a/docs/src/schema/translations.md b/docs/src/schema/translations.md index d58f858f2..da24ae05a 100644 --- a/docs/src/schema/translations.md +++ b/docs/src/schema/translations.md @@ -76,6 +76,14 @@ Clients resolve a preferred language `L` with the same chain everywhere: In `@tomic/lib` this is the `localizeText(value, preferred, defaultLanguage?)` helper; in Rust it is `Value::to_localized_string`. +## What the GUI supports today + +- **LocalizedText columns** in [Tables](../atomicserver/gui/tables.md): per-language input, language switcher, and optional split-by-language columns. +- **Website templates** can declare `defaultLanguage` / `languages` and ship multi-locale starter content. +- **Document-level** `language` + `translationOf` are in the data model; a dedicated "Translate this page" / translations bar UX is still landing. Until then you create sibling resources and set the properties manually (or via the assistant). + +UI chrome strings in the data-browser (menus, buttons) are localized separately via the app's own i18n tooling (Wuchale) and are unrelated to content `LocalizedText`. + ## History An earlier concept on this page (`TranslationBox`, a property per language under `https://atomicdata.dev/languages/{tag}`) was never implemented and is superseded by the model above. diff --git a/docs/src/usecases/headless-cms.md b/docs/src/usecases/headless-cms.md index 9508a16ec..47b94ecd0 100644 --- a/docs/src/usecases/headless-cms.md +++ b/docs/src/usecases/headless-cms.md @@ -11,21 +11,23 @@ This approach has some issues regarding performance and flexibility that headles ## Atomic Server -The [Atomic-Server](https://github.com/atomicdata-dev/atomic-server/blob/master/server/README.md) project may be the right choice for you if you're looking for a Headless CMS: +The [Atomic-Server](https://github.com/atomicdata-dev/atomic-server/blob/master/server/README.md) project may be the right choice for you if you're looking for a Headless CMS. +For the product model (drafts, forks, websites, publishing), see **[Headless CMS, drafts, and forks](../atomicserver/cms.md)**. - - **Free and open source**. MIT licensed, no strings attached. +- **Local-first**. Editors can work offline; the browser runs the same store through WASM. See [Local-first](../atomicserver/local-first.md). - **Easy to use API**. Atomic-Server is built using the [Atomic Data specification](../atomic-data-overview.md). It is well-documented, and uses conventions that most web developers are already familiar with. - **Typescript & React libraries**. Use the existing react hooks to make your own fully editable, live-reloaded web application. -- **Fast**. 1ms responses on my laptop. It's written in Rust, so it squeezes out every cycle of your server. -- **Lightweight**. It's a single 8MB binary, no external dependencies needed. +- **Fast**. Written in Rust; collection queries and search are tuned for interactive editing. +- **Lightweight**. A single self-contained binary (~70MB) with server, web app, search, and database β€” no external runtime dependencies. - **Easy to setup**. Just run the binary and open the address. Even HTTPS support is built-in. -- **Clean, powerful admin GUI**. The Atomic-Data-Browser front-end gives you a very easy interface to manage your content. +- **Clean, powerful admin GUI**. Tables, documents, presence, and ontology editing in one app. +- **Drafts & forks**. Unpublished content lives in a private folder; proposed edits are Forks you merge when ready. - **Share your data models**. Atomic Data is designed to achieve a more decentralized web. You can easily re-use existing data models, or share the ones you built. - **Files / Attachments**. Upload and preview files. -- **Pagination / sorting / filtering**. Query your data. -- **Versioning**. Built-in history, where each transaction is saved. -- **Websockets**. If you need live updates and highly interactive apps (collaborative documents and chatrooms), we've got your back. +- **Pagination / sorting / filtering**. Query your data; table views add aggregates and live filters. +- **Versioning**. Built-in history, where each transaction is a signed commit. +- **Websockets & sync**. Live updates, device pairing, and CRDT collaborative documents. - **Full-text search**. No need for a big elasticsearch server - atomic-server has one built-in. - **Translations / i18n**. Localize content per language β€” see below. diff --git a/planning/README.md b/planning/README.md index 04fea2742..bbfd9cd3d 100644 --- a/planning/README.md +++ b/planning/README.md @@ -50,6 +50,7 @@ scratch document. When a plan becomes obsolete, delete it. | [`drafts-and-suggestions.md`](./drafts-and-suggestions.md) | **Active (2026-07-15):** two generic capabilities, no Website-specific path. A **Fork** (`Fork` class + `originalSubject`) proposes a change to an existing resource and merges onto it; a **Draft** is unpublished new content defined by location (a resource in a private Drafts folder) and publishes by moving. Visibility is location; needs no deny rule. Milestone 11 (headless CMS): #467, mechanism half of #1000. | | [`content-i18n.md`](./content-i18n.md) | **Proposal (2026-07-20):** content translations (#1069) as document-level localization β€” a translation is an ordinary resource with `language` + `translationOf`, drafts/queries/zones/templates compose for free; locale-per-branch rejected (LWW divergence), `TranslationBox` concept dead; field-level `LocalizedText` datatype is the planned second layer for shared-structure fragments (nav labels, feature cards), gated on the primitive-first `Value` reshape. Milestone 11 (headless CMS). | +| [`docs-refresh.md`](./docs-refresh.md) | **Active (2026-08-04):** audit + drafting checklist to bring public docs in line with local-first, DID, sync, tables, i18n, CMS, and presence. | | [`android-data-reuse.md`](./android-data-reuse.md) | **Draft (2026-07-23):** one Atomic store/agent/Iroh node per Android device β€” an elected host app owns the store and serves other apps over Binder (ContentProvider/AIDL, on-demand, cert-bound caller identity); commits are the IPC write protocol; canvas gains a client mode and reuses an installed atomic-server. Supersedes the localhost-daemon transport in `on-device-atomic-daemon.md`. | | [`index-performance.md`](./index-performance.md) | **Diagnosis (2026-07-24):** comparative benchmark vs. NextGraph found collection queries costing ~100Β΅s/matching resource. Root cause: every match is fully Loro-decoded plus permission-checked (which itself re-decodes the drive resource) even when only subjects are requested. One fix shipped (redundant snapshot re-export); the structural fix for the permission-check half is `zones.md`'s zone index, not yet built. | | [`partial-sync.md`](./partial-sync.md) | **Proposal (2026-07-31):** replicate part of a drive per device. Absence is currently overloaded ("I lack X" β‡’ "send me X"), so a partial replica gets refilled and never matches the whole-drive hash. Devices declare a positive scope (subtree/zone roots + a blob-bytes axis) plus fill state; enforcement in both bulk reconcile and live fan-out; eviction must not tombstone. Requires keying RBSR by `zone \|\| subject` before `drive-reconciliation.md` Phase 2 builds the tree. | diff --git a/planning/docs-refresh.md b/planning/docs-refresh.md new file mode 100644 index 000000000..18e5ffbe0 --- /dev/null +++ b/planning/docs-refresh.md @@ -0,0 +1,68 @@ +# Public docs refresh (0.41 local-first) + +> **Status:** Draft in progress (2026-08-04). Audit of what the public docs +> must catch up on after the local-first / DID / sync / CMS / presence work. +> Product docs live in `docs/`; this file tracks the gap analysis and the +> drafting checklist. + +The public docs are uneven. Spec/protocol pages that rode the rewrite +(`websockets.md`, `commits/concepts.md`, much of `did.md`, +`schema/translations.md`) are largely current. **Product-facing and GUI docs +are the weak spot**: tables, CMS, presence, pairing, and local-first UX are +thin, empty, or absent from the TOC, while overview pages still claim Mainline +DHT resolution and leave WIP banners on shipped identity. + +## What shipped (docs must reflect) + +| Area | Shipped reality | Docs before this refresh | +| --- | --- | --- | +| Local-first | Browser WASM + OPFS (per-agent encrypted), Flutter via `atomic_lib`, offline edits + outbox | Feature bullet only; no architecture page; Store docs still server-URL-centric | +| Sync protocol | Binary WS v2 + Iroh QUIC; QR/pairing codes; server = always-on peer | Solid wire ref (`websockets.md`); no user-facing sync/pairing guide | +| `did:ad` | Production identity for agents, resources, commits, blobs, nodes | Full page but `_status: work in progress_`; discovery narrative overclaims Mainline DHT / Reticulum | +| Tables | Views (table/kanban/calendar/timer), filters, templates, computed columns, aggregates, row actions, LocalizedText columns | ~30-line page: sort/copy/CSV only | +| i18n | `LocalizedText` shipped; document-level model documented; full Translate UX partial | Spec page good; GUI support not spelled out | +| CMS | Website templates, Drafts (location), Forks (`originalSubject` + merge) | Stub `headless-cms.md`; usecase page is 2021 marketing; `commits/suggestions.md` is obsolete HTTP Inbox design | +| Presence | Drive presence, cursors, follow-me, meetings, table cell rings, facepile | Almost undocumented; EPHEMERAL tag only | + +Discovery **today**: pkarr announce + Iroh (`discovery_n0` / local network). +Reticulum is proposal-only. Marketing copy that says "resolve over Mainline DHT" +should say peer discovery (pkarr) + sync (Iroh / WS). + +## Checklist + +### New / rewritten pages + +- [x] `atomicserver/gui/tables.md` β€” full feature guide +- [x] `atomicserver/cms.md` β€” Website, drafts, forks, publishing, i18n +- [x] `atomicserver/gui/presence.md` β€” presence, follow-me, meetings +- [x] `atomicserver/local-first.md` β€” OPFS/WASM, encryption, offline +- [x] `atomicserver/gui/sync-and-pairing.md` β€” QR, devices, cloud +- [x] `commits/suggestions.md` β†’ Forks (shipped model) + +### Corrections + +- [x] `did.md` β€” remove WIP banner; split identity (shipped) vs discovery transports +- [x] `commits/intro.md` β€” drop "prone to change" disclaimer +- [x] `atomic-server.md` / `roadmap.md` β€” Mainline DHT overclaim +- [x] `SUMMARY.md` + `extended-table.md` β€” surface new pages +- [x] `atomic-data-overview.md` / `get-started.md` β€” monorepo links, local-first blurb +- [x] `usecases/headless-cms.md` β€” binary size, drafts/forks, link to CMS page +- [x] `authentication.md` β€” `did:ad:agent:` example +- [x] `atomicserver/when-to-use.md` β€” aggregates exist on tables +- [x] `atomicserver/gui.md` β€” passkeys / recovery; link new GUI pages +- [x] `schema/translations.md` β€” GUI support subsection +- [x] sled β†’ redb leftovers (`plugins.md`, `interoperability/sql.md`) +- [x] Delete or replace empty `headless-cms.md` stub + +### Deferred (follow-ups) + +- [ ] `@tomic/lib` Store docs for ClientDb / DID subjects +- [ ] `@tomic/react` presence hooks (`useDrivePresence`, `useResourcePresence`) +- [ ] Broader `atomic-data-browser` β†’ monorepo link sweep across usecases +- [ ] Document-level Translate action / TranslationsBar when that UX ships + +## Priority rationale + +Highest leverage for readers: Tables rewrite, CMS + Presence + Sync + +Local-first pages, then factual fixes (DID status, discovery, sled, aggregates). +Wire protocol docs stay as the reference; product pages link into them.