External data integrations

[danielrothmann]

Motivation

Firm doesn't exist in isolation. While some aspects may be managed with firm over other solutions, a modern business will still have external tools that they use - accounting tools, CRM, wordpress, etc.

We need a way to bring external information into the firm graph so it can be queried in a holistic context.

Proposal

Not exactly sure what would be the most ergonomic way of doing this.

Idea: External block

Mimics in some aspects the Terraform data block, except it would run arbitrary scripts to get the data.

external social_integration {
    script = path"./get_social_data.py"
    arguments = "--id some_id"
}

An external block could define a script (Python? JS?) which outputs an entity. When the graph is built, the script is evaluated to produce the entity. Other entities might then be able to reference that external:

person from_social {
   name = external.social_integration.name
}

Idea: Expand block

A block which expands into a DSL entity.

expand social_integration {
    script = path"./get_social_data.py"
    arguments = "--id some_id"
}

An external block could also define a script which outputs an entity. The difference being that this gets resolved when the DSL is parsed, generating a new DSL entity block next to the expand block. We would need some way of tracking where the DSL came from, so it's not expanded twice and could potentially be updated if we have changes.

person from_social {
   name = "Expanded name"
   expanded_from = expand.social_integration
}

Backward compatibility

I reckon this can be introduced as a new language type, keeping entities / schemas intact.

Considerations

What happens when an external integration fails to run or produce the expected entity? How to test? Fail early or best effort?

[Wilkenfeld]

@danielrothmann I personally like the idea of having an external block rather than an expand block, since the idea would be to have datasources from an external service, which has a broader scope than just adding fields to an entity. In the declaration of the external entity there might be a definition of the exposed fields, which then will be returned by the script. (I'm just sketching some ideas and I've just started using firm, but this is a very important issue to solve for my use cases)

[danielrothmann]

@Wilkenfeld agreed, external blocks would be a good fit for representing a distinct type of entity.

You could theoretically also define a custom schema with the existing tools and use external/expand block to populate an entity of that type. But since that would require 3 blocks, I think it might be more ergonomic to let it live inside the external block, optionally with some schema information, as you said.

External block is well suited to dynamic data - that could change each time you get it. The tradeoff would be that needs to be retrieved every time the firm graph is built, unless the information is cached somehow (but then what is the criteria for invalidating it?). I suppose an expand block would have the same problem if it's updatable.

[Wilkenfeld]

@danielrothmann I think that for that issue the possible solutions are either living with that (specifying that in the docs, so that the scripts would handle caching by themselves) or have some fields like invalidate_if (e.g. if a value changes somewhere) or invalidate_every (e.g. every 5 seconds), so that caching can be managed on firm side.

[stappersg]

Crossreference to #21, CardDAV Sync.

[Wilkenfeld]

@stappersg I think that'd be a great example and test case! I had Jira in mind when I thought about the look and shape for this feature, but I didn't think about CardDAV!

[danielrothmann]

Thanks, that's a good potential first use case for data sync.

Still mulling over how to best approach this, as I want to get it right, especially if it's going to be a language feature as well.

I think it would be best if external data still maps to a Firm schema, so things can be queried in a predictable way. If you want to represent external data in your Firm workspace, you would then first define a custom schema, if there is not a built-in entity for the type of data you want to represent.

@stappersg can I ask you what you are specifically looking to achieve with CardDAV sync? You have your contacts in a separate system, and you want them represented in the Firm workspace why? Assuming to link other activities to them?

I'm starting to think there are 2 different use cases here:

Data import: static external data

Provide data from an external system and put it in your Firm workspace. Data import would result in one or more entities represented in the Firm DSL. You could achieve this today by using the Firm library crates and generate DSL based on your own integration. Or it could be integrated in the language and libraries.

Maybe this is the thing that fits the CardDAV use case?

Data sync: dynamic external data

Provide data from an external system when a Firm graph is being queried. This data can change over time. That's kind of expensive because external systems need to be accessed each time.

Idea: import block

Similar to the expand idea from earlier. Consider being able to define a block that will produce a set of entities which gets written to DSL on import.

import my_contacts {
    method = "script"
    script = path"./get_contacts.py"
}

// Generated post-import
imported contact my_contact_1 {
    name = "Imported 1"
    imported_from = import.my_contacts
}

imported contact my_contact_2 {
    name = "Imported 2"
    imported_from = import.my_contacts
}

You could provide several built-in import methods. Running a python script as the catch-all, but maybe also specific integrations like CardDAV. The import method will provide the entities including their type, ID and fields, and the Firm library will generate DSL from it.

Each imported block is annotated at the language level and has a link back to its import source. If IDs on import are deterministic, this could potentially be used to allow for "re-import" to update some entities or add new ones.

The import action could something you run once (or rarely) using the CLI. firm import -> expand import blocks to imported entities. firm build build the graph including imported entities.

Idea: external block

Extrapolating this design to "live" data, you could have external blocks like so:

// Gotten each time graph is built
external contact my_contact_1 {
    method = "script"
    script = path"./get_contact.py"
    arguments = "--id first"
}

external contact my_contact_2 {
    method = "script"
    script = path"./get_contact.py"
    arguments = "--id second"
}

External blocks would work the same as import blocks, except they provide a single dynamic entity which adheres to a Firm schema. Using firm build, the external data would be retrieved before the graph is built.

Not sure though, if there's any real use case for this?

[stappersg]

On Mon, Oct 20, 2025 at 04:35:38AM -0700, Daniel Rothmann wrote: @stappersg can I ask you what you are specifically looking to achieve with CardDAV sync? You have your contacts in a separate system, and you want them represented in the Firm workspace why? Assuming to link other activities to them?

My wish: Avoiding duplicate data / Avoiding drift between data sets. I have no idea how realistic that wish is. Do know there is a wisdom that says: If you please everbody, nobody will like it. Groeten Geert Stappers -- Silence is hard to parse