Rules for code placement in Islet directories and a separate repository

[bokdeuk-jeong]

App provisioning developers are considering creating additional repositories for the following purposes.

Name	Description
realm-metadata-tool	A tool for generating files containing the the realm metadata.
realm-manager	For host daemon and realm daemon with protocol and client crates
image-registry	OCI and RA-TLS compatibile application image registry server and client.

It would be good to decide and clarify which codes should go into the Islet repo's directory and which ones should be placed in a separate repo.

[L0czek]

The realm-manager and image-registry would be used to store tools and libraries that are mostly agnostic to islet. Since they primarily require only an ARM EAC-something compatible rmm I think they should be kept in separate repos. The islet repository already contains the rmm, cli and many other crates related to for ex. remote attestation. I my opinion adding a bunch of directories will not make islet easier to dive in and maintain. You may want to keep everything in one repo to make integration with the build system straightforward. However, Islet already relies on git submodules so this also shouldn't be an issue.

[bokdeuk-jeong]

I agree with your point that current islet repository contains a lot.
Your suggestion seems to make minimal changes to the current setup. However, I thought it would be great if we could use your request as an opportunity for a broader discussion on how to best position each module.
For example, we already have a tools directory in islet. If we have another repo containing another tool, how can we determine where to look up to find a specific tool?

[L0czek]

For example, we already have a tools directory in islet

What exact directory do you have in mind? I just run find . -name tools -type d | grep -v third-party and didn't find anything.
Nevertheless, I thinks that we should start distinguishing tools by their purpose. We already have a repository for remote-attestation, where we keep libraries related to RA-TLS and linux-rsi. Similarly, we may want to have another repository for examples. cli tooling and application provisioning. The components of the provisioning setup itself shouldn't be stores in a single repo as for example the REE host daemon doesn't have much in common with the image registry server. To make the navigation easier in such organization with a lot of repositories we can introduce proper topics like 'remote-attestation' or 'application-provisioning'. Naturally, having the tools linked on some landing page in islet (the main README.md) with some short description might also be a good idea.

[bokdeuk-jeong]

My mistake with the name 'tools'. However, that doesn't mean we have no issues to disuss.
For example, we have linux-rsi and rsi-test at {islet}/realm/linux-rsi too. There is cli in the islet repo. These all are for the measurement and generating attestation reports. I thought that remote-attestation only include code for 'remote' attestation.
However, I don't believe that the cli directory in islet is intended to only place the measurement code.

[zos]

If we have another repo containing another tool, how can we determine where to look up to find a specific tool?

This is a very valid point. But I don't think this concerns this particular case. I would not call realm-manager of image-registry tools for islet. They are both independent from how Islet operates. Both could be used in any Arm CCA compliant project. For me a tool has functionality tied to a specific solution. We would not put programs like qemu or kvmtool as tools in islet repository.
As for the linux-rsi - this is something required for building Realm image in islet, so I would call it a dependency more than a tool. rsi-test for me seems to be a way for Islet to check if Realm itself can work properly in terms of CCA API. Neither realm-manager nor image-registry are required for that - they use Islet for their purposes.

[p-sawicki2]

I fully agree with @L0czek . These repositories should be placed in the islet-project organization instead of the islet repository. We had similar case when we were developing the remote attestation mechanism. Back then, we decided to put our components into separate repositories and use the git submodules mechanism.

If we have another repo containing another tool, how can we determine where to look up to find a specific tool?

This is the role of documentation. There should be some page that briefly describes what each repository contains and what its purpose is. The short description about the Islet project and the URLs pointing to the documentation (how to build the project, what is the structure of the repositories/organization) should be placed in the README file directly on the landing page (https://github.com/islet-project). I believe that it would help new developers from the community to familiarize with the project.

[Havner]

As for the linux-rsi - this is something required for building Realm image in islet, so I would call it a dependency more than a tool.

No, linux-rsi is not a dependency, it's not required to build anything. It's just a kernel module. Truth is, that even linux-rsi should be in a separate repository. Creating one big bag with everything in it is a maintenance hell and working in islet repository as it is now (with so much things in it) is already difficult and I'd really like to avoid making more of such bags. Every single piece of code that can live by itself and is useful by itself should have a separate repository.

[TomaszSwierczek]

I strongly believe that working with separate git repositories for each element will be much easier to manage.

One topic - is that its easier to manage external projects this way in general (forked ones) and we are referencing a lot of external projects already (which do have their separate repos under the organization), but I think that for external elements of the stack its easy to understand such division (tf-a, acs being examples, we already do this and I believe we should).

For our own, project-specific code/modules, I think similar approach would be better.

Keeping a structure of repositories that corresponds to actual build artifacts & runtime usage will be easier to understand for external developers. If, for instance, we keep all items needed for provisioning in "application provisioning" repo (suggesting that repo is corresponding to functionality/an example usage), we totally hide the fact that for "application provisioning" actual changes are in many other places (hes, rmm/the Islet itself, kvm, ...). A hypothetical repository called "application-provisioning" with many elements that do not even link together (image registry client & server, host & realm daemon...) - such repository on its own is just a random collection, thats why, I think, naming a repository after "feature" or "example functionality" , IMHO is actually misleading.

Another argument is re-usability (even within the project) - what if for some reason later, we may want to use ie. some image signer used for provisioning in another future usecase? will we copy it to a new repository, supposedly named after new usecase?

Also, with any other aggregation we risk another thing - harder configuration of automation, code style (possibly even later some formal verification, though I am not expert on how this will be done?) & license / static / architectural checking (protex? any SAM or other tool we might want to use for analysis later on?). Typical unit of such analysis is a git repository - we may have different requirements for each of these.

TL;DR: I think that the git repositories should correspond to architectural elements of the stack/build artifacts - this way it will be much easier to manage. Less mess in intertwining pull requests and git history, issues separated for each, etc.

Of course, if there are any arguments against it, lets discuss.

[jinbpark]

First of all, could anyone give me a pointer to what exactly realm-manager and image-registry do? I've read documents in the confluence. (realm metadata, sealing key derivation, vendor specific smc allocation)

realm-manager means "Application Manager" in the document?
If so, I prefer putting realm-manager and image-registry into a separate repo.

As for realm-metadata-tool, I think it would be better to have a separate repository that deals with "tools" more broadly.
For example, we can make a repository named "tools (or something similar)" meaning that it has a bunch of different tools used in ISLET across different places.

It can contain "realm-metadata-tool", "kvmtool-rim-measurer", "tools in remote-attestation repo", ...
In short, it is having a central repository to access all kinds of tools while each library/service (e.g., remote-attestation or realm-manager) has their own repos. (of course, such library or service can link to a part of tools)

[nook1208]

TL;DR: I think that the git repositories should correspond to architectural elements of the stack/build artifacts - this way it will be much easier to manage. Less mess in intertwining pull requests and git history, issues separated for each, etc.

I agree with it.

I think that most of libraries and tools should be maintained in a separate repository for better maintenance and history management.

On the other hand, components that belong to the RMM architecture (RSI, RMI, PageTable, ETC..) and modules that have dependencies with that components (e.g, stat module) should be managed in islet as a directory.
However, if a new directory is created, it would be desirable to have a document with a brief description of the directory.

[hihi-wang]

I think we should keep materials on main repository if it has build dependency for building islet.

For those realm-metadata-tool, realm-manager and image-registry, I don't see any build dependency(correct me if I'm wrong), so I guess it's good to separate to another repo.

[zpzigi754]

I totally agree with @TomaszSwierczek's points. Also, I think that the priority should be put on the newcomer's usability. If too many components are separated, it might harm the usability. Another criteria I have in mind for the separation is whether it can exist or be built on its own like ACS-tests. I am open towards maintaining realm-metadata-tool, realm-manager, and image-registry in separate repositories, as they seem to have these independent characteristics.

[bitboom]

I voted to separate realm-metadata-tool, realm-manager and image-registry.

I believe that organizing repositories in an atomic manner based on functionality is beneficial.

This approach offers additional advantages such as ensuring that developers primarily contributing to each repository can maintain high code quality with a sense of ownership. Although managing multiple repositories can pose challenges (e.g., Rust version management, linter tool application, licensing), these can be addressed with proper documentation and tools.

For instance, Gramine even separates examples into different repositories, and Veracruz splits modules by repository, showcasing successful implementations of this strategy.

[bokdeuk-jeong]

I think having too much fine-grained repositories would end up with manageability disaster rather than providing a easy maintenance.
For example:

• I wouldn't want a change made to one repo due to some interface changes triggers subsequent changes to other repos that are dependent on it,
• It's uncomfortable to change the heads for so many submodules from each main repo for every update.
• I wouldn't want to have a separate repo just for small size of code unless it is very unique and needs to be solo.

Thus, I think they should be grouped in a single repo if they are highly coupled and have the same purpose/use. When code size grows too big, we can split them anytime. In case that a module/component is commonly used by multiple repos, we may split it into a separate repo.
Then, repos may have similar hierarchy or directory name such as 'cli', 'tools' or 'lib', but we will know these are for its own uses.
So, realm-metadata-tool, realm-manager and image-registry looks like they have the same purpose, app provisioning. Unless the code size is too big, I prefer putting them together.

I think the islet repo should mainly contain CC platform code such as RMM and HES. I think it is acceptable that the islet repo serves as the anchor point to run the CC platform due to the importance of the name.
On the other hand, I propose to place sdk and examples out of the islet repo. To me, it is always troublesome to figure out the use of modules in lib because modules from the different architecture layer is mixed together in a single directory and placed flat. Once they sdk and examples are moved out of islet, modules in lib will only for platform purposes.

Regarding kvm and some other libs, they are developed by different community. I don't think they have other options other than separating them. That's not the case that we need to take a reference.

[zos]

I think having too much fine-grained repositories would end up with manageability disaster rather than providing a easy maintenance.

I do not agree, that it would be a disaster. Many projects chose this approach (like examples @bitboom quoted). See also how Rust Crypto namespace keeps it repositories. It is really fine-grained, but description in the namespace helps to clarify how to navigate around them.

For me it would be especially important to have it fine-grained, because we might (in the future) want to present a different usage for e.g. image-registry. And yes, we can split repository in the future, but it will have more severe consequences, because each dependant will have to update git path to our separated repository. This doesn't happen normally in other projects and we shouldn't really consider it as a "Plan B".

As for changes that would trigger updates in dependant repositories - this is something we have to take into the consideration, when designing each component. Of course it happens with opensource libraries from crates.io or other sources, but shouldn't happen too frequently, as it entails breaking changes in other modules. We should also take that into consideration and design appropriately.

It's uncomfortable to change the heads for so many submodules from each main repo for every update.

We do not have to keep Rust crates in submodules. We can directly add Github dependencies in Cargo.toml.

This approach offers additional advantages such as ensuring that developers primarily contributing to each repository can maintain high code quality with a sense of ownership.

This is a very good point. Having less fine-grained repositories would e.g. burden developers with unnecesary interrelated rebases, issues, PR-s for unrelated modules/components. Functionality is more clear with fine-grained repositories and other developers could find it easier to find modules that they would like to contribute to (for me, personally, it would be more discouraging to contribute to complex repository with multiple components and functionalities).

All of the remarks are valid and at the same time it seems that most of us are in the aggrement that the fine-grained solution will benefit future development, and I'm sure we will manage any complications that may come.