Islet refactoring

[p-sawicki2]

Here is a prioritized list of tasks (the most important tasks are first) that we think can improve the project, and make the development process easier. This list is the outcome of our internal discussion, based on our experiences during the development of the application provisioning mechanism. Please feel free to share any suggestions.

Review / refactor of the islet repository

1 Remove optee project from the build system

Optee provides only a couple of commands that are used in the build system. It's unnecessary dependency that can be easily replaced with few script commands (like it was done in fvp-cca-scripts).

Reason: the build system is a complex mix of python, bash and some makefile scripts from optee project. It should be more unified as currently it can be hard to find and patch something.

2 Splitting repos from islet-project/islet.

We intend to make islet-project/islet repo a simple repo with just the build system that collects everything, builds and runs the CCA stack. Islet-rmm and every other binary entity residing in the monorepo should be moved to a separate one.

Reason:

a. Repositories that are required when compiling/running islet as more than a cargo dependency can be added to git submodules.
b. Having multiple repos adds ease of reuse by us or some third parties. We also had problems with reusing components. Currently, the linux-rsi component is located both in islet monorepo and in realm-manager as cloning entire islet monorepo just to access 2 files isn’t feasible.
c. Keeping each standalone tool in a separate repository helps lower down the complexity of the directory structure and allow for easier modification of the whole CCA stack.
d. The advantage of having separate repositories was already discussed here #345

Entities from islet monorepo that in our opinion should be transferred:

• Islet-rmm
  Keeping the islet-rmm in a separate repository will visually separate the rmm code from other components that doesn’t need to be upheld to such vigorous security standards and formal verification.

• Linux-rsi
  Kernel module for RSI API. It can be reused in multiple places of the stack thus should be kept in a separate repository.
   

• islet/examples
  Having a repository with just the examples in such big project helps in attracting open source contributors and then getting them familiar with the project. Learning from examples is a way more entertaining than reading plain documentation so a well-structured example repository can be handy.

• islet-hes
  Islet-hes is just a temporary solution to emulate HES component of the CCA stack. It is supposed to run on a separate CPU core. For this reason, it should be separated as a distinct software and hardware component.

3 Removing unnecessary entities from islet

• Islet cli tool

Reason: The current islet cli tool is a modified fork of rsictl from islet-project/remote-attestation. This fork is not being updated and has become outdated and not compatible with the current RSI API. We think that we should deprecate this cli tools and just use the rsictl instead. We basically think the same as mentioned by Sangwan in #382 (comment)

4 Splitting repos from remote-attestation

Reason: Similar argument like with splitting repositories from islet. Additionally, working with all those crates in one repository causes some issues for the cargo tool. When creating dependencies for crates here we need to point all crates to one branch, even though the changes required for that branch were only in one crate. Cargo tool has some issues in such use case. Splitting that to separate repositories makes working with those crates much easier.

Entities from remote-attestation monorepo that in our opinion should be transferred:

• Ratls library
  TLS wrapper implementing additional security layer for verification based on CCA token.

• Realm-verifier
  Library for verifying realm measurements. Might be unnecessary as Veraison implemented realm token verification. Although, we didn’t test it yet.

• Rust-rsi
  Rust implementation of various APIs and helpers for working with linux-rsi API and CCA tokens.

• Rocli
  Tool used to feed Veraison with endorsements and reference values.

• Rsictl
  Rust cmd line tool for working with linux-rsi kernel module to perform RSI operations (attestation, measurements, sealing) in the realm.

5 Clean up the third-party repository chaos

Remove islet third party (islet-project/assets) repository altogether.

Reason: This repository is way too large. When you need to add anything you must download GB of data. Additionally, shallow copy inside islet/third-parties are really hard to work with. Any modification of the shallow submodules requires downloading the history. This makes it really hard to work with. Furthermore adding new forks on another branch in the assets repo feels like abusing git. Organizations and forking we invented to fix the issue that islet/assets made. Since we have an organization we should use it fully and not keep the project in a transient state where some forks are kept as standalone repos under organization and some are just separate branches in the assets repository.

Our recommendation:

• Each repo that is modified should be forked under islet-project. We can prefix those forks to indicate that they are not exclusively added by islet project.
• Each repo that is used unmodified should use upstream source.

6 Fix dependencies in fvp-cca scripts

Reason: Some of the apt packages that are installed aren’t really required to build islet and conflict with default Ubuntu’s compilers. In detail, stuff like gcc/g++ multilib is unnecessary and removes Ubuntu's aarch64 compiler.

7 Add QEMU emulator

Reason: QEMU allows us to use gdb, which will be a necessity to provide debugging capabilities. Furthermore, we will be able to implement our custom hardware components. For example we could implement RPMB to provide proper rollback protection.

8 Network setup cleanup

In general we should make the network configuration as robust and as simple as humanly possible. While making it work at all in the first place.

Reason: Currently, the network setup is dispersed over a lot of different files and even repositories. Some is done in the fvp-cca script, some in “configure_tap.sh” (and other bash files) and some is even done in the forked optee-build repo.

[bitboom]

First of all, thank you for your insights and efforts. 😄 It seems there has been a in-depth discussion on this.

I agree strongly with most of the 8 proposed tasks. (I plan to work on Task 3 in November.)

2 Splitting repos from islet-project/islet.

I have a slightly different perspective on Task 2. The islet-rmm is essential for running islet, so I think it would be better to keep it within the islet repository. Since rmm needs to be updated and developed continuously, separating it could slow down this process. The directory structure below is intuitive and helps convey the structure in which islet operates:

islet
  ├── rmm
  ├── plat
      ├── fvp
      └── qemu (to be added)

the rmm code from other components that doesn’t need to be upheld to such vigorous security standards and formal verification.

I agree that efforts like formal verification or safety improvements could make rmm more complex or harder to understand. For these cases, I think moving them into separate repositories as a good alternative.
(I agree with separating linux-rsi, islet-examples, and hes into separate repositories.)

---

Additionally, I suggest detaching aosp related code and third-parties into islet-aosp. It would be preferable for islet to contain only the essential components needed to run the CCA stack.

[jinbpark]

Thanks for this well-written description on refactoring!
I agree with all you mention here. Here is my personal opinions--

---

islet/examples (Learning from examples is a way more entertaining)

It's not quite having to do with refactoring.
I always feel like the examples we have are too big to get started and to attract other developers. (since they were the outcome of the collaboration with VMware for CCSummit).

I strongly believe we need to create much simpler examples (e.g., hello_world, remote_attestation_hello_world, ...) with automated scripting and proper documentation (our github page).

Specifically, enabling the following procedure would be nice--

1. write a simple program and cross-compile it --> named "user_prog".
2. ./run_inside_realm /path/to/user_prog
3. .... running user_prog (from external developer perspectives, they don't need to do something inside FVP Host and Realm in order to run user_prog).

---

Add QEMU emulator

That's what I want to see the most. I'm wondering how much performance benefits we can gain from using QEMU in place of FVP.
Performance issue is critical in particular to platform developers like us. Of course, using QEMU would be much better to other external developers (most engineers are much more familiar with QEMU)

---

Network setup cleanup

Agreed. I made network configurations in a way that allows users to assign a custom IP address to each ones (e.g., tap device ip, realm ip, and so on). I think using hard-coded IP addresses makes things simple.
For example, kvmtool by default creates a tap device with a default interface name (tap0) and ip address (192.168.33.1?? I don't remember exactly). We can use this default setting. And, embedding an hard-coded IP address in realm-rootfs might help eliminate unnecessary network configuration scripts.

---

Splitting repos from islet-project/islet.

Personally, I'm standing with @bitboom 's opinion.
We'd better prioritize which repositories should be moved first. (of course, linux-rsi, examples, hes, ... should be the first)
And it would be not late to re-discuss whether 'islet-rmm' should be moved or not, after getting other stuffs done first.

---

Clean up the third-party repository chaos

Do we need to revisit all the third-party repositories to check if which one should just point to the upstream source?
I mean, some changes that we've made to a forked repo can be discarded entirely.
So, we'd better investigate on it first.

[zpzigi754]

I agree with most of the points mentioned.

I think that maintaining linux-rsi and islet-hes in a separate repository would be totally acceptable. I also agree with adding QEMU emulator and cleaning up network setup.

I am personally against moving RMM into another repository, as I think that it could be the common component to be referred to.
I also think that simple examples such as hello-realm can be located in the existing location (islet/examples) to help run and develop the test quickly.

[nook1208]

Thank you for your great suggestions from a refactoring perspective.
I agree with most of the content and relate to it as well.

2 Splitting repos from islet-project/islet.

Like @bitboom's opinion, I believe that islet-rmm should remain in islet because it is a core component of islet.

5 Clean up the third-party repository chaos

Totally agree with you. Managing it in forked repo form would make it much less complex.

[bokdeuk-jeong]

AOSP is either a host OS or a realm OS, like Linux or Linux-net, so having its own separate repo would follow the same logic. But creating a separate repo called ‘islet-aosp’ doesn’t seem consistent with our criteria for separation, so I’m not in favor of that. The only part left related to AOSP in the islet repo is for fvp-cca, and since AOSP will be a main part we’re focusing on going forward, it should ideally be executable with the fvp-cca script.

Here’s a summary of where we stand:

Most of Piotr's proposals have reached consensus, but on the topic of separating the RMM repository, everyone who responded here agreed that RMM’s core components should stay in the main islet repository. However, islet’s core components actually include both RMM and HES. So, we need to decide: should we split both RMM and HES into separate repositories, or should both stay in islet’s main repository? Please share your thoughts on which option we should go with.

[p-sawicki2]

Arguments against keeping the HES source code in the main Islet repo:

• Easier maintenance of the source code of both Islet and HES repositories
• Islet HES is intended to run on a separate MCU that is not the part of the main Application Processor
• In the future Islet HES is supposed to be ported to a dedicated Secure Operating System running on a dedicated MCU. Having separate repository will make the porting process easier.
• Islet may not only use our HES implementation but also Arm's RSE (Runtime Security Engine)

[Havner]

My take on it is as follows:

Islet as a project actually consists of several subprojects (code wise) and it's really hard to tell which is core component and which is not. The whole stack requires several components to even run it. Islet RMM is one of them. Just as TF-A, Linux kernels. I wouldn't say any of those is more core than others. The only thing that separates Islet RMM from the others is that it's completely written in house.

To actually refactor this cleanly I would like to see the main Islet repository without any code. There is already so much stuff in there (documentation, starting scripts, build scripts, examples, several use cases) that such a repository would be something that glue everything together. Especially that like I said, IMO we don't have any one specific component that could be considered core (in terms of running the whole stack).

So I would like to split everything to separate repositories (islet rmm and hes included) and have to main repository as something that is a central hub for all of this, but not contain any code by itself.

[p-sawicki2]

@bokdeuk-jeong Since we have a consensus on most of the refactoring points, would you agree to start working on parts that are less invasive to the main Islet repository? For example, the 4th task "Splitting repos from remote-attestation" seems like a good candidate to begin with. @Havner could take care of it.

[Havner]

A question, for the refactoring we'll be needing to fork several upstream repositories into islet-project (e.g. to cleanup assets or to freeze some upstream dependency like veraison with which we've had lots of issues before).

I suggest to prefix such repos with some string so they are easier to distinguish than projects written in-house and sorted.

I suggest 3rd-. So we'd have islet-project/3rd-veraison-services
Any ideas/suggestions?

[bokdeuk-jeong]

@p-sawicki2
Most of items in your initial proposal have reached an agreement, including #4, 'Splitting repos from remote-attestation'. Let's proceed with you proposal as suggested, except splitting islet-rmm and islet-hes mentioned in in #2, 'Splitting repos from islet-project/islet'.
@zpzigi754
I will leave the task of adding this nice introductory example to you.

"I also think that simple examples such as hello-realm can be located in the existing location (islet/examples) to help run and develop the test quickly."

@Havner
Adding the prefix '3rd' to the forked repos for our own usage looks good. Let's do it.
Neverthless, it is important to remember that we should try to avoid forking repos and utilize the upstream instead.

As we spent enough time for the dicussion, I will close this discussion.