bug: Blight journal is empty

[F1o0T]

Hello,

Today, I have started to use MATE, I was able to compile and go through the notes.c example. However, I wasn't able to compile my example project that is compiled using 'make' after the Makefile generation through cmake. The Makefile is created under the build dir in the root src dir.

 mate-cli oneshot exampleProject/                                                          
✖ bb83b24f0e4242648ab3f9592ea1e0f7: failed

=======================
EXITED WITH: 2
=======================

=======================
STDOUT:

=======================

=======================
STDERR:
make: /usr/bin/cmake: Command not found
make: *** [Makefile:4932: cmake_check_build_system] Error 127

=======================
Although, I already have cmake at /usr/bin/cmake

whereis cmake                                                                     
cmake: **/usr/bin/cmake** /usr/lib/x86_64-linux-gnu/cmake /usr/lib/cmake /usr/share/cmake /usr/share/man/man1/cmake.1.gz /usr/src/googletest/googletest/cmake

I have also tried to creat a tar out of the exampleProject/ dir
tar czf exampleProject.tar.gz exampleProject/
mate-cli artifact create compile-target:tarball exampleProject.tar.gz
mate-cli compile create --wait --make-targets exampleProject.elf --artifact-id 6226e34a987942589c5c9cadb089c9ea

Still, it didn't work!

Any idea what I could did wrong, thank you!

[thinkmoore]

By default, compilation takes place in a docker image that does not have access to your local environment and cmake is not present. Two workarounds are:

1. Edit the makefile to install the necessary dependencies before the step that breaks
2. Use the more detailed mate-cli workflow to specify a container to use for the build (https://galoisinc.github.io/MATE/cli-overview.html#a-basic-workflow-with-mate-cli, https://galoisinc.github.io/MATE/cli.html#mate-cli-compile-create)

Unfortunately, the documentation for the latter is currently lacking...

[F1o0T]

Hi @thinkmoore,

Thanks for your reply, I have modified the Makefile to install cmake, and now it is able to see cmake, but I think command mate-cli is also still not mounting the sourcr directory into the container.

Now I get the error:

STDERR:
CMake Error: The source directory "/home/user/projects/testMate/exampleProject" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
make: *** [Makefile:4933: cmake_check_build_system] Error 1

May I ask if you can give some hints about how to use the more detailed mate-cli workflow to specify a container to use for the build, since as you said, the link you shared doesn't have enough information about that.

[F1o0T]

I have fixed the error of not being able to find the source directoy by mouning that directory into the container using the docker-compose.yml file by adding an entry to the executer under the volume section.

Now, I am facing another issue, my project is compiling successfully, but the build command is not showing any output. I understand that the build command is building the CPG graph. However, it is not showing anything.

Is it maybe because I am compiling for riscv not x86?

Thanks in advance for your support.

[thinkmoore]

0 builds completed successfully makes me wonder if something went wrong with the build interposition and we're not seeing any bitcode files. Can you run the first command here https://galoisinc.github.io/MATE/debugging-builds.html#debugging-a-build and see if there are any bitcode_artifact entries?

You might also check docker service logs mate_executor_1 for errors.

[F1o0T]

Here is the output of docker container logs mate-executer-one

 [2023-07-02 11:12:37,969: INFO/MainProcess] Task mate.tasks.compile.compile_artifact[8449e833-5b2c-4549-942b-54dab6120080] received
 [2023-07-02 11:12:37,970: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7fb71ade7700> (args:('mate.tasks.compile.compile_artifact', '8449e833-5b2c-4549-942b-54dab6120080', {'lang': 'py', 'task': 'mate.tasks.compile.compile_artifact', 'id': '8449e833-5b2c-4549-942b-54dab6120080', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8449e833-5b2c-4549-942b-54dab6120080', 'parent_id': None, 'argsrepr': "('d0a1495933d7400a88cab6310bec5c96', CompileOptions(testbed=None, containerized=False, experimental_embed_bitcode=False, docker_image=None, containerized_infer_build=True, make_targets=None, extra_compiler_flags=[]))", 'kwargsrepr': "{'compilation_id': 'fa0403f986a74d4984d9949918e245b9'}", 'origin': 'gen29@6fb04f90469d', 'ignore_result': False, 'properties': {'content_type': 'application/x-python-serialize', 'content_encoding': 'binary', 'application_headers': {'lang': 'py', 'task': 'mate.tasks.compile.compile_artifact', 'id': '8449e833-5b2c-4549-942b-54dab6120080', 'shadow': None, 'eta': None, 'expires': None, 'group':... kwargs:{})
 [2023-07-02 11:12:37,994: DEBUG/ForkPoolWorker-8] http://storage:9000 "GET /artifacts/d0a1495933d7400a88cab6310bec5c96 HTTP/1.1" 200 272442946
 [2023-07-02 11:12:42,450: DEBUG/ForkPoolWorker-8] starting at /tmp/tmpobg741j0/s400; searching ["I have intentially deleted file paths here since I am using it on proprietary codebase"]
 [2023-07-02 11:12:42,450: DEBUG/ForkPoolWorker-8] /tmp/tmpobg741j0/exampleProject does not contain a Makefile; checking the next layer
 [2023-07-02 11:12:42,451: DEBUG/ForkPoolWorker-8] found a makefile in /tmp/tmpobg741j0/exampleProject/build
 [2023-07-02 11:12:42,451: DEBUG/ForkPoolWorker-8] running args=['blight-exec', '--swizzle-path', '--guess-wrapped', '--', 'make']
 [2023-07-02 11:12:42,616: DEBUG/ForkPoolWorker-8] http://storage:9000 "PUT /artifacts/084ed03d58a5421199e5737ef10402a6 HTTP/1.1" 200 0
 [2023-07-02 11:12:42,617: ERROR/ForkPoolWorker-8]
 You have encountered a bug in MATE's implementation. Please report it.

 Additional information:
 artifact: given an empty fileobj to upload self.uuid='e601f9e652194a53872c44820731f976' self.kind=<ArtifactKind.BlightJournal: 'blight:journal'>
 [2023-07-02 11:12:42,621: DEBUG/ForkPoolWorker-8] http://storage:9000 "PUT /artifacts/e601f9e652194a53872c44820731f976 HTTP/1.1" 200 0
 [2023-07-02 11:12:42,892: INFO/ForkPoolWorker-8] Task mate.tasks.compile.compile_artifact[8449e833-5b2c-4549-942b-54dab6120080] succeeded in 4.921525783999641s: None

When I compare that log output with the log output of the notes.c provided example, it seems that the get-bc command is not called since an error occured while executing blight-exec.

[langston-barrett]

Here is that assertion.

[F1o0T]

Hi @langston-barrett,

I am still not sure how to work-around that issue, I have prepared a very simple cmake/make based project with the notes.c source file only. It seems that Mate/blight is not able to intercept the build commands so that it creates bitcode.

[woodruffw]

@xreinheitx could you check which version of blight you're running with? It's unlikely, but this may have been a regression on my part (blight is maintained externally, and has undergone changes since MATE was last released).

[F1o0T]

@woodruffw , I think Mate is using blight which is shipped with the container ghcr.io/galoisinc/mate-dist:main, I am not sure how can I get the version of that one.

[thinkmoore]

I wonder if the problem is that CMAKE is hardcoding a path for cc, etc. and so blight's PATH-based interposition isn't working...

[matteasu]

@xreinheitx you can run a shell on that specific container and then check the version using pip list
https://kodekloud.com/blog/docker-exec/

[F1o0T]

Hi @matteasu,

Thanks for the hint, the version is 0.0.41

root@7f4a8e195da9:~# pip list | grep blight
blight                 0.0.41

[F1o0T]

@thinkmoore, FYI, I grepped grep -r '$(CC)' . the build dir that CMake used to store the Makefiles, and I couldn't see that the CC is used!