refactoring: Isolate clarity VM behind feature gate for lightweight serialization

[Jiloc]

Description

This PR builds on top of #6238, and will be rebased once the other PR will be merged.

The primary goal is to separate the core Clarity data types and their serialization/deserialization logic from the heavyweight components of the full Clarity Virtual Machine.

Currently, any project using the clarity crate, even for simple value serialization, must include the entire VM with all its heavy dependencies. This results in unnecessarily large binaries and slower compile times for tools that only need to interact with Clarity data structures.

This PR introduces a vm feature flag that gates all VM-related code, including parsing, analysis, and execution. This allows developers to use the clarity crate in a lightweight, serialization-only mode by disabling default features.

Key Changes:

• Introduced vm Feature Flag:

  • A new vm feature has been added to Cargo.toml.
  • Many dependencies (rand, integer-sqrt, serde_stacker, rusqlite, etc.) are now optional and included as part of the vm feature.
  • The testing, developer-mode, devtools, and disable-costs features now imply the vm feature.

• Gated VM Modules and Functionality:

  • The core VM engine, analysis passes (type_checker, read_only_checker, etc.), and execution logic are now conditionally compiled with #[cfg(feature = "vm")].
  • This ensures that projects only needing data structures and serialization do not compile the full virtual machine.

comparison between buidling with default flags and --no-default-flags

franc@ASUS-VIVOBOOK:~/stacks-core$ cargo clean && time cargo build --package clarity
real    0m57.571s
user    5m32.915s
sys     0m14.679s

franc@ASUS-VIVOBOOK:~/stacks-core$ cargo clean && time cargo build --package clarity --no-default-features
real    0m18.373s
user    3m16.690s
sys     0m9.888s

This change is fully backward compatible. The default features in Cargo.toml are configured to include vm, so existing projects will experience no breaking changes.

OPEN TO SUGGESTIONS TO IMPROVE IT!

Applicable issues

• fixes Refactor clarity to enable lightweight serialization crate #6236

Additional info (benefits, drawbacks, caveats)

Checklist

• Test coverage for new or modified code paths
• Changelog is updated
• Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo
• New integration test(s) added to bitcoin-tests.yml

[raress96]

Hello,

There is an issue when compiling with this for CosmWasm context.

The issue comes from getrandom dependency, which is used in rand.

error: the wasm*-unknown-unknown targets are not supported by default, you may need to enable the "js" feature. For more information see: https://docs.rs/getrandom/#webassembly-support
   --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/getrandom-0.2.16/src/lib.rs:346:9
    |
346 | /         compile_error!("the wasm*-unknown-unknown targets are not supported by \
347 | |                         default, you may need to enable the \"js\" feature. \
348 | |                         For more information see: \
349 | |                         https://docs.rs/getrandom/#webassembly-support");

It is similar to this issue:
cosmos/solidity-ibc-eureka#155
I also found a similar discussion here:
CosmWasm/cosmwasm#1143

The js feature can't be used since it is not available in CosmWasm context.

Can you take a look at the rand / getrandom dependency and how this is used in the clarity or stacks-common crate? Maybe it can be conditionally enabled.

To reproduce locally, you can get the code from this branch:
Trust-Machines/axelar-amplifier#8

And run the CosmWasm Optimizer build using Docker:

docker run --rm -v "$(pwd)":/code \
  --mount type=volume,source="$(basename "$(pwd)")_cache",target=/target \
  --mount type=volume,source=registry_cache,target=/usr/local/cargo/registry \
  cosmwasm/optimizer:0.16.1

[raress96]

Hello, commit d64ad0b88671d841959334ff1be2108c14073402 successfully compiled in CosmWasm context. However, the latest commit 526c0d15808847096df4fc8a752d3b6dcb0b7069 no longer compiles, it fails with these errors:

--> /usr/local/cargo/git/checkouts/stacks-core-a7710d651838f98d/526c0d1/stacks-common/./src/util/secp256k1/wasm.rs:200:43
    |
200 |     let recovered_pub_key = libsecp256k1::recover(&message, &signature, &recovery_id)?;
    |                                           ^^^^^^^ not found in `libsecp256k1`
    |
note: found an item that was configured out
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:808:8
    |
808 | pub fn recover(
    |        ^^^^^^^
note: the item is gated here
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:806:1
    |
806 | #[cfg(any(feature = "static-context", feature = "lazy-static-context"))]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0425]: cannot find function `verify` in crate `libsecp256k1`
   --> /usr/local/cargo/git/checkouts/stacks-core-a7710d651838f98d/526c0d1/stacks-common/./src/util/secp256k1/wasm.rs:217:29
    |
217 |     let res = libsecp256k1::verify(&message, &signature, &pubkey);
    |                             ^^^^^^ not found in `libsecp256k1`
    |
note: found an item that was configured out
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:790:8
    |
790 | pub fn verify(message: &Message, signature: &Signature, pubkey: &PublicKey) -> bool {
    |        ^^^^^^
note: the item is gated here
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:788:1
    |
788 | #[cfg(any(feature = "static-context", feature = "lazy-static-context"))]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0425]: cannot find function `sign` in crate `libsecp256k1`
   --> /usr/local/cargo/git/checkouts/stacks-core-a7710d651838f98d/526c0d1/stacks-common/./src/util/secp256k1/wasm.rs:338:42
    |
338 |         let (sig, recid) = libsecp256k1::sign(&message, &self.key);
    |                                          ^^^^ not found in `libsecp256k1`
    |
note: found an item that was configured out
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:857:8
    |
857 | pub fn sign(message: &Message, seckey: &SecretKey) -> (Signature, RecoveryId) {
    |        ^^^^
note: the item is gated here
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:852:1
    |
852 | / #[cfg(all(
853 | |     feature = "hmac",
854 | |     any(feature = "static-context", feature = "lazy-static-context")
855 | | ))]
    | |___^

error[E0599]: no function or associated item named `from_secret_key` found for struct `libsecp256k1::PublicKey` in the current scope
   --> /usr/local/cargo/git/checkouts/stacks-core-a7710d651838f98d/526c0d1/stacks-common/./src/util/secp256k1/wasm.rs:107:42
    |
107 |         let key = LibSecp256k1PublicKey::from_secret_key(&privk.key);
    |                                          ^^^^^^^^^^^^^^^ function or associated item not found in `PublicKey`
    |
note: if you're trying to build a new `libsecp256k1::PublicKey` consider using one of the following associated functions:
      libsecp256k1::PublicKey::from_secret_key_with_context
      libsecp256k1::PublicKey::parse_slice
      libsecp256k1::PublicKey::parse
      libsecp256k1::PublicKey::parse_compressed
      libsecp256k1::PublicKey::combine
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:107:5
    |
107 | /     pub fn from_secret_key_with_context(
108 | |         seckey: &SecretKey,
109 | |         context: &ECMultGenContext,
110 | |     ) -> PublicKey {
    | |__________________^
...
123 |       pub fn parse_slice(p: &[u8], format: Option<PublicKeyFormat>) -> Result<PublicKey, Error> {
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
158 |       pub fn parse(p: &[u8; util::FULL_PUBLIC_KEY_SIZE]) -> Result<PublicKey, Error> {
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
192 | /     pub fn parse_compressed(
193 | |         p: &[u8; util::COMPRESSED_PUBLIC_KEY_SIZE],
194 | |     ) -> Result<PublicKey, Error> {
    | |_________________________________^
...
299 |       pub fn combine(keys: &[PublicKey]) -> Result<Self, Error> {
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: there is an associated function `from_secret_key_with_context` with a similar name
   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsecp256k1-0.7.2/src/lib.rs:107:5
    |
107 | /     pub fn from_secret_key_with_context(
108 | |         seckey: &SecretKey,
109 | |         context: &ECMultGenContext,
110 | |     ) -> PublicKey {
    | |__________________^

[Jiloc]

@raress96 can you try building stacks-core with the 'wasm-deterministic' feature flag?

stacks-common = { git = "https://github.com/Jiloc/stacks-core", branch = "chore/clarity-vm-feature-gate", default-features = false, features = ["wasm-deterministic"] }

[raress96]

@raress96 can you try building stacks-core with the 'wasm-deterministic' feature flag?

stacks-common = { git = "https://github.com/Jiloc/stacks-core", branch = "chore/clarity-vm-feature-gate", default-features = false, features = ["wasm-deterministic"] }

It compiles fine with this, thanks.