aws-lc-rs causes substantial binary size increase

[landonxjames]

Problem:

The AWS Rust SDK recently updated our default version of rustls to one that uses aws-lc-rs as the crypto provider. This has caused a substantial (consistently ~4MB in my tests) increase in binary size. This has caused some pain points for customers: smithy-lang/smithy-rs#4049. In our debugging using cargo-bloat it appears that much of this size comes from aws-lc functions as can be seen in the output below from a small test application.

$ cargo bloat --release -n 50
 File  .text     Size                  Crate Name
 1.0%   2.1% 173.1KiB             aws_sdk_s3 aws_sdk_s3::config::endpoint::internals::resolve_endpoint
 0.3%   0.6%  47.2KiB              [Unknown] edwards25519_scalarmulbase_alt_edwards25519_gtable
 0.3%   0.6%  47.2KiB              [Unknown] edwards25519_scalarmulbase_edwards25519_gtable
 0.3%   0.6%  47.2KiB              [Unknown] curve25519_x25519base_byte_alt_edwards25519_gtable
 0.3%   0.6%  47.2KiB              [Unknown] curve25519_x25519base_byte_edwards25519_gtable
 0.2%   0.4%  30.7KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber512_ref_indcpa_keypair_derand
 0.2%   0.4%  28.7KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber768_ref_indcpa_keypair_derand
 0.2%   0.3%  25.7KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber768_ref_indcpa_enc
 0.1%   0.3%  24.9KiB              [Unknown] p384_montjscalarmul_p384_montjadd
 0.1%   0.3%  23.1KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber512_ref_indcpa_enc
 0.1%   0.3%  21.8KiB             aws_config aws_config::loader::ConfigLoader::load::{{closure}}
 0.1%   0.3%  21.8KiB             aws_config aws_config::loader::ConfigLoader::load::{{closure}}
 0.1%   0.3%  21.8KiB             aws_config aws_config::loader::ConfigLoader::load::{{closure}}
 0.1%   0.2%  17.1KiB              [Unknown] p384_montjscalarmul_alt_p384_montjadd
 0.1%   0.2%  17.0KiB            aws_sdk_sts aws_sdk_sts::config::endpoint::internals::resolve_endpoint
 0.1%   0.2%  16.4KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber1024_ref_indcpa_keypair_derand
 0.1%   0.2%  16.0KiB             aws_config aws_config::sso::credentials::load_sso_credentials::{{closure}}
 0.1%   0.2%  15.9KiB             aws_config aws_config::ecs::EcsCredentialsProvider::credentials::{{closure}}
 0.1%   0.2%  14.0KiB              [Unknown] _ec_GFp_nistp224_point_get_affine_coordinates
 0.1%   0.2%  14.0KiB             aws_config aws_config::sso::token::SsoTokenProvider::refresh_cached_token::{{closure}}
 0.1%   0.2%  13.1KiB             aws_lc_sys _aws_lc_0_27_0_pqcrystals_kyber1024_ref_indcpa_enc
 0.1%   0.2%  12.7KiB                   http http::header::name::StandardHeader::from_bytes
 0.1%   0.2%  12.7KiB                   http http::header::name::StandardHeader::from_bytes
 0.1%   0.2%  12.4KiB             aws_config <core::pin::Pin<P> as core::future::future::Future>::poll
 0.1%   0.2%  12.4KiB             aws_config aws_config::imds::credentials::ImdsCredentialsProvider::credentials::{{closure}}
 0.1%   0.1%  12.2KiB              [Unknown] p256_montjscalarmul_p256_montjadd
 0.1%   0.1%  12.2KiB              aws_sigv4 aws_sigv4::http_request::canonical_request::CanonicalRequest::from
 0.1%   0.1%  12.1KiB              [Unknown] p384_montjscalarmul_p384_montjdouble
 0.1%   0.1%  12.1KiB             aws_lc_sys _p384_montjdouble
 0.1%   0.1%  12.0KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  12.0KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  11.9KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  11.9KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  11.8KiB             aws_lc_sys _aws_lc_0_27_0_ml_kem_indcpa_enc_ref
 0.1%   0.1%  11.7KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  11.7KiB     aws_smithy_runtime aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}}
 0.1%   0.1%  11.6KiB             hyper_util hyper_util::client::legacy::client::Client<C,B>::send_request::{{closure}}
 0.1%   0.1%  11.5KiB             aws_lc_sys _aws_lc_0_27_0_ml_kem_indcpa_keypair_derand_ref
 0.1%   0.1%  11.4KiB             aws_sdk_s3 aws_sdk_s3::protocol_serde::shape_put_object::ser_put_object_headers
 0.1%   0.1%  11.3KiB             aws_config aws_config::sso::token::SsoTokenProvider::refresh_cached_token::{{closure}}
 0.1%   0.1%  11.3KiB             aws_config aws_config::profile::credentials::ProfileFileCredentialsProvider::load_credentials::{{closure}}
 0.1%   0.1%  11.0KiB             aws_sdk_s3 aws_sdk_s3::s3_express::identity_provider::DefaultS3ExpressIdentityProvider::identity::{{closure}}
 0.1%   0.1%  10.9KiB             aws_lc_sys _aws_lc_0_27_0_ml_dsa_sign_internal
 0.1%   0.1%  10.7KiB             aws_config aws_config::web_identity_token::WebIdentityTokenCredentialsProvider::credentials::{{closure}}
 0.1%   0.1%  10.7KiB            aws_sdk_sts <aws_sdk_sts::operation::assume_role_with_web_identity::AssumeRoleWithWebIdentityResponseDeserializer as aws_smithy_runtime_api::client::ser_de::DeserializeResponse>::deseriali...
 0.1%   0.1%  10.6KiB                     h2 h2::codec::framed_read::decode_frame
 0.1%   0.1%  10.6KiB                     h2 h2::codec::framed_read::decode_frame
 0.1%   0.1%  10.5KiB            aws_config? <aws_config::imds::client::token::TokenResolver as aws_smithy_runtime_api::client::identity::ResolveIdentity>::resolve_identity::{{closure}}
 0.1%   0.1%  10.5KiB             aws_lc_sys _p521_jdouble_alt
 0.1%   0.1%  10.3KiB aws_smithy_http_client h2::proto::connection::Connection<T,P,B>::poll
42.0%  87.6%   7.0MiB                        And 23518 smaller methods. Use -n N to show more.
47.9% 100.0%   7.9MiB                        .text section size, the file size is 16.6MiB

Solution:

I'm afraid I don't have many suggestions for a full solution to this. I suspect that since aws-lc is a mostly C based dependency that much of rustc's dead code elimination doesn't work for it so we are getting stuck pulling in functions we aren't actually using. The cheeky suggestion there is rewrite it in Rust, but that isn't a reasonable solution on any kind of short timescale.

Potentially splitting the various algorithms into their own crates could help rustc clean up better?

[cxou]

Cross-language LTO might offer some benefits, but it’s a complex beast and not guaranteed to deliver. The Rust compiler appears to support only ThinLTO rather than Fat/Full LTO, which seems to imply it’s more of a Minimum Viable Product than a generally available feature.
https://blog.llvm.org/2019/09/closing-gap-cross-language-lto-between.html
https://doc.rust-lang.org/rustc/linker-plugin-lto.html