SES service allocates huge amounts of memory

[dziedrius]

Description

We've noticed huge memory traffic in our app after we started to send larger attachments with SES, so I've composed small app to see what is going on. To send ~5MB email message SES SDK allocates > 250MB of memory, which is 50 times more than message itself, to send 4 such messages in parallel we need to allocate more than 1GB of RAM, which in my opinion is not right.

Screenshots are taken from dotMemory profiler, I was running console app below with allocations recording.
In total sending email allocated:

The call tree is very deep, hence did not fit in single screenshot, but main parts are two, CreateWebRequest:

and signing:

Some strange things, I have to admit that I haven't checked the source code:

1. Seems that GetParametersAsString was called twice, hence allocating twice.
2. Why you need to sign request - I'm pretty sure it travels through HTTPS.
3. Why you need to encode everything as url and why it takes so much memory.

Reproduction Steps

Project:

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net5.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="AWSSDK.SimpleEmail" Version="3.7.0.73" />
      <PackageReference Include="MimeKit" Version="2.15.1" />
    </ItemGroup>

</Project>

Program.cs

using System;
using System.IO;
using System.Threading.Tasks;
using Amazon;
using Amazon.Runtime;
using Amazon.SimpleEmail;
using Amazon.SimpleEmail.Model;
using MimeKit;

namespace SesMemory
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var options = FormatOptions.Default.Clone();
            
            var sesKey = "my-ses-key";
            var sesSecret = "my-ses-secret";
            var region = "us-west-2";
            var amazonSimpleEmailService = new AmazonSimpleEmailServiceClient(
                new BasicAWSCredentials(sesKey, sesSecret),
                RegionEndpoint.GetBySystemName(region));
            
            var mimeMessage = CreateMimeMessage();

            // entry point for memory profiler capture
            Console.WriteLine("press key to start");
            Console.ReadKey();
            
            using (var memoryStream = new MemoryStream())
            {
                await mimeMessage.WriteToAsync(options, memoryStream);
                
                var sendRequest = new SendRawEmailRequest  { RawMessage = new RawMessage(memoryStream)};
                var response = await amazonSimpleEmailService.SendRawEmailAsync(sendRequest);
            }
            
            // exit point for memory profiler capture
            Console.WriteLine("press key to exit");
            Console.ReadKey();
        }

        private static MimeMessage CreateMimeMessage()
        {
            var emailMessage = new MimeMessage();

            emailMessage.From.Add(new MailboxAddress("my@address.com", "my@address.com"));
            emailMessage.To.Add(new MailboxAddress(string.Empty, "me@address.com"));
            emailMessage.Subject = "test subject";

            var emailBody = new BodyBuilder
            {
                TextBody = "text body",
                HtmlBody = "html body"
            };
            
            // ~5mb png
            emailBody.Attachments.Add("test.jpg", File.ReadAllBytes(@"C:\temp\test-image.png"));
            
            emailMessage.Body = emailBody.ToMessageBody();

            return emailMessage;
        }
    }
}

Environment

• SDK Version: 3.7.0.73
• Package Version: 3.7.0.73
• OS Info: Windows 10 | Linux Alpine
• Build Environment: Rider
• Targeted .NET Platform: 5.0

Resolution

Optimize memory allocations so that it would not be 50 times size of the message.

---

This is a 🐛 bug-report

[dziedrius]

Four months later no reaction :)

[danielmarbach]

@dziedrius I'm just some community rando and not working for AWS. I briefly looked into this and it seems it would be "easily" possible to get rid of most of the allocations caused by GetParameterAsString by using a ValueStringBuilder approach that rents from the buffer pool.

That would at least address the issues you see in your screenshot but it would require to

• Take a dependency to <PackageReference Include="System.Buffers" Version="4.5.1" /> and <PackageReference Include="System.Memory" Version="4.5.5" /> both in the NET 45 and the netstandard target
• Create a dedicated method for bcl45+netstandard that uses the value string builder

On top of that, I looked at the ComposeUrl Method and the pipeline and there is a lot more allocations happening there that could also benefit from those two packages being pulled in that could occupy my desire to make things faster for days 🎆

Given that also involves copying MIT licensed code from the .NET runtime repo I'm hesitant to invest more time until I hear something from the team and especially @normj since I would be essentially contributing non-trivial amount of my spare time to this.

[dziedrius]

@danielmarbach thanks for looking into issue, I'm not a AWS employee too, but it would be nice to hear someone from AWS on this issue, as it was opened 16 months ago.

[andriusrazinskas]

I have encountered the same issue. Latest AWSSDK.SimpleEmail 3.7.300.7 version still behaves the same - GetParametersAsString is allocating a lot, and twice.

[danielmarbach]

I started #3042 but then got sidetracked due to travel and other commitments. :( I will eventually pick it up as a community effort but can't make any firm promises.

[parkinsona]

I have noticed this issue as well when sending large attachments via SES.
I saw the other #3042 , and it mentioned version 4 of the SDK has newer libraries added.
Not sure if they fixed the actual issue though.
I've tried version 4, and the memory usage is still high (if not higher) . Was previously 3.5GB for 100 recipients with 1mb each, but it's now sitting at 4GB.