Cloudwatch Agent SIGHUP Log Rotation Fix

[Paramadon]

Description of the issue

Currently Cloudwatch agent stops detecting log file rotation after recieving a SIGHUP signal. This issue can be resolving by having a persistent context for monitoring only to ensure that log rotation happens correclty.

Note: This is a solution to this problem, this solution makes the log Agent context persistent meaning clean up only happens in agent shut down. If we do prefer a alternative solution, we can work towards that. This is a good initial step.

Tests

Tested this locally by running

ps aux | grep "/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent"
sudo kill -HUP <process ID>

//Performing log rotation
rm test.log 
echo "test" > test.log

Log below shows the testing below

root      ***  0.2  0.5 1234 123  0:00 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json -otelconfig /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml -pidfile /opt/aws/amazon-cloudwatch-agent/var/amazon-cloudwatch-agent.pid
root      ***  0.0  0.0 1234  1234  grep --color=auto /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent
[root***]# sudo kill -HUP <process ID>
[root@***]# sudo kill -HUP 337309
[root@***]# tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
2025-04-09T23:09:51Z D! [logagent] open file count, 1
2025-04-09T23:09:52Z D! [logagent] open file count, 1
2025-04-09T23:09:53Z D! [logagent] open file count, 1
2025-04-09T23:09:54Z D! [logagent] open file count, 1
2025-04-09T23:09:55Z D! [logagent] open file count, 1
[root@***]# ^C
[root@***]# ^C
[root@***]# sudo rm test.log
[root@***]# tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
2025-04-09T23:09:59Z D! [logagent] open file count, 1
2025-04-09T23:10:00Z D! [logagent] open file count, 1
2025-04-09T23:10:01Z D! [logagent] open file count, 1
2025-04-09T23:10:01Z W! [inputs.tail] Stopping tail as file no longer exists: /home/ec2-user/test.log
2025-04-09T23:10:01Z W! [logfile] deleting state file /opt/aws/amazon-cloudwatch-agent/logs/state/_home_ec2-user_test.log
2025-04-09T23:10:01Z I! [logagent] Log src has stopped for i-***/i-***/tmp1(/home/ec2-user/test.log)
2025-04-09T23:10:02Z D! [logagent] open file count, 0
2025-04-09T23:10:02Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log: no such file or directory
2025-04-09T23:10:03Z D! [logagent] open file count, 0
2025-04-09T23:10:03Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log: no such file or directory
2025-04-09T23:10:04Z D! [logagent] open file count, 0
2025-04-09T23:10:04Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log:
[root@***]# echo "test2" > test.log
[root@***]# tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
2025-04-09T23:10:05Z D! [logagent] open file count, 0
2025-04-09T23:10:05Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log: no such file or directory
2025-04-09T23:10:06Z D! [logagent] open file count, 0
2025-04-09T23:10:06Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log: no such file or directory
2025-04-09T23:10:07Z D! [logagent] open file count, 0
2025-04-09T23:10:07Z D! Stat file /home/ec2-user/test.log failed due to stat /home/ec2-user/test.log: no such file or directory
2025-04-09T23:10:08Z D! [logagent] open file count, 0
2025-04-09T23:10:08Z D! [inputs.logfile] The state file /opt/aws/amazon-cloudwatch-agent/logs/state/_home_ec2-user_test.log for /home/ec2-user/test.log does not exist: stat /opt/aws/amazon-cloudwatch-agent/logs/state/_home_ec2-user_test.log: no such file or directory
2025-04-09T23:10:08Z I! [logagent] piping log from i-***/i-***/tmp1(/home/ec2-user/test.log) to cloudwatchlogs with retention -1
2025-04-09T23:10:09Z D! [logagent] open file count, 1
2025-04-09T23:10:10Z D! [logagent] open file count, 1
2025-04-09T23:10:11Z D! [logagent] open file count, 1

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[github-actions]

This PR was marked stale due to lack of activity.