Stream agent logs remotely #1774

jan-law · 2023-10-16T20:12:49Z

Fixes #1655

Depends on #1642

Example usage: opni debug agent-logs --level=debug --follow <cluster-id>

jan-law

Added two questions about writing plugin loggers to log files. Once those are resolved, this PR is ready for review

pkg/logger/logger.go

plugins/metrics/pkg/agent/drivers/opni_manager_otel/otel_driver.go

jan-law · 2023-10-18T21:09:10Z

When running the opni debug agent-logs in my k3s cluster, I get an Unrecognized remote plugin message for one of the log messages. Do you know where this log message is created?

Tried searching here and couldn't find it: https://github.com/search?q=repo%3Aopen-telemetry%2Fopentelemetry-go-contrib+%22Tracer+created%22&type=code

2023 Oct 18 20:51:14 ERROR agent.pluginloader plugins/loader.go:156 failed to load plugin plugin=github.com/rancher/opni/plugins/logging err=Unrecognized remote plugin message: 2023 Oct 18 20:51:14 DEBUG global/internal_logging.go:53 Tracer created logger=tracing name=go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc version=0.42.0 schemaURL=""
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /var/lib/opni-agent/plugins/plugin_logging
  Mode: -rwxr-xr-x
  Owner: 0 [root] (current: 0 [root])
  Group: 0 [root] (current: 0 [root])
  ELF architecture: EM_X86_64 (current architecture: amd64)

kralicky · 2023-10-19T00:07:09Z

@jan-law that message is generated by the go-plugin library when writing to a logger that isn't set up to forward its log messages to the host process.

jan-law · 2023-10-20T19:38:39Z

that message is generated by the go-plugin library when writing to a logger that isn't set up to forward its log messages to the host process.

Update - it seems that go-plugin expects logs to be sent to stderr instead of stdout. This issue affects the slog branch too. Making a fix for it now.

jan-law · 2023-10-23T19:46:07Z

pkg/logger/logger.go

 	DefaultTimeFormat = "2006 Jan 02 15:04:05"
 	errKey            = "err"
 )

 var logSampler = &sampler{}

+func init() {
+	logFs = afero.NewMemMapFs()


The plugin logs do not appear in the debug agent-logs output. Do the plugins share the same logFs here if they are different processes?

No, they will be separate. The plugins should be sending their logs to the host process though, maybe that needs some refactoring.

Although I wasn't able to stream the logs over SyncStderr, the Logger in the ClientConfig can forward the log messages to a writer, where I can log them with a logger on the host side.

Logger: hclog.New(&hclog.LoggerOptions{ Level: hclog.Debug, Output: logger.PluginFileWriter, }),

Go-plugin can't directly forward protobuf message bytes unfortunately, as it's hardcoded to cast all non-hclog output to a string: https://github.com/hashicorp/go-plugin/blob/017b758bf4d495212a55db3de61b2d95ab104e53/client.go#L1155

The forwarded logs look like this. What do you think of the formatting?

Log streaming is working with the forked go-plugin patch in my k3s cluster. I replaced the hclog forwarding logger with io.Copy(c.config.Stderr, r). I'm working on fixing the test env output because the plugins are logging the gprc bytes to stderr

jan-law · 2023-11-06T16:36:18Z

Depends on alexandreLamarre/go-plugin#2

jan-law · 2023-11-09T21:43:31Z

Setting this PR back to draft mode while I work on the plugin loggers for the standalone test env

pkg/opni/commands/debug.go

silence non-opni logs share plugin file lock with agent logger

fix plugin log interleaving prevent duplicate file writing in subprocesses create plugin logger without storing writer in context

pass mode to real plugin loader

pass agent id to real cluster plugin loggers

fixup error handling fix lint fixup after rebase

run linter remove unused file desciptors

jan-law commented Oct 16, 2023

View reviewed changes

pkg/logger/logger.go Outdated Show resolved Hide resolved

plugins/metrics/pkg/agent/drivers/opni_manager_otel/otel_driver.go Outdated Show resolved Hide resolved

jan-law marked this pull request as ready for review October 16, 2023 20:33

jan-law requested a review from kralicky October 16, 2023 20:33

jan-law force-pushed the agent-logs branch 6 times, most recently from 82e1abf to b5c314a Compare October 18, 2023 20:00

jan-law force-pushed the agent-logs branch 2 times, most recently from 7079f5d to eca919f Compare October 23, 2023 19:41

jan-law commented Oct 23, 2023

View reviewed changes

jan-law force-pushed the agent-logs branch 3 times, most recently from 7b9df91 to 61e4eb7 Compare November 2, 2023 16:26

jan-law mentioned this pull request Nov 6, 2023

forward plugin stderr to host without hclog alexandreLamarre/go-plugin#2

Merged

jan-law force-pushed the agent-logs branch 2 times, most recently from 22a45f0 to 79f58fb Compare November 6, 2023 18:13

jan-law marked this pull request as draft November 9, 2023 21:43

kralicky reviewed Nov 10, 2023

View reviewed changes

pkg/opni/commands/debug.go Show resolved Hide resolved

pkg/opni/commands/debug.go Show resolved Hide resolved

jan-law force-pushed the agent-logs branch 2 times, most recently from 60922f0 to bf3716e Compare November 14, 2023 19:18

jan-law force-pushed the agent-logs branch 3 times, most recently from b2397dd to 9b25384 Compare November 27, 2023 22:07

Janelle Law added 26 commits November 27, 2023 22:17

add proto handler tests

c5ad6b4

follow logs with nested streaming

7913bef

configure plugin loggers with agent id

cffdfaa

ensure atomic writes

e116eca

attempt to fix data race

be5de83

fix plugin logger forwarding

0f25210

forward plugin logs without hclog

5cf5963

silence non-opni logs share plugin file lock with agent logger

fix lock synchronization

82bf9c0

add command flag completion

f868eab

remove plugin loggers outside plugins

16a3628

fix plugin name colors

dbcf761

setup context plugin logger

f68af02

move aiops plugin lg to ctx

53bf78e

move alerting plugin lg to ctx

213d660

move logging plugin lg to ctx

702efce

move metrics plugin lg to ctx

e50be00

move slo plugin lg to ctx

61ab617

move topology plugin lg to ctx

bea0d56

move remaining plugin lg to ctx

9822a82

init plugin loggers with module name instead of agent id

5743999

fix plugin log interleaving prevent duplicate file writing in subprocesses create plugin logger without storing writer in context

store mode in plugin context

31256e2

pass mode to real plugin loader

separate test plugin logs by agent id

5ab672e

pass agent id to real cluster plugin loggers

group related functions together

def5fd0

fixup error handling fix lint fixup after rebase

set otel tracing logger to error level

e2e7aa0

share lock among log file writers

a389903

run linter remove unused file desciptors

attempt to fix web test data race

9e6a6a5

jan-law force-pushed the agent-logs branch from 9b25384 to 9e6a6a5 Compare November 28, 2023 19:53

Janelle Law added 3 commits November 29, 2023 16:03

fixup ctx data races

84223d0

init global gateway plugin loggers without streaming

8a04996

close log streaming file on agent restart

a1b1fb5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream agent logs remotely #1774

Stream agent logs remotely #1774

jan-law commented Oct 16, 2023 •

edited

Loading

jan-law left a comment

jan-law commented Oct 18, 2023

kralicky commented Oct 19, 2023

jan-law commented Oct 20, 2023

jan-law Oct 23, 2023

kralicky Oct 23, 2023

jan-law Nov 2, 2023

jan-law Nov 2, 2023

jan-law Nov 6, 2023

jan-law commented Nov 6, 2023

jan-law commented Nov 9, 2023

Stream agent logs remotely #1774

Are you sure you want to change the base?

Stream agent logs remotely #1774

Conversation

jan-law commented Oct 16, 2023 • edited Loading

jan-law left a comment

Choose a reason for hiding this comment

jan-law commented Oct 18, 2023

kralicky commented Oct 19, 2023

jan-law commented Oct 20, 2023

jan-law Oct 23, 2023

Choose a reason for hiding this comment

kralicky Oct 23, 2023

Choose a reason for hiding this comment

jan-law Nov 2, 2023

Choose a reason for hiding this comment

jan-law Nov 2, 2023

Choose a reason for hiding this comment

jan-law Nov 6, 2023

Choose a reason for hiding this comment

jan-law commented Nov 6, 2023

jan-law commented Nov 9, 2023

jan-law commented Oct 16, 2023 •

edited

Loading