kubernetes: audit logging guide

[philips]

Kubernetes keeps an event log that can be viewed using kubectl events. But, many people would also like to have these events go to their centralized logging system as well. This guide will show you how to get these audit logs of kubernetes events in both a best effort and fully consistent manner and inject them using log aggregation systems.

WIP docs: https://docs.google.com/document/d/11_DIkLMmwkGazh02x_2N6eoUdJLY91-cnylog9D-71A/edit#

[philips]

Assigning this to @kbrwn to get the first draft done.

[philips]

Ping @kbrwn or @joshix ?

[kbrwn]

@philips this is currently blocked by the hyperkube ignoring flag arguments. colin has a PR to fix this kubernetes/kubernetes#25512

[philips]

@kbrwn #25512 is now merged!

[philips]

@colhom do you think you could put in a cherry pick backport to v1.2 for #25512?

[colhom]

@philips that would be good to do. we have some blockers in coreos-kubernetes on this one as well.

[colhom]

v1.2 cherry-pick pr kubernetes/kubernetes#26754 is lgtm. Should have #25512 in v1.2.5!

\cc @kbrwn @philips

[chancez]

In the heapster project there's an eventer binary which can now be used which effectively does the same as kubectl get events but it supports sending to stdout (basically exactly what kubectl does) and sending to elasticsearch/google cloud storage.

I'm going to look into enabling eventer in coreos-kubernetes and also start investigating more advanced audit logging.

[pop]

@philips @colegleason @joshix @chancez @kbrwn

Any update on the state of this?

[chancez]

Today audit logging is basically limited to the API Server's logs, and its fairly low detail. These can be collected with normal centralized logging tooling, assuming the API server is running as a pod.

There are also events, as mentioned above, which can be sent to your logging destination of choice using the eventer container, which exists in the heapster image. Just note that events are not at all "audit" material, mostly "pod deleted, pod created, liveness probe failed, new node showed up" types of information. It never includes the "who", so it's not really for auditing.

See https://github.com/kubernetes/heapster/tree/master/events for details on eventer, and https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md for the sink configuration.