|
1 | | -# lol, logging. |
| 1 | +# logstash cookbook! |
2 | 2 |
|
3 | | -## Visit the site: |
| 3 | +A logstash community-driven site for documentation, shared experiences, etc. |
4 | 4 |
|
5 | | -[lol, logging](http://jordansissel.github.com/lol-logging) |
| 5 | +[Visit the site](http://cookbook.logstash.net/) |
6 | 6 |
|
7 | | -## Background |
8 | | - |
9 | | -We all do it poorly, best to laugh about it, and perhaps find a better future. |
10 | | - |
11 | | -This project aims to document logging stuff. I want to encourage the good, and |
12 | | -derail the bad. I want to provide solid data that helps you make the right |
13 | | -decisions about how/why/when you are logging and consuming logs. |
14 | | - |
15 | | -I'm going to include bad things I do (and wish I did better) as well in this |
16 | | -documentary. |
17 | | - |
18 | | -Here are some of my ideas: |
19 | | - |
20 | | -## cultural battles around logging protocols |
21 | | - |
22 | | -Sigh: |
23 | | - |
24 | | -* "I need human readable raw log data" |
25 | | -* "We just dump random stuff over syslog!" |
26 | | -* "We log key=value!" |
27 | | - |
28 | | -## "Standards" and other destructive forces |
29 | | - |
30 | | -Why each of the following are complete bad, and why, and perhaps what we |
31 | | -can try doing, in general, to fix things. |
32 | | - |
33 | | -* timestamps. ugh. |
34 | | -* RFC3164, 5424, 5425. Why each are bad. |
35 | | -* ArcSight's bad ["Common Event Format"](http://www.arcsight.com/collateral/CEFstandards.pdf) |
36 | | -* Splunk's bad ["Common Information Model"](http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/UnderstandandusetheCommonInformationModel) |
37 | | -* LogStash's bad JSON schema (this would be a link if this was actually documented ☹) |
38 | | -* Graylog2's bad [JSON schema](https://github.com/Graylog2/graylog2-docs/wiki/GELF) |
39 | | -* CEE's [bad](http://cee.mitre.org/docs/profiles.html), |
40 | | - [well](http://cee.mitre.org/docs/cls.html), |
41 | | - [everything](http://cee.mitre.org/docs/clt.html). 4 serialization formats, 4 |
42 | | - conformance levels, 2+ transport mechanisms == 30+ combinations of bad. |
43 | | - |
44 | | -## explorations of bad logging. |
45 | | - |
46 | | -* mysql (binary log, slow query log, debug log; all are completely different formats) |
47 | | -* more? |
48 | | - |
49 | | -## explorations of good logging. |
50 | | - |
51 | | -Please tell me someone has an example of good logging in an application. We |
52 | | -can't be all totally screwing this up across the world. |
53 | | - |
54 | | -Anybody? |
55 | | - |
56 | | -## logging libraries |
57 | | - |
58 | | -Sigh: |
59 | | - |
60 | | -* printf-style loggers like: ruby logger, python logging, etc. |
61 | | - |
62 | | -Hurray: |
63 | | - |
64 | | -* log4j MDC/NDC, ruby-cabin, etc |
65 | | - |
66 | | -## types of logs |
67 | | - |
68 | | -* tracing (for the purposes of debugging) |
69 | | -* accounting (for numerical applications like billing, metrics, etc) |
70 | | -* transaction log (for the purposes of rollback and replay) |
71 | | - |
72 | | -## typical problems |
73 | | - |
74 | | -* Fat logs: "I have 300 gigs of logs, how can I make this useful?" |
75 | | -* Fast logs: "I have 50,000 events logged per second, how can I make this useful?" |
76 | | -* Lawn mowing: "I don't really use our logs much, but we have to use complex |
77 | | - logrotate rules or otherwise we run out of disk and it takes down production" |
78 | | -* Syntax vs domain: to answer domain questions (how many customers |
79 | | - signed up?), you require syntax knowledge (how to parse apache logs) |
80 | | -* Wrong audience: Giving the user stack traces instead of English. |
| 7 | +♥ Fork and contribute ♥ |
0 commit comments