|
1 |
| -# lol, logging. |
| 1 | +# logstash cookbook! |
2 | 2 |
|
3 |
| -## Visit the site: |
| 3 | +A logstash community-driven site for documentation, shared experiences, etc. |
4 | 4 |
|
5 |
| -[lol, logging](http://jordansissel.github.com/lol-logging) |
| 5 | +[Visit the site](http://cookbook.logstash.net/) |
6 | 6 |
|
7 |
| -## Background |
8 |
| - |
9 |
| -We all do it poorly, best to laugh about it, and perhaps find a better future. |
10 |
| - |
11 |
| -This project aims to document logging stuff. I want to encourage the good, and |
12 |
| -derail the bad. I want to provide solid data that helps you make the right |
13 |
| -decisions about how/why/when you are logging and consuming logs. |
14 |
| - |
15 |
| -I'm going to include bad things I do (and wish I did better) as well in this |
16 |
| -documentary. |
17 |
| - |
18 |
| -Here are some of my ideas: |
19 |
| - |
20 |
| -## cultural battles around logging protocols |
21 |
| - |
22 |
| -Sigh: |
23 |
| - |
24 |
| -* "I need human readable raw log data" |
25 |
| -* "We just dump random stuff over syslog!" |
26 |
| -* "We log key=value!" |
27 |
| - |
28 |
| -## "Standards" and other destructive forces |
29 |
| - |
30 |
| -Why each of the following are complete bad, and why, and perhaps what we |
31 |
| -can try doing, in general, to fix things. |
32 |
| - |
33 |
| -* timestamps. ugh. |
34 |
| -* RFC3164, 5424, 5425. Why each are bad. |
35 |
| -* ArcSight's bad ["Common Event Format"](http://www.arcsight.com/collateral/CEFstandards.pdf) |
36 |
| -* Splunk's bad ["Common Information Model"](http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/UnderstandandusetheCommonInformationModel) |
37 |
| -* LogStash's bad JSON schema (this would be a link if this was actually documented ☹) |
38 |
| -* Graylog2's bad [JSON schema](https://github.com/Graylog2/graylog2-docs/wiki/GELF) |
39 |
| -* CEE's [bad](http://cee.mitre.org/docs/profiles.html), |
40 |
| - [well](http://cee.mitre.org/docs/cls.html), |
41 |
| - [everything](http://cee.mitre.org/docs/clt.html). 4 serialization formats, 4 |
42 |
| - conformance levels, 2+ transport mechanisms == 30+ combinations of bad. |
43 |
| - |
44 |
| -## explorations of bad logging. |
45 |
| - |
46 |
| -* mysql (binary log, slow query log, debug log; all are completely different formats) |
47 |
| -* more? |
48 |
| - |
49 |
| -## explorations of good logging. |
50 |
| - |
51 |
| -Please tell me someone has an example of good logging in an application. We |
52 |
| -can't be all totally screwing this up across the world. |
53 |
| - |
54 |
| -Anybody? |
55 |
| - |
56 |
| -## logging libraries |
57 |
| - |
58 |
| -Sigh: |
59 |
| - |
60 |
| -* printf-style loggers like: ruby logger, python logging, etc. |
61 |
| - |
62 |
| -Hurray: |
63 |
| - |
64 |
| -* log4j MDC/NDC, ruby-cabin, etc |
65 |
| - |
66 |
| -## types of logs |
67 |
| - |
68 |
| -* tracing (for the purposes of debugging) |
69 |
| -* accounting (for numerical applications like billing, metrics, etc) |
70 |
| -* transaction log (for the purposes of rollback and replay) |
71 |
| - |
72 |
| -## typical problems |
73 |
| - |
74 |
| -* Fat logs: "I have 300 gigs of logs, how can I make this useful?" |
75 |
| -* Fast logs: "I have 50,000 events logged per second, how can I make this useful?" |
76 |
| -* Lawn mowing: "I don't really use our logs much, but we have to use complex |
77 |
| - logrotate rules or otherwise we run out of disk and it takes down production" |
78 |
| -* Syntax vs domain: to answer domain questions (how many customers |
79 |
| - signed up?), you require syntax knowledge (how to parse apache logs) |
80 |
| -* Wrong audience: Giving the user stack traces instead of English. |
| 7 | +♥ Fork and contribute ♥ |
0 commit comments