FEATURE:: Adding -oaddFieldsPosition flag to capture fields position

[EmileDqy]

Adding -oaddFieldsPosition flag to liblognorm

The -oaddFieldsPosition flag is made to enable the tracking of the position of each field's value in the input log.

The output is a new field under "metadata" which has the same structure as the base fields but the values are replaced with an array of two indices.

Example

$ cat sample.log | head -1 | lognormalizer -r rulebase.rb -oaddFieldsPosition | jq

{
  "parent_data": {
    "logver": "60",
    "timestamp": "1572361274",
    "tz": "UTC+1",
    "srcserver": "1"
  },
  "priority": "189",
  "metadata": {
    "fields_position": {
      "priority": [
        1,
        4
      ],
      "parent_data": {
        "logver": [
          12,
          14
        ],
        "timestamp": [
          25,
          35
        ],
        "tz": [
          40,
          45
        ],
        "srcserver": [
          943,
          944
        ],
        "__fieldposition": [
          5,
          944
        ]
      }
    }
  }
}

A __fieldposition field is also added under each new nested level of json so as to track the whole position of the group.
For example here, the field __fieldposition was added under metadata.fields_position.parent_data and says that the parent_data field started at index 5 and ended at index 944.

Limitations

This feature does not work for the fields created by the following types:

• json: it uses libfastjson under the hood which breaks the data lineage.
• checkpoint-lea: Not implemented
• cisco-interface-spec: Not implemented
• iptables: Not implemented
• cisco-interface-spec: Not implemented
• alternative: Not implemented
• repeat: Not implemented
• cee-syslog: Not implemented

[frikilax]

i-1 ?
isn't it i+1 ?
And I don't think all fields are 1 char long

[EmileDqy]

I tried with i-1. It works great.

[frikilax]

why not?

Suggested change

	if (fieldName[0] == '.' && fieldName[1] == '\0') {
	if (fieldName[0] == '.' && fieldName[1] == NULL) {

[frikilax]

wrong indentation