Offer the possibility to output logs in json format

[fmassot]

For devops purpose, it could be nice to configure the tracing_subcriber to output logs in json.

This is possible with tokio-tracing but this needs to be tested to see how json logs are structured.

[fmassot]

Json output with an indexer below.

It's possible to flatten the fields at the root level but I would prefer to keep things separated.

{"timestamp":"2022-11-09T13:42:45.842Z","level":"INFO","fields":{"version":"0.3.1-nightly","commit":"31b2c0e17"},"target":"quickwit"}
{"timestamp":"2022-11-09T13:42:45.851Z","level":"INFO","fields":{"message":"Using listen address as advertise address.","advertise_address":"127.0.0.1"},"target":"quickwit_config::config"}
{"timestamp":"2022-11-09T13:42:45.855Z","level":"WARN","fields":{"message":"Cluster ID is not set, falling back to default value.","cluster_id":"quickwit-default-cluster"},"target":"quickwit_config::config"}
{"timestamp":"2022-11-09T13:42:45.855Z","level":"WARN","fields":{"message":"Peer seed list is empty."},"target":"quickwit_config::config"}
{"timestamp":"2022-11-09T13:42:45.855Z","level":"INFO","fields":{"message":"Loaded Quickwit config.","config_uri":"file:///Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/config/quickwit.yaml","config":"QuickwitConfig { version: 0, cluster_id: \"quickwit-default-cluster\", node_id: \"node-falling-BhQb\", enabled_services: {Searcher, Metastore, Indexer, Janitor}, rest_listen_addr: 127.0.0.1:7280, gossip_listen_addr: 127.0.0.1:7280, grpc_listen_addr: 127.0.0.1:7281, gossip_advertise_addr: 127.0.0.1:7280, grpc_advertise_addr: 127.0.0.1:7281, peer_seeds: [], data_dir_path: \"/Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/quickwit/qwdata\", metastore_uri: Uri { uri: \"file:///Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/quickwit/qwdata/indexes#polling_interval=30s\" }, default_index_root_uri: Uri { uri: \"file:///Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/quickwit/qwdata/indexes\" }, indexer_config: IndexerConfig { enable_opentelemetry_otlp_service: false, split_store_max_num_bytes: Byte(100000000000), split_store_max_num_splits: 1000, max_concurrent_split_uploads: 12 }, searcher_config: SearcherConfig { enable_jaeger_service: false, fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_searches: 100, max_num_concurrent_split_streams: 100 } }"},"target":"quickwit_cli"}
{"timestamp":"2022-11-09T13:42:45.855Z","level":"INFO","fields":{"message":"Setting services from override.","services":"indexer, metastore"},"target":"quickwit_cli::service"}
{"timestamp":"2022-11-09T13:42:45.856Z","level":"INFO","fields":{"message":"Joining cluster.","cluster_id":"quickwit-default-cluster","node_id":"node-falling-BhQb","grpc_public_addr":"127.0.0.1:7281","available_services":"{Indexer, Metastore}","gossip_listen_addr":"127.0.0.1:7280","gossip_public_addr":"127.0.0.1:7280","peer_seed_addrs":""},"target":"quickwit_cluster::cluster"}
{"timestamp":"2022-11-09T13:42:45.887Z","level":"INFO","fields":{"message":"Starting indexer service."},"target":"quickwit_indexing"}
{"timestamp":"2022-11-09T13:42:45.898Z","level":"INFO","fields":{"message":"Spawning indexing pipeline(s).","index_ids":"test, wikipedia"},"target":"quickwit_indexing"}
{"timestamp":"2022-11-09T13:42:45.901Z","level":"INFO","fields":{"message":"ingest-api-garbage-collect-operation"},"target":"quickwit_indexing::actors::ingest_api_garbage_collector"}
{"timestamp":"2022-11-09T13:42:45.901Z","level":"INFO","fields":{"message":"Spawning indexing pipeline.","index_id":"wikipedia","source_id":"_ingest-api","pipeline_ord":"0","root_dir":"/Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/quickwit/qwdata/indexing/wikipedia/_ingest-api"},"target":"quickwit_indexing::actors::indexing_pipeline","span":{"gen":0,"index":"wikipedia","name":"spawn_pipeline"},"spans":[{"gen":0,"index":"wikipedia","name":"spawn_pipeline"}]}
{"timestamp":"2022-11-09T13:42:45.901Z","level":"INFO","fields":{"message":"Spawning merge pipeline.","index_id":"wikipedia","source_id":"_ingest-api","pipeline_ord":"0","root_dir":"/Users/francois.massot@contentsquare-ext.com/Documents/quickwit/repos/quickwit/quickwit/qwdata/indexing/wikipedia/_ingest-api","merge_policy":"StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12 }, split_num_docs_target: 10000000 }"},"target":"quickwit_indexing::actors::merge_pipeline","span":{"gen":0,"index":"wikipedia","name":""},"spans":[{"gen":0,"index":"wikipedia","name":""}]}
{"timestamp":"2022-11-09T13:42:45.903Z","level":"INFO","fields":{"message":"Starting gRPC server.","enabled_grpc_services":"{\"metastore\"}","grpc_listen_addr":"127.0.0.1:7281"},"target":"quickwit_serve::grpc"}
{"timestamp":"2022-11-09T13:42:45.903Z","level":"INFO","fields":{"message":"Starting REST server.","rest_listen_addr":"127.0.0.1:7280"},"target":"quickwit_serve::rest"}
{"timestamp":"2022-11-09T13:42:45.903Z","level":"INFO","fields":{"message":"Searcher ready to accept requests at http://127.0.0.1:7280/"},"target":"quickwit_serve::rest"}
{"timestamp":"2022-11-09T13:43:23.340Z","level":"INFO","fields":{"message":"new-split","split_id":"01GHEA344VD4712D3CZZA35TCB","partition_id":0},"target":"quickwit_indexing::actors::indexer"}
{"timestamp":"2022-11-09T13:44:23.039Z","level":"INFO","fields":{"message":"send-to-index-serializer","commit_trigger":"Timeout","split_ids":"01GHEA344VD4712D3CZZA35TCB","num_docs":3},"target":"quickwit_indexing::actors::indexer"}
{"timestamp":"2022-11-09T13:44:23.229Z","level":"INFO","fields":{"message":"start-packaging-splits","split_ids":"[\"01GHEA344VD4712D3CZZA35TCB\"]"},"target":"quickwit_indexing::actors::packager","span":{"name":"packager"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"name":"packager"}]}
{"timestamp":"2022-11-09T13:44:23.232Z","level":"INFO","fields":{"message":"create-packaged-split","split_id":"01GHEA344VD4712D3CZZA35TCB"},"target":"quickwit_indexing::actors::packager","span":{"name":"packager"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"name":"packager"}]}
{"timestamp":"2022-11-09T13:44:23.283Z","level":"INFO","fields":{"message":"start-stage-and-store-splits","split_ids":"[\"01GHEA344VD4712D3CZZA35TCB\"]"},"target":"quickwit_indexing::actors::uploader","span":{"name":"uploader"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"name":"uploader"}]}
{"timestamp":"2022-11-09T13:44:23.297Z","level":"INFO","fields":{"message":"store-split-remote-success","split_size_in_megabytes":"0.006784","num_docs":"3","elapsed_secs":"0.011484708","throughput_mb_s":"0.59069854","is_mature":false},"target":"quickwit_indexing::split_store::indexing_split_store","span":{"name":"store_split"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"name":"uploader"},{"split":"01GHEA344VD4712D3CZZA35TCB","name":"stage_and_upload"},{"name":"store_split"}]}
{"timestamp":"2022-11-09T13:44:23.297Z","level":"INFO","fields":{"message":"store-in-cache"},"target":"quickwit_indexing::split_store::indexing_split_store","span":{"name":"store_split"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"name":"uploader"},{"split":"01GHEA344VD4712D3CZZA35TCB","name":"stage_and_upload"},{"name":"store_split"}]}
{"timestamp":"2022-11-09T13:44:23.301Z","level":"INFO","fields":{"delta":"∆(wikipedia:(00000000000000000005..00000000000000000008])","checkpoint":"Ckpt(wikipedia:00000000000000000005)"},"target":"quickwit_metastore::checkpoint","span":{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"}]}
{"timestamp":"2022-11-09T13:44:23.310Z","level":"INFO","fields":{"message":"publish-new-splits","new_splits":"[\"01GHEA344VD4712D3CZZA35TCB\"]","checkpoint_delta":"Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008]))"},"target":"quickwit_indexing::actors::publisher","span":{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"}]}
{"timestamp":"2022-11-09T13:44:23.311Z","level":"INFO","fields":{"delta":"∆(wikipedia:(00000000000000000005..00000000000000000008])","checkpoint":"Ckpt()"},"target":"quickwit_metastore::checkpoint","span":{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"},"spans":[{"index_id":"wikipedia","pipeline_ord":"0","source_id":"_ingest-api","name":"index_batch"},{"split_update":"SplitsUpdate { index_id: \"wikipedia\", new_splits: \"01GHEA344VD4712D3CZZA35TCB\", checkpoint_delta: Some(_ingest-api:∆(wikipedia:(00000000000000000005..00000000000000000008])) }","name":"publisher"}]}

[fmassot]

It could be quite useful by coupling that with vector that will push logs into quickwit.