/mylogs/**/*.log;/var/log/**/*.log. Mount your server log files into the container using a Docker volume e.g.
-v /var/log:/mylogs+* **LOGAGENT_ARGS**: Additional [command line arguments for Logagent](https://sematext.com/docs/logagent/cli-parameters/)
LOGAGENT_ARGS="-n httpd"to specify a log source name or
LOGAGENT_ARGS="-u 514"to act as syslog server. Please refer to Logagent command line arguments in the [Logagent Documentation](https://sematext.com/docs/logagent/cli-parameters/) + +### Docker Run Example + +The most basic start method is using docker run command: + +``` +docker pull sematext/logagent +docker run -d --name logagent \ +-e LOGS_TOKEN=YOUR_LOGS_TOKEN \ +-e LOGS_RECEIVER_URL="https://logsene-receiver.sematext.com" +-v /var/run/docker.sock:/var/run/docker.sock sematext/logagent +``` + +# Documentation + +For further information please read the [setup manual](https://sematext.com/docs/logagent/installation-docker/). +You find in the manual all configuration options and the setup instructions for Kubernetes, OpenShift, Mesos, Docker Enterprise, etc. diff --git a/node_modules/@sematext/logagent/dockerhub/docs/logo.png b/node_modules/@sematext/logagent/dockerhub/docs/logo.png new file mode 100644 index 00000000..4e8fdf33 Binary files /dev/null and b/node_modules/@sematext/logagent/dockerhub/docs/logo.png differ diff --git a/node_modules/@sematext/logagent/dockerhub/help.md b/node_modules/@sematext/logagent/dockerhub/help.md new file mode 100644 index 00000000..7f9cefb2 --- /dev/null +++ b/node_modules/@sematext/logagent/dockerhub/help.md @@ -0,0 +1,505 @@ +## Installation for Docker + +Logagent is a general purpose log shipper. The Logagent Docker image is pre-configured for the log collection on container platforms. It runs as a tiny container on every Docker host and collects logs for all cluster nodes and their containers. +All container logs are enriched with Kubernetes and Docker EE/Swarm metadata. + +See: [sematext/logagent](https://hub.docker.com/r/sematext/logagent/) on Docker Hub. + +## Getting started + +To run Logagent you will need a Logs App Token. +If you don't have Logs Apps yet, you can [create Apps now](https://apps.sematext.com/ui/integrations). + +The [Logagent](https://sematext.com/logagent) docker container can be configured through the following environment variables: + +* **REGION**: Sematext Cloud region **US** or **EU** (default: US). The receiver URL will be set to EU/US default values. When using REGION, you don't need to set `LOGS_RECEIVER_URL` (see below). +* **LOGS_RECEIVER_URL**: The URL of your Elasticsearch Endpoint _(defaults to Sematext Cloud US `https://logsene-receiver.sematext.com`)_. + + * For Sematext Europe use `https://logsene-receiver.eu.sematext.com`. + * For Elasticsearch `https://elasticserch-server-name:9200`. + +* **LOGS_TOKEN**: The index where the agent should log to _(for [Sematext Cloud](https://sematext.com/cloud) users the logs token)_ +* **LOGAGENT_ARGS**: Additional [command line arguments for Logagent](https://sematext.com/docs/logagent/cli-parameters/)
LOGAGENT_ARGS="-n httpd"to specify a log source name or
LOGAGENT_ARGS="-u 514"to act as syslog server. Please refer to Logagent command line argumetns in the [Logagent Documentation](https://sematext.com/docs/logagent/cli-parameters/) +* **LOG_GLOB**: Semicolon-separated list of file globs
/mylogs/**/*.log;/var/log/**/*.logMount your server log files into the container using a Docker volume e.g.
-v /var/log:/mylogs. +* **-v /var/run/docker.sock:/var/run/docker.sock** - Collect container logs by mounting the docker socket (mandatory) + + +### Docker Run + +The most basic start method is using docker run command: + +``` +docker pull sematext/logagent +docker run -d --restart=always --name logagent \ +-e LOGS_TOKEN=YOUR_LOGS_TOKEN \ +-e LOGS_RECEIVER_URL="https://logsene-receiver.sematext.com" \ +-v /var/run/docker.sock:/var/run/docker.sock sematext/logagent +``` + + +### Docker Compose + +To use [Docker Compose](https://docs.docker.com/compose/) create docker-compose.yml as follows and insert real tokens: + +``` + +# docker-compose.yml +logagent: + image: 'sematext/logagent:latest' + environment: + - LOGS_TOKEN=YOUR_LOGS_TOKEN + - LOGS_RECEIVER_URL="https://logsene-receiver.sematext.com" + cap_add: + - SYS_ADMIN + restart: always + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + +``` + +Then start Logagent with the docker-compose file: + +``` +docker-compose up -d +``` + +### Docker Swarm and Docker Enterprise + +Connect your Docker client to Swarm or UCP remote API endpoint and +deploy Logagent with following docker command with your Logs Tokens: + +``` +docker service create --restart=always -mode global -name logagent \ +-mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ +-e LOGS_TOKEN="REPLACE THIS WITH YOUR LOGS TOKEN" \ +-e LOGS_RECEIVER_URL="https://logsene-receiver.sematext.com" \ +sematext/logagent +``` + +### Kubernetes and OpenShift + +Run Logagent as [Kubernetes DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset). + +First, create [logagent-daemonset.yml](https://github.com/sematext/logagent-js/blob/master/kubernetes/logagent-daemonset.yml) + +``` +curl -o logagent-daemonset.yml https://raw.githubusercontent.com/sematext/logagent-js/master/kubernetes/logagent-daemonset.yml +``` + + +Then run the DaemonSet: + +``` +kubectl create -f logagent-daemonset.yml +``` + +On Red Hat OpenShift use the "oc" command instead of kubectl. + +``` +oc apply -f logagent-daemonset.yml +``` + +### Kubernetes with containerd and IBM Cloud + +Kubernetes can use cointainerd as container engine. In this case Logagent can't use the Docker remote API to retrieve logs and metadata. +Instead logs are collected from containerd log files and requires access to the relevant directories. +The logagent input-filter for containerd supports: + +* Tailing log files from `/var/log/containers/`, `/var/log/pods` and `/var/data/kubeletlogs` +* Enrichment of logs with podName, namespace, containerName, containerId +* Parsing containerd log headers (timestamp, stream, flags) +* Parsing message content with logagent parser library + +Run Logagent as [Kubernetes DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset). + +First, create [ibm-cloud-logagent-ds.yml](https://github.com/sematext/logagent-js/blob/master/kubernetes/ibm-cloud-logagent-ds.yml) + +``` +curl -o ibm-cloud-logagent-ds.yml https://raw.githubusercontent.com/sematext/logagent-js/master/kubernetes/ibm-cloud-logagent-ds.yml +``` + +Set your Logs Token in the spec.env section in the `ibm-cloud-logagent-ds.yml` file. + +Then run the DaemonSet: + +``` +kubectl create -f ibm-cloud-logagent-ds.yml +``` + + +### Mesos / Marathon + +The following configuration will activate Logagent on every node in the Mesos cluster. Please note that you have to specify the number of Mesos nodes (instances) and Logs Token. An example call to the Marathon API: + +``` +curl -XPOST -H "Content-type: application/json" http://your_marathon_server:8080/v2/apps -d ' +{ + "container": { + "type": "DOCKER", + "docker": { + "image": "sematext/logagent" + }, + "volumes": [ + { + "containerPath": "/var/run/docker.sock", + "hostPath": "/var/run/docker.sock", + "mode": "RW" + } + ], + "network": "BRIDGE" + }, + "env": { + "LOGS_TOKEN": "YOUR_LOGS_TOKEN", + "LOGS_RECEIVER_URL": "https://logsene-receiver.sematext.com" + + }, + "id": "sematext-logagent", + "instances": 1, + "cpus": 0.1, + "mem": 100, + "constraints": [ + [ + "hostname", + "UNIQUE" + ] + ] +} +``` + + +## Configuration Parameters + +### Mandatory Parameters + +
| Parameter / Environment variable+ | Description+ | 
|---|---|
| LOGS_TOKEN+ | Logs Token enables logging to Sematext Cloud, see logging specific parameters for filter options and Log Routing section to route logs from different containers to separate Logs Apps (indices)+ | 
| -v /var/run/docker.sock+ | Path to the docker socket+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| REGION+ | Sematext Cloud region US or EU (default: US). The receiver URL will be set to EU/US default values. When using REGION, you don't need to set LOGS_RECEIVER_URL (see below).+ | 
| LOG_GLOB+ | Semicolon-separated list of file globs (e.g. /var/log//*.log;/mylogs//*.log) to collect log files from the host, assuming the log files are mounted to /mylogs using Docker -v /var/logs:/mylogs+ | 
| LOGAGENT_ARGS+ | Additional command line arguments for Logagent (e.g. LOGAGENT_ARGS="-n httpd" to specify a log source name or LOGAGENT_ARGS="-u 514" to act as syslog server)+ | 
| --privileged+ | The parameter might be helpful when Logagent could not start because of limited permission to connect and write to the Docker socket /var/run/docker.sock. The privileged mode is a potential security risk, we recommend to enable the appropriate security. Please read about Docker security: https://docs.docker.com/engine/security/security/+ | 
| HTTPS_PROXY+ | URL for a proxy server (behind firewalls)+ | 
| LOGS_RECEIVER_URL+ | URL for bulk inserts into Sematext Cloud. Required for Sematext Enterprise (local IP:PORT) or Sematext Cloud Europe: https://logsene-receiver.eu.sematext.com+ | 
| JOURNALD_UPLOAD_PORT+ | Port number for the collection of journald logs, forwarded by systemd-journal-upload.service. Equals to Logagent argument --journald PORT.+ | 
| SYSTEMD_UNIT_FILTER+ | A regular expression to filter journald logs by systemd unit name, e.g. "ssh.service|docker.service". The default value is ".*".+ | 
| CONFIG_FILE+ | Path to the configuration file, containing environment variables +key=value. Default value:/run/secrets/logagent. Create a secret withdocker secret create logagent ./logagent.cfg. Start Logagent with `docker service create --mode global --secret logagent --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock sematext/logagent | 
| --privileged+ | The parameter might be helpful when Logagent could not start because of limited permission to connect and write to the Docker socket /var/run/docker.sock. The privileged mode is a potential security risk, we recommend to enable the appropriate security. Please read about Docker security: https://docs.docker.com/engine/security/security/+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| TAGGING_LABELS+ | A list of docker label names or environment variable names to tag container logs. Supporting wildcards. Default value: +com.docker.*,io.kubernetes.*,annotation.io.* | 
| IGNORE_LOGS_PATTERN+ | Filter logs by a JS regular expression. E.g. +IGNORE_LOGS_PATTERN=\/healthcheck|\/ping | 
| LOGSENE_ENABLED_DEFAULT+ | Enables log collection for containers having no explicit label/environment variable LOGSENE_ENABLED set. Default value: true. See section Log Routing.+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| MATCH_BY_NAME+ | Regular expression to white list container names+ | 
| MATCH_BY_IMAGE+ | Regular expression to white list image names+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| SKIP_BY_NAME+ | Regular expression to black list container names+ | 
| SKIP_BY_IMAGE+ | Regular expression to black list image names for logging+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| PATTERNS_URL+ | Load pattern.yml via HTTP e.g. +PATTERNS_URL=https://myserver/patterns.yml+ | 
| LOGAGENT_PATTERNS+ | Pass patterns.yml via env. variable e.g. +LOGAGENT_PATTERNS="$(cat ./patters.yml)" | 
| LOGAGENT_PATTERNS_BASE64+ | Set to "true" if the LOGAGENT_PATTERNS patterns file you are passing in via env. variable is base64 encoded e.g 
+ +LOGAGENT_PATTERNS_BASE64="$(cat ./patterns.yml | base64)". Useful if your patterns file is not getting set properly due to shell interpretation or otherwise. | 
| PATTERN_MATCHING_ENABLED+ | Activate logagent-js parser, default value is true. To disable the log parser set the value to false. This could increase the throughput of log processing for nodes with a very high log volume.+ | 
| -v /patterns.yml:/etc/logagent/patterns.yml+ | Mount a patterns file to customize patterns for log parsing+ | 
| Parameter / Environment variable+ | Description+ | 
|---|---|
| -v /tmp:/log-buffer+ | Directory to store logs, in a case of a network or service outage. Docker Agent deletes these files after successful transmission.+ | 
| GEOIP_ENABLED+ | Enables GeoIP lookups in the log parser, default value: "false"+ | 
| MAXMIND_DB_DIR+ | Directory for the Geo-IP lite database, must end with /. Storing the DB in a volume could save downloads for updates after restarts. Using /tmp/ (ramdisk) could speed up Geo-IP lookups (requires add. ~30 MB main memory).+ | 
| REMOVE_FIELDS+ | Removes fields from parsed/enriched logs. E.g. +REMOVE_FIELDS=password,creditCardNo | 
+IGNORE_LOGS_PATTERNS=\/healthcheck|\/ping ++ +### Container Log Parsing + +In Docker, logs are console output streams from containers. They might be a mix of plain text messages from start scripts and structured logs from applications. The problem is obvious – you can’t just take a stream of log events all mixed up and treat them like a blob. You need to be able to tell which log event belongs to what container, what app, parse it correctly in order to structure it so you can later derive more insight and operational intelligence from logs, etc. + +Logagent analyzes the event format, parses out data, and turns logs into structured JSON. This is important because the value of logs increases when you structure them — you can then slice and dice them and gain a lot more insight about how your containers, servers, and applications operate. + +Traditionally it was necessary to use log shippers like Logstash, Fluentd or Rsyslog to parse log messages. The problem is that such setups are typically deployed in a very static fashion and configured for each input source. That does not work well in the hyper-dynamic world of containers! We have seen people struggling with the Syslog drivers, parsers configurations, log routing, and more! With its integrated automatic format detection, Logagent eliminates this struggle — and the waste of resources — both computing and human time that goes into dealing with such things! This integration has a low footprint, doesn’t need retransmissions of logs to external services, and it detects log types for the most popular applications and generic JSON and line-oriented log formats out of the box! + + + +For example, Logagent can parse logs from official images like: + +- Nginx, Apache, Redis, MongoDB, MySQL +- Elasticsearch, Solr, Kafka, Zookeeper +- Hadoop, HBase, Cassandra +- Any JSON output with special support for Logstash or Bunyan format +- Plain text messages with or without timestamps in various formats +- Various Linux and Mac OSX system logs + +### Adding log parsing patterns + +In addition, you can define your own patterns for any log format you need to be able to parse and structure. There are three options to pass individual log parser patterns: + +- Configuration file in a mounted volume: ```-v PATH_TO_YOUR_FILE:/etc/logagent/patterns.yml``` + - Kubernetes ConfigMap example: [Template for patterns.yml as ConfigMap](https://github.com/sematext/logagent-js/blob/master/kubernetes/configMapExample.yml) +- Content of the configuration file in an environment variable: ```-e LOGAGENT_PATTERNS=”$(cat patterns.yml)”``` +- Set patterns URL as environment variable: ```-e PATTERNS_URL=http://yourserver/patterns.yml``` + +The file format for the patterns.yml file is based on JS-YAML, in short: + +- `–` indicates an array element +- `!js/regexp` – indicates a JavaScript regular expression +- `!!js/function >` – indicates a JavaScript function + + +The file has the following properties: + +- patterns: list of patterns, each pattern starts with “-“ + - match: a list of pattern definition for a specific log source (image/container) + - sourceName: a regular expression matching the name of the log source. The source name is a combination of image name and container name. + - regex: JS regular expression + - fields: field list of extracted match groups from the regex + - type: type used in Sematext Cloud (Elasticsearch Mapping) + - dateFormat: format of the special fields ‘ts’, if the date format matches, a new field @timestamp is generated + - transform: A JavaScript function to manipulate the result of regex and date parsing + +The following example shows pattern definitions for web server logs, which is one of the patterns available by default: + + + +This example shows a few very interesting features: + +- Masking sensitive data with “autohash” property, listing fields to be replaced with a hash code +- Automatic Geo-IP lookups including automatic updates for Maxmind Geo-IP lite database +- Post-processing of parsed logs with JavaScript functions + +The component for detecting and parsing log messages — [logagent-js](http://sematext.com/docs/logagent/parser/) — is open source and contributions for even more log formats are welcome. + + +### Log Routing + +Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN```. +Logagent inspects the containers for this label and ships the logs to the specified Logs App. + +__Example:__ +The following command will start Nginx webserver and logs for this container will be shipped to the related Logs App. + +``` +docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx +# or use environment variable on Kubernetes (no support for Docker labels) +# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOG_TOKEN -p 80:80 nginx +``` + +All other container logs will be shipped to the Logs App specified in the docker run command for ```sematext/logagent``` with the environment variable ```LOGSENE_TOKEN```. + +By default, all logs from all containers are collected and sent to Sematext Cloud/Elasticsearch. You can change this default by setting the ```LOGSENE_ENABLED_DEFAULT=false``` label for the Logagent container. This default can be overridden, on each container, through the ```LOGSENE_ENABLED``` label. + +Please refer to [Docker Log Management & Enrichment](https://sematext.com/blog/2017/05/15/docker-log-management-enrichment/) for further details. + + +## Known Issues + +**Conflict with Docker logging-drivers. Logagent is running +with a valid Logs Token, but Sematext Cloud does not show container logs. ** + +Please note that Logagent collects logs via Docker Remote +API. If you use a Docker logging-driver other than the default json-file +driver, logs will not be available via the Docker Remote API. Please +make sure that your container or docker daemon uses json-file logging +driver. This ensures that logs are exposed via Docker Remote API. To +check, run the "docker logs" command. If "docker logs CID" shows +container logs then Logagent should be able to collect the +logs as well. + +Please check the parsed timestamps! +Logs with timestamps in the future (or several months or years in the past) might not be displayed in Sematext Cloud. diff --git a/node_modules/@sematext/logagent/elasticsearch-http-test/docker-compose.yml b/node_modules/@sematext/logagent/elasticsearch-http-test/docker-compose.yml new file mode 100644 index 00000000..07e184f5 --- /dev/null +++ b/node_modules/@sematext/logagent/elasticsearch-http-test/docker-compose.yml @@ -0,0 +1,23 @@ +version: '2' +services: + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3 + container_name: elasticsearch + environment: + - path.data=/tmp/elasticsearch + - xpack.security.enabled=false + - discovery.type=single-node + - "ES_JAVA_OPTS=-Xms512m -Xmx512m" + ulimits: + memlock: + soft: -1 + hard: -1 + nofile: + soft: 65536 + hard: 65536 + mem_limit: 1g + #volumes: + # - /tmp/elasticsearch/data:/tmp/elasticsearch + ports: + - "9200:9200" + - "9300:9300" \ No newline at end of file diff --git a/node_modules/@sematext/logagent/elasticsearch-http-test/requests b/node_modules/@sematext/logagent/elasticsearch-http-test/requests new file mode 100644 index 00000000..847cc721 --- /dev/null +++ b/node_modules/@sematext/logagent/elasticsearch-http-test/requests @@ -0,0 +1,6 @@ +{ "index" : { "_index" : "mytest", "_type" : "type1", "_id" : "1" } } +{ "field1":"value1" } +{ "index" : { "_index" : "mytest", "_type" : "type1", "_id" : "2" } } +{ "field1":"value1" } +{ "index" : { "_index" : "mytest", "_type" : "type1", "_id" : "3" } } +{ "field1":"value1" } diff --git a/node_modules/@sematext/logagent/elasticsearch-http-test/test.sh b/node_modules/@sematext/logagent/elasticsearch-http-test/test.sh new file mode 100755 index 00000000..17b520c6 --- /dev/null +++ b/node_modules/@sematext/logagent/elasticsearch-http-test/test.sh @@ -0,0 +1,20 @@ +cd elasticsearch-test + +docker-compose version +docker version +if [[ "$(docker ps)" =~ "elastic" ]]; then + docker-compose stop && true + docker-compose rm -f && true +fi +docker-compose up -d +ps -ef |grep elastic-http | awk '{print $2}' | xargs kill +../bin/logagent.js --config ../config/examples/elastic-http-input.yml & +sleep 20 + +curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9900/_bulk --data-binary "@requests" +sleep 30 +curl 'localhost:9200/_cat/indices?v' 2>&1 | grep mytest +curl 'localhost:9200/mytest/_search' | ../bin/logagent.js -y +curl 'localhost:9200/mytest/_count' | ../bin/logagent.js -y +sleep 5 +ps -ef |grep elastic-http | awk '{print $2}' | xargs kill -3 diff --git a/node_modules/@sematext/logagent/kubernetes/configMapExample.yml b/node_modules/@sematext/logagent/kubernetes/configMapExample.yml new file mode 100644 index 00000000..d7b6f8f4 --- /dev/null +++ b/node_modules/@sematext/logagent/kubernetes/configMapExample.yml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: sematext-agent-patterns +data: + patterns.yml: |- + # Put your custom patterns below. + # Watch out on indentation to follow this comment! diff --git a/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-ds.yml b/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-ds.yml new file mode 100644 index 00000000..a7a735ae --- /dev/null +++ b/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-ds.yml @@ -0,0 +1,93 @@ +# Cluster Role bindings for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: sematext-logagent +subjects: +- kind: ServiceAccount + name: sematext-logagent + namespace: default +--- +# Cluster Role for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +rules: +- apiGroups: + - "" + resources: + - events + - pods + verbs: + - list + - get + - watch +--- +# Service Account for Logagent +apiVersion: v1 +kind: ServiceAccount +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +--- +# Daemonset +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: st-logagent +spec: + selector: + matchLabels: + app: st-logagent + template: + metadata: + labels: + app: st-logagent + spec: + nodeSelector: {} + serviceAccountName: sematext-logagent + hostNetwork: true + dnsPolicy: "ClusterFirst" + restartPolicy: "Always" + containers: + - name: st-logagent + image: sematext/logagent + imagePullPolicy: "Always" + env: + - name: LOG_GLOB + value: "/var/log/containers/*.log;/var/log/*.log" + - name: LOGAGENT_ARGS + value: "--k8sContainerd --k8sEnrichment" + - name: LOGS_TOKEN + value: "YOUR_SEMATEXT_LOGS_TOKEN" + - name: REGION + value: "US" + volumeMounts: + - mountPath: /var/log + name: var-log + - mountPath: /var/data/kubeletlogs + name: kubeletlogs + - mountPath: /etc/localtime + name: localtime + securityContext: + privileged: true + volumes: + - name: var-log + hostPath: + path: /var/log + - name: kubeletlogs + hostPath: + path: /var/data/kubeletlogs + - name: localtime + hostPath: + path: /etc/localtime diff --git a/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-with-config-ds.yml b/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-with-config-ds.yml new file mode 100644 index 00000000..7a091559 --- /dev/null +++ b/node_modules/@sematext/logagent/kubernetes/ibm-cloud-logagent-with-config-ds.yml @@ -0,0 +1,95 @@ +# Cluster Role bindings for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: sematext-logagent +subjects: +- kind: ServiceAccount + name: sematext-logagent + namespace: default +--- +# Cluster Role for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +rules: +- apiGroups: + - "" + resources: + - events + - pods + verbs: + - list + - get + - watch +--- +# Service Account for Logagent +apiVersion: v1 +kind: ServiceAccount +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +--- +# Daemonset +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: st-logagent +spec: + selector: + matchLabels: + app: st-logagent + template: + metadata: + labels: + app: st-logagent + spec: + nodeSelector: {} + serviceAccountName: sematext-logagent + hostNetwork: true + dnsPolicy: "ClusterFirst" + restartPolicy: "Always" + containers: + - name: sematext-logagent + image: sematext/logagent + imagePullPolicy: "Always" + env: + - name: LA_CONFIG_OVERRIDE + value: "true" + volumeMounts: + - mountPath: /var/log + name: var-log + - mountPath: /var/data/kubeletlogs + name: kubeletlogs + - mountPath: /etc/localtime + name: localtime + - mountPath: /etc/sematext + name: logagent-config + securityContext: + privileged: true + volumes: + - name: var-log + hostPath: + path: /var/log + - name: kubeletlogs + hostPath: + path: /var/data/kubeletlogs + - name: localtime + hostPath: + path: /etc/localtime + - name: logagent-config + configMap: + name: logagent-config + items: + - key: logagent.conf + path: logagent.conf \ No newline at end of file diff --git a/node_modules/@sematext/logagent/kubernetes/k8s-annotations-for-log-routing-example.yml b/node_modules/@sematext/logagent/kubernetes/k8s-annotations-for-log-routing-example.yml new file mode 100644 index 00000000..3354ea9b --- /dev/null +++ b/node_modules/@sematext/logagent/kubernetes/k8s-annotations-for-log-routing-example.yml @@ -0,0 +1,77 @@ +apiVersion: v1 +kind: Pod +metadata: + name: nginx + annotations: + "sematext.com/logs-enabled": "false" + #"sematext.com/logs-token": "YOUR_LOGS_TOKEN_FOR_NGINX_LOGS" + #"sematext.com/logs-remove-fields": "client_ip,user" + #"sematext.com/logs-receiver-url": "https://logsene-receiver.eu.sematext.com" + imageregistry: "https://hub.docker.com/" +spec: + containers: + - name: nginx + image: nginx + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Pod +metadata: + name: redis1 + annotations: + "sematext.com/logs-enabled": "true" + imageregistry: "https://hub.docker.com/" +spec: + containers: + - name: redis1 + image: redis + ports: + - containerPort: 6379 +--- +apiVersion: v1 +kind: Pod +metadata: + name: redis2 + annotations: + imageregistry: "https://hub.docker.com/" +spec: + containers: + - name: redis + image: redis + ports: + - containerPort: 6379 +--- +# Logagent test pod, in prod it is deployed as daemonset +apiVersion: v1 +kind: Pod +metadata: + name: st-logagent +spec: + containers: + - name: sematext-logagent + image: sematext/logagent:latest + imagePullPolicy: "IfNotPresent" + env: + # verbose output + #- name: LOGAGENT_ARGS + # value: "-v" + #- name: DEBUG + # value: "true" + #- name: SKIP_BY_IMAGE + # value: 'sematext.logagent' + #- name: SKIP_BY_NAME + # value: 'st-logagent' + - name: LOGSENE_ENABLED_DEFAULT + value: "false" + - name: LOGS_TOKEN + value: "YOUR_SEMATEXT_LOGS_TOKEN" + - name: LOGS_RECEIVER_URL + value: "https://logsene-receiver.sematext.com" + volumeMounts: + - mountPath: /var/run/docker.sock + name: docker-sock + volumes: + - name: docker-sock + hostPath: + path: /var/run/docker.sock diff --git a/node_modules/@sematext/logagent/kubernetes/logagent-daemonset.yml b/node_modules/@sematext/logagent/kubernetes/logagent-daemonset.yml new file mode 100644 index 00000000..88212038 --- /dev/null +++ b/node_modules/@sematext/logagent/kubernetes/logagent-daemonset.yml @@ -0,0 +1,92 @@ +# Cluster Role bindings for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: sematext-logagent +subjects: +- kind: ServiceAccount + name: sematext-logagent + namespace: default +--- +# Cluster Role for Logagent +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +rules: +- apiGroups: + - "" + resources: + - events + - pods + verbs: + - list + - get + - watch +--- +# Service Account for Logagent +apiVersion: v1 +kind: ServiceAccount +metadata: + name: sematext-logagent + labels: + app: sematext-logagent +--- +# Daemonset +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: st-logagent +spec: + selector: + matchLabels: + app: st-logagent + template: + metadata: + labels: + app: st-logagent + spec: + nodeSelector: {} + serviceAccountName: sematext-logagent + hostNetwork: true + dnsPolicy: "ClusterFirst" + restartPolicy: "Always" + containers: + - name: sematext-logagent + image: sematext/logagent:latest + imagePullPolicy: "Always" + env: + - name: LOGS_TOKEN + value: "YOUR_LOGS_TOKEN" + - name: LOGS_RECEIVER_URL + value: "https://logsene-receiver.sematext.com" + volumeMounts: + - mountPath: /var/run/docker.sock + name: docker-sock + - mountPath: /etc/localtime + name: localtime +# - mountPath: /etc/logagent +# name: patterns + securityContext: + privileged: true + volumes: + - name: docker-sock + hostPath: + path: /var/run/docker.sock + - name: localtime + hostPath: + path: /etc/localtime +# - name: patterns +# configMap: +# name: sematext-agent-patterns +# items: +# - key: patterns.yml +# path: patterns.yml diff --git a/node_modules/@sematext/logagent/lib/plugins/input-filter/kubernetesContainerd.js b/node_modules/@sematext/logagent/lib/plugins/input-filter/kubernetesContainerd.js new file mode 100644 index 00000000..43e810e8 --- /dev/null +++ b/node_modules/@sematext/logagent/lib/plugins/input-filter/kubernetesContainerd.js @@ -0,0 +1,130 @@ +// const containerdSplitRegexp = /^(.+[stdout|stderr] [F|P]) / // old +const containerdSplitRegexp = /^(.+)\s(stdout|stderr)\s(F|P)\s(.*)/ // new + +// Dictionary to store sources and log lines +/** + * Key: sourceName + * Value (Object): { streamFlag, previousStreamFlag, logLines } + */ +const sources = {} + +/** + * sourceName - origin of the log, e.g. file name + * config - properties from the config section for this plugin + * data - the log message as string + * callback - callback function (err, data). + */ +function parseK8sFileName (sourceName) { + /** + * SAMPLE sourceName * + * ***************** * + * sourceName: /var/log/containers/app-77b4d5595b-hmjxs_default_app-80209fd578c6be842b5b8a2d6389227ccab0196b7b658bd245eed4767c9b843c.log + * fileName: app-77b4d5595b-hmjxs_default_app-80209fd578c6be842b5b8a2d6389227ccab0196b7b658bd245eed4767c9b843c.log + * meta: { + * 0: app-77b4d5595b-hmjxs, // pod + * 1: default // namespace + * 2: app-80209fd578c6be842b5b8a2d6389227ccab0196b7b658bd245eed4767c9b843c.log // container with .log suffix + * } + */ + + // cut path from /var/log/containers/
new BufferList([ callback ])
-  * bl.length
-  * bl.append(buffer)
-  * bl.get(index)
-  * bl.slice([ start[, end ] ])
-  * bl.shallowSlice([ start[, end ] ])
-  * bl.copy(dest, [ destStart, [ srcStart [, srcEnd ] ] ])
-  * bl.duplicate()
-  * bl.consume(bytes)
-  * bl.toString([encoding, [ start, [ end ]]])
-  * bl.readDoubleBE(), bl.readDoubleLE(), bl.readFloatBE(), bl.readFloatLE(), bl.readInt32BE(), bl.readInt32LE(), bl.readUInt32BE(), bl.readUInt32LE(), bl.readInt16BE(), bl.readInt16LE(), bl.readUInt16BE(), bl.readUInt16LE(), bl.readInt8(), bl.readUInt8()
-  * Streams
-
---------------------------------------------------------
-
-### new BufferList([ callback | Buffer | Buffer array | BufferList | BufferList array | String ])
-The constructor takes an optional callback, if supplied, the callback will be called with an error argument followed by a reference to the **bl** instance, when `bl.end()` is called (i.e. from a piped stream). This is a convenient method of collecting the entire contents of a stream, particularly when the stream is *chunky*, such as a network stream.
-
-Normally, no arguments are required for the constructor, but you can initialise the list by passing in a single `Buffer` object or an array of `Buffer` object.
-
-`new` is not strictly required, if you don't instantiate a new object, it will be done automatically for you so you can create a new instance simply with:
-
-```js
-var bl = require('bl')
-var myinstance = bl()
-
-// equivalent to:
-
-var BufferList = require('bl')
-var myinstance = new BufferList()
-```
-
---------------------------------------------------------
-
-### bl.length
-Get the length of the list in bytes. This is the sum of the lengths of all of the buffers contained in the list, minus any initial offset for a semi-consumed buffer at the beginning. Should accurately represent the total number of bytes that can be read from the list.
-
---------------------------------------------------------
-
-### bl.append(Buffer | Buffer array | BufferList | BufferList array | String)
-`append(buffer)` adds an additional buffer or BufferList to the internal list. `this` is returned so it can be chained.
-
---------------------------------------------------------
-
-### bl.get(index)
-`get()` will return the byte at the specified index.
-
---------------------------------------------------------
-
-### bl.slice([ start, [ end ] ])
-`slice()` returns a new `Buffer` object containing the bytes within the range specified. Both `start` and `end` are optional and will default to the beginning and end of the list respectively.
-
-If the requested range spans a single internal buffer then a slice of that buffer will be returned which shares the original memory range of that Buffer. If the range spans multiple buffers then copy operations will likely occur to give you a uniform Buffer.
-
---------------------------------------------------------
-
-### bl.shallowSlice([ start, [ end ] ])
-`shallowSlice()` returns a new `BufferList` object containing the bytes within the range specified. Both `start` and `end` are optional and will default to the beginning and end of the list respectively.
-
-No copies will be performed. All buffers in the result share memory with the original list.
-
---------------------------------------------------------
-
-### bl.copy(dest, [ destStart, [ srcStart [, srcEnd ] ] ])
-`copy()` copies the content of the list in the `dest` buffer, starting from `destStart` and containing the bytes within the range specified with `srcStart` to `srcEnd`. `destStart`, `start` and `end` are optional and will default to the beginning of the `dest` buffer, and the beginning and end of the list respectively.
-
---------------------------------------------------------
-
-### bl.duplicate()
-`duplicate()` performs a **shallow-copy** of the list. The internal Buffers remains the same, so if you change the underlying Buffers, the change will be reflected in both the original and the duplicate. This method is needed if you want to call `consume()` or `pipe()` and still keep the original list.Example:
-
-```js
-var bl = new BufferList()
-
-bl.append('hello')
-bl.append(' world')
-bl.append('\n')
-
-bl.duplicate().pipe(process.stdout, { end: false })
-
-console.log(bl.toString())
-```
-
---------------------------------------------------------
-
-### bl.consume(bytes)
-`consume()` will shift bytes *off the start of the list*. The number of bytes consumed don't need to line up with the sizes of the internal Buffers—initial offsets will be calculated accordingly in order to give you a consistent view of the data.
-
---------------------------------------------------------
-
-### bl.toString([encoding, [ start, [ end ]]])
-`toString()` will return a string representation of the buffer. The optional `start` and `end` arguments are passed on to `slice()`, while the `encoding` is passed on to `toString()` of the resulting Buffer. See the [Buffer#toString()](http://nodejs.org/docs/latest/api/buffer.html#buffer_buf_tostring_encoding_start_end) documentation for more information.
-
---------------------------------------------------------
-
-### bl.readDoubleBE(), bl.readDoubleLE(), bl.readFloatBE(), bl.readFloatLE(), bl.readInt32BE(), bl.readInt32LE(), bl.readUInt32BE(), bl.readUInt32LE(), bl.readInt16BE(), bl.readInt16LE(), bl.readUInt16BE(), bl.readUInt16LE(), bl.readInt8(), bl.readUInt8()
-
-All of the standard byte-reading methods of the `Buffer` interface are implemented and will operate across internal Buffer boundaries transparently.
-
-See the [Buffer](http://nodejs.org/docs/latest/api/buffer.html) documentation for how these work.
-
---------------------------------------------------------
-
-### Streams
-**bl** is a Node **[Duplex Stream](http://nodejs.org/docs/latest/api/stream.html#stream_class_stream_duplex)**, so it can be read from and written to like a standard Node stream. You can also `pipe()` to and from a **bl** instance.
-
---------------------------------------------------------
-
-## Contributors
-
-**bl** is brought to you by the following hackers:
-
- * [Rod Vagg](https://github.com/rvagg)
- * [Matteo Collina](https://github.com/mcollina)
- * [Jarett Cruger](https://github.com/jcrugzz)
-
-=======
-
-
-## License & copyright
-
-Copyright (c) 2013-2016 bl contributors (listed above).
-
-bl is licensed under the MIT license. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE.md file for more details.
diff --git a/node_modules/bl/bl.js b/node_modules/bl/bl.js
deleted file mode 100644
index 0c8de189..00000000
--- a/node_modules/bl/bl.js
+++ /dev/null
@@ -1,290 +0,0 @@
-var DuplexStream = require('readable-stream/duplex')
-  , util         = require('util')
-  , Buffer       = require('safe-buffer').Buffer
-
-
-function BufferList (callback) {
-  if (!(this instanceof BufferList))
-    return new BufferList(callback)
-
-  this._bufs  = []
-  this.length = 0
-
-  if (typeof callback == 'function') {
-    this._callback = callback
-
-    var piper = function piper (err) {
-      if (this._callback) {
-        this._callback(err)
-        this._callback = null
-      }
-    }.bind(this)
-
-    this.on('pipe', function onPipe (src) {
-      src.on('error', piper)
-    })
-    this.on('unpipe', function onUnpipe (src) {
-      src.removeListener('error', piper)
-    })
-  } else {
-    this.append(callback)
-  }
-
-  DuplexStream.call(this)
-}
-
-
-util.inherits(BufferList, DuplexStream)
-
-
-BufferList.prototype._offset = function _offset (offset) {
-  var tot = 0, i = 0, _t
-  if (offset === 0) return [ 0, 0 ]
-  for (; i < this._bufs.length; i++) {
-    _t = tot + this._bufs[i].length
-    if (offset < _t || i == this._bufs.length - 1)
-      return [ i, offset - tot ]
-    tot = _t
-  }
-}
-
-
-BufferList.prototype.append = function append (buf) {
-  var i = 0
-
-  if (Buffer.isBuffer(buf)) {
-    this._appendBuffer(buf);
-  } else if (Array.isArray(buf)) {
-    for (; i < buf.length; i++)
-      this.append(buf[i])
-  } else if (buf instanceof BufferList) {
-    // unwrap argument into individual BufferLists
-    for (; i < buf._bufs.length; i++)
-      this.append(buf._bufs[i])
-  } else if (buf != null) {
-    // coerce number arguments to strings, since Buffer(number) does
-    // uninitialized memory allocation
-    if (typeof buf == 'number')
-      buf = buf.toString()
-
-    this._appendBuffer(Buffer.from(buf));
-  }
-
-  return this
-}
-
-
-BufferList.prototype._appendBuffer = function appendBuffer (buf) {
-  this._bufs.push(buf)
-  this.length += buf.length
-}
-
-
-BufferList.prototype._write = function _write (buf, encoding, callback) {
-  this._appendBuffer(buf)
-
-  if (typeof callback == 'function')
-    callback()
-}
-
-
-BufferList.prototype._read = function _read (size) {
-  if (!this.length)
-    return this.push(null)
-
-  size = Math.min(size, this.length)
-  this.push(this.slice(0, size))
-  this.consume(size)
-}
-
-
-BufferList.prototype.end = function end (chunk) {
-  DuplexStream.prototype.end.call(this, chunk)
-
-  if (this._callback) {
-    this._callback(null, this.slice())
-    this._callback = null
-  }
-}
-
-
-BufferList.prototype.get = function get (index) {
-  return this.slice(index, index + 1)[0]
-}
-
-
-BufferList.prototype.slice = function slice (start, end) {
-  if (typeof start == 'number' && start < 0)
-    start += this.length
-  if (typeof end == 'number' && end < 0)
-    end += this.length
-  return this.copy(null, 0, start, end)
-}
-
-
-BufferList.prototype.copy = function copy (dst, dstStart, srcStart, srcEnd) {
-  if (typeof srcStart != 'number' || srcStart < 0)
-    srcStart = 0
-  if (typeof srcEnd != 'number' || srcEnd > this.length)
-    srcEnd = this.length
-  if (srcStart >= this.length)
-    return dst || Buffer.alloc(0)
-  if (srcEnd <= 0)
-    return dst || Buffer.alloc(0)
-
-  var copy   = !!dst
-    , off    = this._offset(srcStart)
-    , len    = srcEnd - srcStart
-    , bytes  = len
-    , bufoff = (copy && dstStart) || 0
-    , start  = off[1]
-    , l
-    , i
-
-  // copy/slice everything
-  if (srcStart === 0 && srcEnd == this.length) {
-    if (!copy) { // slice, but full concat if multiple buffers
-      return this._bufs.length === 1
-        ? this._bufs[0]
-        : Buffer.concat(this._bufs, this.length)
-    }
-
-    // copy, need to copy individual buffers
-    for (i = 0; i < this._bufs.length; i++) {
-      this._bufs[i].copy(dst, bufoff)
-      bufoff += this._bufs[i].length
-    }
-
-    return dst
-  }
-
-  // easy, cheap case where it's a subset of one of the buffers
-  if (bytes <= this._bufs[off[0]].length - start) {
-    return copy
-      ? this._bufs[off[0]].copy(dst, dstStart, start, start + bytes)
-      : this._bufs[off[0]].slice(start, start + bytes)
-  }
-
-  if (!copy) // a slice, we need something to copy in to
-    dst = Buffer.allocUnsafe(len)
-
-  for (i = off[0]; i < this._bufs.length; i++) {
-    l = this._bufs[i].length - start
-
-    if (bytes > l) {
-      this._bufs[i].copy(dst, bufoff, start)
-      bufoff += l
-    } else {
-      this._bufs[i].copy(dst, bufoff, start, start + bytes)
-      bufoff += l
-      break
-    }
-
-    bytes -= l
-
-    if (start)
-      start = 0
-  }
-
-  // safeguard so that we don't return uninitialized memory
-  if (dst.length > bufoff) return dst.slice(0, bufoff)
-
-  return dst
-}
-
-BufferList.prototype.shallowSlice = function shallowSlice (start, end) {
-  start = start || 0
-  end = end || this.length
-
-  if (start < 0)
-    start += this.length
-  if (end < 0)
-    end += this.length
-
-  var startOffset = this._offset(start)
-    , endOffset = this._offset(end)
-    , buffers = this._bufs.slice(startOffset[0], endOffset[0] + 1)
-
-  if (endOffset[1] == 0)
-    buffers.pop()
-  else
-    buffers[buffers.length-1] = buffers[buffers.length-1].slice(0, endOffset[1])
-
-  if (startOffset[1] != 0)
-    buffers[0] = buffers[0].slice(startOffset[1])
-
-  return new BufferList(buffers)
-}
-
-BufferList.prototype.toString = function toString (encoding, start, end) {
-  return this.slice(start, end).toString(encoding)
-}
-
-BufferList.prototype.consume = function consume (bytes) {
-  // first, normalize the argument, in accordance with how Buffer does it
-  bytes = Math.trunc(bytes)
-  // do nothing if not a positive number
-  if (Number.isNaN(bytes) || bytes <= 0) return this
-
-  while (this._bufs.length) {
-    if (bytes >= this._bufs[0].length) {
-      bytes -= this._bufs[0].length
-      this.length -= this._bufs[0].length
-      this._bufs.shift()
-    } else {
-      this._bufs[0] = this._bufs[0].slice(bytes)
-      this.length -= bytes
-      break
-    }
-  }
-  return this
-}
-
-
-BufferList.prototype.duplicate = function duplicate () {
-  var i = 0
-    , copy = new BufferList()
-
-  for (; i < this._bufs.length; i++)
-    copy.append(this._bufs[i])
-
-  return copy
-}
-
-
-BufferList.prototype.destroy = function destroy () {
-  this._bufs.length = 0
-  this.length = 0
-  this.push(null)
-}
-
-
-;(function () {
-  var methods = {
-      'readDoubleBE' : 8
-    , 'readDoubleLE' : 8
-    , 'readFloatBE'  : 4
-    , 'readFloatLE'  : 4
-    , 'readInt32BE'  : 4
-    , 'readInt32LE'  : 4
-    , 'readUInt32BE' : 4
-    , 'readUInt32LE' : 4
-    , 'readInt16BE'  : 2
-    , 'readInt16LE'  : 2
-    , 'readUInt16BE' : 2
-    , 'readUInt16LE' : 2
-    , 'readInt8'     : 1
-    , 'readUInt8'    : 1
-  }
-
-  for (var m in methods) {
-    (function (m) {
-      BufferList.prototype[m] = function (offset) {
-        return this.slice(offset, offset + methods[m])[m](0)
-      }
-    }(m))
-  }
-}())
-
-
-module.exports = BufferList
diff --git a/node_modules/bl/package.json b/node_modules/bl/package.json
deleted file mode 100644
index 1ffd2526..00000000
--- a/node_modules/bl/package.json
+++ /dev/null
@@ -1,35 +0,0 @@
-{
-  "name": "bl",
-  "version": "1.2.3",
-  "description": "Buffer List: collect buffers and access with a standard readable Buffer interface, streamable too!",
-  "main": "bl.js",
-  "scripts": {
-    "test": "node test/test.js | faucet"
-  },
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/rvagg/bl.git"
-  },
-  "homepage": "https://github.com/rvagg/bl",
-  "authors": [
-    "Rod Vagg 
+  
+    
+  
+
+  date-fns provides the most comprehensive, yet simple and consistent toolset
+  
+  for manipulating JavaScript dates in a browser & Node.js.
+