diff --git a/docs.kosli.com/content/faq/_index.md b/docs.kosli.com/content/faq/_index.md index f34f476b4..9e69268c4 100644 --- a/docs.kosli.com/content/faq/_index.md +++ b/docs.kosli.com/content/faq/_index.md @@ -131,8 +131,8 @@ API-TOKEN = "123456abcdef" When calling Kosli command you can skip the file extension. For example, to list environments with `org` and `api-token` in the configuration file you would run: -``` -$ kosli list environments --config-file kosli-conf +```shell {.command} +kosli list environments --config-file kosli-conf ``` `--config-file` defaults to `kosli`, so if you name your file `kosli.` and the file is in the same location as where you run Kosli commands from, you can skip the `--config-file` altogether. @@ -163,8 +163,8 @@ part of the template, the state of the extra evidence will affect the overall co The `--compliant` flag is a [boolean flag](#boolean-flags). To report generic evidence as non-compliant use `--compliant=false`, as in this example: -``` -$ kosli report evidence artifact generic server:1.0 \ +```shell {.command} +kosli report evidence artifact generic server:1.0 \ --artifact-type docker \ --name test \ --description "generic test evidence" \ diff --git a/docs.kosli.com/content/getting_started/approvals.md b/docs.kosli.com/content/getting_started/approvals.md index a37ddf56f..d6b89b62d 100644 --- a/docs.kosli.com/content/getting_started/approvals.md +++ b/docs.kosli.com/content/getting_started/approvals.md @@ -1,10 +1,10 @@ --- -title: "Part 9: Approvals" +title: "Part 10: Approvals" bookCollapseSection: false weight: 300 summary: "When an artifact is ready to be deployed to a given environment, an approval may be reported to Kosli. An approval can be requested which will require a manual action, or reported automatically. This will be recorded in Kosli so the decision made outside your CI system won't be lost." --- -# Part 9: Approvals +# Part 10: Approvals When an artifact is ready to be deployed to a given [environment](/getting_started/environments/), an approval may be reported to Kosli. An approval can be requested which will require a manual action, or reported automatically. This will be recorded in Kosli so the decision made outside your CI system won't be lost. diff --git a/docs.kosli.com/content/getting_started/artifacts.md b/docs.kosli.com/content/getting_started/artifacts.md index f796e2223..5b6233036 100644 --- a/docs.kosli.com/content/getting_started/artifacts.md +++ b/docs.kosli.com/content/getting_started/artifacts.md @@ -28,8 +28,8 @@ By establishing and maintaining binary provenance for artifacts, Kosli enables y To attest an artifact, you can run a command similar to the one below: -```shell -$ kosli attest artifact project-a-app.bin \ +```shell {.command} +kosli attest artifact project-a-app.bin \ --artifact-type file \ --build-url https://exampleci.com \ --commit-url https://github.com/ProjectA/ProjectAApp/commit/e67f2f2b121f9325ebf166b7b3c707f73cb48b14 \ @@ -63,8 +63,8 @@ This will allow you to instantly turn off all Kosli CLI commands if Kosli is dow The `--dry-run` flag is also useful when trying commands locally. For example: -```shell -$ kosli attest artifact cyberdojo/differ:dde3b2a \ +```shell {.command} +kosli attest artifact cyberdojo/differ:dde3b2a \ --artifact-type=docker \ --org=cyber-dojo \ --flow=differ-ci \ diff --git a/docs.kosli.com/content/getting_started/attestations.md b/docs.kosli.com/content/getting_started/attestations.md index edb0ebd72..defb713a3 100644 --- a/docs.kosli.com/content/getting_started/attestations.md +++ b/docs.kosli.com/content/getting_started/attestations.md @@ -39,8 +39,8 @@ When you make an attestation, you have the choice of what `name` to attach it to The `jira-ticket` attestation belongs to a single trail and is not linked to a specific artifact. In this example, the id of the trail is the git commit. -```shell -$ kosli attest jira \ +```shell {.command} +kosli attest jira \ --flow backend-ci \ --trail $(git rev-parse HEAD) \ --name jira-ticket @@ -51,8 +51,8 @@ $ kosli attest jira \ Some attestations are attached to a specific artifact, like the unit tests for the `backend` artifact. Often, evidence like unit tests are created _before_ the artifact is built. To attach the evidence to the artifact before its creation, use `backend` (the artifact's `name` from the template), as well as `unit-tests` (the attestation's `name` from the template). -```shell -$ kosli attest junit \ +```shell {.command} +kosli attest junit \ --name backend.unit-tests \ --flow backend-ci \ --trail $(git rev-parse HEAD) \ @@ -65,8 +65,8 @@ This attestation belongs to any artifact attested with the matching `name` from Once the artifact has been built, it can be attested with the following command. -```shell -$ kosli attest artifact my_company/backend:latest \ +```shell {.command} +kosli attest artifact my_company/backend:latest \ --artifact-type docker \ --flow backend-ci \ --trail $(git rev-parse HEAD) \ @@ -88,8 +88,8 @@ Often, evidence like snyk reports are created _after_ the artifact is built. In The following attestation will only belong to the artifact `my_company/backend:latest` attested above and its fingerprint, in this case calculated by the Kosli CLI. -```shell -$ kosli attest snyk \ +```shell {.command} +kosli attest snyk \ --artifact-type docker my_company/backend:latest \ --name backend.security-scan \ --flow backend-ci \ @@ -107,12 +107,12 @@ Typically, the Flow and Trail are explicitly setup before making the attestation This is done with the `create flow` and `begin trail` commands, either of which can specify the name of the template yaml file above (e.g. `.kosli.yml`) whose contents define overall compliance. For example: -```shell -$ kosli create flow backend-ci \ +```shell {.command} +kosli create flow backend-ci \ --template-file .kosli.yml ... -$ kosli begin trail $(git rev-parse HEAD) \ +kosli begin trail $(git rev-parse HEAD) \ --flow backend-ci \ ... ``` @@ -142,7 +142,6 @@ Attestations are append-only immutable records. You can report the same attestat However, only the latest version of the attestation is considered when evaluating compliance. - ## Evidence Vault Along with attestations data, you can attach additional supporting evidence files. These will be securely stored in Kosli's **Evidence Vault** and can easily be retrieved when needed. Alternatively, you can store the evidence files in your own preferred storage and only attach links to it in the Kosli attestation. diff --git a/docs.kosli.com/content/getting_started/environments.md b/docs.kosli.com/content/getting_started/environments.md index f7327496e..f73b16ec2 100644 --- a/docs.kosli.com/content/getting_started/environments.md +++ b/docs.kosli.com/content/getting_started/environments.md @@ -2,11 +2,11 @@ title: "Part 8: Environments" bookCollapseSection: false weight: 280 -summary: "Kosli environments allow you to record the artifacts running in your runtime environments and how they change. Every time an environment change (or a set of changes) is reported, Kosli creates a new environment snapshot containing the status of the environment at a given point in time." +summary: "Kosli environments allow you to record the artifacts running in your runtime environments and how they change. Every time an environment change (or a set of changes) is reported, Kosli creates a new environment snapshot containing the status of the environment at a given point in time. The change record created in Kosli enables you to retrospectively perform runtime forensics about what ran where and when." --- # Part 8: Environments -Kosli environments allow you to record the artifacts running in your runtime environments and how they change. Every time an environment change (or a set of changes) is reported, Kosli creates a new environment snapshot containing the status of the environment at a given point in time. +Kosli environments allow you to record the artifacts running in your runtime environments and how they change. Every time an environment change (or a set of changes) is reported, Kosli creates a new environment snapshot containing the status of the environment at a given point in time. The change record created in Kosli enables you to retrospectively perform runtime forensics about what ran where and when. ## Create an environment @@ -21,7 +21,7 @@ Make sure that type of Kosli environment matches the type of the environment you To create an environment via CLI, you would run a command like this: ```shell {.command} -$ kosli create environment quickstart \ +kosli create environment quickstart \ --environment-type docker \ --description "quickstart environment for tutorial" ``` @@ -42,8 +42,8 @@ After the new environment is created you'll be redirected to its page, which wil ## Snapshoting an environment -To record the current status of your environment you need to make Kosli CLI snapshot the running artifacts in it and report it to Kosli. -When Kosli receives an environment report, if the received list of running artifacts is different than what is in the latest environment snapshot, a new environment snapshot is created. Snapshots are immutable and can't be tampered with. +To record the current status of your environment you need to use the Kosli CLI to snapshot the running artifacts in it and report it to Kosli. +When Kosli receives an environment report, if the received list of running artifacts is different than what is in the latest environment snapshot, a new snapshot is created. Snapshots are immutable and can't be tampered with. Currently, the following environment types are supported: - Kubernetes @@ -52,6 +52,7 @@ Currently, the following environment types are supported: - AWS Simple Storage Service (S3) - AWS Lambda - AWS Elastic Container Service (ECS) +- Azure Web Apps and Function Apps You can report environment snapshots manually using the `kosli snapshot [...]` commands for testing. For production use, however, you would configure the reporting to happen automatically on regular intervals, e.g. via a cron job or scheduled CI job, or on certain events. diff --git a/docs.kosli.com/content/getting_started/flows.md b/docs.kosli.com/content/getting_started/flows.md index d2c9a73cc..e87bed654 100644 --- a/docs.kosli.com/content/getting_started/flows.md +++ b/docs.kosli.com/content/getting_started/flows.md @@ -16,8 +16,8 @@ In all the commands below we skip the required `--api-token` and `--org` flags f To create a Flow, you can run: -```shell -$ kosli create flow process-a --description "My SW delivery process" \ +```shell {.command} +kosli create flow process-a --description "My SW delivery process" \ --use-empty-template ``` @@ -46,8 +46,8 @@ trail: To create a Flow with a template, you can run: -```shell -$ kosli create flow process-a --description "My SW delivery process" \ +```shell {.command} +kosli create flow process-a --description "My SW delivery process" \ --template-file sw-delivery-template.yml ``` diff --git a/docs.kosli.com/content/getting_started/install.md b/docs.kosli.com/content/getting_started/install.md index 807640e59..5616c54e5 100644 --- a/docs.kosli.com/content/getting_started/install.md +++ b/docs.kosli.com/content/getting_started/install.md @@ -163,6 +163,6 @@ API-TOKEN = "123456abcdef" When using the `--config-file` flag you can skip the file extension. For example, to list environments with `org` and `api-token` in the configuration file you would run: -``` -$ kosli list environments --config-file=kosli-conf +```shell {.command} +kosli list environments --config-file=kosli-conf ``` diff --git a/docs.kosli.com/content/getting_started/next.md b/docs.kosli.com/content/getting_started/next.md index fa9097a30..2bc3753a4 100644 --- a/docs.kosli.com/content/getting_started/next.md +++ b/docs.kosli.com/content/getting_started/next.md @@ -1,11 +1,11 @@ --- -title: "Part 10: Next Steps" +title: "Part 11: Next Steps" bookCollapseSection: false weight: 310 summary: "In the previous chapters, you explored Kosli Flows and Environments and have reported some data to Kosli. The next steps would be to harness the benefits of your hard work. Here are a few areas to look at next:" --- -# Part 10: Next Steps +# Part 11: Next Steps In the previous chapters, you explored Kosli Flows and Environments and have reported some data to Kosli. The next steps would be to harness the benefits of your hard work. Here are a few areas to look at next: diff --git a/docs.kosli.com/content/getting_started/policies.md b/docs.kosli.com/content/getting_started/policies.md new file mode 100644 index 000000000..d7d576fdc --- /dev/null +++ b/docs.kosli.com/content/getting_started/policies.md @@ -0,0 +1,227 @@ +--- +title: "Part 9: Environment Policies" +bookCollapseSection: false +weight: 290 +summary: "Environment Policies enable you to define and enforce compliance requirements for artifact deployments across different environments." +--- +# Part 9: Environment Policies + +{{% hint warning %}} +Environment policies is in alpha. It is subject to change, including naming, syntax, CLI commands, etc. +If you want to try this feature, create a policy and attach it to an environment. +{{% /hint %}} +{{% hint warning %}} +Note that once an environment starts using policies, it is not possible to go back to not using them. +{{% /hint %}} + +Environment Policies enable you to define and enforce compliance requirements for artifact deployments across different environments. With Environment Policies, you can: +- Define specific requirements for each environment (e.g, dev, staging, prod) +- Enforce consistent compliance standards across your deployment pipeline +- Prevent non-compliant artifacts from being deployed (via admission controllers) + +Polices are written in YAML and are immutable (updating a policy creates a new version). They can be attached to one or more environments, and an environment can have one or more policies attached to it. + + +## Create a Policy + +You can create a policy via CLI or via the API. Here is a basic policy that requires provenance and specific attestations: + +```yaml {.command} +# prod-policy.yaml +_schema: https://kosli.com/schemas/policy/environment/v1 +artifacts: # the rules apply to artifacts in an environment snapshot + provenance: + required: true # all artifacts must have provenance + attestations: + - name: dependency-scan # all artifacts must have dependency-scan attestation + type: '*' # any attestation type + - name: unit-test # all artifacts must have unit-test attestation + type: junit # must be a 'junit' attestation type +``` + +You can create and manage policies using the Kosli CLI (global flags like org and api-token are omitted for brevity): + +```shell {.command} +kosli create policy prod-requirements prod-policy.yaml +``` + +```shell {.command} +kosli create get policy prod-requirements +``` + +See [kosli create policy](/client_reference/kosli_create_policy/) for usage details and examples. + +{{% hint info %}} +Once you create a policy, you will be able to see it in the UI under `policies` in the left navigation menu. +{{% /hint %}} + +## Declarative Policy Syntax + +A Policy is declaratively defined according to the following schema: + +```yaml +_schema: https://kosli.com/schemas/policy/environment/v1 + +artifacts + provenance: + required: true | false (default = false) + exceptions: (default []) + - if: ${{ expression }} + + trail-compliance: + required: true | false (default = false) + exceptions: (default []) + - if: ${{ expression }} + + attestations: (default []) + - if: ${{ expression }} (default = true) + name: str (default = "*") # cannot have both name and type as * + type: oneOf ['*', 'junit', 'jira', 'pull_request', 'snyk', 'sonar', 'generic', 'custom:'] (default = "*") # cannot have both name and type as * +``` + +### Policy Rules + +A policy consists of `rules` which are applied to artifacts in an environment snapshot. + +#### Provenance + +```yaml {.command} +artifacts: + provenance: + required: true # Requires artifact to be part of a Kosli Flow +``` + +#### Trail Compliance + +```yaml {.command} +artifacts: + trail-compliance: + required: true # Requires the trail in which the artifact is attested to be compliant +``` + +#### Specific Attestations + +```yaml {.command} +artifacts: + attestations: + - name: '*' # attestation name can be anything + type: pull-request + - name: acceptance-test + type: '*' # attestation type can be any built-in or existing custom type + - name: security-scan + type: snyk + - name: coverage-metrics + type: custom:my-coverage-metrics # custom attestation type +``` + +### Policy Rules Exceptions + +You can add exceptions to policy rules using expressions. + +```yaml +_schema: https://kosli.com/schemas/policy/environment/v1 + +artifacts + provenance: + required: true + exceptions: + # provenance is required except when one of the expressions evaluates to true + - if: ${{ expression1 }} + - if: ${{ expression2 }} + + trail-compliance: + required: true + exceptions: + # trail-compliance is required except when one of the expressions evaluates to true + - if: ${{ expression1 }} + - if: ${{ expression2 }} + + attestations: + - if: ${{ expression }} # this attestation is only required when expression evaluates to true + name: unit-tests + type: junit +``` + +#### Policy Expressions + +Policy expressions allow you to create conditional rules using a simple and powerful syntax. Expressions are wrapped in `${{ }}` and can be used in policy rules to create dynamic conditions. An expression consists of operands and operators: + +**Operators** + +Expressions support these operators: +- Comparison: `==, !=, <, >, <=, >=` +- Logical: `and, or, not` +- List membership: `in` + +**Operands** + +Operands can be: +- Literal string +- List +- Context variable +- Function call + + +**Available Contexts** + +Contexts are built-in objects which are accessible from an expression. Expressions can access two main contexts: +- `flow` - Information about the Kosli Flow: + - `flow.name` - Name of the flow + - `flow.tags` - Flow tags (accessed via flow.tags.tag_name) +- `artifact` - Information about the artifact: + - `artifact.name` - Name of the artifact + - `artifact.fingerprint` - SHA256 fingerprint + +**Functions** + +Functions are helpers that can be used when constructing conditions. They may or may not accept arguments. Arguments can be literals or context variables. Expressions can use following functions: + +- `exists(arg)` : checks whether the value of arg is not None/Null +- `matches(input, regex)` : checks if input matches regex + + +**Example Expressions** + +- ${{ exists(flow) }} +- ${{ flow.name in ["runner", 'saver', differ] }} +- ${{ matches(artifact.name, "^datadog:.*") }} +- ${{ flow.name == "runner" and matches(artifact.name, "^runner:.*") }} +- ${{ flow.tags.risk-level == "high" or matches(artifact.name, "^runner:.*") }} +- ${{ not flow.tags.risk-level == "high"}} +- ${{ flow.tags.risk-level != "high"}} +- ${{ flow.tags.key.with.dots == "value"}} +- ${{ flow.tags.risk-level >= 2 }} +- ${{ flow.name == 'prod' and (flow.tags.key_name == "value" or artifact.name == 'critical-service') }} +- ${{ flow.name == 'HIGH-RISK' and artifact.fingerprint == "37193ba1f3da2581e93ff1a9bba523241a7982a6c01dd311494b0aff6d349462" }} + + +## Attaching/Detaching Policies to/from Environments + +Once you define your policies, you can attach them to environments via CLI or API: + +```shell {.command} +kosli attach-policy prod-requirements --environment=aws-production +``` + +To detach a policy from an environment: + +```shell {.command} +kosli detach-policy prod-requirements --environment=aws-production +``` + +Any attachment/detachment operation automatically triggers an evaluation of the latest environment snapshot and creates a new one with an updated compliance status. + +{{% hint info %}} +If you detach all attached policies from an environment, the environment will have no defined requirements for artifacts running in it, and therefore, new environment snapshots will have status `unknown` +{{% /hint %}} + + +## Policy Enforcement Gates + +Environment policies enable you to proactively block deploying a non-compliant artifact into an environment. This can be done as a deployment gate in your delivery pipeline or as an admission controller in your environment. + +Regardless of where you place your policy enforcement gate, it will be using the `assert artifact` Kosli CLI command or its equivalent API call. + +```shell {.command} +kosli assert artifact --fingerprint=$SHA256 --environment=aws-production +``` \ No newline at end of file diff --git a/docs.kosli.com/content/getting_started/trails.md b/docs.kosli.com/content/getting_started/trails.md index afe3360c3..dbf29c006 100644 --- a/docs.kosli.com/content/getting_started/trails.md +++ b/docs.kosli.com/content/getting_started/trails.md @@ -20,8 +20,8 @@ Each trail must possess a unique name within the Flow. This name typically follo To begin a Trail, you can run a command similar to the one below: -```shell -$ kosli begin trail trail-1 --flow process-1 --description "My first trail" +```shell {.command} +kosli begin trail trail-1 --flow process-1 --description "My first trail" ``` Rerunning the command with different description or template file will update the Trail. diff --git a/docs.kosli.com/content/template_ref/_index.md b/docs.kosli.com/content/template_ref/_index.md index a62c493dc..8be0fb782 100644 --- a/docs.kosli.com/content/template_ref/_index.md +++ b/docs.kosli.com/content/template_ref/_index.md @@ -16,7 +16,7 @@ trail: # the trail specification (optional) - name: reference name for the artifact (e.g. frontend-app) (required) attestations: # what attestations are required for the artifact to be compliant - name: the attestation name (required) - type: the attestation type. One of [generic, jira, junit, pull_request, snyk] (required) + type: the attestation type. One of [generic, jira, junit, pull_request, snyk, sonar, custom:] (required) ``` ## Example: @@ -40,4 +40,6 @@ trail: attestations: - name: manual-ui-test type: generic + - name: coverage-metrics + type: custom:coverage-metrics ``` diff --git a/docs.kosli.com/content/understand_kosli/concepts.md b/docs.kosli.com/content/understand_kosli/concepts.md index f2130c946..e1484da20 100644 --- a/docs.kosli.com/content/understand_kosli/concepts.md +++ b/docs.kosli.com/content/understand_kosli/concepts.md @@ -12,7 +12,7 @@ This section helps you understand the concepts Kosli is built on. The figure bel ## Organization -A Kosli organization is an account that owns Kosli resources, such as Flows and Environments. Only members within an organization can access its contents. +A Kosli organization is an account that owns Kosli resources, such as Flows and Environments. Only members within an organization can access its resources. When signing up for Kosli, a personal organization is automatically created for you, bearing your username. This personal organization is exclusively accessible to you. Additionally, you can create `Shared` organizations and invite multiple team members to collaborate on different Flows and Environments. @@ -27,15 +27,15 @@ Each Trail must have a unique identifier of your choice, based on your process a #### Artifact -Kosli Artifacts represent the software artifacts generated from every execution, portrayed as a Trail, of your software process depicted as a Flow. These artifacts play a crucial role in enabling **Binary Provenance**, providing a comprehensive chain of custody that records the origin, history, distribution, and execution details of each artifact. +Kosli Artifacts represent the software artifacts generated from every execution, portrayed as a Trail, of your software process depicted as a Flow. These artifacts play a crucial role in enabling **Binary Provenance**, providing a comprehensive chain of custody that records the origin, history and distribution of each artifact. Each Artifact is uniquely identified by its SHA256 fingerprint. Using this fingerprint, Kosli can link the creation of the Artifact with its runtime-related events, such as when the artifact starts or concludes execution within a specific Environment. #### Attestation -An Attestation is a declaration about whether a particular Artifact or Trail adheres to a certain requirement or not. It is normally reported after performing a specific risk control or quality check (e.g. running tests). The attestation encompasses the procedure's results. +An Attestation is a record of compliance checks or controls that have been performed a particular Artifact or Trail. It is normally reported after performing a specific risk control or quality check (e.g. running tests). The attestation encompasses the procedure's results. -Kosli supports reporting specific types of attestations (e.g., a snyk scan, sonarcloud scan, junit tests) and a generic one for other use cases. +Kosli provides specific built-in types of attestations (e.g., a snyk scan, sonar scan, junit tests) and allows to define your own custom types. ##### Evidence Vault @@ -62,6 +62,7 @@ Kosli supports various types of runtime environments: * Amazon Lambda * Physical/virtual server * Docker host +* Azure Web Apps and Function Apps ### Environment Snapshot @@ -70,3 +71,8 @@ An Environment Snapshot represents the reported status (running Artifacts) of yo In each snapshot, Kosli links the running artifacts to the Flows and Trails that produced them. Snapshot compliance relies on the compliance status of each running artifact, while Environment compliance depends on its latest snapshot compliance. Running artifacts that come from 3rd party sources, can be `allow-listed` in an Environment to make them compliant. + + +### Environment Policy + +Environment Policy enables you to define and enforce compliance requirements for artifact deployments across different environments. \ No newline at end of file