diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..ebe4b538 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.idea/ +node_modules \ No newline at end of file diff --git a/CATALOG.md b/CATALOG.md deleted file mode 100644 index 2e37544d..00000000 --- a/CATALOG.md +++ /dev/null @@ -1,13 +0,0 @@ -# Plugin Catalog - -## Stable Plugins - -| Plugin | Description | Tags | -| ------------------------------------ | ------------------- | ------------------- | -| [Deploy Helm](stable/helm/README.md) | Deploy a Helm chart | `kubernetes` `helm` | - -## Incubator Plugins - -| Plugin | Description | Tags | -| ---------------------------------------------------------------- | -------------------------------------------- | -------------------- | -| [Import Docker Images](incubator/import-docker-images/README.md) | Import Docker images metadata into Codefresh | `docker` `codefresh` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ae42fd6e..732e1302 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,8 +9,9 @@ This repository is used by Plugin developers for maintaining the official plugin ### How to Contribute a Plugin -1. Fork this repository, develop and test your Plugin -2. Choose the correct folder for your plugin based on the information in the [Repository Structure](README.md#repository-structure) section +1. Develop and test your plugin +2. Fork this repository +2. Choose the correct folder for your plugin based on the information in the [Repository Structure](docs/plugin.md#plugin-file-structure) section 3. Ensure your Plugin follows the [technical](#technical-requirements) and [documentation](#documentation-requirements) guidelines, described below 4. Update [Plugin Catalog](CATALOG.md) 5. Submit a pull request diff --git a/README.md b/README.md index 3711b872..053bb9be 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,27 @@ -# Codefresh Plugins - -Use this repository to submit official Plugins for Codefresh. Plugins are curated automated step definitions for Codefresh. For more information about using Codefresh, see its -[documentation](https://docs.codefresh.io). - -## How do I install these plugins? - -Codefresh Plugins are available as plain Docker images. As long as Plugin Docker image is accessible, Codefresh can use it for its pipelines. - -## How do I use plugins from the Incubator repository? - -*TBD* - -## Codefresh Plugin Format - -Take a look at the [example plugin](#) for reference when you're writing your first few plugins. - -The Plugin folder must contain `plugin.yaml` and `README.md` files. - -Before contributing a Plugin, become familiar with the format. Note that the project is still under active development and the format may still evolve a bit. - -## Repository Structure - -This GitHub repository contains the source for the packaged and versioned plugins released in the [`gs://codefresh-plugins` Google Storage bucket](https://console.cloud.google.com/storage/browser/codefresh-plugins/) (the Plugins Repository). - -The Plugins in the `stable/` directory in the master branch of this repository match the latest packaged Plugins in the Plugin Repository, though there may be previous versions of a Plugin available in that Plugin Repository. - -The purpose of this repository is to provide a place for maintaining and contributing official Plugins, with CI processes in place for managing the releasing of Plugins into the Plugin Repository. - -The Plugins in this repository are organized into two folders: -* stable -* incubator - -Stable Plugins meet the criteria in the [technical requirements](CONTRIBUTING.md#technical-requirements). - -Incubator Plugins are those that do not meet these criteria. Having the incubator folder allows plugins to be shared and improved on until they are ready to be moved into the stable folder. The plugins in the `incubator/` directory can be found in the [`gs://codefresh-plugins-incubator` Google Storage Bucket](https://console.cloud.google.com/storage/browser/codefresh-plugins-incubator). - -In order to get a Plugin from incubator to stable, Plugin maintainers should open a pull request that moves the plugin folder. - -## Contributing a Plugin - -We'd love for you to contribute a Plugin that provides a useful automated step for Codefresh. Please read our [Contribution Guide](CONTRIBUTING.md) for more information on how you can contribute Plugins. - -## Review Process - -The following outlines the review procedure used by the Plugin repository maintainers. Github labels are used to indicate state change during the review process. - -* ***AWAITING REVIEW*** - Initial triage which indicates that the PR is ready for review by the maintainers team. All e2e tests must pass in-order to move to this state -* ***CHANGES NEEDED*** - Review completed by at least one maintainer and changes needed by contributor (explicit even when using the review feature of Github) -* ***CODE REVIEWED*** - The plugin structure has been reviewed and found to be satisfactory given the [technical requirements](CONTRIBUTING.md#technical-requirements) (may happen in parallel to UX REVIEWED) -* ***UX REVIEWED*** - The plugin installation UX has been reviewed and found to be satisfactory. (may happen in parallel to CODE REVIEWED) -* ***LGTM*** - Added ONLY once both UX/CODE reviewed are both present. Merge must be handled by someone OTHER than the maintainer that added the LGTM label. This label indicates that given a quick pass of the comments this change is ready to merge - -### Stale Pull Requests - -After initial review feedback, if no updates have been made to the pull request for 1 week, the `stale` label will be added. If after another week there are still no updates it will be closed. Please re-open if/when you have made the proper adjustments. - -## Status of the Project - -This project is still under active development, so you might run into [issues](https://github.com/codefresh-io/plugins/issues). If you do, please don't be shy about letting us know, or better yet, contribute a fix or feature. \ No newline at end of file +# Codefresh plugins + +Codefresh Plugins are Docker images made especially for use in Codefresh freestyle steps. Each plugin facilitates a common task that would otherwise be difficult to achieve. +See each plugin readme for more info and usage instructions. + +## Plugins + +| Plugin| Description| Tags| +| --- | --- | --- | +| [Helm](plugins/helm/README.md) | Deploy Helm charts | `kubernetes` `helm`| +| [Codefresh Cli](plugins/codefresh-cli/README.md) | Operate on Codefresh resources | `cli` `codefresh`| +| [Slack](plugins/slack/README.md)| Send message to slack| `slack` `notify`| +| [Deploy to ECS](plugins/ecs-deploy/README.md)| Deploy docker image to ECS| `ecs` `deploy` `containers` `aws` | +| [Deploy Kompose](plugins/kompose/README.md)| Deploy Docker Compose to Kubernetes cluster with Kubernetes [Kompose](http://kompose.io) | `docker` `docker-compose` `kompose` `deploy` `kubernetes` | +| [GitHub PR](plugins/github-pr/README.MD)| Operates on pull requests on GitHub | `github` `pull-request` | +| [Run Jenkins Jobs](plugins/run-jenkins-job/README.md)| Run jenkins job from codefresh pipeline| `jenkins` `job`| +| [Deploy to DCOS](plugins/dcos-app-deploy/README.md) | Deploy application image to DC/OS cluster | `dcos` `deploy` `containers` | +| [Interact with Jira](plugins/jira/README.md) | Interact with Jira from codefresh pipelines| `jira` `workflow`| +| [Makisu](plugins/makisu/README.md) | Building images using the Makisu tool | `makisu` `uber`| +| [release to npm](plugins/release-to-NPM/README.md) | Release npm modules from a pipeline | `npm` | +| [Twistlock](plugins/cfstep-twistlock) | Security scanning of docker images using Twistlock | `security` | +| [Clair](plugins/clair/README.md) | Security scanning of Docker images using Clair | `security` | +| [Import Docker Images](plugins/import-docker-images/README.md) | Import Docker images metadata into Codefresh| `docker` `codefresh`| +| [Google KMS](plugins/google-kms/README.md) | Encryption/Decryption with Google KMS| `KMS` `codefresh`| +| [Github Release](plugins/github-release/README.md) | Managing GitHub releases | `github` `release`| +| [Google GKE](plugins/gke/README.md) | GKE Clusters | `GKE` `codefresh`| +| [Vault](plugins/vault/README.md) | Export Vault Key/Value pairs as ENV variables | `Vault` `codefresh`| diff --git a/docs/plugin.md b/docs/plugin.md index ef390a52..3fab6293 100644 --- a/docs/plugin.md +++ b/docs/plugin.md @@ -6,7 +6,7 @@ then they can be packaged into versioned archives to be deployed. This document explains the plugin format, and provides basic guidance for building plugins. -## The Plugin File Structure +## Plugin File Structure A plugin is organized as a collection of files inside of a directory. The directory name is the name of the plugin (without versioning information). Thus, @@ -17,7 +17,7 @@ Inside of this directory, Codefresh will expect a structure that matches this: ``` kube-deploy/ plugin.yaml # A YAML file containing information about the plugin - LICENSE # OPTIONAL: A plain text file containing the license for the chart + LICENSE # OPTIONAL: A plain text file containing the license for the plugin README.md # OPTIONAL: A human-readable README file NOTES.md # OPTIONAL: A plain text file containing short usage notes ``` @@ -50,6 +50,8 @@ volumes: - name: The volume name required: true | false (default "false") description: A short description for attached volume +context: + - kind: Codefresh context kind to inject automatically to the plugin ``` @@ -93,4 +95,4 @@ cannot be overridden. ### Predefined Volumes and Files - `/codefresh/volume` - same volume mounted to all steps running in Codefresh pipeline -- `/codefresh/volume/env_vars_to_export` - a placeholder file to filled with **exported** environment variables; any exported variable can be used in subsequent pipeline steps \ No newline at end of file +- `/codefresh/volume/env_vars_to_export` - a placeholder file to be filled with **exported** environment variables; any exported variable can be used in subsequent pipeline steps \ No newline at end of file diff --git a/dynamic-catalog.md b/dynamic-catalog.md new file mode 100644 index 00000000..1ce711d4 --- /dev/null +++ b/dynamic-catalog.md @@ -0,0 +1,30 @@ +# **Catalog** +Created at ** Thu Dec 27 2018 22:37:47 GMT+0000 (Coordinated Universal Time)** + + +| Plugin Name | Image | Description | source | Tags | +| ------------- |-------------| ---- |-----|----| + | azure-builder | codefresh/cf-azure-builder | Docker build in Azure ACR | https://github.com/codefresh-io/azure-build-plugin | **`docker`** **`azure`** | + | clair | codefresh/klar:master | Scan an image with Clair | https://github.com/optiopay/klar | **`clair`** **`security`** | + | codefresh-cli | codefresh/cli | Operate on Codefresh resources | https://github.com/codefresh-io/codefresh | **`cli`** | + | dcos-app-deploy | codefresh/cf-deploy-dcos | Deploy an application on dcos cluster | https://github.com/codefresh-io/cf-deploy-dcos | **`dcos`** **`deploy`** **`deployment`** | + | docker-service | codefresh/docker-service | Codefresh docker-service plugin | https://github.com/codefresh-io/docker-service | **`docker`** **`docker-machine`** **`docker-compose`** | + | ecs-deploy | codefresh/ecs | Release a Helm chart (update or install) | https://github.com/codefresh-io/cf-deploy-ecs.git | **`ecs`** **`deploy`** **`containers`** | + | github-pr | codefresh/github-pr-plugin | Operates on GitHub pull requests | https://github.com/codefresh-io/github-pr-plugin | **`docker`** **`github`** **`pull-request`** | + | gitsubmodules | codefresh/cfstep-gitsubmodules | Update git submodules | https://github.com/codefresh-io/cfstep-gitsubmodules | **`git`** **`submodules`** | + | gke | codefresh/plugin-gke | Codefresh gke plugin | https://github.com/codefresh-io/plugin-gke | **`kubernetes`** **`gke`** **`gcloud`** | + | helm | codefresh/cfstep-helm | Release a Helm chart (update or install) | https://github.com/codefresh-contrib/cfplugin-step | **`helm`** **`kubernetes`** | + | helm-legacy | codefresh/plugin-helm | Release a Helm chart (update or install). There is a new Helm plugin with added capabilities, we are keeping this plugin as is for backward-compatibility. The new plugin is at /incubator/helm (https://github.com/codefresh-io/plugins/tree/master/incubator/helm) | https://github.com/codefresh-io/cf-plugin-helm | **`helm`** **`kubernetes`** | + | import-docker-images | codefresh/import-images | Import metadata for existing Docker images into Codefresh | https://github.com/codefresh-io/cf-import-image | **`docker`** | + | jira | otomato/jira-cli | Update a Jira ticket | https://github.com/codefreshdemo/jira-cli-docker | **`jira`** | + | kompose | codefresh/plugin-helm | Release a Docker Compose to Kubernetes | https://github.com/codefresh-io/cf-kompose-plugin | **`docker-compose`** **`docker`** **`kompose`** **`kubernetes`** | + | release-to-NPM | | | | | + | run-jenkins-job | codefresh/run-jenkins-jobs | Run jenkins job from codefresh pipeline | https://github.com/codefresh-io/cf-run-jenkins-jobs | **`docker`** **`jenkins`** | + | sendgrid | codefresh/sendgridplugin | Send e-mail via Sendgrid | https://github.com/codefresh-io/sendgridplugin | **`mail`** **`sendgrid`** | + | slack | codefresh/slack-message-sender | Send message to slack channel | https://github.com/codefresh-io/slack-message-sender | **`slack`** | + | slack-notifier | codefresh/slacknotifier | Send message to slack channel | https://github.com/codefresh-io/slack-notifier | **`slack`** | + | testplugin | codefresh/github-pr-plugin | Creates GitHub pull request | https://github.com/codefresh-io/github-pr-plugin | **`testplugin`** | + | testplugin1 | codefresh/github-pr-plugin | test plugin1! | https://github.com/codefresh-io/github-pr-plugin | **`testplugin1`** | + | twilio | codefresh/twilioplugin | Send SMS message via Twilio | https://github.com/codefresh-io/twillio-plugin | **`sms`** **`twilio`** | + | versioner | docker.io/codefresh/versioner | semver versions builder | https://github.com/codefresh-io/cf-plugin-versioner.git | **`versioner 1.0`** | + diff --git a/stable/relese-to-NPM/NOTES.md b/plugins/anchore/NOTES.txt similarity index 100% rename from stable/relese-to-NPM/NOTES.md rename to plugins/anchore/NOTES.txt diff --git a/plugins/anchore/README.md b/plugins/anchore/README.md new file mode 100644 index 00000000..a7e5c7f6 --- /dev/null +++ b/plugins/anchore/README.md @@ -0,0 +1,77 @@ +# Codefresh Anchore Plugin + +Anchore is a service that analyzes Docker images and generates a detailed manifest of the image, a virtual ‘bill of materials’ that includes official operating system packages, unofficial packages, configuration files, and language modules and artifacts. Anchore policies can they be defined to govern security vulnerabilities, package whitelists and blacklists, configuration file contents, presence of credentials in image, manifest changes, exposed ports or any user defined checks. These policies can be deployed site wide or customized for specific images or categories of applications. + +For more information view the github repo here: https://github.com/anchore/anchore-engine + +## Prerequisites + +- Codefresh subscription +- Running Anchore Engine service + +### Reference + +- Example `codefresh.yml`: https://raw.githubusercontent.com/valancej/plugins/master/plugins/anchore/codefresh.yml +- Github repo containing Dockerfile: https://github.com/valancej/node_critical_fail +- Anchore Documentation: https://anchore.freshdesk.com/support/home +- Anchore CLI Image: https://hub.docker.com/r/anchore/engine-cli/ + +## Example + +In this example, we will scan an image built by Codefresh. Depending on the result of the Anchore policy evaluation, we will choose to push the image to Dockerhub or not. + +### Setup + +The example setup is described below. + +### Environment Variables + +These environment variables can be set within Codefresh pipeline configuration. + +Name|Required|Description +---|---|--- +ANCHORE_CLI_URL|Yes|The address of the Anchore server +ANCHORE_CLI_USER|Yes|Anchore account name +ANCHORE_CLI_PASS|Yes|Anchore account password +ANCHORE_FAIL_ON_POLICY|No|Fail build if policy evaluation fails +QA_IMAGE|No|Image built and scanned +dockerhubUsername|No|Dockerhub account name +dockerhubPassword|No|Dockerhub account password + +### Codefresh.yml + +```yaml +version: '1.0' +steps: + MyDockerImage: + title: Building Docker Image + type: build + image_name: ${{QA_IMAGE}} + working_directory: ./ + tag: latest + dockerfile: Dockerfile + metadata: + set: + - QA: Pending Anchore scan before push to Dockerhub.. + ScanMyImage: + title: Scanning Docker Image + image: anchore/engine-cli:latest + commands: + - echo "Scanning image with Anchore" + - anchore-cli image add ${{QA_IMAGE}} + - echo "Waiting for analysis to complete" + - anchore-cli image wait ${{QA_IMAGE}} + - echo "Analysis complete" + - if [ "${{ANCHORE_FAIL_ON_POLICY}}" == "true" ] ; then anchore-cli evaluate check ${{QA_IMAGE}}; fi + PushImage: + title: Pushing Docker Image + description: Pushing Docker Image to Dockerhub... + type: push + candidate: '${{MyDockerImage}}' + image_name: jvalance/node_critical_fail + tag: latest + registry: docker.io + credentials: + username: '${{dockerhubUsername}}' + password: '${{dockerhubPassword}}' +``` \ No newline at end of file diff --git a/plugins/anchore/codefresh.yml b/plugins/anchore/codefresh.yml new file mode 100644 index 00000000..5b5aff51 --- /dev/null +++ b/plugins/anchore/codefresh.yml @@ -0,0 +1,33 @@ +version: '1.0' +steps: + MyDockerImage: + title: Building Docker Image + type: build + image_name: ${{QA_IMAGE}} + working_directory: ./ + tag: latest + dockerfile: Dockerfile + metadata: + set: + - QA: Pending Anchore scan.. + ScanMyImage: + title: Scanning Docker Image + image: anchore/engine-cli:latest + commands: + - echo "Scanning image with Anchore" + - anchore-cli image add ${{QA_IMAGE}} + - echo "Waiting for analysis to complete" + - anchore-cli image wait ${{QA_IMAGE}} + - echo "Analysis complete" + - if [ "${{ANCHORE_FAIL_ON_POLICY}}" == "true" ] ; then anchore-cli evaluate check ${{QA_IMAGE}}; fi + PushImage: + title: Pushing Docker Image + description: Pushing Docker Image to Dockerhub... + type: push + candidate: '${{MyDockerImage}}' + image_name: jvalance/node_critical_fail + tag: latest + registry: docker.io + credentials: + username: '${{dockerhubUsername}}' + password: '${{dockerhubPassword}}' \ No newline at end of file diff --git a/plugins/aqua-scan/Dockerfile b/plugins/aqua-scan/Dockerfile new file mode 100644 index 00000000..9f511082 --- /dev/null +++ b/plugins/aqua-scan/Dockerfile @@ -0,0 +1,50 @@ +FROM ubuntu:xenial + +ENV LANG C.UTF-8 + +RUN { \ + echo '#!/bin/sh'; \ + echo 'set -e'; \ + echo; \ + echo 'dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"'; \ + } > /usr/local/bin/docker-java-home && \ + chmod +x /usr/local/bin/docker-java-home + +RUN apt-get update && apt-get install -y --no-install-recommends \ + bzip2 \ + unzip \ + xz-utils \ + apt-transport-https \ + ca-certificates \ + curl \ + software-properties-common \ + python3-openssl && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \ + add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" && \ + apt-get update && apt-get install -y --no-install-recommends \ + docker-ce=17.09.0~ce-0~ubuntu && \ + apt-get install -y \ + openjdk-8-jre \ + ; \ + rm -rf /var/lib/apt/lists/*; \ + \ + [ "$JAVA_HOME" = "$(docker-java-home)" ]; \ + \ + update-alternatives --get-selections | awk -v home="$JAVA_HOME" 'index($3, home) == 1 { $2 = "manual"; print | "update-alternatives --set-selections" }'; \ + update-alternatives --query java | grep -q 'Status: manual' && \ + mkdir /packages && \ + curl -o /packages/twistcli https://cdn.twistlock.com/support/twistcli && \ + curl -o /packages/nexus-iq-cli-1.38.0-02.jar https://download.sonatype.com/clm/scanner/nexus-iq-cli-1.38.0-02.jar + +COPY scripts /scripts + +RUN chmod +x -R /packages +RUN chmod +x -R /scripts + +WORKDIR /scripts + +ENTRYPOINT ["/usr/bin/python3"] +CMD [""] diff --git a/plugins/aqua-scan/LICENSE.md b/plugins/aqua-scan/LICENSE.md new file mode 100644 index 00000000..d39b0063 --- /dev/null +++ b/plugins/aqua-scan/LICENSE.md @@ -0,0 +1,7 @@ +© 2017 Steelcase Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/plugins/aqua-scan/README.md b/plugins/aqua-scan/README.md new file mode 100644 index 00000000..150dcd3c --- /dev/null +++ b/plugins/aqua-scan/README.md @@ -0,0 +1,105 @@ +# Security Scanning Tools [![Codefresh build status]( https://g.codefresh.io/api/badges/build?repoOwner=SC-TechDev&repoName=docker-security-scanner&branch=master&pipelineName=docker-security-scanner&accountName=sctechdevservice&type=cf-1)]( https://g.codefresh.io/repositories/SC-TechDev/docker-security-scanner/builds?filter=trigger:build;branch:master;service:59e62c5410e3d100019e7f3d~docker-security-scanner) + +Docker image which invokes security script using TwistCLI (Nexus coming soon) + +### Prerequisites: + +Codefresh Subscription (Dedicated Infrastructure) - https://codefresh.io/ + +Twistlock Subscription - https://www.twistlock.com/ + +### Documentation: + +Twistlock CLI: https://twistlock.desk.com/customer/en/portal/articles/2875595-twistcli?b_id=16619 + +Nexus IQ CLI: TBD + +## Script Library + +### twistlock.py + +Executes TwistCLI to scan Docker image given. + +### options + +To use an ENVIRONMENT VARIABLE you need to add the variables to your Codefresh Pipeline and also to your codefresh.yaml. + + +Example `codefresh.yml` build is below with required ENVIRONMENT VARIABLES in place. + + +| ENVIRONMENT VARIABLE | SCRIPT ARGUMENT | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|----------------------------|--------------------------------------|----------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------| +| CF_METADATA | [ -c, --cf_metadata ] | null | boolean | No | In combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation | +| TL_CONSOLE_HOSTNAME | [ -C, --tl_console_hostname ] | null | string | Yes | hostname/ip | +| TL_CONSOLE_PORT | [ -P, --tl_console_port ] | null | string | Yes | port | +| TL_CONSOLE_USERNAME | [ -U, --tl_console_username ] | null | string | Yes | username | +| TL_CONSOLE_PASSWORD | [ -X, --tl_console_password ] | null | string | Yes | password | +| TL_ONLY | [ -Z, --tl_only ] | null | boolean | Yes | Twistlock Console Only (Required for now Nexus TBD) | +| TL_TLS_ENABLED | [ -T, --tl_tls_enabled ] | null | boolean | No | enable TLS | +| TL_HASH | [ -H, --tl_hash ] | [ sha1 ] | string | No | [ md5, sha1, sha256 ] hashing algorithm | +| TL_UPLOAD | [ -R, --tl_upload ] | null | boolean | No | ( ignores all options below if set and only returns report url ) uploads report to Twistlock to be used later via Twistlock API | +| TL_DETAILS | [ -D, --tl_details ] | null | boolean | No | prints an itemized list of each vulnerability found by the scanner | +| TL_ONLY_FIXED | [ -O, --tl_only_fixed ] | null | boolean | No | reports just the vulnerabilites that have fixes available | +| TL_COMPLIANCE_THRESHOLD | [ -M, --tl_compliance_threshold ] | null | string | No | [ low, medium, high ] sets the the minimal severity compliance issue that returns a fail exit code | +| TL_VULNERABILITY_THRESHOLD | [ -V, --tl_vulnerability_threshold ] | null | string | No | [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code | + +### codefresh.yml + +Codefresh Build Step to execute Twistlock scan. +All `${{var}}` variables must be put into Codefresh Build Parameters +codefresh.yml +```console + buildimage: + type: build + title: Build Runtime Image + dockerfile: Dockerfile + image_name: # Image you're building/scanning [repository/image] + tag: latest-cf-build-candidate + + nexus_iq_scan_build_stage: + type: composition + composition: + version: '2' + services: + imagebuild: + image: ${{buildimage}} + command: sh -c "exit 0" + labels: + build.image.id: ${{CF_BUILD_ID}} + composition_candidates: + scan_service: + image: sctechdev/docker-security-scanner + environment: + - TL_CONSOLE_HOSTNAME=${{TL_CONSOLE_HOSTNAME}} + - TL_CONSOLE_PORT=${{TL_CONSOLE_PORT}} + - TL_CONSOLE_USERNAME=${{TL_CONSOLE_USERNAME}} + - TL_CONSOLE_PASSWORD=${{TL_CONSOLE_PASSWORD}} + - TL_ONLY=${{TL_ONLY}} + command: twistlock.py -i "$$(docker inspect $$(docker inspect $$(docker ps -aqf label=build.image.id=${{CF_BUILD_ID}}) -f {{.Config.Image}}) -f {{.Id}} | sed 's/sha256://g')" + depends_on: + - imagebuild + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /var/lib/docker:/var/lib/docker + # Everything below this line is Optional for CF_METADATA + - '${{CF_VOLUME_NAME}}:/codefresh/volume' + add_flow_volume_to_composition: true + + export: + title: "Exporting variables..." + image: alpine + commands: + - echo "Exporting variables..." + + set_metadata: + title: "Setting metadata on image..." + image: alpine + commands: + - echo "Setting metadata on image..." + on_finish: + metadata: + set: + - '${{build_step.imageId}}': + - TwistlockSecurityReport: ${{TL_REPORT_URL}} +``` \ No newline at end of file diff --git a/plugins/aqua-scan/codefresh.yml b/plugins/aqua-scan/codefresh.yml new file mode 100644 index 00000000..31881bcb --- /dev/null +++ b/plugins/aqua-scan/codefresh.yml @@ -0,0 +1,41 @@ +version: '1.0' + +steps: + + buildimage: + type: build + description: image build step + dockerfile: Dockerfile + image_name: sctechdev/docker-security-scanner + tag: latest-cf-build-candidate + + push_image: + type: push + candidate: ${{buildimage}} + tag: latest + when: + branch: + only: + - master + push_image1: + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + + push_image_nexus_latest: + title: Push to Nexus Repo (latest) + type: push + candidate: ${{buildimage}} + tag: latest + registry: sonatype-docker-internal + when: + branch: + only: + - master + + push_image_neuxs_gitbranch_gitsha: + title: Push to Nexus Repo (gitbranch + gitsha) + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + registry: sonatype-docker-internal diff --git a/plugins/aqua-scan/plugin.yaml b/plugins/aqua-scan/plugin.yaml new file mode 100644 index 00000000..526a3162 --- /dev/null +++ b/plugins/aqua-scan/plugin.yaml @@ -0,0 +1,65 @@ +image: docker.io/sctechdev/docker-security-scanner +tag: master-c81e6d4 +version: 2.2 +description: Execute Twistlock image scan as build step +keywords: + - aqua 2.2 +home: https://hub.docker.com/r/sctechdev/docker-security-scanner/ +sources: + - https://github.com/SC-TechDev/docker-security-scanner +maintainers: + - name: Dustin Van Buskirk + email: dev@vanbuskirk.me + - name: Varun Tagore + email: rondevops@gmail.com +icon: A URL to an SVG or PNG image to be used as an icon (optional) +envs: + - name: CF_METADATA + type: required + description: Boolean; combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation + - name: TL_CONSOLE_HOSTNAME + type: required + description: Hostname or IP of Twistlock Console + - name: TL_CONSOLE_PORT + type: required + description: Port of Twistlock Console + - name: TL_CONSOLE_USERNAME + type: required + description: Username of Twistlock Console + - name: TL_CONSOLE_PASSWORD + type: required + description: Password of Twistlock Console User + - name: TL_ONLY + type: required + description: Twistlock Console Scan Only (No Nexus) + - name: TL_TLS_ENABLED + type: optional + description: Boolean; Enable TLS connection to Twistlock Console + - name: TL_HASH + type: optional + description: Hashing Algorithm to use + - name: TL_UPLOAD + type: optional + description: Upload report to Twistlock Console and return URL (Overrides all other options only returns URL) + - name: TL_DETAILS + type: optional + description: Prints an itemized list of each vulnerability found by the scanner + - name: TL_ONLY_FIXED + type: optional + description: reports just the vulnerabilites that have fixes available + - name: TL_COMPLIANCE_THRESHOLD + type: optional + description: [ low, medium, high ] sets the the minimal severity compliance issue that returns a fail exit code + - name: TL_VULNERABILITY_THRESHOLD + type: optional + description: [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code +volumes: + - name: /var/run/docker.sock:/var/run/docker.sock + required: true + description: Docker socket for DIND + - name: /var/lib/docker:/var/lib/docker + required: true + description: Docker lib access for DIND + - name: '${{CF_VOLUME_NAME}}:/codefresh/volume' + required: false + description: Volume required if setting Docker image metadata using Codefresh \ No newline at end of file diff --git a/plugins/aqua-scan/scripts/twistlock.py b/plugins/aqua-scan/scripts/twistlock.py new file mode 100644 index 00000000..50510ebf --- /dev/null +++ b/plugins/aqua-scan/scripts/twistlock.py @@ -0,0 +1,195 @@ +import sys +import subprocess +import time +import os +import getopt +import ssl +import re + +def main(argv): + try: + st_scanner_jar = '/packages/nexus-iq-cli-1.38.0-02.jar' + tl_scanner_exec = '/packages/twistcli' + cf_metadata = os.environ.get('CF_METADATA') + docker_image_id = os.environ.get('DOCKER_IMAGE_ID') + st_application_id = os.environ.get('NEXUS_IQ_APPLICATION_ID') + st_url = os.environ.get('NEXUS_IQ_URL') + st_username = os.environ.get('NEXUS_IQ_USERNAME') + st_password = os.environ.get('NEXUS_IQ_PASSWORD') + st_stage = os.environ.get('NEXUS_IQ_STAGE', 'Build') + tl_console_hostname = os.environ.get('TL_CONSOLE_HOSTNAME') + tl_console_port = os.environ.get('TL_CONSOLE_PORT') + tl_console_username = os.environ.get('TL_CONSOLE_USERNAME') + tl_console_password = os.environ.get('TL_CONSOLE_PASSWORD') + tl_only = os.environ.get('TL_ONLY') + tl_tls_enabled = os.environ.get('TL_TLS_ENABLED') + tl_hash = os.environ.get('TL_HASH', 'sha1') + tl_include_package_files = os.environ.get('TL_INCLUDE_PACKAGE_FILES') + tl_upload = os.environ.get('TL_UPLOAD') + tl_details = os.environ.get('TL_DETAILS') + tl_only_fixed = os.environ.get('TL_ONLY_FIXED') + tl_compliance_threshold = os.environ.get('TL_COMPLIANCE_THRESHOLD') + tl_vulnerability_threshold = os.environ.get('TL_VULNERABILITY_THRESHOLD') + java_home = os.environ.get('JAVA_HOME', '/usr/lib/jvm/java-8-openjdk-amd64') + java_keystore_password = os.environ.get('JAVA_KEYSTORE_PASSWORD', 'changeit') + opts, args = getopt.getopt(argv,"h:c:i:a:j:u:p:s:t:E:C:P:U:X:Z:J:K:T:H:F:R:D:O:M:V:", + ["help", "docker_image_id=", "cf_metadata", "st_application_id=", "st_scanner_jar=", "st_url=", "st_username=", "st_password=", "st_stage=", + "tl_scanner_exec=", "tl_console_hostname", "tl_console_port", "tl_console_username=", "tl_console_password=", "tl_only", + "tl_tls_enabled", "tl_hash", "tl_include_package_files", "tl_upload", "tl_details", "tl_only_fixed", "tl_compliance_threshold", + "tl_vulnerability_threshold", "java_home=", "java_keystore_password" + ] + ) + except getopt.GetoptError: + print('Unrecognized Argument! See arguments list using -h or --help. Ex. twistlock.py --help') + sys.exit(2) + for opt, arg in opts: + if opt == ("h","--help"): + print('twistlock.py --arg value or twistlock.py -a value') + print('-c --cf_metadata - Adds scanner info to Docker image metadata for Codefresh builds') + print('-i --docker_image_id [DOCKER_IMAGE_ID] - Docker Image ID short or long IDs accepted') + print('-a --st_application_id [NEXUS_IQ_APPLICATION_ID] - Applications ID in Nexus IQ') + print('-j --st_scanner_jar - Location of nexus-iq-cli*.jar file') + print('-u --st_username [NEXUS_IQ_USERNAME] - Nexus IQ Username') + print('-p --st_password [NEXUS_IQ_PASSWORD] - Password for Nexus IQ Username') + print('-s --st_url [NEXUS_IQ_URL] - Sonatype URL must be HTTPS with Valid Cert') + print('-t --st_stage [NEXUS_IQ_STAGE] - Sonatype Stage') + print('-E --tl_scanner_exec - Location of twistlock-scanner executable') + print('-C --tl_console_hostname [TL_CONSOLE_HOSTNAME] - Hostname/IP for Twistlock Console') + print('-P --tl_console_port [TL_CONSOLE_PORT] - Twistock Console port') + print('-U --tl_console_username [TL_CONSOLE_USERNAME] - Twistlock Console Username') + print('-X --tl_console_password [TL_CONSOLE_PASSWORD] - Password for Twistlock Console Username') + print('-Z --tl_only [TL_ONLY] - Run a stand-alone Twistlock scan') + print('-T --tl_tls_enabled [TL_TLS_ENABLED] - Enabled TLS/HTTPS for Twistlock scan') + print('-H --tl_hash [TL_HASH] - Specifies the hashing algorithm. Supported values are md5, sha1, and sha256') + print('-F --tl_include_package_files [TL_INCLUDE_PACKAGE_FILES] - List all packages in the image') + print('-R --tl_upload [TL_UPLOAD] - Whether to upload the scan result') + print('-D --tl_details [TL_DETAILS] - Prints an itemized list of each vulnerability found by the scanner') + print('-O --tl_only_fixed [TL_ONLY_FIXED] - Reports just the vulnerabilities that have fixes available') + print('-M --tl_compliance_threshold [TL_COMPLIANCE_THRESHOLD] - Sets the minimum severity compliance issue that returns a fail exit code') + print('-V --tl_vulnerability_threshold [TL_VULNERABILITY_THRESHOLD] - Sets the minimum severity vulnerability that returns a fail exit code') + print('-J --java_home [JAVA_HOME] - Java Home Directory (no trailing /)') + print('-K --java_keystore_password [JAVA_KEYSTORE_PASSWORD] - Java Keystore Password') + sys.exit() + elif opt in ("-c", "--cf_metadata"): + cf_metadata = arg + elif opt in ("-i", "--docker_image_id"): + docker_image_id = arg + elif opt in ("-a", "--st_application_id"): + st_application_id = arg + elif opt in ("-j", "--st_scanner_jar"): + st_scanner_jar = arg + elif opt in ("-s", "--st_url"): + st_url = arg + elif opt in ("-u", "--st_username"): + st_username = arg + elif opt in ("-p", "--st_password"): + st_password = arg + elif opt in ("-t", "--st_stage"): + st_stage = arg + elif opt in ("-E", "--tl_scanner_exec"): + tl_scanner_exec = arg + elif opt in ("-C", "--tl_console_hostname"): + tl_console_hostname = arg + elif opt in ('-P', "--tl_console_port"): + tl_console_port = arg + elif opt in ("-U", "--tl_console_username"): + tl_console_username = arg + elif opt in ('-X', "--tl_console_password"): + tl_console_password = arg + elif opt in ('-Z', "--tl_only"): + tl_only = arg + elif opt in ('-T', "--tl_tls_enabled"): + tl_tls_enabled = arg + elif opt in ('-H', "--tl_hash"): + tl_hash = arg + elif opt in ('-F', "--tl_include_package_files"): + tl_include_package_files = arg + elif opt in ('-R', "--tl_upload"): + tl_upload = arg + elif opt in ('-D', "--tl_details"): + tl_details = arg + elif opt in ('-O', "--tl_only_fixed"): + tl_only_fixed = arg + elif opt in ('-M', "--tl_compliance_threshold"): + tl_compliance_threshold = arg + elif opt in ('-V', "--tl_vulnerability_threshold"): + tl_vulnerability_threshold = arg + elif opt in ('-J', "--java_home"): + java_home = arg + elif opt in ('-K', "--java_keystore_password"): + java_keystore_password = arg + + # Determine if TLS is required + if not (tl_only or tl_tls_enable): + # Download and store Twistlock Console site cert + cert = ssl.get_server_certificate((tl_console_hostname, tl_console_port)) + cert, file=open("twistlock.cer", "w") + + # Run stand-alone Twistlock Scan + if tl_only: + + # Determine Protocol + tl_console_protocol = 'https' if tl_tls_enabled else 'http' + + # Base twistcli commnad to scan images + twistcli_base_command = '/packages/twistcli images scan' + + # Required twistcli options to successfully scan image + twistcli_required_options = ("--address '{}://{}:{}' --user '{}' --password '{}' --hash '{}'" + .format(tl_console_protocol, tl_console_hostname, tl_console_port, tl_console_username, tl_console_password, tl_hash)) + + # Optional twistcli options + options = [] + if tl_include_package_files: + options.append("--include-package-files") + if tl_upload: + options.append("--upload") + if tl_details: + options.append("--details") + if tl_compliance_threshold: + options.append("--compliance-threshold '{}'".format(tl_compliance_threshold)) + if tl_vulnerability_threshold: + options.append("--vulnerability-threshold '{}'".format(tl_vulnerability_threshold)) + twistcli_optional_options = ' '.join(options) + + # Concatenate twistcli executable with command + twistcli_exec = ' '.join([twistcli_base_command, twistcli_required_options, twistcli_optional_options, docker_image_id]) + # Execute command but pipe stdout to variable and parse for Twistlock URL + if cf_metadata: + proc = subprocess.Popen(twistcli_exec, shell=True, stdout=subprocess.PIPE) + stdout = proc.communicate()[0].decode('utf-8').strip('\n') + tl_report_url = ''.join(re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', stdout)) + with open('/codefresh/volume/env_vars_to_export', 'a') as f: + print('Twistlock Report: ' + tl_report_url) + f.write('TL_REPORT_URL=' + tl_report_url) + f.close() + # Execute command and send stdout to console + else: + proc = subprocess.Popen(twistcli_exec, shell=True) + stdout, stderr = proc.communicate() + if proc.returncode != 0: + sys.exit(1) + + else: + + # Import site cert into java keystore + command = ['keytool -importcert -noprompt -file twistlock.cer -alias twistlock -storepass {} -keystore {}/jre/lib/security/cacerts' + .format(java_keystore_password, java_home) + ] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + + # Start Docker + command = ['for i in {1..5}; do service docker start && break || sleep 15; done'] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + + # Run Twistlock Scan and send file to Sonatype + command = ["java -cp {} com.sonatype.insight.scan.cli.TwistlockPolicyEvaluatorCli -i {} -a '{}:{}' -s '{}' --twistlock-scanner-executable {} --twistlock-console-url https://{}:{} --twistlock-console-username {} --twistlock-console-password '{}' --stage '{}' {}" + .format(st_scanner_jar, st_application_id, st_username, st_password, st_url, tl_scanner_exec, tl_console_hostname, tl_console_port, tl_console_username, tl_console_password, st_stage, docker_image_id) + ] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + +if __name__ == "__main__": + main(sys.argv[1:]) \ No newline at end of file diff --git a/plugins/azure-builder/README.md b/plugins/azure-builder/README.md new file mode 100644 index 00000000..9bf29f80 --- /dev/null +++ b/plugins/azure-builder/README.md @@ -0,0 +1,56 @@ +# Docker build using Azure ACR + + +## Run locally +`docker run -it codefresh/cf-azure-builder` +``` +NAME: + cf-azure-builder + +DESCRIPTION: + Build + +## Mandatory Parameters: + + AUTH - Authentication mode. By default it is Azure user credentials. + (Use 'service-princpal' value in case you want to authenticate using service principal) + USER - Azure user name (not needed if authentication mode is service-principal) + IMAGE - Image name + TAG - Tag name + ACR_NAME - ACR registry name + APP_ID - Azure service principal application id (only needed if authentication mode is service-principal) + PASSWORD - Azure user\service principal password + TENANT - Azure ad tenant id (only needed if authentication mode is service-principal) + DOCKERFILE_PATH - Dockerfile path (default - working_dir/Dockerfile) + +## Output Variables + + AZURE_IMAGE - Azure image full name in ACR that can be used in later step + +## Usage Example: + +## service principal + +version: '1.0' +steps: + cf-az-build: + image: codefresh/cf-azure-builder + environment: + - AUTH=service-principal + - IMAGE= + - TAG= + - ACR_NAME= + - APP_ID= + - PASSWORD= + - TENANT= + - DOCKERFILE_PATH= + +## user credentials + +image: 'codefresh/cf-azure-builder' +environment: + - IMAGE= + - TAG= + - ACR_NAME= + - USER= + - PASSWORD= diff --git a/plugins/azure-builder/plugin.yaml b/plugins/azure-builder/plugin.yaml new file mode 100644 index 00000000..cdfb9516 --- /dev/null +++ b/plugins/azure-builder/plugin.yaml @@ -0,0 +1,44 @@ +image: codefresh/cf-azure-builder +tag: latest +version: 0.1.0 +description: Docker build in Azure ACR +keywords: + - docker + - azure +home: https://github.com/codefresh-io/azure-build-plugin +sources: + - https://github.com/codefresh-io/azure-build-plugin +maintainers: # (optional) + - name: Amir Gabay + email: amir.gabay@codefresh.io +icon: https://cdn-images-1.medium.com/max/1600/1*c9mvlJQ0o-zQbHM3SL8zkg.png + envs: + - name: AUTH + type: optional + description: Only if authentication mode is by service principal + - name: USER + type: required + description: azure user name (not need in service principal authentication mode) + - name: IMAGE + type: required + description: image name + - name: TAG + type: required + description: tag name + - name: ACR_NAME + type: required + description: acr registry name + - name: APP_ID + type: required + description: azure service principal application id (service principal authentication mode only) + - name: PASSWORD + type: required + description: azure service principal password or user password (based on authentication mode) + - name: TENANT + type: required + description: azure ad tenant id (service principal authentication mode only) + - name: DOCKERFILE_PATH + type: required + description: dockerfile path (default - working_dir/Dockerfile) + + diff --git a/plugins/cfstep-paclair/README.md b/plugins/cfstep-paclair/README.md new file mode 100644 index 00000000..b63f1a93 --- /dev/null +++ b/plugins/cfstep-paclair/README.md @@ -0,0 +1,129 @@ +# cfstep-paclair [![Codefresh build status]( https://g.codefresh.io/api/badges/pipeline/codefresh-inc/codefresh-contrib%2Fcfstep-paclair%2Fcfstep-paclair?branch=master&type=cf-1)]( https://g.codefresh.io/repositories/codefresh-contrib/cfstep-paclair/builds?filter=trigger:build;branch:master;service:5bbe7af8a3686e081e4e1b91~cfstep-paclair) + +Custom Docker image to support clair image scanning from Codefresh pipeline + +### OOTB Step DockerHub + +https://hub.docker.com/r/codefresh/cfstep-paclair/ + +### OOTB Step Project Repository + +https://github.com/codefresh-contrib/cfstep-paclair + +### Prerequisites: + +Codefresh Subscription - https://codefresh.io/ + +Running Clair Instance - +Helm Chart is available to install here: https://github.com/coreos/clair/tree/master/contrib/helm + +### Documentation: + +paclair: https://github.com/yebinama/paclair + +### Tested Registries + +Codefresh Registry - No special setup required. + +Username is your Codefresh Username and Docker Registry keys can be created here https://g.codefresh.io/user/settings + +ECR - Requires AWS CLI credentials with access to ECR. +`REGISTRY=ecr` will find the proper ECR registry using your credentials and image. + +AWS CLI Credentials required for ECR: +https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html + +| ENVIRONMENT VARIABLE | +| --------------------- | +| AWS_ACCESS_KEY_ID | +| AWS_DEFAULT_REGION | +| AWS_SECRET_ACCESS_KEY | + +Registries with Basic auth and Token based auth should work. + +### Full List of Options + +To use an ENVIRONMENT VARIABLE you need to add the variables to your Codefresh Pipeline and also to your codefresh.yml. + +Example `codefresh.yml` build is below with required ENVIRONMENT VARIABLES in place. + +| ENVIRONMENT VARIABLE | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|----------------------------|----------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------| +| API_PREFIX | null | string | No | Prefix for API to Docker Registry | +| CF_ACCOUNT | null | string | No | Codefresh Account Name | +| CLAIR_URL | null | string | Yes | https://clair.domain.com:6060 | +| IMAGE | null | string | Yes | Docker Image Name | +| PROTOCOL | https | string | No | Docker Registry Protocol | +| REGISTRY | r.cfcr.io | string | No | For ECR use `ecr` else use domain name for Docker Registry | +| REGISTRY_PASSWORD | null | string | Yes | Docker Registry Password | +| REGISTRY_USERNAME | null | string | Yes | Docker Registry Username | +| SEVERITY_THRESHOLD | null | string | No | critical, high, medium, low, negligible, unknown | +| TOKEN | null | string | No | Docker Registry Auth Token | +| TOKEN_TYPE | Bearer | string | No | Docker Registry Auth Token Type | +| TOKEN_URL | null | string | No | Docker Registry Auth Token URL | +| TAG | null | string | Yes | Docker Image Tag | + +### SEVERITY_THRESHOLD + +If variable is set step will check that the threshold is not met or exceeded. + +For example, high would fail your build if you had high or critical vulnerabilties on your Docker image. + +### codefresh.yml + +Codefresh Build Step to execute Clair scan. +All `${{var}}` variables must be put into Codefresh Build Parameters +codefresh.yml + +``` console +version: '1.0' +steps: + BuildingDockerImage: + title: Building Docker Image + type: build + image_name: codefresh/demochat # Replace with your Docker image name + working_directory: ./ + dockerfile: Dockerfile + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + CheckClair: + image: codefresh/cfstep-paclair:3.1.0 + environment: + - CF_ACCOUNT=dustinvanbuskirk + - IMAGE=example-voting-app/worker # Replace with your Docker image name + - TAG=${{CF_BRANCH_TAG_NORMALIZED}} + on_success: # Execute only once the step succeeded + metadata: # Declare the metadata attribute + set: # Specify the set operation + - ${{BuildingDockerImage.imageId}}: # Select any number of target images + - SECURITY_SCAN: true + on_fail: # Execute only once the step failed + metadata: # Declare the metadata attribute + set: # Specify the set operation + - ${{BuildingDockerImage.imageId}}: # Select any number of target images + - SECURITY_SCAN: false + ArchiveReport: + image: mesosphere/aws-cli + commands: + - aws s3 cp ./reports/clair-scan-example-voting-app-worker-${{CF_BRANCH_TAG_NORMALIZED}}.html s3://${{S3_BUCKETNAME}}/${{CF_BUILD_ID}}/clair-scan-example-voting-app-worker-${{CF_BRANCH_TAG_NORMALIZED}}.html --acl public-read + on_success: + metadata: + set: + - ${{BuildingDockerImage.imageId}}: + - CLAIR_REPORT: "https://s3.amazonaws.com/${{S3_BUCKETNAME}}/${{CF_BUILD_ID}}/clair-scan-example-voting-app-worker-${{CF_BRANCH_TAG_NORMALIZED}}.html" +``` + +The HTML report is stored in `./reports/clair-scan-{image name}-{image tag}.html` +Any `/` characters in `{image name}` are replaced with `-` + +Optional Storage Step Variables for AWS S3: + +| ENVIRONMENT VARIABLE | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|----------------------------|----------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------| +| AWS_ACCESS_KEY_ID | null | string | No | AWS Access Key of S3 Bucket | +| AWS_DEFAULT_REGION | null | string | Yes | AWS Region of S3 Bucket | +| AWS_SECRET_ACCESS_KEY | null | string | Yes | AWS Secret Key of S3 Bucket | +| S3_BUCKETNAME | null | string | Yes | Name of S3 Bucket to Store Reports | + +### Notes + +Not yet supporting manual Cert validation. Coming soon along with tests. \ No newline at end of file diff --git a/plugins/cfstep-twistlock/README.md b/plugins/cfstep-twistlock/README.md new file mode 100644 index 00000000..cac9587d --- /dev/null +++ b/plugins/cfstep-twistlock/README.md @@ -0,0 +1,216 @@ +# Codefresh Twistlock Plugin + +Dockerhub repo: https://hub.docker.com/r/codefresh/cfstep-twistlock/tags/ + +The Docker image uses the Twistlock API v2.3: https://twistlock.desk.com/customer/en/portal/articles/2912404-twistlock-api-2-3 + +Plugin that allow users to perform Twistlock Security Scans on images. + +This plugin **does not** require access to Docker Daemon. + + +## Prerequisites: + +- Codefresh Subscription - https://codefresh.io/ +- Twistlock Subscription - https://www.twistlock.com/ + +## Options +These options are set as Environment Variables at your pipeline (either at Pipeline configuraion, and/or Step definition) +To use an ENVIRONMENT VARIABLE you need to add the variables to your Codefresh Pipeline and also to your codefresh.yaml. + +| ENVIRONMENT VARIABLE | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|--|--|--|--|--| +| TL_CONSOLE_HOSTNAME | null | string | Yes | hostname/ip | +| TL_CONSOLE_PORT | null | string | Yes | port | +| TL_CONSOLE_USERNAME | null | string | Yes | username | +| TL_CONSOLE_PASSWORD | null | string | Yes | password | +| TL_COMPLIANCE_THRESHOLD | null | string | Yes | [ low, medium, high, critical ] sets the the minimal severity compliance issue that returns a fail exit code | +| TL_VULNERABILITY_THRESHOLD | null | string | Yes | [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code | +| TL_REGISTRY | null | string | Yes | Registry URL. (e.g.: docker.io, cfcr.io). This should match the Registry URL set at Twistlock Console | +| TL_IMAGE_NAME | null | string | Yes | The full image name (excluding the registry URL) (e.g.: myrepo/myimage) | +| TL_IMAGE_TAG | null | string | Yes | The tag of the image to scan. | + +> **Threshold description** +> +> - low: the most **restrictive**. When thresholds are set to this level, the scanning process will fail with any issue or vulnearability found. +> - critical: the most **permissive**. When thresholds are set to this level, the scanning process will fail only if a critical issue or vulnearability is found (or a combination of lower level vulnerabilities that summed up result in a risk score higher than 1000). + + + +## How to use it (examples) + +Summary: in this example, we're going to scan an image built by Codefresh. + +The image's Dockerfile is defined in this sample repo: https://github.com/francisco-codefresh/twistlock_demo + +In order for this to work, the registry to scan must be previously added to Twistlock Console. + +Once the security scan finishes, we annote the image based on the Security Report created by Twistlock. + +In our example pipeline, if the compliance and vulnerability thresholds are not exceeded (which means the scan doesn't fail), then, we push the resulting image to our final, curated, registry. + +### Preparation step: Set up a pipeline with the following configuration +Now, create a pipeline associated to your repo, in this case, our demo repo is "twistlock_demo" (mentioned above) + +#### Environment Variables (configured at Pipeline Configuration): + +``` +TL_CONSOLE_HOSTNAME=169.254.169.254 +TL_CONSOLE_PORT=8083 +TL_CONSOLE_USERNAME=myuser +TL_CONSOLE_PASSWORD=mypassword +TL_COMPLIANCE_THRESHOLD=critical +TL_VULNERABILITY_THRESHOLD=critical +``` + +For this example, we're being permissive (critical for both thresholds). Of course those values can be set to any of the other options. + +### Example 1 - Scanning an image from Codefresh Docker registry +This is a great way to take advantage of the built-in registry provided by Codefresh (for free). +Since evey image built in Codefresh is automatically pushed to this registry, you don't need to worry for explicitly pushising the image to scan. + + +#### Configure the Codefresh registry in Twistlock +In your Twistlock dashboard go to `#!/defend/vulnerabilities/registry` . And add a new "registry settings" record. + +These are the settings used for Codefresh Private Registry: + +- Version: Docker Registry v2 +- Registry: https://r.cf-cd.com *(notice this is not the common r.cfcr.io domain)* +- Repository name: / (e.g.: francisco-codefresh/myimage) +- Tag: +- Username: +- Password: (you can generate one at https://g.codefresh.io/user/settings) + +#### Set up a pipeline with the following configuration + +**Environment Variables (configured at Pipeline Configuration):** + +``` +TL_CONSOLE_HOSTNAME=169.254.169.254 +TL_CONSOLE_PORT=8083 +TL_CONSOLE_USERNAME=my_tl_user +TL_CONSOLE_PASSWORD=my_tl_password +TL_COMPLIANCE_THRESHOLD=critical +TL_VULNERABILITY_THRESHOLD=critical +``` + +For this example, we're being permissive (`critical` for both thresholds). Of course those values can be set to any of the other options. + +**Pipeline YAML (codefresh.yml)** + +```yaml +version: '1.0' +steps: + BuildingDockerImage: + title: Building Docker Image + type: build + image_name: franciscocodefresh/twistlockdemo + tag: '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + + TLScan: + title: Twistlock Scan + image: codefresh/cfstep-twistlock + environment: + - TL_REGISTRY=https://r.cf-cd.com + - TL_IMAGE_NAME=francisco-codefresh/franciscocodefresh/twistlockdemo + - TL_IMAGE_TAG=${{CF_SHORT_REVISION}} + on_success: + metadata: + set: + - ${{BuildingDockerImage.imageId}}: + - SECURITY_SCAN: true + on_fail: + metadata: + set: + - ${{BuildingDockerImage.imageId}}: + - SECURITY_SCAN: false + # If image scan (previous step) fails, the build will fail, thus the image won't be pushed to the curated registry + # If image scan succeeds, the image will be pushed to the curated registry + PushingDockerRegistry: + title: Pushing to FINAL Docker Registry (curated registry of scanned images) + type: push + candidate: '${{BuildingDockerImage}}' + image_name: franciscocodefresh/twistlockdemo + tags: + - '${{CF_SHORT_REVISION}}' +``` + +### Example 2 - Scanning an image from a temporary external registry +In this example, we are going to use Docker Hub as our temporary registry, which can be considered as a "*Registry of unscanned images*" (to be scanned). Once there, we can initiate the scan in Twistlock console. + +#### Configure the registry to scan in Twistlock + +In your Twistlock dashboard go to `#!/defend/vulnerabilities/registry` . And add a new "registry settings" record. + +These are the settings used: + +- Version: Docker Registry v2 +- Registry: docker.io +- Repository name: franciscocodefresh/twistlockdemo-temp +- Tag: +- Username: +- Password: + +#### Set up a pipeline with the following configuration + +**Environment Variables (configured at Pipeline Configuration):** + +``` +TL_CONSOLE_HOSTNAME=169.254.169.254 +TL_CONSOLE_PORT=8083 +TL_CONSOLE_USERNAME=myuser +TL_CONSOLE_PASSWORD=mypassword +TL_COMPLIANCE_THRESHOLD=critical +TL_VULNERABILITY_THRESHOLD=critical +``` + +For this example, we're being permissive (`critical` for both thresholds). Of course those values can be set to any of the other options. + +**Pipeline YAML (codefresh.yml)** + +```yaml +version: '1.0' +steps: + BuildingDockerImage: + title: Building Docker Image + type: build + image_name: franciscocodefresh/twistlockdemo + tag: '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + + PushingToTempDockerRegistry: + title: Pushing to Temporal Docker Registry (for unscanned images -> to be scanned) + type: push + candidate: '${{BuildingDockerImage}}' + image_name: franciscocodefresh/twistlockdemo-temp + tags: + - '${{CF_SHORT_REVISION}}' + + TLScan: + title: Twistlock Scan + image: codefresh/cfstep-twistlock + environment: + - TL_REGISTRY=docker.io + - TL_IMAGE_NAME=franciscocodefresh/twistlockdemo-temp + - TL_IMAGE_TAG=${{CF_SHORT_REVISION}} + on_success: + metadata: + set: + - ${{BuildingDockerImage.imageId}}: + - SECURITY_SCAN: true + on_fail: + metadata: + set: + - ${{BuildingDockerImage.imageId}}: + - SECURITY_SCAN: false + + PushingDockerRegistry: + title: Pushing to FINAL Docker Registry (curated registry of scanned images) + type: push + candidate: '${{BuildingDockerImage}}' + image_name: franciscocodefresh/twistlockdemo + tags: + - '${{CF_SHORT_REVISION}}' +``` diff --git a/plugins/clair/README.md b/plugins/clair/README.md new file mode 100644 index 00000000..50520b3d --- /dev/null +++ b/plugins/clair/README.md @@ -0,0 +1,29 @@ +# Codefresh Clair Plugin + +Clair is an open source Docker Image securitu scanning server. For more information see here: https://github.com/coreos/clair/ + +There's an open source CLI tool for clair, called 'klar' that you can use to integrate secrurity scan into your pipeline. More info on klar: https://github.com/optiopay/klar + +## Usage + +Set environment variables described below, and run the command with the given image to scan: + +```yaml +scan: + image: 'codefresh/klar:master' + commands: + - /klar codefresh/helm:2.8.1 +``` + +(in this example we are scanning the helm image tagged 2.8.1 under codefresh organization in Docker Hub) + +## Environment Variables + +The minimal setup is described below. Please see Klar documentation for additional configuration. + +Name|Required|Description +---|---|--- +CLAIR_ADDR|Yes|The address of the clair server +DOCKER_USER|No|Docker registry account name +DOCKER_PASSWORD|No|Docker registry account password + diff --git a/plugins/clair/plugin.yaml b/plugins/clair/plugin.yaml new file mode 100644 index 00000000..b95d7ef2 --- /dev/null +++ b/plugins/clair/plugin.yaml @@ -0,0 +1,20 @@ +image: codefresh/klar:master +version: 1.0.0 +description: Scan an image with Clair +keywords: + - clair + - security +home: https://github.com/codefresh-io/pugins/tree/master/incubator/clair +sources: + - https://github.com/optiopay/klar +envs: + - name: CLAIR_ADDR + type: required + description: The address of the clair server + - name: DOCKER_USER + type: required + description: Docker registry account name. + - name: DOCKER_PASSWORD + description: Docker registry account password. +context: + - kind: Clair diff --git a/plugins/codefresh-cli/NOTES.md b/plugins/codefresh-cli/NOTES.md new file mode 100644 index 00000000..842a96bc --- /dev/null +++ b/plugins/codefresh-cli/NOTES.md @@ -0,0 +1 @@ +## Notes \ No newline at end of file diff --git a/plugins/codefresh-cli/README.md b/plugins/codefresh-cli/README.md new file mode 100644 index 00000000..bbca3395 --- /dev/null +++ b/plugins/codefresh-cli/README.md @@ -0,0 +1,31 @@ +# Codefresh Cli Plugin + +Use Codefresh Cli plugin to perform operations on your Codefresh resources + +## Usage + + +```yaml +--- +version: '1.0' + +steps: + + ... + + annotate_image: + image: codefresh/cli + description: annotates image with metadata + command: annotate image IMAGE_ID -a key1=value1 -a key2=value2 + + run_pipeline: + image: codefresh/cli + description: run a pipeline + command: run PIPELINE_ID -b master + ... + +``` + +## Environment Variables + +- `CFCONFIG` - Path for cfconfig file path (default: ${HOME}/.cfconfig diff --git a/plugins/codefresh-cli/plugin.yaml b/plugins/codefresh-cli/plugin.yaml new file mode 100644 index 00000000..b3cf301f --- /dev/null +++ b/plugins/codefresh-cli/plugin.yaml @@ -0,0 +1,12 @@ +image: codefresh/cli +version: 0.1.0 +description: Operate on Codefresh resources +keywords: + - cli +home: https://github.com/codefresh-io/codefresh +sources: + - https://github.com/codefresh-io/codefresh +icon: http://cdn.osxdaily.com/wp-content/uploads/2014/08/terminal-icon-osx.png +envs: + - name: CFCONFIG + description: Path for cfconfig file path \ No newline at end of file diff --git a/plugins/dcos-app-deploy/NOTES.md b/plugins/dcos-app-deploy/NOTES.md new file mode 100644 index 00000000..7dc7fc44 --- /dev/null +++ b/plugins/dcos-app-deploy/NOTES.md @@ -0,0 +1 @@ +## Notes diff --git a/plugins/dcos-app-deploy/README.md b/plugins/dcos-app-deploy/README.md new file mode 100644 index 00000000..6431b8a7 --- /dev/null +++ b/plugins/dcos-app-deploy/README.md @@ -0,0 +1,101 @@ +# cf-deploy-dcos +The repository contains code for `codefresh/cf-deploy-dcos` image building. This image makes an application deployment on DC/OS cluster using application definition json file. +It takes application deployment template (deployment.tmpl) and generates application deployment json file substituting variables in the template with the the environment variables values then applies the application deployment json file to DC/OS cluster using DC/OS CLI command. + + +# Usage +In order to use the `codefresh/cf-deploy-dcos` image we need to do the following: + +1. Define environment variables in Codefresh pipeline. + +- `DCOS_URL` **required** - DC/OS cluster URL +- `DCOS_CLUSTER_NAME` **required** - DC/OS cluster name +- `DCOS_CLUSTER_ID` **required** - DC/OS cluster ID +- `DCOS_DCOS_ACS_TOKEN` **required** - DC/OS cluster existing user's token (make it encrypted) +- `DCOS_SSL_VERIFY` default is true, if we want to bypass SSL certificate verification - set it to `false` +- `APP_ID` - application name +- `IMAGE_NAME` - application image name +- `IMAGE_TAG` - application image tag + +2. Create deployment.tmpl and codefresh.yml files in an application repository at the root level. + +``` +codefresh.yml +--- +version: '1.0' +steps: + BuildingDockerImage: + type: build + image_name: applcation/image + ... + + PushToRegistry: + type: push + candidate: ${{BuildingDockerImage}} + ... + + DeployToDcos: + image: codefresh/cf-deploy-dcos:latest + working_directory: ${{main_clone}} + commands: + - /cf-deploy-dcos deployment.tmpl + environment: + - DCOS_URL=${{DCOS_URL}} + - DCOS_CLUSTER_NAME=${{DCOS_CLUSTER_NAME}} + - DCOS_CLUSTER_ID=${{DCOS_CLUSTER_ID}} + - DCOS_DCOS_ACS_TOKEN=${{DCOS_DCOS_ACS_TOKEN}} + - DCOS_SSL_VERIFY=${{DCOS_SSL_VERIFY}} + - APP_ID=${{APP_ID}} + - IMAGE_NAME=${{IMAGE_NAME}} + - IMAGE_TAG=${{IMAGE_TAG}} +``` +We define freestyle step (DeployToDcos in the example above) and environment variables the same as in the Codefresh pipeline. + +``` +deployment.tmpl + +{ + "id": "{{APP_ID}}", + "instances": 1, + "cpus": 0.1, + "mem": 64, + "container": { + "type" : "DOCKER", + "docker": { + "image": "{{IMAGE_NAME}}:{{IMAGE_TAG}}", + "forcePullImage": true, + "privileged": false, + "network": "BRIDGE", + "portMappings": [ + { "hostPort": 80, "containerPort": 8081, "protocol": "tcp", "name": "http"} + ] + } + }, + "acceptedResourceRoles": [ + "slave_public" + ] +} +``` +`APP_ID`, `IMAGE_NAME` and `IMAGE_TAG` variables are just examples. We can parametrise any value in application deployment template depending on our requirements. +But if we set some parameter `{{PARAMETER}}` in application deployment template we should ensure that this parameter is set both in Codefresh pipeline and in codefresh.yml freestyle step as well. + +Notes: we can use already configured DC/OS CLI dcos command to get DC/OS cluster parameters. +Example: +``` +dcos cluster list --attached --json + +[ + { + "attached": true, + "cluster_id": "9d50f776-****-433c-****-ebb01eaafbbc", + "name": "dcos-master", + "url": "https://dcos-master.cloudapp.azure.com", + "version": "1.10.2" + } +] +``` +``` +dcos config show core.dcos_acs_token + +eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ik9UQkVOakZFTWtWQ09VRTRPRVpGTlRNMFJrWXlRa015Tnprd1JrSkVRemRCTWpBM1FqYzVOZyJ9.ey******** +``` diff --git a/plugins/dcos-app-deploy/plugin.yaml b/plugins/dcos-app-deploy/plugin.yaml new file mode 100644 index 00000000..6187c301 --- /dev/null +++ b/plugins/dcos-app-deploy/plugin.yaml @@ -0,0 +1,34 @@ +image: codefresh/cf-deploy-dcos +tag: latest +version: 0.1.0 +description: Deploy an application on dcos cluster +keywords: + - dcos + - deploy + - deployment +home: https://github.com/codefresh-io/cf-deploy-dcos +sources: + - https://github.com/codefresh-io/cf-deploy-dcos +maintainers: # (optional) + - name: Eugene Semirski + email: eugene@codefresh.io +icon: https://avatars0.githubusercontent.com/u/29493517?v=4&s=400 +envs: + - name: DCOS_URL + type: required + description: DC/OS cluster URL + - name: DCOS_CLUSTER_NAME + type: required + description: DC/OS cluster name + - name: DCOS_CLUSTER_ID + type: required + description: DC/OS cluster ID + - name: DCOS_DCOS_ACS_TOKEN + type: required + description: DC/OS cluster existing user's token + - name: DCOS_SSL_VERIFY + type: not_required + description: default is true, if we want to bypass SSL certificate verification - set it to false + - name: PARAMETER + type: not_required + description: see README.md for details diff --git a/plugins/docker-service/README.md b/plugins/docker-service/README.md new file mode 100644 index 00000000..4d4ad14c --- /dev/null +++ b/plugins/docker-service/README.md @@ -0,0 +1,6 @@ +# Codefresh docker-service plugin + +Use Codefresh docker-service plugin to create docker daemon and then use run on it any of +`docker build|run|...` or `docker-compose` on your repository + +See https://github.com/codefresh-io/docker-service \ No newline at end of file diff --git a/plugins/docker-service/plugin.yaml b/plugins/docker-service/plugin.yaml new file mode 100644 index 00000000..5357ec51 --- /dev/null +++ b/plugins/docker-service/plugin.yaml @@ -0,0 +1,14 @@ +image: codefresh/docker-service +tag: master +version: 0.1.0 +description: Codefresh docker-service plugin +keywords: + - docker + - docker-machine + - docker-compose +home: https://github.com/codefresh-io/docker-service +sources: + - https://github.com/codefresh-io/docker-service +maintainers: # (optional) + - name: Kosta Klevensky + email: kosta@codefresh.io \ No newline at end of file diff --git a/plugins/ecs-deploy/README.md b/plugins/ecs-deploy/README.md new file mode 100644 index 00000000..2c9c73cc --- /dev/null +++ b/plugins/ecs-deploy/README.md @@ -0,0 +1,114 @@ + +# cf-ecs-deploy +Deployment to Amazon ECS Service + +### Prerequiests +- Configured ECS Cluster with at least one running instance. +- Configured ECS Service and task definition with an image being deployed. + See http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html + +- AWS Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) with following priviledges: +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1479146904000", + "Effect": "Allow", + "Action": [ + "ecs:DescribeServices", + "ecs:DescribeTaskDefinition", + "ecs:DescribeTasks", + "ecs:ListClusters", + "ecs:ListServices", + "ecs:ListTasks", + "ecs:RegisterTaskDefinition", + "ecs:UpdateService" + ], + "Resource": [ + "*" + ] + } + ] +} +``` + +### Deployment with Codefresh +- Add encrypted environment variables for aws credentials. + * AWS_ACCESS_KEY_ID + * AWS_SECRET_ACCESS_KEY +- Add "deploy to ecs" step to codefresh.yml which runs codefresh/cf-deploy-ecs image with command cfecs-update + Specify the aws region, ecs cluster and service names. See `cfecs-update -h` for parameter references + +```yaml +# codefresh.yml example with deploy to ecs step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push to registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + deploy to ecs: + image: codefresh/cf-deploy-ecs + commands: + - cfecs-update + environment: + - AWS_ACCESS_KEY_ID=${{AWS_ACCESS_KEY_ID}} + - AWS_SECRET_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + + when: + - name: "Execute for 'master' branch" + condition: "'${{CF_BRANCH}}' == 'master'" +``` + + +### Deployment Flow +- get ECS service by specified aws region, ecs cluster and service names +- create new revision from current task definition of the service. If --image-name and --image-tag are provided, replace the tag of the image +- launch update-service with new task definition revision +- wait for deployment to complete (by default, if running withou --no-wait) + * deployment is considered as completed successfully if runningCount == desiredCount for PRIMARY deployment - see `aws ecs describe-service` + * cfecs-update exits with timeout if after --timeout (default = 900s) runningCount != desiredCount script exits with timeout + * cfecs-update exits with error if --max-failed (default = 2) or more ecs tasks were stopped with error for the task definition being deployed. + ECS retries failed tasks continuously + +### Usage with docker + +```bash +docker run --rm -it -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** codefresh/cf-ecs-deploy cfecs-update [options] +``` + +### cfecs-update -h +``` +usage: cfecs-update [-h] [-i IMAGE_NAME] [-t IMAGE_TAG] [--wait | --no-wait] + [--timeout TIMEOUT] [--max-failed MAX_FAILED] [--debug] + region_name cluster_name service_name + +Codefresh ECS Deploy + +positional arguments: + region_name AWS Region, ex. us-east-1 + cluster_name ECS Cluster Name + service_name ECS Service Name + +optional arguments: + -h, --help show this help message and exit + --wait Wait for deployment to complete (default) + --no-wait No Wait for deployment to complete + --timeout TIMEOUT deployment wait timeout (default 900s) + --max-failed MAX_FAILED + max failed tasks to consider deployment as failed + (default 2) + --debug show debug messages + + -i IMAGE_NAME, --image-name IMAGE_NAME + Image Name in ECS Task Definition to set new tag + -t IMAGE_TAG, --image-tag IMAGE_TAG + Tag for the image +``` diff --git a/plugins/ecs-deploy/plugin.yaml b/plugins/ecs-deploy/plugin.yaml new file mode 100644 index 00000000..632a213b --- /dev/null +++ b/plugins/ecs-deploy/plugin.yaml @@ -0,0 +1,36 @@ +image: codefresh/ecs +tag: 2.7.2 +version: 1.1.0 +description: Release a Helm chart (update or install) +keywords: + - ecs + - deploy + - containers +home: https://github.com/codefresh-io/cf-deploy-ecs.git +sources: + - https://github.com/codefresh-io/cf-deploy-ecs.git +maintainers: # (optional) + - name: Oleg Verhovsky + email: oleg@codefresh.io +icon: https://www.metricly.com/wp-content/uploads/2017/07/ECSIcon.png +envs: + - name: AWS_ACCESS_KEY_ID + type: required + description: Amazon access key ID + - name: AWS_SECRET_ACCESS_KEY + type: required + description: amazon secret key (make sure it's encrypted) + - name: aws-region + type: required + description: Helm release name + - name : region + type : required + description : aws region + - name : cluster_name + type : required + description : ecs cluster name + - name : service_name + type : required + description : ecs service name +context: + - kind: Amazon ECS diff --git a/plugins/gitclonerssh/README.md b/plugins/gitclonerssh/README.md new file mode 100644 index 00000000..9c2a4831 --- /dev/null +++ b/plugins/gitclonerssh/README.md @@ -0,0 +1,30 @@ +# Clone repository via SSH +Plugin to clone git repositories via SSH. + +Dockerhub repo: https://hub.docker.com/r/codefresh/cfstep-gitclonerssh + +## Options +| ENVIRONMENT VARIABLE | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|--|--|--|--|--| +| REMOTE_URL | null | string | Yes | Reporitory SSH URL (e.g. `git@github.com:my-user/my-repo.git`) | +| BRANCH | master | string | No | Branch name to checkout (e.g. `master`) | +| SSH_KEY | null | string | Yes | Private SSH key to access the repository. To convert it to single line string, and set a value for this var you can execute: `cat ~/.ssh/my_ssh_key_file | tr '\n' ','`. This assumes that `SPLIT_CHAR` will be set to `,` | +| SPLIT_CHAR | null | string | Yes | Split character you’ve used to replace newline in SSH key (`SSH_KEY`). Recommendation: use `,` (comma character)| +| CLONE_PATH | working directory | string | No | Path where `git clone` is going to be executed. A "`repo-name`" directory will be created there| + +## Usage Example: +This example clones a private repo using a private SSH key. +This example assumes that `SSH_KEY` var is already saved as an encrypted-pipeline-var. And that `SPLIT_CHAR` has a value of ','. + +```yaml +version: '1.0' +steps: + clone_repo_via_ssh: + image: codefresh/cfstep-gitclonerssh + environment: + - REMOTE_URL=git@github.com:my-user/my-repo.git + - BRANCH=my-branch + - SSH_KEY=${{SSH_KEY}} + - SPLIT_CHAR=${{SPLIT_CHAR}} + - CLONE_PATH=/codefresh/volume +``` \ No newline at end of file diff --git a/plugins/gitclonerssh/plugin.yaml b/plugins/gitclonerssh/plugin.yaml new file mode 100644 index 00000000..db99ef77 --- /dev/null +++ b/plugins/gitclonerssh/plugin.yaml @@ -0,0 +1,31 @@ +image: codefresh/cfstep-gitclonerssh +tag: latest +version: 0.1.0 +description: clone git repositories via SSH. +keywords: + - git + - clone + - ssh +home: https://github.com/codefresh-io/cfstep-gitclonerssh +sources: + - https://github.com/codefresh-io/cfstep-gitclonerssh +maintainers: + - name: Francisco Cocozza + email: francisco@codefresh.io +icon: https://git-scm.com/images/logos/downloads/Git-Icon-1788C.png + envs: + - name: REMOTE_URL + type: required + description: Reporitory SSH URL (e.g. `git@github.com:my-user/my-repo.git`) + - name: BRANCH + type: optional + description: Branch name to checkout (e.g. `master`) + - name: SSH_KEY + type: required + description: Private SSH key to access the repository. + - name: SPLIT_CHAR + type: required + description: Split character you’ve used to replace newline in SSH key + - name: CLONE_PATH + type: optional + description: Path where `git clone` is going to be executed. A "`repo-name`" directory will be created there \ No newline at end of file diff --git a/plugins/github-pr/README.MD b/plugins/github-pr/README.MD new file mode 100644 index 00000000..2b167688 --- /dev/null +++ b/plugins/github-pr/README.MD @@ -0,0 +1,99 @@ +# GitHub Pull Request Codefresh Plugin + +Operates on pull requests in GitHub. You can create a pull request, update it, open or close. + +## Environment Variables + +- `GITHUB_TOKEN`: token for access to GitHub +- `GITHUB_REPO_OWNER`: name of repo owner +- `GITHUB_REPO_NAME`: name of repo +- `GITHUB_PR_OPERATION`: operation on pull request (choices: create, update, open, close) `(default: create)` +- `GITHUB_PR_NUMBER`: number of your pull request `(required for: update, open, close)` +- `HEAD`: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this: username:branch. `(required for: create)` +- `BASE`: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. `(required for: create)` +- `TITLE`: The title of the pull request `(required for: create)` + +## Deployment with Codefresh +- Add encrypted environment variables for: + * GITHUB_TOKEN + +- Add "github-pr" step as described below + +```yaml +# codefresh.yml example with github for pr creating step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push_to_registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + github-pr: + image: codefresh/github-pr-plugin + environment: + - GITHUB_REPO_OWNER=${{CF_REPO_OWNER}} + - GITHUB_REPO_NAME=${{CF_REPO_NAME}} + - BASE=master + - HEAD=${{CF_BRANCH}} + - TITLE=Codefresh PR for ${{CF_BRANCH}} +``` + +- or: + +```yaml + +# codefresh.yml example with github pr updating step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push_to_registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + github-pr: + image: codefresh/github-pr-plugin + environment: + - GITHUB_PR_OPERATION=update + - GITHUB_PR_NUMBER=2 # your pr number here + - GITHUB_REPO_OWNER=${{CF_REPO_OWNER}} + - GITHUB_REPO_NAME=${{CF_REPO_NAME}} + - TITLE=Updated title for ${{CF_BRANCH}} # optional + - BASE=master # optional + - HEAD=${{CF_BRANCH}} # optional +``` + +- or: + +```yaml + +# codefresh.yml example with github pr open/close step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push_to_registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + github-pr: + image: codefresh/github-pr-plugin + environment: + - GITHUB_PR_OPERATION=open # same as 'close' + - GITHUB_PR_NUMBER=2 # your pr number here + - GITHUB_REPO_OWNER=${{CF_REPO_OWNER}} + - GITHUB_REPO_NAME=${{CF_REPO_NAME}} +``` diff --git a/plugins/github-pr/plugin.yaml b/plugins/github-pr/plugin.yaml new file mode 100644 index 00000000..fb8afbbc --- /dev/null +++ b/plugins/github-pr/plugin.yaml @@ -0,0 +1,37 @@ +image: codefresh/github-pr-plugin +tag: master +version: 0.2.0 +description: Operates on GitHub pull requests +keywords: + - docker + - github + - pull-request +home: https://github.com/codefresh-io/github-pr-plugin +sources: + - https://github.com/codefresh-io/github-pr-plugin +maintainers: # (optional) + - name: Nick Sakovich + email: nick.sakovich@codefresh.io + - name: Yaroslav Drachenko + email: yaroslav@codefresh.io +icon: https://assets-cdn.github.com/images/modules/logos_page/GitHub-Mark.png +envs: + - name: GITHUB_TOKEN + type: required + description: Token for access to GitHub + - name: GITHUB_REPO_OWNER + type: required + description: Name of repo owner + - name: GITHUB_REPO_NAME + type: required + description: Name of repo + - name: GITHUB_PR_NUMBER + description: The number of updated pull request. Required for open/close/update operations + - name: GITHUB_PR_OPERATION + description: Operation on pull request. Options - create, update, open, close. Default - create. + - name: HEAD + description: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this - username:branch. Required for 'create' operation. + - name: BASE + description: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. Required for 'create' operation. + - name: TITLE + description: The title of the pull request. Required for 'create' operation. diff --git a/plugins/github-release/README.md b/plugins/github-release/README.md new file mode 100644 index 00000000..524a5275 --- /dev/null +++ b/plugins/github-release/README.md @@ -0,0 +1,39 @@ +# GitHub release Codefresh Plugin + +A quick plugin to cover specific use case: create releases in GitHub and upload files for them. The plugin also allows to do more complex actions overriding the command manually. + +## Basic usage + +This example creates a release and uploads files to it: + +``` +github_prerelease: + image: codefresh/cfstep-github-release + environment: + - GITHUB_TOKEN=${{GITHUB_TOKEN}} + - FILES=bin/app-* + - PRERELEASE=true +``` +## Advanced usage + +If one wants to do more actions to manage releases than just to create them, it is possible to override the behaviour with custom commands: + +``` +github_prerelease: + image: codefresh/cfstep-github-release + commands: + - github-release edit --user $CF_REPO_OWNER --repo $CF_REPO_NAME --tag $CF_BRANCH_TAG_NORMALIZED --name "$CF_BRANCH_TAG_NORMALIZED" + - github-release delete --user $CF_REPO_OWNER --repo $CF_REPO_NAME --tag $CF_BRANCH_TAG_NORMALIZED + - github-release --help +``` + +More details about the paramaters and examples see [here](https://github.com/aktau/github-release) + +## Environment Variables + +- `GITHUB_TOKEN`: token for access to GitHub +- `CF_REPO_OWNER`: Codefresh provided variable containing repository owner name +- `CF_REPO_NAME`: Codefresh provided variable containing repository name +- `CF_BRANCH_TAG_NORMALIZED`: Codefresh provided variable containing branch/tag name +- `PRERELEASE`: If true, this variable tells the plugin to create a pre-release +- `FILES`: A glob expression for the list of the files to be uploaded \ No newline at end of file diff --git a/plugins/github-release/github-release/README.md b/plugins/github-release/github-release/README.md new file mode 100644 index 00000000..524a5275 --- /dev/null +++ b/plugins/github-release/github-release/README.md @@ -0,0 +1,39 @@ +# GitHub release Codefresh Plugin + +A quick plugin to cover specific use case: create releases in GitHub and upload files for them. The plugin also allows to do more complex actions overriding the command manually. + +## Basic usage + +This example creates a release and uploads files to it: + +``` +github_prerelease: + image: codefresh/cfstep-github-release + environment: + - GITHUB_TOKEN=${{GITHUB_TOKEN}} + - FILES=bin/app-* + - PRERELEASE=true +``` +## Advanced usage + +If one wants to do more actions to manage releases than just to create them, it is possible to override the behaviour with custom commands: + +``` +github_prerelease: + image: codefresh/cfstep-github-release + commands: + - github-release edit --user $CF_REPO_OWNER --repo $CF_REPO_NAME --tag $CF_BRANCH_TAG_NORMALIZED --name "$CF_BRANCH_TAG_NORMALIZED" + - github-release delete --user $CF_REPO_OWNER --repo $CF_REPO_NAME --tag $CF_BRANCH_TAG_NORMALIZED + - github-release --help +``` + +More details about the paramaters and examples see [here](https://github.com/aktau/github-release) + +## Environment Variables + +- `GITHUB_TOKEN`: token for access to GitHub +- `CF_REPO_OWNER`: Codefresh provided variable containing repository owner name +- `CF_REPO_NAME`: Codefresh provided variable containing repository name +- `CF_BRANCH_TAG_NORMALIZED`: Codefresh provided variable containing branch/tag name +- `PRERELEASE`: If true, this variable tells the plugin to create a pre-release +- `FILES`: A glob expression for the list of the files to be uploaded \ No newline at end of file diff --git a/plugins/github-release/github-release/plugin.yaml b/plugins/github-release/github-release/plugin.yaml new file mode 100644 index 00000000..61d1973b --- /dev/null +++ b/plugins/github-release/github-release/plugin.yaml @@ -0,0 +1,32 @@ +image: docker.io/codefresh/cfstep-github-release +tag: master +version: 0.1.0 +description: Plugin to make github releases +keywords: + - github + - release +home: https://github.com/codefresh-contrib/cfstep-github-release +sources: + - https://github.com/codefresh-contrib/cfstep-github-release +maintainers: + - name: Alex Cheshko + email: a.cheshko@codefresh.io +icon: A URL to an SVG or PNG image to be used as an icon (optional) +envs: + - name: GITHUB_TOKEN + type: required + description: Codefresh provided variable containing repository owner name + - name: CF_REPO_OWNER + type: required + description: Codefresh provided variable containing repository owner name + - name: CF_REPO_NAME + type: required + description: Codefresh provided variable containing repository name + - name: CF_BRANCH_TAG_NORMALIZED + type: required + description: Codefresh provided variable containing branch/tag name + - name: PRERELEASE + description: If true, this variable tells the plugin to create a pre-release + - name: FILES + description: A glob expression for the list of the files to be uploaded + \ No newline at end of file diff --git a/plugins/github-release/plugin.yaml b/plugins/github-release/plugin.yaml new file mode 100644 index 00000000..61d1973b --- /dev/null +++ b/plugins/github-release/plugin.yaml @@ -0,0 +1,32 @@ +image: docker.io/codefresh/cfstep-github-release +tag: master +version: 0.1.0 +description: Plugin to make github releases +keywords: + - github + - release +home: https://github.com/codefresh-contrib/cfstep-github-release +sources: + - https://github.com/codefresh-contrib/cfstep-github-release +maintainers: + - name: Alex Cheshko + email: a.cheshko@codefresh.io +icon: A URL to an SVG or PNG image to be used as an icon (optional) +envs: + - name: GITHUB_TOKEN + type: required + description: Codefresh provided variable containing repository owner name + - name: CF_REPO_OWNER + type: required + description: Codefresh provided variable containing repository owner name + - name: CF_REPO_NAME + type: required + description: Codefresh provided variable containing repository name + - name: CF_BRANCH_TAG_NORMALIZED + type: required + description: Codefresh provided variable containing branch/tag name + - name: PRERELEASE + description: If true, this variable tells the plugin to create a pre-release + - name: FILES + description: A glob expression for the list of the files to be uploaded + \ No newline at end of file diff --git a/plugins/gitsubmodules/README.md b/plugins/gitsubmodules/README.md new file mode 100644 index 00000000..468e3dd6 --- /dev/null +++ b/plugins/gitsubmodules/README.md @@ -0,0 +1,28 @@ +# Update git submodules +Plugin to update git submodules of an already cloned repo. + +Dockerhub repo: https://hub.docker.com/r/codefresh/cfstep-gitsubmodules/tags + +## Options +| ENVIRONMENT VARIABLE | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|--|--|--|--|--| +| GITHUB_TOKEN | null | string | Yes | GitHub Personal Token | +| CF_SUBMODULE_SYNC | null | boolean | No | If set to 'true', the step will perform 'git submodule sync' command | +| CF_SUBMODULE_UPDATE_RECURSIVE | null | boolean | No | If set to 'true', the step will perform 'git submodule update --init' command with '--recursive' option | + +## Usage Example: + +This example updates submodule of a cloned repo. + +The step assumes that the working directory is the cloned repo (which is the default working directory for any free style step) + +```yaml +version: '1.0' +steps: + updateSubmodules: + image: codefresh/cfstep-gitsubmodules + environment: + - GITHUB_TOKEN= + - CF_SUBMODULE_SYNC= + - CF_SUBMODULE_UPDATE_RECURSIVE= +``` \ No newline at end of file diff --git a/plugins/gitsubmodules/plugin.yaml b/plugins/gitsubmodules/plugin.yaml new file mode 100644 index 00000000..a82522b3 --- /dev/null +++ b/plugins/gitsubmodules/plugin.yaml @@ -0,0 +1,24 @@ +image: codefresh/cfstep-gitsubmodules +tag: latest +version: 0.1.0 +description: Update git submodules +keywords: + - git + - submodules +home: https://github.com/codefresh-io/cfstep-gitsubmodules +sources: + - https://github.com/codefresh-io/cfstep-gitsubmodules +maintainers: + - name: Francisco Cocozza + email: francisco@codefresh.io +icon: https://git-scm.com/images/logos/downloads/Git-Icon-1788C.png + envs: + - name: GITHUB_TOKEN + type: required + description: GitHub Token required to access the repository + - name: CF_SUBMODULE_SYNC + type: optional + description: if var is set to 'true', the step will perform 'git submodule sync' command + - name: CF_SUBMODULE_UPDATE_RECURSIVE + type: optional + description: if set to 'true', the step will perform 'git submodule update --init' command with '--recursive' option \ No newline at end of file diff --git a/plugins/gitter-notifier/NOTES.md b/plugins/gitter-notifier/NOTES.md new file mode 100644 index 00000000..7dc7fc44 --- /dev/null +++ b/plugins/gitter-notifier/NOTES.md @@ -0,0 +1 @@ +## Notes diff --git a/plugins/gitter-notifier/README.md b/plugins/gitter-notifier/README.md new file mode 100644 index 00000000..c86bb657 --- /dev/null +++ b/plugins/gitter-notifier/README.md @@ -0,0 +1,45 @@ +# Gitter notify plugin + +Gitter plugin which gives the opportunity send any messages to activity feed. + +## Using example + +```yaml +version: '1.0' +fail_fast: false +... +steps: + ... + sendMessage: + image: codefresh/gitternotifier + environment: + - GITTER_WEBHOOK=https://webhooks.gitter.im/e/123abc +``` + +## Required variables + +- `GITTER_WEBHOOK` - webhook uri from your [gitter](https://gitter.im) room integration + +## Optional variables + +**if you not provide this variables, plugin send info about build** + +- `GITTER_STATUS` + - **ok** - for info messages + - **error** - for error messages (red icon, red text) +- `GITTER_MESSAGE` - text of custom message which will be send, with [Handlebars.js](https://github.com/wycats/handlebars.js/) + - available vars: + - `{{buildTrigger}}` + - `{{buildInitiator}}` + - `{{buildId}}` + - `{{buildTimestamp}}` + - `{{buildUrl}}` + - `{{repoOwner}}` + - `{{repoName}}` + - `{{branch}}` + - `{{revision}}` + - `{{commitAuthor}}` + - `{{commitUrl}}` + - `{{commitMessage}}` + + - for text markup use **Markdown** diff --git a/plugins/gitter-notifier/plugin.yaml b/plugins/gitter-notifier/plugin.yaml new file mode 100644 index 00000000..404bf8a1 --- /dev/null +++ b/plugins/gitter-notifier/plugin.yaml @@ -0,0 +1,20 @@ +image: codefresh/gitternotifier +version: latest +description: Send message to gitter room +keywords: + - gitter +home: https://github.com/codefresh-io/gitter-notifier +sources: + - https://github.com/codefresh-io/gitter-notifier +maintainers: # (optional) + - name: Andrii Shaforostov + email: andrii@codefresh.io +icon: https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Font_Awesome_5_brands_gitter.svg/109px-Font_Awesome_5_brands_gitter.svg.png +envs: + - name: GITTER_WEBHOOK + type: required + description: gitter webhook + - name: GITTER_STATUS + description: send info about current build + - name: GITTER_MESSAGE + description: text of message which will be sent in room diff --git a/plugins/gke/README.md b/plugins/gke/README.md new file mode 100644 index 00000000..1d1ae954 --- /dev/null +++ b/plugins/gke/README.md @@ -0,0 +1,5 @@ +# Codefresh GKE plugin + +Use Codefresh GKE plugin to spin up GKE Kubernetes Cluster and then run any kubectl workload on it + +See https://github.com/codefresh-io/plugin-gke \ No newline at end of file diff --git a/plugins/gke/plugin.yaml b/plugins/gke/plugin.yaml new file mode 100644 index 00000000..79479a7d --- /dev/null +++ b/plugins/gke/plugin.yaml @@ -0,0 +1,28 @@ +image: codefresh/plugin-gke +tag: latest +version: 0.1.0 +description: Codefresh gke plugin +keywords: + - kubernetes + - gke + - gcloud +home: https://github.com/codefresh-io/plugin-gke +sources: + - https://github.com/codefresh-io/plugin-gke +maintainers: + - name: Kosta Klevensky + email: kosta@codefresh.io +icon: https://cloud.google.com/images/products/logos/cloud-symbol.png +envs: + - name: GOOGLE_SERVICE_ACCOUNT_KEY + type: required + description: Google Service Account Key value + - name: CLOUDSDK_COMPUTE_ZONE + type: optional + description: one of valid Google Compute zones + - name: CLOUDSDK_COMPUTE_REGION + type: optional + description: one of valid Google Compute regions. If both CLOUDSDK_COMPUTE_ZONE and CLOUDSDK_COMPUTE_REGION are not set, default is us-central1 + - name: GKE_CLUSTER_NAME + type: optional + description: Name of the cluster to create/delete. Can be also set from gke-create command line. It is generated if not set diff --git a/plugins/google-kms/Dockerfile b/plugins/google-kms/Dockerfile new file mode 100644 index 00000000..5ba4f610 --- /dev/null +++ b/plugins/google-kms/Dockerfile @@ -0,0 +1,10 @@ +FROM google/cloud-sdk:alpine + +WORKDIR /kms + +RUN apk -U add jq bash +ENV PATH=${PATH}:/kms + +COPY google-kms.sh ./kms + + diff --git a/plugins/google-kms/README.md b/plugins/google-kms/README.md new file mode 100644 index 00000000..57c78d1f --- /dev/null +++ b/plugins/google-kms/README.md @@ -0,0 +1,27 @@ +Сodefresh Google KMS plugin + +This plugin facilitates work with Google Key Management Service for such operations like *encrypting* and *decrypting* + +# Usage + +kms [OPERATION] [VALUE_1] [VALUE_n...] + +Set the plugin required environment variables for the pipeline and use the plugin as a freestyle step with a command like: + +```yaml +GoogleKMS: + image: codefresh/google-kms + commands: + - kms encrypt VALUE_1 VALUE_n +``` +where VALUE_1 and VALUE_n are the **names** of the environment variables containing the values you need to encrypt or decrypt. + +The operation is mutable and when the step finishes the variables with the same names will contain encrypted values. For decryption the process is similar + +# Required environment variables + +- `KMS_PROJECT` - GCP project name in which your KMS entities are present +- `KMS_LOCATION` - Google KMS location +- `KMS_KEYRING` - Google KMS keyring +- `KMS_KEY` - Google KMS key +- `GCP_SA_KEY` - [Google Service Account Key (JSON)](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) diff --git a/plugins/google-kms/google-kms.sh b/plugins/google-kms/google-kms.sh new file mode 100755 index 00000000..7eead9b9 --- /dev/null +++ b/plugins/google-kms/google-kms.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +for pluginVar in KMS_PROJECT KMS_LOCATION KMS_KEYRING KMS_KEY + do + if [ -z ${!pluginVar} ]; then echo $pluginVar is not set, stopping...; exit 1; fi + done + +echo $GCP_SA_KEY > google-app-creds.json +export GOOGLE_APPLICATION_CREDENTIALS=$(realpath google-app-creds.json) +operation=$1 + + +function encrypt () { + + hashedtext=$(echo $2 | base64 | tr -d '\n') + cf_export $1=$(curl -s -X POST "https://cloudkms.googleapis.com/v1/projects/$KMS_PROJECT/locations/$KMS_LOCATION/keyRings/$KMS_KEYRING/cryptoKeys/$KMS_KEY:encrypt" \ + -d "{\"plaintext\":\"$hashedtext\"}" \ + -H "Authorization:Bearer $(gcloud auth application-default print-access-token)" \ + -H "Content-Type:application/json" | jq '.ciphertext' --raw-output ) + + } + +function decrypt { + + cf_export $1=$(curl -s -X POST "https://cloudkms.googleapis.com/v1/projects/$KMS_PROJECT/locations/$KMS_LOCATION/keyRings/$KMS_KEYRING/cryptoKeys/$KMS_KEY:decrypt" \ + -d "{\"ciphertext\":\"$2\"}" \ + -H "Authorization:Bearer $(gcloud auth application-default print-access-token)" \ + -H "Content-Type:application/json" | jq '.plaintext' --raw-output | base64 -d) + + } + +for secret in "${@: 2}" + do + $operation $secret ${!secret} + done diff --git a/plugins/helm-legacy/.swp b/plugins/helm-legacy/.swp new file mode 100644 index 00000000..28879d2b Binary files /dev/null and b/plugins/helm-legacy/.swp differ diff --git a/stable/helm/NOTES.md b/plugins/helm-legacy/NOTES.md similarity index 100% rename from stable/helm/NOTES.md rename to plugins/helm-legacy/NOTES.md diff --git a/stable/helm/README.md b/plugins/helm-legacy/README.md similarity index 81% rename from stable/helm/README.md rename to plugins/helm-legacy/README.md index 708a7c06..e70cf24d 100644 --- a/stable/helm/README.md +++ b/plugins/helm-legacy/README.md @@ -1,3 +1,6 @@ +> There is a new Helm plugin with added capabilities, we are keeping this plugin as is for backward-compatibility. +The new plugin can be found here: [/incubator/helm](https://github.com/codefresh-io/plugins/tree/master/incubator/helm) + # Codefresh Helm Plugin Use Codefresh [Helm](https://helm.sh) plugin to deploy a Helm chart into specified (by context) Kubernetes cluster. diff --git a/stable/helm/plugin.yaml b/plugins/helm-legacy/plugin.yaml similarity index 79% rename from stable/helm/plugin.yaml rename to plugins/helm-legacy/plugin.yaml index f95b4c5e..fed4a017 100644 --- a/stable/helm/plugin.yaml +++ b/plugins/helm-legacy/plugin.yaml @@ -1,7 +1,7 @@ image: codefresh/plugin-helm tag: 2.7.2 version: 1.1.0 -description: Release a Helm chart (update or install) +description: Release a Helm chart (update or install). There is a new Helm plugin with added capabilities, we are keeping this plugin as is for backward-compatibility. The new plugin is at /incubator/helm (https://github.com/codefresh-io/plugins/tree/master/incubator/helm) keywords: - helm - kubernetes @@ -13,7 +13,7 @@ maintainers: # (optional) email: alexei@codefresh.io icon: https://avatars0.githubusercontent.com/u/29493517?v=4&s=400 envs: - - name: KUBE_CONTEXT + - name: KUBE_CONTEXT type: required description: Kubernetes context to use - name: CHART_NAME diff --git a/plugins/helm/README.md b/plugins/helm/README.md new file mode 100644 index 00000000..df6d49fc --- /dev/null +++ b/plugins/helm/README.md @@ -0,0 +1,2 @@ +# Codefresh Helm Plugin +The Codefresh Helm plugin facilitates authentication, configuration, and execution of Helm CLI commands in a Codefresh pipeline. For more info, see the documentation here: [https://codefresh.io/docs/docs/new-helm/using-helm-in-codefresh-pipeline/](https://codefresh.io/docs/docs/new-helm/using-helm-in-codefresh-pipeline/) diff --git a/plugins/helm/plugin.yaml b/plugins/helm/plugin.yaml new file mode 100644 index 00000000..53d2c471 --- /dev/null +++ b/plugins/helm/plugin.yaml @@ -0,0 +1,36 @@ +image: codefresh/cfstep-helm +version: 1.0.0 +description: Release a Helm chart (update or install) +keywords: + - helm + - kubernetes +home: https://github.com/codefresh-io/pugins/tree/master/incubator/helm +sources: + - https://github.com/codefresh-contrib/cfplugin-step +maintainers: # (optional) + - name: Itay Shakury + email: itaysk@codefresh.io +icon: https://avatars0.githubusercontent.com/u/29493517?v=4&s=400 +envs: + - name: KUBE_CONTEXT + type: required + description: Kubernetes context to use (the name of the cluster as configured in Codefresh) + - name: CHART_NAME + type: required + description: Helm chart name to release (path to chart folder, or name of packaged chart) + - name: RELEASE_NAME + description: Helm release name + - name: NAMESPACE + description: target Kubernetes namespace + - name: TILLER_NAMESPACE + description: Kubernetes namespace where tiller is at + - name: CHART_VERSION + description: application chart version to install + - name: CHART_REPO_URL + description: Helm chart repository URL (overriden by injected Helm repository context) + - name: CUSTOMFILE_ + description: Values file to provide to Helm (as --file). see usage information below. + - name: CUSTOM_ + description: Value to provide to Helm (as --set). see usage information below. +context: + - kind: kubernetes diff --git a/incubator/import-docker-images/README.md b/plugins/import-docker-images/README.md similarity index 100% rename from incubator/import-docker-images/README.md rename to plugins/import-docker-images/README.md diff --git a/incubator/import-docker-images/plugin.yaml b/plugins/import-docker-images/plugin.yaml similarity index 100% rename from incubator/import-docker-images/plugin.yaml rename to plugins/import-docker-images/plugin.yaml diff --git a/plugins/jira/NOTES.md b/plugins/jira/NOTES.md new file mode 100644 index 00000000..842a96bc --- /dev/null +++ b/plugins/jira/NOTES.md @@ -0,0 +1 @@ +## Notes \ No newline at end of file diff --git a/plugins/jira/README.md b/plugins/jira/README.md new file mode 100644 index 00000000..99f95e1e --- /dev/null +++ b/plugins/jira/README.md @@ -0,0 +1,33 @@ +# Codefresh Jira Plugin + +Use Codefresh Jira plugin to interact with a Jira instance. + +## Usage + + +```yaml +--- +version: '1.0' + +steps: + + ... + + UpdateJira: + title: Update Jira Issue + image: otomato/jira-cli:alpine + commands: + - yes n | jira-cli update ${JIRAID} --comment 'New docker image otomato/bringon:${{CF_SHORT_REVISION}}. Build log is here ${{CF_BUILD_URL}}' --jira-url ${JIRA_URL} -u ${JIRA_USR} -p ${JIRA_PWD} + when: + condition: + all: + JiraIdFound: 'match("${JIRAID}", "[A-Z]+-[0-9]+", true)' + ... + +``` + +## Environment Variables + +- **required** `JIRA_URL` - Url to Jira instance +- **required** `JIRA_USR` - Jira user +- **required** `JIRA_PWD` - - Jira password diff --git a/plugins/jira/plugin.yaml b/plugins/jira/plugin.yaml new file mode 100644 index 00000000..33f31aac --- /dev/null +++ b/plugins/jira/plugin.yaml @@ -0,0 +1,23 @@ +image: otomato/jira-cli +tag: alpine +version: 0.1.0 +description: Update a Jira ticket +keywords: + - jira +home: https://github.com/codefreshdemo/jira-cli-docker +sources: + - https://github.com/codefreshdemo/jira-cli-docker +maintainers: # (optional) + - name: Ant Weiss + email: anton@otomato.link +icon: https://github.com/codefreshdemo/jira-cli-docker/blob/master/jira-logo.png +envs: + - name: JIRA_URL + type: required + description: Url to Jira instance + - name: JIRA_USR + type: required + description: Jira user + - name: JIRA_PWD + type: required + description: Jira password diff --git a/plugins/kompose/README.md b/plugins/kompose/README.md new file mode 100644 index 00000000..fe11227d --- /dev/null +++ b/plugins/kompose/README.md @@ -0,0 +1,47 @@ +[![Codefresh build status]( https://g.codefresh.io/api/badges/build?repoOwner=codefresh-io&repoName=plugins&branch=master&pipelineName=Kompose%20Plugin&accountName=codefresh-inc&type=cf-1)]( https://g.codefresh.io/repositories/codefresh-io/plugins/builds?filter=trigger:build;branch:master;service:5a1c225585e11a0001cb2de1~Kompose%20Plugin) + +# Codefresh Kompose Plugin + +Use Codefresh [Kompose](http://kompose.io) plugin to deploy or convert a Docker Compose file into Kubernetes resources. + +## Usage + +Set required and optional environment variable and add the following step to your Codefresh pipeline: + +```yaml +--- +version: '1.0' + +steps: + + ... + + release_to_env: + image: codefresh/plugin-kompose:v1.5.0 + + ... + +``` + +## Environment Variables + +- **required** `KUBE_CONTEXT` - Kubernetes context to use +- `FILE` - Docker Compose file to deploy (default `docker-compose.yaml` file) +- `NAMESPACE` - target Kubernetes namespace (default `default` namespace) +- `REPLICAS` - specify the number of replicas generated (default `1`) +- `VOLUMES` - volumes to be generated (`persistentVolumeClaim`|`emptyDir`) (default `persistentVolumeClaim`) +- `DRY_RUN` - do a "dry run" (print out) deployment (do not install anything, useful for Debug) +- `DEBUG` - print verbose install output + + +## Kubernetes Configuration + +Add Kubernetes integration to Codefresh: `> Account Settings > Integration > Kubernetes`. From now on, you can use added Kubernetes cluster in Codefresh pipeline, addressing its context by the name you see in `Clusters` menu. + +## Building Plugin + +Use `docker build` command to build the plugin. +Two build arguments can be provided to override default `kubectl`, `helm` and `kompose` version: + +- `HELM_VERSION` - default to `latest` +- `KOMPOSE_VERSION` - default to `v1.5.0` \ No newline at end of file diff --git a/plugins/kompose/example/codefresh.yaml b/plugins/kompose/example/codefresh.yaml new file mode 100644 index 00000000..67e8f0d0 --- /dev/null +++ b/plugins/kompose/example/codefresh.yaml @@ -0,0 +1,18 @@ +version: '1.0' + +steps: + + dry_run: + image: codefresh/plugin-kompose:v1.5.0 + environment: + - NAMESPACE=kompose-test + - FILE=stable/kompose/example/docker-compose.yaml + - DEBUG=true + - DRY_RUN=true + + deploy_to_cluster: + image: codefresh/plugin-kompose:v1.5.0 + environment: + - NAMESPACE=kompose-test + - FILE=stable/kompose/example/docker-compose.yaml + - DEBUG=true diff --git a/plugins/kompose/example/docker-compose.yaml b/plugins/kompose/example/docker-compose.yaml new file mode 100644 index 00000000..5f847a4f --- /dev/null +++ b/plugins/kompose/example/docker-compose.yaml @@ -0,0 +1,24 @@ +version: "2" + +services: + + redis-master: + image: gcr.io/google_containers/redis:e2e + ports: + - "6379" + + redis-slave: + image: gcr.io/google_samples/gb-redisslave:v1 + ports: + - "6379" + environment: + - GET_HOSTS_FROM=dns + + frontend: + image: gcr.io/google-samples/gb-frontend:v4 + ports: + - "80:80" + environment: + - GET_HOSTS_FROM=dns + labels: + kompose.service.type: LoadBalancer diff --git a/plugins/kompose/plugin.yaml b/plugins/kompose/plugin.yaml new file mode 100644 index 00000000..ea49cec3 --- /dev/null +++ b/plugins/kompose/plugin.yaml @@ -0,0 +1,34 @@ +image: codefresh/plugin-helm +tag: v1.5.0 +version: 1.0.0 +description: Release a Docker Compose to Kubernetes +keywords: + - docker-compose + - docker + - kompose + - kubernetes +home: https://github.com/codefresh-io/cf-kompose-plugin +sources: + - https://github.com/codefresh-io/cf-kompose-plugin +maintainers: # (optional) + - name: Alexei Ledenev + email: alexei@codefresh.io +icon: http://cdn.rancher.com/wp-content/uploads/2016/04/20182217/compose-300x295.png +envs: + - name: KUBE_CONTEXT + type: required + description: Kubernetes context to use + - name: FILE + description: Docker Compose file to deploy (default "docker-compose.yaml` file) + - name: NAMESPACE + description: target Kubernetes namespace (default "default") + - name: VOLUMES + description: volumes to be generated ("persistentVolumeClaim"|"emptyDir") (default "persistentVolumeClaim") + - name: REPLICAS + description: specify the number of replicas generated (default 1) + - name: DRY_RUN + description: do a "dry run" (print out) deployment (do not install anything, useful for Debug) + - name: DEBUG + description: print verbose install output +context: + - kind: kubernetes diff --git a/plugins/makisu/README.md b/plugins/makisu/README.md new file mode 100644 index 00000000..3c0acba8 --- /dev/null +++ b/plugins/makisu/README.md @@ -0,0 +1,45 @@ +# Makisu plugin + +The plugin makes using [Makisu build tool](https://github.com/uber/makisu) easier in a Codefresh pipeline. + +### Requirements + +The plugin requires access to the docker daemon enabled, so one have to request it from Codefresh administrators, unless the user uses a Hybrid solution (running builds on his own infrastructure. + +### Basic usage + +This example covers the most common case - to build and push an image using Makisu distributed cache and flexible layer generation features. + +``` +makisuBuildStep: + image: codefresh/cfstep-makisu + environment: + - REGISTRY_HOSTNAME=docker.io + - R_USER=my_username + - R_PASSWORD=my_password + - IMAGE_NAME_TAG=image/name:tag +``` + +By default the makisu context is the main clone directory, but it can be changed with the WORKING_DIRECTORY environment variable. The dockerfile path can also be specified by the DOCKERFILE variable + +### Advanced usage + +If a user needs more flexibility, it is not a problem to add custom flags to the makisu buildcommand: + +``` +.... + environment: + - CUSTOM_FLAGS='--compression=speed ...' +``` + +or to completely override the makisu command: + +``` +makisuBuildStep: + image: codefresh/cfstep-makisu + environment: + - REGISTRY_HOSTNAME=docker.io + - R_USER=my_username + - R_PASSWORD=my_password + - MAKISU_COMMAND='makisu build -t myimage/name:tag --storage /codefresh/volume/makisu --modifyfs=true --commit=explicit --registry-config=/makisu-internal/registry-conf.yml --push docker.io --compression=speed .' +``` \ No newline at end of file diff --git a/plugins/makisu/plugin.yml b/plugins/makisu/plugin.yml new file mode 100644 index 00000000..02c110ee --- /dev/null +++ b/plugins/makisu/plugin.yml @@ -0,0 +1,33 @@ +image: codefresh/cfstep-makisu +tag: latest +version: 0.1.0 +description: The plugin makes using Makisu build tool easier in a Codefresh pipeline +keywords: + - makisu + - uber +home: https://github.com/codefresh-contrib/cfstep-makisu +sources: + - https://github.com/codefresh-contrib/cfstep-makisu +maintainers: # (optional) + - name: Alexander Cheshko + email: a.cheshko@codefresh.io +envs: + - name: IMAGE_NAME_TAG + type: required + description: "Name and tag of the image to build" + - name: REGISTRY_HOSTNAME + type: required + description: "Docker registry host name. Example: docker.io" + - name: R_USER + type: required + description: "User name to authenticate against the registry" + - name: R_PASSWORD + type: required + description: "Password to authenticate against the registry" + - name: DOCKERFILE + - name: CUSTOM_FLAGS + description: "Add custom flags to the makisu build command" + - name: MAKISU_COMMAND + description: "This is to override the makisu build command" + - name: WORKING_DIRECTORY + description: "Changes the makisu context directory. Default is the main clone directory" diff --git a/plugins/release-to-NPM/NOTES.md b/plugins/release-to-NPM/NOTES.md new file mode 100644 index 00000000..e69de29b diff --git a/stable/relese-to-NPM/README.md b/plugins/release-to-NPM/README.md similarity index 100% rename from stable/relese-to-NPM/README.md rename to plugins/release-to-NPM/README.md diff --git a/stable/relese-to-NPM/plugin.yaml b/plugins/release-to-NPM/plugin.yaml similarity index 100% rename from stable/relese-to-NPM/plugin.yaml rename to plugins/release-to-NPM/plugin.yaml diff --git a/plugins/run-jenkins-job/README.md b/plugins/run-jenkins-job/README.md new file mode 100644 index 00000000..df11e088 --- /dev/null +++ b/plugins/run-jenkins-job/README.md @@ -0,0 +1,31 @@ +# Trigger Jenkins Job + + +## Run locally +`go get codefresh-io/cf-run-jenkins-job` +``` +NAME: + cf-run-jenkins-job + +DESCRIPTION: + Trigger Jenkins Job + +## Mandatory Parameters: + + JENKINS_URL - Jenkins Master URL + JENKINS_USER - Jenkins User Name + JENKINS_TOKEN - Jenkins Token + JENKINS_JOB - Jenkins Job Name + +## Usage Example: + +version: '1.0' +steps: + RunJenkins: + title: Triggering Jenkins Job + image: codefresh/cf-run-jenkins-job + environment: + - JENKINS_URL=http://: + - JENKINS_USER= + - JENKINS_TOKEN= + - JENKINS_JOB= diff --git a/plugins/run-jenkins-job/plugin.yaml b/plugins/run-jenkins-job/plugin.yaml new file mode 100644 index 00000000..7a1a9283 --- /dev/null +++ b/plugins/run-jenkins-job/plugin.yaml @@ -0,0 +1,29 @@ +image: codefresh/run-jenkins-jobs +tag: master +version: 0.1.0 +description: Run jenkins job from codefresh pipeline +keywords: + - docker + - jenkins +home: https://github.com/codefresh-io/cf-run-jenkins-jobs +sources: + - https://github.com/codefresh-io/cf-run-jenkins-jobs +maintainers: # (optional) + - name: Vadim Waisman + email: vadim@codefresh.io + - name: Jenny Passi + email: jenny@codefresh.io +icon: https://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2016/11/1479211772build-image-with-dockerfile_feature.jpg +envs: + - name: JENKINS_USERNAME + type: required + description: jenkins username + - name: JENKINS_TOKEN + type: required + description: jenkins token + - name: JENKINS_URL + type: required + description: jenkins machine url + - name: JENKINS_JOB + type: required + description: list of jenkins jobs to run separated by spaces diff --git a/plugins/sendgrid/README.md b/plugins/sendgrid/README.md new file mode 100644 index 00000000..a471ab9a --- /dev/null +++ b/plugins/sendgrid/README.md @@ -0,0 +1,31 @@ +# Codefresh plugin for send e-mail letters + +Codefresh plugin for send e-mail notification via SendGrid + +## Main env variables +- `SENDGRID_APIKEY` - API key from SendGrid +- `SENDGRID_MAIL` - mail where the letter will be sent, you can use _comma_ divider to send on multiple mails (ex. `mail1@example.com, mail2@exmaple.com`) +- `SENDGRID_FROM` - from header of mail +- `SENDGRID_SUBJECT` - subject header of mail +- `SENDGRID_TYPE` - type of mail [build, message, custom] + +## Mail types +### build +Info about current build +### message +Send simple message with text from `SENDGRID_MESSAGE` +### custom +Send message with custom template via [ejs](https://www.npmjs.com/package/ejs) provided `SENDGRID_TEMPLATE` + +## Config for codefresh.yml +``` +version: '1.0' +... +steps: + ... + TestMail: + title: Test Mail + image: 'codefresh/sendgridplugin:latest' + ... +... +``` diff --git a/plugins/sendgrid/plugin.yaml b/plugins/sendgrid/plugin.yaml new file mode 100644 index 00000000..8816102d --- /dev/null +++ b/plugins/sendgrid/plugin.yaml @@ -0,0 +1,33 @@ +image: codefresh/sendgridplugin +version: latest +description: Send e-mail via Sendgrid +keywords: + - mail + - sendgrid +home: https://github.com/codefresh-io/sendgridplugin +sources: + - https://github.com/codefresh-io/sendgridplugin +maintainers: + - name: Pavel Kostohrys + email: pavel@codefresh.io +icon: https://images.ctfassets.net/bx16dovk9m7p/XNo9W4lwAMsai8KOikAae/35f6df5bf800dd4993c2d98376448f0d/a37a5b5ab495b5531dd406d40baa8a5c.png +envs: + - name: SENDGRID_APIKEY + type: required + description: API key from SendGrid + - name: SENDGRID_MAIL + type: required + description: mail where the letter will be sent, you can use comma divider to send on multiple mails (ex.: mail1@example.com, mail2@exmaple.com) + - name: SENDGRID_FROM + type: required + description: from header of mail + - name: SENDGRID_SUBJECT + type: required + description: subject header of mail + - name: SENDGRID_TYPE + type: required + description: type of mail [build, message, custom] + - name: SENDGRID_MESSAGE + description: Text of message for type message + - name: SENDGRID_TEMPLATE + description: Template of message for type message diff --git a/plugins/slack-notifier/NOTES.md b/plugins/slack-notifier/NOTES.md new file mode 100644 index 00000000..842a96bc --- /dev/null +++ b/plugins/slack-notifier/NOTES.md @@ -0,0 +1 @@ +## Notes \ No newline at end of file diff --git a/plugins/slack-notifier/README.md b/plugins/slack-notifier/README.md new file mode 100644 index 00000000..21789c9b --- /dev/null +++ b/plugins/slack-notifier/README.md @@ -0,0 +1,80 @@ +# Codefresh Slack Plugin + +Use Codefresh Slack plugin to send a message to a channel. + +## Usage + +Simple mode - just send text or attachment to slack channel + +```yaml +--- +version: '1.0' + +steps: + + ... + + SendToSlack: + title: Sending message to slack + image: codefresh/slacknotifier + environment: + - SLACK_HOOK_URL=${{SLACK_HOOK_URL}} + - SLACK_TEXT=${{SLACK_TEXT}} + - SLACK_ATTACHMENTS=${{SLACK_ATTACHMENTS}} + ... + +``` + +Template mode - ability to have one template body and just override fields + +```yaml +--- +version: '1.0' + +steps: + + ... + + SendToSlack: + title: Sending message to slack + image: codefresh/slacknotifier + environment: + - SLACK_HOOK_URL=${{SLACK_HOOK_URL}} + - SLACK_TEXT=cool + - SLACK_TEMPLATE_FIELDS=${{SLACK_TEMPLATE_FIELDS}} + - SLACK_TEMPLATE_BODY=${{SLACK_TEMPLATE_BODY}} + - MODE=template + ... + +``` + +Default template mode - send information about build + +```yaml +--- +version: '1.0' + +steps: + + ... + + SendToSlack: + title: Sending message to slack + image: codefresh/slacknotifier + environment: + - SLACK_HOOK_URL=${{SLACK_HOOK_URL}} + - MODE=default-template + ... + +``` + + + +## Environment Variables + +- **required** `SLACK_HOOK_URL` - Url to the channel. Slack official [docs](https://api.slack.com/incoming-webhooks) +- **required** `SLACK_TEXT` - The message that will be sent +- `SLACK_ATTACHMENTS` - print verbose output +- `MODE` - template | simple mode +- `SLACK_TEMPLATE_FIELDS` - fields for override if use template mode +- `SLACK_TEMPLATE_BODY` - message body if use template mode diff --git a/plugins/slack-notifier/plugin.yaml b/plugins/slack-notifier/plugin.yaml new file mode 100644 index 00000000..e9a8def6 --- /dev/null +++ b/plugins/slack-notifier/plugin.yaml @@ -0,0 +1,29 @@ +image: codefresh/slacknotifier +version: latest +description: Send message to slack channel +keywords: + - slack +home: https://github.com/codefresh-io/slack-notifier +sources: + - https://github.com/codefresh-io/slack-notifier +maintainers: # (optional) + - name: Pavel Kostohrys + email: pavel@codefresh.io +icon: https://upload.wikimedia.org/wikipedia/commons/7/76/Slack_Icon.png +envs: + - name: SLACK_HOOK_URL + type: required + description: Url to the channel + - name: SLACK_TEXT + type: required + description: Message to send + - name: SLACK_ATTACHMENTS + description: Attachments to send. Documentation https://api.slack.com/docs/message-attachments + - name: MODE + description: template | simple | default-template, by default - simple. In case with template mode you can have general body in attachment SLACK_TEMPLATE_BODY and different fields SLACK_TEMPLATE_FIELDS + - name: SLACK_TEMPLATE_BODY + type: required + description: Required in template mode, general body, should be not array. Support only one attachment in body. Documentation https://api.slack.com/docs/message-attachments + - name: SLACK_TEMPLATE_FIELDS + type: required + description: Required in template mode, Override fields in SLACK_TEMPLATE_BODY, should be array. Documentation https://api.slack.com/docs/message-attachments diff --git a/plugins/slack/NOTES.md b/plugins/slack/NOTES.md new file mode 100644 index 00000000..842a96bc --- /dev/null +++ b/plugins/slack/NOTES.md @@ -0,0 +1 @@ +## Notes \ No newline at end of file diff --git a/plugins/slack/README.md b/plugins/slack/README.md new file mode 100644 index 00000000..6b3470ec --- /dev/null +++ b/plugins/slack/README.md @@ -0,0 +1,32 @@ +# Codefresh Slack Plugin + +Use Codefresh Slack plugin to send a message to a channel. + +## Usage + + +```yaml +--- +version: '1.0' + +steps: + + ... + + SendToSlack: + title: Sending message to slack + image: codefresh/slack-message-sender + commands: + - slack-message-sender send + environment: + - WEBHOOK_URL=${{SLACK_WEBHOOK_URL}} + - SLACK_MESSAGE=${{CF_COMMIT_MESSAGE}} + ... + +``` + +## Environment Variables + +- **required** `WEBHOOK_URL` - Url to the channel. Slack official [docs](https://api.slack.com/incoming-webhooks) +- **required** `SLACK_MESSAGE` - The message that will be sent +- `DEBUG` - print verbose output diff --git a/plugins/slack/plugin.yaml b/plugins/slack/plugin.yaml new file mode 100644 index 00000000..25d623d9 --- /dev/null +++ b/plugins/slack/plugin.yaml @@ -0,0 +1,21 @@ +image: codefresh/slack-message-sender +version: 0.1.0 +description: Send message to slack channel +keywords: + - slack +home: https://github.com/codefresh-io/slack-message-sender +sources: + - https://github.com/codefresh-io/slack-message-sender +maintainers: # (optional) + - name: Oleg Sucharevich + email: olegs@codefresh.io +icon: https://upload.wikimedia.org/wikipedia/commons/7/76/Slack_Icon.png +envs: + - name: WEBHOOK_URL + type: required + description: Url to the channel + - name: SLACK_MESSAGE + type: required + description: Message to send + - name: DEBUG + description: print verbose install output \ No newline at end of file diff --git a/plugins/telegram-notifier/NOTES.md b/plugins/telegram-notifier/NOTES.md new file mode 100644 index 00000000..842a96bc --- /dev/null +++ b/plugins/telegram-notifier/NOTES.md @@ -0,0 +1 @@ +## Notes \ No newline at end of file diff --git a/plugins/telegram-notifier/README.md b/plugins/telegram-notifier/README.md new file mode 100644 index 00000000..7d14ca02 --- /dev/null +++ b/plugins/telegram-notifier/README.md @@ -0,0 +1,47 @@ +# Telegram notify plugin + +Telegram plugin which gives the opportunity send any messages to users via bot. + +## Usage + +```yaml +version: '1.0' +... +steps: + ... + sendMessage: + image: codefresh/telegramnotifier + environment: + - TELEGRAM_TOKEN=TOKEN + - TELEGRAM_TO=99999999 + - TELEGRAM_MESSAGE=Hello {{{userLink}}}, how are you + - TELEGRAM_IMAGES=https://codefresh.io/docs/assets/brand/codefresh-social.png +``` + +## Required variables + +- `TELEGRAM_TOKEN` - token of your bot (cat get from [@BotFather](https://t.me/BotFather)) +- `TELEGRAM_TO` - array of bot`s user id who will receive a message separated by comma (id you can retrieve from [@myidbot](https://t.me/myidbot)) + +## Optional variables + +- `TELEGRAM_STATUS` - send info about current build, **if pass - all others variables will be ignored** +- `TELEGRAM_MESSAGE` - text of message which will be sent to user, with [Handlebars.js](https://github.com/wycats/handlebars.js/), + - available vars: + - `{{buildTrigger}}` + - `{{buildInitiator}}` + - `{{buildId}}` + - `{{buildTimestamp}}` + - `{{buildUrl}}` + - `{{repoOwner}}` + - `{{repoName}}` + - `{{branch}}` + - `{{revision}}` + - `{{commitAuthor}}` + - `{{commitUrl}}` + - `{{commitMessage}}` + - `{{userID}}` - id of current telegram user + - `{{{userLink}}}` - link to current telegram user + + - for text markup use Markdown +- `TELEGRAM_IMAGES` - array of image links for attaching to message diff --git a/plugins/telegram-notifier/plugin.yaml b/plugins/telegram-notifier/plugin.yaml new file mode 100644 index 00000000..027c7493 --- /dev/null +++ b/plugins/telegram-notifier/plugin.yaml @@ -0,0 +1,25 @@ +image: codefresh/telegramnotifier +version: latest +description: Send message to telegram channel +keywords: + - telegram +home: https://github.com/codefresh-io/telegram-notifier +sources: + - https://github.com/codefresh-io/telegram-notifier +maintainers: # (optional) + - name: Andrii Shaforostov + email: andrii@codefresh.io +icon: https://upload.wikimedia.org/wikipedia/commons/5/5c/Telegram_Messenger.png +envs: + - name: TELEGRAM_TOKEN + type: required + description: token of your bot + - name: TELEGRAM_TO + type: required + description: array of user ids who will receive a message separated by comma + - name: TELEGRAM_STATUS + description: send info about current build + - name: TELEGRAM_MESSAGE + description: text of message which will be sent to user + - name: TELEGRAM_IMAGES + description: array of image links for attaching to message diff --git a/plugins/testplugin/README.MD b/plugins/testplugin/README.MD new file mode 100644 index 00000000..27c33099 --- /dev/null +++ b/plugins/testplugin/README.MD @@ -0,0 +1,42 @@ +# GitHub Pull Request Codefresh Plugin + +Creates a new pull request in GitHub + +## Environment Variables + +- `GITHUB_TOKEN`: token for access to GitHub +- `GITHUB_REPO_OWNER`: name of repo owner +- `GITHUB_REPO_NAME`: name of repo +- `HEAD`: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this: username:branch +- `BASE`: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. +- `TITLE`: The title of the pull request + +## Deployment with Codefresh +- Add encrypted environment variables for: + * GITHUB_TOKEN + +- Add "github-pr" step as descibed below + +```yaml +# codefresh.yml example with github pr step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push to registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + github-pr: + image: codefresh/github-pr-plugin + environment: + - GITHUB_REPO_OWNER=${{CF_REPO_OWNER}} + - GITHUB_REPO_NAME=${{CF_REPO_NAME}} + - BASE=master + - HEAD=${{CF_BRANCH}} + - TITLE=Codefresh PR for ${{CF_BRANCH}} +``` diff --git a/plugins/testplugin/plugin.yaml b/plugins/testplugin/plugin.yaml new file mode 100644 index 00000000..f3c970b9 --- /dev/null +++ b/plugins/testplugin/plugin.yaml @@ -0,0 +1,33 @@ +image: codefresh/github-pr-plugin +tag: master +version: 0.1.0 +description: Creates GitHub pull request +keywords: + - testplugin + +home: https://github.com/codefresh-io/github-pr-plugin +sources: + - https://github.com/codefresh-io/github-pr-plugin +maintainers: # (optional) + - name: Oleg Verhovsky + email: oleg@codefresh.io +icon: https://assets-cdn.github.com/images/modules/logos_page/GitHub-Mark.png +envs: + - name: GITHUB_TOKEN + type: required + description: Token for access to GitHub + - name: GITHUB_REPO_OWNER + type: required + description: Name of repo owner + - name: GITHUB_REPO_NAME + type: required + description: Name of repo + - name: HEAD + type: required + description: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this - username:branch + - name: BASE + type: required + description: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. + - name: TITLE + type: required + description: The title of the pull request diff --git a/plugins/testplugin1/README.MD b/plugins/testplugin1/README.MD new file mode 100644 index 00000000..27c33099 --- /dev/null +++ b/plugins/testplugin1/README.MD @@ -0,0 +1,42 @@ +# GitHub Pull Request Codefresh Plugin + +Creates a new pull request in GitHub + +## Environment Variables + +- `GITHUB_TOKEN`: token for access to GitHub +- `GITHUB_REPO_OWNER`: name of repo owner +- `GITHUB_REPO_NAME`: name of repo +- `HEAD`: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this: username:branch +- `BASE`: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. +- `TITLE`: The title of the pull request + +## Deployment with Codefresh +- Add encrypted environment variables for: + * GITHUB_TOKEN + +- Add "github-pr" step as descibed below + +```yaml +# codefresh.yml example with github pr step +version: '1.0' + +steps: + build-step: + type: build + image-name: repo/image:tag + + push to registry: + type: push + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + github-pr: + image: codefresh/github-pr-plugin + environment: + - GITHUB_REPO_OWNER=${{CF_REPO_OWNER}} + - GITHUB_REPO_NAME=${{CF_REPO_NAME}} + - BASE=master + - HEAD=${{CF_BRANCH}} + - TITLE=Codefresh PR for ${{CF_BRANCH}} +``` diff --git a/plugins/testplugin1/plugin.yaml b/plugins/testplugin1/plugin.yaml new file mode 100644 index 00000000..2c5a092b --- /dev/null +++ b/plugins/testplugin1/plugin.yaml @@ -0,0 +1,34 @@ +image: codefresh/github-pr-plugin +tag: master +version: 0.1.0 +description: test plugin1! +keywords: + - testplugin1 + + +home: https://github.com/codefresh-io/github-pr-plugin +sources: + - https://github.com/codefresh-io/github-pr-plugin +maintainers: # (optional) + - name: Oleg Verhovsky + email: oleg@codefresh.io +icon: https://assets-cdn.github.com/images/modules/logos_page/GitHub-Mark.png +envs: + - name: GITHUB_TOKEN + type: required + description: Token for access to GitHub + - name: GITHUB_REPO_OWNER + type: required + description: Name of repo owner + - name: GITHUB_REPO_NAME + type: required + description: Name of repo + - name: HEAD + type: required + description: The name of the branch where your changes are implemented. For cross-repository pull requests in the same network, namespace head with a user like this - username:branch + - name: BASE + type: required + description: The name of the branch you want the changes pulled into. This should be an existing branch on the current repository. You cannot submit a pull request to one repository that requests a merge to a base of another repository. + - name: TITLE + type: required + description: The title of the pull request diff --git a/plugins/twilio/README.md b/plugins/twilio/README.md new file mode 100644 index 00000000..da160f45 --- /dev/null +++ b/plugins/twilio/README.md @@ -0,0 +1,27 @@ +# Codefresh plugin for send SMS notification + +Codefresh plugin for send SMS notification via Twilio + +[![Codefresh build status]( https://g.codefresh.io/api/badges/pipeline/codefresh-inc/codefresh-io%2Ftwillio-plugin%2Ftwillio-plugin?branch=master&key=eyJhbGciOiJIUzI1NiJ9.NTY3MmQ4ZGViNjcyNGI2ZTM1OWFkZjYy.AN2wExsAsq7FseTbVxxWls8muNx_bBUnQWQVS8IgDTI&type=cf-2)]( https://g.codefresh.io/pipelines/twillio-plugin/builds?repoOwner=codefresh-io&repoName=twillio-plugin&serviceName=codefresh-io%2Ftwillio-plugin&filter=trigger:build~Build;branch:master;pipeline:5c1a73926ecec326b46fca2b~twillio-plugin) + +## Main env variables +- `TWILIO_SID` - Your account SID from Twilio console +- `TWILIO_TOKEN` - Your API Auth Token from Twilio console +- `TWILIO_PHONE_FROM` - Phone number from which messages will be sent +- `TWILIO_PHONE_TO` - Phone number to which messages will be sent +- `TWILIO_TYPE` - Type of your message [build - send info about your build via Codefresh, default - Send message with statc text] + +For **message** type you must provide `TWILIO_MESSAGE` env + +## Config for codefresh.yml +``` +version: '1.0' +... +steps: + ... + TestSMS: + title: Test SMS + image: codefresh/twilioplugin:latest + ... +... +``` diff --git a/plugins/twilio/plugin.yaml b/plugins/twilio/plugin.yaml new file mode 100644 index 00000000..5464cbaf --- /dev/null +++ b/plugins/twilio/plugin.yaml @@ -0,0 +1,30 @@ +image: codefresh/twilioplugin +version: latest +description: Send SMS message via Twilio +keywords: + - sms + - twilio +home: https://github.com/codefresh-io/twillio-plugin +sources: + - https://github.com/codefresh-io/twillio-plugin +maintainers: + - name: Pavel Kostohrys + email: pavel@codefresh.io +icon: https://cdn.iconscout.com/icon/free/png-256/twilio-1-285957.png +envs: + - name: TWILIO_SID + type: required + description: Your account SID from Twilio console + - name: TWILIO_TOKEN + type: required + description: Your API Auth Token from Twilio console + - name: TWILIO_PHONE_FROM + type: required + description: Phone number from which messages will be sent + - name: TWILIO_PHONE_TO + type: required + description: Phone number to which messages will be sent + - name: TWILIO_TYPE + description: Type of your message [build - send info about your build via Codefresh, default - Send message with statc text] + - name: TWILIO_MESSAGE + description: Required in default mode, text for message diff --git a/plugins/twistlock-scan/Dockerfile b/plugins/twistlock-scan/Dockerfile new file mode 100644 index 00000000..9f511082 --- /dev/null +++ b/plugins/twistlock-scan/Dockerfile @@ -0,0 +1,50 @@ +FROM ubuntu:xenial + +ENV LANG C.UTF-8 + +RUN { \ + echo '#!/bin/sh'; \ + echo 'set -e'; \ + echo; \ + echo 'dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"'; \ + } > /usr/local/bin/docker-java-home && \ + chmod +x /usr/local/bin/docker-java-home + +RUN apt-get update && apt-get install -y --no-install-recommends \ + bzip2 \ + unzip \ + xz-utils \ + apt-transport-https \ + ca-certificates \ + curl \ + software-properties-common \ + python3-openssl && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \ + add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" && \ + apt-get update && apt-get install -y --no-install-recommends \ + docker-ce=17.09.0~ce-0~ubuntu && \ + apt-get install -y \ + openjdk-8-jre \ + ; \ + rm -rf /var/lib/apt/lists/*; \ + \ + [ "$JAVA_HOME" = "$(docker-java-home)" ]; \ + \ + update-alternatives --get-selections | awk -v home="$JAVA_HOME" 'index($3, home) == 1 { $2 = "manual"; print | "update-alternatives --set-selections" }'; \ + update-alternatives --query java | grep -q 'Status: manual' && \ + mkdir /packages && \ + curl -o /packages/twistcli https://cdn.twistlock.com/support/twistcli && \ + curl -o /packages/nexus-iq-cli-1.38.0-02.jar https://download.sonatype.com/clm/scanner/nexus-iq-cli-1.38.0-02.jar + +COPY scripts /scripts + +RUN chmod +x -R /packages +RUN chmod +x -R /scripts + +WORKDIR /scripts + +ENTRYPOINT ["/usr/bin/python3"] +CMD [""] diff --git a/plugins/twistlock-scan/LICENSE.md b/plugins/twistlock-scan/LICENSE.md new file mode 100644 index 00000000..d39b0063 --- /dev/null +++ b/plugins/twistlock-scan/LICENSE.md @@ -0,0 +1,7 @@ +© 2017 Steelcase Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/plugins/twistlock-scan/README.md b/plugins/twistlock-scan/README.md new file mode 100644 index 00000000..150dcd3c --- /dev/null +++ b/plugins/twistlock-scan/README.md @@ -0,0 +1,105 @@ +# Security Scanning Tools [![Codefresh build status]( https://g.codefresh.io/api/badges/build?repoOwner=SC-TechDev&repoName=docker-security-scanner&branch=master&pipelineName=docker-security-scanner&accountName=sctechdevservice&type=cf-1)]( https://g.codefresh.io/repositories/SC-TechDev/docker-security-scanner/builds?filter=trigger:build;branch:master;service:59e62c5410e3d100019e7f3d~docker-security-scanner) + +Docker image which invokes security script using TwistCLI (Nexus coming soon) + +### Prerequisites: + +Codefresh Subscription (Dedicated Infrastructure) - https://codefresh.io/ + +Twistlock Subscription - https://www.twistlock.com/ + +### Documentation: + +Twistlock CLI: https://twistlock.desk.com/customer/en/portal/articles/2875595-twistcli?b_id=16619 + +Nexus IQ CLI: TBD + +## Script Library + +### twistlock.py + +Executes TwistCLI to scan Docker image given. + +### options + +To use an ENVIRONMENT VARIABLE you need to add the variables to your Codefresh Pipeline and also to your codefresh.yaml. + + +Example `codefresh.yml` build is below with required ENVIRONMENT VARIABLES in place. + + +| ENVIRONMENT VARIABLE | SCRIPT ARGUMENT | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|----------------------------|--------------------------------------|----------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------| +| CF_METADATA | [ -c, --cf_metadata ] | null | boolean | No | In combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation | +| TL_CONSOLE_HOSTNAME | [ -C, --tl_console_hostname ] | null | string | Yes | hostname/ip | +| TL_CONSOLE_PORT | [ -P, --tl_console_port ] | null | string | Yes | port | +| TL_CONSOLE_USERNAME | [ -U, --tl_console_username ] | null | string | Yes | username | +| TL_CONSOLE_PASSWORD | [ -X, --tl_console_password ] | null | string | Yes | password | +| TL_ONLY | [ -Z, --tl_only ] | null | boolean | Yes | Twistlock Console Only (Required for now Nexus TBD) | +| TL_TLS_ENABLED | [ -T, --tl_tls_enabled ] | null | boolean | No | enable TLS | +| TL_HASH | [ -H, --tl_hash ] | [ sha1 ] | string | No | [ md5, sha1, sha256 ] hashing algorithm | +| TL_UPLOAD | [ -R, --tl_upload ] | null | boolean | No | ( ignores all options below if set and only returns report url ) uploads report to Twistlock to be used later via Twistlock API | +| TL_DETAILS | [ -D, --tl_details ] | null | boolean | No | prints an itemized list of each vulnerability found by the scanner | +| TL_ONLY_FIXED | [ -O, --tl_only_fixed ] | null | boolean | No | reports just the vulnerabilites that have fixes available | +| TL_COMPLIANCE_THRESHOLD | [ -M, --tl_compliance_threshold ] | null | string | No | [ low, medium, high ] sets the the minimal severity compliance issue that returns a fail exit code | +| TL_VULNERABILITY_THRESHOLD | [ -V, --tl_vulnerability_threshold ] | null | string | No | [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code | + +### codefresh.yml + +Codefresh Build Step to execute Twistlock scan. +All `${{var}}` variables must be put into Codefresh Build Parameters +codefresh.yml +```console + buildimage: + type: build + title: Build Runtime Image + dockerfile: Dockerfile + image_name: # Image you're building/scanning [repository/image] + tag: latest-cf-build-candidate + + nexus_iq_scan_build_stage: + type: composition + composition: + version: '2' + services: + imagebuild: + image: ${{buildimage}} + command: sh -c "exit 0" + labels: + build.image.id: ${{CF_BUILD_ID}} + composition_candidates: + scan_service: + image: sctechdev/docker-security-scanner + environment: + - TL_CONSOLE_HOSTNAME=${{TL_CONSOLE_HOSTNAME}} + - TL_CONSOLE_PORT=${{TL_CONSOLE_PORT}} + - TL_CONSOLE_USERNAME=${{TL_CONSOLE_USERNAME}} + - TL_CONSOLE_PASSWORD=${{TL_CONSOLE_PASSWORD}} + - TL_ONLY=${{TL_ONLY}} + command: twistlock.py -i "$$(docker inspect $$(docker inspect $$(docker ps -aqf label=build.image.id=${{CF_BUILD_ID}}) -f {{.Config.Image}}) -f {{.Id}} | sed 's/sha256://g')" + depends_on: + - imagebuild + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /var/lib/docker:/var/lib/docker + # Everything below this line is Optional for CF_METADATA + - '${{CF_VOLUME_NAME}}:/codefresh/volume' + add_flow_volume_to_composition: true + + export: + title: "Exporting variables..." + image: alpine + commands: + - echo "Exporting variables..." + + set_metadata: + title: "Setting metadata on image..." + image: alpine + commands: + - echo "Setting metadata on image..." + on_finish: + metadata: + set: + - '${{build_step.imageId}}': + - TwistlockSecurityReport: ${{TL_REPORT_URL}} +``` \ No newline at end of file diff --git a/plugins/twistlock-scan/codefresh.yml b/plugins/twistlock-scan/codefresh.yml new file mode 100644 index 00000000..31881bcb --- /dev/null +++ b/plugins/twistlock-scan/codefresh.yml @@ -0,0 +1,41 @@ +version: '1.0' + +steps: + + buildimage: + type: build + description: image build step + dockerfile: Dockerfile + image_name: sctechdev/docker-security-scanner + tag: latest-cf-build-candidate + + push_image: + type: push + candidate: ${{buildimage}} + tag: latest + when: + branch: + only: + - master + push_image1: + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + + push_image_nexus_latest: + title: Push to Nexus Repo (latest) + type: push + candidate: ${{buildimage}} + tag: latest + registry: sonatype-docker-internal + when: + branch: + only: + - master + + push_image_neuxs_gitbranch_gitsha: + title: Push to Nexus Repo (gitbranch + gitsha) + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + registry: sonatype-docker-internal diff --git a/plugins/twistlock-scan/plugin.yaml b/plugins/twistlock-scan/plugin.yaml new file mode 100644 index 00000000..2aa31afd --- /dev/null +++ b/plugins/twistlock-scan/plugin.yaml @@ -0,0 +1,65 @@ +image: docker.io/sctechdev/docker-security-scanner +tag: master-c81e6d4 +version: 2.2 +description: Execute Twistlock image scan as build step +keywords: + - Twistlock 2.2 +home: https://hub.docker.com/r/sctechdev/docker-security-scanner/ +sources: + - https://github.com/SC-TechDev/docker-security-scanner +maintainers: + - name: Dustin Van Buskirk + email: dev@vanbuskirk.me + - name: Varun Tagore + email: rondevops@gmail.com +icon: A URL to an SVG or PNG image to be used as an icon (optional) +envs: + - name: CF_METADATA + type: required + description: Boolean; combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation + - name: TL_CONSOLE_HOSTNAME + type: required + description: Hostname or IP of Twistlock Console + - name: TL_CONSOLE_PORT + type: required + description: Port of Twistlock Console + - name: TL_CONSOLE_USERNAME + type: required + description: Username of Twistlock Console + - name: TL_CONSOLE_PASSWORD + type: required + description: Password of Twistlock Console User + - name: TL_ONLY + type: required + description: Twistlock Console Scan Only (No Nexus) + - name: TL_TLS_ENABLED + type: optional + description: Boolean; Enable TLS connection to Twistlock Console + - name: TL_HASH + type: optional + description: Hashing Algorithm to use + - name: TL_UPLOAD + type: optional + description: Upload report to Twistlock Console and return URL (Overrides all other options only returns URL) + - name: TL_DETAILS + type: optional + description: Prints an itemized list of each vulnerability found by the scanner + - name: TL_ONLY_FIXED + type: optional + description: reports just the vulnerabilites that have fixes available + - name: TL_COMPLIANCE_THRESHOLD + type: optional + description: [ low, medium, high ] sets the the minimal severity compliance issue that returns a fail exit code + - name: TL_VULNERABILITY_THRESHOLD + type: optional + description: [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code +volumes: + - name: /var/run/docker.sock:/var/run/docker.sock + required: true + description: Docker socket for DIND + - name: /var/lib/docker:/var/lib/docker + required: true + description: Docker lib access for DIND + - name: '${{CF_VOLUME_NAME}}:/codefresh/volume' + required: false + description: Volume required if setting Docker image metadata using Codefresh \ No newline at end of file diff --git a/plugins/twistlock-scan/scripts/twistlock.py b/plugins/twistlock-scan/scripts/twistlock.py new file mode 100644 index 00000000..50510ebf --- /dev/null +++ b/plugins/twistlock-scan/scripts/twistlock.py @@ -0,0 +1,195 @@ +import sys +import subprocess +import time +import os +import getopt +import ssl +import re + +def main(argv): + try: + st_scanner_jar = '/packages/nexus-iq-cli-1.38.0-02.jar' + tl_scanner_exec = '/packages/twistcli' + cf_metadata = os.environ.get('CF_METADATA') + docker_image_id = os.environ.get('DOCKER_IMAGE_ID') + st_application_id = os.environ.get('NEXUS_IQ_APPLICATION_ID') + st_url = os.environ.get('NEXUS_IQ_URL') + st_username = os.environ.get('NEXUS_IQ_USERNAME') + st_password = os.environ.get('NEXUS_IQ_PASSWORD') + st_stage = os.environ.get('NEXUS_IQ_STAGE', 'Build') + tl_console_hostname = os.environ.get('TL_CONSOLE_HOSTNAME') + tl_console_port = os.environ.get('TL_CONSOLE_PORT') + tl_console_username = os.environ.get('TL_CONSOLE_USERNAME') + tl_console_password = os.environ.get('TL_CONSOLE_PASSWORD') + tl_only = os.environ.get('TL_ONLY') + tl_tls_enabled = os.environ.get('TL_TLS_ENABLED') + tl_hash = os.environ.get('TL_HASH', 'sha1') + tl_include_package_files = os.environ.get('TL_INCLUDE_PACKAGE_FILES') + tl_upload = os.environ.get('TL_UPLOAD') + tl_details = os.environ.get('TL_DETAILS') + tl_only_fixed = os.environ.get('TL_ONLY_FIXED') + tl_compliance_threshold = os.environ.get('TL_COMPLIANCE_THRESHOLD') + tl_vulnerability_threshold = os.environ.get('TL_VULNERABILITY_THRESHOLD') + java_home = os.environ.get('JAVA_HOME', '/usr/lib/jvm/java-8-openjdk-amd64') + java_keystore_password = os.environ.get('JAVA_KEYSTORE_PASSWORD', 'changeit') + opts, args = getopt.getopt(argv,"h:c:i:a:j:u:p:s:t:E:C:P:U:X:Z:J:K:T:H:F:R:D:O:M:V:", + ["help", "docker_image_id=", "cf_metadata", "st_application_id=", "st_scanner_jar=", "st_url=", "st_username=", "st_password=", "st_stage=", + "tl_scanner_exec=", "tl_console_hostname", "tl_console_port", "tl_console_username=", "tl_console_password=", "tl_only", + "tl_tls_enabled", "tl_hash", "tl_include_package_files", "tl_upload", "tl_details", "tl_only_fixed", "tl_compliance_threshold", + "tl_vulnerability_threshold", "java_home=", "java_keystore_password" + ] + ) + except getopt.GetoptError: + print('Unrecognized Argument! See arguments list using -h or --help. Ex. twistlock.py --help') + sys.exit(2) + for opt, arg in opts: + if opt == ("h","--help"): + print('twistlock.py --arg value or twistlock.py -a value') + print('-c --cf_metadata - Adds scanner info to Docker image metadata for Codefresh builds') + print('-i --docker_image_id [DOCKER_IMAGE_ID] - Docker Image ID short or long IDs accepted') + print('-a --st_application_id [NEXUS_IQ_APPLICATION_ID] - Applications ID in Nexus IQ') + print('-j --st_scanner_jar - Location of nexus-iq-cli*.jar file') + print('-u --st_username [NEXUS_IQ_USERNAME] - Nexus IQ Username') + print('-p --st_password [NEXUS_IQ_PASSWORD] - Password for Nexus IQ Username') + print('-s --st_url [NEXUS_IQ_URL] - Sonatype URL must be HTTPS with Valid Cert') + print('-t --st_stage [NEXUS_IQ_STAGE] - Sonatype Stage') + print('-E --tl_scanner_exec - Location of twistlock-scanner executable') + print('-C --tl_console_hostname [TL_CONSOLE_HOSTNAME] - Hostname/IP for Twistlock Console') + print('-P --tl_console_port [TL_CONSOLE_PORT] - Twistock Console port') + print('-U --tl_console_username [TL_CONSOLE_USERNAME] - Twistlock Console Username') + print('-X --tl_console_password [TL_CONSOLE_PASSWORD] - Password for Twistlock Console Username') + print('-Z --tl_only [TL_ONLY] - Run a stand-alone Twistlock scan') + print('-T --tl_tls_enabled [TL_TLS_ENABLED] - Enabled TLS/HTTPS for Twistlock scan') + print('-H --tl_hash [TL_HASH] - Specifies the hashing algorithm. Supported values are md5, sha1, and sha256') + print('-F --tl_include_package_files [TL_INCLUDE_PACKAGE_FILES] - List all packages in the image') + print('-R --tl_upload [TL_UPLOAD] - Whether to upload the scan result') + print('-D --tl_details [TL_DETAILS] - Prints an itemized list of each vulnerability found by the scanner') + print('-O --tl_only_fixed [TL_ONLY_FIXED] - Reports just the vulnerabilities that have fixes available') + print('-M --tl_compliance_threshold [TL_COMPLIANCE_THRESHOLD] - Sets the minimum severity compliance issue that returns a fail exit code') + print('-V --tl_vulnerability_threshold [TL_VULNERABILITY_THRESHOLD] - Sets the minimum severity vulnerability that returns a fail exit code') + print('-J --java_home [JAVA_HOME] - Java Home Directory (no trailing /)') + print('-K --java_keystore_password [JAVA_KEYSTORE_PASSWORD] - Java Keystore Password') + sys.exit() + elif opt in ("-c", "--cf_metadata"): + cf_metadata = arg + elif opt in ("-i", "--docker_image_id"): + docker_image_id = arg + elif opt in ("-a", "--st_application_id"): + st_application_id = arg + elif opt in ("-j", "--st_scanner_jar"): + st_scanner_jar = arg + elif opt in ("-s", "--st_url"): + st_url = arg + elif opt in ("-u", "--st_username"): + st_username = arg + elif opt in ("-p", "--st_password"): + st_password = arg + elif opt in ("-t", "--st_stage"): + st_stage = arg + elif opt in ("-E", "--tl_scanner_exec"): + tl_scanner_exec = arg + elif opt in ("-C", "--tl_console_hostname"): + tl_console_hostname = arg + elif opt in ('-P', "--tl_console_port"): + tl_console_port = arg + elif opt in ("-U", "--tl_console_username"): + tl_console_username = arg + elif opt in ('-X', "--tl_console_password"): + tl_console_password = arg + elif opt in ('-Z', "--tl_only"): + tl_only = arg + elif opt in ('-T', "--tl_tls_enabled"): + tl_tls_enabled = arg + elif opt in ('-H', "--tl_hash"): + tl_hash = arg + elif opt in ('-F', "--tl_include_package_files"): + tl_include_package_files = arg + elif opt in ('-R', "--tl_upload"): + tl_upload = arg + elif opt in ('-D', "--tl_details"): + tl_details = arg + elif opt in ('-O', "--tl_only_fixed"): + tl_only_fixed = arg + elif opt in ('-M', "--tl_compliance_threshold"): + tl_compliance_threshold = arg + elif opt in ('-V', "--tl_vulnerability_threshold"): + tl_vulnerability_threshold = arg + elif opt in ('-J', "--java_home"): + java_home = arg + elif opt in ('-K', "--java_keystore_password"): + java_keystore_password = arg + + # Determine if TLS is required + if not (tl_only or tl_tls_enable): + # Download and store Twistlock Console site cert + cert = ssl.get_server_certificate((tl_console_hostname, tl_console_port)) + cert, file=open("twistlock.cer", "w") + + # Run stand-alone Twistlock Scan + if tl_only: + + # Determine Protocol + tl_console_protocol = 'https' if tl_tls_enabled else 'http' + + # Base twistcli commnad to scan images + twistcli_base_command = '/packages/twistcli images scan' + + # Required twistcli options to successfully scan image + twistcli_required_options = ("--address '{}://{}:{}' --user '{}' --password '{}' --hash '{}'" + .format(tl_console_protocol, tl_console_hostname, tl_console_port, tl_console_username, tl_console_password, tl_hash)) + + # Optional twistcli options + options = [] + if tl_include_package_files: + options.append("--include-package-files") + if tl_upload: + options.append("--upload") + if tl_details: + options.append("--details") + if tl_compliance_threshold: + options.append("--compliance-threshold '{}'".format(tl_compliance_threshold)) + if tl_vulnerability_threshold: + options.append("--vulnerability-threshold '{}'".format(tl_vulnerability_threshold)) + twistcli_optional_options = ' '.join(options) + + # Concatenate twistcli executable with command + twistcli_exec = ' '.join([twistcli_base_command, twistcli_required_options, twistcli_optional_options, docker_image_id]) + # Execute command but pipe stdout to variable and parse for Twistlock URL + if cf_metadata: + proc = subprocess.Popen(twistcli_exec, shell=True, stdout=subprocess.PIPE) + stdout = proc.communicate()[0].decode('utf-8').strip('\n') + tl_report_url = ''.join(re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', stdout)) + with open('/codefresh/volume/env_vars_to_export', 'a') as f: + print('Twistlock Report: ' + tl_report_url) + f.write('TL_REPORT_URL=' + tl_report_url) + f.close() + # Execute command and send stdout to console + else: + proc = subprocess.Popen(twistcli_exec, shell=True) + stdout, stderr = proc.communicate() + if proc.returncode != 0: + sys.exit(1) + + else: + + # Import site cert into java keystore + command = ['keytool -importcert -noprompt -file twistlock.cer -alias twistlock -storepass {} -keystore {}/jre/lib/security/cacerts' + .format(java_keystore_password, java_home) + ] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + + # Start Docker + command = ['for i in {1..5}; do service docker start && break || sleep 15; done'] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + + # Run Twistlock Scan and send file to Sonatype + command = ["java -cp {} com.sonatype.insight.scan.cli.TwistlockPolicyEvaluatorCli -i {} -a '{}:{}' -s '{}' --twistlock-scanner-executable {} --twistlock-console-url https://{}:{} --twistlock-console-username {} --twistlock-console-password '{}' --stage '{}' {}" + .format(st_scanner_jar, st_application_id, st_username, st_password, st_url, tl_scanner_exec, tl_console_hostname, tl_console_port, tl_console_username, tl_console_password, st_stage, docker_image_id) + ] + proc = subprocess.Popen(command, shell=True) + stdout, stderr = proc.communicate() + +if __name__ == "__main__": + main(sys.argv[1:]) \ No newline at end of file diff --git a/plugins/vault/README.md b/plugins/vault/README.md new file mode 100644 index 00000000..855f936a --- /dev/null +++ b/plugins/vault/README.md @@ -0,0 +1,51 @@ +# cf-vault-plugin + +Use Codefresh [Vault](https://www.vaultproject.io) plugin to make key-value pairs stored in a vault available as environment variables for further steps. + +NOTE: this plugin currently supports token authentication and Key/Value secrets engine only. + + +## Usage + +Set required and optional environment variables and add the following step to your Codefresh pipeline: + +Example Variables: + +The example below will authenticate to vault server `https://vault.testdomain.io:8200` using token `s.4wtaMJuZ7dv0c4XuRaasLUOG` and export all secrets found in path `secret/codefreshsecret` as ENV variables available for further steps. + +```text +VAULT_ADDR=https://vault.testdomain.io:8200 +VAULT_PATH=secret/codefreshsecret +VAULT_AUTH_TOKEN=s.4wtaMJuZ7dv0c4XuRaasLUOG +``` + + +```yaml +--- +version: '1.0' + +steps: + + ... + + Vault_to_Env: + title: Importing vault values + image: 'codefresh/cf-vault-plugin' + environment: + - VAULT_ADDR=${{VAULT_ADDR}} + - VAULT_PATH=${{VAULT_PATH}} + - VAULT_AUTH_TOKEN=${{VAULT_AUTH_TOKEN}} + + ... + +``` + +## Environment Variables + +| Variables | Required | Default | Description | +|----------------|----------|---------|-----------------------------------------------------------------------------------------| +| VAULT_ADDR | YES | | Vault server URI | +| VAULT_PATH | YES | | Path to secrets in vault | +| VAULT_AUTH_TOKEN | YES | | Vault authentication token | +| VAULT_CLIENT_CERT_BASE64 | NO | | Base64 encoded client cerificate | +| VAULT_CLIENT_KEY_BASE64 | NO | | Base64 encoded client key diff --git a/plugins/vault/plugin.yaml b/plugins/vault/plugin.yaml new file mode 100644 index 00000000..09f71819 --- /dev/null +++ b/plugins/vault/plugin.yaml @@ -0,0 +1,27 @@ +image: codefresh/cf-vault-plugin +tag: latest +version: 0.1.0 +description: The plugin exports KV pairs from Hashicorp Vault to Codefresh pipeline ENV variables +keywords: + - vault + - hashicorp +home: https://github.com/codefresh-io/cf-vault-plugin +sources: + - https://github.com/codefresh-io/cf-vault-plugin +maintainers: # (optional) + - name: Alexander Aladov + email: a.aladov@codefresh.io +envs: + - name: VAULT_ADDR + type: required + description: "Vault server URI. Example: https://vault.testdomain.io:8200" + - name: VAULT_PATH + type: required + description: "Path to secrets in vault. Example: secret/codefreshsecret" + - name: VAULT_AUTH_TOKEN + type: required + description: "Vault authentication token" + - name: VAULT_CLIENT_CERT_BASE64 + description: "Base64 encoded client cerificate" + - name: VAULT_CLIENT_KEY_BASE64 + description: "Base64 encoded client key" diff --git a/plugins/versioner/Dockerfile b/plugins/versioner/Dockerfile new file mode 100644 index 00000000..9f511082 --- /dev/null +++ b/plugins/versioner/Dockerfile @@ -0,0 +1,50 @@ +FROM ubuntu:xenial + +ENV LANG C.UTF-8 + +RUN { \ + echo '#!/bin/sh'; \ + echo 'set -e'; \ + echo; \ + echo 'dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"'; \ + } > /usr/local/bin/docker-java-home && \ + chmod +x /usr/local/bin/docker-java-home + +RUN apt-get update && apt-get install -y --no-install-recommends \ + bzip2 \ + unzip \ + xz-utils \ + apt-transport-https \ + ca-certificates \ + curl \ + software-properties-common \ + python3-openssl && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \ + add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" && \ + apt-get update && apt-get install -y --no-install-recommends \ + docker-ce=17.09.0~ce-0~ubuntu && \ + apt-get install -y \ + openjdk-8-jre \ + ; \ + rm -rf /var/lib/apt/lists/*; \ + \ + [ "$JAVA_HOME" = "$(docker-java-home)" ]; \ + \ + update-alternatives --get-selections | awk -v home="$JAVA_HOME" 'index($3, home) == 1 { $2 = "manual"; print | "update-alternatives --set-selections" }'; \ + update-alternatives --query java | grep -q 'Status: manual' && \ + mkdir /packages && \ + curl -o /packages/twistcli https://cdn.twistlock.com/support/twistcli && \ + curl -o /packages/nexus-iq-cli-1.38.0-02.jar https://download.sonatype.com/clm/scanner/nexus-iq-cli-1.38.0-02.jar + +COPY scripts /scripts + +RUN chmod +x -R /packages +RUN chmod +x -R /scripts + +WORKDIR /scripts + +ENTRYPOINT ["/usr/bin/python3"] +CMD [""] diff --git a/plugins/versioner/LICENSE.md b/plugins/versioner/LICENSE.md new file mode 100644 index 00000000..d39b0063 --- /dev/null +++ b/plugins/versioner/LICENSE.md @@ -0,0 +1,7 @@ +© 2017 Steelcase Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/plugins/versioner/README.md b/plugins/versioner/README.md new file mode 100644 index 00000000..150dcd3c --- /dev/null +++ b/plugins/versioner/README.md @@ -0,0 +1,105 @@ +# Security Scanning Tools [![Codefresh build status]( https://g.codefresh.io/api/badges/build?repoOwner=SC-TechDev&repoName=docker-security-scanner&branch=master&pipelineName=docker-security-scanner&accountName=sctechdevservice&type=cf-1)]( https://g.codefresh.io/repositories/SC-TechDev/docker-security-scanner/builds?filter=trigger:build;branch:master;service:59e62c5410e3d100019e7f3d~docker-security-scanner) + +Docker image which invokes security script using TwistCLI (Nexus coming soon) + +### Prerequisites: + +Codefresh Subscription (Dedicated Infrastructure) - https://codefresh.io/ + +Twistlock Subscription - https://www.twistlock.com/ + +### Documentation: + +Twistlock CLI: https://twistlock.desk.com/customer/en/portal/articles/2875595-twistcli?b_id=16619 + +Nexus IQ CLI: TBD + +## Script Library + +### twistlock.py + +Executes TwistCLI to scan Docker image given. + +### options + +To use an ENVIRONMENT VARIABLE you need to add the variables to your Codefresh Pipeline and also to your codefresh.yaml. + + +Example `codefresh.yml` build is below with required ENVIRONMENT VARIABLES in place. + + +| ENVIRONMENT VARIABLE | SCRIPT ARGUMENT | DEFAULT | TYPE | REQUIRED | DESCRIPTION | +|----------------------------|--------------------------------------|----------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------| +| CF_METADATA | [ -c, --cf_metadata ] | null | boolean | No | In combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation | +| TL_CONSOLE_HOSTNAME | [ -C, --tl_console_hostname ] | null | string | Yes | hostname/ip | +| TL_CONSOLE_PORT | [ -P, --tl_console_port ] | null | string | Yes | port | +| TL_CONSOLE_USERNAME | [ -U, --tl_console_username ] | null | string | Yes | username | +| TL_CONSOLE_PASSWORD | [ -X, --tl_console_password ] | null | string | Yes | password | +| TL_ONLY | [ -Z, --tl_only ] | null | boolean | Yes | Twistlock Console Only (Required for now Nexus TBD) | +| TL_TLS_ENABLED | [ -T, --tl_tls_enabled ] | null | boolean | No | enable TLS | +| TL_HASH | [ -H, --tl_hash ] | [ sha1 ] | string | No | [ md5, sha1, sha256 ] hashing algorithm | +| TL_UPLOAD | [ -R, --tl_upload ] | null | boolean | No | ( ignores all options below if set and only returns report url ) uploads report to Twistlock to be used later via Twistlock API | +| TL_DETAILS | [ -D, --tl_details ] | null | boolean | No | prints an itemized list of each vulnerability found by the scanner | +| TL_ONLY_FIXED | [ -O, --tl_only_fixed ] | null | boolean | No | reports just the vulnerabilites that have fixes available | +| TL_COMPLIANCE_THRESHOLD | [ -M, --tl_compliance_threshold ] | null | string | No | [ low, medium, high ] sets the the minimal severity compliance issue that returns a fail exit code | +| TL_VULNERABILITY_THRESHOLD | [ -V, --tl_vulnerability_threshold ] | null | string | No | [ low, medium, high, critical ] sets the minimal severity vulnerability that returns a fail exit code | + +### codefresh.yml + +Codefresh Build Step to execute Twistlock scan. +All `${{var}}` variables must be put into Codefresh Build Parameters +codefresh.yml +```console + buildimage: + type: build + title: Build Runtime Image + dockerfile: Dockerfile + image_name: # Image you're building/scanning [repository/image] + tag: latest-cf-build-candidate + + nexus_iq_scan_build_stage: + type: composition + composition: + version: '2' + services: + imagebuild: + image: ${{buildimage}} + command: sh -c "exit 0" + labels: + build.image.id: ${{CF_BUILD_ID}} + composition_candidates: + scan_service: + image: sctechdev/docker-security-scanner + environment: + - TL_CONSOLE_HOSTNAME=${{TL_CONSOLE_HOSTNAME}} + - TL_CONSOLE_PORT=${{TL_CONSOLE_PORT}} + - TL_CONSOLE_USERNAME=${{TL_CONSOLE_USERNAME}} + - TL_CONSOLE_PASSWORD=${{TL_CONSOLE_PASSWORD}} + - TL_ONLY=${{TL_ONLY}} + command: twistlock.py -i "$$(docker inspect $$(docker inspect $$(docker ps -aqf label=build.image.id=${{CF_BUILD_ID}}) -f {{.Config.Image}}) -f {{.Id}} | sed 's/sha256://g')" + depends_on: + - imagebuild + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /var/lib/docker:/var/lib/docker + # Everything below this line is Optional for CF_METADATA + - '${{CF_VOLUME_NAME}}:/codefresh/volume' + add_flow_volume_to_composition: true + + export: + title: "Exporting variables..." + image: alpine + commands: + - echo "Exporting variables..." + + set_metadata: + title: "Setting metadata on image..." + image: alpine + commands: + - echo "Setting metadata on image..." + on_finish: + metadata: + set: + - '${{build_step.imageId}}': + - TwistlockSecurityReport: ${{TL_REPORT_URL}} +``` \ No newline at end of file diff --git a/plugins/versioner/codefresh.yml b/plugins/versioner/codefresh.yml new file mode 100644 index 00000000..31881bcb --- /dev/null +++ b/plugins/versioner/codefresh.yml @@ -0,0 +1,41 @@ +version: '1.0' + +steps: + + buildimage: + type: build + description: image build step + dockerfile: Dockerfile + image_name: sctechdev/docker-security-scanner + tag: latest-cf-build-candidate + + push_image: + type: push + candidate: ${{buildimage}} + tag: latest + when: + branch: + only: + - master + push_image1: + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + + push_image_nexus_latest: + title: Push to Nexus Repo (latest) + type: push + candidate: ${{buildimage}} + tag: latest + registry: sonatype-docker-internal + when: + branch: + only: + - master + + push_image_neuxs_gitbranch_gitsha: + title: Push to Nexus Repo (gitbranch + gitsha) + type: push + candidate: ${{buildimage}} + tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}} + registry: sonatype-docker-internal diff --git a/plugins/versioner/plugin.yaml b/plugins/versioner/plugin.yaml new file mode 100644 index 00000000..2c9c70e5 --- /dev/null +++ b/plugins/versioner/plugin.yaml @@ -0,0 +1,32 @@ +image: docker.io/codefresh/versioner +tag: latest +version: 1.0 +description: semver versions builder +keywords: + - versioner 1.0 +#home: https://hub.docker.com/r/sctechdev/docker-security-scanner/ +sources: + - https://github.com/codefresh-io/cf-plugin-versioner.git +maintainers: + - name: Oleg Verhovsky + email: oleg@codefresh.io + + +#icon: A URL to an SVG or PNG image to be used as an icon (optional) +envs: + + - name: CURRENT_VERSION_SOURCE # from_file , from_tag, + type : required + description : action + - name: MAJOR + type: optional + - name: MINOR + type: optional + description: Boolean; combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation + - name: PATCH + type: optional + description: Boolean; combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation + - name: BUILD_NUM + type: optional + description: Boolean; combination with TL_UPLOAD stores Twistlock Report URL in TL_REPORT_URL var for Codefresh metadata annotation + \ No newline at end of file diff --git a/tools/Dockerfile b/tools/Dockerfile new file mode 100644 index 00000000..c115655b --- /dev/null +++ b/tools/Dockerfile @@ -0,0 +1,7 @@ +FROM node:alpine +WORKDIR /src +COPY ./package.json /src +RUN npm install +COPY . /src + +CMD ["node", "/src/index.js"] \ No newline at end of file diff --git a/tools/index.js b/tools/index.js new file mode 100644 index 00000000..57bf1f33 --- /dev/null +++ b/tools/index.js @@ -0,0 +1,77 @@ + + +//list folders +//build catalog + +const { lstatSync, readdirSync } = require('fs') +const { basename, join } = require('path') +const kefir = require('kefir'); +const _ = require('lodash'); +const YAML = require('yamljs'); +const Mustache = require('mustache'); +const fs = require('fs'); +const debug = require('debug'); + +const pluginsDir = process.env.PLUGINS || "../plugins" +debug(`plugins path is ${pluginsDir}`); +const isDirectory = source => lstatSync(source).isDirectory() +const getDirectories = source => + readdirSync(source) + .map(name => join(source, name)).filter(isDirectory); +const getContent = source => + readdirSync(source) + .map(name => join(source, name)) + +const validatePlugin = (p)=>{return p;} +const catalog = "dynamic-catalog.md"; +let plugins = kefir.sequentially(0, getDirectories(pluginsDir)).map(validatePlugin) + + +const createMD = (template , data)=>{ + Mustache.parse(template); // optional, speeds up future uses + data.date = new Date(); + var rendered = Mustache.render(template, data); + return rendered; +} + + let pluginData = plugins.flatMap((plugin)=>{ + let yaml = _.chain(getContent(plugin)).map((f)=>{ + + return f; + }).thru((f)=>{ + return f; + }).filter((file)=> + (basename(file) === "plugin.yaml")) + .first().value(); + let pluginMeta; + try{ + pluginMeta = YAML.load(yaml) + }catch(e){ + return kefir.constantError(e) + } + return (_.isUndefined(yaml)) ? kefir.never() : kefir.constant(pluginMeta) + + }).ignoreErrors().scan((plugins , p)=>{ + plugins.push(p); + return plugins; + }, []).spy().last() + + let template = kefir.fromNodeCallback(_.partial(fs.readFile, "template.md")) + .map((f)=>new Buffer(f).toString()).spy(); + + + kefir.concat([template, pluginData]).scan((prev, next)=>{ + prev.push(next); + return prev; + }, []).last().spy('->').map((data)=>{ + let t = _.first(data); + let plugins = _.last(data); + // plugins = {"plugins": [{image:"1"}, {"image":"2"}]} + return createMD(t, {plugins}); + }).flatMap((data)=>{ + return kefir.fromNodeCallback( + _.partial(fs.writeFile, catalog, data) + ) + }).log(); + + \ No newline at end of file diff --git a/tools/package-lock.json b/tools/package-lock.json new file mode 100644 index 00000000..6898b1f4 --- /dev/null +++ b/tools/package-lock.json @@ -0,0 +1,107 @@ +{ + "name": "catalog-creator", + "version": "1.0.0", + "lockfileVersion": 1, + "dependencies": { + "argparse": { + "version": "1.0.10", + "resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.10.tgz", + "integrity": "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg==" + }, + "balanced-match": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.0.tgz", + "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c=" + }, + "brace-expansion": { + "version": "1.1.11", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", + "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==" + }, + "concat-map": { + "version": "0.0.1", + "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", + "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s=" + }, + "debug": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.1.0.tgz", + "integrity": "sha512-heNPJUJIqC+xB6ayLAMHaIrmN9HKa7aQO8MGqKpvCA+uJYVcvR6l5kgdrhRuwPFHU7P5/A1w0BjByPHwpfTDKg==" + }, + "fs.realpath": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", + "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8=" + }, + "glob": { + "version": "7.1.3", + "resolved": "https://registry.npmjs.org/glob/-/glob-7.1.3.tgz", + "integrity": "sha512-vcfuiIxogLV4DlGBHIUOwI0IbrJ8HWPc4MU7HzviGeNho/UJDfi6B5p3sHeWIQ0KGIU0Jpxi5ZHxemQfLkkAwQ==" + }, + "inflight": { + "version": "1.0.6", + "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", + "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=" + }, + "inherits": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", + "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" + }, + "kefir": { + "version": "3.8.5", + "resolved": "https://registry.npmjs.org/kefir/-/kefir-3.8.5.tgz", + "integrity": "sha512-u4UxHyIvdOOM62Y/yAtYPeYEg/yUfwl5/QF3ksrTRxEdhpa7LAFChntZxVqbcf0gCGblZzL/JnV/gZYWOps3Qw==" + }, + "lodash": { + "version": "4.17.11", + "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz", + "integrity": "sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg==" + }, + "minimatch": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz", + "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==" + }, + "ms": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz", + "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg==" + }, + "mustache": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/mustache/-/mustache-3.0.1.tgz", + "integrity": "sha512-jFI/4UVRsRYdUbuDTKT7KzfOp7FiD5WzYmmwNwXyUVypC0xjoTL78Fqc0jHUPIvvGD+6DQSPHIt1NE7D1ArsqA==" + }, + "once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=" + }, + "path-is-absolute": { + "version": "1.0.1", + "resolved": "http://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", + "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18=" + }, + "sprintf-js": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.0.3.tgz", + "integrity": "sha1-BOaSb2YolTVPPdAVIDYzuFcpfiw=" + }, + "symbol-observable": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/symbol-observable/-/symbol-observable-1.0.4.tgz", + "integrity": "sha1-Kb9hXUqnEhvdiYsi1LP5vE4qoD0=" + }, + "wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=" + }, + "yamljs": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/yamljs/-/yamljs-0.3.0.tgz", + "integrity": "sha512-C/FsVVhht4iPQYXOInoxUM/1ELSf9EsgKH34FofQOp6hwCPrW4vG4w5++TED3xRUo8gD7l0P1J1dLlDYzODsTQ==" + } + } +} diff --git a/tools/package.json b/tools/package.json new file mode 100644 index 00000000..bf352200 --- /dev/null +++ b/tools/package.json @@ -0,0 +1,18 @@ +{ + "name": "catalog-creator", + "version": "1.0.0", + "description": "", + "main": "index.js", + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + }, + "author": "", + "license": "ISC", + "dependencies": { + "debug": "^4.1.0", + "kefir": "^3.8.5", + "lodash": "^4.17.11", + "mustache": "^3.0.1", + "yamljs": "^0.3.0" + } +} diff --git a/tools/t.yaml b/tools/t.yaml new file mode 100644 index 00000000..bd6bce3e --- /dev/null +++ b/tools/t.yaml @@ -0,0 +1,15 @@ +version: ‘1.0’ + +steps: + + BuildingDockerImage: + title: Building Docker Image + type: build + image_name: EXAMPLEUSER/EXAMPLEREPO + working_directory: ./ + dockerfile: Dockerfile + tag: ‘${{CF_BRANCH_TAG_NORMALIZED}}’ + metadata: + set: + - commit_url: ‘${{CF_COMMIT_URL}}’ + - note: ‘example note’ \ No newline at end of file diff --git a/tools/template.md b/tools/template.md new file mode 100644 index 00000000..10662ef4 --- /dev/null +++ b/tools/template.md @@ -0,0 +1,10 @@ +# **Catalog** +** {{date}}** + +# ?? +| Plugin Name | Description | source | Tags | +| ------------- |:-------------:| -----:|----:| + {{#plugins}} + | {{image}} | {{description}} | {{#sources}}{{.}}{{/sources}} | {{#keywords}} **`{{.}}`** {{/keywords}}| +{{/plugins}} + diff --git a/version b/version new file mode 100644 index 00000000..3eefcb9d --- /dev/null +++ b/version @@ -0,0 +1 @@ +1.0.0