Debugging Common Development Problems

Jenkins Deployments

Because we use zero-downtime deployment, it can sometimes be difficult to see the logs of a failed build. To gain visibility into the logs:

1. Restart the deployment job in Jenkins.
2. Log into the environment in the cloud foundry command line. cf login --sso
3. Wait for the deploy script in Jenkins to switch the APP_NAME to APP_NAME-venerable.
4. Begin trailing the logs in cloud foundry cf APP_NAME.
5. Debug accordingly.

Nightly recycle-production task failure

Occasionally, the nightly recycle-production task will fail due to a known bug in the the Cloud Foundry recycle plugin not receiving the ping that the deployment was successful, (See https://github.com/Comcast/cf-recycle-plugin/issues/6). The logs will reflect this by repeatedly displaying Instance 0 Status: running until a timeout is reached. The deployment can be confirmed by interrogating the Cloud.gov logs or looking at the application details with cf app <app-name>.

Snyk updates

When a nightly deploy, or a PR triggers a snyk failure there are a number of remediations. The first is to run snyk wizard to see if there is an available patch. If there is no available patch, look at the Snyk vulnerability report an ensure that it is not a false positive. If not, consider removing the dependency.

Viewing log errors

1. Log into the Kibana instance hosted on cloud.gov.
2. If you know that the application error comes from a particular service (app) within Open Forest, you can filter by that application, by clicking on Add a filter and then in the Fields dropdown selecting @cf.app.
3. For the Operators dropdown select is.
4. For the Values type in the app name most likely fs-intake-api (the production server). This will either be:
   • fs-middlelayer-api-staging
   • fs-middlelayer-api
   • open-forest-platform-api
   • open-forest-platform-api-staging
   • open-forest-platform-frontend
   • open-forest-platform-frontend-staging
5. Filter @source.type is APP
6. Enter a query of what to search. If you want to search the message text enter @message : YOURSEARCHTERM. NOTE: If you like to search one of the commonly audit-able logging events you may want to try the following queries:
   • @message : CONTROLLER: to view what user has updated, created or accessed a resource
   • @message : AUTHENTICATION: to view login and logout events
   • @message : ERROR: to view application errors.

Christmas Tree on-call monitoring response steps

• Check cloud.gov status for issues Is it operational? Are there issues? (and subscribe for updates)
• Check usda-forest-service org in cloud.gov
  • Go to the dasboard
  • Select “usda-forest-service”
  • Select “public-production”
  • Check that the applications are green
  • Confirm that the url is associated: Bound route: openforest.fs.usda.gov is still there
  • If something is gray or red, consider restarting the app
    • Restart typically takes about four minutes, won’t prevent another problem from popping up
• New Relic (if something goes down, we should get emails)
  • When you get a New Relic alert, go into cloud.gov and see if the app is green
  • Go into FS-Open-Forest-Platform-API in New Relic
  • See if there is an error type that is particularly high. FYI, Known issues include IE, authorization errors
• Check that the deployment on master worked by looking at the CircleCi Badge README (scroll down below the folders)

• Check the last Pull requests into master to see if issues started at the same time as the issue

  • Check the last few deployments in circleCI

    • If red, click onto the “master / build-deploy”. Do not do anything if it says “master / nightly”

    • If “prod-deploy” is red, click on the “prod-deploy” box.

    • Consider restarting the workflow

• View the log errors:

  • Log into Kibana
  • Go to this link only once you are logged in

• If nothing is resolved open a new issue in the Platform repo