Skip to content

Commit 133c89f

Browse files
ahmedxgoudaarkid15r
ahmedxgouda
and
arkid15r
authored
Establish an e2e backend instance locally and in CI/CD (#2429)
* Add dockerfile and compose and update makefile * Dump db-data * Reorder makefile * Dump db-data * Remove e2e dockerfile * Add CI/CD * Apply rabbit's suggestions * Add postgres instead of pgvector * Remove needs * Update envs * Fix migrations step * Add envs to docker * Remove migrations step * Remove --without test from dockerfile * Copy tests in dockerfile and add needs to gh workflow * Update dockerfile * Apply suggestion * Use the e2e instance in the frontend e2e tests * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD and dockerfile * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Update CI/CD * Restore needs for the job * Update Makefiles * Update docs * Apply suggestions * Update load-data * Skip sonar error * update port numbers * Update docs * Update code * Add required shell property back * Bump config.webServer timeout * Update code --------- Co-authored-by: Arkadii Yakovets <[email protected]>
1 parent fffe02d commit 133c89f

File tree

16 files changed

+256
-8
lines changed

16 files changed

+256
-8
lines changed

‎.github/workflows/run-ci-cd.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,13 +213,30 @@ jobs:
213213
- scan-code
214214
- scan-ci-dependencies
215215
runs-on: ubuntu-latest
216+
services:
217+
db:
218+
image: pgvector/pgvector:pg16
219+
env:
220+
POSTGRES_DB: nest_db_e2e
221+
POSTGRES_PASSWORD: nest_user_e2e_password
222+
POSTGRES_USER: nest_user_e2e
223+
options: >-
224+
--health-cmd="pg_isready -U nest_user_e2e -d nest_db_e2e -h localhost -p 5432"
225+
--health-interval=5s
226+
--health-timeout=5s
227+
--health-retries=5
228+
ports:
229+
- 5432:5432
216230
steps:
217231
- name: Check out repository
218232
uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3
219233

220234
- name: Set up Docker buildx
221235
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435
222236

237+
- name: Setup E2E environment
238+
uses: ./.github/workflows/setup-e2e-environment
239+
223240
- name: Build frontend end-to-end testing image
224241
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83
225242
with:

‎.github/workflows/setup-e2e-environment/action.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Set up E2E environment
2+
3+
description: Sets up the environment for end-to-end testing.
4+
5+
runs:
6+
using: composite
7+
steps:
8+
- name: Wait for database to be ready
9+
run: |
10+
until docker exec ${{ job.services.db.id }} pg_isready -U nest_user_e2e -d nest_db_e2e; do
11+
echo "Waiting for database..."
12+
sleep 5
13+
done
14+
shell: bash
15+
16+
- name: Install PostgreSQL client
17+
run: sudo apt-get install -y postgresql-client
18+
shell: bash
19+
20+
- name: Load Postgres data
21+
env:
22+
PGPASSWORD: nest_user_e2e_password
23+
run: |
24+
gunzip -c backend/data/nest-e2e.sql.gz | psql -h localhost -U nest_user_e2e -d nest_db_e2e
25+
shell: bash
26+
27+
- name: Build backend e2e image
28+
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83
29+
with:
30+
cache-from: |
31+
type=gha
32+
cache-to: |
33+
type=gha,compression=zstd
34+
context: backend
35+
file: backend/docker/Dockerfile
36+
load: true
37+
platforms: linux/amd64
38+
tags: owasp/nest:test-backend-e2e-latest
39+
40+
- name: Start Backend in the background
41+
run: |
42+
docker run -d --rm --name e2e-nest-backend \
43+
--env-file backend/.env.e2e.example \
44+
--network host \
45+
-p 9000:9000 \
46+
owasp/nest:test-backend-e2e-latest \
47+
sh -c '
48+
gunicorn wsgi:application --bind 0.0.0.0:9000
49+
'
50+
shell: bash
51+
52+
- name: Waiting for the backend to be ready
53+
run: |
54+
until wget --spider http://localhost:9000/a; do
55+
echo "Waiting for backend..."
56+
sleep 5
57+
done
58+
echo "Backend is up!"
59+
shell: bash

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ __pycache__
66
.DS_Store
77
.env*
88
!.env.example
9+
!.env.e2e.example
910
.github/instructions/snyk_rules.instructions.md
1011
.idea
1112
.lighthouseci/

‎CONTRIBUTING.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,32 @@ make test
415415
This command runs tests and checks that coverage threshold requirements are satisfied for both backend and frontend.
416416
**Please note your PR won't be merged if it fails the code tests checks.**
417417

418+
### Setting Up e2e Testing Environment
419+
420+
Follow these steps to setup your e2e testing environment:
421+
422+
1. Make sure you have `gzip` installed on your machine.
423+
424+
2. Run the e2e backend instance with the following command:
425+
426+
```bash
427+
make run-backend-e2e
428+
```
429+
430+
3. Load the data into the e2e db with the following command (in another terminal session):
431+
432+
```bash
433+
make load-data-e2e
434+
```
435+
436+
4. Now, you can stop the backend instance, and run the frontend e2e tests with the following command:
437+
438+
```bash
439+
make test-frontend-e2e
440+
```
441+
442+
**Please note that you only need to do these steps once.**
443+
418444
### Test Coverage
419445

420446
- There is a **minimum test coverage requirement** for the **backend** code -- see [pyproject.toml](https://github.com/OWASP/Nest/blob/main/backend/pyproject.toml).

‎backend/.env.e2e.example‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
DJANGO_ALGOLIA_APPLICATION_ID=None
2+
DJANGO_ALGOLIA_EXCLUDED_LOCAL_INDEX_NAMES=None
3+
DJANGO_ALGOLIA_WRITE_API_KEY=None
4+
DJANGO_ALLOWED_HOSTS=*
5+
DJANGO_AWS_ACCESS_KEY_ID=None
6+
DJANGO_AWS_SECRET_ACCESS_KEY=None
7+
DJANGO_SETTINGS_MODULE=settings.e2e
8+
DJANGO_CONFIGURATION=E2E
9+
DJANGO_DB_HOST=None
10+
DJANGO_DB_NAME=nest_db_e2e
11+
DJANGO_DB_USER=nest_user_e2e
12+
DJANGO_DB_PASSWORD=nest_user_e2e_password
13+
DJANGO_DB_PORT=5432
14+
DJANGO_OPEN_AI_SECRET_KEY=None
15+
DJANGO_PUBLIC_IP_ADDRESS="127.0.0.1"
16+
DJANGO_REDIS_HOST=None
17+
DJANGO_REDIS_PASSWORD=None
18+
DJANGO_RELEASE_VERSION=None
19+
DJANGO_SECRET_KEY=None
20+
DJANGO_SENTRY_DSN=None
21+
DJANGO_SLACK_BOT_TOKEN=None
22+
DJANGO_SLACK_SIGNING_SECRET=None
23+
GITHUB_TOKEN=None

‎backend/Makefile‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,15 @@ exec-backend-command:
2929
exec-backend-command-it:
3030
@docker exec -it nest-backend $(CMD) 2>/dev/null
3131

32+
exec-backend-command-e2e:
33+
@docker exec -it e2e-nest-backend $(CMD)
34+
3235
exec-db-command-it:
3336
@docker exec -it nest-db $(CMD)
3437

38+
exec-db-command-e2e:
39+
@docker exec -it e2e-nest-db $(CMD)
40+
3541
clear-cache:
3642
@CMD="python manage.py clear_cache" $(MAKE) exec-backend-command
3743

@@ -56,6 +62,10 @@ dump-data:
5662
@CMD="sed -E -i 's/(\"[^\"]*email\"): *\"([^\"]|\\\")*\"/\1: \"\"/g' data/nest.json" $(MAKE) exec-backend-command
5763
@CMD="gzip -f data/nest.json" $(MAKE) exec-backend-command
5864

65+
dump-data-e2e:
66+
@echo "Dumping Nest e2e data"
67+
@CMD="pg_dumpall -U nest_user_e2e --clean | gzip -9 > backend/data/nest-e2e.sql.gz" $(MAKE) exec-db-command-e2e
68+
5969
enrich-data: \
6070
github-enrich-issues \
6171
owasp-enrich-chapters \
@@ -76,6 +86,10 @@ load-data:
7686
@echo "Loading Nest data"
7787
@CMD="python manage.py load_data" $(MAKE) exec-backend-command
7888

89+
load-data-e2e:
90+
@echo "Loading Nest e2e data"
91+
@gunzip -c backend/data/nest-e2e.sql.gz | docker exec -i e2e-nest-db psql -U nest_user_e2e -d nest_db_e2e
92+
7993
merge-migrations:
8094
@CMD="python manage.py makemigrations --merge" $(MAKE) exec-backend-command
8195

@@ -99,6 +113,10 @@ restore-backup:
99113
@echo "Restoring Nest backup"
100114
@CMD="python manage.py restore_backup" $(MAKE) exec-backend-command
101115

116+
run-backend-e2e:
117+
@DOCKER_BUILDKIT=1 \
118+
docker compose --project-name nest-e2e -f docker-compose/e2e.yaml up --build --remove-orphans --abort-on-container-exit backend db
119+
102120
save-backup:
103121
@echo "Saving Nest backup"
104122
@CMD="python manage.py dumpdata --natural-primary --natural-foreign --indent=2" $(MAKE) exec-backend-command > backend/data/backup.json

‎backend/apps/common/utils.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ def get_user_ip_address(request: HttpRequest) -> str:
100100
str: The user's IP address.
101101
102102
"""
103-
if settings.IS_LOCAL_ENVIRONMENT:
103+
if settings.IS_LOCAL_ENVIRONMENT or settings.IS_E2E_ENVIRONMENT:
104104
return settings.PUBLIC_IP_ADDRESS
105105

106106
x_forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR")

‎backend/data/nest-e2e.sql.gz‎

25.4 MB
Binary file not shown.

‎backend/docker/Dockerfile‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,5 +63,3 @@ RUN rm -rf /home/owasp/.cache && \
6363
chmod +x /home/owasp/entrypoint.sh
6464

6565
USER owasp
66-
67-
CMD ["/home/owasp/entrypoint.sh"]

‎backend/settings/base.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ class Base(Configuration):
1919
DEBUG = False
2020
GITHUB_APP_ID = None
2121
GITHUB_APP_INSTALLATION_ID = None
22+
IS_E2E_ENVIRONMENT = False
2223
IS_LOCAL_ENVIRONMENT = False
2324
IS_PRODUCTION_ENVIRONMENT = False
2425
IS_STAGING_ENVIRONMENT = False

0 commit comments

Comments
 (0)