Skip to content

Commit e4030d5

Browse files
heynemannheynemann
authored
[FEAT] Improvement of our build and CircleCI (thumbor#1341)
**Is backwards compatible**: yes This commit adds CircleCI and creates a new testing docker image for Thumbor allowing our contributors to run tests easily without the requirements leaking to their own environment.
1 parent 520c9f8 commit e4030d5

11 files changed

+238
-3
lines changed

‎.circleci/config.yml

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
version: 2.1
2+
3+
orbs:
4+
# The python orb contains a set of prepackaged circleci configuration you can use repeatedly in your configurations files
5+
# Orb commands and jobs help you with common scripting around a language/tool
6+
# so you dont have to copy and paste it everywhere.
7+
# See the orb documentation here: https://circleci.com/developer/orbs/orb/circleci/python
8+
python: circleci/[email protected]
9+
10+
workflows:
11+
ci:
12+
jobs:
13+
- build-and-test-python36
14+
- build-and-test-python37
15+
- build-and-test-python38
16+
- build-and-test-python39
17+
18+
jobs:
19+
build-and-test-python36:
20+
working_directory: /app
21+
docker:
22+
- image: thumbororg/thumbor-test:36
23+
steps:
24+
- checkout
25+
- run:
26+
name: Compile Extensions
27+
command: make compile_ext
28+
- run:
29+
name: Redis
30+
command: make redis
31+
- run:
32+
name: Run Unit Tests
33+
command: make sequential-unit
34+
- run:
35+
name: Run Integration Tests
36+
command: env ASYNC_TEST_TIMEOUT=30 make integration_run
37+
- run:
38+
name: Lint
39+
command: make flake
40+
41+
build-and-test-python37:
42+
working_directory: /app
43+
docker:
44+
- image: thumbororg/thumbor-test:37
45+
steps:
46+
- checkout
47+
- run:
48+
name: Compile Extensions
49+
command: make compile_ext
50+
- run:
51+
name: Redis
52+
command: make redis
53+
- run:
54+
name: Run Unit Tests
55+
command: make sequential-unit
56+
- run:
57+
name: Run Integration Tests
58+
command: env ASYNC_TEST_TIMEOUT=30 make integration_run
59+
- run:
60+
name: Lint
61+
command: make flake
62+
63+
build-and-test-python38:
64+
working_directory: /app
65+
docker:
66+
- image: thumbororg/thumbor-test:38
67+
steps:
68+
- checkout
69+
- run:
70+
name: Compile Extensions
71+
command: make compile_ext
72+
- run:
73+
name: Redis
74+
command: make redis
75+
- run:
76+
name: Run Unit Tests
77+
command: make sequential-unit
78+
- run:
79+
name: Run Integration Tests
80+
command: env ASYNC_TEST_TIMEOUT=30 make integration_run
81+
- run:
82+
name: Lint
83+
command: make flake
84+
85+
build-and-test-python39:
86+
working_directory: /app
87+
docker:
88+
- image: thumbororg/thumbor-test:39
89+
steps:
90+
- checkout
91+
- run:
92+
name: Compile Extensions
93+
command: make compile_ext
94+
- run:
95+
name: Redis
96+
command: make redis
97+
- run:
98+
name: Run Unit Tests
99+
command: make sequential-unit
100+
- run:
101+
name: Run Integration Tests
102+
command: env ASYNC_TEST_TIMEOUT=30 make integration_run
103+
- run:
104+
name: Lint
105+
command: make flake

‎.python-version

-1
This file was deleted.

‎Makefile

+49
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,9 @@ coverage:
3939
unit:
4040
@pytest -n `nproc` --cov=thumbor tests/
4141

42+
sequential-unit:
43+
@pytest -sv --cov=thumbor tests/
44+
4245
kill_redis:
4346
@-redis-cli -p 6668 -a hey_you shutdown
4447

@@ -148,3 +151,49 @@ sample_images:
148151
cp tests/fixtures/filters/watermark.png tests/fixtures/images/watermark.png
149152
# the watermark filter's logic is too complicated to reproduce with IM, the watermark test images can't be generated here
150153
# similarly, the noise, colorize, redeye and fill filters generate output too unique to be reproduce with IM and can't be generated here
154+
155+
test-docker-build: test-docker-36-build test-docker-37-build test-docker-38-build test-docker-39-build
156+
157+
test-docker-run: test-docker-36-run test-docker-37-run test-docker-38-run test-docker-39-run
158+
159+
test-docker-publish: test-docker-36-publish test-docker-37-publish test-docker-38-publish test-docker-39-publish
160+
161+
test-docker-36-build:
162+
@docker build -f TestDockerfile36 -t thumbor-test-36 .
163+
164+
test-docker-36-run:
165+
@docker run -v "$$(pwd):/app" thumbororg/thumbor-test:36 make compile_ext redis sequential-unit integration flake
166+
167+
test-docker-36-publish:
168+
@docker image tag thumbor-test-36:latest thumbororg/thumbor-test:36
169+
@docker push thumbororg/thumbor-test:36
170+
171+
test-docker-37-build:
172+
@docker build -f TestDockerfile37 -t thumbor-test-37 .
173+
174+
test-docker-37-run:
175+
@docker run -v "$$(pwd):/app" thumbororg/thumbor-test:37 make compile_ext redis sequential-unit integration flake
176+
177+
test-docker-37-publish:
178+
@docker image tag thumbor-test-37:latest thumbororg/thumbor-test:37
179+
@docker push thumbororg/thumbor-test:37
180+
181+
test-docker-38-build:
182+
@docker build -f TestDockerfile38 -t thumbor-test-38 .
183+
184+
test-docker-38-run:
185+
@docker run -v "$$(pwd):/app" thumbororg/thumbor-test:38 make compile_ext redis sequential-unit integration flake
186+
187+
test-docker-38-publish:
188+
@docker image tag thumbor-test-38:latest thumbororg/thumbor-test:38
189+
@docker push thumbororg/thumbor-test:38
190+
191+
test-docker-39-build:
192+
@docker build -f TestDockerfile39 -t thumbor-test-39 .
193+
194+
test-docker-39-run:
195+
@docker run -v "$$(pwd):/app" thumbororg/thumbor-test:39 make compile_ext redis sequential-unit integration flake
196+
197+
test-docker-39-publish:
198+
@docker image tag thumbor-test-39:latest thumbororg/thumbor-test:39
199+
@docker push thumbororg/thumbor-test:39

‎README.mkd

+1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ If you use thumbor, please take 1 minute and answer [this survey](http://t.co/qP
55
[<img src="https://raw.github.com/thumbor/thumbor/master/logo-thumbor.png">](https://github.com/thumbor/thumbor)
66

77
[<img src="https://secure.travis-ci.org/thumbor/thumbor.png?branch=master">](http://travis-ci.org/thumbor/thumbor)
8+
[![CircleCI Status](https://circleci.com/gh/thumbor/thumbor.svg?style=shield&circle-token=:circle-token)](https://circleci.com/gh/thumbor/thumbor)
89
[![Coverage Status](https://coveralls.io/repos/thumbor/thumbor/badge.svg?branch=master&service=github)](https://coveralls.io/github/thumbor/thumbor?branch=master)
910
[![Code Climate](https://codeclimate.com/github/thumbor/thumbor/badges/gpa.svg)](https://codeclimate.com/github/thumbor/thumbor)
1011
[![Codacy Badge](https://api.codacy.com/project/badge/373e13c719c0417f84f0d7d363c9d539)](https://www.codacy.com/app/heynemann/thumbor)

‎TestDockerfile36

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM python:3.6-slim
2+
WORKDIR /app
3+
RUN apt-get update -y && apt-get install -y python3-dev libcurl4-openssl-dev libgnutls28-dev libjpeg-progs libimage-exiftool-perl gifsicle scons python3-all-dev libboost-python-dev libexiv2-dev ffmpeg make zlib1g-dev gcc libssl-dev libjpeg-dev libwebp-dev redis && apt-get clean
4+
RUN apt-get update -y && apt-get install --reinstall -y build-essential && apt-get clean
5+
RUN pip install --upgrade pip
6+
COPY setup.py /app/setup.py
7+
RUN mkdir -p /app/thumbor
8+
RUN touch /app/thumbor/__init__.py
9+
RUN ln -s /usr/lib/x86_64-linux-gnu/libboost_python3-py37.so /usr/lib/libboost_python36.so
10+
RUN python -m pip install -e .[tests] && rm -rf ~/.cache/pip
11+
RUN pip install coveralls
12+
RUN pip install --upgrade pytest
13+
RUN rm -rf /app

‎TestDockerfile37

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM python:3.7-slim
2+
WORKDIR /app
3+
RUN apt-get update -y && apt-get install -y python3-dev libcurl4-openssl-dev libgnutls28-dev libjpeg-progs libimage-exiftool-perl gifsicle scons python3-all-dev libboost-python-dev libexiv2-dev ffmpeg make zlib1g-dev gcc libssl-dev libjpeg-dev libwebp-dev redis && apt-get clean
4+
RUN apt-get update -y && apt-get install --reinstall -y build-essential && apt-get clean
5+
RUN pip install --upgrade pip
6+
COPY setup.py /app/setup.py
7+
RUN mkdir -p /app/thumbor
8+
RUN touch /app/thumbor/__init__.py
9+
RUN ln -s /usr/lib/x86_64-linux-gnu/libboost_python3-py37.so /usr/lib/libboost_python37.so
10+
RUN python -m pip install -e .[tests] && rm -rf ~/.cache/pip
11+
RUN pip install coveralls
12+
RUN pip install --upgrade pytest
13+
RUN rm -rf /app

‎TestDockerfile38

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM python:3.8-slim
2+
WORKDIR /app
3+
RUN apt-get update -y && apt-get install -y python3-dev libcurl4-openssl-dev libgnutls28-dev libjpeg-progs libimage-exiftool-perl gifsicle scons python3-all-dev libboost-python-dev libexiv2-dev ffmpeg make zlib1g-dev gcc libssl-dev libjpeg-dev libwebp-dev redis && apt-get clean
4+
RUN apt-get update -y && apt-get install --reinstall -y build-essential && apt-get clean
5+
RUN pip install --upgrade pip
6+
COPY setup.py /app/setup.py
7+
RUN mkdir -p /app/thumbor
8+
RUN touch /app/thumbor/__init__.py
9+
RUN ln -s /usr/lib/x86_64-linux-gnu/libboost_python3-py37.so /usr/lib/libboost_python38.so
10+
RUN python -m pip install -e .[tests] && rm -rf ~/.cache/pip
11+
RUN pip install coveralls
12+
RUN pip install --upgrade pytest
13+
RUN rm -rf /app

‎TestDockerfile39

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM python:3.9-slim
2+
WORKDIR /app
3+
RUN apt-get update -y && apt-get install -y python3-dev libcurl4-openssl-dev libgnutls28-dev libjpeg-progs libimage-exiftool-perl gifsicle scons python3-all-dev libboost-python-dev libexiv2-dev ffmpeg make zlib1g-dev gcc libssl-dev libjpeg-dev libwebp-dev redis && apt-get clean
4+
RUN apt-get update -y && apt-get install --reinstall -y build-essential && apt-get clean
5+
RUN pip install --upgrade pip
6+
COPY setup.py /app/setup.py
7+
RUN mkdir -p /app/thumbor
8+
RUN touch /app/thumbor/__init__.py
9+
RUN ln -s /usr/lib/x86_64-linux-gnu/libboost_python3-py37.so /usr/lib/libboost_python39.so
10+
RUN python -m pip install -e .[tests] && rm -rf ~/.cache/pip
11+
RUN pip install coveralls
12+
RUN pip install --upgrade pytest
13+
RUN rm -rf /app

‎docs/hacking_on_thumbor.rst

+26
Original file line numberDiff line numberDiff line change
@@ -78,3 +78,29 @@ To merge thumbor's master with your fork::
7878

7979
If there was anything to merge, just run your tests again. If they pass,
8080
`send a pull request <http://help.github.com/send-pull-requests/>`__.
81+
82+
Introducing a new Dependency
83+
----------------------------
84+
85+
If we introduce a new dependency, the testing docker images need to be updated.
86+
87+
If the new dependency requires changes to the docker image, make sure to update the TestDockerfile36, TestDockerfile37, TestDockerfile38 and TestDockerfile39 files.
88+
89+
Then build and publish with::
90+
91+
make test-docker-build test-docker-publish
92+
93+
Remember that you must be logged in with your docker hub account and you must be part of the `thumbororg <https://hub.docker.com/repository/docker/thumbororg/thumbor-test>` team of administrators.
94+
95+
Running tests in docker
96+
-----------------------
97+
98+
If you do not wish to configure your environment with thumbor's dependencies, you can use our docker image to run tests with::
99+
100+
make test-docker-run
101+
102+
Or if you want to run a specific python version with your tests::
103+
104+
make test-docker-39-run
105+
106+
Just replace '39' with the python version you want: 36, 37, 38 or 39.

‎integration_tests/urls_helpers.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ async def try_url(self, url):
124124
failed = False
125125

126126
try:
127-
result = await self.http_client.fetch(url, request_timeout=30)
127+
result = await self.http_client.fetch(url, request_timeout=60)
128128
except Exception as err: # pylint: disable=broad-except
129129
logging.exception("Error in %s: %s", url, err)
130130
error = err

‎setup.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,10 @@
1414

1515
from setuptools import Extension, setup
1616

17-
from thumbor import __version__
17+
try:
18+
from thumbor import __version__
19+
except ImportError:
20+
__version__ = "0.0.0"
1821

1922
TESTS_REQUIREMENTS = [
2023
"cairosvg!=1.0.21,<2.0.0,>=1.0.0",

0 commit comments

Comments
 (0)