Skip to content

Commit

Permalink
Publish warning about installing packages dynamically (apache#450)
Browse files Browse the repository at this point in the history
  • Loading branch information
potiuk authored Jul 12, 2021
1 parent 8230c7f commit 2c887ee
Show file tree
Hide file tree
Showing 4 changed files with 43 additions and 7 deletions.
21 changes: 19 additions & 2 deletions docs-archive/docker-stack/_sources/entrypoint.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -262,11 +262,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and
Installing additional requirements
..................................

.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for
testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
production environment. We have deliberately chose to make it a development/test dependency and we print
a warning, whenever it is used. There is an inherent security-related issue with using this method in
production. Installing the requirements this way can happen at literally any time - when your containers
get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
literally any time. This opens you up to a serious vulnerability where your production environment
might be brought down by a single dependency being removed from PyPI - or even dependency of your
dependency. This means that you put your production service availability in hands of 3rd-party developers.
At any time, any moment including weekends and holidays those 3rd party developers might bring your
production Airflow instance down, without you even knowing it. This is a serious vulnerability that
is similar to the infamous
`leftpad <https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/>`_
problem. You can fully protect against this case by building your own, immutable custom image, where the
dependencies are baked in. You have been warned.

Installing additional requirements can be done by specifying ``_PIP_ADDITIONAL_REQUIREMENTS`` variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
it must install new packages. Therefore this option should only be used for testing. When testing is
finished, you should create your custom image with dependencies baked in.
it must install new packages and it opens up huge potential security vulnerability when used in production
(see below). Therefore this option should only be used for testing. When testing is finished,
you should create your custom image with dependencies baked in.

Example:

Expand Down
4 changes: 2 additions & 2 deletions docs-archive/docker-stack/build.html
Original file line number Diff line number Diff line change
Expand Up @@ -1283,7 +1283,7 @@ <h2>Examples of image customizing<a class="headerlink" href="#examples-of-image-
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt <span class="o">||</span> <span class="nb">true</span>

curl -Lo <span class="s2">&quot;docker-context-files/constraints-3.7.txt&quot;</span> <span class="se">\</span>
https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt
https://raw.githubusercontent.com/apache/airflow/constraints-2.1.0/constraints-3.7.txt

<span class="c1"># For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.</span>
pip install <span class="nv">pip</span><span class="o">==</span><span class="m">20</span>.2.4
Expand Down Expand Up @@ -1323,7 +1323,7 @@ <h2>Examples of image customizing<a class="headerlink" href="#examples-of-image-
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker build . <span class="se">\</span>
--build-arg <span class="nv">PYTHON_BASE_IMAGE</span><span class="o">=</span><span class="s2">&quot;python:3.7-slim-buster&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_INSTALLATION_METHOD</span><span class="o">=</span><span class="s2">&quot;apache-airflow&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">&quot;2.0.2&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">&quot;2.1.0&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_MYSQL_CLIENT</span><span class="o">=</span><span class="s2">&quot;false&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_PRE_CACHED_PIP_PACKAGES</span><span class="o">=</span><span class="s2">&quot;false&quot;</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_FROM_DOCKER_CONTEXT_FILES</span><span class="o">=</span><span class="s2">&quot;true&quot;</span> <span class="se">\</span>
Expand Down
23 changes: 21 additions & 2 deletions docs-archive/docker-stack/entrypoint.html
Original file line number Diff line number Diff line change
Expand Up @@ -805,11 +805,30 @@ <h3>Creating admin user<a class="headerlink" href="#creating-admin-user" title="
</div>
<div class="section" id="installing-additional-requirements">
<h3>Installing additional requirements<a class="headerlink" href="#installing-additional-requirements" title="Permalink to this headline"></a></h3>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Installing requirements this way is a very convenient method of running Airflow, very useful for
testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
production environment. We have deliberately chose to make it a development/test dependency and we print
a warning, whenever it is used. There is an inherent security-related issue with using this method in
production. Installing the requirements this way can happen at literally any time - when your containers
get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
literally any time. This opens you up to a serious vulnerability where your production environment
might be brought down by a single dependency being removed from PyPI - or even dependency of your
dependency. This means that you put your production service availability in hands of 3rd-party developers.
At any time, any moment including weekends and holidays those 3rd party developers might bring your
production Airflow instance down, without you even knowing it. This is a serious vulnerability that
is similar to the infamous
<a class="reference external" href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/">leftpad</a>
problem. You can fully protect against this case by building your own, immutable custom image, where the
dependencies are baked in. You have been warned.</p>
</div>
<p>Installing additional requirements can be done by specifying <code class="docutils literal notranslate"><span class="pre">_PIP_ADDITIONAL_REQUIREMENTS</span></code> variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
it must install new packages. Therefore this option should only be used for testing. When testing is
finished, you should create your custom image with dependencies baked in.</p>
it must install new packages and it opens up huge potential security vulnerability when used in production
(see below). Therefore this option should only be used for testing. When testing is finished,
you should create your custom image with dependencies baked in.</p>
<p>Example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it -p <span class="m">8080</span>:8080 <span class="se">\</span>
--env <span class="s2">&quot;_PIP_ADDITIONAL_REQUIREMENTS=lxml==4.6.3 charset-normalizer==1.4.1&quot;</span> <span class="se">\</span>
Expand Down
Loading

0 comments on commit 2c887ee

Please sign in to comment.