Merge branch 'release/4.5.4'

Author: GrahamDumpleton

docs/installation.rst

@@ -18,6 +18,7 @@ For instructions on how to compile mod_wsgi from source code for UNIX like
 operating systems such as Linux and MacOS X see:
 
   * :doc:`user-guides/quick-installation-guide`
+  * :doc:`user-guides/installation-on-macosx`
 
 If you are on Windows, you should instead use:
 

docs/release-notes.rst

@@ -5,6 +5,7 @@ Release Notes
 .. toctree::
    :maxdepth: 2
 
+   release-notes/version-4.5.4
    release-notes/version-4.5.3
    release-notes/version-4.5.2
    release-notes/version-4.5.1

docs/release-notes/version-4.5.4.rst

@@ -0,0 +1,44 @@
+=============
+Version 4.5.4
+=============
+
+Version 4.5.4 of mod_wsgi can be obtained from:
+
+  https://codeload.github.com/GrahamDumpleton/mod_wsgi/tar.gz/4.5.4
+
+Bugs Fixed
+----------
+
+1. When using Apache 2.4 and daemon mode, the connection and request log
+   IDs from the Apache child worker processes were not being copied across
+   to the daemon process so that log messages generated against the request
+   would use the same ID in logs when using the ``%L`` format modifier.
+
+2. When using Apache 2.4 and daemon mode, the remote client port
+   information was not being cached such that log messages generated
+   against the request would use the port in logs when using the ``%a``
+   format modifier.
+
+Features Changed
+----------------
+
+1. If ``sys.stdout`` and ``sys.stderr`` are used in the context of the
+   thread handling a request, calls against them to log messages will be
+   routed back via ``wsgi.errors`` from the per request WSGI ``environ``
+   dictionary. This avoids the danger of logged messages from different
+   request handlers being intermixed as buffering will now be done on a per
+   request basis. Such messages will also be logged with the correct
+   connection and request log ID if the ``%L`` formatter is used in the
+   error log format.
+
+New Features
+------------
+
+1. Added new option ``--error-log-format`` to ``mod_wsgi-express`` to allow
+   the error log message format to be specified.
+
+2. Pass through to the WSGI per request ``environ`` dictionary new values
+   for ``mod_wsgi.connection_id`` and ``mod_wsgi.request_id``. These are
+   the Apache log IDs for the connection and request that it uses in log
+   messages when using the ``%L`` format modifier. This only applies to
+   Apache 2.4 and later.

docs/user-guides.rst

@@ -6,6 +6,7 @@ User Guides
    :maxdepth: 1
 
    user-guides/quick-installation-guide
+   user-guides/installation-on-macosx
    user-guides/quick-configuration-guide
    user-guides/configuration-guidelines
    user-guides/installation-issues

docs/user-guides/installation-on-macosx.rst

@@ -0,0 +1,163 @@
+=======================
+Installation On MacOS X
+=======================
+
+If you are using MacOS X, mod_wsgi can be compiled from source code
+against the standard versions of Python and Apache httpd server supplied
+with the operating system. To do this though you will first need to have
+installed the Xcode command line tools.
+
+The Xcode command line tools package provides a C compiler, along with
+header files and support tools for the Apache httpd server. If you have
+already set up your system so as to be able to install additional Python
+packages which include C extensions, you likely will already have the
+Xcode command line tools.
+
+Install Xcode command line tools
+--------------------------------
+
+To install the Xcode command line tools you should run the command::
+
+    xcode-select --install
+
+If this gives you back the error message::
+
+    xcode-select: error: command line tools are already installed, use "Software Update" to install updates
+
+then the tools have already been installed. As noted by the warning
+message, do make sure you have run a system software update to ensure
+that you have the latest versions of these tools.
+
+If you do not already have the Xcode command line tools installed, running
+that ``xcode-select`` command should result in you being prompted to
+install them. This may ask you to provide the details of an administrator
+account along with the password for that account.
+
+Note that it is not necessary to install the whole of the Xcode
+developer application from the MacOS X App Store, only the command line
+tools using ``xcode-select``. If you have installed the Xcode developer
+application, still ensure that the command line tools are installed and
+ensure you have run the system software update.
+
+Configuring and building mod_wsgi
+---------------------------------
+
+If you are using the Python and Apache httpd server packages provided with
+the operating system, all you need to do to configure the mod_wsgi source
+code before building it is to run in the mod_wsgi source code directory::
+
+    ./configure
+
+This should yield output similar to::
+
+    checking for apxs2... no
+    checking for apxs... /usr/sbin/apxs
+    checking for gcc... gcc
+    checking whether the C compiler works... yes
+    checking for C compiler default output file name... a.out
+    checking for suffix of executables...
+    checking whether we are cross compiling... no
+    checking for suffix of object files... o
+    checking whether we are using the GNU C compiler... yes
+    checking whether gcc accepts -g... yes
+    checking for gcc option to accept ISO C89... none needed
+    checking for prctl... no
+    checking Apache version... 2.4.18
+    checking for python... /usr/bin/python
+    configure: creating ./config.status
+    config.status: creating Makefile
+
+The ``configure`` script should show that it has detected ``apxs`` as being
+located at ``/usr/sbin/apxs`` and ``python`` as being at ``/usr/bin/python``.
+
+If you get different values for ``apxs`` and ``python`` then it means
+that you likely have a separate installation of Python or the Apache
+httpd server installed on your system. If this is the case, to ensure that
+you use the versions of Python and Apache httpd server provided with the
+operating system instead use the command::
+
+    ./configure --with-python=/usr/bin/python --with-apxs=/usr/sbin/apxs
+
+Once you have configured the source code by running ``configure``, you
+can build mod_wsgi using the command::
+
+    make
+
+This will compile the mod_wsgi source code and produce a single
+``mod_wsgi.so`` file which then needs to be installed into a common
+location so that the Apache httpd server can use it.
+
+Installing the mod_wsgi module
+------------------------------
+
+What you need to do to install the mod_wsgi module depends on which version
+of MacOS X you are using.
+
+For the Apache httpd server provided by the operating system, the directory
+``/usr/libexec/apache2`` is used to store the compiled modules. Prior to
+MacOS X El Capitan (10.11) this directory was writable and the mod_wsgi
+module could be installed here along with all the default modules. With the
+introduction of the System Integrity Protection (SIP_) feature in MacOS X
+El Capitan this directory is not writable, not even to the root user.
+
+Because of this, if you are using a version of MacOS X prior to MacOS X El
+Capitan (10.11) you can use the command::
+
+    sudo make install
+
+to install the mod_wsgi module. As ``sudo`` is being run, you will be
+prompted for your password. The module will be installed into the
+directory ``/usr/libexec/apache2``. Within the Apache httpd server
+configuration file you can then use the standard ``LoadModule`` line
+of::
+
+    LoadModule wsgi_module libexec/apache2/mod_wsgi.so
+
+If however you are using MacOS X El Capitan (10.11) or later, the mod_wsgi
+module will need to be installed into a different location. If you don't
+and try to run just ``sudo make install``, it will fail with the output::
+
+    ./apxs -i -S LIBEXECDIR=/usr/libexec/apache2 -n 'mod_wsgi' src/server/mod_wsgi.la
+    /usr/share/httpd/build/instdso.sh SH_LIBTOOL='./libtool' src/server/mod_wsgi.la /usr/libexec/apache2
+    ./libtool --mode=install install src/server/mod_wsgi.la /usr/libexec/apache2/
+    libtool: install: install src/server/.libs/mod_wsgi.so /usr/libexec/apache2/mod_wsgi.so
+    install: /usr/libexec/apache2/mod_wsgi.so: Operation not permitted
+    apxs:Error: Command failed with rc=4653056
+    .
+    make: *** [install] Error 1
+
+The directory you use to install the mod_wsgi module is up to you, but
+one suggested option is that you use the directory
+``/usr/local/httpd/modules``. Just ensure that this isn't already used
+by a separate installation of the Apache httpd server.
+
+To install the mod_wsgi module into this directory use the command::
+
+    sudo make install LIBEXECDIR=/usr/local/httpd/modules
+
+The output from the command will be similar to::
+
+    mkdir -p /usr/local/httpd/modules
+    ./apxs -i -S LIBEXECDIR=/usr/local/httpd/modules -n 'mod_wsgi' src/server/mod_wsgi.la
+    /usr/share/httpd/build/instdso.sh SH_LIBTOOL='./libtool' src/server/mod_wsgi.la /usr/local/httpd/modules
+    ./libtool --mode=install install src/server/mod_wsgi.la /usr/local/httpd/modules/
+    libtool: install: install src/server/.libs/mod_wsgi.so /usr/local/httpd/modules/mod_wsgi.so
+    libtool: install: install src/server/.libs/mod_wsgi.lai /usr/local/httpd/modules/mod_wsgi.la
+    libtool: install: install src/server/.libs/mod_wsgi.a /usr/local/httpd/modules/mod_wsgi.a
+    libtool: install: chmod 644 /usr/local/httpd/modules/mod_wsgi.a
+    libtool: install: ranlib /usr/local/httpd/modules/mod_wsgi.a
+    libtool: install: warning: remember to run `libtool --finish /usr/libexec/apache2'
+    chmod 755 /usr/local/httpd/modules/mod_wsgi.so
+
+The warning about needing to run ``libtool --finish`` can be ignored as it
+is not required for everything to work.
+
+With the mod_wsgi module installed in this location, the ``LoadModule`` line
+in the Apache httpd configuration file should be::
+
+    LoadModule wsgi_module /usr/local/httpd/modules/mod_wsgi.so
+
+Normal steps to then configure the Apache httpd server and mod_wsgi for
+your specific WSGI application would then be followed.
+
+.. _SIP: https://en.wikipedia.org/wiki/System_Integrity_Protection

src/server/__init__.py

@@ -394,6 +394,10 @@ def find_mimetypes():
 </IfDefine>
 LogLevel %(log_level)s
 
+<IfDefine MOD_WSGI_ERROR_LOG_FORMAT>
+ErrorLogFormat "%(error_log_format)s"
+</IfDefine>
+
 <IfDefine MOD_WSGI_ACCESS_LOG>
 <IfModule !log_config_module>
 LoadModule log_config_module ${MOD_WSGI_MODULES_DIRECTORY}/mod_log_config.so
@@ -2257,6 +2261,8 @@ def check_percentage(option, opt_str, value, parser):
 
     optparse.make_option('--access-log-format', metavar='FORMAT',
             help='Specify the format of the access log records.'),
+    optparse.make_option('--error-log-format', metavar='FORMAT',
+            help='Specify the format of the error log records.'),
 
     optparse.make_option('--error-log-name', metavar='FILE-NAME',
             default='error_log', help='Specify the name of the error '
@@ -2646,6 +2652,10 @@ def _cmd_setup_server(command, args, options):
     options['access_log_format'] = options['access_log_format'].replace(
             '\"', '\\"')
 
+    if options['error_log_format']:
+        options['error_log_format'] = options['error_log_format'].replace(
+                '\"', '\\"')
+
     options['pid_file'] = ((options['pid_file'] and os.path.abspath(
             options['pid_file'])) or os.path.join(options['server_root'],
             'httpd.pid'))
@@ -3006,6 +3016,8 @@ def _cmd_setup_server(command, args, options):
         options['httpd_arguments_list'].append('-DMOD_WSGI_DIRECTORY_INDEX')
     if options['directory_listing']:
         options['httpd_arguments_list'].append('-DMOD_WSGI_DIRECTORY_LISTING')
+    if options['error_log_format']:
+        options['httpd_arguments_list'].append('-DMOD_WSGI_ERROR_LOG_FORMAT')
     if options['access_log']:
         options['httpd_arguments_list'].append('-DMOD_WSGI_ACCESS_LOG')
     if options['rotate_logs']:

src/server/mod_wsgi.c

@@ -3749,6 +3749,8 @@ static int wsgi_execute_script(request_rec *r)
 
     int status;
 
+    WSGIThreadInfo *thread_info = NULL;
+
     /* Grab request configuration. */
 
     config = (WSGIRequestConfig *)ap_get_module_config(r->request_config,
@@ -3958,7 +3960,7 @@ static int wsgi_execute_script(request_rec *r)
 
     /* Setup metrics for start of request. */
 
-    wsgi_start_request();
+    thread_info = wsgi_start_request();
 
     /* Load module if not already loaded. */
 
@@ -3995,6 +3997,9 @@ static int wsgi_execute_script(request_rec *r)
                 PyObject *method = NULL;
                 PyObject *args = NULL;
 
+                Py_INCREF(adapter->log);
+                thread_info->log = adapter->log;
+
                 Py_INCREF(object);
                 status = Adapter_run(adapter, object);
                 Py_DECREF(object);
@@ -4029,6 +4034,8 @@ static int wsgi_execute_script(request_rec *r)
                 Py_XDECREF(object);
                 Py_XDECREF(method);
 
+                Py_CLEAR(thread_info->log);
+
                 adapter->bb = NULL;
             }
 
@@ -6053,6 +6060,27 @@ static void wsgi_build_environment(request_rec *r)
 
     apr_table_setn(r->subprocess_env, "mod_wsgi.request_start",
                    apr_psprintf(r->pool, "%" APR_TIME_T_FMT, r->request_time));
+
+#if AP_MODULE_MAGIC_AT_LEAST(20100923,2)
+    if (!r->log_id || !r->connection->log_id) {
+        const char **id;
+
+        /* Need to cast const away. */
+
+        if (r)
+            id = &((request_rec *)r)->log_id;
+        else
+            id = &((conn_rec *)c)->log_id;
+
+        ap_run_generate_log_id(c, r, id);
+    }
+
+    if (r->log_id)
+        apr_table_setn(r->subprocess_env, "mod_wsgi.request_id", r->log_id);
+    if (r->connection->log_id)
+        apr_table_setn(r->subprocess_env, "mod_wsgi.connection_id",
+                       r->connection->log_id);
+#endif
 }
 
 typedef struct {
@@ -12325,9 +12353,13 @@ static int wsgi_hook_daemon_handler(conn_rec *c)
 #if AP_MODULE_MAGIC_AT_LEAST(20111130,0)
     r->connection->client_ip = (char *)apr_table_get(r->subprocess_env,
                                                      "REMOTE_ADDR");
+    r->connection->client_addr->port = atoi(apr_table_get(r->subprocess_env,
+                                                          "REMOTE_PORT"));
 #else
     r->connection->remote_ip = (char *)apr_table_get(r->subprocess_env,
                                                      "REMOTE_ADDR");
+    r->connection->remote_addr->port = atoi(apr_table_get(r->subprocess_env,
+                                                          "REMOTE_PORT"));
 #endif
 
 #if AP_MODULE_MAGIC_AT_LEAST(20111130,0)
@@ -12428,7 +12460,9 @@ static int wsgi_hook_daemon_handler(conn_rec *c)
         errno = 0;
         config->request_start = apr_strtoi64(item, (char **)&item, 10);
 
-        if (!(!*item && errno != ERANGE))
+        if (!*item && errno != ERANGE)
+            r->request_time = config->request_start;
+        else
             config->request_start = 0.0;
     }
 
@@ -12448,6 +12482,18 @@ static int wsgi_hook_daemon_handler(conn_rec *c)
                    apr_psprintf(r->pool, "%" APR_TIME_T_FMT,
                    config->daemon_start));
 
+#if AP_MODULE_MAGIC_AT_LEAST(20100923,2)
+    item = apr_table_get(r->subprocess_env, "mod_wsgi.request_id");
+
+    if (item)
+        r->log_id = item;
+
+    item = apr_table_get(r->subprocess_env, "mod_wsgi.connection_id");
+
+    if (item)
+        r->connection->log_id = item;
+#endif
+
     /*
      * Install the standard HTTP input filter and set header for
      * chunked transfer encoding to force it to dechunk the input.