v.1.8.4 update

Author: drlight17

CHANGES.md

@@ -12,6 +12,18 @@ Unreleased VER.
 - provide variables in .env for user defined parser regexps override
 - parse log file from the new lines since last parsing
 
+Ver. 1.8.4
+- ~~GUI: add ability to config gui max log show interval (now it is 3 days) by .env variable~~
+- ~~BUG: gui add non-breaking space ( ) instead of space in multiple recipients row between number and recipient~~
+- ~~BUG: add flex-wrap: nowrap; to .multiple_with_aliases .multiple_mailtos to create proper vertical column of recipients in #mail_to_details~~
+- ~~BUG: mobile narrow gui users edit page fix~~
+- ~~BUG: add error handler for overlimit query (if query output is overlimit RETHINK_ARR_LIMIT env var)~~
+- ~~GUI: add ability to turn off gui parser operating block (by .env variable)~~
+- ~~GUI: remove > [...] in tips modal~~
+- ~~dependabot security fix (python 3.9, quart 0.20.0, werkzeug 3.0.6)~~
+- ~~BUG: in user edit menu changing gui settings cause infinite loading circle~~
+- ~~BUG: add ability to set length of date string in log file to extract timestamp (by .env variable)~~
+
 VER. 1.8.3
 - ~~BUG: multipe recipients view in mobile narrow gui~~
 - ~~BUG: block also autorefresh in case of GUI blocking while parsing~~

Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile for deploy
-FROM python:3.8-alpine
+FROM python:3.9-alpine
 RUN apk update
 RUN apk add bash ncurses musl-dev gcc flock openldap-dev
 # upgrade pip
@@ -10,4 +10,4 @@ COPY Pipfile /app/
 # uncomment to append ypur ca cert for TLS connections i.e. LDAPS
 #COPY ca.crt /usr/local/share/ca-certificates/ca.crt
 #RUN cat /usr/local/share/ca-certificates/ca.crt >> /etc/ssl/certs/ca-certificates.crt
-RUN pipenv install
+RUN pipenv install

Pipfile

@@ -12,8 +12,8 @@ python-dateutil = "*"
 pytz = "*"
 rethinkdb = ">=2.4.7"
 python-dotenv = "*"
-quart = "=0.19.4"
-werkzeug = "=3.0.3"
+quart = "=0.20.0"
+werkzeug = "=3.0.6"
 moment = "*"
 datefinder = "*"
 python-ldap = "*"

README.md

@@ -41,41 +41,35 @@ All maintained docker images could be found [here](https://hub.docker.com/r/drli
 - Docker.io (tested on 23.0.2)
 - Docker-compose (tested on 2.9.0)
 
-
-If you want to use complete image downloaded from docker hub (for production) then you don't have to git clone full repo. You will only need docker-compose.yaml from the root path and create .env file (see description below).
 ```
+If you want to use complete image downloaded from docker hub (fro production) then you don't have to git clone full repo. You will only need docker-compose.yaml from the root path and create .env file (see decription below).
 git clone https://github.com/drlight17/mta-log-parser
 cd mta-log-parser
 cp example.env .env
-```
-If you want to build your image of mta-log-parser then copy Dockerfile and docker-compose.yaml files from build
-dir to the root path (recommended for development). If you want to use complete image downloaded from docker hub then pass this step (recommended for production).
-```
+
+# If you want to build your image of mta-log-parser then copy Dockerfile and docker-compose.yaml files from build
+# dir to the root path (recommended for development). If you want to use complete image downloaded from docker hub then pass this step (recommended for production).
 yes | cp -rf ./build/Dockerfile ./Dockerfile
 yes | cp -rf ./build/docker-compose.yaml ./docker-compose.yaml
-```
-Adjust the example .env as needed. Make sure you set SECRET_KEY to a long random string
-to the password you want to use to log into the web application. Other variables are described. 
-```
+
+# Adjust the example .env as needed. Make sure you set SECRET_KEY to a long random string
+# to the password you want to use to log into the web application. Other variables are described. 
 nano .env
-```
-To build and run app with web GUI run
-```
+
+# To build and run app with web GUI run
 docker-compose up -d
-```
 
-To stop app with web GUI app run
-```
+# To stop app with web GUI app run
 docker-compose down
-```
 
-To schedule log parsing add to your crontab (every minute in example)
-```
+# To schedule log parsing add to your crontab (every minute in example)
 crontab -e
 */1  *   *   *   *   docker exec -t mta-log-parser flock /tmp/lck_mlp /app/run.sh cron
-```
 
 
+
+
+```
 GUI access
 ========
 Rethinkdb web gui is available on the port 8080 (you may change expose port in .env).
@@ -101,13 +95,13 @@ Simply run upgrade.sh
 
 **Application will detect the finished upgrade procedure and will be forced to clear all cached data, cookies and will logout current user session!**
 
-Install in your system (not recommended)
+Install in your system (not recommended as I provide **no support** for this type of usage (issues will be closed, questions will not be answered))
 ========
 
 **Pre-requirements**
 
  - [RethinkDB](https://rethinkdb.com/) (for storing the queryable log data)
- - Python 3.7 MINIMUM (will not work on earlier versions)
+ - Python 3.8 MINIMUM (will not work on earlier versions)
  - Pipenv (`python3 -m pip install pipenv`) - for creating a virtualenv + installing dependencies
 
 
@@ -119,9 +113,7 @@ apt install -y python3 python3-dev python3-pip libsasl2-dev python-dev-is-python
 python3 -m pip install -U pipenv
 
 adduser --gecos "" --disabled-password mailparser
-```
-To ensure that the parser is able to read the mail.log, add the user to the appropriate groups
-```
+# To ensure that the parser is able to read the mail.log, add the user to the appropriate groups
 gpasswd -a mailparser syslog adm postfix
 
 su - mailparser
@@ -131,29 +123,33 @@ cd mta-log-parser
 pipenv install
 
 cp example.env .env
-```
-Adjust the example .env as needed. Make sure you set SECRET_KEY to a long random string
-to the password you want to use to log into the web application.
-```
+# Adjust the example .env as needed. Make sure you set SECRET_KEY to a long random string
+# to the password you want to use to log into the web application.
 nano .env
-```
-Add a crontab entry to run the parse/import script every minute or so
-You should use a file lock utility such as `flock` (included by default on Ubuntu) or `lckdo` to prevent the
-cron overlapping if there's a lot to parse.
-```
+
+# Add a crontab entry to run the parse/import script every minute or so
+# You should use a file lock utility such as `flock` (included by default on Ubuntu) or `lckdo` to prevent the
+# cron overlapping if there's a lot to parse.
+
 crontab -e
 # *  *   *   *   *    flock /tmp/lck_mailparser /home/mailparser/mta-log-parser/run.sh cron
-```
-Development
-===========
-```
+
+####
+# DEVELOPMENT
+####
+
 ./run.sh dev         # Run the development server with automatic restart on edits
 ./run.sh parse       # Import MAIL_LOG immediately
-```
 
-Production systemd service for the WebUI
-===================
-```
+####
+# PRODUCTION
+####
+
+exit
+
+# (AS ROOT)
+
+# Production systemd service for the WebUI
 install -m 644 /home/mailparser/mta-log-parser/mta-log-parser.service /etc/systemd/system/
 systemctl daemon-reload
 systemctl enable mta-log-parser.service

build/Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile for building image
-FROM python:3.8-alpine
+FROM python:3.9-alpine
 RUN apk update
 RUN apk add bash ncurses musl-dev gcc flock openldap-dev
 # upgrade pip

build/docker-compose.yaml

@@ -41,4 +41,4 @@ networks:
         - subnet: "172.20.0.0/16"
     driver_opts:
       com.docker.network.enable_ipv6: "false"
-      com.docker.network.bridge.name: mlp_iface
+      com.docker.network.bridge.name: mlp_iface

deploy/Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile for deploy
-FROM python:3.8-alpine
+FROM python:3.9-alpine
 RUN apk update
 RUN apk add bash ncurses musl-dev gcc flock openldap-dev
 # upgrade pip
@@ -10,4 +10,4 @@ COPY Pipfile /app/
 # uncomment to append ypur ca cert for TLS connections i.e. LDAPS
 #COPY ca.crt /usr/local/share/ca-certificates/ca.crt
 #RUN cat /usr/local/share/ca-certificates/ca.crt >> /etc/ssl/certs/ca-certificates.crt
-RUN pipenv install
+RUN pipenv install

deploy/docker-compose.yaml

@@ -38,4 +38,4 @@ networks:
         - subnet: "172.20.0.0/16"
     driver_opts:
       com.docker.network.enable_ipv6: "false"
-      com.docker.network.bridge.name: mlp_iface
+      com.docker.network.bridge.name: mlp_iface

example.env

@@ -41,6 +41,10 @@ PORT=8487
 # make date time format (default is "HH:mm:ss, DD.MM.YYYY") to use in GUI for emails list timestamps and date picker according to moment.js templates (https://momentjs.com/)
 #DATETIME_FORMAT='HH:mm:ss, DD.MM.YYYY'
 
+# set length of date time string in your log file to process by parser - eliminates many parsing errors, make sure to set properly so that parser won't get extra symbols to further process (20 chars by default)
+# for example if your date in log looks like '2025-02-15 00:00:03.474' value 20 will use '2025-02-15 00:00:03.' to extract datetime
+#DATETIME_INPUT_LENGTH = 20
+
 # convert related logs timestamp according to the DATETIME_FORMAT. High CPU demanding! (false by default)
 #MAIL_LOG_TIMESTAMP_CONVERT=false
 
@@ -54,7 +58,10 @@ PORT=8487
 #EXCLUDE_FROM_TOP_RECIPIENTS='[email protected],[email protected]'
 
 # block GUI refresh while parser worker is running (true by default)
-GUI_REFRESH_BLOCK=true
+#GUI_REFRESH_BLOCK=false
+
+# set GUI max log period in user setting menu (4320 minutes = 3 days by default)
+#GUI_MAX_LOG_PERIOD=4320
 
 # set startup command
 # dev to run in python developer mode

mlp/main.py

@@ -45,7 +45,7 @@
 
 # !!! change version upon update !!!
 global VERSION
-VERSION ="1.8.3"
+VERSION ="1.8.4"
 
 lockfile = "processing.lock"
 # postf_match += r'([A-F0-9]{11})\:[ \t]+?(.*)'
@@ -63,7 +63,8 @@
 #exim_match += r'([0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}).(.+)'
 #exim_match += r'([0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}).(.+)'
 #exim_match = r'.*([0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}).(.+)'
-exim_match = r'([0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}).(.+)'
+#exim_match = r'([0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}).(.+)'
+exim_match = r'([0-9A-Za-z]{6}-[0-9A-Za-z]{6,11}-[0-9A-Za-z]{2,4})\s(.+)'
 """Regex to match the (1) Message ID and the (2) Log Message"""
 
 # sendmail regexp
@@ -189,7 +190,10 @@ async def import_log(logfile: str) -> Dict[str, PostfixMessage]:
             # TODO test new universal datetime extractor instead of regexps (cut first 20 symbols from the string)
             # change to line for exim
             #dtimes = datefinder.find_dates(m.group(0)[:20])
-            dtimes = datefinder.find_dates(line[:20])
+            # add datetime_input_length 
+            #dtimes = datefinder.find_dates(line[:20])
+            dtimes = datefinder.find_dates(line[:settings.datetime_input_length])
+            #print(dtimes)
             #print(line[:20])
 
             for dtime in dtimes:
@@ -392,8 +396,18 @@ async def main():
                 log.info('Housekeeping has deleted NOTHING. There are no emails %d day(s) ago or something went wrong.', housekeeping_days)
         else:
             log.info('Housekeeping is not configured so there will be no deletion of old data! Pay attention to the disk space!')
+
+        # TODO create log lines preparation function before call import_log (1.8.4???)
+        '''log.info('Prepare %s log file', settings.mta)
+        prepared_log = await prepare_log(settings.mail_log)
+
+        log.info('Importing %s log file', settings.mta)
+        import_output = import_log(prepared_log)'''
+
         log.info('Importing %s log file', settings.mta)
         import_output = await import_log(settings.mail_log)
+
+
         log.info('Converting %s log data into list',settings.mta)
         multiple_recipients_qids = import_output['multiple_recipients_qids']
         multiple_recipients = import_output['multiple_recipients']