After celery lib upgrade to 5.3.3, get sql exception.

[rj-wuwei1]

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

Details

N/A

Minimally Reproducible Test Case

Details

Expected Behavior

Actual Behavior

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/billiard/pool.py", line 1793, in safe_apply_callback
fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/worker/request.py", line 624, in on_failure
self.task.backend.mark_as_failure(
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure
self.store_result(task_id, exc, state,
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 526, in store_result
self._store_result(task_id, result, state, traceback,
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/init.py", line 47, in _inner
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/init.py", line 117, in _store_result
task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2861, in iter
result = self._iter()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2875, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
return self._execute_internal(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2124, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 252, in orm_execute_statement
result = conn.execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1414, in execute
return meth(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 489, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
return self._exec_single_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2325, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 747, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ..._taskmeta.traceback AS celery_taskmeta_traceback, celery_tas...
^

[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback, celery_taskmeta.children AS celery_taskmeta_children
FROM celery_taskmeta
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': 'b9e469f3-e53c-4243-92aa-adeaaf5a9757'}]
(Background on this error at: https://sqlalche.me/e/20/f405)

[rj-wuwei1]

Rollback to 5.3.1 version resolved the exception.

[auvipy]

UndefinedColumn can be seen, can you check the logs to find what commit caused this?

[Nusnus]

Maybe this? #8338
Due to error:

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ..._taskmeta.traceback AS celery_taskmeta_traceback, celery_tas...

[auvipy]

since that introduced a new model field, that field need to be applied on DB as well! missing migrations strikes here most probably!

[Nusnus]

since that introduced a new model field, that field need to be applied on DB as well! missing migrations strikes here most probably!

This is a bit of a big deal, I agree.
If we can't resolve this very soon I think we'll need to revert the other PR and release another version as this may cause a lot of issues! @auvipy

[Nusnus]

@Pokerface9711 Thank you for the report! ❤️

[auvipy]

I will handle that. lets wait

[ldacey]

I ran into this with Airflow 2.7.0 when I updated without using constraints:

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ...ry_taskmeta.date_done, celery_taskmeta.traceback, celery_tas...
                                                             ^
[SQL: SELECT celery_taskmeta.id, celery_taskmeta.task_id, celery_taskmeta.status, celery_taskmeta.result, celery_taskmeta.date_done, celery_taskmeta.traceback, celery_taskmeta.children 
FROM celery_taskmeta 

[potiuk]

Question - we are about to release Airflow 2.7.1 with limitted constraints and we will exclude both 5.3.2, 5.3.3 from our celery provider requirements - assuming that it will be fixed by added migrations to celery in 5.3.4.

Is it a sound assumption :)?

[auvipy]

Question - we are about to release Airflow 2.7.1 with limitted constraints and we will exclude both 5.3.2, 5.3.3 from our celery provider requirements - assuming that it will be fixed by added migrations to celery in 5.3.4.

Is it a sound assumption :)?

I would suggest sticking with celery 5.3.1 for that and wait for a new celery point release with proper schema migrations support!

[potiuk]

I would suggest sticking with celery 5.3.1 for that and wait for a new celery point release with proper schema migrations support!

Slightly off-topic but FYI - Just in case you might be interested how it works for us @auvipy: this is what effectively we did by setting our !=5.3.2, !=5.3.3 limits.

The way how airflow dependencies work - we do not "pin" dependencies (because airflow is both library and applicaiton) - we produce constraint files that can be use to make sure the installation is reproducible, but we do not limit the dependencies to be upgraded separately (for example to make it possible to update to newer versions with bugfixes and new features and security fixes). We have a complete mechanism to automatically build the set of constraints for airflow when released. Any by excluding 5.3.2 and 5.3.3 our constraints automatically landed at 5.3.1. See main and 2.7 branch constraints automatically pushed by our CI. Also we realised that the sqlalchemy configuration of celery is not (yet) "integration tested" by Airflow and we added a task for that apache/airflow#34032 otherwise we would have detected this isssue a bit earlier (basically right after release).

but whenever you release 5.3.4 we will run our tests and when they pass we will upgrade our constraints to 5.3.4 (all fully automated). We have 650 transitive dependencies in total,so we had to automate it :) .

We very rarely upper-bind the releases, only when we know that they can cause big issues (for example we now have celery <6 because we know celery generally only introduce breaking changes in major version releases and those are really breaking - we had to deliberately made efforts to make Airlfow Celery 5-compatible and that's cool).

Also as discussed in apache/airflow#34031 (comment) PyPI has protections against re-releasing already released versions (so celery will have to go with 5.3.4 effectively) so - all is cool I think for us.

So effectively: we are sticking with 5.3.1 as you suggested, while we are stil open to upgrade to new versions of celery automaticaly when it happens.

[auvipy]

yes I learnt from PyPI today that we can't release a same version again! we yanked them and will release 5.3.4 very soon.