Is there any module for handling database changes that leverages mongoengine to perform the migrations ?

[DachuanZhao]

https://stackoverflow.com/questions/56160585/is-there-any-module-for-handling-database-changes-that-leverages-mongoengine-to
Like alembic and sqlalchemy.
Is there any module for handling database changes that leverages mongoengine to perform the migrations ?

[bagerard]

Hi @DachuanZhao, Unfortunately no. The way we are handling migrations at my company is by using pymongo directly

[ghonchesefidi]

@bagerard Can you please explain more about migrations using pymongo?
I'm using Flask and Mongoengine as my DB.
How can we automate migrations process ?

[ghonchesefidi]

It will be nice if you give a clue about this subject then I will try it on my own project and if it works I will write and update Mongoengine documents about it.
@bagerard

[ghonchesefidi]

I had googled a lot and almost fount nothing except this package which I couldn't work with it due to defective documentation.

[stefanvanburen]

@ghonchesefidi I haven't used it myself, but have you tried something similar to what's outlined here: https://blog.sneawo.com/blog/2017/04/14/migrations-for-mongoengine/ ?

[ghonchesefidi]

I'm working on it now. Let's see how it will work. I will share the result.

[ghonchesefidi]

@bagerard I couldn't make use of it. How should I import and use it? as I said here documentation is not good

[bagerard]

Hi, I've never used the flask-mongoengine-migration package, the way we handle migration at my company is simply by keeping a version attribute in the database, and then we just have a python script (organized by migration tasks) wrapped with a cli interface. It checks the version and run the migrate tasks accordingly. Such script runs after deploying a new version. We don't try to introspect the mongoengine schemas or anything.

As for the migration tasks themselves, it depends on what you need to migrate but for tasks such as:

• adding a new field with a default value
• removing a field from a collection
• removing a collection
• renaming a field in a collection

The easiest is to get the corresponding pymongo collection of your models and apply the change through the pymongo api.

I wanted to do this for a while so I've actually starting to write docs on migration of models, its currently on this branch if you want to have a look: https://github.com/bagerard/mongoengine/blob/add_migration_documentation/docs/guide/migration.rst

[ghonchesefidi]

@bagerard Thanks for your reply. I will check the link you provided. I think now I'm able to solve migration problem.
but automating migrations will be a good feature for mongoengine to add ! I hope to see it in next versions.

[amcgregor]

Since "migrations" in the MongoDB sense are essentially a transcript of generally atomic operations, I don't bother with extensive migration tracking or coding tools. A function issuing a few PyMongo requests is more than enough. As such, my "migration runner" literally just uses pkg_resources.resource_listdir to search for all .py files a la package/*/migration/*.py (with an area-based source organization, e.g. that * mapping to "account", or "invoice", or "company", or…), then collects all top-level non-underscore-prefixed functions, with the migration functions themselves able to identify if they need to run or not.

Simple, effective. Made excessively pretty with ANSI color sequences and rate of progress estimations. And a spinner that goes . o O 0 O o . in a cycle. Interesting not-quite-migration use of this, pre-aggregated analytics. We have an "analytic" collection with hourly activity periods generated from a "hits" collection (as the clicks come in). In development, we won't DB restore the "analytic" collection, there's no need as a migration checks for existing analytic periods, and if not found, generates them from the actual hit data automatically.

[moonkwoo]

If I rename a field on collection, is there any method for loading values from older documents where the field is still using the former name Like Alsoload from doctrine? Like this, https://www.doctrine-project.org/projects/doctrine-mongodb-odm/en/2.0/reference/migrating-schemas.html

[amcgregor]

@moonkwoo There are two Python programming approaches to this problem which do not require any support from the data access layer at all… not really.

I tend to strongly prefer "transparent live migration on access" for particularly complex objects or very large datasets, where issuing a single $rename operation under a lock (blocking other access during the migration) is impractical.

Re-Naming a Field in Python, Preserving Access via Old Name

Initial Version

class Person(Document):
	username = StringField()

Whoops, that should have been the primary key.

class Person(Document):
	_id = StringField('_id')
	
	username = _id  # Same field, different accessible name Python-side.

One may wish to warn on access to deprecated / legacy attribute names, and possibly make such access read-only, using a property. (A setter to warn and assign to the updated name can also be added.)

class Person(Document):
	_id = StringField('_id')
	
	@property
	def username(self):
		warnings.warn("Username is now primary key.", DeprecationWarning, stacklevel=1)
		return self._id

Transparent Live Migration On Access

Now we start to step out of the DAO a bit. If the collection may contain mixed documents with old and new keys (by gum, try to avoid this) things may seem to get a little out of hand. But detecting such legacy objects may provide a convenient point to transparently fix them.

This gets a bit tricky by accessing the "internal" MongoDB data representation to "swap" the fields upon detection of a legacy document, but it does this seemingly complex step to avoid marking the active document as "dirty", and to avoid an extra round-trip by utilizing reload().

class Person(Document):
	_old = StringField('username', default=None)
	_new = StringField('_id')
	
	@property
	def username(self):
		if Person._old.db_field in self._data:  # detect the legacy attribute
			# update remotely; needs to be CAS atomic, though!
			self.update(__raw__={'$rename': {Person._old.db_field: Person._new.db_field}})
			# update ("swap") locally w/o setting the dirty flag
			self._data[Person._new.db_field] = self._data.pop(Person._old.db_field)
		
		return self._new

Also needing a setter+deleter, but that handles the "seamless, live migration" of old objects. It should be noticeable how generic the above code actually is. This can absolutely be isolated into a reusable property-generator function, or class decorator.

Take heed of the CAS ("compare and swap") warning, though. If two processes attempt this, because of the delay between initially loading the document and eventually accessing the attribute, two may attempt the rename. Make it conditional (find().update()) on the field actually still existing ($exists). No error can be generated, then, from the update failing outright, and the result is irrelevant; if this process did it, or a different one did, we still swap our local values and continue business as usual.