cffi: update to use libyang v4.2.2

[steweg]

This PR introduces several commits needed for proper functionality with libyang v4.1.0

[steweg]

This PR apparently needs to wait for official release of libyang v4.1.0. If you would like to test it by yourself in the mean time, use libyang devel branch

[choppsv1]

What will happen if the underlying system hasn't upgraded to libyang v4?

[steweg]

What will happen if the underlying system hasn't upgraded to libyang v4?

It won't work. As libyang v3 vs v4 makes backward incompatible changes, the similar will happen with libyang-python. I tried to keep the libyang-python API unchanged as much as possible, but some options are simply gone and AFAIK there is no other way around. I believe it is similar to when libyang was upgraded from v2 to v3.

[choppsv1]

What will happen if the underlying system hasn't upgraded to libyang v4?

It won't work. As libyang v3 vs v4 makes backward incompatible changes, the similar will happen with libyang-python. I tried to keep the libyang-python API unchanged as much as possible, but some options are simply gone and AFAIK there is no other way around. I believe it is similar to when libyang was upgraded from v2 to v3.

I think for our use we won't be able to require our users upgrade to 4. So I think the main problem is going to be making sure they get a working version of the python bindings. It might make sense to create a libyang3 stable branch to track critical fixes. Also not sure how to handle 2 versions of the bindings with pypi. Does pypi let you update multiple versions that vary by major?

[steweg]

What will happen if the underlying system hasn't upgraded to libyang v4?

It won't work. As libyang v3 vs v4 makes backward incompatible changes, the similar will happen with libyang-python. I tried to keep the libyang-python API unchanged as much as possible, but some options are simply gone and AFAIK there is no other way around. I believe it is similar to when libyang was upgraded from v2 to v3.

I think for our use we won't be able to require our users upgrade to 4. So I think the main problem is going to be making sure they get a working version of the python bindings. It might make sense to create a libyang3 stable branch to track critical fixes. Also not sure how to handle 2 versions of the bindings with pypi. Does pypi let you update multiple versions that vary by major?

Frankly speaking, the requirement of having specific version or higher of libyang is kind of dependency that I would expect to be there since the first version of libyang-python, including some checks during first installation and also during upgrade process. The simple reason is that even new APIs are introduced in libyang within the same major version and if you want to introduce that within libyang-python, you still need to make sure about the version of libyang. Otherwise this will never work correctly. As I am not maintainer of this repo, I never looked in specific how this is being achieved besides just putting version restriction within cffi/source.c file. There is also one more important note. I was told by libyang maintainer that bugfixes are not always backported to older versions (see this example: CESNET/libyang#2428 (comment)). So at some point the upgrade of underlying libyang becomes inevitable.

Regarding the pypi versioning. In general updates are possible and can support multiple major versions, but it depends on maintainers, how many stable versions they want to support and all the staff along with it (backporting bugfixes....).

[rjarry]

My time for maintaining libyang-python is very limited. I will not be able to support maintenance/stable branches. When libyang 4.x is released, I will accept this PR and we can bump the python project version to v4.0.0 to follow libyang version scheme.

[rjarry]

Could you squash all in a single commit? I want the git history to remain bisectable.

Also, could you use imperative mood in your commit message?

libyang-python/README.rst

Lines 258 to 261 in ddb7e99

	- Describe your changes in imperative mood, e.g. *"make xyzzy do frotz"*
	instead of *"[This patch] makes xyzzy do frotz"* or *"[I] changed xyzzy to
	do frotz"*, as if you are giving orders to the codebase to change its
	behaviour.

[steweg]

Could you squash all in a single commit? I want the git history to remain bisectable.

Also, could you use imperative mood in your commit message?

libyang-python/README.rst

Lines 258 to 261 in ddb7e99

	- Describe your changes in imperative mood, e.g. *"make xyzzy do frotz"*
	instead of *"[This patch] makes xyzzy do frotz"* or *"[I] changed xyzzy to
	do frotz"*, as if you are giving orders to the codebase to change its
	behaviour.

Done

[choppsv1]

My time for maintaining libyang-python is very limited. I will not be able to support maintenance/stable branches. When libyang 4.x is released, I will accept this PR and we can bump the python project version to v4.0.0 to follow libyang version scheme.

Perhaps I could offer to help? I'd prefer not to have to fork and do this alone, so a collaboration would be preferable. Maybe you could create a v3 stable branch and grant permissions to that, and I (for a time) could maintain it, doing the back-ports required (in FRR we use mergifyio to make this as painless as possible). That way v3 would still be updated and fetchable the same way from pypi.

[rjarry]

Perhaps I could offer to help? I'd prefer not to have to fork and do this alone, so a collaboration would be preferable. Maybe you could create a v3 stable branch and grant permissions to that, and I (for a time) could maintain it, doing the back-ports required (in FRR we use mergifyio to make this as painless as possible). That way v3 would still be updated and fetchable the same way from pypi.

Oh that would be much appreciated 👍

Sorry, I had missed your message and only see it now. I have invited you as a member of the project with write/push access. I have created a b_3.x branch which should be ready to work.

master...b_3.x

In order to create new releases on pypi.org, all you need to do is push a signed/annotated tag on that branch following the pattern v3.x.y where x and y are positive integers. Try to use semantic versioning where it makes sense. The CI will take care of the rest.

libyang-python/.github/workflows/ci.yml

Lines 94 to 114 in 3815807

	deploy:
	needs: [lint, test]
	if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
	runs-on: ubuntu-24.04
	steps:
	- uses: actions/checkout@v3
	- uses: actions/setup-python@v4
	with:
	python-version: 3.x
	- uses: actions/cache@v3
	with:
	path: ~/.cache/pip
	key: pip
	restore-keys: pip
	- run: python -m pip install --upgrade pip setuptools wheel
	- run: python -m pip install --upgrade pep517 twine
	- run: python -m pep517.build --out-dir dist/ --source .
	- uses: pypa/gh-action-pypi-publish@release/v1
	with:
	user: ${{ secrets.pypi_user }}
	password: ${{ secrets.pypi_password }}

Thanks for stepping up ❤️

[choppsv1]

In order to create new releases on pypi.org, all you need to do is push a signed/annotated tag on that branch following the pattern v3.x.y where x and y are positive integers. Try to use semantic versioning where it makes sense. The CI will take care of the rest.

Got it.

Might need to do something with tox.ini/tox-install.sh to have it fetch the correct v3 of libyang too. I'll take a look at that.

Thanks for stepping up ❤️

Thanks for the invitation. :)

[Krisscut]

Looks like libyang released last week: https://github.com/CESNET/libyang/releases/tag/v4.2.2

[steweg]

Looks like libyang released last week: https://github.com/CESNET/libyang/releases/tag/v4.2.2

I adjusted the patch to work with v4.2.2. Unfortunately there is still one test failing due to very strange issue, for which I have asked help of libyang maintainers (CESNET/libyang#2457).

[jeremie6wind]

Hi,
Do you need to do something like this?

------------------------- tests/yang/yang-library.json -------------------------
index 3dbf2568a99f..f9664845d6d8 100644
@@ -1,16 +1,31 @@
 {
   "ietf-yang-library:modules-state": {
     "module-set-id": "e595da11ace92c0d881995fa7e56bbe86f1f48e9",
     "module": [
       {
         "name": "yolo-system",
         "revision": "1999-04-01",
         "namespace": "urn:yang:yolo:system",
         "conformance-type": "implement"
       }
     ]
   },
   "ietf-yang-library:yang-library": {
+    "module-set": [
+      {
+        "name": "complete",
+        "module": [
+          {
+            "name": "yang",
+            "revision": "2025-01-29",
+            "namespace": "urn:ietf:params:xml:ns:yang:1",
+            "location": [
+              "file://@internal"
+            ]
+          }
+        ]
+      }
+    ],
     "content-id": "321566"
   }
 }

[steweg]

Hi, Do you need to do something like this?

------------------------- tests/yang/yang-library.json -------------------------
index 3dbf2568a99f..f9664845d6d8 100644
@@ -1,16 +1,31 @@
 {
   "ietf-yang-library:modules-state": {
     "module-set-id": "e595da11ace92c0d881995fa7e56bbe86f1f48e9",
     "module": [
       {
         "name": "yolo-system",
         "revision": "1999-04-01",
         "namespace": "urn:yang:yolo:system",
         "conformance-type": "implement"
       }
     ]
   },
   "ietf-yang-library:yang-library": {
+    "module-set": [
+      {
+        "name": "complete",
+        "module": [
+          {
+            "name": "yang",
+            "revision": "2025-01-29",
+            "namespace": "urn:ietf:params:xml:ns:yang:1",
+            "location": [
+              "file://@internal"
+            ]
+          }
+        ]
+      }
+    ],
     "content-id": "321566"
   }
 }

Unfortunately this is little bit outside of my expertise as I don't use this feature of libyang, I was just trying to fix the tests. The issue was actually taken care of directly by libyang maintainer. It is already fixed, but it will be in the next release. The question now would be that, which option should we go through:

• adjust the test as was proposed (I am assuming that it works) - side effect could be that if there is a user which uses exactly the same approach as was used in the original test, needs to adapt
• remove the test for good - according to indication from libyang maintainer it can be that is completely wrong use-case. I cannot judge this. And it has the same side effect as in previous point.
• wait for fixed version of libyang and keep the test as-is

My approach was to minimize the number of backward incompatible changes to bare minimum. But adding one more is not that big of deal. So I will leave it up to maintainers of this repo to decide