feat: add per-picture forced IDR API

[ylatuya]

Introduce VVENC_PIC_FLAG_FORCE_IDR on vvencYUVBuffer, and thread the request through preprocessing and GOP handling so that a selected input frame is encoded as a forced IDR/CRA picture, respecting the configured refresh type.

I tried to mimic the API of other encoders that use per-picture flags to define optional encode requests.

The new API is only enabled with VVENC_USE_UNSTABLE_API.

Unit tests have been added for the new behavior.

Fixes #311

[K-os]

Thank you very much for the contribution. From a first glance it looks very good to me, but I think we need to talk about this internally.

I currently just have one small remark: I think the #if VVENC_USE_UNSTABLE_API blocks are only needed for the externally visible interface. So they should be in vvenc.cpp and vvencimpl.cpp, but I don't think we need them in EncoderLib and CommonLib files.

Furthermore, you might want to add your name to the AUTHORS.md file.

[ylatuya]

Thank you very much for the contribution. From a first glance it looks very good to me, but I think we need to talk about this internally.

Hi! Thanks for your feedback. I opened the PR to start the discussion. I understand this is an important change API-wise and that it needs to be reviewed and approved internally.

Furthermore, you might want to add your name to the AUTHORS.md file.

I will do that as well.

[ylatuya]

I currently just have one small remark: I think the #if VVENC_USE_UNSTABLE_API blocks are only needed for the externally visible interface. So they should be in vvenc.cpp and vvencimpl.cpp, but I don't think we need them in EncoderLib and CommonLib files.

I had to guard the implementation with VVENC_USE_UNSTABLE_API ifdef's because they depend on the newly added enum and the new picFlags field. The userData field does not require it, since the implementation does not access it.

I will review everything to see how to omit them in EncoderLib and CommonLib

[K-os]

Ah, ok. I had the impression that #ifdefs in EncoderLib and CommonLib were not necessarily needed. But if they are, that's fine. If removing the #ifdefs there complicates the implementation, just leave it as it is.

[ylatuya]

Ah, ok. I had the impression that #ifdefs in EncoderLib and CommonLib were not necessarily needed. But if they are, that's fine. If removing the #ifdefs there complicates the implementation, just leave it as it is.

Most of them can be removed. I have updated the PR with that fixed.

[chrs303]

Hey! I had a short look at the PR and I'd like to run some tests. Therefore I'd like to know how can I configure the encoder to insert the IDRs. I want to run tests with fixQP and rate control and check other test cases. Could you give me an example configuration?

[ylatuya]

Hey! I had a short look at the PR and I'd like to run some tests. Therefore I'd like to know how can I configure the encoder to insert the IDRs. I want to run tests with fixQP and rate control and check other test cases. Could you give me an example configuration?

No special configuration is required for the encoder. Insertion of IDRs is done programmatically, by setting VVENC_PIC_FLAG_FORCE_IDR in the picFlags of the input picture. You can use the unit tests as a reference. The test configures the encoder with a GOP size of 16 pictures, it encodes 12 pictures, and forces an IDR in the 6th picture.

https://github.com/fluendo/vvenc/blob/57f10ae53549d7b1e222e1a3d64b6a0e33a14b48/test/vvenclibtest/vvenclibtest.cpp#L357

[ylatuya]

@K-os One thing I have doubts about is the name IDR in VVENC_PIC_FLAG_FORCE_IDR. Depending on the configured refresh type, it can generate a CRA, an IDR_W_RADL, or an IDR_N_LP. Is it fine as it is, or do you have suggestions for a better name?

[chrs303]

Hey! I had a short look at the PR and I'd like to run some tests. Therefore I'd like to know how can I configure the encoder to insert the IDRs. I want to run tests with fixQP and rate control and check other test cases. Could you give me an example configuration?

No special configuration is required for the encoder. Insertion of IDRs is done programmatically, by setting VVENC_PIC_FLAG_FORCE_IDR in the picFlags of the input picture. You can use the unit tests as a reference. The test configures the encoder with a GOP size of 16 pictures, it encodes 12 pictures, and forces an IDR in the 6th picture.

https://github.com/fluendo/vvenc/blob/57f10ae53549d7b1e222e1a3d64b6a0e33a14b48/test/vvenclibtest/vvenclibtest.cpp#L357

Did you test it with more frames too? When I set framesToEncode >= 24 only the first frame is encoded and he stops processing more frames. It seems like a flush is missing. I'm having a deeper look.

And FYI, I added config parameters, i.e. the number and list of the intra POCs

[ylatuya]

Did you test it with more frames too? When I set framesToEncode >= 24 only the first frame is encoded and he stops processing more frames. It seems like a flush is missing. I'm having a deeper look.

I didn't test with more frames. I think the issue come from the fact that when a new GOP is force-started, the previous GOP has gaps in the coding numbers. This causes the process to stall after the first frame with cn=0, waiting for the next coding number with cn=1 that does not exist in https://github.com/fluendo/vvenc/blob/force-idr/source/Lib/EncoderLib/EncGOP.cpp#L1485. I think a solution is being able to mark the GOP as incomplete so that cn gaps are allowed.

This is the instrumented output from encodePicture

[encodePicture] loop=0 flush=0 inputPending=0 picsRcvd=9 queueEmpty=0 accessUnitOutputStarted=0 waitAndStay=0 auSize=0 pendingAu=0
  PreProcess: queued=4 free=2 minQueue=1 startPoc=0 flushSeen=0 flushAll=1 nonBlocking=0
    [0] poc=5 cn=8 gopNum=1 entryPoc=5 slice=I tid=0 startGop=1 startIntra=1 forcedIdr=1 sharedForcedIdr=1 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [1] poc=6 cn=21 gopNum=1 entryPoc=6 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [2] poc=7 cn=20 gopNum=1 entryPoc=7 slice=B tid=3 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [3] poc=8 cn=22 gopNum=1 entryPoc=8 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
  MCTF: queued=9 free=0 minQueue=7 startPoc=0 flushSeen=0 flushAll=1 nonBlocking=0
    [0] poc=0 cn=0 gopNum=0 entryPoc=0 slice=I tid=0 startGop=1 startIntra=1 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [1] poc=1 cn=5 gopNum=0 entryPoc=1 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [2] poc=2 cn=4 gopNum=0 entryPoc=2 slice=B tid=3 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [3] poc=3 cn=6 gopNum=0 entryPoc=3 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [4] poc=4 cn=3 gopNum=0 entryPoc=4 slice=B tid=2 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [5] poc=5 cn=8 gopNum=1 entryPoc=5 slice=I tid=0 startGop=1 startIntra=1 forcedIdr=1 sharedForcedIdr=1 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [6] poc=6 cn=21 gopNum=1 entryPoc=6 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [7] poc=7 cn=20 gopNum=1 entryPoc=7 slice=B tid=3 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [8] poc=8 cn=22 gopNum=1 entryPoc=8 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
  EncGOP: queued=3 free=0 minQueue=17 startPoc=0 flushSeen=0 flushAll=0 nonBlocking=0
    [0] poc=0 cn=0 gopNum=0 entryPoc=0 slice=I tid=0 startGop=1 startIntra=1 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [1] poc=2 cn=4 gopNum=0 entryPoc=2 slice=B tid=3 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0
    [2] poc=1 cn=5 gopNum=0 entryPoc=1 slice=B tid=4 startGop=0 startIntra=0 forcedIdr=0 sharedForcedIdr=0 init=0 recon=0 referenced=1 neededOut=1 finished=0 flush=0 inProc=0 refCnt=0

[ylatuya]

It should be fixed now. I will also expand the test suite to verify that forcing an IDR at any picture from the GOP works.

[chrs303]

Thanks for your fix. Unfortunately there are still issues when I do more tests. For example:

• with setting forcedPoc = 16 the encoder ends up in a loop and stops processing pictures
• with setting e.g. framesToEncode = 49 checkForcedPerFrameIdrNoRadl fails
• with setting vvencParams.m_poc0idr = 0 all tests are failing

[ylatuya]

Thanks for your findings. I'm planning to expand the test suite to catch all these errors and probably others. I'm new to this codebase and I am using this to learn about it in more depth.

[ylatuya]

I have extended the test suite to cover the following scenarios with all their permutations:

• The first picture is an IDR (pic0idr=1)
• The first picture is not an IDR (pic0idr=0)
• The IDR is forced within the initial GOP
• The IDR is forced after the first GOP is completed
• Forcing different pictures: the 2 first pictures of the GOP, 2 pictures mid-gop, the last 2 pictures of the GOP

All these scenarios are now working correctly.

I am now trying to fix the other issue you detected, encoding 50 frames.