Kroxylicious 1.0

[tombentley]

This PR proposes:

• a release of Kroxylicious 1.0 in the near future,
• a commitment to making patch releases

[k-wall]

@tombentley, thanks for writing this up.

[SamBarker]

While I'm supportive of a move to 1.0 I have a few caveats around the proposed model.

[SamBarker]

I'm wondering if the example would be better expressed as a table.

[SamBarker]

Given what we say about API(java/config) stability why would we offer anything other than roll forward to the latest patch release?

I am very wary about building up a branch of un-released commits/fixes. While I accept releases have a cost, ours are fairly cheap and getting cheaper (we keep automating stuff) I think we should aim to release every minor release branch we start. However i'd also advocate for lazy branching around this. Only start a 1.x-1 branch if and when requires

[robobario]

The bleeding edge is more or less the first time our new code comes into contact with the real world. It's a likely place for performance etc to degrade. Conservative Users may be putting that version through it's paces in test environments before adopting in production, so supporting a couple of minor versions could be a good idea.

[tombentley]

@SamBarker the language around the number of maintained minor releases was based on what I thought we had discussed and agreed out of band, but maybe I misunderstood. I added the 6 month thing off my own bat purely as a way of making the commitment less open-ended. I'm happy to change it if that's not the consensus we reached, but we (the maintainers on the hook for actually making good on this) all need to agree what we're willing to do here.

Personally I feel that just "roll forward" is not really good enough. It means that users have to accept the additional risk of taking a bunch of unrelated features in order to get a fix. That's just not always acceptable in production, and I think it therefore stands to harm adoption in production environments.

Therefore I would advocate for:

• Creating a 1.x branch at the time we release 1.x.
• Cherry picking to this branch. What gets cherry-picked is partly up to the community. If someone from the community it willing to open a charry-pick PR targeted at 1.x that's great, and we should endevour to get that into a release. This is why a 1.x branch needs to be created when releasing 1.x.0, so that anyone can just go ahead and open such a PR.
• However, I don't think we should be doing a release simply because a single commit got made to the 1.x branch. The costs of a release are not just bourne by the maintainers who are clicking the buttons on some automation (however fancy). There is a cost for the whole community in terms of assessing whether or not to pick up the new 1.x.y release, and in actually rolling it out. It's no good the project pushing out a new patch release every week if users cannot keep up. We also want the community to participate in testing. For example, if Billy Bob opened a cherry-pick PR that's going to be included in a release then Billy Bob might well want to confirm that the proposed 1.x.y release fixes their problem. Bear in mind what's in the release might be more than just Billy Bob's PR, so it reasonable to want to be sure that none of the other fixes as regressed Billy's fix. Also it's possible that the maintainers haven't been able to reproduce Billy's problem (e.g. maybe it's hardware specific), so their feedback is invaluable.
• For all these reasons it's I think it's inevitable that some number of fixes get merged to the 1.x branch, building up, before a release of 1.x.y happens. This is OK! We can still decide to expedite a release if a particularly terrible bug/CVE is found. But otherwise batching a bunch of fixes actually makes the eventual release more consumable, and more likely to actually be used (because the risk/benefit looks more favourable if a release fixes several problems rather than a single one). And releases being used is, after all, why we do them.

We need to decide how many 'live' release branches we want on the go. Obviously the effort here is multiplicative, so we need to be realistic. Apache Kafka generally manages to release two patch releases for each minor (see https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan), and they're usually spaced about 3-4 months apart. Kafka also aims for 3 a.b.0 releases a year. This averages out to there being 3 'next releases' at any one time: A .0, a .1 and a .2. Personally my gut feeling is we could sustain something similar, but note the implication that we slow down the pace on the .0 releases.

So then the final loose end is when and how we stop making releases from a minor branch. Here the Apache Kafka project doesn't really have anything written down. But in practice patches do get merged to minor branches which are never publicly released. However, they are available to those motivated to do their own building and testing. That seems like a reasonable balance to me: The project is not obliged to release them, but the community has the ability (and some motivation) to contribute those patches to the project.

cc @k-wall @gracegrimwood @robobario @hrishabhg, because this discussion is kinda central to the whole thing.

[robobario]

Only start a 1.x-1 branch if and when requires

On this point I would value having the release branches created as part of the standard workflow so that we don't have diverging release flows (release latest off main, vs other releases off release branch). Having them exist before we need them will enable community members to PR backports directly to them without having to request a branch creation. It makes things more democratic and approachable if all the processes are in place and Contributors just have to review/approve things and pull the release levers.

[robobario]

I'm also wondering if it would be manageable to have things more time based. So have a scheme like:

1. the latest minor version is actively supported
2. when a new minor version is created, on date X ,the previous minor version is moved to a different support level
3. the previous minor now has support dates set based on date X, like 3 months of bugfixing and 6 months of security fixing for example.

This would give Users an easy to interpret table of dates for when a minor goes EOL. And users would know that they are getting at least that many months of support for the latest minor version. Under this scheme there would always be a transition period when we put out a new minor version where we support two versions, where I think with the current proposal if you didn't release for 6 months the prior minor version would be unsupported on day 0 of the new release.

current actively supported version : 1.4.x

old versions:

version    | bugfix support until       | security support until (EOL)   |
1.3.x      | 01-03-2026                 | 01-06-2026                     |
1.2.x      | 01-02-2026                 | 01-05-2026                     |
...

But this could put us on the hook for more than the proposed 3 minor versions of maintenance. It feels like, as Tom said, we will probably slow down the minors due to these new processes and guarantees though. We may end up following Kafka's cadence, to keep up we have to bump kafka-clients and take minor or major changes to the Filter API.

Maybe those concerns are manageable by keeping the lengths of time tight enough

[gracegrimwood]

This got suuuuper long, sorry 😅

Here's a sort of TL;DR of my thoughts:

• We should be actively creating and maintaining new patch releases for the last two minor versions, as well as the current minor version (i.e. 1.x, 1.x-1, and 1.x-2). To limit the overhead, there should be no more than 3 actively maintained minor versions at any time.
• The EOL date for the oldest active minor version 1.x-2 should be upon the release of the third subsequent minor version 1.x+1 or six months after the initial release of the expiring version 1.x-2, whichever comes first.
• We shouldn't try to schedule the EOL or maintenance windows of releases ahead of time, so we don't overcommit ourselves. We could revisit this after we've reached 1.0.0 and tried out a less rigid maintenance schedule for a bit.
• We should stop merging PRs to a release branch once that version is no longer receiving new patch releases; the purpose of these branches is to create patch releases to serve users of the proxy, anything beyond that is unnecessary extra work. We could revisit this later if there is demand for it.
• We should have release branches for minor releases in addition to the existing release tags, and these should be created during their initial (minor.0) release by our release automation. These branches can then accept PRs for backporting fixes, and new patch releases for their respective versions can be made from those branches. Patch releases only need release tags, not branches.
• Backports should be community-driven via raising PRs for backports against active release branches. Maintainers would be responsible for approving these PRs, backporting high-severity security fixes, and for cutting patch releases.
• The timing of patch releases should be driven by the community through a proposal process (separate from this repo), where a member of the community proposes a new patch release and others can vote on it. Urgent patch releases (such as for high-severity security issues) should bypass this. If we find we are creating patch releases too often, we could look at implementing rules to limit this.
• Maintaining multiple minor releases might mean we're inclined to put out minor releases less often but it shouldn't mean the pace of development is meaningfully slower. Also, being more mindful about what we're releasing and how often might not be a bad thing.
• Maintaining multiple minor releases will mean more things for us to review and more times we have to run the release workflow, but that would happen to some extent anyway as the project grows.
• Not providing any active maintenance for prior releases and instead taking a stance of telling users to just roll forward to the latest version risks alienating a significant number of people who might otherwise use Kroxylicious, and isn't in the best interests of the project.
• We really should not be talking about any of this as "support". It muddies the water as to what it is we're trying to accomplish as a project, so we should use the term "active maintenance" or just "maintenance" if we can.

---

the language around the number of maintained minor releases was based on what I thought we had discussed and agreed out of band

We discussed the idea of maintaining and issuing patches for 1.x, 1.x-1, and 1.x-2, but I don't know if we actually reached an agreement. Either way, it sounds like a good number to me; any less isn't really worth it, any more could become unsustainable.

I added the 6 month thing off my own bat purely as a way of making the commitment less open-ended.

I think 6 months is fine as a sort of hard-coded "expiry date" for minor releases. We currently release much more frequently than that, so I reckon it's unlikely any release would hit that deadline anyway. Even if we did let the oldest minor release "expire", having another minor release in there between oldest and latest means users still have a buffer zone to work in, and 6 months seems like plenty of time to arrange a proxy upgrade to a new minor version. We can revisit that number later on if we need to, but I think it's fine.

Personally I feel that just "roll forward" is not really good enough. It means that users have to accept the additional risk of taking a bunch of unrelated features in order to get a fix. That's just not always acceptable in production, and I think it therefore stands to harm adoption in production environments.

I fully agree. We can't know every user's environment and risk profile, and we risk alienating potential users if our response to requests for backported fixes is to just install the latest version. I don't think that stance is in the best interests of the project, even if it would mean less maintenance work for us. Besides, if it was feasible for everyone to just roll forward all the time, nobody would make any money selling LTS versions of things 😉

• Creating a 1.x branch at the time we release 1.x.
• Cherry picking to this branch. What gets cherry-picked is partly up to the community.

Agreed, starting out with a major.minor minor release branch at initial major.minor.0 release time to serve as the source of all future major.minor.x patch releases is the way to go. As @robobario suggested, let's have our release automation create it for us, perhaps when it creates the release tag? Then we let the community put up PRs against these minor release branches with backported fixes, and our job is mostly to approve or not, as we currently do with PRs to main.

Patch releases can then be tagged in git as usual but won't get their own release branches. We shouldn't put out patches of patch releases like major.minor.patch.x, as per semver, so any fixes should instead be backported to the respective minor branch and then released in the next patch release of that minor version. For example, if someone wanted to backport a fix to release major.minor.patch then instead they would raise a PR against the major.minor release branch, and that would get released as major.minor.patch+1.

I think there's still definitely a bigger role for us as maintainers in this process, like urgent backporting of high-severity exploitable CVEs, but otherwise I think this process really should be driven by the people using the proxy on the daily, as they're best placed to know which fixes they need and don't need. We can step in when something that isn't a fix (i.e. a feature) gets suggested, for anything urgent, or for anything that's too complicated to backport, but for the most part I see this as the kind of system that starts to run itself.

• However, I don't think we should be doing a release simply because a single commit got made to the 1.x branch. The costs of a release are not just bourne by the maintainers who are clicking the buttons on some automation (however fancy). There is a cost for the whole community in terms of assessing whether or not to pick up the new 1.x.y release, and in actually rolling it out. It's no good the project pushing out a new patch release every week if users cannot keep up. [...]
• For all these reasons it's I think it's inevitable that some number of fixes get merged to the 1.x branch, building up, before a release of 1.x.y happens. This is OK! We can still decide to expedite a release if a particularly terrible bug/CVE is found. But otherwise batching a bunch of fixes actually makes the eventual release more consumable, and more likely to actually be used

Yeah, batching fixes is the way to go. As for timing, I think the we should handle this through proposals, honestly. Maybe not anything as big or elaborate as what we do here in this repo, but having a forum or something (like Slack or GitHub discussions or just raising an issue with a special tag on it) where community members can suggest that they think it's time for a patch release of whatever active minor version and then others can vote on that idea. Obviously for anything high-risk we could just go ahead with creating a patch release anyway, but for everything else this lets us take a more hands-off approach.

If, by some chance, that proposal process gets out of hand and folks do start requesting new patch releases every week, then we could look at instituting some kind of limit on how often we will release new patch versions, but that's the kind of bridge we should cross if/when we get to it. I think trying to come up with a schedule for this is a bit of a fool's errand because there will be some times we get to a patch release date and there's nothing worth releasing, and some times when something will show up well ahead of schedule and we'll want to put patches out ASAP. We're better off letting the people who use the proxy determine (for the most part) when new patch releases are required.

We need to decide how many 'live' release branches we want on the go. Obviously the effort here is multiplicative, so we need to be realistic.

I think if we stick to only having branches for minor releases then the number of actively maintained "live" branches comes down to how many minor releases we intend to create new patch releases for. Like I said above, I think the current and the two previous minor releases (1.x, 1.x-1, and 1.x-2) is a good number. I don't think it means we will have to slow development of new features (.0's) because we'd be leaning on the community to do most of the heavy lifting as far as maintenance effort goes, but it might change how often we cut new releases from main and make us think more carefully about what goes into them, which isn't necessarily a bad thing. And of course, all of this will mean more things for us to review and more releases, but we're going to get more things to review and more stuff to release as our contributor base grows anyway.

We'll also have to keep in mind what happens when we hit a new major version, but I don't think that's an unnavigable problem, nor a particularly pressing one at this juncture.

So then the final loose end is when and how we stop making releases from a minor branch. Here the Apache Kafka project doesn't really have anything written down. But in practice patches do get merged to minor branches which are never publicly released. However, they are available to those motivated to do their own building and testing. That seems like a reasonable balance to me: The project is not obliged to release them, but the community has the ability (and some motivation) to contribute those patches to the project.

I've sort of said this already, but if we're only promising patch releases for the previous two minor versions, then the answer to "when do we stop releasing patches for 1.x-2?" is "once we've released 1.x+1", or at that 6 month expiration date, whichever comes first. Obviously, if we're churning out a new minor release every week then we're going to need to reassess both how many minor versions we're maintaining and how long we're maintaining them for. Though, if we were in a position to be churning out minor releases that frequently I think this whole discussion would be very different.

I think "once we've released 1.x+1" also answers the question of "when do we stop accepting PRs to the release branch for 1.x-2?", because if it's not going in a patch release then it doesn't need backporting IMO. The point of this is to make releases and upgrades of Kroxylicious safer, more stable, and more reliable for the people who use it, so merging backports that aren't going to end up in releases seems a bit like wasted effort. In saying that, there is some reasonable demand for this in the future, and we feel it wouldn't add any unreasonable burden on us to maintain, then we could do so. If we draw a hard line now we can always bend it later if we need to, but I don't think we need to do so pre-emptively.

I'm also wondering if it would be manageable to have things more time based. So have a scheme like:

1. the latest minor version is actively supported
2. when a new minor version is created, on date X ,the previous minor version is moved to a different support level
3. the previous minor now has support dates set based on date X, like 3 months of bugfixing and 6 months of security fixing for example.

This would give Users an easy to interpret table of dates for when a minor goes EOL. And users would know that they are getting at least that many months of support for the latest minor version.

This is what NodeJS does and, to an extent, what Ubuntu does ^{_{something something pro subscriptions and paid support tiers}}.

I don't dislike this model (quite the opposite actually, as a user it's great) but it feels very strict and rigid given where the project is at currently, and I just don't feel that it's the right fit for us right now. I think it also implies a level of "support" (I don't mean to nitpick on language but I really don't like using this word here, I think we should be calling this "maintenance" because that's what we're actually proposing to do) that I don't think we're trying to offer. It is not within our scope as an open source project to be offering —or appearing to offer— anything close to enterprise support, which is why I think we should be talking about "actively maintained" releases rather than "supported" releases.

If we are wanting some kind of tapered maintenance model then we could say something like "version 1.x-1 gets all the backports while version 1.x-2 only gets CVE fixes until we release 1.x+1", but trying to come up with a schedule for this at this stage isn't really feasible because we have no way of knowing how realistic these kinds of timelines are yet. We can guess, but we don't actually know. We aren't Kafka or NodeJS or Ubuntu; our development effort is not their development effort, our maintenance burden is not their maintenance burden, and we can only model ourselves off of them to a point. A system like this, with stricter timelines and more assurance for users, is the kind of thing we could revisit in the near future once we've actually reached 1.0.0 and tried out something a little more flexible, at which point we'll have a better idea of how much extra work we've actually created for ourselves and what kinds of timelines are actually practical.

Anyway, I'm off to bed. If you read to the end of this: here, have a cookie 🍪

[gracegrimwood]

I'm really happy we've started this discussion, I think it's a great move for the project. Thanks @tombentley for putting this together! 🎉

[robobario]

Is there a distinction between the current latest minor and the 2 prior ones here? I'm wondering if somewhere we should say the latest minor will be bugfix supported beyond 6 months.

[hrishabhg]

Should it be more like this? (current version + 2 patches or current version, (current version - 1) patch and (current version - 2) patch.

Suggested change

	We propose to provide patch releases for current `1.x`, plus up to the previous two minor releases. I.e. If the current release is `1.x.y` then we are prepared to release a `1.x.(y+1)`, `1.(x-1).z` and `1.(x-2).z`.
	We propose to provide patch releases for current `1.x`, plus up to the previous two minor releases. I.e. If the current release is `1.x.y` then we are prepared to release a `1.(x-1).(y+1)` and `1.(x-2).(y+2)`.

OR

Suggested change

	We propose to provide patch releases for current `1.x`, plus up to the previous two minor releases. I.e. If the current release is `1.x.y` then we are prepared to release a `1.x.(y+1)`, `1.(x-1).z` and `1.(x-2).z`.
	We propose to provide patch releases for current `1.x`, plus up to the previous two minor releases. I.e. If the current release is `1.x.y` then we are prepared to release a `1.x.(y+1)` and `1.x.(y+2)`.

[tombentley]

Holding off any change here while we work though the conversation above.

[tombentley]

Thanks for the initial round of reviews folks. I've made some of the more minor changes. I think we need to resolve this conversation to make further progress.

[alanrobinson-dwp]

Hi all, for what it's worth from our perspective this sounds like it would work well for us.

Our security posture means we shouldn't be too far away from latest releases. If we were any further out than 3 we'd be getting asked to upgrade regardless.

If we were one or two out and there was a fix release for a security vulnerability it'd be easier getting our current version + security vulnerability or bug into Prod than jumping possibly 2 versions and having to make sure we covered off testing for extra features in order to get that security vulnerability or bug.

[mimaison]

This only covers compatibility between Kroxylicious versions. I couldn't see anywhere compatibility with regards to Kafka. My understand is that Kroxy always tries to adopt the latest Kafka dependencies. Is that true? Should that be documented? If so it's takes the same Kafka compatibility guarantees as Kafka.
So for example if Kafka 5.0 drops support for Kafka < 3.0 (just a random example, this is not based on actual plans), Kroxylicious would also drop support for Kafka < 3.0.