Problem: C4Social only addresses a sub-process in the development method

[aschrijver]

What C4Social now defines to me seem like a sub-process of the development method: the more technical-oriented flow of how maintainers and contributors can ensure a larger role for community involvement.

Yesterday I looked at a bunch of projects that use C4 in some form or other, and they had some things in common: the project deliverables (i.e. the product) were all quite technical components, and as such the target audience were mostly Developers who - when reading through a list of Problems - would be able to more or less accurately weigh and understand the ins and outs, pros and cons, and impact of validating a problem and adding an implementation to address them.

C4 to me seems tailored to this type of project + audience.

The weight of the process is in the first 2 steps now: Problem definition and validity check. This is not just technical, but there's a world of product development considerations involved here, especially for products where the UX/UI and evolution thereof are an important part of project success.

These first 2 steps encompass requirements elicitation and product design, where the interests of a range of stakeholders that are part of the community need to be considered. Doing this in an issue that is later closed may not cut it (though a 'Definition of Done' may ensure that user and design docs etc. are updated).

Having different stakeholders with their own needs, plus the requirement for the process to be 'Social' to me indicates that these two steps are more involved: there's a larger process to them. Some examples:

• Say in the community there's overwhelming consensus that a certain Problem, a feature request, is valid. Members involved are mostly Fedizens with diverse backgrounds. Great! They are 'the customer', so they expressed their need. But now a single "other stakeholder" steps in, adding from their perspective:
  • Developer: "This violates our KISS and YAGNI, and need to keep stuff easy-to-use for other devs"
  • Developer: "This would complicate our architecture and codebase, would need overhaul and redesign"
  • Designer: "This clutters the UI, does not fit UX and product principles. Product will lose direction and focus"
  • QA: "This will deteriorate the NFR's we set to adhere to. This will be too high maintenance burden"
  • etcetera ..

These are a bit contrived examples, but who wins? You might argue that in the 2-step loop a consensus is reached, but it is quite involved and many of these considerations may just not be understood by a non-technical audience. There might be a need for rules where one stakeholders arguments can overrule a majority.

Other example of more intricate process is where a Problem is too large in scope (an Epic really, not just a User Story) and needs to be broken down, maybe even into 10 or more sub-Problems. Now you have dependencies on your hands that need to be managed. You'll get a backlog that needs to be prioritized. And while you are implementing them, new and independent Problems may become related and there's even more dependencies to consider (e.g. to avoid considerable tech debt).

I am not advocating to write complex sub-flows to the process. Maybe capturing these things in some clear rules and guidance is all that is needed. All I'm saying is that the single diamond decision-making step is now the place where most "Social community engagement" takes place.

---

Update: 2 more concrete examples.. imagine that both Signal and Keybase were both C4Social projects.

• Would BDFL decisions be overruled? In Signal in order to "protect NFR's" the backend is centralized, and no third-party clients are allowed to interact in the core network.
• Would scope creep and project direction be fully organic? Keybase went from verified identity provider to DM + filesharing social network. It caused a shift in audience that was interested in it.
• Could the project be hijacked by a high influx of certain stakeholders to the community? Both Signal and Keybase are peddling with cryptocurrencies to the horror of many (majority?) in their user base. What if an active community of cryptofans embrace your project for adoption, and immediately outnumbered the existing active members in your community

[aschrijver]

Elaborating a bit further.. So the C4Social is currently more or less like this:

1. problem --> 2. validation --> 3. implementation --> 4. merge

For 3) you want to make it contributors as easy as possible, and entice them to pick any problem that matches their interest. For 4) you facilitate both contributors and maintainers. The contributor wants the satisfaction to see their code quickly integrated, and the maintainer wants to be able to do this with minimum workload. These steps are great, requirements clear.

Imagine in a project this process works exceedingly well. In quick succession well-scoped validated problems are implemented + merged. Project evolves at blazing speed. Then suddenly these Problems are filed:

• "Product lights up like a Christmas tree with all unrelated features we added. End-users no longer understand, need support"
• "Formerly unrelated features start to get a life of their own, and are branching every more away from the original scope"

Their may be clear community consensus on this, so the problems are valid. However, the solutions are not so obvious now, because scope creep cannot easily be undone. E.g. a solution may be to drop be certain features:

• "Can't do, I use feature A, cut feature B instead!" reacted to with "Now way, feature B is essential", etcetera
• "Making this change will break backwards-compatibility" or "will break 3rd-party integrations", etcetera

In many cases there's no way back. To mitigate somewhat the process might be altered to:

1. problem --> 2. validation --> 3. design --> 4. implement --> 5. merge

Or, similar, but more explicit:

1. problem --> 2. validation --> 3. design --> 4. validation --> 5. implement --> 6. merge

In this process:

• In 1) and 2) there's full community engagement
• In 3) and 4) opinion of certain stakeholders has higher weight, might overrule
• And 5) and 6) are just 'doing the work'
• There can be feedback loops to any prior step

[ledcoyote]

What you wrote here, and what some folks in the diaspora dev community wrote has given me a lot to think about here. I have a few thoughts to list here.

First, relating to @aschrijver's post above, there does seem to be a need for organic leadership to arise within the c4 model for projects as complex as social platforms like Mastodon, etc. This could allow focal points for considerations such as UX, security, etc. And they could surely be chosen and removed by a consensus process.

Second, to the point about chunks of work being of epic (i.e. multi issue) scope, this is going to be a likely scenario for this kind of software. There needs to be some attention put in by maintainers to not allow other commits to disrupt an epic that is in progress. Not sure exactly what that looks like. Maybe the epic contributors should be looped in to review changes to code that is in active development by the epic.

Third, I do think we should stop merging upstream PRs. There is a bit of surprise factor for an author if they suddenly see their commits in main on another fork, which we should probably avoid. I think that if a PR looks stalled, but is otherwise a valid change we could invite the author to write an issue and PR into a c4 fork. That might be a bit extra overhead, but perhaps within the purvue of a maintainer's duties.

But anyways, this is some really good discussion, and I look forward to continuing it 👋

[weex]

Terminology

C4 is the protocol we're discussing and C4Social is this community that is creating magic through evolution of the Fediverse. In other words, C4Social is applying C4 but is distinct and will be evolving to suit this community's needs.

BDFL is benevolent dictator for life, but @aschrijver what are the NFR and DM you mention?

Getting to the points:

1. There is an argument to be made that C4 is only for technical projects. That use-cases involving in-depth domain knowledge or user experience, would be stream-rolled by hastily merged code. I can see this being where a contributor throws the PR over the wall in an environment where nobody's watching, i.e. too small a community. We should definitely highlight the increased responsibility and staking of reputation by the developer. Reputation is implicitly staked with the standard ad hoc process as well, but here it's made explicit. Merge+revert to record recklessness is part of the protocol.
   Tough talk aside, I see C4Social following an evolutionary community process where contrary evidence should generate Problems.

2. Several of your concerns are around what the community pays attention to in terms of goals, direction, design, architecture and epics. I think a major challenge is going to be building consensus around the idea that, those aren't very helpful for a community, especially if they detract from shared purpose.
   Hintjens made a point that when gets an architectural idea, how throws it away. "It's trash," he said, which gave me a chuckle. This is an approach I've had here as well. Trying like hell not to design anything, but to curate this growing list of problems, making solutions when and where I can. It feels great to forget about the grandiose design I worked on before finding C4.

but who wins? You might argue that in the 2-step loop a consensus is reached, but it is quite involved and many of these considerations may just not be understood by a non-technical audience. There might be a need for rules where one stakeholders arguments can overrule a majority.

3. Cases of stakeholder conflict should be resolved with discussion, options/plugins, and forks/competition in that order.
   Side thought: More forking might be the thing that finally incentivizes modularity and don't-repeat-yourself in FFLOSS or FLOF :) (that extra F being for federated) What if forking creates a green field where what emerges is the software equivalent of a Cambrian radiation? 🧘

where a Problem is too large in scope

4. That's a good point and they should be broken down, with a goal of normalizing the sub-problems so they are independent. APIs and some rules around API stability, deprecation, testing are important for this. I'd encourage the meta-problem to stay open and unlinked or moved to a higher-level issue tracker. Maybe the problem is one which another software project solves?
   Side thought: I've focused on responding to your points here, but this thread is a great example of a pretty high level problem and if broken down my proposed solutions would be to use corresponding responses as source material for a set of FAQs

5. To scope creep and project direction, all development under C4 is intentionally toward solving problems so any oozing will be toward value. There is no project direction except that which emerges from shared purpose (see point 2 above) in the community. Differences of opinion should be encouraged to be talked out, but if unresolvable settings, plugins, addons, maintained patches, and finally forks are options.

(to be cont'd)

[weex]

Third, I do think we should stop merging upstream PRs.

@ledcoyote Do you mind creating a discussion topic for this or a separate problem statement? We had some discussion also in a c4social/diaspora issue IIRC.

[weex]

Continuing to @aschrijver's second post...

"Product lights up like a Christmas tree with all unrelated features we added. End-users no longer understand, need support"

6. This is a Problem, and I suspect a healthy community will not see them suddenly, but let's say a project had a flag day where on that date a bunch of new stuff was turned on globally, which is plausible.
   I think the Problems filed that day would be highly instructive and great opportunity for community growth in the anti-fragile sense. Flag days would come into question. End-user testing might become a higher priority for the community. Any number of solutions might be proposed including changing C4 or C4' itself. Solutions that affect a core process would in my view deserve the highest level of community discussion around accuracy, value, proof and evidence.

7. I address basic scope creep in 5. above, but you raise a good point about unwanted features which I think can mostly be optional, but I would like to point out a bias toward removing unused code. I see unused code as a unique problem. It is unnecessary complexity and a maintenance cost, so CI should definitely alert on unexercised code paths. I feel like there's more to say on this, but it's not coming to me.

8. Looking at your modified processes with one or two additional steps, reminds me that contributors should be encouraged to work how they want, whether alone or in groups. Smaller patches are better but those extra steps can certainly fit into the implementation box. Draft PRs are an excellent way to get community feedback on any patch but are not required by the protocol.

[aschrijver]

Sorry, should not just drop acronyms. DM is Direct Messaging and NFR is Non-Functional Requirement.

I just read the long response on the Diaspora forum, and it has a number of direct matches to this issue, most importantly the type of project for which C4 may be best-suited. And a whole bunch of additional and very concrete examples where applying C4 principles would be problematic for other project types and the complex development workflows that have emerged for them over time.

As you say this Problem is a top-level one that should be broken down into constituent parts, and I advice to start with the Diaspora feedback and thoroughly distill it, because there's so much to address there. On the whole I agree with most of the points there (though they are somewhat harshly formulated to my taste).

Where I mention Design you should read "all the other stuff that needs attention other than coding + merging". I find your attention is too heavily focused on developer experience at the end of the pipeline, rather than community processes and other stakeholder needs. In no way do I advocate for "grandiose design" in a waterfall kinda process. Adding my design step to the process is just conceptual. To make it work a lot of elaboration should happen, so that comprehensive guidance is available (and most likely more steps to the process).

Several of your concerns are around what the community pays attention to in terms of goals, direction, design, architecture and epics. I think a major challenge is going to be building consensus around the idea that, those aren't very helpful for a community, especially if they detract from shared purpose.

Want to address this briefly, as it is also mentioned in the Diaspora comment. A rather large and active community is an objective and in that many people are end-users, non-technical. You want to keep them away from all the technical discussion. But in C4 this is intricately tied to the Problem issue they filed, which must be elaborated enough for a developer to start coding on it. Diaspora separated these technical / non-technical concerns with a Discourse forum, and I think that is a good approach.

Re:Design. I mentioned 'Definition of Done'. Once you close a Problem you have an issue and PR with potentially a ton of discussion in it. Most of that may now be irrelevant, but some decisions that were made along the way are crucial. These must be extracted, summarized and somewhere documented (e.g. in an Architecture Notebook, or a UX Styleguide, etc).

I see unused code as a unique problem. It is unnecessary complexity and a maintenance cost, so CI should definitely alert on unexercised code paths.

This is different than what I meant to say. Yours is a technical concern. I refer to returning to a more comprehensive feature set, and removing features that are used by some users but are not core to what the product stands for. Ideally you would never come in this place, but making it as-easy-as-possible to merge PR's can land you there.

Smaller patches are better but those extra steps can certainly fit into the implementation box.

You could squash the whole process in a single 'implement' step, but leaving steps out either makes the process messier, or you get the additional requirement to document how one should 'behave' in the step. The processes are there, whether you explicitly define them or not. And the same holds for design. Without explicit design steps your codebase still reflects a design (though probably one that looks more like an ants nest, than a modern skyscraper 😜 )

Draft PRs are an excellent way to get community feedback on any patch but are not required by the protocol.

This makes me wonder once more who you see as the community? A PR deals with code. You might get feedback from coders, and that's about it, I guess.

[aschrijver]

@ledcoyote wrote:

First, relating to @aschrijver's post above, there does seem to be a need for organic leadership to arise within the c4 model for projects as complex as social platforms like Mastodon, etc. This could allow focal points for considerations such as UX, security, etc. And they could surely be chosen and removed by a consensus process.

Yes, fully agree. In my still simplified process extension there's a handover from non-technical to the more technical design phase. In that phase there can be still community involvement (by all stakeholders), but some voices now carry more weight.

Terminology

C4 is the protocol we're discussing and C4Social is this community that is creating magic through evolution of the Fediverse. In other words, C4Social is applying C4 but is distinct and will be evolving to suit this community's needs.

I think C4 just doesn't cut it, and that what you are trying to add to it is more than an "addendum to C4". I would propose that you define "C4Social" as a separate process, a fork of C4. And FFLOSS communities based on it are C4Social communities. As communities evolve they contribute their lessons-learned to the C4Social process, helping it mature.

One more remark on applicability of C4 as it currently stands: IMHO right now it can't tackle complex existing projects such as Diaspora or Mastodon. Trying to do so sets yourself a huge task, and enormous workload. As I mentioned above, FLOSS development processes emerge over time, based on what works and what not.

My advice would be not to try and take on the role of community manager AND allround maintainer of 2 fullblown mature FOSS projects AND the design lead of a new development method / protocol.

Where C4Social may work is in combination with a brand new federation project. At the start of such project you have mostly a developer audience. Things are still simple and easy at the first beginnings. C4Social may be great to progress rapidly towards implementation of a MVP (Minimum Viable Product).

You might say that using C4Social at the start of a project is overkill and just amounts to extra work, and you are probably right. But maturing C4Social is part of the objectives, so in this case it is ideal to explore both project/code + process at the same time. Along the way you can figure things out, such community-building and how that relates to the process.

[weex]

@aschrijver wrote:

start with the Diaspora feedback and thoroughly distill it, because there's so much to address there

When that post went up, I examined it closely. Today I responded with links to all of the Issues it generated. I agree about the tone but knew that the contents were valuable and would be tracked.

attention is too heavily focused on developer experience at the end of the pipeline, rather than community processes and other stakeholder needs

Tunnel vision on developer experience was apparent when I visited the various sites and repos pertaining to C4S with other roles in mind. There wasn't much for an end-user or podmin or anyone besides myself so created some issues around that after identifying a bunch of problems. I'd like to share a list but they're basically with the readmes and the way the repos and org are presented.

A PR deals with code. You might get feedback from coders, and that's about it, I guess.

I don't want to make many rules very early but the way I envision it, there'd be threads everywhere they need to be to support discussion around a PR. It's also not lost on me that the software we're working on builds such spaces as well. It's too much to worry about so I think we're just trying to keep it simple with the kernel that helps code and community operate together. Maybe C4's is like RISC?

IMHO right now it can't tackle complex existing projects such as Diaspora or Mastodon. Trying to do so sets yourself a huge task, and enormous workload.

This is what excites me personally and I'm keen to figure out how to explain it to others. I've coded a lot in the past and built things and I've experienced burnout when the why wasn't there. Never before has there been IMHO as comprehensive an approach for tackling the hardest problems. (I've created like 4 issue trackers outside this project with good results)

That being said, the tech can be tricky and building a community is to me an almost total mystery but maintaining (even two massive projects) under C4 is pretty easy.

[weex]

Had another thought about this issue today because you're right in that C4 isn't comprehensive. If C4 is a kernel, then what is userspace? Is that a place where more extensive design system calls are made into that implementation diamond?

Re-focusing on this problem definition, I agree that it's accurate but am less sure of the value. Putting it another way, if we're not sure the C4 hypothesis will work but haven't conducted a trial (for say 6-12 months) then what is the value in pursuing another hypothesis?

[aschrijver]

I agree a trial is in order. But what to trial?

1. A greenfield project that evolves along with its process.
2. Existing large-scale established project that you try to whip into shape.

Note that while we talk about Process, it is actually about the combination of both culture + process, and in community-driven project culture is at least as important.

I believe that choosing 2) you either risk doing no justice to that, or you'll make concessions to C4Social just to make it fit for the case at hand. I'd personally opt for 1).

[weex]

C4-tools began out of the need to make PRs more easily and I see that as a greenfield project but I'm still not sure how to address the wider culture question. If you can put it in problem form, I'd encourage you to make issues in c4-tools otherwise I'd like to take this to https://github.com/c4social/c4social.org/discussions. If it's to be a meta-protocol then I'd choose a completely different name than C4. It'd be more our version of 0MQ's Community Guide.

To give you an example of a problem I find interesting and in C4-tools scope: I was looking at Hometown but since my personal GitHub account already has a fork of Mastodon, I couldn't do anything in a linked and sensible way there. GitHub doesn't make it very easy for a developer to contribute to more than one fork. I'll report that to them but it does give me a reason to search for a better PLATFORM.

[aschrijver]

C4-tools began out of the need to make PRs more easily

That is a very limited scope. If that is indeed the desired scope, then this issue might be closed. C4 will then be some building block that can be 'dropped in' to a larger process in any one project, but you don't have to consider that larger process (which will differ from project to project) to be in scope.

Question is if such scope is realistic. It may be. Work then becomes focussed on how integration can be as seamless as possible. Trying to 'drop in' in a complex project such as Diaspora is then a way to find where friction occurs, and improve accordingly.

I still think, if this were the approach, it is a tough road ahead and you should repeat the drop-in process in multiple different project types / organizations / cultures to hone the building block to something that is intuitive enough.

But I had the notion that you were aiming - just like the original C4 - to offer an end-to-end process, but with the USP's (Unique Selling Points) that it was tailored to support Social + Federated apps, plus the communities involved in creating them. I.e. a holistic vision.

I see that as a greenfield project but I'm still not sure how to address the wider culture question.

Finding the new, improved C4 process (which I prefer to call "C4Social" process) can be seen as a greenfield process. But the accompanying federated projects that are your 'helpers' to evolve the process are anything but greenfield. In my proposal for a trial both the process + the app would be greenfield. That would ease maturing and evolving both side-by-side according to objectives and requirements you set out to achieve. It would provide more control.

The contributors to the app would gradually familiarize with the process and be active helpers to improve it. In the other trial option, evolving based on mature app projects, you'd have 2 separate audiences: One interested in the app, and another more process-oriented crowd interested in evolving the process.

how to address the wider culture question.

Culture is a strange concept. It emerges over time. There's only so much you can do to "actively build a culture". You can depict / envision how you'd like you ideal culture to be. Based on that you can create a Manifesto, describe a set of values. Then while building procedures and setting rules continuously reflect how it would impact culture. And then - after introducing them - 'measure' afterwards based on actual dynamics you observe.

This also is much easier with a fully greenfield approach. Dropping C4 into a mature project will immediately lead to big disruption, and more of a 'crisis management' to get all the wrinkles figured out.

It'd be more our version of 0MQ's Community Guide.

As implied above I see Process + Community evolving alongside each other. So these are not independent artifacts to deliver (if your scope is holistic).

To give you an example of a problem I find interesting and in C4-tools scope

This example highlights another requirement to consider carefully: What is the importance of facilitating forks as well as possible in C4Social? A lot of complexity in maturing C4 can arise from the position you take here.

1. C4Social is particularly tailored to make forking of existing projects (e.g. those with lotsa 'stuck' PR's) easy, to get a fork that 'sprints forwards' with ease.
2. C4Social makes it particularly easy to be forked, in such way that its forks can easily contribute their changes upstream.
3. C4Social must ensure that both 1) and 2) are supported really well.
4. C4Social places no special attention to having an USP in ease-of-forking.

Option 4) means that forking is done like in any FOSS project, where the fork maintainers make their own decisions afterwards.

I would opt for option 4, because that aligns best with a MVP (Minimum Viable Product), and leaves options 1) and 2) to be addressed further up the roadmap if there's a need.

This means in the case of forking Diaspora (if forking mature project is in scope), the fork would be limited to getting a copy of the codebase. All the process + cultural aspects of the original project would be 'ripped out' and refactored to a native C4Social implementation. With that done the fork starts to be operational, and can be considered more or less greenfield. You'd still be able to do upstream PR's (and adhere to the particular process + rules in the upstream project).

[aschrijver]

The subject of this issue came up in chat recently and led me to write a Proposal to be part of Social Coding Movement.