feat: Activity Detection (RPC)

[vesaber]

Summary

• What changed: with the help of @dogbonewish i added activity detection that uses the fluxerapp detectable repo and the discord ipc.
• Why it is correct: I'm not quite sure I understand this but this allows users to show their cool activities if the app supports discord ipc or is registered in the detectable repo
• Risk: If we look back to the CSAM attack that the Fluxer platform had like a couple weeks ago (or a month?) I'm guessing this could be abused by showing porn as your status and other offensive things, including but not limited to: titles, descriptions and buttons.

Verification

• Tests run: I've used and modified some tests that @dogbonewish created. Most of them can be found at fluxer_desktop/src/main/__tests__ in this PR
• Manual checks: I spun up a dev server multiple times and used both the desktop app aswell as browser sessions to test if the activities properly propagate and dont have any issues
• Screenshots or recordings: will add later

Checklist

• I understand every change in this PR.
• I can explain what it does and why it is correct.
• I disclosed any LLM coding help below.

LLM Disclosure

No I did not use LLM in any of my additions but I can confidently say I vibecoded to JPEGMAFIA, Nirvana, Queens of the Stone Age and Jane Remover

[Kamalaja]

Risk: If we look back to the CSAM attack that the Fluxer platform had like a couple weeks ago (or a month?) I'm guessing this could be abused by showing porn as your status and other offensive things, including but not limited to: titles, descriptions and buttons.

Specifically, this implementation only allows for UGC in text form correct, or can users also upload image assets?

[vesaber]

Risk: If we look back to the CSAM attack that the Fluxer platform had like a couple weeks ago (or a month?) I'm guessing this could be abused by showing porn as your status and other offensive things, including but not limited to: titles, descriptions and buttons.

Specifically, this implementation only allows for UGC in text form correct, or can users also upload image assets?

Yes, UGC in text form

[Kamalaja]

@vesaber

Yes, UGC in text form

Would you be open to extending this to show your currently active activity in admin, specifically under the account information setting under "Pronouns". We would then also want a button under the account page under "Clear Profile Fields" for Activity

[vesaber]

@vesaber

Yes, UGC in text form

Would you be open to extending this to show your currently active activity in admin, specifically under the account information setting under "Pronouns". We would then also want a button under the account page under "Clear Profile Fields" for Activity

thumbs up!

[Kamalaja]

Documenting; no action needed - things needed for merge:

1. Admin controls mentioned above
2. Desktop / Mobile implementation
   2a. If one of them is not available, we need a feature flag in admin for us to lock this feature behind.

[vesaber]

Mobile implementation

Does this address the PWA or the Flutter client?

[Kamalaja]

To that end, how does this implementation handle if you have multiple running pieces of software that could be detected?

Also, want to check in, have any privacy controls been implemented? We should make this opt-in instead of opt-out.

These were not defined in the activity detection bounty, so they do not need to be implemented to claim the bounty - but for mass rollout, we should do it. That's on me, apologies.

[vesaber]

It's worth noting that integrating with the Spotify API itself likely will be unfeasible because of how that requires an approved developer account on their platform (which each instance would need credentials for too, on the off chance that they mange to get approved for production usage).

However, what could be done instead is this (which coincidentally will work with a wide variety of other media player applications, not just Spotify):

* Windows: [System Media Transport Controls (SMTC)](https://devblogs.microsoft.com/oldnewthing/20231108-00/?p=108980)

* Linux: [MPRIS over D-Bus](https://app.readthedocs.org/projects/mpris2/downloads/pdf/latest/)

* macOS: [MediaRemote through `mediaremote-adapter`](https://github.com/ungive/mediaremote-adapter)

I did manage to integrate MPRIS in earlier development of this PR but it deemed useless to me and removed it later on while cleaning the code.
Now what if I do integrate MPRIS and the other varieties? Mobile users still won't be able to project their Spotify status on their profile.
A small thought that I had about this integration is that users can just make their own API keys on Spotify and enter them into the Fluxer integration system. With that, every activity they project from Spotify arrives at Fluxer and then Fluxer can proxy whatever arrives so that the API key doesn't get constantly hit with requests.

[vesaber]

To that end, how does this implementation handle if you have multiple running pieces of software that could be detected?

Also, want to check in, have any privacy controls been implemented? We should make this opt-in instead of opt-out.

These were not defined in the activity detection bounty, so they do not need to be implemented to claim the bounty - but for mass rollout, we should do it. That's on me, apologies.

It should hold up fine with multiple pieces of software running in the background, I'll showcase this later on with some screenshots and videos when I'm close to finishing this PR.
As to privacy controls and opt-in & out, I don't think I implemented that. that just came to my mind too.
I'll implement it tomorrow.

[Kamalaja]

I did manage to integrate MPRIS in earlier development of this PR but it deemed useless to me and removed it later on while cleaning the code. Now what if I do integrate MPRIS and the other varieties? Mobile users still won't be able to project their Spotify status on their profile. A small thought that I had about this integration is that users can just make their own API keys on Spotify and enter them into the Fluxer integration system. With that, every activity they project from Spotify arrives at Fluxer and then Fluxer can proxy whatever arrives so that the API key doesn't get constantly hit with requests.

At this time, we don't wish to act as a proxy gateway for selfhosted instances (we want to be air-gapped). So I don't think this structure is viable. We may consider it as part of some follow-up work, but for now, we should go with a system that each instance can manage independently.

[vesaber]

At this time, we don't wish to act as a proxy gateway for selfhosted instances

the self-hosted instances will be their own proxies.

[Kamalaja]

It should hold up fine with multiple pieces of software running in the background, I'll showcase this later on with some screenshots and videos when I'm close to finishing this PR. As to privacy controls and opt-in & out, I don't think I implemented that. that just came to my mind too. I'll implement it tomorrow.

Ok, didn't write a spec originally so here is what is needed for rollout:

1. We need privacy controls, with an opt-in framing - @hampus-fluxer could you share teh new canary format? We should set 3 levels - No one, Friends, Community Members.

2. We need a notification to alert users that this is live. Perhaps a callout attached to the settings icon, or a nagbar at the top of the desktop client?

3. In the settings, we need some language letting users know that if their program is not detected, to go visit the detectables github and submit it there.

[Kamalaja]

At this time, we don't wish to act as a proxy gateway for selfhosted instances

the self-hosted instances will be their own proxies.

Ah, I misunderstood, I don't know if it's worth going that far right now, vs just MPRIs and accepting a mobile deficit. Is there a similar API layer on mobile apps to utilize 🤔

[vesaber]

It should hold up fine with multiple pieces of software running in the background, I'll showcase this later on with some screenshots and videos when I'm close to finishing this PR. As to privacy controls and opt-in & out, I don't think I implemented that. that just came to my mind too. I'll implement it tomorrow.

Ok, didn't write a spec originally so here is what is needed for rollout:

1. We need privacy controls, with an opt-in framing - @hampus-fluxer could you share teh new canary format? We should set 3 levels - No one, Friends, Community Members.

2. We need a notification to alert users that this is live. Perhaps a callout attached to the settings icon, or a nagbar at the top of the desktop client?

3. In the settings, we need some language letting users know that if their program is not detected, to go visit the detectables github and submit it there.

everything sounds fine to me, though I'd like to get small directions for 2.

[vesaber]

At this time, we don't wish to act as a proxy gateway for selfhosted instances

the self-hosted instances will be their own proxies.

Ah, I misunderstood, I don't know if it's worth going that far right now, vs just MPRIs and accepting a mobile deficit. Is there a similar API layer on mobile apps to utilize 🤔

welp, I'm not sure myself.. I'm not a mobile dev at the very least (even though I'm looking forward to learning swift).
But I do know stuff like scrobbling is hard to do on IOS (keeping an app open).

[edeuss]

@Kamalaja @vesaber I think the mobile implementation (for flutter_client) will just be viewing the activities on peoples profiles/status's. Not a good way to do detection on mobile even on a limited sense.

[vesaber]

@Kamalaja @vesaber I think the mobile implementation (for flutter_client) will just be viewing the activities on peoples profiles/status's. Not a good way to do detection on mobile even on a limited sense.

thats totally fine, i imagined that this would be the outcome

[vesaber]

I implemented privacy controls. Which took embarrassingly long. Before I commit this, I would love to know if this looks fine and if the wording would need any changes?

[hampus-fluxer]

I implemented privacy controls. Which took embarrassingly long. Before I commit this, I would love to know if this looks fine and if the wording would need any changes?

looks good, the heading of the radio group could probably be renamed "Who can see your detected activity" for consistency – though I wonder if perhaps having a toggle under it for explicitly enabling and disabling detection entirely would make sense, and making "no one" then act as nobody but yourself being able to see it – then you could test your activity detection out fully locally, and it also becomes clear what to do if you don't want Fluxer to do any detection, not even visible only to yourself, with that split

[probablyjassin]

Looks awesome to me!

[vesaber]

the heading of the radio group could probably be renamed "Who can see your detected activity" for consistency

Right that's a flaw I missed. flip.
About the toggle. I think it does make a lot of sense, and I might just work on it since the other 2 specs that @Kamalaja mentioned aren't as important as what you said. Thanks for the feedback!
I myself noticed a flaw that the buttons are reversed. so I'll also just change that just for consistency.

[Kamalaja]

@vesaber - agree with @hampus-fluxer's design recommendation. @hampus-fluxer if you have insight, would anti-cheat detection be more forgiving of it then? My understanding of most client-side anticheats is that they detect at time of hooking, not by just presence of hooking code.

[Kamalaja]

@vesaber for 2. what direction were you looking for?

[hampus-fluxer]

@vesaber - agree with @hampus-fluxer's design recommendation. @hampus-fluxer if you have insight, would anti-cheat detection be more forgiving of it then? My understanding of most client-side anticheats is that they detect at time of hooking, not by just presence of hooking code.

I don't think any anti-cheat engine will care about an application probing the system for running processes – that's more or less just what any task manager would do – to my knowledge, detecting your activity should not require injecting a DLL into a running application like how fullscreen game capture works when screen sharing, there's nothing violating the integrity of other applications on your system in order to detect what's running 👀

[vesaber]

@vesaber for 2. what direction were you looking for?

none. You already gave me the right direction. Just needed small feedback for refinement.

[Kamalaja]

@vesaber for 2. what direction were you looking for?

none. You already gave me the right direction. Just needed small feedback for refinement.

Sounds good - just let me know if you need anything :)

[vesaber]

[vesaber]

@hampus-fluxer is this fine?

[Kamalaja]

@vesaber The two pictures are meant to convey the toggle on/off state correct?

Some feedback:

I think the line "Who can see your detected activity" should be below the enable toggle and its description. Enabling it is the top-level, and then once it's enabled, the visibility settings follow.

[hampus-fluxer]

I think it might make sense if activity detection as a whole is separated from profile privacy, perhaps, such that it would be:

• heading: Profile privacy
  • "who can see your full profile"
• heading: Activity detection
  • toggle to turn activity detection on or off entirely (when off, the radio group section below it becomes disabled, since it is implicitly disabled as Lilith mentioned)
  • "who can see your detected activity"

[vesaber]

I think it might make sense if activity detection as a whole is separated from profile privacy, perhaps, such that it would be:

• heading: Profile privacy

  • "who can see your full profile"

• heading: Activity detection

  • toggle to turn activity detection on or off entirely (when off, the radio group section below it becomes disabled, since it is implicitly disabled as Lilith mentioned)

  • "who can see your detected activity"

thats totally fine and it definitely sounds better to do so. may i wait for the new sfu to release so i dont have any major issues with merging?

[hampus-fluxer]

sure, you can wait until I land my enormous PR if you'd like

[witchlliee]

I built fluxer with this branch but I still can't get it to detect games or anything. Should I wait?

[vesaber]

I built fluxer with this branch but I still can't get it to detect games or anything. Should I wait?

did u also build the desktop app?

[witchlliee]

I built fluxer with this branch but I still can't get it to detect games or anything. Should I wait?

did u also build the desktop app?

Yes, I guess. I just used a PKGBUILD like this:

Details

pkgname=fluxer-git _pkgname=fluxer pkgver=r285.1f044a4 pkgrel=1 pkgdesc="A free and open source instant messaging and VoIP platform built for friends, groups, and communities." arch=('x86_64' 'aarch64') url="https://fluxer.app/" license=('AGPL-3.0-or-later') options=('!lto') provides=('fluxer') conflicts=('fluxer') _electronver=39 _nodever=20 makedepends=( 'git' "nodejs>=$_nodever" "electron$_electronver" 'npm' 'pnpm' 'python' 'rust' 'clang' 'libfido2' ) source=( "$_pkgname::git+https://github.com/vesaber/fluxer.git#branch=feature/activity-detection" ) sha256sums=( 'SKIP' )

pkgver() {
cd "$_pkgname"
printf "r%s.%s" "$(git rev-list --count HEAD)" "$(git rev-parse --short=7 HEAD)"
}

prepare() {
cd "$_pkgname/fluxer_desktop"
pnpm install --frozen-lockfile
}

build() {
local electron_builder_options=(
--linux
--dir
-c.electronDist=/usr/lib/electron$_electronver
)
cd "$_pkgname/fluxer_desktop"
export NODE_ENV=production
pnpm build
pnpm exec electron-builder --config electron-builder.config.cjs ${electron_builder_options[@]}
}

package() {
cd "$_pkgname/fluxer_desktop"
sed -i 's|Exec=app.fluxer.Fluxer|Exec=/usr/lib/fluxer/fluxer|' packaging/linux/app.fluxer.Fluxer.desktop
install -vDm 644 packaging/linux/app.fluxer.Fluxer.desktop -t "$pkgdir/usr/share/applications/"
install -vDm 644 packaging/linux/app.fluxer.Fluxer.svg "$pkgdir/usr/share/icons/hicolor/scalable/apps/app.fluxer.Fluxer.svg"
install -vdm 755 "$pkgdir/usr/lib/$_pkgname/"
cp -rv dist-electron/linux-unpacked/* "$pkgdir/usr/lib/$_pkgname/"
}