gds: init louvain #5155

sdht0 · 2025-03-28T23:13:48Z

Implements basic parallel Louvain.

References:

https://en.wikipedia.org/wiki/Louvain_method
Parallel Heuristics for Scalable Community Detection. Hao Lu, Mahantesh Halappanavar, and Ananth Kalyanaraman: https://arxiv.org/abs/1410.1237

codecov · 2025-04-10T14:52:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.57%. Comparing base (e1239d6) to head (d2933f4).
Report is 11 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5155      +/-   ##
==========================================
- Coverage   86.58%   86.57%   -0.01%     
==========================================
  Files        1410     1410              
  Lines       61973    61978       +5     
  Branches     7606     7607       +1     
==========================================
+ Hits        53657    53660       +3     
- Misses       8141     8143       +2     
  Partials      175      175

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sdht0 · 2025-04-10T20:12:48Z

Cannot reproduce the apple clang error locally. Perhaps failing because of an older version in the builder.

test/test_files/function/gds/louvain.test

sdht0 · 2025-04-10T20:28:48Z

src/function/gds/louvain.cpp

+    return nextCommId;
+}
+
+void aggregateCommunities(offset_t newCommCount, PhaseState& state) {


Not parallelized yet. Might be a bottleneck on large graphs, although communities quickly get smaller in each phase.

TODO: check performance on large graphs

src/include/function/gds/compute_mem.h

src/include/graph/graph_mem.h

src/include/function/gds/gds_utils_mem.h

src/function/gds/louvain.cpp

semihsalihoglu-uw · 2025-04-11T16:28:32Z

src/function/gds/louvain.cpp

+    PhaseState& state;
+};
+
+class WriteResultsVC : public GDSResultVertexCompute {


Is it possible to refactor and re-use some other ResultsWriter? What is needed so that we do not keep writing our own results writers.

I don't see this addressed.

Will work on this in a future PR.

semihsalihoglu-uw

I'm approving in case you want to merge but again we need to do the inmemgraph differently and I don't think it should be a lot of work. So you might do it as part of this PR. But if you do so, I want to see again before you merge.

src/include/function/gds/gds_object_manager.h

src/include/graph/graph_mem.h

src/function/gds/louvain.cpp

semihsalihoglu-uw · 2025-04-18T04:39:31Z

src/function/gds/louvain.cpp

+    PhaseState& state;
+};
+
+class WriteResultsVC : public GDSResultVertexCompute {


I don't see this addressed.

src/include/graph/graph_mem.h

src/include/function/gds/gds_object_manager.h

andyfengHKU

I didn't go into details of the algorithm implementation. Let's test an its performance on various dataset and see.

I can already tell supporting multi labeled graph will not be trivial. So let's not do it. Supporting filtered graph is possible. I'll need to double check a few cases with InMemGraph.

@ray6080 check the allocation for gds_object_manager.

andyfengHKU · 2025-04-29T16:20:08Z

src/include/graph/graph_mem.h

+namespace kuzu {
+namespace graph {
+
+using weight_t = common::offset_t;


I would keep this inside InMemGraph or Neighbor because not all algorithm's weight_t is integer. Some (weighted shortest path) could be arbitrary numerical value.

Will refactor once this is needed. Might have to switch InMemGraph to a template for the weight type.

src/include/graph/graph_mem.h

src/graph/graph_mem.cpp

src/include/function/gds/gds_object_manager.h

extension/gds/src/function/louvain.cpp

andyfengHKU · 2025-04-29T16:53:34Z

extension/gds/src/function/louvain.cpp

+struct PhaseState {
+    InMemGraph graph;
+    AtomicObjectArray<weight_t> nodeWeightedDegrees;
+    ObjectArray<CommInfo> currCommInfos;


In the future it would be good to abstract

class CommInfos { private ObjectArray<CommInfo> ...; } class Communities { private AtomicObjectArray<offset_t> ... }

And expose more meaningful interface like Communities::getCommunity(offset) instead of state.currComm.get(offset)

Leaving to a future PR.

src/include/function/gds/gds_object_manager.h

Makefile

src/include/common/assert.h

src/include/common/task_system/task.h

src/include/storage/buffer_manager/memory_manager.h

extension/gds/src/include/main/graph_mem.h

src/include/storage/buffer_manager/memory_manager.h

src/include/function/gds/gds_object_manager.h

src/include/storage/buffer_manager/memory_manager.h

Implements basic parallel Louvain References: - https://en.wikipedia.org/wiki/Louvain_method - Parallel Heuristics for Scalable Community Detection. Hao Lu, Mahantesh Halappanavar, and Ananth Kalyanaraman: https://arxiv.org/abs/1410.1237

This comment was marked as outdated.

Sign in to view

sdht0 force-pushed the gds-louvain branch from caf5340 to 577c12c Compare April 1, 2025 07:09

This comment was marked as outdated.

Sign in to view

sdht0 force-pushed the gds-louvain branch from 962fdb9 to a89e1b7 Compare April 9, 2025 14:23

This comment was marked as outdated.

Sign in to view

sdht0 force-pushed the gds-louvain branch 2 times, most recently from e454fcb to 39edafc Compare April 9, 2025 15:12

This comment was marked as outdated.

Sign in to view

sdht0 marked this pull request as ready for review April 10, 2025 14:24

This comment was marked as outdated.

Sign in to view

sdht0 requested review from andyfengHKU and semihsalihoglu-uw April 10, 2025 20:09

sdht0 commented Apr 10, 2025

View reviewed changes

test/test_files/function/gds/louvain.test Outdated Show resolved Hide resolved

sdht0 commented Apr 10, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

semihsalihoglu-uw requested changes Apr 11, 2025

View reviewed changes

sdht0 force-pushed the gds-louvain branch 2 times, most recently from bd6e7e3 to eb43f48 Compare April 17, 2025 04:04

This comment was marked as outdated.

Sign in to view

sdht0 force-pushed the gds-louvain branch from 7090569 to 3662f3f Compare April 17, 2025 05:24

This comment was marked as outdated.

Sign in to view

semihsalihoglu-uw approved these changes Apr 18, 2025

View reviewed changes

sdht0 force-pushed the gds-louvain branch 2 times, most recently from 2d15eb9 to 0e4a06d Compare April 18, 2025 13:58

andyfengHKU reviewed Apr 21, 2025

View reviewed changes

src/include/function/gds/gds_object_manager.h Outdated Show resolved Hide resolved

Init parallel louvain

04f6dab

sdht0 force-pushed the gds-louvain branch from a261586 to 04f6dab Compare April 29, 2025 11:22

Run clang-format

7cf8812

This comment was marked as outdated.

Sign in to view

Remove const

06b549e

This comment was marked as outdated.

Sign in to view

andyfengHKU approved these changes Apr 29, 2025

View reviewed changes

sdht0 added 2 commits April 30, 2025 10:56

Init mm-based allocator

5c09e68

Make Louvain work as extension

4d64891

sdht0 requested a review from benjaminwinger as a code owner April 30, 2025 15:06

sdht0 commented Apr 30, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

ray6080 reviewed Apr 30, 2025

View reviewed changes

Address comments

7010ace

sdht0 force-pushed the gds-louvain branch from 5c67faf to 7010ace Compare April 30, 2025 18:14

This comment was marked as outdated.

Sign in to view

sdht0 added 2 commits April 30, 2025 15:55

Fix CI

67d497a

Fmt

d3c3984

This comment was marked as outdated.

Sign in to view

tidy

7ff18e7

sdht0 force-pushed the gds-louvain branch from 096a0fe to 7ff18e7 Compare April 30, 2025 21:04

This comment was marked as outdated.

Sign in to view

Fix

d2933f4

This comment was marked as outdated.

Sign in to view

sdht0 merged commit d53c8b2 into master Apr 30, 2025
28 checks passed

sdht0 deleted the gds-louvain branch April 30, 2025 23:58

gds: init louvain #5155

gds: init louvain #5155

Uh oh!

Conversation

sdht0 commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

codecov bot commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

sdht0 commented Apr 10, 2025

Uh oh!

Uh oh!

sdht0 Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

semihsalihoglu-uw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

andyfengHKU left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdht0 commented Mar 28, 2025 •

edited

Loading

codecov bot commented Apr 10, 2025 •

edited

Loading

sdht0 Apr 10, 2025 •

edited

Loading