Code

Optimize skew partition#3105

Code

Closed

turboFei wants to merge 47 commits intoapache:mainapache/celeborn:mainfrom turboFei:optimize-skew-partitionturboFei/incubator-celeborn:optimize-skew-partitionCopy head branch name to clipboard

+2,571-133

Commits

Commits on Jan 19, 2025

• [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files

  wangshengjie123

  committed
  2570b50

  Copy full SHA for 2570b50

• fix unit test

  wangshengjie123

  committed
  43416c3

  Copy full SHA for 43416c3

• fix ut and refactor code

  wangshengjie123

  committed
  355f6b3

  Copy full SHA for 355f6b3

• refactor code and imports

  wangshengjie123

  committed
  2b3ad58

  Copy full SHA for 2b3ad58

• fix NPE and remove unused code and fix scala 2.11 compile error

  wangshengjie123

  committed
  1a0462d

  Copy full SHA for 1a0462d

• format code

  wangshengjie123

  committed
  5d1bcb6

  Copy full SHA for 5d1bcb6

• add spark 3.3 patch

  wangshengjie123

  committed
  2c4b03f

  Copy full SHA for 2c4b03f

• remove unused log and refactor code

  wangshengjie123

  committed
  ef2dacb

  Copy full SHA for ef2dacb

• add unit tests and refactor code

  wangshengjie123

  committed
  b4dae1d

  Copy full SHA for b4dae1d

• split the skewed partition based on the chunk range

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  52fdb96

  Copy full SHA for 52fdb96

• remove CelebornInputStreamSuiteJ

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  deb2913

  Copy full SHA for deb2913

• refactor code for batch open stream

  wangshengjie123

  committed
  f7abfc2

  Copy full SHA for f7abfc2

• Revert "refactor code for batch open stream"

  Show description for 2e3be7d
  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  2e3be7d

  Copy full SHA for 2e3be7d

• celeborn.client.dataPushFailure.tracking.enabled -> celeborn.client.adaptive.optimizeSkewedPartitionRead.enabled

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  cf8dcb2

  Copy full SHA for cf8dcb2

• refactor code for batch open stream

  wangshengjie123

  committed
  6ce0e11

  Copy full SHA for 6ce0e11

• add ut and refactor split logic

  wangshengjie123

  committed
  9b7cc7f

  Copy full SHA for 9b7cc7f

• refactor code according to review suggestions

  wangshengjie123

  committed
  8ac7172

  Copy full SHA for 8ac7172

• add sbt dependency

  wangshengjie123

  committed
  c169aa2

  Copy full SHA for c169aa2

• update spark patch

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  4b9fab6

  Copy full SHA for 4b9fab6

• address comment

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  2e3a1c1

  Copy full SHA for 2e3a1c1

• fix

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  1794fd4

  Copy full SHA for 1794fd4

• add ut

  wangshengjie123

  committed
  e7d8d73

  Copy full SHA for e7d8d73

• update

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  14d5f10

  Copy full SHA for 14d5f10

• refactor

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  85f9445

  Copy full SHA for 85f9445

• add CelebornPartitionUtil.java

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  697b468

  Copy full SHA for 697b468

• fix ut for jdk 8

  

  cfmcgrady

  authored and

  wangshengjie123

  committed
  2b6349d

  Copy full SHA for 2b6349d

• rebase main and fix npe

  wangshengjie123

  committed
  8155619

  Copy full SHA for 8155619

• fix ut

  wangshengjie123

  committed
  aa30ff6

  Copy full SHA for aa30ff6

• fix npe when memory storage enabled

  wangshengjie123

  committed
  6b4bb40

  Copy full SHA for 6b4bb40

• fix code style check error

  wangshengjie123

  committed
  59bef45

  Copy full SHA for 59bef45

• fix comile error

  wangshengjie123

  committed
  031fdb3

  Copy full SHA for 031fdb3

• add spark patch

  wangshengjie123

  committed
  3966f49

  Copy full SHA for 3966f49

• fix ut compile error and add license to spark patch

  wangshengjie123

  committed
  8c17522

  Copy full SHA for 8c17522

• update spark patch to abort stage when rerun skew join stage

  wangshengjie123

  committed
  21f14d9

  Copy full SHA for 21f14d9

• address comment, avoid reading replicate peer when read skew partition without mapRange

  wangshengjie123

  committed
  e9efddc

  Copy full SHA for e9efddc

• update ClientUtils

  wangshengjie123

  committed
  23ca472

  Copy full SHA for 23ca472

• address review comment

  wangshengjie123

  committed
  94cb56d

  Copy full SHA for 94cb56d

• remove duplicate uts

  wangshengjie123

  committed
  60422d1

  Copy full SHA for 60422d1

Commits on Jan 23, 2025

• add spark patch

  wangshengjie123

  committed
  4130227

  Copy full SHA for 4130227

Commits on Jan 24, 2025

• Merge branch 'main' into optimize-skew-partition

  wangshengjie123

  committed
  e9dcfd6

  Copy full SHA for e9dcfd6

• fix codestyle

  wangshengjie123

  committed
  1c58362

  Copy full SHA for 1c58362

• fix uts

  wangshengjie123

  committed
  cbceadf

  Copy full SHA for cbceadf

Commits on Feb 13, 2025

• add method javadoc comment

  wangshengjie123

  committed
  21a0635

  Copy full SHA for 21a0635

• Merge branch 'main' into optimize-skew-partition

  wangshengjie123

  committed
  4307f0c

  Copy full SHA for 4307f0c

Commits on Feb 17, 2025

• Update common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

  Show description for a9a4af1
  

  RexXiong

  and

  pan3793

  authored
  a9a4af1

  Copy full SHA for a9a4af1

• Update docs/configuration/client.md

  RexXiong

  authored
  cbad6f8

  Copy full SHA for cbad6f8

• Merge branch 'main' into optimize-skew-partition

  turboFei

  authored
  400c9be

  Copy full SHA for 400c9be