Code

Parallel sampling eviction#157

Code

Merged

sunggg merged 42 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom masahi:parallel-sampling-evictionmasahi/mlc-llm:parallel-sampling-evictionCopy head branch name to clipboardFeb 2, 2024

+498-185

Commits

Commits on Jan 11, 2024

• add new model for evaluating logits over multiple queries using KV cache

  masahi

  committed
  e0ef4c6

  Copy full SHA for e0ef4c6

• add test

  masahi

  committed
  4ccbb27

  Copy full SHA for 4ccbb27

• clean

  masahi

  committed
  f1314a5

  Copy full SHA for f1314a5

• Only the number of past tokens is needed

  masahi

  committed
  2bee022

  Copy full SHA for 2bee022

• fix build

  masahi

  committed
  756b09f

  Copy full SHA for 756b09f

• fix

  masahi

  committed
  09ef5b3

  Copy full SHA for 09ef5b3

• correctly handle num_past_tokens > sliding_window case

  masahi

  committed
  7b67ba4

  Copy full SHA for 7b67ba4

• wip

  masahi

  committed
  e0517fd

  Copy full SHA for e0517fd

• blac

  masahi

  committed
  cf89a5b

  Copy full SHA for cf89a5b

• wip

  masahi

  committed
  9ca4806

  Copy full SHA for 9ca4806

• wip

  masahi

  committed
  4541b4d

  Copy full SHA for 4541b4d

• remove cancel call back in eviction

  masahi

  committed
  5d376d2

  Copy full SHA for 5d376d2

• Create MultiQueryDecodeRequest

  masahi

  committed
  59c36cc

  Copy full SHA for 59c36cc

• only the number of past tokens is needed

  masahi

  committed
  f58acf7

  Copy full SHA for f58acf7

• wip

  masahi

  committed
  d9dd2ca

  Copy full SHA for d9dd2ca

• wip

  masahi

  committed
  cb11761

  Copy full SHA for cb11761

• wip

  masahi

  committed
  24f7bfa

  Copy full SHA for 24f7bfa

• fix

  masahi

  committed
  34da221

  Copy full SHA for 34da221

• wip

  masahi

  committed
  d94e9d8

  Copy full SHA for d94e9d8

• wip

  masahi

  committed
  4a3bb77

  Copy full SHA for 4a3bb77

• wip

  masahi

  committed
  0c6875e

  Copy full SHA for 0c6875e

• wip

  masahi

  committed
  a46abe1

  Copy full SHA for a46abe1

• working?

  masahi

  committed
  c80bea2

  Copy full SHA for c80bea2

• remove dbg print

  masahi

  committed
  18239a4

  Copy full SHA for 18239a4

• multi gpu works

  masahi

  committed
  fd2b2bd

  Copy full SHA for fd2b2bd

• fixed sliding window logic

  masahi

  committed
  6ac292b

  Copy full SHA for 6ac292b

• remove dbug print

  masahi

  committed
  2f9d1f7

  Copy full SHA for 2f9d1f7

• clean and fix

  masahi

  committed
  3a9f6d6

  Copy full SHA for 3a9f6d6

• mypy

  masahi

  committed
  9fb9261

  Copy full SHA for 9fb9261

Commits on Jan 13, 2024

• generate signature update

  masahi

  committed
  906b23b

  Copy full SHA for 906b23b

• Merge branch 'batch-serving' into parallel-sampling-eviction

  masahi

  committed
  2c1aa04

  Copy full SHA for 2c1aa04

• more

  masahi

  committed
  b197e71

  Copy full SHA for b197e71

• fix mypy

  masahi

  committed
  2dfa28d

  Copy full SHA for 2dfa28d

• fix

  masahi

  committed
  e287c5f

  Copy full SHA for e287c5f

Commits on Jan 31, 2024

• Merge branch 'batch-serving' into parallel-sampling-eviction

  masahi

  committed
  417750c

  Copy full SHA for 417750c

• fix

  masahi

  committed
  c925c52

  Copy full SHA for c925c52

• mypy fix

  masahi

  committed
  a4d6e01

  Copy full SHA for a4d6e01

Commits on Feb 1, 2024

• Merge branch 'batch-serving' into parallel-sampling-eviction

  masahi

  committed
  7360392

  Copy full SHA for 7360392

• refactor

  masahi

  committed
  5dbf73e

  Copy full SHA for 5dbf73e

• fix

  masahi

  committed
  78a6f77

  Copy full SHA for 78a6f77

• rename

  masahi

  committed
  9189697

  Copy full SHA for 9189697

• Disallow preempting when a request has generated more than max_num_batched_tokens

  masahi

  committed
  d4fe2d7

  Copy full SHA for d4fe2d7