Code

Limit number of sequences#220

Code

Merged

masahi merged 11 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom yelite:fix-memory-profilingyelite/mlc-llm:fix-memory-profilingCopy head branch name to clipboardFeb 23, 2024

+45-3

Commits

Commits on Feb 21, 2024

• Add max_num_seq

  yelite

  committed
  29cac68

  Copy full SHA for 29cac68

• add to engine

  yelite

  committed
  549861a

  Copy full SHA for 549861a

Commits on Feb 22, 2024

• Limit max_num_seq when grows the batch

  yelite

  committed
  4cf54ea

  Copy full SHA for 4cf54ea

• Add max num seq to args

  yelite

  committed
  a96e915

  Copy full SHA for a96e915

Commits on Feb 23, 2024

• Add max_num_seq_per_sequence

  yelite

  committed
  1fd2b2e

  Copy full SHA for 1fd2b2e

• Expose gpu memory utilization to engine config

  yelite

  committed
  634a869

  Copy full SHA for 634a869

• Fix dataclass

  yelite

  committed
  48af6d5

  Copy full SHA for 48af6d5

• Remove the default value of gpu_memory_utilization

  yelite

  committed
  38ba3e3

  Copy full SHA for 38ba3e3

• Merge branch 'expose-gpu-util' into fix-memory-profiling

  yelite

  committed
  320a9ba

  Copy full SHA for 320a9ba

• Fix params

  yelite

  committed
  0b574aa

  Copy full SHA for 0b574aa

• Apply to torch model

  yelite

  committed
  cd4f1c8

  Copy full SHA for cd4f1c8