Code

Enable running PyTorch models #207

Code

Merged

sunggg merged 85 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom masahi:pt-modelmasahi/mlc-llm:pt-modelCopy head branch name to clipboardFeb 22, 2024

+850-64

Commits

Commits on Jan 11, 2024

• refactor to separate TVM specific bits from paged_cache_model

  masahi

  committed
  12ce0a3

  Copy full SHA for 12ce0a3

• fix

  masahi

  committed
  7a84f15

  Copy full SHA for 7a84f15

• Remove engine config change for now

  masahi

  committed
  f454b7b

  Copy full SHA for f454b7b

Commits on Jan 12, 2024

• make mypy happy with TextGenerator impl by Model

  masahi

  committed
  afde741

  Copy full SHA for afde741

• stub

  masahi

  committed
  c49ef45

  Copy full SHA for c49ef45

• wip

  masahi

  committed
  d9ac72f

  Copy full SHA for d9ac72f

• wip

  masahi

  committed
  acbf825

  Copy full SHA for acbf825

• wip

  masahi

  committed
  fef750f

  Copy full SHA for fef750f

• PT model memory profiling works

  masahi

  committed
  25a567e

  Copy full SHA for 25a567e

• get rid of vllm prepare_inputs

  masahi

  committed
  3d06f68

  Copy full SHA for 3d06f68

• wip

  masahi

  committed
  3cafc8b

  Copy full SHA for 3cafc8b

• model runs but nan output

  masahi

  committed
  34f77ef

  Copy full SHA for 34f77ef

• mypy improvement

  masahi

  committed
  afb4d4f

  Copy full SHA for afb4d4f

• runs e2e but the result is garbage

  masahi

  committed
  e7212a5

  Copy full SHA for e7212a5

• working

  masahi

  committed
  f27e3b3

  Copy full SHA for f27e3b3

• minor

  masahi

  committed
  2316e37

  Copy full SHA for 2316e37

• do sampling by mlc function

  masahi

  committed
  9b985e8

  Copy full SHA for 9b985e8

Commits on Jan 13, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  f2dcc48

  Copy full SHA for f2dcc48

• merge fix

  masahi

  committed
  4d73e63

  Copy full SHA for 4d73e63

• wip parallel sampling

  masahi

  committed
  15a0d3b

  Copy full SHA for 15a0d3b

• fix test

  masahi

  committed
  959019d

  Copy full SHA for 959019d

Commits on Jan 15, 2024

• wip

  masahi

  committed
  b6050d9

  Copy full SHA for b6050d9

• fix

  masahi

  committed
  ff8eb27

  Copy full SHA for ff8eb27

Commits on Jan 16, 2024

• wip

  masahi

  committed
  8696df5

  Copy full SHA for 8696df5

• wip

  masahi

  committed
  0af3a70

  Copy full SHA for 0af3a70

Commits on Jan 18, 2024

• wip

  masahi

  committed
  90ffccd

  Copy full SHA for 90ffccd

• attach cache_blocks to model

  masahi

  committed
  32686d8

  Copy full SHA for 32686d8

• change get_num_cache_blocks signature

  masahi

  committed
  de2631b

  Copy full SHA for de2631b

• wip

  masahi

  committed
  618ca62

  Copy full SHA for 618ca62

Commits on Jan 19, 2024

• wip

  masahi

  committed
  9ce2f47

  Copy full SHA for 9ce2f47

• wip

  masahi

  committed
  c14c0e9

  Copy full SHA for c14c0e9

• refactor

  masahi

  committed
  4328440

  Copy full SHA for 4328440

• update for qwen

  masahi

  committed
  9fb6358

  Copy full SHA for 9fb6358

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  0c40fe8

  Copy full SHA for 0c40fe8

• mergei fix

  masahi

  committed
  7d89811

  Copy full SHA for 7d89811

• clean

  masahi

  committed
  0bbc41a

  Copy full SHA for 0bbc41a

• KV cache refactor to decouple cache blocks and metadata about them

  masahi

  committed
  08a63ca

  Copy full SHA for 08a63ca

• Merge branch 'kv-refactor' into pt-model

  masahi

  committed
  1dee091

  Copy full SHA for 1dee091

• update for KV refactor

  masahi

  committed
  e098d0b

  Copy full SHA for e098d0b

Commits on Jan 20, 2024

• updated for the latest vllm

  masahi

  committed
  02b7c1b

  Copy full SHA for 02b7c1b

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  5cefe97

  Copy full SHA for 5cefe97

• qwen and phi supported

  masahi

  committed
  c470c36

  Copy full SHA for c470c36

• Make num_shards configuable via engine config

  masahi

  committed
  a4612da

  Copy full SHA for a4612da

• unify Model and ModelRpcClient classes

  masahi

  committed
  4564bd0

  Copy full SHA for 4564bd0

• support PT model in server

  masahi

  committed
  196026c

  Copy full SHA for 196026c

Commits on Jan 21, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  de68a84

  Copy full SHA for de68a84

• properly allocate port

  masahi

  committed
  b502654

  Copy full SHA for b502654

Commits on Jan 22, 2024

• refactor engine creation

  masahi

  committed
  686780c

  Copy full SHA for 686780c

• fix sync point

  masahi

  committed
  bebd7b2

  Copy full SHA for bebd7b2

• do not create executor at each step

  masahi

  committed
  ed46b5e

  Copy full SHA for ed46b5e

• remove dup obtain calls

  masahi

  committed
  62918dd

  Copy full SHA for 62918dd

Commits on Feb 1, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  b98bdce

  Copy full SHA for b98bdce

• fix

  masahi

  committed
  e144517

  Copy full SHA for e144517

Commits on Feb 2, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  e6abcc7

  Copy full SHA for e6abcc7

• use sample_from_logits

  masahi

  committed
  dfbf359

  Copy full SHA for dfbf359

Commits on Feb 6, 2024

• enable TCP NoDelay option to fix slow socket recv issue

  masahi

  committed
  04da3bb

  Copy full SHA for 04da3bb

• Replace TCP with Unix domain socket

  masahi

  committed
  5dfecb2

  Copy full SHA for 5dfecb2

• clean and add note on RPC overhead

  masahi

  committed
  e4bbad9

  Copy full SHA for e4bbad9

• clean

  masahi

  committed
  ee9cdc9

  Copy full SHA for ee9cdc9

• RPC process join works

  masahi

  committed
  2071749

  Copy full SHA for 2071749

• fix mypy

  masahi

  committed
  15a90d0

  Copy full SHA for 15a90d0

Commits on Feb 13, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  1f56ee9

  Copy full SHA for 1f56ee9

• merge fix

  masahi

  committed
  90284fa

  Copy full SHA for 90284fa

• wip test fix

  masahi

  committed
  52ad1ad

  Copy full SHA for 52ad1ad

• fix

  masahi

  committed
  61b680e

  Copy full SHA for 61b680e

• Properly verify sampling params in api handler

  masahi

  committed
  f128fe6

  Copy full SHA for f128fe6

• Create model artifact config before module initialization

  masahi

  committed
  568583a

  Copy full SHA for 568583a

• fix engine start

  masahi

  committed
  762012d

  Copy full SHA for 762012d

• Merge branch 'sampling-params-init-fix' into pt-model

  masahi

  committed
  72f3707

  Copy full SHA for 72f3707

Commits on Feb 14, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  3128329

  Copy full SHA for 3128329

• fix

  masahi

  committed
  dc5fb6e

  Copy full SHA for dc5fb6e

• black

  masahi

  committed
  ebe0b4e

  Copy full SHA for ebe0b4e

• properly handle import failure

  masahi

  committed
  4b2de70

  Copy full SHA for 4b2de70

• add titoken dep

  masahi

  committed
  f09d458

  Copy full SHA for f09d458

• revert logprob change

  masahi

  committed
  c9ac5ba

  Copy full SHA for c9ac5ba

• restored tokenizer.is_fast assert but commented out

  masahi

  committed
  f1cf274

  Copy full SHA for f1cf274

Commits on Feb 15, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  eaa53a7

  Copy full SHA for eaa53a7

• fix vocab siz

  masahi

  committed
  1336fb8

  Copy full SHA for 1336fb8

• properly account for logits storage in memory profiling

  masahi

  committed
  6186ef2

  Copy full SHA for 6186ef2

Commits on Feb 20, 2024

• Merge branch 'batch-serving' into pt-model

  masahi

  committed
  2229324

  Copy full SHA for 2229324

• merge fix

  masahi

  committed
  aa4d477

  Copy full SHA for aa4d477

• validate num_shards in engine creation

  masahi

  committed
  8bb96ed

  Copy full SHA for 8bb96ed

• replace print with structlog

  masahi

  committed
  cf0813d

  Copy full SHA for cf0813d

• add peak memory log for tvm as well

  masahi

  committed
  f716851

  Copy full SHA for f716851

Commits on Feb 21, 2024

• add tokenizer.is_fast warning on creation

  masahi

  committed
  992b1a0

  Copy full SHA for 992b1a0