Stepfun 3.7 Flash support

[heidariamin07]

Can we please get the support for Stepfun 3.7 Flash?

[heidariamin07]

Managed to get it working. There is a PR in the mlx-lm. I just asked my hermes agent to use that PR for the oMLX's mlx-lm evnvironment. Now I can use Step3.7 Flash with oMLX. The TG is a bit lower than on llama cpp, but the PP is higher and stays higher with higher tokens. with over 10k tokens the PP was around 430 t/s on M3 Ultra with 60 core GPU. Here is the step by step tutorial to get it working: Using Unmerged MLX-LM PRs with oMLX

This guide shows how to install an unreleased PR from the mlx-lm repository directly into your local oMLX installation, so you can use new model types before they land in an official release.

Prerequisites
- oMLX installed in /Applications/oMLX.app
- A model checkpoint you want to load (e.g. /Volumes/WD_EXTRA/models/catalyst/Step-3.7-Flash-4bit)
- Terminal access



Step 1: Identify the PR commit hash

Find the PR in the mlx-lm GitHub repo and note the commit hash of the branch you want.
For example, PR #1325 (Step 3.7 Flash support) used commit:

bash
2b23758bdc3379d0b18c4022a8bf0136925803c6




Step 2: Install the PR into oMLX's bundled Python

oMLX ships with its own Python 3.11 environment. Do not use your system pip. Use the bundled Python instead:

bash
APP=/Applications/oMLX.app

"$APP/Contents/Frameworks/cpython-3.11/bin/pip" install \
  --no-deps \
  "git+https://github.com/ml-explore/mlx-lm.git@<COMMIT_HASH>"


- --no-deps prevents overwriting dependencies that oMLX already provides.
- Replace <COMMIT_HASH> with the actual commit hash.



Step 3: Verify the new module loads

bash
PYTHONPATH="$APP/Contents/Resources:$APP/Contents/Frameworks/app-omlx-app/lib/python3.11/site-packages:$APP/Contents/Frameworks/framework-mlx-framework/lib/python3.11/site-packages" \
PYTHONHOME="$APP/Contents/Frameworks/cpython-3.11" \
"$APP/Contents/Frameworks/cpython-3.11/bin/python" -c "from mlx_lm.models import step3p7; print('step3p7 loaded')"


You should see step3p7 loaded if the PR installed correctly.



Step 4: Serve the model

bash
omlx serve --model /path/to/your/model

[fry69]

Sounds like the solution is as easy as updating this line pointing to a newer mlx-lm and rebuild? -> https://github.com/jundot/omlx/blob/main/pyproject.toml#L33

I'd recommend to run pytest after this to check how much this breaks (or not).

Update: This is still a PR (-> ml-explore/mlx-lm@7794077) not a commit on the main branch, so this will not work. I commented too fast, sorry for that.

[heidariamin07]

Sounds like the solution is as easy as updating this line pointing to a newer mlx-lm and rebuild? -> https://github.com/jundot/omlx/blob/main/pyproject.toml#L33

Isn't this the mlx-vlm?? I mean there is no PR in vlm-mlx for Step support yet.

[fry69]

Line 33 points to mlx-lm, line 76 to mlx-vlm, I mixed those up in the first version of my comment, sorry.

[heidariamin07]

Ah. thanks

[beamivalice]

This Step 3.7 Flash crashed every single time on model unload - maybe #1542 could help.

[fry69]

FYI: This commit just dropped which likely adds Stepfun 3.7 Flash support -> 4795fb4