-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't default to OPENBLAS_NUM_THREADS=1
in notebooks
#233
Comments
If you copy the cell like five times, do all results in jupyter not have a NaN and all in julia engine have it? Does setting a random seed make a difference? |
I'm seeing pretty consistent results between QNR and the REPL for that code. No @ajinkya-k Can we confirm that the Manifest.toml's are identical and that the exact same version of |
@jkrumbiegel There are no RNG calls in MixedModels fitting so that should not matter, but I did try setting seed just before running fit, yet the problem persists. Also, copying the chunk multiple times produces the same output as before. |
Fitting this model is a delicate optimization problem. Even slight differences in the floating point evaluations or order of operations in the numerical linear algebra could result in convergence to different local optima. We are most interested in being able to configure QNR so that we reproduce the environment of the Jupyter evaluation. We have engine: julia
julia:
exeflags:
- -tauto
- --project in the header of the document from which this example was taken. For me, the Jupyter kernel configuration julia-1.11 is {
"display_name": "julia 1.11.2",
"argv": [
"/Users/dmbates/.julia/juliaup/julia-1.11.2+0.aarch64.apple.darwin14/bin/julia",
"-i",
"--color=yes",
"-tauto",
"--project",
"/Users/dmbates/.julia/packages/IJulia/dR0lE/src/kernel.jl",
"{connection_file}"
],
"language": "julia",
"env": {},
"interrupt_mode": "signal"
} I would hope that the repetition of the |
As you are on an ARM mac, can you confirm you're running the exact same Julia each time and not once arm and once x64 versions maybe? Just as a sanity check. Other than that, maybe optimization settings could make a difference? You could compare |
I do have an arm macos device, and can confirm that both are using the same binary, the only difference is one of them uses |
I am also on an arm macos device and I do have a I will look into |
Malt is used to run the workers, and it does https://github.com/JuliaPluto/Malt.jl/blob/124c471c574d0958fd349858fd3686b798068ff3/src/Malt.jl#L255. Do we reckon that setting |
Yes, that's the difference % OPENBLAS_NUM_THREADS=1 julia -tauto --project --startup=no
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.2 (2024-12-01)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using MixedModels
julia> m1 = fit(MixedModel, @formula(y ~ 1 + service + (1|s) + (1|d) + (1 + service|dept)), MixedModels.dataset(:InstEval))
Minimizing 219 Time: 0:00:04 (19.19 ms/it)
Linear mixed model fit by maximum likelihood
y ~ 1 + service + (1 | s) + (1 | d) + (1 + service | dept)
logLik -2 logLik AIC AICc BIC
-118825.3984 237650.7968 237666.7968 237666.7987 237740.4285
Variance components:
Column Variance Std.Dev. Corr.
s (Intercept) 0.105160 0.324284
d (Intercept) 0.264545 0.514340
dept (Intercept) 0.000000 0.000000
service: Y 0.022730 0.150766 +NaN
Residual 1.385073 1.176891
Number of obs: 73421; levels of grouping factors: 2972, 1128, 14
Fixed-effects parameters:
─────────────────────────────────────────────────────
Coef. Std. Error z Pr(>|z|)
─────────────────────────────────────────────────────
(Intercept) 3.27728 0.0188109 174.22 <1e-99
service: Y -0.0503874 0.0433777 -1.16 0.2454
───────────────────────────────────────────────────── Thank you for looking into this. I think that in some ways this just shows that the example is too fragile if changes like this in the environment cause convergence to another optimum. Do you have any suggestions of how we could force loading of AppleAccelerate.jl without explicitly doing so in the document? In other words, can we defeat the |
Adding a |
could the hardcoded |
I think the frontmatter would count as "explicitly in the document". There's an env variable |
Ah okay. I misunderstood the code |
yeah maybe we should remove that env var in our vendored Malt version, I'm not sure what it's original purpose was, though. I guess if you set your own env variable in the frontmatter you could override it. |
would the override have to be inside the |
It's possible to call using LinearAlgebra
BLAS.set_number_threads(4) to set the number of BLAS threads manually in an interactive session. If I toggle back and forth between 1 and 2 threads, I see the changed model fit. |
Avoid
Yes, anything set outside of julia:
env: ["OPENBLAS_NUM_THREADS=2"] There's some further context in the Malt PR that added this JuliaPluto/Malt.jl#59. |
I think this issue can be closed as we know the cause of the discrepancy. |
I've read through the threads in Malt that mentioned the openblas settings. Looks like this was inherited from Distributed which set it to avoid OOM errors on windows. I'm not sure if we should also have it, it seems unexpected for users that openblas will behave differently in quarto notebooks than in normal repls. |
ah of course! That fixed it. Sorry for the trouble |
No worries. |
We can only remove that if we are certain that it's not going to cause trouble for any users of this package. Given that it's relatively simple to override the behaviour with an env var it doesn't seem high priority to me. |
Is it not going to cause all kinds of numerical code to be slower than necessary? I'm not sure but I assumed that was why it's a bit problematic |
Yes. Like was said in #233 (comment), if it can be shown that removing the current default don't result in OOM on Windows then we can certainly do it. I don't have easy access to a Windows system to thoroughly test this, so it'll have to rely on someone with access to do so. |
You can easily check the effect of multi-threaded custom BLAS with % OPENBLAS_NUM_THREADS=1 julia --startup=no
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.3 (2025-01-21)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using LinearAlgebra
julia> BLAS.get_config()
LinearAlgebra.BLAS.LBTConfig
Libraries:
└ [ILP64] libopenblas64_.dylib
julia> peakflops()
4.96176686992832e10
julia> using AppleAccelerate
julia> BLAS.get_config()
LinearAlgebra.BLAS.LBTConfig
Libraries:
├ [ LP64] Accelerate
└ [ILP64] Accelerate
julia> peakflops()
4.16963396263107e11
julia> versioninfo()
Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin24.0.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 6 virtual cores)
|
Even if it still causes OOM issues, is the responsibility to avoid that on us or on the users with those systems? We're not doing anything much more special than spinning up a vanilla julia process with a bit of code loaded in, so I find it a bit strange having to handle BLAS issues on some systems in QNR. |
It is our responsibility to provide understandable error messages in the event of a notebook OOM. I've not reproduced the OOM so don't know what it would look like to a user. If we can successfully do that such that they are guided to adding Someone with a Windows system is welcome to check this and propose an alternative behaviour that works for everyone. |
engine: julia
OPENBLAS_NUM_THREADS=1
in notebooks
I am reopening this as an up-for-grabs issue. We can change the behaviour if it can be confirmed that it won't result in confusing errors for users. |
I removed the default OPENBLAS env setting here: QuartoNotebookRunner.jl/src/Malt.jl Line 374 in f736668
"OPENBLAS_DEFAULT_NUM_THREADS" => "1" and "OPENBLAS_MAIN_FREE" => "1" when i printed all the environment variables inside the quarto notebook using for evv in ENV println(evv) end . Note that running the same in REPL did not produce these env vars. Maybe quarto-cli is setting them somewhere? |
In a REPL I am seeing those env vars:
appear to be set by https://github.com/JuliaLang/julia/blob/88c71dd2164030ef9563a57ef3f1072581d8efca/stdlib/OpenBLAS_jll/src/OpenBLAS_jll.jl#L46-L74. |
Sorry I think there was some issue in my session. |
Hello!
engine: julia
produces different output than REPL and alsojupyter: julia-1.11
. Also tagging @dmbates (who is the dev forMixedModels.jl
) here who also had the same issue on his machine.See example with screenshot below.
MWE:
REPL output:
jupyter: julia-1.11
andengine: jupyter
output:The text was updated successfully, but these errors were encountered: