Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Continually updating reactive variables generates numerous GenieSessionFileSession files with enormous footprint >300GB #257

Closed
zygmuntszpak opened this issue Jan 25, 2024 · 1 comment
Closed

Continually updating reactive variables generates numerous GenieSessionFileSession files with enormous footprint >300GB #257

zygmuntszpak opened this issue Jan 25, 2024 · 1 comment

Comments

@zygmuntszpak
Copy link

I opened this issue on Genie (GenieFramework/Genie.jl#702), but it looks like it is a Stipple specific issue.

I'm writing an app (work related) which continually ingests an MQTT stream of IoT sensors, does some analysis and dynamically plots it using Stipple. The app is meant to run continually and the reactive variables are updated approximately every second. Within 24 hours the PC runs out of disk space because of an enormous number of session files that are serialized to a temporary folder. Multiple session files are created within a minute, and each session file eventually becomes 20 MB large. The sheer number of these files eventually crashed the program because there is no more disk space:

**Error stacktrace**
┌ Error: 2024-01-24 12:33:51 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:33:51 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:33:51 Failed to store session data
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:41
┌ Error: 2024-01-24 12:33:51 Failed to store session data
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:41
┌ Error: 2024-01-24 12:33:51 SystemError("close", 28, nothing)
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:42
┌ Error: 2024-01-24 12:33:51 SystemError("close", 28, nothing)
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:42
┌ Error: 2024-01-24 12:33:51 Resetting session
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:46
┌ Error: 2024-01-24 12:33:51 Resetting session
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:46
┌ Error: 2024-01-24 12:33:52 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
Total retrieved records is: 760817
Inside Reading Task Timestamp: 2024-01-24T12:34:37.791
┌ Error: 2024-01-24 12:37:15 Failed to store session data
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:41
┌ Error: 2024-01-24 12:37:15 SystemError("close", 28, nothing)
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:42
┌ Error: 2024-01-24 12:37:15 Resetting session
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:46
┌ Error: 2024-01-24 12:37:22 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:22 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:23 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:23 Failed to store session data
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:41
┌ Error: 2024-01-24 12:37:23 SystemError("close", 28, nothing)
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:42
┌ Error: 2024-01-24 12:37:23 Resetting session
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:46
┌ Error: 2024-01-24 12:37:24 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:25 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:25 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:26 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:27 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:27 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:27 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:28 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:29 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:30 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:30 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:31 SQLite.SQLiteException("database or disk is full")
└ @ SearchLight C:\Users\zygmuntszpak\.julia\packages\SearchLight\Ps2Js\src\SearchLight.jl:213
┌ Error: 2024-01-24 12:37:31 Failed to store session data
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:41
┌ Error: 2024-01-24 12:37:31 SystemError("close", 28, nothing)
└ @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:42
Unhandled Task ERROR: SystemError: flush: No space left on device
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base .\error.jl:176
  [2] #systemerror#82
    @ .\error.jl:175 [inlined]
  [3] systemerror
    @ .\error.jl:175 [inlined]
  [4] flush(s::IOStream)
    @ Base .\iostream.jl:70
  [5] handle_message(::LoggingExtras.FileLogger, ::Base.CoreLogging.LogLevel, ::Vararg{Any};
 kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\Sinks\filelogger.jl:47
  [6] handle_message(::LoggingExtras.FileLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64)
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\Sinks\filelogger.jl:45
  [7] handle_message(::LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, ::Base.CoreLogging.LogLevel, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\tee.jl:24
  [8] handle_message(::LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64)
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\tee.jl:21
  [9] handle_message(::LoggingExtras.TeeLogger{Tuple{LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, Genie.Logger.GenieLogger}}, ::Base.CoreLogging.LogLevel, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\tee.jl:24
 [10] handle_message
    @ C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\tee.jl:21 [inlined]
 [11] #handle_message#13
    @ C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\transformer.jl:28 [inlined]
 [12] handle_message(::LoggingExtras.TransformerLogger{LoggingExtras.TeeLogger{Tuple{LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, Genie.Logger.GenieLogger}}, Genie.Logger.var"#1#2"{String}}, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64)
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\transformer.jl:20
 [13] handle_message(::LoggingExtras.MinLevelLogger{LoggingExtras.TransformerLogger{LoggingExtras.TeeLogger{Tuple{LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, Genie.Logger.GenieLogger}}, Genie.Logger.var"#1#2"{String}}, Base.CoreLogging.LogLevel}, ::Base.CoreLogging.LogLevel, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\minlevelfiltered.jl:17
 [14] handle_message(::LoggingExtras.MinLevelLogger{LoggingExtras.TransformerLogger{LoggingExtras.TeeLogger{Tuple{LoggingExtras.TeeLogger{Tuple{LoggingExtras.FileLogger, Logging.ConsoleLogger}}, Genie.Logger.GenieLogger}}, Genie.Logger.var"#1#2"{String}}, Base.CoreLogging.LogLevel}, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64)
    @ LoggingExtras C:\Users\zygmuntszpak\.julia\packages\LoggingExtras\VLO3o\src\CompositionalLoggers\minlevelfiltered.jl:15
 [15] handle_message(j::VSCodeServer.VSCodeLogger, level::Base.CoreLogging.LogLevel, message::String, _module::Module, group::Symbol, id::Symbol, file::String, line::Int64; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ VSCodeServer c:\Users\zygmuntszpak\.vscode\extensions\julialang.language-julia-1.66.2\scripts\packages\VSCodeServer\src\progress.jl:31
 [16] handle_message(j::VSCodeServer.VSCodeLogger, level::Base.CoreLogging.LogLevel, message::String, _module::Module, group::Symbol, id::Symbol, file::String, line::Int64)
    @ VSCodeServer c:\Users\zygmuntszpak\.vscode\extensions\julialang.language-julia-1.66.2\scripts\packages\VSCodeServer\src\progress.jl:7
 [17] #invokelatest#2
    @ .\essentials.jl:819 [inlined]
 [18] invokelatest
    @ .\essentials.jl:816 [inlined]
 [19] macro expansion
    @ .\logging.jl:330 [inlined]
 [20] write(session::GenieSession.Session)
    @ GenieSessionFileSession C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:55
 [21] persist
    @ C:\Users\zygmuntszpak\.julia\packages\GenieSessionFileSession\otnJC\src\GenieSessionFileSession.jl:116 [inlined]
 [22] set!
    @ C:\Users\zygmuntszpak\.julia\packages\GenieSession\Kmjen\src\GenieSession.jl:182 [inlined]
 [23] set!
    @ C:\Users\zygmuntszpak\.julia\packages\GenieSession\Kmjen\src\GenieSession.jl:187 [inlined]
 [24] (::Stipple.ModelStorage.Sessions.var"#2#4"{Main.App.DataStream.var"Main.App.DataStream_ReactiveModel", Main.App.DataStream.var"##Main.App.DataStream_ReactiveModel!#331"})(#unused#::Vector{PlotlyBase.GenericTrace{Dict{Symbol, Any}}})
    @ Stipple.ModelStorage.Sessions C:\Users\zygmuntszpak\.julia\packages\Stipple\4Csa4\src\ModelStorage.jl:39
 [25] #invokelatest#2
    @ .\essentials.jl:819 [inlined]
 [26] invokelatest
    @ .\essentials.jl:816 [inlined]
 [27] notify
    @ C:\Users\zygmuntszpak\.julia\packages\Observables\YdEbO\src\Observables.jl:206 [inlined]
 [28] setindex!(observable::Observable, val::Any)
    @ Observables C:\Users\zygmuntszpak\.julia\packages\Observables\YdEbO\src\Observables.jl:123
 [29] setindex!
    @ C:\Users\zygmuntszpak\.julia\packages\Observables\YdEbO\src\Observables.jl:109 [inlined]
 [30] update_r1_fcu1_in_temp_plot!(__model__::Main.App.DataStream.var"##Main.App.DataStream_ReactiveModel!#331")
    @ Main.App.DataStream C:\Users\zygmuntszpak\.julia\dev\LineZeroApp\lib\DataStream\room_1_plots.jl:2
 [31] update_plots!(__model__::Main.App.DataStream.var"##Main.App.DataStream_ReactiveModel!#331")
    @ Main.App.DataStream C:\Users\zygmuntszpak\.julia\dev\LineZeroApp\lib\DataStream\DataStream.jl:326
 [32] (::Main.App.DataStream.var"#171#172"{Main.App.DataStream.var"##Main.App.DataStream_ReactiveModel!#331"})()
    @ Main.App.DataStream .\threadingconstructs.jl:410

I don't fully understand what is meant to be stored in a Session. It looks like it is a copy of the Reactive model. In my instance, since I have multiple plots with hundreds of thousands of data points, that probably explains the 20mb filesize. However, I don't understand what constitutes an actual session. I would have thought that a new session only started every time I refreshed the browser. In contrast, I see numerous temporary session files created. My impression is that approximately every time I update a reactive variable, a new session file is logged. I still need to run some further tests to confirm this hypothesis. I wasn't expecting to generate 300GB of temporary files within 24 hours. Is persisting the session files to disk necessary, and should new sessions be spawned so frequently?

The current workaround I am piloting involves continually scanning the folder of temporary session files and deleting files older than one minute.

I would be grateful for any insights or suggestions you may have.

julia> versioninfo()

Julia Version 1.9.3
Commit bed2cd540a (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, tigerlake)
  Threads: 4 on 8 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 4
and
pkg> st

  [6e4b80f9] BenchmarkTools v1.4.0
  [336ed68f] CSV v0.10.12
  [a93c6f00] DataFrames v1.6.1
  [c43c736e] Genie v5.23.8
  [a59fdf5c] GenieFramework v1.26.11
  [033835bb] JLD2 v0.4.45
  [0f8b85d8] JSON3 v1.14.0
  [b9914132] JSONTables v1.0.3
  [db317de6] Mosquitto v0.8.1
  [a03496cd] PlotlyBase v0.8.19
  [1bd9f7bb] RemoteREPL v0.2.17 `C:\Users\zygmuntszpak\.julia\dev\RemoteREPL`
  [0aa819cd] SQLite v1.6.0
  [340e8cb6] SearchLight v2.10.0
  [21a827c4] SearchLightSQLite v2.2.2
  [2913bbd2] StatsBase v0.34.2
  [4acbeb90] Stipple v0.27.33
  [a3c5d34a] StippleUI v0.22.18
  [f269a46b] TimeZones v1.13.0
  [9bd350c2] OpenSSH_jll v9.3.2+0
  [9e88b42a] Serialization
  [8dfed614] Test
@essenciary
Copy link
Member

Closing this in favour of the Genie issue where we are already debugging it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@essenciary @zygmuntszpak