Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions src/legend_data.jl
Original file line number Diff line number Diff line change
Expand Up @@ -488,3 +488,131 @@
)
end
export channel_info

"""
period_channelinfo(data::LegendData, period::DataPeriod; kwargs...)
Get channel information for a given period, combining all runs in that period.

this channel info takes all runs in this period and creates a combined channelinfo
for :usability, :is_blinded, :psd_usability, :low_aoe_status, :high_aoe_status, :lq_status, :ann_status, :coax_rt_status only the 'best' value is taken, i.e. the one with the highest priority in the hierarchy
all the other column entries should remain the same over different runs

all kwargs and filterby can be used as in the usual channelinfo

make sure to include "Tables" by "using Tables" before using this function
"""

function period_channelinfo(data::LegendData, period::DataPeriod; kwargs...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is great as a first draft, however this should become a dispatch onto the existing channelinfo

Suggested change
function period_channelinfo(data::LegendData, period::DataPeriod; kwargs...)
function channelinfo(data::LegendData, sel::PeriodSelLike; kwargs...)

filekey_array = get_filekey_array(data, period)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These functions all need comments; otherwise, this is not maintainable.

chinfo_array = get_chinfo_array(data, filekey_array; kwargs...)
merged_chinfo = vcat(chinfo_array...)
red_chinfo = merge_and_reduce_chinfo(merged_chinfo)
sorted_chinfo = sort_chinfo(red_chinfo)
extended = get(kwargs, :extended, false)
hierarchies = get_hierarchies(extended)
column_order = get_column_order(extended)
final_chinfo = Table(apply_hierarchies(sorted_chinfo, hierarchies, column_order))
return final_chinfo

Check warning on line 515 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L505-L515

Added lines #L505 - L515 were not covered by tests
end

function get_filekey_array(l200, period)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your function arguments need types.

rinfo = runinfo(l200, period) |> filterby(@pf $cal.is_analysis_run)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are actively filtering here by cal analysis runs. This is not a good idea. Usually, the category is given to load the channelinfo, e.g.

channelinfo(l200, :p03, :r000, :cal)

In the case where you load the channelinfo for a specific DataRun. This category should also be used in the case of a DataPeriod channelinfo, by allowing the DataCategory as an additional argument

return [i.cal.startkey for i in rinfo]

Check warning on line 520 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L518-L520

Added lines #L518 - L520 were not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is equivalent to

Suggested change
return [i.cal.startkey for i in rinfo]
rinfo.cal.startkey

end


function get_chinfo_array(l200, filekey_array; kwargs...)
extended = get(kwargs, :extended, false)
return [channelinfo(l200, fk; kwargs...) for fk in filekey_array]

Check warning on line 526 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L524-L526

Added lines #L524 - L526 were not covered by tests
end

function merge_and_reduce_chinfo(merged_chinfo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this function doing? This looks like it could be simplified a lot with standard Tables operations. In particular the pushing into empty Dict or Vector is not necessary here.

grouped = Dict{Any, Dict{Symbol, Vector}}()
for row in Tables.rows(merged_chinfo)
detector = row.detector
detector_group = get!(grouped, detector, Dict(col => [] for col in propertynames(row) if col != :detector))
for col in propertynames(row)
if col != :detector
push!(detector_group[col], getproperty(row, col))

Check warning on line 536 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L529-L536

Added lines #L529 - L536 were not covered by tests
end
end
end
red_chinfo = (; detector = collect(keys(grouped)))
for col in propertynames(first(Tables.rows(merged_chinfo)))
if col != :detector
red_chinfo = merge(red_chinfo, (; (col => [grouped[d][col] for d in keys(grouped)])))

Check warning on line 543 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L538-L543

Added lines #L538 - L543 were not covered by tests
end
end
return red_chinfo

Check warning on line 546 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L545-L546

Added lines #L545 - L546 were not covered by tests
end

function sort_chinfo(red_chinfo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you want to achieve with this function?
Can you not just do

sort(red_chinfo

Types are also missing.

sorted_indices = sortperm(red_chinfo.detector)
sorted_chinfo = (; detector = red_chinfo.detector[sorted_indices])
for col in propertynames(red_chinfo)
if col != :detector
sorted_chinfo = merge(sorted_chinfo, (; (col => red_chinfo[col][sorted_indices])))

Check warning on line 554 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L549-L554

Added lines #L549 - L554 were not covered by tests
end
end
return sorted_chinfo

Check warning on line 557 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L556-L557

Added lines #L556 - L557 were not covered by tests
end

function apply_hierarchies(sorted_chinfo, hierarchies, column_order)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Types to the arguments

final_chinfo = (;)
for col in column_order
if col in keys(hierarchies)
hierarchy = hierarchies[col]
if isempty(hierarchy)

Check warning on line 565 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L560-L565

Added lines #L560 - L565 were not covered by tests
# Skip applying hierarchy if it's empty
final_chinfo = merge(final_chinfo, (; (col => sorted_chinfo[col])))
continue

Check warning on line 568 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L567-L568

Added lines #L567 - L568 were not covered by tests
end
hierarchy_index = Dict(x => i for (i, x) in enumerate(hierarchy))
final_chinfo = merge(final_chinfo, (; (col => [

Check warning on line 571 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L570-L571

Added lines #L570 - L571 were not covered by tests
begin
valid_values = filter(x -> haskey(hierarchy_index, x), sorted_chinfo[col][i])
valid_values == [] ? nothing : valid_values[argmin(hierarchy_index[x] for x in valid_values)]

Check warning on line 574 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L573-L574

Added lines #L573 - L574 were not covered by tests
end for i in 1:length(sorted_chinfo.detector)
])))
else
final_chinfo = merge(final_chinfo, (; (col => [

Check warning on line 578 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L578

Added line #L578 was not covered by tests
sorted_chinfo[col][i] isa AbstractVector ? first(unique(sorted_chinfo[col][i])) : sorted_chinfo[col][i]
for i in 1:length(sorted_chinfo.detector)
])))
end
end
return final_chinfo

Check warning on line 584 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L583-L584

Added lines #L583 - L584 were not covered by tests
end

function get_hierarchies(extended::Bool)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the base_hierarchies are static and do not need to be changed?
In that case you can just make them a const

Suggested change
function get_hierarchies(extended::Bool)
const _base_hierarchies = Dict{Symbol, Vector{Union{Symbol, Bool}}}(
:usability => [:on, :ac, :off],
:is_blinded => [true, false],
:psd_usability => [:on, :off],
:low_aoe_status => [:valid, :present, :missing, :unknown],
:high_aoe_status => [:valid, :present, :missing, :unknown],
:lq_status => [:valid, :present, :missing, :unknown],
:ann_status => [:valid, :present, :missing, :unknown],
:coax_rt_status => [:valid, :present, :missing, :unknown]
)

base_hierarchies = Dict(

Check warning on line 588 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L587-L588

Added lines #L587 - L588 were not covered by tests
:usability => [:on, :ac, :off],
:is_blinded => [true, false],
:psd_usability => [:on, :off],
:low_aoe_status => [:valid, :present, :missing, :unknown],
:high_aoe_status => [:valid, :present, :missing, :unknown],
:lq_status => [:valid, :present, :missing, :unknown],
:ann_status => [:valid, :present, :missing, :unknown],
:coax_rt_status => [:valid, :present, :missing, :unknown]
)
return base_hierarchies

Check warning on line 598 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L598

Added line #L598 was not covered by tests
end

function get_column_order(extended::Bool)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this function doing and can you not move this to the outer function?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order of the columns was changed in the prior steps and I wanted to keep the original order (as in the usual channelinfo)

base_columns = [

Check warning on line 602 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L601-L602

Added lines #L601 - L602 were not covered by tests
:detector, :channel, :fcid, :rawid, :system, :processable,
:usability, :is_blinded, :psd_usability, :low_aoe_status, :high_aoe_status,
:lq_status, :ann_status, :coax_rt_status, :is_bb_like,
:det_type, :location, :detstring, :fiber, :position
]
if extended
extended_columns = [

Check warning on line 609 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L608-L609

Added lines #L608 - L609 were not covered by tests
:cc4ch, :daqcrate, :daqcard, :hvcard, :hvch,
:enrichment, :mass, :total_volume, :active_volume, :fccd
]
return vcat(base_columns, extended_columns)

Check warning on line 613 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L613

Added line #L613 was not covered by tests
else
return base_columns

Check warning on line 615 in src/legend_data.jl

View check run for this annotation

Codecov / codecov/patch

src/legend_data.jl#L615

Added line #L615 was not covered by tests
end
end
export period_channelinfo
Loading