Struggling to understand documentation

[droodman]

I'm continuing to struggle to understand how to use this package. I appreciate all the effort that has gone into documenting it, and I hope that future improvements will get it to the point where it help me look forward to, and recommend, doing econometrics in Julia.

Example: I'm trying to understand numeric formatting codes, and in particular, how I can control the formatting of a log likelihood. (Is that possible?)

I find:
"estimformat is a String that describes the format of the estimate." From that, I can't tell what the acceptable format codes are.

So I search the documentation for "estimformat". I get one other hit. It has some examples, which are interesting, but serve as samples of what is possible without giving the big picture.

At the top of this subsection, I see "Also see Customization of Defaults". So I click that. I get:

Which is overwhelming. And also maybe not what I want since it is about defaults in particular? At that point I gave up.

For comparison, "help esttab" in Stata leads me directly to numeric formatting information. More generally--and this applies to many Julia packages--I think it would help the user a lot to have a systematic and thorough description of all options. I think this is the norm in Stata and Python; not sure about R.

[junder873]

I certainly want to do a lot more with the documentation. Part of the advantage (but challenge) of this package is everything is customizable, and often there are several different ways to do it. This creates a challenge in writing documentation that is thorough but also not overwhelming (as you correctly point out with the customization of defaults page).

You mentioned wanting to change the format of log likelihood (I assume you mean log likelihood summary statistics, similar to $R^2$).

• If all you need is more (or fewer) decimal points, then regtable(...; digits_stats=4) will impact all of the summary statistics. This also impacts other summary statistics, which may or may not matter to you.
• If you want some other statistic format, then regtable(...; statisticformat="%02.5f") can work. This also changes the format of the under statistic (standard error, t-stat). This package relies on the cfmt function from Format.jl for these strings.
• If there is a specific format you want to achieve for log likelihood, you can change how it is represented by:

using Format # this package relies on Format.jl already)
function Base.repr(render::AbstractRenderType, x::LogLikelihood; args...)
    format(RegressionTables.value(x); commas=true, precision=2)
end

The advantage of this method is it only impacts log likelihood.

[droodman]

Thanks. I tried that in this ~MWE and got an error:

using StatsAPI, StatsBase, RegressionTables, Format

struct myresult<:RegressionModel end

StatsAPI.coef(R::myresult) = [1.]
StatsAPI.coefnames(R::myresult) = "coef"
StatsAPI.dof(R::myresult) = 100
StatsAPI.loglikelihood(R::myresult) = -1e6
StatsAPI.vcov(R::myresult) = [1.;;]
StatsAPI.responsename(R::myresult) = "depname"
StatsAPI.islinear(R::myresult) = false
RegressionTables._responsename(x::myresult) = StatsAPI.responsename(x)
RegressionTables._coefnames(x::myresult) = coefnames(x)

function Base.repr(render::AbstractRenderType, x::LogLikelihood; args...)
	format(x; commas=true, precision=2)
end

regtable(myresult(); regression_statistics = [LogLikelihood])

ERROR: MethodError: no method matching format(::LogLikelihood; commas::Bool, precision::Int64)

Closest candidates are:
  format(::Union{AbstractString, FormatExpr}, Any...) got unsupported keyword arguments "commas", "precision"
   @ Format C:\Users\drood\.julia\packages\Format\0LfJN\src\formatexpr.jl:163
  format(::T; width, precision, leftjustified, zeropadding, commas, signed, positivespace, stripzeros, parens, alternative, mixedfraction, mixedfractionsep, fractionsep, fractionwidth, tryden, suffix, autoscale, conversion) where T<:Real
   @ Format C:\Users\drood\.julia\packages\Format\0LfJN\src\cformat.jl:142

Stacktrace:
  [1] repr(render::AsciiTable, x::LogLikelihood; args::@Kwargs{})
    @ Main .\Untitled-1:16
  [2] repr(render::AsciiTable, x::LogLikelihood)
    @ Main .\Untitled-1:15
  [3] _broadcast_getindex_evalf
    @ .\broadcast.jl:709 [inlined]
  [4] _broadcast_getindex
    @ .\broadcast.jl:682 [inlined]
  [5] _getindex
    @ .\broadcast.jl:706 [inlined]
  [6] _broadcast_getindex
    @ .\broadcast.jl:681 [inlined]
  [7] _getindex
    @ .\broadcast.jl:705 [inlined]
  [8] _broadcast_getindex
    @ .\broadcast.jl:681 [inlined]
  [9] getindex
    @ .\broadcast.jl:636 [inlined]
 [10] copyto_nonleaf!(dest::Vector{…}, bc::Base.Broadcast.Broadcasted{…}, iter::Base.OneTo{…}, state::Int64, count::Int64)
    @ Base.Broadcast .\broadcast.jl:1098
 [11] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{…}, Tuple{…}, typeof(+), Tuple{…}})
    @ Base.Broadcast .\broadcast.jl:950
 [12] materialize
    @ .\broadcast.jl:903 [inlined]
 [13] update_widths!(row::DataRow{AsciiTable})
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\datarow.jl:288
 [14] DataRow(data::Vector{…}, align::String, colwidths::Vector{…}, print_underlines::Vector{…}, render::AsciiTable; combine_equals::Bool)
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\datarow.jl:110
 [15] push_DataRow!(data::Vector{…}, vals::Vector{…}, align::String, colwidths::Vector{…}, print_underlines::Bool, render::AsciiTable; combine_equals::Bool)
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\regtable.jl:846
 [16] push_DataRow!(data::Vector{…}, vals::Vector{…}, align::String, colwidths::Vector{…}, print_underlines::Bool, render::AsciiTable)
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\regtable.jl:819
 [17] push_DataRow!(::Vector{DataRow{AsciiTable}}, ::Matrix{Any}, ::String, ::Vararg{Any}; vargs::@Kwargs{})
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\regtable.jl:810
 [18] push_DataRow!
    @ C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\regtable.jl:808 [inlined]
 [19] regtable(rrs::myresult; renderSettings::Nothing, render::AsciiTable, keep::Vector{…}, drop::Vector{…}, order::Vector{…}, fixedeffects::Vector{…}, labels::Dict{…}, align::Symbol, header_align::Symbol, below_statistic::Type, stat_below::Bool, regression_statistics::Vector{…}, groups::Nothing, print_depvar::Bool, number_regressions::Bool, print_estimator_section::Bool, print_fe_section::Bool, file::Nothing, transform_labels::Dict{…}, extralines::Nothing, section_order::Vector{…}, print_fe_suffix::Bool, print_control_indicator::Bool, standardize_coef::Bool, print_clusters::Bool, print_randomeffects::Bool, digits::Nothing, digits_stats::Nothing, estimformat::Nothing, statisticformat::Nothing, below_decoration::Nothing, number_regressions_decoration::Nothing, estim_decoration::Nothing, regressors::Nothing, use_relabeled_values::Bool, confint_level::Float64, kwargs::@Kwargs{})
    @ RegressionTables C:\Users\drood\.julia\packages\RegressionTables\iz9ba\src\regtable.jl:661
 [20] top-level scope
    @ Untitled-1:19
Some type information was truncated. Use `show(err)` to see complete types.

[junder873]

My mistake, should be format(RegressionTables.value(x); commas=true, precision=2). I updated the original.

[droodman]

Maybe what would be best, once the technical intricacies are worked out, is a higher-level layer, like esttab for estout or DataFramesMeta for DataFrames. I think something like Base.repr(render::AbstractRenderType, x::LogLikelihood; args...) = format(RegressionTables.value(x); commas=true, precision=2) is going to deter most users. Compare to stat(ll, fmt(a2)), the esttab equivalent.

[junder873]

Appreciate the feedback, and I am always happy to accept suggestions or pull requests.

I thought about this a little today and put together a pull request #161. I would really appreciate it if you could look at that and make any suggestions on the usability side of things (I think it is much closer to what Stata implements).

[droodman]

Oh interesting.
I also spent time thinking about this. In the example, I'm still rather confused:

• There are/would be two ways to specify a statistic and its format? I see format() clauses and a cfmt() clause. That in itself is confusing, as an example. That said, having additional ways to do the same thing should be OK, as long as one is emphasized in documentation and is easy to understand.
• I find it confusing that, if I understand right, the syntax in both cases has a formatting function wrapping a statistic. Psychologically, for me, this makes formatting primary and what's formatted secondary. But I think of the statistic as primary, so I want a syntax that centers the statistic and makes formatting secondary.
• In Stata, and in Julia if it becomes popular for econometrics, there are lots of estimation packages that produce lots of different return values. So a regression tables package should make it easy and intuitive to report any return value. Neither the user nor the author of the regression package should have to do anything special to make the package work with regtables(): that's the point of standards. I think in a Julia context, we can probably stipulate that the standard is that any return value must be made available through an exported function that takes the estimation result object as its sole argument. That function might be part of StatsAPI, or it might be unique to the estimation package. The user hoping to display the statistic using regtables() should not need to define any functions or types to report it and control its formatting. That's too confusing. Maybe that is already the case. My impression is that it's not, but I'm not sure because of my struggles to understand the package and its documentation.
• I don't understand where in the example the anonymous function is. And I think a regular user should probably not have to deal with that clever a concept.
• It is confusing to me (as a general user) to specify a statistic label with the pairs syntax (format(AdjR2; precision=3) => "My R2"). Statistics have several attributes. Why is the format specified one way and the label a quite different way?
• Stata's options syntax is flexible and gives users a good way to think about and invoke program features. The most analogous Julia constructs are keyword arguments and named tuples, which are of course closely related. As a user I want to translate Stata's stat(ll R2, fmt(a2 a3) label("Log likelihood" "R2")) to statistics=(items=[ll R2], fmt=["a2" "a3"], label=["Log likelihood" "R2"]) or statistics=(items=(ll, R2), fmt=("a2", "a3"), label=("Log likelihood", "R2")) or statistics=(ll=>(fmt="a2", label="Log likelihood"), R2=>(fmt="a3", label="R2")) or statistics=(ll=(fmt="a2", label="Log likelihood"), R2=(fmt="a3", label="R2")). I find such syntaxes less confusing than what I'm encountering in regtables(). All but the last would require ll and R2 to already be defined objects. They would probably be functions ll(::<:RegressionModel) -> <:Number and R2(::<:RegressionModel) -> <:Number, which it is reasonable to expect estimation packages to define and export.
• One could use macros to offer the user an even cleaner syntax, like @opts stat[ll R2; fmt[a2 a3] label["Log likelihood" "R2"]]. Meta.parse() can parse that line!

The issue here--how, in an interactive programming environment, to accept complex user options in a way that is flexible, easy to understand and remember, yet syntactically valid--goes beyond regtables(). Maybe there is an opportunity to define a stand-alone package for this purpose. It would parse options, accept and enforce type requirements, automatically provide reasonable error messages, allow customization of validation and error messages such as through stored @assert statements, could accept abbreviated option names, and could even include a structure to automate/require documentation--though how this would feed into Documenter.jl, I don't know. A huge problem in Julia, in my view, is the lack of package documentation that just lists all the options--what they accept, what they do--in a thorough and standardized way. Contrast that with Stata or Python.

[junder873]

If we could continue this conversation on the pull request (to keep discussions more contained), that would be great.