0
\$\begingroup\$

The Julia package DrWatson.jl is aimed at scientific computing and data analysis. It describes itself as "scientific project assistant software". I recently decided giving it a try and like the functionality so far.

Useful features, in my opinion, include the safesave method, which saves dictionaries to JLD files while making sure no data is lost, and the @tag! macro, which adds key-value pairs to a dictionary that include, e.g., the current git commit hash, as well as a warning, together with a "gitpatch" showing the differences, in case of a dirty repository. It also provides tools to collect the results from all these files back together into DataFrame objects.

I use scripts that have a lot of adjustable parameters and need to run on cluster machines for days to produce output. Shortly after starting a given simulation, I want to be as sure as possible that (assuming the simulation itself runs fine) my program will not crash in the end when trying to save stuff, and that it's easy to keep track of where the result files went, when looking at the console output of a run.

My current workflow is tending into the direction of using DrWatson to just carelessly dump all input parameters and all results of my simulations into .jld files as Dict objects. It may be better to use some kind of JSON-based format to save all simulation parameters. However, I have a lot of different simulations that have mostly disjoint sets of input parameters and results. Overall, putting in the time to develop some kind of system to save all the metadata seems like a major distraction from my actual work (That's what she said while writing a CodeReview post about tools to simplify and automate workflow...).

The issue

While DrWatson seems useful, when adopting it in my code-base, I ended up producing insane amounts of boilerplate code. The repository containing the simulation scripts doesn't contain any of the simulation code and refers to other repositories (Julia packages). Thus, my "simulations" project consists of many completely independent and disjoint scripts, each of which looked like this (notice the boilerplate):

using DrWatson
@quickactivate "PROJECTNAME"

println(repeat("-", 80))
println(repeat("-", 10), " Script $(@__FILE__)")
println(repeat("-", 80))

using SparseArrays
using JLD

using Pkg
@info("Status of custom simulation packages used:")
Pkg.status("<SOMESTUFF>")
# ...

function main()
    p = 32  # blahblahblah
    q = 4  # ...
    # many more parameters...
    # some of these parameters get organized into `struct`'s
    
    simul_params = DrWatson.@strdict p q # a LOT more stuff.
          # Multiple lines of listed parameters, easy to forget something...
    @tag!(simul_params) # add git commit information, etc.
    output_file_name = DrWatson.savename(<important parameters>, ".jld")
    println("Output file complete results: '$output_file_name'")

    # ... simulations can take days to complete
    # some of the simulations actually internally save intermediate results
    # to paths that are part of the parameters. 
    result1 = external_library.some_simulation(parameters)
    result2 = external_library.other_simulation(result1, other_parameters)
    
    simul_params["result1"] = result1
    simul_params["result2"] = result2
    # ...
    # you get the idea

    output_file_path = DrWatson.datadir("sims", "script_name", filename), simul_params)
    DrWatson.safesave(output_file_path, simul_params)
end


main()

Proposed solution

What I ended up doing is a bit similar to this codereview post (I actually don't fully understand what is being done there). However, using DrWatson I get a lot more metadata and my approach seems much more radical in that I save all available variables. Perhaps an approach like the linked post is actually objectively better, I'm not sure how much the dangers/downsides of saving absolutely everything will come back to bite me.

In any case, the below code defines macros that either return as a Dict (or save directly to a JLD file) all global and local variables (except functions, modules, variables with special names, etc.). This is done by using Base.@local to get local variables in the caller's scope and getfield to get global variables of the calling module. The use of getfield is inspired from the source of JLD2.jl.

"""
This file is to be `include`'d into all simulation scripts.
It makes sure that the DrWatson package and its required imports are available.
"""

using DrWatson
@quickactivate "LDPC4QKD"

using SparseArrays
using JLD

using Pkg
@info("Status of RateAdaptiveLDPC used:")
Pkg.status("RateAdaptiveLDPC")



"""
Provides macros for obtaining `Dict`'s of selected local (potentially also global) variables.
"""
module ScopeCrawler

using DrWatson
using SparseArrays
using JLD

export @all_vars_dict
export @safesave_vars

"""
    macro safesave_vars(out_path="", save_globals=true)

Saves the outputs of `@all_vars_dict` to a `JLD` file as a `Dict{String, Any}`.
"""
macro safesave_vars(out_path="", save_globals=true)
    return quote
        output_file_path = $(esc(out_path))

        # auto-generate path if none given.
        if output_file_path == ""
            output_file_path = datadir("autoname", "$(basename(string(__source__.file)))_line$(__source__.line)_$(__module__).jld")
            @warn("@safesave_locals received empty destination path from $(__source__.file). Auto-generated path:
                    '$output_file_path'")
        end

        # make sure the file path has '.jld' extension.
        if length(output_file_path) < 4 || output_file_path[end - 3:end] != ".jld"
            @warn("Appending extension '.jld' to output file path '$output_file_path'.")
            output_file_path *= ".jld"
        end

        results = @all_vars_dict($(esc(save_globals)))
        DrWatson.@tag!(results)

        if isfile(output_file_path)
            @warn("The requested output path $output_file_path already exists. 
                Final path is chosen by `DrWatson.safesave` and will (probably) be
                $(abspath(DrWatson.recursively_clear_path(output_file_path)))")
        else
            @info("Final output file destination: $(abspath(output_file_path))")
        end
        DrWatson.safesave(
            output_file_path, 
            results
        )
    end
end


"""
    macro all_vars_dict(save_globals=true)

Returns all local variables (with reasonable names, no functions and no modules)
in the caller scope and global variables of the module in a `Dict{String, Any}`
(with some additional meta-information). 


Note: this is slow and not optimized. Meant to be called rarely!
TODO: test properly
TODO specify additional information
"""
macro all_vars_dict(save_globals=true)
    # Note: output_file_path should evaluate to a String.
    sourcefile = "unknown"
    sourceline = "unknown"
    try 
        sourcefile = basename(string(__source__.file))
        sourceline = __source__.line
    catch e
        @warn("macro received invalid __source__: $e")
    end

    # could add additional metadata that the macro has access to
    additional_info = Dict{String,Any}()
    try
        additional_info["#source#"] = repr(__source__)
        additional_info["#module#"] = repr(__module__)
        additional_info["#stacktrace#"] = stacktrace()
    catch e
        @warn("Failed to privide additional info in safesave_locals. Reason: $e")
    end

    # helper functions
    symbol_dict_to_strdict(dict::Dict{Symbol,T} where T) = Dict(string(key) => val for (key, val) in dict)

    function combine_conflicting_keys(x, y) # when globals and conflict, resolve
        @warn("Name conflict found between global/local variables while saving: $x vs $y.")
        return Dict("local" => x, "global" => y)
    end

    # Generate expression.
    # Basic idea: 
    # Evaluate `Base.@locals` and function `get_globals_dict` in the scope of the caller.
    return quote
        globals_dict = Dict{String, Any}()
        if $(esc(save_globals))
            globals_dict = $get_globals_dict(@__MODULE__)
        end

        merge($combine_conflicting_keys,
            # `Any` makes sure the right method is for `merge` is called:
            Dict{String, Any}(),
            $symbol_dict_to_strdict(Base.@locals),  
            $additional_info,
            globals_dict
        )
    end
end


"""
    function get_globals_dict(m::Module)

Returns a `Dict{String, Any}` containing global fields of a module that are likely to be user-defined variables.
Ignores modules, functions and non-standard names.

Note: inspired from JLD2.jl, 
see https://github.com/JuliaIO/JLD2.jl/blob/f7535ab19cb65e703ce4f1f2209ccd32e5b9287f/src/loadsave.jl#L78
"""
function get_globals_dict(m::Module)
    results = Dict{String,Any}()

    try
        for vname in names(m; all=true)
            s = string(vname)
            if (!occursin(r"^_+[0-9]*$", s) # skip IJulia history vars
                && !occursin(r"[@#$%^&*()?|\\/,.<>]", s)) # skip variables with special characters
                v = getfield(m, vname)
                if !isa(v, Module) && !isa(v, Function)
                    try
                        results[s] = v
                    catch e
                        if isa(e, PointerException)
                            @warn("Saving globals, skipping $vname because it contains a pointer.")
                        else
                            @warn("Saving globals, skipping $vname because $e")
                        end
                    end
                end
            end
        end
    catch e
        @warn("Saving globals failed because: $e")
    end
    return results
end

end # module ScopeCrawler



"""
poor man's unit tests for the above
Note: running this test at the start of each job using the functionality makes sense 
because if this stuff crashes at the end of a simulation, all the work is lost.
"""
module TestingScopeCrawler
using Test
using ..ScopeCrawler

using JLD

tmptmp_global_var_for_test = "tmptmp_global_var_for_test"

@testset "testing @all_vars_dict" begin
    local_var = 3
    othervar = "asdf"

    @testset "no args" begin
        allvarsdict = @all_vars_dict
        @test allvarsdict["local_var"] == 3
        @test allvarsdict["othervar"] == "asdf"
        @test allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test"
    end
    @testset "arg: true" begin
        allvarsdict = @all_vars_dict true
        @test allvarsdict["local_var"] == 3
        @test allvarsdict["othervar"] == "asdf"
        @test allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test"
    end
    @testset "arg: false" begin
        allvarsdict = @all_vars_dict false
        @test allvarsdict["local_var"] == 3
        @test allvarsdict["othervar"] == "asdf"
        @test_throws(KeyError, allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test")
    end
    @testset "arg: variable name" begin
        variable_containing_false = false
        allvarsdict = @all_vars_dict variable_containing_false
        @test allvarsdict["local_var"] == 3
        @test allvarsdict["othervar"] == "asdf"
        @test_throws(KeyError, allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test")
    end
end


@testset "testing @safesave_locals" begin
    try
        local_var = 3
        othervar = "asdf"

        dest_path = "tmptmp_test_safesave_locals.jld"
        @safesave_vars dest_path
        @test isfile("tmptmp_test_safesave_locals.jld")

        loadagain = load("tmptmp_test_safesave_locals.jld")
        @test loadagain["local_var"] == 3
        @test loadagain["othervar"] == "asdf"
        @test loadagain["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test"
    finally
        rm("tmptmp_test_safesave_locals.jld")  # throws exception `IOError` if the file does not exist.
    end
end

end  # module TestingScopeCrawler

How to use:

With my boilerplate script, the code above simplifies to

include("dr_watson_boilerplate.jl")

function main()
    p = 32  # blahblahblah
    q = 4  # ...
    # many more parameters...
    # some of these parameters get organized into `struct`'s
    
    output_file_name = DrWatson.savename(<important parameters>, ".jld")
    println("Output file complete results: '$output_file_name'")

    # ... simulations can take days to complete
    # some of the simulations actually internally save intermediate results
    # to paths that are part of the parameters. 
    result1 = external_library.some_simulation(parameters)
    result2 = external_library.other_simulation(result1, other_parameters)

    output_file_path = DrWatson.datadir("sims", "script_name", filename), simul_params)
    ScopeCrawler.@safesave_vars output_file_path
end


main()

What do you think?

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.