# Automatic saving of workspace variables for DrWatson.jl

The Julia package DrWatson.jl is aimed at scientific computing and data analysis. It describes itself as "scientific project assistant software". I recently decided giving it a try and like the functionality so far.

Useful features, in my opinion, include the safesave method, which saves dictionaries to JLD files while making sure no data is lost, and the @tag! macro, which adds key-value pairs to a dictionary that include, e.g., the current git commit hash, as well as a warning, together with a "gitpatch" showing the differences, in case of a dirty repository. It also provides tools to collect the results from all these files back together into DataFrame objects.

I use scripts that have a lot of adjustable parameters and need to run on cluster machines for days to produce output. Shortly after starting a given simulation, I want to be as sure as possible that (assuming the simulation itself runs fine) my program will not crash in the end when trying to save stuff, and that it's easy to keep track of where the result files went, when looking at the console output of a run.

My current workflow is tending into the direction of using DrWatson to just carelessly dump all input parameters and all results of my simulations into .jld files as Dict objects. It may be better to use some kind of JSON-based format to save all simulation parameters. However, I have a lot of different simulations that have mostly disjoint sets of input parameters and results. Overall, putting in the time to develop some kind of system to save all the metadata seems like a major distraction from my actual work (That's what she said while writing a CodeReview post about tools to simplify and automate workflow...).

## The issue

While DrWatson seems useful, when adopting it in my code-base, I ended up producing insane amounts of boilerplate code. The repository containing the simulation scripts doesn't contain any of the simulation code and refers to other repositories (Julia packages). Thus, my "simulations" project consists of many completely independent and disjoint scripts, each of which looked like this (notice the boilerplate):

using DrWatson
@quickactivate "PROJECTNAME"

println(repeat("-", 80))
println(repeat("-", 10), " Script $(@__FILE__)") println(repeat("-", 80)) using SparseArrays using JLD using Pkg @info("Status of custom simulation packages used:") Pkg.status("<SOMESTUFF>") # ... function main() p = 32 # blahblahblah q = 4 # ... # many more parameters... # some of these parameters get organized into struct's simul_params = DrWatson.@strdict p q # a LOT more stuff. # Multiple lines of listed parameters, easy to forget something... @tag!(simul_params) # add git commit information, etc. output_file_name = DrWatson.savename(<important parameters>, ".jld") println("Output file complete results: '$output_file_name'")

# ... simulations can take days to complete
# some of the simulations actually internally save intermediate results
# to paths that are part of the parameters.
result1 = external_library.some_simulation(parameters)
result2 = external_library.other_simulation(result1, other_parameters)

simul_params["result1"] = result1
simul_params["result2"] = result2
# ...
# you get the idea

output_file_path = DrWatson.datadir("sims", "script_name", filename), simul_params)
DrWatson.safesave(output_file_path, simul_params)
end

main()


## Proposed solution

What I ended up doing is a bit similar to this codereview post (I actually don't fully understand what is being done there). However, using DrWatson I get a lot more metadata and my approach seems much more radical in that I save all available variables. Perhaps an approach like the linked post is actually objectively better, I'm not sure how much the dangers/downsides of saving absolutely everything will come back to bite me.

In any case, the below code defines macros that either return as a Dict (or save directly to a JLD file) all global and local variables (except functions, modules, variables with special names, etc.). This is done by using Base.@local to get local variables in the caller's scope and getfield to get global variables of the calling module. The use of getfield is inspired from the source of JLD2.jl.

"""
This file is to be include'd into all simulation scripts.
It makes sure that the DrWatson package and its required imports are available.
"""

using DrWatson
@quickactivate "LDPC4QKD"

using SparseArrays
using JLD

using Pkg

"""
Provides macros for obtaining Dict's of selected local (potentially also global) variables.
"""
module ScopeCrawler

using DrWatson
using SparseArrays
using JLD

export @all_vars_dict
export @safesave_vars

"""
macro safesave_vars(out_path="", save_globals=true)

Saves the outputs of @all_vars_dict to a JLD file as a Dict{String, Any}.
"""
macro safesave_vars(out_path="", save_globals=true)
return quote
output_file_path = $(esc(out_path)) # auto-generate path if none given. if output_file_path == "" output_file_path = datadir("autoname", "$(basename(string(__source__.file)))_line$(__source__.line)_$(__module__).jld")
@warn("@safesave_locals received empty destination path from $(__source__.file). Auto-generated path: '$output_file_path'")
end

# make sure the file path has '.jld' extension.
if length(output_file_path) < 4 || output_file_path[end - 3:end] != ".jld"
@warn("Appending extension '.jld' to output file path '$output_file_path'.") output_file_path *= ".jld" end results = @all_vars_dict($(esc(save_globals)))
DrWatson.@tag!(results)

if isfile(output_file_path)
@warn("The requested output path $output_file_path already exists. Final path is chosen by DrWatson.safesave and will (probably) be$(abspath(DrWatson.recursively_clear_path(output_file_path)))")
else
@info("Final output file destination: $(abspath(output_file_path))") end DrWatson.safesave( output_file_path, results ) end end """ macro all_vars_dict(save_globals=true) Returns all local variables (with reasonable names, no functions and no modules) in the caller scope and global variables of the module in a Dict{String, Any} (with some additional meta-information). Note: this is slow and not optimized. Meant to be called rarely! TODO: test properly TODO specify additional information """ macro all_vars_dict(save_globals=true) # Note: output_file_path should evaluate to a String. sourcefile = "unknown" sourceline = "unknown" try sourcefile = basename(string(__source__.file)) sourceline = __source__.line catch e @warn("macro received invalid __source__:$e")
end

try
catch e
@warn("Failed to privide additional info in safesave_locals. Reason: $e") end # helper functions symbol_dict_to_strdict(dict::Dict{Symbol,T} where T) = Dict(string(key) => val for (key, val) in dict) function combine_conflicting_keys(x, y) # when globals and conflict, resolve @warn("Name conflict found between global/local variables while saving:$x vs $y.") return Dict("local" => x, "global" => y) end # Generate expression. # Basic idea: # Evaluate Base.@locals and function get_globals_dict in the scope of the caller. return quote globals_dict = Dict{String, Any}() if$(esc(save_globals))
globals_dict = $get_globals_dict(@__MODULE__) end merge($combine_conflicting_keys,
# Any makes sure the right method is for merge is called:
Dict{String, Any}(),
$symbol_dict_to_strdict(Base.@locals),$additional_info,
globals_dict
)
end
end

"""
function get_globals_dict(m::Module)

Returns a Dict{String, Any} containing global fields of a module that are likely to be user-defined variables.
Ignores modules, functions and non-standard names.

Note: inspired from JLD2.jl,
"""
function get_globals_dict(m::Module)
results = Dict{String,Any}()

try
for vname in names(m; all=true)
s = string(vname)
if (!occursin(r"^_+[0-9]*$", s) # skip IJulia history vars && !occursin(r"[@#$%^&*()?|\\/,.<>]", s)) # skip variables with special characters
v = getfield(m, vname)
if !isa(v, Module) && !isa(v, Function)
try
results[s] = v
catch e
if isa(e, PointerException)
@warn("Saving globals, skipping $vname because it contains a pointer.") else @warn("Saving globals, skipping$vname because $e") end end end end end catch e @warn("Saving globals failed because:$e")
end
return results
end

end # module ScopeCrawler

"""
poor man's unit tests for the above
Note: running this test at the start of each job using the functionality makes sense
because if this stuff crashes at the end of a simulation, all the work is lost.
"""
module TestingScopeCrawler
using Test
using ..ScopeCrawler

using JLD

tmptmp_global_var_for_test = "tmptmp_global_var_for_test"

@testset "testing @all_vars_dict" begin
local_var = 3
othervar = "asdf"

@testset "no args" begin
allvarsdict = @all_vars_dict
@test allvarsdict["local_var"] == 3
@test allvarsdict["othervar"] == "asdf"
@test allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test"
end
@testset "arg: true" begin
allvarsdict = @all_vars_dict true
@test allvarsdict["local_var"] == 3
@test allvarsdict["othervar"] == "asdf"
@test allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test"
end
@testset "arg: false" begin
allvarsdict = @all_vars_dict false
@test allvarsdict["local_var"] == 3
@test allvarsdict["othervar"] == "asdf"
@test_throws(KeyError, allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test")
end
@testset "arg: variable name" begin
variable_containing_false = false
allvarsdict = @all_vars_dict variable_containing_false
@test allvarsdict["local_var"] == 3
@test allvarsdict["othervar"] == "asdf"
@test_throws(KeyError, allvarsdict["tmptmp_global_var_for_test"] == "tmptmp_global_var_for_test")
end
end

@testset "testing @safesave_locals" begin
try
local_var = 3
othervar = "asdf"

dest_path = "tmptmp_test_safesave_locals.jld"
@safesave_vars dest_path
@test isfile("tmptmp_test_safesave_locals.jld")

finally
rm("tmptmp_test_safesave_locals.jld")  # throws exception IOError if the file does not exist.
end
end

end  # module TestingScopeCrawler


## How to use:

With my boilerplate script, the code above simplifies to

include("dr_watson_boilerplate.jl")

function main()
p = 32  # blahblahblah
q = 4  # ...
# many more parameters...
# some of these parameters get organized into struct's

output_file_name = DrWatson.savename(<important parameters>, ".jld")
println("Output file complete results: '\$output_file_name'")

# ... simulations can take days to complete
# some of the simulations actually internally save intermediate results
# to paths that are part of the parameters.
result1 = external_library.some_simulation(parameters)
result2 = external_library.other_simulation(result1, other_parameters)

output_file_path = DrWatson.datadir("sims", "script_name", filename), simul_params)
ScopeCrawler.@safesave_vars output_file_path
end

main()


What do you think?