1
\$\begingroup\$

Update

I have posted a sequel (here) with my prototype for the "missing piece": the combination and propagation of variable names, especially within nested c(...) syntax.


Background

After wrangling one too many lists of column names, I sought an analog in M for tidyselect in R. I call it Selex, and its purposes are threefold:

  1. replicate the ergonomic syntax for selecting and operating on sets of fields; and
  2. extend this syntax to any M object with named fields; and
  3. avoid redundant calls to (say) Table.ColumnNames(), which decimate performance when those names must be (re)loaded from (say) an external source.

Approach

To avoid such redundancy, the field names must be called only once, when the syntax is evaluated in their context. This must apply to arbitrarily complex syntax, like "the complement to the union of fields like a* with fields like *z":

# A tidyselection in R.
!(starts_with("a") | ends_with("z"))
// The equivalent with Selex.
Slx_Not(Slx_Or(Slx_Starts("a"), Slx_Ends("z")))

I exploit functions as closures to achieve the Slx_*() family, which are essentially function factories: they generate functions of the form (vars) =>. When nested in complex syntax, these are ultimately resolved into a composite function of the same form (vars) =>. It is ultimately this composite function that operates once on the field names, to yield a subset of them.

The underlying set operations are outsourced to the Vars_*() family. These functions are simpler in nature: they operate on sets of field names (or positions), and return a list of names.

Overview

Here is tabulated the Selex syntax itself, including

  • its Slx_*() functions; and
  • its underlying Vars_*() implementations; and
  • its symbolic shorthands ("aliases"); and
  • its analogous language in tidyselect.
Syntax Description Alias Implementation Analog
Literals
{...} Combination of selections. c(...)
"var_1" Field named var_1. var_1
0 First field. 1
k k + 1th field. k
-1 Last field. last_col()
-k kth field from the end. last_col(offset = k - 1)
Helpers
Slx_Every() Every field in the object. Vars_Every() everything()
Slx_At() Fields by name or position. Vars_At()
Slx_Any() Any such fields that exist. @ Vars_Any() any_of()
Slx_All() All such fields, or error. Vars_All() all_of()
Slx_With() Fields whose names meet a condition. Vars_With()
Slx_Where() Fields whose values meet a condition. Vars_Where() where()
Slx_Right() Fields to the right of a field. > Vars_Right()
Slx_Left() Fields to the left of a field. < Vars_Left()
Slx_Between() Fields between two fields. : Vars_Between() :
Slx_Starts() Fields starting with a prefix. [ Vars_Starts() starts_with()
Slx_Ends() Fields ending with a suffix. ] Vars_Ends() ends_with()
Slx_Contains() Fields containing a substring. * Vars_Contains() contains()
Slx_Matches() Fields matching a regular expression. ? Vars_Matches() matches()
Slx_Seq() Fields following a numeric sequence. # Vars_Seq() num_range()
Operations
Slx_Not() The complement of a selection. ! Vars_Not() !
Slx_And() The intersection of selections. & Vars_And() &
Slx_Or() The union of selections. | Vars_Or() |
Slx_Xor() The exclusive union of selections. ^ Vars_Xor()
Slx_Diff() The difference of two selections. - Vars_Diff()

API

The interface exposed to users.

"Selexion"

Apply the special selex syntax to an object.

Data_Selex()

Select a subset (selex) of fields from an object.

let Data_Selex = (
    data as any,
    selex as any
    // optional options as nullable record
) as any =>
    let
        fields = Data_Resolve(data, selex), // options
        result = if fields = null then
            null
        else
            Data_SelectFields(data, fields)
    in
        result
in
    Data_Selex

Data_Resolve()

Resolve selex syntax into subset of field names from data.

let Data_Resolve = (
    data as any,
    selex as any
    // optional options as nullable record
) as nullable list =>
    let
        vars = List.Buffer(Data_FieldNames(data)),
        // opts = Record_Buffer(options),
        
        result = Vars_Resolve(vars, selex) // opts
    in
        result
in
    Data_Resolve

Generics

Manipulate fielded objects (tables or records) with generic operations.

Data_SelectFields()

Select a subset (list) of fields.

let Data_SelectFields = (
    data as any,
    fields as list
) as any =>
    if Value.Is(data, type table) then
        Table.SelectColumns(data, fields)
    else if Value.Is(data, type record) then
        Record.SelectFields(data, fields)
    else
        null
in
    Data_SelectFields

Data_FieldNames()

Extract all field names.

let Data_FieldNames = (
    data as any
) as nullable list =>
    if Value.Is(data, type table) then
        Table.ColumnNames(data)
    else if Value.Is(data, type record) then
        Record.FieldNames(data)
    else
        null
in
    Data_FieldNames

Data_FieldValues()

Extract all field values.

let Data_FieldValues = (
    data as any
) as nullable list =>
    if Value.Is(data, type table) then
        Table.ToColumns(data)
    else if Value.Is(data, type record) then
        Record.FieldValues(data)
    else
        null
in
    Data_FieldValues

Data_Field()

Extract a single field value.

let Data_Field = (
    data as any,
    field as text
) as any =>
    if Value.Is(data, type table) then
        Table.Column(data, field)
    else if Value.Is(data, type record) then
        Record.Field(data, field)
    else
        null
in
    Data_Field

Data_FieldOrDefault()

Extract a single field value, and default in its absence.

let Data_FieldOrDefault = (
    data as any,
    field as text,
    optional defaultValue as any
) as any =>
    try Data_Field(data, field) otherwise defaultValue
in
    Data_FieldOrDefault

Data_RenameFields()

Rename a subset of fields.

let Data_RenameFields = (
    data as any,
    renames as list,
    optional missingField as nullable number
) as any =>
    if Value.Is(data, type table) then
        Table.RenameColumns(data, renames, missingField)
    else if Value.Is(data, type record) then
        Record.RenameFields(data, renames, missingField)
    else
        null
in
    Data_RenameFields

Evaluation

Use data as an environment for evaluation.

Data_Mask()

Create an environment (record) from fielded objects, where later objects "mask" the earlier: if their field names collide, the later value takes precedence.

let Data_Mask = (
    masks as list
) as record =>
    Record_Mask(List.Transform(masks, Data_AsRecord))
in
    Data_Mask

Data_Evaluate()

Evaluate an M expression in the context of data, which masks any environment(s) supplied as further context.

let Data_Evaluate = (
    document as text,
    data as any,
    optional context as any
) as any =>
    let
        cxt = if Value.Is(context, type list) then
            context
        else
            {context},
        masks = List.Combine({cxt, {data}}),
        envir = Data_Mask(masks),
        result = Expression.Evaluate(document, envir)
    in
        result
in
    Data_Evaluate

Selex Helpers

Syntax for specifying fields that meet special criteria.

Slx_Every()

The "universe" of every field in the data.

let Slx_Every = (
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Every, vars, { // options
    })
in
    Slx_Every

Slx_At()

Field(s) specified by name or position. Here negatives (like -3) indicate positions (like 3rd) from the end (namely -1).

let Slx_At = (
    at as any,
    // optional unique as nullable logical,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_At, vars, {at // unique, options
    })
in
    Slx_At

Slx_Any()

The subset of such fields that actually exist.

let Slx_Any = (
    at as any,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Any, vars, {at // options
    })
in
    Slx_Any

Slx_All()

All such fields, or an error if any do not exist.

let Slx_All = (
    at as any,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_All, vars, {at // options
    })
in
    Slx_All

Slx_With()

Fields whose names meet a condition.

let Slx_With = (
    condition as function,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_With, vars, {condition // options
    })
in
    Slx_With

Slx_Where()

Fields whose values meet a condition.

let Slx_Where = (
    condition as function,
    data as any,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Where, vars, {condition, data // options
    })
in
    Slx_Where

Slx_Right ()

The count fields moving rightwards from from.

let Slx_Right = (
    optional from as any,
    optional count as nullable number,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Right, vars, {from, count // options
    })
in
    Slx_Right

Slx_Left()

The count fields moving leftwards from from.

let Slx_Left = (
    optional from as any,
    optional count as nullable number,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Left, vars, {from, count // options
    })
in
    Slx_Left

Slx_Between()

Fields between from and to.

let Slx_Between = (
    optional from as any,
    optional to as any,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Between, vars, {from, to // options
    })
in
    Slx_Between

Slx_Starts()

Fields prefixed by match.

let Slx_Starts = (
    match as text,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Starts, vars, {match // options
    })
in
    Slx_Starts

Slx_Ends()

Fields suffixed by match.

let Slx_Ends = (
    match as text,
    // options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Ends, vars, {match // options
    })
in
    Slx_Ends

Slx_Contains()

Fields containing match.

let Slx_Contains = (
    match as text,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Contains, vars, {match // options
    })
in
    Slx_Contains

Slx_Matches()

Fields matching the pattern match, where regex flags are specified like [ignoreCase = true, ...].

let Slx_Matches = (
    match as text,
    optional flags as nullable record,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Matches, vars, {match, flags // options
    })
in
    Slx_Matches

Slx_Seq()

Fields following a numeric sequence, padded by 0s to fit width.

let Slx_Seq = (
    sequence as list,
    optional prefix as nullable text,
    optional suffix as nullable text,
    optional width as nullable number,
    // optional unique as nullable logical,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    Slx_Helper(Vars_Seq, vars, {sequence, prefix, suffix, width // unique, options
    })
in
    Slx_Seq

Selex Operations

Syntax for operating on (sets of) fields, which are specified by Slx_*() helpers or simply by name or position.

Slx_Not()

The complement of selex within that universe.

let Slx_Not = (
    selex as any,
    // optional options as nullable record,
    optional vars as nullable list
) as function =>
    let override = vars
    in (vars as list) as nullable list =>
        Vars_Not(
            Value_IfNull(override, vars),
            Slx_Eval(selex, Value_IfNull(override, vars), false) // options
            // options
        )
in
    Slx_Not

Slx_And()

The intersection of several selexes.

let Slx_And = (
    selexes as list
    // optional options as nullable record
) as function =>
    (vars as list) as nullable list =>
        Vars_And(
            Slx_Eval(selexes, vars, true) // options
            // options
        )
in
    Slx_And

Slx_Or()

The union of several selexes.

let Slx_Or = (
    selexes as list
    // optional options as nullable record
) as function =>
    (vars as list) as nullable list =>
        Vars_Or(
            Slx_Eval(selexes, vars, true) // options
            // options
        )
in
    Slx_Or

Slx_Xor()

The "exclusive union" (XOR) of several selexes.

let Slx_Xor = (
    selexes as list
    // optional options as nullable record
) as function =>
    (vars as list) as nullable list =>
        Vars_Xor(
            Slx_Eval(selexes, vars, true) // options
            // options
        )
in
    Slx_Xor

Slx_Diff()

The (set) difference of two selexes.

let Slx_Diff = (
    selex1 as any,
    selex2 as any
    // optional options as nullable record
) as function =>
    (vars as list) as nullable list =>
        Vars_Diff(
            Slx_Eval(selex1, vars, false), // options
            Slx_Eval(selex2, vars, false) // options
            // options
        )
in
    Slx_Diff

Aliases

Symbolic shorthand for some Slx_*(), with default settings.

@

For Slx_Any().

let #"@" = (at as any) as function =>
    Slx_Any(at)
in #"@"

>

For Slx_Right().

let #">" = (optional from as any, optional count as nullable number) as function =>
    Slx_Right(from, count)
in #">"

<

For Slx_Left().

let #"<" = (optional from as any, optional count as nullable number) as function =>
    Slx_Left(from, count)
in #"<"

:

For Slx_Between().

let #":" = (optional from as any, optional to as any) as function =>
    Slx_Between(from, to)
in #":"

[

For Slx_Starts().

let #"[" = (match as text) as function =>
    Slx_Starts(match)
in #"["

]

For Slx_Ends().

let #"]" = (match as text) as function =>
    Slx_Ends(match)
in #"]"

*

For Slx_Contains().

let #"*" = (match as text) as function =>
    Slx_Contains(match)
in #"*"

?

For Slx_Matches().

let #"?" = (match as text, optional flags as nullable record) as function =>
    Slx_Matches(match, flags)
in #"?"

#

For Slx_Seq().

let #"#" = (
    sequence as list,
    optional prefix as nullable text,
    optional suffix as nullable text,
    optional width as nullable number
) as function =>
    Slx_Seq(sequence, prefix, suffix, width)
in #"#"

!

For Slx_Not() as a unary operator.

let #"!" = (x as any // optional options as nullable record
) as function =>
    Slx_Not(x) // options
in #"!"

&

For Slx_And() as a binary operator.

let #"&" = (x as any, y as any // optional options as nullable record
) as function =>
    Slx_And({x, y}) // options
in #"&"

|

For Slx_Or() as a binary operator.

let #"|" = (x as any, y as any // optional options as nullable record
) as function =>
    Slx_Or({x, y}) // options
in #"|"

^

For Slx_Xor() as a binary operator.

let #"^" = (x as any, y as any // optional options as nullable record
) as function =>
    Slx_Xor({x, y}) // options
in #"^"

-

For Slx_Diff() as a binary operator.

let #"-" = (x as any, y as any // optional options as nullable record
) as function =>
    Slx_Diff({x, y}) // options
in #"-"

Variables

Manipulate a set of field names.

Vars_Index()

Get the positions of fields within vars, specified by name or position.

let Vars_Index = (
    vars as list,
    set as any,
    optional backwards as nullable logical
    // optional options as nullable record
) as list =>
    let
        // Constants.
        DFL_BACKWARDS = false,
        
        // Defaults.
        bwd = Value_IfNull(backwards, DFL_BACKWARDS),
        
        // Workflow.
        count = if bwd then
            List.Count(vars)
        else
            null,
        result = if Value.Is(set, type list) then
            List.Transform(set, each Var_Index(vars, _, bwd, count))  // options
        else
            {Var_Index(vars, set, bwd, count)}  // options
    in
        result
in
    Vars_Index

Vars_Resolve()

Resolve selex syntax into a subset of vars.

let Vars_Resolve = (
    vars as list,
    selex as any
    // optional options as nullable record
) as nullable list =>
    let
        slx_fn = Slx_Resolve(selex), // options
        result = if slx_fn = null then
            null
        else
            slx_fn(vars)
    in
        result
in
    Vars_Resolve

Support

The hidden helpers that support the API.

// ################
// ## Aesthetics ##
// ################

// Convert n into "nth".
let Number_Ordinal = (
    number as nullable number
    // optional culture as nullable text
) as nullable text =>
    let
        // Constants.
        SFX_1 = "st",
        SFX_2 = "nd",
        SFX_3 = "rd",
        SFX_OTHER = "th",
        OTHER = {"11", "12", "13"},
        
        // Workflow.
        result = if number = null then
            null
        else
            let
                txt = Number.ToText(number), // null, culture
                l1 = Text.End(txt, 1),
                l2 = Text.End(txt, 2),
                
                sfx = if List.Contains(OTHER, l2) then SFX_OTHER
                else if l1 = "1" then SFX_1
                else if l1 = "2" then SFX_2
                else if l1 = "3" then SFX_3
                else SFX_OTHER,
                
                ord = txt & sfx
            in
                ord
    in
        result
in
    Number_Ordinal


// Format text as `M code` or an `#"identifier name"`.
let Text_AsCode = (
    text as nullable text,
    id as logical
) as text =>
    let
        // Constants.
        NULL_TXT = "",
        PFX = "`",
        SFX = "`",
        
        // Workflow.
        result = if text = null then
            NULL_TXT
        else
            let
                txt = if id then
                    Expression.Identifier(text)
                else
                    text,
                fmt = PFX & txt & SFX
            in
                fmt
    in
        result
in
    Text_AsCode


// Print a variable by name or position.
let Var_Print = (
    value as any
) as text =>
    let
        NULL_SYM = "null", // "NULL", "null"
        
        result = if Value.Is(value, type text) then
            Text_AsCode(value, true)
        else if Value.Is(value, type number) then
            Number_Ordinal(value)
        else
            NULL_SYM
    in
        result
in
    Var_Print


// #################
// ## Diagnostics ##
// #################

// Error thrown by `Vars_All()` for nonexistent fields.
let Err_All = (
    errs as list
) as record =>
    let
        // Constants.
        LST_SEP = ", ",
        
        // Workflow.
        flds = List.Transform(errs, each Var_Print(_)),
        lst = Text.Combine(flds, LST_SEP),
        
        ttl = "Nonexistent Fields",
        msg = "The " & Text_AsCode("*_All()", false) & " family prevents you selecting fields that do not exist in the data.  Use " & Text_AsCode("*_Any()", false) & " to avoid this.",
        dtl = "The following fields do not exist: " & lst & ".",
        
        result = Error.Record(ttl, msg, dtl)
    in
        result
in
    Err_All


// ################
// ## Evaluation ##
// ################

// Convert a fielded object into an environmental `record`.
let Data_AsRecord = (
    data as any
) as nullable record =>
    if data = null then
        null
    else if Value.Is(data, type record) then
        data
    else
        let
            flds = Data_FieldNames(data),
            vals = Data_FieldValues(data),
            rcd = Record.FromList(vals, flds)
        in
            rcd
in
    Data_AsRecord


// Mask a hierarchy of environmental `record`s.
let Record_Mask = (
    masks as list
) as record =>
    Record.Combine(List.RemoveNulls(masks))
in
    Record_Mask


// ####################
// ## Implementation ##
// ####################

// Generate variable names per a numeric sequence.
let Seq_Generate = (
    sequence as list,
    prefix as text,
    suffix as text,
    width as number,
    optional culture as nullable text
) as list =>
    let
        // Constants.
        PAD_CHAR = "0",
        
        num = List.Transform(sequence, each Number.From(_, culture)),
        int = List.Transform(num, Number_Truncate),
        txt = List.Transform(int, each Number.ToText(_, null, culture)),
        dgt = List.Transform(txt, each Text.PadStart(_, width, PAD_CHAR)),
        result = List.Transform(dgt, each prefix & _ & suffix)
    in
        result
in
    Seq_Generate


// Implements `Vars_Index()` for a single field.
let Var_Index = (
    vars as list,
    item as any,
    backwards as logical,
    optional count as nullable number
    // optional options as nullable record
) as nullable number =>
    let
        pos = Pos_From(vars, item), // options
        result = if backwards then
            pos - count
        else
            pos
    in
        result
in
    Var_Index


// Implements `Slx_Every()`.
let Vars_Every = (
    vars as list
    // optional options as nullable record
) as list =>
    vars
in
    Vars_Every


// Identify a single field specified by name or position.
let Var_At = (
    vars as list,
    at as any
    // optional options as nullable record
) as nullable text =>
    let
        pos = Pos_From(vars, at), // options
        result = try vars{pos} otherwise null
    in
        result
in
    Var_At


// Implements `Slx_At()`.
let Vars_At = (
    vars as list,
    at as any
    // optional unique as nullable logical,
    // optional options as nullable record
) as list =>
    let
        // Constants.
        // DFL_UNIQUE = true,
        
        // Defaults.
        unq = true, // Value_IfNull(unique, DFL_UNIQUE),
        
        // Workflow.
        result = if Value.Is(at, type list) then
            let
                names = List.Transform(at, each Var_At(vars, _)), // options
                set = if unq then
                    List.Distinct(names) // Opts_EquationCriteria(options)
                else
                    names
            in
                set
        else
            {Var_At(vars, at)} // options
    in
        result
in
    Vars_At


// Implements `Slx_Any()`.
let Vars_Any = (
    vars as list,
    at as any
    // optional options as nullable record
) as list =>
    List.RemoveNulls(Vars_At(vars, at)) // options
in
    Vars_Any


// Implements `Slx_All()`.
let Vars_All = (
    vars as list,
    at as any
    // optional options as nullable record
) as list =>
    let
        ats = if Value.Is(at, type list) then at else {at},
        map = List.Transform(ats, each Var_At(vars, _)), // options
        // map = Vars_At(vars, at, false, options)
        
        result = if List.Contains(map, null) then
            let
                fltr = List.Transform(map, each _ = null),
                err = List.Distinct(List_Filter(ats, fltr))
            in
                error Err_All(err)
        else
            Vars_At(vars, at) // true, options
            // map
    in
        result
in
    Vars_All


// Implements `Slx_With()`.
let Vars_With = (
    vars as list,
    condition as function
    // optional options as nullable record
) as list =>
    List.Select(vars, condition)
in
    Vars_With


// Implements `Slx_Where()`.
let Vars_Where = (
    vars as list,
    condition as function,
    data as any
    // optional options as nullable record
) as list =>
    List.Select(vars, each condition(Data_Field(data, _)))
    /* let
        data = Vars_ToData(vars),
        result = if data = null then
            null
        else
            List.Select(vars, each condition(Data_Field(data, _)))
    in
        result */
in
    Vars_Where


// Implements `Slx_Between()`.
let Vars_Between = (
    vars as list,
    from as any,
    to as any
    // optional options as nullable record
) as list =>
    let
        // Constants.
        DFL_FROM = 0,
        DFL_TO = -1,
        
        // Defaults.
        f = Value_IfNull(from, DFL_FROM),
        t = Value_IfNull(to, DFL_TO),
        
        // Workflow.
        pos = Pos_Between(vars, f, t), // options
        result = Vars_At(vars, pos) // false, options
    in
        result
in
    Vars_Between


// Implements `Slx_Right()`.
let Vars_Right = (
    vars as list,
    optional from as any,
    optional count as nullable number
    // optional options as nullable record
) as list =>
    let
        // Constants.
        DFL_FROM = 0,
        DFL_COUNT = 1,
        
        // Defaults.
        frm = Value_IfNull(from, DFL_FROM),
        cnt = Value_IfNull(count, DFL_COUNT),
        
        // Workflow.
        pos = Pos_From(vars, frm), // options
        result = List.Range(vars, pos, cnt)
    in
        result
in
    Vars_Right


// Implements `Slx_Left()`.
let Vars_Left = (
    vars as list,
    optional from as any,
    optional count as nullable number
    // optional options as nullable record
) as list =>
    let
        // Constants.
        DFL_FROM = -1,
        DFL_COUNT = 1,
        
        // Defaults.
        frm = Value_IfNull(from, DFL_FROM),
        cnt = Value_IfNull(count, DFL_COUNT),
        
        // Workflow.
        pos = Pos_From(vars, frm), // options
        untrim = List.Range(vars, 0, pos + 1),
        result = List.LastN(untrim, cnt)
    in
        result
in
    Vars_Left


// Implements `Slx_Starts()`.
let Vars_Starts = (
    vars as list,
    match as text
    // optional options as nullable record
) as list =>
    List.Select(vars, each Text.StartsWith(_, match)) // Opts_Comparer(options)
in
    Vars_Starts


// Implements `Slx_Ends()`.
let Vars_Ends = (
    vars as list,
    match as text
    // optional options as nullable record
) as list =>
    List.Select(vars, each Text.EndsWith(_, match)) // Opts_Comparer(options)
in
    Vars_Ends


// Implements `Slx_Contains()`.
let Vars_Contains = (
    vars as list,
    match as text
    // optional options as nullable record
) as list =>
    List.Select(vars, each Text.Contains(_, match)) // Opts_Comparer(options)
in
    Vars_Contains


// Implements `Slx_Matches()`.
let Vars_Matches = (
    vars as list,
    match as text,
    optional flags as nullable record
    // optional options as nullable record
) as list =>
    let
        // flags = Opts_Regex(options),
        tests = Rgx_Test(vars, match, flags),
        result = List_Filter(vars, tests)
    in
        result
in
    Vars_Matches


// Implements `Slx_Seq()`.
let Vars_Seq = (
    vars as list,
    sequence as list,
    optional prefix as nullable text,
    optional suffix as nullable text,
    optional width as nullable number
    // optional unique as nullable logical
    // optional options as nullable record
) as list =>
    let
        // Constants.
        DFL_PREFIX = "",
        DFL_SUFFIX = "",
        DFL_WIDTH = 0,
        // DFL_UNIQUE = true,
        PAD_CHAR = "0",
        
        // Defaults
        pfx = Value_IfNull(prefix, DFL_PREFIX),
        sfx = Value_IfNull(suffix, DFL_SUFFIX),
        wid = Value_IfNull(width, DFL_WIDTH),
        // unq = Value_IfNull(unique, DFL_UNIQUE),
        
        // Workflow.
        seq = Seq_Generate(sequence, pfx, sfx, wid), // Opts_Culture(options)
        result = Vars_Any(vars, seq) // unq, options
    in
        result
in
    Vars_Seq


// Implements `Slx_Not()`.
let Vars_Not = (
    vars as list,
    set as list
    // optional options as nullable record
) as list =>
    Vars_Diff(vars, set) // options
in
    Vars_Not


// Implements `Slx_And()`.
let Vars_And = (
    sets as list
    // optional options as nullable record
) as list =>
    List.Intersect(sets) // Opts_EquationCriteria(options)
in
    Vars_And


// Implements `Slx_Or()`.
let Vars_Or = (
    sets as list
    // optional options as nullable record
) as list =>
    List.Union(sets) // Opts_EquationCriteria(options)
in
    Vars_Or


// Implements `Slx_Xor()`.
let Vars_Xor = (
    sets as list
    // optional options as nullable record
) as list =>
    List.Accumulate(sets, {}, (state, current) => Vars_BinaryXor(state, current)) // options
in
    Vars_Xor


// Implements `Slx_Diff()`.
let Vars_Diff = (
    set1 as list,
    set2 as list
    // optional options as nullable record
) as list =>
    List.RemoveMatchingItems(set1, set2) // Opts_EquationCriteria(options)
in
    Vars_Diff


// Take the XOR for two sets of fields.
let Vars_BinaryXor = (
    set1 as list,
    set2 as list
    // optional options as nullable record
) as list =>
    Vars_Diff(
        Vars_Or({set1, set2}), // options
        Vars_And({set1, set2}) // options
        // options
    )
in
    Vars_BinaryXor


// ##############
// ## Position ##
// ##############

// Standardize a position as a list index: `0`-indexed and counting "forwards".
let Pos_Calc = (
    cnt as number,
    pos as number
) as nullable number =>
    let
        trn = Number_Truncate(pos),
        off = if trn < 0 then
            cnt + trn
        else
            trn,
        result = if off < 0 or off >= cnt then
            null
        else
            off
    in
        result
in
    Pos_Calc


// Locate a field in `vars` by name.
let Pos_FromName = (
    vars as list,
    name as text,
    optional equationCriteria as any
) as nullable number =>
    let
        pos = List.PositionOf(vars, name, 0, equationCriteria),
        result = if pos < 0 then
            null
        else
            pos
    in
        result
in
    Pos_FromName


// Locate a field by name or position.
let Pos_From = (
    vars as list,
    val as any
    // optional options as nullable record
) as nullable number =>
    if Value.Is(val, type number) then
        Pos_Calc(List.Count(vars), val)
    else if Value.Is(val, type text) then
        Pos_FromName(vars, val) // Opts_EquationCriteria(options)
    else
        null
in
    Pos_From


// Locate the _range_ of positions between two such fields.
let Pos_Between = (
    vars as list,
    from as any,
    to as any
    // optional options as nullable record
) as list =>
    let
        fPos = Pos_From(vars, from), // options
        tPos = Pos_From(vars, to), // options
        
        result = if fPos = null or tPos = null then
            {}
        else
            let
                start = 0,
                end = List.Count(vars) - 1,
                lPos = List.Min({fPos, tPos, end}),
                rPos = List.Max({start, fPos, tPos}),
                n = rPos - lPos + 1,
                pos = if fPos > tPos then
                    List.Numbers(fPos, n, -1)
                else
                    List.Numbers(fPos, n, 1)
            in
                pos
    in
        result
in
    Pos_Between


// ###########
// ## Regex ##
// ###########

// Test a `list` of strings against a regex `pattern`, as prototyped on Stack Overflow: https://stackoverflow.com/a/77791459.
let Rgx_Test = (
    texts as list,
    pattern as text,
    optional flags as nullable record
) as nullable list =>
    let
        // Constants.
        JS_NEW = "#(lf)",
        HTML_TAG = "<br>",
        
        // Workflow.
        result = if List.IsEmpty(texts) then
            {}
        else
            let
                strs = List.Transform(texts, Json_Stringify),
                ptn = Json_Stringify(pattern),
                flg = Json_Stringify(Rgx_Flags(flags)),
                tag = Json_Stringify(HTML_TAG),
                
                lines = List.Transform(strs, each
                    "document.write(tag + rgx.test(" & _ & "));"
                ),
                code = Text.Combine(lines, JS_NEW),
                
                html = "<html><body><script>
                    var ptn = " & ptn & ";
                    var flg = " & flg & ";
                    var tag = " & tag & ";
                    var rgx = new RegExp(ptn, flg);
                    " & code & "
                </script></body></html>",
                
                page = Web.Page(html),
                out = page{0}[Data]{[
                    Name = "HTML"
                ]}[Children]{[
                    Name = "BODY"
                ]}[Children],
                exprs = Table.SelectRows(out, each [Kind] = "Text")[Text],
                
                lgls = try List.Transform(exprs, Logical.FromText) otherwise null
            in
                lgls
    in
        result
in
    Rgx_Test


// Interpret a `record` as a string of regex flags.
let Rgx_Flags = (
    optional flags as nullable record
) as text =>
    let
        // Constants.
        MOD_SEP = "",
        MOD_FLAGS = [
            hasIndices  = "d",
            global      = "g",
            ignoreCase  = "i",
            multiline   = "m",
            dotAll      = "s",
            unicode     = "u",
            unicodeSets = "v",
            sticky      = "y"
        ],
        
        // Defaults.
        flgs = if flags = null then [] else flags, // flgs = Value_IfNull(flags, [])
        
        // Workflow.
        flds = List.Select(
            Record.FieldNames(MOD_FLAGS),
            each try Record.Field(flgs, _) = true otherwise false
        ),
        chrs = Record.FieldValues(Record.SelectFields(MOD_FLAGS, flds)),
        
        result = Text.Combine(chrs, MOD_SEP)
    in
        result
in
    Rgx_Flags


// Interpret an M object as a JSON string.
let Json_Stringify = (
    value as any,
    optional encoding as nullable number
) as nullable text =>
    Text.FromBinary(Json.FromValue(value, encoding), encoding)
in
    Json_Stringify


// ##############
// ## Selexion ##
// ##############

// Evaluate `selex`es in preparation for set operations.
let Slx_Eval = (
    selex as any,
    vars as list,
    plural as logical
    // optional options as nullable record
) as list =>
    if plural then
        List.Transform(selex, each Vars_Resolve(vars, _)) // options
    else
        Vars_Resolve(vars, selex) // options
in
    Slx_Eval


// Resolve a `selex` statement into a single `function`.
let Slx_Resolve = (
    selex as any
    // optional options as nullable record
) as nullable function =>
    if Value.Is(selex, type number) or Value.Is(selex, type text) then
        Slx_At(selex) // options
    else if Value.Is(selex, type function) then
        selex
    else if Value.Is(selex, type list) then
        Slx_Or(selex) // options
    else
        null
in
    Slx_Resolve


// Convert a `Vars_*()` into a `Slx_*()` helper, and optionally override its `vars`.
let Slx_Helper = (
    vars_fn as any,
    vars as nullable list,
    args as list
) as nullable function =>
    let override = vars
    in (vars as list) as list =>
        Function.Invoke(vars_fn, List.Combine({{Value_IfNull(override, vars)}, args}))
in
    Slx_Helper


// ###################
// ## Miscellaneous ##
// ###################

// Filter `list` by a `list` of `logical`s.
let List_Filter = (
    list as list,
    filter as list
) as list =>
    let
        pos = List.Select(List.Positions(list), each try filter{_} = true otherwise false),
        result = List.Transform(pos, each list{_})
    in
        result
in
    List_Filter


// Remove any decimal to leave an integer.
let Number_Truncate = (
    number as nullable number
) as nullable number =>
    Number.IntegerDivide(number, 1)
in
    Number_Truncate


// Return the `value` or a default when `null`.  Useful for default arguments.
let Value_IfNull = (
    value as any,
    default as any
) as any =>
    if value = null then
        default
    else
        value
in
    Value_IfNull
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Your question is rather long! Long questions are more likely to scare off reviewers -- as such I'd recommend you get as close to one code block as possible. Yeah the length would be the same, but the apparent length would be vastly different. \$\endgroup\$
    – Peilonrayz
    Commented Jan 18 at 4:00
  • \$\begingroup\$ @Peilonrayz I appreciate the advice, and I might consolidate the Support functions into a single code block. But for the sake of clarity, I think it's necessary to separate the API functions from one another, so they don't get "lost in the sauce". This also permits individual explanations of their purposes, which is essential for reviewing its functionality. Finally, Power Query is rather annoying when it comes to importing code libraries: some reviewers might have to paste the code separately for each function, in order to populate and test the functions in their environment. \$\endgroup\$
    – Greg
    Commented Jan 18 at 4:47
  • \$\begingroup\$ @Greg Consider consolidating Slx_With() and Slx_Where() into a Slx_Where*() family: Slx_WhereName(), Slx_WhereValue(), and possibly Slx_WhereIndex(). \$\endgroup\$
    – Greg
    Commented May 29 at 18:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.