The function name in the docstring should match the name of the function (get_activation_fn_with_deriv). Change:

    >>> relu, relu_deriv = activation_fn_with_deriv('relu')

to:

    >>> relu, relu_deriv = get_activation_fn_with_deriv('relu')

Since all the function names in the available_fn list are already lower-case, there is no need to loop through them and call lower(). The raise statement can be simplified, also shortening a long line:

raise ValueError(f'Specified activation function not available. Choose from: {available_fn}')

You have nice consistency in how you declare your inner functions.

Here are some points to consider.

The function name in the docstring should match the name of the function (get_activation_fn_with_deriv). Change:

    >>> relu, relu_deriv = activation_fn_with_deriv('relu')

to:

    >>> relu, relu_deriv = get_activation_fn_with_deriv('relu')

Since all the function names in the available_fn list are already lower-case, there is no need to loop through them and call lower(). The raise statement can be simplified, also shortening a long line:

raise ValueError(f'Specified activation function not available. Choose from: {available_fn}')