Working Neural Net from scratch

Question

Asking for a review of this neural network code.

I have officially finished my first neural net that works properly (by my standards right now). But I know there is more than likely some details I am missing or getting wrong that might be affecting the quality of the nets intelligence. I plan on making this a library soon (hence the options for different types of optimization, regularization and activation. I am not too worried about optimal performance but suggestions on improving performance would be much appreciated! I am not proficient in the use of pointers so I did avoid using them, but if I have to, I will.

It is quite a lot of code, but if anyone could just take a quick review, it would really help me in just understanding more. I am 4 months into coding now so Im still very new. (started with c++). So to clarify, I just want to know if anyone can spot any dire problems that will go unnoticed by the untrained eye, or just small improvements I could make. Very motivated to learn so any criticism welcome!
so heres the code: (If anyone finds this helpful for learning, more than welcome to use it)

#include <iostream>
#include <vector>
#include <iomanip>
#include <cmath>
#include <random>
#include <fstream>
#include <chrono>
#include <sstream>
#include <string>


double Relu(double &val)
{
    if (val < 0.0) return 0.01 * (exp(val) - 1.0);
    else return val;
}
double Reluderiv(double &val)
{
    if (val < 0.0) return Relu(val) + 0.01;
    else  return 1.0;
}
double tanhderiv(double& val)
{
    return 1.0 - (tanh(val) * tanh(val));
}
double randdist(double x, double y)
{
    return sqrt(2.0 / (x + y));
}
int randomt(int x, int y)
{
    std::random_device rd;
    std::mt19937 mt(rd());
    std::uniform_real_distribution<double> dist(x, y);
    return round(dist(mt));
}
double randomd(double x, double y)
{
    std::random_device rd;
    std::mt19937 mt(rd());
    std::uniform_real_distribution<double> dist(x, y);
    return dist(mt);
}
class INneuron
{
public:
    double val{};
    double VDW{};
    double SDW{};
    double VDB{};
    double SDB{};
    std::vector <double> weights{};
    std::vector <double> weightderivs{};
    std::vector <double> emavals{};
    std::vector <double> adamvals{};
};
class HIDneuron
{
public:
    double preactval{};
    double actval{};
    double actvalPD{};
    double preactvalPD{};
    double VDW{};
    double SDW{};
    double VDB{};
    double SDB{};
    std::vector <double> weights{};
    std::vector <double> weightderivs{};
    std::vector <double> emavals{};
    std::vector <double> adamvals{};
    double bias{};
    double biasderiv{};
    double biasema{};
    double biasadam{};
};
class OUTneuron
{
public:
    double preactval{};
    double actval{};
    double preactvalPD{};
    double bias{};
    double biasderiv{};
    double VDB{};
    double SDB{};
    double biasema{};
    double biasadam{};

};
class Net
{
public:
    Net(int netdimensions, int hidlayers, int hidneurons, int outneurons, int inneurons, double lambda, double alpha) : 
        NETDIMENSIONS(netdimensions), HIDLAYERS(hidlayers), HIDNEURONS(hidneurons), OUTNEURONS(outneurons), INNEURONS(inneurons), Lambda(lambda), Alpha(alpha) {}
    void defineoptimizer(const std::string& optimizer);
    void defineregularizer(const std::string& regularizer);
    void defineactivation(const std::string& activation);
    void definedescenttype(const std::string& descenttypeS);
    bool Feedforward(const std::vector <double>& invec);
    void Backprop(const std::vector <double>& targets);
    void Updateweights();
    void printvalues(double totalloss);
    void Initweights();
    void softmax();
    double regularize(double weight);
    double lossfunc(const std::vector <double>& target);
    void calcema(const size_t& Layernum, const size_t& neuron, const size_t& weight, const std::string& layer, const std::string& BorW);
    void calcadam(const size_t& Layernum, const size_t& neuron,const size_t& weight, const std::string& layer, const std::string& BorW);
    double iter{};

private:
    INneuron Inn;
    HIDneuron Hidn;
    OUTneuron Outn;
    std::vector <std::vector <HIDneuron>> Hidlayers{};
    std::vector <INneuron> Inlayer{};
    std::vector <OUTneuron> Outlayer{};
    size_t NETDIMENSIONS{};
    size_t HIDLAYERS{};
    size_t HIDNEURONS{};
    size_t OUTNEURONS{};
    size_t INNEURONS{};
    double Lambda{};
    double Alpha{};
    double loss{};
    int optimizerformula{};
    int regularizertype{};
    int activationtype{};
};
void Net::defineoptimizer(const std::string& optimizer)
{
    if (optimizer == "ExpAvrg")
    {
        optimizerformula = 1;
    }
    else if (optimizer == "SGD")
    {
        optimizerformula = 2;
    }
    else if (optimizer == "Adam")
    {
        optimizerformula = 3;
    }
    else {
        std::cout << "no optimizer matching description" << '\n';
        abort();
    }
}
void Net::defineactivation(const std::string& activation)
{
    if (activation == "Relu")activationtype = 1;
    else if (activation == "Tanh") activationtype = 2;
    else {
        std::cout << "no activation determined" << '\n'; abort();
    }
}
void Net::defineregularizer(const std::string& regularizer)
{
    if (regularizer == "L1")
    {
        regularizertype = 1;
    }
    else if (regularizer == "L2")
    {
        regularizertype = 2;
    }
    else std::cout << "no regularizer determined" << '\n';
}
void Net::definedescenttype(const std::string& descenttypeS)
{
    if (descenttypeS == "SGD")
    {
        descenttype = 1;
    }
    else if (descenttypeS == "MiniBatch")
    {
        descenttype = 2;
    }
    else std::cout << "No descenttype chose, default SGD set" << '\n';
}
double Net::regularize(double weight)
{
    if (regularizertype == 1)
    {
        double absval{ weight };
        if (absval > 0.0) return 1.0;
        else if (absval < 0.0) return -1.0;
        else if (absval == 0.0) return 0.0;
        else return 2;
    }
    else if (regularizertype == 2)
    {
        double absval{};
        if (weight < 0.0) absval = weight * -1.0;
        else absval = weight;
        return (2.0 * absval);
    }
    else { std::cout << "no regularizer recognized" << '\n'; abort(); }

}
void Net::softmax()
{
    double sum{};
    for (size_t Osize = 0; Osize < Outlayer.size(); Osize++)
    {
        sum += exp(Outlayer[Osize].preactval);
    }
    for (size_t Osize = 0; Osize < Outlayer.size(); Osize++)
    {
        Outlayer[Osize].actval = exp(Outlayer[Osize].preactval) / sum;
    }
}
void Net::Initweights()
{
    Hidlayers.reserve(HIDLAYERS);   
    Inlayer.reserve(INNEURONS);
    Outlayer.reserve(OUTNEURONS);
    unsigned seed = std::chrono::system_clock::now().time_since_epoch().count();
    std::default_random_engine generator(seed);
    std::normal_distribution<double> distribution(0.0, 1.0);

    for (size_t WD = 0; WD < HIDLAYERS + 1; WD++)
    {
        if (WD == 0)
        {
            for (size_t WL = 0; WL < INNEURONS; WL++)
            {
                Inlayer.push_back(Inn);
                Inlayer.back().weights.reserve(HIDNEURONS);
                Inlayer.back().weightderivs.reserve(HIDNEURONS);
                if (optimizerformula == 1) { Inlayer.back().emavals.reserve(HIDNEURONS); }
                else if (optimizerformula == 3) { Inlayer.back().adamvals.reserve(HIDNEURONS); }
                else;
                for (size_t WK = 0; WK < HIDNEURONS; WK++)
                {
                    double val = distribution(generator) * randdist(INNEURONS, HIDNEURONS);
                    Inlayer.back().weights.push_back(val);
                    Inlayer.back().weightderivs.push_back(0.0);
                    if (optimizerformula == 1){ Inlayer.back().emavals.push_back(0.0); }
                    else if (optimizerformula == 3){ Inlayer.back().adamvals.push_back(0.0); }
                }
                if (optimizerformula == 3)
                {
                    Inlayer.back().SDB = 0.0;
                    Inlayer.back().VDB = 0.0;
                    Inlayer.back().SDW = 0.0;
                    Inlayer.back().VDW = 0.0; 
                }
            }
        }
        else if (WD < HIDLAYERS && WD != 0)
        {
            Hidlayers.push_back(std::vector <HIDneuron>());
            Hidlayers.back().reserve(HIDNEURONS);
            for (size_t WL = 0; WL < HIDNEURONS; WL++)
            {
                Hidlayers.back().push_back(Hidn);
                Hidlayers.back().back().weights.reserve(HIDNEURONS);
                Hidlayers.back().back().weightderivs.reserve(HIDNEURONS);
                if (optimizerformula == 1) { Hidlayers.back().back().emavals.reserve(HIDNEURONS); }
                for (size_t WK = 0; WK < HIDNEURONS; WK++)
                {
                    double val = distribution(generator) * randdist(HIDNEURONS, HIDNEURONS);
                    Hidlayers.back().back().weights.push_back(val);
                    Hidlayers.back().back().weightderivs.push_back(0.0);
                    if (optimizerformula == 1) { Hidlayers.back().back().emavals.push_back(0.0); }
                    else if (optimizerformula == 3) { Hidlayers.back().back().adamvals.push_back(0.0); }
                    //Hidlayers.back().back().adamvals.push_back(0.0);
                }
                Hidlayers.back().back().bias = 0.0;
                Hidlayers.back().back().biasderiv = 0.0;
                if (optimizerformula == 1) { Hidlayers.back().back().biasema = 0.0; }
                else if (optimizerformula == 3) {
                    Hidlayers.back().back().biasadam = 0.0;
                    Hidlayers.back().back().VDB = 0.0;
                    Hidlayers.back().back().SDB = 0.0;
                    Hidlayers.back().back().SDW = 0.0;
                    Hidlayers.back().back().VDW = 0.0;
                }
            }
        }
        else if (WD == HIDLAYERS)
        {
            Hidlayers.push_back(std::vector <HIDneuron>());
            Hidlayers.back().reserve(HIDNEURONS);
            for (size_t WL = 0; WL < HIDNEURONS; WL++)
            {
                Hidlayers.back().push_back(Hidn);
                Hidlayers.back().back().weights.reserve(OUTNEURONS);
                Hidlayers.back().back().weightderivs.reserve(OUTNEURONS);
                if (optimizerformula == 1) { Hidlayers.back().back().emavals.reserve(OUTNEURONS); }
                for (size_t WK = 0; WK < OUTNEURONS; WK++)
                {
                    double val = distribution(generator) * randdist(HIDNEURONS, OUTNEURONS);
                    Hidlayers.back().back().weights.push_back(val);
                    Hidlayers.back().back().weightderivs.push_back(0.0);
                    if (optimizerformula == 1) { Hidlayers.back().back().emavals.push_back(0.0); }
                    else if (optimizerformula == 3) { Hidlayers.back().back().adamvals.push_back(0.0); }
                }
                Hidlayers.back().back().bias = 0.0;
                Hidlayers.back().back().biasderiv = 0.0;
                if (optimizerformula == 1) { Hidlayers.back().back().biasema = 0.0; }
                else if (optimizerformula == 3) {
                    Hidlayers.back().back().biasadam = 0.0;
                    Hidlayers.back().back().VDB = 0.0;
                    Hidlayers.back().back().SDB = 0.0;
                    Hidlayers.back().back().SDW = 0.0;
                    Hidlayers.back().back().VDW = 0.0;
                }
            }
        }
    }
    for (size_t i = 0; i < OUTNEURONS; i++)
    {
        Outlayer.push_back(Outn);
        Outlayer.back().bias = 0.0;
        Outlayer.back().biasderiv = 0.0;
        if (optimizerformula ==1)
        Outlayer.back().biasema = 0.0;
        else if (optimizerformula == 3)
        {
            Outlayer.back().SDB = 0.0;
            Outlayer.back().VDB = 0.0;
        }
    }
}
bool Net::Feedforward(const std::vector <double>& invec)
{
    bool success = 1;
    for (size_t I = 0; I < Inlayer.size(); I++)
    {
        Inlayer[I].val = invec[I];
    }
    for (size_t h = 0; h < Hidlayers[0].size(); h++)
    {
        double preval = Hidlayers[0][h].bias;
        for (size_t I = 0; I < Inlayer.size(); I++)
        {
            preval += Inlayer[I].val * Inlayer[I].weights[h];
        }
        Hidlayers[0][h].preactval = preval;
        if (activationtype == 1)
        {
            Hidlayers[0][h].actval = Relu(preval);
        }
        else if (activationtype == 2)
        {
            Hidlayers[0][h].actval = tanh(preval);
        }
        if (isnan(Hidlayers[0][h].actval)) { std::cout << "isnan at Hidlayers[0][" << h << "].actval"; success = 0; }
        else success = 1;
    }
    for (size_t H = 1; H < Hidlayers.size(); H++)
    {
        size_t prevh = H - 1;
        for (size_t h = 0; h < Hidlayers[H].size(); h++)
        {
            double preval = Hidlayers[H][h].bias;
            for (size_t p = 0; p < Hidlayers[prevh].size(); p++)
            {
                preval += Hidlayers[prevh][p].actval * Hidlayers[prevh][p].weights[h];
            }
            Hidlayers[H][h].preactval = preval;
            if (activationtype == 1)
            {
                Hidlayers[H][h].actval = Relu(preval);
            }
            else if (activationtype == 2)
            {
                Hidlayers[H][h].actval = tanh(preval);
            }
            if (isnan(Hidlayers[H][h].actval)) { std::cout << "isnan at Hidlayers["<< H <<"][" << h << "].actval"; success = 0; }
            else success = 1;
        }
    }
    for (size_t O = 0; O < Outlayer.size(); O++)
    {
        size_t lhid = Hidlayers.size() - 1;
        double preval = Outlayer[O].bias;
        for (size_t h = 0; h < Hidlayers[lhid].size(); h++)
        {
            preval += Hidlayers[lhid][h].actval * Hidlayers[lhid][h].weights[O];
        }
        Outlayer[O].preactval = preval;
    }
    return success;
}
void Net::calcema(const size_t& Layernum, const size_t& neuron, const size_t& weight, const std::string& layer, const std::string& BorW)
{
    static double Beta{ 0.9 };
    if (BorW == "Weight")
    {
        if (layer == "Inlayer")
        {
            double tempval = (Beta * Inlayer[neuron].emavals[weight]) +((1.0 - Beta) * Inlayer[neuron].weightderivs[weight]);
            Inlayer[neuron].emavals[weight] = tempval;
        }
        else if (layer == "Hidlayers")
        {
            double tempval = (Beta * Hidlayers[Layernum][neuron].emavals[weight]) +((1.0 - Beta) * Hidlayers[Layernum][neuron].weightderivs[weight]);
            Hidlayers[Layernum][neuron].emavals[weight] = tempval;
        }
    }
    else if (BorW == "Bias")
    {
        if (layer == "Hidlayers")
        {
            double tempval = (Beta * Hidlayers[Layernum][neuron].biasema) +((1.0 - Beta) * Hidlayers[Layernum][neuron].biasderiv);
            Hidlayers[Layernum][neuron].biasema = tempval;
        }
        else if (layer == "Outlayer")
        {
            double tempval = (Beta * Outlayer[neuron].biasema) + ((1.0 - Beta) * Outlayer[neuron].biasderiv);
            Outlayer[neuron].biasema = tempval;
        }
    }

}
void Net::Backprop(const std::vector <double>& targets)
{
    for (size_t O = 0; O < Outlayer.size(); O++)
    {
        double PDval{};
        PDval = targets[O] - Outlayer[O].actval;
        PDval = PDval * -1.0;
        Outlayer[O].preactvalPD = PDval;
    }
    for (size_t H = Hidlayers.size(); H > 0; H--)
    {
        size_t Top = H;
        size_t Current = H - 1;
        for (size_t h = 0; h < Hidlayers[Current].size(); h++)
        {
            double actPD{};
            double PreactPD{};
            double biasPD{};
            for (size_t hw = 0; hw < Hidlayers[Current][h].weights.size(); hw++)
            {
                double PDval{};
                if (H == Hidlayers.size())
                {
                    PDval = Outlayer[hw].preactvalPD * Hidlayers[Current][h].actval;
                    biasPD = Outlayer[hw].preactvalPD;
                    if (descenttype == 2)
                        Outlayer[hw].biasderiv += biasPD;
                    else Outlayer[hw].biasderiv = biasPD;
                    actPD += Hidlayers[Current][h].weights[hw] * Outlayer[hw].preactvalPD;
                    if (optimizerformula == 1)
                    {
                        calcema(0, hw, 0, "Outlayer", "Bias");
                    }
                    else if (optimizerformula == 3)
                    {
                        calcadam(0, hw, 0, "Outlayer", "Bias");
                    }
                }
                else
                {
                    PDval = Hidlayers[Top][h].preactvalPD * Hidlayers[Current][h].actval;
                    actPD += Hidlayers[Current][h].weights[hw] * Hidlayers[Top][h].preactvalPD;
                }
                if (descenttype == 2)
                    Hidlayers[Current][h].weightderivs[hw] += PDval;
                else Hidlayers[Current][h].weightderivs[hw] = PDval;
                if (optimizerformula == 1)
                {
                    calcema(Current, h, hw, "Hidlayer", "Weight");
                }
                else if (optimizerformula == 3)
                {
                    calcadam(Current, h, hw, "Hidlayer", "Weight");
                }
            }
            if (H != Hidlayers.size())
            {
                biasPD = Hidlayers[Top][h].preactvalPD;
                if (descenttype == 2)
                Hidlayers[Top][h].biasderiv += biasPD;
                else Hidlayers[Top][h].biasderiv = biasPD; 
                if(optimizerformula == 1)
                {
                    calcema(Top, h, 0, "Hidlayer", "Bias");
                }
                else if (optimizerformula == 3)
                {
                    calcadam(Top, h, 0, "Hidlayer", "Bias");
                }
            }
            Hidlayers[Current][h].actvalPD = actPD;
            if (activationtype == 1) {
                PreactPD = Hidlayers[Current][h].actvalPD * Reluderiv(Hidlayers[Current][h].preactval);
            }
            else if (activationtype == 2)
            {
                PreactPD = Hidlayers[Current][h].actvalPD * tanhderiv(Hidlayers[Current][h].preactval);
            }
            Hidlayers[Current][h].preactvalPD = PreactPD;
            actPD = 0;
        }
    }
    for (size_t I = 0; I < Inlayer.size(); I++)
    {
        double PDval{};
        for (size_t hw = 0; hw < Inlayer[I].weights.size(); hw++)
        {
            PDval = Hidlayers[0][hw].preactvalPD * Inlayer[I].val;
            double biasPD = Hidlayers[0][hw].preactvalPD;
            if (descenttype == 2)
            {
                Inlayer[I].weightderivs[hw] += PDval;
                Hidlayers[0][hw].biasderiv += biasPD;
            }
            else
            {
                Inlayer[I].weightderivs[hw] = PDval;
                Hidlayers[0][hw].biasderiv = biasPD;
            }
            if (optimizerformula == 1)
            {
                calcema(0, hw, 0, "Hidlayer", "Bias");
                calcema(0, I, hw, "Inlayer", "Weight");
            }
            else if (optimizerformula == 3)
            {
                calcadam(0, hw, 0, "Hidlayer", "Bias");
                calcadam(0, I, hw, "Inlayer", "Weight");
            }
        }
    }
}
void Net::Updateweights()
{
    for (size_t I = 0; I < Inlayer.size(); I++)
    {
        double PD{};
        for (size_t iw = 0; iw < Inlayer[I].weights.size(); iw++)
        {
            if (optimizerformula == 2)
            {
                PD = (Inlayer[I].weightderivs[iw] * -1.0) - (Lambda * regularize(Inlayer[I].weights[iw]));
                Inlayer[I].weights[iw] = Inlayer[I].weights[iw] + (Alpha * PD);
            }
            else if (optimizerformula == 1)
            {
                PD = ((Inlayer[I].weightderivs[iw] + (0.9 * Inlayer[I].emavals[iw])) * -1.0) - (Lambda * regularize(Inlayer[I].weights[iw]));
                Inlayer[I].weights[iw] = Inlayer[I].weights[iw] + (Alpha * PD);
            }
            else if (optimizerformula == 3)
            {
                PD = ((Inlayer[I].weightderivs[iw] + (0.9 * Inlayer[I].adamvals[iw])) * -1.0); //- (Lambda * regularize(Inlayer[I].weights[iw]));
                Inlayer[I].weights[iw] = Inlayer[I].weights[iw] + (Alpha * PD);
            }
            else std::cout << "no optimizer formula chosen" << '\n';
            if (descenttype == 2) Inlayer[I].weightderivs[iw] = 0.0;
        }
    }

    for (size_t H = 0; H < Hidlayers.size(); H++)
    {
        for (size_t h = 0; h < Hidlayers[H].size(); h++)
        {
            double PD{};
            for (size_t hw = 0; hw < Hidlayers[H][h].weights.size(); hw++)
            {
                if (optimizerformula == 2)
                {
                    PD = (Hidlayers[H][h].weightderivs[hw] * -1.0) - (Lambda * regularize(Hidlayers[H][h].weights[hw]));
                    Hidlayers[H][h].weights[hw] = Hidlayers[H][h].weights[hw] + (Alpha * PD);
                }
                else if (optimizerformula == 1)
                {
                    PD = ((Hidlayers[H][h].weightderivs[hw] + (0.9 * Hidlayers[H][h].emavals[hw])) * -1.0) - (Lambda * regularize(Hidlayers[H][h].weights[hw]));
                    Hidlayers[H][h].weights[hw] = Hidlayers[H][h].weights[hw] + (Alpha * PD);
                }
                else if (optimizerformula == 3)
                {
                    PD = ((Hidlayers[H][h].weightderivs[hw] + (0.9 * Hidlayers[H][h].adamvals[hw])) * -1.0);// - (Lambda * regularize(Hidlayers[H][h].weights[hw]));
                    Hidlayers[H][h].weights[hw] = Hidlayers[H][h].weights[hw] + (Alpha * PD);
                }
                else std::cout << "no optimizer formula chosen" << '\n';
                if (descenttype == 2) Hidlayers[H][h].weightderivs[hw] = 0.0;
            }
            if (optimizerformula == 1)
            {
                PD = ((Hidlayers[H][h].biasderiv + (0.9 * Hidlayers[H][h].biasema)) * -1.0);
                Hidlayers[H][h].bias = Hidlayers[H][h].bias + (Alpha * PD);
            }
            else if (optimizerformula == 2)
            {
                PD = Hidlayers[H][h].biasderiv * -1.0;
                Hidlayers[H][h].bias = Hidlayers[H][h].bias + (Alpha * PD);
            }
            else if (optimizerformula == 3)
            {
                PD = ((Hidlayers[H][h].biasderiv + (0.9 * Hidlayers[H][h].biasadam)) * -1.0);
                Hidlayers[H][h].bias = Hidlayers[H][h].bias + (Alpha * PD);
            }
            else std::cout << "no optimizer formula chosen" << '\n';
            if (descenttype == 2)Hidlayers[H][h].biasderiv = 0;
        }
    }
    for (size_t biases = 0; biases < Outlayer.size(); biases++)
    {
        if (optimizerformula == 2)
        {
            double PD = Outlayer[biases].biasderiv * -1.0;
            Outlayer[biases].bias = Outlayer[biases].bias + (Alpha * PD);
        }
        else if (optimizerformula == 1)
        {
            double PD = ((Outlayer[biases].biasderiv + (0.9 * Outlayer[biases].biasema)) * -1.0);
            Outlayer[biases].bias = Outlayer[biases].bias + (Alpha * PD);
        }
        else if (optimizerformula == 3)
        {
            double PD = ((Outlayer[biases].biasderiv + (0.9* Outlayer[biases].biasadam)) * -1.0);
            Outlayer[biases].bias = Outlayer[biases].bias + (Alpha * PD);
        }
        else std::cout << "no optimizer formula chosen" << '\n';
        if (descenttype == 2)Outlayer[biases].biasderiv = 0.0;
    }
}
void Net::printvalues(double totalloss)
{
    for (size_t Res = 0; Res < Outlayer.size(); Res++)
    {
        std::cout << Outlayer[Res].actval << " / ";
    }
    std::cout << '\n' << "loss = " << totalloss << '\n';
}
double Net::lossfunc(const std::vector <double>& target)
{
    int pos{ -1 };
    double val{};
    for (size_t t = 0; t < target.size(); t++)
    {
        pos += 1;
        if (target[t] > 0)
        {
            break;
        }
    }

    val = -log(Outlayer[pos].actval);

    return val;
}
void Net::calcadam(const size_t& Layernum, const size_t& neuron, const size_t& weight, const std::string& layer, const std::string& BorW)
{
    static const double beta1{0.9};
    static const double beta2{0.999};
    static const double epsilon{0.000000001};
    if (layer == "Inlayer")
    {
        double Vdw = (beta1 * Inlayer[neuron].VDW) + ((1.0 - beta1) * Inlayer[neuron].weightderivs[weight]);
        double Sdw = (beta2 * Inlayer[neuron].SDW) + ((1.0 - beta2) * (Inlayer[neuron].weightderivs[weight] * Inlayer[neuron].weightderivs[weight]));
        double VdwC = Vdw / (1.0 - pow(beta1, iter)); //iter value needs attention, cant have pow(beta1, 10000)
        double SdwC = Sdw / (1.0 - pow(beta2, iter));
        Inlayer[neuron].VDW = VdwC;
        Inlayer[neuron].SDW = SdwC;
        double resW = (VdwC / (sqrt(SdwC) + epsilon));
        Inlayer[neuron].adamvals[weight] = resW;
    }
    else if (layer == "Hidlayers")
    {
        if (BorW == "Weight")
        {
            double Vdw = (beta1 * Hidlayers[Layernum][neuron].VDW) + ((1.0 - beta1) * Hidlayers[Layernum][neuron].weightderivs[weight]); //momentum
            double Sdw = (beta2 * Hidlayers[Layernum][neuron].SDW) + ((1.0 - beta2) * (Hidlayers[Layernum][neuron].weightderivs[weight] * Hidlayers[Layernum][neuron].weightderivs[weight])); // rmsprop
            double VdwC = Vdw / (1 - pow(beta1, iter)); //corrected
            double SdwC = Sdw / (1 - pow(beta2, iter));
            Hidlayers[Layernum][neuron].VDW = VdwC;
            Hidlayers[Layernum][neuron].SDW = SdwC;
            double resW = (VdwC / (sqrt(SdwC) + epsilon));
            Hidlayers[Layernum][neuron].adamvals[weight] = resW;
        }
        else if (BorW == "Bias")
        {
            double Vdb = (beta1 * Hidlayers[Layernum][neuron].VDB) + ((1.0 - beta1) * Hidlayers[Layernum][neuron].biasderiv);
            double Sdb = (beta2 * Hidlayers[Layernum][neuron].SDB) + ((1.0 - beta2) * (Hidlayers[Layernum][neuron].biasderiv * Hidlayers[Layernum][neuron].biasderiv));
            double SdbC = Sdb / (1 - pow(beta2, iter));
            double VdbC = Vdb / (1 - pow(beta1, iter));
            Hidlayers[Layernum][neuron].VDB = VdbC;
            Hidlayers[Layernum][neuron].SDB = SdbC;
            double resB = (VdbC / (sqrt(SdbC) + epsilon));
            Hidlayers[Layernum][neuron].biasadam = resB;
        }
    }
    else if (layer == "Outlayer")
    {
        double Vdb = (beta1 * Outlayer[neuron].VDB) + ((1.0 - beta1) * Outlayer[neuron].biasderiv);
        double Sdb = (beta2 * Outlayer[neuron].SDB) + ((1.0 - beta2) * (Outlayer[neuron].biasderiv * Outlayer[neuron].biasderiv));
        double SdbC = Sdb / (1 - pow(beta2, iter));
        double VdbC = Vdb / (1 - pow(beta1, iter));
        Outlayer[neuron].VDB = VdbC;
        Outlayer[neuron].SDB = SdbC;
        double resB = (VdbC / (sqrt(SdbC) + epsilon));
        Outlayer[neuron].biasadam = resB;
    }
}
int main()
{
    double arrAlpha[100];
    double arrLambda[100];
    for (int i = 0; i < 100; i++)
    {
        arrAlpha[i] = randomd(0.00000, 0.1)/10.0;
        arrLambda[i] = randomd(0.00000, 0.09)/10.0;
    }
    std::vector <double> innums{};
    std::vector <double> outnums{};
    std::vector <std::string> INstrings{};
    std::vector <std::string> OUTstrings{};
    std::string nums{};
    std::string in{};
    std::string out{};
    std::string allin{};
    double totalloss{};
    double loss{};
    double single{};
    double batchcount{ 0.0 };
    double alphaval{};
    double lambdaval{};
    std::ifstream file("N.txt");
    while (file.is_open())
    {
        while (file >> nums)
        {
            if (nums == "in:")
            {
                std::getline(file, in);
                allin += in;
            }
            else if (nums == "out:")
            {
                std::getline(file, out);
                OUTstrings.push_back(out);
                INstrings.push_back(allin);
                allin.clear();
            }
            else;
        }
        break;
    }
    for (int i = 0; i < 100; i++)
    {
        alphaval = arrAlpha[i];
        lambdaval = arrLambda[i];
        Net net(0, 1, 4, 3, 4, lambdaval, alphaval); //dimensions / hiddenlayers / hiddenneurons / outneurons / inneurons
        net.defineoptimizer("Adam");
        net.defineregularizer("L1");
        net.defineactivation("Relu");
        net.definedescenttype("MiniBatch");
        net.Initweights();
        for (int epoch = 0; epoch < 50000; epoch++)
        {
            net.iter += 1;
            int random = randomt(0, 97); // 1300 samples 
            std::cout << "fetching" << '\n';
            std::stringstream in(INstrings[random]);
            std::stringstream out(OUTstrings[random]);
            while (in >> single)
            {
                innums.push_back(single);
            }
            while (out >> single)
            {
                outnums.push_back(single);
            }
            std::cout << "epoch " << epoch << '\n';
            std::cout << '\n' << "targets: " << '\n';
            for (auto element : outnums) std::cout << element << " / ";
            std::cout << '\n';
            batchcount += 1.0;
            if (!net.Feedforward(innums)) { innums.clear(); outnums.clear(); break; }
            else;
            net.softmax();
            loss += net.lossfunc(outnums);
            totalloss = loss / batchcount;
            net.printvalues(totalloss);
            net.Backprop(outnums);
            if (net.descenttype = 2)
            {
                if (net.iter == 124)
                {
                    net.Updateweights();
                    net.iter = 0;
                }
                else;
            }
            else net.Updateweights();
            innums.clear();
            outnums.clear();
        }
        std::ofstream outfiles{ "resultsoverall.txt", std::ios::app };
        while (outfiles.is_open())
        {
            outfiles << "alphaval: " << alphaval << " / lambdaval: " << lambdaval << "| Loss = " << totalloss << '\n';
            outfiles << "iterations: " << batchcount << '\n';
            break;
        }
        totalloss = 0.0; loss = 0.0; batchcount = 0.0;
    }
}

Answer 1

This review is gonna be geared more towards code style, rather than the algorithm since I don't remember much about neural networks.

Mark Relu and Reluderiv as constexpr

Mark those two functions are constexpr so that the compiler can evaluate the result at compile time, if possible. Unfortunately, I don't think tanh and sqrt are constexpr functions, so you can't mark those functions are constexpr.

Pass double by value

You don't need to pass double by reference, since double can easily fit inside a CPU register. In fact, it might even be slower, since it might involve a memory read (if the compiler hadn't optimized it)

Don't recreate std::random_device every time

Your randomt and randomd function recreates std::random_device every time it's called, which can degrade performance. std::random_device is implementation defined. For example, it might be a thin wrapper over fopen("/dev/urandom") which might block, or might be a call to some crypto API (in case of Windows, IIRC). Better to create it just once, and utilize in multiple times.

You can either declare it in a namespace

namespace my_random
{
    std::random_device rd;
}

int randomt(int x, int y)
{
   std::mt19937 gen(my_random::rd());
   ...
}

and use it like that, or alternatively you can declare it static.

int randomt(int x, int y)
{
    static std::random_device rd;
    ...
}

Use std::uniform_int_distribution

You should use std::uniform_int_distribution instead of using std::uniform_real_distribution and rounding the value.

Use struct instead of class

Your INneuron, HIDneuron, OUTneuron class contains only public data members. You should prefer using a struct in this case. Makes no difference to the compiler, but makes your intention clearer to other people looking at your code. A widely accepted convention is that structs should just be containers for data, while classes should be used to implement OOP.

Use inheritance

Notice that you're repeating a lot of data members in your classes. This might be a good opportunity to utilize inheritance. How you might wanna structure your data, I'll leave to you.

Better naming convention

defineoptimizers, definedescenttype, etc. are can be made more readable; something like define_optimizers or defineDescentType.

Consistent naming conventions

The naming of data members and methods is all over the place. optimizerformula vs. NETDIMENSIONS vs. Alpha. Pick one and stick with it.

Use std::string_view instead of const std::string&

C++17 provides you with std::string_view which is a read only string, and is generally preferred nowadays instead of const std::string&.

Prefer using enum class

For cases where might have a fixed set of options, prefer using enum class over std::string. So, for descent type, you might define it like this:

enum class DescentType
{
    SGD,
    MiniBatch
};

Then instead of doing descenttype = 1 or descenttype = 2, you could do descenttype = DescentType::SGD.

Be consistent in your logic

The defineoptimizer and defineactivation abort when no matching value is found, but defineregularizer and definedescenttype just choose a default value and continue working. In fact, they're not even doing that, because the default value would be 0 and you don't have an option when they are set to 0.

You don't need to create a generator in the Initweights method

You have already created a function that would give you a random double. So why are you creating a random engine and distribution again, and why are you seeding using the time? Just use the function again.

Use whitespace

Whitespace is your friend! Use it liberally, it makes your code much more readable.

Cleaner code

Your Initweights method is littered with back() calls, which is quite frankly, ugly. Either construct the object fully and push_back or get the object using reference.

InLayer.push_back(Inn);
auto& vec = InLayer.back();
...

Although, you should prefer the former, rather than the latter.

Also, you Inn member is just a default constructed object of INneuron that you're using the push_back into InLayer. Why not just do InLayers.push_back(INneuron{}) or better yet, InLayers.emplace_back(). You could save quite a bit of memory by removing Inn.

The else is not doing anything, so you could remove it.

You true or false with bool

Your feedforward method assigns a 1 to success which is a bool. Might be legal C++, but why not just use true?

Pass size_t by value

Again, better to pass such small objects by value.

You constexpr instead of const for constants

You should use constexpr instead of const in calcadam for the constants, which is more C++-ish way to define constants.

Your main is doing too much

Split your main into separate functions. For example, there could be just a single function to read the input file and parse the data, and return a struct containing said that. There could also be another function to write to the output file. There could also be another function to actually run the program (the big loop inside main).

while(file.is_open())...break;

This is just a weird way to check if opening a file is successful. You can simply do

std::ifstream file("N.txt")
if(!file)
{
    std::cout << "Cannot open file!\n";
    // handle error
}

// continue with normal program execution

Use better names

Use much more descriptive names that you are right now. It's okay to write more if it means that the person who will be reading it will have an easier time.