0
\$\begingroup\$

I'm currently getting data from an API every 15 seconds. This data contains a json with 100-2000 new items every time. Now i don't need all the items that are in there but still quite some of them and i want to store these items in a database. I will probably delete items that are added longer then 5-7 days.

Now i just want to check if the way i'm doing this now is "normal". Because i will run this all day and it's going to get a lot of items.

The script looks like this now:

function fetchItems() {
  axios
    .get("http://lorem.com")
    .then((response) => {

      for (let i = 0; i < response.data.stashes.length; i++) {
        if (some statements) { //filter some of the stashes here with some statements
          stashes.push(filterItems(response.data.stashes[i])); //filter items in stashes and push to new array
        }
      }
      //filtering done
      //add all items to database
      for (let i = 0; i < stashes.length; i++) {
        for (let j = 0; j < stashes[i].items.length; j++) {
          let item = { data i want to store };
          let sql = 'INSERT INTO items SET ?';
          let query = db.query(sql, item, (error, result) => {
            if (error) console.log(error);
            console.log('item added');
          });
        }
      }
    })
    .catch((error) => {
      console.log(error);
    });
}

Is this the way this should be done?

\$\endgroup\$
  • 1
    \$\begingroup\$ What SQL dialect is this? Does it work? \$\endgroup\$ – slepic Apr 26 at 17:23
1
\$\begingroup\$

Quickly overviewing the code - it would work. If you are into a more readable approach, I may have just a few (minor) comments:

Declare fetchItems as async, and use await for axios.

Instead of a for loop, use response.data.stashes.filter(). Adding all items in separate queries seems odd, is there a way to push all of them in a single transaction? Build the query outside in a helper function.

So your flow may look something like this:

const fetchItems = () => {/*... axios ...*/};
const filterItems = (item) => {/*.. return true or false */};
const buildQuery = (items) => {/*.. return SQL string ..*/};

async function execute() {
  try {
    const items = (await fetchItems()).filter(filterItems);
    const query = buildQuery(items);
    await db.query(sql);
  } catch (err) {
  /* handle it */
  }
}
| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.