1
\$\begingroup\$

This is a simplified version of my app. It has a bunch of scraper classes, and a Job class, which controls the collection of data, and the saving to the db. During scraping it could happen many unforeseen problem due to irregular html structure, date format, missing information etc. I want to have my app to run without stopping, and if there is an unprocessable item, it should log it, and somehow flag it in the database, that there was a problem with the site. After the irregularity is handled in the code, the next time we know that the page has to be scanned again, but this time it will be work fine, and the flag is switched to scanned.

In this example there is some problem with item nr. 4. It is logged (in my app to a file) and an error is thrown. Should I rethrow it in every function call till it reaches the Job class' flush method?

There was a second idea to handle this problem. Instead of throwing errors around the result of Scraper class' scrape method should contain if something went wrong. [1, 2, 3, false, 5, 6 ...] and during the flush handle the falsy item. The problem: this items are obviously not just simple numbers, but complex objects with a lot of links (to media, sub pages etc.) and those links are also scraped with different scrape classes, so I should handle in every for loop and scraper the falsy entries. Maybe there is a better solution.

jsbin: https://jsbin.com/jojiwuyugi/1/edit?js,console

class Scraper {
  check(item) {
    if (item === 4) {
      throw 42
      console.log('There is a problem with ' + item)
      return false
    } else {
      return true
    }
  }

  scrape() {
    const arr = [1, 2, 3, 4, 5, 6, 7, 8, 9]
    const res = []
    for (const item of arr) {
      try {
        if (this.check(item))
          res.push(item)
      } catch(e) {
      }

    }
    return res
  }
}

class SiteScraper {
  scrape() {
    return {
      site: 'Foo'
    }
  }
}

class Job {
  flush(data) {
    try {
      console.log(data.site)
      for (const item of data.items)
        console.log(item)
    } catch(e) {
      console.log('Error while flushing ' + e)
    }
    // here I should switch the flag if everything is fine / wrong 
  }

  collectData() {
    const siteScraper = new SiteScraper()
    const scraper = new Scraper()
    const site = siteScraper.scrape()
    site.items = scraper.scrape()
    return site
  }

  run() {
    const data = this.collectData()
    this.flush(data)
  }

}


const j = new Job()
j.run()
\$\endgroup\$

closed as off-topic by 200_success, Mike Brant, Mast, Grajdeanu Alex., 1201ProgramAlarm Apr 25 at 20:54

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Lacks concrete context: Code Review requires concrete code from a project, with sufficient context for reviewers to understand how that code is used. Pseudocode, stub code, hypothetical code, obfuscated code, and generic best practices are outside the scope of this site." – 200_success, Mike Brant, Mast, Grajdeanu Alex.
If this question can be reworded to fit the rules in the help center, please edit the question.