2
\$\begingroup\$

I my firm a few people have following problem:

A Content Management System is hosted externally. The treaty doesn't include database-access.

In September the treaty will expire. So they have to get the content copied.

I have tried to figure out something. So this programming is a first-try. A prototype in the best-case.

It works only with the HTML source-code of the articles from these news-site: heise.de

I could have used another site as well. It was just for having something to experiment with.

Here's the code I have written so far:

document.getElementById('exportText').addEventListener('click', function() {
  'use strict';
  var exportText = '';
  var re = /(data\-article\-type="meldung">)(.*)(<\/article)/m;
  var reString  = '(<h2 class="article__heading">)(.*?)(<\/h2>)|';
      reString += '(<p class="meldung_anrisstext">)(.*?)(<\/p>)|';
      reString += '(<h3 class="subheading">)(.*?)(<\/h3>)|';
      reString += '(<p>)(.*?)(<\/p>)';
  var matches  = [];
  var mainText = document.getElementById('source')
                         .value                            // Get the content from the first text-box.
                         .replace(/\r?\n|\r|\s{2,}/g, '')  // Remove line breaks; Multiple blanks to ONE blank.
                         .match(re)[0];                    // Restrict the area which is searched.

  re = new RegExp(reString, 'gm');

  // Search in the "meldung-wrapper"-container for h2-, h3-headlines and paragraphs.
  while ((matches = re.exec(mainText)) !== null) {
    exportText += ( matches[2] || matches[5] ||
                    matches[8] || matches[11] || '' ) + '\n\n';
  }

  document.getElementById('target').value = exportText;
});

document.getElementById('reset').addEventListener('click', function() {
  document.getElementById('source').value = '';
  document.getElementById('target').value = '';
});
* {
  background-color: #cdcdcd;
}

.wrap {
  width: 800px;
  margin: 10px auto;
}

label {
  font-weight: 900;
  margin: 10px 0;
  display: block;
}

textarea {
  background-color: #efefef;
  border-radius: 4px;
}

label,
button {
  font-family: verdana;
  sans-serif;
}

.control {
  display: inline-block;
  text-decoration: none;
  color: #fff;
  padding: 6px 8px;
  border: 1px solid;
  text-align: center;
  text-transform: uppercase;
  margin: 12px 5px;
  border-radius: 4px;
  background-color: #9a9a9a;
  font-size: .9rem;
}

.control:hover {
  background-color: #232323;
  color: #efefef;
}

.control:active {
  opacity: 0.6;
}

#target {
  padding: 4px 5px;
}
<!doctype html>
<html>

<head>
  <title>Exporter Script</title>
  <link rel="stylesheet" href="css/style.css" />
</head>

<body>
  <div class="wrap">
    <div>
      <label for="target">HTML Source-Code</label>
      <textarea id="source" rows="10" cols="100"></textarea>
    </div>

    <div>
      <label for="target">Exportierte Inhalte</label>
      <textarea id="target" rows="20" cols="100"></textarea>
    </div>

    <div>
      <a class="control" href="#" id="exportText">Export starten</a>
      <a class="control" href="#" id="reset">Reset Textboxes</a>
    </div>
  </div>
  <script src="js/main.js"></script>
</body>

</html>

You've got to paste the source-code of an article into the textbox "HTML Source Code". Then click the button. Then the second textbox contains the textual content of the article including inline-tags (hyperlinks, bold, italic etc. are desired to be exported too).

Result of Export

What do you think about the approach?

- Is this a good approach? Or is there a better way to accomplish this task?

Please keep in mind that I don't have database-access.

- Are there points which can or must be improved in my programming?

- Any ideas how to accomplish a better structure?

Looking forward to reading your hints, comments and recommendations.

\$\endgroup\$
1
  • \$\begingroup\$ scraping a site to retrieve your own data? that's bologna. i don't care who is hosting your stuff i highly doubt that they don't provide any means for your to get your data back. why do you keep saying "treaty" is your web host the chinese governement or what? \$\endgroup\$ Jul 5 '17 at 15:52
2
\$\begingroup\$

Is this a good approach?

Sadly, nope.

Or is there a better way to accomplish this task?

Yes.

Instead of manually sifting through pages and copying source code, use wget's site mirroring functionality to download the entire site. There are a lot of ways to do this with the various options wget has. HTTrack is also an option if you want the tool to mangle the structure, names, etc. But we only need to download the pages so wget is fine.

Next, instead of having a page that parses the page source for content, consider writing a Node.js script instead*. Use glob to grab the paths of all HTML files, cheerio to traverse and grab content from the HTML like jQuery, and Node's very own fs to read and write to files. Also, regular expressions are the worst tools to parse through HTML. That's a known fact, don't even go there.

In summary, you run wget to download the entire site, and the Node.js script to parse all the HTML that was downloaded. You'll probably spend half a day to a day tops to build this script and the next day you'll have parsed content in seconds. You just saved 2 months worth of work right there.

With the answer above, the rest becomes irrelevant.


*Any language would actually do (Python, Ruby, PHP, etc). But since you wrote in JS, Node.js would probably make more sense.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.