# Tag Info

15

As the code is similar, they can be merged together. I'll suggest you to break down the functions into smaller reusable chunks which will follow the Single Responsibility Principle (SRP). These functions will be testable and reusable. Both functions are Adding a delay of 500 milliseconds Selecting elements from provided page Clicking the first element ...

9

Try this query instead on your data and tell how it performs. Not sure it will return exactly the same as your query but it should be close enough so you can adjust to what you want. select C.CaseName, O.PartNumber, O.NodeName, O.Value as OldValue, N.Value as NewValue from dbo.Cases as C cross apply ( select M.X....

7

Code Flow: you are redefining links on every iterations of the while loop - you basically need to do it once as a while loop exit condition, we can use the fact that there are line numbers in the company list grid - we can simply wait for the number 1000 to show up while scrolling I would also create a class to have the driver and WebDriverWait instance ...

5

I don't see a method to directly turn it into an array -- Array.from doesn't work, eg. The best I could achieve was a little cleanup: var result = document.evaluate("//*[@*[starts-with(name(), 'temp')]]", document, null, XPathResult.ANY_TYPE, null); var node, nodes = [] while (node = result.iterateNext()) nodes.push(node); console.log(nodes); <...

5

Sub Fortune_parser() The procedure is implicitly Public; I'd make it explicitly so. It's also using a rather uncommon casing style I don't even have a name for. In every single VBA type library out there[citation needed], public members are PascalCase. Note that classes that expose public members that have underscores in their names, cannot be used with ...

5

There are no docstrings, so a user of your class is left wondering which methods could be useful to call. In fact, after studying the code, it looks like all the methods are meant to be called by __init__, in a specific order. Makes me wonder if this should be a class at all. You set very many instance attributes, even a self.dummy, but use very few outside ...

5

I think that one of the collections you iterate already has all the data you need. Try this: $linkQuery =$xpath->query("//div[@class='news-archive']//a"); foreach ($linkQuery as$link) { $data[] = array('link' =>$link->getAttribute('href'), 'title' => $link->nodeValue); } Btw. array_map() is another way for traversing two (or more) ... 5 Nitpicks Here are a few small critiques to start with: Single-letter table reference aliases are often discouraged because they do not convey any meaning, such that you have to go back to their sources to understand what they actually reference to. For example, your sub-query aliased as T would make more sense with an alias like CaseChanges. SQL Server/... 5 I would recommend a more modular design - having a LinkedInScraper class, initialized with a login and password and with separate methods for logging in and getting profile links. Also, I think you are overusing the XPaths overall - whenever possible, first explore if you can use "by id", "by name" or "by css selector" locators and fall back to XPath only ... 4 This won't do much for the performance, but maintaining this kind of T-SQL is painful, so proper formatting is essential. The FROM clause is hidden, and one has to pay very close attention to notice the ORDER BY clause as well. Consider: WITH cte (Execution_ID,Intake_Generic_ID, [File_Name],F1,Report_Attributes) AS ( SELECT Execution_ID ... 4 Reading the documentation for Transformer, where it says: A Transformer may be used multiple times. Parameters and output properties are preserved across transformations. , I'd say that not only can the Transformer object be reused across multiple nodes, but you should also be able to use a single StringWriter instead of recreating it for every node; ... 3 There doesn't appear to be a way to get the results directly from document.evaluate(). Taking the concept from this answer, one could get a snapshot result, create an array filled with dummy values (e.g. 0s) and then map calls to snapshotItem(). const query = document.evaluate("//*[@*[starts-with(name(), 'temp')]]", document, null, XPathResult.... 3 There are several major issues in the code: violates the DRY principle there is actually no element presence check - find_element_* method will fail with NoSuchElementException if the element is not found (well, you may be handling the exception outside of the method, but that scatters the logic too much resulting into more "spaghetti code") Instead, I ... 3 According to comments below this is a new, shorter code without sorting, to make it comparable: First I create test data CREATE TABLE Intake_Generic(Execution_ID INT,Intake_Generic_ID INT,[File_Name] VARCHAR(100),F1 VARCHAR(MAX)); INSERt INTO Intake_Generic VALUES (1,2,'SomeFileName','This~is~a~test') ,(2,1,'SomeFileName','And~just~another~test'); GO Now ... 3 I'm having trouble following this code, so I'm going to focus on maintainability. Here are some things you can consider changing: Some members are currently initialized in __init__, some are only initialized in other methods. It's typically good to define invariants of your class that will remain true at all observable times. Since __init__ calls everything ... 2 The "i" is not necessary when you already have the array key. The key applies to both link and title. PHP$linkQuery = array("link1", "link2", "link3", "link4", "link5"); $titleQuery = array("title1", "title2", "title3", "title4", "title5"); foreach($linkQuery as $k =>$link){ $data[$k] = array('link' => $link,'title' =>$titleQuery[$k]); } .... 2 I am suspicious of this line here: my ($time, $report) = ($line =~ /(\d+),(.*)$/); # time, XML file That line needs to scan and group the entire XML doc, but my instinct is that a 2-limited split (see note on LIMIT) would be more efficient: my ($time, $report) = split /,/,$line, 2; Apart from that, really, everything looks sane, and far better than a ...

2

Except for the difference between @id=… and @class=… (and an indentation error on that line), the two functions look identical. You should avoid cut-and-paste code by implementing both getElementsById() and getElementsByClass() in terms of a common helper function — perhaps getElementsByXPath(). In any document, an id is supposed to uniquely identify at ...

2

Really, the best way to optimize performance is to make the HTTP requests in parallel. After you have done that, you could consider further optimizations with regards to parsing. Consider using curl_multi_exec() or similar for this. I have a REST client based on curl_multi_exec() that you can feel free to take a look at for inspiration (or just use as-is ...

2

As long as you don't face noticable performance issues I would chose whatever method seems to be better for you. The XPath works so you may stick to it. I would however build the path in a more generic way by specifying an array of attribute names first: var attributeNames = new[] { "billingLocation", "adminOffice", "homeCity", "origin", ...

2

There are some things I would do differently: create a class - this way you may share a web-scraping session between the methods and also share the base and start urls use more meaningful variable and method names - for instance, parse_doc() could be get_contact_info(); you can re-use page variable names in different methods etc. you can use findtext() ...

2

Good job once again! I would eliminate one loop depth level - there is only a single table that you are interested it - iterate over its rows directly. And, I would switch to more concise CSS selectors to locate table rows and cells. Also, I would rename post to a row and t_data to cell to correspond to what these variables represent: For Each row In ...

2

Here is the summary of the applied improvements: keeping a visited set of links - $O(1)$ lookups using collections.deque() to keep the queue of the next links to scrape - popping left from it is $O(1)$ reusing the same requests.Session() instance, keeping a session instance as an instance variable of your crawler class using urljoin() to join the base ...

2

There are a few major things that immediately caught my eye: bare except clauses should be avoided in favor of catching more specific exceptions element locators quality - you should NOT be relying on layout-oriented classes like padding0 and pure-technical Angular classes like ng-binding and ng-scope wait.until() calls in your case return a WebElement ...

2

The code is quite clean and easy to read, good job! I would only focus on couple things: remove extra spaces around = when it is used to define a keyword argument CSS selectors are more appropriate and reliable when it comes to handling multi-valued class attribute than XPath expressions. Plus, they are more concise and generally faster naming - for links ...

2

Well, your regex solution is obviously badly broken - it will fail if the attributes are in a different order, if they are separated by newlines, if they are delimited by single quotes, etc etc. If you try to replace it with a more correct regex (it will never be 100% correct of course) then you are quite likely to lose some of this speed - perhaps ...

2

Docstrings You should include a docstring at the beginning of every module, class, and function you write. This will allow documentation to identify what your code is supposed to do. Docstrings are great for the understanding the functionality of the larger part of the code, i.e., the general purpose of any class, module or function. Comments are used for ...

1

If you've got a single method, there is probably not much sense in having a class and just have a separate function instead. More info at Stop Writing Classes. If you do need a classmethod, the first argument to such a method is agreed to be named cls and not self. Crawl should be named crawl according to Python PEP8 naming practices. Use string formatting ...

1

VBAs selenium bindings are not as rich in functionality as other language bindings, but I would improve the following: use the "by id" locators whenever possible: .FindElementById("email").SendKeys ("someEmail") --Fill in the email .FindElementById("pass").SendKeys ("password") --Fill in the password .FindElementById("loginbutton").Click if tempting to ...

1

is it bad practice to use relative path(//) twice in a single xpath? It would depend on the HTML source of the code - how many elements are there in the path between the nodes, how unique are the elements and their attributes, in general. In this case though, it is perfectly okay - it is even better than having: //div[@id='a1']/div/div/span[@id='s1'] ...

Only top voted, non community-wiki answers of a minimum length are eligible