18
\$\begingroup\$

Github repo

I wrote this little toy today that probes random web servers and prints out the title of their index page. I'm curious if this code is any good since I'm a bit rusty.

get_titles.py

#!/usr/bin/env python

""" Probe random web server and print out the title of their index page. """

import os
import sys
import lxml.html


def get_title(url):
    '''Extract the title from a webpage.
    If anything goes wrong like the request
    times out or the page does not have a
    title get_title will return None'''
    try:
        title = lxml.html.parse(url)
        return title.find(".//title").text
    except KeyboardInterrupt:
        sys.exit(0)
    except Exception:
        return None

def shell(command):
    """Run a shell command.
    return it's output as a string"""
    return os.popen(command).read()

def scan(nhosts):
    """Run an nmap scan of n hosts.
    This will return IPv4 addresses of hosts
    that are up and have a service running on
    port 80. Simply supply the number of hosts
    you would like to check."""
    results = set(shell('./discover.sh {}'.format(nhosts)).split('\n'))
    if not results:
        print 'Nothing Found.'
        sys.exit(0)
    return ["http://{}".format(ip) for ip in results if ip]

def main():
    """ Run the scan."""
    if not len(sys.argv) >= 2:
        print 'You forgot to tell me how many hosts to check'
        sys.exit(0)
    for url in scan(sys.argv[1]):
        title = get_title(url)
        if title:
            print title, url


if __name__ == '__main__':
    main()

discover.sh

#!/bin/bash

nhosts=$1

nmap \
--host-timeout 10s \
--max-retries 0 \
–min-parallelism 100 \
-T5 \
-Pn \
-n \
-iR $1 \
-p80 \
-oG tmp.txt \
--open \
tmp.txt > /dev/null

cat tmp.txt | grep 'Host' | awk '{print $2}'
rm tmp.txt
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Minor point: cat tmp.txt | grep 'Host' is a Useless Use of Cat and can be shorted to grep 'Host' tmp.txt — and then the entire line can be shorted even further to awk '/Host/ {print $2}' tmp.txt. \$\endgroup\$
    – jwodder
    Jul 12, 2017 at 15:59
  • 1
    \$\begingroup\$ As a sysadmin I wouldn't be happy with you port scanning to find webservers :) How about selecting a random word from a large dictionary and using it as a "i feel lucky" google query (which returns top non-paid result), then parsing that page for the title.... \$\endgroup\$
    – ivanivan
    Jul 12, 2017 at 16:57
  • \$\begingroup\$ "Run a shell command. Return its output as a string." \$\endgroup\$
    – Scimonster
    Jul 13, 2017 at 16:20

4 Answers 4

14
\$\begingroup\$

Overall, the code is pretty clean. Some notes about the Python part:

  • be consistent about the documentation string style - enclose them with triple quotes, start with an upper-case letter, end with a dot (PEP8 reference). I also think that you can use less newline breaks there.
  • if we would follow PEP8 importing recommendations, there should be a newline between the "system" level imports and third-parties:

    import os
    import sys
    
    import lxml.html
    
  • you should have 2 line breaks between the function definitions (reference)

  • use print() as a function for Python 3 compatibility
  • I would avoid unnecessary invert of the logic and replace:

    if not len(sys.argv) >= 2:
    

    with:

    if len(sys.argv) < 2:
    
  • avoid using bare except clauses
  • you can use .findtext() method:

    title.findtext(".//title")
    

    which will return None if title element was not found.

\$\endgroup\$
0
7
\$\begingroup\$

On top of what @alecxe has said I would advise you update your docstrings to include the parameters and descriptions of what they do using some well known convention. Also, I would emphasize you use a well known convention so that way some documentation generator (for example, Sphinx) can use the information to generate readable docs.

Note: Sphinx uses the reST convention.

\$\endgroup\$
3
\$\begingroup\$

Regarding the shell script portion of your project I would indent the subsequent lines of the nmap command like so:

nmap \
    --host-timeout 10s \
    --max-retries 0 \
    –min-parallelism 100 \
    -T5 \
    -Pn \
    -n \
    -iR $1 \
    -p80 \
    -oG tmp.txt \
    --open \
    tmp.txt > /dev/null
\$\endgroup\$
0
\$\begingroup\$

get_titles.py

#!/usr/bin/env python

"""
Probe random web server and print out the title of their index page.
:param nhosts: The number of random IPs you want to probe.
"""

import os
import sys

# third party imports
import lxml.html


def get_title(url):
    """
    Extract the title from a web page..
    :param url: The URL of the page
    :returns: A string if title was found or None if somthing goes wrong
    """
    try:
        title = lxml.html.parse(url)
        return title.find(".//title").text
    except KeyboardInterrupt:
        sys.exit(0)
    except Exception:
        return None

def shell(command):
    """
    Execute a shell command and return it's output.
    :param command: The command you want to execute
    :returns: A string representation of the commands output
    """
    return os.popen(command).read()

def scan(nhosts):
    """
    Scan for web servers.
    This function uses namp to find random systems
    that are running a service on port 80.
    We will assume that if the system is running a service
    on port 80 it's a web server and generate a URL for it.
    :param nhosts:  The number of random IPs you would like to probe
    :returns: A list if URLs
    """
    results = set(shell('./discover.sh {}'.format(nhosts)).split('\n'))
    if not results:
        print 'Nothing Found.'
        sys.exit(0)
    return ["http://{}".format(ip) for ip in results if ip]

def main():
    """
    This function puts everything together
    :returns: None
    """
    if not len(sys.argv) >= 2:
        print 'You forgot to tell me how many hosts to check'
        sys.exit(0)
    for url in scan(sys.argv[1]):
        title = get_title(url)
        if title:
            print title, url


if __name__ == '__main__':
    main()

Diff


@@ -1,17 +1,25 @@
  #!/usr/bin/env python

 -""" Probe random web server and print out the title of their index page. """
 +""" 
 +Probe random web server and print out the title of their index page. 
 +
 +:param nhosts: The number of random IPs you want to probe. 
 +"""

  import os
  import sys
 +
 +# third party imports
  import lxml.html


  def get_title(url):
 -    '''Extract the title from a webpage.
 -    If anything goes wrong like the request
 -    times out or the page does not have a
 -    title get_title will return None'''
 +    """
 +    Extract the title from a web page..
 +
 +    :param url: The URL of the page
 +    :returns: A string if title was found or None if somthing goes wrong
 +    """
      try:
          title = lxml.html.parse(url)
          return title.find(".//title").text
 @@ -21,24 +29,37 @@ def get_title(url):
          return None

  def shell(command):
 -    """Run a shell command.
 -    return it's output as a string"""
 +    """
 +    Execute a shell command and return it's output.
 +    
 +    :param command: The command you want to execute
 +    :returns: A string representation of the commands output
 +    """
      return os.popen(command).read()

  def scan(nhosts):
 -    """Run an nmap scan of n hosts.
 -    This will return IPv4 addresses of hosts
 -    that are up and have a service running on
 -    port 80. Simply supply the number of hosts
 -    you would like to check."""
 +    """
 +    Scan for web servers.
 +    
 +    This function uses namp to find random systems 
 +    that are running a service on port 80.
 +    We will assume that if the system is running a service
 +    on port 80 it's a web server and generate a URL for it.
 +    
 +    :param nhosts:  The number of random IPs you would like to probe
 +    :returns: A list if URLs
 +    """
      results = set(shell('./discover.sh {}'.format(nhosts)).split('\n'))
      if not results:
          print 'Nothing Found.'
          sys.exit(0)
      return ["http://{}".format(ip) for ip in results if ip]

  def main():
 -    """ Run the scan."""
 +    """  
 +    This function puts everything together    
 +    :returns: None
 +    """
      if not len(sys.argv) >= 2:
          print 'You forgot to tell me how many hosts to check'
          sys.exit(0)

discover.sh

  • Removed unused variable nhosts.
  • Fixed indentation.
  • Shortened the parsing portion to just awk.

#!/bin/bash
nmap \
    --host-timeout 10s \
    --max-retries 0 \
    -T5 \
    -Pn \
    -n \
    -iR "$1" \
    -p80 \
    -oG tmp.txt \
    --open \
    tmp.txt > /dev/null
    awk '/Host/ {print $2}' tmp.txt
    rm tmp.txt

Diff


@@ -1,19 +1,16 @@
  #!/bin/bash

 -nhosts=$1
 -
  nmap \
 ---host-timeout 10s \
 ---max-retries 0 \
 -–min-parallelism 100 \
 --T5 \
 --Pn \
 --n \
 --iR $1 \
 --p80 \
 --oG tmp.txt \
 ---open \
 -tmp.txt > /dev/null
 -
 -cat tmp.txt | grep 'Host' | awk '{print $2}'
 -rm tmp.txt 
 +    --host-timeout 10s \
 +    --max-retries 0 \
 +    –min-parallelism 100 \
 +    -T5 \
 +    -Pn \
 +    -n \
 +    -iR "$1" \
 +    -p80 \
 +    -oG tmp.txt \
 +    --open \
 +    tmp.txt > /dev/null
 +    awk '/Host/ {print $2}' tmp.txt
 +    rm tmp.txt
\$\endgroup\$
1
  • \$\begingroup\$ You must also provide the changes that were made. Code-only answers still aren't allowed for self-answers. \$\endgroup\$
    – Jamal
    Jul 15, 2017 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.