4
\$\begingroup\$

Preface

This is my first work script; normally I'm just mucking around in console or writing snippets. This script has a lot of firsts for me (configs, logging instead of print, try/except). Typically when I just write snippets I don't care if things are breaking. Given this is set up as a scheduled task for work, I tried to make it a bit less... um... brittle.

Problem

The web app I'm logging into stores log files on a system admin profile as attachments. I needed to write a script to automatically log in and download these attachments as it only has a small retention period before they disappear.

General Plan of Action

  1. GET webapp/ page to retrieve cookies

  2. GET webapp/login to retrieve form input ids used for posting credentials

    GET login response:

    form    -23nuyjabt3o6jyz20bze4lt19wjxqihqpzcja0kiy0osistaqb,-2jnri62v92sbs0qooyazpdfrwv2561kl49fulslch1thh43c0s,2qvew89xgsjkg0u0uhhmuxjxqp8kb71t6k827iaofkmepu7c47

  3. POST username to webapp/uuid to retrieve UUID

    uuid params:

    userName    testuser

    uuid response:

    uuid    c8637a56-1495-4388-888b-0c35aff86974

  4. Hash password using UUID and random string (SHA3-KACCAK)

  5. POST credentials (hashed pw, username) mapped to the random form IDS from above.

    POST /login params using the form IDS from previous GET /login

    -23nuyjabt3o6jyz20bze4lt19wjxqihqpzcja0kiy0osistaqb 
-2jnri62v92sbs0qooyazpdfrwv2561kl49fulslch1thh43c0s a3a3574d05e0de1fa30d898e8ac425e44be2f19b7f8119a1ae27eeb4e3d0e445679554ba12fd44f120784465bc18cc512eaababec00a4ec9c03f11b2d64208ae
 2qvew89xgsjkg0u0uhhmuxjxqp8kb71t6k827iaofkmepu7c47 testuser

  6. Get the current user's profile page id by finding "View My Profile" link and getting value of kUserID parameter.

  7. GET worker profile page and find all attachments.

  8. Download attachments to disk if they don't already exist.

log_downloader.py

Logs into web app, navigates to own worker profile and downloads attachment files that match the whitelist.

import configparser
import logging
import requests
import sha3
from logging.handlers import RotatingFileHandler
from pathlib import Path
from urllib.parse import urlparse, parse_qsl
from bs4 import BeautifulSoup
from getpass import getpass

logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
fh = RotatingFileHandler('log.txt', maxBytes=1024, backupCount=5)
formatter = logging.Formatter(
        '%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
handler.setFormatter(formatter)
fh.setFormatter(formatter)
logger.addHandler(handler)
logger.addHandler(fh)
logger.setLevel(logging.DEBUG)


config = configparser.ConfigParser()
config.read('config.ini')
URL = config['Penelope']['url']

sess = requests.session()


def get_uuid(username: str):
    """
    Returns a UUID for the provided username.
    :param username: Penelope User Name
    :return: str
    """
    try:
        r = sess.get(f'{URL}acm_loginControl')
        r = sess.post(f'{URL}acm_loginControl/uuid', {'userName': username})
        _uuid = r.json()['uuid']
        logger.debug(f'UUID Retrieved: {_uuid}')
        return _uuid
    except Exception as e:
        print(e)


def digest_password(uuid: str, password: str):
    """
    Digests a password using **SHA3 (Keccak)** to match Crypto.JS lib used in Penelope login
    :param uuid: Generated UUID for username.
    :param password: pw to be digested
    :return: str
    """
    h = sha3.keccak_512()

    h.update(uuid.encode())
    h.update('algo != null && algo !== 0'.encode())  # Random string in Penelope JS script
    h.update(password.encode())

    hexdigest = h.hexdigest()
    logger.debug(f'Password Hashed: {hexdigest}')
    return hexdigest


def login(username: str, digest: str):
    if 'authentype' not in sess.cookies or 'JSESSIONID' not in sess.cookies:
        try:
            # Gets token from login page
            r = sess.get(f'{URL}acm_loginControl')
            # Gets form IDs generated by server based off timestamp
            r = sess.get(f'{URL}acm_loginControl/login')

            # Set Credentials to form IDS
            form_ids = r.json()['form'].split(',')
            creds = {form_ids[0]: '', form_ids[2]: username, form_ids[1]: digest}

            # Post Credentials
            logger.debug('Trying to Log In')
            r = sess.post(f'{URL}acm_loginControl/login', creds)
            logger.debug(r.json())
            if r.json()['state'] == 'ok':
                logger.debug('Logged in Successfully')
                r = sess.get(f'{URL}acm_loginControl/create')
                r.raise_for_status()
            else:
                raise ValueError(r.json()['errorCode'])
            return r
        except ValueError as e:
            logger.error(e)

        except Exception as e:
            logger.error(e)

    else:
        logger.debug('Not Logging in')
        return f'Session already exists: {sess.cookies}'


def navigate_to_workerprofile(user_id: int):
    return sess.get(f'{URL}acm_userProfileControl?actionType=view&kUserID={user_id}')


def get_user_id(login_response: requests.models.Response):
    """Get Worker Profile kUserID (wruser table)"""
    soup = BeautifulSoup(login_response.content, 'html.parser')
    home_page_link = soup.find(id='frm_content_id')['src']
    r = sess.get(f'{URL}{home_page_link}')
    soup = BeautifulSoup(r.content, 'html.parser')
    href = next(link.get('href') for link in soup.find_all('a') if link.text == 'View My Profile')
    qs = urlparse(href)
    params = dict(parse_qsl(qs.query))
    logger.debug(f'kUserID Retrieved: {params["kUserID"]}')
    return params['kUserID']

def find_log_attachments():
    logger.debug('Searching for stdout log files')
    soup = BeautifulSoup(r.content, 'html.parser')
    attachment_table = soup.find(id='attachListTable_worker')
    logs = [link.text for link in attachment_table.find_all('a') if link.text.startswith('stdout')]
    logger.debug(logs)
    return logs


def download_attachment(file_name):
    download_url = f'{URL}acm_attachmentControl?actionType=download' \
                   f'&attachCat=worker&attachCatID={kuserid}&attachName={file_name}'
    logger.debug(f'Downloading {file_name}')
    r = sess.get(download_url)
    if 'JSESSIONID' not in sess.cookies:
        raise ConnectionError
    return r


def save_log_files():
    log_attachments.sort()
    for log_name in log_attachments:
        log_file_path = Path(f'E:/Penelope Logs/mywebapp/{log_name}')

        if not log_file_path.exists():
            try:
                log_download_response = download_attachment(log_name)
                with open(log_file_path, 'wb') as log_file:
                    log_file.write(log_download_response.content)
                    logger.debug(f'Log File {log_file_path} saved to disk')
            except ConnectionError as e:
                logger.error('Connection no longer valid. No session token.')
        else:
            logger.debug(f'{log_file_path} already exists')

if __name__ == '__main__':
    if not config.has_option('Penelope', 'user'):
        user = input('Enter Username to login with')
        config['Penelope']['user'] = user
    if not config.has_option('Penelope', 'password'):
        password = getpass('Enter Password')
        user = config['Penelope']['user']
        uuid = get_uuid(user)
        digest = digest_password(uuid, password)
        config['Penelope']['password'] = digest
        with open('config.ini', 'w') as config_file:
            config.write(config_file)

    user = config['Penelope']['user']
    digest = config['Penelope']['password']

    r = login(user, digest)
    kuserid = get_user_id(r)
    r = navigate_to_workerprofile(kuserid)
    log_attachments = find_log_attachments()
    save_log_files()

Output log file

PW already hashed / 1 new log file to DL

2019-03-01 08:00:00,969 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): mywebapp:443
2019-03-01 08:00:01,099 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl HTTP/1.1" 200 None
2019-03-01 08:00:01,140 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl/login HTTP/1.1" 200 None
2019-03-01 08:00:01,141 root         DEBUG    Trying to Log In
2019-03-01 08:00:01,183 urllib3.connectionpool DEBUG    https://mywebapp:443 "POST /acm_loginControl/login HTTP/1.1" 200 None
2019-03-01 08:00:01,222 root         DEBUG    {'next': 'create', 'state': 'ok'}
2019-03-01 08:00:01,223 root         DEBUG    Logged in Successfully
2019-03-01 08:00:01,331 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl/create HTTP/1.1" 302 83
2019-03-01 08:00:01,378 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_topFrameControl HTTP/1.1" 200 None
2019-03-01 08:00:01,491 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET //acm_homepageControl?actionType=top_frame&curval=1551391201355 HTTP/1.1" 200 None
2019-03-01 08:00:01,823 root         DEBUG    kUserID Retrieved: 1077
2019-03-01 08:00:01,877 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_userProfileControl?actionType=view&kUserID=1077 HTTP/1.1" 200 None
2019-03-01 08:00:01,941 root         DEBUG    Searching for stdout log files
2019-03-01 08:00:01,960 root         DEBUG    ['stdout_2019-02-26.0.log.gz', 'stdout_2019-02-25.2.log.gz', 'stdout_2019-02-25.1.log.gz', 'stdout_2019-02-25.0.log.gz', 'stdout_2019-02-24.0.log.gz', 'stdout_2019-02-23.0.log.gz', 'stdout_2019-02-22.1.log.gz', 'stdout_2019-02-22.0.log.gz', 'stdout_2019-02-28.1.log.gz', 'stdout_2019-02-28.0.log.gz', 'stdout_2019-02-27.1.log.gz', 'stdout_2019-02-27.0.log.gz', 'stdout_2019-02-26.2.log.gz', 'stdout_2019-02-26.1.log.gz']
2019-03-01 08:00:01,960 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-22.0.log.gz already exists
2019-03-01 08:00:01,961 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-22.1.log.gz already exists
2019-03-01 08:00:01,961 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-23.0.log.gz already exists
2019-03-01 08:00:01,961 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-24.0.log.gz already exists
2019-03-01 08:00:01,962 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-25.0.log.gz already exists
2019-03-01 08:00:01,962 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-25.1.log.gz already exists
2019-03-01 08:00:01,962 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-25.2.log.gz already exists
2019-03-01 08:00:01,962 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-26.0.log.gz already exists
2019-03-01 08:00:01,963 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-26.1.log.gz already exists
2019-03-01 08:00:01,963 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-26.2.log.gz already exists
2019-03-01 08:00:01,963 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-27.0.log.gz already exists
2019-03-01 08:00:01,964 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-27.1.log.gz already exists
2019-03-01 08:00:01,964 root         DEBUG    E:\Penelope Logs\mywebapp\stdout_2019-02-28.0.log.gz already exists
2019-03-01 08:00:01,964 root         DEBUG    Downloading stdout_2019-02-28.1.log.gz
2019-03-01 08:00:01,989 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_attachmentControl?actionType=download&attachCat=worker&attachCatID=1077&attachName=stdout_2019-02-28.1.log.gz HTTP/1.1" 200 8964328
2019-03-01 08:00:02,509 root         DEBUG    Log File E:\Penelope Logs\mywebapp\stdout_2019-02-28.1.log.gz saved to disk
\$\endgroup\$

1 Answer 1

Reset to default
3
\$\begingroup\$

Since you handle the parameters of the URLs manually using f-strings, you open yourself to a whole lot of possible bugs. If the file name of an attachment contains even so much as a space, your script will break.

You could fix this by manually URL encoding your parameters, or you could use the features provided by the requests library for this by passing a dictionary with all parameters:

def navigate_to_workerprofile(user_id: int):
    params = {'actionType': 'view', 'kUserID': user_id}
    return sess.get(f'{URL}acm_userProfileControl', params=params)

def download_attachment(file_name):
    params = {'actionType': 'download',
              'attachCat': 'worker',
              'attachCatID': kuserid,
              'attachName': file_name}
    logger.debug(f'Downloading {file_name}')
    r = sess.get(f'{URL}acm_attachmentControl', params=params)
    if 'JSESSIONID' not in sess.cookies:
        raise ConnectionError
    return r

Next, you should not rely on global variables, if at all possible. Currently you are passing some parameters as arguments to your functions and some not. Instead, pass all parameters as arguments (global constants like URL are OK):

BASE_PATH = Path('E:/Penelope Logs/mywebapp')

def save_log_files(log_attachments):
    for log_name in sorted(log_attachments):
        log_file_path = BASE_PATH / log_name

        if not log_file_path.exists():
            try:
                log_download_response = download_attachment(log_name)
                with open(log_file_path, 'wb') as log_file:
                    log_file.write(log_download_response.content)
                    logger.debug(f'Log File {log_file_path} saved to disk')
            except ConnectionError as e:
                logger.error('Connection no longer valid. No session token.')
        else:
            logger.debug(f'{log_file_path} already exists')
\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.