1
\$\begingroup\$

I needed a script to import repositories from other Bitbucket accounts to my own Bitbucket account. I have some 250 repositories in my account and I wanted them to move under Team (which is same as Organization in Github). Though Bitbucket has APIs, I couldn't find any API to import, and instead they provide a HTML form on their site. When I couldn't get my pure Python script to automate and fill the form, I decided to go with Selenium IDE. The following Python script generates Selenium IDE script, which I later run on the IDE.

This is how this script works:

  1. It takes a REST call to Bitbucket, to find all the repositories
  2. The Bitbucket provides a paginated response, the response includes repo URL which is just enough
  3. I make a series of calls to get all repository URLs
  4. Once I have all the URLs, I fill in the Selenium script template and write it back to a file

This script also can be used to import repositories from Github/or any other version control server by importing the generate_sel_script function.

import json

import requests
from requests.auth import HTTPBasicAuth

from templates import sel_script_template
from templates import command_template
from settings import *

def generate_bb_repo_urls():
    def make_rest_call(url):
        auth = HTTPBasicAuth(SRC_AC_USERNAME, SRC_AC_PASSWORD)
        return requests.get(url=url, auth=auth).json()

    seed_url = 'https://bitbucket.org/api/2.0/repositories/' + SRC_AC_USERNAME
    response = make_rest_call(seed_url)
    bb_repo_urls = []

    while('next' in response.keys()):
        for repo in response['values']:
            bb_repo_urls.append(repo['links']['html']['href'])
        response = make_rest_call(response['next'])

    return bb_repo_urls

def generate_sel_script(src_repo_urls):
    commands = "".join(map(lambda url: command_template.substitute({
                                                'bb_username': SRC_AC_USERNAME, 
                                                'bb_password': SRC_AC_PASSWORD, 
                                                'repo_url': url,
                                                'owner_value': IMPORTER_ID
                                                }), src_repo_urls))

    with open('selenium-script.txt', 'w') as sel_file: 
        sel_file.write(sel_script_template.substitute({
                                        'bb_username': IMPORTER_BB_USERNAME, 
                                        'bb_password': IMPORTER_BB_PASSWORD, 
                                        'commands': commands
                                        }))

if __name__ == '__main__':
    generate_sel_script(generate_bb_repo_urls())

settings.py:

SRC_AC_USERNAME = 'bitbucket'
SRC_AC_PASSWORD = 'ohlongjohnson'
IMPORTER_BB_USERNAME = 'bitbucket'
IMPORTER_BB_PASSWORD = 'ohlongjohnson'
IMPORTER_ID = '007007'

templates.py:

from string import Template

sel_script_template = Template("""<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="https://bitbucket.org/account/signin/?next=/repo/import" />
<title>New Test</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">New Test</td></tr>
</thead><tbody>
<tr>
    <td>open</td>
    <td>/account/signin/?next=/repo/import</td>
    <td></td>
</tr>
<tr>
    <td>type</td>
    <td>id=id_username</td>
    <td>$bb_username</td>
</tr>
<tr>
    <td>type</td>
    <td>id=id_password</td>
    <td>$bb_password</td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>name=submit</td>
    <td></td>
</tr>
<tr>
    <td>pause</td>
    <td>100</td>
    <td></td>
</tr>

$commands
</tbody></table>
</body>
</html>
""")

command_template = Template("""<tr>
    <td>pause</td>
    <td>1000</td>
    <td></td>
</tr>
<tr>
    <td>open</td>
    <td>https://bitbucket.org/repo/import</td>
    <td></td>
</tr>
<tr>
    <td>pause</td>
    <td>100</td>
    <td></td>
</tr>
<tr>
    <td>open</td>
    <td>https://bitbucket.org/repo/import</td>
    <td></td>
</tr>
<tr>
    <td>type</td>
    <td>id=id_url</td>
    <td>$repo_url</td>
</tr>
<tr>
    <td>click</td>
    <td>id=id_auth</td>
    <td></td>
</tr>
<tr>
    <td>type</td>
    <td>id=id_username</td>
    <td>$bb_username</td>
</tr>
<tr>
    <td>type</td>
    <td>id=id_password</td>
    <td>$bb_password</td>
</tr>
<tr>
    <td>select</td>
    <td>id=id_owner</td>
    <td>value=$owner_value</td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>xpath=(//button[@type='submit'])[2]</td>
    <td></td>
</tr>""")

What I am looking for:

  • Any general code improvements/comments
  • How do I avoid from settings import *? (or is it okay for import * in this case) I have two options, import settings or five import statements for each variable.
  • Is it possible to avoid this line bb_repo_urls = [], but create a empty list later and append directly in while loop?
  • Any way to improve this line : "".join(map(lambda url...
\$\endgroup\$
  • 1
    \$\begingroup\$ One suggestion: perhaps consider using the keyring module rather than storing Bitbucket credentials in settings.py. \$\endgroup\$ – alexwlchan Feb 14 '14 at 19:02
3
\$\begingroup\$

Its actually very simple, there is no need to write complex scripts.

What I did was, complete one import process using BitBuckets HTML form, and record the post request that form makes using chrome. The chrome will give you the request in cURL form. Then use a simple nodejs script to run the same cURL command by just replacing the repo URL and the repo name.

var shell = require('shelljs');

var arr = {
    "https://laxxxxxxtee.git": "pyyyyye",
    "https://lxxxxxxxxlace.git": "shyyyyyce",
    "https://latttttttwa.git": "syyyyya",
    "https://laxxxxxxxappy-ionic.git": "snayyyyic",
    "https://laxxxxxxxxxxxxnter-ui.git": "viyyyyyyr-ui",
    "https://laxxxxxxxxxnter.git": "vyyyyyer",
    "https://laxxxxxxxxtend.git": "viyyyyy"
};

for (var url in arr) {
    var name = arr[url];
    name = encodeURIComponent(name);
    url = encodeURIComponent(url);
    var command = "curl 'https://bitbucket.org/repo/import?owner=txxxxh' -H 'Pragma: no-cache' -H 'Origin: https://bitbucket.org' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8,en-GB;q=0.6,ml;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://bitbucket.org/repo/import?owner=txxxxxh' -H 'Cookie: optimizelyEndUserId=oexxxxxxxxxx643798; __ar_v4=3coookies-form' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'source_scm=git&source=source-git&goog_project_name=&goog_scm=svn&sourceforge_project_name=&sourceforge_mount_point=&sourceforge_scm=svn&codeplex_project_name=&codeplex_scm=svn&url=" + url + ".git&auth=on&username=nyyyyyys&password=Gvvvvv&owner=36vvvv2&name=" + name + "&description=&is_private=on&forking=no_forks&no_forks=True&no_public_forks=True&has_wiki=on&language=php&csrfmiddlewaretoken=MErrrrrrrrrOZ91r&scm=git' --compressed";

    shell.exec(command, function(code, output) {
        console.log('Exit code:', code);
        console.log('Program output:', output);
    });
}
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.