• response.url is a string and so you shouldn't cast it to one. This confused me as I it made me think response.url would be something that isn't a string.

• You seem to be getting the response as an instance of TextResponse, which contains selector. This also contains the shortcut xpath which is equivalent to selector.xpath. Which allows you to reduce the size of that large command.

• Unfortunately scrapy doesn't document extract_first, but looking at the source they use Parsel. The documentation on extract_first was also kinda lacking, so I read the source code again, and it seems like they always return strings. So you probably don't need the str around extract_first either.

• You don't define pagename unless page is not None. But you go on to use it regardless. This is wrong, and can lead to errors. There are two ways to come at this, either silently fail. Or raise an exception, where NameError: name 'pagename' is not defined is not a good error.

• Finally it's strange to see yield rather than return, you're only going to ever return one thing. And so unless it's a requirement imposed by scrapy I'd change it to return.

BASE_URL = 'http://www.example.com'
USER_NAME = 'UserNameHere'
PASSWORD = 'PasswordHere'
PAGES = ['page1.aspx', 'page2.aspx', 'page3.aspx', 'page4.aspx']

class ShareSpider(scrapy.Spider):
    name = "sharespider"
    start_urls = [BASE_URL + '/public/login.aspx']

    def parse(self, response):
        yield scrapy.FormRequest.from_response(
            response,
            formxpath='//form[@id="login"]',
            formdata={
                'UserName': USER_NAME,             
                'Password': PASSWORD,             
                'Action':'1',
            },
            callback=self.after_login)

    def after_login(self, response):
        base_url = BASE_URL + '/public/'
        for page in PAGES:
            yield Request(
                url=base_url + page + "?id=1",
                callback=self.action)

    def action(self, response):
        page = re.search('public/(.*)id=1', response.url)
        if page:
            page_name = page.group(1)
            title = response.xpath('//title/text()').extract_first().strip()
            item = PageItem()
            item['pagename'] = page_name
            item['description'] = title
            yield item

• response.url is a string and so you shouldn't cast it to one. This confused me as I it made me think response.url would be something that isn't a string.

• You seem to be getting the response as an instance of TextResponse, which contains selector. This also contains the shortcut xpath which is equivalent to selector.xpath. Which allows you to reduce the size of that large command.

• Unfortunately scrapy doesn't document extract_first, but looking at the source they use Parsel. The documentation on extract_first was also kinda lacking, so I read the source code again, and it seems like they always return strings or None. So you probably don't need the str around extract_first if you set default to an empty string.

• You don't define pagename unless page is not None. But you go on to use it regardless. This is wrong, and can lead to errors. There are two ways to come at this, either silently fail. Or raise an exception, where NameError: name 'pagename' is not defined is not a good error.

• Finally it's strange to see yield rather than return, you're only going to ever return one thing. And so unless it's a requirement imposed by scrapy I'd change it to return.

BASE_URL = 'http://www.example.com'
USER_NAME = 'UserNameHere'
PASSWORD = 'PasswordHere'
PAGES = ['page1.aspx', 'page2.aspx', 'page3.aspx', 'page4.aspx']

class ShareSpider(scrapy.Spider):
    name = "sharespider"
    start_urls = [BASE_URL + '/public/login.aspx']

    def parse(self, response):
        yield scrapy.FormRequest.from_response(
            response,
            formxpath='//form[@id="login"]',
            formdata={
                'UserName': USER_NAME,             
                'Password': PASSWORD,             
                'Action':'1',
            },
            callback=self.after_login)

    def after_login(self, response):
        base_url = BASE_URL + '/public/'
        for page in PAGES:
            yield Request(
                url=base_url + page + "?id=1",
                callback=self.action)

    def action(self, response):
        page = re.search('public/(.*)id=1', response.url)
        if page:
            page_name = page.group(1)
            title = response.xpath('//title/text()').extract_first('').strip()
            item = PageItem()
            item['pagename'] = page_name
            item['description'] = title
            yield item