Retrieve Data from TOC

Question

I aim to convert the TOC of The Python Language Reference — Python 3.6.3 documentation to a structured data with the following steps:

1. Copy contents to a plr.md file:

   In [1]: with open('plr.md') as file:
      ...:     content = file.read()
   In [2]: content
   Out[2]: '\n\n- \\1. Introduction\n  - [1.1. Alternate Implementations]
   (https://docs.python.org/3.6/reference/introduction.html#alternate-implementations)\n  - [1.2. Notation](https://docs.python.org/3.6/reference/introduction.html#notation)\n- \\2. Lexical analysis\n  - [2.1. Line structure]
   (https://docs.python.org/3.6/reference/lexical_analysis.html#line-structure)\n  - [2.2. Other tokens](https://docs.python.org/3.6/reference/lexical_analysis.html#other-tokens)\n
   

2. Get chapters:

   In [47]: chapters = content.split('\n- \\')
       ...: #subtract the unqualified part
       ...: chapters = chapters[1:]
   In [50]: chapters[0]
   Out[50]: '1. Introduction\n  - [1.1. Alternate Implementations](https://docs.python.org/3.6/reference/introduction.html#alternate-implementations)
   \n  - [1.2. Notation](https://docs.python.org/3.6/reference/introduction.html#notation)'
   

3. Separate chapter name and section name in each chapters:

   chapter_details = chapters[0].split('\n  -')
   sections = chapter_details[1:]
   chapter = chapter_details[0]
   In [54]: chapter
   Out[54]: '1. Introduction'
   In [55]: sections
   Out[55]:
   [' [1.1. Alternate Implementations](https://docs.python.org/3.6/reference/introduction.html#alternate-implementations)',
    ' [1.2. Notation](https://docs.python.org/3.6/reference/introduction.html#notation)']
   

4. Convert section:

   def convert_section(s):
       start = s.index('[') + 1
       end = s.index(']')
       return s[start:end]
   In [57]: print(convert_section(' [1.1. Alternate Implementations](https://docs.python.org/3.6/reference/i
       ...: ntroduction.html#alternate-implementations)'))
   1.1. Alternate Implementations
   
   sections = map(convert_section, sections)
   sections = list(sections)
   

5. Create a dict:

   key = chapter
   {key:sections}
    {'1. Introduction':['1.1. Alternate Implementations', '1.2. Notation']}
   

6. Encapsulate code in a class and get the result:

   class TOC:
       def __init__(self, filename):
           self.filename = filename
   
       def read(self, filename):
           with open (filename) as file:
               content = file.read()
           return content
   
       def convert_section(self, s):
           start = s.index('[') + 1
           end = s.index(']')
           return s[start:end]
   
       def get_chapters(self, filename):
           content = self.read(filename)
           chapters = content.split('\n- \\')
           #subtract the unqualified part
           chapters = chapters[1:]
           return chapters
   
       def create_chapter_dict(self, chapter):
           chapter_details = chapter.split('\n  -')
           sections = chapter_details[1:]
           key = chapter_details[0]
           value = map(self.convert_section, sections)
           return {key: list(value)}
   
       def get_chapters_dict(self):
           chapters = self.get_chapters(self.filename)
           chapters_dict = {}
           for chapter in chapters:
               chapter_dict = self.create_chapter_dict(chapter)
               chapters_dict.update(chapter_dict)
           return chapters_dict
   

Run and get the result:

In [89]: TOC('plr.md').get_chapters_dict()
Out[89]:
{'1. Introduction': ['1.1. Alternate Implementations', '1.2. Notation'],
 '2. Lexical analysis': ['2.1. Line structure',
  '2.2. Other tokens',
  '2.3. Identifiers and keywords',
  '2.4. Literals',
  '2.5. Operators',
  '2.6. Delimiters'],
 '3. Data model': ['3.1. Objects, values and types',
  '3.2. The standard type hierarchy',
  '3.3. Special method names',
  '3.4. Coroutines'],

This solution is a bit too much for a daily common operation. Is there a standard or easy method for such a task?

Answer 1

I think you are overcomplicating the problem. I would probably go with a proper Markdown parser (like mistune), or parse the generated HTML instead. Here is how I would do it using BeautifulSoup parser:

from pprint import pprint

from bs4 import BeautifulSoup
import requests


response = requests.get("https://docs.python.org/3.6/reference/index.html")

soup = BeautifulSoup(response.content, "html.parser")
contents = soup.select_one("#the-python-language-reference ul")

pprint({
    li.a.get_text(): [li.a.get_text() for li in li("li")]
    for li in contents.find_all("li", recursive=False)
})

Prints:

{'1. Introduction': ['1.1. Alternate Implementations', '1.2. Notation'],
 '10. Full Grammar specification': [],
 '2. Lexical analysis': ['2.1. Line structure',
                         '2.2. Other tokens',
                         '2.3. Identifiers and keywords',
                         '2.4. Literals',
                         '2.5. Operators',
                         '2.6. Delimiters'],
 '3. Data model': ['3.1. Objects, values and types',
                   '3.2. The standard type hierarchy',
                   '3.3. Special method names',
                   '3.4. Coroutines'],
 '4. Execution model': ['4.1. Structure of a program',
                        '4.2. Naming and binding',
                        '4.3. Exceptions'],
 '5. The import system': ['5.1. importlib',
                          '5.2. Packages',
                          '5.3. Searching',
                          '5.4. Loading',
                          '5.5. The Path Based Finder',
                          '5.6. Replacing the standard import system',
                          '5.7. Special considerations for __main__',
                          '5.8. Open issues',
                          '5.9. References'],
 '6. Expressions': ['6.1. Arithmetic conversions',
                    '6.2. Atoms',
                    '6.3. Primaries',
                    '6.4. Await expression',
                    '6.5. The power operator',
                    '6.6. Unary arithmetic and bitwise operations',
                    '6.7. Binary arithmetic operations',
                    '6.8. Shifting operations',
                    '6.9. Binary bitwise operations',
                    '6.10. Comparisons',
                    '6.11. Boolean operations',
                    '6.12. Conditional expressions',
                    '6.13. Lambdas',
                    '6.14. Expression lists',
                    '6.15. Evaluation order',
                    '6.16. Operator precedence'],
 '7. Simple statements': ['7.1. Expression statements',
                          '7.2. Assignment statements',
                          '7.3. The assert statement',
                          '7.4. The pass statement',
                          '7.5. The del statement',
                          '7.6. The return statement',
                          '7.7. The yield statement',
                          '7.8. The raise statement',
                          '7.9. The break statement',
                          '7.10. The continue statement',
                          '7.11. The import statement',
                          '7.12. The global statement',
                          '7.13. The nonlocal statement'],
 '8. Compound statements': ['8.1. The if statement',
                            '8.2. The while statement',
                            '8.3. The for statement',
                            '8.4. The try statement',
                            '8.5. The with statement',
                            '8.6. Function definitions',
                            '8.7. Class definitions',
                            '8.8. Coroutines'],
 '9. Top-level components': ['9.1. Complete Python programs',
                             '9.2. File input',
                             '9.3. Interactive input',
                             '9.4. Expression input']}

Note that this would work for the limited depth of the nested lists. If it would be needed, you can generalize parsing the nested lists using something like this dictify() function.