Unstructured to Structured TOC

Question

The following code tries to convert an unstructured TOC with bounding box layout data given by the output of pdftotext -bbox-layout -f 11 -l 13 new_book.pdf toc.html in a html file into structured data on a DataFrame.

I intend to use this data to build the internal TOC of a pdf document from scratch.

I would like to hear suggestions about how to make better use of pandas.DataFrame functionalities to make the code more efficient.

Updated:

Since there is not yet an answer, and @J_H does have a valid point there (see his comment below), I have updated my code with functions so that it is more readable and hopefully somebody would be keen (and kind) enough to review it for me.

from lxml import etree
import pandas as pd
import numpy as np
import numeral

# This is a rewriting of parse_toc.py to improve on readability.


def build_dataframe(file_path):
    """Build dataframe from pdftotext output."""

    with open(file_path) as f:
        text = f.read()

    tree = etree.HTML(text)

    toc_data = []
    for i, page in enumerate(tree.findall('.//page')):  # .// recursively search beyond the immediate children.
        for f, flow in enumerate(page.findall('.//flow')):
            for j, block in enumerate(flow.findall('.//block')):
                for k, line in enumerate(block.findall('.//line')):
                    for word in line.findall('word'):
                        toc_data.append((word.text, i, f, j, k,
                                         round(float(word.get('xmin'))),  # round-off to smoothen values.
                                         round(float(word.get('ymin'))),
                                         round(float(word.get('xmax'))),
                                         round(float(word.get('ymax')))))

    df = pd.DataFrame(toc_data, columns=["word", "page", "flow", "block", "line", "xMin", "yMin", "xMax", "yMax"])
    df = df.sort_values(by=["page", "yMax", "xMax"])

    return df


def consolidate_bboxes(df):

    df = df.assign(new_yMax=np.where(df['yMax'].diff(-1) == -1, df.yMax + 1, df.yMax))
    df = df.assign(new_yMax=df.groupby(['page', 'yMax'])['new_yMax'].transform('max'))
    df.sort_values(by=['page', 'new_yMax', 'xMax'], inplace=True)
    df.reset_index(drop=True, inplace=True)

    return df


def max_and_min(df):
    """Helper script to explore extremities to filter off."""

    min_values = df.groupby('page')['new_yMax'].min().to_list()
    max_values = df.groupby('page')['new_yMax'].max().to_list()

    min_max = min_values + max_values

    suggested = []
    for i, v in enumerate(min_max):

        candidate = df.loc[df['new_yMax'] == v]

        print(candidate)
        print()

        if len(candidate) < 3:
            suggested.append(v)

    return suggested


def apply_filter(df, filter_list):
    return df.loc[~(df['new_yMax'].isin(filter_list))]


def is_roman_numeral(num: str) -> bool:
    try:
        numeral.roman2int(num)
    except ValueError:
        return False

    return True


def build_toc(df):
    """Iterate through word list to build dataframe with the right TOC labels."""

    # Iterate through word list

    words = list(df['word'])
    labels = []
    pg_no = []
    caption = None
    label = None

    for i, word in enumerate(words):

        word = word.replace(",", "").replace(".", "")

        if word.isnumeric() and int(word) <= 332 or is_roman_numeral(word):
            # 332 is the maximum number of pages in the book.

            pg_no.append(word)

            if caption:
                if caption not in ["CONTENTS"]:
                    labels.append(caption)
                    caption = None
                else:
                    caption = None

            if label:

                label = label.replace("- ", "")
                labels.append(label)

                if label[-1] == "-":
                    pass
                else:
                    label = None

        else:
            if word.isupper() and len(word) > 1:
                if caption:
                    caption += " " + word
                else:
                    caption = word
            else:
                if label:
                    label += " " + word
                else:
                    label = word

    pg_no += [np.nan] * 3

    # Output Dataframe

    df_out = pd.DataFrame({"labels": labels, "page": pg_no})

    return df_out


def build_bookmark_arrays(df, label):
    """Build arrays that locate a certain label within the dataframe."""

    lb = np.where(df['labels'] == label)[0]  # label
    cp = lb + 1  # caption

    location = np.concatenate((lb, cp))
    arr = np.sort(location)

    return arr


def shift_cells_down(df_out, caption):
    """Shift page-label cells down at cardinal captions where no page number is associated."""

    arr_shift = np.where(df_out['labels'] == caption)[0] + 1

    for i in arr_shift:
        df_out.loc[i:, 'page'] = df_out.loc[i:, 'page'].shift(1)

    return df_out


def set_bookmark_levels(df_out):
    """Set bookmark levels with a dictionary."""

    d = BOOKMARK_LEVELS

    for key in d.keys():

        if d[key][0] == "default":
            df_out['BookmarkLevel'] = key
        else:
            df_out.loc[d[key], 'BookmarkLevel'] = key

    return df_out


def set_labels(df_out, loc_arr):
    """Set bookmark labels by string concatenation from different cardinal cells."""

    caption = df_out.loc[loc_arr, ['labels', 'page']]
    caption = caption.stack().groupby(level=0).apply(' '.join)

    this_index = None
    for i, row in enumerate(caption):

        last_index = this_index
        this_index = caption.index[i]

        if this_index and last_index:
            if this_index - last_index == 1:
                caption[this_index] = ": ".join((caption[last_index], caption[this_index]))
                caption[last_index] = np.nan

    return caption


def update_dataframe(df_out):
    """Update labels and rename columns."""

    df_out['labels'].update(parts_lbl)
    df_out['labels'].update(chapters_lbl)

    df_out.drop(df_out[df_out.labels == "Part"].index, inplace=True)
    df_out.drop(df_out[df_out.labels == "Chapter"].index, inplace=True)

    df_out['BookmarkPageNumber'] = None

    df_out.rename(columns={
        "labels": "BookmarkTitle",
        "page": "ActualPageNumber",
    }, inplace=True)

    return df_out


def offset_page_numbers(df_out, offset):
    """Offset page numbers where actual labels or numbers differ from index."""

    for i, n in enumerate(df_out['ActualPageNumber']):
        if not n:  # Skip None values
            continue

        loc = df_out.index[i]

        if is_roman_numeral(n):
            df_out.loc[loc, 'BookmarkPageNumber'] = numeral.roman2int(n)
        else:
            df_out.loc[loc, 'BookmarkPageNumber'] = int(n) + offset

    return df_out


if __name__ == '__main__':

    toc_df_raw = build_dataframe('toc.html')
    toc_df_raw = consolidate_bboxes(toc_df_raw)

    suggested_filter = max_and_min(toc_df_raw)
    toc_df_raw = apply_filter(toc_df_raw, suggested_filter)  # [44, 45, 127, 551]

    toc_df = build_toc(toc_df_raw)

    toc_df = shift_cells_down(toc_df, "Part")

    # Build Bookmark Arrays

    chapters = build_bookmark_arrays(toc_df, "Chapter")
    parts = build_bookmark_arrays(toc_df, "Part")
    BOOKMARK_LEVELS = {
        1: parts,
        2: chapters,
        3: ["default"],
    }

    toc_df = set_bookmark_levels(toc_df)

    parts_lbl = set_labels(toc_df, parts)
    chapters_lbl = set_labels(toc_df, chapters)
    chapters_lbl = chapters_lbl.str.replace(" \d+$", "", regex=True)

    toc_df = update_dataframe(toc_df)
    toc_df = offset_page_numbers(toc_df, 24)

    print(toc_df)

toc.html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Evolution, development, and children&apos;s learning</title>
<meta name="Keywords" content="http://archive.org/details/evolutiondevelop00fish"/>
<meta name="Author" content="Fishbein, Harold D"/>
<meta name="Creator" content="Digitized by the Internet Archive"/>
<meta name="Producer" content="Recoded by LuraDocument PDF v2.53"/>
<meta name="CreationDate" content=""/>
</head>
<body>
<doc>
  <page width="389.000000" height="612.000000">
    <flow>
      <block xMin="16.975480" yMin="110.518904" xMax="115.665742" yMax="126.679434">
        <line xMin="16.975480" yMin="110.518904" xMax="115.665742" yMax="126.679434">
          <word xMin="16.975480" yMin="110.518904" xMax="115.665742" yMax="126.679434">CONTENTS</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="17.263200" yMin="293.420111" xMax="56.250411" yMax="299.804267">
        <line xMin="17.263200" yMin="293.420111" xMax="56.250411" yMax="299.804267">
          <word xMin="17.263200" yMin="293.420111" xMax="56.250411" yMax="299.804267">Foreword</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="74.231760" yMin="295.046148" xMax="81.425335" yMax="299.757939">
        <line xMin="74.231760" yMin="295.046148" xMax="81.425335" yMax="299.757939">
          <word xMin="74.231760" yMin="295.046148" xMax="81.425335" yMax="299.757939">xi</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="17.119340" yMin="315.807962" xMax="149.332434" yMax="322.439750">
        <line xMin="17.119340" yMin="315.807962" xMax="149.332434" yMax="322.439750">
          <word xMin="17.119340" yMin="316.640467" xMax="46.036926" yMax="322.052187">Preface</word>
          <word xMin="49.775560" yMin="315.987128" xMax="64.450143" yMax="322.395030">and</word>
          <word xMin="68.333500" yMin="315.807962" xMax="149.332434" yMax="322.439750">Acknowledgements</word>
        </line>
      </block>
      <block xMin="16.831620" yMin="338.716199" xMax="94.082714" yMax="344.046505">
        <line xMin="16.831620" yMin="338.716199" xMax="94.082714" yMax="344.046505">
          <word xMin="16.831620" yMin="338.760569" xMax="61.857498" yMax="343.675894">Illustration</word>
          <word xMin="65.600160" yMin="338.716199" xMax="94.082714" yMax="344.046505">Credits</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="168.316200" yMin="316.433218" xMax="177.522665" yMax="322.463452">
        <line xMin="168.316200" yMin="316.433218" xMax="177.522665" yMax="322.463452">
          <word xMin="168.316200" yMin="316.433218" xMax="177.522665" yMax="322.463452">xv</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="112.930100" yMin="338.841088" xMax="128.180411" yMax="343.835565">
        <line xMin="112.930100" yMin="338.841088" xMax="128.180411" yMax="343.835565">
          <word xMin="112.930100" yMin="338.841088" xMax="128.180411" yMax="343.835565">xvii</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="16.687760" yMin="358.565974" xMax="49.775272" yMax="366.057124">
        <line xMin="16.687760" yMin="358.565974" xMax="49.775272" yMax="366.057124">
          <word xMin="16.687760" yMin="358.565974" xMax="39.561500" yMax="366.057124">Part</word>
          <word xMin="44.740460" yMin="359.426498" xMax="49.775272" yMax="366.022103">1</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="63.010680" yMin="355.260450" xMax="216.080022" yMax="366.882191">
        <line xMin="63.010680" yMin="355.260450" xMax="216.080022" yMax="366.882191">
          <word xMin="63.010680" yMin="355.889341" xMax="129.183978" yMax="366.725218">OVERVIEW</word>
          <word xMin="133.645940" yMin="355.260450" xMax="160.260615" yMax="366.882191">AND</word>
          <word xMin="165.295140" yMin="355.687553" xMax="216.080022" yMax="366.775585">THEORY</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="17.550920" yMin="381.853339" xMax="56.536692" yMax="387.749015">
        <line xMin="17.550920" yMin="381.853339" xMax="56.536692" yMax="387.749015">
          <word xMin="17.550920" yMin="381.853339" xMax="49.054534" yMax="387.749015">Chapter</word>
          <word xMin="53.228200" yMin="383.246837" xMax="56.536692" yMax="387.580962">1</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="73.368600" yMin="379.620065" xMax="204.569208" yMax="388.665982">
        <line xMin="73.368600" yMin="379.620065" xMax="204.569208" yMax="388.665982">
          <word xMin="73.368600" yMin="380.381084" xMax="121.705560" yMax="388.296261">OVERVIEW</word>
          <word xMin="125.302060" yMin="379.921390" xMax="145.155603" yMax="388.590770">AND</word>
          <word xMin="148.895100" yMin="379.620065" xMax="197.232060" yMax="388.665982">SUMMARY</word>
          <word xMin="201.404000" yMin="383.397047" xMax="204.569208" yMax="387.543469">1</word>
        </line>
      </block>
      <block xMin="73.368600" yMin="396.203985" xMax="340.803765" yMax="402.973982">
        <line xMin="73.368600" yMin="396.203985" xMax="340.803765" yMax="402.973982">
          <word xMin="73.368600" yMin="396.984912" xMax="115.378597" yMax="402.488222">Evolution,</word>
          <word xMin="119.979240" yMin="396.688096" xMax="156.950109" yMax="402.742076">Learning</word>
          <word xMin="161.267060" yMin="396.203985" xMax="176.516508" yMax="402.862911">and</word>
          <word xMin="180.832020" yMin="396.601227" xMax="227.874240" yMax="402.763758">Adaptation</word>
          <word xMin="232.190040" yMin="398.358487" xMax="238.520455" yMax="402.504909">1,</word>
          <word xMin="242.691820" yMin="396.676348" xMax="296.635867" yMax="402.565240">Evolutionary</word>
          <word xMin="300.523540" yMin="396.236577" xMax="330.012538" yMax="402.675008">Theory</word>
          <word xMin="334.186780" yMin="398.639857" xMax="340.803765" yMax="402.973982">3,</word>
        </line>
      </block>
      <block xMin="72.937020" yMin="407.087577" xMax="340.949351" yMax="436.352955">
        <line xMin="73.368600" yMin="407.087577" xMax="340.949351" yMax="414.071343">
          <word xMin="73.368600" yMin="408.205992" xMax="115.378597" yMax="413.709302">Evolution,</word>
          <word xMin="119.259940" yMin="407.399744" xMax="173.930768" yMax="413.910543">Development</word>
          <word xMin="177.667100" yMin="407.475738" xMax="192.771537" yMax="414.071343">and</word>
          <word xMin="196.656620" yMin="407.859423" xMax="241.972520" yMax="413.795806">Behavioral</word>
          <word xMin="245.712880" yMin="407.678447" xMax="292.755100" yMax="413.840978">Adaptation</word>
          <word xMin="296.495460" yMin="409.579567" xMax="302.825875" yMax="413.725989">6,</word>
          <word xMin="306.853380" yMin="407.087577" xMax="322.245249" yMax="413.808693">The</word>
          <word xMin="325.699040" yMin="408.613188" xMax="340.949351" yMax="413.607665">Pri-</word>
        </line>
        <line xMin="73.224740" yMin="418.257984" xMax="340.665084" yMax="425.042421">
          <word xMin="73.224740" yMin="418.484203" xMax="93.077420" yMax="424.985956">mate</word>
          <word xMin="96.673920" yMin="418.855220" xMax="150.338304" yMax="424.713582">Evolutionary</word>
          <word xMin="154.074060" yMin="419.102236" xMax="175.942219" yMax="424.831694">Lines</word>
          <word xMin="179.825000" yMin="420.054138" xMax="191.045217" yMax="424.953632">10,</word>
          <word xMin="194.786440" yMin="418.257984" xMax="210.323320" yMax="425.042421">The</word>
          <word xMin="213.632100" yMin="418.820819" xMax="283.263217" yMax="424.901936">Hunter-Gatherer</word>
          <word xMin="287.000700" yMin="418.855220" xMax="340.665084" yMax="424.713582">Evolutionary</word>
        </line>
        <line xMin="72.937020" yMin="428.400463" xMax="260.385737" yMax="436.352955">
          <word xMin="72.937020" yMin="429.726143" xMax="91.063380" yMax="435.662526">Line</word>
          <word xMin="94.947600" yMin="430.987498" xMax="106.167817" yMax="435.886992">14,</word>
          <word xMin="110.052900" yMin="429.859145" xMax="141.846535" yMax="435.809096">Aspects</word>
          <word xMin="145.586320" yMin="430.247338" xMax="153.929625" yMax="435.712202">of</word>
          <word xMin="156.519680" yMin="428.400463" xMax="186.872701" yMax="436.352955">Human</word>
          <word xMin="190.614500" yMin="429.582236" xMax="244.990991" yMax="436.057981">Development</word>
          <word xMin="249.165520" yMin="430.843638" xMax="260.385737" yMax="435.743132">17.</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="17.407060" yMin="451.481579" xMax="56.537268" yMax="457.428499">
        <line xMin="17.407060" yMin="451.481579" xMax="56.537268" yMax="457.428499">
          <word xMin="17.407060" yMin="451.481579" xMax="48.910674" yMax="457.377255">Chapter</word>
          <word xMin="52.940480" yMin="452.716708" xMax="56.537268" yMax="457.428499">2</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="72.937020" yMin="449.977638" xMax="195.649600" yMax="458.199573">
        <line xMin="72.937020" yMin="449.977638" xMax="195.649600" yMax="458.199573">
          <word xMin="72.937020" yMin="449.977638" xMax="110.194458" yMax="458.112178">THEORY</word>
          <word xMin="113.937120" yMin="450.448159" xMax="125.733065" yMax="458.174503">OF</word>
          <word xMin="129.330140" yMin="450.347717" xMax="183.274187" yMax="458.199573">EVOLUTION</word>
          <word xMin="187.449580" yMin="452.333023" xMax="195.649600" yMax="457.704036">21</word>
        </line>
      </block>
      <block xMin="73.080880" yMin="465.983340" xMax="340.661343" yMax="472.747476">
        <line xMin="73.080880" yMin="465.983340" xMax="340.661343" yMax="472.747476">
          <word xMin="73.080880" yMin="466.758806" xMax="103.436491" yMax="472.439642">History</word>
          <word xMin="108.326580" yMin="467.621124" xMax="119.691808" yMax="472.583940">21,</word>
          <word xMin="124.582760" yMin="466.954244" xMax="167.455917" yMax="472.570628">Precursors</word>
          <word xMin="172.632000" yMin="467.451928" xMax="180.257155" yMax="472.446405">of</word>
          <word xMin="184.572380" yMin="466.245723" xMax="214.351400" yMax="472.747476">Darwin</word>
          <word xMin="219.098780" yMin="466.320828" xMax="233.773363" yMax="472.728730">and</word>
          <word xMin="239.095320" yMin="466.579624" xMax="271.607968" yMax="472.664134">Spencer</word>
          <word xMin="276.642780" yMin="467.527938" xMax="287.862997" yMax="472.427432">21,</word>
          <word xMin="293.042820" yMin="465.983340" xMax="322.676829" yMax="472.453432">Darwin</word>
          <word xMin="325.986760" yMin="466.033108" xMax="340.661343" yMax="472.441010">and</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="73.224740" yMin="477.304862" xMax="340.806642" yMax="516.833445">
        <line xMin="73.368600" yMin="477.304862" xMax="340.516908" yMax="483.649442">
          <word xMin="73.368600" yMin="477.412558" xMax="105.591226" yMax="483.442792">Wallace</word>
          <word xMin="112.642380" yMin="478.605158" xMax="123.862597" yMax="483.504652">25,</word>
          <word xMin="130.912600" yMin="477.757301" xMax="183.851929" yMax="483.536511">Evolutionary</word>
          <word xMin="190.470640" yMin="477.354630" xMax="219.244942" yMax="483.637019">Theory</word>
          <word xMin="226.004060" yMin="477.304862" xMax="240.533632" yMax="483.649442">and</word>
          <word xMin="247.295340" yMin="477.838741" xMax="281.966751" yMax="483.516184">Genetics</word>
          <word xMin="288.870880" yMin="478.461298" xMax="296.351025" yMax="483.360792">to</word>
          <word xMin="303.544600" yMin="477.374553" xMax="322.102540" yMax="483.452279">1935</word>
          <word xMin="329.151680" yMin="478.410624" xMax="340.516908" yMax="483.373440">26,</word>
        </line>
        <line xMin="73.368600" yMin="487.346947" xMax="340.806642" yMax="494.606943">
          <word xMin="73.368600" yMin="487.346947" xMax="105.882111" yMax="494.445730">Modern</word>
          <word xMin="110.772200" yMin="488.664436" xMax="149.466800" yMax="494.296650">Synthetic</word>
          <word xMin="154.217920" yMin="488.019257" xMax="183.706918" yMax="494.457688">Theory</word>
          <word xMin="188.744320" yMin="489.017323" xMax="196.944340" yMax="494.388336">of</word>
          <word xMin="200.828560" yMin="488.673469" xMax="240.680657" yMax="494.474163">Evolution</word>
          <word xMin="245.425160" yMin="489.581936" xMax="256.932809" yMax="494.606943">27,</word>
          <word xMin="262.256780" yMin="489.168438" xMax="309.727127" yMax="494.350617">Definitions,</word>
          <word xMin="314.621820" yMin="488.596570" xMax="340.806642" yMax="494.313589">Evolu-</word>
        </line>
        <line xMin="73.224740" yMin="499.071140" xMax="340.657891" yMax="505.672670">
          <word xMin="73.224740" yMin="499.752530" xMax="102.287337" yMax="505.191388">tionary</word>
          <word xMin="107.031840" yMin="500.260136" xMax="146.452357" yMax="505.424224">Strategies</word>
          <word xMin="151.340720" yMin="499.264768" xMax="166.015303" yMax="505.672670">and</word>
          <word xMin="170.617960" yMin="499.868323" xMax="213.775960" yMax="505.522021">Inventions</word>
          <word xMin="218.811060" yMin="500.759598" xMax="230.031277" yMax="505.659092">28,</word>
          <word xMin="234.923380" yMin="500.062856" xMax="286.132936" yMax="505.653233">Canalization</word>
          <word xMin="290.597200" yMin="499.071140" xMax="305.414205" yMax="505.541232">and</word>
          <word xMin="310.018300" yMin="499.660217" xMax="340.657891" yMax="505.394197">Genetic</word>
        </line>
        <line xMin="73.512460" yMin="510.141105" xMax="340.804628" yMax="516.833445">
          <word xMin="73.512460" yMin="510.619818" xMax="125.737093" yMax="516.321007">Assimilation</word>
          <word xMin="132.351200" yMin="511.742726" xMax="143.428995" yMax="516.580030">32,</word>
          <word xMin="151.340720" yMin="510.141105" xMax="166.590168" yMax="516.800031">The</word>
          <word xMin="174.214460" yMin="510.945543" xMax="191.477660" yMax="516.599241">Five</word>
          <word xMin="199.246100" yMin="510.727451" xMax="222.551420" yMax="516.833445">Major</word>
          <word xMin="230.176000" yMin="511.283047" xMax="259.093586" yMax="516.694767">Factors</word>
          <word xMin="266.860300" yMin="511.329228" xMax="274.485455" yMax="516.323705">in</word>
          <word xMin="282.109460" yMin="510.557381" xMax="321.813957" yMax="516.336591">Evolution</word>
          <word xMin="329.439400" yMin="511.354564" xMax="340.804628" yMax="516.317380">40,</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="73.512460" yMin="519.982323" xMax="128.609689" yMax="527.466689">
        <line xMin="73.512460" yMin="519.982323" xMax="128.609689" yMax="527.466689">
          <word xMin="73.512460" yMin="519.982323" xMax="113.505252" yMax="527.466689">Summary</word>
          <word xMin="117.102040" yMin="522.094296" xMax="128.609689" yMax="527.119303">43.</word>
        </line>
      </block>
      <block xMin="330.014840" yMin="544.868388" xMax="339.797895" yMax="551.276290">
        <line xMin="330.014840" yMin="544.868388" xMax="339.797895" yMax="551.276290">
          <word xMin="330.014840" yMin="544.868388" xMax="339.797895" yMax="551.276290">vu</word>
        </line>
      </block>
    </flow>
  </page>
  <page width="389.000000" height="612.000000">
    <flow>
      <block xMin="47.617660" yMin="40.654677" xMax="59.988469" yMax="44.706117">
        <line xMin="47.617660" yMin="40.654677" xMax="59.988469" yMax="44.706117">
          <word xMin="47.617660" yMin="40.654677" xMax="59.988469" yMax="44.706117">viii</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="68.333500" yMin="38.651306" xMax="106.167529" yMax="44.846628">
        <line xMin="68.333500" yMin="38.651306" xMax="106.167529" yMax="44.846628">
          <word xMin="68.333500" yMin="38.651306" xMax="106.167529" yMax="44.846628">CONTENTS</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="45.891340" yMin="66.123168" xMax="84.733540" yMax="71.965700">
        <line xMin="45.891340" yMin="66.123168" xMax="84.733540" yMax="71.965700">
          <word xMin="45.891340" yMin="66.123168" xMax="77.110974" yMax="71.965700">Chapter</word>
          <word xMin="81.280900" yMin="67.035302" xMax="84.733540" yMax="71.558261">3</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.277440" yMin="62.968901" xMax="326.273905" yMax="71.879473">
        <line xMin="101.277440" yMin="62.968901" xMax="326.273905" yMax="71.879473">
          <word xMin="101.277440" yMin="63.587759" xMax="164.572963" yMax="71.879473">EMBRYOLOGY</word>
          <word xMin="168.316200" yMin="63.242111" xMax="187.882311" yMax="71.785980">AND</word>
          <word xMin="191.909240" yMin="63.318153" xMax="272.184846" yMax="71.407464">DEVELOPMENTAL</word>
          <word xMin="275.779620" yMin="62.968901" xMax="313.182069" yMax="71.135102">THEORY</word>
          <word xMin="316.635860" yMin="64.451998" xMax="326.273905" yMax="70.764917">46</word>
        </line>
      </block>
      <block xMin="101.277440" yMin="78.797152" xMax="367.274868" yMax="108.549492">
        <line xMin="101.421300" yMin="78.797152" xMax="367.274868" yMax="86.599255">
          <word xMin="101.421300" yMin="80.674745" xMax="132.495923" yMax="86.490139">Aspects</word>
          <word xMin="136.667000" yMin="80.961608" xMax="144.723735" yMax="86.238770">of</word>
          <word xMin="147.888080" yMin="78.797152" xMax="177.667100" yMax="86.599255">Human</word>
          <word xMin="182.126760" yMin="79.693903" xMax="231.758460" yMax="86.195656">Embryology</word>
          <word xMin="235.642680" yMin="80.825003" xMax="247.295340" yMax="85.913331">47,</word>
          <word xMin="251.611140" yMin="79.836922" xMax="309.006964" yMax="85.620655">Psychological</word>
          <word xMin="313.183220" yMin="79.310282" xMax="367.274868" yMax="85.752106">Development</word>
        </line>
        <line xMin="101.421300" yMin="90.675222" xMax="367.273429" yMax="97.647062">
          <word xMin="101.421300" yMin="92.873079" xMax="112.354085" yMax="97.647062">55,</word>
          <word xMin="116.670460" yMin="92.398049" xMax="170.191559" yMax="97.406095">Interactionist</word>
          <word xMin="174.358320" yMin="91.352040" xMax="219.820957" yMax="97.307646">Hypothesis</word>
          <word xMin="224.133880" yMin="92.297639" xMax="235.066665" yMax="97.071622">58,</word>
          <word xMin="239.383040" yMin="90.675222" xMax="293.474688" yMax="97.117046">Development</word>
          <word xMin="297.790200" yMin="90.890599" xMax="320.663940" yMax="96.883519">Leads</word>
          <word xMin="324.404300" yMin="91.840723" xMax="331.741160" yMax="96.646366">to</word>
          <word xMin="335.481520" yMin="91.520395" xMax="367.273429" yMax="96.726320">Increas-</word>
        </line>
        <line xMin="101.277440" yMin="101.478295" xMax="366.988586" yMax="108.549492">
          <word xMin="101.277440" yMin="103.210139" xMax="113.504965" yMax="108.549492">ing</word>
          <word xMin="118.396780" yMin="103.227348" xMax="177.229766" yMax="108.365429">Differentiation</word>
          <word xMin="181.695180" yMin="102.067308" xMax="196.369763" yMax="108.475210">and</word>
          <word xMin="200.828560" yMin="102.443770" xMax="243.408243" yMax="108.021708">Hierarchic</word>
          <word xMin="247.870780" yMin="102.643748" xMax="292.610089" yMax="107.971793">Integration</word>
          <word xMin="297.070900" yMin="103.079884" xMax="308.436128" yMax="108.042700">61,</word>
          <word xMin="313.039360" yMin="101.478295" xMax="366.988586" yMax="107.903158">Development</word>
        </line>
      </block>
      <block xMin="101.277440" yMin="111.685926" xMax="212.194075" yMax="119.226526">
        <line xMin="101.277440" yMin="111.685926" xMax="212.194075" yMax="119.226526">
          <word xMin="101.277440" yMin="114.207778" xMax="108.757585" yMax="119.107272">in</word>
          <word xMin="112.642380" yMin="113.730001" xMax="137.817305" yMax="119.226526">Stages</word>
          <word xMin="141.702100" yMin="113.963476" xMax="153.209749" yMax="118.988483">63,</word>
          <word xMin="157.095120" yMin="111.685926" xMax="197.232923" yMax="119.197429">Summary</word>
          <word xMin="201.116280" yMin="113.682106" xMax="212.194075" yMax="118.519410">66.</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="46.179060" yMin="139.943129" xMax="78.547560" yMax="147.199085">
        <line xMin="46.179060" yMin="139.943129" xMax="78.547560" yMax="147.199085">
          <word xMin="46.179060" yMin="139.943129" xMax="68.334651" yMax="147.199085">Part</word>
          <word xMin="73.368600" yMin="140.176604" xMax="78.547560" yMax="146.961041">2</word>
        </line>
      </block>
      <block xMin="45.891340" yMin="162.754559" xMax="85.020972" yMax="168.650235">
        <line xMin="45.891340" yMin="162.754559" xMax="85.020972" yMax="168.650235">
          <word xMin="45.891340" yMin="162.754559" xMax="77.394954" yMax="168.650235">Chapter</word>
          <word xMin="81.280900" yMin="163.407898" xMax="85.020972" yMax="168.307392">4</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="91.638820" yMin="136.824930" xMax="167.023762" yMax="147.797627">
        <line xMin="91.638820" yMin="136.824930" xMax="167.023762" yMax="147.797627">
          <word xMin="91.638820" yMin="136.824930" xMax="167.023762" yMax="147.797627">PHYLOGENY</word>
        </line>
      </block>
      <block xMin="101.421300" yMin="160.757537" xMax="297.358045" yMax="168.872043">
        <line xMin="101.421300" yMin="160.757537" xMax="297.358045" yMax="168.872043">
          <word xMin="101.421300" yMin="161.145699" xMax="119.115217" yMax="168.872043">THE</word>
          <word xMin="122.856440" yMin="160.812719" xMax="244.423894" yMax="168.775388">PALEOANTHROPOLOGICAL</word>
          <word xMin="248.590080" yMin="160.757537" xMax="284.552778" yMax="168.609393">RECORD</word>
          <word xMin="288.583160" yMin="162.441518" xMax="297.358045" yMax="168.189067">69</word>
        </line>
      </block>
      <block xMin="101.421300" yMin="176.911577" xMax="367.561149" yMax="216.441369">
        <line xMin="101.565160" yMin="176.911577" xMax="367.419015" yMax="183.632693">
          <word xMin="101.565160" yMin="177.162228" xMax="140.697382" yMax="183.570130">Overview</word>
          <word xMin="144.291580" yMin="178.606384" xMax="155.656808" yMax="183.569200">69,</word>
          <word xMin="158.965300" yMin="176.911577" xMax="174.357169" yMax="183.632693">The</word>
          <word xMin="177.954820" yMin="178.192886" xMax="201.405151" yMax="183.312875">Fossil</word>
          <word xMin="205.144360" yMin="177.118810" xMax="233.918662" yMax="183.401199">Record</word>
          <word xMin="237.512860" yMin="176.917927" xMax="252.474875" yMax="183.451340">and</word>
          <word xMin="256.070800" yMin="177.830061" xMax="279.950985" yMax="183.043901">Fossil</word>
          <word xMin="283.404200" yMin="177.326948" xMax="310.163886" yMax="183.169480">Dating</word>
          <word xMin="314.334100" yMin="178.369338" xMax="325.554317" yMax="183.268832">71,</word>
          <word xMin="329.151680" yMin="177.688884" xMax="367.419015" yMax="183.258907">Placental</word>
        </line>
        <line xMin="101.421300" yMin="188.042216" xMax="367.273429" yMax="194.633772">
          <word xMin="101.421300" yMin="188.582398" xMax="151.629591" yMax="194.561749">Adaptations</word>
          <word xMin="155.081080" yMin="189.734278" xMax="166.301297" yMax="194.633772">72,</word>
          <word xMin="169.898660" yMin="188.541678" xMax="202.121286" yMax="194.571912">Primate</word>
          <word xMin="205.432080" yMin="188.614989" xMax="244.996745" yMax="194.373846">Evolution</word>
          <word xMin="248.590080" yMin="188.045820" xMax="263.407085" yMax="194.515912">and</word>
          <word xMin="266.860300" yMin="188.706398" xMax="279.375257" yMax="194.171262">the</word>
          <word xMin="282.684900" yMin="188.042216" xMax="330.737017" yMax="194.337044">Anthropoid</word>
          <word xMin="334.186780" yMin="188.412280" xMax="367.273429" yMax="194.604210">Adapta-</word>
        </line>
        <line xMin="101.565160" yMin="198.130466" xMax="367.561149" yMax="205.279412">
          <word xMin="101.565160" yMin="199.941083" xMax="120.986260" yMax="205.029411">tions</word>
          <word xMin="124.295040" yMin="200.379918" xMax="135.515257" yMax="205.279412">73,</word>
          <word xMin="138.968760" yMin="199.582750" xMax="172.777011" yMax="205.118852">Climate,</word>
          <word xMin="176.372360" yMin="198.504150" xMax="212.196089" yMax="205.208305">Hominid</word>
          <word xMin="215.646140" yMin="199.336655" xMax="257.509400" yMax="204.820742">Evolution,</word>
          <word xMin="261.105900" yMin="198.547600" xMax="275.922905" yMax="205.017692">and</word>
          <word xMin="279.376120" yMin="199.208178" xMax="291.891077" yMax="204.673042">the</word>
          <word xMin="294.913000" yMin="198.130466" xMax="331.310731" yMax="204.942042">Hominid</word>
          <word xMin="334.474500" yMin="198.770200" xMax="367.561149" yMax="204.962130">Adapta-</word>
        </line>
        <line xMin="101.421300" yMin="208.957003" xMax="221.544688" yMax="216.441369">
          <word xMin="101.421300" yMin="210.844582" xMax="120.984821" yMax="215.970224">tions</word>
          <word xMin="124.870480" yMin="211.262604" xMax="136.235708" yMax="216.225420">79,</word>
          <word xMin="140.119640" yMin="208.957003" xMax="180.112432" yMax="216.441369">Summary</word>
          <word xMin="183.996940" yMin="210.067306" xMax="206.581521" yMax="215.984466">Table</word>
          <word xMin="210.179460" yMin="210.831024" xMax="221.544688" yMax="215.793840">85.</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="46.035200" yMin="232.260656" xMax="85.021260" yMax="238.129194">
        <line xMin="46.035200" yMin="232.260656" xMax="85.021260" yMax="238.129194">
          <word xMin="46.035200" yMin="232.260656" xMax="77.393803" yMax="238.129194">Chapter</word>
          <word xMin="81.568620" yMin="233.337462" xMax="85.021260" yMax="237.860421">5</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.852880" yMin="230.119711" xMax="343.106100" yMax="238.663580">
        <line xMin="101.852880" yMin="230.119711" xMax="343.106100" yMax="238.663580">
          <word xMin="101.852880" yMin="231.165752" xMax="129.474000" yMax="238.402485">BRAIN</word>
          <word xMin="133.502080" yMin="230.754937" xMax="180.830869" yMax="238.505026">FUNCTION</word>
          <word xMin="184.572380" yMin="230.119711" xMax="204.138491" yMax="238.663580">AND</word>
          <word xMin="207.877700" yMin="230.723266" xMax="261.394483" yMax="238.512931">EVOLUTION</word>
          <word xMin="265.277840" yMin="231.000159" xMax="276.642205" yMax="238.443818">OF</word>
          <word xMin="280.670860" yMin="230.630079" xMax="298.364777" yMax="238.356423">THE</word>
          <word xMin="302.249860" yMin="231.075264" xMax="330.302560" yMax="238.425071">BRAIN</word>
          <word xMin="334.042920" yMin="232.350223" xMax="343.106100" yMax="238.286606">86</word>
        </line>
      </block>
      <block xMin="101.996740" yMin="246.628511" xMax="141.267067" yMax="253.059027">
        <line xMin="101.996740" yMin="246.628511" xMax="141.267067" yMax="253.059027">
          <word xMin="101.996740" yMin="246.628511" xMax="141.267067" yMax="253.059027">Overview</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="147.168780" yMin="248.090764" xMax="158.534008" yMax="253.053580">
        <line xMin="147.168780" yMin="248.090764" xMax="158.534008" yMax="253.053580">
          <word xMin="147.168780" yMin="248.090764" xMax="158.534008" yMax="253.053580">86,</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="164.719700" yMin="246.345284" xMax="367.993880" yMax="253.129721">
        <line xMin="164.719700" yMin="246.345284" xMax="367.993880" yMax="253.129721">
          <word xMin="164.719700" yMin="246.517211" xMax="199.824417" yMax="253.086808">Methods</word>
          <word xMin="205.575940" yMin="247.627498" xMax="213.487665" yMax="252.809677">of</word>
          <word xMin="218.523340" yMin="246.948838" xMax="255.349198" yMax="252.979072">Studying</word>
          <word xMin="261.393620" yMin="247.039326" xMax="283.978201" yMax="252.956486">Brain</word>
          <word xMin="289.734040" yMin="247.132529" xMax="329.586137" yMax="252.933223">Evolution</word>
          <word xMin="335.193800" yMin="248.090764" xMax="346.559028" yMax="253.053580">88,</word>
          <word xMin="352.457000" yMin="246.345284" xMax="367.993880" yMax="253.129721">The</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.852880" yMin="257.479527" xMax="368.281600" yMax="264.012940">
        <line xMin="101.852880" yMin="257.479527" xMax="368.281600" yMax="264.012940">
          <word xMin="101.852880" yMin="258.492103" xMax="125.158200" yMax="263.580431">Fossil</word>
          <word xMin="128.754700" yMin="257.561887" xMax="157.383991" yMax="263.812615">Record</word>
          <word xMin="160.979340" yMin="258.930938" xMax="172.199557" yMax="263.830432">90,</word>
          <word xMin="175.653060" yMin="258.217004" xMax="214.207828" yMax="263.828864">Increased</word>
          <word xMin="217.516320" yMin="258.213384" xMax="238.952899" yMax="263.829768">Brain</word>
          <word xMin="242.260240" yMin="258.334638" xMax="258.946849" yMax="263.799502">Size</word>
          <word xMin="262.256780" yMin="258.880264" xMax="273.622008" yMax="263.843080">90,</word>
          <word xMin="277.362080" yMin="258.039663" xMax="298.941080" yMax="263.693361">Brain</word>
          <word xMin="302.106000" yMin="258.296633" xMax="318.937620" yMax="263.808988">Size</word>
          <word xMin="321.814820" yMin="257.479527" xMax="336.776835" yMax="264.012940">and</word>
          <word xMin="339.797320" yMin="258.145518" xMax="357.205531" yMax="263.846707">Rate</word>
          <word xMin="360.513160" yMin="258.635963" xMax="368.281600" yMax="263.724291">of</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.996740" yMin="268.269027" xMax="367.707311" yMax="274.944719">
        <line xMin="101.996740" yMin="268.269027" xMax="367.707311" yMax="274.944719">
          <word xMin="101.996740" yMin="268.981151" xMax="146.449480" yMax="274.804460">Maturation</word>
          <word xMin="150.765280" yMin="269.813624" xMax="162.130508" yMax="274.776440">93,</word>
          <word xMin="166.446020" yMin="268.825512" xMax="235.352083" yMax="274.843308">Hunter-Gatherer</word>
          <word xMin="239.526900" yMin="268.658125" xMax="290.304876" yMax="274.705320">Adaptations</word>
          <word xMin="294.481420" yMin="268.269027" xMax="309.443435" yMax="274.802440">and</word>
          <word xMin="314.046380" yMin="268.445478" xMax="323.684425" yMax="274.758397">an</word>
          <word xMin="327.856940" yMin="268.419221" xMax="367.707311" yMax="274.944719">Overview</word>
        </line>
      </block>
      <block xMin="101.421300" yMin="279.145364" xMax="367.565177" yMax="285.979477">
        <line xMin="101.421300" yMin="279.145364" xMax="367.565177" yMax="285.979477">
          <word xMin="101.421300" yMin="280.201358" xMax="109.764605" yMax="285.666222">of</word>
          <word xMin="112.498520" yMin="279.446688" xMax="136.956159" yMax="285.854590">Man&apos;s</word>
          <word xMin="140.263500" yMin="279.627664" xMax="163.857979" yMax="285.809418">Three</word>
          <word xMin="167.596900" yMin="279.924465" xMax="194.211575" yMax="285.735336">Brains</word>
          <word xMin="197.951360" yMin="280.941518" xMax="209.171577" yMax="285.841012">95,</word>
          <word xMin="212.768940" yMin="279.196037" xMax="228.160809" yMax="285.917153">The</word>
          <word xMin="231.614600" yMin="279.792337" xMax="259.808858" yMax="285.948083">Limbic</word>
          <word xMin="263.695380" yMin="279.666558" xMax="292.609514" yMax="285.979477">System</word>
          <word xMin="296.639320" yMin="280.639288" xMax="312.752791" yMax="285.916450">102,</word>
          <word xMin="316.348140" yMin="279.145364" xMax="331.885020" yMax="285.929801">The</word>
          <word xMin="335.481520" yMin="279.913590" xMax="367.565177" yMax="285.917818">Neocor-</word>
        </line>
      </block>
      <block xMin="101.996740" yMin="289.985537" xMax="280.527575" yMax="307.941749">
        <line xMin="119.979240" yMin="289.985537" xMax="280.527575" yMax="296.706653">
          <word xMin="119.979240" yMin="291.466793" xMax="135.947700" yMax="296.696463">104,</word>
          <word xMin="142.709120" yMin="289.985537" xMax="173.492858" yMax="296.706653">Growth</word>
          <word xMin="180.112720" yMin="290.186420" xMax="194.929725" yMax="296.656512">and</word>
          <word xMin="201.835580" yMin="290.507652" xMax="248.161377" yMax="296.576332">Maturation</word>
          <word xMin="254.632200" yMin="291.134718" xMax="262.975505" yMax="296.599582">of</word>
          <word xMin="268.155040" yMin="291.184486" xMax="280.527575" yMax="296.587160">the</word>
        </line>
        <line xMin="101.996740" yMin="300.430246" xMax="227.297649" yMax="307.941749">
          <word xMin="101.996740" yMin="301.665407" xMax="142.998566" yMax="307.633450">Neocortex</word>
          <word xMin="147.312640" yMin="302.506008" xMax="163.426111" yMax="307.783170">109,</word>
          <word xMin="167.309180" yMin="300.430246" xMax="207.446983" yMax="307.941749">Summary</word>
          <word xMin="211.474200" yMin="302.294298" xMax="227.297649" yMax="307.476477">111.</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.565160" yMin="291.292183" xMax="113.217820" yMax="296.380511">
        <line xMin="101.565160" yMin="291.292183" xMax="113.217820" yMax="296.380511">
          <word xMin="101.565160" yMin="291.292183" xMax="113.217820" yMax="296.380511">tex</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="46.466780" yMin="323.344848" xMax="85.452840" yMax="329.187380">
        <line xMin="46.466780" yMin="323.344848" xMax="85.452840" yMax="329.187380">
          <word xMin="46.466780" yMin="323.344848" xMax="77.686414" yMax="329.187380">Chapter</word>
          <word xMin="81.568620" yMin="323.948403" xMax="85.452840" yMax="329.036731">6</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="286.856840" yMin="289.029984" xMax="367.849157" yMax="296.945161">
        <line xMin="286.856840" yMin="289.029984" xMax="367.849157" yMax="296.945161">
          <word xMin="286.856840" yMin="289.029984" xMax="317.067440" yMax="296.945161">Human</word>
          <word xMin="323.685000" yMin="290.772766" xMax="346.269581" yMax="296.689926">Brain</word>
          <word xMin="352.744720" yMin="290.085978" xMax="367.849157" yMax="296.681583">and</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.709020" yMin="321.870784" xMax="285.563251" yMax="329.735078">
        <line xMin="101.709020" yMin="321.870784" xMax="285.563251" yMax="329.735078">
          <word xMin="101.709020" yMin="321.935935" xMax="161.120323" yMax="329.718816">CATARRHINE</word>
          <word xMin="165.007420" yMin="322.433619" xMax="197.805774" yMax="329.594593">SOCIAL</word>
          <word xMin="201.835580" yMin="321.870784" xMax="267.871636" yMax="329.735078">ADAPTATIONS</word>
          <word xMin="272.470840" yMin="323.589150" xMax="285.563251" yMax="329.306169">113</word>
        </line>
      </block>
      <block xMin="100.989720" yMin="336.984338" xMax="368.281025" yMax="376.437139">
        <line xMin="101.133580" yMin="336.984338" xMax="366.985709" yMax="343.695570">
          <word xMin="101.133580" yMin="337.241308" xMax="140.548918" yMax="343.695570">Overview</word>
          <word xMin="147.024920" yMin="338.327148" xMax="163.138391" yMax="343.604310">113,</word>
          <word xMin="169.754800" yMin="337.819526" xMax="222.259096" yMax="343.551245">Canalization</word>
          <word xMin="228.593540" yMin="338.183288" xMax="236.650275" yMax="343.460450">of</word>
          <word xMin="242.116380" yMin="337.523631" xMax="279.377271" yMax="343.625102">Behavior</word>
          <word xMin="285.418240" yMin="336.984338" xMax="300.522677" yMax="343.579943">and</word>
          <word xMin="306.853380" yMin="337.398773" xMax="343.969260" yMax="343.476499">Learning</word>
          <word xMin="350.730680" yMin="338.002328" xMax="366.985709" yMax="343.325850">114,</word>
        </line>
        <line xMin="100.989720" yMin="347.963816" xMax="368.281025" yMax="354.861327">
          <word xMin="100.989720" yMin="347.963816" xMax="127.316100" yMax="354.861327">Which</word>
          <word xMin="132.495060" yMin="348.751966" xMax="168.602769" yMax="354.664603">Primates</word>
          <word xMin="173.782880" yMin="349.637843" xMax="181.119740" yMax="354.443486">to</word>
          <word xMin="186.442560" yMin="348.111311" xMax="227.439207" yMax="354.824512">Compare?</word>
          <word xMin="232.909340" yMin="349.511128" xMax="249.164369" yMax="354.834650">117,</word>
          <word xMin="254.488340" yMin="348.427130" xMax="287.291009" yMax="354.565915">General</word>
          <word xMin="292.611240" yMin="348.852439" xMax="354.758760" yMax="354.279989">Characteristics</word>
          <word xMin="359.937720" yMin="348.678718" xMax="368.281025" yMax="354.143582">of</word>
        </line>
        <line xMin="100.989720" yMin="358.994918" xMax="367.416138" yMax="365.605403">
          <word xMin="100.989720" yMin="359.967648" xMax="134.942982" yMax="365.527495">Selected</word>
          <word xMin="138.681040" yMin="359.810199" xMax="187.018863" yMax="365.566794">Catarrhines</word>
          <word xMin="191.189940" yMin="360.375733" xMax="207.158400" yMax="365.605403">119,</word>
          <word xMin="211.186480" yMin="360.204726" xMax="267.438617" yMax="365.468319">Socialization,</word>
          <word xMin="271.319960" yMin="359.356870" xMax="318.215443" yMax="365.500178">Maturation</word>
          <word xMin="321.958680" yMin="358.994918" xMax="337.063117" yMax="365.590523">and</word>
          <word xMin="340.516620" yMin="359.429276" xMax="367.416138" yMax="365.302338">Learn-</word>
        </line>
        <line xMin="101.133580" yMin="369.589885" xMax="367.417865" yMax="376.437139">
          <word xMin="101.133580" yMin="370.645879" xMax="113.361105" yMax="375.985232">ing</word>
          <word xMin="117.677480" yMin="370.983368" xMax="133.790951" yMax="376.260530">123,</word>
          <word xMin="137.674020" yMin="370.559900" xMax="167.739609" yMax="376.186460">Infancy</word>
          <word xMin="171.912700" yMin="371.021373" xMax="187.881160" yMax="376.251043">124,</word>
          <word xMin="192.053100" yMin="369.834187" xMax="207.015115" yMax="376.367600">The</word>
          <word xMin="210.611040" yMin="370.996020" xMax="243.839247" yMax="376.437139">Juvenile</word>
          <word xMin="247.583060" yMin="370.652182" xMax="269.304481" yMax="376.343194">Stage</word>
          <word xMin="273.477860" yMin="370.877513" xMax="289.446320" yMax="376.107183">127,</word>
          <word xMin="293.474400" yMin="369.589885" xMax="308.723848" yMax="376.248811">The</word>
          <word xMin="312.463920" yMin="370.130098" xMax="349.006662" yMax="376.113972">Subadult</word>
          <word xMin="352.600860" yMin="369.741000" xMax="367.417865" yMax="376.211092">and</word>
        </line>
      </block>
      <block xMin="101.133580" yMin="380.150403" xMax="308.146969" yMax="387.634769">
        <line xMin="101.133580" yMin="380.150403" xMax="308.146969" yMax="387.634769">
          <word xMin="101.133580" yMin="381.343939" xMax="124.007320" yMax="387.336859">Adult</word>
          <word xMin="128.035400" yMin="381.834368" xMax="153.500346" yMax="387.394215">Stages</word>
          <word xMin="157.814420" yMin="382.173698" xMax="173.496311" yMax="387.309517">129,</word>
          <word xMin="177.810960" yMin="381.184696" xMax="223.705177" yMax="387.196839">Aggression</word>
          <word xMin="227.730380" yMin="382.204448" xMax="243.843851" yMax="387.481610">131,</word>
          <word xMin="248.014640" yMin="380.150403" xMax="288.007432" yMax="387.634769">Summary</word>
          <word xMin="292.323520" yMin="381.848878" xMax="308.146969" yMax="387.031057">134.</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="45.891340" yMin="402.467848" xMax="84.589392" yMax="408.310380">
        <line xMin="45.891340" yMin="402.467848" xMax="84.589392" yMax="408.310380">
          <word xMin="45.891340" yMin="402.467848" xMax="77.110974" yMax="408.310380">Chapter</word>
          <word xMin="81.280900" yMin="403.818817" xMax="84.589392" yMax="408.152942">7</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.277440" yMin="401.177458" xMax="315.484692" yMax="409.152573">
        <line xMin="101.277440" yMin="401.177458" xMax="315.484692" yMax="409.152573">
          <word xMin="101.277440" yMin="401.177458" xMax="190.756921" yMax="408.992000">HUNTER-GATHERER</word>
          <word xMin="194.354860" yMin="401.819002" xMax="227.298225" yMax="409.011637">SOCIAL</word>
          <word xMin="231.183020" yMin="401.254357" xMax="297.503919" yMax="409.152573">ADAPTATIONS</word>
          <word xMin="302.249860" yMin="402.950101" xMax="315.484692" yMax="408.729311">135</word>
        </line>
      </block>
      <block xMin="101.133580" yMin="417.102611" xMax="367.276594" yMax="423.991120">
        <line xMin="101.133580" yMin="417.102611" xMax="367.276594" yMax="423.991120">
          <word xMin="101.133580" yMin="417.102611" xMax="140.403907" yMax="423.533127">Overview</word>
          <word xMin="145.442460" yMin="418.457168" xMax="161.555931" yMax="423.734330">135,</word>
          <word xMin="167.165320" yMin="417.580340" xMax="220.677787" yMax="423.953188">Assumptions</word>
          <word xMin="226.147920" yMin="418.148588" xMax="248.447659" yMax="423.991120">about</word>
          <word xMin="253.912900" yMin="418.080738" xMax="306.562207" yMax="423.828287">Canalization</word>
          <word xMin="312.176200" yMin="418.639033" xMax="328.144660" yMax="423.868703">138,</word>
          <word xMin="333.035900" yMin="417.408428" xMax="367.276594" yMax="423.816330">Contem-</word>
        </line>
      </block>
      <block xMin="101.277440" yMin="428.287543" xMax="367.131871" yMax="456.668783">
        <line xMin="101.421300" yMin="428.287543" xMax="367.131871" yMax="434.667690">
          <word xMin="101.421300" yMin="428.287543" xMax="128.610840" yMax="434.223926">porary</word>
          <word xMin="132.063480" yMin="428.403352" xMax="204.997047" yMax="434.374788">Hunter-Gatherers</word>
          <word xMin="208.740860" yMin="429.390528" xMax="224.854331" yMax="434.667690">139,</word>
          <word xMin="228.449680" yMin="428.549927" xMax="269.451506" yMax="434.517970">Canalized</word>
          <word xMin="273.190140" yMin="428.443151" xMax="310.451031" yMax="434.544622">Behavior</word>
          <word xMin="313.183220" yMin="428.858490" xMax="347.274587" yMax="434.440952">Patterns</word>
          <word xMin="351.018400" yMin="429.246668" xMax="367.131871" yMax="434.523830">141,</word>
        </line>
        <line xMin="101.277440" yMin="439.260686" xMax="366.985709" yMax="445.826910">
          <word xMin="101.277440" yMin="439.508639" xMax="149.615263" yMax="445.265234">Cooperative</word>
          <word xMin="153.786340" yMin="439.260686" xMax="187.163011" yMax="445.506892">Hunting</word>
          <word xMin="192.053100" yMin="440.361893" xMax="208.021560" yMax="445.591563">141,</word>
          <word xMin="212.337360" yMin="439.690488" xMax="230.318709" yMax="445.579380">Tool</word>
          <word xMin="234.779520" yMin="439.495034" xMax="286.279098" yMax="445.628166">Manufacture</word>
          <word xMin="290.597200" yMin="439.325822" xMax="305.126772" yMax="445.670402">and</word>
          <word xMin="309.299000" yMin="439.652483" xMax="327.425360" yMax="445.588866">Tool</word>
          <word xMin="331.741160" yMin="439.419008" xMax="346.415743" yMax="445.826910">Use</word>
          <word xMin="351.162260" yMin="440.256038" xMax="366.985709" yMax="445.438217">144,</word>
        </line>
        <line xMin="101.421300" yMin="449.701823" xMax="366.984558" yMax="456.668783">
          <word xMin="101.421300" yMin="449.701823" xMax="121.273980" yMax="456.203576">Food</word>
          <word xMin="126.740660" yMin="450.394945" xMax="157.815283" yMax="456.210339">Sharing</word>
          <word xMin="164.000400" yMin="451.226498" xMax="179.682291" yMax="456.362317">144,</word>
          <word xMin="185.291680" yMin="450.233845" xMax="243.699991" yMax="456.610086">Husband-Wife</word>
          <word xMin="249.165520" yMin="451.246389" xMax="301.668953" yMax="456.537120">Reciprocities</word>
          <word xMin="307.572680" yMin="451.439113" xMax="323.541140" yMax="456.668783">146,</word>
          <word xMin="329.295540" yMin="450.397628" xMax="366.984558" yMax="456.569205">Symbolic</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.133580" yMin="460.570937" xMax="166.451487" yMax="467.152972">
        <line xMin="101.133580" yMin="460.570937" xMax="166.451487" yMax="467.152972">
          <word xMin="101.133580" yMin="460.570937" xMax="166.451487" yMax="467.152972">Communication</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="173.782880" yMin="462.084753" xMax="189.751340" yMax="467.314423">
        <line xMin="173.782880" yMin="462.084753" xMax="189.751340" yMax="467.314423">
          <word xMin="173.782880" yMin="462.084753" xMax="189.751340" yMax="467.314423">147,</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="196.656620" yMin="461.141868" xMax="343.251399" yMax="467.549770">
        <line xMin="196.656620" yMin="461.141868" xMax="343.251399" yMax="467.549770">
          <word xMin="196.656620" yMin="461.481198" xMax="214.927991" yMax="467.465072">Rule</word>
          <word xMin="221.544400" yMin="461.544540" xMax="248.588929" yMax="467.449262">Giving</word>
          <word xMin="255.351500" yMin="461.141868" xMax="270.026083" yMax="467.549770">and</word>
          <word xMin="276.930500" yMin="461.481198" xMax="295.201871" yMax="467.465072">Rule</word>
          <word xMin="301.962140" yMin="461.460386" xMax="343.251399" yMax="467.470267">Following</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="350.874540" yMin="462.122758" xMax="366.697989" yMax="467.304937">
        <line xMin="350.874540" yMin="462.122758" xMax="366.697989" yMax="467.304937">
          <word xMin="350.874540" yMin="462.122758" xMax="366.697989" yMax="467.304937">149,</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.421300" yMin="470.702590" xMax="161.123200" yMax="478.106673">
        <line xMin="101.421300" yMin="470.702590" xMax="161.123200" yMax="478.106673">
          <word xMin="101.421300" yMin="470.702590" xMax="140.985102" yMax="478.106673">Summary</word>
          <word xMin="145.154740" yMin="472.586533" xMax="161.123200" yMax="477.816203">151.</word>
        </line>
      </block>
      <block xMin="45.747480" yMin="494.089349" xMax="78.259552" yMax="501.538223">
        <line xMin="45.747480" yMin="494.089349" xMax="78.259552" yMax="501.538223">
          <word xMin="45.747480" yMin="494.089349" xMax="68.044629" yMax="501.391665">Part</word>
          <word xMin="73.224740" yMin="494.942618" xMax="78.259552" yMax="501.538223">3</word>
        </line>
      </block>
      <block xMin="46.322920" yMin="516.858265" xMax="85.021260" yMax="522.960011">
        <line xMin="46.322920" yMin="516.858265" xMax="85.021260" yMax="522.960011">
          <word xMin="46.322920" yMin="516.858265" xMax="77.397543" yMax="522.673659">Chapter</word>
          <word xMin="81.137040" yMin="517.871683" xMax="85.021260" yMax="522.960011">8</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="91.351100" yMin="491.047128" xMax="160.258889" yMax="502.330779">
        <line xMin="91.351100" yMin="491.047128" xMax="160.258889" yMax="502.330779">
          <word xMin="91.351100" yMin="491.047128" xMax="160.258889" yMax="502.330779">ONTOGENY</word>
        </line>
      </block>
      <block xMin="101.709020" yMin="515.005930" xMax="257.940692" yMax="523.695850">
        <line xMin="101.709020" yMin="515.005930" xMax="257.940692" yMax="523.695850">
          <word xMin="101.709020" yMin="515.005930" xMax="134.798259" yMax="523.675310">MOTOR</word>
          <word xMin="138.537180" yMin="516.833770" xMax="168.606222" yMax="523.398844">SKILLS</word>
          <word xMin="172.775860" yMin="515.643854" xMax="240.388046" yMax="523.695850">DEVELOPMENT</word>
          <word xMin="244.705860" yMin="517.750381" xMax="257.940692" yMax="523.529591">153</word>
        </line>
      </block>
      <block xMin="101.133580" yMin="530.879504" xMax="367.129569" yMax="549.416273">
        <line xMin="101.565160" yMin="530.879504" xMax="367.128994" yMax="538.553356">
          <word xMin="101.565160" yMin="530.879504" xMax="112.786240" yMax="538.229311">An</word>
          <word xMin="117.677480" yMin="531.577166" xMax="157.237829" yMax="538.055173">Overview</word>
          <word xMin="162.130220" yMin="532.682008" xMax="170.186955" yMax="537.959170">of</word>
          <word xMin="173.782880" yMin="532.675658" xMax="198.812794" yMax="538.140522">Social</word>
          <word xMin="203.849620" yMin="531.899256" xMax="254.344480" yMax="538.514082">Competency</word>
          <word xMin="259.379580" yMin="533.182343" xMax="275.779620" yMax="538.553356">153,</word>
          <word xMin="280.670860" yMin="531.770778" xMax="305.844921" yMax="538.366383">Motor</word>
          <word xMin="310.306020" yMin="533.091918" xMax="335.767514" yMax="537.856855">Skills:</word>
          <word xMin="340.804340" yMin="532.161718" xMax="367.128994" yMax="537.909267">Evolu-</word>
        </line>
        <line xMin="101.133580" yMin="542.498778" xMax="367.129569" yMax="549.416273">
          <word xMin="101.133580" yMin="543.299581" xMax="130.480157" yMax="548.791583">tionary</word>
          <word xMin="133.358220" yMin="543.217221" xMax="195.072434" yMax="548.991908">Considerations</word>
          <word xMin="198.095220" yMin="544.084953" xMax="214.063680" yMax="549.314623">157,</word>
          <word xMin="216.797020" yMin="542.957489" xMax="271.031089" yMax="549.416273">Development</word>
          <word xMin="273.621720" yMin="543.676868" xMax="282.110035" yMax="549.236715">of</word>
          <word xMin="283.548060" yMin="543.250686" xMax="319.655769" yMax="549.163323">Reaction</word>
          <word xMin="321.958680" yMin="542.981952" xMax="345.121579" yMax="549.050632">Time,</word>
          <word xMin="347.853480" yMin="542.498778" xMax="367.129569" yMax="548.811697">Tim-</word>
        </line>
      </block>
      <block xMin="101.852880" yMin="553.029451" xMax="367.272854" yMax="560.205773">
        <line xMin="101.852880" yMin="553.029451" xMax="367.272854" yMax="560.205773">
          <word xMin="101.852880" yMin="554.494483" xMax="116.526600" yMax="559.300126">ing,</word>
          <word xMin="120.410820" yMin="553.255687" xMax="135.372835" yMax="559.789100">and</word>
          <word xMin="139.400340" yMin="553.029451" xMax="182.123307" yMax="560.025337">Movement</word>
          <word xMin="185.723260" yMin="553.304519" xMax="206.584111" yMax="560.136447">Time</word>
          <word xMin="210.611040" yMin="554.874453" xMax="226.579500" yMax="560.104123">161,</word>
          <word xMin="230.607580" yMin="553.746989" xMax="284.841649" yMax="560.205773">Development</word>
          <word xMin="288.439300" yMin="554.398518" xMax="296.782605" yMax="559.863382">of</word>
          <word xMin="299.660380" yMin="554.197650" xMax="333.468631" yMax="559.733752">Visually</word>
          <word xMin="337.063980" yMin="553.205918" xMax="367.272854" yMax="559.801523">Guided</word>
        </line>
      </block>
    </flow>
    <flow>
      <block xMin="101.996740" yMin="563.556584" xMax="366.986860" yMax="570.700778">
        <line xMin="101.996740" yMin="563.556584" xMax="366.986860" yMax="570.700778">
          <word xMin="101.996740" yMin="563.884166" xMax="139.830769" yMax="570.079488">Reaching</word>
          <word xMin="144.867020" yMin="563.857908" xMax="159.541603" yMax="570.265810">and</word>
          <word xMin="164.144260" yMin="564.152883" xMax="202.123300" yMax="570.371951">Grasping</word>
          <word xMin="207.302260" yMin="565.376233" xMax="223.270720" yMax="570.605903">167,</word>
          <word xMin="228.161960" yMin="564.275915" xMax="282.111186" yMax="570.700778">Development</word>
          <word xMin="286.712980" yMin="564.975403" xMax="294.913000" yMax="570.346416">of</word>
          <word xMin="298.365640" yMin="564.856879" xMax="310.593165" yMax="570.196232">the</word>
          <word xMin="315.197260" yMin="563.556584" xMax="366.986860" yMax="570.341021">Complemen-</word>
        </line>
      </block>
    </flow>
  </page>
</doc>
</body>
</html>

Desired Outcome:

| BookmarkTitle                                    | BookmarkLevel |       PageNumber |
|--------------------------------------------------+---------------+------------------|
| Forward                                          |             1 |               xi |
| Preface and Acknowledgements                     |             1 |               xv |
| Illustration Credits                             |             1 |             xvii |
| Part 1 Overview and Theory                       |             1 |                1 |
| Chapter 1 OVERVIEW AND SUMMARY                   |             2 |                1 |
| Evolution, Learning and Adaptation               |             3 |                1 |
| Evolutionary Theory                              |             3 |                3 |
| Evolution, Development and Behavioral Adaptation |             3 |                6 |
| ...                                              |           ... |              ... |
| Chapter 2 THEORY OF EVOLUTION                    |             2 |               21 |
| History                                          |             3 |               21 |
| Precursors of Darwin and Spencer                 |             3 |               21 |

```

Wow, that's really frightening. I see a ton of code, and not a single def fn(): definition to help structure it, to help tame the forest of globals? Not sure I want to even begin reading that in earnest. (Oh, wait, I see, there's a lovely three lines of is_roman_numeral(), that much looks good. Though better to simply assign set(numeral.upper()).) — J_H, Commented Dec 31, 2022 at 5:41
@J_H, It does what I want for this specific case. Not quite sure how to organize this into more readable functions because the globals are everywhere, and I end up having large chunks of code within each function, which increases the complexity and defeats the point. It would be nice if suggestions can be made as to how to improve on readability and generalize the functions as well. Good point there. — Sati, Commented Dec 31, 2022 at 7:23
@J_H I have updated the code with functions. Hopefully it is more reader friendly now. — Sati, Commented Dec 31, 2022 at 13:06
I have rolled back Rev 6 → 5. Please see What to do when someone answers. — Sᴀᴍ Onᴇᴌᴀ, Commented Jan 1, 2023 at 6:41

J_H · Accepted Answer · 2022-12-31 18:33:00Z

updated my code with functions

Thank you, that's a lovely refactoring. Now I am brave enough to wade into it.

Such structuring helps the Gentle Reader in a few ways:

The name documents the one thing this chunk of code tries to do.
Local variables go out of scope upon exit, so the Reader doesn't have to hold dozens of facts in his head.

In build_dataframe I'm reading

    df = df.sort_values(by=["page", "yMax", "xMax"])

I wonder if it might be helpful to promote those three to Index? Certainly in an RDBMS table I would define that as the compound PK. I will stay tuned and see how it shakes out.

I'm reading consolidate_bboxes, and I have a thing about inplace. In the design of pandas API, it (A.) isn't necessary and (B.) tends to create problems. Nevermind some poor caller that got his supposedly readonly dataframe mutated. Worse, it can lead to chained assignment bugs. Pandas might possibly phase it out in future. So as a knee-jerk response, without even thinking about it, I offer this Tiny Nit. Rephrase as:

    df = df.sort_values( ... )
    df = df.reset_index( ... )

As a bigger item, I confess I cannot explain exactly what this does:

    df = df.assign(new_yMax=np.where(df['yMax'].diff(-1) == -1, df.yMax + 1, df.yMax))

It seems to be consolidating roundoff issues, so two lines that are almost the same come out the same.

But I don't see how that relates to the high level business goal of producing a nicely formatted ToC, and it's not clear that a window of 1 suffices. Perhaps a "small" window should be an input parameter that defaults to 1? Perhaps we'd like another technique for "smoosh them together" or "go with the majority value", such as .median(), so [11, 12, 13, 14] doesn't become new sequential numbers? The identifier new_Ymax is nice enough, but it doesn't help me understand the higher level goal.

And then the .groupby() I'm not quite getting, either. We're looking at all words (xMax) on a line, and finding the extent of the line?

So, pardon my density, but I'm looking for a little help, which could take any of these forms:

A unit test that demonstrates in / out values for this function.
Variable name(s) that spell out the intent more clearly.
This is already pretty small, but maybe an even tinier helper function that has an informative name.
# comments
"""docstring""" on this or on a tiny helper.

tl;dr: If you ask me what this does, I couldn't properly explain it.

I feel like I should, but I'm not yet seeing it. Perhaps it will become clearer as I read more code. What I'm going for ATM is Local Analysis -- I should be able to say "this is correct", or that it isn't, and so far I cannot say that conclusively. If you assigned me a maintenance task, I would have some trouble maintaining this particular function.

BTW, the function's name is terrific, and offered me a valuable clue. I am just unclear on the details of consolidating those bounding boxes. In part that's because the row count didn't change, so I suppose we're trying to winnow many distinct values down to fewer values.

def max_and_min(df):
    """Helper script to explore extremities to filter off."""

tiny nit: recommend you call it "helper function" or just "helper". I think of a script as a module, a file, which runs top to bottom, such as the part protected by "if name is main". And "explore" tends to describe a person's behavior, so maybe prefer "identify" here.

Oh, maybe this is about interactively examining and exploring, as I see there's still a debug print here, nevermind. If the code is stable enough that it's ready for production, maybe remove the debug?

That parameter 3 is a magic number. Please give it a name. It tends to be convenient to make it a defaulted kwarg in the function signature. That way callers, such as unit tests, are free to override the default if they wish.

I'm not very fond of the identifier min_max. It is clearly accurate, but it's not as helpful as suggested and candidate. However, I confess that I'm still not sure if we're returning per-page vertical extents, or something more fine grained.

I do like apply_filter, short and sweet.

def is_roman_numeral(num: str) -> bool:

Brilliant! Self explanatory, it leaves no questions. The conventional is_ prefix goes with the boolean return.

In build_toc, consider eliding this redundant comment:

    # Iterate through word list

We see the for word in words, we've got it.

The code gives the specifics. Reserve # comments for the general, for the poetic remarks, for higher level items not apparent in the code.

Again, <= 332 is a magic number that probably should become a kwarg defaulted parameter. But thank you for the comment explaining its meaning.

def build_toc(df, max_pages=332):

I imagine we're range limiting it so things like calendar year can't be confused with page numbers.

tiniest of nits:

                    caption = None
                else:
                    caption = None

Rather than an else clause, we might just unconditionally assign that after the if.

Similarly, you might use not equal, if label[-1] != "-":, to collapse a clause.

The if ... label += " " + word idiom commonly comes up, in many languages. The conditional can be slightly distracting. One approach is to init to " " and finally use .lstrip(). Another is to init to [] and finally use " ".join( ... ). No biggie.

    pg_no += [np.nan] * 3

Again, consider promoting magic number to a kwarg defaulted parameter. Or make this dependent on len(labels).

    cp = lb + 1  # caption

On the one hand, you did a service to the Reader, thank you. OTOH, listen to what the code is telling you. The cp identifier was too obscure, so it needed a comment. Well, that suggests renaming it to caption, right?

In any event, kudos, it is much clearer than np.concatenate((lb, lb + 1)).

(And BTW lb was perfectly clear from context, it is fine as a local variable, since they have lower documentation burden.)

Naming the arr temp var is maybe useful. But consider returning the sort results directly, and telling the Reader the return type via an optional type annotation in the signature.

Or maybe the function name told us that already.

If I could quibble about the verb for a moment, I would prefer get_bookmarks() (with annotation) or perhaps the more verbose get_bookmark_arrays(). The "build" suggests to me that we're evaluating for side effects and the build product is stored someplace to be accessed later. Kind of like a "print_foo()" prefix, which suggests a single side effect on sys.stdout. But that's just me, reasonable people will differ, YMMV. I'm just revealing my thought process as I read it.

def shift_cells_down(df_out, caption):
    """Shift page-label cells down at cardinal captions where no page number is associated."""

Oooohh! A vocabulary word.

I'm sure it has a precise meaning for editors and type setters. I confess I don't know what a cardinal caption is, and google results were entertaining but uninformative. But that's cool, I don't need to understand everything, and I appreciate that you're trying to help me out, "A" for effort. Perhaps it denotes "large font", or "first on page".

This is the sort of thing that a unit test can conveniently illustrate.

I am reading set_bookmark_levels.

The way you're using .loc(), it appears you're extending the dataframe by a row on each iteration.

Now, you only iterate a handful of times so it doesn't really matter. But it's not a terrific habit to be in.

Pandas, and the underlying numpy, kind of wants to know how many rows to allocate at init time. And then we refrain from adding rows. So a common pattern is to build up a list of dicts (or a dict of lists) which we hand in to pd.DataFrame().

Why? Because numpy allocations, unlike list allocations, are expensive. They may involve copying many data values from small buffer to slightly larger buffer. And this can lead to quadratic performance, something you really don't want.

tl;dr: Consider creating that Series with a list comprehension.

def set_labels(df_out, loc_arr):

Sorry, I don't like the identifier df_out, I don't think it's accurate. Seems pretty clear it is an input which is never altered.

Calling it df or maybe df_in would be fine. Or invent a new name, some business term that more precisely describes its contents.

        if this_index and last_index:
            if this_index - last_index == 1:

tiny nit: The guard, as written, is perfectly lovely. Consider changing 2nd if to and, for same effect.

In update_dataframe, usual nit about maybe not using inplace.

    df_out['BookmarkPageNumber'] = None

Consider moving that initialization down to next function, so we see init + assignments all together.

In offset_page_numbers, the "skip None" clause is perfectly clear as written. Maybe accomplish that with .dropna()? Whatever.

The is_roman_numeral / roman / int calculation is perfectly clear. Consider combining it into a temp var or helper function. No biggie.

At toplevel, the # [44, 45, 127, 551] comment is (I'm sure!) helpful to you, but also cryptic, possibly describing troublesome page numbers. Consider writing it as a longer comment on a separate line. Consider immortalizing those page numbers in their very own unit test, which demonstrates the trouble.

Consider the trivial refactor of putting all this within def main(): ... return toc_df. Why? So the temp vars go out-of-scope upon function exit. That way we're not polluting the global namespace.

Overall?

I have to say, I am pretty impressed with two aspects of this code.

well organized pipeline of distinct processing steps
well chosen identifiers

I suspect it was well organized even before functions were imposed on it. And the identifiers are helpful to the Reader, short where local scope makes it appropriate, longer when they're global and so must be self-explanatory.

Could others usefully maintain this code? Yes, I think so.

Thanks for the nice review. df_out distinguishes on a global scale a different dataframe from the raw one derived directly out of toc.html. This is reflected by toc_df and toc_df_raw in the global scope. I know it doesn't make sense at the functional level. — Sati, Commented Jan 1, 2023 at 0:12
df = df.assign(new_yMax=np.where(df['yMax'].diff(-1) == -1, df.yMax + 1, df.yMax)) and the following transform simply collapses yMax values close to one another down to a single value. It's been lifted almost unchanged from this answer to my question. The task, as you suspected, was to consolidate yMax values into reliable line indicators. Curiously, the original toc_raw does provide the line of each word within a nested context of page -> flow -> block -> line but it is totally unreliable. — Sati, Commented Jan 1, 2023 at 0:35
Not quite sure how to implement the unit tests to serve your purposes, though. — Sati, Commented Jan 1, 2023 at 0:40
I'm not sure if I've got you right for the case of set_bookmark_levels. Please refer to the updated further discussion above. — Sati, Commented Jan 1, 2023 at 3:19
I think I didn't understand what df_out looked like on the way in and the way out. Sorry. I was concerned that each iteration appended a row, but now it appears that it overwrote some default value, which is much cheaper. — J_H, Commented Jan 1, 2023 at 5:43

2 revs · Accepted Answer · 2023-01-01 06:59:18Z

Further Discussion

This is a response to @J_H's answer above.

I have refactored set_bookmark_levels as such:

def set_bookmark_levels(df_out):
    """Set bookmark levels with a dictionary."""

    d = BOOKMARK_LEVELS
    levels = []

    for key in d.keys():
        if d[key][0] == "default":
            levels = [key] * len(df_out)

    for key in d.keys():
        if d[key][0] != "default":
            for idx in d[key]:
                levels[idx] = key

    df_out['BookmarkLevel'] = levels

    return df_out

Compare with the original:

def set_bookmark_levels(df_out):
    """Set bookmark levels with a dictionary."""

    d = BOOKMARK_LEVELS

    for key in d.keys():

        if d[key][0] == "default":
            df_out['BookmarkLevel'] = key  
# This doesn't work as expected. Because the default in this case is 3, it ends up overwriting the other levels. A separate loop should do the trick. 
        else:
            df_out.loc[d[key], 'BookmarkLevel'] = key

    return df_out

It doesn't seem apparent to me that the new update is definitely better than the old version. Given the interception of a list, it is definitely not easier to read and there are obviously more steps involved in the process.

Pandas, and the underlying numpy, kind of wants to know how many rows to allocate at init time. And then we refrain from adding rows. So a common pattern is to build up a list of dicts (or a dict of lists) which we hand in to pd.DataFrame().

Did I misunderstand your statement or is updating dataframes through the intervention of lists really the way to go?

One more clarification: I am just adding a new column and updating row values with the array of indices given in d[key].

I suppose this answer doesn't really answer the original question, but is a "comment" on an existing answer. The StackExchange site is not intended for discussion, but rather for clear questions that have concise answers. As such, if you have a follow-up question, it's better to ask it in a separate question. — Juho, Commented Jan 1, 2023 at 10:08
It's actually a follow-up question that happens specifically within this context of exchange, not meant to standalone as an independent question on its own right. It doesn't mean anything when read out of context. I cannot even be sure that @J_H sees it if I post this as a new question. I think, even though this is not a full answer to my own question but a reply to an answer, it still fits the description: "Posting a self-answer. If you want to show everyone how you improved your code, but don't want to ask another question, then post an answer to your own question. " — Sati, Commented Jan 1, 2023 at 11:37
I did try to improve my code, even though I merely shared a snippet of it over here. There is no way I could clarify my doubts if this too is not allowed. — Sati, Commented Jan 1, 2023 at 11:41

Stack Exchange Network

Unstructured to Structured TOC

Updated:

toc.html:

Desired Outcome:

2 Answers 2

Further Discussion

Your Answer

Not the answer you're looking for? Browse other questions tagged
python
pandas
pdf
dataframe
or ask your own question.

Hot Network Questions

Unstructured to Structured TOC

Updated:

toc.html:

Desired Outcome:

2 Answers 2

Further Discussion

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged pythonpandaspdfdataframe or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
python
pandas
pdf
dataframe
or ask your own question.