1
\$\begingroup\$

Given a txt input which represents extracted resume (or “CV” outside of the US) data, pass the text to an AI model and return a JSON version of the original input.

I have created the below script, which tries to prompt gpt3-turbo in order to process and return the JSON.

As well as the usual review, I am also looking for any guidance on the below list:

  1. Is a GPT even needed to achieve this?
  2. If so is the gpt3-turbo model ideal for this sort of work?
  3. Can I prompt GPT any better to achieve high confidence scoring?
  4. How could I improve the standardisation of the response so that in the case we get an unexpected field, it doesn't explode a client
  5. Is there a way to make the processing go faster? I know the bottleneck is the call to OpenAI - can I do anything with my inputs to make it faster?

It's worth noting that the input txt can change in an arbitrary way. It is most likely never the exact same format.

Code

import json

from openai import OpenAI, Stream
from openai.types.chat import ChatCompletion, ChatCompletionChunk


OPENAI_API_KEY = "sk-*"
# Low Temperature (e.g., 0.0 to 0.3)
#   A low temperature makes the model's responses more deterministic and conservative.
TEMPERATURE = 0.1
MAX_TOKENS = 2500
RAW_DATA = """
Forename SURNAME
e-mail: professional email address tel: UK landline or mobile
Education and Qualifications

2000-2003
University/Universities
Location; City and Country
Degree and Subject
applicable additional info




Work Experience

Sep-07 – Aug-10
Official Company Name
Job title
  Please use 3-4 bullets maximum to describe your job function &
City, Country
responsibilities
  Concentrate on your achievements, and what you have distinctly
contributed to in each role, using quantitative examples where possible
  Examples that may assist you –

“Advised client’s Digital Media division on £3M international expansion,
coordinating a team of 8 analysts during initial research phase”
“Structured and negotiated equipment deal financing including credit
purchases, rentals, and 31 lease contracts worth $745k”


Jun-05 – Sep-07
Official Company Name
Job title
  Make sure your work experience comes to life, consider what someone
City, Country
reading your CV would be most interested in
  Avoid any negativity or short comings on your CV that may raise the
wrong questions
  Try to avoid having your CV read like a job description


Mar-04 – Jun-05
Official Company Name
Job title
  Try to ensure your CV is easy to scan, start bullet points with relevant
City, Country
action verbs
  You can also include significant relevant voluntary experience in your
work experience if it is applicable
  Try to avoid industry jargon that may not be understood

Aug-03 – Mar-04
Official Company Name
Job title
  Use past tense for roles you have completed
  Please set dates using the abbreviated month and two digits for the year,
City, Country
you must include months as well as years
  Make sure your CV is an accurate reflection of you and what you want to
highlight about your experience
  Stick to facts you can easily discuss. Avoid subjective comments

Additional Information

Interests:
Concentrate on activities you participate in and are willing to talk about. You
should highlight achievements in those activities. Eg. rather than just listing
‘running’ say ‘running – participated in several marathons, President of the
Oxford Runners Club’

Achievements:
List academic or other achievements here, for example
First Class Honours, Previous University
Study abroad scholarship (selected 3 out of 600 students)
Principal Cellist of London Youth Orchestra

Nationality:
your nationality, dual nationality, and any additional work authorization if
applicable
Languages:
languages other than English and ability level eg. German (fluent)
"""


def parse_cv(client: OpenAI) -> ChatCompletion | Stream[ChatCompletionChunk]:
    conversation = [
        {
            "role": "system",
            "content": (
                "You are a sophisticated resume parser. "
                "Your task is to analyze, extract, and structure data from resumes. "
                "You should identify and separate key sections such as "
                "personal details, professional experience, education, skills, and additional information "
                "like certifications or personal projects. "
                "For each section, provide structured information in a clear, concise, and organized manner."
            )
        },
        {
            "role": "user",
            "content": (
                "Please parse the following resume and return all data structured as JSON. "
                "For each section, include a confidence score indicating your certainty that the information is "
                "correctly extracted and true. "
                "Focus on accurately identifying and categorizing the data into predefined fields such as name, "
                "contact information, work history, educational background, skills, and any other relevant sections. "
                "Handle ambiguities or unclear information gracefully, "
                "and indicate areas of uncertainty in your confidence scores."
            )
        },
        {
            "role": "user",
            "content": RAW_DATA
        }
    ]
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=conversation,
        max_tokens=MAX_TOKENS,
        temperature=TEMPERATURE,
        stop=None
    )
    return response


def gpt_verify() -> ChatCompletion | Stream[ChatCompletionChunk]:
    client = OpenAI(api_key=OPENAI_API_KEY)
    cv_data = parse_cv(client)
    return cv_data


if __name__ == "__main__":
    result = gpt_verify()
    str_result = result.choices[0].message.content
    json_result = json.loads(str_result)
    print(json_result)

Current Output

{
  "name": {
    "forename": "Forename",
    "surname": "SURNAME",
    "confidence": 0.9
  },
  "contact_information": {
    "email": "professional email address",
    "phone": "UK landline or mobile",
    "confidence": 0.8
  },
  "education": {
    "items": [
      {
        "start_date": "2000",
        "end_date": "2003",
        "institution": "University/Universities",
        "location": "City and Country",
        "degree": "Degree and Subject",
        "additional_info": "applicable additional info",
        "confidence": 0.9
      }
    ]
  },
  "work_experience": {
    "items": [
      {
        "start_date": "Sep-07",
        "end_date": "Aug-10",
        "company": "Official Company Name",
        "job_title": "Job title",
        "responsibilities": [
          "Please use 3-4 bullets maximum to describe your job function & responsibilities",
          "Concentrate on your achievements, and what you have distinctly contributed to in each role, using quantitative examples where possible",
          "Examples that may assist you –",
          "“Advised client’s Digital Media division on £3M international expansion, coordinating a team of 8 analysts during initial research phase”",
          "“Structured and negotiated equipment deal financing including credit purchases, rentals, and 31 lease contracts worth $745k”"
        ],
        "location": "City, Country",
        "confidence": 0.8
      },
      {
        "start_date": "Jun-05",
        "end_date": "Sep-07",
        "company": "Official Company Name",
        "job_title": "Job title",
        "responsibilities": [
          "Make sure your work experience comes to life, consider what someone reading your CV would be most interested in",
          "Avoid any negativity or shortcomings on your CV that may raise the wrong questions",
          "Try to avoid having your CV read like a job description"
        ],
        "location": "City, Country",
        "confidence": 0.8
      },
      {
        "start_date": "Mar-04",
        "end_date": "Jun-05",
        "company": "Official Company Name",
        "job_title": "Job title",
        "responsibilities": [
          "Try to ensure your CV is easy to scan, start bullet points with relevant action verbs",
          "You can also include significant relevant voluntary experience in your work experience if it is applicable",
          "Try to avoid industry jargon that may not be understood"
        ],
        "location": "City, Country",
        "confidence": 0.8
      },
      {
        "start_date": "Aug-03",
        "end_date": "Mar-04",
        "company": "Official Company Name",
        "job_title": "Job title",
        "responsibilities": [
          "Use past tense for roles you have completed",
          "Please set dates using the abbreviated month and two digits for the year, you must include months as well as years",
          "Make sure your CV is an accurate reflection of you and what you want to highlight about your experience",
          "Stick to facts you can easily discuss. Avoid subjective comments"
        ],
        "location": "City, Country",
        "confidence": 0.8
      }
    ]
  },
  "additional_information": {
    "interests": "Concentrate on activities you participate in and are willing to talk about. You should highlight achievements in those activities. Eg. rather than just listing ‘running’ say ‘running – participated in several marathons, President of the Oxford Runners Club’",
    "achievements": "List academic or other achievements here, for example\nFirst Class Honours, Previous University\nStudy abroad scholarship (selected 3 out of 600 students)\nPrincipal Cellist of London Youth Orchestra",
    "nationality": "your nationality, dual nationality, and any additional work authorization if applicable",
    "languages": "languages other than English and ability level eg. German (fluent)",
    "confidence": 0.7
  }
}
\$\endgroup\$
3
  • \$\begingroup\$ Can I see an example? \$\endgroup\$
    – Bob
    Commented Jan 8 at 21:16
  • \$\begingroup\$ Please note we have a fairly strict policy on content generated by generative artificial intelligence tools. \$\endgroup\$
    – Mast
    Commented Feb 25 at 17:00
  • \$\begingroup\$ Noted, but those policies refer to content created by generative ai not on question and answers about the topic right? \$\endgroup\$
    – Bob
    Commented Feb 27 at 4:12

1 Answer 1

2
\$\begingroup\$

Current code

  • The parse_cv function should already take the CV text as an input argument
  • There should be no variables declared in the global scope to avoid redeclarations and conflicts: everything under if __name__ == __main__: should be moved to a def main() -> None: function

Data validation

With your current implementation, the LLM model might be inconsistent, and asking it to generate a data format based on an informal description will make its response impossible to validate and exploit systematically.

Sending a precise data structure to ChatGPT might help it understand what you expect. You could use simple dataclasses, but a more powerful choice would be to use pydantic models, that allow you to define data structures, and validate the data before instantiating the classes.

A format could look like this (defined in a file named candidate.py):

import datetime
from typing import Sequence

from pydantic import BaseModel, EmailStr, Field


class Name(BaseModel):
    forename: str
    surname: str
    confidence: float


class ContactInformation(BaseModel):
    email: EmailStr
    phone: str
    confidence: float


class Location(BaseModel):
    city: str
    country: str
    confidence: float


class Period(BaseModel):
    start_date: datetime.date = Field(description="The start date of the period, in a YYYY-MM-DD format")
    end_date: datetime.date = Field(description="The end date of the period, in a YYYY-MM-DD format")


class EducationItem(BaseModel):
    period: Period
    institution: str
    location: Location
    degree: str
    additional_info: str
    confidence: float


class WorkExperienceItem(BaseModel):
    period: Period
    company: str
    job_title: str
    responsibilities: Sequence[str]
    location: Location
    confidence: float


class AdditionalInformation(BaseModel):
    interests: Sequence[str]
    achievements: Sequence[str]
    nationality: str
    languages: Sequence[str]
    confidence: float


class Candidate(BaseModel):
    name: Name
    contact_information: ContactInformation
    education: Sequence[EducationItem]
    work_experience: Sequence[WorkExperienceItem]
    additional_information: AdditionalInformation

Some fields can be defined as optional to make the validation more flexible to missing information in the resume, and adding Field descriptions and docstrings might help ChatGPT for the data extraction.

The Candidate model can be instantiated with the following JSON (slightly modified from your version):

{
    "name": {"forename": "Forename", "surname": "SURNAME", "confidence": 0.9},
    "contact_information": {
        "email": "[email protected]",
        "phone": "UK landline or mobile",
        "confidence": 0.8
    },
    "education": [
        {
            "period": {"start_date": "2000-09-01", "end_date": "2003-06-01"},
            "institution": "University/Universities",
            "location": {
                "city": "City",
                "country": "Country"
            },
            "degree": "Degree and Subject",
            "additional_info": "applicable additional info",
            "confidence": 0.9
        }
    ],
    "work_experience": [
        {
            "period": {"start_date": "2007-09-01", "end_date": "2010-08-01"},
            "location": {
                "city": "City",
                "country": "Country"
            },
            "company": "Official Company Name",
            "job_title": "Job title",
            "responsibilities": [
                "Please use 3-4 bullets maximum to describe your job function & responsibilities"
            ],
            "confidence": 0.8
        }
    ],
    "additional_information": {
        "interests": ["Table tennis", "Scrabble"],
        "achievements": ["Multiple patents", "Academic award"],
        "nationality": "your nationality, dual nationality, and any additional work authorization if applicable",
        "languages": ["Mandarin", "Javanese", "Cajun"],
        "confidence": 0.7
    }
}

With the file candidate.py in the same folder:

import json
from pathlib import Path
from .candidate import Candidate


def parse_cv(cv_text: str) -> str:
    candidate_def = Path("candidate.py").read_text(encoding="utf-8")
    # candidate_def can be injected in the prompt
    ...
    return json_result


def main() -> None:
    cv_text = ...
    json_result: str = parse_cv(cv_text)
    candidate = Candidate(**json.loads(json_result))
    print(candidate.work_experience[0].period.start_date.year)
    # 2007


if __name__ == "__main__":
    main()

If the JSON is not valid according to the Candidate model, a ValidationError is raised:

from .candidate import Candidate


invalid_json = {
    "name": {"forename": "", "surname": "", "confidence": 0.9},
    "contact_information": {
        "email": "a@b",
        "phone": "",
        "confidence": 0.8,
    },
    "education": [],
    "work_experience": [],
    "additional_information": {
        "interests": [],
        "achievements": [],
        "nationality": "",
        "languages": [],
        "confidence": 0.7,
    },
}
candidate = Candidate(**invalid_json)
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Candidate
# contact_information.email
#   value is not a valid email address: The part after the @-sign is not valid. It should have a period. [type=value_error, input_value='a@b', input_type=str]
```
\$\endgroup\$
1
  • \$\begingroup\$ <s>Do you think I should also alter the prompt that's inputted into ChatGPT?</s> EDIT Just seen your comment about injecting into the prompt - nice! \$\endgroup\$
    – Bob
    Commented Jan 9 at 23:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.