​
Background

Key Features:

  • Uses /parse with LLM calls to determine what pages match your description
  • Can be chained with /extract to determine which pages match the type that you would want to extract from

Key Inputs:

  1. split_description

    • name: Name of the category of pages you’re trying to extract
    • description: Details describing the characteristics of the pages you want
    • partition_key (optional): Identifier for splitting sections by. In the output the partition_key will be filled in with values from the content and appended after the split_description name
      • Examples: account number, index, name, holding type

  2. split_rules

    • Optional field for changing the default rules
      • Example: Each page can only be classified into a single category

​
Example Use Cases

  1. Scenario 1: You have a long document and the information that you plan to extract is only on some subset of the pages.
  2. Scenario 2: You have a long document with information that can be clearly broken down into subsections.

​
Scenario 1:

​
You have a long document and the information that you plan to extract is only on some subset of the pages.

  1. Run /parse on the file and get the parse job id

  2. Run /split on the parse output. Example config:

    {
    "document_url": "jobid://{parse_job_id}",
    "split_description": [
        {
            "name": "Clinical Scenarios",
            "description": "Pages of the Clinical Scenarios"
        },
        {
            "name": "Recommendation",
            "description": "Page that includes the recommendation at the top of the page"
        }
    ]
}

    Example response:

    {
  "usage": {
    "num_pages": 20
  },
  "result": {
    "section_mapping": {
      "Clinical Scenarios": [
        1,
        2,
        3
      ],
      "Recommendation": [
        1
      ]
    }
  }
}

  3. Run /extract on the parse output using the same document_url you sent into split. Set the page range as the pages returned from split:

    {
    "document_url": "jobid://{parse_job_id}",
    "advanced_options": 
    { "page_range": [
            {
                "start": 1,
                "end": 2
            }
        ] },
    "schema": extract_schema
}

​
Scenario 2:

​
You have a long document with information that can be clearly broken down into subsections.

An example of this would be a financial statement with multiple accounts. Each account will have a unique account number. Consider the following schema:

{  
  "type": "object",  
  "properties": {  
    "acct_holdings": {  
      "type": "array",  
      "items": {  
        "type": "object",  
        "properties": {  
          "price": {  
            "type": "number",  
            "description": "The price of the holding per unit."  
          },  
          "quantity": {  
            "type": "number",  
            "description": "The quantity of the holding"  
          },  
          "ticker": {  
            "type": "string",  
            "description": "The ticker of the holding."  
          },  
          "category": {  
            "type": "string",  
            "description": "The category of the holding, such as equity, fixed income, or cash."  
          },  
          "acct_number": {  
            "type": "string",  
            "description": "The unique number for the account."  
          }  
        }  
      },  
      "description": "A list of holdings across all the accounts."  
    }  
  }  
}

The schema is extracting the holdings. So we can make a split request to get the holdings for each account. From the schema description, it’s clear that the acct_number is a unique field. By setting the account number as the partition key, we will get the account holdings returned in multiple sections where each section has a different account number. If the partition key is not set in this case, all of the account holdings would be returned in a single array.

Example config:

  "split_description": [  
    {  
      "name": "account holdings",  
      "description": "Pages for each separate account holdings",  
      "partition_key": "account number" 
    }  
  ],  
  "split_rules": "Split the document into the applicable sections"  
}

Example response:

{  
  "usage": {  
    "num_pages": 58  
  },  
  "result": {  
    "section_mapping": {  
      "account holdings 1111": [  
        1,  
        2,  
        3  
      ],  
      "account holdings 2222": [  
        7,  
        8,  
        9,  
        10,  
        11,  
        12  
      ],
      "account holdings 3333": [  
        19,  
        20,  
        21,  
        22,  
        23,  
        24  
      ]
    }  
  }  
}

You can then run extract on each subset of the pages. The “acct_holdings” can be concatenated across the extract calls since its an array.

Extraction PipeliningPolicies