Getting Started With DocuPipe

DocuPipe is here to do one thing: convert complex documents into a consistent, structured payload that has the same fields and data types every time.

The flow comprises of three parts:

POST a document, which may include complex entities like tables, key-value pairs, checkboxes, signatures, etc. This triggers our AI processing, and returns a documentId and jobId.
GET by jobId to see when the document is processed. This gets a plain text representation of that document that is easily readable for both humans and AI readers.
POST a standardize request. A standardize request has a document ID, as well as a schema, and generates a structured JSON

This getting started guide will walk through one example. Let's say we have this PDF file on our local machine : example_document.pdf

Authentication

Every request to DocuPipe needs to include an API key. You may obtain your API key by signing up and going to this link, where your API key is visible at the bottom of the page.

Posting a Document

The first thing you want to do, is take that file and post it to DocuPipe. Replace YOUR_API_KEY with your actual API key obtained in the previous step. Supported file formats are: PDF, images (JPG, PNG, WEBP), text files, and JSONs. Regardless of the format, use base64 encoding as shown below.

import base64
import requests

url = "https://app.docupipe.ai/document"
api_key = "YOUR_API_KEY"

payload = {"document": {"file": {
    "contents": base64.b64encode(open("example_document.pdf", 'rb').read()).decode(),
    "filename": "example_document.pdf"
}}}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "X-API-Key": api_key
}

response = requests.post(url, json=payload, headers=headers)
document_id = response.json()['documentId']

const fetch = require('node-fetch');
const fs = require('fs');

// Replace with your actual DocuPipe API key
const api_key = "YOUR_API_KEY";
const url = "https://app.DocuPipe.io/document";

// Read and encode the file in base64
const filePath = "example_document.pdf";
const fileContents = fs.readFileSync(filePath);
const base64Content = Buffer.from(fileContents).toString('base64');

// Construct the JSON payload
const payload = {
    document: {
        file: {
            contents: base64Content,
            filename: filePath
        }
    }
};

// Make the POST request with JSON payload
fetch(url, {
    method: 'POST',
    headers: {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "X-API-Key": api_key
    },
    body: JSON.stringify(payload)
})
.then(response => response.json())
.then(data => {
    const document_id = data.documentId;
    console.log(document_id); // Output the document ID
})
.catch(error => console.error('Error:', error));

If we print our the response, we'll see it gives us an ID for the document, which we may later use to query for its AI and human readable result:

print(response)
=> {'documentId': '96dde1aa', 'jobId': '42ace16a'}

What you get is essentially a pointer that you can then use to query for the document's results, which you do with a GET request. Typical processing time is about 5-15 seconds for a entire document, depending on its size.

Polling for Job Completion

As soon as you upload a document, DocuPipe will extract its underlying text, tables, and build a good textual representation for both human and AI readers. This takes a bit of time: about 5 seconds for a single page, and as much as a minute for 100+ page documents.

You need to poll for document state and wait for it complete processing

import time
import requests

job_id = '42ace16a'
url = f"https://app.docupipe.ai/job/{job_id}"
headers = {
    "accept": "application/json",
    "X-API-Key": "YOUR_API_KEY"
}

status = "processing"
wait_seconds = 2

while status == "processing":
    response = requests.get(url, headers=headers)
    status = response.json()["status"]
    print(status)
    time.sleep(wait_seconds)
    # exponential backoff is recommended when waiting for things to complete
    wait_seconds *= 2

const fetch = require("node-fetch");
const jobId = '42ace16a';
const url = `https://app.docupipe.ai/job/${jobId}`;

const headers = {
  accept: "application/json",
  "X-API-Key": "YOUR_API_KEY",
};

let status = "processing";
let waitSeconds = 2000; // milliseconds

async function poll(url) {
  while (status === "processing") {
    const response = await fetch(url, { headers });
    const data = await response.json();
    status = data.status;
    console.log(status);
    await new Promise((r) => setTimeout(r, waitSeconds));
    // exponential backoff
    waitSeconds *= 2;
  }
}

// usage
poll(url);

Note you could avoid polling for the document to become prepared, by using webhooks, which you can read about in this guide.

Retrieving Document Results

DocuPipe is a RESTful API, which means you append document_id to the URL path when you want to get the results.

url = f"https://app.docupipe.ai/document/{document_id}"

headers = {
    "accept": "application/json",
    "X-API-Key": "YOUR_API_KEY"
}

response = requests.get(url, headers=headers)

print(response.json())

const getDocumentResults = (documentId) => {
    const url = `https://app.DocuPipe.io/document/${documentId}`;

    fetch(url, {
        method: 'GET',
        headers: {
            "accept": "application/json",
            "X-API-Key": "demo-api-key"
        }
    })
    .then(response => response.json())
    .then(data => console.log(data))
    .catch(error =>
console.error(error));
};

getDocumentResults(document_id); // Replace document_id with the actual ID

This will get you the result below. Our parsed text is human-readable, but more importantly it is AI readable, which lets you reason about it and ask your favorite Large Language Model questions about its content.

{
    "documentId": "e95af17c",
    "status": "completed",
    "result": {
        "pages": [
          {
            "sections": [
              {
                "type": "text",
                "text": "the first paragraph in your doucment",
                "bbox": [
                  0.1, 0.12, 0.4, 0.2
                ],
                "header": "paragraph",
                
              }
            ],
            "text": "string",
            "pageNum": 0
          },
        ],
        "numPages": 0,
        "text": "string",
        "language": "string"
      },
}

Standardizing a Document - convert it to JSON with a structure

On DocuPIpe, you usually want to build a schema first, which lets you get consistent JSONs with predictable format. You can create schemas using our user interface. We assume you've already created a schema, and this is how you call it:

import requests

url = "https://app.docupipe.ai/v2/standardize/batch"

payload = {
    "documentIds": ["exampleDocumentId"],
    "schemaId": "exampleSchemaId"
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "X-API-Key": "YOUR_API_KEY_HERE"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())

const url = 'https://app.DocuPipe.io/standardize/batch';
const options = {
  method: 'POST',
  headers: {
    accept: 'application/json',
    'content-type': 'application/json',
    'X-API-Key': 'YOUR_API_KEY_HERE'
  },
  body: JSON.stringify({documentIds: ['exampleDocumentId'], schemaId: 'exampleSchemaId'})
};

fetch(url, options)
  .then(res => res.json())
  .then(json => console.log(json))
  .catch(err => console.error('error:' + err));

This will return a standardizationId, which again you can poll for using a GET request, which will finally give you our JSON view of the document, which we call a standardization.

What that standardization will include depends on the exact schema used. A schema can let you specify exactly what you want to surface from each document.

As an example. maybe the input is a rental contract, and you set up a schema to extract some basic fields. The output may look like this:

{"monthlyAmount": 2000,
 "currency":"USD",
 "moveInDate":"2020-31-01", 
 "depositAmount": 3000, 
 "depositCurrency":"USD"}

Using our schema creation dashboard, you can create very complex schemas that are specific to your use case. You can add an exact field for an annual payment for a rental contract, or have a field to describe whether tenants are likely allowed to keep a pet crocodile in the house. Schemas let you understand documents in a way that can be entirely unique to your use case.

There's plenty more to DocuPipe API. You can:

Classify documents by type for triaging downstream actions (standardize using different schemas depending on classification outcome).
Split a long document into smaller atomic sub-documents using AI to intelligently decide where one ends and another begins.
Generate a visual review of a standardization to see exactly what pixels justify every decision made by our AI.
Use our Workflow concept to automate a sequence of events (e.g. upload -> classify -> standardize) in a single call. See here for code example.
Use Webhooks for more efficiency in receiving output payloads.