DEV Community: Mindee

Document Classification API

Jonathan Grandperrin — Sun, 28 Mar 2021 10:27:19 +0000

Workflows often involve document processing. And sometimes, you need to classify those documents automatically in your software. One reason can be that your users upload a bunch of different data in a unique flow, or they upload a single pdf including many different documents. It can be very tricky to automate this depending on your use case.

In this article, we’ll show you how to build an accurate document classification API that fits exactly your needs. In minutes, you’ll get your API up and running and you’ll be able to process millions of documents synchronously.

Our use case

Let’s take an example where your users are uploading documents on a single endpoint of your backend, and you want to classify them into 5 categories:

-W9
-1040 Forms
-Invoices
-Payslip
-Other

Once our API is trained, we’ll be able to launch specific workflows on those different types of documents.

Set up our document classification API

Create an account here: https://platform.mindee.com/signup and sign in. You’ll land on our home page.

Click on the “Create a new API” button.

Fill out a few information about your API. Give it a name, a description, and a cover image if you want.

Then click on “Next”.

This is the step where you are going to define your classes. Add a Classification field:

A popup will show up, we need now to input our different possible classes. Let’s fill the form with the classes defined earlier.

Once you click the "Add this classification field" button, we are all set.

Click the “Start training your model” button.

Train your document classifier

Your API was just deployed! Now we need to train the model.

To do so, we’ll need data, 15 samples for each type should be enough to get very high performances, but it’s up to you to train with more if you want to. It’s going to take you no more than 10 minutes to annotate your data once it’s uploaded.

The training interface looks like this:

On the left part of the screen, you can upload images, pdf, or zip archives. If you have all your training data in a folder on your laptop, just zip it and drag and drop it on the upload interface. You can mix pdfs and images, it’s not a problem as our backend will take care of this.

Gathering your samples for training is actually the most boring part of the process.

In my example, I have a total of 90 data, equally distributed. As it’s a dummy example, I’ve put random documents for the “other” class, but in your real-world use case, it’s better to use real data from your flow that you’d consider as “other”.

My zip file is ready. When I drag and drop the file on the left part of the screen, the data management pane opens:

Each data will appear automatically in the pane when it’s ready for annotation.

To make the annotation process easier, click on the setting icon in the header, and check the automatic data loading:

Let’s start annotating the data.

Click on “Your data set” on the left part of the screen, and click on the first document you see in the list.

Now, it’s very simple. Click on the desired class for each data on the right part of the interface:

Validate, and repeat.

It took me 4 minutes and 51 seconds to annotate my 89 data.

A model is trained every 20 data, and each of them is automatically deployed on your API under new versions:

V1.0 = no model
V1.1 = 1st model (20 data)
V2.2 = 2nd model (40 data)
…

You get an email when a model is deployed. My last model was deployed 15 minutes after I finished my 89 annotations. The first one was ready before I finished.

To know the performances or your model, ask the chat, we’ll give you the accuracy of your model. I got an overall accuracy of 96%, with confusion coming from invoices being classified as others. Adding a few more invoices and other documents would fix this.

Use the API

Once your first model is deployed you can test it right away with new data.

Hit the “Live interface” button on the sidebar, drag and drop a document. You should see something like this:

The latest version of your API (i.e the latest trained model) is automatically set for the live interface.

To integrate your API in your application, you can now hit the “Documentation” button in the sidebar.

There is everything you need to use the API:

API Reference: base url, request body, headers and sample codes
Response scheme
Limitations: technical limitations in payload size, rate limit
Open API: you can download the open API configuration to build your swagger collection, create automatically an SDK etc...

Conclusion

In under an hour, we’ve trained and deployed an API for classifying documents from 5 different classes. Either you have to process a few hundred documents per month, or tens of millions, you can safely use your API in your production environment. Our whole architecture scales automatically as the number of requests grows.

Feel free to contact us using the chat on https://mindee.com if you have any questions or if you just want to chat and understand how our algorithm works.

Extending License Plate Extraction

Doug Sillars — Thu, 25 Feb 2021 22:22:39 +0000

If you’re like me, you’ve gotten a speeding ticket. Even more like me, it was a speeding ticket…from a camera. My last ticket was in Dresden. I was crossing a bridge over the Elbe, and the signs all of a sudden said 50 km/hr.. And I was…. Well I was going over that. A few weeks later, a ticket arrived in our mailbox.

The ticket had a photo of my car (with me driving). You could see our license plate, but to show that there was no possibility of mistake, they also gave a closeup of our plate. With the details from my UK plates, they were able to look up my address and assign the ticket to me.

How did they scan the image for a license plate? Likely using an OCR or machine learning API like Mindee’s License plate API. When this API is provided a photo of a car (where the license plates are visible), it will find the region of the plate, and extract the details form that part of the plate.

For example, when I submit this image (from Google Image search)

I received the response “LR33TEE’. Read all about how to use our License plate API in our tutorial.

What can I to with the license tag information

Governments are able to identify us based on our plate. But what data can we extract from license plates? I know that in some countries, like Ireland and Germany, you can identify the region of the car’s origin from the letters on the plate. BUt I want more information.

But can we learn more? The UK government has a free API through the DVLA that uses the license plate number and provide information about the vehicle:

The call (using Axios in Node) looks like this:

  config: {

    url: 'https://driver-vehicle-licensing.api.gov.uk/vehicle-enquiry/v1/vehicles',

    method: 'post',

    data: '{"registrationNumber":"LR33TEE"}',

    headers: {

      Accept: 'application/json, text/plain, */*',

      'Content-Type': 'application/json',

      'x-api-key': <API TOKEN>',

      'User-Agent': 'axios/0.20.0',

      'Content-Length': 32

Now, it turns out, "LR33TEE" is not in the DVLA database. It's probably a fake reg tag Photoshopped for the article. So, we'll try again:

When you call this API, the response looks like this:

DVLA response {
  registrationNumber: 'S638GWH',
  engineCapacity: 1396,
  markedForExport: false,
  fuelType: 'PETROL',
  motStatus: 'Not valid',
  colour: 'SILVER',
  make: 'HONDA',
  yearOfManufacture: 1999,
  taxDueDate: '2016-11-04',
  taxStatus: 'Untaxed',
  dateOfLastV5CIssued: '2015-02-23',
  motExpiryDate: '2016-12-15',
  wheelplan: '2 AXLE RIGID BODY',
  monthOfFirstRegistration: '1999-01'

Now, this image is from an article with CCTV camera footage about a person stealing petrol. It is an older article, and as you can see - an older car, that is no longer taxed or has a valid MOT - probably in a junkyard somewhere.

What can we do with this data? For me, this is great, I can never remember when my MOT is due, or when I need to retax my car. Now I can use this API to quickly look up my car - just by knowing the numbers on my license plate. (They have a website where you can type this in).

But it can be easier!

Combining DVLA and Mindee’s License plate API, I can submit a photo of a vehicle, and it will extract the registration number automatically. Then this text string can be sent to the DVLA for a response.

I’ve built a super simple demo - use your phone to take a photo of a car with a UK license plate, and you can know the MOT date:

https://demo-license-plates.mindee.com/MOT

Want to know when your tax is due?

https://demo-license-plates.mindee.com/TAX

will tell you when your taxes need to be renewed.

Finally, https://demo-license-plates.mindee.com/CAMERA will give you the full output from the DVLA API.

Watch a video on how this magic works:

Want to try out the code? It's available on Github.

Going Further

What else could the combination of Mindee with DVLA information be used for? Thinking about the CCTV image at the petrol station. What if the API looked up the car for everyone pumping gas. The API contains the type of fuel for the vehicle. According to insurance estimates, over 150,000 vehicles in the UK accidentally pump the incorrect fuel into their gas tank. By taking a photo with the CCTV, the station can know what type of fuel should be used, and can stop the pump if the incorrect fuel line is connected.

Again - thinking of the image above: perhaps cars with expired tax and MOT are more likely to not pay for petrol, so these cars could be given a lower volume of petrol as a prepay.

Finally, if the station is also a service station - the receipt could also print a coupon for MOT service -reminding the customer that their MOT expires in 3 months.. 2 months.. 1 month, and give an appropriate discount for scheduling their service early.

Conclusion

The Mindee License plate API is both elegant and simple. You get useful information (the license plate number) just from a photo. Automatic parsing of a photo for this information now opens up a gateway of new opportunities. Now you can use additional APIs to probe for more information about a specific car or truck.

How to design and deploy your first document parsing API

Doug Sillars — Thu, 25 Feb 2021 21:55:27 +0000

This tutorial describes the steps required to create a custom document data extraction API using Mindee.

Over the course of this tutorial, you will learn how to train a deep learning model that can parse and extract the data of your choice in your documents.

Once the ML model is trained, you will be able to use the custom REST API backed by that model in your application code using the language of your choice.

Let’s see how to build and deploy your first document data extraction REST API using the following, fictitious scenario: as a developer, you want to extract the name of any burger food store that delivers orders to your door, along with its address, average user rating, user votes count as well as its first menu item name and price.

Set up your API

Prerequisites

You will need a free beta account. Sign up and confirm your email address before proceeding.
Download this data model configuration file.
Download this sample training set.

Once you have signed up and logged in at https://beta.mindee.net, press the I’m ready, start building button, as shown below:

First, we need some basic information about your new, custom document scanning API. That info will help us highlight your API in your Mindee API Hub (and in other users’ API Hubs if you choose to share your API with them). The requested properties are:

Document type: The common name of the document type your API will be trained on. Examples: 1040 US Tax Form, Pay Stub, Bank Statement
API name: The resource name of your API URL. We automatically generate one for you based on your document type but you can customize it at your own convenience. Note however that you can’t use special or accented characters. The format of your API URL complies with the following scheme: https://api-beta.mindee.net/[username]/[API_name]
Description: An optional, short blurb aimed at conveying the purpose of the API. This description will appear below the image in your API Hub (and others if you choose to share your API with other users). We suggest you make it as clear and concise as possible to maximize its effectiveness.
Image: An uploaded image that appropriately illustrates your API. Other users may see it in their API Hub if you choose to share it with them.

Here is a possible configuration for our first burger stores API:

Fill out the form with the following values (or pick your own):

Document type: Burger Stores and Menus
API Name: burger_stores
Description: An API used to extract data (such as store name, ratings and address) from popular burger stores
Image: upload this image

Once you’re done setting up your API, press the Next step: define your data model button to create the list of fields we’d like to extract from our burger stores documents.

Define your document data model

The following page shows up:

You can either manually add each field or upload a json-formatted file, typically generated from an existing data model someone previously built (and exported).

The following information is required for each field:

Field name: The legible field name used in your API documentation
Api response key: The field key available in your API response to carry the extracted field value for each document
Field type: The data type of the field you are trying to extract. The supported data types currently are: strings, numbers, dates, email addresses, phone numbers and URLs. Specifying a field type helps our deep learning selector better understand the data you’re looking for by restricting the number of potential candidates for each field. Learn more about the data types we support.

For the purpose of this tutorial, we want to extract the following information from our image or PDF documents:

The burger store name (a string)
The burger store address (a string)
The store average user rating (a floating number)
The votes count number (an integer)
The name of the first item on the menu (a string)
The price of the first item (an amount)

Below is a screenshot of one of our training files with each field above highlighted in green:

The Mindee representation of these fields after setting them up one by one is the following:

You can manually add them one by one or you can upload a configuration file to automatically populate your data model (this file is typically generated using the Config Export feature available to existing models in the Settings section of your API dashboard).

Press the Choose file button to upload the json-formatted config file (the one mentioned in the Prerequisites section above):

Once you’re done setting up your data model, press the Data model ready? start training button at the top of the screen:

Train the model

Now is the time to train our Burger Stores deep learning model in the Training section of our API:

For each of our documents we must assign a value to each field by selecting a box (or several consecutive boxes) displayed in the uploaded document. The Mindee selector highlights all the potential candidates for each field in blue, as shown below:

When you find the blue box that matches the field you are looking for, simply click that box. This populates the field text box with the selected value and shows the selected box in an identifiable color. For instance, the screenshot below highlights the box we selected for the average_rating field in orange:

If the content you are looking for is composed of more than one box, simply select all the consecutive boxes that match your field (non-consecutive boxes may result in an inefficient model). For example, he screenshot below shows the address field populated with 9 consecutive boxes:

You can download this sample dataset to train your Burger Stores deep learning model. Unzip the file into a folder and upload each file in that folder one-by-one to the Training page.

Once you have selected the proper box(es) for each of your fields (as displayed on the right-hand side of the document), press the validate button and repeat the tagging process for the next document.

The progress bar above each picture indicates the advancement of your tagging efforts and identifies the milestones at which a new model is generated.

Every 20 documents, a new model is being trained and automatically deployed into your API environment. You will receive an email whenever your latest model has been trained and is up and running:

Note that the more documents you feed into your model, the more accurate it will be.

With that being said, we will generate the first version of your trained model once you reach the 20 documents threshold (uploaded and validated).

At this point, you can experience the true power of the Mindee data extraction engine every time you upload a new training file, as the model will start filling out the fields on the right-hand side with the most likely values it was able to identify in the uploaded file.

From now on, the Burger Stores model will help you annotate new data in the Training section:

With a 20-documents training set, the model algorithm starts working pretty well on most fields, but still makes a few mistakes on the complex fields, such as the address field (extra words are predicted).

Once again, the more data you’re feeding to the model, the more accurate your API will get. So your job will now consists in correcting your model until you reach the next 20 documents threshold, which will kick off the generation of a new, more efficient model. You may repeat this task as often as you want.

Settings, API keys and documentation

If you select the Documentation tab, the following page is displayed:

This page allows you to explore your API endpoints and download your API definition as an OpenAPI-formatted file.

The API Keys tab allows you to create and manage your custom API tokens:

The Settings tab allows you to update general information about your API such as its name or its description. It also allows you to download your data model field configuration, which can be used to create a new, enhanced data model without having to start from scratch.

That’s it! You should now be able to set up a real world use case now and build whichever document parsing API you may need.

If you have any questions or feedback about this tutorial, feel free to reach out to us in the #tutorials channel of the Mindee Community on Slack (the Get Started email you have received when signing up contains an invitation to join it).

Parsing Invoices with Mindee's API

Doug Sillars — Thu, 25 Feb 2021 21:51:15 +0000

Parsing Invoices with Mindee’s API

Scanning and inputting data from invoices into accounting systems is a slow and tedious process. Using Mindee’s invoice parsing API, all of the pertinent data can be automatically and accurately extracted from your invoice in seconds, allowing for fast and painless data entry.

Interested in learning more? In this tutorial, we will walk through the steps to use Mindee’s Invoice API.

API Prerequisites

You’ll need Mindee account. Sign up for free. Then confirm your email to login.
An invoice. Use a recently received invoice, or do a Google Image search for a invoice and download a few to test with.

Setting up the API

Log into you Mindee account and access your Invoices API environment by clicking the Invoices card :

To activate the API, click the “Try for Free” button to access the free tier. You’ll land on the dashboard page - this page will show the usage of your account with the Invoice API - of course, it is empty now. On the left navigation, there are links to “Documentation”, “Credentials” and “Live Interface”. The docs tab has all of the technical details you’ll need to build for the Invoce API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API, so click on “Credentials” to create an API token.

Add a new token. In this example, I’ve named it “Tutorial”

Click “Add New Key” and you’ll be able to see your API token.

Now, we are ready to make an API call. In this example, we’ll be using cURL.

curl -X POST \ https://api.mindee.net/products/invoices/v1/predict  
    -H 'X-Inferuser-Token: {apiToken}'  
    -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'  
    -F file=@/path/to/your/file.png

Simply replace ```{apiToken}``` with your new API token and ```/path/to/your/file/png``` with the path to your invoice - wither an image or a pdf.

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

Here’s the invoice I used - the roofer who fixed the gasket around our chimney this spring (I’ve obfuscated my home address in the image):

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details. The full JSON can be accessed here (link to JSON file). Since the response is quite verbose, we will walk through the various fields section by section.

Extracted fields

Summary & Documents section

The first two sections of the response contain information about the API call made:

    "call": {
        "endpoint": { "name": "invoices", "version": "1.0" },
        "finished_at": "2020-09-11T18:39:58+00:00",
        "id": "3b4ac75a-f800-466b-bfc4-56c8f8e75134",
        "n_documents": 1,
        "n_inputs": 1,
        "processing_time": 1.047,
        "started_at": "2020-09-11T18:39:57+00:00"
    },
    "documents": [
        {
            "id": "5d72b0c2-16dd-481d-9e8b-031c5d1ab33a",
            "name": "Screenshot 2020-09-11 at 19.39.13.png"
        }
    ],

The call section tells us that we ran on the v1 of the invoice endpoint, uploading one document that is one page long. After a second, the file was processed, and the response transmitted back to me. The documents section gives the Mindee id for the file, and the filename (and you can tell I used a screenshot of the original invoice).

Predictions:

This is the fun part, where the API extraxcts pertinent details form teh invoice automatcilly, saving tedious manual data entry. Every invoice is different, and will have different fields. This invoice, being sent from the UNited States will not have some common European features like and IBAN or Tax Id.

Other peices are rpesent in this invoice, but not picked up by the API. We are planning multiple additional releases to this API in 2020 that will continue to improve the accuracy and precision of the results.

Company Number

In this case, there is no company number in the invoice:

"company_number": {
                "probability": 0,
                "segmentation": { "bounding_box": [] },
                "type": "N/A",
                "value": "N/A"
            },

Due Date

My invoice does have a due date, but it was not extracted.

            "due_date": {
                "iso": "N/A",
                "probability": 0,
                "raw": "N/A",
                "segmentation": { "bounding_box": [] }
            },

Invoice Date

The work was done in late June of this year, and the API correctly extracts the invoice date, along with four [x,y] points boxing in the value:

            "invoice_date": {
                "iso": "2020-06-29",
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [ 0.766, 0.392 ],
                        [ 0.855, 0.392 ],
                        [ 0.855, 0.416 ],
                        [ 0.766, 0.416 ]
                    ]
                }
            },

Invoice Number

The invoice is numbered 1277, which is extracted by the API, along with the four [x,y] corrdinates denoting the location in the image.

            "invoice_number": {
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [ 0.766, 0.363 ],
                        [ 0.807, 0.363 ],
                        [ 0.807, 0.382 ],
                        [ 0.766, 0.382 ]
                    ]
                },
                "value": "1277"
            },

Locale

Based on the information on the invoice, the API is able to predict (with 82% confidence) that this bill originated in the United States, is in USD, and is in English.

"locale": { "currency": "USD", "language": "en", "probability": 0.82 },

Orientation

The invoice did not require rotation to be parsed.

"orientation": { "degrees": 0, "probability": 0.99 },

Payment Details

The invoice does not contain any payment details, so were not extracted.

"payment_details": {
                "iban": "N/A",
                "probability": 0,
                "segmentation": { "bounding_box": [] }
            },

Supplier

The API is only 50% confident in this result, but it does extract the name of the company (despite without any spaces in the name).

            "supplier": {
                "probability": 0.5,
                "segmentation": {
                    "bounding_box": [
                        [ 0.446, 0.066 ],
                        [ 0.561, 0.066 ],
                        [ 0.561, 0.083 ],
                        [ 0.446, 0.083 ]
                    ]
                },
                "value": "SILVERHAMMER"
            },

Tax Id

Companies in the US generally do not list their Tax IDs on invoices, so this was not found.

            "tax_id": {
                "probability": 0,
                "segmentation": { "bounding_box": [] },
                "value": "N/A"
            },

Taxes

The API did not extract the texes, or the percentage tax rate (we have an update later this year t better extract US tax details.)

"taxes": [],

Total without Taxes

The API did not extract the taxes, but did get the cost before taxes with 99% confidence.

    "total_excl": {
                "amount": 350,
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [ 0.874, 0.803 ],
                        [ 0.931, 0.803 ],
                        [ 0.931, 0.825 ],
                        [ 0.874, 0.825 ]
                    ]
                }
            },

Total with Taxes

The total with taxes was alos extracted accurately with 99% confidence, meaning that the tax value could easily be calculated with a simple difference.

"total_incl": {
                "amount": 380.45,
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [ 0.842, 0.897 ],
                        [ 0.933, 0.897 ],
                        [ 0.933, 0.93 ],
                        [ 0.842, 0.93 ]
                    ]
                }
            }

Conclusion

Manual entry of invoices is a time consuming task. Adding the Mindee API to your system will simplify the extraction and make your accounts paybale processes run more accurately and smoothly. Do you have more questions? Click the chat button at the bottom right.

Parsing international passports with Mindee's API

Doug Sillars — Thu, 25 Feb 2021 21:48:44 +0000

Parsing international passports with Mindee's API

Many onboarding processes in mobile or web apps require to extract some data from ID documents. In this tutorial, you will learn how to automatically extract data from passports to offer to your users the best onboarding experience.

We will walk through the steps to use Mindee’s Internation Passport Parsing API. Let’s get started!

API Prerequisites

You’ll need a free Mindee account. Sign up and confirm your email to login.
A picture of a passport. You can use yours safely as data protection is one of our priority and we won't send your images to any third party application. You can also download a fake one here.

Setting up the API

Log into your Mindee account and access your passport API environment by clicking the International Passport card:

To activate the API, click the “Try for Free” button. This will give you access the API for free 50 times a month. There are four sections on the API landing page, noted in the left navigation. You are currently on the dashboard, and there are additional links to “Documentation”, “Credentials” and “Live Interface”. The docs tab has all of the technical details you’ll need to build for the passports API endpoint, and the Live Interface is a cool interactive demo.

We'll start by clicking "credentials" and creating a new API token, in this case, named Tutorial:

Click “Add New Key” and you’ll be able to see your API token,.

If you move back to the documentation tab, you can pick your API token, and the language, and create an API call. Here is the API call in cURL:

curl -X POST \ https://api.mindee.net/products/passport/v1/predict \ 
-H 'X-Inferuser-Token: {apiToken}’ \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F file=@/path/to/your/file.png

Simply replace {apiToken} with your new API token and /path/to/your/file/png with the path to your receipt.

That's it! We're all set to run this command in the terminal, and see the data that Mindee can extract.

In this tutirla, I'll use a sample [Cyprus passport](https://commons.wikimedia.org/wiki/File:Cyprus_passport_data_page.jpg) from Wikimedia.

Extracted fields

The JSON response can be a bit verbose, so we will break it into sections to describe what we are seeing. The initial sections describe the API and the API response:

"call": {
        "endpoint": {
            "name": "passport",
            "version": "1.0"
        },
        "finished_at": "2020-09-04T20:57:51+00:00",
        "id": "7d81f5fe-fce1-4b63-b3de-cc5948c39b55",
        "n_documents": 1,
        "n_inputs": 1,
        "processing_time": 1.988,
        "started_at": "2020-09-04T20:57:49+00:00"
    },
    "documents": [
        {
            "id": "57aded44-a216-45be-859c-f949cc5b011d",
            "name": "1024px-Cyprus_passport_data_page.jpg"
        }
    ],

Here, we can see that passport API was called, and it processed one document in about 2 seconds. What we are really interested in seeing is the results, and these can be found in the predictions section of the JSON file:

Predictions

The API response is in alphabetical order of items extracted from the document:

Birth Date

"birth_date": {
                "probability": 1.0,
                "segmentation": {
                    "bounding_box": [
                        [.053,  0.912],
                        [0.955,0.912],
                        [0.955,0.969],
                        [0.053,0.969]
                    ]
                },
                "value": "1970-01-01"
            },

The API matched the brithdate in the MRX fields (more on that in a minute), so is 100% confident that the birthdate is 1 January, 1970. The bounding box gives four (x,y) points in the image where the date of birth can be found.

Country of issuance

The API then attempts to parse the country that issued the passport.

"country": {
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [0.479,0.199],
                        [0.522,0.199],
                        [0.522,0.234],
                        [0.479,0.234]
                    ]
                },
                "value": "CYP"
            },

The API is 99% certain that Cyprus is the issuing country. This is really incredible, as the API team tells me that there are no Cyprus passports in the data training set. Again, 4 (x,y) coordinates in the image point out where CYP is used.

Expiration Date

"expiry_date": {
                "probability": 1.0,
                "segmentation": {
                    "bounding_box": [
                        [0.053,0.912],
                        [0.955,0.912],
                        [0.955,0.969],
                        [0.053,0.969]
                    ]
                },
                "value": "2020-12-01"
            },

Many countries do not allow entry with under six months of validity left on the passport. This woman should probably start the renewal process, as the API has identified that this passport will expire on 1 December, 2020.

Gender

            "gender": {
                "probability": 0.1,
                "segmentation": {
                    "bounding_box": [
                        [0.053,0.912],
                        [0.955,0.912],
                        [0.955,0.969],
                        [0.053,0.969]
                    ]
                },
                "value": "F"
            },

The API's confidence is not high on this one, just 10%, but it does correctly identify the gender as female.

Given Name

            "given_names": [
                {
                    "probability": 0.99,
                    "segmentation": {
                        "bounding_box": [
                            [0.048,0.844],
                            [0.952,0.844],
                            [0.952,0.895],
                            [0.048,0.895]
                        ]
                    },
                    "value": "AFRODITI"
                }
            ],

The API is really certain (99% confidence) that the given name on the passport is Afroditi, meaning that our passport holder was named after the Greek goddess of beauty and love.

ID Number

            "id_number": {
                "probability": 1.0,
                "segmentation": {
                    "bounding_box": [
                        [0.673,0.197],
                        [0.774,0.197],
                        [0.774,0.233],
                        [0.673,0.233]
                    ]
                },
                "value": "K00000413"
            },

While we are not numbers, in many ways, we are identified by numbers. In this case, the passport ID number, which must be used when booking flights, checking into hotels, etc. The API has found the location (marked by the 4 (x,y) points in the bounding box, for the Passport number, which is K00000413.

Issuance Date

Contining the alphabetical trip through the passport fields, we next come to the date the passport was issued:

            "issuance_date": {
                "probability": 0.18,
                "segmentation": {
                    "bounding_box": [
                        [0.353,0.586],
                        [0.426,0.586],
                        [0.426,0.62],
                        [0.353,0.62]
                    ]
                },
                "value": "2014-01-21"
            },

In this case, the API missed the issuance date. However, while our API does miss in it's predictions, in this case, the issue is due to the fact that the passport is a sample (we'll have a blog post on this soon). The issuance date is missing from the MRZ zone of the sample passport, so the API is unable to accurately measure the issuance date.

Surname

"surname": {
                "probability": 0.98,
                "segmentation": {
                    "bounding_box": [
                        [0.33,0.272],
                        [0.444,0.272],
                        [0.444,0.309],
                        [0.33,0.309]
                    ]
                },
                "value": "ANONYMOU"
            }

Our fake passport holder has a surname of Anonymou, which is correctly identified by the API, along with bounding boxes.

MRZ1 and MRZ2

The last two regions identified by the API (which I pulled *slightly* out of alphabetical order for readability) are the 2 Machine Readable Zone (MRZ) lines:

"mrz1": {
                "probability": 0.99,
                "segmentation": {
                    "bounding_box": [
                        [0.048,0.844],
                        [0.952,0.844],
                        [0.952,0.895],
                        [0.048,0.895]
                    ]
                },
                "value": "P<CYPANONYMOU<<AFRODITI<<<<<<<<<<<<<<<<<<<<<"
            },
"mrz2": {
                "probability": 0.1,
                "segmentation": {
                    "bounding_box": [
                        [0.053,0.912],
                        [0.955,0.912],
                        [0.955,0.969],
                        [0.053,0.969]
                    ]
                },
                "value": "K000004134CYP7001017F2012010<<<<<<<<<<<<<<<4"
            },

The MRZ zones (if you cannot tell from the JSON above) are the two lines of machine readable text at the bottom of every passport page. MRZ1 is line one, and MRZ2 is line 2. Most of the identifying information can be pulled from this area. The confidence interval for line one is high, as the fake passport has this area filled out correctly. However, the confidence level is low, as the algorithm expected more characters other than "<" to supply the issuance date.

Conclusion

And there you have it! Mindee's passport API parsed a passport in under 2 seconds, matching nearly all of the sections of the passport quickly and accurately. Give it try with our free tier, and let us know in the chat how it works for you!

Invoice parsing automation Microsoft Flow, Mindee and Excel

Doug Sillars — Thu, 25 Feb 2021 21:18:28 +0000

Invoice parsing automation Microsoft Flow: Mindee and Excel

Mindee’s invoice parsing API makes the extraction of invoice details quick and easy. Our RESTful API takes an image or PDF of an invoice and extracts all of the pertinent details of the invoice. In under a second, it returns a JSON response with the extracted elements along with coordinates of their visual location in the document and the confidence of our ML algorithm. We’ve blogged on how to use the Invoice API with Curl, NodeJS, and Python.

We realize that this is just part of the invoice ‘process’ at your company. In this post, we’re going to extend the API further using Microsoft Power Automate. By creating a ‘flow’ for your invoices you can build automation connecting different Office 365 services to automate your workflows.

Robotic Process Automation (RPA)

RPA is using computers and automation to streamline mundane tasks. By reducing the number of time employees have to do simple and repetitive tasks, the employees have more time for high-value work - increasing productivity and reducing human errors that might occur during the process. The process described below certainly removes a lot of process steps to the invoice process, so it is an RPA.

Low Code/ No Code

Even better, this process also can be considered a low code implementation. There's no need to pull one of your developers off a project to implement. Simply following these instructions (and having access to an Office 365 account), any member of staff should be able to add this automation step to the invoice process.

Complete Invoice Processing

Extracting the pertinent data from the invoice is a huge step in simplifying the processing of an invoice, but it is just one step in the required workflow. In today’s digital age, many invoices arrive via email and must be rescued from your inbox for processing. Then all the details must be entered into a spreadsheet (or another database) - all this before you actually *PAY* the invoice.

Many of these steps can be automated, and with Mindee’s Invoices API, the most tedious part (the data entry into the payment system) can be automated as well. If your company is using Microsoft Office 365, this automation can be done using Microsoft's Power Automate flows.

In the following example, we’ll automate the process from email delivery to storing the invoice data into an Excel Online spreadsheet. Inside the workflow itself, we’ll save the invoice to a document library SharePoint Online, call the Mindee’s Invoices API and populate additional, custom columns in the SharePoint library and. A picture is worth a thousand words, so here is a diagram of the flow as we built it:

Test the Flow

You can try this flow by sending an email to doug@riolinx.com (make sure you attach an invoice in PDF or JPG format). Microsoft Flows seem to run every 5 minutes or so, so you’ll soon see your invoice parsed and added to the Excel document. Feel free to try it out as often as you’d like!

While this is a fu example, it does not do a lot of good for your company's invoices to end up in a publically available spreadsheet on the internet. You'll want to create your own (private) version of this Flow for your company's invoices. Read on to learn how:

Importing the Flow

The flow we created is available on Github. Download the "MindeeInvoiceswithMicrosoftFlow.zip" from the repository. (You'll also need the invoice_excel.xlsx file in a bit, so you might as well download that now as well.).

To add the Flow to your Microsoft account, navigate to https://flow.microsoft.com and log in with your work credentials. You may have to create a free trial of Power Apps if you do not already have Flow/Power apps enabled.

Click on the “My Flows” in the left navigation.

On this page, there is a link to import a flow:

Click import to begin the upload of the Zip file. After the Zip file is uploaded, the page will ask you to name the flow and to connect it to your Outlook account. The connection is required in order for the Flow to have access to your emails.

Step 1: Under name - import setup - Create as new, and give the Flow a name that makes sense for your organization.

Step 2: Connect your Outlook - click “select during import.” In my case, I did not have a Flow connection set to Outlook, so I had to click “Create New” and connect to my SharePoint account. Clicking the refresh allowed me to add the connection:

Here I am connecting to a new email address in a new Microsoft Flow account.

Now, once all the connections have been made, you can press the blue import button at the bottom of the page. After a few seconds, the package will import successfully.

Editing the flow

In order for this flow to succeed in your Office 365 environment, you’ll need to make a few edits. Head back to “My Flows” and select the Mindee integration and press the edit button.

Add your email account

The email account that receives invoices in your organization. In the flow, it is currently set to doug@riolinx.com - which does your organization very little good. We’ll need to update that:

Click the “On new email” and select advanced options:

Now, edit the “To” field to correspond to the address that will be receiving invoices in your organization.

Mindee API Key

The next step in the process is your Mindee API Key. You’ll need to paste in your API key in the Value field. To obtain a Mindee Invoice API Key, visit Mindee.com and follow these steps to get your key.

Now you can save your Flow. To turn it on, head back to the main view of your flow, and select “Flow Checker”. You'll see a warning that the flow is off - and a button to turn it on.

This will flag another warning - don’t worry - it is just telling you that the flow has never run successfully. Of course, it hasn’t - you haven’t had a chance yet!

Test it out

Send an invoice to the email address you configured in the flow (don't forget to attach an invoice!). The Flow will run in the next few minutes, and you’ll see the successful run in your 28-day run history.

Clicking the date and time (in the 28-day run history table) allows you to check the status. If you scroll to the bottom of the Flow, you’ll see the “apply to each attachment in the email.” Open this, and then Open “Update Excel Online Spreadsheet”, and open one of the variables:

The screenshot above shows that the API correctly extracted the invoice date as 29 June 2020. This is super cool - as we now have a Flow that interacts with the Mindee API to extract data from submitted invoices.

But your Flow does not actually DO anything with that data yet. In the next sections, we’ll connect the flow to Excel and to SharePoint, and submit the data to these Office tools for future manipulation.

Connect an Excel Online Spreadsheet

In the Github repo, there is an excel file: Invoice_excel. Upload this into your Excel Online account. To do this: Open Excel in a new tab:

Upload the file into Excel. It will open, and you’ll find an empty table with headers that look like they will intake invoice data. This table is named “invoices2021”

Connect the Flow to the Excel table:

Back in your Flow, click the edit button and open the “apply to each Attachment in the email” (at the bottom). Open the “Update Excel spreadsheet” entry and add an action at the bottom(the circle with a '+' sign in it). We’re going to “add a row to a table”:

Choose the Invoice_excel file you just uploaded:

And the table is “invoices2021.” Now that this is added, all of the columns in the table will appear in the Flow. Add the variables that were assigned earlier by clicking each entry box in the form, and adding the dynamic variables as shown below (the purple dynamic entries correspond to the variables that exit the Mindee flow):

Click save, and test the flow. You can use the trigger you used in the initial test to run the flow immediately.

The difference is now that the invoice details are added automatically to the Excel file.

Yes!!!

Saving the file to SharePoint

In My SharePoint site, I have created a new document library called "invoice". When it is created, I land on a page with a table where my files will end up.

Looking at the column names, I see that I can add columns to my table! We can add the extracted Mindee data right into the SharePoint Table:

Clicking "add column" (note: always choose "single line of text" - in case of the API responds with a "N/A") opens this dialogue:

For this example, I picked 4 variables: supplier name, invoice date, due date, and total due. I could have added more - it is up to you what data you want to appear here.

Connect the Flow to SharePoint

Now that we've created the library, we need to wire it to connect to our automated Flow.

First, we'll want to insert the file into the document library. So between the last variable, and before we add to the Excel table, insert a step Sharepoint: Create File:

Fill in the form - the SharePoint and Folder path are your company's SharePoint, and the library you just created for your invoices. Then use the dynamic variables to add the "Attachment Name" and "attachment contents" to the last two fields.

The file is now added, but to fill the new columns created in the library, we need yet another connector: SharePoint: Update File Properties:

We again connect this to our new SharePoint Library. To specify which line we apply, choose ItemId - this refers back to the document we just created one step back. When you add the library name, it will find all of the columns, you added. Then we can use the variables from the Mindee Flow to fill out the columns of the SharePoint library, as shown above.

Adding the SharePoint Path to Excel

The Excel table has fields to help you find the file in SharePoint, so go back to the "add a row to excel: and insert the SharePoint Filename and path to the flow, and connect the dots between all the files and connections we've made with this flow.

Now we're ready to save the whole flow, and test!

The test

Once we send an email to the account, we see that not only is the invoice added to the SharePoint Library, but the details are also added to the excel table (with the SharePoint file details in the Excel file):

Conclusion

To increase productivity, many organizations are using Robotic Process Automation and low/no-code tools to automate tedious and slow tasks. The processing of invoices is one such process that is readily suited for automation.

Using Mindee's API to extract data from an invoice is just one piece of the invoice pipeline. In this demo, we've extended the use of the Mindee invoice extraction API with Microsoft Flow - taking an email attachment, saving it in SharePoint, and extracting the details as column data in SharePoint as well as creating a row in Excel.

With all of the tools inside Microsoft 365, this is just the tip of the iceberg - you could potentially integrate with any other product in the suite. Try out our Mindee Flow to extract data, and then use the variables to wire the data into your favorite Microsoft Project. If you have any questions (how to reach out, etc.)

Passport Machine Readable Zone: protect your identity online

Doug Sillars — Thu, 25 Feb 2021 21:09:55 +0000

Passport Machine Readable Zone: protect your identity online

Your passport. Accepted as photo identification nearly everywhere, and required for international travel. The data on your passport can be used to create accounts at banks, and potentially can lead to identity theft. So, how do you protect your passport details from theft?

Mr. Bean's passport

It should be pretty clear that you should not post your passport on social media, Of course, this does not stop people, and they are unwittingly exposing their private data.

The Human Readable Zone

The Top of your passport has all of the information you expect to see. In the example of Mr. Bean (above), we learn that he was born 6 Jan 1955 in Enfield, UK. we can also read the passport number, and the date the passport was issued (and when it expires).

The Machine Readable Zone

The Machine Readable Zone (MRZ for short) is the 2 lines of text at the bottom of the passport. If you look at your passport, the first line of the MRZ has a 3 letter abbreviation for your country, followed by your name (all separated by "<"). The Mr. Bean passport is clearly fake, as it only states "GB" instead of the expected "GBR" that is used officially.

The second line of the MRZ appears to be more gibberish, but there are important values there (and this is where you might accidently leak your private information. It kicks off with your passport number + a checksum digit), followied by your nationality, your date of birth (+1 digit checksum), your sex, and finally the expiration date of your passport.

The Mr. Bean passport above correctly shows his passport number and nationality. It begins to break down on date of birth - the MRZ reports as: 6 November 1981 (different from the human readable zone by several decades). It also shows the issuance date to be 42 April 2000 (hmmm).

Publicly posting your passport

Even if you carefully obfuscate your passport, you can leak information from the MRZ section of your passport. Here is a photo of my passport, where I've drawn a giant yellow box over the entire human readable region:

Trust me, this is my passport.

If I run this passport (with the MRZ zone exposed) through the Mindee Passport Extraction API, it correctly identifies all of my information - as it can extract it from the MRZ. The curl command looks like:

curl -X POST 
  https://api.mindee.net/products/passport/v1/predict 
  -H 'X-Inferuser-Token: {myAPItoken}' 
  -H 'content-type: multipart/form-data;
  -F file=@/path/to/my/passport.jpg

I receive a JSON result (here are a few snippets of 'sharable' information that were correctly extracted):

    "country": {
                "probability": 1,
                "segmentation": { "bounding_box": [] },
                "value": "USA"
            },
    "given_names": [
                {
                    "probability": 0.2,
                    "segmentation": {
                        "bounding_box": [
                            [ 0.056, 0.915 ],
                            [ 0.479, 0.915 ],
                            [ 0.479, 0.939 ],
                            [ 0.056, 0.939 ]
                        ]
                    },
                    "value": "DOUGLASS"
                }
            ],

The API can read the MRZ and extract all of the private data that I so carefully hid in the human readable zone.

Exposing the MRZ

So, what are people exposing on social media?

The young man knew not to post the entire photo of his passport, but the MRZ gave away his passport number (I've contacted him to let him know).

Finding Fake Passports

Poorly forged documents (like Mr. Bean's passport above) will have discrepancies between the Human Readable and the MRZ. In a recent article in Angola, a woman claims that her passport was forged and used in crimes. The article includes an image of "her" passport:

If we read the bottom line of the MRZ, nothing matches the top of the passport. The can be tested with the Mindee API:

The MRZ Zones are predicted as:

            "mrz1": {
                "probability": 0.27,
                "segmentation": {
                    "bounding_box": [
                        [ 0.147, 0.803 ],
                        [ 0.861, 0.803 ],
                        [ 0.861, 0.882 ],
                        [ 0.147, 0.882 ]
                    ]
                },
                "value": "PNAGOISABEL<<DOS<SANTOS<<<<<<<<<<<<<<<<<<<<<"
            },
            "mrz2": {
                "probability": 1,
                "segmentation": {
                    "bounding_box": [
                        [ 0.148, 0.867 ],
                        [ 0.865, 0.867 ],
                        [ 0.865, 0.951 ],
                        [ 0.148, 0.951 ]
                    ]
                },
                "value": "N1473613<8AGO8909176M18090600444874<N01<1356"
            },

Extracting the passport number "N1473613" - it clearly does not match the number on the passport "N1471383".

Issuance date in the MRZ: 17 Sept 2008. On the passport it states 6 Sept 2018.

The "M" following the issuance date provides the incorrect gender.

The Expiration date appears as 18-Sept 1989, while the passport states 6 Sept 2021.

This passport is clearly fake based on these details. The icing on the forgery cake is the signature. This woman is clearly not "Bruce Lee"

Conclusion

The MRZ zone is often overlooked by passport holders, who are unaware that private data is encoded in these two lines. These two lines should be kept as secure as the rest of your passport, as they have exactly the same information in them. To learn more about information extraction from passports, look no further than the Mindee API.

An API for my API?

Doug Sillars — Fri, 09 Oct 2020 20:41:50 +0000

Please Add Your API

Last weekend, the kids and I hopped into the car for a long day trip. I drove the car a few hours south to Folkestone, UK. We had our passports examined, and after a short wait, I drove our car onto a train. While we were still inside the car. 30 minutes later, the train stopped in Calais, and we were in France.

The service is called Eurotunnel, and it is an easy way to get your right hand drive car to Europe, and drive on the right side of the road (or left hand drive car to drive on the left side in the UK).

Leaving the Eurotunnel train. Photo by my daughter.

In our case, we literally drove 2 roundabouts, and then took the train back to the UK (its a strange visa thing, probably better that you don't ask :))

What does any of this have to do with APIs?

Boy, I'm glad you asked, but I was getting there... really.

When you book a trip on the Eurotunnel, you head over to Eurotunnel.com, and enter your travel dates, and your car's license plate. They use an API to lookup the make and model of your car - so that they can establish the cost for you to travel. (We have a large van, so we had to go in the oversized car section). See, there. I talked about an API!

You pick your dates, and your departure times - just like choosing a flight (remember when we could book a flight and take a holiday?). You see the price, enter your card number, and you end up on a success page! Hooray, you're heading to France!

Not so fast

A bit later, you check your email, and you see your confirmation email from Eurotunnel. At the top, it tells you about the times and that you need to get to the terminal 45 minutes before departure....blah blah blah. There's a section on traveling during Coronavirus - which you might glance at.

If you're still in the email, but before you get to the "follow Eurotunnel on Twitter" button, there was a section that drew my attention:

Ensure your API is Complete

Oh - like an Open API Specification - make sure that my API is complete... Then I remember that I am not reading an industry e-mail, I'm reading my travel itinerary to take a train with my car.

"If you do not supply your API before travel" - that seems oddly specific. Again, I bet most people do not have their own API.

"Add an API to your booking". Is this like a hackathon challenge? Will Eurotunnel refund some of my ticket if I build something cool? Alright - I'm actually going to click that link!!

I land on a webpage that reads like like weird and demented developer clickbait.

Need help populating your API - yes! yes I do! Are you going to provision some awesome API tokens or something?

API just got easier Awesome. Is there like a Zapier integration that I can leverage to connect your API? Will you help me become RESTful?

Yeah, it is way less cool than that

For developers, API means Application Programming Interface. What Eurotunnel really wanted was:

Advance Passenger Information? Lame.

The above screenshot (with the happy green checkbox) is what it looks like AFTER you add your API. Beforehand, all the text is red, with a pink background, and it warns you that without adding your API, you might not get past the ticket booth, or that the border control might not let you on the train.. Or that they'll let you on the train, but never let you off.... I don't remember all of the details, but there were dire consequences.

Needless to say, I provided my API. And it turns out, all they wanted was the information from my passport.

First Name, Surname, Nationality,

passport number,

Date of Birth,

Passport Expiration.

You know, the stuff we used to fill out on travel booking websites, back when we could travel internationally.

This got me thinking

If you've ever applied for a job online, you know that you upload your resume, and the website scans all the text and autofills your name, address, email, and all your experience in a long form that you can then submit to a mailbox that nobody monitors.

It's kind of cool that computers can read your whole resume and then ignore you.

But why can't airlines (or the Eurotunnel) take a picture of my passport and extract all this standard passport data for me (and my kids)? Why did I fill out an online form for every flight (for for everyone traveling), when just uploading an image could have accomplished exactly the same thing?

The Passport API exists

Mindee.com has a passport extraction API. There is a tutorial on how to use it on their website (and if you sign up, there is a live demo that you can test for free. Basically, you upload a photo of your passport photo page to their API endpoint, and it will extract all the information you want into a JSON object.

The algorithm also gives you bounding boxes around the location of the information, allowing you to build images like this (this is the live demo on the API page):

This is a fake passport.

You also get a nice form with all of the extracted fields:

Country, passport number and name - all extracted automatically from the image.

An API for my API!

Perhaps my family travels more than average, but I have images saved of all of my family member's passports on my computer. Imagine how easy check-in would be if I could just upload 6 images, and the website could fill in all the passport dates (issue, expiration, birthdate) passport number and name for each traveller.

There would probably be fewer errors for Eurotunnel, and it would be a lot faster and easier for their users.

Then I thought...what if all the airlines did this? Give people an option. "Fill in this form, or upload a picture of your passport" (Kind of sounds like how you might apply for a job on LinkedIn, amirite?) It would make the whole process easier for customers!

ProTip: Ryanair- you could even offer automatic passport upload as an "add-on" (For €1.50 per traveller, would you like to upload your passport rather than enter the values manually?)

Manual Data Entry Sucks

Why doesn't EVERY travel company in the world offer this? It is not a security thing - many passport control gates look at the passport photo and decide if you are automatically allowed into the country or not. This should be a no-brainer for EVERY travel site in the world.

Conclusion

My initial joking about using APIs for passport data entry let me to look for an API that can actually parse passport data automatically. I found Mindee's and found it very easy to implement. Solutions like this will make check-in process faster, have fewer errors, and easier for customers. I cannot wait to see it in service in 2021, the next time I plan to travel internationally.

Parsing receipts with Mindee's Machine Learning API

Doug Sillars — Thu, 03 Sep 2020 09:26:26 +0000

Anyone who has filed an expense report can tell you: receipt tracking and expense logging is a headache. Enter Mindee’s receipt parsing API, which uses deep learning to automatically, accurately and instantaneously parse your receipt details.

In this tutorial, we will walk through the steps to use Mindee’s Receipt Parsing API. Let’s get started!

API Prerequisites

You’ll need a free Mindee account. Sign up and confirm your email to login.
A receipt. Look in your bag/wallet for a recent one, or do a Google Image search for a receipt and download a few to test with.

Setting up the API

Log into you Mindee account and access your Expense Receipt API environment by clicking the Expense Receipts card:

To activate the API, click the “Try for Free” button to access our generous free tier. You’ll land on the dashboard page - where you can quickly see API usage (you have none right now, but that will change). On the left navigation, there are links to “Documentation”, “Credentials” and “Live Interface”. The docs tab has all of the technical details you’ll need to build for the receipts API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API, so click on “Credentials” to create an API token.

Add a new token. In this example, I’ve named it “Tutorial”:

Click “Add New Key” and you’ll be able to see your API token,.

Now, we are ready to make an API call. In this example, we’ll be using cURL.

curl -X POST \ https://api.mindee.net/products/expense_receipts/v2/predict \ 

-H 'X-Inferuser-Token: {apiToken}’ \

-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \

-F file=@/path/to/your/file.png

Simply replace

{apiToken}

with your new API token and /path/to/your/file/png with the path to your receipt.

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

In this example, I used a receipt from the grocery store in Koln airport (my last business trip in 2020):

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details. Since the response is quite verbose, we will walk through the various fields section by section.

API Response: Parsing Results

Summary & Documents section:

The first two sections of the response contain information about the API call made:

"call": {
    "endpoint": {
        "name": "expense_receipts",
        "version": "2.1"
    },
    "finished_at": "2020-08-29T18:01:22+00:00",
    "id": "e47d8654-0df7-4839-a282-2c04bf293886",
    "n_documents": 1,
    "n_inputs": 1,
    "processing_time": 1.087,
    "started_at": "2020-08-29T18:01:21+00:00"
},
"documents": [
    {
    "id": "66d9adc6-76cf-4c42-8622-3dadb660ac32",
    "name": "IMG_20200301_073354.jpg"
    }
],

The call section tells us that we ran on the expense receipts endpoint, uploading one document that is one page long, and after just about 1 second, the file was processed. The documents section gives the Mindee id for the file, and the filename.

Predictions:

Now we are getting to the exciting stuff. The Predictions section is broken into several sections. Several of these are identifying fields on the receipt, and others are using Machine Learning to deduce information from the receipt. Let’s go through each section:

Date

Identified from text on the receipt and converted into ISO format. This purchase was made on February 3, 2020, and the model is 99% confident in that choice. The segmentation bounding box provides 4 (x,y) coordinates indicating where the date was pulled from the receipt [(0,0) is the upper left corner, (1,1) is the bottom right corner].

"date": {
    "iso": "2020-02-03",
    "probability": 0.99,
    "raw": "03-02-2020",
    "segmentation": {
    "bounding_box": [
      [0.64,0.661],
      [0.801,0.661],
      [0.801,0.686],
      [0.64,0.686]
  ]
}

Locale

Using data from the receipt, the API can predict where the purchase was made, the language and the currency:. Check the documentation for the latest support. At the time of writing, support is centered on Europe and North America.

"locale": {
    "country": "DE",
    "currency": "EUR",
    "language": "de",
    "probability": 0.77,
    "value": "de-DE"
},

In the case of my receipt, it is 77% confident that the purchase is in German, made in Germany, and in euros.

Merchant

"merchant": {
    "name": "REWE",
    "probability": 0.91,
    "segmentation": {
    "bounding_box": [
      [0.279, 0.135],
      [0.719, 0.135],
      [0.719,0.23],
      [0.279,0.23]
     ]
}

The API correctly predicted (it was 91% sure) that it was a REWE store. Again, four (x,y) points mark the location of the text naming the Merchant on the image.

Orientation

"orientation": {
   "degrees":0,
   "probability": 0.99
},

Did the document require rotation before parsing? Measured in 90 degree increments [0.90.180.270]. In this case, it did not require any rotation.

Taxes

"taxes": [],

If any taxes are identified in the receipt, the will appear here. In this case, no taxes were found (but this is the correct result).

Time

"time": {
    "iso": "15:50",
    "probability": 0.99,
    "raw": "15:50",
    "segmentation": {
    "bounding_box": [
      [0.649,0.898],
      [0.732,0.898],
      [0.732,0.925],
      [0.649,0.925]
    ]
}

Time the receipt was printed, confidence, and the (x,y) coordinates that bound the field in the image.

Total

Perhaps the most important part of the receipt, the total spent, along with confidence and the box indicating the location on the receipt.

"total": {
    "amount": 17.74,
    "probability": 0.99,
    "segmentation": {
    "bounding_box": [
     [0.663,0.589],
     [0.765,0.589],
     [0.765,0.617],
     [0.663,0.617]
   ]
}

Summary:

In just over 1 second, a receipt was uploaded, parsed and the response returned to the end user. We know that €17.74 was spent at a REWE in Germany on February 3, 2020 at 15:50. Using the bounding boxes, we can have the user validate the values, and then input this data into an expense management system.

Conclusion

Using the MIndee receipt parsing API, you can quickly validate receipts, allowing for faster, more accurate (and less painful) expense management for our users. If you have questions, please reach out to us in the chat widget in the bottom right.