A POC that uses GPT 4 Vision API to generate a digital form from an Image using JSON Forms from https://jsonforms.io/

💭 Inspired by:

1. screenshot-to-code：https://github.com/abi/screenshot-to-code
2. draw-a-ui：https://github.com/SawyerHood/draw-a-ui

Both repositories demonstrate that the GPT4 Vision API can be used to generate a UI from an image and can recognize the patterns and structure of the layout provided in the image.

Click the thumbnail to watch on YouTube:

https://nathanfhh.github.io/Digital-Form-with-GPT4-Vision-API/

I am using pdf.js to process the PDF file and request to OpenAI's API to generate the response entirely in the browser.

1. cd into frontend directory

cd ai-json-form

2. Install Packages and run

npm install
npm run dev

1. cd into directory

cd backend

2. Install Packages

poetry install
# alternatively, you can use pip install
pip install -r requirements.txt

3. Setup Environment Variables

export OPENAI_API_KEY=
# optional
export OPENAI_ORG=

If you plan to use the Mock response only, you should set OPENAI_API_KEY to any value.

4. Run

python main.py

1. export the environment variables

echo "OPENAI_API_KEY=YOUR_API_KEY" > .env
# The following is optional
echo "OPENAI_ORG=YOUR_ORG" >> .env

2. Run the docker-compose

docker-compose up --build

3. Open the browser and visit http://localhost:8080/aijsv/

I am new to Vue, so the code might not be the best practice. I am still learning and improving. Should you have any suggestions, please feel free to PR.

1. Upload PDF files of up to three pages from the frontend

   If you want to adjust the number of pages, you can change the MAX_PDF_PAGES variable in backend/app/socket.py

2. When the backend receives the PDF file in Base64 string format, it does the following processes:

   • Convert the URL String Back to Bytes
   • Read the PDF file, convert it to a JPG image, and save it to the /tmp folder using the package pdf2image.
   • Extract the strings from the same PDF file using the package PyPDF2. The extracted strings will become part of the prompt sent to the GPT4 model to enhance accuracy.
   • Prepare the prompts and send them along with the PDF screenshot to the GPT4 Vision API
   • Send the chunk to the frontend via Socket.IO incrementally.

3. Whenever the frontend receives the chunk, it appends it to the codemirror editor, and checks if the current content is a valid YAML. If it's a valid YAML, it will apply it to the JSON Scheme to force the UI to re-render.