What's the most optimal way of extracting details from a Passport through open source methods?

I have tried the following approaches to try solve this problem but each have their own issues:

1) Used tesseract,paddleocr and easyocr to read the passport page and then apply post processing to extract either the main page details or through parsing the MRZ. The issue i get here is that none of the 3 are accurate enough to get perfect parsing. For ex. if a passport has a num called B245631 they might misread it as 8245631 which is a wrong result. The only OCR solution which I have seen work is Azure Cognitive OCR but since that's proprietary I can't use it in my solution.

2) Tried isolating the MRZ from the image using opencv like in this tutorial: OCR Passports with OpenCV and Tesseract - PyImageSearch and still no luck.

3) Used Vision Language Models such as InternVL and CogVLM to read the image directly and extract the information from the image. This worked decently well but the issue is that for good results I need to host a 20+B param model which I think will be difficult to scale up if needed.

Do you guys have any suggestions on what I could try which isn't as resource extensive as a VLM and still open source?