Ai Invoice Processing — Architecture & Technical Analysis

1. Executive Abstract, Problem Formulation & Background

An Efficient Deep Learning-Based Approach to Automating Invoice Document Validation Aziz Amari Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia azizamari@ieee.org Mariem Makni and and and and and Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia mariem.makni@insat.ucar.tn Wissal Fnaich Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia wissal.fnaich@insat.ucar.tn Akram Lahmar Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia akram.lahmar@insat.ucar.tn Fedi Koubaa Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia fedi.koubaa@insat.ucar.tn Oumayma Charrad Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia oumayma.charrad@ieee.org Mohamed Ali Zormati Affiliation: National Institute of Applied Sciences and Technology (INSAT) University of Carthage Tunis, Tunisia zormati@ieee.org Rabaa Youssef Douss Affiliation: National Institute of Applied Sciences and Technology (INSAT) COSIM Lab. LR11TIC01 University of Carthage Tunis, Tunisia rabaa.youssef@insat.ucar.tn Abstract In large organizations, the number of financial transactions can grow rapidly, driving the need for fast and accurate multi-criteria invoice validation. Manual processing remains error-prone and time-consuming, while current automated solutions are limited by their inability to support a variety of constraints, such as documents that are partially handwritten or photographed with a mobile phone. In this paper, we propose to automate the validation of machine written invoices using document layout analysis and object detection techniques based on recent deep learning (DL) models. We introduce a novel dataset consisting of manually annotated real-world invoices and a multi-criteria validation process. We fine-tune and benchmark the most relevant DL models on our dataset. Experimental results show the effectiveness of the proposed pipeline and selected DL models in terms of achieving fast and accurate validation of invoices. Index Terms: invoice processing, deep learning, document layout analysis, object detection, optical character recognition. I Introduction In large organizations, it is a common case to manually process a steady stream of invoice documents, for example, to reimburse staff expenses. Invoice processing for validation purposes is quite challenging. In fact, proof of authenticity mainly relies on the presence of a handwritten signature and a stamp [ 1 ] , and the verification of key elements of the document (e.g., document type such as receipt and invoice, client name, date of issue, etc.). It is therefore a necessity to consider automated processing, which requires handling the variability of document layout and quality [ 2 ] , in order to guarantee reliable invoice validation. To enable efficient understanding and processing of invoice documents, it is appropriate to consider deep learning (DL) techniques, which have revolutionized the accuracy and efficiency of both text recognition and object detection and have shown remarkable results in various applications [ 3 , 4 ] . For the text recognition needed for invoice validation, many recent methods, such as LiLT [ 5 ] , LayoutLMv2 [ 6 ] , and LayoutLMv3 [ 7 ] are at the forefront, allowing the understanding of complex structured documents, such as invoices. These attention-based models include embedded optical character recognition (OCR) to enhance document understanding. Regarding object detection, a wide range of deep architectures are proposed in the literature, including FCOS [ 8 ] , Faster R-CNN [ 9 ] , RetinaNet [ 10 ] , and SSD [ 11 ] . These DL models can potentially meet the need of identifying and localizing stamp and signature, which is an underlying task of invoice validation. When considering DL models for invoice validation, and since these models are not specifically trained on this particular task, it is relevant to consider transfer learning, where models pre-trained on large datasets can be fine-tuned on a smaller dataset specific to invoices. Recent works applied transfer learning on financial document datasets [ 12 , 13 , 14 ] , which improved model performance with limited data. However, after reviewing these datasets, we note that some of them are of receipt type [ 12 , 13 ] , others consider a variety of forms [ 14 , 15 ] , which differ slightly from a regular machine written invoice. In [ 16 ] , the authors propose a large dataset of 10000 synthetic invoices, specifically tailored for document analysis and understanding. However, these documents do not always reflect real-world scenarios where invoices are often photographed with mobile phones, introducing significant distortions. We also notice that many of the datasets lack the presence of stamp and signature components. In order to fill this gap and to effectively apply DL techniques, we propose a novel, fully annotated dataset composed of real machine written invoice documents with various imperfections, such as blurriness, class imbalance, low-quality phone-shot images, and varying formats. In our case study, we consider a machine written invoice document to be valid only if it contains a stamp, a signature, and certain selected fields. After collecting and annotating the dataset, we select the most relevant DL-based approaches for implementation and benchmarking. We progressively evaluate and improve the results until obtaining a robust pipeline that maximizes accuracy and reliability, while maintaining reasonable performance. The remainder of this paper is structured as follows. In Section II , we present the material considered and the methods proposed. The results are reported and discussed in Section III , while Section IV concludes the paper and discusses the future perspectives. II Materials and Methods In this section, we present the materials, detailing the data collection and processing pipeline, and then describe the methods, including the selection of the most recent and relevant models for implementation. Since invoice validation consists of two main tasks, we divide our work into two independent steps: keyword detection and object detection. The adopted training pipeline is shown in Figure 1 , which shows that keyword detection is preceded by an OCR phase, which allows for an OCR-augmented dataset. For object detection, we aim to verify the presence of signatures and stamps to ensure the authenticity of financial documents. Fig. 1: Adopted training pipeline. II-A Materials We outline the dataset used for our DL pipeline by first describing the data annotation process, followed by data augmentation and preparation. II-A 1 Data annotation We collected 381 real financial documents, of which 84% (320 documents) are invoices. Among these, 240 are machine-written. We manually annotated the documents with bounding boxes around the necessary elements to establish the validity of the invoices, as shown in Table I . All classes are represented as bounding boxes, except for a boolean attribute indicating whether handwriting is present. TABLE I: Annotated dataset description. Class Description Title Document type (e.g., invoice, receipt, etc.). Client Client name. Stamp Stamp for document authenticity. Signature Signature for document authenticity. Date Issue date, different formats. Total Total label. Total Value Total numeric value. Handwritten True if handwritten (not machine-written). An example of an annotated document is shown in Figure 2 . For confidentiality reasons, some parts of the financial documents have been blurred. Fig. 2: Example of an annotated document. Our dataset includes various real invoices; many are distorted due to mobile phone capture as depicted in Figure 3 , and some are handwritten, potentially appearing skewed during capture, as illustrated by Figure 4 . Fig. 3: Example of a distorted document. Fig. 4: Example of a skewed document. This work focuses on the processing of digital invoices, since handwritten invoices are sparse in the current dataset, and present additional challenges for augmentation and processing. II-A 2 Data augmentation and preparation In order to improve the ability of the DL models to generalize, data augmentation and preparation techniques are required due to the limited size of collected dataset. Given that the adopted workflow is divided into two main tasks, namely keyword detection and object detection, each task is subject to specific and adapted augmentation techniques. For the keyword detection task, the challenges encountered by a DL model include the presence of unclear characters and image distortions such as rotation, shear, or scale transformations. Therefore, we choose to augment the dataset using the following techniques: • Median blur and color jitter: improves clarity by reducing noise and simulating various lighting conditions, particularly in documents with complex backgrounds. • Padding: prevents critical text from being cut off during data preparation, with a probability set to 1. • Rotation: is limited to 5 degrees using inter-cubic interpolation to minimize data loss. In order to preserve information and standardize image dimensions, an additional step ensures no cropping occurs closer than 20 pixels from any bounding box after rotation. • Resize: is performed using bi-linear sampling to maintain consistent image dimensions, supporting uniform model performance. For the stamp and signature detection task, our goal is to present these objects under varying conditions. The following outlines the augmentation techniques we apply: • Rotation: is performed with a maximum of 10 degrees to simulate the common skew in scanned documents. The rotation transformation is applied with a probability of 0.5 to ensure data diversity. • Color jitter: is performed with a probability of 0.4 to account for color variation and lighting conditions, which enhances the model’s resilience in detecting signatures and stamps across different documents. • Multiplicative noise: is added on a per-channel basis with a multiplier ranging from 0.5 to 1.0, applied with a probability of 0.3. This noise simulates digital scanning artifacts and helps train the model to be less sensitive to such variations. Figure 5 illustrates the distribution of the final dataset after data augmentation. It is important to note that the class distribution remains unchanged following the augmentation process, as the applied operations only affect the number of instances per class. Fig. 5: Final dataset class distribution. II-B Methods This subsection outlines the deep learning models we plan to train and evaluate for both keyword detection and stamp and signature detection tasks. II-B 1 Keyword detection We perform keyword detection to identify significant parts of the document (e.g., customer name, document type, date, etc.). To achieve this, we use (i) an OCR step followed by (ii) a document layout analysis step. This approach is necessary for attention-based models that incorporate embedded OCR to enhance document understanding. Below, we present the selected methods for both steps and provide a comparison to determine the most effective combination of OCR and document layout analysis methods for keyword detection. An OCR step for character extraction from input images, utilizing robust modules, is essential for the document layout analysis task. Choosing the appropriate OCR model requires comparing the most relevant OCR tools. The most relevant OCR models in the literature are tested on the FUNSD dataset [ 14 ] , which stands for Form Understanding in Noisy Scanned Documents. To evaluate the similarity between the extracted and the ground truth text in the dataset, we consider two metrics from the original papers: accuracy and similarity based on Levenshtein distance [ 19 ] , which is particularly suited for measuring the distance between characters in a word. TABLE II: Comparison of OCR models on FUNSD [ 14 ] . Model Accuracy Similarity EasyOCR 0.284 0.719 Tesseract [ 12 ] 0.306 0.569 Amazon Textract 0.345 0.513 Google OCR 0.576 0.892 Microsoft Azure OCR 0.633 0.915 Table II summarizes the results of the most relevant OCR models according to original papers, showing that Microsoft Azure OCR outperforms all methods and is the best cloud-based solution. Therefore, Microsoft Azure OCR is selected as our primary method. Additionally, we consider using Tesseract during implementation to explore the potential of a local model and to confirm or refute our decision. After the OCR step, we use document layout analysis models to automate the extraction, categorization, and interpretation of information from the invoice documents. Recently, attention-based architectures have become commonly used, since attention mechanisms allow models to focus on the most relevant parts of the input data. This improves their ability to handle complex documents. Three relevant models are considered in this work. First, LayoutLMv2Large [ 6 ] combines textual and visual data to interpret complex document structures. The model employs a multimodal transformer architecture with the following pre-training objectives: masked visual-language modeling to enhance context understanding, text-image alignment for better simultaneous processing of visual and textual data, and text-image matching to ensure that the model can distinguish between matched and unmatched text-image pairs. This method uses a cross-entropy loss function for masked language modeling tasks, optimizing accuracy and relevancy of predictions. Second, we consider LayoutLMv3Large [ 7 ] , which also combines textual and visual data. It leverages a multimodal transformer architecture with several pre-training objectives: masked language modeling to enhance the understanding of textual context, masked image modeling to learn document layouts, and word-patch alignment to improve simultaneous processing of visual and textual data. As for LayoutLMv2Large [ 6 ] , this mùodel uses a cross-entropy loss function for masked language modeling tasks. Third, LiLT [ 5 ] is also a relevant model that employs a dual-stream Transformer architecture, processing textual and layout information in parallel by integrating through a bi-directional attention complementation mechanism. It is pre-trained using: masked visual-language modeling to enhance language comprehension while preserving layout awareness, and key point location and cross-modal alignment identification to understand and align textual and visual elements within documents. This method also uses cross-entropy loss for masked visual-language modeling to handle multi-class classification problems effectively. TABLE III: Comparison of document layout analysis models on RVL-CDIP [ 17 ] . Model Accuracy Parameters LayoutLMv2Large [ 6 ] 0.9564 425.9 millions LayoutLMv3Large [ 7 ] 0.9593 356.0 millions LiLT [ 5 ] 0.9568 130.0 millions In Table III , we summarize the accuracy results of the three relevant document layout analysis models as evaluated on the RVL-CDIP dataset [ 17 ] according to original papers. We also include parameter count for the three models. From these results, we can deduce that LayoutLMv3Large [ 7 ] and LayoutLMv2Large [ 6 ] show high accuracy rates, making them strong candidates. LiLT [ 5 ] also performs well, though it has fewer parameters compared to other models, making it a good candidate when reduced inference time is needed. II-B 2 Signature and stamp detection Object detection models locate instances of objects in images by estimating bounding boxes. In our case, we need to detect the presence of stamps and signatures to guarantee the authenticity of documents. We select four promising models for this purpose. First, RetinaNet [ 10 ] , which is renowned for its approach in single-stage object detection, significantly enhances detection capabilities through feature pyramid networks and focal loss. Focal loss reduces the loss contribution from easy examples and increases the importance of correcting misclassified examples. Second, the Fully Convolutional One-Stage (FCOS) model [ 8 ] operates on a per-pixel prediction basis, eliminating the need for anchor boxes, simplifying the detection pipeline, and reducing computational complexity. FCOS introduces several novel components in its loss function to handle object detection challenges effectively: classification loss, which is a binary cross-entropy loss that classifies whether each location is the center of an object, center-ness loss, which is a novel loss that down-weights the contribution of low-quality detections during training, focusing more on the center regions of objects, and regression loss, which is a bounding box regression loss using IoU-loss that measures the accuracy of the predicted boxes against the ground truth. Third, Faster R-CNN [ 9 ] is renowned for its two-stage object detection approach, which significantly improves both speed and accuracy. The loss function in Faster R-CNN is a multi-task loss that combines classification loss (a softmax loss that evaluates the predicted class probabilities against the ground truth class labels) and bounding box regression loss (a smooth L1 loss that measures the difference between the predicted bounding box coordinates and the ground truth coordinates). Finally, Single Shot MultiBox Detector (SSD) [ 11 ] is a single-stage object detection algorithm known for its balance between speed and accuracy. The loss function in SSD, as for Faster RCNN, is a combination of classification loss and bounding box regression loss. TABLE IV: Comparison of object detection models on COCO [ 20 ] . Model mAP Parameters RetinaNet [ 10 ] 41.5 38.2 millions FCOS [ 8 ] 39.2 32.3 millions Faster R-CNN [ 9 ] 37 41.8 millions SSD [ 11 ] 25.1 35.6 millions We present a comparison of the models using the number of parameters and their mean average precision (mAP) score on the Microsoft COCO dataset [ 20 ] . The results are summarized in Table IV . Among these models, RetinaNet [ 10 ] has the highest mAP, indicating superior performance in detecting and classifying objects within documents. We move forward with fine-tuning and testing the presented methods for stamp and signature detection. Based on the comparisons made for keyword detection and stamp and signature detection, we summarize our choices as follows. We select Microsoft Azure OCR and Tesseract for OCR step, based on their high accuracy and similarity scores. Regarding document layout analysis, we move forward with LayoutLMv3, LayoutLMv2, and LiLT, to be mainly evaluated using the F1-score. It is worth noting that we consider standard models (not large ones, i.e., LayoutLMv3Large and LayoutLMv2Large) for their lower resource comsumption and better fit for our data. For object detection, we benchmark RetinaNet, FCOS, Faster R-CNN, and SSD, using the mAP metric. These models are chosen based on their performance on benchmark datasets and their suitability for our specific tasks for invoice document validation. III Results and Discussion In this section, we present the results of the benchmarking performed on the selected models after fine-tuning them on our data. We also examine the challenge of class imbalance within the OCR-augmented dataset and its implications. In our experimental evaluation, we focus on precision, for a highly reliable invoice validation pipeline, with consideration of limited computational resources. The implementation of the models is performed using hardware consisting of an NVIDIA GeForce RTX 3050 GPU and the PyTorch framework. We first present and discuss results for keyword detection, then the results for signature and stamp detection. III-A Evaluation of selected OCR models To ensure keyword detection reliability, we evaluate the two selected OCR methods: Microsoft Azure OCR and Tesseract. Since our dataset was not fully annotated for the OCR step, we propose to use these two OCR models in inference mode in order to produce annotations. Subsequently, we fine-tune the LayoutLMv3 model using these annotations and evaluate which of the two OCRs performs better with the fixed layout document analysis model. The training of LayoutLMv3 is performed under identical conditions, maintaining the same number of epochs and other hyperparameters. TABLE V: Comparison of fine-tuning LayoutLMv3 using Tesseract versus Microsoft Azure OCR. Method Validation Loss F1-Score Recall Precision Tesseract [ 12 ] 1 ​ e − 2 1e^{-2} 0.81 0.77 0.87 Azure OCR 3.6 ​ e − 2 3.6e^{-2} 0.56 0.48 0.68 As shown in Table V , Tesseract [ 12 ] demonstrates higher effectiveness with an F1-score of 0.81, indicating robust detection capabilities. In contrast, Microsoft Azure OCR’s results are more conservative, with an F1-score of 0.56 and a precision of 0.68. However, the variation in the dataset size handled by each OCR tool is substantial, as depicted in Table VI . Azure OCR’s ability to extract more instances on average provides a significant advantage, allowing the model to more accurately distinguish between irrelevant other (denoted as ’O’) labels and targeted fields. TABLE VI: Comparison of class instance counts. Class O Title Date Client Total Total Value Azure OCR 31715 217 242 553 315 221 Tesseract 6272 189 201 161 198 192 We present the inference results with annotations powered by Tesseract (cf. Figure 6 ) and Microsoft Azure OCR (cf. Figure 7 ). As discussed earlier following the results of Table VI , Azure OCR provides more selective and stable results, capturing target fields with minimal errors, whereas Tesseract exhibits less precision, often mislabeling or capturing non-target fields. In conclusion, the robustness and selectivity of Microsoft Azure OCR make it the preferred choice for the critical task of text recognition. Fig. 6: LayoutLMv3 output using Tesseract. Fig. 7: LayoutLMv3 output using Azure OCR. III-B Final keyword detection results For document layout analysis, we train models on 50 epochs with a batch size of 4. We opt for Adam optimizer and L2 regularization with a weight decay of 10 − 3 10^{-3} . The learning rate is also fixed at 5 ​ e − 5 5e^{-5} for the three models. TABLE VII: Comparison of document layout analysis models. Model GFlops Inference Time (s) Parameters Validation Loss F1-Score Precision LayoutLMv3 44.83 2.08 125.93 millions 3.6e-2 0.56 0.68 LiLT 43.2 1.04 130 millions 3e-4 0.53 0.61 LayoutLMv2 43.7 2.20 200 millions 3.53e-2 0.47 0.49 (a) LayoutLMv3 (b) LiLT (c) LayoutLMv2 Fig. 8: Training and validation loss comparison for keyword detection. (a) LayoutLMv3 (b) LiLT (c) LayoutLMv2 Fig. 9: F1-score over training steps comparison for keyword detection. Addressing data imbalance is a crucial step in this process. We opt for using the focal loss, which is an altered version of weighted cross entropy designed to address class imbalance more effectively by focusing on hard-to-classify examples. The remainder of this subsection presents the evaluation results of the three selected state-of-the-art document layout analysis models. Our analysis emphasizes both on the F1-score, for a balanced measure of precision and on recall, to highlight true positive rates. Table VII summarizes the evaluation results and suggests that LayoutLMv3 performed the best, while LiLT required less inference time than the two other models. Figure 8 shows the training and validation loss of the retained models. In all three plots, the training loss consistently decreases as the number of training steps increases. This suggests that the models are learning the training data effectively. However, the ideal scenario is for the validation loss to also decrease alongside the training loss. However, we found that the validation loss for LayoutLMv3 (cf. Figure 8a ) starts to increase by around 1,250 steps, which is a clear indication of overfitting. LiLT (cf. Figure 8b ) follows a similar but less severe pattern. LayoutLMv2 has the best performance, as depicted in Figure 8c . The validation loss, after an initial slight decrease, can be seen to be fairly flat, thus indicating good generalization without overfitting the training data. Based on these results, we conclude that overfitting was observed with both the LiLT and LayoutLMv3 models. However, they remain the top-performing models based on their least F1-scores. The overfitting issue could be mitigated by implementing an early stopping strategy. For interpretation purposes, we chose to continue training all models for the same number of epochs to show their overall tendencies. In future work, these models should be re-trained with early stopping, halting the training when no improvement in validation loss is observed after a certain number of steps. Figure 9b depicts the F1-scores over training for each model. All models show an increasing trend in F1-scores with more training steps, though the rate of improvement varies: LayoutLMv3 achieves the highest overall F1-score, indicating superior performance (Figure 9a ), while both LiLT and LayoutLMv2 follow. Although not the fastest, LayoutLMv3’s inference time remains acceptable (see Table VII ). III-C Signature and stamp detection In this section, we present the results of training and validation of object detection models for signature and stamp detection. We mainly focus on training and validation loss, as well as mAP over epochs for each model. Results are summarized in Table VIII . All models were trained with a momentum of 0.9 and L2 regularization (weight decay of 0.0005). Key parameters for training include a batch size equal to 8, 50 epochs, and a learning rate of 0.001. TABLE VIII: Comparison of signature and stamp detection models. Model GFlops Inference Time (s) Parameters Training Loss Validation Loss mAP@0.50 mAP@50:95 RetinaNet 128.32 0.29 36.4 millions 0.22 0.38 83.96 45.70 Faster R-CNN 134.38 0.18 41 millions 0.11 0.20 81.351 42.085 FCOS 80.34 0.17 32.1 millions 0.73 1.18 76.94 41.77 SSD 30.48 0.11 23.9 millions 1.33 3.59 68.33 31.19 (a) RetinaNet (b) Faster R-CNN (c) FCOS (d) SSD Fig. 10: Training and validation loss comparison for signature and stamp detection. (a) RetinaNet (b) Faster R-CNN (c) FCOS (d) SSD Fig. 11: mAP comparison for signature and stamp detection Examining the training and validation loss is important for evaluating the performance of object detection models. Based on the results shown in Figure 10 , we note that RetinaNet (Figure 10a ) shows a significant decrease in training loss, indicating effective learning, while validation loss initially decreases but later fluctuates, indicating potential overfitting. Faster R-CNN (Figure 10b ) follows a similar trend. Meanwhile, both FCOS (Figure 10c ) and SSD (Figure 10d ) have decreasing training losses with relatively high validation losses. We conclude that RetinaNet demonstrates superior generalization ability, Faster R-CNN remains competitive with some overfitting, while FCOS and SSD suffer from significant overfitting, impacting their overall performance and generalization. Figure 11 depicts the mAP over epochs for the four considered models. We note that RetinaNet (Figure 11a ) quickly stabilizes around 0.84 mAP@0.50, indicating high detection accuracy, with a steady increase in mAP@50:95. Faster R-CNN (Figure 11b ) achieves 0.81 mAP@0.50, slightly below RetinaNet, and maintains consistent performance in mAP@50:95. FCOS (Figure 11c ) reaches 0.77 mAP@0.50 but shows lower and more variable mAP@50:95 compared to previous models. Finally, SSD (Figure 11d ) has the lowest 0.68 mAP@0.50 and variability in mAP@50:95 highlighting challenges in achieving good precision. Fig. 12: Inference results for RetinaNet. In summary, RetinaNet has superior detection accuracy and stability across IoU thresholds, while Faster R-CNN and FCOS perform well but lag behind in accuracy under stricter IoU conditions. SSD faces significant challenges in achieving high precision across all thresholds. Given these results, RetinaNet stands out as the best model. It offers the highest detection accuracy with reasonable computational efficiency, although not the fastest. The inference results for RetinaNet (see Figure 12 ) demonstrate its robustness in accurately detecting and localizing signatures and stamps across various document types, further reinforcing its suitability for the signature and stamp detection task. Based on the comparative analysis of the different models, and to guarantee a robust and efficient pipeline, we select the following methods: • For keyword recognition, we move forward with Microsoft Azure OCR for the OCR step, and LayoutLMv3 for document layout analysis step. • For signature and stamp detection, we move forward with RetinaNet. IV Conclusion To address the urgent need for efficient solutions to automate the validation of invoice documents, we propose a novel, fully annotated dataset composed of real invoices with various imperfections, along with a robust approach that accounts for different document layouts and real-world data variations. This automation not only streamlines the validation process but also significantly reduces human error and enhances operational efficiency. Our contribution is twofold: first, we focus on keyword detection using OCR and document layout analysis models; second, we identify stamps and signatures using object detection models. For each task, we benchmark various DL models to establish a strong foundation that balances efficiency and performance. Based on our evaluation results, we select an end-to-end system combining Microsoft Azure OCR with LayoutLMv3 for the keyword detection task, while RetinaNet proves to be the best choice for stamp and signature detection. Future work will involve further hyperparameter tuning and refinement of the DL models. Additionally, we plan to focus on handwritten invoices, which pose unique challenges due to their high variability in styles and legibility issues. Acknowledgments This work is the result of a collaboration between the IEEE Tunisia Section Volunteers Tools Coordination and the National Institute of Applied Sciences and Technology (INSAT). The authors would like to acknowledge all the members of the IEEE Tunisia Section Units Accounting Platform development team for their valuable efforts. References [1] L. D. Jaller, S. Gaillard, M. Molinuevo, and Others, ‘The regulation of Digital Trade’, World Bank Other Operational Studies, 2020. [2] M. G. Marne, P. R. Futane, S. B. Kolekar, A. D. Lakhadive and S. K. Marathe, ”Identification of Optimal Optical Character Recognition (OCR) Engine for Proposed System,” 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), Pune, India, 2018, pp. 1-4. [3] X.-F. Wang, Z.-H. He, K. Wang, Y.-F. Wang, L. Zou, and Z.-Z. Wu, ‘A survey of text detection and recognition algorithms based on deep learning technology’, Neurocomputing, vol. 556, p. 126702, 2023. [4] K. L. Masita, A. N. Hasan, and T. Shongwe, ‘Deep learning in object detection: A review’, in 2020 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD), 2020, pp. 1–11. [5] J. Wang, L. Jin, and K. Ding, ‘Lilt: A simple yet effective language-independent layout transformer for structured document understanding’, arXiv preprint arXiv:2202. 13669, 2022. [6] Y. Xu et al., ‘Layoutlmv2: Multi-modal pre-training for visually-rich document understanding’, arXiv preprint arXiv:2012. 14740, 2020. [7] Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, ‘Layoutlmv3: Pre-training for document ai with unified text and image masking’, in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4083–4091. [8] Z. Tian, X. Chu, X. Wang, X. Wei, and C. Shen, ‘Fully convolutional one-stage 3d object detection on lidar range images’, in Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 34899–34911. [9] S. Ren, K. He, R. Girshick and J. Sun, ”Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017, doi: 10.1109/TPAMI.2016.2577031. [10] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, ‘Focal Loss for Dense Object Detection’, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318–327, 2018. [11] W. Liu et al., ‘Ssd: Single shot multibox detector’, in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, 2016, pp. 21–37. [12] S. Park et al., ‘CORD: a consolidated receipt dataset for post-OCR parsing’, in Workshop on Document Intelligence at NeurIPS 2019, 2019. [13] Z. Huang et al., ”ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction,” 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, NSW, Australia, 2019, pp. 1516-1520. [14] G. Jaume, H. Kemal Ekenel and J. -P. Thiran, ”FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents,” 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), Sydney, NSW, Australia, 2019, pp. 1-6. [15] D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard, ‘Building a test collection for complex document information processing’, in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 665–666. [16] M. Limam, M. Dhiaf, and Y. Kessentini, ‘FATURA: A Multi-Layout Invoice Image Dataset for Document Analysis and Understanding’, arXiv preprint arXiv:2311. 11856, 2023. [17] T.-Y. Lin et al., ‘Microsoft coco: Common objects in context’, in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 2014, pp. 740–755. [18] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, ‘Res2Net: A New Multi-Scale Backbone Architecture’, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 2, pp. 652–662, 2021. [19] S. D. Chowdhury, U. Bhattacharya and S. K. Parui, ”Online Handwriting Recognition Using Levenshtein Distance Metric,” 2013 12th International Conference on Document Analysis and Recognition, Washington, DC, USA, 2013, pp. 79-83. [20] T.-Y. Lin et al., ‘Microsoft coco: Common objects in context’, in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 2014, pp. 740–755.

2. Related Work, Literature Survey & Algorithmic Foundations

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

Invoice Information Extraction: Methods and Performance Evaluation Sai Yashwant † † thanks: Use footnote for providing further information about author (webpage, alternative address)— not for acknowledging funding agencies. Affiliation: Technology Product Leader – Data Science and AI Affiliation: ManpowerGroup Services India Pvt. Ltd. Email: sai.yashwant@manpowergroup.com Anurag Dubey Affiliation: Executive Data Scientist Affiliation: ManpowerGroup Services India Pvt. Ltd. Email: anurag.dubey@manpowergroup.com Praneeth Paikray Affiliation: Senior Gen AI Specialist Affiliation: ManpowerGroup Services India Pvt. Ltd. Email: praneeth.paikray@manpowergroup.com Gantala Thulsiram Affiliation: Assistant Professor Affiliation: Indian Institute of Technology, Hyderabad Email: thulsiramg@mae.iith.ac.in Abstract This paper presents methods for extracting structured information from invoice documents and proposes a set of evaluation metrics (EM) to assess the accuracy of the extracted data against annotated ground truth. The approach involves pre-processing scanned or digital invoices, applying Docling and LlamaCloud Services to identify and extract key fields such as invoice number, date, total amount, and vendor details. To ensure the reliability of the extraction process, we establish a robust evaluation framework comprising field-level precision, consistency check failures, and exact match accuracy. The proposed metrics provide a standardized way to compare different extraction methods and highlight strengths and weaknesses in field-specific performance. A Preprint Keywords Evaluation Metrics, Key-Value Pair Extraction, Information Retrieval, Text Detection 1 Introduction Invoices are the key document for an organization that stores information on organizational transactions and serves as a proof of purchases and are necessary for accounting and tax purposes [ 1 ] . Invoices contain information such as purchase date, description of the product , quantity, unit price, and tax deductions. Managing and analyzing invoices is a costly affair for an organization which can be both credit and debit. Extracting and automating information from invoices can significantly increase the productivity of an organization. And hence there is a lot of demand to automate the invoice extraction process. The extracted information must be evaluated on defined metrics to maintain the accuracy of the extraction. With numerous research studies conducted, there are several techniques available for the extraction and evaluation of information extraction from invoices. Although there have been several approaches for the extraction of information from invoices such as graph-based extraction [ 2 ] [ 3 ] , grid-based approaches [ 4 ] , and approaches based on positional embeddings [ 5 ] , a concrete approach to evaluate the information extracted is still lacking. In recent years, invoice information extraction (IIE) has gained significant attention as part of the broader field of document information extraction, primarily due to the growing emphasis on business process automation and digital transformation [ 19 ] . Unlike structured documents such as forms, invoices often exhibit heterogeneous layouts depending on vendors, languages, fonts, and formatting practices [ 16 ] , making extraction tasks extremely challenging. Rule-driven approaches based on traditional optical character recognition (OCR) were heavily dependent on template-specific heuristics, which had limited generalization [ 17 ] . However, machine learning and deep learning methods have introduced more robust techniques for extracting invoice data using multimodal representations of textual, visual, and spatial information [ 11 , 18 ] . Despite advances, the challenge of the stent lies in maintaining consistent extraction accuracy and reliability in domains and invoice types. Many state-of-the-art models focus exclusively on improving extraction performance, yet less attention has been paid to establishing standardized evaluation strategies [ 20 ] . Given that extracted outputs directly impact downstream tasks such as accounting, fraud detection, and enterprise resource planning (ERP) integration, performance evaluation frameworks are crucial. Standard metrics such as precision, recall, and the F1 score are commonly employed but may not capture the business-critical precision of structured fields such as invoice numbers, tax IDs, or payment terms [ 21 ] . Thus, there is an increasing need for domain-specific evaluation methodologies that can assess both syntactic accuracy and semantic consistency in the extraction of invoice information. Furthermore, large-scale industry datasets for the extraction of invoices remain scarce due to privacy and confidentiality concerns [ 20 ] , which limits the benchmarking and reproducibility of research results. Consequently, many works rely on synthetic datasets or proprietary collections, which hinders fair comparison between methods. This research gap highlights the urgency for developing not only effective extraction algorithms, but also robust evaluation frameworks and benchmark datasets. Establishing the EM for invoice data extraction should have a clear ground truth for each targeted field (e.g. invoice number, date, supplier, totals, line-item fields) and measured field level from extraction process [ 6 ] . A robust EM should cover the accuracy, reliability, business risk, and operational performance from start to finish. Beyond aggregate scores, performance should be stratified by document conditions - digital vs scanned PDF, skew / noise, stamps / handwriting, language, currencies and unseen supplier templates - to measure robustness and generalization, especially for long-tail suppliers where layouts vary widely [ 7 ] . Business-centric key performance indicators (KPIs), such as the straight-through processing rate, the manual review rate, and the false auto-approval rate, are critical to reflect the risk of operation and automation effectiveness in real workflows [ 8 ] . Finally, calibration of confidence thresholds with precision recall trade-offs per field, plus continuous benchmarking on fixed canary sets, helps tune auto-approve vs. human review gates and monitor drift over time, ensuring consistent and reproducible comparisons between models and vendors [ 9 ] . Overall, while numerous extraction approaches continue to evolve, ranging from graph-based to transformer-based representations, the question of performance evaluation under realistic enterprise constraints remains underexplored. This paper seeks to address this research gap by systematically discussing extraction methods, identifying their strengths and weaknesses, and proposing evaluation strategies suitable for the extraction of invoice information. This paper is organized as follows. Section 2 discusses the literature survey. Section 3 discusses the methodologies involved in extracting invoice information. The results and analysis are given in Section 4, after which the paper concludes an outlook on further work in Section 5. 2 Literature Survey Automated invoice information extraction is a critical task in business process automation, aiming to convert unstructured or semi-structured invoice documents into structured data. This process is often called Intelligent Document Processing (IDP), significantly reducing manual efforts, minimizing errors, and accelerating financial workflows. The extraction methods have evolved from traditional template-based methods to sophisticated deep learning models. 2.1 Traditional and Rule Based Methods Early approaches to obtaining invoice information relied heavily on OCR combined with rule-based systems. After converting an invoice image to machine-readable text using an OCR engine like Tesseract, these systems would apply handcrafted rules or templates to locate and extract key information. These methods often used regular expressions (regex) to find specific patterns such as dates (e.g., dd-mm-yyyy) invoice numbers (e.g., INV-), or monetary values. For invoices with a consistent layout, template-based methods are popular. In this approach, zones or coordinates for specific fields (e.g., vendor name, total amount) are predefined for each invoice template. Although effective for a known set of invoice layouts, these systems are brittle and do not generalize to new or unseen invoice formats, which is a significant limitation given the wide diversity of invoice designs [ 10 ] . Maintenance overhead is also substantial, as a new template is required for each new invoice layout. 2.2 Machine Learning and Deep Learning Approaches Machine learning (ML) and deep learning (DL) approaches are widely used to extract invoice information because of the difficulties of traditional rule-based systems in handling the various and frequently unstructured formats of invoices. The limitations of rule-based systems allow for the adoption of ML and DL models, which can learn to extract information from diverse layouts without explicit rules. By immediately learning patterns from annotated data, ML and DL models can generalize across different formats, in contrast to rule-based techniques that require human development of templates and heuristics for every invoice style. To discover and categorize important invoice fields, machine learning techniques, such as support vector machines (SVMs) and conditional random fields (CRFs), usually rely on designed features. However, by automatically learning rich contextual and geographic representations of invoice content, deep learning techniques, especially those that use transformer-based architectures (e.g., BERT variations) and sequence models such as BiLSTM-CRF, have shown greater results. These models are capable of handling multi-language invoices, noisy OCR outputs, and complex document structures, making them more robust and scalable for real-world applications. As a result, the ML and DL approaches have become the foundation of modern invoice processing systems. 2.2.1 Text and Layout-Aware Models Modern approaches treat invoice extraction as a sequence labeling or question-answering task, incorporating not just the textual content but also the spatial layout of the document. A seminar work by [ 11 ] , LayoutLM revolutionized document artificial intelligence (AI) by pre-training a transformer-based model on large-scale document image datasets. It extends the BERT architecture by incorporating 2D positional embeddings (bounding-box coordinates) alongside text and visual embeddings. This allows the model to understand the spatial arrangement of words on a page, which is crucial to identifying fields such as headers, line items, and totals. Following LayoutLM, several variants have been proposed. DocFormer [ 12 ] introduced a multi-modal self-attention mechanism that learns correlations between text, visual features, and spatial locations more effectively. These models are typically fine-tuned for specific downstream tasks, such as obtaining information from invoices or receipts. 2.2.2 Graph-Based Methods Recognizing that documents possess an inherent graphical, another line of research focused on the application of graph neural network (GNN). In this paradigm, a document is represented as a graph G = (V, E), where the nodes V correspond to text segments (for example, words or text lines) and the edges E represent the spatial and logical relationships between them. Edges are typically constructed on the basis of heuristics such as proximity and vertical/horizontal alignment. A GNN is then applied to this graph structure to learn node representations that encode relational context. The final node embeddings are used for downstream tasks, such as node classification, where each text segment is assigned a semantic label (e.g., invoice date). GNNs are particularly effective at capturing complex, non-local dependencies and modeling tabular structures. The message-passing mechanism inherent to GNNs allows information to propagate across the document, allowing the model to associate entities that are spatially distant but semantically related. For many businesses, the most critical and challenging data are contained within the line items of an invoice. Graph-based methods are specifically engineered to excel at this. This technique models the invoice as a network of interconnected text elements (a graph). Words are nodes, and their spatial relationships (e.g., being adjacent, being in the same row/column) from the edges. A GNN then analyzes this network to understand complex relationships, such as matching the description of each item of the line to its quantity, unit price, and total [ 2 ] . The ROI of this method lies in its ability to enable deep process automation and financial control. By accurately extracting line item data, businesses can automate the three-way matching process (comparing the invoice against the purchase order and good receipt note). This is crucial to prevent overpayment, ensure contract compliance, and generate granular spending analytics that can be used for budget optimization and supplier negotiation. 2.2.3 Large Language Models in Document Information Extraction The advent of LLMs has marked a significant paradigm shift in the field of information extraction, moving away from the conventional supervised learning approach that requires extensive task-specific fine-tuning. LLMs leverage vast amounts of pre-trained knowledge to perform complex reasoning and extraction tasks with minimal or no task-specific examples. Traditional DL models, including the layout-aware architectures previously discussed, operate under a supervised learning paradigm. This necessitates the creation of large, domain-specific, and meticulously annotated datasets for fine-tuning, which is both time-consuming and cost-prohibitive. LLMs introduce the concept of in-context learning [ 13 ] . Instead of updating the model weights, the model is conditioned at inference time using a natural language prompt that describes the task. This approach allows for rapid adaptation to new tasks and document types without a dedicated training phase. The primary modalities for this approach in document extraction are text-based and multi-modal. • Text-based extraction via prompt engineering: The most direct application of LLMs involves processing the text output from an OCR engine. The methodology relies on carefully constructing a prompt that guides the model’s behavior. – Zero-shot learning: In this setting, the LLM is given a high-level task description without any concrete examples. The model relies entirely on its pre-trained knowledge to understand the task and extract the required information. – Few-shot learning: To improve performance and guide the model towards the desired output format, the prompt is augmented with a few examples (typically 1 to 5) of input documents and their corresponding structured outputs. This technique has been shown to significantly enhance extraction accuracy (Brown et al., 2020). Prompt engineering has emerged as a critical discipline in itself. Structure, wording, and examples within the prompt can have a substantial impact on the quality of the extraction. A typical prompt for invoice extraction would include the role, the task instructions, the desired output format (e.g., JSON schema), the few-shot examples (if any), and the OCR-extracted text of the target invoice. • Multimodal models for visual document understanding; a limitation of the text-based approach is its dependency on the quality of the upstream OCR engine. Errors or structural misinterpretations from the OCR phase are propagated to the LLM. Multimodal LLMs, also known as Vision-Language Models (VLMs), mitigate this by processing the document image directly. These models, such as GPT-4V [ 14 ] and open-source alternatives such as LLaVA [ 15 ] , integrate a vision encoder with a large language model. The vision encoder processes the raw image to extract visual features, which are then passed to the language model along with the textual prompt. This allows the model to ground the text in its visual context, using layout, font styles, logos, and table structures that are often lost in plain text. This approach is more closely aligned with the comprehension of human-like documents. 3 Methodology The study implemented in this paper employs a comparative experimental design to evaluate the performance of two distinct methodological paradigms of obtaining information from invoices: a dedicated document understanding model (Dockling) and a LlamaExtractor. The methodology adopted in the work tragets key value fields and line items typically present in business invoices, aligned with community benchmarks on key information localization and extraction (KILE) and line item recognition (LIR) from DocILE and related tasks. The following subsections detail the implementation specifics and evaluation metrics for each approach. 3.1 Dataset and Preprocessing Effective data set preparation is foundational for the performance and generalization of invoice information extraction systems. To address the scarcity of annotated resources and ensure model robustness, prior work emphasizes the importance of creating diverse, well-annotated invoice datasets. A curated data set of 102 digitally born and scanned invoice documents in PDF and JPEG format is utilized. Invoices vary in layout, language (e.g., English, German) and complexity. Each invoice is manually annotated to create a ground truth JSON schema containing key fields such as invoice_number, issue_date, due_date, seller_name, seller_address, total_amount, tax_amount, and line_items. Curation part also invoved the variability of the input image by rotating the images at different angles to creat few synthetic dataset [ 22 ] . Comparsion of ground truth with the results obtained from different methods listed helps us in setting up the evaluation metrics. This combined approach—diverse real data, synthetic augmentation, and robust preprocessing lays a strong foundation for a reliable evaluation of extraction methods across varied invoice instances. 3.2 Method 1: Docling-Based Extraction Docling is used to parse invoices from diverse formats (PDF, images, Office files) into a unified Docling document representation and export machine-readable JSON/Markdown suitable for downstream extraction. It’s an end-to-end document understanding model specifically designed to parse complex documents into structured JSON data. Its architecture bypasses traditional OCR and layout detection pipelines, treating the document as a unified modality. We utilized a pre-trained Docling-base model. Each document is parsed into blocks (text, tables, images) with reading order and structure preserved, then exported to JSON to standardize inputs for rule-based mapping. Table regions are identified and normalized to rows/columns to support line-item extraction downstream. Docling’s CV-first parsing reduces OCR error propagation in these steps. Key fields (e.g., vendor_name, dates, totals) are localized using layout-aware text blocks and table structures; extracted values are serialized with bounding boxes to satisfy the KILE/LIR evaluation The model’s strength lies in its ability to understand spatial relationships and textual semantics concurrently [ 12 ] . The model is prompted with a textual description of the desired output schema. The architectural diagram representing the overall extraction flow is shown in Fig. 1 Figure 1: Architecture Diagram of the Docling System 3.3 Method 2: Llama Extractor The LlamaExtractor uses schema-guided, LLM-based structured extraction to return deterministic JSON that conforms to a declared schema for invoice fields and line items. To enforce structure, the extractor relies on structured output capabilities, such as JSON Schema-constrained decoding and function/tool signatures, that guarantee type-safe outputs and minimize parsing failures. Implementations are performed with LlamaIndex structured extraction utilities or managed extraction services (e.g., LlamaExtract) that take a Pydantic/JSON schema and emit well-typed JSON from unstructured or semi-structured inputs. Schema-constrained output is achieved by attaching the schema to the LLM (e.g., through LlamaIndex structured outputs or the Llama API’s structured output interface), which steers decoding to produce only valid fields, types, and nesting. A predefined ’invoice’ schema is commonly available in Llama-driven document agents and can be used as is or extended to custom fields without additional fine-tuning. Prompts describe the task, the schema, and any field-level instructions (e.g., normalization hints, required formats, and units), optionally augmented by few-shot exemplars that illustrate desired JSON outputs for invoices with and without line items. The LLM receives the parsed invoice text as context and is instructed to extract only the requested fields and to leave any missing fields null or empty as defined by the schema. The extractor can produce a confidence score and brief rationale per field as part of the structured output schema to facilitate triage, auditing, and human-in-the-loop workflows. The architectural diagram representing the general extraction flow is shown in Fig. 2 . The workflow of extracting information is as follows - • Invoices from different countries with different languages enter the system. • A custom router in place routes the invoice to different parsing/extraction strategies depending on the complexity of the invoice. • After routing the invoice to a particular channel, parsing prcocess of the invoice start where it identifies the documents key regions (headers, tables, footers, line items) and prepares the structured context input to for extraction. • After parsing, the system detects the invoice language and triggers translation workflows if the content is not in English. • Once the translation of the invoice text into English is completed, the structured data extraction process is initiated. This process relies on a predefined schema which outlines the expected fields such as invoice number, date, vendor details, and line items. Figure 2: Architecture Diagram of LlamaExtractor 3.4 Evaluation Metrics Performance is evaluated at field and document levels, with metrics aligned with DocILE tasks: key information localization / extraction and line item recognition, including zero / few shot analyzes [ 25 ] . • Field-level metrics: These metrics assess the accuracy of identifying and extracting structured header fields such as invoice numbers, dates, buyer/seller details, and totals. Evaluation includes: – Exact match and relaxed match scoring to account for strict correctness versus partial or fuzzy matches (e.g., text normalization or OCR-induced variants). – Tolerance windows for numeric fields, such as amounts and dates, allow for acceptable deviations due to formatting or rounding. – F1 scores for binary classification of field presence and correctness, enabling nuanced evaluation of whether fields are correctly detected, even if their values are partially incorrect. • Line item metrics: Given the complexity of tabular data in financial documents, line-item performance is evaluated through: – Row-wise assignment accuracy, which measures how well the model groups and extracts values into coherent line items. – Tolerance-based numerical checks for quantities and monetary values, taking into account common OCR or format-related errors. – Table-level completeness, which evaluates the extent to which the model captures all relevant line-items without omissions or duplication. • Robustness: Beyond static accuracy metrics, robustness is a critical consideration for real-world deployments. Evaluations include: – Cross-vendor performance, measuring consistency when documents originate from multiple sources or layouts. – Template generalization in zero-shot (unseen layouts) and few-shot (limited training examples) settings, to test adaptability across diverse document types and vendors. 4 Results and Analysis This section presents a comparative evaluation of two distinct methods applied to the task of obtaining invoice information: Docling and LlamaExtractor. The performance of each approach is assessed based on accuracy, efficiency, and robustness in extracting key invoice components such as invoice number, date, vendor details, and total amount. The results have been anlyzed on two different parameters listed below - • Accuracy distribution measures the percentage of invoice fields correctly extracted by each method, highlighting strengths and weaknesses across different data points. • Consistency Check Failures count instances where extracted data violates predefined business rules, such as mismatched gross and net amounts, indicating extraction errors. The extraction and evaluation schema consists of several key entities, each representing different aspects of the invoice data. The bill-to-entity includes fields related to the customer or business unit responsible for payment. The supplier entity captures vendor-specific information such as tax IDs and bank details. The invoice entity encompasses general transactional details, including invoice numbers, dates, and payment terms. Detailed transaction lines are covered under the Line Item entity, which records individual services or products with pricing and tax information. Finally, the tax line summary entity summarizes the tax-related fields in the invoice to ensure compliance and accurate reporting. 4.1 Result - Docling The documenting methodology is evaluated on a benchmark invoice data set, focusing on field-level accuracy and document-level consistency along with consistency checks. • The methodology achieved an overall accuracy of 63%, demonstrating moderate performance in recognizing and extracting key information from the invoice. • Accuracy varied significantly across entities, with entities like invoice_name, invoice_number showing the highest extraction accuracy at 90%, benefiting from the Docling’s layout understanding and parsing approach. • The tax_amount, net_amount, currency_format entities exhibited lower accuracy at 60%, primarily due to limitations in handling unstructured text fields, complex table structures, multi-line descriptions. • The Line Item entity, being the most granular and diverse in content, achieved an accuracy of 58%. This performance reflects the difficulty of documenting with complex nested table layouts. Beyond individual field accuracy, consistency checks were performed to ensure that the extracted data adhered to the logical and business rules encoded in the schema. For example, net amount + tax amount + roundoff amount = gross amount in the invoice entity. The Docling methodology achieved a 80% pass rate on these consistency checks, with 20% of extractions failing due to incorrect numerical field extraction leading to calculation mismatches/misalignment between line item totals and invoice summaries/errors in tax calculation fields. Further analysis revealed that most consistency failures were concentrated in invoices with complex multi-tax scenarios and failures predominantly occurred in invoices with non-standard formatting. 4.2 Result - LlamaExtractor The LlamaExtractor is evaluated on a curated dataset annotated with the predefined schema covering five key entities: bill-to, supplier, invoice, line item, and tax line summary. The overall accuracy of the LlamaExtractor is calculated as the proportion of correctly extracted fields relative to the total annotated fields throughout the dataset. • The extractor achieved an accuracy of 94%, demonstrating strong performance in recognizing and extracting key invoice information. • Accuracy varied between entities, with the invoice and bill to entities showing the highest extraction precision due to their relatively standardized field formats. • The line item entity, being the most granular and diverse in content, exhibited a slightly lower accuracy of 91%, reflecting challenges in handling variable descriptions and numeric values on different invoice layouts. Beyond individual field accuracy, consistency checks were performed to ensure that the extracted data adhered to the logical and business rules encoded in the schema. For example, net amount + tax amount + round-off amount = gross amount in the invoice entity. The LlamaExtractor passed 93% of these general consistency checks. 4.3 Comparative Analysis The comparative evaluation between the Docling and the LlamaExtractor method highlights distinct trade-offs in accuracy, efficiency, technical architecture, and practical usability for automated invoice data extraction. 4.3.1 Overall Performance The LlamaExtractor shows superior accuracy, achieving an overall extraction accuracy of 94%, compared to 63% for the Docling method. This significant performance gap underscores the stronger ability of the LlamaParser model to interpret and structure invoice content in various formats. In terms of processing efficiency, the Docling method exhibits faster average processing times (10 seconds per document) relative to the LlamaExtractor (30 seconds per document). This suggests that Docling prioritizes lightweight processing, likely due to its simpler OCR and layout-based approach, while the LlamaExtractor’s deep language modeling introduces higher computational overhead but delivers better semantic understanding. Table 1 shows the overall comparitive performance between LlamaExtractor and Docling method. Table 1: Overall Performance and Consistency Validation Comparison Metric Llama Extractor Docling Method Overall Performance Overall Accuracy (%) 94 63 Processing Time (avg/doc) 30 sec 10 sec Consistency Validation Consistency Check Pass Rate (%) 93 80 Mathematical Validation Errors 0 20 4.3.2 Consistency Validation The LlamaExtractor achieves a consistency check pass rate 93%, indicating highly stable and reliable output, even across invoices with varying layouts or linguistic nuances. In contrast, the Docling method records an 80% pass rate, revealing occasional inconsistencies, especially in field alignment and numerical validation. Furthermore, the LlamaExtractor recorded zero mathematical validation errors, while the documentation exhibited 20 errors, suggesting that the internal reasoning capabilities of the Llama enhance the logical verification of the extracted numeric data, such as totals, taxes and line item calculations. Table 1 shows the comparitive consistency validation between LlamaExtractor and Docling method. 4.3.3 Technical Characteristics Both methods support similar input formats (raw text, PDF, JPG), enabling flexible document ingestion. However, their underlying architectures differ significantly. The LlamaExtractor operates as an LLM, using contextual understanding and reasoning to interpret complex invoice structures. This allows it to be generalized across unseen templates without explicit layout training. In contrast, the documenting method depends on layout analysis and OCR pipelines, making it more deterministic, but less adaptable to diverse invoice templates. Consequently, Llama requires only minimal preprocessing, while Docling necessitates explicit document structuring and OCR preprocessing to function optimally. From a computational point of view, the LlamaExtractor requires medium resources, while the low resource consumption of Docling makes it more suitable for small- or large-scale deployments with limited computational infrastructure. Table 2 shows the comparison between technical characteristics of LlamaExtractor and Docling method. Table 2: Technical Characteristics of Invoice Extraction Methods Metric Llama Extractor Docling Method Input Format Raw Text/PDF/JPG Raw Text/PDF/JPG Model Architecture Large Language Model Layout Analysis + OCR Pre-processing Required Minimal Required Computational Resources Medium Low 4.3.4 Error Analysis The error patterns further reveal the fundamental differences in the two systems. The primary sources of error of the LlamaExtractor stem from variable field descriptions and numeric formatting variations, such as unconventional representations of dates or currency. On the other hand, docing errors arise predominantly from mathematical validation issues and incomplete line item extractions, which can be attributed to OCR misreads or inconsistent mapping of the bounding box. In terms of structural robustness, the LlamaExtractor handles complex layouts more effectively (rated “Good”) compared to Docling (“Fair”). Additionally, Llama supports multi-page document parsing, enabling end-to-end extraction from lengthy invoices, while Docling’s support is limited, reducing its applicability for multi-page commercial invoices. Table 3 shows the comparitive error analysis of LlamaExtractor and Docling method. Table 3: Error Analysis of Llama Extractor and Docling Method Metric Llama Extractor Docling Method Primary Error Sources Variable descriptions, numeric formatting Mathematical validation, line item description Complex Layout Handling Good Fair Multi-page Document Support Yes Limited 4.3.5 Practical Considerations From an operational standpoint, the LlamaExtractor exhibits medium setup complexity, primarily due to model integration and configuration requirements. The Docling method, being a rule-driven OCR pipeline, offers low setup complexity and can be more easily deployed in traditional environments. Regarding scalability, the LlamaExtractor demonstrates excellent scalability, benefiting from its ability to generalize without layout retraining. Docling, though functional, scales only fairly, as its OCR-based pipeline may require custom adjustments for new formats. In terms of cost efficiency, Docling performs better (High) because of its lower resource demands and simpler architecture, whereas the LlamaExtractor incurs medium operational costs due to higher inference time and model complexity. Both systems, however, are capable of real-time processing, making them viable for near-instantaneous invoice validation and automation workflows. Table 4 shows the practical considerations for LlamaExtractor and Docling method. Table 4: Practical Considerations for Deployment and Scalability Metric Llama Extractor Docling Method Setup Complexity Medium Low Scalability Excellent Fair Cost Efficiency Medium High Real-time Processing Yes Yes 4.3.6 Summary In summary, the LlamaExtractor provides a more intelligent, semantically aware, and accurate solution for invoice data extraction, particularly suited for enterprises requiring high reliability and scalability across varied invoice templates. The Docling method, while less accurate, offers a lightweight, faster, and more cost-efficient alternative, suitable for applications where speed and simplicity outweigh the need for high precision. Thus, the choice between the two depends on the specific operational priorities: accuracy and adaptability (Llama) versus cost-effectiveness (Docling). 5 Conclusion and Future Discussion This study presented a comprehensive comparative analysis between two different invoice extraction methodologies: the LlamaExtractor, an LLM-based system, and the Docling method, an OCR-based approach and layout-driven approach. Through detailed evaluation across performance, consistency, technical characteristics, and practical dimensions, the analysis highlights that the LlamaExtractor consistently outperforms traditional document parsing frameworks in terms of accuracy, semantic understanding, and robustness across diverse invoice structures. The results show that while the LlamaExtractor achieves a significantly higher overall accuracy of 94%, it does so with moderately higher computational requirements and processing time. Its reasoning-driven architecture ensures near-perfect consistency validation and eliminates most numerical or structural errors, making it particularly suitable for enterprise-grade invoice automation systems where precision and reliability are critical. In contrast, the Docling Method, though less accurate, offers faster execution and lower computational cost, making it better suited for lightweight or cost-sensitive deployments. In general, this comparative assessment underscores the trade-off between semantic accuracy and computational efficiency. It establishes that modern LLM-based extractors, when properly optimized, can redefine the state of automated document understanding, bridging the gap between human-like reasoning and structured data extraction from unstructured financial documents. Future research can extend this work in several promising directions. First, integrating a hybrid extraction pipeline that combines the semantic reasoning of LLMs with the structured layout awareness of OCR systems could yield a more balanced and efficient approach. Such hybrid models may significantly reduce latency while maintaining high accuracy. Second, exploring domain-adaptive fine-tuning of language models in invoice-specific corpora could enhance contextual accuracy, particularly for vendor-specific templates, regional tax formats, and multilingual invoices. Incorporating self-correction mechanisms or confidence-based validation layers could further improve reliability during large-scale deployments. Another potential avenue involves the development of a benchmark data set for the evaluation of invoice extraction that includes various layouts, languages, and industries. This would enable standardized comparison across methods and foster reproducibility in research. Lastly, expanding this investigation to include other financial documents such as purchase orders, receipts, and contracts could provide a broader understanding of how LLM-based and OCR-based systems generalize across related domains. Such explorations would be instrumental in evolving towards fully autonomous document understanding systems within enterprise ecosystems. In summary, while LLM-based extraction methods, such as the LlamaExtractor, have shown clear advantages, future work should focus on optimizing their performance, scalability, and cost-efficiency to ensure widespread adoption in real-world production-grade invoice processing pipelines. References [1] T. Saout, F. Lardeux, and F. Saubion, “An overview of data extraction from invoices,” IEEE Access , vol. 12, pp. 19872–19886, 2024, doi:10.1109/ACCESS.2024.3360528. [2] X. Liu, F. Gao, Q. Zhang, and H. Zhao, “Graph convolution for multimodal information extraction from visually rich documents,” arXiv preprint arXiv:1903.11279 , 2019. [3] F. Krieger, P. Drews, B. Funk, and T. Wobbe, “Information extraction from invoices: a graph neural network approach for datasets with high layout variety,” in International Conference on Wirtschaftsinformatik , 2021, pp. 5–20. [4] T. I. Denk and C. Reisswig, “Bertgrid: Contextualized embedding for 2d document representation and understanding,” arXiv preprint arXiv:1909.04948 , 2019. [5] B. P. Majumder, N. Potti, S. Tata, J. B. Wendt, Q. Zhao, and M. Najork, “Representation learning for information extraction from form-like documents,” in Proc. of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 6495–6504. [6] J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. Rosen, G. Ceder, K. Persson, and A. Jain, “Structured information extraction from scientific text with large language models,” Nature Communications , vol. 15, no. 1, p. 1418, 2024. [7] F. Krieger, P. Drews, and B. Funk, “Automated invoice processing: Machine learning-based information extraction for long tail suppliers,” Intelligent Systems with Applications , vol. 20, p. 200285, 2023. [8] M. Bardvall and I. Hassle, “Automating invoice recognition: A comparative study of large language models and ocr/ml technologies,” 2024. [9] V. Jijkoun, J. Mur, and M. de Rijke, “Information extraction for question answering: Improving recall through syntactic patterns,” in COLING 2004: Proc. of the 20th Int. Conf. on Computational Linguistics , 2004, pp. 1284–1290. [10] A. R. Katti, C. Reisswig, C. Guder, S. Brarda, S. Bickel, J. Höhne, and J. Faddoul, “Chargrid: Towards understanding 2d documents,” arXiv preprint arXiv:1809.08799 , 2018. [11] Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, “Layoutlm: Pre-training of text and layout for document image understanding,” in Proc. of the 26th ACM SIGKDD Int. Conf. on Knowledge Discovery & Data Mining , 2020, pp. 1192–1200. [12] S. Appalaraju, B. Jasani, B. U. Kota, Y. Xie, and R. Manmatha, “Docformer: End-to-end transformer for document understanding,” in Proc. IEEE/CVF Int. Conf. on Computer Vision , 2021, pp. 993–1003. [13] T. Brown et al. , “Language models are few-shot learners,” Advances in Neural Information Processing Systems , vol. 33, pp. 1877–1901, 2020. [14] OpenAI, GPT, “4V (ision) System Card,” 2023. [15] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in Neural Information Processing Systems , vol. 36, pp. 34892–34916, 2023. [16] Ł. Borchmann, M. Pietruszka, T. Stanislawek, D. Jurkiewicz, M. Turski, K. Szyndler, and F. Graliński, “Due: End-to-end document understanding benchmark,” in 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021. [17] I. Gallo, L. Noce, A. Zamberletti, and A. Calefati, “Deep neural networks for page stream segmentation and classification,” in 2016 Int. Conf. on Digital Image Computing: Techniques and Applications (DICTA) , 2016, pp. 1–7. [18] Y. Xu, T. Lv, L. Cui, G. Wang, Y. Lu, D. Florencio, C. Zhang, and F. Wei, “Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding,” arXiv preprint arXiv:2104.08836 , 2021. [19] J. Chen, T. Lv, L. Cui, C. Zhang, and F. Wei, “XDoc: Unified pre-training for cross-format document understanding,” arXiv preprint arXiv:2210.02849 , 2022. [20] T. Stanislawek, F. Graliński, A. Wroblewska, D. Lipinski, A. Kaliska, P. Rosalska, B. Topolski, and P. Biecek, “Kleister: key information extraction datasets involving long documents with complex layouts,” in Int. Conf. on Document Analysis and Recognition , 2021, pp. 564–579. [21] C. Trivedi and U. P. Rao, “Secrecy aware key management scheme for Internet of Healthcare Things,” The Journal of Supercomputing , vol. 79, no. 11, pp. 12492–12522, 2023. [22] A. P. V. Ananthakrishnan, R. KR, B. Wilson, et al. , “Generating synthetic invoices via layout-preserving content replacement,” arXiv preprint arXiv:2508.03754 , 2025. [23] Š. Šimsa, M. Šulc, M. Uřičář, Y. Patel, A. Hamdi, M. Kocián, M. Skalický, J. Matas, A. Doucet, M. Coustaty, et al. , “Docile benchmark for document information localization and extraction,” in Int. Conf. on Document Analysis and Recognition , 2023, pp. 147–166. [24] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in Int. Conf. on Learning Representations (ICLR) , 2023. [25] Š. Šimsa, M. Uřičář, M. Šulc, Y. Patel, A. Hamdi, M. Kocián, M. Skalický, J. Matas, A. Doucet, M. Coustaty, et al. , “Overview of DocILE 2023: document information localization and extraction,” in Int. Conf. of the Cross-Language Evaluation Forum for European Languages , 2023, pp. 276–293.

4. Neural Network Backbone, Layer Specifications & Loss Formulations

5. Training Protocol, Convergence Behavior & Experimental Evaluation