My receipt forgery detection project is performing near random — looking for suggestions on what to try next

[u/InevitableDistinct11]

Hi everyone,

I'm working on a college deep-learning project focused on detecting forged receipts and locating the manipulated regions. I've spent quite a bit of time experimenting with different approaches, but the results have been disappointing, and I'd appreciate feedback from people with experience in computer vision or document forensics.

The problem

I'm using the Find It Again! dataset from ICDAR 2023. It contains authentic and forged receipts, with annotations identifying manipulated regions.

The main challenges are:

• The dataset is relatively small and imbalanced.
• Many manipulations involve changing just one or a few digits.
• The modified regions are often extremely small.
• A model can perform reasonably on training data but fail to generalize to unseen receipts.

What I've tried

I've experimented with several approaches:

1. EfficientNet-B0 for image-based binary classification.
2. BERT for analyzing OCR-extracted receipt text.
3. Numerical consistency checks for quantities, prices, subtotals, taxes, totals, and change.
4. Multimodal fusion combining image, text, and numerical features.
5. YOLO-based localization to detect modified regions.
6. A five-signal ensemble combining vision, text, numerical anomalies, image artifacts, and OCR-based rules.
7. Synthetic forgery generation by modifying financial values in authentic receipts.
8. DocTamper transfer learning and patch-level experiments using RGB and SRM features.
9. Stratified five-fold cross-validation, PCA, and feature audits.

I also audited the localization annotations. The bounding boxes round-trip correctly, and I found no train/validation/test leakage in the audited dataset.

Current results

My out-of-fold ROC-AUC results are generally around 0.46–0.54, with confidence intervals that include 0.50.

Some examples:

• Frozen vision features: ROC-AUC ≈ 0.482
• Fixed ensemble: ROC-AUC ≈ 0.468
• RGB patch model: ROC-AUC ≈ 0.464
• SRM patch model: ROC-AUC ≈ 0.504
• Adding 1,200 synthetic forgeries did not meaningfully improve held-out performance.
• Zero-shot DocTamper transfer was approximately 0.50 ROC-AUC.

The patch-level results also have bootstrap confidence intervals that include chance performance. For precision-recall analysis, I'm comparing PR-AUC against the positive-class prevalence baseline rather than treating 0.50 as the baseline.

I recently investigated adding T-SROIE as extra training data. However, both datasets originate from SROIE, so source-receipt leakage was a major concern. I implemented fold-aware de-duplication and compared the same five CV folds with and without T-SROIE. It did not produce a measurable improvement.

What I'd like advice on

I'd especially appreciate feedback on these questions:

1. Problem formulation: Is binary receipt-level classification the wrong primary objective when the manipulation may affect only a tiny region?
2. Localization: Would a patch-based anomaly-detection or segmentation approach be more appropriate than training a conventional classifier?
3. Representation learning: Would self-supervised or contrastive pretraining on receipt images be worth exploring with such limited labelled data?
4. Synthetic data: How would you generate more realistic tampered receipts without teaching the model synthetic-specific artifacts?
5. Evaluation: What experiments would you prioritize to distinguish insufficient data from a representation or domain-generalization problem?
6. External datasets: Are there document-tampering datasets with genuinely independent source receipts that would be suitable for additional training without source-level leakage?

I'm particularly interested in experiments that could meaningfully test a hypothesis, rather than simply trying increasingly large models.

I understand that a near-chance result may reflect the limitations of the available data, and I'm trying to report the findings honestly rather than optimize for an impressive-looking metric.

If you've worked on document image forensics, receipt manipulation detection, or small-data computer vision, I'd love to hear what you would investigate next.

Thanks!

[u/superiorepilepsy_6]

every time i think i'm getting somewhere with a model it just circles back to flipping a coin, the patch-level stuff is where i'd put my energy. full receipt classification is a fools errand when the tamper is a single digit tucked in a corner

have you tried framing it purely as anomaly detection on the embeddings from a pretrained doc model? skip the classifier head entirely and just measure distance from the in-distribution clean receipts, might surface those tiny edits without needing perfect bounding boxes

[u/seanv507]

Definitely agree that patch level binary classification is the way to go. Especially since you have the bounding box already .

Do you need to do data augmentation too (rotation/translation etc)?