HFCMA:Hierarchical Fine-grained Cross-modal Alignment via Dynamic Granularity Discovery and Optimal Transport

Implementation of Hierarchical Fine-grained Cross-modal Alignment via Dynamic Granularity Discovery and Optimal Transport.

HFCMA is a fine-grained image-text retrieval framework built on a CLIP-style dual encoder. It progressively organizes low-level patch/token representations into adaptive visual regions and textual phrases, then performs structured region-phrase matching with entropy-regularized optimal transport.

Abstract

Fine-grained image-text retrieval requires a model not only to assess global semantic consistency between an image and a text, but also to model visual evidence for local textual semantics, including objects, attributes, quantities, actions, and relations, so as to distinguish semantically similar candidates. Existing methods commonly use fixed image patches, detected regions, or text tokens as the basic matching units and then aggregate local interactions through max pooling, cross attention, or similarity accumulation. Such strategies remain limited in semantic unit construction and global matching consistency. This paper proposes a hierarchical fine-grained cross-modal alignment framework, termed HFCMA. The framework progressively transforms low-level patch and token representations into adaptive visual regions and textual phrases, and then performs structured region-phrase matching within a global retrieval scoring process. Specifically, HFCMA first adopts dynamic granularity discovery (DGD) to aggregate raw patches and tokens into semantically compact visual regions and textual phrases. It then formulates region-phrase matching as an entropy-regularized optimal transport (OT) problem, thereby estimating a structured alignment matrix under global marginal constraints. To reduce background noise and local semantic drift, HFCMA further introduces background debiasing, textual importance weighting, and prototype consistency regularization. Experiments on Flickr30K and MS-COCO show that HFCMA effectively combines dynamic semantic unit construction with OT-based structured matching under a dual-encoder retrieval framework. The resulting fine-grained score provides structured local alignment information for candidate ranking and improves the discrimination of subtle semantic differences.

Repository Structure

hfcma-public/
|-- configs/                 # dataset and hyperparameter examples
|-- docs/                    # data format and reproducibility notes
|-- scripts/                 # training and evaluation commands
|-- src/hfcma/
|   |-- datasets/            # JSON retrieval dataset loader
|   |-- evaluation/          # retrieval metrics
|   |-- models/              # HFCMA modules and OpenCLIP wrapper
|   |-- losses.py            # retrieval and auxiliary training losses
|   |-- rerank.py            # top-K fine-grained reranking utilities
|   |-- train.py             # training and validation entry point
|   `-- evaluate.py          # evaluation-only entry point
|-- tests/                   # lightweight correctness checks
|-- tools/                   # annotation conversion utilities
|-- requirements.txt
|-- pyproject.toml
`-- README.md

Installation

conda create -n hfcma python=3.10 -y
conda activate hfcma
pip install -r requirements.txt
pip install -e .

The default configuration uses OpenAI CLIP ViT-L/14 through OpenCLIP. A CUDA GPU is recommended for training and reranking.

Data Format

Prepare each dataset as a JSON list. Each entry corresponds to one image and its captions:

{
  "image": "train2014/COCO_train2014_000000000009.jpg",
  "captions": [
    "A group of people standing near a bus.",
    "People are waiting beside a bus."
  ],
  "image_id": 9,
  "split": "train"
}

Recommended local layout:

data/
|-- flickr30k/
|   |-- images/
|   `-- cache/flickr30k_all.json
`-- coco/
    |-- images/
    `-- cache/coco_all.json

See docs/DATA.md for supported fields and evaluation protocol notes.

If your annotations follow the Karpathy-style JSON format, convert them with:

python tools/convert_retrieval_json.py \
  --input path/to/dataset_karpathy.json \
  --output data/flickr30k/cache/flickr30k_all.json

Training

Flickr30K:

bash scripts/train_flickr30k.sh

MS-COCO:

bash scripts/train_coco.sh

The scripts train the HFCMA head with the CLIP backbone frozen.

Evaluation

Flickr30K:

bash scripts/eval_flickr30k.sh

MS-COCO:

bash scripts/eval_coco.sh

Evaluation computes global CLIP similarities and reranks top-ranked candidates with the HFCMA fine-grained OT score.

Sanity Checks

Run the lightweight tests before launching a full experiment:

pytest tests

These tests verify retrieval-metric computation and the marginal constraints of the Sinkhorn transport plan.

Main Configuration

Dataset	Visual/text slots	Prototypes	Sinkhorn steps	OT epsilon	Rerank top-K
Flickr30K	8	32	10	0.10	128
MS-COCO	12	48	15	0.07	128

Expected Results

The manuscript reports the following main RSUM values. Minor variation is expected across hardware, CUDA version, annotation conversion, and random seed.

Dataset/protocol	Full HFCMA	HFCMA-Fusion-Calib
Flickr30K-1K	535.78	542.92
MS-COCO 1K	522.95	532.62
MS-COCO 5K	439.59	442.27

Reproducibility

See docs/REPRODUCIBILITY.md for settings that should be reported when reproducing the experiments, including backbone, slot number, OT settings, candidate pool size, random seed, and dataset split.

Citation

@article{hfcma2026,
  title   = {Hierarchical Fine-grained Cross-modal Alignment via Dynamic Granularity Discovery and Optimal Transport},
  author  = {Pengfeng Song},
  journal = {To appear},
  year    = {2026}
}

License

The license will be updated before the final release.