DeBERTa-v3-large-clause-metaphor

This model is a fine-tuned version of Microsoft DeBERTa-v3-large for binary indirect metaphor detection in English at token level, trained on the VU Amsterdam Metaphor Corpus (VUAMC). Training uses clause-level segmentation (SpaCy ROOT + advcl/ccomp/relcl subtrees) and a two-class scheme (non-metaphor / metaphor), following the MIPVU annotation guidelines (Steen et al. 2010). Only indirect metaphors (metaphor_type == "met") are labeled positive.

Model description

• Base model: microsoft/deberta-v3-large
• Task: Token classification (binary metaphor detection)
• Training unit: Clauses (SpaCy ROOT + advcl/ccomp/relcl), with word-level labels
• Label scheme: binary — 0 = non_metaphor, 1 = metaphor

Training & evaluation data

• Dataset: VUAMC (VU Amsterdam Metaphor Corpus), covering four registers: News, Academic, Fiction, Conversation.
• Split: A 10-document preliminary held-out set drawn from the VUAMC corpus (distinct from the NAACL FLP 2018 official 27-document split). A random 10% of training sentences is held out as dev set.
• Test set: 10 documents, 2,588 sentences, 29,241 tokens, 2,509 gold metaphor tokens (8.6%).

Test set document list

Fragment	Genre	Tokens	Gold metaphors
a1e-fragment01	News	671	86
a3k-fragment11	News	1,413	186
a7s-fragment03	News	983	121
amm-fragment02	Academic	4,313	491
b17-fragment02	Academic	1,806	256
ccw-fragment03	Fiction	2,443	181
faj-fragment17	Fiction	5,049	290
fpb-fragment01	Fiction	5,012	447
kbd-fragment21	Conversation	3,299	208
kcu-fragment02	Conversation	4,252	243

Training hyperparameters

Parameter	Value
Loss	Cross-Entropy
Epochs	4
Learning rate	2e-5
Batch size	2
Gradient accumulation steps	8 (effective batch 16)
Max sequence length	192
Optimizer / scheduler	AdamW, linear decay

Results

Evaluated on the 10-document held-out test set (29,241 tokens, 2,509 gold metaphor tokens):

Metric	Value
F1	77.03
Precision	79.12
Recall	75.05
Accuracy	96.16

By genre

Genre follows the BNC-Baby document ID prefix mapping (News: a*; Academic: amm, b17; Fiction: ccw, faj, fpb; Conversation: k*).

Genre	Docs	Tokens	Metaphors	Precision	Recall	F1	Accuracy
News	3	3,067	393	84.32	90.33	87.22	96.61
Fiction	3	12,504	918	80.11	80.28	80.20	97.09
Academic	2	6,119	747	82.45	66.67	73.72	94.20
Conversation	2	7,551	451	67.36	64.97	66.14	96.03

Key observations:

• News (F1 = 87.22%) achieves the highest performance in this test set, driven by exceptionally high recall (90.33%). Journalistic metaphors in the three sampled fragments are lexically conventional and well-covered by the training distribution.
• Fiction (F1 = 80.20%) shows balanced precision and recall (80.11% / 80.28%), reflecting the model's reliable handling of narrative-register metaphors across diverse syntactic structures.
• Academic (F1 = 73.72%) suffers from low recall (66.67%) despite strong precision (82.45%). Academic texts contain a high density of domain-specific metaphors (amm: 491 gold positives in 4,313 tokens = 11.4%), and the model tends to under-predict in technically dense passages.
• Conversation (F1 = 66.14%) is the hardest register, consistent with findings across metaphor detection systems. Spoken language metaphors are fragmented, elliptical, and context-dependent, making surface-form detection difficult.

By document

Fragment	Genre	Tokens	Metaphors	Precision	Recall	F1
a7s-fragment03	News	983	121	87.60	87.60	87.60
a3k-fragment11	News	1,413	186	85.71	90.32	87.96
b17-fragment02	Academic	1,806	256	84.82	85.16	84.99
fpb-fragment01	Fiction	5,012	447	83.22	82.10	82.66
faj-fragment17	Fiction	5,049	290	79.10	84.83	81.86
a1e-fragment01	News	671	86	77.88	94.19	85.26
ccw-fragment03	Fiction	2,443	181	73.81	68.51	71.06
amm-fragment02	Academic	4,313	491	80.69	57.03	66.83
kbd-fragment21	Conversation	3,299	208	70.97	63.46	67.01
kcu-fragment02	Conversation	4,252	243	64.66	66.26	65.45

By part of speech (Universal POS)

POS tags with zero gold metaphors in the test set are omitted.

POS	Full name	Tokens	Metaphors	Precision	Recall	F1
DET	Determiner	2,145	129	94.62	95.35	94.98
ADP	Preposition	2,308	735	89.91	86.12	87.98
PRON	Pronoun	3,279	160	89.71	76.25	82.43
VERB	Verb	3,149	712	76.28	73.17	74.70
ADJ	Adjective	1,712	221	77.84	68.33	72.77
SCONJ	Subord. Conjunction	583	37	72.97	72.97	72.97
ADV	Adverb	1,228	90	72.94	68.89	70.86
NOUN	Noun	4,216	415	77.42	57.83	66.21
AUX	Auxiliary	2,039	4	66.67	50.00	57.14
PART	Particle	859	2	33.33	50.00	40.00
PROPN	Proper noun	1,081	4	50.00	25.00	33.33

Key observations:

• DET (Determiners, F1 = 94.98%) achieves the highest F1. Determiner metaphors (e.g., article-level MRW) are rare but highly lexically predictable; precision and recall are nearly identical (94.62% / 95.35%).
• ADP (Prepositions, F1 = 87.98%) is the largest positive class (735 gold tokens, 29.3% of all metaphors). Prepositional metaphors (e.g., under pressure, over time) are highly conventionalized and well-represented in training data. High precision (89.91%) and recall (86.12%) demonstrate the model's strength here.
• PRON (Pronouns, F1 = 82.43%) shows strong precision (89.71%) but moderate recall (76.25%), suggesting the model is conservative about predicting pronouns as metaphorical while being accurate when it does.
• VERB (Verbs, F1 = 74.70%) is the second-largest positive class (712 gold tokens). Verbs are central to predicative metaphor and are well-covered by the training signal, though their semantic diversity limits recall.
• ADJ (Adjectives, F1 = 72.77%) and SCONJ (Subordinating Conjunctions, F1 = 72.97%) show comparable moderate performance. Adjectival metaphors are often subtle and context-dependent; SCONJ metaphors (e.g., as if, like) can be borderline with literal comparatives.
• ADV (Adverbs, F1 = 70.86%) mirrors findings in other metaphor detection systems: adverbial metaphors are semantically diverse and harder to detect from local context.
• NOUN (Nouns, F1 = 66.21%) has notably low recall (57.83%) despite reasonable precision (77.42%). Noun metaphors are the most frequent content-word class but show the widest range of metaphoricity, with many borderline cases (metonymy, dead metaphor) that the model misses.
• AUX / PART / PROPN have very few gold positives (≤ 4 each) in this test set; F1 estimates are not reliable at this sample size.

Label dictionary

{
  "0": "non_metaphor",
  "1": "metaphor"
}

Subwords are aligned to words via the tokenizer's word_ids; the first subword of each word is used for prediction.

Usage example

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

model_path = "tommyleo2077/deberta-v3-large-clause-metaphor"  # or local path
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForTokenClassification.from_pretrained(model_path)
model.eval()

words = ["The", "government", "attacked", "the", "proposal", "."]
inputs = tokenizer(
    words,
    is_split_into_words=True,
    return_tensors="pt",
    truncation=True,
    max_length=192,
)
word_ids = inputs.word_ids(batch_index=0)

with torch.no_grad():
    logits = model(**inputs).logits
preds = logits.argmax(dim=-1)[0].tolist()

word_preds = {}
for i, wid in enumerate(word_ids):
    if wid is not None and wid not in word_preds:
        word_preds[wid] = preds[i]

for i, w in enumerate(words):
    label = "metaphor" if word_preds.get(i, 0) == 1 else "non_metaphor"
    print(f"{w}\t{label}")

Citation

Model author: Tommy Leo — 1683619168tl@gmail.com

Dataset (MIPVU / VUAMC):

@book{steen2010method,
  title     = {A Method for Linguistic Metaphor Identification: From {MIP} to {MIPVU}},
  author    = {Steen, Gerard and Dorst, Aletta G. and Herrmann, J. Berenike and Kaal, Anna and Krennmayr, Tina and Pasma, Thea},
  year      = {2010},
  publisher = {John Benjamins}
}

Train/test split:

@inproceedings{leong2018vua,
  title     = {A Report on the 2018 {VUA} Metaphor Detection Shared Task},
  author    = {Leong, Chee Wee and Beigman Klebanov, Beata and Shutova, Ekaterina},
  booktitle = {Proceedings of the Workshop on Figurative Language Processing at NAACL-HLT 2018},
  year      = {2018}
}

Base model: Microsoft DeBERTa

This model:

@misc{leo2025debertav3largeclausemetaphor,
  title        = {deberta-v3-large-clause-metaphor: Metaphor Detection at Clause Level},
  author       = {Leo, Tommy},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/tommyleo2077/deberta-v3-large-clause-metaphor}},
  note         = {Contact: 1683619168tl@gmail.com}
}