Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

Beomyoung Kim; Sung Ju Hwang

ECCV 2026 · Accepted

Phoenix

Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

Keep your segmenter. Add Phoenix. Turn coarse, noisy masks into precise object boundaries with a model-agnostic refinement layer trained on realistic adversarial errors.

Beomyoung Kim^1,2 Sung Ju Hwang^1,3

1 KAIST2 NAVER Cloud, Image Vision3 DeepAuto.ai

Paper Code Demo

Phoenix in practice

Refine masks across your pipeline.

Phoenix takes an image and a noisy binary mask, then returns a cleaner mask. That small interface makes it useful as a post-processing layer, a data-quality step, or an on-demand vision tool.

Upgrade masks, not your model.

Phoenix sits behind the existing segmentation service and refines its output. The upstream model remains untouched.

Mask2Former · +3.8 AP^mask

Existing modelMask2Former / MRCNN / …→ Coarse maskunchanged API output→ Phoenixrefinement add-on→ Better masksame downstream contract

PhoenixModel output

Base model untouched

A model-agnostic finishing layer

Evaluated across Mask R-CNN, SOLO, CondInst, RefineMask, Mask2Former, ViTDet, and MaskDINO.

refined = phoenix(image, coarse_mask)

Choose an example 01 / 03 · Person

Numbers shown are reported paper results on COCO val with LVIS annotations.

Abstract

What doesn't kill you makes you stronger

Despite significant advances in image segmentation, even state-of-the-art models produce masks with imperfect boundaries, semantic inconsistencies, and structural errors. Mask refinement addresses these limitations, yet current approaches rely on simplistic synthetic noise that fails to capture the complex error patterns of real segmentation models. We introduce Phoenix, a framework that leverages adversarial learning to generate semantically meaningful noise patterns and contrastive learning to model refinement relationships. Our approach consists of two innovations: (1) Adversarial Mask Perturbation (AMP), which employs embedding-space attacks to create semantic-aware noise that mimics real segmentation errors, and (2) Contrastive Mask Refinement Learning (CMRL), a tri-directional framework that ensures feature consistency within semantic regions while maintaining separation between classes. Experiments demonstrate that Phoenix significantly outperforms existing methods across diverse tasks, while consistently enhancing state-of-the-art segmentation models with substantial improvements.

TL;DR

Two ideas, one synergy

AMP and CMRL are mutually constitutive: combined they exceed the sum of their parts (+10.4 IoU vs. +6.2 and +2.5 alone).

AMP

🎯

Adversarial, not destructive

We repurpose embedding-space FGSM as a constructive noise generator. Perturbations concentrate exactly where models are uncertain — challenging boundaries and ambiguous regions.

🎛️

Controllable noise

Guidance masks and an IoU threshold τ let us dial error type (over/under-segmentation) and intensity, covering the full spectrum of realistic failures.

CMRL

🧲

Tri-directional contrast

Six regions over target / noisy / refined masks drive intra-class consistency, inter-class separation, and self-improvement — tailored to refinement, not generic representation learning.

Method

The Phoenix framework

Built on a frozen SAM encoder; only the lightweight decoder (4M params) is fine-tuned.

Overview. AMP injects learnable perturbation embeddings into the frozen decoder to synthesize realistic noisy masks; CMRL then supervises refinement in feature space across six target/noisy/refined regions.

AMP

Semantic-aware noise via embedding attacks

Unlike morphological operations that apply random geometric distortions, AMP runs FGSM in the embedding space of a frozen decoder. The resulting noise is image-conditional and lands on semantically hard regions — over-/under-segmentation, contextual confusion, and boundary imprecision — at only ~6 ms per update.

Morphological (left) vs. our adversarial (right) noise masks.

Tri-directional contrastive structure over six pixel regions.

CMRL

Contrastive refinement in feature space

CMRL partitions pixels into true / success / failure × foreground / background regions and applies three objectives — intra-class consistency, inter-class contrast, and self-improvement bootstrapping — guiding failures toward already-corrected regions within the same image.

Why it works

Adversarial noise is semantically richer

Pearson correlation between noise location and image edge/texture maps on LVIS val.

Morphological: narrowly concentrated near 0 — semantically uncorrelated.

Adversarial (ours): spans [−0.6, 0.8] — aligns with semantic structures and homogeneous regions, mirroring real-model errors.

Noise pattern comparison vs real model errors

Realism. AMP reproduces the failure patterns of real models (1% PointWSSIS, DIS-UNet) that morphological operations cannot synthesize.

Results

State-of-the-art across tasks

Weakly semi-supervised instance segmentation — PointWSSIS, COCO train5K (LVIS annotations)

Method	Supervision	AP^mask	AP^boundary
PointWSSIS	𝓕 1% + 𝓟 99%	12.6	6.3
+ SegRefiner		14.7	9.5
+ SAMRefiner		21.8	16.4
+ Phoenix (ours)		28.7 +16.1	23.6 +17.3
PointWSSIS	𝓕 5% + 𝓟 95%	25.7	16.4
+ SegRefiner		27.6	20.2
+ SAMRefiner		32.8	26.2
+ Phoenix (ours)		36.3 +10.6	30.2 +13.8
PointWSSIS	𝓕 10% + 𝓟 90%	30.2	20.4
+ SegRefiner		31.7	23.8
+ SAMRefiner		36.6	29.7
+ Phoenix (ours)		38.9 +8.7	32.6 +12.2

𝓕: full mask annotations · 𝓟: object-center point annotations. Gains shown vs. the unrefined baseline.

Modern instance segmentation models — COCO val (LVIS annotations)

Base model	AP^mask	+ Phoenix	AP^boundary	+ Phoenix
Mask R-CNN (R50)	39.8	46.9 +7.1	27.3	38.8 +11.5
SOLO	37.4	46.2 +8.8	24.7	37.7 +13.0
Mask2Former	46.8	50.6 +3.8	37.0	42.1 +5.1
MaskDINO	56.8	58.0 +1.2	46.5	48.6 +2.1

Selected results from Table 2. Phoenix consistently improves both classical and state-of-the-art segmentation architectures without modifying the base model.

Fine-grained segmentation — DIS benchmark, average over 5 splits

Base model	Refiner	IoU	Boundary 𝓕
U-Net	—	54.7	68.4
	+ SegRefiner	58.7	72.6
	+ Phoenix	75.7 +21.0	84.4 +16.0
PSPNet	—	55.8	69.8
	+ Phoenix	75.9 +20.1	84.8 +15.0
HRNet	—	60.5	74.9
	+ Phoenix	76.0 +15.5	85.1 +10.2
ISNet	—	66.1	79.6
	+ Phoenix	76.6 +11.0	85.9 +6.3

Gallery

See the refinement

LVIS instance segmentation: noisy input → SAMRefiner → Phoenix. Phoenix restores missing object structure and cleaner boundaries.

DIS fine-grained refinement: thin structures and intricate contours recovered with high fidelity.

The IoU threshold τ controls noise intensity — from aggressive (low τ) to subtle (high τ) perturbations.

Honest limitations: ambiguous targets, totally mislocalized inputs, and class-agnostic confusions — discussed in the supplementary.

Analysis

Efficient, stable, and synergistic

Component synergy

AMP	CMRL	AP¹	IoU¹
✗	✗	22.3	65.3
✓	✗	26.9 +4.6	71.5 +6.2
✗	✓	23.8 +1.5	67.8 +2.5
✓	✓	28.7 +6.4	75.7 +10.4

Combined gain (+10.4 IoU) exceeds the sum of parts — true synergy.

Encoder efficiency

Encoder	AP¹	VRAM	FPS
ViT-H	28.7	4.9 GB	3.5
ViT-L	26.6	4.1 GB	5.6
ViT-B	22.4	2.8 GB	12.4
EfficientViT-XL1	28.0	0.8 GB	45.2

EfficientViT-XL1 retains accuracy at a 17× memory cut and 45 FPS. Over 5 runs Phoenix is stable: AP¹=28.7±0.5, AP²=46.9±0.3.

Citation

BibTeX

@inproceedings{kim2026phoenix,
  title     = {Learning from Adversity: Semantic-Aware Mask Refinement
               through Adversarial Perturbation},
  author    = {Kim, Beomyoung and Hwang, Sung Ju},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}