DiffPS: Leveraging Prior Knowledge of Diffusion Model for Person Search

Method

Prior Knowledge in Diffusion Model

DiffPS uses a frozen diffusion model backbone to provide rich spatial features, and separates task-specific features for detection and Re-ID to avoid gradient interference. This decoupled design ensures stability and better representation learning. To fully exploit the capabilities of the pre-trained diffusion model, DiffPS draws upon four core priors:

• 1) Text Conditioning 🔽
  Pre-trained diffusion models exhibit strong cross-modal alignment between text and image features, enabling precise localization of person-related regions (e.g., body parts, clothing). This alignment enhances the model’s ability to suppress background clutter and overcome occlusion, providing more robust feature representations for person search. This motivates us to design DGRPN and SFAN, which explicitly exploit this alignment for detection and identification tasks, respectively.
• 2) Timestep Semantics 🔽
  The informativeness of diffusion features varies with timestep. For complex scenes like person search, earlier to mid-stage steps (e.g., t=300) offer the best balance between denoising and detail preservation. These steps retain fine-grained features while minimizing the impact of both synthetic and real-world noise, leading to improved person-specific representation.
• 3) Hierarchical Structure 🔽
  Diffusion models follow a UNet-based hierarchical structure, where up-stage layers merge global context and local details through skip connections. While up-stage features are generally effective for both detection and re-ID, their usefulness varies across layers. Carefully selecting specific up-stage outputs leads to stronger spatial precision and identity discrimination.
• 4) Shape Bias 🔽
  As noted in “Intriguing properties of generative classifiers”, generative models often exhibit a shape bias, emphasizing global structures over fine details. For person search, where both global structure and fine-grained details are essential, leveraging diffusion features while further enhancing high-frequency information can lead to even greater discriminative power This motivates us to design MSFRN, which explicitly counteracts the shape bias by reinforcing fine-grained representations.

Framework

DiffPS features a frozen diffusion backbone with decoupled detection (DGRPN) and re-ID (MSFRN, SFAN) branches, each tailored to leverage distinct diffusion priors for robust person search.

BibTeX