Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition

Text-to-image (T2I) diffusion models exhibit impressive photorealistic image generation capabilities, yet they struggle in compositional image generation. In this work, we introduce RoleBench, a benchmark focused on evaluating compositional generalization in action-based relations (e.g., "mouse chasing cat"). We show that state-of-the-art T2I models and compositional approaches consistently default to frequent reversed relations (i.e., cat chasing mouse), a phenomenon we call RoleCollapse. Related works attribute this to the model's architectural limitation or being underrepresented in the data. Our key insight reveals that while models fail on rare compositions when their inversions are common, they can successfully generate similar intermediate compositions (e.g., "mouse chasing boy"), suggesting that this limitation is due to the presence of frequent counterparts rather than the absence of rare compositions. Motivated by this, we hypothesize that directional decomposition can gradually mitigate role collapse. We test this via ReBind, a lightweight framework that teaches role bindings using carefully selected active/passive intermediaries. Experiments suggest that intermediate compositions through intermediate fine-tuning can significantly mitigate role bias, with humans preferring more than 78% compared to state-of-the-art methods. Our findings highlight the role of distributional asymmetries in compositional failures and offer a simple, effective path to improving generalization.