How to Post-Train Text-to-Image Models: Combining Preference and Rubric Rewards

Large-scale preference data provides a strong and general signal, but it does not explicitly check whether every detail in the prompt has been correctly followed. To explicitly measure prompt following, we introduce a faithfulness reward based on automatically generated, prompt-specific rubrics.

For each training prompt, we use a language model to decompose the prompt into a tree-structured checklist of concrete yes/no questions. Each question evaluates one aspect of the requested image, such as whether an object is present, whether it has the correct attribute, whether two objects have the requested spatial relationship, or whether a specified style is preserved. Dependencies between questions are also captured—for example, checking the color of an object only makes sense if that object is present.

A vision-language model then evaluates the generated image against these questions, and the fraction of satisfied criteria becomes the faithfulness reward. More details can be found in our paper [1].

Faithfulness autorubric methods can construct questions that check basic object presence, attributes, and relationships. However, they are less suited to checking constraints, such as whether the user wants to avoid an undesired style, especially when these constraints are implicit. More importantly, these autorubrics are derived from the prompt before training and therefore cannot anticipate reward-hacking behaviors that emerge during RL training.

To complement this, we introduce rubric-based rewards for constraint satisfaction and reward-hacking prevention. The constraint reward checks whether the model introduces content that conflicts with the user’s intent—for example, adding unnecessary objects, changing a requested style, or violating negative instructions. Because some prompts are open-ended and allow creative freedom, we apply this reward only when the prompt calls for relatively strict adherence.

We also use rubric-based rewards for recurring reward-hacking behaviors that emerge during RL, such as adding garbled text or drifting toward photorealism when a non-photographic style is requested.

Rather than treating these anti-reward hack rewards as independent reward objectives, we use them to shape the preference reward through a simple gating function. Specifically, let $r_{\mathrm{pref}}$ denote the preference reward and we define a binary indicator $d_{\mathrm{hack}} \in {0,1}$ indicating whether a reward-hacking behavior is detected. We use the reward hack prevention reward to modulate the normalized preference reward:

$r_{\mathrm{pref}}^{\mathrm{gated}} = \begin{cases} r_{\mathrm{pref}}, & d_{\mathrm{hack}} = 0,\\ \min(r_{\mathrm{pref}}, 0), & d_{\mathrm{hack}} = 1. \end{cases}$

Thus, the preference reward is preserved when no reward-hacking behavior is detected, while it is canceled when the anti-reward-hacking signal is flagged. This allows the detector to suppress known exploits without introducing another independent objective that the policy can optimize or game.

Finally, we found that models trained with different reward configurations often develop complementary strengths. We therefore ensemble independently optimized policies directly in weight space by averaging their weight updates [2]. Together with independent reward normalization and prompt-dependent gating, this gives us a simple reward-composition recipe that works consistently without extensive reward-weight tuning.