A team of researchers associated with Peking University, Pika, and Stanford University has introduced RPG (Recaption, Plan, and Generate). The proposed RPG framework is the new state-of-the-art in the context of text-to-image conversion, especially in handling complex text prompts involving multiple objects with various attributes and relationships. The existing models which have shown exceptional results with simple prompts, often need help with accurately following complex prompts that require the composition of multiple entities into a single image
Previous approaches introduced additional layouts or boxes, leveraging prompt-aware attention guidance, or using image understanding feedback for refining diffusion generation. These methods have few limitations in handling overlapped objects and increasing training costs with complex prompts. The proposed method is a novel training-free text-to-image generation framework named. RPG harnesses multimodal Large Language Models (MLLMs) for improved compositionality in text-to-image diffusion models.
The model is composed of three core strategies: Multimodal Recaptioning, Chain-of-Thought Planning, and Complementary Regional Diffusion. Each separate strategy helps in enhancing the flexibility and precision of long text-to-image generation. Unlike existing techniques, RPG uses editing in a closed loop which improves its generative power.
Coming to what each strategy does:
- In Multimodal Recaptioning, MLLMs transform text prompts into highly descriptive ones, decomposing them into distinct subprompts.
- Chain-of-thought planning involves partitioning the image space into complementary subregions, assigning different subprompts to each subregion, and leveraging MLLMs for efficient region division.
- Complementary Regional Diffusion facilitates region-wise compositional generation by independently generating image content guided by subprompts within designated regions and subsequently merging them spatially.
The proposed RPG framework uses GPT-4 as the reception and CoT planner, with SDXL as the base diffusion backbone. Extensive experiments demonstrate RPG’s superiority over state-of-the-art models, particularly in multi-category object composition and text-image semantic alignment. The method is also shown to generalize well to different MLLM architectures and diffusion backbones.
RPG framework has demonstrated exceptional performance compared to other existing models in both quantitative and qualitative evaluations. The model surpassed ten known text-to-image generating models in attribute binding, recognizing object relationships, and the complexity of the prompt. The image generated by the proposed model is detailed and successfully includes all the elements in the text in the image. It outperforms other diffusion models in precision, flexibility, and generative ability. Overall, RPG offers a promising avenue for advancing the field of text-to-image synthesis.
Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our Telegram Channel
Pragati Jhunjhunwala is a consulting intern at MarktechPost. She is currently pursuing her B.Tech from the Indian Institute of Technology(IIT), Kharagpur. She is a tech enthusiast and has a keen interest in the scope of software and data science applications. She is always reading about the developments in different field of AI and ML.
