Our approach shows strong generalization and versatility in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney. It also enables easy editing and multi-concept prompt generation.

Prompt engineering for personalized image generation is labor-intensive, requires model-specific tuning, limiting generalization.

PRISM uses VLMs and iterative in-context learning to automatically generate effective, human-readable prompts using only black-box access to image generation models.

With small perturbations (less than 5% of total web page pixels), attackers can execute targeted adversarial goals with up to 67% success rates.

We also find that inference-time compute that is often used to improve model performance can introduce new vulnerabilities and harm robustness.

4/4 Llama 3.1 70B agents successfully complete 16.7% of tasks on 150k websites. Agents trained on human-annotated data from Mind2Web and WebLINX struggle to generalize to real-world websites. Adding synthetic data significantly improves generalization.

With B Trabucco, G Sigurdsson, R Piramuthu

3/4 Language models perform competitively with human annotators, achieving:
- 97% accuracy in detecting and filtering harmful content
- 89% success rate in generating feasible tasks
- 82% accuracy in judging successful task completions

2/4 The pipeline follows a three-step process:
- LLM generates tasks for 150k websites
- LLM agents complete these tasks and produce trajectories
- LLM reviews the trajectories and evaluates their success

3/3 Joint work with Tiffani Min, Yue Wu, Jimin Sun, Max Kaufmann, Fahim Tajwar, Yonatan Bisk

2/3 Offline-collected state transitions are evaluated using PRMs to determine optimal intervention timing, creating labeled trajectories for training the helper model.

This minimizes costly intervention calls during training while leveraging PRMs to enhance robustness to off-policy data.

🌲 Ruslan Salakhutdinov (@rsalakhu.bsky.social) from CMU (@scsatcmu.bsky.social) opened the workshop with a talk on Tree Search for Language Model Agents.

Timestamp 36:20 in neurips.cc/virtual/2024...

📎 arxiv.org/abs/2407.01476

#NeurIPS2024 #AdaptiveFoundationModels

2/2 Our findings show that even when unlearning a single fact, current methods either fail to properly unlearn with high recall or end up unlearning many other irrelevant facts.

Paper: arxiv.org/abs/2410.15153
Code+Dataset: github.com/wrh14/deep_u...

joint work R Wu, C Yadav, K Chaudhuri.