Error analysis reveals that unresolved bugs are not randomly distributed; they cluster around specific bug types and higher complexity profiles. In particular, Program Anomaly, Network, and GUI bugs remain the most challenging for both models.

Evaluated on 314 real-world Python bugs, we observed consistent gains in both #fixed and Pass@k scores for Llama 3.3 and GPT-4o-mini, demonstrating a 23% improvement over prior work.

This layered approach offers several advantages.
Allows simpler bugs to be fixed with minimal input, conserving tokens and computation.
Scales context progressively, injecting more information only when necessary. Enables analysis of bug types & complexity.

1️⃣ Bug Knowledge (e.g., immediate code and test context)
2️⃣ Repository Knowledge (e.g., related files, dependencies, commit history)
3️⃣ Project Knowledge (e.g., documentation, past bug fixes)

💡 If you are building, evaluating, or relying on LLMs for software development, please ask yourself: Did it warn you about the hidden security risk?

As a preliminary solution to this problem, we built a CLI tool prototype that integrates static analysis with LLM prompting, aiming to make AI code suggestions more secure by design.

However, when LLMs do warn you, they tend to offer more complete explanations, including potential causes of the vulnerability, exploits, and even fixes.

We evaluated GPT-4, Claude 3, and Llama 3 across 300 real-world Stack Overflow posts containing vulnerable code.

The results?
⚠️<40% of vulns flagged
⚠️As low as 12.6% when code was obfuscated
⚠️Common issues (e.g., unsanitized input) often missed - unless explicitly prompted

Congrats!!

Can you please add me here