The Nugget

LLM-based agents in SE combine concepts from LLMs and agents to enhance software engineering tasks by utilizing a framework with three key modules: perception, memory, and action. Despite the utility of LLMs, addressing challenges like diverse input modalities, knowledge retrieval bases, and multi-agent collaboration efficiency is essential for advancing their effectiveness in SE.

Make it stick

💡 The three core components of LLM-based agents in SE are perception, memory, and action.
🔍 In SE, LLM-based agents can use token-based, tree/graph-based, and hybrid-based inputs for code understanding.
🧠 Semantic memory relies on external knowledge bases, while episodic memory stores context-related information.
🎯 Future opportunities include enhancing multi-modality input, establishing a comprehensive knowledge base, and improving multi-agent collaboration efficiency.

Perception Module:
- Textual Input: Includes token-based (treats code as natural language), tree/graph-based (models structural code info), and hybrid-based inputs (combines multiple forms for richer context).
- Visual Input: Uses images like UI sketches for integrating visual context into code analysis.
- Auditory Input: Incorporates auditory data for interacting with LLMs through speech.
Memory Module:
- Semantic Memory: Includes documents, libraries, and API information that provide world knowledge.
- Episodic Memory: Records current case-related content and historical interactions to improve reasoning accuracy.
- Procedural Memory: Involves implicit knowledge in LLM weights and explicit knowledge coded into agents.
Action Module:
- Internal Actions:
  - Reasoning Actions: Generate high-quality answers using techniques like Chain-of-Thought (CoT).
  - Retrieval Actions: Gather relevant information from knowledge bases.
  - Learning Actions: Continuously update knowledge by learning from feedback.
- External Actions:
  - Dialogue: Agents interact with humans and other agents to refine answers.
  - Digital Environments: Agents use tools like compilers and completion engines for self-optimization.

Exploring Perception Module:
- Lack of exploration in tree/graph-based inputs, as well as visual and auditory inputs.
Role-playing Abilities:
- Agents need multi-faceted capabilities to handle diverse tasks in SE.
Knowledge Retrieval Base:
- Absence of a comprehensive and authoritative code-related knowledge base.
Addressing Hallucinations:
- LLMs need methods to mitigate hallucinations (e.g., generating non-existent APIs) to enhance agent reliability.
Efficiency in Multi-agent Collaboration:
- Improving efficiency by managing computing resources and minimizing communication overhead.
Integrating SE Technologies:
- Leveraging advanced SE techniques like software testing to boost agent development.

"A single agent contains three key modules: perception, memory, and action."
"Exploring different modalities for input and refining the perception module could significantly enhance the capabilities of LLM-based agents."
"Semantic memory can be updated by incorporating a recognised external knowledge base, potentially enriching the agent's ability to make informed decisions."
"Hallucinations of LLM-based agents remain a critical challenge; addressing this could significantly improve their reliability and overall performance."
"Collaborative efficiency among multiple agents can be improved by optimizing the management and synchronization of shared resources."

This summary contains AI-generated information and may have important inaccuracies or omissions.