Class AgentFeedbackCollector

java.lang.Object
neqsim.util.agentic.AgentFeedbackCollector
All Implemented Interfaces:
Serializable
public class AgentFeedbackCollector extends Object implements Serializable
Collects and aggregates feedback metrics from agent sessions for continuous improvement.

Maintains a rolling log of agent session outcomes, categorized by task type, agent name, and failure mode. Provides summary statistics that can be used to identify recurring failure patterns and guide improvements to both the AI agent prompts and the underlying NeqSim Java codebase.

Key Metrics Tracked:

  • Success rate per agent and task type
  • Common failure modes (convergence, missing API, invalid input)
  • Simulation run success rate across sessions
  • Average phase durations (scope, research, analysis, validation, reporting)
  • NeqSim API gaps discovered during sessions

Usage:


AgentFeedbackCollector collector = new AgentFeedbackCollector();

// After each session completes
collector.recordSession(session);

// Get aggregate statistics
String report = collector.getSummaryReport();
double successRate = collector.getSuccessRate("solve.task");

// Get improvement recommendations
List<String> gaps = collector.getDiscoveredAPIGaps();
Version:
1.0
Author:
Even Solbraa
See Also:

  • Field Details

  • Constructor Details

    • AgentFeedbackCollector

      public AgentFeedbackCollector()
      Create a new feedback collector.

  • Method Details

    • recordSession

      public void recordSession(AgentSession session)
      Record a completed agent session.
      Parameters:
      session - the completed session

    • recordAPIGap

      public void recordAPIGap(String description, String suggestedPackage, String priority)
      Record a discovered API gap (missing capability in NeqSim).
      Parameters:
      description - what is missing
      suggestedPackage - where the implementation should go
      priority - priority level ("critical", "important", "nice-to-have")

    • getOverallSuccessRate

      public double getOverallSuccessRate()
      Get the overall success rate across all recorded sessions.
      Returns:
      success rate as a fraction (0.0 to 1.0), or 0.0 if no sessions

    • getSuccessRate

      public double getSuccessRate(String agentName)
      Get the success rate for a specific agent.
      Parameters:
      agentName - agent to filter by
      Returns:
      success rate as a fraction (0.0 to 1.0), or 0.0 if no sessions for that agent

    • getSessionCount

      public int getSessionCount()
      Get the total number of recorded sessions.
      Returns:
      session count

    • getDiscoveredAPIGaps

      public List<AgentFeedbackCollector.APIGap> getDiscoveredAPIGaps()
      Get all discovered API gaps.
      Returns:
      unmodifiable list of API gaps

    • getFailureCategoryCounts

      public Map<String,Integer> getFailureCategoryCounts()
      Get the most common failure categories across all sessions.
      Returns:
      map of failure category to count, sorted by frequency

    • getAverageSimulationSuccessRate

      public double getAverageSimulationSuccessRate()
      Get average simulation success rate across all sessions.
      Returns:
      average simulation success rate (0.0 to 1.0)

    • getSummaryReport

      public String getSummaryReport()
      Generate a summary report as JSON.
      Returns:
      JSON summary report

    • generateRemediations

      public List<Map<String,Object>> generateRemediations()
      Generate automated remediation recommendations based on recurring failure patterns.

      Analyzes failure frequency by category and agent, and returns prioritized, actionable recommendations. This is the core of the cross-session learning loop described in the framework: failures drive recommendations, which in turn drive framework improvements.

      Returns:
      list of remediation recommendation maps, each with "priority", "category", "recommendation", "affectedAgent", and "frequency"

    • computeLearningTrend

      public List<Map<String,Object>> computeLearningTrend()
      Compute a learning trend showing success rate over sliding windows.

      Divides sessions into windows of 10 and computes the success rate in each window. An increasing trend indicates that the framework is improving over time through accumulated knowledge (skill updates, API gap fixes, new error-handling patterns).

      Returns:
      list of maps with "window", "sessions", "successRate" keys, one per window

    • getMostImpactfulGap

      public AgentFeedbackCollector.APIGap getMostImpactfulGap()
      Identify the most impactful API gap to fix next.

      Cross-references discovered API gaps with failure frequency to determine which missing capability would fix the most failures if implemented. This drives the development flywheel described in the framework: Gap discovery leads to targeted implementation.

      Returns:
      the highest-impact API gap, or null if none recorded

    • getRemediationText

      private String getRemediationText(String category, String agent, int count)
      Get remediation text for a failure category.
      Parameters:
      category - the failure category name
      agent - the affected agent name
      count - how many times this failure occurred
      Returns:
      human-readable remediation recommendation

    • classifyFailure

      public static AgentFeedbackCollector.FailureCategory classifyFailure(String failureReason)
      Classify a failure reason into a category.
      Parameters:
      failureReason - the failure reason string from AgentSession
      Returns:
      classified failure category