New: Auto-Improve prompts feature + other tweaks

[tinomarques]

• New: Auto-improve feature
• Update: Added support for streaming .reasoning from a message
• Misc: Variable changes no longer cause a dirty state
• Misc: Other ui tweaks
• Refactor: Variable state data type
• WIP (paused): Prompt evals panel

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
helicone	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2025 7:04pm
helicone-bifrost	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2025 7:04pm
helicone-eu	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2025 7:04pm

[fumedev]

Summary

• Added new Auto-Improve feature that analyzes and suggests improvements for prompts with reasoning
• Added support for streaming reasoning from message responses in LLM API
• Modified variable state data type to use optional idx instead of isMessage boolean
• Variable changes no longer trigger dirty state updates
• Added UniversalPopup component for reusable popup functionality
• Added initial scaffolding for prompt evaluations panel (work in progress)

🔍 Fume is reviewing this PR!

🔗 Track the review progress here:
https://app.fumedev.com/chat/0203d289-c3fd-4e59-8f10-a85f40220ad2

[fumedev]

Empty interfaces can cause type checking issues. Either remove this or add expected properties.

🔨 See the suggested fix

@@ -43,9 +43,14 @@
   idx?: number;
 }
 
-export interface StateEval {}
+export interface StateEval {
+  id: string;
+  name: string;
+  type: "classifier" | "test";
+  enabled: boolean;
+}
 
-// DB INTERFACE
+// DB INTERFACE 
 export interface PromptVersionReference {
   id: string;
   minor_version: number;
@@ -58,4 +63,4 @@
   parent_prompt_version?: string | null;
   experiment_id?: string | null;
   updated_at?: string;
-}
+}

_{⚠️ Fume is an LLM-based tool and can make mistakes.}

[fumedev]

No error handling if JSON parse fails. This could cause silent failures in the streaming response.

🔨 See the suggested fix

@@ -114,35 +114,25 @@
               V{version + 1} <span className="">(Suggested)</span>
             </h3>
             <div className="flex flex-col gap-2">
-              {parseImprovedMessages(improvement.content).map((msg, index) => (
-                <MiniMessage
-                  key={index}
-                  role={msg.role}
-                  content={msg.content}
-                />
-              ))}
+              {(() => {
+                try {
+                  return parseImprovedMessages(improvement.content).map((msg, index) => (
+                    <MiniMessage
+                      key={index}
+                      role={msg.role}
+                      content={msg.content}
+                    />
+                  ));
+                } catch (error) {
+                  return (
+                    <div className="text-red-500 text-sm p-4">
+                      Error parsing improvement content. Please try again.
+                    </div>
+                  );
+                }
+              })()}
             </div>
           </div>
-        </div>
-      )}
-
-      {/* Actions */}
-      {improvement && !isImproving && (
-        <div className="flex justify-center gap-2">
-          <Button variant="outline" onClick={onCancel}>
-            Cancel
-          </Button>
-          <Button
-            className="bg-green-500 dark:bg-green-500 hover:bg-green-500/90 dark:hover:bg-green-500/90"
-            onClick={onApplyImprovement}
-          >
-            Save Suggested Version
-          </Button>
-        </div>
-      )}
-    </div>
-  );
-}
 
 const MiniMessage = ({ role, content }: { role: string; content: string }) => (
   <div className="flex flex-col gap-1">

_{⚠️ Fume is an LLM-based tool and can make mistakes.}

[fumedev]

Add a guard clause to return early if state is null to prevent runtime errors.

[fumedev]

Testing

• Attempted Browser Testing

  • Tried to test login: Failed due to unresponsive form and relogin warnings
  • Tried to start app locally: Failed due to missing Vercel auth

• Code Review Testing

  • Found empty StateEval interface that would cause type errors
  • Identified race condition in streaming state updates in AutoImprove
  • Found missing error handling for JSON parsing in streaming response
  • Found missing null check in handleImprove that could cause runtime errors

• Not Tested (Blocked)

  • Auto-Improve feature functionality
  • Variable state management changes
  • Streaming response handling

[chitalian]

Why is this here?