Shipflow

Automated product improvement pipeline. Feedback from Discord is classified, ticketed, built, tested with a cohort, and shipped to production.

Overview Issues Trends Evaluations Signals

Recent evaluations

Click a row to view source messages and AI reasoning.

12,651 evaluations

Verdict

Summary

Signals

Conf.

Date

New ticket

The agent successfully inputs a command into a terminal-like application but then enters a long wait loop (12 consecutive waits) because it misinterprets the continuous output as a single, unfinished process. It's waiting for a specific 'progress vector format' that it never seems to find, causing the loop-guard to terminate the session. The screenshots confirm the agent is looking at a terminal that is actively generating text, but the agent model is stuck waiting instead of acting.

100%

Aug 15

New ticket

The agent successfully completed the user's request to find popular apps in the Play Store by step 13. However, instead of terminating the session, it continued with erroneous actions. It sent confusing messages to the user about not being able to 'follow' something (steps 14-15), and then launched an unrelated app (step 16, attempting to launch TikTok but screenshot shows Facebook). This is a clear case of an 'OVERRUN' failure, where the agent does not correctly identify the task completion state and proceeds with irrelevant actions.

100%

Aug 15

New ticket

The agent completely failed to follow the user's instructions. The user provided a very specific, long, and multi-sentence text to be sent in a chat. The agent, instead of typing this text, generated its own series of shorter, simpler, and incorrect messages that were only vaguely related in tone. This is a critical failure in instruction following, where the agent hallucinates a different task instead of executing the explicit one given.

100%

Aug 15

New ticket

The agent clicks the 'Agree and Continue' button at step 5, but the screenshots for steps 5, 6, and 7 show the screen is unchanged. This indicates the click action failed. Despite the failure, the agent proceeds to message the user at step 6 that it has successfully accepted the terms. This is a silent failure where the agent hallucinates success without verifying the outcome of its action.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

New ticket

The agent correctly launches the Builda app but then makes a critical navigation error. In step 3, it clicks on a user's profile instead of the chat input field, as confirmed by the screenshot for step 5 which shows it landed on the profile page for 'Pancho'. Although it correctly identifies the mistake and presses back in step 4, its state representation becomes corrupted. In step 5, its thought process is completely detached from the actual screen state; it believes the message is already typed and ready to send, when it had just navigated back to the main chat screen. This indicates a failure in both element identification (tapping the profile) and state tracking after a corrective action.

95%

Aug 15

New ticket

The agent fails to follow a consistent `type -> send` pattern for a continuous chat task. It frequently types multiple messages into the input field before clicking send, which results in concatenated and jumbled messages. For example, in steps 15-16, it performs two `type` actions in a row without a `click(send)` in between. This pattern repeats. Additionally, the agent gets into small loops, like repeating the same `type` action (Steps 10-11) or repeatedly clicking the input field (Steps 1-4).

95%

Aug 15

New ticket

The agent was given a multi-step task: open Snapchat, read the terms of service, and summarize them. The log shows the agent successfully completed the first step of opening the app. The provided screenshots, although from later unlogged steps, confirm that the app opened to the correct 'Terms of Service' update screen. However, the session terminated immediately after step 2, with no attempt to interact with the screen, click the links to the terms, or perform the requested summarization. This is a clear failure to complete the task.

100%

Aug 15

New ticket

The agent completely fails to recognize its own UI. The screenshots at Step 1 and 2 clearly show the agent's accessibility permission screen, but the agent's thought process indicates it believes it is already in the Android Settings app. Its first action is a drag, which is inappropriate for the screen it's on. Its second thought process is a hallucination, describing a different screen (AnyDesk settings) than the one shown in the screenshot. The agent is stuck before it can even begin the user's task due to a fundamental failure in state perception.

100%

Aug 15

New ticket

The agent gets stuck in a navigation loop between steps 3 and 10, repeatedly trying to enter the correct chat. More importantly, after the user provides follow-up instructions (after step 12), the agent unnecessarily relaunches the app for every single message. Steps 13, 18, and 23 all use `start_app` to perform an action in an app that should already be open and in focus. This demonstrates a significant state management failure where the agent loses context between turns, leading to extreme inefficiency (BLOAT).

95%

Aug 15

Dismissed

The agent successfully engaged in the chat conversation as requested by the user. The log shows a consistent pattern of reading messages, typing responses in the correct persona, and sending them. The screenshots from late in the session confirm the agent is actively participating in a live chat. This is a successful execution of a creative, open-ended task.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

Dismissed

The agent was tasked with entering a chat and role-playing as a girlfriend. The provided logs and screenshots show the agent successfully engaged in a continuous, multi-turn conversation, adapting its responses to user messages while maintaining the requested persona. There is no indication of a failure; the agent is performing the open-ended task as intended.

100%

Aug 15

New ticket

The agent repeatedly re-launches the 'Builda' app (Steps 1, 5, 11) despite being already inside it. The agent's own thought process reveals this confusion, stating in Step 5 and 11 that it believes it's on the AGI app's screen when it is likely already in the target app. This indicates a failure in state awareness or screen recognition, leading to unnecessary and inefficient actions. The agent eventually performs the core task of chatting, but the repeated re-launching is a clear bug.

95%

Aug 15

New ticket

The agent was tasked with opening the 'Builda' app. It correctly identified the task and used the `start_app` tool. However, the screenshots clearly show that the agent did not leave the application it started in. The screenshot at Step 1 shows a chat app, and the screenshot at Step 2 shows the same chat app. Despite this, the agent reported in Step 2 that it had successfully opened 'Builda' and proceeded to describe the chat interface as if it were the target app. This is a clear case of hallucination where the agent fails to verify the outcome of its action and falsely reports success.

95%

Aug 15

Dismissed

The agent was tasked with continuously chatting in a specific persona. The logs show a repeating pattern of typing a message and clicking to send it. This is the correct behavior for the given task. The screenshots, although from later in the session, confirm the agent was successfully participating in the chat, with its messages appearing in the chat history. There is no failure pattern here; the agent is performing the user's open-ended creative task as expected. The session likely ended because the user was satisfied or manually stopped it.

98%

Aug 15

New ticket

The agent is stuck in a first-time user onboarding flow for what appears to be the Picsart app. The screenshots clearly show a multi-step questionnaire in Spanish that the agent is unable to navigate correctly. It repeatedly clicks the 'Siguiente' (Next) button without always fulfilling the prerequisites on each screen, such as selecting an option. This prevents it from ever reaching the core functionality of the app required to complete the user's goal.

100%

Aug 15

New ticket

The agent gets stuck in a loop from steps 8-14 trying to click a text input field in the Builda app. It performs multiple clicks around the same y-coordinate (~890-905) without successfully focusing the field. It also repeatedly attempts to type into the field before it is focused (e.g., Step 7, 18, 23), indicating a state detection problem. It eventually recovers but the initial loop and repeated errors are significant.

90%

Aug 15

New ticket

The agent failed to send the chat message. In Step 6, it intended to click the 'Enviar' (Send) button but instead clicked coordinates [856, 545], which are located in the middle of the screen on the chat history, not on the button itself. The screenshot at Step 7 confirms the message was never sent; it does not appear in the conversation log. The agent then incorrectly concluded it had succeeded and ended the session.

100%

Aug 15

Dismissed

The agent's task was to chat in an app for 10 minutes. The provided logs and screenshots show the agent successfully typing messages, sending them, and receiving replies within a chat. Screenshot at step 62 shows a message sent by the agent, and the screenshot at step 63 shows a reply from another user, which the agent correctly observes in its thought process for step 60. This behavior directly aligns with the user's goal. No failure is observed.

100%

Aug 15

Dismissed

The agent successfully completed the task. It correctly identified the chat input field, tapped it, typed a relevant message, and then used `needs_confirmation` to ask the user if it should continue. This is the expected behavior for this type of open-ended task.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

New ticket

The agent repeatedly fails to identify the correct action after typing a message in a chat interface. Instead of clicking the 'Send' button, it clicks the text input field again. This led to an erroneous navigation to a different chat screen at step 6, forcing a recovery sequence. The same incorrect action pattern is repeated multiple times throughout the session (e.g., steps 14, 20, 25), indicating a fundamental misunderstanding of the UI state.

95%

Aug 15

New ticket

The agent attempts to install an app, which likely requires opening the Play Store. It correctly opens the app drawer via a drag action at Step 39. However, it then enters a loop at Steps 40-42, repeatedly clicking coordinates [396, 130]. The screenshot for step 47 shows this coordinate corresponds to the 'Buscar apps' (Search apps) bar in the app drawer. The clicks are ineffective, causing the agent to press back at Step 43 and ultimately get stuck in a larger loop, as indicated by the loop-guard at Step 46. The core failure is the inability to successfully interact with the search input field.

95%

Aug 15

New ticket

The agent is attempting to chat in an app, but it fails to recognize the 'Tap to chat' button that is necessary to activate the text input field. The screenshots for steps 11 and 12 clearly show this button is present, but the agent's actions and thoughts indicate it believes it's interacting with a standard, always-visible text field. It clicks in the general vicinity of the button but never activates it, causing it to get stuck.

100%

Aug 15

Dismissed

The agent successfully completed the task of opening the camera. The screenshot from Step 5 clearly shows the camera application is open and active, confirming the agent's `finished` action in Step 4 was correct. The task was successful.

100%

Aug 15

Dismissed

The agent successfully launched the Picsart app, encountered a sign-in screen, and correctly escalated to the user with `needs_confirmation` at step 6. This is the expected and desired behavior. The subsequent steps show the agent navigating the app's initial onboarding flow, which is a prerequisite to editing images. Although the log is short and the task is incomplete, there is no evidence of a failure, loop, or incorrect action. The agent appears to be proceeding correctly.

90%

Aug 15

New ticket

The agent is stuck in a loop trying to type into a chat input field in the Builda app. It repeatedly clicks the input area but the subsequent 'type' actions fail to register text. Out of numerous attempts, only one message (from step 15) successfully appears in the chat, as confirmed by the screenshot at step 25. The agent also attempts to click a non-existent 'send' button at step 16. This indicates a fundamental problem with element interaction, specifically focusing and typing into this app's UI.

95%

Aug 15

New ticket

The agent fails to ever leave its own UI. After attempting to launch the 'Builda' app in Step 1, it proceeds to click on its own 'Tap to chat' bar, believing it's interacting with the target app. Screenshots from steps 11-13 confirm it is stuck in the AGI UI. Critically, in Step 7, the agent hallucinates that it has completed the task and sends a `finished` event, which is a severe failure of state awareness. The agent is both stuck and incorrectly reports success.

100%

Aug 15

Dismissed

The agent successfully completed the task of opening the camera. Step 1 goes to the home screen and Step 2 directly launches the camera app via `start_app`. The screenshot provided for 'Step 5' clearly shows the camera interface open and functional. The task was completed successfully and efficiently.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

New ticket

The agent successfully opened Messenger and searched for the specified contact. However, it stopped after typing the name and did not proceed to the second part of the task: initiating the call. The session ends prematurely, making this an incomplete task.

100%

Aug 15

Dismissed

The agent successfully navigated the Google Play Store, searched for the correct app, requested user confirmation, and then initiated the installation. The series of long waits at the end (steps 11-16) is the expected behavior while an app downloads and installs. There are no signs of a loop, error, or incorrect action. The task appears to have been completed successfully. The screenshots provided are for steps 46-48, which are not in the log, so they were disregarded.

95%

Aug 15

Dismissed

The agent correctly identified the user's request as a potential attempt to generate harmful or manipulative content, violating its safety policies. It appropriately declined the task and notified the user. This is an example of the safety filter working as intended, not a bug.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

New ticket

The agent was tasked with opening the Shopee app. It issued the `start_app` command twice (Steps 1 and 3). However, the screenshots at Step 3 and 4 clearly show the agent is viewing the TikTok app. This indicates a fundamental failure of the `start_app` primitive, as the agent was unable to switch from the current app (TikTok) to the target app (Shopee).

100%

Aug 15

Dismissed

The provided step log is too short to identify a failure. It only shows the first 4 steps where the agent correctly launches Messenger and begins typing the contact's name. The screenshots, which are from later steps not included in the log (steps 8, 9, 10), show the agent successfully in a conversation with the correct contact ('ธนภัทร'). This visual evidence strongly suggests the task was completed successfully and the log is simply incomplete.

95%

Aug 15

Dismissed

The agent correctly typed the message as requested in the Builda app. It then correctly identified that sending a message is an irreversible action and used `needs_confirmation` to ask the user for permission before proceeding. This is the intended and correct behavior. The session ended pending user input, which is not an agent failure.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 15

New ticket

The agent is stuck in a loop trying to type into a web-based chat interface (Google's Gemini/Spark). It repeatedly fails to locate and focus the correct text input field. The logs show a repeating pattern of `click` and `type` actions, with the agent's internal monologue confirming that it believes the typing action is failing. The screenshots from the end of the session confirm this analysis: the on-screen keyboard is visible, but the text input field is not, suggesting the agent has scrolled past it or is clicking on the wrong area (the chat history). The core failure is the inability to correctly identify and interact with the web text input element.

100%

Aug 14

New ticket

The agent got stuck in a loop between the chat screen and the group info screen. It repeatedly clicked the three-dots menu and the back button for over 10 steps. The root cause was its failure to recognize that it needed to scroll down on the group info screen to find the 'change alias' option. It only broke the loop after finally performing a scroll action in step 24, which revealed the necessary UI element.

100%

Aug 14

New ticket

The agent successfully sends an initial message, but then completely deviates from the user's stated goal. The user wanted the agent to *wait* for a reply, but the agent hallucinates a new, complex goal to 'send 5 successive, escalating messages'. While attempting this self-assigned (and incorrect) task, it gets confused, clicks wrong targets, loses track of its state (e.g., believing a message was sent when it wasn't), and performs redundant actions. This is a critical failure in goal adherence and state tracking.

100%

Aug 14

New ticket

The agent was tasked to maintain a specific persona ('Olie' with a 'rocky vibe') in a chat. While the provided logs (steps 22-46) show it performing the task correctly, the screenshots from later in the session (steps 64-66) clearly show that the agent has completely broken character. It has reverted to a generic AI assistant, talking about its own capabilities ('soy una IA con múltiples capacidades'). This is a critical failure of long-term instruction following and persona maintenance.

100%

Aug 14

Dismissed

The provided step log is only 2 steps long, showing the agent successfully launching the app. The screenshots are from much later (steps 61-63) and show the agent has correctly navigated to the target chat screen. Without the logs for steps 3-60, it's impossible to determine if a failure occurred or what the cause was. The available evidence suggests the agent was on the path to success, so there is no actionable bug to report.

10%

Aug 14

Dismissed

The agent successfully completed the task. It correctly interpreted the user's request to monitor the screen for a reply, waited an appropriate amount of time, detected the screen change when the reply appeared, and notified the user as requested. The agent's actions perfectly align with the goal, and it demonstrated successful screen monitoring and comprehension.

100%

Aug 14

Dismissed

Noise filter: no actionable issues detected.

100%

Aug 14

Loading more…