Shipflow

Automated product improvement pipeline. Feedback from Discord is classified, ticketed, built, tested with a cohort, and shipped to production.

Overview Issues Trends Evaluations Signals

Recent evaluations

Click a row to view source messages and AI reasoning.

4,860 evaluations

Verdict

Summary

Signals

Conf.

Date

New ticket

The agent gets stuck in two distinct loops. First, after successfully creating a folder, it attempts to rename it but repeats the `type` action three times (Steps 7, 8, 10) followed by confused clicks. This indicates a failure to recognize that the typing action was successful. Second, it enters a severe loop from steps 16-25, repeatedly trying to `press_and_hold` the same coordinates ([385, 345]) to start creating a second folder, but never succeeds in initiating a drag. The task is incomplete and the agent is stuck.

95%

Jul 13

New ticket

The agent enters a clear and unproductive loop from step 9 to step 25. It repeatedly performs a `press_and_hold` at the same coordinates followed immediately by a `press_back`. This indicates it's trying to initiate an action (likely entering home screen edit mode or selecting an app) but fails to proceed and just cancels, achieving no progress towards the goal of organizing apps. The provided screenshots are for steps 59-61, which do not exist in the 25-step log, so they were disregarded.

100%

Jul 13

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 13

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 13

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 13

New ticket

The agent repeatedly enters selection mode by using `press_and_hold`, selects a few photos, scrolls, and then starts a completely new selection with another `press_and_hold`. This indicates it's losing the selection state and never progresses to the actual deletion step. It's stuck in a loop of selecting photos without ever acting on them, making the task incomplete. The provided screenshots, while mismatched with the step numbers in the log, show a photo editor, suggesting the agent eventually got completely sidetracked from its goal of deleting photos in a gallery view.

95%

Jul 13

New ticket

The agent is stuck in a loop, repeatedly trying to launch the eBay app. The log shows 10 consecutive `start_app` actions for the same package. This indicates the agent is failing to recognize that the app has successfully launched and is not proceeding to the next logical step of interacting with the UI.

100%

Jul 13

Dismissed

The agent was tasked with selecting a photo and making multiple edits. The log shows a series of clicks and drags consistent with using photo editing tools. The provided screenshots, despite a mismatch in step numbers with the log, confirm the task was successfully completed. The screenshot for step 41 shows the edited photo ready to be saved, and the screenshot for step 42 shows the 'Saved!' confirmation screen. Therefore, the agent achieved the user's goal.

100%

Jul 13

Dismissed

The agent successfully pasted the required text into a new thread. The final step in the log (Step 7) is a click on the 'Yes' button of a confirmation dialog to post the content, as shown in the screenshot for that step. The session ends immediately after this successful confirmation, strongly suggesting the task was completed as requested. The click on the keyboard at Step 6 was unnecessary but did not cause a failure.

95%

Jul 13

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 13

Dismissed

The session consists of a single tap action with an 'Unknown' completion status. Without screenshots or further steps, it's impossible to determine if the agent succeeded or failed. The session is too short and lacks context to be actionable.

10%

Jul 13

Dismissed

The agent correctly interpreted the user's request to "Tell me when this is done" as a command to immediately finish the task and report its completion. The agent's single action was `finished({"content":"Done"})`, which is the correct and most efficient way to handle this meta-task. This is a successful session.

100%

Jul 13

New ticket

The agent successfully launches the correct application, 'Photo Collage', but then fails to complete the task. The log shows a series of clicks, including a repeated one, suggesting it might be lost. The screenshots confirm the failure decisively. The screenshot for step 61 shows an incoming call, a likely interruption. The screenshots for steps 62 and 63 show the agent has completely left the photo editing app and is back in its own chat UI, discussing a different task entirely ('reschedule it tomorrow'). This indicates the agent abandoned the original task after being interrupted.

100%

Jul 13

New ticket

The agent only performed a single step, launching the eBay app, and then the session terminated. It never attempted the core part of the task, which was to search for an item. This is a clear failure, likely due to a system-level issue or crash immediately after the app launch.

100%

Jul 13

Dismissed

The agent successfully opened the Threads app, navigated to the new post screen, and typed the full, correct content. The session ends with a `needs_confirmation` step, which is the expected behavior for a task that involves publishing content. The screenshot at step 7 confirms the agent is correctly asking for user confirmation before posting. This is a successful session.

100%

Jul 13

New ticket

The agent correctly navigates to the Play Store and searches for an app. However, it gets into a minor loop by typing the search query twice (Steps 8 & 10). The main failure occurs after it likely initiates an app install (Step 17). It then waits for a long time (13 seconds across steps 14-16, 18-19, 21) and proceeds to click in random, unproductive locations (Steps 20, 22-24), indicating it is lost and doesn't know how to proceed or confirm the task is complete. The provided screenshots are from a different session and are ignored.

172

90%

Jul 12

New ticket

The agent fails to complete the primary goal of finding and installing an app. It starts in an existing app, leaves it, and then gets stuck in a significant loop for 14 steps (steps 5-18), repeatedly scrolling and clicking the same coordinate on what is likely the home screen or app drawer. While the final screenshots show it eventually found the correct feature within the app, the long loop and failure to follow the initial instructions constitute a clear bug.

95%

Jul 12

New ticket

The agent session began with a 'max steps' error popup from a previous session already on screen. The agent correctly identified and attempted to click the 'OK' button in steps 1 and 2 to dismiss it, as confirmed by the screenshots. However, the popup persisted, blocking any further action. The agent correctly recognized it was blocked and terminated the task. This indicates a failure in the environment's state-reset logic between sessions, not a failure of the agent's task-solving ability for the specified goal.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent was explicitly told 'Do not post it until you have reviewed the entire thread'. However, after drafting the post, it proceeded to click the 'Post' button in step 14. The final screenshot confirms this action with a 'Posting...' toast notification at the bottom of the screen. This is a clear violation of a negative constraint. The agent also exhibited overrun behavior by first creating a completely different thread (steps 1-10) before starting the correct one (steps 11-15).

100%

Jul 12

New ticket

The agent attempted to send an audio file via WhatsApp. It correctly identified a file and asked for confirmation. However, its own internal monologue at step 22 states that it sent the wrong file. Following this error, the agent became stuck, performing a series of aimless clicks and failing to recover or complete the original task. The repetitive clicks on what appears to be a send button (steps 18, 19) also indicate a problem. The provided screenshots are for steps that do not exist in the log (35-37), so they cannot be used for analysis.

90%

Jul 12

New ticket

The agent correctly types a review but then clicks the 'Publish' button twice in a row (Step 3 and Step 4). The screenshots confirm that the UI element targeted is the same in both steps and that the screen does not change after the first click. This indicates a potential issue with the agent not correctly detecting the state change after a submission click, leading it to repeat the action unnecessarily.

95%

Jul 12

New ticket

The agent gets stuck in a loop trying to update a TikTok bio. It types the full text, then seems to fail, and then proceeds to type progressively shorter versions of the same text. This strongly suggests it is hitting a character limit, failing to save, and then trying a flawed recovery strategy of shortening the input. The initial three attempts to start the app are also redundant. The provided screenshots are from steps 59-61, while the log only goes to step 25, so they are not relevant to this analysis.

107

90%

Jul 12

New ticket

The agent correctly identifies that it needs to wait for user input in Step 1, but then immediately gets derailed. In steps 2 and 3, it sends unsolicited and irrelevant messages about PDF uploads, a topic the user never mentioned. It repeats the same incorrect message, indicating it's stuck in a conversational or reasoning loop. The task is not completed because the agent never receives the necessary information from the user. The screenshots provided are for steps that are not in the log, so they cannot be used for analysis, but the failure is clear from the log alone.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent is stuck in a loop trying to dismiss a 'max steps reached' popup. The screenshots from steps 7 and 8 clearly show this popup on the screen. The agent repeatedly clicks at coordinates `[871, 85]`, which corresponds to the text at the top of the popup, not the 'OK' button. This is a clear wrong target identification failure on a system dialog.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent spends the first 24 steps struggling to navigate the Settings app to find the app list. It gets stuck in loops (typing 'Apps' multiple times, clicking the same coordinates) and uses `press_back` frequently, indicating confusion. It only starts searching for the correct app name at step 25. This is highly inefficient. Although screenshots provided (for steps beyond the log's range) suggest it eventually finds the app, the initial navigation is severely flawed. This is a BLOAT and STUCK issue.

90%

Jul 12

New ticket

The agent failed the first half of a multi-app task. After searching for headphones on Amazon, it appears to have failed to apply a filter or sort at Step 5. It then gets lost, presses the home button, re-enters the app, and clicks aimlessly (Steps 11-17) before abandoning the Amazon task entirely and moving on to AliExpress. This constitutes an INCOMPLETE failure pattern, as it never found the item on Amazon nor performed the requested comparison.

95%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

102

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent initially tries to perform the task by opening YouTube and clicking in the top-right corner, likely looking for the cast icon. It gets stuck in a loop between steps 5 and 12, clicking the top-right and top-left corners repeatedly. After failing and relaunching the app, the agent completely loses the context of the original task. From step 16 onwards, it starts sending user messages about making a phone call, which is completely unrelated to the YouTube casting goal. This indicates a severe failure in state management or goal tracking after an initial action failure.

95%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

115

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent correctly completes the task at step 2, even calling the 'finished' action. However, it then continues to perform 24 additional, unrequested actions, including typing multiple messages and eventually pressing the home button. This is a classic overrun failure where the agent's termination logic fails after task completion. The provided screenshots are for steps 64-66, which do not exist in the provided log (ends at step 26), so they cannot be used for analysis.

100%

Jul 12

New ticket

The agent correctly identifies the arrow keys for the game but lacks any strategy to play it effectively as requested by the user ("properly with good mind"). It enters a repetitive loop of clicking the same few buttons, which quickly leads to a "Game Over" state as seen in the final screenshot. This is a model limitation in handling complex, stateful tasks like playing a game.

90%

Jul 12

New ticket

The agent failed to execute the task due to a backend service error. The screenshot for Step 2 clearly shows an 'HTTP 503' error with the detail 'Hermes tenant capacity exhausted'. This is not an on-device agent logic failure but a critical infrastructure or service-level issue that prevented the agent from operating.

100%

Jul 12

Dismissed

The session contains only a single 'start_app' action and the task goal is unknown. There is no evidence of an agent failure. The session likely ended immediately for reasons outside the agent's control, such as user cancellation or a system issue. This is not an actionable bug.

100%

Jul 12

Dismissed

The agent was invoked while the user's notification shade was open. The agent correctly identified this state and prompted the user for a task. This is the expected behavior, not a failure. The session ended after this single, appropriate action because no task was given.

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

120

100%

Jul 12

Dismissed

Noise filter: no actionable issues detected.

100%

Jul 12

New ticket

The agent receives a file path as a task goal. It correctly opens the 'Files by Google' app but then fails to navigate. It presses home and relaunches the app, indicating it's stuck. The session ends without completing the task. The provided screenshots are from steps 8-10, but the log only has 7 steps, and they show a completely different task and a backend infrastructure error ('Hermes SSE connection failed'). Therefore, the ticket is based solely on the provided step log, which shows a clear agent failure.

90%

Jul 12

Loading more…