{"id":86895,"title":"Skyvern 2.0 - State-of-the-art web navigation with 85.8% on WebVoyager Eval","tagline":"Open source AI Web Agent","body":"We’ve been working hard, cooking up something new to share with you all!\n\nSkyvern 2.0 **scored state-of-the-art 85.85%** on the WebVoyager Eval.\n\nThis is the best-in-class performance of all WebAgents, giving advanced closed-source web agents like Google Mariner a run for their money.\n\n![uploaded image](/media/?type=post\u0026id=86895\u0026key=user_uploads/1120458/4f04949f-e01d-42cb-b568-a2460f7ac7f5)\n\n### TL;DR\n\n* **Real-World Tests:** We ran all of the tests in Skyvern Cloud to get a better representation of autonomous browser operations (ie, they didn’t run on any local machines)\n* **Open-Sourced Results:** All of the runs can be [seen here](https://eval.skyvern.com/) through our UI.\n* **We’re just getting started**. Try [Skyvern Cloud](https://app.skyvern.com/) or [Skyvern Open Source](https://github.com/Skyvern-AI/Skyvern) out for yourself and see Skyvern in action!\n\n## Agent Architecture\n\nAchieving this SOTA result required expanding Skyvern’s original architecture. Skyvern 1.0 involved a single prompt operating in a loop both making decisions and taking actions on a website. This approach was a good starting point but scored \\~45% on the WebVoyager benchmark because it had insufficient memory of previous actions and could not do complex reasoning.\n\nTo solve this problem, we created a self-reflection feedback loop within Skyvern. This resulted in 2 main changes:\n\n1. We added a “Planner” phase, which could decompose very complex objectives down into smaller achievable goals\n2. This allowed Skyvern to have a working memory of things it had completed and things that were still waiting to be finished\n   * This allows Skyvern to work with long, complex prompts without increasing the hallucination rate\n3. We added a “Validator” phase, which confirmed whether or not the original goals the “Planner” generates are successfully completed\n4. This acts as a supervisor function to confirm that the Task executor is achieving its objectives as expected and report any errors/tweaks back to the Planner so it can make adjustments in real-time as needed\n\n![](https://blog.skyvern.com/content/images/2025/01/architecture.png)\n\n## Test Setup\n\nAll tests were run in [Skyvern Cloud](https://app.skyvern.com/) with an **async cloud browser** and used a combination of GPT-4o and GPT-4o-mini as the primary decision-making LLMs. The goal of this test is to assert real-world quality — the quality represented by this benchmark is the same as what you would experience with [Skyvern](https://app.skyvern.com/)’s browsers running asynchronously.\n\n💡 **Why is this important?** Most benchmarks are run on local browsers with a relatively safe IP address and an impressive browser fingerprint. This is not representative of how Autonomous agents will run in the cloud, and we wanted our benchmark to represent how agents would behave in production\n\nIn addition to the above, we’ve made a few minor tweaks to the dataset to bring it up to date:\n\n1. We’ve removed 8 tasks from the dataset because the results are no longer valid. For example, one of the tasks asked to go to [apple.com](http://apple.com) and check when the Apple Vision Pro will be released — in 2025, it’s already been released and forgotten\n2. Many of the flight/hotel booking tasks referenced old dates. We updated both the prompt and the answer to more modern dates for this evaluation\n\n🔍 **For the curious**:\n\nThe full dataset can be seen here: \u003chttps://github.com/Skyvern-AI/skyvern/tree/main/evaluation/datasets\u003e\n\nThe full list of modifications can be seen here: \u003chttps://github.com/Skyvern-AI/skyvern/pull/1576/commits/60dc48f4cf3b113ff1850e5267a197c84254edf1\u003e\n\n## Test Results\n\nWe’re doing something out of the ordinary. In addition to the results, we’re making our entire benchmark run public.\n\n💡 **Why is this important?** Most benchmarks are run behind closed doors, with impressive results being published without any accompanying material to verify the results. This makes it hard to understand how things like hallucinations or website drift over time play an effect on agent performance\n\nWe believe this isn’t aligned with our open source mission, and have decided to publish the entire eval results to the public.\n\n📊 All individual run results can be seen here: \u003chttps://eval.skyvern.com\u003e\n\n🔍 The entire Eval dataset can be seen here: \u003chttps://github.com/Skyvern-AI/skyvern/tree/main/evaluation/datasets\u003e\n\n## Limitations of the WebVoyager benchmark\n\nThe WebVoyager benchmark is a comprehensive benchmark testing a variety of prompts on 15 different websites. While this acts as a good first step in testing Web agents, this only captures 15 hand-picked websites of the millions of active websites on the internet.\n\nWe think there is tremendous opportunity here to better evaluate web agents against one another with a more comprehensive benchmark similar to SWE-Bench.\n\n## What’s on the horizon\n\nBrowser automation is still a nascent space with tons of room for improvement. While we’ve achieved a major milestone in agent performance, a few important issues are next to be solved:\n\n1. Can we improve Skyvern’s reasoning to operate efficiently in situations with more uncertainty? Examples include vague prompts, ambiguous or highly complex websites/tools, websites with extremely poor UX (legacy portals)\n2. Can we give Skyvern access to more tools so it can effectively log into websites, make purchases, and behave more like a human?\n3. Can we have Skyvern better memorize things it has already done in the past so it can do them again at a lower price point?\n\n## References\n\n* [Google Mariner Report](https://deepmind.google/technologies/project-mariner/)\n* [Claude Computer Use Report](https://docs.anthropic.com/en/docs/build-with-claude/computer-use)\n* [WebVoyager Report](https://arxiv.org/html/2401.13919v4)\n* [WILBUR Report](https://arxiv.org/html/2404.05902v1)\n* [AgentE Report](https://arxiv.org/html/2407.13032v1)\n* [HCompany Report](https://www.hcompany.ai/blog/a-research-update)","slug":"MbX-skyvern-2-0-state-of-the-art-web-navigation-with-85-8-on-webvoyager-eval","created_at":"2025-01-16T16:48:00.855Z","updated_at":"2026-05-25T01:59:07.838Z","total_vote_count":10,"url":"https://www.ycombinator.com/launches/MbX-skyvern-2-0-state-of-the-art-web-navigation-with-85-8-on-webvoyager-eval","share_image_url":"https://www.ycombinator.com/media/?type=post\u0026id=86895\u0026key=user_uploads/1120458/4f04949f-e01d-42cb-b568-a2460f7ac7f5","company":{"id":28887,"name":"Skyvern","slug":"skyvern","url":"https://www.skyvern.com/","logo":"https://bookface-images.s3.amazonaws.com/small_logos/87d977bf0d4b7032c71830901e118beb0ca9946c.png","batch":"Summer 2023","industry":"B2B","tags":["Artificial Intelligence","Robotic Process Automation","Workflow Automation","Open Source","API"],"search_path":"https://bookface.ycombinator.com/company/28887"}}