Table of Contents
DOM navigation: Why e-commerce sites are so hard for agents
Sophia Willows
Head of Engineering @ Rye
Aug 27, 2025
DOM changes break most shopping agents. Rye’s Universal Checkout API adapts with self-healing workflows to enable true agentic commerce.
E-commerce is vast and fragmented. In the U.S. alone, there are nearly 12 million live merchant sites, powered by hundreds of different platforms, custom builds, and marketplaces.
On the path to building a universal checkout agent, some companies are tackling the challenge incrementally, starting with one or two major platforms like Shopify. Using APIs or carefully crafted scripts certainly makes it easier to get to something you can launch, but it doesn’t solve the real challenge. Agentic commerce, by definition, has to work across the open web. Consumers don’t restrict their shopping to a handful of platforms, and neither should their agents.
That’s why DOM adaptability matters. To be credible, an agent must be able to navigate the structure of any site, no matter how it’s built or how often it changes. And that’s where most systems fail, or don’t even try.
E-commerce is vast and fragmented. In the U.S. alone, there are nearly 12 million live merchant sites, powered by hundreds of different platforms, custom builds, and marketplaces.
On the path to building a universal checkout agent, some companies are tackling the challenge incrementally, starting with one or two major platforms like Shopify. Using APIs or carefully crafted scripts certainly makes it easier to get to something you can launch, but it doesn’t solve the real challenge. Agentic commerce, by definition, has to work across the open web. Consumers don’t restrict their shopping to a handful of platforms, and neither should their agents.
That’s why DOM adaptability matters. To be credible, an agent must be able to navigate the structure of any site, no matter how it’s built or how often it changes. And that’s where most systems fail, or don’t even try.
E-commerce is vast and fragmented. In the U.S. alone, there are nearly 12 million live merchant sites, powered by hundreds of different platforms, custom builds, and marketplaces.
On the path to building a universal checkout agent, some companies are tackling the challenge incrementally, starting with one or two major platforms like Shopify. Using APIs or carefully crafted scripts certainly makes it easier to get to something you can launch, but it doesn’t solve the real challenge. Agentic commerce, by definition, has to work across the open web. Consumers don’t restrict their shopping to a handful of platforms, and neither should their agents.
That’s why DOM adaptability matters. To be credible, an agent must be able to navigate the structure of any site, no matter how it’s built or how often it changes. And that’s where most systems fail, or don’t even try.
The real challenges of DOM navigation
More than many other kinds of websites, e-commerce sites create many challenges for agents, notably:
Dynamic layouts: Merchants redesign pages frequently and unpredictably. A button that was in the same place for months may shift IDs, change location, or disappear in an A/B test tomorrow.
Conditional flows: Different users see different checkout paths based on region, shipping rules, or inventory availability. An agent that succeeds in buying a golf club in Florida may be unable to buy an umbrella in Idaho from the same site.
Popups and overlays: Newsletter signups, discount offers, or cookie banners often obscure the checkout path. A human dismisses them as casually as swatting a fly; an agent can get stuck and fail.
Honeypots: Some sites insert invisible elements under the outdated premise that all bots trying to buy things are bad. They don’t affect humans, but they can mislead automated flows into taking actions that mark them as suspicious.
Each of these agentic browsing challenges has occupied many research teams. An agent that doesn’t have a strong approach for all of them cannot credibly be considered universal.
Rye's approach
Rye designed its Universal Checkout system to handle this reality from the start. The first time an agent encounters a site, we first pre-process sites to strip out unnecessary elements like popups. Then, guided by heuristics and RLHF, the LLM explores like a human: moving through the DOM, filling forms, and completing checkout. That first successful run is captured as an execution trace, which Rye compiles into a deterministic plan keyed to the site’s structure.
On subsequent visits, the plan takes over to execute quickly, at about 1/50th of the computational cost and 3 to 4 times the speed of LLMs. But when something changes, Rye doesn’t fail, it adapts. The runtime falls back to the agent for the specific step that broke, completes the flow, and regenerates the plan to reflect the new layout. In practice, this self-healing loop between exploration and reuse is what allows Rye to stay both resilient and fast, even as e-commerce sites shift around it. (In full candor: we have programmatically integrated with Shopify and Amazon, simply because the high volumes and seconds-fast checkout make the effort worthwhile.)
Rye designed its Universal Checkout system to handle this reality from the start. The first time an agent encounters a site, we first pre-process sites to strip out unnecessary elements like popups. Then, guided by heuristics and RLHF, the LLM explores like a human: moving through the DOM, filling forms, and completing checkout. That first successful run is captured as an execution trace, which Rye compiles into a deterministic plan keyed to the site’s structure.
On subsequent visits, the plan takes over to execute quickly, at about 1/50th of the computational cost and 3 to 4 times the speed of LLMs. But when something changes, Rye doesn’t fail, it adapts. The runtime falls back to the agent for the specific step that broke, completes the flow, and regenerates the plan to reflect the new layout. In practice, this self-healing loop between exploration and reuse is what allows Rye to stay both resilient and fast, even as e-commerce sites shift around it. (In full candor: we have programmatically integrated with Shopify and Amazon, simply because the high volumes and seconds-fast checkout make the effort worthwhile.)
Rye designed its Universal Checkout system to handle this reality from the start. The first time an agent encounters a site, we first pre-process sites to strip out unnecessary elements like popups. Then, guided by heuristics and RLHF, the LLM explores like a human: moving through the DOM, filling forms, and completing checkout. That first successful run is captured as an execution trace, which Rye compiles into a deterministic plan keyed to the site’s structure.
On subsequent visits, the plan takes over to execute quickly, at about 1/50th of the computational cost and 3 to 4 times the speed of LLMs. But when something changes, Rye doesn’t fail, it adapts. The runtime falls back to the agent for the specific step that broke, completes the flow, and regenerates the plan to reflect the new layout. In practice, this self-healing loop between exploration and reuse is what allows Rye to stay both resilient and fast, even as e-commerce sites shift around it. (In full candor: we have programmatically integrated with Shopify and Amazon, simply because the high volumes and seconds-fast checkout make the effort worthwhile.)
Why solving DOM navigation matters
Without the ability to handle arbitrary sites, agentic commerce reduces to a handful of controlled integrations. That’s not the open-web shopping experience consumers expect.
Rye has been working on these challenges since before ChatGPT launched. In building what we now call the Sync API, we’ve processed hundreds of millions of product description pages (PDPs) on tens of thousands of e-commerce sites. We’ve learned how to dodge popups, evade hidden fields, and reduce our reliance on LLMs to browse.
By solving DOM navigation, Rye ensures that agents can buy anywhere, reliably. For consumers, that means pressing “buy” just works. For implementers, it means one API abstracts away all the details to completing a purchase, no matter how messy the site underneath might be.
Without the ability to handle arbitrary sites, agentic commerce reduces to a handful of controlled integrations. That’s not the open-web shopping experience consumers expect.
Rye has been working on these challenges since before ChatGPT launched. In building what we now call the Sync API, we’ve processed hundreds of millions of product description pages (PDPs) on tens of thousands of e-commerce sites. We’ve learned how to dodge popups, evade hidden fields, and reduce our reliance on LLMs to browse.
By solving DOM navigation, Rye ensures that agents can buy anywhere, reliably. For consumers, that means pressing “buy” just works. For implementers, it means one API abstracts away all the details to completing a purchase, no matter how messy the site underneath might be.
Without the ability to handle arbitrary sites, agentic commerce reduces to a handful of controlled integrations. That’s not the open-web shopping experience consumers expect.
Rye has been working on these challenges since before ChatGPT launched. In building what we now call the Sync API, we’ve processed hundreds of millions of product description pages (PDPs) on tens of thousands of e-commerce sites. We’ve learned how to dodge popups, evade hidden fields, and reduce our reliance on LLMs to browse.
By solving DOM navigation, Rye ensures that agents can buy anywhere, reliably. For consumers, that means pressing “buy” just works. For implementers, it means one API abstracts away all the details to completing a purchase, no matter how messy the site underneath might be.
Learn more
If you’re building a recommendation engine or other shopping tool, check out the docs and see how the Universal Checkout API is the key to offering your users truly seamless buying.
If you’re building a recommendation engine or other shopping tool, check out the docs and see how the Universal Checkout API is the key to offering your users truly seamless buying.
If you’re building a recommendation engine or other shopping tool, check out the docs and see how the Universal Checkout API is the key to offering your users truly seamless buying.
Related articles
Related articles
Monetize
your AI platform
with shopping.
Monetize
your AI platform
with shopping.
Monetize
your AI platform
with shopping.
Product
Resources