Why you can’t “track” personal agents

Summary

• Personal agents deliver personalization based on rich context, which means measuring them like ChatGPT or Gemini is a mistake. Each interaction with Muse or Instinct builds the app’s picture of who the user is. As that picture gets more complex, recommendations and answers drift further from the average distribution.
• Most vendors offer Muse tracking via the Muse LLM API, which doesn’t reflect Muse in real-world conditions. Instinct has no underlying model API and offers no way to manage context directly, making it a non-starter.
• Personalization changes AI output, meaning automated accounts don’t fix it either. Besides potential ToS violations, simulating realistic users at scale is very complex.
• Server-side tracking is not always viable (for now). Muse and Instinct don’t always identify themselves, and even an identified visit doesn’t show the context that led to it. Emerging standards may change that in the future.
• Seeing where Muse and Instinct struggle to use your site is valuable, but few vendors offer it yet. Manually checking where personal agents encounter blocks or fail to convert on your site gives you immediate feedback, but it’s not the same as visibility tracking.

What makes personal agents different?

The value proposition of personal agents centers on personalization and integration.

It’s all over their branding. Instinct pitches itself as “a personal assistant that understands what you’re working on and what’s important to you.” Muse “gets to know you and your goals, working across all aspects of your life.”

Personal agents hook into as many of a user’s accounts as possible. They can receive access to your social accounts, email, or anything else, then use them on your behalf. Importantly, all of these things—and your entire chat history—form a continuous context about who you are and what you like.

As we reported earlier this month, Muse continuously builds a user profile in the form of daily notes, goal files, a shopping profile, and regular monitoring of your email and calendar.

Tracking ChatGPT with fresh, context-free chats works reasonably well because most real chats are also short and light on context. Personal agents are the opposite. A typical user has weeks of history, memory files, and connected accounts behind every question. If personalization changes AI behavior, then a context-free sample doesn’t look anything like a real user.

Personalization changes AI behavior

In our comparison of the Muse app and API, both recommended the same products when they had similar context. But as context in our Muse account grew, that context asymmetry played a decisive role, leading to product recommendations that never appeared in the API’s answers. The decisive information was what Muse had recorded about us in the background, not something we put in our prompt.

That was one account over a few days. Existing studies focus on the impact of specific context elements in other scenarios:

• iPullRank published a study in which seeding relevant brand names in Google accounts connected to AI Mode increased their mentions from 23.9% to 66.8%. AI Mode was reading the user’s emails and using them to inform its final output.
• Unusual, an AI visibility vendor, prepended buyer personas to queries like “best CRM software” and found the recommended brands changed significantly depending on who was asking. Mid-market brands swapped out up to 75% of the set, while category leaders stayed mostly stable.
• Profound gave Muse Spark and ChatGPT memory files modeled on Muse’s to simulate users, then ran shopping prompts with and without that context. Recommended products for the sessions with context and without it overlapped half as much as repeat prompts for the same simulated user. Muse Spark’s searches also included details about the user 54% of the time.

More research is needed, but it seems likely that context and personalization have a larger effect on AI behavior than many marketers would like. A context-free sample captures the model’s defaults, but maybe not what a user with context actually sees.

Measuring personal agents is uniquely difficult

Analytics vendors currently offer three ways to track brand visibility in AI: API calls, app automation, and server log tracking. For personal agents, the first two fall short, and the third isn’t yet viable.

Why the Muse API doesn’t work

UI scraping is the preferred data collection method among leading AI visibility tools. Chat UI answers differ significantly from API responses in length, number of brands mentioned, and number of citations.

In cases where no public, unauthenticated UI is available (like Claude), API calls are considered a best-effort fallback.

We’ve seen some vendors rush to use the Muse Spark API as a proxy for the Muse personal agent. Even when vendors acknowledge that the API is just a proxy, there are reasons to be skeptical.

The API doesn’t have Muse’s tools. In our report, we found that Muse checks the user’s shopping profile, always searches its memory, checks daily notes and other agent-managed files that didn’t come up in the search, and only then runs a Meta product catalog search and a browser search in parallel. The browser agent supplied 64% of the products Muse showed us, and 11 of its 13 top recommendations.

The Muse Spark API offers the model web search, but none of the tools that Muse relies on before ever searching the web.

We did find significant overlap in the brands mentioned when Muse had sparse context. The model does play a role. Muse’s main agent sometimes named products and brands in the browser agent’s instructions before it had received any search results.

However, recommendations diverged as context grew, and the API named far more brands (52 vs. 22) and cited an almost entirely different set of sources (43 shared domains out of 1,907), with or without context. Measuring Muse via the API may help you see if the model is aware of your brand at all, but it’s far too noisy to be a factor in marketing decisions.

In the case of Instinct, there is no underlying model API at all, and it doesn’t expose its steps and activity to the user like Muse does.

Sampling an underlying LLM is sampling a distribution, but it’s unlikely to resemble the distribution of what users actually see.

Account automation isn’t the solution

A few smaller vendors offer Instinct and Muse tracking by automating in-app interactions, such as WhatsApp messages for Instinct or the Muse web app for Muse. On the surface, this sounds better, as it’s much closer to how visibility vendors already track platforms like ChatGPT.

But aggregate visibility only means something if the samples represent real users. Vendors typically don’t disclose how their accounts are set up, and each option has a problem.

Shared accounts. In our testing, Muse carried “they prefer quiet gear” from a keyboard request into a later mouse search, through notes that it wrote itself. On a shared account, every query can shape the next one. Instinct in particular is extra difficult. It currently offers no method to manage or reset context. You can’t sample the same query multiple times in the same conditions.

Refreshed context every time. For agents that offer controls to manage stored context and data, resetting the context every time doesn’t reflect personal agents in real-world use. Your real customers ask questions in accounts with weeks or months of context to shape the final recommendations.

Multiple accounts per customer with seeded personas. An accurate simulation would likely require extremely detailed customer profiles and rich simulated context. First, running so many fake accounts almost certainly violates the agent provider’s terms of service. Second, it’s hard to rebuild a representative person. Managing this kind of simulation would be risky, costly, and complex.

Without knowing which of these a vendor is doing, you can’t know what its numbers mean.

Personal agents don’t always identify themselves on your server

Many AI crawlers identify themselves via user-agent strings or request signatures. Currently, Instinct doesn’t, and Muse only does sometimes.

DataDome reported that across 29.1M Muse requests in the agent’s first 12 days, it never identified itself. But since mid-October, Meta has started signing a portion of Muse requests with Web Bot Auth. If the trend continues, it will make server-side attribution easier.

As sunbeam Research reported last month, Instinct doesn’t identify itself at all. For direct web fetches, it uses Exa, with no additional signature or user-agent modification. For full browser visits, it appears exactly like desktop Chrome on Linux, coming from a residential IP.

Identification wouldn’t solve the whole problem, anyway.

In our testing, 36% of the products Muse showed us came from Meta’s catalog, a search that happens on Meta’s infrastructure and never touches your site. Its browser agent mostly searched large retailers like Best Buy, Amazon, and Newegg. A brand’s own server would only see visits that land on its own pages, like Muse checking price and stock on a product page.

While you would know something led the agent to visit your site, the rest of the journey is entirely opaque. In our testing, the most specific browser instructions Muse’s main agent wrote came from context, which doesn’t show up in your logs.

Protocols like WebMCP or the Personal Agent Protocol could pave the way for more transparent access and monitoring of agent activity on your site. PAP in particular is being actively developed in partnership with Instinct and Meta, and is supposed to ensure that site administrators can reliably identify agents and the users for whom they’re acting.

What you can do instead

The data you gather is only as useful as it is representative. The hurdles to gathering visibility data may be so great that it’s not worth the effort you could spend on something else.

Personalization affects which option an agent picks, but the options still come mostly from the web. That makes your site and product pages the part you can control.

Gather evidence of where agents fail on your site: blocked requests, unclear shipping policies, missing availability information, or pages the agent can’t use.

You don’t need complex automation or simulated users to do this. One person using one account can test agent accessibility, especially in agents like Muse, which show their steps. Blocks and missing information are properties of your site, not of the user. Context doesn’t apply.

By asking a personal agent to complete actions on your site, you can surface invisible issues that block potential customers when their agents do show up.

Few vendors offer this, and the ones that do bundle it with visibility metrics that likely don’t hold up.

All the analytics in the world won’t make a difference if your brand’s online presence isn’t well-structured, informative, and readable to both humans and agents.