How do coding agents choose deployment vendors?

Summary

• The repo and context mattered most. Agents cited details and concerns about specific requirements when explaining their recommendations. The repo changed the pick 89% of the time, versus 32% for a plain rerun.
• Model effects were present, but typically small. The models mostly agreed. But on some repos, models showed different preferences. GPT-6.1 Sol, released during the study, showed high agreement with GPT-6 Astra, which suggests that preferences may run in model families.
• A single deployment artifact can change the recommendation. In one repo, a netlify.toml took Netlify from zero to unanimous recommendations from one model, replacing Cloudflare.
• Agents mostly searched to confirm, not compare. Most didn’t search at all. Nearly all searches were performed by OpenAI models. Of those that searched, 61% of search queries used site:, and all directly fetched pages came from vendors’ own sites. Agents checked pricing, plan restrictions, and deployment details.
• Agents made incorrect claims about vendors and didn’t check. In 18 of 80 answers about one repo, agents warned that Upstash might not support a feature its docs say it does. None looked it up.
• Being mentioned isn’t the same as being recommended, but it can tell us about models’ prior knowledge. Vercel appeared in 510 of the 640 answers but was the top pick in only 29. When measuring AI visibility, differentiate between top recommendations, alternatives, and rejections.
• For vendors: get into the repo early. The repo outweighed the model and harness, and a single config file could change the recommendation. Distributing open-source resources like templates and boilerplates that already include your config may be the highest-leverage thing you can do.

AI is changing vendor selection

In a survey of 1,100 developers, Sonar reported that 48% had tried vibe coding. When they broke down the data by the size of the company where the devs worked, 67% of developers at SMBs rated agents effective for vibe coding tasks.

When AI takes responsibility for a large part or all of code generation, countless decisions get baked in, including vendors for everything from email to hosting.

Beyond the terminal, G2 surveyed 1,000 B2B software buyers, finding that “71% rely on AI Chatbots somewhere in the software research process.” Agents increasingly either make the choice independently or are trusted to present the options.

For vendors, this new reality poses some questions about how to grow when adoption may be decided by an LLM instead of a human. Some marketers have coined the term Agent Led Growth (ALG) to describe this phenomenon. One of the most important advertising surfaces may now be a shortlist in a chat UI or terminal.

How does the shortlist get built?

In order to plan how to get on the shortlist, we have to understand how the shortlist gets created. There are a few key elements that influence the final recommendation.

1. The model’s prior knowledge. This is everything the model “knows” before the prompt is sent.
2. The context. The context includes the prompt, the contents of the directory where it’s working, skills/rules, and any other user-created information.
3. The tools. The harness determines what the agent can do to inform its recommendation. Can it fetch pages? Does it have a domain allow-list? A native system prompt?
4. Retrieved information. If the model searches, what does it search for, and at what point in the process? Does it search widely or narrowly? Does it search to build a list of candidates, or for information about a vendor it already has in mind?

What we don’t know is what exerts the strongest influence on vendor selection. The answer to that question tells vendors and marketers where to focus their efforts: documentation, open-source templates, blog content, or something else entirely.

Existing research on LLM vendor selection and our experiment

There is plentiful research about the behavior of consumer AI surfaces, like ChatGPT’s web UI. However, there is significantly less about the retrieval and recommendation behavior of coding agents specifically. Agent CLIs execute tasks and make decisions in an environment substantially different from other consumer surfaces. To make informed decisions about ALG, marketers need to understand how those differences may change outcomes.

Armature analyzed 5,292 sessions and found that Codex, Cursor, and Claude Code all approach vendor selection differently. They report that Codex frequently searches the web with site: search operators, Claude Code tends to rely on its prior knowledge, and Cursor shows a more even split between the two behaviors.

A paper from Catal et al. found that frontier models show a statistically significant preference for affiliated vendors when models are asked to integrate third-party services into an incomplete codebase.

While these studies tell us what agents may choose, they do not necessarily tell us how.

To answer that question, we ran a full factorial experiment: every combination of repo, model, and harness. Because we compared runs that differed by only one factor, we can say how much each one moves the recommendation, rather than just observing what agents pick.

Methodology

We tested four coding harnesses: Claude Code, Codex, Pi, and OpenCode. We paired them with two frontier models, GPT-6 Astra and Claude Opus 5.5, and two open-weight models, GLM 5.3 Flash and DeepSeek V4.1 Flash.

We chose eight real-world, deployable open-source apps covering common tech stacks, distributed across three rough categories. The tech stack selection was informed by the Stack Overflow 2025 Developer Survey:

We chose less popular repos to limit memorization. We stripped deployment configs, hosting hints, and text identifying the original project. If a model consistently named the original project during testing, we replaced the repo.

Every run started with the same, simple prompt: “Where is the best place to deploy my app?” Each repo ran with every model in every harness, five times: 8 × 4 × 4 × 5 = 640 runs. The harnesses used their default settings, with model calls routed through OpenRouter or native provider APIs. We restricted OpenRouter calls to each model’s native provider.

We recorded answers, tool calls, and available reasoning, then classified vendor mentions as top picks, alternatives, conditional recommendations, passing mentions, or rejections. We classified a subset of answers by hand and used an LLM for the rest, spot-checking its labels against ours.

To compare repo, model, and harness effects, we measured how often changing one factor changed the top pick, above the variation from rerunning the same setup (the “noise floor”). This is a form of categorical variance decomposition, similar to ANOVA but built for outcomes that are categories, like vendor names, rather than numbers.

For specific preferences, like Astra’s lean toward Render, we used permutation tests: we shuffled model labels between runs that shared a repo and harness, and checked whether the real gap beat the shuffled ones.

Outside the main grid, we ran 80 tests through native Codex and Claude Code subscriptions to compare subscription behavior with API access. We also added up to 15 runs to each of the 46 setups with low agreement, bringing those setups up to 20 runs each.

We tested whether adding vendor-specific deployment config files changed the recommendation, with five runs per variant. We also compared common stack-vendor pairings with historical Stack Overflow mentions.

When GPT-6.1 Sol came out during the study, we ran it through the same eight repos and four harnesses, five times each. We report those 160 runs and the other follow-up checks separately. The main figures use the original 640 runs.

The context beats everything else

Changing the repo changed the top pick 89% of the time. Changing the model changed it 43% of the time, and changing the harness 37%. A plain rerun: 32%.

After accounting for the noise floor, the repo had by far the largest effect: +57 percentage points (95% CI 54–61), compared with +11 (CI 8–15) for the model and +6 (CI 2–9) for the harness. We tested one repo per stack, so this measures the whole repo: its framework and its specific requirements together.

The static sites mostly went to static hosts. Astro went to Cloudflare in 79 of 80 runs, while the plain HTML site went to GitHub Pages in 76. The Express and Next.js apps mostly went to managed app platforms, with Render and Railway leading their respective recommendations.

Part of this is common sense: a static site and a Python API need different kinds of hosts. But the FastAPI repo shows why the contents matter beyond the stack. The app keeps Gemini session cookies on disk, and agents recommended a self-hosted server in 61 of 80 runs. Every one of those 61 answers mentioned those requirements. Of those answers, 39 also raised IP reputation, including the difference between residential and datacenter IPs. These are requirements that “Python + FastAPI” alone wouldn’t tell an agent.

Small details in the repo made the difference between a recommendation and a rejection.

The Go API uses Redis keyspace notifications to expire user sessions, a setting listed in its .env.example and docs. In 54 of 80 Go answers, the agent flagged it as a requirement for whichever Redis provider the user chose. In 18, the agent warned that Upstash didn’t or might not support it. Upstash’s documentation says it does, and none of the agents looked it up.

The claim also counted against Fly.io, whose managed Redis runs on Upstash. Seven of the 18 warnings were framed as a drawback of Fly.io, and one answer passed over Fly.io for that reason alone:

“I’d avoid Fly.io as a firstpick here only because its Redis offering (Upstash) doesn’t surface keyspace-notify config the way your REDIS_NOTIFY_KEYSPACE_EVENTS=Ex setting needs…”

Some models treated it as settled fact in their reasoning:

“Upstash does NOT support keyspace notifications (it’s a known limitation…)”

The React app calls Bmob, a Chinese backend service, and its README is in Chinese. In 46 of 80 React answers, the agent added Chinese hosts such as Tencent COS and Aliyun OSS, usually as an alternative for users in mainland China:

“…such as Tencent Cloud COS or Alibaba Cloud OSS with their CDN. Those need an ICP license…”

That didn’t change the winner. Cloudflare Pages still took 67 of 80 React runs, and a Chinese provider was the top pick once. In both repos, one detail changed who else was named and who was ruled out.

Vercel lost on its own framework, but has immense mindshare

Vercel makes Next.js. You’d expect it to be the default for a Next.js app. It was the top pick in only 2 of 80 runs.

The agent made this decision based on the contents of the repo itself. The exemplar repo requires a long-running server, uses a custom cache, and talks to Redis over HTTP.

Many of the answers did acknowledge Vercel as a common option, but typically ruled it out.

Astra, running in Pi, said:

“Vercel is usually a strong default for Next.js…”

DeepSeek, running in Claude Code, said:

“[Vercel is] the default reflex for Next.js…”

The key observation is that while most models had a prior connection between Next.js and Vercel, they ruled against it in nearly every single run after examining the repo contents. The final vendor choice hinged on the technical requirements of the app itself.

While it was not measured in our study, it’s a reasonable hypothesis that a simpler full-stack Next.js app might result in a high rate of Vercel recommendations.

The monorepo that paired Next.js with a Go API did yield recommendations for Vercel in 16 of 80 runs. Not a majority by any means, but another example of how influential the repo was on the vendor selection.

A config file can change the pick

We took two setups with a near-unanimous pick and added deployment config files to the repo. Nothing else changed. We wanted to see whether an existing deployment configuration could change the recommendation.

On the Astro repo, a two-line netlify.toml took Netlify from 0/5 to 5/5. The file specified the build command and output directory, and Opus named it as the reason for recommending Netlify over Cloudflare:

“…the repo already has a netlify.toml (build with npm run build, publish dist). There’s no server code, so you just need a static host.”

“Why Netlify:netlify.toml already sets the build command (npm run build) and the publish folder (dist).”

The Express repo gave a different result. Adding a fly.toml got Fly.io onto the shortlist, but Astra still chose Render in all 5 runs. Fly.io had not appeared in the baseline answers; with the file, it was the named alternative every time. Astra acknowledged the existing config without treating it as a reason to change its recommendation:

“You already have a minimal configuration in fly.toml. If you’ve started setting up Fly, I wouldn’t switch just for this app…”

Astra’s searches also stayed the same. All 5 runs searched only Render’s docs, using the same queries as without the file. It never looked up Fly.io.

Adding a railway.json alongside fly.tomldid change the result. Astra picked Railway in all 5 runs and didn’t mention Fly.io once. Render, the original winner, became an alternative in 3 answers. In the control variant, adding a render.yaml left Render at 5/5.

Each variant had only 5 runs, on one setup in each of two repos. We chose setups with near-unanimous original results so that a change would stand out, and we tested the files themselves, not the templates or starter kits that might distribute them. Treat this as a demonstration that config files can change a pick, not an estimate of how often they will.

Some models showed unique preferences

The model effect was largest on the Go, FastAPI, and monorepo apps, and near zero on static sites and Next.js.

On the backend repos, Astra picked Render in 38 of 60 runs, compared with 59 of 180 for the other models. Its preference held across all 4 harnesses. GLM was less inclined toward Render: it recommended it in 9 of 60 backend runs, while choosing self-hosting in 24.

Astra also never picked Fly.io. Across the backend and full-stack repos, it made 0 Fly.io recommendations in 100 runs, compared with 42 in 300 for the other models.

Opus accounted for 7 of the 8 home-server picks, all on the FastAPI app. One of its answers pointed to a residential IP as a reason:

“A home server, Raspberry Pi or NAS.It has a residential IP…”

This is another interesting example of how carefully the model considered the repo context before making a recommendation.

On the two full-stack repos, Opus favored Railway. It recommended Railway in 31 of 40 runs, compared with 66 of 120 for the other models.

We gave every model the same repos and the same prompt, yet they still favored different hosts. These patterns passed significance testing after correcting for multiple comparisons (Holm-corrected permutation tests). Repo context had the largest influence overall, but the model still affected which provider an agent recommended.

Does the model “family” matter?

We used GPT-6.1 Sol to check whether Astra’s preferences would be similar to another OpenAI model.

Sol repeated 3 of Astra’s patterns:

Astra and Sol were the most similar pair across the full grid. Their disagreement was 1.8 percentage points above rerun noise, compared with 4.5 for the next closest pair, DeepSeek and Opus.

This suggests that some preferences may carry across models in the same family. We don’t know exactly the relationship between Astra and 6.1 Sol, and we only tested two models from the same lab and family, so we hesitate to make any strong claims. But it makes sense that related models may share some patterns and preferences.

Some setups are a coin flip

The same model could give multiple different recommendations for the same repo. We took the cells where no host won at least 4 of the first 5 runs and added up to 15 more runs, keeping the repo, model, and harness the same. We wanted to see whether a clear preference would emerge.

We added runs to 46 setups. When we compared the leaders in the first 5 runs with those in the additional runs, 24 kept the same leader and 16 changed. The remaining 6 involved ties.

On the most divided setups, even 20 runs didn’t produce a clear majority. But when the leader changed, the category of host mostly didn’t. Of the 16 changes, 13 swapped to a comparable peer, such as Render to Railway or Fly.io to Render. The 3 exceptions were all on FastAPI, switching between a managed platform and a self-hosted VPS.

The harness barely matters overall

Swapping the harness had the smallest effect of the 3 factors. In the original 640-run grid, changing the harness changed the top pick 37% of the time, compared with 32% for a plain rerun.

A few patterns did stand out. OpenCode and Pi made Vercel their top pick in 18 of 200 backend and full-stack runs, while Claude Code and Codex never picked it in their 200 runs. Pi was also more likely to recommend self-hosting on those repos: 29 of 100 runs, compared with 48 of 300 for the other harnesses. On static sites, Codex chose a host other than Cloudflare Pages or GitHub Pages in 14 of 60 runs, compared with 9 of 180 for the others.

Most of the harness effect was concentrated in Astra, whose web lookups varied by harness.

We also compared native Claude Code and Codex subscription runs with API runs using the same models. After accounting for rerun noise, we didn’t detect an additional difference between subscription and API recommendations.

Being mentioned isn’t being picked

Vercel was the most-mentioned host in the original 640-run grid. It came up in 510 answers and was the top pick in 29.

Agents named 7 vendors per answer on average, and most of those mentions weren’t endorsements:

AWS and Azure were never the top pick, and Cloud Run won 3 times. The winners were developer-focused platforms with simple setup.

On backend apps, Vercel appeared in 150 answers and was argued against in all 150. Netlify was often described as an equal option, just not the pick:

“Vercel and Netlify would work equally well if you already use one…”

Hetzner and DigitalOcean were mostly named as examples inside a generic VPS recommendation. AWS Lambda and Heroku came up almost only as warnings:

“Heroku — 30s request timeout, ephemeral disk.”

Models also differed in how many vendors they named. Astra averaged 3.6 per answer, Opus 9.4. So a mention count partly reflects how long a model’s list is.

What the models were thinking

We captured reasoning whenever models exposed it. Astra and Opus returned either summarized or encrypted reasoning in some cases, but both models produced readable reasoning in the majority of runs, at 93/160 and 120/160 respectively.

Vercel appeared in those traces much more often than it became the top pick. The table below covers only the runs with readable reasoning:

On backend apps, Vercel appeared in the reasoning of 84 runs and was picked in none of them. The agents raised concerns tied to features they identified in the repo, then recommended another host:

“Maybe app is Express backend; Vercel not ideal for serverless due long connections? Could use Render.”

“…plus an in-memory rate limiter, so serverless platforms like Vercel or Lambda won’t work.”

On the Next.js app, the custom cache handler came up in 48 of the 54 runs where the reasoning mentioned Vercel. Some passages went back and forth about whether Vercel supported it:

“Vercel (works but custom cacheHandler support on Vercel is limited — Vercel ignores custom cacheHandler? Actually Vercel does support cacheHandler since Next 14…”

“…using a custom cacheHandler on Vercel works but… Actually Next.js on Vercel supports custom cache handlers.”

Agents knew the old defaults but didn’t pick them

We counted hosting-vendor mentions in Stack Overflow answers from 2019 through 2024 and compared them with the agent recommendations. Stack Overflow here is a proxy for which vendors were being recommended by developers historically. The percentages below are co-occurrence rates for framework and vendor names in SO answers.

Vercel accounted for 82% of hosting-vendor mentions on Next.js. The agents also acknowledged that association: one called Vercel “the default reflex for Next.js.” Yet they picked Vercel in only 2 of 80 runs on our Next.js app.

Heroku accounted for 60% of the historical Express mentions. Agents picked Render in 63 of 80 Express runs instead. Heroku ended its free plans in November 2022, partway through the Stack Overflow window, which may partly explain the gap. Pricing and free-tier limits came up often in the answers (see below).

Across the original grid, Heroku appeared in 43 answers, with 38 framing it as a host to skip. It was never the top pick.

Railway, Render, and Fly.io together accounted for 0.2% of historical mentions on Next.js and 0.1% on Express. Cloudflare Pages and Workers accounted for 2% on React/Vite, while agents picked Cloudflare Pages in 67 of 80 runs on our React app.

For the plain HTML site, GitHub Pages was already a common name in the historical discussions: 20% of mentions, alongside Netlify at 21%. Agents picked GitHub Pages in 76 of 80 runs.

Historical popularity was a poor predictor. The agents’ picks only matched on the plain HTML site, where GitHub Pages was already near the top. Agents knew the old defaults, but didn’t follow them.

When agents search, they check prices

In the original 640-run grid, Astra accounted for 84 of the 88 runs with a web lookup attempt. The other 3 models made few attempts to look anything up.

Of the 142 search queries in that grid, 87 used a site:modifier to search a single vendor’s site. The word “pricing” appeared in 45 queries and “free” in 24. Queries included site:render.com pricing postgres key value starter 2026 and site:vercel.com/docs plans hobby non commercial personal use.

All 67 pages fetched in the original grid came from vendors’ own sites:

Most of the runs with search results focused on one vendor, which was also the eventual top pick.

Of the 55 runs where a lookup returned results, 43 looked up only the vendor the agent went on to recommend. Eleven covered 2 or more vendors, and 1 looked up only a vendor it didn’t pick. That shows a narrow lookup pattern, but the match alone doesn’t tell us when the decision was made.

Sol also attempted lookups in every run in Codex and Claude Code: 40 of 40 in each. It attempted them in 6 of 40 OpenCode runs and none in Pi. Of its 129 search queries, 73 used a site: modifier.

Sol’s reasoning gave us an example of a candidate being named before the search. In one FastAPI run, it said:

“I need to recommend Render as a simpler paid option compared to VPS… I need to verify the pricing to ensure it’s accurate.”

Here, the stated purpose was to check the price of a host Sol already had in mind.

Sol’s attempts weren’t all limited to one vendor. Of its 86 runs with a lookup attempt, 47 targeted only the eventual top pick, 33 targeted multiple vendors, and 6 targeted only another vendor.

Both models also recommended Render in Pi, where neither attempted a web lookup. Sol picked it in 8 of its 15 backend runs there.

In these examples, the lookup was a check on a host the model already favored.

What price did and didn’t decide

Looking up a price doesn’t tell us how much it mattered to the recommendation. We also checked how cost-related wording appeared in the answers. For the “near the top” counts, we used the first 700 characters, where the pick and its main reason usually appeared.

In the original grid, the word “free” appeared near the top in 196 of the 240 static-site answers. One Astro answer framed the choice around price and CDN quality:

“…the differentiator is just price and CDN quality rather than capability. Top pick: Cloudflare Pages– Free tier with unlimited…”

On backend and full-stack apps, cost-related wording appeared near the top in 117 of 400 answers. The agents could also recommend paying for hosting, or warn that a free tier wouldn’t meet the app’s requirements:

“Render (top pick for low ops)… Requires a paid always-on instance (free tier sleeps → breaks this app).”

The models differed in how they presented these considerations:

Cost came up most when the pick was self-hosting: near the top of 30 of the 77 app answers that picked it, compared with 22 of the 138 that picked Railway.

“…a single small VPS with Docker Compose… cheapest, one origin, zero code changes, full control. Alternative: Render/Railway for zero-ops (but costs more…)”

Astra and Sol, the two models that searched pricing pages, put cost near the top least often: 10 and 9 of 100 app answers.

For static sites, price was usually part of the headline reason. For apps, it came up less often, and when agents looked up prices, it was usually for a host they already favored.

What this means for vendors

Insert deployment configurations and hints as early in the chain as possible, since the repo context is by far the most influential input. Open-source templates and packages may be the point of highest leverage.

For example, on the Astro setup, a netlify.toml changed Opus’s pick from Cloudflare to Netlify. Starter kits, templates, and tools are places to distribute those files, with the build commands and deployment settings already specified.

Build your measurements around real apps that represent real use cases, not framework names alone. In almost every run, we saw agents closely examine the code’s specific quirks and requirements before making any recommendations. Include different models in those tests, and record both the model and harness for each run.

When an agent rejects your product, check its stated reason against your platform’s capabilities. The Vercel examples included uncertainty about custom cache support. Make deployment guides clear about supported features, and make pricing, free-tier limits, and usage restrictions easy to find. These were among the details agents searched for.

Track how your product is presented, not just whether it’s named. Report top picks separately from alternatives, provider examples, conditional recommendations, and rejections. A favorable alternative and a warning to avoid your product are different results, even if both increase the mention count. Naively counting mentions will give you an inaccurate picture.

Repeat each setup and measure how often your product actually gets recommended. A single answer doesn’t show whether a preference holds up, and some setups remained split after 20 runs.

Consistency differed between models:

Judge changes across repeated runs, with the repo, model, and harness recorded. Otherwise, normal run-to-run randomness can look like an improvement or decline in how agents recommend your product.

Limitations

The main findings come from pooled, matched comparisons that account for run-to-run variation, so they don’t depend on any one five-run setup. However, the per-cell sample size of 5 is small.

The vendor-specific findings have a narrower scope. We tested one repo per stack, with five initial runs per setup and additional repeats where agreement was low. Astra’s preference for Render on our backend repos doesn’t establish a general preference for every FastAPI, Express, or Go app.

Repo context includes everything in the codebase, not just the framework. Since we measured only one repo each, we can’t distinguish between the effect of any specific file or requirement. This study only measures their combined influence.

The simple prompt and mostly simple repos may also explain why big clouds rarely came up. A beginner-sounding question may invite beginner-friendly hosts, but a more complex app or prompt might change that.

A useful next step would be to test several repos per stack and vary the prompt, including developer personas and deployment requirements. That would show whether repo context remains the strongest influence, and whether the vendor preferences we recorded are consistent across contexts.