Running a 12,700-Product Catalog as a Team of One

RigPlan is a planning platform for camper vans, RVs, trailers, and tiny homes. Users pick a battery chemistry and system voltage, then assemble a complete off-grid power build — parts list, cost, weight — from a curated product catalog. That catalog is the product. It currently holds 12,700 products (the number moves weekly as imports run), and I am the only person who maintains it.

This post is about the system that makes that sentence possible: an import pipeline that turns retailer product pages into structured draft records, and an agentic QA layer that decides which drafts are fit to publish. I built and operate both, and I'll describe what's actually in the code — including the parts that didn't work.

1. The problem

"Managing" a catalog product sounds like one job. It's six:

• A description that's accurate and readable — not scraped boilerplate, not empty.
• A photo that's actually the product, at usable resolution — not a retailer's logo or promo banner.
• Specs with clean labels and no duplicates, extracted from pages that format them ten different ways.
• Categorization into a two-level taxonomy that decides where the product appears in the planner.
• Planning options — the structured attributes the build math runs on. In RigPlan's schema, system voltage, battery capacity, wattage, and weight are not columns on the product; they live in an options subsystem (product_option_definitions → product_selected_options). A battery without a selected system-voltage is invisible to the electrical planner no matter how good its description is.
• Freshness — prices and availability come from retailer listings and go stale continuously. A product's displayed price is the minimum across its listings; a discontinued listing nobody notices is a broken promise to the user.

At 12,700 products, doing this by hand doesn't fail gracefully — it just doesn't happen. At three minutes per product, one full pass over the catalog is about 16 weeks of full-time work. The realistic manual failure mode is worse than slowness: you stop looking, and the catalog silently rots.

So the design constraint was never "make imports fast." It was: one person's attention is the scarcest resource in the system, so every component must spend as little of it as possible — and only where a machine's judgment can't be trusted.

2. The import pipeline

Products enter as draft, and only published products are publicly visible. Everything between those two states is the pipeline.

Ingestion is three Supabase Edge Functions coordinated through Postgres queues (pgmq): a sitemap scraper that reads the integrated retailer's sitemap index and enqueues product URLs in batches of 100; a URL processor that drains the queue on a pg_cron schedule (every few minutes, up to 5 retries on exponential backoff before a message is poisoned rather than retried forever); and a catalog sync that refreshes price and availability on a weekly plan, with a worker draining page chunks. The sync is deliberately update-only — it can never create products — and deliberately portable: when the retailer's bot protection started blocking cloud egress, the same update-only sync logic moved to a weekly job on a trusted host instead of the Edge cron.

For each URL, the processor needs page content to extract from. My first approach was pure scraping: Firecrawl fetches the page as Markdown and an LLM extracts structured fields. It worked, but it was the pipeline's biggest source of inaccuracy — Markdown rendered from an arbitrary product page is lossy in unpredictable ways. The fix came from noticing that my largest affiliate partner runs WooCommerce, which exposes a public Store API. Structured JSON beats scraped Markdown every time it's available, so the processor now tries the Store API first and falls back to Firecrawl (which returns Markdown and HTML; the pipeline feeds the LLM whichever is longer, capped at 125,000 characters). The fallback makes any product URL importable; the structured path makes imports accurate, and I'm building per-retailer adapters like it as affiliate partnerships expand.

Extraction is a single generateObject call through the Vercel AI SDK, with a Zod schema enforcing the output shape: name, brand, model number, description, flat specs, category and component type names, an https-only image URL, purchasability, and price in integer cents. Extraction runs on a small, cheap model (Haiku-class) — extraction against a schema is a cheap-model job. Description rewriting, where prose quality matters, gets a stronger Sonnet-class model. Matching model cost to task difficulty is most of the cost story.

The LLM's output is treated as a hostile witness. Before a draft lands, deterministic code runs four classes of checks:

• Dedupe. Incoming products match against existing records in strict precedence: source page URL, then canonical listing URL, then retailer + external key, then unique brand + model. Re-imports update; they don't clone.
• Category guard. The taxonomy has two "importer-magnet" buckets — Exterior and Interior — where lazy categorization accumulates. The guard tries to remap into a specific category using keyword hints; if it can't, the product is created as needs_review with the blocker reason written to its admin notes. Nothing lands in the vague buckets silently.
• Spec cleanup. Near-duplicate spec keys are collapsed, brand/model values hiding in specs get promoted to first-class fields, and specs already covered by a structured option are dropped.
• Image plausibility. Images smaller than 300px on either edge are rejected by reading the actual image headers. This rule exists because it didn't at first: 139 products shipped with page chrome — logos, badges — as their product photo before the check went in.

The output is a draft that is usually publishable as-is. My operating estimate is that 80–90% of imports need no correction beyond what the pipeline already did — an operator's estimate, not an instrumented metric, which I'll come back to in section 6. The rest cluster into predictable failures: products that don't belong on a van-build site at all, thin or oddly ordered specs, descriptions that mention the source merchant by name, missing images, and products stuck in the importer-magnet categories.

3. The agentic QA layer

Deciding which drafts are publishable is a judgment task, so it's done by an agent — through a deliberately narrow interface.

RigPlan exposes an MCP server: dozens of narrowly scoped tools, all data-only PostgREST operations, authenticated by scoped API keys (with OAuth 2.0 + PKCE layered on top). There is no exec_sql tool and never will be; the repo's standing instructions to any coding agent say so explicitly.

The agent driving it is an OpenClaw instance deployed to the cloud — in practice, a 24-hour employee. On an hourly schedule it sweeps the latest draft imports and QAs each one: the image, the description, the specifications, whether planning options were selected correctly (a battery with the wrong system-voltage is the canonical failure), whether the pricing checks out, and site fit — does this product belong on an off-grid build platform at all. It touches the catalog exclusively through the scoped MCP and REST endpoints; those credentials, not its prompt, are the real boundary of what it can do. The same agent is wired into Google Search Console and the site's APIs, so outside the QA loop I can ask it about analytics or anything else happening on the site. (One caveat for readers checking my work: the agent's schedule and wiring live in my ops environment, not the product repo — more on why that bothers me in section 6.)

Alerting has gone through one migration of its own: originally the agent reached me over Telegram whenever something was wrong or needed my attention; since moving it to the cloud, day-to-day communication runs through a Slack workspace where it sits alongside the rest of my tooling.

The QA decision rule is asymmetric by design: publish only when every check passes with high confidence; on any failure or any doubt, set needs_review with a recorded reason. Judging and fixing are separated, and the fixing side has an explicit autonomy boundary:

Allowed without me: writing a description where none exists, collapsing duplicate spec keys, migrating a spec value into a first-class field or structured option.

Requires me: rewriting an existing-but-poor description, swapping an image, anything failing site fit, and any deletion. Deletion additionally requires a products:delete scope the routine's credentials simply don't have — and the MCP server hides tools the caller isn't scoped for, so an unscoped agent doesn't even see a delete tool to be tempted by.

Below the agent-level rules sits a harder layer of guardrails in the MCP handlers themselves, which don't depend on the agent behaving:

• Product updates go through a strict field allowlist (additionalProperties: false); status is a three-value enum.
• The image URL field can't be set to an arbitrary string — it must be a managed storage URL that provably belongs to that product.
• Transitioning a product into needs_review is blocked unless a reason is recorded, so the escalation queue can't become a mystery pile.
• Every write lands in a product revisions table, so any agent action can be audited and reversed.

The escalations are where I actually spend my attention, and they're small and concrete. A recent example: batteries whose extracted system voltage fell outside the planner's canonical set of {12, 24, 48}. An agent-authored fix initially proposed treating 36V as valid — that PR was closed as factually wrong, because 36 isn't a voltage the planner can represent. The corrected migration maps only unambiguous cases (a 12V+24V multi-pack set becomes 24V) and explicitly leaves 6V, 16V, 36V, and 51.2V products unmapped pending a policy decision from me. "Do not invent a value" is written into the rules table. That episode is the whole system in miniature: the agent proposes, deterministic constraints bound what's possible, and genuinely ambiguous policy comes to a human.

4. The road not taken

The MCP server wasn't built for a scheduled agent. I built it so that Claude and I could QA the catalog together: I'd sit in a terminal with a Claude agent connected to the server and work through problem products one at a time — the agent proposing fixes, me approving each one.

The quality was fine. The economics weren't. Interactive QA didn't remove me from the loop; it made the loop more pleasant. I still went through every product, just with Claude's help — which was genuinely helpful and saved approximately zero time. The queue grew whenever I wasn't at the keyboard, coupling the catalog's health to my availability: the exact property I was trying to remove. What those sessions did produce was the autonomy boundary in section 3. Watching an agent work under supervision is how I learned what it could be trusted with — descriptions yes, image swaps no. That boundary wasn't designed up front; it was extracted from dozens of supervised sessions.

So the architecture shifted: the scheduled OpenClaw agent took over QA, driving the same endpoints — and it turned out to be better at the job than me steering Claude by hand: more consistent, and it never gets bored on product four hundred. I want to be precise about what got replaced, because "I replaced the MCP prototype with agentic QA" — the way I've sometimes summarized it — is sloppy. The MCP server wasn't replaced; it's the substrate the agent runs on, and the interactive workflow still exists for working the escalation queue. What got replaced was me as the scheduler and the default reviewer. The tradeoff is real: a conservative batch judge publishes less aggressively than I would and escalates things I'd have waved through. I accept that cost, because the currency it saves — my attention — is the one I can't buy more of.

5. Results and operations

Numbers I can support, and their limits:

• Catalog size: 12,700 products in production, maintained by one person.
• Cadence: import processor every few minutes; hourly agentic QA sweep over new drafts; price/availability refresh weekly — from Supabase Edge or the trusted host, depending on what the retailer's bot protection allows; production HTTP canary every 30 minutes.
• Publish-ready rate: 80–90% of imports publish without correction, by my operational estimate. The system does not currently record this — there is no pass-rate metric in the codebase, and I consider that a defect (section 6).
• Escalation and cost: likewise not instrumented. The cost structure is at least designed sensibly — cheap model for schema-constrained extraction, better model only for prose — but "designed to be cheap" and "measured to be cheap" are different claims, and I can only make the first.

The agent's value shows up outside the QA loop too. When I was running paid acquisition, the site saw 100–200 users a day. Asking the OpenClaw agent to dig into Search Console showed that much of that ad traffic came from regions RigPlan doesn't serve — so I turned the ads off rather than keep paying for vanity numbers, and the site runs on organic traffic while I build out SEO and affiliate coverage. That's a decision I might have deferred for months if the analysis had required a manual deep-dive.

Day-to-day, operating this looks like: checking Slack for anything the agent flagged overnight, a pass over the escalation queue (every item arrives with a recorded reason, so triage is fast), and occasional incident response. The instructive incident so far: the catalog sync's discontinue sweep — which marks listings not seen in a crawl as discontinued — falsely discontinued about 11,900 listings after a partial run. Recovery was a restore plus new guards, and the sync now logs a marked_discontinued count per run specifically so that number gets watched. The lesson wasn't "automation is dangerous"; it was that any automated sweep needs its blast radius bounded and its most destructive counter surfaced as a first-class metric.

6. What I'd do differently

Instrument the funnel from day one. The most embarrassing admission in this post is that my headline metrics — publish-ready rate, escalation rate, cost per product — are operator estimates rather than rows in a table. A qa_runs table with counts per verdict would have cost an afternoon, and "80–90%" would be a query instead of a guess.

Make the QA verdict a first-class record. Subtler: the status-change tool accepts a reason argument, but it's only logged server-side — the durable review reason travels through a separate admin-notes mechanism. The split means the QA agent's verdicts aren't queryable history. The revisions table audits what changed; I'd also want why, structurally.

No silent zeros in user-facing math. Two schema decisions used to leak into build totals: a listing with a null price summed as $0, and a product with no weight option contributed 0 lb — both wrong in the direction that flatters the build. Price is largely fixed now: a missing price is explicit, and the plan rollup tracks it as an unknown rather than a free part. Weight unknowns still need the same honesty treatment. The lesson stands either way: represent unknowns as unknowns, because the QA layer flagging them after the fact is a patch, not a fix.

Start per-retailer structured imports sooner. Scrape-and-extract generalized to any URL, which felt like the right generality to build first — but the accuracy jump from the one structured Store API adapter says I had the order backwards. Structured-first per retailer, scraping as the true fallback, is now the roadmap.

Keep the automation auditable from the repo. The OpenClaw agent's schedule and alert wiring live on a separate ops host, which was expedient but means the product repo can't testify to how the catalog is operated — I felt this acutely while fact-checking this post against the code. Operational configuration deserves the same review-and-version discipline as migrations.

The honest summary: the LLM calls were the easy part. The system earns its keep through the boring machinery around them — queues with poison-message handling, Zod schemas, field allowlists, scoped credentials, revision trails, and a decision rule that routes doubt to a human. That's what lets one person operate a 12,700-product catalog without it operating them.