StackEngine CMS

Tag: agentic seo pipeline

  • How to Build an Agentic SEO Pipeline with Claude Code and MCP

    How to Build an Agentic SEO Pipeline with Claude Code and MCP

    An agentic SEO pipeline uses Claude Code as the orchestrator, MCP servers as the data layer, and Airtable as the storage backend. Instead of switching between keyword tools, spreadsheets, and AI chat windows, a single system runs the full research loop — from brand analysis through SERP deep dives — in one session. This post walks through the 13-step architecture behind SearchScope, part of the broader agentic SEO approach we’ve been building at StackEngine.

    Why MCP Changes the Architecture

    TL;DR: MCP lets Claude Code call external APIs as native tools, which means your AI agent can read, write, and act on live data without custom middleware.

    Model Context Protocol (MCP) turns external services into tool calls that Claude Code can invoke directly. DataForSEO becomes a keyword research tool. Airtable becomes a database. No API wrappers. No intermediate scripts. Claude Code reads the MCP tool definitions, understands what each one does, and calls them as part of its reasoning loop.

    This matters because the bottleneck in traditional SEO research isn’t the data — it’s the assembly. You pull keywords from one tool, check SERPs in another, paste results into a spreadsheet, then manually cross-reference. An MCP-based pipeline collapses that into a single orchestrated flow where Claude Code decides what to query, interprets the results, and writes structured records directly to Airtable.

    The practical difference: what used to take an afternoon of tab-switching now runs in one Claude Code session. And because the agent handles the data plumbing, you spend your time on decisions — which keywords matter, which clusters to pursue, where the gaps are.

    The 13-Step Pipeline

    TL;DR: The pipeline moves through four phases — setup, broad research, clustering, and deep dives — each building on the data from the previous phase.

    Here’s the full sequence:

    1. Verify MCP servers — Confirm DataForSEO and Airtable MCP connections are live. If either is down, the pipeline stops early instead of failing mid-run.
    2. Run brand profile — Analyze the target website, extract core topics, and determine the brand’s current positioning. This grounds every keyword decision in what the brand actually does.
    3. Expand topics into keywords — Use DataForSEO to generate keyword ideas, suggestions, and related terms for each core topic.
    4. Write keywords to Airtable — Store raw keyword data with search volume, difficulty, and intent classification. Claude Code creates the Airtable table schema through MCP if it doesn’t exist yet.
    5. Landscape analysis — Pull SERP data for priority keywords. Identify who ranks, what content types dominate, and where the gaps are.
    6. AI platform assessments — Query ChatGPT, Perplexity, Claude, and Gemini with the same prompts to see which brands and pages they cite. This is AI platform visibility tracking applied at the keyword level.
    7. Cluster results — Group keywords into semantic clusters based on SERP overlap, topic similarity, and intent alignment.
    8. Score and categorize — Rank clusters by opportunity: search volume, difficulty, AI citation gaps, and alignment with the brand’s existing content.
    9. Write results to Airtable — Store clusters with scores, categories, and priority rankings.
    10. Deep dive: full SERP analysis — For priority clusters, pull full SERP results with featured snippets, People Also Ask, and site links.
    11. Deep dive: AI platform analysis — Detailed AI visibility checks for priority terms — who gets cited, in what context, and how often.
    12. Deep dive: Google AI Mode check — Test whether Google’s AI overview triggers for priority keywords, and what content it pulls from.
    13. Write updated cluster records — Push deep dive data back to Airtable, enriching the cluster records with SERP and AI citation details.

    Steps 1–4 are setup and broad collection. Steps 5–9 are analysis and organization. Steps 10–13 are targeted deep dives on the clusters that matter most.

    How Claude Code Orchestrates the Flow

    TL;DR: Claude Code acts as the decision layer — it doesn’t just execute steps sequentially, it reads intermediate results and adjusts what happens next.

    The key architectural insight: Claude Code isn’t running a fixed script. It’s making judgment calls at each step based on what the data shows.

    After step 3, if DataForSEO returns 2,000 keyword ideas, Claude Code doesn’t blindly process all of them. It filters by relevance, deduplicates, and selects the set worth writing to Airtable. After step 5, if the landscape analysis shows a keyword cluster is dominated by massive authority sites with no realistic entry point, Claude Code deprioritizes it.

    Sub-agents handle parallelizable work. When running AI platform assessments in step 6, separate sub-agents can query each platform simultaneously instead of sequentially. This is where the architecture went through three iterations — early versions hit context window limits trying to hold all the data in a single agent’s memory. The solution: specialized sub-agents that handle discrete tasks and write results to Airtable, keeping the orchestrator’s context lean.

    The pattern looks like this:

    Orchestrator (Claude Code)
├── Brand Analyzer        → reads website, writes brand profile
├── Keyword Expander      → calls DataForSEO, writes to Airtable
├── SERP Analyzer         → pulls rankings, identifies gaps
├── AI Visibility Checker → queries 4 AI platforms in parallel
├── Cluster Engine        → groups and scores keywords
└── Deep Dive Agents      → detailed analysis on priority clusters

    Each sub-agent gets a focused task, a clear input, and writes its output to Airtable. The orchestrator reads those results and decides what runs next.

    What Three Architecture Versions Taught Us

    TL;DR: Context windows are the hard constraint. The system works when you treat Airtable as shared memory and keep each agent’s scope narrow.

    Version 1 tried to run everything in a single Claude Code session. It worked for small keyword sets. At scale, the context window filled up with raw API responses, and the agent lost track of earlier analysis.

    Version 2 introduced sub-agents but kept intermediate results in memory. Better, but still fragile — if one sub-agent returned unexpectedly large results, it could crowd out the orchestrator’s reasoning space.

    Version 3 — the current architecture — treats Airtable as the system’s shared memory. Each agent writes its results to Airtable immediately. The orchestrator reads summaries, not raw data. This mirrors how traditional keyword research breaks down at scale — the difference is that the agentic version solves the assembly problem instead of just generating more data.

    The takeaway for anyone building similar systems: design for the context window from day one. If an agent needs to hold more than a few hundred rows of data to make a decision, restructure so it doesn’t.

    Getting Started

    TL;DR: You need Claude Code, two MCP servers, and an Airtable base. The pipeline creates its own schema.

    The prerequisites are straightforward:

    • Claude Code — the orchestrator
    • DataForSEO MCP server — provides keyword data, SERP results, and AI platform queries
    • Airtable MCP server — provides structured data storage with API access
    • An Airtable base — the pipeline creates tables and fields through MCP, so you start with an empty base

    You don’t need to predefine the Airtable schema. Claude Code creates the tables it needs — keywords, clusters, SERP data, AI citations — through the Airtable MCP. This is one of the things that surprised us: the agent can design its own data model based on what it’s collecting. You review and adjust, but the initial schema is functional out of the box.

    The pipeline fits within the broader agentic SEO framework where AI agents handle research, monitoring, and analysis — and you handle the decisions about what to pursue. The hidden gem keywords that emerge from steps 7–8 are often the most valuable output: clusters with real search volume that competitors haven’t targeted because they require cross-referencing data that manual processes can’t assemble efficiently.

    Start with a single topic area. Run the pipeline end to end. Review what it finds. Then expand.

  • Traditional Keyword Research vs Agentic SEO: What Actually Changes

    Traditional Keyword Research vs Agentic SEO: What Actually Changes

    You open a keyword tool, pull a list, sort by volume, export to a spreadsheet, and start writing content modeled on what already ranks. That’s been the workflow for a decade. Agentic SEO replaces most of it — not with a better keyword tool, but with an AI agent that reasons through your data and gives you strategy instead of spreadsheets.

    This post breaks down exactly what changes when you move from traditional keyword research to an agentic approach — what you lose, what you gain, and where the real shift happens.

    Traditional Keyword Research Gives You Data

    TL;DR: Keyword tools are data retrieval systems. They surface numbers. They don’t tell you what to do with them.

    A typical keyword research session looks like this: you enter a seed term into Ahrefs, Semrush, or Google Keyword Planner. You get back a list — search volume, keyword difficulty, CPC, trend data. Maybe you expand into related terms or questions. Then you export everything to a spreadsheet and start making decisions manually.

    The output is a flat list of keywords with metrics attached. What you do next — which keywords to target, what content to create, how to position against competitors — is entirely on you.

    That’s not a flaw. These tools are good at what they do. But they’re data retrieval systems, not strategy systems. They answer “what are people searching for?” They don’t answer “where should your brand show up in search?”

    The gap between those two questions is where most SEO work actually happens. And it’s almost entirely manual.

    What You Get What You Don’t Get
    Search volume per keyword Which keywords fit your brand positioning
    Keyword difficulty scores Whether you can realistically compete
    CPC and trend data How to cluster keywords into a content plan
    Related keyword suggestions What content format matches the intent
    SERP feature indicators How AI platforms treat the topic

    Agentic SEO Gives You Strategy

    TL;DR: An AI agent doesn’t just pull data — it reasons through your brand context, competitive landscape, and audience to produce ranked, justified recommendations.

    An agentic SEO system like SearchScope works differently. Instead of returning a keyword list, it runs a pipeline: brand profiling, topic suggestions, keyword expansion, clustering, scoring, SERP analysis, AI platform assessments, and deep dive reports.

    The critical difference: the agent has your brand context loaded. It knows your audience, your topics, your voice, your competitive positioning. When it evaluates a keyword, it’s not just checking volume and difficulty. It’s reasoning through whether that keyword makes sense for your brand to pursue.

    The output isn’t a spreadsheet. It’s a set of recommendations with reasoning attached — why this keyword cluster matters, what the competitive landscape looks like, where the content gaps are, and what format the content should take.

    Here’s what that looks like in practice:

    • Topic suggestion: The agent proposes topics based on your brand profile and audience, not just search volume
    • Keyword clustering: Related keywords get grouped into content themes automatically, with a parent-child hierarchy
    • Competitive analysis: The agent checks what’s currently ranking, evaluates the content quality, and identifies gaps you can fill
    • AI platform assessment: It checks how AI platforms like ChatGPT, Perplexity, and Claude treat the topic — something no traditional keyword tool does
    • Strategic scoring: Keywords get scored against your specific brand fit, not just generic difficulty metrics

    The Real Shift: From “What Keywords Exist” to “Where Should We Show Up”

    TL;DR: Traditional tools answer what’s out there. Agentic systems answer what’s right for you — and why.

    The fundamental shift isn’t about better data. It’s about moving from data retrieval to strategic reasoning.

    Traditional keyword research asks: “What are people searching for in this topic?” You get a list. You interpret it yourself. You decide what to write based on your own analysis of the numbers.

    Agentic SEO asks: “Given this brand, this audience, and this competitive landscape — where should we show up in search?” The agent interprets the data, weighs it against your context, and delivers recommendations with reasoning.

    This matters because the hard part of SEO was never finding keywords. Tools have been good at that for years. The hard part was figuring out which keywords to pursue, in what order, with what content, positioned against which competitors. That’s strategy. And until recently, it required a human analyst spending hours in spreadsheets.

    An AI agent compresses that analysis. Not perfectly — you still review and direct. But the first draft of your content strategy comes with reasoning attached, not just numbers.

    What an Agent Catches That a Keyword Tool Misses

    TL;DR: Agents find opportunities in the gaps between data points — brand-fit keywords, underserved intents, and hidden gems that surface from reasoning, not sorting.

    Keyword tools rank by volume and difficulty. Sort descending, pick from the top. That’s a reasonable heuristic, but it misses opportunities that only show up when you reason across multiple dimensions.

    An agentic system catches things like:

    • Brand-fit keywords with low competition that a human might skip because the volume looks small — but the audience match is exact
    • Content gaps where competitors have thin or outdated content and a well-structured piece could rank quickly
    • AI platform opportunities where a topic gets asked frequently in ChatGPT or Perplexity but has weak source material available for citation
    • Cluster opportunities where targeting three related long-tail keywords with one piece of content is more effective than chasing a single high-volume head term
    • Intent mismatches where the current top results don’t actually answer what the searcher wants

    These aren’t visible in a keyword spreadsheet. They emerge from the agent reasoning across data points — cross-referencing SERP content quality, AI platform coverage, brand positioning, and audience need simultaneously.

    When to Use Which Approach

    TL;DR: Use keyword tools for quick data lookups. Use an agentic system when you need a content strategy, not a keyword list.

    This isn’t an either-or decision. Keyword tools still have a place.

    Use traditional keyword tools when you need:

    • Quick search volume checks for a specific term
    • CPC data for paid search planning
    • Historical trend data for a known keyword
    • A fast export for a client report

    Use an agentic SEO system when you need:

    • A content strategy for a new topic cluster
    • Competitive gap analysis with actionable recommendations
    • AI platform visibility assessment
    • Keyword prioritization based on brand fit, not just volume
    • A full pipeline from research to content briefs

    The tools give you ingredients. The agent gives you the recipe — and explains why it chose those ingredients for your kitchen.

    What This Means for Your Workflow

    The practical change is this: keyword research stops being a standalone step that produces a spreadsheet. It becomes part of a pipeline that produces strategy.

    If you’re building AI systems for search — and documenting the process as you go — the agentic approach fits naturally. The agent reasons through your data the same way you would, just faster and with more consistency across large keyword sets.

    The parent guide on Agentic SEO covers the full framework. Start there if you want the complete picture of how AI agents are replacing the traditional keyword research workflow.