Purpose
Manual competitor research eats a full day and still misses the one thing that matters: the opening your competitor has left for you.
This SOP replaces that day of digging with a repeatable workflow. Claude, connected to Apify, reads a competitor across their website, social, sales pages, and podcast, then returns a structured brief. The final line surfaces the market gap that is hard to spot by hand, because you are usually too close to your own category to see it.
Who this is for
Founders, marketers, and operators who want fast, structured competitor intelligence without hiring an analyst or spending a day in browser tabs. No technical background is required. If you can paste a link and read a brief, you can run this.
Tools needed
- A Claude account at claude.ai. Connectors are available on paid plans, so confirm your plan supports them.
- An Apify account. A free account is enough to start. Apify runs on usage credits, so heavy scraping may need a paid tier.
- One competitor to start with: a website URL, a social handle, or a brand name.
Apify is a library of ready-made scrapers and automation tools called actors. Connected to Claude, it becomes the eyes: it pulls live data, and Claude reasons over what it finds.
Open Claude connectors
- Go to claude.ai and log in.
- Open Settings from the menu, then find the Connectors area.
- Select Add custom connector so you can point Claude at Apify directly.
Connect Apify
- In the connector dialog, paste the Apify MCP endpoint into the URL field.
- Give it a clear name like Apify so it is easy to spot later.
- Save, then authorize the connection. You will be sent to Apify to sign in and grant access, then returned to Claude.
You only do this once. After Apify is authorized it stays connected to your account. From then on you just switch it on inside the chat where you want to use it.
Start a fresh chat
- Open a new chat so the context is clean and focused on this one competitor.
- Confirm the Apify connector is toggled on for the conversation. Look for it in the tools or connectors control near the message box.
Add the competitor and the prompt
Paste one competitor, then the prompt below. Copy it as is, and swap in your competitor where prompted.
Keep the numbered structure. It tells Claude exactly what to return and keeps the brief scannable. The fifth line, the gap, is the payoff, so never drop it.
Let it run
- Send the prompt and watch Claude work. It will pick the right Apify actors and go read each source you named.
- If Claude asks to run a specific actor or needs a confirmation, approve it so it can pull live data.
- Give it a few minutes. It reads the website, the social profiles, the sales pages, and the podcast, then assembles the brief.
Read the gap
- Read the full teardown top to bottom. Positioning, offers, pricing, repeated promises, and true audience are all laid out.
- Go straight to the last section: the gap they are leaving open. This is the angle you can own that they are not defending.
- Turn the gap into action. Draft a positioning line, an offer, or a content angle that fills it before they do.
Run this across three or four competitors and patterns appear fast: shared blind spots become your clearest opening, and repeated promises show you where the category is crowded.
Lead generation use cases we have built with Apify
The same connection that powers competitor teardowns also builds prospect lists. Here are workflows we have set up with Apify for lead generation.
Local business lists from maps
Pull businesses in a city and niche from map listings, complete with name, category, phone, website, and rating. A ready outreach list for agencies and local service sellers.
Instagram prospect enrichment
Feed in a list of handles and pull public profile data: bio, follower range, contact link, and niche signals. Claude then scores who fits your ideal client and drafts the first line of outreach.
LinkedIn intent signals
Track companies posting specific roles or keywords, a strong signal of budget and priorities. Turn that stream into a warm list of accounts worth approaching this month.
Contact extraction from sites
Crawl a batch of company websites and pull public emails, socials, and contact pages into one clean sheet. Skips the manual copy and paste across dozens of tabs.
Competitor audience mining
Study who engages with a competitor's public content, then build a lookalike prospect list of people already interested in the category but not yet loyal to anyone.
Directory and marketplace pulls
Extract sellers, vendors, or members from niche directories and marketplaces. Ideal when your buyers cluster on one platform and you want the whole list, structured.
Only collect public data, and follow each platform's terms plus the privacy rules that apply to you, such as GDPR. Use scraped contacts for compliant, relevant outreach, not spam.
FAQ
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| Apify does not appear as a connector option | Plan does not support connectors, or the custom connector was not saved | Confirm your Claude plan includes connectors, then re-add the MCP endpoint and save before authorizing. |
| Claude replies without reading any live pages | The Apify connector is toggled off for that chat | Switch Apify on in the tools or connectors control near the message box, then resend the prompt. |
| A source comes back empty | The page blocks scraping, or the URL or handle was wrong | Double check the link, then ask Claude to try a different actor or skip that one source and continue. |
| The run stops partway | Apify credits ran out mid-task | Check your Apify usage. Top up credits or narrow the request to fewer sources per run. |
| The brief skips the market gap | The fifth prompt line was removed or the competitor input was thin | Restore the numbered structure and ask directly: what is the single gap this competitor leaves open. |
Want this built with you, live?
Join the live masterclass and watch competitor intelligence and lead generation workflows built end to end, so you can copy them the same day.
Save your seat →