build-scrape

# Build Scrape Skill

Scrapes a target URL with Playwright headless Chromium, enforces ethical guardrails (robots.txt check + copyright disclaimer + interactive confirmation), sanitises all content with AI-generated placeholder text and dimensions, and produces a clean HTML/CSS directory that feeds directly into the `build-visual` FSE pipeline - as if the user had exported a Figma or Canva design.

**Critical sequencing:** This skill runs AFTER `build-scaffold` and `build-mcp` complete (WordPress is installed, MCP adapter is active) and BEFORE `build-visual` Sections 2-6 (CSS token extraction, theme scaffolding, font download, activation, SETUP.md).

This skill expects the following variables to already be set by the calling command:

- `SOURCE_URL` - the URL to scrape (set by COMMAND.md Section 1 argument parsing)
- `BUILD_DIR` - absolute path to the build directory (set by build-scaffold Section 2)
- `SLUG` - build slug (set by COMMAND.md Section 1)
- `SITE_TITLE` - site title (set by COMMAND.md Section 1)
- `PLUGIN_DIR` - absolute path to the CoWork plugin directory
- `WP` - WP-CLI command prefix (set by build-scaffold Section 4)

---

## Section 0: Prerequisite Check

Check that Playwright and robots-parser are installed before proceeding. Create the temp scrape directory and register an EXIT trap for cleanup.

```bash
echo "[Build] build-scrape: checking prerequisites..."

# Check Playwright
if ! node -e "require('playwright')" 2>/dev/null; then
  echo ""
  echo "ERROR: Playwright is required for URL clone mode."
  echo "  Install with:"
  echo "    npm install playwright"
  echo "    npx playwright install chromium"
  echo ""
  exit 1
fi

# Check robots-parser
if ! node -e "require('robots-parser')" 2>/dev/null; then
  echo ""
  echo "ERROR: robots-parser is required for URL clone mode."
  echo "  Install with:"
  echo "    npm install robots-parser"
  echo ""
  exit 1
fi

echo "[Build] Prerequisites: playwright OK, robots-parser OK"

# Create temp scrape directory
SCRAPE_DIR="/tmp/scrape_${SLUG}_$$"
mkdir -p "$SCRAPE_DIR"

# EXIT trap: always clean up temp directory (Pitfall 3)
trap "rm -rf '$SCRAPE_DIR'" EXIT

echo "[Build] Scrape temp directory: $SCRAPE_DIR"
```

---

## Section 1: Pre-Scrape Guard (robots.txt + Copyright Disclaimer + Confirmation)

Fetch and parse robots.txt, display the review banner with the copyright disclaimer, and require explicit interactive confirmation before any page request to the target site.

**IMPORTANT:** No network requests to `$SOURCE_URL` are made until after the user confirms.

### 1a: robots.txt Check

```bash
echo "[Build] Checking robots.txt for $SOURCE_URL..."

ROBOTS_CHECK=$(SOURCE_URL="$SOURCE_URL" node - <<'NODEOF'
const robotsParser = require('robots-parser');
const https = require('https');
const http  = require('http');
const { URL } = require('url');

const sourceUrl = process.env.SOURCE_URL;
const base      = new URL(sourceUrl);
const robotsUrl = `${base.origin}/robots.txt`;

const client = robotsUrl.startsWith('https') ? https : http;

const req = client.get(robotsUrl, { timeout: 10000 }, (res) => {
  let body = '';
  res.on('data', d => body += d);
  res.on('end', () => {
    const robots   = robotsParser(robotsUrl, body);
    const allowed  = robots.isAllowed(sourceUrl, 'WPCoWork/1.0');
    const disallowed = robots.isDisallowed(sourceUrl, 'WPCoWork/1.0');
    process.stdout.write(JSON.stringify({
      robots_url:  robotsUrl,
      source_url:  sourceUrl,
      allowed:     allowed !== false,
      disallowed:  disallowed === true,
      raw_preview: body.substring(0, 500)
    }) + '\n');
  });
});

req.on('error', () => {
  // No robots.txt or unreachable - treat as allowed per RFC 9309
  process.stdout.write(JSON.stringify({
    allowed: true,
    disallowed: false,
    error: 'No robots.txt found or unreachable - treating as allowed'
  }) + '\n');
});

req.on('timeout', () => {
  req.destroy();
  process.stdout.write(JSON.stringify({
    allowed: true,
    disallowed: false,
    error: 'robots.txt fetch timed out - treating as allowed'
  }) + '\n');
});
NODEOF
)

ROBOTS_ALLOWED=$(echo "$ROBOTS_CHECK" | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print('true' if d.get('allowed', True) else 'false')")
```

### 1b: Display Review Banner and Copyright Disclaimer

```bash
echo ""
echo "┌─────────────────────────────────────────────────────────────────────────┐"
echo "│                    URL CLONE MODE - REVIEW REQUIRED                    │"
echo "└─────────────────────────────────────────────────────────────────────────┘"
echo ""
echo "Source URL: $SOURCE_URL"
echo ""

if [ "$ROBOTS_ALLOWED" = "false" ]; then
  echo "WARNING: robots.txt for this site disallows scraping of this URL."
  echo "         Proceeding is your responsibility under applicable laws."
else
  echo "robots.txt check: OK (crawling not restricted for this URL)"
fi

echo ""
echo "IMPORTANT: This tool reproduces visual layout structure only."
echo "           All text and images are replaced with AI-generated placeholders."
echo "           You are responsible for ensuring your use complies with"
echo "           applicable copyright and terms of service laws."
echo ""
echo "The generated site will contain:"
echo "  - Source site's color palette and typography (flagged in SETUP.md)"
echo "  - AI-generated fictional placeholder content (no scraped text)"
echo "  - Dimension-matched placeholder images (no source images downloaded)"
echo ""
```

### 1c: Interactive Confirmation Gate

```bash
# TTY check required - read -p hangs in non-interactive contexts (Pitfall 6)
if [ -t 0 ]; then
  read -p "Proceed with URL clone? (y/N): " USER_CONFIRM
else
  echo "ERROR: URL clone mode requires an interactive terminal (TTY not detected)."
  echo "       Run this command in an interactive shell, not a piped or CI context."
  echo ""
  exit 1
fi

if [ "$USER_CONFIRM" != "y" ] && [ "$USER_CONFIRM" != "Y" ]; then
  echo ""
  echo "URL clone cancelled."
  echo ""
  exit 0
fi

echo ""
echo "[Build] User confirmed. Recording robots status..."

# Store robots status for scrape.json update in Section 3
if [ "$ROBOTS_ALLOWED" = "true" ]; then
  ROBOTS_STATUS="allowed"
else
  ROBOTS_STATUS="disallowed-override"
fi

echo "[Build] robots_status=$ROBOTS_STATUS"
echo "[Build] Beginning Playwright scrape..."
```

---

## Section 2: Playwright Scraping

Invoke `scraper.js` as a Node.js subprocess. Parse the JSON result and handle homepage failures as build-aborting errors. Inner page failures are already handled gracefully inside `scraper.js` (they appear in `failed_pages`).

```bash
SCRAPE_RESULT=$(node "$PLUGIN_DIR/skills/build-scrape/scraper.js" "$SOURCE_URL" "$SCRAPE_DIR" 2>&1)

# Check whether the scrape succeeded
SCRAPE_SUCCESS=$(echo "$SCRAPE_RESULT" | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print('true' if d.get('success') else 'false')" 2>/dev/null || echo "false")

if [ "$SCRAPE_SUCCESS" != "true" ]; then
  SCRAPE_ERROR=$(echo "$SCRAPE_RESULT" | python3 -c \
    "import json,sys; d=json.load(sys.stdin); print(d.get('error','Unknown scrape error'))" 2>/dev/null || echo "$SCRAPE_RESULT")
  echo ""
  echo "ERROR: Failed to scrape $SOURCE_URL"
  echo "       $SCRAPE_ERROR"
  echo ""
  exit 1
fi

SCRAPE_MANIFEST="$SCRAPE_DIR/scrape.json"

# Log scrape summary
PAGES_COUNT=$(python3 -c \
  "import json; d=json.load(open('$SCRAPE_MANIFEST')); print(len(d.get('pages',[])))" 2>/dev/null || echo "?")
CSS_SIZE=$(python3 -c \
  "import json; d=json.load(open('$SCRAPE_MANIFEST')); print(d.get('css_size_bytes',0))" 2>/dev/null || echo "?")
FEATURES=$(python3 -c \
  "import json; d=json.load(open('$SCRAPE_MANIFEST')); print(', '.join(d.get('dynamic_features',[])) or 'none')" 2>/dev/null || echo "?")
IMAGES_COUNT=$(python3 -c \
  "import json; d=json.load(open('$SCRAPE_MANIFEST')); print(len(d.get('images',[])))" 2>/dev/null || echo "?")

echo "[Build] Scrape complete:"
echo "[Build]   Pages scraped:     $PAGES_COUNT"
echo "[Build]   CSS size (bytes):  $CSS_SIZE"
echo "[Build]   Dynamic features:  $FEATURES"
echo "[Build]   Images noted:      $IMAGES_COUNT (dimensions only - no downloads)"
```

---

## Section 3: Content Sanitisation (the coding agent In-Context Step)

This section is executed by the coding agent as an in-context judgment step. the coding agent reads each HTML file written by `scraper.js` and rewrites it with fictional placeholder content. No external tool is called - the coding agent performs all replacements using its own language capability.

**the coding agent performs the following replacements on each `.html` file in `$SCRAPE_DIR`:**

### 3a: HTML Content Replacements (per file)

1. **Text content replacement:** For all text inside `<p>`, `<span>`, `<li>`, `<td>`, `<h1>`, `<h2>`, `<h3>`, `<h4>`, `<h5>`, `<h6>` tags - replace with fictional placeholder text that matches the approximate word count. Use generic business/site copy relevant to a generic version of the detected site type (e.g. professional services, retail, portfolio, blog). Do NOT use Lorem Ipsum - use readable placeholder prose ("Providing expert solutions for modern businesses", "Our team brings over a decade of experience", etc.).

2. **Brand name replacement:** Detect the source brand name from: `<title>`, `<meta name="description">`, logo `<img alt="...">`, prominent `<h1>`. Replace all occurrences of the detected brand name throughout the file with "Your Brand" or "Company Name".

3. **Logo image replacement:** For `<img>` elements that function as the site logo (identified by: alt text containing brand name, `class*="logo"`, `id*="logo"`, inside `<a>` pointing to `/`): remove the `src` attribute, add `data-placeholder="logo"` attribute. Example: `<img class="site-logo" data-placeholder="logo" alt="Company Name Logo">`.

4. **Navigation link text replacement:** For all `<a>` elements inside `<nav>`, `<header>`, or elements with `role="navigation"`: replace the link text with generic navigation labels (Home, About, Services, Work, Portfolio, Blog, Contact, Team, Pricing, FAQ). Keep `href` attributes intact - layout preservation depends on the link structure remaining valid.

5. **Script tag removal:** Remove all `<script>` tags (both inline and external `<script src="...">` tags). JS is not needed for design token extraction and its presence may interfere with static analysis in the build-visual pipeline.

6. **External stylesheet link removal:** Remove `<link rel="stylesheet">` tags that point to the source domain or its CDN (the CSS is already captured in `styles/main.css`). Keep `<link rel="stylesheet">` tags for Google Fonts (these are useful font references that the build-visual pipeline can leverage).

7. **Brand-identifying meta tag removal:** Strip the following `<meta>` tags to prevent brand information from leaking into the theme: `og:title`, `og:site_name`, `og:description`, `twitter:title`, `twitter:site`, `twitter:description`, `name="description"` (the original site description). Replace the `<title>` tag content with `{SITE_TITLE}` (the user's chosen site title from the build arguments).

8. **Dynamic feature placeholder insertion:** Where a dynamic feature was detected on a page (cross-reference with that page's `features` array in `scrape.json`), insert a styled placeholder `<div>` immediately after the detected element's parent section. Use this format:

   ```html
   <div style="border: 2px dashed #aaa; padding: 2rem; margin: 1rem 0; text-align: center; color: #666; background: #f9f9f9;">
     [Search functionality - manual setup required. See SETUP.md for recommended plugins.]
   </div>
   ```

   Use the appropriate feature name in the placeholder text (Search, Contact Form, E-commerce Cart, Member Login, Map, Video Embed, Social Feed).

**the coding agent writes the sanitised HTML back to the same file path**, overwriting the raw scraped version.

### 3b: CSS Sanitisation (Python script - automated)

Strip `content: "..."` CSS declarations that contain text (Pitfall 5):

```bash
python3 -c "
import re, sys
css_file = '$SCRAPE_DIR/styles/main.css'
with open(css_file, 'r', errors='ignore') as f:
    css = f.read()
# Replace content: '...' or content: \"...\" with content: \"\"
css = re.sub(r'content\s*:\s*[\"\\']([^\"\\'>{]+)[\"\\']', 'content: \"\"', css)
with open(css_file, 'w') as f:
    f.write(css)
print('[Build] CSS content: declarations sanitised')
"
```

### 3c: Update scrape.json with robots_status

```bash
python3 -c "
import json
manifest_path = '$SCRAPE_DIR/scrape.json'
with open(manifest_path, 'r') as f:
    manifest = json.load(f)
manifest['robots_status'] = '$ROBOTS_STATUS'
with open(manifest_path, 'w') as f:
    json.dump(manifest, f, indent=2)
print('[Build] scrape.json updated with robots_status=$ROBOTS_STATUS')
"
```

---

## Section 4: SETUP.md URL Clone Appendix

After `build-visual` Section 6 writes the base `SETUP.md`, this section **appends** URL-clone-specific content. This section is run by COMMAND.md Section 3c after the build-visual pipeline completes.

the coding agent reads `$SCRAPE_DIR/scrape.json` and generates the following markdown, then appends it to `$BUILD_DIR/SETUP.md`:

```bash
CLONE_TIMESTAMP=$(date -u +"%Y-%m-%d %H:%M UTC")

python3 - <<PYEOF
import json, sys, os

manifest_path = os.environ.get('SCRAPE_MANIFEST', '$SCRAPE_DIR/scrape.json')
build_dir     = os.environ.get('BUILD_DIR', '$BUILD_DIR')
setup_md_path = os.path.join(build_dir, 'SETUP.md')
clone_ts      = '$CLONE_TIMESTAMP'

with open(manifest_path, 'r') as f:
    manifest = json.load(f)

source_url   = manifest.get('source_url', 'unknown')
dynamic_feats = manifest.get('dynamic_features', [])
pages        = manifest.get('pages', [])
failed_pages = manifest.get('failed_pages', [])
spa_fallback = manifest.get('spa_fallback_used', False)

# Dynamic feature to WP.org plugin mapping (all 7 features)
PLUGIN_MAP = {
    'ecommerce':    ('E-commerce (cart, checkout)',       'WooCommerce',                              'wordpress.org/plugins/woocommerce/'),
    'search':       ('Site search',                       'WordPress built-in search (no plugin needed)', ''),
    'login':        ('Member login / registration',       'Theme My Login',                           'wordpress.org/plugins/theme-my-login/'),
    'contact-form': ('Contact form',                      'WPForms Lite',                             'wordpress.org/plugins/wpforms-lite/'),
    'form':         ('Generic form',                      'WPForms Lite',                             'wordpress.org/plugins/wpforms-lite/'),
    'maps':         ('Map embed',                         'WP Google Maps',                           'wordpress.org/plugins/wp-google-maps/'),
    'video-embed':  ('Video embed',                       'WordPress built-in Video block (no plugin needed)', ''),
    'social-feed':  ('Social media feed',                 'Smash Balloon Social Post Feed',           'wordpress.org/plugins/custom-facebook-feed/'),
}

# Build which pages each feature was found on
feature_pages = {}
for page in pages:
    for feat in page.get('features', []):
        feature_pages.setdefault(feat, []).append(page.get('url', 'unknown'))

appendix = []
appendix.append('')
appendix.append('---')
appendix.append('')
appendix.append('## URL Clone Information')
appendix.append('')
appendix.append(f'**Cloned from:** [{source_url}]({source_url})')
appendix.append(f'**Cloned at:** {clone_ts}')
appendix.append('')
appendix.append('### Color Palette Attribution')
appendix.append('')
appendix.append('The color palette used in this theme was extracted from the source URL.')
appendix.append('**Before going live, replace with your own brand colors:**')
appendix.append('')
appendix.append('1. Open Appearance > Editor > Styles > Colors')
appendix.append(f'2. Current palette was sourced from: {source_url}')
appendix.append('3. Colors are not copyrighted by themselves, but this theme\'s visual')
appendix.append('   resemblance to the source is your responsibility.')
appendix.append('')

if dynamic_feats:
    appendix.append('### Dynamic Features Detected (Not Cloned)')
    appendix.append('')
    appendix.append('The following interactive features were detected on the source site but cannot')
    appendix.append('be automatically reproduced from a URL scrape. Visual placeholder blocks have')
    appendix.append('been inserted in the templates. Manual setup required:')
    appendix.append('')
    appendix.append('| Feature | Detected On | Recommended WP Plugin |')
    appendix.append('|---------|-------------|----------------------|')
    for feat in dynamic_feats:
        if feat in PLUGIN_MAP:
            label, plugin, url = PLUGIN_MAP[feat]
            detected_on = ', '.join(feature_pages.get(feat, ['unknown']))
            if url:
                plugin_link = f'[{plugin}]({url})'
            else:
                plugin_link = plugin
            appendix.append(f'| {label} | {detected_on} | {plugin_link} |')
    appendix.append('')
else:
    appendix.append('### Dynamic Features Detected')
    appendix.append('')
    appendix.append('No dynamic features (e-commerce, forms, search, maps, video, social feeds) were detected on the scraped pages.')
    appendix.append('')

appendix.append('### Pages Scraped')
appendix.append('')
appendix.append('| Page | Filename | Status |')
appendix.append('|------|----------|--------|')
for page in pages:
    url = page.get('url', 'unknown')
    filename = page.get('filename', 'unknown')
    appendix.append(f'| {url} | {filename} | Scraped |')
for fp in failed_pages:
    url   = fp.get('url', 'unknown')
    error = fp.get('error', 'unknown error')
    appendix.append(f'| {url} | - | Failed: {error[:80]} |')
appendix.append('')

if spa_fallback:
    appendix.append('> **Note:** One or more pages used the `domcontentloaded` fallback due to')
    appendix.append('> continuous background network activity. Dynamic feature detection may be')
    appendix.append('> incomplete for JavaScript-heavy or SPA-rendered pages.')
    appendix.append('')

appendix.append('---')
appendix.append('')
appendix.append('*URL Clone mode - build-scrape skill + build-visual FSE pipeline*')
appendix.append(f'*Source: {source_url}*')

with open(setup_md_path, 'a') as f:
    f.write('\n'.join(appendix) + '\n')

print('[Build] SETUP.md URL clone appendix written.')
PYEOF
```

---

## Section 5: Output Variables

The following variables are set by this skill for consumption by COMMAND.md Section 3c and the downstream `build-visual` skill:

```bash
# Set by Section 0
# SCRAPE_DIR="/tmp/scrape_${SLUG}_$$"   - already set above

# Set by Section 2
# SCRAPE_MANIFEST="$SCRAPE_DIR/scrape.json"

# Set by Section 1c
# ROBOTS_STATUS="allowed" or "disallowed-override"

# Set for build-visual consumption
VISUAL_PATH="$SCRAPE_DIR"     # build-visual Section 2 reads CSS from this directory
VISUAL_MODE="html-css"        # ALWAYS html-css for URL builds - NEVER "screenshot"

echo "[Build] Output variables:"
echo "[Build]   SCRAPE_DIR=$SCRAPE_DIR"
echo "[Build]   SCRAPE_MANIFEST=$SCRAPE_MANIFEST"
echo "[Build]   ROBOTS_STATUS=$ROBOTS_STATUS"
echo "[Build]   VISUAL_PATH=$VISUAL_PATH"
echo "[Build]   VISUAL_MODE=$VISUAL_MODE"
```

**Variable reference:**

| Variable         | Value                              | Consumer                           |
|------------------|------------------------------------|------------------------------------|
| `SCRAPE_DIR`     | `/tmp/scrape_{SLUG}_{PID}`         | COMMAND.md Section 3c, build-visual |
| `SCRAPE_MANIFEST`| `$SCRAPE_DIR/scrape.json`          | Section 4 (SETUP.md appendix)      |
| `ROBOTS_STATUS`  | `"allowed"` or `"disallowed-override"` | Section 3c, scrape.json          |
| `VISUAL_PATH`    | `$SCRAPE_DIR`                      | build-visual Section 2a (CSS token extraction) |
| `VISUAL_MODE`    | `"html-css"` (always)              | build-visual Section 1a (input detection skipped - mode pre-set) |

---

## Implementati

Shown under the source repo’s MIT license. Read the full folder on GitHub.