Files
mmcghen 8d382e723f feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details.
- Introduced a stealth browser module to handle bot protection and improve scraping reliability.
- Created a SQLite database for tracking product history, price changes, stock events, and user favorites.
- Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores.
- Implemented a favorites system to manage priority products and categories with custom notification settings.
- Added news aggregation module to fetch and analyze Pokemon TCG news from various sources.
- Created tools for API discovery and monitoring, including a backend monitor for detecting new products.
- Added unit tests for database operations, product filtering, and API endpoints to ensure functionality.
- Enhanced existing modules with improved error handling and logging for better maintainability.
2026-03-27 23:08:09 -04:00

7.8 KiB

PokemonCenter Backend API Monitor

Overview

This document outlines how to build a backend API monitor that detects new products before they appear on the website - similar to what accounts like @pokepullzhq do.

Why Backend Monitoring?

  • Faster detection: Products are loaded into the backend/API before the frontend displays them
  • Less aggressive: API calls are lighter than full page loads
  • Avoids bot detection: Direct API calls look different than browser automation
  • More reliable: JSON responses are easier to parse than HTML

Step 1: Find the API Endpoints

How to Investigate

  1. Open https://www.pokemoncenter.com/category/tcg-cards in Chrome
  2. Open DevTools (F12)
  3. Go to Network tab
  4. Check Preserve log
  5. Filter by Fetch/XHR
  6. Refresh the page
  7. Look for API calls that return product data

What to Look For

Common patterns:

  • GraphQL endpoints: /graphql or /api/graphql
  • REST APIs: /api/products, /api/catalog, /api/search
  • Third-party services:
    • Algolia (search): *.algolia.net or *.algolianet.com
    • Contentful (CMS): cdn.contentful.com
    • Commercetools (e-commerce): *.commercetools.com
    • Salesforce Commerce: *.demandware.net

Signs you found the right endpoint:

  • Response contains product names, prices, SKUs
  • Response has inStock, availability, or similar fields
  • Response includes product URLs or IDs

Example Findings to Document

For each endpoint found, note:

URL: https://api.pokemoncenter.com/products?category=tcg
Method: GET
Headers:
  - Authorization: Bearer xxx (if any)
  - x-api-key: xxx (if any)
Response format: JSON
Contains: productId, name, price, availability, url

Step 2: API Response Analysis

Once you find the product API, analyze the response structure:

Key Fields to Track

{
  "products": [
    {
      "id": "12345",           // Unique product ID
      "sku": "PKM-CR-ETB",     // SKU code
      "name": "Chaos Rising ETB",
      "url": "/product/chaos-rising-etb",
      "price": 54.99,
      "availability": {
        "inStock": false,       // Current stock status
        "preorder": true,       // Pre-order available?
        "releaseDate": "2026-04-15"
      },
      "status": "ACTIVE",       // May be "HIDDEN", "DRAFT" before launch
      "publishedAt": null       // null = not yet visible on site
    }
  ]
}

Detection Strategies

  1. New Product Detection

    • Compare product IDs against known list
    • New ID = new product added to backend
  2. Pre-Launch Detection

    • Product exists but publishedAt is null
    • Product has status: "DRAFT" or similar
    • Product availability.inStock changes from false to true
  3. Stock Change Detection

    • Track inStock or availability field changes

Step 3: Implementation Plan

Option A: Add to Chrome Extension

// In background.js - add API monitoring alongside page monitoring

async function checkBackendAPI() {
  const API_URL = "https://api.pokemoncenter.com/products?category=tcg";

  try {
    const response = await fetch(API_URL, {
      headers: {
        // Add any required headers discovered during investigation
        'Accept': 'application/json',
      }
    });

    const data = await response.json();
    const products = data.products || [];

    // Compare against known products
    for (const product of products) {
      if (!knownBackendProducts[product.id]) {
        // NEW PRODUCT DETECTED!
        await sendDiscordNotification({
          name: product.name,
          url: `https://www.pokemoncenter.com${product.url}`,
          price: `$${product.price}`,
          inStock: product.availability?.inStock
        }, "backend_detect");

        knownBackendProducts[product.id] = product;
      }
    }
  } catch (error) {
    console.error("Backend API check failed:", error);
  }
}

Option B: Standalone Python Script

# backend_monitor.py
import requests
import time
import json
from discord_webhook import DiscordWebhook, DiscordEmbed

API_URL = "https://api.pokemoncenter.com/products"
DISCORD_WEBHOOK = "your-webhook-url"
KNOWN_PRODUCTS_FILE = "known_backend_products.json"
CHECK_INTERVAL = 30  # seconds - can be faster for API

def load_known_products():
    try:
        with open(KNOWN_PRODUCTS_FILE, 'r') as f:
            return json.load(f)
    except FileNotFoundError:
        return {}

def save_known_products(products):
    with open(KNOWN_PRODUCTS_FILE, 'w') as f:
        json.dump(products, f)

def check_api():
    headers = {
        'User-Agent': 'Mozilla/5.0...',
        # Add discovered headers
    }

    response = requests.get(API_URL, headers=headers)
    return response.json()

def send_alert(product, alert_type):
    webhook = DiscordWebhook(url=DISCORD_WEBHOOK, content="@everyone")

    embed = DiscordEmbed(
        title=f"🚨 {alert_type.upper()}",
        description=f"**{product['name']}**",
        color=0xFF0000 if alert_type == "BACKEND DETECT" else 0x00FF00
    )
    embed.add_embed_field(name="Price", value=f"${product.get('price', 'TBD')}")
    embed.add_embed_field(name="Status", value=product.get('status', 'Unknown'))
    embed.add_embed_field(name="Link", value=f"[VIEW]({product['url']})", inline=False)

    webhook.add_embed(embed)
    webhook.execute()

def main():
    known = load_known_products()
    print(f"Loaded {len(known)} known products")

    while True:
        try:
            data = check_api()
            products = data.get('products', [])

            for product in products:
                pid = product['id']

                if pid not in known:
                    print(f"NEW BACKEND PRODUCT: {product['name']}")
                    send_alert(product, "BACKEND DETECT")
                    known[pid] = product
                    save_known_products(known)

                # Check for status changes
                elif known[pid].get('status') != product.get('status'):
                    print(f"STATUS CHANGE: {product['name']}")
                    send_alert(product, "STATUS CHANGE")
                    known[pid] = product
                    save_known_products(known)

            print(f"Checked {len(products)} products")

        except Exception as e:
            print(f"Error: {e}")

        time.sleep(CHECK_INTERVAL)

if __name__ == "__main__":
    main()

Step 4: Rate Limiting Considerations

API vs Page Scraping

Approach Safe Interval Notes
Full page load 60-90 sec Heavy, triggers bot detection
API call 15-30 sec Lighter, more tolerant
GraphQL query 15-30 sec Depends on complexity

Best Practices

  1. Use proper headers: Include realistic User-Agent, Accept, etc.
  2. Don't hammer: Even APIs have rate limits
  3. Cache responses: Don't re-process unchanged data
  4. Handle 429s gracefully: Back off when rate limited

Step 5: Advanced - Multiple Detection Layers

For maximum coverage, run both:

  1. Backend API Monitor (every 30 sec)

    • Fast detection of new products in system
    • Lighter on resources
  2. Frontend Page Monitor (every 90 sec)

    • Confirms products are live on website
    • Catches anything API might miss

Alert Priority

BACKEND DETECT (API) = "Product loaded, drop imminent!"
FRONTEND DETECT (Page) = "Product is LIVE, buy now!"
RESTOCK = "Back in stock!"

Next Steps

  1. Wait for rate limit to clear (~30 min)
  2. Investigate the API using DevTools Network tab
  3. Document the endpoints you find
  4. Share the findings so we can build the monitor

Questions to Answer During Investigation

  • What URL serves product data?
  • What headers are required?
  • Is authentication needed?
  • What does the response structure look like?
  • Are there pagination parameters?
  • Is there a "hidden" or "draft" status visible in API?