feat: Implement Walmart scraper and integrate with existing architecture

- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
2026-03-27 23:08:09 -04:00
parent cddae24e34
commit 8d382e723f
64 changed files with 15433 additions and 443 deletions
@@ -0,0 +1,285 @@
+# PokemonCenter Backend API Monitor
+
+## Overview
+
+This document outlines how to build a backend API monitor that detects new products before they appear on the website - similar to what accounts like @pokepullzhq do.
+
+## Why Backend Monitoring?
+
+- **Faster detection**: Products are loaded into the backend/API before the frontend displays them
+- **Less aggressive**: API calls are lighter than full page loads
+- **Avoids bot detection**: Direct API calls look different than browser automation
+- **More reliable**: JSON responses are easier to parse than HTML
+
+---
+
+## Step 1: Find the API Endpoints
+
+### How to Investigate
+
+1. Open https://www.pokemoncenter.com/category/tcg-cards in Chrome
+2. Open DevTools (F12)
+3. Go to **Network** tab
+4. Check **Preserve log**
+5. Filter by **Fetch/XHR**
+6. Refresh the page
+7. Look for API calls that return product data
+
+### What to Look For
+
+**Common patterns:**
+- GraphQL endpoints: `/graphql` or `/api/graphql`
+- REST APIs: `/api/products`, `/api/catalog`, `/api/search`
+- Third-party services:
+  - **Algolia** (search): `*.algolia.net` or `*.algolianet.com`
+  - **Contentful** (CMS): `cdn.contentful.com`
+  - **Commercetools** (e-commerce): `*.commercetools.com`
+  - **Salesforce Commerce**: `*.demandware.net`
+
+**Signs you found the right endpoint:**
+- Response contains product names, prices, SKUs
+- Response has `inStock`, `availability`, or similar fields
+- Response includes product URLs or IDs
+
+### Example Findings to Document
+
+For each endpoint found, note:
+```
+URL: https://api.pokemoncenter.com/products?category=tcg
+Method: GET
+Headers:
+  - Authorization: Bearer xxx (if any)
+  - x-api-key: xxx (if any)
+Response format: JSON
+Contains: productId, name, price, availability, url
+```
+
+---
+
+## Step 2: API Response Analysis
+
+Once you find the product API, analyze the response structure:
+
+### Key Fields to Track
+
+```json
+{
+  "products": [
+    {
+      "id": "12345",           // Unique product ID
+      "sku": "PKM-CR-ETB",     // SKU code
+      "name": "Chaos Rising ETB",
+      "url": "/product/chaos-rising-etb",
+      "price": 54.99,
+      "availability": {
+        "inStock": false,       // Current stock status
+        "preorder": true,       // Pre-order available?
+        "releaseDate": "2026-04-15"
+      },
+      "status": "ACTIVE",       // May be "HIDDEN", "DRAFT" before launch
+      "publishedAt": null       // null = not yet visible on site
+    }
+  ]
+}
+```
+
+### Detection Strategies
+
+1. **New Product Detection**
+   - Compare product IDs against known list
+   - New ID = new product added to backend
+
+2. **Pre-Launch Detection**
+   - Product exists but `publishedAt` is null
+   - Product has `status: "DRAFT"` or similar
+   - Product `availability.inStock` changes from false to true
+
+3. **Stock Change Detection**
+   - Track `inStock` or `availability` field changes
+
+---
+
+## Step 3: Implementation Plan
+
+### Option A: Add to Chrome Extension
+
+```javascript
+// In background.js - add API monitoring alongside page monitoring
+
+async function checkBackendAPI() {
+  const API_URL = "https://api.pokemoncenter.com/products?category=tcg";
+
+  try {
+    const response = await fetch(API_URL, {
+      headers: {
+        // Add any required headers discovered during investigation
+        'Accept': 'application/json',
+      }
+    });
+
+    const data = await response.json();
+    const products = data.products || [];
+
+    // Compare against known products
+    for (const product of products) {
+      if (!knownBackendProducts[product.id]) {
+        // NEW PRODUCT DETECTED!
+        await sendDiscordNotification({
+          name: product.name,
+          url: `https://www.pokemoncenter.com${product.url}`,
+          price: `$${product.price}`,
+          inStock: product.availability?.inStock
+        }, "backend_detect");
+
+        knownBackendProducts[product.id] = product;
+      }
+    }
+  } catch (error) {
+    console.error("Backend API check failed:", error);
+  }
+}
+```
+
+### Option B: Standalone Python Script
+
+```python
+# backend_monitor.py
+import requests
+import time
+import json
+from discord_webhook import DiscordWebhook, DiscordEmbed
+
+API_URL = "https://api.pokemoncenter.com/products"
+DISCORD_WEBHOOK = "your-webhook-url"
+KNOWN_PRODUCTS_FILE = "known_backend_products.json"
+CHECK_INTERVAL = 30  # seconds - can be faster for API
+
+def load_known_products():
+    try:
+        with open(KNOWN_PRODUCTS_FILE, 'r') as f:
+            return json.load(f)
+    except FileNotFoundError:
+        return {}
+
+def save_known_products(products):
+    with open(KNOWN_PRODUCTS_FILE, 'w') as f:
+        json.dump(products, f)
+
+def check_api():
+    headers = {
+        'User-Agent': 'Mozilla/5.0...',
+        # Add discovered headers
+    }
+
+    response = requests.get(API_URL, headers=headers)
+    return response.json()
+
+def send_alert(product, alert_type):
+    webhook = DiscordWebhook(url=DISCORD_WEBHOOK, content="@everyone")
+
+    embed = DiscordEmbed(
+        title=f"🚨 {alert_type.upper()}",
+        description=f"**{product['name']}**",
+        color=0xFF0000 if alert_type == "BACKEND DETECT" else 0x00FF00
+    )
+    embed.add_embed_field(name="Price", value=f"${product.get('price', 'TBD')}")
+    embed.add_embed_field(name="Status", value=product.get('status', 'Unknown'))
+    embed.add_embed_field(name="Link", value=f"[VIEW]({product['url']})", inline=False)
+
+    webhook.add_embed(embed)
+    webhook.execute()
+
+def main():
+    known = load_known_products()
+    print(f"Loaded {len(known)} known products")
+
+    while True:
+        try:
+            data = check_api()
+            products = data.get('products', [])
+
+            for product in products:
+                pid = product['id']
+
+                if pid not in known:
+                    print(f"NEW BACKEND PRODUCT: {product['name']}")
+                    send_alert(product, "BACKEND DETECT")
+                    known[pid] = product
+                    save_known_products(known)
+
+                # Check for status changes
+                elif known[pid].get('status') != product.get('status'):
+                    print(f"STATUS CHANGE: {product['name']}")
+                    send_alert(product, "STATUS CHANGE")
+                    known[pid] = product
+                    save_known_products(known)
+
+            print(f"Checked {len(products)} products")
+
+        except Exception as e:
+            print(f"Error: {e}")
+
+        time.sleep(CHECK_INTERVAL)
+
+if __name__ == "__main__":
+    main()
+```
+
+---
+
+## Step 4: Rate Limiting Considerations
+
+### API vs Page Scraping
+
+| Approach | Safe Interval | Notes |
+|----------|---------------|-------|
+| Full page load | 60-90 sec | Heavy, triggers bot detection |
+| API call | 15-30 sec | Lighter, more tolerant |
+| GraphQL query | 15-30 sec | Depends on complexity |
+
+### Best Practices
+
+1. **Use proper headers**: Include realistic User-Agent, Accept, etc.
+2. **Don't hammer**: Even APIs have rate limits
+3. **Cache responses**: Don't re-process unchanged data
+4. **Handle 429s gracefully**: Back off when rate limited
+
+---
+
+## Step 5: Advanced - Multiple Detection Layers
+
+For maximum coverage, run both:
+
+1. **Backend API Monitor** (every 30 sec)
+   - Fast detection of new products in system
+   - Lighter on resources
+
+2. **Frontend Page Monitor** (every 90 sec)
+   - Confirms products are live on website
+   - Catches anything API might miss
+
+### Alert Priority
+
+```
+BACKEND DETECT (API) = "Product loaded, drop imminent!"
+FRONTEND DETECT (Page) = "Product is LIVE, buy now!"
+RESTOCK = "Back in stock!"
+```
+
+---
+
+## Next Steps
+
+1. **Wait for rate limit to clear** (~30 min)
+2. **Investigate the API** using DevTools Network tab
+3. **Document the endpoints** you find
+4. **Share the findings** so we can build the monitor
+
+### Questions to Answer During Investigation
+
+- [ ] What URL serves product data?
+- [ ] What headers are required?
+- [ ] Is authentication needed?
+- [ ] What does the response structure look like?
+- [ ] Are there pagination parameters?
+- [ ] Is there a "hidden" or "draft" status visible in API?