Files
mmcghen 8d382e723f feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details.
- Introduced a stealth browser module to handle bot protection and improve scraping reliability.
- Created a SQLite database for tracking product history, price changes, stock events, and user favorites.
- Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores.
- Implemented a favorites system to manage priority products and categories with custom notification settings.
- Added news aggregation module to fetch and analyze Pokemon TCG news from various sources.
- Created tools for API discovery and monitoring, including a backend monitor for detecting new products.
- Added unit tests for database operations, product filtering, and API endpoints to ensure functionality.
- Enhanced existing modules with improved error handling and logging for better maintainability.
2026-03-27 23:08:09 -04:00

5.7 KiB

Smart Monitor Design Document

Overview

Design for an adaptive monitoring system that mimics human behavior and automatically switches modes based on detected activity.


Monitoring Modes

1. STEALTH MODE (Default)

  • Check interval: 45-90 seconds (randomized around 1 minute)
  • Behavior: Human-like patterns, decoy requests
  • Purpose: Avoid detection during normal monitoring

2. ALERT MODE (Triggered by new SKU)

  • Check interval: 15-30 seconds (faster, still randomized)
  • Duration: 30-60 minutes after detection
  • Purpose: Catch the drop going live quickly

3. COOLDOWN MODE (After alert period)

  • Gradually slow back down to stealth mode
  • Prevents sudden behavior change which could trigger detection

4. SLEEP MODE (Optional - night hours)

  • Check interval: 5-10 minutes
  • Active hours: e.g., 2am-6am local time
  • Purpose: Real humans sleep, bots don't

Human-Like Obfuscation Techniques

Timing Randomization

# Instead of exact intervals:
time.sleep(60)  # BAD - robotic

# Use gaussian distribution around target:
import random
base_interval = 60
jitter = random.gauss(0, 10)  # +/- 10 seconds standard deviation
time.sleep(max(30, base_interval + jitter))  # GOOD - human-like

Decoy Requests

Mix in non-API requests to look like a real browser:

  • Occasionally fetch an image from the site
  • Load the homepage or a random category page
  • Request favicon, robots.txt, etc.
DECOY_URLS = [
    "/favicon.ico",
    "/",
    "/category/plush",
    "/category/accessories",
]

def make_decoy_request():
    """Occasionally make a non-API request"""
    if random.random() < 0.2:  # 20% of the time
        url = random.choice(DECOY_URLS)
        session.get(API_BASE + url)

Referrer Rotation

Change the Referer header to look like you're browsing:

REFERRERS = [
    "https://www.pokemoncenter.com/",
    "https://www.pokemoncenter.com/category/tcg-cards",
    "https://www.pokemoncenter.com/category/new-releases",
    "https://www.google.com/",
]

headers["Referer"] = random.choice(REFERRERS)

User-Agent Variation

Slightly vary the User-Agent (within reason):

# Base UA with minor variations
chrome_versions = ["146.0.0.0", "145.0.0.0", "146.0.7680.154"]
version = random.choice(chrome_versions)
headers["User-Agent"] = f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{version} Safari/537.36"

Session Breaks

Occasionally "take a break" like a human would:

def maybe_take_break():
    """Occasionally pause for a longer period"""
    if random.random() < 0.05:  # 5% chance
        break_duration = random.randint(120, 300)  # 2-5 minutes
        logger.info(f"Taking a break for {break_duration}s...")
        time.sleep(break_duration)

Time-of-Day Awareness

Adjust behavior based on time:

from datetime import datetime

def get_interval_for_time():
    hour = datetime.now().hour

    if 2 <= hour < 6:
        # Late night - slower checks
        return random.randint(300, 600)  # 5-10 min
    elif 9 <= hour < 17:
        # Business hours - prime drop time
        return random.randint(45, 90)  # ~1 min
    else:
        # Evening/early morning
        return random.randint(60, 120)  # 1-2 min

Mode Transition Logic

                    ┌─────────────────┐
                    │  STEALTH MODE   │
                    │  (45-90 sec)    │
                    └────────┬────────┘
                             │
                    New SKU detected?
                             │
              YES ───────────┴─────────── NO
               │                          │
               ▼                          │
    ┌─────────────────┐                   │
    │   ALERT MODE    │                   │
    │  (15-30 sec)    │◄──────────────────┘
    └────────┬────────┘
             │
      30-60 min elapsed?
             │
          YES │
             ▼
    ┌─────────────────┐
    │  COOLDOWN MODE  │
    │  (gradually     │
    │   slow down)    │
    └────────┬────────┘
             │
             ▼
    Back to STEALTH MODE

Detection Tracking

When a new product is detected:

  1. Log detection with timestamp
  2. Fetch full product details
  3. Send Discord alert
  4. Enter ALERT MODE
  5. Track if product becomes "announced":
    • Visible on homepage
    • Linked from category pages
    • Social media announcement

When product is announced, return to STEALTH MODE.


Implementation Priority

  1. Basic backend monitor (done)
  2. Add timing randomization
  3. Implement mode switching
  4. Add decoy requests
  5. Add time-of-day awareness
  6. Add session breaks
  7. Integrate with Discord notifier

Testing Plan

  1. Test with cookies: Run warmup, verify API access works
  2. Test SKU detection: Manually add a fake SKU to known list, verify detection
  3. Test mode switching: Simulate detection, verify faster checks activate
  4. Monitor for blocks: Run for extended period, note when cookies expire

Risk Mitigation

Risk Mitigation
Cookies expire Auto-detect 403, prompt for re-warmup
IP blocked Support proxy rotation (already built)
Pattern detected Randomization + decoys + breaks
Rate limited Back off on 429 responses