8d382e723f
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
5.7 KiB
5.7 KiB
Smart Monitor Design Document
Overview
Design for an adaptive monitoring system that mimics human behavior and automatically switches modes based on detected activity.
Monitoring Modes
1. STEALTH MODE (Default)
- Check interval: 45-90 seconds (randomized around 1 minute)
- Behavior: Human-like patterns, decoy requests
- Purpose: Avoid detection during normal monitoring
2. ALERT MODE (Triggered by new SKU)
- Check interval: 15-30 seconds (faster, still randomized)
- Duration: 30-60 minutes after detection
- Purpose: Catch the drop going live quickly
3. COOLDOWN MODE (After alert period)
- Gradually slow back down to stealth mode
- Prevents sudden behavior change which could trigger detection
4. SLEEP MODE (Optional - night hours)
- Check interval: 5-10 minutes
- Active hours: e.g., 2am-6am local time
- Purpose: Real humans sleep, bots don't
Human-Like Obfuscation Techniques
Timing Randomization
# Instead of exact intervals:
time.sleep(60) # BAD - robotic
# Use gaussian distribution around target:
import random
base_interval = 60
jitter = random.gauss(0, 10) # +/- 10 seconds standard deviation
time.sleep(max(30, base_interval + jitter)) # GOOD - human-like
Decoy Requests
Mix in non-API requests to look like a real browser:
- Occasionally fetch an image from the site
- Load the homepage or a random category page
- Request favicon, robots.txt, etc.
DECOY_URLS = [
"/favicon.ico",
"/",
"/category/plush",
"/category/accessories",
]
def make_decoy_request():
"""Occasionally make a non-API request"""
if random.random() < 0.2: # 20% of the time
url = random.choice(DECOY_URLS)
session.get(API_BASE + url)
Referrer Rotation
Change the Referer header to look like you're browsing:
REFERRERS = [
"https://www.pokemoncenter.com/",
"https://www.pokemoncenter.com/category/tcg-cards",
"https://www.pokemoncenter.com/category/new-releases",
"https://www.google.com/",
]
headers["Referer"] = random.choice(REFERRERS)
User-Agent Variation
Slightly vary the User-Agent (within reason):
# Base UA with minor variations
chrome_versions = ["146.0.0.0", "145.0.0.0", "146.0.7680.154"]
version = random.choice(chrome_versions)
headers["User-Agent"] = f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{version} Safari/537.36"
Session Breaks
Occasionally "take a break" like a human would:
def maybe_take_break():
"""Occasionally pause for a longer period"""
if random.random() < 0.05: # 5% chance
break_duration = random.randint(120, 300) # 2-5 minutes
logger.info(f"Taking a break for {break_duration}s...")
time.sleep(break_duration)
Time-of-Day Awareness
Adjust behavior based on time:
from datetime import datetime
def get_interval_for_time():
hour = datetime.now().hour
if 2 <= hour < 6:
# Late night - slower checks
return random.randint(300, 600) # 5-10 min
elif 9 <= hour < 17:
# Business hours - prime drop time
return random.randint(45, 90) # ~1 min
else:
# Evening/early morning
return random.randint(60, 120) # 1-2 min
Mode Transition Logic
┌─────────────────┐
│ STEALTH MODE │
│ (45-90 sec) │
└────────┬────────┘
│
New SKU detected?
│
YES ───────────┴─────────── NO
│ │
▼ │
┌─────────────────┐ │
│ ALERT MODE │ │
│ (15-30 sec) │◄──────────────────┘
└────────┬────────┘
│
30-60 min elapsed?
│
YES │
▼
┌─────────────────┐
│ COOLDOWN MODE │
│ (gradually │
│ slow down) │
└────────┬────────┘
│
▼
Back to STEALTH MODE
Detection Tracking
When a new product is detected:
- Log detection with timestamp
- Fetch full product details
- Send Discord alert
- Enter ALERT MODE
- Track if product becomes "announced":
- Visible on homepage
- Linked from category pages
- Social media announcement
When product is announced, return to STEALTH MODE.
Implementation Priority
- Basic backend monitor (done)
- Add timing randomization
- Implement mode switching
- Add decoy requests
- Add time-of-day awareness
- Add session breaks
- Integrate with Discord notifier
Testing Plan
- Test with cookies: Run warmup, verify API access works
- Test SKU detection: Manually add a fake SKU to known list, verify detection
- Test mode switching: Simulate detection, verify faster checks activate
- Monitor for blocks: Run for extended period, note when cookies expire
Risk Mitigation
| Risk | Mitigation |
|---|---|
| Cookies expire | Auto-detect 403, prompt for re-warmup |
| IP blocked | Support proxy rotation (already built) |
| Pattern detected | Randomization + decoys + breaks |
| Rate limited | Back off on 429 responses |