Files
pokemon-stock-checker/docs/SMART_MONITOR_DESIGN.md
T
mmcghen 8d382e723f feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details.
- Introduced a stealth browser module to handle bot protection and improve scraping reliability.
- Created a SQLite database for tracking product history, price changes, stock events, and user favorites.
- Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores.
- Implemented a favorites system to manage priority products and categories with custom notification settings.
- Added news aggregation module to fetch and analyze Pokemon TCG news from various sources.
- Created tools for API discovery and monitoring, including a backend monitor for detecting new products.
- Added unit tests for database operations, product filtering, and API endpoints to ensure functionality.
- Enhanced existing modules with improved error handling and logging for better maintainability.
2026-03-27 23:08:09 -04:00

200 lines
5.7 KiB
Markdown

# Smart Monitor Design Document
## Overview
Design for an adaptive monitoring system that mimics human behavior and automatically switches modes based on detected activity.
---
## Monitoring Modes
### 1. STEALTH MODE (Default)
- **Check interval**: 45-90 seconds (randomized around 1 minute)
- **Behavior**: Human-like patterns, decoy requests
- **Purpose**: Avoid detection during normal monitoring
### 2. ALERT MODE (Triggered by new SKU)
- **Check interval**: 15-30 seconds (faster, still randomized)
- **Duration**: 30-60 minutes after detection
- **Purpose**: Catch the drop going live quickly
### 3. COOLDOWN MODE (After alert period)
- **Gradually slow back down** to stealth mode
- **Prevents sudden behavior change** which could trigger detection
### 4. SLEEP MODE (Optional - night hours)
- **Check interval**: 5-10 minutes
- **Active hours**: e.g., 2am-6am local time
- **Purpose**: Real humans sleep, bots don't
---
## Human-Like Obfuscation Techniques
### Timing Randomization
```python
# Instead of exact intervals:
time.sleep(60) # BAD - robotic
# Use gaussian distribution around target:
import random
base_interval = 60
jitter = random.gauss(0, 10) # +/- 10 seconds standard deviation
time.sleep(max(30, base_interval + jitter)) # GOOD - human-like
```
### Decoy Requests
Mix in non-API requests to look like a real browser:
- Occasionally fetch an image from the site
- Load the homepage or a random category page
- Request favicon, robots.txt, etc.
```python
DECOY_URLS = [
"/favicon.ico",
"/",
"/category/plush",
"/category/accessories",
]
def make_decoy_request():
"""Occasionally make a non-API request"""
if random.random() < 0.2: # 20% of the time
url = random.choice(DECOY_URLS)
session.get(API_BASE + url)
```
### Referrer Rotation
Change the Referer header to look like you're browsing:
```python
REFERRERS = [
"https://www.pokemoncenter.com/",
"https://www.pokemoncenter.com/category/tcg-cards",
"https://www.pokemoncenter.com/category/new-releases",
"https://www.google.com/",
]
headers["Referer"] = random.choice(REFERRERS)
```
### User-Agent Variation
Slightly vary the User-Agent (within reason):
```python
# Base UA with minor variations
chrome_versions = ["146.0.0.0", "145.0.0.0", "146.0.7680.154"]
version = random.choice(chrome_versions)
headers["User-Agent"] = f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{version} Safari/537.36"
```
### Session Breaks
Occasionally "take a break" like a human would:
```python
def maybe_take_break():
"""Occasionally pause for a longer period"""
if random.random() < 0.05: # 5% chance
break_duration = random.randint(120, 300) # 2-5 minutes
logger.info(f"Taking a break for {break_duration}s...")
time.sleep(break_duration)
```
### Time-of-Day Awareness
Adjust behavior based on time:
```python
from datetime import datetime
def get_interval_for_time():
hour = datetime.now().hour
if 2 <= hour < 6:
# Late night - slower checks
return random.randint(300, 600) # 5-10 min
elif 9 <= hour < 17:
# Business hours - prime drop time
return random.randint(45, 90) # ~1 min
else:
# Evening/early morning
return random.randint(60, 120) # 1-2 min
```
---
## Mode Transition Logic
```
┌─────────────────┐
│ STEALTH MODE │
│ (45-90 sec) │
└────────┬────────┘
New SKU detected?
YES ───────────┴─────────── NO
│ │
▼ │
┌─────────────────┐ │
│ ALERT MODE │ │
│ (15-30 sec) │◄──────────────────┘
└────────┬────────┘
30-60 min elapsed?
YES │
┌─────────────────┐
│ COOLDOWN MODE │
│ (gradually │
│ slow down) │
└────────┬────────┘
Back to STEALTH MODE
```
---
## Detection Tracking
When a new product is detected:
1. **Log detection** with timestamp
2. **Fetch full product details**
3. **Send Discord alert**
4. **Enter ALERT MODE**
5. **Track if product becomes "announced"**:
- Visible on homepage
- Linked from category pages
- Social media announcement
When product is announced, return to STEALTH MODE.
---
## Implementation Priority
1. [x] Basic backend monitor (done)
2. [ ] Add timing randomization
3. [ ] Implement mode switching
4. [ ] Add decoy requests
5. [ ] Add time-of-day awareness
6. [ ] Add session breaks
7. [ ] Integrate with Discord notifier
---
## Testing Plan
1. **Test with cookies**: Run warmup, verify API access works
2. **Test SKU detection**: Manually add a fake SKU to known list, verify detection
3. **Test mode switching**: Simulate detection, verify faster checks activate
4. **Monitor for blocks**: Run for extended period, note when cookies expire
---
## Risk Mitigation
| Risk | Mitigation |
|------|------------|
| Cookies expire | Auto-detect 403, prompt for re-warmup |
| IP blocked | Support proxy rotation (already built) |
| Pattern detected | Randomization + decoys + breaks |
| Rate limited | Back off on 429 responses |