feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# Smart Monitor Design Document
|
||||
|
||||
## Overview
|
||||
Design for an adaptive monitoring system that mimics human behavior and automatically switches modes based on detected activity.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Modes
|
||||
|
||||
### 1. STEALTH MODE (Default)
|
||||
- **Check interval**: 45-90 seconds (randomized around 1 minute)
|
||||
- **Behavior**: Human-like patterns, decoy requests
|
||||
- **Purpose**: Avoid detection during normal monitoring
|
||||
|
||||
### 2. ALERT MODE (Triggered by new SKU)
|
||||
- **Check interval**: 15-30 seconds (faster, still randomized)
|
||||
- **Duration**: 30-60 minutes after detection
|
||||
- **Purpose**: Catch the drop going live quickly
|
||||
|
||||
### 3. COOLDOWN MODE (After alert period)
|
||||
- **Gradually slow back down** to stealth mode
|
||||
- **Prevents sudden behavior change** which could trigger detection
|
||||
|
||||
### 4. SLEEP MODE (Optional - night hours)
|
||||
- **Check interval**: 5-10 minutes
|
||||
- **Active hours**: e.g., 2am-6am local time
|
||||
- **Purpose**: Real humans sleep, bots don't
|
||||
|
||||
---
|
||||
|
||||
## Human-Like Obfuscation Techniques
|
||||
|
||||
### Timing Randomization
|
||||
```python
|
||||
# Instead of exact intervals:
|
||||
time.sleep(60) # BAD - robotic
|
||||
|
||||
# Use gaussian distribution around target:
|
||||
import random
|
||||
base_interval = 60
|
||||
jitter = random.gauss(0, 10) # +/- 10 seconds standard deviation
|
||||
time.sleep(max(30, base_interval + jitter)) # GOOD - human-like
|
||||
```
|
||||
|
||||
### Decoy Requests
|
||||
Mix in non-API requests to look like a real browser:
|
||||
- Occasionally fetch an image from the site
|
||||
- Load the homepage or a random category page
|
||||
- Request favicon, robots.txt, etc.
|
||||
|
||||
```python
|
||||
DECOY_URLS = [
|
||||
"/favicon.ico",
|
||||
"/",
|
||||
"/category/plush",
|
||||
"/category/accessories",
|
||||
]
|
||||
|
||||
def make_decoy_request():
|
||||
"""Occasionally make a non-API request"""
|
||||
if random.random() < 0.2: # 20% of the time
|
||||
url = random.choice(DECOY_URLS)
|
||||
session.get(API_BASE + url)
|
||||
```
|
||||
|
||||
### Referrer Rotation
|
||||
Change the Referer header to look like you're browsing:
|
||||
```python
|
||||
REFERRERS = [
|
||||
"https://www.pokemoncenter.com/",
|
||||
"https://www.pokemoncenter.com/category/tcg-cards",
|
||||
"https://www.pokemoncenter.com/category/new-releases",
|
||||
"https://www.google.com/",
|
||||
]
|
||||
|
||||
headers["Referer"] = random.choice(REFERRERS)
|
||||
```
|
||||
|
||||
### User-Agent Variation
|
||||
Slightly vary the User-Agent (within reason):
|
||||
```python
|
||||
# Base UA with minor variations
|
||||
chrome_versions = ["146.0.0.0", "145.0.0.0", "146.0.7680.154"]
|
||||
version = random.choice(chrome_versions)
|
||||
headers["User-Agent"] = f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{version} Safari/537.36"
|
||||
```
|
||||
|
||||
### Session Breaks
|
||||
Occasionally "take a break" like a human would:
|
||||
```python
|
||||
def maybe_take_break():
|
||||
"""Occasionally pause for a longer period"""
|
||||
if random.random() < 0.05: # 5% chance
|
||||
break_duration = random.randint(120, 300) # 2-5 minutes
|
||||
logger.info(f"Taking a break for {break_duration}s...")
|
||||
time.sleep(break_duration)
|
||||
```
|
||||
|
||||
### Time-of-Day Awareness
|
||||
Adjust behavior based on time:
|
||||
```python
|
||||
from datetime import datetime
|
||||
|
||||
def get_interval_for_time():
|
||||
hour = datetime.now().hour
|
||||
|
||||
if 2 <= hour < 6:
|
||||
# Late night - slower checks
|
||||
return random.randint(300, 600) # 5-10 min
|
||||
elif 9 <= hour < 17:
|
||||
# Business hours - prime drop time
|
||||
return random.randint(45, 90) # ~1 min
|
||||
else:
|
||||
# Evening/early morning
|
||||
return random.randint(60, 120) # 1-2 min
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mode Transition Logic
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ STEALTH MODE │
|
||||
│ (45-90 sec) │
|
||||
└────────┬────────┘
|
||||
│
|
||||
New SKU detected?
|
||||
│
|
||||
YES ───────────┴─────────── NO
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│ ALERT MODE │ │
|
||||
│ (15-30 sec) │◄──────────────────┘
|
||||
└────────┬────────┘
|
||||
│
|
||||
30-60 min elapsed?
|
||||
│
|
||||
YES │
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ COOLDOWN MODE │
|
||||
│ (gradually │
|
||||
│ slow down) │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
Back to STEALTH MODE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detection Tracking
|
||||
|
||||
When a new product is detected:
|
||||
|
||||
1. **Log detection** with timestamp
|
||||
2. **Fetch full product details**
|
||||
3. **Send Discord alert**
|
||||
4. **Enter ALERT MODE**
|
||||
5. **Track if product becomes "announced"**:
|
||||
- Visible on homepage
|
||||
- Linked from category pages
|
||||
- Social media announcement
|
||||
|
||||
When product is announced, return to STEALTH MODE.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
1. [x] Basic backend monitor (done)
|
||||
2. [ ] Add timing randomization
|
||||
3. [ ] Implement mode switching
|
||||
4. [ ] Add decoy requests
|
||||
5. [ ] Add time-of-day awareness
|
||||
6. [ ] Add session breaks
|
||||
7. [ ] Integrate with Discord notifier
|
||||
|
||||
---
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Test with cookies**: Run warmup, verify API access works
|
||||
2. **Test SKU detection**: Manually add a fake SKU to known list, verify detection
|
||||
3. **Test mode switching**: Simulate detection, verify faster checks activate
|
||||
4. **Monitor for blocks**: Run for extended period, note when cookies expire
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Cookies expire | Auto-detect 403, prompt for re-warmup |
|
||||
| IP blocked | Support proxy rotation (already built) |
|
||||
| Pattern detected | Randomization + decoys + breaks |
|
||||
| Rate limited | Back off on 429 responses |
|
||||
Reference in New Issue
Block a user