feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
# Pokemon Center Bot Detection - Strategies & Solutions
|
||||
|
||||
Pokemon Center uses **Imperva (Incapsula)** bot protection, which is one of the most aggressive anti-bot systems. Here are practical approaches to work around it.
|
||||
|
||||
## Current Problem
|
||||
- Direct API calls: **403 Forbidden**
|
||||
- Sitemap.xml: **Blocked by Imperva**
|
||||
- Browser automation: **CAPTCHA after a few requests**
|
||||
|
||||
---
|
||||
|
||||
## Strategy 1: Third-Party Stock Trackers (Easiest)
|
||||
|
||||
Instead of scraping Pokemon Center directly, integrate with existing stock tracking services:
|
||||
|
||||
### Discord Bots/Webhooks
|
||||
Several Discord servers track Pokemon Center in real-time:
|
||||
- **Pokemon TCG Drops** - Dedicated Pokemon TCG stock alerts
|
||||
- **Stock Informer** - General retail stock tracking
|
||||
- Search Discord for "Pokemon Center stock alerts"
|
||||
|
||||
You can join these servers and set up webhook forwarding to your own Discord.
|
||||
|
||||
### Stock Alert Services
|
||||
- **NowInStock.net** - Has Pokemon Center tracking
|
||||
- **Distill.io** - Browser extension for change detection
|
||||
- **Visualping** - Monitors page changes
|
||||
|
||||
### Implementation
|
||||
```python
|
||||
# Forward alerts from a public tracker to your system
|
||||
# Set up a Discord bot to listen to stock alert channels
|
||||
# Then forward to your own notification system
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy 2: Undetected Chrome Driver
|
||||
|
||||
Use `undetected-chromedriver` which patches Chrome to avoid detection:
|
||||
|
||||
```bash
|
||||
pip install undetected-chromedriver
|
||||
```
|
||||
|
||||
```python
|
||||
import undetected_chromedriver as uc
|
||||
|
||||
driver = uc.Chrome(headless=False) # Headless often gets detected
|
||||
driver.get("https://www.pokemoncenter.com/category/tcg-cards")
|
||||
|
||||
# Add human-like delays
|
||||
import time
|
||||
import random
|
||||
time.sleep(random.uniform(3, 7))
|
||||
|
||||
# Scroll like a human
|
||||
driver.execute_script("window.scrollBy(0, 500)")
|
||||
time.sleep(random.uniform(1, 3))
|
||||
```
|
||||
|
||||
**Pros:** Often bypasses Imperva
|
||||
**Cons:** Slower, still may trigger CAPTCHA eventually
|
||||
|
||||
---
|
||||
|
||||
## Strategy 3: Session Persistence
|
||||
|
||||
Keep a browser session alive and logged in:
|
||||
|
||||
1. **Manual login once** - Complete any CAPTCHA manually
|
||||
2. **Save cookies** - Export session cookies
|
||||
3. **Reuse session** - Load cookies on each check
|
||||
4. **Longer intervals** - Check every 3-5 minutes instead of 60-90 seconds
|
||||
|
||||
```python
|
||||
# Save cookies after manual login
|
||||
import pickle
|
||||
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))
|
||||
|
||||
# Load cookies on subsequent runs
|
||||
cookies = pickle.load(open("cookies.pkl", "rb"))
|
||||
for cookie in cookies:
|
||||
driver.add_cookie(cookie)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy 4: Residential Proxy Rotation
|
||||
|
||||
Use residential proxies that look like real home internet connections:
|
||||
|
||||
### Providers
|
||||
- **Bright Data** (formerly Luminati) - Best but expensive
|
||||
- **Oxylabs** - Good quality
|
||||
- **Smartproxy** - Budget option
|
||||
- **IPRoyal** - Pay-per-GB
|
||||
|
||||
### Cost
|
||||
- ~$10-15/GB for residential proxies
|
||||
- ~50-100 requests per MB depending on page size
|
||||
|
||||
```python
|
||||
proxies = {
|
||||
"http": "http://user:pass@proxy.provider.com:port",
|
||||
"https": "http://user:pass@proxy.provider.com:port"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy 5: Human-Like Behavior Pattern
|
||||
|
||||
If continuing with browser automation:
|
||||
|
||||
```python
|
||||
import random
|
||||
import time
|
||||
|
||||
def human_like_check():
|
||||
# Random delay between checks (2-5 minutes)
|
||||
delay = random.uniform(120, 300)
|
||||
|
||||
# Random user agents
|
||||
user_agents = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
|
||||
]
|
||||
|
||||
# Vary timing throughout the day
|
||||
# Check more during business hours, less at night
|
||||
hour = datetime.now().hour
|
||||
if 2 <= hour <= 6: # 2am-6am
|
||||
delay *= 2 # Slower at night
|
||||
|
||||
# Add random mouse movements
|
||||
# Add random scrolling
|
||||
# Occasionally visit other pages (home, about)
|
||||
|
||||
return delay
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy 6: Webhook from Manual Monitoring
|
||||
|
||||
The most reliable approach for Pokemon Center:
|
||||
|
||||
1. **Open Pokemon Center in your actual browser**
|
||||
2. **Use Distill.io browser extension** to monitor changes
|
||||
3. **Set up webhook** to forward alerts to your system
|
||||
|
||||
This way:
|
||||
- You're using a real browser with real cookies
|
||||
- Imperva sees normal human behavior
|
||||
- Change detection triggers your notification system
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach
|
||||
|
||||
Given Pokemon Center's aggressive protection, I recommend a **hybrid approach**:
|
||||
|
||||
1. **Primary: Join existing stock alert Discord servers**
|
||||
- Let others deal with the bot detection
|
||||
- Forward their alerts to your system
|
||||
|
||||
2. **Secondary: Use undetected-chromedriver with long intervals**
|
||||
- Check every 3-5 minutes
|
||||
- Use session persistence
|
||||
- Add human-like behavior
|
||||
|
||||
3. **Backup: Manual Distill.io monitoring**
|
||||
- For specific products you really care about
|
||||
- Most reliable but requires keeping browser open
|
||||
|
||||
---
|
||||
|
||||
## Code Changes Needed
|
||||
|
||||
To implement these strategies, we would need to:
|
||||
|
||||
1. Add `undetected-chromedriver` as an option
|
||||
2. Implement cookie/session persistence
|
||||
3. Add configurable random delays
|
||||
4. Add proxy support
|
||||
5. Consider adding Discord bot listener for third-party alerts
|
||||
|
||||
Would you like me to implement any of these strategies?
|
||||
@@ -0,0 +1,13 @@
|
||||
# Pokemon Stock Monitor - Notes
|
||||
|
||||
## Bot Detection / CAPTCHA Log
|
||||
|
||||
| Date | Time | Site | Notes |
|
||||
|------|------|------|-------|
|
||||
| 2026-03-25 | ~10:30 AM | PokemonCenter | CAPTCHAs appeared, bot protection triggered |
|
||||
|
||||
## Tips
|
||||
- Pokemon Center uses Imperva bot protection
|
||||
- CAPTCHAs more likely during high traffic (product drops)
|
||||
- Using Chrome extension with real browser avoids most detection
|
||||
- If blocked, clear cookies and wait before retrying
|
||||
@@ -0,0 +1,46 @@
|
||||
# Pokemon Center Drop Strategy Notes
|
||||
|
||||
Source: https://www.pokepullz.ca/blog/pokemon-center-drops-guide.html
|
||||
|
||||
## Queue System
|
||||
- Pokemon Center uses a **randomized virtual queue** that activates before major drops
|
||||
- Position is NOT based on when you arrived - it's assigned randomly once queue activates
|
||||
- This prevents site crashes and distributes access fairly
|
||||
|
||||
## Timing Signals
|
||||
- Security changes on the Pokemon Center site serve as early warning signals that drops are imminent
|
||||
- Products sometimes load in waves rather than all at once
|
||||
|
||||
## Multi-Device Strategy
|
||||
1. Join queue on primary device (desktop on WiFi) first
|
||||
2. Wait a few minutes, then join on second device (phone on mobile data)
|
||||
3. This gives you two separate queue positions and accounts for staggered product loading
|
||||
|
||||
## Technical Tips
|
||||
|
||||
### The Cart Trick
|
||||
When Add to Cart button isn't displaying during glitchy drops:
|
||||
1. Click the cart icon to go to your cart page
|
||||
2. Hit browser back button to return to product page
|
||||
3. This forces a page reload that may show the button
|
||||
|
||||
## Order Flagging Risks (Avoid These!)
|
||||
- Typos in billing/shipping information
|
||||
- Mismatched address data
|
||||
- VPNs or browser extensions
|
||||
- Multiple rapid orders
|
||||
- Overloaded carts
|
||||
- Queue-checking tools or auto-refreshers
|
||||
|
||||
## Pre-Drop Prep Checklist
|
||||
- [ ] Log into account days ahead
|
||||
- [ ] Have devices ready
|
||||
- [ ] Monitor Discord alerts
|
||||
- [ ] Disable browser extensions
|
||||
- [ ] Turn off VPN
|
||||
|
||||
## Implications for Our Monitor
|
||||
1. **Don't use VPN/proxies during actual purchases** - only for monitoring
|
||||
2. **Queue position is random** - early detection still helps you GET in queue early
|
||||
3. **Watch for security changes** - could indicate imminent drop
|
||||
4. **Multi-wave loading** - may need multiple checks to catch all products
|
||||
@@ -0,0 +1,199 @@
|
||||
# Smart Monitor Design Document
|
||||
|
||||
## Overview
|
||||
Design for an adaptive monitoring system that mimics human behavior and automatically switches modes based on detected activity.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Modes
|
||||
|
||||
### 1. STEALTH MODE (Default)
|
||||
- **Check interval**: 45-90 seconds (randomized around 1 minute)
|
||||
- **Behavior**: Human-like patterns, decoy requests
|
||||
- **Purpose**: Avoid detection during normal monitoring
|
||||
|
||||
### 2. ALERT MODE (Triggered by new SKU)
|
||||
- **Check interval**: 15-30 seconds (faster, still randomized)
|
||||
- **Duration**: 30-60 minutes after detection
|
||||
- **Purpose**: Catch the drop going live quickly
|
||||
|
||||
### 3. COOLDOWN MODE (After alert period)
|
||||
- **Gradually slow back down** to stealth mode
|
||||
- **Prevents sudden behavior change** which could trigger detection
|
||||
|
||||
### 4. SLEEP MODE (Optional - night hours)
|
||||
- **Check interval**: 5-10 minutes
|
||||
- **Active hours**: e.g., 2am-6am local time
|
||||
- **Purpose**: Real humans sleep, bots don't
|
||||
|
||||
---
|
||||
|
||||
## Human-Like Obfuscation Techniques
|
||||
|
||||
### Timing Randomization
|
||||
```python
|
||||
# Instead of exact intervals:
|
||||
time.sleep(60) # BAD - robotic
|
||||
|
||||
# Use gaussian distribution around target:
|
||||
import random
|
||||
base_interval = 60
|
||||
jitter = random.gauss(0, 10) # +/- 10 seconds standard deviation
|
||||
time.sleep(max(30, base_interval + jitter)) # GOOD - human-like
|
||||
```
|
||||
|
||||
### Decoy Requests
|
||||
Mix in non-API requests to look like a real browser:
|
||||
- Occasionally fetch an image from the site
|
||||
- Load the homepage or a random category page
|
||||
- Request favicon, robots.txt, etc.
|
||||
|
||||
```python
|
||||
DECOY_URLS = [
|
||||
"/favicon.ico",
|
||||
"/",
|
||||
"/category/plush",
|
||||
"/category/accessories",
|
||||
]
|
||||
|
||||
def make_decoy_request():
|
||||
"""Occasionally make a non-API request"""
|
||||
if random.random() < 0.2: # 20% of the time
|
||||
url = random.choice(DECOY_URLS)
|
||||
session.get(API_BASE + url)
|
||||
```
|
||||
|
||||
### Referrer Rotation
|
||||
Change the Referer header to look like you're browsing:
|
||||
```python
|
||||
REFERRERS = [
|
||||
"https://www.pokemoncenter.com/",
|
||||
"https://www.pokemoncenter.com/category/tcg-cards",
|
||||
"https://www.pokemoncenter.com/category/new-releases",
|
||||
"https://www.google.com/",
|
||||
]
|
||||
|
||||
headers["Referer"] = random.choice(REFERRERS)
|
||||
```
|
||||
|
||||
### User-Agent Variation
|
||||
Slightly vary the User-Agent (within reason):
|
||||
```python
|
||||
# Base UA with minor variations
|
||||
chrome_versions = ["146.0.0.0", "145.0.0.0", "146.0.7680.154"]
|
||||
version = random.choice(chrome_versions)
|
||||
headers["User-Agent"] = f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{version} Safari/537.36"
|
||||
```
|
||||
|
||||
### Session Breaks
|
||||
Occasionally "take a break" like a human would:
|
||||
```python
|
||||
def maybe_take_break():
|
||||
"""Occasionally pause for a longer period"""
|
||||
if random.random() < 0.05: # 5% chance
|
||||
break_duration = random.randint(120, 300) # 2-5 minutes
|
||||
logger.info(f"Taking a break for {break_duration}s...")
|
||||
time.sleep(break_duration)
|
||||
```
|
||||
|
||||
### Time-of-Day Awareness
|
||||
Adjust behavior based on time:
|
||||
```python
|
||||
from datetime import datetime
|
||||
|
||||
def get_interval_for_time():
|
||||
hour = datetime.now().hour
|
||||
|
||||
if 2 <= hour < 6:
|
||||
# Late night - slower checks
|
||||
return random.randint(300, 600) # 5-10 min
|
||||
elif 9 <= hour < 17:
|
||||
# Business hours - prime drop time
|
||||
return random.randint(45, 90) # ~1 min
|
||||
else:
|
||||
# Evening/early morning
|
||||
return random.randint(60, 120) # 1-2 min
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mode Transition Logic
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ STEALTH MODE │
|
||||
│ (45-90 sec) │
|
||||
└────────┬────────┘
|
||||
│
|
||||
New SKU detected?
|
||||
│
|
||||
YES ───────────┴─────────── NO
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│ ALERT MODE │ │
|
||||
│ (15-30 sec) │◄──────────────────┘
|
||||
└────────┬────────┘
|
||||
│
|
||||
30-60 min elapsed?
|
||||
│
|
||||
YES │
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ COOLDOWN MODE │
|
||||
│ (gradually │
|
||||
│ slow down) │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
Back to STEALTH MODE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detection Tracking
|
||||
|
||||
When a new product is detected:
|
||||
|
||||
1. **Log detection** with timestamp
|
||||
2. **Fetch full product details**
|
||||
3. **Send Discord alert**
|
||||
4. **Enter ALERT MODE**
|
||||
5. **Track if product becomes "announced"**:
|
||||
- Visible on homepage
|
||||
- Linked from category pages
|
||||
- Social media announcement
|
||||
|
||||
When product is announced, return to STEALTH MODE.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
1. [x] Basic backend monitor (done)
|
||||
2. [ ] Add timing randomization
|
||||
3. [ ] Implement mode switching
|
||||
4. [ ] Add decoy requests
|
||||
5. [ ] Add time-of-day awareness
|
||||
6. [ ] Add session breaks
|
||||
7. [ ] Integrate with Discord notifier
|
||||
|
||||
---
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Test with cookies**: Run warmup, verify API access works
|
||||
2. **Test SKU detection**: Manually add a fake SKU to known list, verify detection
|
||||
3. **Test mode switching**: Simulate detection, verify faster checks activate
|
||||
4. **Monitor for blocks**: Run for extended period, note when cookies expire
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Cookies expire | Auto-detect 403, prompt for re-warmup |
|
||||
| IP blocked | Support proxy rotation (already built) |
|
||||
| Pattern detected | Randomization + decoys + breaks |
|
||||
| Rate limited | Back off on 429 responses |
|
||||
@@ -0,0 +1,285 @@
|
||||
# PokemonCenter Backend API Monitor
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines how to build a backend API monitor that detects new products before they appear on the website - similar to what accounts like @pokepullzhq do.
|
||||
|
||||
## Why Backend Monitoring?
|
||||
|
||||
- **Faster detection**: Products are loaded into the backend/API before the frontend displays them
|
||||
- **Less aggressive**: API calls are lighter than full page loads
|
||||
- **Avoids bot detection**: Direct API calls look different than browser automation
|
||||
- **More reliable**: JSON responses are easier to parse than HTML
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Find the API Endpoints
|
||||
|
||||
### How to Investigate
|
||||
|
||||
1. Open https://www.pokemoncenter.com/category/tcg-cards in Chrome
|
||||
2. Open DevTools (F12)
|
||||
3. Go to **Network** tab
|
||||
4. Check **Preserve log**
|
||||
5. Filter by **Fetch/XHR**
|
||||
6. Refresh the page
|
||||
7. Look for API calls that return product data
|
||||
|
||||
### What to Look For
|
||||
|
||||
**Common patterns:**
|
||||
- GraphQL endpoints: `/graphql` or `/api/graphql`
|
||||
- REST APIs: `/api/products`, `/api/catalog`, `/api/search`
|
||||
- Third-party services:
|
||||
- **Algolia** (search): `*.algolia.net` or `*.algolianet.com`
|
||||
- **Contentful** (CMS): `cdn.contentful.com`
|
||||
- **Commercetools** (e-commerce): `*.commercetools.com`
|
||||
- **Salesforce Commerce**: `*.demandware.net`
|
||||
|
||||
**Signs you found the right endpoint:**
|
||||
- Response contains product names, prices, SKUs
|
||||
- Response has `inStock`, `availability`, or similar fields
|
||||
- Response includes product URLs or IDs
|
||||
|
||||
### Example Findings to Document
|
||||
|
||||
For each endpoint found, note:
|
||||
```
|
||||
URL: https://api.pokemoncenter.com/products?category=tcg
|
||||
Method: GET
|
||||
Headers:
|
||||
- Authorization: Bearer xxx (if any)
|
||||
- x-api-key: xxx (if any)
|
||||
Response format: JSON
|
||||
Contains: productId, name, price, availability, url
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: API Response Analysis
|
||||
|
||||
Once you find the product API, analyze the response structure:
|
||||
|
||||
### Key Fields to Track
|
||||
|
||||
```json
|
||||
{
|
||||
"products": [
|
||||
{
|
||||
"id": "12345", // Unique product ID
|
||||
"sku": "PKM-CR-ETB", // SKU code
|
||||
"name": "Chaos Rising ETB",
|
||||
"url": "/product/chaos-rising-etb",
|
||||
"price": 54.99,
|
||||
"availability": {
|
||||
"inStock": false, // Current stock status
|
||||
"preorder": true, // Pre-order available?
|
||||
"releaseDate": "2026-04-15"
|
||||
},
|
||||
"status": "ACTIVE", // May be "HIDDEN", "DRAFT" before launch
|
||||
"publishedAt": null // null = not yet visible on site
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Detection Strategies
|
||||
|
||||
1. **New Product Detection**
|
||||
- Compare product IDs against known list
|
||||
- New ID = new product added to backend
|
||||
|
||||
2. **Pre-Launch Detection**
|
||||
- Product exists but `publishedAt` is null
|
||||
- Product has `status: "DRAFT"` or similar
|
||||
- Product `availability.inStock` changes from false to true
|
||||
|
||||
3. **Stock Change Detection**
|
||||
- Track `inStock` or `availability` field changes
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Implementation Plan
|
||||
|
||||
### Option A: Add to Chrome Extension
|
||||
|
||||
```javascript
|
||||
// In background.js - add API monitoring alongside page monitoring
|
||||
|
||||
async function checkBackendAPI() {
|
||||
const API_URL = "https://api.pokemoncenter.com/products?category=tcg";
|
||||
|
||||
try {
|
||||
const response = await fetch(API_URL, {
|
||||
headers: {
|
||||
// Add any required headers discovered during investigation
|
||||
'Accept': 'application/json',
|
||||
}
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
const products = data.products || [];
|
||||
|
||||
// Compare against known products
|
||||
for (const product of products) {
|
||||
if (!knownBackendProducts[product.id]) {
|
||||
// NEW PRODUCT DETECTED!
|
||||
await sendDiscordNotification({
|
||||
name: product.name,
|
||||
url: `https://www.pokemoncenter.com${product.url}`,
|
||||
price: `$${product.price}`,
|
||||
inStock: product.availability?.inStock
|
||||
}, "backend_detect");
|
||||
|
||||
knownBackendProducts[product.id] = product;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Backend API check failed:", error);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Option B: Standalone Python Script
|
||||
|
||||
```python
|
||||
# backend_monitor.py
|
||||
import requests
|
||||
import time
|
||||
import json
|
||||
from discord_webhook import DiscordWebhook, DiscordEmbed
|
||||
|
||||
API_URL = "https://api.pokemoncenter.com/products"
|
||||
DISCORD_WEBHOOK = "your-webhook-url"
|
||||
KNOWN_PRODUCTS_FILE = "known_backend_products.json"
|
||||
CHECK_INTERVAL = 30 # seconds - can be faster for API
|
||||
|
||||
def load_known_products():
|
||||
try:
|
||||
with open(KNOWN_PRODUCTS_FILE, 'r') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
return {}
|
||||
|
||||
def save_known_products(products):
|
||||
with open(KNOWN_PRODUCTS_FILE, 'w') as f:
|
||||
json.dump(products, f)
|
||||
|
||||
def check_api():
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0...',
|
||||
# Add discovered headers
|
||||
}
|
||||
|
||||
response = requests.get(API_URL, headers=headers)
|
||||
return response.json()
|
||||
|
||||
def send_alert(product, alert_type):
|
||||
webhook = DiscordWebhook(url=DISCORD_WEBHOOK, content="@everyone")
|
||||
|
||||
embed = DiscordEmbed(
|
||||
title=f"🚨 {alert_type.upper()}",
|
||||
description=f"**{product['name']}**",
|
||||
color=0xFF0000 if alert_type == "BACKEND DETECT" else 0x00FF00
|
||||
)
|
||||
embed.add_embed_field(name="Price", value=f"${product.get('price', 'TBD')}")
|
||||
embed.add_embed_field(name="Status", value=product.get('status', 'Unknown'))
|
||||
embed.add_embed_field(name="Link", value=f"[VIEW]({product['url']})", inline=False)
|
||||
|
||||
webhook.add_embed(embed)
|
||||
webhook.execute()
|
||||
|
||||
def main():
|
||||
known = load_known_products()
|
||||
print(f"Loaded {len(known)} known products")
|
||||
|
||||
while True:
|
||||
try:
|
||||
data = check_api()
|
||||
products = data.get('products', [])
|
||||
|
||||
for product in products:
|
||||
pid = product['id']
|
||||
|
||||
if pid not in known:
|
||||
print(f"NEW BACKEND PRODUCT: {product['name']}")
|
||||
send_alert(product, "BACKEND DETECT")
|
||||
known[pid] = product
|
||||
save_known_products(known)
|
||||
|
||||
# Check for status changes
|
||||
elif known[pid].get('status') != product.get('status'):
|
||||
print(f"STATUS CHANGE: {product['name']}")
|
||||
send_alert(product, "STATUS CHANGE")
|
||||
known[pid] = product
|
||||
save_known_products(known)
|
||||
|
||||
print(f"Checked {len(products)} products")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
|
||||
time.sleep(CHECK_INTERVAL)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Rate Limiting Considerations
|
||||
|
||||
### API vs Page Scraping
|
||||
|
||||
| Approach | Safe Interval | Notes |
|
||||
|----------|---------------|-------|
|
||||
| Full page load | 60-90 sec | Heavy, triggers bot detection |
|
||||
| API call | 15-30 sec | Lighter, more tolerant |
|
||||
| GraphQL query | 15-30 sec | Depends on complexity |
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use proper headers**: Include realistic User-Agent, Accept, etc.
|
||||
2. **Don't hammer**: Even APIs have rate limits
|
||||
3. **Cache responses**: Don't re-process unchanged data
|
||||
4. **Handle 429s gracefully**: Back off when rate limited
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Advanced - Multiple Detection Layers
|
||||
|
||||
For maximum coverage, run both:
|
||||
|
||||
1. **Backend API Monitor** (every 30 sec)
|
||||
- Fast detection of new products in system
|
||||
- Lighter on resources
|
||||
|
||||
2. **Frontend Page Monitor** (every 90 sec)
|
||||
- Confirms products are live on website
|
||||
- Catches anything API might miss
|
||||
|
||||
### Alert Priority
|
||||
|
||||
```
|
||||
BACKEND DETECT (API) = "Product loaded, drop imminent!"
|
||||
FRONTEND DETECT (Page) = "Product is LIVE, buy now!"
|
||||
RESTOCK = "Back in stock!"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Wait for rate limit to clear** (~30 min)
|
||||
2. **Investigate the API** using DevTools Network tab
|
||||
3. **Document the endpoints** you find
|
||||
4. **Share the findings** so we can build the monitor
|
||||
|
||||
### Questions to Answer During Investigation
|
||||
|
||||
- [ ] What URL serves product data?
|
||||
- [ ] What headers are required?
|
||||
- [ ] Is authentication needed?
|
||||
- [ ] What does the response structure look like?
|
||||
- [ ] Are there pagination parameters?
|
||||
- [ ] Is there a "hidden" or "draft" status visible in API?
|
||||
Reference in New Issue
Block a user