- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
4.9 KiB
Pokemon Center Bot Detection - Strategies & Solutions
Pokemon Center uses Imperva (Incapsula) bot protection, which is one of the most aggressive anti-bot systems. Here are practical approaches to work around it.
Current Problem
- Direct API calls: 403 Forbidden
- Sitemap.xml: Blocked by Imperva
- Browser automation: CAPTCHA after a few requests
Strategy 1: Third-Party Stock Trackers (Easiest)
Instead of scraping Pokemon Center directly, integrate with existing stock tracking services:
Discord Bots/Webhooks
Several Discord servers track Pokemon Center in real-time:
- Pokemon TCG Drops - Dedicated Pokemon TCG stock alerts
- Stock Informer - General retail stock tracking
- Search Discord for "Pokemon Center stock alerts"
You can join these servers and set up webhook forwarding to your own Discord.
Stock Alert Services
- NowInStock.net - Has Pokemon Center tracking
- Distill.io - Browser extension for change detection
- Visualping - Monitors page changes
Implementation
# Forward alerts from a public tracker to your system
# Set up a Discord bot to listen to stock alert channels
# Then forward to your own notification system
Strategy 2: Undetected Chrome Driver
Use undetected-chromedriver which patches Chrome to avoid detection:
pip install undetected-chromedriver
import undetected_chromedriver as uc
driver = uc.Chrome(headless=False) # Headless often gets detected
driver.get("https://www.pokemoncenter.com/category/tcg-cards")
# Add human-like delays
import time
import random
time.sleep(random.uniform(3, 7))
# Scroll like a human
driver.execute_script("window.scrollBy(0, 500)")
time.sleep(random.uniform(1, 3))
Pros: Often bypasses Imperva Cons: Slower, still may trigger CAPTCHA eventually
Strategy 3: Session Persistence
Keep a browser session alive and logged in:
- Manual login once - Complete any CAPTCHA manually
- Save cookies - Export session cookies
- Reuse session - Load cookies on each check
- Longer intervals - Check every 3-5 minutes instead of 60-90 seconds
# Save cookies after manual login
import pickle
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))
# Load cookies on subsequent runs
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
Strategy 4: Residential Proxy Rotation
Use residential proxies that look like real home internet connections:
Providers
- Bright Data (formerly Luminati) - Best but expensive
- Oxylabs - Good quality
- Smartproxy - Budget option
- IPRoyal - Pay-per-GB
Cost
- ~$10-15/GB for residential proxies
- ~50-100 requests per MB depending on page size
proxies = {
"http": "http://user:pass@proxy.provider.com:port",
"https": "http://user:pass@proxy.provider.com:port"
}
Strategy 5: Human-Like Behavior Pattern
If continuing with browser automation:
import random
import time
def human_like_check():
# Random delay between checks (2-5 minutes)
delay = random.uniform(120, 300)
# Random user agents
user_agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
]
# Vary timing throughout the day
# Check more during business hours, less at night
hour = datetime.now().hour
if 2 <= hour <= 6: # 2am-6am
delay *= 2 # Slower at night
# Add random mouse movements
# Add random scrolling
# Occasionally visit other pages (home, about)
return delay
Strategy 6: Webhook from Manual Monitoring
The most reliable approach for Pokemon Center:
- Open Pokemon Center in your actual browser
- Use Distill.io browser extension to monitor changes
- Set up webhook to forward alerts to your system
This way:
- You're using a real browser with real cookies
- Imperva sees normal human behavior
- Change detection triggers your notification system
Recommended Approach
Given Pokemon Center's aggressive protection, I recommend a hybrid approach:
-
Primary: Join existing stock alert Discord servers
- Let others deal with the bot detection
- Forward their alerts to your system
-
Secondary: Use undetected-chromedriver with long intervals
- Check every 3-5 minutes
- Use session persistence
- Add human-like behavior
-
Backup: Manual Distill.io monitoring
- For specific products you really care about
- Most reliable but requires keeping browser open
Code Changes Needed
To implement these strategies, we would need to:
- Add
undetected-chromedriveras an option - Implement cookie/session persistence
- Add configurable random delays
- Add proxy support
- Consider adding Discord bot listener for third-party alerts
Would you like me to implement any of these strategies?