feat: Implement Walmart scraper and integrate with existing architecture
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
"""
|
||||
Session Warmup Tool for Pokemon Center
|
||||
|
||||
Run this BEFORE starting the monitor to:
|
||||
1. Open a stealth browser
|
||||
2. Let you manually browse and solve any CAPTCHAs
|
||||
3. Save cookies/session for the monitor to reuse
|
||||
|
||||
Usage:
|
||||
python warmup_session.py
|
||||
|
||||
After running:
|
||||
1. Browse Pokemon Center normally for a few minutes
|
||||
2. Add items to cart, look at products, etc.
|
||||
3. Solve any CAPTCHAs that appear
|
||||
4. Press Enter in this terminal when done
|
||||
5. The session will be saved and reused by the monitor
|
||||
"""
|
||||
|
||||
import sys
|
||||
import time
|
||||
|
||||
print("=" * 60)
|
||||
print("Pokemon Center Session Warmup")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# Import stealth browser
|
||||
try:
|
||||
from .stealth_browser import StealthBrowser
|
||||
except ImportError as e:
|
||||
print(f"Error importing stealth browser: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
print("Starting stealth browser...")
|
||||
print("This will open a Chrome window.")
|
||||
print()
|
||||
|
||||
browser = StealthBrowser(
|
||||
headless=False,
|
||||
session_name="pokemoncenter" # Same name used by monitor
|
||||
)
|
||||
|
||||
try:
|
||||
browser.start()
|
||||
print("[OK] Browser started!")
|
||||
print()
|
||||
|
||||
# Navigate to Pokemon Center
|
||||
print("Navigating to Pokemon Center...")
|
||||
browser.driver.get("https://www.pokemoncenter.com")
|
||||
|
||||
time.sleep(3)
|
||||
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("WARMUP INSTRUCTIONS")
|
||||
print("=" * 60)
|
||||
print("""
|
||||
1. If you see a CAPTCHA or "Pardon Our Interruption":
|
||||
- Solve it in the browser window
|
||||
- Wait for the page to load
|
||||
|
||||
2. Browse naturally for 2-3 minutes:
|
||||
- Click on some products
|
||||
- Look at different categories
|
||||
- Add something to cart (you don't have to buy)
|
||||
- This builds a legitimate browsing profile
|
||||
|
||||
3. Navigate to the TCG section:
|
||||
- https://www.pokemoncenter.com/category/tcg-cards
|
||||
|
||||
4. Once you're browsing normally without issues:
|
||||
- Come back here
|
||||
- Press ENTER to save the session
|
||||
""")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
input("Press ENTER when you're done browsing to save the session...")
|
||||
|
||||
print()
|
||||
print("Saving session cookies...")
|
||||
browser._save_cookies()
|
||||
|
||||
# Check how many cookies we got
|
||||
cookies = browser.driver.get_cookies()
|
||||
print(f"[OK] Saved {len(cookies)} cookies")
|
||||
|
||||
# Get current URL for reference
|
||||
current_url = browser.driver.current_url
|
||||
print(f"[OK] Final URL: {current_url}")
|
||||
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("SESSION SAVED!")
|
||||
print("=" * 60)
|
||||
print("""
|
||||
Your session has been saved. The monitor will now use these
|
||||
cookies when checking Pokemon Center.
|
||||
|
||||
Tips for best results:
|
||||
- Run the monitor with 3-5 minute check intervals
|
||||
- Keep USE_STEALTH_BROWSER = True in config.py
|
||||
- If you get blocked again, run this warmup again
|
||||
|
||||
Session file: data/sessions/pokemoncenter_cookies.pkl
|
||||
""")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
print("Closing browser...")
|
||||
browser.stop()
|
||||
print("Done!")
|
||||
Reference in New Issue
Block a user