8d382e723f
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
4.1 KiB
4.1 KiB
Pokemon Stock Monitor - Project Guide
Overview
A Pokemon TCG stock monitoring system that tracks retail sites (Target, PokemonCenter, GameStop, Best Buy, Walmart) for restocks and new drops, sending Discord notifications when products become available.
Architecture
User -> Chrome Extension (PokemonCenter API interception)
-> Python Monitor (main.py) -> Scrapers -> ProductTracker -> Discord Notifications
-> Flask Dashboard (localhost:5000) -> REST API
Directory Structure
Core Application
main.py- Entry point, scheduling loop, orchestrates scrapingconfig.py- All configuration (intervals, URLs, Discord webhook, browser settings)
Source Code (src/)
browser.py- Playwright/Chrome browser managementproduct_tracker.py- Detects new products and restocks, manages statediscord_notifier.py- Sends Discord webhook notificationsdiscord_bot.py- Interactive Discord bot commandsdatabase.py- SQLite database for historical trackingfavorites.py- User favorites/watchlist managementscraper_state.py- Persists scraper state between runsstore_locator.py- Target store location lookup
Scrapers (scrapers/)
base.py-Productdataclass,BaseScraperabstract classpokemoncenter.py- PokemonCenter scraper (needs stealth browser)target.py- Target.com scrapergamestop.py- GameStop.com scraper (light bot protection)bestbuy.py- BestBuy.com scraper (moderate bot protection)walmart.py- Walmart.com scraper (aggressive bot protection, may need stealth)
Chrome Extension (chrome-extension/)
manifest.json- Extension config (MV3)background.js- Service worker, manages alarms and notificationscontent.js- Injected into PokemonCenter pagesapi-interceptor.js- Intercepts PokemonCenter API responsespopup.html/js- Extension popup UI
Dashboard (dashboard/)
app.py- Flask app, routesapi.py- REST API endpoints (/api/*)templates/index.html- Main templatestatic/app.js- Frontend JavaScript
Data (data/)
products.json- Tracked products statescraper_state.json- Persisted scraper stateknown_skus.json- Known product SKUsproxies.json- Proxy configurationuser_locations.json- User store locations
Tools (tools/)
Development and debugging utilities:
api_monitor.py,backend_monitor.py- API monitoringstealth_browser.py- Anti-detection browsercapture_api_calls.py- HAR capture for debugging
Key Classes
Product (scrapers/base.py)
@dataclass
class Product:
name: str
url: str
price: Optional[str]
in_stock: bool
image_url: Optional[str]
site: str
product_id: Optional[str]
ProductTracker (src/product_tracker.py)
process_products(products)->(new_products, restocked_products)get_stats()-> dashboard statistics
BaseScraper (scrapers/base.py)
scrape_category_page(url)->List[Product]check_product_stock(url)->(in_stock, price)
Common Tasks
Add a new retailer scraper
- Create
scrapers/newsite.pyextendingBaseScraper - Implement
scrape_category_page()andcheck_product_stock() - Add to
scrapers/__init__.py - Add config in
config.py(URLs, enable flag) - Add check function in
main.py
Modify Discord notifications
- Webhook format:
src/discord_notifier.py - Bot commands:
src/discord_bot.py
Change check intervals or URLs
- Edit
config.py
Dashboard changes
- Routes:
dashboard/app.py - API:
dashboard/api.py - Frontend:
dashboard/static/app.js
Chrome extension changes
- Popup UI:
chrome-extension/popup.html,popup.js - API interception:
chrome-extension/api-interceptor.js - Background logic:
chrome-extension/background.js
Conventions
- Python 3.13+
- Logging via
loggingmodule (configured in main.py) - Dataclasses for data structures
- Flask for web/API
- Playwright for browser automation
- SQLite for persistence (
stats.db)
Running
# Main monitor
python main.py
# Dashboard only
python dashboard/app.py
# Discord bot
python src/discord_bot.py
Testing
python -m pytest tests/