feat: Implement Walmart scraper and integrate with existing architecture

- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details.
- Introduced a stealth browser module to handle bot protection and improve scraping reliability.
- Created a SQLite database for tracking product history, price changes, stock events, and user favorites.
- Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores.
- Implemented a favorites system to manage priority products and categories with custom notification settings.
- Added news aggregation module to fetch and analyze Pokemon TCG news from various sources.
- Created tools for API discovery and monitoring, including a backend monitor for detecting new products.
- Added unit tests for database operations, product filtering, and API endpoints to ensure functionality.
- Enhanced existing modules with improved error handling and logging for better maintainability.
This commit is contained in:
2026-03-27 23:08:09 -04:00
parent cddae24e34
commit 8d382e723f
64 changed files with 15433 additions and 443 deletions
+137
View File
@@ -0,0 +1,137 @@
# Pokemon Stock Monitor - Project Guide
## Overview
A Pokemon TCG stock monitoring system that tracks retail sites (Target, PokemonCenter, GameStop, Best Buy, Walmart) for restocks and new drops, sending Discord notifications when products become available.
## Architecture
```
User -> Chrome Extension (PokemonCenter API interception)
-> Python Monitor (main.py) -> Scrapers -> ProductTracker -> Discord Notifications
-> Flask Dashboard (localhost:5000) -> REST API
```
## Directory Structure
### Core Application
- `main.py` - Entry point, scheduling loop, orchestrates scraping
- `config.py` - All configuration (intervals, URLs, Discord webhook, browser settings)
### Source Code (`src/`)
- `browser.py` - Playwright/Chrome browser management
- `product_tracker.py` - Detects new products and restocks, manages state
- `discord_notifier.py` - Sends Discord webhook notifications
- `discord_bot.py` - Interactive Discord bot commands
- `database.py` - SQLite database for historical tracking
- `favorites.py` - User favorites/watchlist management
- `scraper_state.py` - Persists scraper state between runs
- `store_locator.py` - Target store location lookup
### Scrapers (`scrapers/`)
- `base.py` - `Product` dataclass, `BaseScraper` abstract class
- `pokemoncenter.py` - PokemonCenter scraper (needs stealth browser)
- `target.py` - Target.com scraper
- `gamestop.py` - GameStop.com scraper (light bot protection)
- `bestbuy.py` - BestBuy.com scraper (moderate bot protection)
- `walmart.py` - Walmart.com scraper (aggressive bot protection, may need stealth)
### Chrome Extension (`chrome-extension/`)
- `manifest.json` - Extension config (MV3)
- `background.js` - Service worker, manages alarms and notifications
- `content.js` - Injected into PokemonCenter pages
- `api-interceptor.js` - Intercepts PokemonCenter API responses
- `popup.html/js` - Extension popup UI
### Dashboard (`dashboard/`)
- `app.py` - Flask app, routes
- `api.py` - REST API endpoints (`/api/*`)
- `templates/index.html` - Main template
- `static/app.js` - Frontend JavaScript
### Data (`data/`)
- `products.json` - Tracked products state
- `scraper_state.json` - Persisted scraper state
- `known_skus.json` - Known product SKUs
- `proxies.json` - Proxy configuration
- `user_locations.json` - User store locations
### Tools (`tools/`)
Development and debugging utilities:
- `api_monitor.py`, `backend_monitor.py` - API monitoring
- `stealth_browser.py` - Anti-detection browser
- `capture_api_calls.py` - HAR capture for debugging
## Key Classes
### `Product` (scrapers/base.py)
```python
@dataclass
class Product:
name: str
url: str
price: Optional[str]
in_stock: bool
image_url: Optional[str]
site: str
product_id: Optional[str]
```
### `ProductTracker` (src/product_tracker.py)
- `process_products(products)` -> `(new_products, restocked_products)`
- `get_stats()` -> dashboard statistics
### `BaseScraper` (scrapers/base.py)
- `scrape_category_page(url)` -> `List[Product]`
- `check_product_stock(url)` -> `(in_stock, price)`
## Common Tasks
### Add a new retailer scraper
1. Create `scrapers/newsite.py` extending `BaseScraper`
2. Implement `scrape_category_page()` and `check_product_stock()`
3. Add to `scrapers/__init__.py`
4. Add config in `config.py` (URLs, enable flag)
5. Add check function in `main.py`
### Modify Discord notifications
- Webhook format: `src/discord_notifier.py`
- Bot commands: `src/discord_bot.py`
### Change check intervals or URLs
- Edit `config.py`
### Dashboard changes
- Routes: `dashboard/app.py`
- API: `dashboard/api.py`
- Frontend: `dashboard/static/app.js`
### Chrome extension changes
- Popup UI: `chrome-extension/popup.html`, `popup.js`
- API interception: `chrome-extension/api-interceptor.js`
- Background logic: `chrome-extension/background.js`
## Conventions
- Python 3.13+
- Logging via `logging` module (configured in main.py)
- Dataclasses for data structures
- Flask for web/API
- Playwright for browser automation
- SQLite for persistence (`stats.db`)
## Running
```bash
# Main monitor
python main.py
# Dashboard only
python dashboard/app.py
# Discord bot
python src/discord_bot.py
```
## Testing
```bash
python -m pytest tests/
```