Add deduplication and normalize names/URLs

Add product name cleanup and URL normalization across backend, scrapers, and frontend; introduce product/event deduplication. Key changes:
- Clean up SVG/CSS garbage, trailing "X Reviews", mojibake and extra spaces in product names (api, scrapers, JS).
- Normalize URLs (lowercase host, strip tracking/query params, trim trailing slashes) in scrapers and Database and use normalized URL for comparisons/storage.
- Database: use normalized URLs when creating/getting products, prevent duplicate stock events within 1 hour, adjust recent events query to avoid duplicate products.
- Implement deduplication routines in Database: deduplicate_products (by product_id and normalized/clean name) and deduplicate_events (remove duplicate events within same hour).
- API: add POST endpoints /products/deduplicate and /events/deduplicate to trigger deduplication and return stats.
- Frontend: add UI buttons and handlers to call deduplication endpoints and display results.

These changes reduce duplicate product records/events caused by minor URL/name variations and stray CSS/svg artifacts.
This commit is contained in:
2026-03-28 13:31:30 -04:00
parent ad5d179ba3
commit f01bde54a5
5 changed files with 366 additions and 20 deletions
+10
View File
@@ -356,6 +356,16 @@
<button id="checkBrokenUrlsBtn" class="btn-secondary">Check Broken URLs</button>
<button id="cleanupBrokenUrlsBtn" class="btn-danger">Cleanup Broken URLs</button>
<p id="brokenUrlsResult" class="help-text"></p>
<h3>Duplicate Products</h3>
<p class="help-text">Remove duplicate products from Target, GameStop, and other stores</p>
<button id="deduplicateBtn" class="btn-secondary">Remove Duplicates</button>
<p id="deduplicateResult" class="help-text"></p>
<h3>Duplicate Events</h3>
<p class="help-text">Remove duplicate events from the activity feed (same product/type within 1 hour)</p>
<button id="deduplicateEventsBtn" class="btn-secondary">Clean Up Events</button>
<p id="deduplicateEventsResult" class="help-text"></p>
</div>
</section>
</main>