go-event-alerts

GO Event Alerts - Backend

This directory contains the Python backend for the Pokémon GO Event Notifier project. The backend is responsible for scraping event information, processing it with a series of LLMs, and storing it in DynamoDB.

Table of Contents

• Overview
• Features
• Setup & Installation
• Configuration
• Running the Pipeline

Overview

The backend is a data processing pipeline that performs the following steps:

1. Scrapes event data from pokemongolive.com and leekduck.com.
2. Processes the raw HTML content using a multi-stage LLM pipeline to extract, refine, and deduplicate structured data.
3. Stores the final, clean event data in an AWS DynamoDB table.

Features

• Modular Design: The pipeline is broken down into modules for scraping, AI processing, data cleaning, and cloud interaction.
• LLM Integration: Leverages the power of LLMs to understand and structure unstructured text.
• Timezone Aware: Correctly handles timezones for global and local events.
• Idempotent: The pipeline can be run multiple times without creating duplicate entries.

LLM Processing Pipeline

A key feature of this backend is its multi-stage approach to data refinement using several targeted LLM calls. This strategy, favoring multiple, specialized calls to smaller models over a single call to a large one, ensures both high data quality and cost-effectiveness.

1. Initial Structuring

• Source-Specific Prompts: Raw, unstructured text from each data source (pokemongolive.com and leekduck.com) is processed using unique, hand-crafted prompts. These prompts provides the LLM with extensive context, examples, and precise formatting instructions to convert the messy input into a clean, structured JSON object.

2. Quality Assurance & Classification

• Type Correction: After initial structuring, each event undergoes a quality check. A dedicated, rule-based LLM call (correct_event_type) analyzes the event’s details to fix common classification errors (e.g., correctly identifying a “Live Event” based on a city name in the summary).
• Importance of Classification: Accurate event type classification is fundamental to the app’s user experience, as it allows users to subscribe to notifications for the specific categories they care about (e.g., “Community Day”, “Raid Day”).

3. Semantic Deduplication

Since the same event is often announced on multiple sites with slightly different wording and timing, a sophisticated deduplication process is required.

• Heuristic Pre-filtering: To minimize expensive LLM calls, a cheap heuristic check is performed first. It compares the start/end times and durations of potential duplicates. Only if the events are temporally close does the process continue.
• LLM-Powered Duplicate Detection: If the heuristic check passes, a specialized LLM call (are_events_duplicates_llm) determines if the two events are semantically the same, despite differences in their descriptions.
• Quality-Based Selection: When a duplicate pair is confirmed, a final LLM call (choose_better_event_llm) compares the two JSON objects and selects the one with the most complete and specific information (e.g., preferring an exact time over a placeholder, or a more descriptive summary). This ensures the data stored in the database is of the highest possible quality.

Setup & Installation

1. Clone the repository:

   git clone [https://github.com/sthoresen/go-event-alerts.git](https://github.com/sthoresen/go-event-alerts.git)
   cd go-event-alerts/backend
   

2. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
   

3. Install the dependencies:

   pip install -r requirements.txt
   

Configuration

The backend requires the following environment variables to be set in a .env file:

• OPENAI_API_KEY: Your API key for OpenAI.
• AWS_ACCESS_KEY_ID: Your AWS access key.
• AWS_SECRET_ACCESS_KEY: Your AWS secret access key.
• AWS_REGION: The AWS region where your DynamoDB tables are located.

Running the Pipeline

To run the data processing pipeline manually, execute the lambda_function.py script:

```bash python lambda_function.py

This site is open source. Improve this page.