# Reddit Scraper App Architecture

## Overview
This document outlines the architecture for a local desktop app that helps users find Reddit posts and comments where they can add value and comment on them using Qwen2.5 3b (these model is chosen because of his reasoning level and speed).

## High-Level System Diagram

```mermaid
graph TD
    A[User] --> B[GUI] - The user interface
    B --> C[Scraper] - It does the scraping and finds the material
    C --> D[Reddit] - It checks posts and comments on Reddit
    D --> C
    C --> E[Relevance Scorer] - It scores them based on relevance to the user's input and what the user is looking for
    E --> F[Kilo Knowledge Base] - Because this project was made for Kilo, it was trained on the full documentation and available information about Kilo to find the best matches
    E --> G[Results]
    G --> B

## Modules and Responsibilities

### 1. GUI Module
- **Responsibility**: User interaction and display of results.
- **Functionality**:
  - Search bar for user input.
  - Results display showing link, type, subreddit, age, and relevance score.
  - Filters for relevance, age, etc.

### 2. Scraper Module
- **Responsibility**: Scraping Reddit posts and comments.
- **Functionality**:
  - HTML scraping using BeautifulSoup.
  - Dynamic content scraping using Puppeteer.
  - Anti-ban strategies (rate limits, delays, headers).

### 3. Relevance Scorer Module
- **Responsibility**: Scoring the relevance of posts and comments.
- **Functionality**:
  - Uses the Kilo Knowledge Base to determine relevance.
  - Computes relevance score based on keyword matches and contextual relevance.

### 4. Kilo Knowledge Base
- **Responsibility**: Storing information about Kilo Code.
- **Functionality**:
  - Stores keywords and contextual information.
  - Used by the Relevance Scorer to determine relevance.

## Data Flow

1. **User Input**: The user inputs a search query through the GUI.
2. **Scraping**: The Scraper Module scrapes Reddit for posts and comments based on the query.
3. **Relevance Scoring**: The Relevance Scorer Module scores the scraped data using the Kilo Knowledge Base.
4. **Results Display**: The GUI displays the scored results to the user.

## Anti-Ban Strategy

1. **Rate Limits**: Implement delays between requests to avoid triggering rate limits.
2. **Headers**: Use realistic headers to mimic a human user.
3. **User-Agent Rotation**: Rotate user agents to avoid detection.
4. **Proxy Rotation**: Use proxies to distribute requests.

## Kilo Knowledge Storage

The Kilo Knowledge Base will be stored locally in a structured format (e.g., JSON or SQLite). This will include:
- Keywords related to Kilo Code.
- Contextual information about Kilo Code.
- Relevance scoring criteria.

## Scraping Strategy

**Option A**: Direct HTML scraping using libraries like BeautifulSoup.
**Option B**: Use a headless browser like Puppeteer for dynamic content.

**Combined Strategy**:
- Use BeautifulSoup for static content.
- Use Puppeteer for dynamic content and to handle JavaScript-rendered pages.

## Scoring Logic

The relevance score will be computed based on:
- Keyword matches in the post/comment.
- Contextual relevance to Kilo Code.
- Age of the post/comment (newer posts may be more relevant).

## GUI Structure

1. **Search Bar**: For user input.
2. **Results Display**: Shows the link, type, subreddit, age, and relevance score.
3. **Filters**: Allow the user to filter results by relevance, age, etc.

## Tech Stack

- **Language**: Python (for its rich ecosystem of scraping and GUI libraries).
- **Libraries**:
  - BeautifulSoup for HTML scraping.
  - Puppeteer for dynamic content.
  - Tkinter or PyQt for the GUI.
  - SQLite for local storage of the Kilo Knowledge Base.

## MVP vs Future Upgrades

**MVP**:
- Basic GUI with search functionality.
- Simple scraping using BeautifulSoup.
- Basic relevance scoring.

**Future Upgrades**:
- Advanced scraping with Puppeteer.
- Enhanced relevance scoring with machine learning.
- Support for more platforms (e.g., Twitter, Hacker News).