AI-driven system for processing receipt images, classifying line items and tracking personal spending.
Spendify is a comprehensive receipt processing and classification system that combines Discord bot integration, Google Cloud AI services, and multi-agent AI pipelines to automatically categorize and analyze receipt data.
Discord Bot → Flask API → Google Document AI → ADK Classification Pipeline → Firebase
- Discord Bot – receives receipt images from users and forwards them to the API.
- Flask API – orchestrates OCR, classification and data storage, also serving a simple dashboard.
- ADK Pipeline – multi-agent system that groups and validates receipt items.
- Firebase – persists raw OCR data and summarised spending information.
USERDATA/ ├── {primary_id}/ │ ├── sources/ │ │ └── {source}: {identifier} │ └── metadata SESSIONS/ ├── {session_id}/ │ ├── timestamp │ ├── user_id │ └── source DATA/ ├── RAW_DATA/ │ └── {date}/ │ └── {session_id}: raw_ocr_data ├── RECEIPTS/ │ └── {date}/ │ └── {session_id}: extracted_entities └── SUMMARIES/ └── {date}/ └── {session_id}: classified_data
| Folder | Description |
|---|---|
discord_bot |
Bot for collecting images from Discord users. |
flask_api |
Flask service performing OCR and classification. |
adk_pipeline |
Agent Development Kit pipeline for receipts. |
Spendify/
├── discord_bot/ # Discord bot module
│ ├── bot.py # Main Discord bot
│ ├── requirements.txt # Bot dependencies
│ ├── deploy-bot.md # Bot deployment guide
│ ├── .env.template # Environment template
│ ├── img_content/ # Bot image content
│ └── README.md # Bot documentation
├── flask_api/ # Main API server module
│ ├── main_api.py # Flask API server
│ ├── gcp_docai.py # OCR processing
│ ├── firebase_store.py # Data storage
│ ├── gcp_adk_classification.py # ADK client
│ ├── requirements.txt # API dependencies
│ ├── deploy-api.md # API deployment guide
│ ├── deploy.sh # Deployment script
│ ├── Dockerfile # Docker configuration
│ ├── firebaseConfig.json.template # Firebase config template
│ ├── serviceAccountKey.json.template # Service account template
│ ├── .env.template # Environment template
│ ├── dashboard/ # Web dashboard
│ │ ├── index.html # Main dashboard
│ │ ├── login.html # Login page
│ │ ├── register.html # Registration page
│ │ ├── upload.html # Upload page
│ │ ├── chat.html # Chat interface
│ │ └── already_registered.html # Registration status
│ ├── uploads/ # API file uploads
│ └── README.md # API documentation
├── adk_pipeline/ # Agent Development Kit pipeline
│ ├── receipt_classifier/ # Agent pipeline
│ │ ├── agent.py # Root agent orchestrator
│ │ ├── requirements.txt # Agent dependencies
│ │ ├── firebase_credentials.json.template # Firebase template
│ │ ├── .env.template # Environment template
│ │ ├── __init__.py # Package initialization
│ │ └── subagents/ # Individual processing agents
│ │ ├── classifier_init/ # Initial classification agent
│ │ │ ├── agent.py # Categorizes line items
│ │ │ └── __init__.py # Agent initialization
│ │ ├── classification_grouper/ # Item grouping agent
│ │ │ ├── agent.py # Groups items by category
│ │ │ ├── tools.py # Grouping utilities
│ │ │ └── __init__.py # Agent initialization
│ │ ├── classification_reviewer/ # Validation agent
│ │ │ ├── agent.py # Validates classifications
│ │ │ ├── tools.py # Review utilities
│ │ │ └── __init__.py # Agent initialization
│ │ ├── classification_refiner/ # Correction agent
│ │ │ ├── agent.py # Refines misclassifications
│ │ │ └── __init__.py # Agent initialization
│ │ └── classification_response/ # Final response agent
│ │ ├── agent.py # Generates final summary
│ │ ├── tools.py # Response utilities
│ │ ├── firebase_store.py # Data persistence
│ │ └── __init__.py # Agent initialization
│ ├── evals/ # Evaluation data
│ ├── deploy-adk.md # ADK deployment guide
│ ├── deploy.sh # Deployment script
│ ├── flow.png # Pipeline flow diagram
│ ├── firebase_credentials.json # Firebase credentials
│ ├── .env.template # Environment template
│ └── README.md # ADK documentation
├── process.png # System process diagram
└── README.md # Project documentation
- Install Python requirements in each module & Configure
.envfiles using the provided templates. - Start the ADK Server:
cd adk_pipeline adk web - In another terminal, run the Flask API:
cd flask_api python main_api.py - In another terminal, run the Discord bot:
cd discord_bot python bot.py - Access the dashboard at
http://localhost:8080/(or your configured port).
- Accepts image uploads from Discord users
- Handles user registration and authentication
- Manages file uploads and session tracking
- Provides real-time feedback to users
- Central Flask API server
- Orchestrates the entire processing pipeline
- Handles user management and data flow
- Integrates all system components
- Extracts structured data from receipt images
- Identifies line items, totals, taxes, and merchant information
- Provides entity recognition and text extraction
- Manages all data persistence operations
- Stores user data, sessions, raw OCR data, and classifications
- Provides structured data organization across collections
- HTTP client for Google Agent Development Kit
- Handles communication with multi-agent systems
- Manages session creation and event streaming
Multi-agent system with sequential processing:
- Initial Classifier: Categorizes line items (Groceries, Fast Food, etc.)
- Grouping Agent: Groups items by category with totals
- Validation Loop:
- Reviewer: Validates classification accuracy
- Refiner: Corrects misclassifications
- Response Agent: Generates final summary and saves to Firebase
- User uploads receipt image via Discord
- Bot saves image locally and calls API
- API processes image through GCP Document AI
- OCR extracts entities (items, totals, taxes)
- Raw data stored in Firebase
- Classification pipeline triggered via ADK
- Multi-agent system processes receipt:
- Classifies items by category
- Groups and calculates totals
- Validates against receipt total
- Refines if validation fails
- Final classified data saved to Firebase
- Update the List of Classification Options.
- Modify the ADK agents under
adk_pipeline/receipt_classifierto tweak classification behaviour or enhance them. - Adjust
firebase_store.pyif you prefer another database backend. - Update the
discord_bot/bot.pywith more integrations. - Adapt
discord_bot/bot.pyfor different chat platforms. - Update the Dashboad Options
- Add a Oauth2 Login using GMAIL to be done through API (including requested through bot).
- Create a Regression Model to Predict Individual User Spending.
- Fork the repository
- Create feature branch
- Add tests for new functionality
- Update documentation
- Submit pull request