FamilyFinance: Self-Hosted Finance Tracker with AI Categorization

The Problem

I got tired of the tradeoff. Every personal finance app I tried wanted my bank credentials. Mint, before it shut down, wanted a direct connection to every account. YNAB wants Plaid access. Copilot, the same. The pitch is always: give us your banking credentials, and we’ll give you pretty charts.

That never sat right with me. I work in infrastructure. I know what happens to data once it leaves your control. And frankly, after Mint’s shutdown in 2024 left millions of users scrambling to export their data, the risk of vendor lock-in stopped being theoretical.

But the alternative was miserable. I tried spreadsheets. I tried Ledger CLI. The problem is the same every time: manual data entry. Every bank exports statements differently. Some give you CSVs with inconsistent column headers. Some give you OFX or QFX files that generic parsers choke on. And none of them handle PDF statements, which is what most brokerages and tax documents actually come as.

I wanted something that could ingest whatever format my banks threw at me, categorize transactions intelligently, and give me a real financial picture — spending trends, net worth, tax prep, the works — all running on my own hardware.

The Approach

6-Service Architecture

FamilyFinance runs as a 6-service Docker Compose deployment. Each service has a clear responsibility:

1. FastAPI backend — API routes, business logic via a strict service layer, SQLAlchemy 2.0 models with async sessions
2. React frontend — TypeScript, Vite, Tailwind CSS, TanStack Query for server state, Zustand for client state
3. Celery worker — processes import tasks, runs AI categorization batches
4. Celery Beat scheduler — triggers file watch scans every 30 seconds
5. PostgreSQL 16 — the source of truth for all transaction and account data
6. Redis — task queue broker and result backend for Celery

The separation matters. The Celery worker can churn through a bulk import of thousands of transactions without the API server missing a beat. The Beat scheduler runs independently, scanning the watch directory on its 30-second interval regardless of what else is happening.

The Import Pipeline

The import pipeline is where I spent the most design time. It runs as a 3-stage Celery task chain:

Stage 1: Scan. Celery Beat checks the watch directory every 30 seconds. New files get queued. The automatic parser detection looks at file extensions and content signatures to route CSV, OFX, QFX, and PDF files to the right parser.

Stage 2: Parse. This is where the plugin system earns its keep. CSV files use custom parser schemas — saved column mappings specific to each bank’s export format. You map the columns once, save the schema, and every future import from that bank uses it automatically. OFX and QFX files go through standard financial data parsers. PDF files hit the AI-powered extraction engine.

Stage 3: Categorize. Uncategorized transactions get batched and sent to Anthropic Claude for categorization. Batching is deliberate — sending transactions one at a time would be wasteful. The system groups them, sends a batch, and applies the results.

The pipeline tracks status through discrete states (PENDING, PROCESSING, CATEGORIZING, COMPLETED) with real-time progress available via API endpoints that the frontend polls to update the UI.

PDF Intelligence

The PDF parsing was the hardest part and the feature I’m most satisfied with. Every bank formats their PDF statements differently. Chase puts transaction data in a completely different layout than Bank of America. Brokerage reports from Schwab look nothing like Fidelity’s.

I built 8+ bank-specific PDF parsers:

• Bank statements: Ally, Chase, Bank of America, PennyMac, American Express
• Brokerage reports: Schwab, Fidelity
• Tax forms: W-2, 1099 (various types), 1098

Each parser knows the exact layout of its target document type. It’s not generic OCR hoping for the best — it’s targeted extraction that knows where Chase puts the date column versus where BofA puts it. When the built-in parsers don’t match, the system falls back to AI-powered extraction that can handle unfamiliar formats.

The Plugin System

Early on I realized I’d be adding new bank formats and data sources indefinitely. So I built a plugin system with 4 distinct plugin types:

1. FileParser — handles parsing logic for specific file formats or bank-specific layouts
2. DataSource — integrates external data (market prices, exchange rates) for net worth calculations
3. AIProvider — swappable AI backends for categorization (currently Anthropic Claude)
4. Notification — alerts when imports complete, errors occur, or spending thresholds are hit

Plugins are discovered at startup via a registry pattern. Drop a new parser module in the right directory, and it’s available on the next restart. No core code changes required.

The Dashboard

The frontend went through a major visual redesign. The current version uses a modern fintech aesthetic: teal and emerald palette, gradient cards, collapsible sidebar navigation, and a suite of interactive Recharts visualizations.

Financial overview: KPI cards showing key metrics at a glance, balance sheet view, spending breakdown by category, and subscription tracking.

Spending analysis: Interactive trend charts (area and line), category donut breakdowns, and summary tables with date range filtering.

Net worth tracking: Brokerage holdings pulled from imported statements, net worth over time charting, and asset allocation breakdowns.

Tax insights: Document checklist tracking which tax forms have been imported, income breakdown across sources, and deduction tracking for estimated returns.

CLI and Automation

Beyond the web interface, there’s a full CLI for power users. Bulk import commands can process an entire directory of statements. User management (creation, role assignment, password resets) works without touching the database directly. And the file watch mode means day-to-day usage is genuinely hands-off: drop a statement PDF in the watch folder, and within 30 seconds it’s parsed, categorized, and reflected in your dashboard.

What I Learned

Bank data formats are a mess, and that’s the actual product. The technical architecture — FastAPI, Celery, React — is standard full-stack work. The hard part is handling the chaos of real-world financial data. Every bank exports CSVs with different column names. Date formats vary. Amounts are sometimes negative for debits, sometimes positive with a separate column for transaction type. The custom parser schema system (save and reuse column mappings per bank) came directly from hitting this wall repeatedly.

PDF parsing is a domain problem, not an NLP problem. My first instinct was to throw AI at every PDF and let it figure out the structure. That works in a pinch, but it’s slow, expensive, and unreliable for documents you’ll process regularly. The 8+ bank-specific parsers exist because knowing the exact layout of a Chase statement and extracting data deterministically is faster, cheaper, and more accurate than asking an LLM to interpret it every time. AI extraction is the fallback for unknown formats, not the primary path.

Plugin systems need to justify themselves early. I’ve seen projects build elaborate plugin architectures for two integrations. I almost did the same thing. What saved it was that I genuinely needed 4 different extension points (parsers, data sources, AI providers, notifications) within the first month of development. If you’re going to add the complexity of a plugin system, you need to be adding plugins regularly or the abstraction is just overhead.

Docker Compose is the right deployment target for personal tools. I briefly considered packaging this as a desktop Electron app. Docker Compose won because the target user (someone technical enough to care about self-hosting their finances) already has Docker. Six services sounds like a lot, but docker compose up makes it a single command. And the separation means I can upgrade PostgreSQL or swap Redis for a different broker without touching application code.

Celery Beat for file watching is surprisingly elegant. I originally had a Python watchdog process monitoring the directory. Replacing it with a Celery Beat task that scans every 30 seconds was simpler, more reliable, and eliminated a whole class of file-locking issues. The scan is cheap — just a directory listing — and it integrates naturally with the existing task pipeline.