Playto Pay

Overview

Playto Pay is a hiring-challenge-style backend that models a real problem fintechs face: a merchant collects payments from customers in USD, and the platform owes them a payout in INR. This project scopes down to just that payout leg — given a merchant's accumulated balance, let them request a payout, safely subtract it from their available funds, hand it off to a background worker to "settle," and handle every way that process can go wrong: concurrent requests, duplicate retries, crashed workers, and failed bank transfers.

The interesting part isn't the CRUD — it's that money bugs don't announce themselves. A race condition in an e-commerce cart shows a wrong item; a race condition in a payout engine silently pays a merchant twice. The project is built end-to-end around eliminating that category of bug through database guarantees rather than application-level carefulness.

Problem

A merchant payout system has to answer, correctly, under concurrent load:

• How much money can this merchant actually withdraw right now? — not "as of the last read," but as of this exact instant, accounting for money already reserved by other in-flight payout requests.
• What happens if the same request arrives twice? — a flaky network, an anxious frontend retry, or a client-side double-click should never result in two payouts.
• What happens when a background worker dies mid-transfer? — the payout can't be left in limbo, and it can't silently double-process when a second worker picks it back up.
• What happens when the bank transfer itself fails? — the merchant's held funds have to come back, without ever deleting the record of what was attempted.

Solving all four "correctly" rules out the obvious shortcut of a balance integer column that gets decremented in application code — that pattern is exactly where race conditions and lost updates live.

Constraints

• Correctness over features. As a hiring-challenge scope, the project favors a small, airtight payout flow over a broad product surface.
• Auditability is non-negotiable. Every paisa moved has to be traceable to a specific, immutable record — money history can't be reconstructed from a mutable balance field.
• Concurrency has to be handled at the database, not the application. Since multiple API requests and multiple Celery workers can touch the same merchant simultaneously, correctness can't depend on Python-level checks.
• Idempotency has to survive partial failures. A request that reserves an idempotency key and then crashes before responding can't be allowed to look like "safe to retry from scratch."
• SQLite/Postgres duality. The system needs to run on SQLite for fast local iteration, while the concurrency-critical paths only make sense — and are only tested — against PostgreSQL's row-level locking.

Key Engineering Decisions

An append-only ledger instead of a mutable balance column

Every credit and debit is its own immutable LedgerEntry row — CREDIT_CUSTOMER_PAYMENT, DEBIT_PAYOUT_HOLD, CREDIT_PAYOUT_REFUND. A merchant's available balance isn't stored anywhere; it's computed on demand as sum(credits) − sum(payout holds), with a separate held_balance tracking funds already reserved by pending or processing payouts.

Reason: Deriving balance from history rather than storing it directly means the balance can never silently drift from the transactions that produced it — the ledger is the source of truth, and it can be replayed or audited at any point.

Tradeoff: Every balance check now requires an aggregation query over ledger rows instead of a single column read, trading a little query cost for a system that can't lie about how it got to a number.

Money as integer paise, never floats

All amounts are stored as BigIntegerField counts of paise (India's smallest currency unit) rather than FloatField or even DecimalField.

Reason: Floating-point arithmetic introduces rounding drift that is unacceptable when the units are money; representing the smallest currency unit as a plain integer sidesteps the entire class of precision bugs without needing Decimal's extra complexity for a domain that never needs fractional paise.

Tradeoff: Every amount in the system and its API has to be paise-denominated end to end — any code that assumes rupees has to explicitly convert, which is a discipline the codebase has to maintain consistently rather than something the type system enforces for free.

Row-level locking instead of optimistic concurrency

A payout request wraps the merchant row, the idempotency key row, the balance check, and the payout/ledger creation in a single database transaction, taking a select_for_update() lock on the merchant row for the duration.

Reason: Two concurrent requests reading "available balance: ₹100" and both approving a ₹60 payout is a real, common race — Python-level balance checks operate on stale reads by the time they act on them. A row lock forces the second transaction to wait until the first commits, so it recalculates balance against the post-hold state rather than a snapshot that's already out of date. Optimistic concurrency control was explicitly ruled out because money movement needs strict serialization, not retry-after-conflict.

Tradeoff: The merchant row becomes a serialization point — all payout requests for one merchant queue behind each other, which caps how many concurrent payouts a single merchant can have in flight, in exchange for a guarantee that no combination of concurrent requests can overdraft the account.

Idempotency keys scoped per merchant, with in-flight protection

Each request carries an Idempotency-Key header; the key is stored per-merchant with a database UniqueConstraint, alongside a SHA-256 hash of the canonical request body and an in_progress flag.

Reason: A retried request with the same key and same body gets back the exact stored response with no new payout created; the same key with a different body gets a 409 Conflict instead of silently doing the wrong thing. The in_progress marker specifically covers the case where a worker reserves the key and then crashes before writing a response — without it, a retried request could look like a fresh, unclaimed key and re-run the whole payout.

Tradeoff: Keys have to expire (24 hours here) so that legitimately-reusable keys don't pile up forever, which means the system's guarantee is bounded in time rather than permanent — a key reused a week later is treated as brand new.

A four-state payout state machine with no illegal transitions

Payouts move through PENDING → PROCESSING → COMPLETED or PROCESSING → FAILED, enforced by an explicit transition table where COMPLETED and FAILED map to empty next-state sets — so failed → completed or completed → pending simply cannot happen in code.

Reason: Payment state machines are exactly the kind of thing that "should never happen" bugs love to violate under concurrent access; making illegal transitions structurally impossible (rather than just discouraged by convention) removes an entire category of production incidents where a payout ends up in a state nobody intended.

Tradeoff: Every new payout outcome has to be modeled as an explicit state and explicit transition up front, which is more design overhead than just setting a status field directly — but it's overhead paid once, at design time, instead of debugged forever in production.

Failure handled as a compensating transaction, never a rewrite

When a bank transfer fails, the system doesn't delete or edit the original DEBIT_PAYOUT_HOLD entry — it transitions the payout to FAILED and writes a new CREDIT_PAYOUT_REFUND entry in the same transaction, restoring available balance while leaving the full history intact.

Reason: This keeps the ledger's core invariant — that it is only ever appended to — true even in the failure path, which is what makes the system genuinely auditable rather than auditable "as long as nothing went wrong."

Tradeoff: The ledger accumulates more rows over time for every retried or failed attempt, rather than staying minimal — a deliberate trade of storage and query volume for a complete, honest history.

Two-worker safety net: skip_locked claiming plus stale-payout retry

A Celery task claims pending payouts using select_for_update(skip_locked=True), so multiple worker processes can pull from the same queue without two workers grabbing the same row. A second scheduled task, retry_stale_processing_payouts, finds payouts that have sat in PROCESSING for more than 30 seconds and either retries them with exponential backoff or fails them out (with a refund) after three attempts.

Reason: skip_locked turns "don't double-process a payout" from an application-level convention into a database-enforced guarantee under concurrent workers. The stale-payout sweep exists because a worker can die after claiming a payout but before finishing it — without a watchdog, that payout would sit in PROCESSING forever, silently holding the merchant's funds hostage.

Tradeoff: The 30-second staleness threshold and 3-attempt cap are judgment calls, not derived constants — too aggressive and a slow-but-healthy transfer gets retried unnecessarily; too lenient and a genuinely stuck payout sits unresolved longer than it should.

The AI Audit

The project's EXPLAINER.md deliberately walks through an "obviously reasonable" first-draft implementation of payout creation — one that checks balance, then creates the payout and ledger hold as separate, unlocked steps — and shows exactly how it breaks: two concurrent requests can both read the same available balance, both pass the insufficient-funds check, and both insert holds, overdrafting the account, all without idempotency ever entering the picture. The corrected version folds the merchant lock, the idempotency check, and the balance check into one atomic block. Documenting the wrong version alongside the right one turns the design decision into something a reviewer can independently verify, instead of asking them to trust that "it's handled."

Results

• A REST API (/api/v1/merchants, /balance, /ledger, /payouts) backed by an append-only ledger where balance is always derived, never stored.
• Idempotent payout creation verified under a dedicated concurrency test that only runs against PostgreSQL, since SQLite has no row-level SELECT FOR UPDATE.
• A background processing pipeline — process_pending_payouts, process_payout, retry_stale_processing_payouts — that claims work safely across multiple Celery workers and resolves stuck payouts automatically.
• A seeded demo environment (three merchants, bank accounts, and ledger history) runnable end-to-end with a single docker compose up --build.
• A companion EXPLAINER.md that documents the exact locking, idempotency, and state-machine code alongside the reasoning for each choice — turning the design rationale into a reviewable artifact rather than an oral history.

Takeaways

Payout systems fail quietly, not loudly — the bug that matters most is the one where two requests both succeed instead of one clearly failing. Building this project pushed the correctness burden down to the database (row locks, unique constraints, atomic transactions) rather than up into application logic, which is the difference between "we're careful" and "this cannot happen."

Key lessons:

• Deriving balance from an immutable ledger, instead of storing and mutating it directly, is what makes a payments system auditable rather than merely functional.
• Concurrency correctness for money has to be enforced at the database layer — optimistic checks and in-process carefulness aren't strong enough guarantees for real financial state.
• Idempotency is only as strong as its handling of the in-flight case — a key that's reserved but not yet resolved is a different state than "not seen" or "resolved," and conflating them reintroduces the exact bug idempotency exists to prevent.
• Making illegal state transitions structurally impossible catches an entire class of bugs before they can reach production, at the one-time cost of designing the state machine explicitly.