Engineering Case Study
TalentMatch
Production-oriented job matching backend
A modular monolith split by execution model, not by business entity: a synchronous Fastify API owns request/response use cases, an asynchronous BullMQ worker owns retryable side effects, and both consume the same domain and infrastructure contracts from shared workspace packages. MongoDB is the durable source of truth; OpenSearch is a rebuildable read model that can always be recreated from it.
Overview
What TalentMatch is
Publishing a job and applying to one look synchronous from the outside, but the work behind them (search indexing, candidate scoring) is retryable and shouldn't block a response. TalentMatch splits on exactly that line: the API handles the request/response use cases, a BullMQ worker handles the asynchronous effects, and MongoDB stays the one place domain truth actually lives.
Deployed and live, not a local-only demo: a self-hosted instance runs the real API against real MongoDB/Redis/OpenSearch, alongside a full AWS ECS/Fargate deployment configuration (immutable per-commit images, OIDC-authenticated GitHub Actions deploys, Secrets Manager injection) for the production target.
Architecture
How a publish becomes a searchable job
- 1Employer publishes a job → API atomically marks it published and stores a pending indexSync marker in the same MongoDB update
- 2API tries an immediate BullMQ enqueue for low latency, using the event ID as the job ID, so a crash between enqueue and acknowledgement can't create a duplicate
- 3If enqueueing fails, the worker's relay discovers the pending marker on its own and retries. Durability lives in the data, not in Redis staying up
- 4Index worker reloads the canonical job from MongoDB and upserts or deletes the OpenSearch document. Old commands arriving late still converge correctly
- 5Failed indexing attempts retry with exponential backoff; after 5 attempts, a diagnostic copy lands on a dead-letter queue for safe, idempotent replay
- 6Candidate applies with an Idempotency-Key → a unique index plus a SHA-256 fingerprint tell a safe replay apart from an accidental key reuse with different input
Why it's built this way
MongoDB and Redis can't be written atomically, so the embedded marker pattern makes durability a property of one document update instead of two systems staying in sync. Delete is a tombstone rather than a hard delete, specifically so the only durable record of an undelivered OpenSearch delete command can never be erased before it's acted on.
Verified, Not Assumed
Engineering Highlights
- Two deployables from one codebase: publish/delete write a durable indexSync marker in the same MongoDB update as the state transition, so an index command surviving a crash is a property of the data model, not of Redis staying up.
- Idempotent applications enforced by a unique (candidateId, idempotencyKey) index plus a SHA-256 request fingerprint. A replayed request returns the original result; a reused key with different input returns a real 409, not a silent overwrite.
- Deterministic candidate scoring (skills, location, salary, experience) against an immutable job snapshot captured at application time, so a later job edit or deletion can never change or break an already-computed score.
- Search cache coherence via a generation counter, not a TTL alone: any mutation increments search:version, old query keys become unreachable without an unsafe Redis KEYS scan, and the index worker invalidates again after OpenSearch actually confirms the write.
- A dead-letter queue for indexing failures after 5 retries with exponential backoff. Replay is safe because OpenSearch upserts/deletes are idempotent and the worker always rereads MongoDB rather than trusting the queued payload.
- Guarded atomic transitions: every MongoDB update includes the expected current state in its filter, so two concurrent publish requests can't both win, and a missing resource (404) is distinguished from an invalid transition (409).
- OIDC bearer-JWT auth in production (signature/issuer/audience/lifetime verified via JWKS); Redis-backed rate limiting that fails open on a Redis error, because availability matters more once requests are already being authenticated cryptographically.
- Liveness and readiness are deliberately different questions: liveness excludes external dependencies so an outage never causes a restart loop; readiness checks Mongo/Redis/OpenSearch concurrently and pulls the task from traffic without killing the process.
Named, Not Hidden
A documented trade-off
The reindex command is simple on purpose, and that has a real cost
OpenSearch is explicitly a read model: job detail reads always go to MongoDB, so a stale or missing search document can never redefine domain truth. But the current reindex command clears the live index and re-enqueues every published job ID, which creates a real window where search results are reduced while it catches up. This is documented in docs/architecture.md as an accepted MVP-scale trade-off, not discovered after the fact. Production-scale reindexing would build a versioned physical index, verify document counts, and atomically swap a read alias instead, a real, named next step, not a silent gap.
5 Documented Decisions
Key Decisions
Decision 01
Two deployables, one codebase
Separating the API and worker protects request latency and allows independent scaling, while a pnpm monorepo keeps changes atomic and avoids a service-per-domain operational burden neither the team size nor the traffic justifies.
Decision 02
Infrastructure clients connect before listen
The API fails startup rather than briefly advertising readiness with a broken dependency graph. Appropriate because every planned use case genuinely requires Mongo/Redis/OpenSearch, not an optional dependency being treated as mandatory.
Decision 03
OpenSearch as a read model, never a source of truth
Search documents may lag MongoDB and can always be recreated from it; job detail endpoints read MongoDB directly so a missing or stale index entry degrades search, never correctness.
Decision 04
Zod at every trust boundary
TypeScript types alone provide no runtime safety. Environment variables and request payloads are parsed with Zod before use, and only the parsed output is allowed into application services.
Decision 05
Guarded atomic transitions over optimistic assumptions
State-changing MongoDB updates include the expected current state in their own filter, so the database itself, not application logic, is the concurrency boundary between two racing requests.
Numbers That Are Real
Measured Metrics
Test suite (run in this environment)
Measured46 passed, 8 skipped (the skips are real Mongo/Redis/OpenSearch integration tests needing live infra)
TypeScript typecheck
MeasuredClean across all 6 workspace packages/apps
Indexing retry policy
Measured5 attempts, exponential backoff, then a diagnostic dead-letter queue entry
Search cache TTL
Measured120 seconds, invalidated immediately on mutation via a generation counter
Rate limit default
Measured100 requests / 60s per actor or IP, fails open on a Redis error
Documented architecture decisions
Measured5, in docs/architecture.md
Technology