Work log: March 23, 2026

What shipped today

Simplified idea-to-content pipeline (v2.2)

The article processing chain was completely eliminated. A new daily.scan_and_generate_ideas handler replaces the old 5-step pipeline (google search → fetch articles → score → store → generate ideas) with a single handler that passes search result snippets directly to the idea generator as ephemeral context. No articles are stored — search results are used once and discarded. The daily pipeline collapsed to: page scrape → scan and generate ideas → briefing. ~4,200 lines of dead article handler code removed (7 handler files, 7 test files). The search_google() utility was extracted to engine/services/serper.py before deletion.

Briefings were gutted — removed Claude-generated article sections, pyramid layout, lifecycle prompts, and onboarding drips. Briefings are now pure data assembly: new ideas + content status. No AI call needed. -1,391 lines.

Stage_data audit and flat column migration complete

All remaining stage_data reads across engine and web were replaced with flat column reads. The gate issue (#1607) added the 8 flat columns to fetch_content(), then #1608 (engine, 11 files) and #1609 (web, 2 files) switched all reads. The _smartify_quotes post-processor was added to the generation service to guarantee curly quotes and em dashes in all output, regardless of what the AI produces.

Content generation quality improvements

The AI model was upgraded from Claude Sonnet 4.5 to Claude Sonnet 4.6. Guidepost prompts were rewritten with mandatory formatting requirements (3+ headings, 2+ bold phrases per section, 2+ standalone emphasis paragraphs) and a self-check checklist. Intro prompts were fixed to produce one paragraph with no title (was including titles and running to full target length). Draft prompts explicitly prohibit H1 titles. Transcript parsing was switched to Haiku for cost efficiency.

Distribution fixes

Blog queue slot-filling was implemented — blog posts now auto-schedule via slots, mirroring the social queue pattern. The rescheduleBlogQueue() function, API endpoint, and scheduler polling were all added with 4 new tests. Publishing logs now track all outbound activity — social publish and blog publish handlers write to publishing_logs on both success and failure.

Cleanup and bug fixes

PRODUCT.md fully rewritten for the 3-act product. Deprecated engine handlers (content_stage_start, content_field_generate) deleted (-1,845 lines). Orphaned route directories (articles, feeds, scans) deleted (-2,760 lines). Stale stage_status writes replaced with status column. Build cache type errors fixed via tsconfig. Stale dashboard queries and admin settings for removed features cleaned up. Idea “Write” button fixed to pre-populate content creation form.

Multiple engine deploy failures from a Python 3.11 f-string backslash syntax error in briefing_template.py — Railway reverted to old builds until the fix landed. Field redo was setting status to generating and never resetting — fixed by removing the status change (redo stays in review). Duplicate dual-write logic in content service caused database locks — removed the duplicate.

Completed

v2.2 — Simplified pipeline

• #1596 — New handler: daily.scan_and_generate_ideas
• #1597 — Simplify briefing to ideas + content status
• #1598 — Remove article scoring, fetching, and scan handlers

Stage_data audit

• #1606 — Audit and replace all remaining stage_data reads (decomposed)
• #1607 — Gate: add flat columns to fetch_content SELECT
• #1608 — Engine: replace stage_data reads with flat column reads
• #1609 — Web: replace stage_data reads with flat column reads

Cleanup and fixes

• #1587 — Update PRODUCT.md for simplified product
• #1588 — Replace stage_status writes with status
• #1589 — Delete orphaned articles, feeds, scans routes
• #1590 — Remove deprecated engine handlers
• #1591 — Clear .next build cache type errors
• #1601, #1603-1605 — Deploy failures resolved
• #1615 — Track all outbound publishing in publishing_logs
• #1617 — Blog queue slot-filling
• #1619 — Fix idea “Write” button
• #1620 — Remove stale dashboard/admin queries

Release progress

• v2.0 — Product simplification: 0/25 open (complete)
• v2.1 — Content model refactor: 0/15 open (complete)
• v2.2 — Simplified pipeline: 0/3 open (complete)
• v1.5: 1/50 open (#743 dashboard redesign, backlog)

Carry-over

• Guidepost prompt quality — rewritten with mandatory checklist but needs more real-world testing
• Smart quotes post-processing — deployed but not yet verified on fresh content
• Python 3.11 vs local Python version mismatch — caused f-string syntax error that passed local checks but failed Railway. Consider upgrading Dockerfile (#927 backlog)
• #1543 (Refactor API, MCP, and Apple app) — needs clarification, waiting on reply

Risks

• Railway engine deployed with 4 consecutive failures before the f-string fix landed. The revert-to-last-working-build behavior means code changes aren’t deployed until all syntax errors are fixed — local Python 3.14 doesn’t catch 3.11-only syntax restrictions
• The daily pipeline now depends on Serper API for idea generation — if the API key expires or rate limits hit, no new ideas are generated

Flags and watch-outs

• 36 commits in one day across engine and web — continued high volume
• The transcript parser now uses Haiku instead of Sonnet — monitor quality of answer extraction
• Blog slot-filling is new and untested in production — watch for scheduling issues when blog slots are first configured
• The content creation form now accepts idea_id param — verify this works end-to-end with the “Write” button on the ideas page