Batch Message Analysis

The batch message analysis system processes unanalyzed messages across all organizations on an hourly schedule. For each contact with new messages, it updates the contact’s AI memory and extracts actionable tasks — extending the contact memory pipeline beyond voice calls to text-based messaging.

How It Works

Hourly cron job fires
  → For each organization:
    → Load watermark (last_processed_at) from contact_memory_analysis_state
    → Query messages newer than watermark (up to 10,000)
    → Group messages by contact_id
    → Sort contacts by last_contacted_at DESC (most active first)
    → Process up to 20 contacts concurrently:
        Phase A: Update contact memory with message transcript
        Phase B: Extract tasks from conversation context
    → Advance watermark to newest processed message timestamp

On first run for an organization, the watermark seeds to 30 days ago — so initial processing covers the last month of messages.

Data Model

contact_memory_analysis_state

Tracks per-organization processing progress.

Column	Type	Description
id	UUID	Primary key
organization_id	UUID	FK → organization (unique constraint)
last_processed_at	timestamptz	Watermark — newest message timestamp processed
created_at	timestamptz	Row creation time
updated_at	timestamptz	Last update time

The watermark only advances when at least one contact succeeds. On total failure, it stays put so the next run retries the same batch.

Message Selection

The job queries messages matching all of these conditions:

• created_at > last_processed_at (newer than watermark)
• contact_id IS NOT NULL (linked to a contact)
• body IS NOT NULL AND body != '' (has content)
• Ordered by created_at ASC
• Capped at 10,000 messages per organization per run

Individual message bodies are truncated to 1,000 characters. The total transcript per contact is capped at 50,000 characters.

Phase A: Memory Update

Messages are formatted as a timestamped transcript:

[2026-04-07 14:30 inbound] Hey, I wanted to follow up on our contract discussion.
[2026-04-07 14:35 outbound] Sure! Let me pull up your file.

This transcript feeds into the existing update_contact_memory service with MemoryUpdateSource::Messages. The service synthesizes the transcript into the contact’s system memory note — a structured summary with 9 sections:

• Relationship, Communication style, Preferences, Situation, Pain points, Sensitivities, Pending items, Key dates, Notes

Organization-scoped AI rules for the MemoryUpdate area are injected into the prompt, letting admins customize how memory is synthesized.

Phase B: Task Extraction

After memory updates, the system extracts up to 5 tasks per contact. The LLM receives:

• The contact’s updated memory
• All existing open tasks (for deduplication)
• The message transcript
• Organization-scoped AI rules for the TaskCreator area

ExtractedTask Schema

struct ExtractedTask {
    title: String,
    category: String,       // follow-up, callback, send-info, meeting, general
    description: Option<String>,
    priority: Option<String>,  // urgent, high, medium, low
    due_date_offset_days: Option<i64>,
}

Tasks are deduplicated via case-insensitive substring matching against existing open tasks. If the contact has exactly one owner, new tasks are auto-assigned to that owner.

Manual Trigger

You can trigger analysis outside the hourly schedule:

POST /api/messages/trigger-analysis
Permission: MessageCollection:Create

This runs the same pipeline immediately for all organizations.

AI Rules Integration

This feature introduces the MemoryUpdate and TaskCreator AI rule areas. Admins configure these rules in Settings → AI Rules by selecting the target area. Rules are injected as priority directives in the respective prompts.

See Multi-Area AI Rules for the full list of areas and how prompt injection works.

Key Files

File	Purpose
mods/messaging/jobs/analyze_messages_job.rs	Hourly cron job — iterates organizations
mods/messaging/services/process_unanalyzed_messages_service.rs	Per-org orchestration, watermark management
mods/contact/services/ai/analyze_contact_messages_service.rs	Per-contact analysis (memory + task extraction)
mods/contact/services/notes/update_contact_memory_service.rs	Unified memory update with MemoryUpdateSource enum
mods/settings/services/get_active_rules_for_area_service.rs	Fetches rules for a specific AI area
migration/src/m20260407_120000_create_contact_memory_analysis_state.rs	Creates watermark table