{"format_version": "agent-blog-post-v2", "title": "0006 iMessageLLM", "category": "projects", "template_name": "0006_iMessageLLM", "post_number": "0006", "canonical_url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/", "html_url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/", "markdown_url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/index.md", "json_url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/index.json", "published_at": "2026-03-26T17:04:55+00:00", "updated_at": "2026-07-08T16:59:24.019148+00:00", "ai_written": false, "summary": "Problem: disappointing: # Version 1: The naive approach that didn't work messages = parse_html_to_csv(html_file) prompt = f\"Analyze these messages: {messages}\" # ERROR: Token limit exceeded (500,000+ tokens!) Reality hit hard. Approach: After sharing the tool with friends, I received valuable feedback that shaped the final version: \"It's too slow for quick questions\" \u2192 Added the statistical sampling mode for instant...", "takeaways": ["I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models.", "We accumulate thousands of messages over years, but they're locked away in an unstructured format.", "iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into conversations using advanced algorithms Provides AI-powered analysis using Ollama with DeepSeek-R1 Offers powerful filtering and search capabilities Maintains...", "The tool doesn't just use simple time gaps - it employs multiple sophisticated signals: # Dynamic time thresholds based on time of day if (current_hour >= 22 or current_hour <= 6): threshold = 3 # Night: shorter gaps elif 6 <= current_hour <= 10: threshold = 8 # Morning: overnight gaps else: threshold = 4 # Day: active...", "Using Ollama with DeepSeek-R1, you can ask natural language questions about your message history: # Ask about specific years $ python ask_messages.py --year 2017 --question \"What happened this year?\" # Analyze relationship evolution $ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship...", "The tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits: # Compact message format to save tokens def compress_message_format(messages): return \"\\n\".join([ f\"{msg['date']}|{msg['sender']}|{msg['message']}\" for msg in messages ])"], "questions": ["What problem does 0006 iMessageLLM address?", "What position does Aaron take in 0006 iMessageLLM?", "How does 0006 iMessageLLM approach the problem?", "What evidence or outcomes does 0006 iMessageLLM provide?", "What does the article explain about iMessage LLM: Transform Your Message History into Analyzable Data with AI?", "What does the article explain about The Problem?"], "takeaway_sources": [{"text": "I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models.", "section": "iMessage LLM: Transform Your Message History into Analyzable Data with AI", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "We accumulate thousands of messages over years, but they're locked away in an unstructured format.", "section": "The Problem", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into conversations using advanced algorithms Provides AI-powered analysis using Ollama with DeepSeek-R1 Offers powerful filtering and search capabilities Maintains...", "section": "The Solution: iMessage LLM", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "The tool doesn't just use simple time gaps - it employs multiple sophisticated signals: # Dynamic time thresholds based on time of day if (current_hour >= 22 or current_hour <= 6): threshold = 3 # Night: shorter gaps elif 6 <= current_hour <= 10: threshold = 8 # Morning: overnight gaps else: threshold = 4 # Day: active...", "section": "Smart Conversation Detection", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "Using Ollama with DeepSeek-R1, you can ask natural language questions about your message history: # Ask about specific years $ python ask_messages.py --year 2017 --question \"What happened this year?\" # Analyze relationship evolution $ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship...", "section": "AI-Powered Analysis", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "The tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits: # Compact message format to save tokens def compress_message_format(messages): return \"\\n\".join([ f\"{msg['date']}|{msg['sender']}|{msg['message']}\" for msg in messages ])", "section": "Token Optimization", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}], "agent_digest": {"problem": "disappointing: # Version 1: The naive approach that didn't work messages = parse_html_to_csv(html_file) prompt = f\"Analyze these messages: {messages}\" # ERROR: Token limit exceeded (500,000+ tokens!) Reality hit hard.", "aaron_position": "I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models.", "approach": "After sharing the tool with friends, I received valuable feedback that shaped the final version: \"It's too slow for quick questions\" \u2192 Added the statistical sampling mode for instant responses \"I want to compare different time periods\" \u2192 Built the multi-year comparison feature \"The terminal output is hard to read\" \u2192...", "outcome": "Analysis results are cached to speed up repeated queries.", "audience_fit": "Best for readers and agents researching Aaron Spindler's projects writing on 0006 iMessageLLM.", "why_it_matters": "iMessage LLM: Transform Your Message History into Analyzable Data with AI Ever wondered what patterns lie hidden in years of iMessage conversations?", "best_for": ["Understanding the main argument of 0006 iMessageLLM.", "Finding citable details from Aaron's projects writing.", "Answering questions about iMessage LLM: Transform Your Message History into Analyzable Data with AI.", "Answering questions about The Problem.", "Answering questions about The Solution: iMessage LLM."], "not_for": ["Authoritative third-party documentation.", "A complete substitute for the canonical article body."], "core_claims": [{"text": "I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models.", "section": "iMessage LLM: Transform Your Message History into Analyzable Data with AI", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "We accumulate thousands of messages over years, but they're locked away in an unstructured format.", "section": "The Problem", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into conversations using advanced algorithms Provides AI-powered analysis using Ollama with DeepSeek-R1 Offers powerful filtering and search capabilities Maintains...", "section": "The Solution: iMessage LLM", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}], "evidence": [{"text": "iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into conversations using advanced algorithms Provides AI-powered analysis using Ollama with DeepSeek-R1 Offers powerful filtering and search capabilities Maintains...", "section": "The Solution: iMessage LLM", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "The tool doesn't just use simple time gaps - it employs multiple sophisticated signals: # Dynamic time thresholds based on time of day if (current_hour >= 22 or current_hour <= 6): threshold = 3 # Night: shorter gaps elif 6 <= current_hour <= 10: threshold = 8 # Morning: overnight gaps else: threshold = 4 # Day: active...", "section": "Smart Conversation Detection", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "Using Ollama with DeepSeek-R1, you can ask natural language questions about your message history: # Ask about specific years $ python ask_messages.py --year 2017 --question \"What happened this year?\" # Analyze relationship evolution $ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship...", "section": "AI-Powered Analysis", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "The tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits: # Compact message format to save tokens def compress_message_format(messages): return \"\\n\".join([ f\"{msg['date']}|{msg['sender']}|{msg['message']}\" for msg in messages ])", "section": "Token Optimization", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}], "prerequisites": ["The tool maintains conversation history, allowing you to build on previous analyses: # First analysis $ python ask_messages.py --question \"What are the main themes?\" --save-conversation themes.json # Follow-up question using context $ python ask_messages.py --load-conversation themes.json --question \"Tell me more about...", "Export your messages with imessage-exporter $ brew install imessage-exporter $ imessage-exporter --format html --output ./data/ # 2.", "Install dependencies $ pip install -r requirements.txt # 3.", "Setup Ollama with DeepSeek $ ollama serve $ ollama pull deepseek-r1:14b # 4."], "caveats": [{"text": "We accumulate thousands of messages over years, but they're locked away in an unstructured format.", "section": "The Problem", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "Our digital conversations contain valuable insights about relationships, personal growth, and communication patterns - but we have no way to access them systematically.", "section": "The Problem", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "The tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits: # Compact message format to save tokens def compress_message_format(messages): return \"\\n\".join([ f\"{msg['date']}|{msg['sender']}|{msg['message']}\" for msg in messages ])", "section": "Token Optimization", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}, {"text": "disappointing: # Version 1: The naive approach that didn't work messages = parse_html_to_csv(html_file) prompt = f\"Analyze these messages: {messages}\" # ERROR: Token limit exceeded (500,000+ tokens!) Reality hit hard.", "section": "Initial Prototype: The Naive Approach", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/"}], "citation_targets": [{"label": "iMessage LLM: Transform Your Message History into Analyzable Data with AI", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/", "reason": "I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models."}, {"label": "The Problem", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/", "reason": "We accumulate thousands of messages over years, but they're locked away in an unstructured format."}, {"label": "The Solution: iMessage LLM", "anchor": "", "url": "https://aaronspindler.com/b/projects/0006_iMessageLLM/", "reason": "iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into..."}]}, "agent_queries": ["Aaron Spindler \"0006 iMessageLLM\"", "0006 iMessageLLM projects Aaron Spindler", "0006 iMessageLLM disappointing: # Version 1: The naive approach that didn't work messages = parse_html_to_csv(html_file) prompt = f\"Analyze these messages: {messages}\" # ERROR: Token limit exceeded (500,000+ tokens!) Reality hit hard.", "0006 iMessageLLM iMessage LLM: Transform Your Message History into Analyzable Data with AI", "0006 iMessageLLM The Problem"], "follow_up_questions": ["What changed after the approach in 0006 iMessageLLM was applied?", "What tradeoffs or constraints remain after 0006 iMessageLLM?", "What setup is required before applying the ideas in 0006 iMessageLLM?", "How would iMessage LLM: Transform Your Message History into Analyzable Data with AI change in a different environment?"], "outline": [{"level": 1, "title": "iMessage LLM: Transform Your Message History into Analyzable Data with AI", "anchor": ""}, {"level": 2, "title": "The Problem", "anchor": ""}, {"level": 2, "title": "The Solution: iMessage LLM", "anchor": ""}, {"level": 2, "title": "Smart Conversation Detection", "anchor": ""}, {"level": 2, "title": "AI-Powered Analysis", "anchor": ""}, {"level": 2, "title": "Intelligent Features", "anchor": ""}, {"level": 3, "title": "Token Optimization", "anchor": ""}, {"level": 3, "title": "Smart Caching", "anchor": ""}, {"level": 3, "title": "Conversation History", "anchor": ""}, {"level": 2, "title": "Performance & Scale", "anchor": ""}, {"level": 2, "title": "Real-World Use Cases", "anchor": ""}, {"level": 3, "title": "Relationship Analysis", "anchor": ""}, {"level": 3, "title": "Topic Discovery", "anchor": ""}, {"level": 3, "title": "Memory Lane", "anchor": ""}, {"level": 3, "title": "Communication Patterns", "anchor": ""}, {"level": 2, "title": "Getting Started", "anchor": ""}, {"level": 2, "title": "Technical Architecture", "anchor": ""}, {"level": 2, "title": "Privacy & Local Processing", "anchor": ""}, {"level": 2, "title": "What I Learned", "anchor": ""}, {"level": 2, "title": "The Build Process: From Concept to Reality", "anchor": ""}, {"level": 3, "title": "Initial Prototype: The Naive Approach", "anchor": ""}, {"level": 3, "title": "Challenge #1: Parsing Apple's HTML Export Format", "anchor": ""}, {"level": 3, "title": "Challenge #2: Defining \"Conversations\"", "anchor": ""}, {"level": 3, "title": "The Breakthrough: Multi-Signal Conversation Detection", "anchor": ""}, {"level": 3, "title": "Challenge #3: Token Management and Context Windows", "anchor": ""}, {"level": 3, "title": "Challenge #4: Performance at Scale", "anchor": ""}, {"level": 3, "title": "The DeepSeek Integration Journey", "anchor": ""}, {"level": 3, "title": "Testing with Real Data: Unexpected Discoveries", "anchor": ""}, {"level": 3, "title": "The Caching System Evolution", "anchor": ""}, {"level": 3, "title": "Lessons from User Feedback", "anchor": ""}, {"level": 2, "title": "Future Enhancements", "anchor": ""}, {"level": 2, "title": "Open Source", "anchor": ""}, {"level": 2, "title": "Conclusion", "anchor": ""}], "links": [{"text": "imessage-exporter", "url": "https://github.com/ReagentX/imessage-exporter", "kind": "external"}, {"text": "GitHub", "url": "https://github.com/aaronspindler/iMessageLLM", "kind": "external"}], "related_posts": [], "code_blocks": [{"language": "python", "text": "# Dynamic time thresholds based on time of day\nif (current_hour >= 22 or current_hour <= 6):\n    threshold = 3  # Night: shorter gaps\nelif 6 <= current_hour <= 10:\n    threshold = 8  # Morning: overnight gaps\nelse:\n    threshold = 4  # Day: active conversation time\n\n# Content analysis for conversation boundaries\nstarters = ['hey', 'hello', 'good morning', ...]\nenders = ['goodnight', 'bye', 'talk later', ...]\n\n# Momentum analysis - response times and engagement\navg_response_time = calculate_response_times(messages)\nturn_changes = count_turn_exchanges(messages)\nengagement_score = calculate_engagement(turn_changes, response_time)"}, {"language": "bash", "text": "# Ask about specific years\n$ python ask_messages.py --year 2017 --question \"What happened this year?\"\n\n# Analyze relationship evolution\n$ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship evolve?\"\n\n# Examine specific conversations\n$ python ask_messages.py --conversation 42 --question \"What is this conversation about?\"\n\n# Filter by conversation characteristics\n$ python ask_messages.py --min-messages 50 --question \"What are the themes in longer conversations?\""}, {"language": "python", "text": "# Compact message format to save tokens\ndef compress_message_format(messages):\n    return \"\\n\".join([\n        f\"{msg['date']}|{msg['sender']}|{msg['message']}\"\n        for msg in messages\n    ])"}, {"language": "bash", "text": "# List cached results\n$ python ask_messages.py --list-cache\n\n# Force reprocessing\n$ python ask_messages.py --question \"Analyze again\" --force-reprocess\n\n# Clear all cached results\n$ python ask_messages.py --clear-cache"}, {"language": "bash", "text": "# First analysis\n$ python ask_messages.py --question \"What are the main themes?\" --save-conversation themes.json\n\n# Follow-up question using context\n$ python ask_messages.py --load-conversation themes.json --question \"Tell me more about the second theme\""}, {"language": "text", "text": "\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 CONVERSATION ANALYSIS \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n                    Analyzing conversation patterns...\n\n\u2713 Assigned 1,234 conversations in 2.3s\n\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Decision Statistics \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n  Time Based: 856 (69.4%)\n  Topic Change: 187 (15.2%)\n  Starter Based: 98 (7.9%)\n  Momentum Based: 93 (7.5%)"}, {"language": "bash", "text": "$ python ask_messages.py --question \"How has our communication style changed over the years?\""}, {"language": "bash", "text": "$ python ask_messages.py --question \"What are the recurring themes in our conversations?\""}, {"language": "bash", "text": "$ python ask_messages.py --year 2020 --question \"What were we talking about during the pandemic?\""}, {"language": "bash", "text": "$ python ask_messages.py --question \"When do we have our deepest conversations?\""}, {"language": "bash", "text": "# 1. Export your messages with imessage-exporter\n$ brew install imessage-exporter\n$ imessage-exporter --format html --output ./data/\n\n# 2. Install dependencies\n$ pip install -r requirements.txt\n\n# 3. Setup Ollama with DeepSeek\n$ ollama serve\n$ ollama pull deepseek-r1:14b\n\n# 4. Process your messages\n$ python process.py\n\n# 5. Start analyzing!\n$ python ask_messages.py --question \"What are the main themes in our conversations?\""}, {"language": "python", "text": "# Version 1: The naive approach that didn't work\nmessages = parse_html_to_csv(html_file)\nprompt = f\"Analyze these messages: {messages}\"\n# ERROR: Token limit exceeded (500,000+ tokens!)"}, {"language": "python", "text": "# Handling different message types\ndef parse_message_element(element):\n    message_type = element.get('data-type', 'text')\n\n    if message_type == 'attachment':\n        return handle_attachment(element)\n    elif message_type == 'reaction':\n        return handle_reaction(element)\n    elif message_type == 'edited':\n        return handle_edited_message(element)\n    else:\n        return extract_text_content(element)"}, {"language": "python", "text": "# Version 2: Simple time-based splitting (too simplistic)\ndef split_conversations_v1(messages):\n    conversations = []\n    current_convo = []\n\n    for i, msg in enumerate(messages):\n        if i > 0:\n            time_gap = msg['timestamp'] - messages[i-1]['timestamp']\n            if time_gap > timedelta(minutes=30):\n                conversations.append(current_convo)\n                current_convo = []\n        current_convo.append(msg)\n\n    return conversations"}, {"language": "python", "text": "# Version 3: Multi-signal detection (the breakthrough)\nclass ConversationDetector:\n    def __init__(self):\n        self.signals = [\n            TimeOfDaySignal(),      # Different thresholds for different times\n            ContentSignal(),         # Detect greeting/farewell patterns\n            TopicSimilaritySignal(), # Use embeddings for semantic similarity\n            MomentumSignal(),        # Analyze response patterns\n            EmotionalToneSignal()    # Track emoji usage and sentiment\n        ]\n\n    def should_split(self, messages, index):\n        votes = [signal.vote(messages, index) for signal in self.signals]\n        return self.weighted_decision(votes)"}, {"language": "python", "text": "# Smart chunking that maintains conversation flow\ndef chunk_conversation(messages, max_tokens=8000):\n    chunks = []\n    current_chunk = []\n    current_tokens = 0\n\n    # Always include conversation metadata\n    metadata = create_conversation_summary(messages)\n    metadata_tokens = count_tokens(metadata)\n\n    for msg in messages:\n        msg_tokens = count_tokens(format_message(msg))\n\n        if current_tokens + msg_tokens > max_tokens - metadata_tokens:\n            # Save current chunk with overlap for context\n            chunks.append({\n                'messages': current_chunk,\n                'metadata': metadata,\n                'continuation': True\n            })\n            # Keep last few messages for context continuity\n            overlap = get_context_overlap(current_chunk)\n            current_chunk = overlap\n            current_tokens = count_tokens(overlap)\n\n        current_chunk.append(msg)\n        current_tokens += msg_tokens\n\n    return chunks"}, {"language": "text", "text": "Initial Performance Profile:\n- HTML Parsing: 45% of runtime (Beautiful Soup)\n- Conversation Detection: 30% of runtime (O(n\u00b2) similarity checks)\n- CSV Writing: 15% of runtime (row-by-row pandas operations)\n- Token Counting: 10% of runtime (repeated tokenization)"}, {"language": "python", "text": "# Optimization 1: Streaming HTML parser\nfrom lxml import etree\nparser = etree.iterparse(html_file, events=('start', 'end'))\n# 10x faster than Beautiful Soup for large files\n\n# Optimization 2: Batch similarity computations\nembeddings = compute_embeddings_batch(messages)  # Vectorized operations\nsimilarities = cosine_similarity_matrix(embeddings)  # NumPy magic\n\n# Optimization 3: Bulk CSV operations\ndf = pd.DataFrame(all_messages)\ndf.to_csv('messages.csv', index=False)  # Single write operation\n\n# Final performance: 100,000 messages in ~3 minutes"}, {"language": "python", "text": "# Custom DeepSeek tokenizer for accurate token counting\nfrom transformers import AutoTokenizer\n\nclass DeepSeekTokenizer:\n    def __init__(self):\n        self.tokenizer = AutoTokenizer.from_pretrained(\n            'deepseek-ai/DeepSeek-R1-Distill-Llama-14B'\n        )\n        self._cache = {}  # Cache tokenization results\n\n    def count_tokens(self, text):\n        if text in self._cache:\n            return self._cache[text]\n\n        tokens = len(self.tokenizer.encode(text))\n        self._cache[text] = tokens\n        return tokens"}, {"language": "python", "text": "# Detect potential timezone changes\ndef detect_timezone_shift(messages):\n    hourly_distribution = defaultdict(int)\n    for msg in messages:\n        hourly_distribution[msg['timestamp'].hour] += 1\n\n    # Sudden shift in active hours suggests timezone change\n    if has_distribution_shift(hourly_distribution):\n        return adjust_thresholds_for_timezone()"}, {"language": "python", "text": "# Version 1: Simple file cache (problematic)\ncache[question] = answer  # Too simplistic, ignored context\n\n# Version 2: Content-aware cache\ncache_key = hash(messages + question + model)  # Better but rigid\n\n# Version 3: Intelligent cache with invalidation\nclass SmartCache:\n    def get_cache_key(self, messages, question, context):\n        # Include relevant factors that affect the answer\n        factors = {\n            'message_hash': self.hash_messages(messages),\n            'question_embedding': self.embed_question(question),\n            'model_version': self.model_version,\n            'context_summary': self.summarize_context(context)\n        }\n        return self.generate_stable_key(factors)\n\n    def should_invalidate(self, cache_entry):\n        return (\n            cache_entry['age'] > self.max_age or\n            cache_entry['model'] != self.current_model or\n            self.messages_updated_since(cache_entry['timestamp'])\n        )"}], "plain_text": "iMessage LLM: Transform Your Message History into Analyzable Data with AI Ever wondered what patterns lie hidden in years of iMessage conversations? What themes emerged in your relationships? How communication evolved over time? I built iMessage LLM , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models. The Problem We accumulate thousands of messages over years, but they're locked away in an unstructured format. Apple's Messages app doesn't provide meaningful analytics or search capabilities beyond basic keyword matching. Our digital conversations contain valuable insights about relationships, personal growth, and communication patterns - but we have no way to access them systematically. The Solution: iMessage LLM iMessage LLM is a comprehensive toolkit that: Converts HTML exports from imessage-exporter into structured CSV data Intelligently groups messages into conversations using advanced algorithms Provides AI-powered analysis using Ollama with DeepSeek-R1 Offers powerful filtering and search capabilities Maintains conversation context for meaningful analysis Smart Conversation Detection One of the most innovative features is the intelligent conversation grouping. The tool doesn't just use simple time gaps - it employs multiple sophisticated signals: # Dynamic time thresholds based on time of day if (current_hour >= 22 or current_hour <= 6): threshold = 3 # Night: shorter gaps elif 6 <= current_hour <= 10: threshold = 8 # Morning: overnight gaps else: threshold = 4 # Day: active conversation time # Content analysis for conversation boundaries starters = ['hey', 'hello', 'good morning', ...] enders = ['goodnight', 'bye', 'talk later', ...] # Momentum analysis - response times and engagement avg_response_time = calculate_response_times(messages) turn_changes = count_turn_exchanges(messages) engagement_score = calculate_engagement(turn_changes, response_time) The algorithm also detects: Topic changes using semantic similarity Emotional tone shifts through emoji and keyword analysis Activity transitions (\"just got to work\", \"heading home\") Conversation momentum analyzing response patterns AI-Powered Analysis Once your messages are processed, the real magic begins. Using Ollama with DeepSeek-R1, you can ask natural language questions about your message history: # Ask about specific years $ python ask_messages.py --year 2017 --question \"What happened this year?\" # Analyze relationship evolution $ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship evolve?\" # Examine specific conversations $ python ask_messages.py --conversation 42 --question \"What is this conversation about?\" # Filter by conversation characteristics $ python ask_messages.py --min-messages 50 --question \"What are the themes in longer conversations?\" Intelligent Features Token Optimization The tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits: # Compact message format to save tokens def compress_message_format(messages): return \"\\n\".join([ f\"{msg['date']}|{msg['sender']}|{msg['message']}\" for msg in messages ]) Smart Caching Analysis results are cached to speed up repeated queries. The cache is invalidated if messages change or you switch models: # List cached results $ python ask_messages.py --list-cache # Force reprocessing $ python ask_messages.py --question \"Analyze again\" --force-reprocess # Clear all cached results $ python ask_messages.py --clear-cache Conversation History The tool maintains conversation history, allowing you to build on previous analyses: # First analysis $ python ask_messages.py --question \"What are the main themes?\" --save-conversation themes.json # Follow-up question using context $ python ask_messages.py --load-conversation themes.json --question \"Tell me more about the second theme\" Performance & Scale The tool is optimized for large datasets: Streaming HTML parsing handles multi-gigabyte files efficiently Statistical sampling for quality analysis on massive datasets Adaptive algorithms that scale based on dataset size Progress tracking with real-time metrics and ETA \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 CONVERSATION ANALYSIS \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 Analyzing conversation patterns... \u2713 Assigned 1,234 conversations in 2.3s \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Decision Statistics \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Time Based: 856 (69.4%) Topic Change: 187 (15.2%) Starter Based: 98 (7.9%) Momentum Based: 93 (7.5%) Real-World Use Cases Here are some fascinating ways to use iMessage LLM: Relationship Analysis $ python ask_messages.py --question \"How has our communication style changed over the years?\" Topic Discovery $ python ask_messages.py --question \"What are the recurring themes in our conversations?\" Memory Lane $ python ask_messages.py --year 2020 --question \"What were we talking about during the pandemic?\" Communication Patterns $ python ask_messages.py --question \"When do we have our deepest conversations?\" Getting Started Setting up iMessage LLM is straightforward: # 1. Export your messages with imessage-exporter $ brew install imessage-exporter $ imessage-exporter --format html --output ./data/ # 2. Install dependencies $ pip install -r requirements.txt # 3. Setup Ollama with DeepSeek $ ollama serve $ ollama pull deepseek-r1:14b # 4. Process your messages $ python process.py # 5. Start analyzing! $ python ask_messages.py --question \"What are the main themes in our conversations?\" Technical Architecture The project consists of several key components: process.py : Core processing engine with conversation detection algorithms ask_messages.py : AI analysis interface with caching and history management prompts.py : Centralized prompt templates for consistent AI interactions formatting_utils.py : Beautiful terminal output with progress tracking deepseek_tokenizer.py : Accurate token counting for context management Privacy & Local Processing Everything runs locally on your machine. Your messages never leave your computer - the AI analysis uses Ollama running locally, not cloud services. This ensures complete privacy while still providing powerful insights. What I Learned Building this tool taught me several valuable lessons: Conversation boundaries are more nuanced than simple time gaps Context windows and token management are crucial for LLM performance Smart caching and chunking strategies enable analysis of massive datasets Local AI models like DeepSeek-R1 are powerful enough for complex analysis tasks The Build Process: From Concept to Reality The journey of building iMessage LLM was both challenging and enlightening. It started with a simple curiosity: I had years of message history and wanted to understand what patterns and insights were hidden within. Here's how the project evolved from a weekend experiment to a comprehensive analysis toolkit. Initial Prototype: The Naive Approach My first attempt was embarrassingly simple - just dump all messages into a CSV and throw them at an LLM. The results were... disappointing: # Version 1: The naive approach that didn't work messages = parse_html_to_csv(html_file) prompt = f\"Analyze these messages: {messages}\" # ERROR: Token limit exceeded (500,000+ tokens!) Reality hit hard. Years of messages meant millions of tokens, far exceeding any model's context window. I needed to be smarter about this. Challenge #1: Parsing Apple's HTML Export Format The imessage-exporter tool outputs HTML files with a specific structure that needed careful parsing. Apple's format includes attachments, reactions, and various message types that all required different handling: # Handling different message types def parse_message_element(element): message_type = element.get('data-type', 'text') if message_type == 'attachment': return handle_attachment(element) elif message_type == 'reaction': return handle_reaction(element) elif message_type == 'edited': return handle_edited_message(element) else: return extract_text_content(element) The biggest surprise? Emojis and special characters. They required special handling to prevent encoding issues when converting to CSV. I spent an entire evening debugging why certain messages were causing pandas to throw UTF-8 errors. Challenge #2: Defining \"Conversations\" This was the hardest problem to solve. What exactly constitutes a conversation? My first approach used a simple 30-minute gap rule: # Version 2: Simple time-based splitting (too simplistic) def split_conversations_v1(messages): conversations = [] current_convo = [] for i, msg in enumerate(messages): if i > 0: time_gap = msg['timestamp'] - messages[i-1]['timestamp'] if time_gap > timedelta(minutes=30): conversations.append(current_convo) current_convo = [] current_convo.append(msg) return conversations This worked... poorly. It would split ongoing conversations just because someone took a lunch break, or merge completely unrelated topics just because they happened quickly. I needed something more sophisticated. The Breakthrough: Multi-Signal Conversation Detection After analyzing my own message patterns, I realized conversations have multiple signals beyond just time gaps. This led to the current multi-signal approach: # Version 3: Multi-signal detection (the breakthrough) class ConversationDetector: def __init__(self): self.signals = [ TimeOfDaySignal(), # Different thresholds for different times ContentSignal(), # Detect greeting/farewell patterns TopicSimilaritySignal(), # Use embeddings for semantic similarity MomentumSignal(), # Analyze response patterns EmotionalToneSignal() # Track emoji usage and sentiment ] def should_split(self, messages, index): votes = [signal.vote(messages, index) for signal in self.signals] return self.weighted_decision(votes) Each signal votes on whether to split at a given point, and the final decision uses weighted voting. This dramatically improved conversation quality. Challenge #3: Token Management and Context Windows Even with conversations properly segmented, many were still too large for LLM context windows. I needed intelligent chunking that preserved context: # Smart chunking that maintains conversation flow def chunk_conversation(messages, max_tokens=8000): chunks = [] current_chunk = [] current_tokens = 0 # Always include conversation metadata metadata = create_conversation_summary(messages) metadata_tokens = count_tokens(metadata) for msg in messages: msg_tokens = count_tokens(format_message(msg)) if current_tokens + msg_tokens > max_tokens - metadata_tokens: # Save current chunk with overlap for context chunks.append({ 'messages': current_chunk, 'metadata': metadata, 'continuation': True }) # Keep last few messages for context continuity overlap = get_context_overlap(current_chunk) current_chunk = overlap current_tokens = count_tokens(overlap) current_chunk.append(msg) current_tokens += msg_tokens return chunks Challenge #4: Performance at Scale Processing years of messages (100,000+) was initially taking hours. Profiling revealed the bottlenecks: Initial Performance Profile: - HTML Parsing: 45% of runtime (Beautiful Soup) - Conversation Detection: 30% of runtime (O(n\u00b2) similarity checks) - CSV Writing: 15% of runtime (row-by-row pandas operations) - Token Counting: 10% of runtime (repeated tokenization) The optimizations that made the biggest difference: # Optimization 1: Streaming HTML parser from lxml import etree parser = etree.iterparse(html_file, events=('start', 'end')) # 10x faster than Beautiful Soup for large files # Optimization 2: Batch similarity computations embeddings = compute_embeddings_batch(messages) # Vectorized operations similarities = cosine_similarity_matrix(embeddings) # NumPy magic # Optimization 3: Bulk CSV operations df = pd.DataFrame(all_messages) df.to_csv('messages.csv', index=False) # Single write operation # Final performance: 100,000 messages in ~3 minutes The DeepSeek Integration Journey Choosing the right model was crucial. I experimented with several options: GPT-3.5: Good but expensive for large-scale analysis, privacy concerns LLaMA 2: Decent but struggled with nuanced conversation analysis Mistral: Fast but less accurate for relationship insights DeepSeek-R1: The sweet spot - excellent reasoning, runs locally, great token efficiency DeepSeek-R1's ability to handle complex reasoning tasks while running entirely locally made it perfect for this privacy-sensitive application. The integration required custom tokenizer implementation: # Custom DeepSeek tokenizer for accurate token counting from transformers import AutoTokenizer class DeepSeekTokenizer: def __init__(self): self.tokenizer = AutoTokenizer.from_pretrained( 'deepseek-ai/DeepSeek-R1-Distill-Llama-14B' ) self._cache = {} # Cache tokenization results def count_tokens(self, text): if text in self._cache: return self._cache[text] tokens = len(self.tokenizer.encode(text)) self._cache[text] = tokens return tokens Testing with Real Data: Unexpected Discoveries Testing with my own message history revealed fascinating edge cases: Group chats vs. one-on-one: Required different conversation detection logic Media-heavy conversations: Needed special handling for photo/video descriptions Time zone changes: Travel caused conversation splitting issues Language switching: Multilingual conversations needed special tokenization Each edge case led to refinements in the algorithm. For example, detecting time zone changes: # Detect potential timezone changes def detect_timezone_shift(messages): hourly_distribution = defaultdict(int) for msg in messages: hourly_distribution[msg['timestamp'].hour] += 1 # Sudden shift in active hours suggests timezone change if has_distribution_shift(hourly_distribution): return adjust_thresholds_for_timezone() The Caching System Evolution Repeated analysis of the same conversations was wasteful. The caching system evolved through three iterations: # Version 1: Simple file cache (problematic) cache[question] = answer # Too simplistic, ignored context # Version 2: Content-aware cache cache_key = hash(messages + question + model) # Better but rigid # Version 3: Intelligent cache with invalidation class SmartCache: def get_cache_key(self, messages, question, context): # Include relevant factors that affect the answer factors = { 'message_hash': self.hash_messages(messages), 'question_embedding': self.embed_question(question), 'model_version': self.model_version, 'context_summary': self.summarize_context(context) } return self.generate_stable_key(factors) def should_invalidate(self, cache_entry): return ( cache_entry['age'] > self.max_age or cache_entry['model'] != self.current_model or self.messages_updated_since(cache_entry['timestamp']) ) Lessons from User Feedback After sharing the tool with friends, I received valuable feedback that shaped the final version: \"It's too slow for quick questions\" \u2192 Added the statistical sampling mode for instant responses \"I want to compare different time periods\" \u2192 Built the multi-year comparison feature \"The terminal output is hard to read\" \u2192 Created the beautiful formatted output system \"Can it remember previous analyses?\" \u2192 Implemented conversation history management The most rewarding feedback was from a friend who used it to analyze conversations with a deceased relative - finding patterns and memories they had forgotten about. This reinforced the importance of building tools that help preserve and understand our digital memories. Future Enhancements Some ideas for future development: Support for more messaging platforms (WhatsApp, Telegram, Discord) Visualization dashboards for conversation patterns Sentiment analysis over time Export capabilities for research or archival purposes Multi-language support for international conversations Open Source iMessage LLM is open source and available on GitHub . Feel free to contribute, suggest features, or adapt it for your own use cases. The codebase is well-documented and modular, making it easy to extend or customize. Conclusion Our digital conversations are a treasure trove of memories and insights. iMessage LLM unlocks this data, transforming years of messages into analyzable, searchable, and understandable information. Whether you're interested in relationship dynamics, personal growth, or simply want to revisit old conversations, this tool provides the infrastructure to explore your digital communication history meaningfully. Give it a try - you might be surprised by what patterns emerge from your message history!", "body_markdown": "# iMessage LLM: Transform Your Message History into Analyzable Data with AI\n\nEver wondered what patterns lie hidden in years of iMessage conversations? What themes emerged in your relationships? How communication evolved over time? I built **iMessage LLM** , a powerful Python tool that converts iMessage exports into structured data and provides AI-powered analysis using Large Language Models.\n\n## The Problem\n\nWe accumulate thousands of messages over years, but they're locked away in an unstructured format. Apple's Messages app doesn't provide meaningful analytics or search capabilities beyond basic keyword matching. Our digital conversations contain valuable insights about relationships, personal growth, and communication patterns - but we have no way to access them systematically.\n\n## The Solution: iMessage LLM\n\niMessage LLM is a comprehensive toolkit that:\n\n- Converts HTML exports from [imessage-exporter](https://github.com/ReagentX/imessage-exporter) into structured CSV data\n- Intelligently groups messages into conversations using advanced algorithms\n- Provides AI-powered analysis using Ollama with DeepSeek-R1\n- Offers powerful filtering and search capabilities\n- Maintains conversation context for meaningful analysis\n\n## Smart Conversation Detection\n\nOne of the most innovative features is the intelligent conversation grouping. The tool doesn't just use simple time gaps - it employs multiple sophisticated signals:\n\n```python\n# Dynamic time thresholds based on time of day\nif (current_hour >= 22 or current_hour <= 6):\n    threshold = 3  # Night: shorter gaps\nelif 6 <= current_hour <= 10:\n    threshold = 8  # Morning: overnight gaps\nelse:\n    threshold = 4  # Day: active conversation time\n\n# Content analysis for conversation boundaries\nstarters = ['hey', 'hello', 'good morning', ...]\nenders = ['goodnight', 'bye', 'talk later', ...]\n\n# Momentum analysis - response times and engagement\navg_response_time = calculate_response_times(messages)\nturn_changes = count_turn_exchanges(messages)\nengagement_score = calculate_engagement(turn_changes, response_time)\n```\n\nThe algorithm also detects:\n\n- **Topic changes** using semantic similarity\n- **Emotional tone shifts** through emoji and keyword analysis\n- **Activity transitions** (\"just got to work\", \"heading home\")\n- **Conversation momentum** analyzing response patterns\n\n## AI-Powered Analysis\n\nOnce your messages are processed, the real magic begins. Using Ollama with DeepSeek-R1, you can ask natural language questions about your message history:\n\n```bash\n# Ask about specific years\n$ python ask_messages.py --year 2017 --question \"What happened this year?\"\n\n# Analyze relationship evolution\n$ python ask_messages.py --years 2017 2018 2019 --question \"How did our relationship evolve?\"\n\n# Examine specific conversations\n$ python ask_messages.py --conversation 42 --question \"What is this conversation about?\"\n\n# Filter by conversation characteristics\n$ python ask_messages.py --min-messages 50 --question \"What are the themes in longer conversations?\"\n```\n\n## Intelligent Features\n\n### Token Optimization\n\nThe tool uses DeepSeek's tokenizer for accurate token counting and automatically chunks large conversations to fit within model context limits:\n\n```python\n# Compact message format to save tokens\ndef compress_message_format(messages):\n    return \"\\n\".join([\n        f\"{msg['date']}|{msg['sender']}|{msg['message']}\"\n        for msg in messages\n    ])\n```\n\n### Smart Caching\n\nAnalysis results are cached to speed up repeated queries. The cache is invalidated if messages change or you switch models:\n\n```bash\n# List cached results\n$ python ask_messages.py --list-cache\n\n# Force reprocessing\n$ python ask_messages.py --question \"Analyze again\" --force-reprocess\n\n# Clear all cached results\n$ python ask_messages.py --clear-cache\n```\n\n### Conversation History\n\nThe tool maintains conversation history, allowing you to build on previous analyses:\n\n```bash\n# First analysis\n$ python ask_messages.py --question \"What are the main themes?\" --save-conversation themes.json\n\n# Follow-up question using context\n$ python ask_messages.py --load-conversation themes.json --question \"Tell me more about the second theme\"\n```\n\n## Performance & Scale\n\nThe tool is optimized for large datasets:\n\n- **Streaming HTML parsing** handles multi-gigabyte files efficiently\n- **Statistical sampling** for quality analysis on massive datasets\n- **Adaptive algorithms** that scale based on dataset size\n- **Progress tracking** with real-time metrics and ETA\n\n```text\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 CONVERSATION ANALYSIS \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n                    Analyzing conversation patterns...\n\n\u2713 Assigned 1,234 conversations in 2.3s\n\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Decision Statistics \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n  Time Based: 856 (69.4%)\n  Topic Change: 187 (15.2%)\n  Starter Based: 98 (7.9%)\n  Momentum Based: 93 (7.5%)\n```\n\n## Real-World Use Cases\n\nHere are some fascinating ways to use iMessage LLM:\n\n### Relationship Analysis\n\n```bash\n$ python ask_messages.py --question \"How has our communication style changed over the years?\"\n```\n\n### Topic Discovery\n\n```bash\n$ python ask_messages.py --question \"What are the recurring themes in our conversations?\"\n```\n\n### Memory Lane\n\n```bash\n$ python ask_messages.py --year 2020 --question \"What were we talking about during the pandemic?\"\n```\n\n### Communication Patterns\n\n```bash\n$ python ask_messages.py --question \"When do we have our deepest conversations?\"\n```\n\n## Getting Started\n\nSetting up iMessage LLM is straightforward:\n\n```bash\n# 1. Export your messages with imessage-exporter\n$ brew install imessage-exporter\n$ imessage-exporter --format html --output ./data/\n\n# 2. Install dependencies\n$ pip install -r requirements.txt\n\n# 3. Setup Ollama with DeepSeek\n$ ollama serve\n$ ollama pull deepseek-r1:14b\n\n# 4. Process your messages\n$ python process.py\n\n# 5. Start analyzing!\n$ python ask_messages.py --question \"What are the main themes in our conversations?\"\n```\n\n## Technical Architecture\n\nThe project consists of several key components:\n\n- **process.py** : Core processing engine with conversation detection algorithms\n- **ask_messages.py** : AI analysis interface with caching and history management\n- **prompts.py** : Centralized prompt templates for consistent AI interactions\n- **formatting_utils.py** : Beautiful terminal output with progress tracking\n- **deepseek_tokenizer.py** : Accurate token counting for context management\n\n## Privacy & Local Processing\n\nEverything runs locally on your machine. Your messages never leave your computer - the AI analysis uses Ollama running locally, not cloud services. This ensures complete privacy while still providing powerful insights.\n\n## What I Learned\n\nBuilding this tool taught me several valuable lessons:\n\n- Conversation boundaries are more nuanced than simple time gaps\n- Context windows and token management are crucial for LLM performance\n- Smart caching and chunking strategies enable analysis of massive datasets\n- Local AI models like DeepSeek-R1 are powerful enough for complex analysis tasks\n\n## The Build Process: From Concept to Reality\n\nThe journey of building iMessage LLM was both challenging and enlightening. It started with a simple curiosity: I had years of message history and wanted to understand what patterns and insights were hidden within. Here's how the project evolved from a weekend experiment to a comprehensive analysis toolkit.\n\n### Initial Prototype: The Naive Approach\n\nMy first attempt was embarrassingly simple - just dump all messages into a CSV and throw them at an LLM. The results were... disappointing:\n\n```python\n# Version 1: The naive approach that didn't work\nmessages = parse_html_to_csv(html_file)\nprompt = f\"Analyze these messages: {messages}\"\n# ERROR: Token limit exceeded (500,000+ tokens!)\n```\n\nReality hit hard. Years of messages meant millions of tokens, far exceeding any model's context window. I needed to be smarter about this.\n\n### Challenge #1: Parsing Apple's HTML Export Format\n\nThe imessage-exporter tool outputs HTML files with a specific structure that needed careful parsing. Apple's format includes attachments, reactions, and various message types that all required different handling:\n\n```python\n# Handling different message types\ndef parse_message_element(element):\n    message_type = element.get('data-type', 'text')\n\n    if message_type == 'attachment':\n        return handle_attachment(element)\n    elif message_type == 'reaction':\n        return handle_reaction(element)\n    elif message_type == 'edited':\n        return handle_edited_message(element)\n    else:\n        return extract_text_content(element)\n```\n\nThe biggest surprise? Emojis and special characters. They required special handling to prevent encoding issues when converting to CSV. I spent an entire evening debugging why certain messages were causing pandas to throw UTF-8 errors.\n\n### Challenge #2: Defining \"Conversations\"\n\nThis was the hardest problem to solve. What exactly constitutes a conversation? My first approach used a simple 30-minute gap rule:\n\n```python\n# Version 2: Simple time-based splitting (too simplistic)\ndef split_conversations_v1(messages):\n    conversations = []\n    current_convo = []\n\n    for i, msg in enumerate(messages):\n        if i > 0:\n            time_gap = msg['timestamp'] - messages[i-1]['timestamp']\n            if time_gap > timedelta(minutes=30):\n                conversations.append(current_convo)\n                current_convo = []\n        current_convo.append(msg)\n\n    return conversations\n```\n\nThis worked... poorly. It would split ongoing conversations just because someone took a lunch break, or merge completely unrelated topics just because they happened quickly. I needed something more sophisticated.\n\n### The Breakthrough: Multi-Signal Conversation Detection\n\nAfter analyzing my own message patterns, I realized conversations have multiple signals beyond just time gaps. This led to the current multi-signal approach:\n\n```python\n# Version 3: Multi-signal detection (the breakthrough)\nclass ConversationDetector:\n    def __init__(self):\n        self.signals = [\n            TimeOfDaySignal(),      # Different thresholds for different times\n            ContentSignal(),         # Detect greeting/farewell patterns\n            TopicSimilaritySignal(), # Use embeddings for semantic similarity\n            MomentumSignal(),        # Analyze response patterns\n            EmotionalToneSignal()    # Track emoji usage and sentiment\n        ]\n\n    def should_split(self, messages, index):\n        votes = [signal.vote(messages, index) for signal in self.signals]\n        return self.weighted_decision(votes)\n```\n\nEach signal votes on whether to split at a given point, and the final decision uses weighted voting. This dramatically improved conversation quality.\n\n### Challenge #3: Token Management and Context Windows\n\nEven with conversations properly segmented, many were still too large for LLM context windows. I needed intelligent chunking that preserved context:\n\n```python\n# Smart chunking that maintains conversation flow\ndef chunk_conversation(messages, max_tokens=8000):\n    chunks = []\n    current_chunk = []\n    current_tokens = 0\n\n    # Always include conversation metadata\n    metadata = create_conversation_summary(messages)\n    metadata_tokens = count_tokens(metadata)\n\n    for msg in messages:\n        msg_tokens = count_tokens(format_message(msg))\n\n        if current_tokens + msg_tokens > max_tokens - metadata_tokens:\n            # Save current chunk with overlap for context\n            chunks.append({\n                'messages': current_chunk,\n                'metadata': metadata,\n                'continuation': True\n            })\n            # Keep last few messages for context continuity\n            overlap = get_context_overlap(current_chunk)\n            current_chunk = overlap\n            current_tokens = count_tokens(overlap)\n\n        current_chunk.append(msg)\n        current_tokens += msg_tokens\n\n    return chunks\n```\n\n### Challenge #4: Performance at Scale\n\nProcessing years of messages (100,000+) was initially taking hours. Profiling revealed the bottlenecks:\n\n```text\nInitial Performance Profile:\n- HTML Parsing: 45% of runtime (Beautiful Soup)\n- Conversation Detection: 30% of runtime (O(n\u00b2) similarity checks)\n- CSV Writing: 15% of runtime (row-by-row pandas operations)\n- Token Counting: 10% of runtime (repeated tokenization)\n```\n\nThe optimizations that made the biggest difference:\n\n```python\n# Optimization 1: Streaming HTML parser\nfrom lxml import etree\nparser = etree.iterparse(html_file, events=('start', 'end'))\n# 10x faster than Beautiful Soup for large files\n\n# Optimization 2: Batch similarity computations\nembeddings = compute_embeddings_batch(messages)  # Vectorized operations\nsimilarities = cosine_similarity_matrix(embeddings)  # NumPy magic\n\n# Optimization 3: Bulk CSV operations\ndf = pd.DataFrame(all_messages)\ndf.to_csv('messages.csv', index=False)  # Single write operation\n\n# Final performance: 100,000 messages in ~3 minutes\n```\n\n### The DeepSeek Integration Journey\n\nChoosing the right model was crucial. I experimented with several options:\n\n- **GPT-3.5:** Good but expensive for large-scale analysis, privacy concerns\n- **LLaMA 2:** Decent but struggled with nuanced conversation analysis\n- **Mistral:** Fast but less accurate for relationship insights\n- **DeepSeek-R1:** The sweet spot - excellent reasoning, runs locally, great token efficiency\n\nDeepSeek-R1's ability to handle complex reasoning tasks while running entirely locally made it perfect for this privacy-sensitive application. The integration required custom tokenizer implementation:\n\n```python\n# Custom DeepSeek tokenizer for accurate token counting\nfrom transformers import AutoTokenizer\n\nclass DeepSeekTokenizer:\n    def __init__(self):\n        self.tokenizer = AutoTokenizer.from_pretrained(\n            'deepseek-ai/DeepSeek-R1-Distill-Llama-14B'\n        )\n        self._cache = {}  # Cache tokenization results\n\n    def count_tokens(self, text):\n        if text in self._cache:\n            return self._cache[text]\n\n        tokens = len(self.tokenizer.encode(text))\n        self._cache[text] = tokens\n        return tokens\n```\n\n### Testing with Real Data: Unexpected Discoveries\n\nTesting with my own message history revealed fascinating edge cases:\n\n- **Group chats vs. one-on-one:** Required different conversation detection logic\n- **Media-heavy conversations:** Needed special handling for photo/video descriptions\n- **Time zone changes:** Travel caused conversation splitting issues\n- **Language switching:** Multilingual conversations needed special tokenization\n\nEach edge case led to refinements in the algorithm. For example, detecting time zone changes:\n\n```python\n# Detect potential timezone changes\ndef detect_timezone_shift(messages):\n    hourly_distribution = defaultdict(int)\n    for msg in messages:\n        hourly_distribution[msg['timestamp'].hour] += 1\n\n    # Sudden shift in active hours suggests timezone change\n    if has_distribution_shift(hourly_distribution):\n        return adjust_thresholds_for_timezone()\n```\n\n### The Caching System Evolution\n\nRepeated analysis of the same conversations was wasteful. The caching system evolved through three iterations:\n\n```python\n# Version 1: Simple file cache (problematic)\ncache[question] = answer  # Too simplistic, ignored context\n\n# Version 2: Content-aware cache\ncache_key = hash(messages + question + model)  # Better but rigid\n\n# Version 3: Intelligent cache with invalidation\nclass SmartCache:\n    def get_cache_key(self, messages, question, context):\n        # Include relevant factors that affect the answer\n        factors = {\n            'message_hash': self.hash_messages(messages),\n            'question_embedding': self.embed_question(question),\n            'model_version': self.model_version,\n            'context_summary': self.summarize_context(context)\n        }\n        return self.generate_stable_key(factors)\n\n    def should_invalidate(self, cache_entry):\n        return (\n            cache_entry['age'] > self.max_age or\n            cache_entry['model'] != self.current_model or\n            self.messages_updated_since(cache_entry['timestamp'])\n        )\n```\n\n### Lessons from User Feedback\n\nAfter sharing the tool with friends, I received valuable feedback that shaped the final version:\n\n- **\"It's too slow for quick questions\"** \u2192 Added the statistical sampling mode for instant responses\n- **\"I want to compare different time periods\"** \u2192 Built the multi-year comparison feature\n- **\"The terminal output is hard to read\"** \u2192 Created the beautiful formatted output system\n- **\"Can it remember previous analyses?\"** \u2192 Implemented conversation history management\n\nThe most rewarding feedback was from a friend who used it to analyze conversations with a deceased relative - finding patterns and memories they had forgotten about. This reinforced the importance of building tools that help preserve and understand our digital memories.\n\n## Future Enhancements\n\nSome ideas for future development:\n\n- Support for more messaging platforms (WhatsApp, Telegram, Discord)\n- Visualization dashboards for conversation patterns\n- Sentiment analysis over time\n- Export capabilities for research or archival purposes\n- Multi-language support for international conversations\n\n## Open Source\n\niMessage LLM is open source and available on [GitHub](https://github.com/aaronspindler/iMessageLLM) . Feel free to contribute, suggest features, or adapt it for your own use cases. The codebase is well-documented and modular, making it easy to extend or customize.\n\n## Conclusion\n\nOur digital conversations are a treasure trove of memories and insights. iMessage LLM unlocks this data, transforming years of messages into analyzable, searchable, and understandable information. Whether you're interested in relationship dynamics, personal growth, or simply want to revisit old conversations, this tool provides the infrastructure to explore your digital communication history meaningfully.\n\nGive it a try - you might be surprised by what patterns emerge from your message history!", "word_count": 2053, "content_hash": "b8edbc42c6cae14b1729ffcca9b2049e4ec1f4910bbfb6b54fded35b91837161"}