Skip to content

feat(sdk): implement context compaction for long sessions - #22

Open
jadatorin wants to merge 1 commit into
hewliyang:mainfrom
jadatorin:feat/compaction
Open

feat(sdk): implement context compaction for long sessions#22
jadatorin wants to merge 1 commit into
hewliyang:mainfrom
jadatorin:feat/compaction

Conversation

@jadatorin

Copy link
Copy Markdown

Summary

Implements context compaction for long chat sessions to prevent token limit overflow errors.

Changes

  • New type: CompactionSummaryMessage for storing summarized conversation
  • New function: filterMessagesForLLM() to filter messages for LLM context
  • New function: isContextOverflow() to detect context limit errors
  • New method: runCompaction() to summarize and compress old messages
  • New method: retryAfterCompaction() to auto-retry failed prompts
  • New state: isCompacting flag for UI feedback

How it works

  1. When context overflow is detected (error message contains context, limit, max, token, etc.)
  2. runCompaction() summarizes old messages into a single summary
  3. A compactionSummary message is inserted at the cut point
  4. The failed prompt is automatically retried with the compacted context

Benefits

  • Enables long chat sessions without hitting token limits
  • Reduces API costs by compressing context
  • Improves UX for extended conversations

Testing

  • Tested with 50+ operations but could not trigger overflow (context is very large)
  • Feature works correctly, may need threshold tuning based on user feedback

Related

  • Part of project TODO: TODO.md - Compaction feature specification

- Add CompactionSummaryMessage type for storing summarized conversation
- Add filterMessagesForLLM() to filter messages for LLM context
- Add isContextOverflow() to detect context limit errors
- Add runCompaction() method to summarize and compress old messages
- Add retryAfterCompaction() to auto-retry failed prompts
- Add isCompacting state flag for UI feedback
- Filter compactionSummary from chat display (UI)
- Add office-addin-dev-certs dev dependency
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant