Add implementation strategy with epics and tickets · jauntywk.bsky.social/opencode-session-rs@da6e17c

+643

1 changed file

expand all

doc

discovery

genesis.md

+643

doc/discovery/genesis.md

··· 499 499 5. Do we need to support git snapshot operations? 500 500 6. What's the right balance between index size and lookup speed? 501 501 7. How to handle concurrent reads during updates? 502 + 503 + --- 504 + 505 + # Implementation Strategy 506 + 507 + ## Development Philosophy 508 + 509 + Build incrementally from core types to full materializer, with each phase producing usable functionality: 510 + 511 + 1. **Types first** - Define Rust types matching opencode schemas 512 + 2. **Storage layer** - Read individual files from disk 513 + 3. **Session loading** - Assemble complete sessions from files 514 + 4. **Index layer** - Fast lookups without parsing everything 515 + 5. **Watching** - Real-time updates via watchman 516 + 517 + ## Phase Overview 518 + 519 + ``` 520 + ┌─────────────────────────────────────────────────────────────────────┐ 521 + │ Phase 1: Foundation │ 522 + │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 523 + │ │ core-id │→│ core-type │→│ core-err │→│ stor-path │ │ 524 + │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ 525 + └─────────────────────────────────────────────────────────────────────┘ 526 + ↓ 527 + ┌─────────────────────────────────────────────────────────────────────┐ 528 + │ Phase 2: File Reading │ 529 + │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 530 + │ │ stor-mmap│→│ stor-read │→│ pars-sess │→│ pars-part │ │ 531 + │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ 532 + └─────────────────────────────────────────────────────────────────────┘ 533 + ↓ 534 + ┌─────────────────────────────────────────────────────────────────────┐ 535 + │ Phase 3: Session Assembly │ 536 + │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 537 + │ │ load-sess│→│ load-msg │→│ load-part │ │ 538 + │ └──────────┘ └──────────┘ └──────────┘ │ 539 + └─────────────────────────────────────────────────────────────────────┘ 540 + ↓ 541 + ┌─────────────────────────────────────────────────────────────────────┐ 542 + │ Phase 4: Index & Materializer │ 543 + │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 544 + │ │ idx-meta │→│ idx-build │→│ mat-sess │→│ mat-query │ │ 545 + │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ 546 + └─────────────────────────────────────────────────────────────────────┘ 547 + ↓ 548 + ┌─────────────────────────────────────────────────────────────────────┐ 549 + │ Phase 5: Real-time Watching │ 550 + │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 551 + │ │ watch-ev │→│ watch-sess│→│ watch-idx │ │ 552 + │ └──────────┘ └──────────┘ └──────────┘ │ 553 + └─────────────────────────────────────────────────────────────────────┘ 554 + ``` 555 + 556 + --- 557 + 558 + ## Epic 1: Core Foundation 559 + 560 + > Establish the fundamental types and error handling that everything else builds upon 561 + 562 + ### ticket: core-id 563 + 564 + **Title**: ID Types with Timestamp Extraction 565 + 566 + Define typed identifiers for sessions, messages, and parts: 567 + 568 + ```rust 569 + pub struct SessionId(String); // "ses_xxx" 570 + pub struct MessageId(String); // "msg_xxx" 571 + pub struct PartId(String); // "prt_xxx" 572 + 573 + impl SessionId { 574 + pub fn timestamp(&self) -> i64 { /* extract from hex */ } 575 + pub fn prefix(&self) -> &str { "ses" } 576 + pub fn as_str(&self) -> &str { &self.0 } 577 + } 578 + ``` 579 + 580 + **Acceptance**: 581 + - [ ] SessionId, MessageId, PartId types 582 + - [ ] Timestamp extraction from ID 583 + - [ ] Parse from string, display, debug 584 + - [ ] PartialEq, Eq, Hash, Clone, Serialize, Deserialize 585 + 586 + --- 587 + 588 + ### ticket: core-type 589 + 590 + **Title**: Core Data Types 591 + 592 + Define Rust structs matching opencode's TypeScript/Zod schemas: 593 + 594 + ```rust 595 + // Session metadata 596 + pub struct SessionInfo { id, title, project_id, directory, time, ... } 597 + 598 + // Messages (discriminated union) 599 + pub enum Message { User(UserMessage), Assistant(AssistantMessage) } 600 + 601 + // Parts (discriminated union) 602 + pub enum Part { Text(TextPart), Tool(ToolPart), ... } 603 + ``` 604 + 605 + **Acceptance**: 606 + - [ ] SessionInfo, SessionTime, SessionSummary 607 + - [ ] UserMessage, AssistantMessage 608 + - [ ] All 12 Part variants with full field types 609 + - [ ] FileDiff, ToolState variants 610 + - [ ] serde derive for JSON serialization 611 + 612 + --- 613 + 614 + ### ticket: core-err 615 + 616 + **Title**: Error Type and Result Alias 617 + 618 + Define a unified error type for the crate: 619 + 620 + ```rust 621 + pub enum Error { 622 + Io(std::io::Error), 623 + Json(serde_json::Error), 624 + InvalidId { id: String, reason: &'static str }, 625 + NotFound { entity: &'static str, id: String }, 626 + Migration { version: u32 }, 627 + Watchman(String), 628 + } 629 + 630 + pub type Result<T> = std::result::Result<T, Error>; 631 + ``` 632 + 633 + **Acceptance**: 634 + - [ ] Error enum with all variants 635 + - [ ] impl From for common errors 636 + - [ ] impl std::error::Error 637 + - [ ] impl Display with helpful messages 638 + - [ ] Result<T> type alias 639 + 640 + --- 641 + 642 + ### ticket: stor-path 643 + 644 + **Title**: Storage Path Resolution 645 + 646 + Implement XDG-compliant path discovery: 647 + 648 + ```rust 649 + pub struct StoragePaths { 650 + pub root: PathBuf, // ~/.local/share/opencode/storage 651 + pub session: PathBuf, // root/session 652 + pub message: PathBuf, // root/message 653 + pub part: PathBuf, // root/part 654 + pub diff: PathBuf, // root/session_diff 655 + } 656 + 657 + impl StoragePaths { 658 + pub fn detect() -> Result<Self>; 659 + pub fn session_file(&self, project_id: &str, session_id: &SessionId) -> PathBuf; 660 + pub fn message_dir(&self, session_id: &SessionId) -> PathBuf; 661 + pub fn part_dir(&self, message_id: &MessageId) -> PathBuf; 662 + } 663 + ``` 664 + 665 + **Acceptance**: 666 + - [ ] XDG detection via `xdg` crate 667 + - [ ] Path builders for each entity type 668 + - [ ] Test with OPENCODE_TEST_HOME override 669 + - [ ] Platform-specific path handling 670 + 671 + --- 672 + 673 + ## Epic 2: File Reading 674 + 675 + > Low-level file access with zero-copy techniques 676 + 677 + ### ticket: stor-mmap 678 + 679 + **Title**: Memory-Mapped File Wrapper 680 + 681 + Create a safe wrapper around memmap2: 682 + 683 + ```rust 684 + pub struct MappedFile { 685 + path: PathBuf, 686 + mmap: Mmap, 687 + } 688 + 689 + impl MappedFile { 690 + pub fn open(path: &Path) -> Result<Self>; 691 + pub fn as_bytes(&self) -> &[u8]; 692 + pub fn path(&self) -> &Path; 693 + pub fn len(&self) -> usize; 694 + } 695 + ``` 696 + 697 + **Acceptance**: 698 + - [ ] Safe mmap wrapper 699 + - [ ] Error handling for missing files 700 + - [ ] Arc for shared ownership 701 + - [ ] Document safety assumptions 702 + 703 + --- 704 + 705 + ### ticket: stor-read 706 + 707 + **Title**: File Reader with JSON Parsing 708 + 709 + Read and parse JSON files: 710 + 711 + ```rust 712 + pub struct FileReader { 713 + paths: StoragePaths, 714 + } 715 + 716 + impl FileReader { 717 + pub fn read_session(&self, project_id: &str, id: &SessionId) -> Result<SessionInfo>; 718 + pub fn read_message(&self, session_id: &SessionId, id: &MessageId) -> Result<Message>; 719 + pub fn read_part(&self, message_id: &MessageId, id: &PartId) -> Result<Part>; 720 + pub fn read_diff(&self, session_id: &SessionId) -> Result<Vec<FileDiff>>; 721 + pub fn read_mapped(&self, path: &Path) -> Result<Arc<MappedFile>>; 722 + } 723 + ``` 724 + 725 + **Acceptance**: 726 + - [ ] Read each entity type 727 + - [ ] Cache mapped files by path 728 + - [ ] Handle missing files gracefully 729 + 730 + --- 731 + 732 + ### ticket: pars-sess 733 + 734 + **Title**: Session File Parser 735 + 736 + Parse session JSON into typed struct: 737 + 738 + ```rust 739 + pub fn parse_session(data: &[u8]) -> Result<SessionInfo>; 740 + pub fn parse_session_from_file(path: &Path) -> Result<SessionInfo>; 741 + ``` 742 + 743 + **Acceptance**: 744 + - [ ] Parse from byte slice 745 + - [ ] Validate required fields 746 + - [ ] Handle optional fields 747 + - [ ] Test with real session files 748 + 749 + --- 750 + 751 + ### ticket: pars-part 752 + 753 + **Title**: Part Parser with All Variants 754 + 755 + Parse part JSON with discriminated union: 756 + 757 + ```rust 758 + pub fn parse_part(data: &[u8]) -> Result<Part>; 759 + pub fn parse_part_from_file(path: &Path) -> Result<Part>; 760 + ``` 761 + 762 + **Acceptance**: 763 + - [ ] Parse all 12 part types 764 + - [ ] Handle tool state variants 765 + - [ ] Test each variant with example data 766 + 767 + --- 768 + 769 + ## Epic 3: Session Assembly 770 + 771 + > Load complete sessions from individual files 772 + 773 + ### ticket: load-sess 774 + 775 + **Title**: Session Loader 776 + 777 + Load a complete session with all metadata: 778 + 779 + ```rust 780 + pub struct SessionLoader { 781 + reader: FileReader, 782 + } 783 + 784 + impl SessionLoader { 785 + pub fn load(&self, project_id: &str, session_id: &SessionId) -> Result<LoadedSession>; 786 + } 787 + 788 + pub struct LoadedSession { 789 + pub info: SessionInfo, 790 + pub diff: Option<Vec<FileDiff>>, 791 + } 792 + ``` 793 + 794 + **Acceptance**: 795 + - [ ] Load session info 796 + - [ ] Load associated diff file 797 + - [ ] Validate session integrity 798 + 799 + --- 800 + 801 + ### ticket: load-msg 802 + 803 + **Title**: Message Loader 804 + 805 + Load messages for a session: 806 + 807 + ```rust 808 + impl SessionLoader { 809 + pub fn load_messages(&self, session_id: &SessionId) -> Result<Vec<Message>>; 810 + pub fn load_message(&self, session_id: &SessionId, message_id: &MessageId) -> Result<Message>; 811 + } 812 + ``` 813 + 814 + **Acceptance**: 815 + - [ ] List message files in directory 816 + - [ ] Sort by ID (ascending = chronological) 817 + - [ ] Parse each message 818 + 819 + --- 820 + 821 + ### ticket: load-part 822 + 823 + **Title**: Part Loader 824 + 825 + Load parts for a message: 826 + 827 + ```rust 828 + impl SessionLoader { 829 + pub fn load_parts(&self, message_id: &MessageId) -> Result<Vec<Part>>; 830 + pub fn load_part(&self, message_id: &MessageId, part_id: &PartId) -> Result<Part>; 831 + } 832 + 833 + pub struct MessageWithParts { 834 + pub message: Message, 835 + pub parts: Vec<Part>, 836 + } 837 + ``` 838 + 839 + **Acceptance**: 840 + - [ ] List part files 841 + - [ ] Sort by ID 842 + - [ ] Parse each part 843 + - [ ] Assemble message with parts 844 + 845 + --- 846 + 847 + ## Epic 4: Index & Materializer 848 + 849 + > Fast in-memory index with lazy content loading 850 + 851 + ### ticket: idx-meta 852 + 853 + **Title**: Index Metadata Types 854 + 855 + Define the index data structures: 856 + 857 + ```rust 858 + pub struct SessionMeta { 859 + pub id: SessionId, 860 + pub title: String, 861 + pub created: i64, 862 + pub updated: i64, 863 + pub project_id: String, 864 + pub message_count: usize, 865 + } 866 + 867 + pub struct MessageMeta { 868 + pub id: MessageId, 869 + pub session_id: SessionId, 870 + pub role: Role, 871 + pub part_count: usize, 872 + } 873 + 874 + pub struct PartRef { 875 + pub id: PartId, 876 + pub message_id: MessageId, 877 + pub part_type: PartType, 878 + pub path: PathBuf, 879 + pub mmap: Option<Arc<MappedFile>>, 880 + } 881 + ``` 882 + 883 + **Acceptance**: 884 + - [ ] Define all index types 885 + - [ ] Separation from full content types 886 + 887 + --- 888 + 889 + ### ticket: idx-build 890 + 891 + **Title**: Index Builder 892 + 893 + Build index from storage: 894 + 895 + ```rust 896 + pub struct SessionIndex { 897 + pub sessions: HashMap<SessionId, SessionMeta>, 898 + pub messages: HashMap<MessageId, MessageMeta>, 899 + pub parts: HashMap<PartId, PartRef>, 900 + pub by_session: HashMap<SessionId, Vec<MessageId>>, 901 + pub by_message: HashMap<MessageId, Vec<PartId>>, 902 + } 903 + 904 + impl SessionIndex { 905 + pub fn build(paths: &StoragePaths) -> Result<Self>; 906 + pub fn build_for_project(paths: &StoragePaths, project_id: &str) -> Result<Self>; 907 + } 908 + ``` 909 + 910 + **Acceptance**: 911 + - [ ] Scan storage directories 912 + - [ ] Parse metadata only (not full content) 913 + - [ ] Build relationship maps 914 + - [ ] Handle missing files gracefully 915 + 916 + --- 917 + 918 + ### ticket: mat-sess 919 + 920 + **Title**: Session Materializer 921 + 922 + Implement the main materializer: 923 + 924 + ```rust 925 + pub struct SessionMaterializer { 926 + paths: StoragePaths, 927 + index: SessionIndex, 928 + files: HashMap<PathBuf, Arc<MappedFile>>, 929 + } 930 + 931 + impl SessionMaterializer { 932 + pub fn new() -> Result<Self>; 933 + pub fn for_project(project_id: &str) -> Result<Self>; 934 + 935 + // Index access 936 + pub fn sessions(&self) -> &HashMap<SessionId, SessionMeta>; 937 + pub fn session(&self, id: &SessionId) -> Option<&SessionMeta>; 938 + 939 + // Lazy content loading 940 + pub fn load_session(&self, id: &SessionId) -> Result<SessionInfo>; 941 + pub fn load_message(&self, id: &MessageId) -> Result<Message>; 942 + pub fn load_part(&self, id: &PartId) -> Result<Part>; 943 + } 944 + ``` 945 + 946 + **Acceptance**: 947 + - [ ] Initialize from storage 948 + - [ ] Index-based lookups 949 + - [ ] Lazy content loading 950 + - [ ] Mmap caching 951 + 952 + --- 953 + 954 + ### ticket: mat-query 955 + 956 + **Title**: Query Interface 957 + 958 + High-level query API: 959 + 960 + ```rust 961 + impl SessionMaterializer { 962 + // Get complete session tree 963 + pub fn get_session_tree(&self, id: &SessionId) -> Result<SessionTree>; 964 + 965 + // Filter sessions 966 + pub fn sessions_by_time(&self, since: i64) -> Vec<&SessionMeta>; 967 + pub fn sessions_by_project(&self, project_id: &str) -> Vec<&SessionMeta>; 968 + 969 + // Navigate relationships 970 + pub fn messages_for_session(&self, session_id: &SessionId) -> Vec<&MessageMeta>; 971 + pub fn parts_for_message(&self, message_id: &MessageId) -> Vec<&PartRef>; 972 + } 973 + 974 + pub struct SessionTree { 975 + pub session: SessionInfo, 976 + pub messages: Vec<MessageWithParts>, 977 + pub diff: Option<Vec<FileDiff>>, 978 + } 979 + ``` 980 + 981 + **Acceptance**: 982 + - [ ] Session tree assembly 983 + - [ ] Time-based filtering 984 + - [ ] Project filtering 985 + - [ ] Relationship navigation 986 + 987 + --- 988 + 989 + ## Epic 5: Real-time Watching 990 + 991 + > Watch for file changes and update materializer 992 + 993 + ### ticket: watch-ev 994 + 995 + **Title**: Watch Event Types 996 + 997 + Define change events: 998 + 999 + ```rust 1000 + pub enum SessionEvent { 1001 + SessionCreated { project_id: String, session_id: SessionId }, 1002 + SessionUpdated { project_id: String, session_id: SessionId }, 1003 + SessionDeleted { project_id: String, session_id: SessionId }, 1004 + MessageCreated { session_id: SessionId, message_id: MessageId }, 1005 + MessageDeleted { session_id: SessionId, message_id: MessageId }, 1006 + PartCreated { message_id: MessageId, part_id: PartId }, 1007 + PartUpdated { message_id: MessageId, part_id: PartId }, 1008 + PartDeleted { message_id: MessageId, part_id: PartId }, 1009 + DiffUpdated { session_id: SessionId }, 1010 + FullRebuild, 1011 + } 1012 + ``` 1013 + 1014 + **Acceptance**: 1015 + - [ ] Event enum for all change types 1016 + - [ ] Path → Event parser 1017 + 1018 + --- 1019 + 1020 + ### ticket: watch-sess 1021 + 1022 + **Title**: Session Watcher 1023 + 1024 + Watch storage directory for changes: 1025 + 1026 + ```rust 1027 + pub struct SessionWatcher { 1028 + // watchman or notify backend 1029 + } 1030 + 1031 + impl SessionWatcher { 1032 + pub async fn new(paths: &StoragePaths) -> Result<Self>; 1033 + pub async fn next_event(&mut self) -> Result<SessionEvent>; 1034 + pub async fn initial_scan(&self) -> Result<Vec<SessionEvent>>; 1035 + } 1036 + ``` 1037 + 1038 + **Acceptance**: 1039 + - [ ] Watchman client integration 1040 + - [ ] Fallback to notify crate 1041 + - [ ] Fallback to polling 1042 + - [ ] Initial directory scan 1043 + 1044 + --- 1045 + 1046 + ### ticket: watch-idx 1047 + 1048 + **Title**: Live Index Updates 1049 + 1050 + Update materializer from events: 1051 + 1052 + ```rust 1053 + impl SessionMaterializer { 1054 + pub async fn watch(&mut self) -> Result<()>; 1055 + pub fn apply_event(&mut self, event: SessionEvent) -> Result<()>; 1056 + } 1057 + 1058 + // Usage 1059 + let mut mat = SessionMaterializer::new()?; 1060 + tokio::spawn(async move { 1061 + mat.watch().await 1062 + }); 1063 + ``` 1064 + 1065 + **Acceptance**: 1066 + - [ ] Start watching on materializer 1067 + - [ ] Apply events to index 1068 + - [ ] Update mmap cache 1069 + - [ ] Handle full rebuild 1070 + 1071 + --- 1072 + 1073 + ## Dependency Graph 1074 + 1075 + ``` 1076 + core-id ─────────────────────────────────────────────────────────┐ 1077 + │ │ 1078 + ▼ │ 1079 + core-type ◄──────────────────────────────────────────────────────┤ 1080 + │ │ 1081 + ▼ │ 1082 + core-err ◄───────────────────────────────────────────────────────┤ 1083 + │ │ 1084 + ▼ │ 1085 + stor-path ───────────────────────────────────────────────────────┤ 1086 + │ │ 1087 + ├──────────────┐ │ 1088 + ▼ ▼ │ 1089 + stor-mmap pars-sess │ 1090 + │ │ │ 1091 + ▼ ▼ │ 1092 + stor-read ◄── pars-part │ 1093 + │ │ │ 1094 + ├──────────────┴──────────────┐ │ 1095 + ▼ ▼ ▼ │ 1096 + load-sess load-msg load-part │ 1097 + │ │ │ │ 1098 + └──────────────┴──────────────┘ │ 1099 + │ │ 1100 + ▼ │ 1101 + idx-meta │ 1102 + │ │ 1103 + ▼ │ 1104 + idx-build │ 1105 + │ │ 1106 + ▼ │ 1107 + mat-sess ◄───────────────────────────────────────────┘ 1108 + │ 1109 + ▼ 1110 + mat-query 1111 + │ 1112 + ▼ 1113 + watch-ev 1114 + │ 1115 + ▼ 1116 + watch-sess 1117 + │ 1118 + ▼ 1119 + watch-idx 1120 + ``` 1121 + 1122 + --- 1123 + 1124 + ## Milestone Deliverables 1125 + 1126 + ### Milestone 1: Read Session Files (End of Phase 2) 1127 + 1128 + - Can read and parse any session/message/part file 1129 + - CLI tool to dump session info: `opencode-session dump <session-id>` 1130 + 1131 + ### Milestone 2: Load Complete Sessions (End of Phase 3) 1132 + 1133 + - Can load full session trees 1134 + - CLI tool to export session: `opencode-session export <session-id>` 1135 + 1136 + ### Milestone 3: Fast Queries (End of Phase 4) 1137 + 1138 + - Index-based lookups without parsing everything 1139 + - CLI tool to list sessions: `opencode-session list --project <id>` 1140 + 1141 + ### Milestone 4: Real-time Updates (End of Phase 5) 1142 + 1143 + - Live watching of session changes 1144 + - Example daemon: `opencode-session watch --project <id>`

Configure Feed

Configure Feed