this repo has no description
syntax highlighting; bugfix for line end
+330
-9
+1
docs/README.md
+1
docs/README.md
···
8
8
- `docs/roadmap/editor_livecoding.md`: Livecoding editor plan (TIC‑80 UI vibes): CODE + CONSOLE only.
9
9
- `docs/roadmap/editor_shortcuts.md`: Implementation checklist for PageUp/Down, Shift variants, Ctrl+Home/End, and block indent/outdent.
10
10
- `docs/roadmap/editor_shortcuts_phase2.md`: Plan for word navigation/edit, smart line bounds, and document selection (Ctrl/Cmd+Shift+Home/End).
11
+
- `docs/roadmap/editor_syntax_highlighting.md`: Plan and TODOs for Lua syntax highlighting with CODE_THEME colors.
11
12
- `docs/roadmap/todos_code_review.md`: Rolling TODOs from code review (high/medium/low priority) with checkboxes.
12
13
- `docs/roadmap/todos_from_ai_review.md`: Rolling TODOs from AI code review (2025-08-27).
13
14
+88
docs/roadmap/editor_syntax_highlighting.md
+88
docs/roadmap/editor_syntax_highlighting.md
···
1
+
2
+
# Editor Syntax Highlighting — Plan & TODOs (CODE View)
3
+
4
+
Status: planning (tests-first, incremental)
5
+
6
+
This document is the implementation plan and checklist for adding syntax highlighting to the CODE view with TIC‑80 parity in spirit and practical constraints of the 240×136 framebuffer. We will implement Lua first (TIC‑80 default), use CODE_THEME colors, and integrate with the existing renderer, selection, and caret.
7
+
8
+
## Goals
9
+
10
+
- Lua (5.3) highlighting with a pragmatic tokenizer:
11
+
- Keywords, identifiers, numbers, strings (short/long), comments (line/block), signs/punctuation, API names.
12
+
- Multiline constructs (long strings/comments) carry state across lines.
13
+
- Theme‑driven colors from CODE_THEME:
14
+
- `BG`, `FG`, `STRING`, `NUMBER`, `KEYWORD`, `API`, `COMMENT`, `SIGN`, `SELECT`, `CURSOR`.
15
+
- Low overhead: tokenize only visible lines (plus minimal context) with per‑line state cache and dirty invalidation on edits.
16
+
- Selection/caret precedence: keep current selection overlay and caret behavior; highlighting is suppressed when selection inverts glyphs.
17
+
18
+
## Token Kinds and Color Map
19
+
20
+
- `Whitespace` (no color change; use previous).
21
+
- `Identifier` → `FG` (unless API or keyword).
22
+
- `Keyword` (Lua 5.3 set): `and, break, do, else, elseif, end, false, for, function, goto, if, in, local, nil, not, or, repeat, return, then, true, until, while` → `KEYWORD`.
23
+
- `Number` → `NUMBER` (support decimal, hex `0x..`, floats with exponent; pragmatic subset).
24
+
- `StringShort` (single/double quoted with escapes) → `STRING`.
25
+
- `StringLong` (Lua long brackets `[[ .. ]]`, `[=[ .. ]=]` nesting level) → `STRING`.
26
+
- `CommentLine` (`-- ...`) → `COMMENT`.
27
+
- `CommentBlock` (`--[[ .. ]]`, equal‑sign variants) → `COMMENT`.
28
+
- `API` (known TIC‑80 API names: `cls, pix, line, rect, rectb, circ, circb, elli, ellib, tri, trib, clip, print, peek, poke, memcpy, memset, fft*, vqt*` …) → `API`.
29
+
- `Sign` (operators, punctuation) → `SIGN`.
30
+
31
+
## Architecture
32
+
33
+
- Module: `editor/highlight.rs`
34
+
- `struct LineTok { runs: Vec<(col_start, col_end, Kind)>; state_out: State }`
35
+
- `enum State { Normal, InLongString { level: u32 }, InBlockComment { level: u32 } }`
36
+
- `fn lex_line(text: &str, state_in: State) -> LineTok`
37
+
- Color resolver: `fn color_for(kind: Kind, theme: &Theme) -> u8`
38
+
39
+
- Cache and invalidation
40
+
- Store per‑line `State` and compact token runs in `CodeBuffer` (or sibling `HighlighterCache`).
41
+
- On edit: mark changed line..end as dirty; recompute forward until state stabilizes (no change) or visible limit.
42
+
43
+
- Rendering hooks (in `CodeBuffer::draw`)
44
+
- Compute visible line range; request `LineTok` for each visible line.
45
+
- Before drawing monospace glyphs, set color per token run.
46
+
- Selection overlay: if selected, use current selection path (shadow + fill + dark glyph) and skip token color.
47
+
- Caret overlay unchanged.
48
+
49
+
- Theme load
50
+
- Parse CODE_THEME from TIC-80 `config.tic` already embedded; map palette indices directly (Sweetie16 with white=12, greys 13/14/15).
51
+
52
+
## Tests (TDD)
53
+
54
+
- Tokenization unit tests (`highlight_tests.rs`)
55
+
- Keywords vs identifiers; numbers (int/float/hex); strings short/long across lines; comments line/block with bracket levels.
56
+
- State propagation across lines for long strings/comments.
57
+
58
+
- Rendering tests (framebuffer)
59
+
- Draw a small snippet and assert pixel colors at representative positions:
60
+
- keyword, number, string, comment, API function call, sign.
61
+
- Verify selection overlay precedence (colored tokens become dark under selection).
62
+
63
+
- Performance sanity
64
+
- Tokenize visible range only; tests ensure caches invalidate on edits and recompute affected lines (without exhaustively re‑lexing entire file).
65
+
66
+
## TODO Checklist
67
+
68
+
- [ ] Skeleton module `editor/highlight.rs` (tokens, state, API).
69
+
- [ ] Theme plumbing: expose CODE_THEME colors (BG/FG/STRING/NUMBER/KEYWORD/API/COMMENT/SIGN).
70
+
- [ ] Lua tokenizer (short strings, numbers, keywords, comments, signs, identifiers).
71
+
- [ ] Long strings / block comments with `[[` and `[=[` nesting; cross‑line state.
72
+
- [ ] API name set and classification.
73
+
- [ ] Highlighter cache in editor: dirty range strategy + forward recompute.
74
+
- [ ] Integrate with `CodeBuffer::draw` (colors for tokens; selection/caret precedence).
75
+
- [ ] Unit tests: tokenization coverage.
76
+
- [ ] Framebuffer tests: colored pixels for representative tokens; selection precedence.
77
+
- [ ] Config flag to disable highlighting (optional; default on).
78
+
- [ ] Documentation updates (implementation status, testing catalog).
79
+
80
+
## Out of Scope (later phases)
81
+
82
+
- Language modes beyond Lua (Wren, Squirrel, MoonScript, JS/Python): pluggable lexers.
83
+
- Identifier‑based semantic highlighting (locals/upvalues vs globals).
84
+
- Multi‑threaded/background lexing (current size/perf doesn’t require it).
85
+
86
+
---
87
+
88
+
Implementation will follow this plan and check off items as they land with tests.
+48
-9
tic80_rust/src/editor/code.rs
+48
-9
tic80_rust/src/editor/code.rs
···
198
198
#[allow(clippy::missing_const_for_fn)]
199
199
pub fn end(&mut self) {
200
200
self.reset_caret_blink();
201
-
self.caret_col = self.line_len(self.caret_line);
201
+
// Move to visual end of line (exclude trailing newline)
202
+
let len_vis = if self.caret_line >= self.rope.len_lines() { 0 } else {
203
+
let seg = self.rope.line(self.caret_line);
204
+
let len = seg.len_chars();
205
+
if len > 0 && seg.chars().last() == Some('\n') { len - 1 } else { len }
206
+
};
207
+
self.caret_col = len_vis;
208
+
self.desired_col = Some(self.caret_col);
202
209
}
203
210
204
211
#[must_use]
···
233
240
234
241
pub fn move_right(&mut self) {
235
242
self.reset_caret_blink();
236
-
let len = self.line_len(self.caret_line);
243
+
let len = {
244
+
let seg = self.rope.line(self.caret_line);
245
+
let l = seg.len_chars();
246
+
if l > 0 && seg.chars().last() == Some('\n') { l - 1 } else { l }
247
+
};
237
248
if self.caret_col < len {
238
249
self.caret_col += 1;
239
250
} else if self.caret_line + 1 < self.line_count() {
···
248
259
let want = self.desired_col.unwrap_or(self.caret_col);
249
260
if self.caret_line > 0 {
250
261
self.caret_line -= 1;
251
-
self.caret_col = self.line_len(self.caret_line).min(want);
262
+
let len_vis = {
263
+
let seg = self.rope.line(self.caret_line);
264
+
let l = seg.len_chars(); if l > 0 && seg.chars().last() == Some('\n') { l - 1 } else { l }
265
+
};
266
+
self.caret_col = len_vis.min(want);
252
267
}
253
268
self.desired_col = Some(want);
254
269
}
···
258
273
let want = self.desired_col.unwrap_or(self.caret_col);
259
274
if self.caret_line + 1 < self.line_count() {
260
275
self.caret_line += 1;
261
-
self.caret_col = self.line_len(self.caret_line).min(want);
276
+
let len_vis = {
277
+
let seg = self.rope.line(self.caret_line);
278
+
let l = seg.len_chars(); if l > 0 && seg.chars().last() == Some('\n') { l - 1 } else { l }
279
+
};
280
+
self.caret_col = len_vis.min(want);
262
281
}
263
282
self.desired_col = Some(want);
264
283
}
···
367
386
if lc == 0 { return; }
368
387
let delta = vis.min(lc.saturating_sub(1) - self.caret_line);
369
388
self.caret_line += delta;
370
-
self.caret_col = self.line_len(self.caret_line).min(want);
389
+
let len_vis = {
390
+
let seg = self.rope.line(self.caret_line);
391
+
let l = seg.len_chars(); if l > 0 && seg.chars().last() == Some('\n') { l - 1 } else { l }
392
+
};
393
+
self.caret_col = len_vis.min(want);
371
394
self.desired_col = Some(want);
372
395
}
373
396
pub fn page_up(&mut self, vis: usize) {
···
376
399
let want = self.caret_col;
377
400
let delta = vis.min(self.caret_line);
378
401
self.caret_line -= delta;
379
-
self.caret_col = self.line_len(self.caret_line).min(want);
402
+
let len_vis = {
403
+
let seg = self.rope.line(self.caret_line);
404
+
let l = seg.len_chars(); if l > 0 && seg.chars().last() == Some('\n') { l - 1 } else { l }
405
+
};
406
+
self.caret_col = len_vis.min(want);
380
407
self.desired_col = Some(want);
381
408
}
382
409
···
502
529
// Draw characters cell-by-cell, applying selection overlays where needed
503
530
let line_char_start = self.rope.line_to_char(line_idx);
504
531
let sel = self.selection_range_idx();
532
+
// Syntax highlighting: get token runs and map to per-column colors
533
+
let mut col_colors: Vec<u8> = vec![crate::editor::highlight::default_theme().fg; line.chars().count()];
534
+
{
535
+
use crate::editor::highlight as hl;
536
+
let toks = hl::lex_line(&line, hl::State::Normal);
537
+
let theme = hl::default_theme();
538
+
for (a, b, k) in toks.runs {
539
+
let color = hl::color_for(k, theme);
540
+
let end = b.min(col_colors.len());
541
+
if a < end { col_colors[a..end].fill(color); }
542
+
}
543
+
}
505
544
for (i_vis, ch) in vis.chars().enumerate() {
506
545
let cell_x = area.x + gutter_w + gap + i32::try_from(i_vis).unwrap_or(0) * 6;
507
546
let cell_y = gutter_y;
···
516
555
let s = ch.to_string();
517
556
let _ = fb.print_text(&s, cell_x, cell_y, 15, true, 1, true);
518
557
} else {
519
-
// Normal glyph (no selection overlay)
558
+
// Normal glyph (no selection overlay): use token color
520
559
let s = ch.to_string();
521
-
// TIC default text color (no syntax) is white (12), 6px tall
522
-
let _ = fb.print_text(&s, cell_x, cell_y, 12, true, 1, true);
560
+
let color = *col_colors.get(i_vis).unwrap_or(&12u8);
561
+
let _ = fb.print_text(&s, cell_x, cell_y, color, true, 1, true);
523
562
}
524
563
}
525
564
// Extra selection cell for newline (EOL) when selected
+173
tic80_rust/src/editor/highlight.rs
+173
tic80_rust/src/editor/highlight.rs
···
1
+
2
+
#![allow(
3
+
clippy::cognitive_complexity,
4
+
clippy::single_match_else,
5
+
clippy::must_use_candidate,
6
+
clippy::missing_const_for_fn,
7
+
clippy::derivable_impls,
8
+
clippy::use_self,
9
+
unused_assignments,
10
+
unused_mut
11
+
)]
12
+
13
+
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
14
+
pub enum Kind {
15
+
Whitespace,
16
+
Identifier,
17
+
Keyword,
18
+
Number,
19
+
StringShort,
20
+
StringLong,
21
+
CommentLine,
22
+
CommentBlock,
23
+
Api,
24
+
Sign,
25
+
}
26
+
27
+
#[derive(Clone, Debug, PartialEq, Eq)]
28
+
pub enum State {
29
+
Normal,
30
+
InLongString { level: u32 },
31
+
InBlockComment { level: u32 },
32
+
}
33
+
34
+
impl Default for State { fn default() -> Self { Self::Normal } }
35
+
36
+
#[derive(Clone, Debug, Default)]
37
+
pub struct LineTok {
38
+
pub runs: Vec<(usize, usize, Kind)>, // [start,end) in columns
39
+
pub state_out: State,
40
+
}
41
+
42
+
#[derive(Clone, Copy, Debug)]
43
+
pub struct Theme {
44
+
pub bg: u8,
45
+
pub fg: u8,
46
+
pub string: u8,
47
+
pub number: u8,
48
+
pub keyword: u8,
49
+
pub api: u8,
50
+
pub comment: u8,
51
+
pub sign: u8,
52
+
}
53
+
54
+
#[must_use]
55
+
pub const fn default_theme() -> Theme {
56
+
Theme { bg: 15, fg: 12, string: 4, number: 11, keyword: 3, api: 5, comment: 14, sign: 13 }
57
+
}
58
+
59
+
const fn is_ident_start(c: char) -> bool { c == '_' || c.is_ascii_alphabetic() }
60
+
const fn is_ident_continue(c: char) -> bool { c == '_' || c.is_ascii_alphanumeric() }
61
+
62
+
fn is_keyword(s: &str) -> bool {
63
+
matches!(s,
64
+
"and"|"break"|"do"|"else"|"elseif"|"end"|"false"|"for"|"function"|
65
+
"goto"|"if"|"in"|"local"|"nil"|"not"|"or"|"repeat"|"return"|
66
+
"then"|"true"|"until"|"while")
67
+
}
68
+
69
+
fn is_api(s: &str) -> bool {
70
+
matches!(s,
71
+
"cls"|"pix"|"line"|"rect"|"rectb"|"circ"|"circb"|"elli"|"ellib"|
72
+
"tri"|"trib"|"clip"|"print"|"peek"|"poke"|"memcpy"|"memset"|
73
+
"fft"|"ffts"|"fftr"|"fftrs"|"vqt"|"vqts"|"vqtr"|"vqtrs")
74
+
}
75
+
76
+
#[must_use]
77
+
pub fn lex_line(text: &str, mut state: State) -> LineTok {
78
+
let mut out = LineTok::default();
79
+
let chars: Vec<char> = text.chars().collect();
80
+
let mut i = 0usize;
81
+
// Handle continuing states
82
+
match state {
83
+
State::InLongString { level } => {
84
+
// search for ]=...=]
85
+
while i < chars.len() {
86
+
if chars[i] == ']' {
87
+
// check = level then ]
88
+
let mut j = 0; while i + 1 + j < chars.len() && j < level as usize && chars[i+1 + j] == '=' { j += 1; }
89
+
if j == level as usize && i + 1 + j < chars.len() && chars[i+1 + j] == ']' {
90
+
// include closing delimiter
91
+
out.runs.push((0, i + 2 + j, Kind::StringLong));
92
+
i += 2 + j; state = State::Normal; break;
93
+
}
94
+
}
95
+
i += 1;
96
+
}
97
+
if matches!(state, State::InLongString { .. }) { out.runs.push((0, chars.len(), Kind::StringLong)); out.state_out = state; return out; }
98
+
}
99
+
State::InBlockComment { level } => {
100
+
while i < chars.len() {
101
+
if chars[i] == ']' {
102
+
let mut j = 0; while i + 1 + j < chars.len() && j < level as usize && chars[i+1 + j] == '=' { j += 1; }
103
+
if j == level as usize && i + 1 + j < chars.len() && chars[i+1 + j] == ']' { out.runs.push((0, i + 2 + j, Kind::CommentBlock)); i += 2 + j; state = State::Normal; break; }
104
+
}
105
+
i += 1;
106
+
}
107
+
if matches!(state, State::InBlockComment { .. }) { out.runs.push((0, chars.len(), Kind::CommentBlock)); out.state_out = state; return out; }
108
+
}
109
+
State::Normal => {}
110
+
}
111
+
112
+
// Normal scanning
113
+
i = 0;
114
+
let push_run = |start: usize, end: usize, kind: Kind, out: &mut LineTok| { if end > start { out.runs.push((start, end, kind)); } };
115
+
while i < chars.len() {
116
+
let c = chars[i];
117
+
// whitespace
118
+
if c.is_whitespace() { let start = i; while i < chars.len() && chars[i].is_whitespace() { i += 1; } push_run(start, i, Kind::Whitespace, &mut out); continue; }
119
+
// comment
120
+
if c == '-' && i + 1 < chars.len() && chars[i+1] == '-' {
121
+
// block comment?
122
+
if i + 3 < chars.len() && chars[i+2] == '[' {
123
+
let mut j = i + 3; let mut level = 0u32; while j < chars.len() && chars[j] == '=' { level += 1; j += 1; }
124
+
if j < chars.len() && chars[j] == '[' {
125
+
let mut k = j + 1; let mut closed = None;
126
+
while k < chars.len() { if chars[k] == ']' {
127
+
let mut eq = 0usize; while k + 1 + eq < chars.len() && eq < level as usize && chars[k+1+eq] == '=' { eq += 1; }
128
+
if eq == level as usize && k + 1 + eq < chars.len() && chars[k+1+eq] == ']' { closed = Some(k + 2 + eq); break; }
129
+
} k += 1; }
130
+
match closed { Some(end) => { push_run(i, end, Kind::CommentBlock, &mut out); i = end; continue; }
131
+
None => { push_run(i, chars.len(), Kind::CommentBlock, &mut out); out.state_out = State::InBlockComment { level }; return out; } }
132
+
}
133
+
}
134
+
push_run(i, chars.len(), Kind::CommentLine, &mut out); i = chars.len(); break;
135
+
}
136
+
// short strings
137
+
if c == '"' || c == '\'' {
138
+
let quote = c; let start = i; i += 1;
139
+
while i < chars.len() {
140
+
if chars[i] == '\\' { i += 2; continue; }
141
+
if chars[i] == quote { i += 1; break; }
142
+
i += 1;
143
+
}
144
+
push_run(start, i, Kind::StringShort, &mut out); continue;
145
+
}
146
+
// long string
147
+
if c == '[' { let mut j = i + 1; let mut level = 0u32; while j < chars.len() && chars[j] == '=' { level += 1; j += 1; } if j < chars.len() && chars[j] == '[' {
148
+
let start = i; let mut k = j + 1; let mut closed = None; while k < chars.len() { if chars[k] == ']' {
149
+
let mut eq = 0usize; while k + 1 + eq < chars.len() && eq < level as usize && chars[k+1+eq] == '=' { eq += 1; }
150
+
if eq == level as usize && k + 1 + eq < chars.len() && chars[k+1+eq] == ']' { closed = Some(k + 2 + eq); break; }
151
+
} k += 1; } match closed { Some(end) => { push_run(start, end, Kind::StringLong, &mut out); i = end; continue; } None => { push_run(start, chars.len(), Kind::StringLong, &mut out); out.state_out = State::InLongString { level }; return out; } } } }
152
+
// number
153
+
if c.is_ascii_digit() { let start = i; i += 1; while i < chars.len() && (chars[i].is_ascii_hexdigit() || matches!(chars[i], '.'|'x'|'X'|'e'|'E'|'+'|'-'|'_')) { i += 1; } push_run(start, i, Kind::Number, &mut out); continue; }
154
+
// identifier / keyword / api
155
+
if is_ident_start(c) { let start = i; i += 1; while i < chars.len() && is_ident_continue(chars[i]) { i += 1; } let s: String = chars[start..i].iter().copied().collect(); let kind = if is_keyword(&s) { Kind::Keyword } else if is_api(&s) { Kind::Api } else { Kind::Identifier }; push_run(start, i, kind, &mut out); continue; }
156
+
// sign
157
+
push_run(i, i+1, Kind::Sign, &mut out); i += 1;
158
+
}
159
+
out.state_out = state; out
160
+
}
161
+
162
+
#[must_use]
163
+
pub const fn color_for(kind: Kind, th: Theme) -> u8 {
164
+
match kind {
165
+
Kind::Whitespace | Kind::Identifier => th.fg,
166
+
Kind::Keyword => th.keyword,
167
+
Kind::Number => th.number,
168
+
Kind::StringShort | Kind::StringLong => th.string,
169
+
Kind::CommentLine | Kind::CommentBlock => th.comment,
170
+
Kind::Api => th.api,
171
+
Kind::Sign => th.sign,
172
+
}
173
+
}
+1
tic80_rust/src/lib.rs
+1
tic80_rust/src/lib.rs
+19
tic80_rust/tests/highlight_lexer_tests.rs
+19
tic80_rust/tests/highlight_lexer_tests.rs
···
1
+
2
+
use tic80_rust::editor::highlight as hl;
3
+
4
+
#[test]
5
+
fn lex_keywords_ident_numbers() {
6
+
let line = "local x = 42";
7
+
let tok = hl::lex_line(line, hl::State::Normal);
8
+
assert!(tok.runs.iter().any(|r| matches!(r, (0, 5, hl::Kind::Keyword))));
9
+
assert!(tok.runs.iter().any(|r| matches!(r, (6, 7, hl::Kind::Identifier))));
10
+
assert!(tok.runs.iter().any(|r| matches!(r, (10, 12, hl::Kind::Number))));
11
+
}
12
+
13
+
#[test]
14
+
fn lex_strings_and_comments() {
15
+
let line = "print(\"hi\") -- ok";
16
+
let tok = hl::lex_line(line, hl::State::Normal);
17
+
assert!(tok.runs.iter().any(|(_,_,k)| matches!(k, hl::Kind::StringShort)));
18
+
assert!(tok.runs.iter().any(|(_,_,k)| matches!(k, hl::Kind::CommentLine)));
19
+
}