parse EOL as a token

README.md

@@ -8,7 +8,7 @@ standard Lua 5.2 (compound assignments, `?` print shorthand, single-line
 operators `@`/`%`/`$`, the rotate/logical-shift family `<<>` / `>><` / `>>>`,
 and `#include`).
 
-## Status — v0.2
+## Status — v0.3 (unreleased)
 
 Working today:
 
@@ -51,22 +51,22 @@ their PICO-8 work continue to get standard Lua treatment.
 | `#include path` directive | ✓ |
 | `_init` / `_update` / `_update60` / `_draw` highlighted as builtins | ✓ |
 
+### Line-significance (resolved in v0.3)
+
+PICO-8's shorthand `if (cond) ...` and `while (cond) ...` are
+line-bounded: a later-line `else` belongs to an enclosing standard
+`if`, not the shorthand, and a multi-statement single-line shorthand
+body collects every statement on the line. The external scanner emits
+a zero-width `LINE_END` token at `\n` / `\r` / EOF when (and only
+when) the parser is at the body-or-terminator decision point of a
+shorthand statement, so the AST now matches PICO-8 semantics — see
+[`grammars/pico-8-lua/KNOWN_LIMITATIONS.md`](grammars/pico-8-lua/KNOWN_LIMITATIONS.md)
+for the wiring detail and
+[`grammars/pico-8-lua/test/corpus/shorthand_line_end.txt`](grammars/pico-8-lua/test/corpus/shorthand_line_end.txt)
+for the test corpus.
+
 ### Known limitations
 
-- **Line-significant dialect features are not modeled.** PICO-8's
-  shorthand `if (cond) stmt [else stmt]` is line-bounded — a
-  later-line `else` belongs to an enclosing standard `if`, not the
-  shorthand. Without an external scanner, the grammar can't see
-  newlines, so it greedily binds `else` to the nearest `if` ( the
-  C / Java convention ) and treats a multi-statement single-line
-  shorthand body as one statement plus a sequence of unconditional
-  follow-ups. The parse is structurally wrong but **tokens still
-  classify correctly**, so syntax highlighting renders identically
-  to a correct parse; only auto-indent and semantic selection are
-  subtly affected. Full write-up:
-  [`grammars/pico-8-lua/KNOWN_LIMITATIONS.md`](grammars/pico-8-lua/KNOWN_LIMITATIONS.md).
-  Slated for v0.3 work alongside LSP integration ( which needs a
-  correct AST ).
 - **No language server.** No completion, hover docs, or diagnostics for
   PICO-8 builtins yet — only a static `function.builtin` highlight on
   recognized names. See Roadmap.
@@ -173,17 +173,22 @@ a Lua identifier resembles a section marker ( e.g. `local __foo__ = 1`
 must remain a `line`, not be re-tokenized as a marker ), and the
 fallback `unknown_section` rule.
 
+The Lua grammar has a corpus under `grammars/pico-8-lua/test/corpus/` —
+run `( cd grammars/pico-8-lua && npx tree-sitter test )`. The corpus
+exercises shorthand `if`/`while` line-end behavior: dangling-else,
+multi-statement bodies, EOF termination, nested same-line shorthands,
+and coexistence with standard `if (parenthesized) then ... end`.
+
 ## Roadmap
 
 ### v0.3 — Language server integration
 
-Prerequisite: an external scanner for `tree-sitter-pico8-lua` so the
-shorthand-if and shorthand-while bodies are line-bounded the way PICO-8
-defines them. See [`grammars/pico-8-lua/KNOWN_LIMITATIONS.md`](grammars/pico-8-lua/KNOWN_LIMITATIONS.md).
-LSP features that walk the AST ( unreachable-code lint, goto-definition
-through a conditional branch ) need correct structure.
+The line-significance prerequisite is now satisfied (see *Line-significance*
+above), so LSP features that walk the AST — unreachable-code lint,
+goto-definition through a conditional branch — have a correct structure
+to work against.
 
-Then wire up [`japhib/pico8-ls`](https://github.com/japhib/pico8-ls) ( or whichever
+Wire up [`japhib/pico8-ls`](https://github.com/japhib/pico8-ls) ( or whichever
 PICO-8 LSP is most maintained at the time ) for:
 
 - Completion of PICO-8 builtins ( `spr`, `circfill`, `btn`, `flr`, … ).