Constrained sampling of language model already works great, even with a (e.g. JSON) schema where the parser is built on the fly. I already considered implementing a tree-sitter version of the sampling, which poses some interesting challenges that go deep into parser design. Note: Treesitter is not even the right tool for the job, but a lot of parsers are available and poses an interesting challenge.

Going a step further, LSPs in constrained sampling would be just a great fit. The sampling is then not only aware of the language but also of my own project. It is also a great extension point for all text-based (semi?-) structured text. I could for example make it file-system aware, add other plug-ins like a identifier-name-generator, snippets, documentation and so on. Particularly interesting: Refactoring-Actions. Everything as extension, everything project-specific, everything local everything hackable. Like vom for language.