parent:
f446e2800b994f797ef5d5c74b41c0ff29304839
nmyk <nick@nmyk.io>
2026-11-14T15:24:28-05:00
add To/FromTree
diff --git a/PLAN.md b/PLAN.md
index 70b41d2eabef27e66190b1057bbbbb872d86a32d..6c2af0e54b34e30a38351d15ef9310aa4c50ff02 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -101,10 +101,10 @@ 14. **Node OIDs** (`ast`)
- Build: the `GOV` root constant; `EncodeOID(ns, id, text) x509.OID` and `DecodeOID(x509.OID) (ns, id, text, error)`; payload grammar checks per namespace.
- Learn: what an OID is (a path in a global naming tree), arcs, the `2.25` UUID arc, and how Go's `x509.OID` holds arcs too big for `uint64`.
- Done when: every row of the design's namespace table round-trips; surrogates, empty symbols and `"1."` as a float are rejected.
-15. **Lowering and raising** (`ast`)
- - Build: `Lower(program []sexpr.Value) *Tree`, which classifies reserved names into ns 0 and numbers nodes in pre-order; `Raise(*Tree) []sexpr.Value`, the inverse, which also keeps a map from each produced value to its node for error locations.
+15. **ToTree and FromTree** (`ast`)
+ - Build: `ToTree(program []sexpr.Value) *Tree`, which classifies reserved names into ns 0 and numbers nodes in pre-order; `FromTree(*Tree) []sexpr.Value`, the inverse, which also keeps a map from each produced value to its node for error locations.
- Learn: an AST as a separate structure from source text; why sibling order must be explicit data once you leave a text format.
- - Done when: `Raise(Lower(p))` equals `p` for every `testdata` program, and `'+` raises to the symbol `+`.
+ - Done when: `FromTree(ToTree(p))` equals `p` for every `testdata` program, and `'+` becomes the symbol `+`.
16. **Layering** (`ast`)
- Build: `Layer(*Tree) Layers`, which computes depth n and returns per-depth node sets plus parent→child edges. No padding: certs 2..n assert `anyPolicy`, and the validator copies each unmapped leaf down.
- Learn: why the policy tree forces every leaf to depth n (RFC 5280 §6.1.3 step d.3 pruning), and how `anyPolicy` (RFC 9618 step d.2) makes the validator do the padding.
@@ -135,7 +135,7 @@ - Build: hand-encode `certificatePolicies` (OID `2.5.29.32`) as a SEQUENCE of PolicyInformation, OIDs only for now, leaving room for qualifiers in Phase 7. Cert 1 lists the top-level nodes; certs 2..n list `anyPolicy` (`2.5.29.32.0`).
- Learn: policy OIDs as assertions a CA makes about a certificate, which is the hook GOVERNMENT abuses. Read RFC 5280 §4.2.1.4.
- Done when: `ParseCertificate` returns the OIDs in `Policies`, including one with the 128-bit `GOV` arc.
22. **The transpiler** (`lisp2gov`, `cmd/government`)
- - Build: `Compile(src) ([]byte, error)`: read, lower, layer, then one cert per depth (cert 1 with the top-level nodes, the rest with `anyPolicy`), each with the mappings for its depth's outgoing edges; per-depth keys from the seed; PEM with the source as a header comment. Wire up `government compile`.
+ - Build: `Compile(src) ([]byte, error)`: read, build the tree, layer, then one cert per depth (cert 1 with the top-level nodes, the rest with `anyPolicy`), each with the mappings for its depth's outgoing edges; per-depth keys from the seed; PEM with the source as a header comment. Wire up `government compile`.
- Learn: how a certification path is a chain of issuer→subject links; how a whole program becomes n+1 signed objects.
- Done when: `government compile examples/sq.lisp -o sq.gov` writes a file that `openssl x509` can print block by block, and compiling twice gives identical bytes. **Milestone: GOVERNMENT programs exist.**
@@ -177,7 +177,7 @@ - Build: after processing, every asserted policy is a node at its depth, every mapping produced an edge, and node ids are exactly 1..N (else an orphaned-policy error naming the OID); mappings in cert *n* rejected; `anyPolicy` missing from a cert 2..n, or asserted in cert 1, rejected; `-max-nodes` budget.
- Learn: how silent pruning makes RFC processing forgiving, and why a language needs it to be unforgiving; policy-processing DoS (CVE-2026-32281 in Go).
- Done when: deleting one mapping from a valid file gives an error (a gap in the node ids, or a list node left with no elements), not a shorter program, except for the subtree with the highest ids, which the test documents.
30. **The full pipeline** (`cmd/government`)
- - Build: `government run`: load → ratify → policy graph → `Unlayer` → `Raise` → evaluate; errors carry the node OID and depth; exit codes 1 and 2. Switch the test runner to compile every `testdata` file and run the resulting bytes.
+ - Build: `government run`: load → ratify → policy graph → `Unlayer` → `FromTree` → evaluate; errors carry the node OID and depth; exit codes 1 and 2. Switch the test runner to compile every `testdata` file and run the resulting bytes.
- Learn: how the stages compose into a compiler-like pipeline with a trust boundary in the middle.
- Done when: every `testdata/*.lisp` file passes through the full pipeline with no test changes. **Milestone: GOVERNMENT runs.**
@@ -187,7 +187,7 @@ Make programs inspectable. These commands are also the best debugging aids for Phase 7.
31. **tree and decompile** (`cmd/government`, `gov2lisp`)
- Build: `government tree` prints the policy graph with `GOV` abbreviated, payloads decoded and depth shown; `government decompile` prints Lisp from the decoded AST via the `sexpr` printer. Add the round-trip test: `decompile(compile(src))` equals the normalized source for every `testdata` file.
- - Learn: decompilation as the inverse of every lowering step; why a round-trip test catches bugs in both directions at once.
+ - Learn: decompilation as the inverse of every compile step; why a round-trip test catches bugs in both directions at once.
- Done when: `government tree` on the worked example matches the design doc's drawing, and the round-trip test passes.
32. **verify** (`cmd/government`)
- Build: `government verify` ratifies and decodes without evaluating, then prints a certificate of ratification: anchor subject, key fingerprint (SHA-256 of the SPKI), window in force (the overlap of all validity periods), depth and node count.
diff --git a/ast/tree.go b/ast/tree.go
new file mode 100644
index 0000000000000000000000000000000000000000..39464736312426ebd1c4e19776400f13d1cd7e64
--- /dev/null
+++ b/ast/tree.go
@@ -0,0 +1,216 @@
+package ast
+
+import (
+ "crypto/x509"
+ "fmt"
+ "strconv"
+
+ "government/sexpr"
+)
+
+// Node is one node of the program tree, and one policy OID in the
+// certificates. Only list nodes have children; an atom's payload is all
+// there is to it.
+type Node struct {
+ ID uint64
+ NS Namespace
+ Text string
+ Children []*Node
+}
+
+// Tree is a whole program. Its forms are the children of the policy tree's
+// anyPolicy root.
+type Tree struct {
+ Forms []*Node
+}
+
+func (n *Node) OID() (x509.OID, error) {
+ return EncodeOID(n.NS, n.ID, n.Text)
+}
+
+func (n *Node) IsList() bool {
+ return n.NS == NSBuiltin && n.Text == ListNode
+}
+
+// ToTree turns a parsed program into a tree, numbering nodes 1, 2, 3, ... in
+// pre-order (a list before its elements, elements left to right). Because
+// ids are given in reading order, sorting siblings by id recovers argument
+// order, which certificates do not preserve.
+func ToTree(program []sexpr.Value) (*Tree, error) {
+ b := &builder{}
+ t := &Tree{}
+ for _, form := range program {
+ n, err := b.node(form)
+ if err != nil {
+ return nil, err
+ }
+ t.Forms = append(t.Forms, n)
+ }
+ return t, nil
+}
+
+type builder struct {
+ lastID uint64
+}
+
+func (b *builder) node(v sexpr.Value) (*Node, error) {
+ b.lastID++
+ n := &Node{ID: b.lastID}
+ switch v := v.(type) {
+ case *sexpr.Pair:
+ n.NS, n.Text = NSBuiltin, ListNode
+ return n, b.elements(n, v)
+ case sexpr.Symbol:
+ n.NS, n.Text = NSSymbol, string(v)
+ if isReservedSymbol(n.Text) {
+ n.NS = NSBuiltin
+ }
+ case sexpr.Int:
+ n.NS, n.Text = NSInt, strconv.FormatInt(int64(v), 10)
+ case sexpr.Float:
+ // The reader keeps values, not spellings, so 3.10 is stored as 3.1.
+ // Print gives the shortest text that reads back as the same float.
+ n.NS, n.Text = NSFloat, sexpr.Print(v)
+ case sexpr.String:
+ n.NS, n.Text = NSString, string(v)
+ case sexpr.Bool:
+ n.NS, n.Text = NSBuiltin, False
+ if v {
+ n.Text = True
+ }
+ case sexpr.Empty:
+ n.NS, n.Text = NSBuiltin, Empty
+ default:
+ return nil, fmt.Errorf("%s cannot be in a program tree", sexpr.Print(v))
+ }
+ if n.NS != NSBuiltin {
+ if err := checkPayload(n.NS, n.Text); err != nil {
+ return nil, err
+ }
+ }
+ return n, nil
+}
+
+// elements makes each element of the list p a child of n. An improper
+// tail becomes a "." child followed by the tail, mirroring the source text
+// (a b . c).
+func (b *builder) elements(n *Node, p *sexpr.Pair) error {
+ var rest sexpr.Value = p
+ for {
+ switch r := rest.(type) {
+ case *sexpr.Pair:
+ child, err := b.node(r.Car)
+ if err != nil {
+ return err
+ }
+ n.Children = append(n.Children, child)
+ rest = r.Cdr
+ continue
+ case sexpr.Empty:
+ return nil
+ }
+ b.lastID++
+ n.Children = append(n.Children, &Node{ID: b.lastID, NS: NSBuiltin, Text: Dot})
+ tail, err := b.node(rest)
+ if err != nil {
+ return err
+ }
+ n.Children = append(n.Children, tail)
+ return nil
+ }
+}
+
+// isReservedSymbol reports whether a source symbol names a builtin. The
+// structural names never come out of the reader as symbols; if one did, it
+// must stay a symbol rather than turn into structure.
+func isReservedSymbol(name string) bool {
+ switch name {
+ case ListNode, Dot, True, False, Empty:
+ return false
+ }
+ return IsBuiltin(name)
+}
+
+// Origins maps each list FromTree produced back to its node, so an evaluation
+// error inside a .gov program can name the OID it came from, the way the
+// reader's positions name a line and column.
+type Origins map[*sexpr.Pair]*Node
+
+// FromTree is ToTree's inverse. A builtin name becomes the plain symbol, so
+// '+ quotes the symbol +, exactly as in source.
+func FromTree(t *Tree) ([]sexpr.Value, Origins, error) {
+ origins := Origins{}
+ forms := make([]sexpr.Value, len(t.Forms))
+ for i, n := range t.Forms {
+ v, err := value(n, origins)
+ if err != nil {
+ return nil, nil, err
+ }
+ forms[i] = v
+ }
+ return forms, origins, nil
+}
+
+func value(n *Node, origins Origins) (sexpr.Value, error) {
+ if n.IsList() {
+ return listValue(n, origins)
+ }
+ if len(n.Children) > 0 {
+ return nil, fmt.Errorf("node %d: %s %q cannot have children", n.ID, n.NS, n.Text)
+ }
+ switch n.NS {
+ case NSBuiltin:
+ switch n.Text {
+ case True:
+ return sexpr.T, nil
+ case False:
+ return sexpr.F, nil
+ case Empty:
+ return sexpr.Nil, nil
+ case Dot:
+ return nil, fmt.Errorf("node %d: . is only allowed second-to-last in a list", n.ID)
+ }
+ return sexpr.Symbol(n.Text), nil
+ case NSSymbol:
+ return sexpr.Symbol(n.Text), nil
+ case NSInt, NSFloat:
+ v, _, err := sexpr.ParseNumber(n.Text)
+ if err != nil {
+ return nil, fmt.Errorf("node %d: %w", n.ID, err)
+ }
+ return v, nil
+ case NSString:
+ return sexpr.String(n.Text), nil
+ }
+ return nil, fmt.Errorf("node %d: unknown %s", n.ID, n.NS)
+}
+
+func listValue(n *Node, origins Origins) (sexpr.Value, error) {
+ if len(n.Children) == 0 {
+ return nil, fmt.Errorf("node %d: a list node needs at least one element; () is its own builtin", n.ID)
+ }
+ children := n.Children
+ var tail sexpr.Value = sexpr.Nil
+ if k := len(children); k >= 3 && isDot(children[k-2]) {
+ t, err := value(children[k-1], origins)
+ if err != nil {
+ return nil, err
+ }
+ tail, children = t, children[:k-2]
+ }
+ items := make([]sexpr.Value, len(children))
+ for i, c := range children {
+ v, err := value(c, origins)
+ if err != nil {
+ return nil, err
+ }
+ items[i] = v
+ }
+ list := sexpr.ListWithTail(tail, items...).(*sexpr.Pair)
+ origins[list] = n
+ return list, nil
+}
+
+func isDot(n *Node) bool {
+ return n.NS == NSBuiltin && n.Text == Dot
+}
diff --git a/ast/tree_test.go b/ast/tree_test.go
new file mode 100644
index 0000000000000000000000000000000000000000..05c43b83c2dc499998945ebee1fb060d9f2a866d
--- /dev/null
+++ b/ast/tree_test.go
@@ -0,0 +1,240 @@
+package ast
+
+import (
+ "os"
+ "path/filepath"
+ "reflect"
+ "strconv"
+ "strings"
+ "testing"
+
+ "government/sexpr"
+)
+
+func mustRead(t *testing.T, src string) []sexpr.Value {
+ t.Helper()
+ forms, err := sexpr.ReadAll(src)
+ if err != nil {
+ t.Fatalf("ReadAll(%q): %v", src, err)
+ }
+ return forms
+}
+
+// outline prints a tree as "text#id" atoms and nested lists, so tests can
+// state the expected numbering compactly.
+func outline(nodes []*Node) string {
+ parts := make([]string, len(nodes))
+ for i, n := range nodes {
+ label := n.Text + "#" + itoa(n.ID)
+ if n.IsList() {
+ label = "(#" + itoa(n.ID) + " " + outline(n.Children) + ")"
+ }
+ parts[i] = label
+ }
+ return strings.Join(parts, " ")
+}
+
+func itoa(n uint64) string { return strconv.FormatUint(n, 10) }
+
+func TestToTreeNumbersInPreOrder(t *testing.T) {
+ tests := []struct {
+ src, want string
+ }{
+ // The design's worked example.
+ {"(+ 1 (* 2 3))", "(#1 +#2 1#3 (#4 *#5 2#6 3#7))"},
+ {"1 2", "1#1 2#2"},
+ {"'(a . b)", "(#1 quote#2 (#3 a#4 .#5 b#6))"},
+ {"(f () T F)", "(#1 f#2 ()#3 T#4 F#5)"},
+ }
+ for _, tt := range tests {
+ tree, err := ToTree(mustRead(t, tt.src))
+ if err != nil {
+ t.Errorf("ToTree(%s): %v", tt.src, err)
+ continue
+ }
+ if got := outline(tree.Forms); got != tt.want {
+ t.Errorf("ToTree(%s) = %s, want %s", tt.src, got, tt.want)
+ }
+ }
+}
+
+func TestToTreeNamespaces(t *testing.T) {
+ tree, err := ToTree(mustRead(t, `(define x "s") foo 1.50 -3 '+`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ f := tree.Forms
+ quoted := f[4].Children[1]
+ checks := []struct {
+ n *Node
+ ns Namespace
+ text string
+ }{
+ {f[0].Children[0], NSBuiltin, "define"},
+ {f[0].Children[1], NSSymbol, "x"},
+ {f[0].Children[2], NSString, "s"},
+ {f[1], NSSymbol, "foo"},
+ {f[2], NSFloat, "1.5"},
+ {f[3], NSInt, "-3"},
+ // Quoting does not change a name's namespace; it only changes how
+ // the evaluator treats the node.
+ {quoted, NSBuiltin, "+"},
+ }
+ for _, c := range checks {
+ if c.n.NS != c.ns || c.n.Text != c.text {
+ t.Errorf("node %d = %s %q, want %s %q", c.n.ID, c.n.NS, c.n.Text, c.ns, c.text)
+ }
+ }
+}
+
+func TestQuotedBuiltinBecomesSymbol(t *testing.T) {
+ tree, err := ToTree(mustRead(t, "'+"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ forms, _, err := FromTree(tree)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := sexpr.List(sexpr.Symbol("quote"), sexpr.Symbol("+"))
+ if !reflect.DeepEqual(forms[0], want) {
+ t.Errorf("FromTree = %s, want %s", sexpr.Print(forms[0]), sexpr.Print(want))
+ }
+}
+
+func TestRoundTripTestdata(t *testing.T) {
+ files, err := filepath.Glob("../testdata/*.lisp")
+ if err != nil || len(files) == 0 {
+ t.Fatalf("no testdata: %v", err)
+ }
+ for _, file := range files {
+ src, err := os.ReadFile(file)
+ if err != nil {
+ t.Fatal(err)
+ }
+ program := mustRead(t, string(src))
+ tree, err := ToTree(program)
+ if err != nil {
+ t.Errorf("%s: ToTree: %v", file, err)
+ continue
+ }
+ got, _, err := FromTree(tree)
+ if err != nil {
+ t.Errorf("%s: FromTree: %v", file, err)
+ continue
+ }
+ if !reflect.DeepEqual(got, program) {
+ t.Errorf("%s: FromTree(ToTree(p)) differs from p", file)
+ }
+ }
+}
+
+func TestRoundTripEdgeCases(t *testing.T) {
+ for _, src := range []string{
+ `'(1 2 . 3)`,
+ `'((a . b) . (c . ()))`,
+ `"" "😀" "\u{0}"`,
+ `1.0e+21 -0.0 1.0e-7 0.1`,
+ `-9223372036854775808`,
+ `'(quote lambda T F ())`,
+ } {
+ program := mustRead(t, src)
+ tree, err := ToTree(program)
+ if err != nil {
+ t.Errorf("ToTree(%s): %v", src, err)
+ continue
+ }
+ got, _, err := FromTree(tree)
+ if err != nil {
+ t.Errorf("FromTree(ToTree(%s)): %v", src, err)
+ continue
+ }
+ if !reflect.DeepEqual(got, program) {
+ t.Errorf("FromTree(ToTree(%s)) = %v", src, got)
+ }
+ }
+}
+
+func TestEveryTreeNodeEncodes(t *testing.T) {
+ tree, err := ToTree(mustRead(t, `(define (f) '(1 . "x")) 2.5 T`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var walk func([]*Node)
+ walk = func(nodes []*Node) {
+ for _, n := range nodes {
+ if _, err := n.OID(); err != nil {
+ t.Errorf("node %d: %v", n.ID, err)
+ }
+ walk(n.Children)
+ }
+ }
+ walk(tree.Forms)
+}
+
+func TestOrigins(t *testing.T) {
+ tree, err := ToTree(mustRead(t, "(+ 1 (* 2 3))"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ forms, origins, err := FromTree(tree)
+ if err != nil {
+ t.Fatal(err)
+ }
+ outer := forms[0].(*sexpr.Pair)
+ inner := outer.Cdr.(*sexpr.Pair).Cdr.(*sexpr.Pair).Car.(*sexpr.Pair)
+ if origins[outer].ID != 1 || origins[inner].ID != 4 {
+ t.Errorf("origins: outer #%d, inner #%d; want #1, #4", origins[outer].ID, origins[inner].ID)
+ }
+}
+
+func TestFromTreeRejects(t *testing.T) {
+ atom := func(id uint64, ns Namespace, text string) *Node { return &Node{ID: id, NS: ns, Text: text} }
+ list := func(id uint64, children ...*Node) *Node {
+ return &Node{ID: id, NS: NSBuiltin, Text: ListNode, Children: children}
+ }
+ tests := []struct {
+ name string
+ form *Node
+ want string
+ }{
+ {"empty list node", list(1), "at least one element"},
+ {"atom with a child", &Node{ID: 1, NS: NSInt, Text: "1", Children: []*Node{atom(2, NSInt, "2")}}, "cannot have children"},
+ {"dot at the start", list(1, atom(2, NSBuiltin, Dot), atom(3, NSSymbol, "a")), "second-to-last"},
+ {"dot at the end", list(1, atom(2, NSSymbol, "a"), atom(3, NSBuiltin, Dot)), "second-to-last"},
+ {"two tails", list(1, atom(2, NSSymbol, "a"), atom(3, NSBuiltin, Dot), atom(4, NSSymbol, "b"), atom(5, NSSymbol, "c")), "second-to-last"},
+ {"bad integer", atom(1, NSInt, "007"), "malformed"},
+ }
+ for _, tt := range tests {
+ _, _, err := FromTree(&Tree{Forms: []*Node{tt.form}})
+ if err == nil || !strings.Contains(err.Error(), tt.want) {
+ t.Errorf("%s: error = %v, want %q", tt.name, err, tt.want)
+ }
+ }
+}
+
+func TestToTreeRejects(t *testing.T) {
+ tests := []struct {
+ name string
+ form sexpr.Value
+ want string
+ }{
+ // The reader never makes these; ToTree must not turn them into
+ // structure if something else does.
+ {"symbol named .", sexpr.Symbol(Dot), "builtin name"},
+ {"symbol named (...)", sexpr.Symbol(ListNode), "builtin name"},
+ {"empty symbol", sexpr.Symbol(""), "empty symbol"},
+ {"non-literal value", fakeFunction{}, "cannot be in a program tree"},
+ }
+ for _, tt := range tests {
+ _, err := ToTree([]sexpr.Value{tt.form})
+ if err == nil || !strings.Contains(err.Error(), tt.want) {
+ t.Errorf("%s: error = %v, want %q", tt.name, err, tt.want)
+ }
+ }
+}
+
+type fakeFunction struct{ sexpr.FunctionBase }
+
+func (fakeFunction) Builtin() bool { return false }
+func (fakeFunction) Name() string { return "" }