government

commit 3608a1b3f346ed04e58a598e4f092b09f515c3fd

tree

parent:
f446e2800b994f797ef5d5c74b41c0ff29304839

nmyk <nick@nmyk.io>

2026-11-14T15:24:28-05:00

add To/FromTree

diff --git a/PLAN.md b/PLAN.md
index 70b41d2eabef27e66190b1057bbbbb872d86a32d..6c2af0e54b34e30a38351d15ef9310aa4c50ff02 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -101,10 +101,10 @@ 14. **Node OIDs** (`ast`)
     - Build: the `GOV` root constant; `EncodeOID(ns, id, text) x509.OID` and `DecodeOID(x509.OID) (ns, id, text, error)`; payload grammar checks per namespace.
     - Learn: what an OID is (a path in a global naming tree), arcs, the `2.25` UUID arc, and how Go's `x509.OID` holds arcs too big for `uint64`.
     - Done when: every row of the design's namespace table round-trips; surrogates, empty symbols and `"1."` as a float are rejected.
-15. **Lowering and raising** (`ast`)
-    - Build: `Lower(program []sexpr.Value) *Tree`, which classifies reserved names into ns 0 and numbers nodes in pre-order; `Raise(*Tree) []sexpr.Value`, the inverse, which also keeps a map from each produced value to its node for error locations.
+15. **ToTree and FromTree** (`ast`)
+    - Build: `ToTree(program []sexpr.Value) *Tree`, which classifies reserved names into ns 0 and numbers nodes in pre-order; `FromTree(*Tree) []sexpr.Value`, the inverse, which also keeps a map from each produced value to its node for error locations.
     - Learn: an AST as a separate structure from source text; why sibling order must be explicit data once you leave a text format.
-    - Done when: `Raise(Lower(p))` equals `p` for every `testdata` program, and `'+` raises to the symbol `+`.
+    - Done when: `FromTree(ToTree(p))` equals `p` for every `testdata` program, and `'+` becomes the symbol `+`.
 16. **Layering** (`ast`)
     - Build: `Layer(*Tree) Layers`, which computes depth n and returns per-depth node sets plus parent→child edges. No padding: certs 2..n assert `anyPolicy`, and the validator copies each unmapped leaf down.
     - Learn: why the policy tree forces every leaf to depth n (RFC 5280 §6.1.3 step d.3 pruning), and how `anyPolicy` (RFC 9618 step d.2) makes the validator do the padding.
@@ -135,7 +135,7 @@     - Build: hand-encode `certificatePolicies` (OID `2.5.29.32`) as a SEQUENCE of PolicyInformation, OIDs only for now, leaving room for qualifiers in Phase 7. Cert 1 lists the top-level nodes; certs 2..n list `anyPolicy` (`2.5.29.32.0`).
     - Learn: policy OIDs as assertions a CA makes about a certificate, which is the hook GOVERNMENT abuses. Read RFC 5280 §4.2.1.4.
     - Done when: `ParseCertificate` returns the OIDs in `Policies`, including one with the 128-bit `GOV` arc.
 22. **The transpiler** (`lisp2gov`, `cmd/government`)
-    - Build: `Compile(src) ([]byte, error)`: read, lower, layer, then one cert per depth (cert 1 with the top-level nodes, the rest with `anyPolicy`), each with the mappings for its depth's outgoing edges; per-depth keys from the seed; PEM with the source as a header comment. Wire up `government compile`.
+    - Build: `Compile(src) ([]byte, error)`: read, build the tree, layer, then one cert per depth (cert 1 with the top-level nodes, the rest with `anyPolicy`), each with the mappings for its depth's outgoing edges; per-depth keys from the seed; PEM with the source as a header comment. Wire up `government compile`.
     - Learn: how a certification path is a chain of issuer→subject links; how a whole program becomes n+1 signed objects.
     - Done when: `government compile examples/sq.lisp -o sq.gov` writes a file that `openssl x509` can print block by block, and compiling twice gives identical bytes. **Milestone: GOVERNMENT programs exist.**
 
@@ -177,7 +177,7 @@     - Build: after processing, every asserted policy is a node at its depth, every mapping produced an edge, and node ids are exactly 1..N (else an orphaned-policy error naming the OID); mappings in cert *n* rejected; `anyPolicy` missing from a cert 2..n, or asserted in cert 1, rejected; `-max-nodes` budget.
     - Learn: how silent pruning makes RFC processing forgiving, and why a language needs it to be unforgiving; policy-processing DoS (CVE-2026-32281 in Go).
     - Done when: deleting one mapping from a valid file gives an error (a gap in the node ids, or a list node left with no elements), not a shorter program, except for the subtree with the highest ids, which the test documents.
 30. **The full pipeline** (`cmd/government`)
-    - Build: `government run`: load → ratify → policy graph → `Unlayer` → `Raise` → evaluate; errors carry the node OID and depth; exit codes 1 and 2. Switch the test runner to compile every `testdata` file and run the resulting bytes.
+    - Build: `government run`: load → ratify → policy graph → `Unlayer` → `FromTree` → evaluate; errors carry the node OID and depth; exit codes 1 and 2. Switch the test runner to compile every `testdata` file and run the resulting bytes.
     - Learn: how the stages compose into a compiler-like pipeline with a trust boundary in the middle.
     - Done when: every `testdata/*.lisp` file passes through the full pipeline with no test changes. **Milestone: GOVERNMENT runs.**
 
@@ -187,7 +187,7 @@ Make programs inspectable. These commands are also the best debugging aids for Phase 7.
 
 31. **tree and decompile** (`cmd/government`, `gov2lisp`)
     - Build: `government tree` prints the policy graph with `GOV` abbreviated, payloads decoded and depth shown; `government decompile` prints Lisp from the decoded AST via the `sexpr` printer. Add the round-trip test: `decompile(compile(src))` equals the normalized source for every `testdata` file.
-    - Learn: decompilation as the inverse of every lowering step; why a round-trip test catches bugs in both directions at once.
+    - Learn: decompilation as the inverse of every compile step; why a round-trip test catches bugs in both directions at once.
     - Done when: `government tree` on the worked example matches the design doc's drawing, and the round-trip test passes.
 32. **verify** (`cmd/government`)
     - Build: `government verify` ratifies and decodes without evaluating, then prints a certificate of ratification: anchor subject, key fingerprint (SHA-256 of the SPKI), window in force (the overlap of all validity periods), depth and node count.
diff --git a/ast/tree.go b/ast/tree.go
new file mode 100644
index 0000000000000000000000000000000000000000..39464736312426ebd1c4e19776400f13d1cd7e64
--- /dev/null
+++ b/ast/tree.go
@@ -0,0 +1,216 @@
+package ast
+
+import (
+	"crypto/x509"
+	"fmt"
+	"strconv"
+
+	"government/sexpr"
+)
+
+// Node is one node of the program tree, and one policy OID in the
+// certificates. Only list nodes have children; an atom's payload is all
+// there is to it.
+type Node struct {
+	ID       uint64
+	NS       Namespace
+	Text     string
+	Children []*Node
+}
+
+// Tree is a whole program. Its forms are the children of the policy tree's
+// anyPolicy root.
+type Tree struct {
+	Forms []*Node
+}
+
+func (n *Node) OID() (x509.OID, error) {
+	return EncodeOID(n.NS, n.ID, n.Text)
+}
+
+func (n *Node) IsList() bool {
+	return n.NS == NSBuiltin && n.Text == ListNode
+}
+
+// ToTree turns a parsed program into a tree, numbering nodes 1, 2, 3, ... in
+// pre-order (a list before its elements, elements left to right). Because
+// ids are given in reading order, sorting siblings by id recovers argument
+// order, which certificates do not preserve.
+func ToTree(program []sexpr.Value) (*Tree, error) {
+	b := &builder{}
+	t := &Tree{}
+	for _, form := range program {
+		n, err := b.node(form)
+		if err != nil {
+			return nil, err
+		}
+		t.Forms = append(t.Forms, n)
+	}
+	return t, nil
+}
+
+type builder struct {
+	lastID uint64
+}
+
+func (b *builder) node(v sexpr.Value) (*Node, error) {
+	b.lastID++
+	n := &Node{ID: b.lastID}
+	switch v := v.(type) {
+	case *sexpr.Pair:
+		n.NS, n.Text = NSBuiltin, ListNode
+		return n, b.elements(n, v)
+	case sexpr.Symbol:
+		n.NS, n.Text = NSSymbol, string(v)
+		if isReservedSymbol(n.Text) {
+			n.NS = NSBuiltin
+		}
+	case sexpr.Int:
+		n.NS, n.Text = NSInt, strconv.FormatInt(int64(v), 10)
+	case sexpr.Float:
+		// The reader keeps values, not spellings, so 3.10 is stored as 3.1.
+		// Print gives the shortest text that reads back as the same float.
+		n.NS, n.Text = NSFloat, sexpr.Print(v)
+	case sexpr.String:
+		n.NS, n.Text = NSString, string(v)
+	case sexpr.Bool:
+		n.NS, n.Text = NSBuiltin, False
+		if v {
+			n.Text = True
+		}
+	case sexpr.Empty:
+		n.NS, n.Text = NSBuiltin, Empty
+	default:
+		return nil, fmt.Errorf("%s cannot be in a program tree", sexpr.Print(v))
+	}
+	if n.NS != NSBuiltin {
+		if err := checkPayload(n.NS, n.Text); err != nil {
+			return nil, err
+		}
+	}
+	return n, nil
+}
+
+// elements makes each element of the list p a child of n. An improper
+// tail becomes a "." child followed by the tail, mirroring the source text
+// (a b . c).
+func (b *builder) elements(n *Node, p *sexpr.Pair) error {
+	var rest sexpr.Value = p
+	for {
+		switch r := rest.(type) {
+		case *sexpr.Pair:
+			child, err := b.node(r.Car)
+			if err != nil {
+				return err
+			}
+			n.Children = append(n.Children, child)
+			rest = r.Cdr
+			continue
+		case sexpr.Empty:
+			return nil
+		}
+		b.lastID++
+		n.Children = append(n.Children, &Node{ID: b.lastID, NS: NSBuiltin, Text: Dot})
+		tail, err := b.node(rest)
+		if err != nil {
+			return err
+		}
+		n.Children = append(n.Children, tail)
+		return nil
+	}
+}
+
+// isReservedSymbol reports whether a source symbol names a builtin. The
+// structural names never come out of the reader as symbols; if one did, it
+// must stay a symbol rather than turn into structure.
+func isReservedSymbol(name string) bool {
+	switch name {
+	case ListNode, Dot, True, False, Empty:
+		return false
+	}
+	return IsBuiltin(name)
+}
+
+// Origins maps each list FromTree produced back to its node, so an evaluation
+// error inside a .gov program can name the OID it came from, the way the
+// reader's positions name a line and column.
+type Origins map[*sexpr.Pair]*Node
+
+// FromTree is ToTree's inverse. A builtin name becomes the plain symbol, so
+// '+ quotes the symbol +, exactly as in source.
+func FromTree(t *Tree) ([]sexpr.Value, Origins, error) {
+	origins := Origins{}
+	forms := make([]sexpr.Value, len(t.Forms))
+	for i, n := range t.Forms {
+		v, err := value(n, origins)
+		if err != nil {
+			return nil, nil, err
+		}
+		forms[i] = v
+	}
+	return forms, origins, nil
+}
+
+func value(n *Node, origins Origins) (sexpr.Value, error) {
+	if n.IsList() {
+		return listValue(n, origins)
+	}
+	if len(n.Children) > 0 {
+		return nil, fmt.Errorf("node %d: %s %q cannot have children", n.ID, n.NS, n.Text)
+	}
+	switch n.NS {
+	case NSBuiltin:
+		switch n.Text {
+		case True:
+			return sexpr.T, nil
+		case False:
+			return sexpr.F, nil
+		case Empty:
+			return sexpr.Nil, nil
+		case Dot:
+			return nil, fmt.Errorf("node %d: . is only allowed second-to-last in a list", n.ID)
+		}
+		return sexpr.Symbol(n.Text), nil
+	case NSSymbol:
+		return sexpr.Symbol(n.Text), nil
+	case NSInt, NSFloat:
+		v, _, err := sexpr.ParseNumber(n.Text)
+		if err != nil {
+			return nil, fmt.Errorf("node %d: %w", n.ID, err)
+		}
+		return v, nil
+	case NSString:
+		return sexpr.String(n.Text), nil
+	}
+	return nil, fmt.Errorf("node %d: unknown %s", n.ID, n.NS)
+}
+
+func listValue(n *Node, origins Origins) (sexpr.Value, error) {
+	if len(n.Children) == 0 {
+		return nil, fmt.Errorf("node %d: a list node needs at least one element; () is its own builtin", n.ID)
+	}
+	children := n.Children
+	var tail sexpr.Value = sexpr.Nil
+	if k := len(children); k >= 3 && isDot(children[k-2]) {
+		t, err := value(children[k-1], origins)
+		if err != nil {
+			return nil, err
+		}
+		tail, children = t, children[:k-2]
+	}
+	items := make([]sexpr.Value, len(children))
+	for i, c := range children {
+		v, err := value(c, origins)
+		if err != nil {
+			return nil, err
+		}
+		items[i] = v
+	}
+	list := sexpr.ListWithTail(tail, items...).(*sexpr.Pair)
+	origins[list] = n
+	return list, nil
+}
+
+func isDot(n *Node) bool {
+	return n.NS == NSBuiltin && n.Text == Dot
+}
diff --git a/ast/tree_test.go b/ast/tree_test.go
new file mode 100644
index 0000000000000000000000000000000000000000..05c43b83c2dc499998945ebee1fb060d9f2a866d
--- /dev/null
+++ b/ast/tree_test.go
@@ -0,0 +1,240 @@
+package ast
+
+import (
+	"os"
+	"path/filepath"
+	"reflect"
+	"strconv"
+	"strings"
+	"testing"
+
+	"government/sexpr"
+)
+
+func mustRead(t *testing.T, src string) []sexpr.Value {
+	t.Helper()
+	forms, err := sexpr.ReadAll(src)
+	if err != nil {
+		t.Fatalf("ReadAll(%q): %v", src, err)
+	}
+	return forms
+}
+
+// outline prints a tree as "text#id" atoms and nested lists, so tests can
+// state the expected numbering compactly.
+func outline(nodes []*Node) string {
+	parts := make([]string, len(nodes))
+	for i, n := range nodes {
+		label := n.Text + "#" + itoa(n.ID)
+		if n.IsList() {
+			label = "(#" + itoa(n.ID) + " " + outline(n.Children) + ")"
+		}
+		parts[i] = label
+	}
+	return strings.Join(parts, " ")
+}
+
+func itoa(n uint64) string { return strconv.FormatUint(n, 10) }
+
+func TestToTreeNumbersInPreOrder(t *testing.T) {
+	tests := []struct {
+		src, want string
+	}{
+		// The design's worked example.
+		{"(+ 1 (* 2 3))", "(#1 +#2 1#3 (#4 *#5 2#6 3#7))"},
+		{"1 2", "1#1 2#2"},
+		{"'(a . b)", "(#1 quote#2 (#3 a#4 .#5 b#6))"},
+		{"(f () T F)", "(#1 f#2 ()#3 T#4 F#5)"},
+	}
+	for _, tt := range tests {
+		tree, err := ToTree(mustRead(t, tt.src))
+		if err != nil {
+			t.Errorf("ToTree(%s): %v", tt.src, err)
+			continue
+		}
+		if got := outline(tree.Forms); got != tt.want {
+			t.Errorf("ToTree(%s) = %s, want %s", tt.src, got, tt.want)
+		}
+	}
+}
+
+func TestToTreeNamespaces(t *testing.T) {
+	tree, err := ToTree(mustRead(t, `(define x "s") foo 1.50 -3 '+`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	f := tree.Forms
+	quoted := f[4].Children[1]
+	checks := []struct {
+		n    *Node
+		ns   Namespace
+		text string
+	}{
+		{f[0].Children[0], NSBuiltin, "define"},
+		{f[0].Children[1], NSSymbol, "x"},
+		{f[0].Children[2], NSString, "s"},
+		{f[1], NSSymbol, "foo"},
+		{f[2], NSFloat, "1.5"},
+		{f[3], NSInt, "-3"},
+		// Quoting does not change a name's namespace; it only changes how
+		// the evaluator treats the node.
+		{quoted, NSBuiltin, "+"},
+	}
+	for _, c := range checks {
+		if c.n.NS != c.ns || c.n.Text != c.text {
+			t.Errorf("node %d = %s %q, want %s %q", c.n.ID, c.n.NS, c.n.Text, c.ns, c.text)
+		}
+	}
+}
+
+func TestQuotedBuiltinBecomesSymbol(t *testing.T) {
+	tree, err := ToTree(mustRead(t, "'+"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	forms, _, err := FromTree(tree)
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := sexpr.List(sexpr.Symbol("quote"), sexpr.Symbol("+"))
+	if !reflect.DeepEqual(forms[0], want) {
+		t.Errorf("FromTree = %s, want %s", sexpr.Print(forms[0]), sexpr.Print(want))
+	}
+}
+
+func TestRoundTripTestdata(t *testing.T) {
+	files, err := filepath.Glob("../testdata/*.lisp")
+	if err != nil || len(files) == 0 {
+		t.Fatalf("no testdata: %v", err)
+	}
+	for _, file := range files {
+		src, err := os.ReadFile(file)
+		if err != nil {
+			t.Fatal(err)
+		}
+		program := mustRead(t, string(src))
+		tree, err := ToTree(program)
+		if err != nil {
+			t.Errorf("%s: ToTree: %v", file, err)
+			continue
+		}
+		got, _, err := FromTree(tree)
+		if err != nil {
+			t.Errorf("%s: FromTree: %v", file, err)
+			continue
+		}
+		if !reflect.DeepEqual(got, program) {
+			t.Errorf("%s: FromTree(ToTree(p)) differs from p", file)
+		}
+	}
+}
+
+func TestRoundTripEdgeCases(t *testing.T) {
+	for _, src := range []string{
+		`'(1 2 . 3)`,
+		`'((a . b) . (c . ()))`,
+		`"" "😀" "\u{0}"`,
+		`1.0e+21 -0.0 1.0e-7 0.1`,
+		`-9223372036854775808`,
+		`'(quote lambda T F ())`,
+	} {
+		program := mustRead(t, src)
+		tree, err := ToTree(program)
+		if err != nil {
+			t.Errorf("ToTree(%s): %v", src, err)
+			continue
+		}
+		got, _, err := FromTree(tree)
+		if err != nil {
+			t.Errorf("FromTree(ToTree(%s)): %v", src, err)
+			continue
+		}
+		if !reflect.DeepEqual(got, program) {
+			t.Errorf("FromTree(ToTree(%s)) = %v", src, got)
+		}
+	}
+}
+
+func TestEveryTreeNodeEncodes(t *testing.T) {
+	tree, err := ToTree(mustRead(t, `(define (f) '(1 . "x")) 2.5 T`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	var walk func([]*Node)
+	walk = func(nodes []*Node) {
+		for _, n := range nodes {
+			if _, err := n.OID(); err != nil {
+				t.Errorf("node %d: %v", n.ID, err)
+			}
+			walk(n.Children)
+		}
+	}
+	walk(tree.Forms)
+}
+
+func TestOrigins(t *testing.T) {
+	tree, err := ToTree(mustRead(t, "(+ 1 (* 2 3))"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	forms, origins, err := FromTree(tree)
+	if err != nil {
+		t.Fatal(err)
+	}
+	outer := forms[0].(*sexpr.Pair)
+	inner := outer.Cdr.(*sexpr.Pair).Cdr.(*sexpr.Pair).Car.(*sexpr.Pair)
+	if origins[outer].ID != 1 || origins[inner].ID != 4 {
+		t.Errorf("origins: outer #%d, inner #%d; want #1, #4", origins[outer].ID, origins[inner].ID)
+	}
+}
+
+func TestFromTreeRejects(t *testing.T) {
+	atom := func(id uint64, ns Namespace, text string) *Node { return &Node{ID: id, NS: ns, Text: text} }
+	list := func(id uint64, children ...*Node) *Node {
+		return &Node{ID: id, NS: NSBuiltin, Text: ListNode, Children: children}
+	}
+	tests := []struct {
+		name string
+		form *Node
+		want string
+	}{
+		{"empty list node", list(1), "at least one element"},
+		{"atom with a child", &Node{ID: 1, NS: NSInt, Text: "1", Children: []*Node{atom(2, NSInt, "2")}}, "cannot have children"},
+		{"dot at the start", list(1, atom(2, NSBuiltin, Dot), atom(3, NSSymbol, "a")), "second-to-last"},
+		{"dot at the end", list(1, atom(2, NSSymbol, "a"), atom(3, NSBuiltin, Dot)), "second-to-last"},
+		{"two tails", list(1, atom(2, NSSymbol, "a"), atom(3, NSBuiltin, Dot), atom(4, NSSymbol, "b"), atom(5, NSSymbol, "c")), "second-to-last"},
+		{"bad integer", atom(1, NSInt, "007"), "malformed"},
+	}
+	for _, tt := range tests {
+		_, _, err := FromTree(&Tree{Forms: []*Node{tt.form}})
+		if err == nil || !strings.Contains(err.Error(), tt.want) {
+			t.Errorf("%s: error = %v, want %q", tt.name, err, tt.want)
+		}
+	}
+}
+
+func TestToTreeRejects(t *testing.T) {
+	tests := []struct {
+		name string
+		form sexpr.Value
+		want string
+	}{
+		// The reader never makes these; ToTree must not turn them into
+		// structure if something else does.
+		{"symbol named .", sexpr.Symbol(Dot), "builtin name"},
+		{"symbol named (...)", sexpr.Symbol(ListNode), "builtin name"},
+		{"empty symbol", sexpr.Symbol(""), "empty symbol"},
+		{"non-literal value", fakeFunction{}, "cannot be in a program tree"},
+	}
+	for _, tt := range tests {
+		_, err := ToTree([]sexpr.Value{tt.form})
+		if err == nil || !strings.Contains(err.Error(), tt.want) {
+			t.Errorf("%s: error = %v, want %q", tt.name, err, tt.want)
+		}
+	}
+}
+
+type fakeFunction struct{ sexpr.FunctionBase }
+
+func (fakeFunction) Builtin() bool { return false }
+func (fakeFunction) Name() string  { return "" }