Watch
1
0
Fork
You've already forked packages
0

packages/ at markup-kit dd1e6eb5270e

A snapshot of packages/ in the Weft repository; the official packages are published from these commits.

Markup-Kit-Commit: dd1e6eb5270e62e0c8b6c202b3d01f0da6e00ddc
This commit is contained in:
Weft packages 2026-11-19 20:43:26 +00:00
commit 306b143104
72 changed files with 10915 additions and 0 deletions

102
markdown/README.md Normal file
View file

@ -0,0 +1,102 @@
# @weft/markdown
CommonMark and GitHub Flavored Markdown for Weft: a syntax tree, HTML, and a sanitiser. It is written in Weft, needs no
capabilities (it does no I/O), and is safe for untrusted input by default.
```sh
weft add weft/markdown
```
```weft
import { parse, render, sanitize, toHtml, toText, visit } from "@weft/markdown"
toHtml("# Hello\n\n**bold** <script>alert(1)</script>")
// <h1>Hello</h1>
// <p><strong>bold</strong> &lt;script&gt;alert(1)&lt;/script&gt;</p>
```
## What it gives you
| | |
|---|---|
| `toHtml(source, options?)` | Markdown to HTML |
| `parse(source, options?)` | the syntax tree (`Document`, typed in `@weft/markdown/ast`) |
| `render(doc, options?)` | a tree to HTML, for a tree you have changed |
| `toText(doc)` | plain text, for previews, search and the text part of an email |
| `visit(doc, f)` | every inline of a document, to collect links or images |
| `sanitize(html, policy?)` | the allowlist sanitiser on its own, for HTML from anywhere |
GFM is on by default: tables, task lists, strikethrough and bare links (`www.`, `https://`, email addresses). `gfm: false`
gives plain CommonMark.
## Safe by default
- **Raw HTML is shown as text.** `html: "sanitize"` keeps what an allowlist permits; `html: "trust"` passes it through and
is for Markdown you wrote yourself.
- **Links and images only use `http`, `https`, `mailto` and `tel`**, or are relative (`schemes` changes the list). Images
may also be `data:image/` PNG, GIF, JPEG or WebP, never SVG. A link that isn't allowed renders as its text.
- **The sanitiser is an allowlist over a tokenizer**, and what it writes is well formed with every text escaped. Scripts,
styles, frames, forms, SVG, MathML, event handlers and unsafe URLs are removed. The only `<input>` kept is a disabled
checkbox.
- **Hostile documents can't take the server down.** Quotes, lists and inlines stop nesting at 64 levels, the text a link
reference expands to is bounded, and every input in `tests/hostile.weft` takes time proportional to its size.
```weft
toHtml("a <kbd>b</kbd> <script>c()</script>", { html: "sanitize" }) // <p>a <kbd>b</kbd> </p>
sanitize("<a href=\"javascript:x()\" onclick=\"y()\">hi</a>") // <a>hi</a>
```
## Options
```weft
toHtml(source, {
headingOffset: 1, // `# Title` renders as <h2>
linkTarget: "_blank", // adds rel="noopener noreferrer"
rewriteUrl: (url, kind) => url.startsWith("upload:") ? cdn(url.slice(7)) : null,
})
```
`rewriteUrl` runs before the safety check, so an app's own scheme (`upload:<id>`) can resolve to a real address, and one it
can't resolve is dropped rather than rendered.
## Conformance
`weft run tests/spec.weft` runs the specs' examples:
| | |
|---|---|
| CommonMark 0.31.2 | 652 of 652 |
| GitHub Flavored Markdown | 660 of 669, and the other 9 are below |
The 9 are examples where the two specs give different HTML for the same text: the GFM spec was written against CommonMark
0.29, and its renderer folds nested `<strong>`, which CommonMark 0.31.2's own examples keep. This package follows CommonMark
there, and the runner checks each such example against CommonMark's expected output. GFM's `tagfilter` example isn't run:
`html: "escape"` and `"sanitize"` already remove those tags, and `"trust"` means what it says.
## Speed
Against markdown-it 15.0.3 on Node 24, rendering the CommonMark spec's examples joined and repeated (`scripts/bench.mjs`;
one cold run each, in a 6 GB container, so markdown-it's JIT is warming up over the small sizes):
| document | Weft VM | markdown-it (JIT) | markdown-it (`--jitless`) |
|---|---|---|---|
| 16 KB | 52 ms | 44 ms | 47 ms |
| 157 KB | 112 ms | 105 ms | 242 ms |
| 787 KB | 1.2 s | 0.54 s | 1.6 s |
| 3.1 MB | 5.3 s | 1.5 s | 5.7 s |
## Limits
- A turn's default limits (32 MB of allocation, and 250 ms when serving) fit documents up to about 150 KB. Above that, raise
`WEFT_MEMORY_MB` and `WEFT_DEADLINE_MS`, or render outside a request.
## Development
```sh
weft test tests/markdown.test.weft # the API and its safety
weft run tests/spec.weft # the specs' examples (-v prints each failure)
node scripts/hostile.mjs # hostile input: nothing stops, nothing is quadratic
node scripts/bench.mjs commonmark.json # beside markdown-it (MARKDOWN_IT=<its folder>)
node scripts/gen-spec.mjs # regenerates tests/cases.weft from the specs
node scripts/gen-tables.mjs # regenerates lib/entities.weft and lib/unicode.weft
```

83
markdown/ast.weft Normal file
View file

@ -0,0 +1,83 @@
// The Markdown syntax tree and the options of every function in `@weft/markdown`.
/// A parsed document: its blocks, in order.
export type Document = { children: Block[] }
/// A table column's alignment, `null` when its delimiter row doesn't say.
export type Align = "left" | "center" | "right" | null
/// A block: what a document, a quote or a list item is made of.
///
/// - `CodeBlock`: `info` is the text after the opening fence (`""` for an indented block).
/// - `HtmlBlock`: raw HTML as written, rendered according to `Options.html`.
/// - `BulletList`, `OrderedList`: a `tight` list has no blank lines between its items; `start` is the first number.
/// - `Table`: a GFM table, with one cell per column in every row.
export type Block =
| Paragraph { children: Inline[] }
| Heading { level: float, children: Inline[] }
| ThematicBreak
| CodeBlock { info: string, code: string }
| HtmlBlock { html: string }
| Blockquote { children: Block[] }
| BulletList { tight: boolean, items: ListItem[] }
| OrderedList { tight: boolean, start: float, items: ListItem[] }
| Table { align: Align[], head: TableCell[], rows: TableCell[][] }
/// A list item. `checked` is `null` unless the item is a GFM task (`- [x] done`).
export type ListItem = { children: Block[], checked: boolean? }
/// A table cell.
export type TableCell = { children: Inline[] }
/// An inline: what paragraphs, headings and cells are made of. `HtmlInline` is raw HTML as written, rendered according
/// to `Options.html`.
export type Inline =
| Str { text: string }
| SoftBreak
| HardBreak
| CodeSpan { code: string }
| Emphasis { children: Inline[] }
| Strong { children: Inline[] }
| Strikethrough { children: Inline[] }
| Link { url: string, title: string?, children: Inline[] }
| Picture { url: string, title: string?, alt: Inline[] }
| HtmlInline { html: string }
/// What happens to raw HTML in the source:
/// - `"escape"` (the default) shows it as text;
/// - `"sanitize"` keeps what the `policy` allows and removes the rest;
/// - `"trust"` passes it through. Use it only for Markdown you wrote yourself.
export type HtmlMode = "escape" | "sanitize" | "trust"
/// What `sanitize` keeps. Anything left out takes the default: common formatting tags, no scripts, styles, event
/// handlers or forms.
export type Policy = {
/// Lowercase tag names to keep. A tag not listed is removed and its content kept.
tags?: string[],
/// Attributes to keep, per lowercase tag name.
attributes?: Map<string, string[]>,
/// URL schemes `href` and `src` may use. Relative URLs are always allowed.
schemes?: string[],
}
/// Whether a URL is a link's or an image's.
export type UrlKind = "link" | "image"
/// Options for parsing and rendering. Every field is optional.
export type Options = {
/// GitHub Flavored Markdown: tables, task lists, strikethrough and bare autolinks. On by default.
gfm?: boolean,
/// What happens to raw HTML. `"escape"` by default.
html?: HtmlMode,
/// What `html: "sanitize"` keeps.
policy?: Policy,
/// URL schemes links and images may use: `http`, `https`, `mailto` and `tel` by default. Relative URLs are always
/// allowed, and images may also be `data:image/` PNG, GIF, JPEG or WebP. A link to anything else renders as its text.
schemes?: string[],
/// Added to every heading's level, up to 6: `1` renders `# Title` as `<h2>`.
headingOffset?: float,
/// `target` for every link. `"_blank"` also adds `rel="noopener noreferrer"`.
linkTarget?: string,
/// Rewrites a URL before it is checked and rendered, as an app resolves `upload:<id>` to a real address.
/// Returning `null` keeps the URL as written.
rewriteUrl?: (url: string, kind: UrlKind) => string?,
}

12
markdown/bench/bench.weft Normal file
View file

@ -0,0 +1,12 @@
// `weft run bench/bench.weft <copies>`: renders the CommonMark spec's examples, joined and repeated, and prints the size.
// scripts/bench.mjs times it from outside the VM, beside markdown-it on the same text.
import { args } from "std/process"
import { log } from "std/log"
import { toHtml } from "../markdown"
import { COMMONMARK } from "../tests/cases"
let copies = try int(args().at(0) ?? "1") else 1
let text = COMMONMARK.map(c => c.markdown).join("\n\n").repeat(copies)
if copies > 0 {
log(toHtml(text).length)
}

1145
markdown/lib/blocks.weft Normal file

File diff suppressed because it is too large Load diff

242
markdown/lib/chars.weft Normal file
View file

@ -0,0 +1,242 @@
// Character classes and the text transformations CommonMark defines: escapes, entities, URLs.
import { entity } from "./entities"
import { PUNCTUATION } from "./unicode"
/// The code unit at `i`, or -1 past the end.
export fn unit(s: string, i: int): int {
return s.codeUnitAt(i) ?? -1
}
export fn isSpaceOrTab(c: float): boolean {
return c == 32 || c == 9
}
export fn isDigit(c: float): boolean {
return c >= 48 && c <= 57
}
export fn isAlpha(c: float): boolean {
return (c >= 65 && c <= 90) || (c >= 97 && c <= 122)
}
export fn isAlnum(c: float): boolean {
return isAlpha(c) || isDigit(c)
}
export fn isHex(c: float): boolean {
return isDigit(c) || (c >= 65 && c <= 70) || (c >= 97 && c <= 102)
}
/// ASCII punctuation: what a backslash can escape.
export fn isAsciiPunct(c: float): boolean {
return (c >= 33 && c <= 47) ||
(c >= 58 && c <= 64) ||
(c >= 91 && c <= 96) ||
(c >= 123 && c <= 126)
}
/// Unicode whitespace: the `Zs` category, tab, line feed, form feed and carriage return.
export fn isUnicodeSpace(c: float): boolean {
if c < 128 {
return c == 32 || c == 9 || c == 10 || c == 12 || c == 13
}
return c == 0xa0 ||
c == 0x1680 ||
(c >= 0x2000 && c <= 0x200a) ||
c == 0x202f ||
c == 0x205f ||
c == 0x3000
}
/// Unicode punctuation: the `P` and `S` categories.
export fn isUnicodePunct(c: float): boolean {
if c < 128 {
return isAsciiPunct(c)
}
let mut lo = 0
let mut hi = PUNCTUATION.length.div(2) - 1
while lo <= hi {
let mid = (lo + hi) >> 1
if c < PUNCTUATION[mid * 2] {
hi = mid - 1
} else if c > PUNCTUATION[mid * 2 + 1] {
lo = mid + 1
} else {
return true
}
}
return false
}
/// Whether `s` holds only spaces and tabs (or nothing).
export fn isBlank(s: string): boolean {
for i of 0..s.length {
let c = s.codeUnitAt(i) ?? 32
if c != 32 && c != 9 && c != 10 && c != 13 {
return false
}
}
return true
}
/// The code point ending just before `pos`, or a line feed at the start of `s`.
export fn codePointBefore(s: string, pos: int): int {
if pos <= 0 {
return 10
}
let low = s.codeUnitAt(pos - 1) ?? 10
if low >= 0xdc00 && low <= 0xdfff && pos >= 2 {
let high = s.codeUnitAt(pos - 2) ?? 0
if high >= 0xd800 && high <= 0xdbff {
return (high - 0xd800) * 0x400 + (low - 0xdc00) + 0x10000
}
}
return low
}
/// The character reference starting at the `&` at `pos`: its text and where it ends, or `null` when there is none.
export fn scanEntity(s: string, pos: int): [string, int]? {
let mut i = pos + 1
if s.codeUnitAt(i) == 35 {
i += 1
let c = s.codeUnitAt(i) ?? -1
let hex = c == 120 || c == 88
if hex {
i += 1
}
let start = i
while i - start < (hex ? 6 : 7) &&
(hex ? isHex(s.codeUnitAt(i) ?? -1) : isDigit(s.codeUnitAt(i) ?? -1)) {
i += 1
}
if i == start || s.codeUnitAt(i) != 59 {
return null
}
let n = try int(s.slice(start, i), hex ? 16 : 10) else 0
let valid = n != 0 && n <= 0x10ffff && !(n >= 0xd800 && n <= 0xdfff)
return [String.fromCodePoint(valid ? n : 0xfffd), i + 1]
}
let start = i
while i - start < 32 && isAlnum(s.codeUnitAt(i) ?? -1) {
i += 1
}
if i == start || s.codeUnitAt(i) != 59 {
return null
}
let text = entity(s.slice(start, i))
if text == null {
return null
}
return [text, i + 1]
}
/// `s` with its backslash escapes and character references replaced by what they stand for.
export fn unescape(s: string): string {
if !s.includes("\\") && !s.includes("&") {
return s
}
let mut parts: string[] = []
let mut from = 0
let mut i = 0
while i < s.length {
let c = s.codeUnitAt(i) ?? 0
if c == 92 && isAsciiPunct(s.codeUnitAt(i + 1) ?? 0) {
parts.push(s.slice(from, i))
from = i + 1
i += 2
} else if c == 38 {
let found = scanEntity(s, i)
if found != null {
parts.push(s.slice(from, i))
parts.push(found[0])
i = found[1]
from = i
} else {
i += 1
}
} else {
i += 1
}
}
parts.push(s.slice(from))
return parts.join("")
}
/// `s` escaped for HTML text and attribute values.
export fn escapeHtml(s: string): string {
let mut i = 0
while i < s.length {
let c = s.codeUnitAt(i) ?? 0
if c == 38 || c == 60 || c == 62 || c == 34 {
break
}
i += 1
}
if i == s.length {
return s
}
let mut parts: string[] = [s.slice(0, i)]
let mut from = i
while i < s.length {
let c = s.codeUnitAt(i) ?? 0
let rep = c == 38 ? "&amp;" : c == 60 ? "&lt;" : c == 62 ? "&gt;" : c == 34 ? "&quot;" : ""
if rep != "" {
parts.push(s.slice(from, i))
parts.push(rep)
from = i + 1
}
i += 1
}
parts.push(s.slice(from))
return parts.join("")
}
fn keptInUrl(c: int): boolean {
// Letters, digits and what URLs use as syntax; everything else is percent-encoded.
return isAlnum(c) || (c < 128 && ";/?:@&=+$,-_.!~*'()#".includes(String.fromCharCode(c)))
}
/// `url` percent-encoded for an `href`: what is already encoded stays, and a lone surrogate becomes U+FFFD.
export fn normalizeUrl(url: string): string {
let mut parts: string[] = []
let mut from = 0
let mut i = 0
while i < url.length {
let c = url.codeUnitAt(i) ?? 0
if c == 37 && isHex(url.codeUnitAt(i + 1) ?? -1) && isHex(url.codeUnitAt(i + 2) ?? -1) {
i += 3
} else if keptInUrl(c) {
i += 1
} else {
parts.push(url.slice(from, i))
let point = url.codePointAt(i) ?? 0xfffd
let width = point > 0xffff ? 2 : 1
let safe = point >= 0xd800 && point <= 0xdfff ? 0xfffd : point
for b of Bytes.fromUtf8(String.fromCodePoint(safe)).toList() {
parts.push("%" + str(b, { radix: 16 }).toUpperCase().padStart(2, "0"))
}
i += width
from = i
}
}
parts.push(url.slice(from))
return parts.join("")
}
/// A link label as references are matched: case-folded, with whitespace collapsed.
export fn normalizeLabel(label: string): string {
let mut out: string[] = []
let mut space = false
for ch of label.trim() {
if ch == " " || ch == "\t" || ch == "\n" || ch == "\r" {
space = true
} else {
if space {
out.push(" ")
space = false
}
out.push(ch)
}
}
return out.join("").toLowerCase().toUpperCase()
}

File diff suppressed because one or more lines are too long

1397
markdown/lib/inlines.weft Normal file

File diff suppressed because it is too large Load diff

255
markdown/lib/render.weft Normal file
View file

@ -0,0 +1,255 @@
// The HTML renderer: the output CommonMark's and GFM's spec tests expect.
import type {
Align,
Block,
Document,
HtmlMode,
Inline,
ListItem,
Options,
TableCell,
UrlKind,
} from "../ast"
import { escapeHtml } from "./chars"
import { DEFAULT_SCHEMES, isSafeUrl, sanitize } from "./sanitize"
type Out = {
parts: string[],
atLineStart: boolean,
mode: HtmlMode,
schemes: string[],
offset: float,
target: string?,
}
fn put(mut o: Out, text: string) {
if text != "" {
o.parts.push(text)
o.atLineStart = text.endsWith("\n")
}
}
fn targetOf(o: Out): string? {
return o.target
}
fn isEscaping(o: Out): boolean {
return o.mode == "escape"
}
fn isAtLineStart(o: Out): boolean {
return o.atLineStart
}
fn cr(mut o: Out) {
if !isAtLineStart(o) {
put(&o, "\n")
}
}
fn rawHtml(o: Out, html: string): string {
return o.mode == "escape" ? escapeHtml(html) : html
}
fn plainText(inlines: Inline[]): string {
let mut out: string[] = []
for n of inlines {
match n {
Str { text } => out.push(text),
SoftBreak | HardBreak => out.push(" "),
CodeSpan { code } => out.push(code),
Emphasis { children } | Strong { children } | Strikethrough { children } => out.push(
plainText(children),
),
Link { children } => out.push(plainText(children)),
Picture { alt } => out.push(plainText(alt)),
HtmlInline { html } => out.push(html),
}
}
return out.join("")
}
fn wrap(mut o: Out, tag: string, children: Inline[], options: Options): void {
put(&o, `<${tag}>`)
renderInlines(&o, children, options)
put(&o, `</${tag}>`)
}
/// The URL to render, or `null` when it isn't safe to.
fn finalUrl(url: string, kind: UrlKind, o: Out, options: Options): string? {
let rewrite = options.rewriteUrl
let rewritten = rewrite != null ? rewrite(url, kind) ?? url : url
return o.mode == "trust" || isSafeUrl(rewritten, o.schemes, kind == "image") ? rewritten : null
}
fn renderInlines(mut o: Out, inlines: Inline[], options: Options): void {
for n of inlines {
match n {
Str { text } => put(&o, escapeHtml(text)),
SoftBreak => put(&o, "\n"),
HardBreak => put(&o, "<br />\n"),
CodeSpan { code } => put(&o, `<code>${escapeHtml(code)}</code>`),
Emphasis { children } => wrap(&o, "em", children, options),
Strong { children } => wrap(&o, "strong", children, options),
Strikethrough { children } => wrap(&o, "del", children, options),
HtmlInline { html } => {
let raw = rawHtml(o, html)
put(&o, raw)
},
Link { url, title, children } => {
let href = finalUrl(url, "link", o, options)
if href == null {
renderInlines(&o, children, options)
} else {
let titled = title != null && title != "" ? ` title="${escapeHtml(title)}"` : ""
let linkTarget = targetOf(o)
let target = linkTarget != null ? ` target="${escapeHtml(linkTarget)}"` : ""
let rel = linkTarget == "_blank" ? " rel=\"noopener noreferrer\"" : ""
put(&o, `<a href="${escapeHtml(href)}"${titled}${target}${rel}>`)
renderInlines(&o, children, options)
put(&o, "</a>")
}
},
Picture { url, title, alt } => {
let src = finalUrl(url, "image", o, options)
if src == null {
put(&o, escapeHtml(plainText(alt)))
} else {
let titled = title != null && title != "" ? ` title="${escapeHtml(title)}"` : ""
put(&o, `<img src="${escapeHtml(src)}" alt="${escapeHtml(plainText(alt))}"${titled} />`)
}
},
}
}
}
fn renderItem(mut o: Out, item: ListItem, tight: boolean, options: Options): void {
put(&o, "<li>")
if item.checked != null {
put(
&o,
item.checked
? "<input checked=\"\" disabled=\"\" type=\"checkbox\"> "
: "<input disabled=\"\" type=\"checkbox\"> ",
)
}
renderBlocks(&o, item.children, tight, options)
put(&o, "</li>")
cr(&o)
}
fn renderCells(mut o: Out, tag: string, cells: TableCell[], align: Align[], options: Options) {
put(&o, "<tr>\n")
for [i, cell] of cells.entries() {
let a = align.at(i) ?? null
put(&o, a != null ? `<${tag} align="${a}">` : `<${tag}>`)
renderInlines(&o, cell.children, options)
put(&o, `</${tag}>\n`)
}
put(&o, "</tr>\n")
}
fn renderBlocks(mut o: Out, blocks: Block[], tight: boolean, options: Options): void {
for block of blocks {
match block {
Paragraph { children } => {
if tight {
renderInlines(&o, children, options)
} else {
cr(&o)
wrap(&o, "p", children, options)
cr(&o)
}
},
Heading { level, children } => {
let shifted = level + o.offset
let shown = shifted > 6 ? 6 : shifted < 1 ? 1 : shifted
cr(&o)
wrap(&o, `h${shown}`, children, options)
cr(&o)
},
ThematicBreak => {
cr(&o)
put(&o, "<hr />")
cr(&o)
},
CodeBlock { info, code } => {
cr(&o)
let language = info.trim().split(" ")[0].split("\t")[0]
put(
&o,
language != "" ? `<pre><code class="language-${escapeHtml(language)}">` : "<pre><code>",
)
put(&o, escapeHtml(code))
put(&o, "</code></pre>")
cr(&o)
},
HtmlBlock { html } => {
cr(&o)
if isEscaping(o) {
put(&o, `<p>${escapeHtml(html)}</p>`)
} else {
put(&o, html)
}
cr(&o)
},
Blockquote { children } => {
cr(&o)
put(&o, "<blockquote>\n")
renderBlocks(&o, children, false, options)
cr(&o)
put(&o, "</blockquote>")
cr(&o)
},
BulletList { tight: itemsTight, items } => {
cr(&o)
put(&o, "<ul>\n")
for item of items {
renderItem(&o, item, itemsTight, options)
}
put(&o, "</ul>")
cr(&o)
},
OrderedList { tight: itemsTight, start, items } => {
cr(&o)
put(&o, start != 1 ? `<ol start="${start}">\n` : "<ol>\n")
for item of items {
renderItem(&o, item, itemsTight, options)
}
put(&o, "</ol>")
cr(&o)
},
Table { align, head, rows } => {
cr(&o)
put(&o, "<table>\n<thead>\n")
renderCells(&o, "th", head, align, options)
put(&o, "</thead>\n")
if rows.length > 0 {
put(&o, "<tbody>\n")
for row of rows {
renderCells(&o, "td", row, align, options)
}
put(&o, "</tbody>\n")
}
put(&o, "</table>")
cr(&o)
},
}
}
}
/// Renders a document to HTML.
export fn renderDocument(doc: Document, options: Options): string {
let mode = options.html ?? "escape"
let mut o: Out = {
parts: [],
atLineStart: true,
mode,
schemes: options.schemes ?? DEFAULT_SCHEMES,
offset: options.headingOffset ?? 0,
target: options.linkTarget,
}
renderBlocks(&o, doc.children, false, options)
let html = o.parts.join("")
return mode == "sanitize" ? sanitize(html, options.policy) : html
}

403
markdown/lib/sanitize.weft Normal file
View file

@ -0,0 +1,403 @@
// The HTML sanitiser: an allowlist over a tokenizer. What comes out is well formed and holds only allowed tags and
// attributes, every text escaped, so nothing a browser parses differently can bring a removed element back.
import type { Policy } from "../ast"
import { escapeHtml, isAlpha, scanEntity } from "./chars"
let DEFAULT_TAGS = [
"a",
"abbr",
"b",
"blockquote",
"br",
"code",
"dd",
"del",
"details",
"div",
"dl",
"dt",
"em",
"figcaption",
"figure",
"h1",
"h2",
"h3",
"h4",
"h5",
"h6",
"hr",
"i",
"img",
"input",
"ins",
"kbd",
"li",
"mark",
"ol",
"p",
"pre",
"q",
"s",
"samp",
"small",
"span",
"strong",
"sub",
"summary",
"sup",
"table",
"tbody",
"td",
"tfoot",
"th",
"thead",
"tr",
"u",
"ul",
"var",
]
let DEFAULT_ATTRIBUTES = Map<string, string[]>(
[
["a", ["href", "title", "target"]],
["img", ["src", "alt", "title", "width", "height"]],
["ol", ["start"]],
["code", ["class"]],
["td", ["align", "colspan", "rowspan"]],
["th", ["align", "colspan", "rowspan", "scope"]],
["input", ["type", "checked", "disabled"]],
["details", ["open"]],
],
)
/// URL schemes allowed unless the options say otherwise.
export let DEFAULT_SCHEMES = ["http", "https", "mailto", "tel"]
let VOID_TAGS = Set(
[
"br",
"hr",
"img",
"input",
"wbr",
"area",
"base",
"col",
"embed",
"link",
"meta",
"param",
"source",
"track",
],
)
/// Elements removed with everything inside them.
let DROPPED = Set(
[
"script",
"style",
"template",
"iframe",
"object",
"embed",
"applet",
"noscript",
"noembed",
"noframes",
"title",
"textarea",
"xmp",
"plaintext",
"svg",
"math",
"select",
"frameset",
"head",
],
)
fn isDataImage(url: string): boolean {
let lower = url.slice(0, 16).toLowerCase()
return lower.startsWith("data:image/png") ||
lower.startsWith("data:image/gif") ||
lower.startsWith("data:image/jpeg") ||
lower.startsWith("data:image/jpg") ||
lower.startsWith("data:image/webp")
}
/// Whether a URL is relative or uses one of `schemes`. Browsers ignore tabs, line breaks and leading control characters
/// in a scheme, so they are ignored here too.
export fn isSafeUrl(url: string, schemes: string[], image: boolean): boolean {
let mut compact: string[] = []
for ch of url {
let c = ch.codeUnitAt(0) ?? 0
if c > 32 && c != 127 {
compact.push(ch)
}
}
let text = compact.join("")
let mut colon = -1
for i of 0..text.length {
let c = text.codeUnitAt(i) ?? 0
if c == 58 {
colon = i
break
}
if c == 47 || c == 63 || c == 35 {
return true
}
}
if colon < 0 {
return true
}
if image && isDataImage(text) {
return true
}
return schemes.includes(text.slice(0, colon).toLowerCase())
}
/// Text decoded from an attribute value: its character references replaced.
fn decodeAttribute(value: string): string {
if !value.includes("&") {
return value
}
let mut parts: string[] = []
let mut from = 0
let mut i = value.indexOf("&")
while i >= 0 {
let found = scanEntity(value, i)
if found != null {
parts.push(value.slice(from, i))
parts.push(found[0])
from = found[1]
i = value.indexOf("&", from)
} else {
i = value.indexOf("&", i + 1)
}
}
parts.push(value.slice(from))
return parts.join("")
}
/// Text as it may appear between tags: a character reference stays, anything else that means something is escaped.
fn cleanText(text: string): string {
if !text.includes("&") && !text.includes("<") && !text.includes(">") {
return text
}
let mut parts: string[] = []
let mut from = 0
let mut i = 0
while i < text.length {
let c = text.codeUnitAt(i) ?? 0
if c == 38 {
let found = scanEntity(text, i)
if found != null {
i = found[1]
} else {
parts.push(text.slice(from, i))
parts.push("&amp;")
i += 1
from = i
}
} else if c == 60 || c == 62 {
parts.push(text.slice(from, i))
parts.push(c == 60 ? "&lt;" : "&gt;")
i += 1
from = i
} else {
i += 1
}
}
parts.push(text.slice(from))
return parts.join("")
}
type Attr = { name: string, value: string }
type Tag = { name: string, attrs: Attr[], end: int }
fn isSpace(c: int): boolean {
return c == 32 || c == 9 || c == 10 || c == 12 || c == 13
}
/// Reads the tag starting at `<` at `pos` (its name begins at `start`).
fn readTag(html: string, start: int): Tag {
let mut i = start
while i < html.length {
let c = html.codeUnitAt(i) ?? 32
if isSpace(c) || c == 62 || c == 47 {
break
}
i += 1
}
let name = html.slice(start, i).toLowerCase()
let mut attrs: Attr[] = []
while i < html.length {
let c = html.codeUnitAt(i) ?? 32
if c == 62 {
return { name, attrs, end: i + 1 }
}
if isSpace(c) || c == 47 {
i += 1
continue
}
let nameStart = i
i += 1
while i < html.length {
let u = html.codeUnitAt(i) ?? 32
if isSpace(u) || u == 61 || u == 62 || u == 47 {
break
}
i += 1
}
let attr = html.slice(nameStart, i).toLowerCase()
while isSpace(html.codeUnitAt(i) ?? 0) {
i += 1
}
let mut value = ""
if html.codeUnitAt(i) == 61 {
i += 1
while isSpace(html.codeUnitAt(i) ?? 0) {
i += 1
}
let q = html.codeUnitAt(i) ?? -1
if q == 34 || q == 39 {
let close = html.indexOf(String.fromCharCode(q), i + 1)
let stop = close < 0 ? html.length : close
value = html.slice(i + 1, stop)
i = close < 0 ? html.length : close + 1
} else {
let from = i
while i < html.length {
let u = html.codeUnitAt(i) ?? 32
if isSpace(u) || u == 62 {
break
}
i += 1
}
value = html.slice(from, i)
}
}
attrs.push({ name: attr, value: decodeAttribute(value) })
}
return { name, attrs, end: html.length }
}
/// Where the content of the dropped element `name` ends: after its closing tag, or at the end of the text.
fn skipElement(html: string, name: string, from: int): int {
let mut i = html.indexOf("</", from)
while i >= 0 {
if html.slice(i + 2, i + 2 + name.length).toLowerCase() == name {
let after = html.codeUnitAt(i + 2 + name.length) ?? 62
if isSpace(after) || after == 62 || after == 47 {
let close = html.indexOf(">", i)
return close < 0 ? html.length : close + 1
}
}
i = html.indexOf("</", i + 2)
}
return html.length
}
fn keepAttribute(tag: string, a: Attr, schemes: string[]): boolean {
if a.name == "href" || a.name == "src" {
return isSafeUrl(a.value, schemes, tag == "img")
}
if a.name == "target" {
return a.value == "_blank"
}
if a.name == "type" && tag == "input" {
return a.value.toLowerCase() == "checkbox"
}
if a.name == "class" {
return a.value.startsWith("language-") && !a.value.includes(" ")
}
return true
}
/// The only input kept is a task list's checkbox, always disabled: a text field in rendered content can phish.
fn isCheckbox(tag: Tag): boolean {
return tag.attrs.some(a => a.name == "type" && a.value.toLowerCase() == "checkbox")
}
fn openTag(tag: Tag, allowed: string[], schemes: string[]): string {
let mut out: string[] = ["<", tag.name]
let mut blank = false
let mut seen = Set<string>()
for a of tag.attrs {
if allowed.includes(a.name) && !seen.has(a.name) && keepAttribute(tag.name, a, schemes) {
seen.add(a.name)
out.push(` ${a.name}="${escapeHtml(a.value)}"`)
blank = blank || (a.name == "target" && a.value == "_blank")
}
}
if blank {
out.push(" rel=\"noopener noreferrer\"")
}
if tag.name == "input" && !seen.has("disabled") {
out.push(" disabled=\"\"")
}
out.push(VOID_TAGS.has(tag.name) ? " />" : ">")
return out.join("")
}
/// Removes everything `policy` doesn't allow from `html`: scripts, styles, event handlers and unsafe URLs by default.
/// The result is well formed: stray closing tags are dropped and open ones closed.
///
/// ```weft
/// sanitize("<p onclick=\"x()\">hi<script>bad()</script></p>") // "<p>hi</p>"
/// ```
export fn sanitize(html: string, policy?: Policy): string {
let tags = Set(policy?.tags ?? DEFAULT_TAGS)
let attributes = policy?.attributes ?? DEFAULT_ATTRIBUTES
let schemes = policy?.schemes ?? DEFAULT_SCHEMES
let mut out: string[] = []
let mut open: string[] = []
let mut i = 0
while i < html.length {
let lt = html.indexOf("<", i)
if lt < 0 {
out.push(cleanText(html.slice(i)))
break
}
if lt > i {
out.push(cleanText(html.slice(i, lt)))
}
let next = html.codeUnitAt(lt + 1) ?? -1
if html.startsWith("<!--", lt) {
let end = html.indexOf("-->", lt + 4)
i = end < 0 ? html.length : end + 3
} else if next == 33 || next == 63 {
let end = html.indexOf(">", lt + 2)
i = end < 0 ? html.length : end + 1
} else if next == 47 && isAlpha(html.codeUnitAt(lt + 2) ?? -1) {
let tag = readTag(html, lt + 2)
i = tag.end
let at = open.lastIndexOf(tag.name)
if at >= 0 {
while open.length > at {
out.push(`</${open.pop() ?? ""}>`)
}
}
} else if isAlpha(next) {
let tag = readTag(html, lt + 1)
i = tag.end
if DROPPED.has(tag.name) {
i = VOID_TAGS.has(tag.name) ? i : skipElement(html, tag.name, i)
} else if tags.has(tag.name) && (tag.name != "input" || isCheckbox(tag)) {
out.push(openTag(tag, attributes.get(tag.name) ?? [], schemes))
if !VOID_TAGS.has(tag.name) {
open.push(tag.name)
}
}
} else {
out.push("&lt;")
i = lt + 1
}
}
while open.length > 0 {
out.push(`</${open.pop() ?? ""}>`)
}
return out.join("")
}

50
markdown/lib/unicode.weft Normal file
View file

@ -0,0 +1,50 @@
// Generated by scripts/gen-tables.mjs from the engine's Unicode tables (general categories P and S). Don't edit it.
/// Sorted, inclusive ranges of code points: start, end, start, end, …
export let PUNCTUATION: int[] = [
33, 47, 58, 64, 91, 96, 123, 126, 161, 169, 171, 172, 174, 177, 180, 180,
182, 184, 187, 187, 191, 191, 215, 215, 247, 247, 706, 709, 722, 735, 741, 747,
749, 749, 751, 767, 885, 885, 894, 894, 900, 901, 903, 903, 1014, 1014, 1154, 1154,
1370, 1375, 1417, 1418, 1421, 1423, 1470, 1470, 1472, 1472, 1475, 1475, 1478, 1478, 1523, 1524,
1542, 1551, 1563, 1563, 1565, 1567, 1642, 1645, 1748, 1748, 1758, 1758, 1769, 1769, 1789, 1790,
1792, 1805, 2038, 2041, 2046, 2047, 2096, 2110, 2142, 2142, 2184, 2184, 2404, 2405, 2416, 2416,
2546, 2547, 2554, 2555, 2557, 2557, 2678, 2678, 2800, 2801, 2928, 2928, 3059, 3066, 3191, 3191,
3199, 3199, 3204, 3204, 3407, 3407, 3449, 3449, 3572, 3572, 3647, 3647, 3663, 3663, 3674, 3675,
3841, 3863, 3866, 3871, 3892, 3892, 3894, 3894, 3896, 3896, 3898, 3901, 3973, 3973, 4030, 4037,
4039, 4044, 4046, 4058, 4170, 4175, 4254, 4255, 4347, 4347, 4960, 4968, 5008, 5017, 5120, 5120,
5741, 5742, 5787, 5788, 5867, 5869, 5941, 5942, 6100, 6102, 6104, 6107, 6144, 6154, 6464, 6464,
6468, 6469, 6622, 6655, 6686, 6687, 6816, 6822, 6824, 6829, 6990, 6991, 7002, 7018, 7028, 7039,
7164, 7167, 7227, 7231, 7294, 7295, 7360, 7367, 7379, 7379, 8125, 8125, 8127, 8129, 8141, 8143,
8157, 8159, 8173, 8175, 8189, 8190, 8208, 8231, 8240, 8286, 8314, 8318, 8330, 8334, 8352, 8385,
8448, 8449, 8451, 8454, 8456, 8457, 8468, 8468, 8470, 8472, 8478, 8483, 8485, 8485, 8487, 8487,
8489, 8489, 8494, 8494, 8506, 8507, 8512, 8516, 8522, 8525, 8527, 8527, 8586, 8587, 8592, 9257,
9280, 9290, 9372, 9449, 9472, 10101, 10132, 11123, 11126, 11263, 11493, 11498, 11513, 11516, 11518, 11519,
11632, 11632, 11776, 11822, 11824, 11869, 11904, 11929, 11931, 12019, 12032, 12245, 12272, 12287, 12289, 12292,
12296, 12320, 12336, 12336, 12342, 12343, 12349, 12351, 12443, 12444, 12448, 12448, 12539, 12539, 12688, 12689,
12694, 12703, 12736, 12773, 12783, 12783, 12800, 12830, 12842, 12871, 12880, 12880, 12896, 12927, 12938, 12976,
12992, 13311, 19904, 19967, 42128, 42182, 42238, 42239, 42509, 42511, 42611, 42611, 42622, 42622, 42738, 42743,
42752, 42774, 42784, 42785, 42889, 42890, 43048, 43051, 43062, 43065, 43124, 43127, 43214, 43215, 43256, 43258,
43260, 43260, 43310, 43311, 43359, 43359, 43457, 43469, 43486, 43487, 43612, 43615, 43639, 43641, 43742, 43743,
43760, 43761, 43867, 43867, 43882, 43883, 44011, 44011, 64297, 64297, 64434, 64466, 64830, 64847, 64912, 64913,
64968, 64975, 65020, 65023, 65040, 65049, 65072, 65106, 65108, 65126, 65128, 65131, 65281, 65295, 65306, 65312,
65339, 65344, 65371, 65381, 65504, 65510, 65512, 65518, 65532, 65533, 65792, 65794, 65847, 65855, 65913, 65929,
65932, 65934, 65936, 65948, 65952, 65952, 66000, 66044, 66463, 66463, 66512, 66512, 66927, 66927, 67671, 67671,
67703, 67704, 67871, 67871, 67903, 67903, 68176, 68184, 68223, 68223, 68296, 68296, 68336, 68342, 68409, 68415,
68505, 68508, 68974, 68974, 69006, 69007, 69293, 69293, 69328, 69336, 69461, 69465, 69510, 69513, 69703, 69709,
69819, 69820, 69822, 69825, 69952, 69955, 70004, 70005, 70085, 70088, 70093, 70093, 70107, 70107, 70109, 70111,
70200, 70205, 70313, 70313, 70612, 70613, 70615, 70616, 70731, 70735, 70746, 70747, 70749, 70749, 70854, 70854,
71105, 71127, 71233, 71235, 71264, 71276, 71353, 71353, 71484, 71487, 71739, 71739, 72004, 72006, 72162, 72162,
72255, 72262, 72346, 72348, 72350, 72354, 72448, 72457, 72673, 72673, 72769, 72773, 72816, 72817, 73463, 73464,
73539, 73551, 73685, 73713, 73727, 73727, 74864, 74868, 77809, 77810, 92782, 92783, 92917, 92917, 92983, 92991,
92996, 92997, 93549, 93551, 93847, 93850, 94178, 94178, 113820, 113820, 113823, 113823, 117760, 117999, 118010, 118012,
118016, 118451, 118458, 118480, 118496, 118512, 118608, 118723, 118784, 119029, 119040, 119078, 119081, 119140, 119146, 119148,
119171, 119172, 119180, 119209, 119214, 119274, 119296, 119361, 119365, 119365, 119552, 119638, 120513, 120513, 120539, 120539,
120571, 120571, 120597, 120597, 120629, 120629, 120655, 120655, 120687, 120687, 120713, 120713, 120745, 120745, 120771, 120771,
120832, 121343, 121399, 121402, 121453, 121460, 121462, 121475, 121477, 121483, 123215, 123215, 123647, 123647, 124415, 124415,
125278, 125279, 126124, 126124, 126128, 126128, 126254, 126254, 126704, 126705, 126976, 127019, 127024, 127123, 127136, 127150,
127153, 127167, 127169, 127183, 127185, 127221, 127245, 127405, 127462, 127490, 127504, 127547, 127552, 127560, 127568, 127569,
127584, 127589, 127744, 128728, 128732, 128748, 128752, 128764, 128768, 128985, 128992, 129003, 129008, 129008, 129024, 129035,
129040, 129095, 129104, 129113, 129120, 129159, 129168, 129197, 129200, 129211, 129216, 129217, 129232, 129240, 129280, 129623,
129632, 129645, 129648, 129660, 129664, 129674, 129678, 129734, 129736, 129736, 129741, 129756, 129759, 129770, 129775, 129784,
129792, 129938, 129940, 130031, 130042, 130042,
]

156
markdown/markdown.weft Normal file
View file

@ -0,0 +1,156 @@
// `@weft/markdown`: CommonMark and GitHub Flavored Markdown.
import type { Block, Document, Inline, Options, Policy } from "./ast"
import { parseDocument } from "./lib/blocks"
import { renderDocument } from "./lib/render"
import { sanitize as clean } from "./lib/sanitize"
/// Parses Markdown into its syntax tree. Parsing never fails: any text is a document.
///
/// ```weft
/// let doc = parse("# Hello\n\nSome *text*.")
/// doc.children.length // 2
/// ```
export fn parse(source: string, options: Options = {}): Document {
return parseDocument(source, options.gfm ?? true)
}
/// Renders a syntax tree to HTML. Raw HTML is escaped and unsafe URLs are removed unless `options` say otherwise.
export fn render(doc: Document, options: Options = {}): string {
return renderDocument(doc, options)
}
/// Markdown to HTML, safe for untrusted input by default.
///
/// ```weft
/// toHtml("**bold** <script>alert(1)</script>")
/// // "<p><strong>bold</strong> &lt;script&gt;alert(1)&lt;/script&gt;</p>\n"
///
/// toHtml("# Title", { headingOffset: 1, linkTarget: "_blank" })
/// // "<h2>Title</h2>\n"
/// ```
export fn toHtml(source: string, options: Options = {}): string {
return renderDocument(parseDocument(source, options.gfm ?? true), options)
}
/// Removes everything `policy` doesn't allow from HTML, whatever produced it. See `Policy` for the defaults.
///
/// ```weft
/// sanitize("<a href=\"javascript:x()\" title=\"t\">hi</a>") // "<a title=\"t\">hi</a>"
/// ```
export fn sanitize(html: string, policy?: Policy): string {
return clean(html, policy)
}
fn inlineText(inlines: Inline[]): string {
let mut out: string[] = []
for n of inlines {
match n {
Str { text } => out.push(text),
SoftBreak => out.push(" "),
HardBreak => out.push("\n"),
CodeSpan { code } => out.push(code),
Emphasis { children } | Strong { children } | Strikethrough { children } => out.push(
inlineText(children),
),
Link { url, children } => {
let label = inlineText(children)
out.push(label == url || `mailto:${label}` == url ? label : `${label} (${url})`)
},
Picture { alt } => {
let label = inlineText(alt)
out.push(label == "" ? "[image]" : `[image: ${label}]`)
},
HtmlInline => {},
}
}
return out.join("")
}
fn blockText(blocks: Block[], mut out: string[]): void {
for block of blocks {
match block {
Paragraph { children } | Heading { children } => out.push(inlineText(children)),
ThematicBreak => out.push("---"),
CodeBlock { code } => out.push(code.trimEnd()),
HtmlBlock => {},
Blockquote { children } => blockText(children, &out),
BulletList { items } | OrderedList { items } => {
for item of items {
let mut inner: string[] = []
blockText(item.children, &inner)
let mark = item.checked == null ? "- " : item.checked ? "- [x] " : "- [ ] "
out.push(mark + inner.join("\n "))
}
},
Table { head, rows } => {
out.push(head.map(c => inlineText(c.children)).join(" | "))
for row of rows {
out.push(row.map(c => inlineText(c.children)).join(" | "))
}
},
}
}
}
/// The document as plain text, for previews, search and the text part of an email. Links keep their address and images
/// become `[image: caption]`.
///
/// ```weft
/// toText(parse("Read [the guide](https://weft.dev)."))
/// // "Read the guide (https://weft.dev)."
/// ```
export fn toText(doc: Document): string {
let mut out: string[] = []
blockText(doc.children, &out)
return out.join("\n\n")
}
fn visitInlines(inlines: Inline[], f: (node: Inline) => void): void {
for n of inlines {
f(n)
match n {
Emphasis { children } | Strong { children } | Strikethrough { children } | Link {
children,
} => visitInlines(children, f),
Picture { alt } => visitInlines(alt, f),
_ => {},
}
}
}
fn visitBlocks(blocks: Block[], f: (node: Inline) => void): void {
for block of blocks {
match block {
Paragraph { children } | Heading { children } => visitInlines(children, f),
Blockquote { children } => visitBlocks(children, f),
BulletList { items } | OrderedList { items } => {
for item of items {
visitBlocks(item.children, f)
}
},
Table { head, rows } => {
for cell of head {
visitInlines(cell.children, f)
}
for row of rows {
for cell of row {
visitInlines(cell.children, f)
}
}
},
_ => {},
}
}
}
/// Calls `f` with every inline of the document, in order, outermost first.
///
/// ```weft
/// let mut images: string[] = []
/// visit(parse(source), n => {
/// if let Picture { url } = n { images.push(url) }
/// })
/// ```
export fn visit(doc: Document, f: (node: Inline) => void) {
visitBlocks(doc.children, f)
}

View file

@ -0,0 +1,49 @@
// Times @weft/markdown on the VM against markdown-it on Node (JIT and --jitless), on the same text: the CommonMark
// spec's examples, joined and repeated. Needs markdown-it: MARKDOWN_IT=<path to its folder>.
//
// node packages/markdown/scripts/bench.mjs commonmark.json 1,10,50
import { spawnSync } from "node:child_process"
import { readFileSync, writeFileSync } from "node:fs"
import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"
const root = join(dirname(fileURLToPath(import.meta.url)), "..")
const weft = process.env.WEFT_BIN ?? join(root, "../../core/target/release/weft")
const spec = process.argv[2]
const sizes = (process.argv[3] ?? "1,10,50").split(",").map(Number)
const lib = process.env.MARKDOWN_IT
const script = `
const MarkdownIt = require(${JSON.stringify(lib)})
const fs = require("fs")
const cases = JSON.parse(fs.readFileSync(${JSON.stringify(spec)}, "utf8"))
const text = cases.map((c) => c.markdown).join("\\n\\n").repeat(Number(process.argv[2]))
const md = new MarkdownIt({ html: false, linkify: true })
const start = process.hrtime.bigint()
const html = md.render(text)
console.log(Number(process.hrtime.bigint() - start) / 1e6, text.length, html.length)
`
const file = join(process.env.TMPDIR ?? "/workspace/core/target/tools", "bench-markdown-it.cjs")
writeFileSync(file, script)
function ms(cmd, args, env = {}) {
const start = process.hrtime.bigint()
const r = spawnSync(cmd, args, { cwd: root, encoding: "utf8", env: { ...process.env, ...env } })
if (r.status !== 0) throw new Error(r.stderr)
return { ms: Number(process.hrtime.bigint() - start) / 1e6, out: r.stdout.trim() }
}
const limits = { WEFT_FUEL: "100000000000", WEFT_MEMORY_MB: "4000" }
const base = Math.min(...[0, 1, 2].map(() => ms(weft, ["run", "bench/bench.weft", "0"], limits).ms))
console.log(`VM start-up (loading the package and the spec's text): ${base.toFixed(0)} ms`)
console.log("size KB | weft VM ms | markdown-it JIT ms | markdown-it --jitless ms | VM / JIT | VM / jitless")
for (const n of sizes) {
const vm = Math.min(...[0, 1, 2].map(() => ms(weft, ["run", "bench/bench.weft", String(n)], limits).ms)) - base
const best = (flags) => {
const runs = [0, 1, 2].map(() => ms("node", [...flags, file, String(n)]).out.split(" ").map(Number))
return { ms: Math.min(...runs.map((r) => r[0])), size: runs[0][1] }
}
const jit = best([])
const nojit = best(["--jitless"])
console.log(`${(jit.size / 1024).toFixed(0).padStart(7)} | ${vm.toFixed(0).padStart(10)} | ${jit.ms.toFixed(1).padStart(18)} | ${nojit.ms.toFixed(1).padStart(24)} | ${(vm / jit.ms).toFixed(1).padStart(8)} | ${(vm / nojit.ms).toFixed(1).padStart(12)}`)
}

View file

@ -0,0 +1,69 @@
// Generates tests/cases.weft from the CommonMark spec's examples and cmark-gfm's spec text.
//
// curl -o commonmark.json https://spec.commonmark.org/0.31.2/spec.json
// curl -o gfm.txt https://raw.githubusercontent.com/github/cmark-gfm/master/test/spec.txt
// node packages/markdown/scripts/gen-spec.mjs commonmark.json gfm.txt
import { mkdirSync, readFileSync, writeFileSync } from "node:fs"
import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"
const tests = join(dirname(fileURLToPath(import.meta.url)), "..", "tests")
mkdirSync(tests, { recursive: true })
const commonmark = JSON.parse(readFileSync(process.argv[2], "utf8")).map((c) => ({
id: c.example,
section: c.section,
markdown: c.markdown,
html: c.html,
}))
// cmark-gfm's spec.txt: examples are fenced by 32 backticks, `example` then the extension's name.
const gfm = []
let section = ""
let number = 0
const lines = readFileSync(process.argv[3], "utf8").split("\n")
for (let i = 0; i < lines.length; i++) {
const heading = lines[i].match(/^#{1,6} (.*)$/)
if (heading) section = heading[1]
const open = lines[i].match(/^`{32} example(.*)$/)
if (!open) continue
number++
const extensions = open[1].trim().split(/\s+/).filter(Boolean)
let j = i + 1
const source = []
while (lines[j] !== ".") source.push(lines[j++])
j++
const html = []
while (!lines[j].startsWith("`".repeat(32))) html.push(lines[j++])
i = j
// `disabled` examples describe behaviour cmark-gfm itself doesn't have. `tagfilter` is for pages that embed raw HTML:
// here `html: "escape"` and `"sanitize"` already remove those tags, and `"trust"` means what it says.
if (extensions.includes("disabled") || extensions.includes("tagfilter")) continue
const text = (l) => l.map((x) => x + "\n").join("").replaceAll("→", "\t")
gfm.push({ id: number, section, markdown: text(source), html: text(html), extensions })
}
const str = (s) => JSON.stringify(s).replaceAll("\u2028", "\\u2028").replaceAll("\u2029", "\\u2029").replaceAll("$", "\\$")
const row = (c) =>
` { id: ${c.id}, section: ${str(c.section)}, markdown: ${str(c.markdown)}, html: ${str(c.html)}, gfm: ${(c.extensions ?? []).length > 0} },`
writeFileSync(
join(tests, "cases.weft"),
`// Generated by scripts/gen-spec.mjs. Don't edit it.
/// One example. With gfm set it needs GFM's extensions; cmark-gfm runs the others without them.
export type Case = { id: number, section: string, markdown: string, html: string, gfm: boolean }
/// The examples of the CommonMark spec, version 0.31.2.
export let COMMONMARK: Case[] = [
${commonmark.map(row).join("\n")}
]
/// The examples of the GitHub Flavored Markdown spec.
export let GFM: Case[] = [
${gfm.map(row).join("\n")}
]
`,
)
console.log(`${commonmark.length} CommonMark examples, ${gfm.length} GFM examples`)

View file

@ -0,0 +1,73 @@
// Generates lib/entities.weft (the HTML5 named character references) and lib/unicode.weft (Unicode
// punctuation and symbols, what CommonMark's delimiter rules call punctuation).
//
// curl -o entities.json https://html.spec.whatwg.org/entities.json
// node packages/markdown/scripts/gen-tables.mjs entities.json
import { readFileSync, writeFileSync } from "node:fs"
import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"
const lib = join(dirname(fileURLToPath(import.meta.url)), "..", "lib")
const entities = JSON.parse(readFileSync(process.argv[2], "utf8"))
const pairs = Object.entries(entities)
.filter(([name]) => name.endsWith(";"))
.map(([name, v]) => `${name.slice(1, -1)}=${v.codepoints.map((c) => c.toString(16)).join(",")}`)
.sort()
writeFileSync(
join(lib, "entities.weft"),
`// Generated by scripts/gen-tables.mjs from https://html.spec.whatwg.org/entities.json. Don't edit it.
import { parseInt } from "std/math"
let DATA = "${pairs.join(" ")}"
fn build(): Map<string, string> {
let mut table = Map<string, string>()
for entry of DATA.split(" ") {
let at = entry.indexOf("=")
let mut text = ""
for hex of entry.slice(at + 1).split(",") {
text += String.fromCodePoint(try parseInt(hex, 16) else 0xfffd)
}
table.set(entry.slice(0, at), text)
}
return table
}
let ENTITIES = build()
/// The text of a named character reference (\`amp\` for \`&amp;\`), or \`null\` when HTML has no such name.
export fn entity(name: string): string? {
return ENTITIES.get(name)
}
`,
)
const punct = /[\p{P}\p{S}]/u
const ranges = []
let start = -1
for (let c = 0; c <= 0x110000; c++) {
const is = c < 0x110000 && !(c >= 0xd800 && c <= 0xdfff) && punct.test(String.fromCodePoint(c))
if (is && start < 0) start = c
if (!is && start >= 0) {
ranges.push(start, c - 1)
start = -1
}
}
const rows = []
for (let i = 0; i < ranges.length; i += 16) rows.push(" " + ranges.slice(i, i + 16).join(", ") + ",")
writeFileSync(
join(lib, "unicode.weft"),
`// Generated by scripts/gen-tables.mjs from the engine's Unicode tables (general categories P and S). Don't edit it.
/// Sorted, inclusive ranges of code points: start, end, start, end, …
export let PUNCTUATION: number[] = [
${rows.join("\n")}
]
`,
)
console.log(`${pairs.length} entities, ${ranges.length / 2} punctuation ranges`)

View file

@ -0,0 +1,39 @@
// Runs tests/hostile.weft: each input at n and 2n, timed from outside the VM (the clock doesn't move within a turn).
// A size that doubles must not take much more than twice as long.
//
// node packages/markdown/scripts/hostile.mjs [n]
import { spawnSync } from "node:child_process"
import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"
const root = join(dirname(fileURLToPath(import.meta.url)), "..")
const weft = process.env.WEFT_BIN ?? join(root, "../../core/target/release/weft")
const n = Number(process.argv[2] ?? 8000)
// The limits are lifted so that a slow path shows up as time rather than as a stop.
const env = { ...process.env, WEFT_FUEL: "20000000000", WEFT_MEMORY_MB: "1500" }
function run(args) {
const start = process.hrtime.bigint()
const r = spawnSync(weft, ["run", "tests/hostile.weft", ...args], { cwd: root, env, encoding: "utf8", timeout: 120000 })
return { ms: Number(process.hrtime.bigint() - start) / 1e6, out: r.stdout.trim(), err: r.stderr.trim(), status: r.status }
}
const names = run(["--list"]).out.split("\n")
const base = run(["lines", "0"]).ms
let bad = 0
for (const name of names) {
const a = run([name, String(n)])
const b = run([name, String(n * 2)])
const [ta, tb] = [Math.max(a.ms - base, 1), Math.max(b.ms - base, 1)]
const stopped = a.status !== 0 || b.status !== 0
const quadratic = tb > 1500 && tb / ta > 3.2
if (stopped || quadratic) bad++
const note = stopped ? ` STOPPED: ${(b.err || a.err).split("\n")[0]}` : quadratic ? " QUADRATIC" : ""
console.log(`${name.padEnd(24)} ${ta.toFixed(0).padStart(7)} ms ${tb.toFixed(0).padStart(7)} ms${note}`)
}
const expansion = run(["expansion"])
const size = Number(expansion.out)
if (!(size > 0 && size < 1000000)) bad++
console.log(`reference expansion: ${size} bytes of HTML from 50 KB of Markdown`)
console.log(bad === 0 ? "ok" : `${bad} failed`)
process.exit(bad === 0 ? 0 : 1)

1333
markdown/tests/cases.weft Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,47 @@
// Hostile input, one per run: `weft run tests/hostile.weft <name> <n>` renders it and prints the HTML's length.
// scripts/hostile.mjs runs them all and checks that the time each takes grows in proportion to its size.
import { log } from "std/log"
import { args } from "std/process"
import { toHtml } from "../markdown"
type Input = { name: string, make: (n: int) => string }
let inputs: Input[] = [
{ name: "nested quotes", make: n => ">".repeat(n) + " x" },
{ name: "nested lists", make: n => "- ".repeat(n) + "x" },
{ name: "nested emphasis", make: n => "*a **b ".repeat(n) + "c" + "** d*".repeat(n) },
{ name: "nested links", make: n => "[a ".repeat(n) + "](/x)".repeat(n) },
{ name: "nested images", make: n => "![".repeat(n) + "x" + "](/y)".repeat(n) },
{ name: "open emphasis", make: n => "*a ".repeat(n) },
{ name: "closed emphasis", make: n => "_a_ ".repeat(n) },
{ name: "open brackets", make: n => "[".repeat(n) },
{ name: "close brackets", make: n => "]".repeat(n) },
{ name: "unfinished links", make: n => "[a](".repeat(n) },
{ name: "backtick runs", make: n => "`a ``b ".repeat(n) },
{ name: "open tags", make: n => "<a ".repeat(n) },
{ name: "open comments", make: n => "a <!-- ".repeat(n) },
{ name: "open cdata", make: n => "a <![CDATA[ ".repeat(n) },
{ name: "open instructions", make: n => "a <? ".repeat(n) },
{ name: "entities", make: n => "&amp".repeat(n) },
{ name: "backslashes", make: n => "\\".repeat(n) },
{ name: "lines", make: n => "a\n".repeat(n) },
{ name: "reference definitions", make: n => "[a]: /b\n".repeat(n) },
{ name: "list items", make: n => "- a\n".repeat(n) },
{ name: "trailing parens", make: n => "http://a.dev/" + ")".repeat(n) },
{ name: "at signs", make: n => "a@".repeat(n) },
{ name: "table rows", make: n => "| a | b |\n|---|---|\n" + "| 1 | 2 |\n".repeat(n) },
{ name: "wide table", make: n => "| a ".repeat(n) + "\n" + "|-".repeat(n) },
]
let name = args().at(0) ?? "--list"
if name == "--list" {
log(inputs.map(i => i.name).join("\n"))
} else if name == "expansion" {
// A definition used many times can't multiply the output.
let url = "/" + "x".repeat(10000)
log(toHtml(`[a]: ${url}\n\n` + "[a] ".repeat(10000)).length)
} else {
let n = try int(args().at(1) ?? "1000") else 1000
let input = inputs.find(i => i.name == name)
log(input == null ? -1 : toHtml(input.make(n)).length)
}

View file

@ -0,0 +1,220 @@
// `weft test`: the package's API and its safety. The spec's examples are in tests/spec.weft, hostile input in
// tests/hostile.weft (both are programs: they run past a test's deadline).
import { assert, assertEq } from "std/testing"
import { parse, render, sanitize, toHtml, toText, visit } from "../markdown"
// ---- safe by default ----
test "raw HTML is shown as text" {
assertEq(toHtml("<script>alert(1)</script>"), "<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>\n")
assertEq(
toHtml("a <b onclick=\"x()\">b</b>"),
"<p>a &lt;b onclick=&quot;x()&quot;&gt;b&lt;/b&gt;</p>\n",
)
}
test "unsafe link schemes render as their text" {
let attacks = [
"[x](javascript:alert(1))",
"[x](JaVaScRiPt:alert(1))",
"[x](vbscript:msgbox(1))",
"[x](data:text/html;base64,PHNjcmlwdD4=)",
"[x](&#106;avascript:alert(1))",
"[x](javascript&colon;alert(1))",
"[x](file:///etc/passwd)",
]
for a of attacks {
assertEq(toHtml(a), "<p>x</p>\n")
}
assertEq(toHtml("<javascript:alert(1)>"), "<p>javascript:alert(1)</p>\n")
}
test "unsafe images render as their alt text" {
assertEq(toHtml("![cat](javascript:alert(1))"), "<p>cat</p>\n")
assertEq(toHtml("![cat](data:image/svg+xml;base64,PHN2Zz4=)"), "<p>cat</p>\n")
assertEq(
toHtml("![cat](data:image/png;base64,AAAA)"),
"<p><img src=\"data:image/png;base64,AAAA\" alt=\"cat\" /></p>\n",
)
}
test "safe URLs are kept" {
assertEq(
toHtml("[a](https://x.dev/?q=1&r=2)"),
"<p><a href=\"https://x.dev/?q=1&amp;r=2\">a</a></p>\n",
)
assertEq(toHtml("[a](/docs/x:y)"), "<p><a href=\"/docs/x:y\">a</a></p>\n")
assertEq(toHtml("[a](#top)"), "<p><a href=\"#top\">a</a></p>\n")
assertEq(toHtml("[a](mailto:me@x.dev)"), "<p><a href=\"mailto:me@x.dev\">a</a></p>\n")
assertEq(toHtml("[a](tel:+38640123456)"), "<p><a href=\"tel:+38640123456\">a</a></p>\n")
}
test "titles, code and attributes are escaped" {
assertEq(
toHtml("[a](/x \"\\\"><script>\")"),
"<p><a href=\"/x\" title=\"&quot;&gt;&lt;script&gt;\">a</a></p>\n",
)
assertEq(toHtml("`<b>`"), "<p><code>&lt;b&gt;</code></p>\n")
assertEq(
toHtml("```\"><script>\nx\n```"),
"<pre><code class=\"language-&quot;&gt;&lt;script&gt;\">x\n</code></pre>\n",
)
}
// ---- the sanitiser ----
test "sanitize removes scripts, handlers and unsafe URLs" {
assertEq(sanitize("<p onclick=\"x()\">hi<script>bad()</script></p>"), "<p>hi</p>")
assertEq(sanitize("<a href=\"javascript:x()\" title=\"t\">hi</a>"), "<a title=\"t\">hi</a>")
assertEq(sanitize("<a href=\"java&#115;cript:x()\">hi</a>"), "<a>hi</a>")
assertEq(sanitize("<a href=\"java&#9;script:x()\">hi</a>"), "<a>hi</a>")
assertEq(sanitize("<a href=\" &#14; javascript:x()\">hi</a>"), "<a>hi</a>")
assertEq(sanitize("<img src=x onerror=alert(1)>"), "<img src=\"x\" />")
assertEq(sanitize("<IMG SRC=\"JAVASCRIPT:x\">"), "<img />")
assertEq(sanitize("<svg><script>x</script></svg>ok"), "ok")
assertEq(sanitize("<math><mi>x</mi></math>ok"), "ok")
assertEq(sanitize("<style>*{display:none}</style>ok"), "ok")
assertEq(sanitize("<iframe src=\"https://evil\"></iframe>ok"), "ok")
assertEq(sanitize("<ScRiPt >x</sCrIpT >ok"), "ok")
assertEq(sanitize("<script>never closed"), "")
assertEq(sanitize("<!-- <script>x</script> -->ok"), "ok")
assertEq(sanitize("<form action=\"/x\"><button>go</button></form>"), "go")
}
test "sanitize keeps only a disabled checkbox" {
assertEq(sanitize("<input type=\"text\" value=\"pw\">"), "")
assertEq(sanitize("<input>"), "")
assertEq(
sanitize("<input type=\"checkbox\" checked>"),
"<input type=\"checkbox\" checked=\"\" disabled=\"\" />",
)
}
test "sanitize gives well-formed output" {
assertEq(sanitize("<b><i>x</b>y"), "<b><i>x</i></b>y")
assertEq(sanitize("</div>x"), "x")
assertEq(sanitize("<p>a"), "<p>a</p>")
assertEq(sanitize("1 < 2 & 3 > 2 &amp; ok"), "1 &lt; 2 &amp; 3 &gt; 2 &amp; ok")
assertEq(sanitize("<a title=\"a&quot;b\">x</a>"), "<a title=\"a&quot;b\">x</a>")
assertEq(
sanitize("<a target=\"_blank\" href=\"/x\">x</a>"),
"<a target=\"_blank\" href=\"/x\" rel=\"noopener noreferrer\">x</a>",
)
assertEq(sanitize("<a target=\"evil\" href=\"/x\">x</a>"), "<a href=\"/x\">x</a>")
}
test "a policy narrows what is kept" {
assertEq(sanitize("<b>x</b><i>y</i>", { tags: ["b"] }), "<b>x</b>y")
assertEq(sanitize("<a href=\"ftp://x\">y</a>", { schemes: ["ftp"] }), "<a href=\"ftp://x\">y</a>")
assertEq(sanitize("<a href=\"https://x\">y</a>", { schemes: ["ftp"] }), "<a>y</a>")
let attributes = Map<string, string[]>([["b", ["title"]]])
assertEq(
sanitize("<b title=\"t\" id=\"i\">x</b>", { tags: ["b"], attributes }),
"<b title=\"t\">x</b>",
)
}
test "html: sanitize keeps safe raw HTML in Markdown" {
assertEq(
toHtml("<div onclick=\"x()\">\n*hi*\n</div>", { html: "sanitize" }),
"<div>\n*hi*\n</div>\n",
)
assertEq(
toHtml("a <kbd>b</kbd> <script>c</script>", { html: "sanitize" }),
"<p>a <kbd>b</kbd> </p>\n",
)
assertEq(toHtml("<script>\nbad()\n\n</script>\n\nok", { html: "sanitize" }), "\n<p>ok</p>\n")
assertEq(
toHtml("- [x] done", { html: "sanitize" }),
"<ul>\n<li><input checked=\"\" disabled=\"\" type=\"checkbox\" /> done</li>\n</ul>\n",
)
}
test "html: trust passes raw HTML through" {
assertEq(
toHtml("<div onclick=\"x()\">a</div>", { html: "trust" }),
"<div onclick=\"x()\">a</div>\n",
)
}
// ---- GFM ----
test "tables" {
let html = toHtml("| a | b |\n|:--|--:|\n| 1 | `x\\|y` |")
assertEq(
html,
"<table>\n<thead>\n<tr>\n<th align=\"left\">a</th>\n<th align=\"right\">b</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td align=\"left\">1</td>\n<td align=\"right\"><code>x|y</code></td>\n</tr>\n</tbody>\n</table>\n",
)
}
test "task lists" {
let doc = parse("- [ ] todo\n- [x] done\n- plain")
match doc.children[0] {
BulletList { items } => {
let checked: boolean?[] = items.map(i => i.checked)
let expected: boolean?[] = [false, true, null]
assertEq(checked, expected)
},
_ => assert(false, "a list"),
}
}
test "strikethrough and bare links" {
assertEq(toHtml("~~gone~~"), "<p><del>gone</del></p>\n")
assertEq(
toHtml("see https://x.dev/a_b."),
"<p>see <a href=\"https://x.dev/a_b\">https://x.dev/a_b</a>.</p>\n",
)
assertEq(toHtml("www.x.dev"), "<p><a href=\"http://www.x.dev\">www.x.dev</a></p>\n")
assertEq(toHtml("me@x.dev"), "<p><a href=\"mailto:me@x.dev\">me@x.dev</a></p>\n")
}
test "gfm: false is plain CommonMark" {
assertEq(toHtml("~~a~~ https://x.dev", { gfm: false }), "<p>~~a~~ https://x.dev</p>\n")
assertEq(toHtml("| a |\n|---|", { gfm: false }), "<p>| a |\n|---|</p>\n")
}
// ---- what Vidni's briefs need ----
test "heading offset, link target and URL rewriting" {
assertEq(toHtml("# T", { headingOffset: 1 }), "<h2>T</h2>\n")
assertEq(toHtml("###### T", { headingOffset: 3 }), "<h6>T</h6>\n")
assertEq(
toHtml("[a](https://x.dev)", { linkTarget: "_blank" }),
"<p><a href=\"https://x.dev\" target=\"_blank\" rel=\"noopener noreferrer\">a</a></p>\n",
)
let resolve = (url: string, kind: "link" | "image") => url.startsWith("upload:")
? `https://cdn.x.dev/${url.slice(7)}`
: null
assertEq(
toHtml("![c](upload:abc)", { rewriteUrl: resolve }),
"<p><img src=\"https://cdn.x.dev/abc\" alt=\"c\" /></p>\n",
)
// Not resolved: `upload:` isn't an allowed scheme, so the caption stays.
assertEq(toHtml("![c](upload:abc)"), "<p>c</p>\n")
}
test "toText for previews and emails" {
assertEq(
toText(parse("# Hi\n\nRead **this** [guide](https://x.dev).\n\n![cat](a.png)\n\n- one\n- two")),
"Hi\n\nRead this guide (https://x.dev).\n\n[image: cat]\n\n- one\n\n- two",
)
}
test "visit finds the images of a document" {
let mut urls: string[] = []
visit(parse("![a](upload:1)\n\n> *![b](upload:2)*\n\n| ![c](upload:3) |\n|---|"), n => {
if let Picture { url } = n {
urls.push(url)
}
})
assertEq(urls, ["upload:1", "upload:2", "upload:3"])
}
test "render uses a tree that was changed" {
let doc = parse("hello")
assertEq(render({ children: [...doc.children, ...doc.children] }), "<p>hello</p>\n<p>hello</p>\n")
}
test "strange characters survive" {
assertEq(toHtml("a\u{0}b"), "<p>a\u{fffd}b</p>\n")
assertEq(toHtml("line\r\nnext\rlast"), "<p>line\nnext\nlast</p>\n")
assertEq(toHtml("😀 *é*"), "<p>😀 <em>é</em></p>\n")
assertEq(toHtml(""), "")
}

54
markdown/tests/spec.weft Normal file
View file

@ -0,0 +1,54 @@
// Runs the spec's examples: `weft run tests/spec.weft [commonmark|gfm] [-v]` prints every failure and a summary.
import { args, exit } from "std/process"
import { log } from "std/log"
import { toHtml } from "../markdown"
import { COMMONMARK, GFM } from "./cases"
import type { Case } from "./cases"
/// Where the GFM spec (written against CommonMark 0.29) and CommonMark 0.31.2 give different HTML for the same text,
/// CommonMark wins: cmark-gfm's renderer folds nested `<strong>`, which CommonMark's examples keep.
fn followsCommonMark(c: Case, got: string): boolean {
return COMMONMARK.some(o => o.markdown == c.markdown && o.html == got)
}
fn runAll(name: string, cases: Case[], verbose: boolean): int {
let mut failed = 0
let mut disagreements = 0
let mut bySection = Map<string, int>()
for c of cases {
let got = toHtml(c.markdown, { gfm: c.gfm, html: "trust" })
if got != c.html && followsCommonMark(c, got) {
disagreements += 1
} else if got != c.html {
failed += 1
bySection.set(c.section, (bySection.get(c.section) ?? 0) + 1)
if verbose {
log(`--- ${name} example ${c.id} (${c.section})`)
log(`markdown: ${c.markdown}`)
log(`expected: ${c.html}`)
log(`got: ${got}`)
}
}
}
log(
`${name}: ${cases.length - failed - disagreements} of ${cases.length} pass, ${disagreements} follow CommonMark 0.31.2 where the specs disagree, ${failed} fail`,
)
for [section, n] of bySection {
log(` ${n} failing in ${section}`)
}
return failed
}
let argv = args()
let verbose = argv.includes("-v")
let only = argv.filter(a => a != "-v").at(0)
let mut failures = 0
if only == null || only == "commonmark" {
failures += runAll("CommonMark", COMMONMARK, verbose)
}
if only == null || only == "gfm" {
failures += runAll("GFM", GFM, verbose)
}
if failures > 0 {
exit(1)
}

10
markdown/weft.toml Normal file
View file

@ -0,0 +1,10 @@
[package]
name = "weft/markdown"
version = "0.1.0"
description = "CommonMark and GitHub Flavored Markdown: an AST, HTML and a sanitiser, safe by default"
license = "MIT OR Apache-2.0"
# A placeholder until the public repository exists (packages/README.md, Publishing for real).
repository = "https://git.weft-lang.dev/weft/packages"
exports = ["markdown", "ast"]
main = "markdown"
include = ["markdown.weft", "ast.weft", "lib", "README.md"]