packages/ at markup-kit dd1e6eb5270e
A snapshot of packages/ in the Weft repository; the official packages are published from these commits. Markup-Kit-Commit: dd1e6eb5270e62e0c8b6c202b3d01f0da6e00ddc
This commit is contained in:
commit
306b143104
72 changed files with 10915 additions and 0 deletions
102
markdown/README.md
Normal file
102
markdown/README.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# @weft/markdown
|
||||
|
||||
CommonMark and GitHub Flavored Markdown for Weft: a syntax tree, HTML, and a sanitiser. It is written in Weft, needs no
|
||||
capabilities (it does no I/O), and is safe for untrusted input by default.
|
||||
|
||||
```sh
|
||||
weft add weft/markdown
|
||||
```
|
||||
|
||||
```weft
|
||||
import { parse, render, sanitize, toHtml, toText, visit } from "@weft/markdown"
|
||||
|
||||
toHtml("# Hello\n\n**bold** <script>alert(1)</script>")
|
||||
// <h1>Hello</h1>
|
||||
// <p><strong>bold</strong> <script>alert(1)</script></p>
|
||||
```
|
||||
|
||||
## What it gives you
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `toHtml(source, options?)` | Markdown to HTML |
|
||||
| `parse(source, options?)` | the syntax tree (`Document`, typed in `@weft/markdown/ast`) |
|
||||
| `render(doc, options?)` | a tree to HTML, for a tree you have changed |
|
||||
| `toText(doc)` | plain text, for previews, search and the text part of an email |
|
||||
| `visit(doc, f)` | every inline of a document, to collect links or images |
|
||||
| `sanitize(html, policy?)` | the allowlist sanitiser on its own, for HTML from anywhere |
|
||||
|
||||
GFM is on by default: tables, task lists, strikethrough and bare links (`www.`, `https://`, email addresses). `gfm: false`
|
||||
gives plain CommonMark.
|
||||
|
||||
## Safe by default
|
||||
|
||||
- **Raw HTML is shown as text.** `html: "sanitize"` keeps what an allowlist permits; `html: "trust"` passes it through and
|
||||
is for Markdown you wrote yourself.
|
||||
- **Links and images only use `http`, `https`, `mailto` and `tel`**, or are relative (`schemes` changes the list). Images
|
||||
may also be `data:image/` PNG, GIF, JPEG or WebP, never SVG. A link that isn't allowed renders as its text.
|
||||
- **The sanitiser is an allowlist over a tokenizer**, and what it writes is well formed with every text escaped. Scripts,
|
||||
styles, frames, forms, SVG, MathML, event handlers and unsafe URLs are removed. The only `<input>` kept is a disabled
|
||||
checkbox.
|
||||
- **Hostile documents can't take the server down.** Quotes, lists and inlines stop nesting at 64 levels, the text a link
|
||||
reference expands to is bounded, and every input in `tests/hostile.weft` takes time proportional to its size.
|
||||
|
||||
```weft
|
||||
toHtml("a <kbd>b</kbd> <script>c()</script>", { html: "sanitize" }) // <p>a <kbd>b</kbd> </p>
|
||||
sanitize("<a href=\"javascript:x()\" onclick=\"y()\">hi</a>") // <a>hi</a>
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
```weft
|
||||
toHtml(source, {
|
||||
headingOffset: 1, // `# Title` renders as <h2>
|
||||
linkTarget: "_blank", // adds rel="noopener noreferrer"
|
||||
rewriteUrl: (url, kind) => url.startsWith("upload:") ? cdn(url.slice(7)) : null,
|
||||
})
|
||||
```
|
||||
|
||||
`rewriteUrl` runs before the safety check, so an app's own scheme (`upload:<id>`) can resolve to a real address, and one it
|
||||
can't resolve is dropped rather than rendered.
|
||||
|
||||
## Conformance
|
||||
|
||||
`weft run tests/spec.weft` runs the specs' examples:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| CommonMark 0.31.2 | 652 of 652 |
|
||||
| GitHub Flavored Markdown | 660 of 669, and the other 9 are below |
|
||||
|
||||
The 9 are examples where the two specs give different HTML for the same text: the GFM spec was written against CommonMark
|
||||
0.29, and its renderer folds nested `<strong>`, which CommonMark 0.31.2's own examples keep. This package follows CommonMark
|
||||
there, and the runner checks each such example against CommonMark's expected output. GFM's `tagfilter` example isn't run:
|
||||
`html: "escape"` and `"sanitize"` already remove those tags, and `"trust"` means what it says.
|
||||
|
||||
## Speed
|
||||
|
||||
Against markdown-it 15.0.3 on Node 24, rendering the CommonMark spec's examples joined and repeated (`scripts/bench.mjs`;
|
||||
one cold run each, in a 6 GB container, so markdown-it's JIT is warming up over the small sizes):
|
||||
|
||||
| document | Weft VM | markdown-it (JIT) | markdown-it (`--jitless`) |
|
||||
|---|---|---|---|
|
||||
| 16 KB | 52 ms | 44 ms | 47 ms |
|
||||
| 157 KB | 112 ms | 105 ms | 242 ms |
|
||||
| 787 KB | 1.2 s | 0.54 s | 1.6 s |
|
||||
| 3.1 MB | 5.3 s | 1.5 s | 5.7 s |
|
||||
|
||||
## Limits
|
||||
|
||||
- A turn's default limits (32 MB of allocation, and 250 ms when serving) fit documents up to about 150 KB. Above that, raise
|
||||
`WEFT_MEMORY_MB` and `WEFT_DEADLINE_MS`, or render outside a request.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
weft test tests/markdown.test.weft # the API and its safety
|
||||
weft run tests/spec.weft # the specs' examples (-v prints each failure)
|
||||
node scripts/hostile.mjs # hostile input: nothing stops, nothing is quadratic
|
||||
node scripts/bench.mjs commonmark.json # beside markdown-it (MARKDOWN_IT=<its folder>)
|
||||
node scripts/gen-spec.mjs # regenerates tests/cases.weft from the specs
|
||||
node scripts/gen-tables.mjs # regenerates lib/entities.weft and lib/unicode.weft
|
||||
```
|
||||
83
markdown/ast.weft
Normal file
83
markdown/ast.weft
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
// The Markdown syntax tree and the options of every function in `@weft/markdown`.
|
||||
/// A parsed document: its blocks, in order.
|
||||
export type Document = { children: Block[] }
|
||||
|
||||
/// A table column's alignment, `null` when its delimiter row doesn't say.
|
||||
export type Align = "left" | "center" | "right" | null
|
||||
|
||||
/// A block: what a document, a quote or a list item is made of.
|
||||
///
|
||||
/// - `CodeBlock`: `info` is the text after the opening fence (`""` for an indented block).
|
||||
/// - `HtmlBlock`: raw HTML as written, rendered according to `Options.html`.
|
||||
/// - `BulletList`, `OrderedList`: a `tight` list has no blank lines between its items; `start` is the first number.
|
||||
/// - `Table`: a GFM table, with one cell per column in every row.
|
||||
export type Block =
|
||||
| Paragraph { children: Inline[] }
|
||||
| Heading { level: float, children: Inline[] }
|
||||
| ThematicBreak
|
||||
| CodeBlock { info: string, code: string }
|
||||
| HtmlBlock { html: string }
|
||||
| Blockquote { children: Block[] }
|
||||
| BulletList { tight: boolean, items: ListItem[] }
|
||||
| OrderedList { tight: boolean, start: float, items: ListItem[] }
|
||||
| Table { align: Align[], head: TableCell[], rows: TableCell[][] }
|
||||
|
||||
/// A list item. `checked` is `null` unless the item is a GFM task (`- [x] done`).
|
||||
export type ListItem = { children: Block[], checked: boolean? }
|
||||
|
||||
/// A table cell.
|
||||
export type TableCell = { children: Inline[] }
|
||||
|
||||
/// An inline: what paragraphs, headings and cells are made of. `HtmlInline` is raw HTML as written, rendered according
|
||||
/// to `Options.html`.
|
||||
export type Inline =
|
||||
| Str { text: string }
|
||||
| SoftBreak
|
||||
| HardBreak
|
||||
| CodeSpan { code: string }
|
||||
| Emphasis { children: Inline[] }
|
||||
| Strong { children: Inline[] }
|
||||
| Strikethrough { children: Inline[] }
|
||||
| Link { url: string, title: string?, children: Inline[] }
|
||||
| Picture { url: string, title: string?, alt: Inline[] }
|
||||
| HtmlInline { html: string }
|
||||
|
||||
/// What happens to raw HTML in the source:
|
||||
/// - `"escape"` (the default) shows it as text;
|
||||
/// - `"sanitize"` keeps what the `policy` allows and removes the rest;
|
||||
/// - `"trust"` passes it through. Use it only for Markdown you wrote yourself.
|
||||
export type HtmlMode = "escape" | "sanitize" | "trust"
|
||||
|
||||
/// What `sanitize` keeps. Anything left out takes the default: common formatting tags, no scripts, styles, event
|
||||
/// handlers or forms.
|
||||
export type Policy = {
|
||||
/// Lowercase tag names to keep. A tag not listed is removed and its content kept.
|
||||
tags?: string[],
|
||||
/// Attributes to keep, per lowercase tag name.
|
||||
attributes?: Map<string, string[]>,
|
||||
/// URL schemes `href` and `src` may use. Relative URLs are always allowed.
|
||||
schemes?: string[],
|
||||
}
|
||||
|
||||
/// Whether a URL is a link's or an image's.
|
||||
export type UrlKind = "link" | "image"
|
||||
|
||||
/// Options for parsing and rendering. Every field is optional.
|
||||
export type Options = {
|
||||
/// GitHub Flavored Markdown: tables, task lists, strikethrough and bare autolinks. On by default.
|
||||
gfm?: boolean,
|
||||
/// What happens to raw HTML. `"escape"` by default.
|
||||
html?: HtmlMode,
|
||||
/// What `html: "sanitize"` keeps.
|
||||
policy?: Policy,
|
||||
/// URL schemes links and images may use: `http`, `https`, `mailto` and `tel` by default. Relative URLs are always
|
||||
/// allowed, and images may also be `data:image/` PNG, GIF, JPEG or WebP. A link to anything else renders as its text.
|
||||
schemes?: string[],
|
||||
/// Added to every heading's level, up to 6: `1` renders `# Title` as `<h2>`.
|
||||
headingOffset?: float,
|
||||
/// `target` for every link. `"_blank"` also adds `rel="noopener noreferrer"`.
|
||||
linkTarget?: string,
|
||||
/// Rewrites a URL before it is checked and rendered, as an app resolves `upload:<id>` to a real address.
|
||||
/// Returning `null` keeps the URL as written.
|
||||
rewriteUrl?: (url: string, kind: UrlKind) => string?,
|
||||
}
|
||||
12
markdown/bench/bench.weft
Normal file
12
markdown/bench/bench.weft
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
// `weft run bench/bench.weft <copies>`: renders the CommonMark spec's examples, joined and repeated, and prints the size.
|
||||
// scripts/bench.mjs times it from outside the VM, beside markdown-it on the same text.
|
||||
import { args } from "std/process"
|
||||
import { log } from "std/log"
|
||||
import { toHtml } from "../markdown"
|
||||
import { COMMONMARK } from "../tests/cases"
|
||||
|
||||
let copies = try int(args().at(0) ?? "1") else 1
|
||||
let text = COMMONMARK.map(c => c.markdown).join("\n\n").repeat(copies)
|
||||
if copies > 0 {
|
||||
log(toHtml(text).length)
|
||||
}
|
||||
1145
markdown/lib/blocks.weft
Normal file
1145
markdown/lib/blocks.weft
Normal file
File diff suppressed because it is too large
Load diff
242
markdown/lib/chars.weft
Normal file
242
markdown/lib/chars.weft
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
// Character classes and the text transformations CommonMark defines: escapes, entities, URLs.
|
||||
import { entity } from "./entities"
|
||||
import { PUNCTUATION } from "./unicode"
|
||||
|
||||
/// The code unit at `i`, or -1 past the end.
|
||||
export fn unit(s: string, i: int): int {
|
||||
return s.codeUnitAt(i) ?? -1
|
||||
}
|
||||
|
||||
export fn isSpaceOrTab(c: float): boolean {
|
||||
return c == 32 || c == 9
|
||||
}
|
||||
|
||||
export fn isDigit(c: float): boolean {
|
||||
return c >= 48 && c <= 57
|
||||
}
|
||||
|
||||
export fn isAlpha(c: float): boolean {
|
||||
return (c >= 65 && c <= 90) || (c >= 97 && c <= 122)
|
||||
}
|
||||
|
||||
export fn isAlnum(c: float): boolean {
|
||||
return isAlpha(c) || isDigit(c)
|
||||
}
|
||||
|
||||
export fn isHex(c: float): boolean {
|
||||
return isDigit(c) || (c >= 65 && c <= 70) || (c >= 97 && c <= 102)
|
||||
}
|
||||
|
||||
/// ASCII punctuation: what a backslash can escape.
|
||||
export fn isAsciiPunct(c: float): boolean {
|
||||
return (c >= 33 && c <= 47) ||
|
||||
(c >= 58 && c <= 64) ||
|
||||
(c >= 91 && c <= 96) ||
|
||||
(c >= 123 && c <= 126)
|
||||
}
|
||||
|
||||
/// Unicode whitespace: the `Zs` category, tab, line feed, form feed and carriage return.
|
||||
export fn isUnicodeSpace(c: float): boolean {
|
||||
if c < 128 {
|
||||
return c == 32 || c == 9 || c == 10 || c == 12 || c == 13
|
||||
}
|
||||
return c == 0xa0 ||
|
||||
c == 0x1680 ||
|
||||
(c >= 0x2000 && c <= 0x200a) ||
|
||||
c == 0x202f ||
|
||||
c == 0x205f ||
|
||||
c == 0x3000
|
||||
}
|
||||
|
||||
/// Unicode punctuation: the `P` and `S` categories.
|
||||
export fn isUnicodePunct(c: float): boolean {
|
||||
if c < 128 {
|
||||
return isAsciiPunct(c)
|
||||
}
|
||||
let mut lo = 0
|
||||
let mut hi = PUNCTUATION.length.div(2) - 1
|
||||
while lo <= hi {
|
||||
let mid = (lo + hi) >> 1
|
||||
if c < PUNCTUATION[mid * 2] {
|
||||
hi = mid - 1
|
||||
} else if c > PUNCTUATION[mid * 2 + 1] {
|
||||
lo = mid + 1
|
||||
} else {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/// Whether `s` holds only spaces and tabs (or nothing).
|
||||
export fn isBlank(s: string): boolean {
|
||||
for i of 0..s.length {
|
||||
let c = s.codeUnitAt(i) ?? 32
|
||||
if c != 32 && c != 9 && c != 10 && c != 13 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
/// The code point ending just before `pos`, or a line feed at the start of `s`.
|
||||
export fn codePointBefore(s: string, pos: int): int {
|
||||
if pos <= 0 {
|
||||
return 10
|
||||
}
|
||||
let low = s.codeUnitAt(pos - 1) ?? 10
|
||||
if low >= 0xdc00 && low <= 0xdfff && pos >= 2 {
|
||||
let high = s.codeUnitAt(pos - 2) ?? 0
|
||||
if high >= 0xd800 && high <= 0xdbff {
|
||||
return (high - 0xd800) * 0x400 + (low - 0xdc00) + 0x10000
|
||||
}
|
||||
}
|
||||
return low
|
||||
}
|
||||
|
||||
/// The character reference starting at the `&` at `pos`: its text and where it ends, or `null` when there is none.
|
||||
export fn scanEntity(s: string, pos: int): [string, int]? {
|
||||
let mut i = pos + 1
|
||||
if s.codeUnitAt(i) == 35 {
|
||||
i += 1
|
||||
let c = s.codeUnitAt(i) ?? -1
|
||||
let hex = c == 120 || c == 88
|
||||
if hex {
|
||||
i += 1
|
||||
}
|
||||
let start = i
|
||||
while i - start < (hex ? 6 : 7) &&
|
||||
(hex ? isHex(s.codeUnitAt(i) ?? -1) : isDigit(s.codeUnitAt(i) ?? -1)) {
|
||||
i += 1
|
||||
}
|
||||
if i == start || s.codeUnitAt(i) != 59 {
|
||||
return null
|
||||
}
|
||||
let n = try int(s.slice(start, i), hex ? 16 : 10) else 0
|
||||
let valid = n != 0 && n <= 0x10ffff && !(n >= 0xd800 && n <= 0xdfff)
|
||||
return [String.fromCodePoint(valid ? n : 0xfffd), i + 1]
|
||||
}
|
||||
let start = i
|
||||
while i - start < 32 && isAlnum(s.codeUnitAt(i) ?? -1) {
|
||||
i += 1
|
||||
}
|
||||
if i == start || s.codeUnitAt(i) != 59 {
|
||||
return null
|
||||
}
|
||||
let text = entity(s.slice(start, i))
|
||||
if text == null {
|
||||
return null
|
||||
}
|
||||
return [text, i + 1]
|
||||
}
|
||||
|
||||
/// `s` with its backslash escapes and character references replaced by what they stand for.
|
||||
export fn unescape(s: string): string {
|
||||
if !s.includes("\\") && !s.includes("&") {
|
||||
return s
|
||||
}
|
||||
let mut parts: string[] = []
|
||||
let mut from = 0
|
||||
let mut i = 0
|
||||
while i < s.length {
|
||||
let c = s.codeUnitAt(i) ?? 0
|
||||
if c == 92 && isAsciiPunct(s.codeUnitAt(i + 1) ?? 0) {
|
||||
parts.push(s.slice(from, i))
|
||||
from = i + 1
|
||||
i += 2
|
||||
} else if c == 38 {
|
||||
let found = scanEntity(s, i)
|
||||
if found != null {
|
||||
parts.push(s.slice(from, i))
|
||||
parts.push(found[0])
|
||||
i = found[1]
|
||||
from = i
|
||||
} else {
|
||||
i += 1
|
||||
}
|
||||
} else {
|
||||
i += 1
|
||||
}
|
||||
}
|
||||
parts.push(s.slice(from))
|
||||
return parts.join("")
|
||||
}
|
||||
|
||||
/// `s` escaped for HTML text and attribute values.
|
||||
export fn escapeHtml(s: string): string {
|
||||
let mut i = 0
|
||||
while i < s.length {
|
||||
let c = s.codeUnitAt(i) ?? 0
|
||||
if c == 38 || c == 60 || c == 62 || c == 34 {
|
||||
break
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
if i == s.length {
|
||||
return s
|
||||
}
|
||||
let mut parts: string[] = [s.slice(0, i)]
|
||||
let mut from = i
|
||||
while i < s.length {
|
||||
let c = s.codeUnitAt(i) ?? 0
|
||||
let rep = c == 38 ? "&" : c == 60 ? "<" : c == 62 ? ">" : c == 34 ? """ : ""
|
||||
if rep != "" {
|
||||
parts.push(s.slice(from, i))
|
||||
parts.push(rep)
|
||||
from = i + 1
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
parts.push(s.slice(from))
|
||||
return parts.join("")
|
||||
}
|
||||
|
||||
fn keptInUrl(c: int): boolean {
|
||||
// Letters, digits and what URLs use as syntax; everything else is percent-encoded.
|
||||
return isAlnum(c) || (c < 128 && ";/?:@&=+$,-_.!~*'()#".includes(String.fromCharCode(c)))
|
||||
}
|
||||
|
||||
/// `url` percent-encoded for an `href`: what is already encoded stays, and a lone surrogate becomes U+FFFD.
|
||||
export fn normalizeUrl(url: string): string {
|
||||
let mut parts: string[] = []
|
||||
let mut from = 0
|
||||
let mut i = 0
|
||||
while i < url.length {
|
||||
let c = url.codeUnitAt(i) ?? 0
|
||||
if c == 37 && isHex(url.codeUnitAt(i + 1) ?? -1) && isHex(url.codeUnitAt(i + 2) ?? -1) {
|
||||
i += 3
|
||||
} else if keptInUrl(c) {
|
||||
i += 1
|
||||
} else {
|
||||
parts.push(url.slice(from, i))
|
||||
let point = url.codePointAt(i) ?? 0xfffd
|
||||
let width = point > 0xffff ? 2 : 1
|
||||
let safe = point >= 0xd800 && point <= 0xdfff ? 0xfffd : point
|
||||
for b of Bytes.fromUtf8(String.fromCodePoint(safe)).toList() {
|
||||
parts.push("%" + str(b, { radix: 16 }).toUpperCase().padStart(2, "0"))
|
||||
}
|
||||
i += width
|
||||
from = i
|
||||
}
|
||||
}
|
||||
parts.push(url.slice(from))
|
||||
return parts.join("")
|
||||
}
|
||||
|
||||
/// A link label as references are matched: case-folded, with whitespace collapsed.
|
||||
export fn normalizeLabel(label: string): string {
|
||||
let mut out: string[] = []
|
||||
let mut space = false
|
||||
for ch of label.trim() {
|
||||
if ch == " " || ch == "\t" || ch == "\n" || ch == "\r" {
|
||||
space = true
|
||||
} else {
|
||||
if space {
|
||||
out.push(" ")
|
||||
space = false
|
||||
}
|
||||
out.push(ch)
|
||||
}
|
||||
}
|
||||
return out.join("").toLowerCase().toUpperCase()
|
||||
}
|
||||
23
markdown/lib/entities.weft
Normal file
23
markdown/lib/entities.weft
Normal file
File diff suppressed because one or more lines are too long
1397
markdown/lib/inlines.weft
Normal file
1397
markdown/lib/inlines.weft
Normal file
File diff suppressed because it is too large
Load diff
255
markdown/lib/render.weft
Normal file
255
markdown/lib/render.weft
Normal file
|
|
@ -0,0 +1,255 @@
|
|||
// The HTML renderer: the output CommonMark's and GFM's spec tests expect.
|
||||
import type {
|
||||
Align,
|
||||
Block,
|
||||
Document,
|
||||
HtmlMode,
|
||||
Inline,
|
||||
ListItem,
|
||||
Options,
|
||||
TableCell,
|
||||
UrlKind,
|
||||
} from "../ast"
|
||||
import { escapeHtml } from "./chars"
|
||||
import { DEFAULT_SCHEMES, isSafeUrl, sanitize } from "./sanitize"
|
||||
|
||||
type Out = {
|
||||
parts: string[],
|
||||
atLineStart: boolean,
|
||||
mode: HtmlMode,
|
||||
schemes: string[],
|
||||
offset: float,
|
||||
target: string?,
|
||||
}
|
||||
|
||||
fn put(mut o: Out, text: string) {
|
||||
if text != "" {
|
||||
o.parts.push(text)
|
||||
o.atLineStart = text.endsWith("\n")
|
||||
}
|
||||
}
|
||||
|
||||
fn targetOf(o: Out): string? {
|
||||
return o.target
|
||||
}
|
||||
|
||||
fn isEscaping(o: Out): boolean {
|
||||
return o.mode == "escape"
|
||||
}
|
||||
|
||||
fn isAtLineStart(o: Out): boolean {
|
||||
return o.atLineStart
|
||||
}
|
||||
|
||||
fn cr(mut o: Out) {
|
||||
if !isAtLineStart(o) {
|
||||
put(&o, "\n")
|
||||
}
|
||||
}
|
||||
|
||||
fn rawHtml(o: Out, html: string): string {
|
||||
return o.mode == "escape" ? escapeHtml(html) : html
|
||||
}
|
||||
|
||||
fn plainText(inlines: Inline[]): string {
|
||||
let mut out: string[] = []
|
||||
for n of inlines {
|
||||
match n {
|
||||
Str { text } => out.push(text),
|
||||
SoftBreak | HardBreak => out.push(" "),
|
||||
CodeSpan { code } => out.push(code),
|
||||
Emphasis { children } | Strong { children } | Strikethrough { children } => out.push(
|
||||
plainText(children),
|
||||
),
|
||||
Link { children } => out.push(plainText(children)),
|
||||
Picture { alt } => out.push(plainText(alt)),
|
||||
HtmlInline { html } => out.push(html),
|
||||
}
|
||||
}
|
||||
return out.join("")
|
||||
}
|
||||
|
||||
fn wrap(mut o: Out, tag: string, children: Inline[], options: Options): void {
|
||||
put(&o, `<${tag}>`)
|
||||
renderInlines(&o, children, options)
|
||||
put(&o, `</${tag}>`)
|
||||
}
|
||||
|
||||
/// The URL to render, or `null` when it isn't safe to.
|
||||
fn finalUrl(url: string, kind: UrlKind, o: Out, options: Options): string? {
|
||||
let rewrite = options.rewriteUrl
|
||||
let rewritten = rewrite != null ? rewrite(url, kind) ?? url : url
|
||||
return o.mode == "trust" || isSafeUrl(rewritten, o.schemes, kind == "image") ? rewritten : null
|
||||
}
|
||||
|
||||
fn renderInlines(mut o: Out, inlines: Inline[], options: Options): void {
|
||||
for n of inlines {
|
||||
match n {
|
||||
Str { text } => put(&o, escapeHtml(text)),
|
||||
SoftBreak => put(&o, "\n"),
|
||||
HardBreak => put(&o, "<br />\n"),
|
||||
CodeSpan { code } => put(&o, `<code>${escapeHtml(code)}</code>`),
|
||||
Emphasis { children } => wrap(&o, "em", children, options),
|
||||
Strong { children } => wrap(&o, "strong", children, options),
|
||||
Strikethrough { children } => wrap(&o, "del", children, options),
|
||||
HtmlInline { html } => {
|
||||
let raw = rawHtml(o, html)
|
||||
put(&o, raw)
|
||||
},
|
||||
Link { url, title, children } => {
|
||||
let href = finalUrl(url, "link", o, options)
|
||||
if href == null {
|
||||
renderInlines(&o, children, options)
|
||||
} else {
|
||||
let titled = title != null && title != "" ? ` title="${escapeHtml(title)}"` : ""
|
||||
let linkTarget = targetOf(o)
|
||||
let target = linkTarget != null ? ` target="${escapeHtml(linkTarget)}"` : ""
|
||||
let rel = linkTarget == "_blank" ? " rel=\"noopener noreferrer\"" : ""
|
||||
put(&o, `<a href="${escapeHtml(href)}"${titled}${target}${rel}>`)
|
||||
renderInlines(&o, children, options)
|
||||
put(&o, "</a>")
|
||||
}
|
||||
},
|
||||
Picture { url, title, alt } => {
|
||||
let src = finalUrl(url, "image", o, options)
|
||||
if src == null {
|
||||
put(&o, escapeHtml(plainText(alt)))
|
||||
} else {
|
||||
let titled = title != null && title != "" ? ` title="${escapeHtml(title)}"` : ""
|
||||
put(&o, `<img src="${escapeHtml(src)}" alt="${escapeHtml(plainText(alt))}"${titled} />`)
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn renderItem(mut o: Out, item: ListItem, tight: boolean, options: Options): void {
|
||||
put(&o, "<li>")
|
||||
if item.checked != null {
|
||||
put(
|
||||
&o,
|
||||
item.checked
|
||||
? "<input checked=\"\" disabled=\"\" type=\"checkbox\"> "
|
||||
: "<input disabled=\"\" type=\"checkbox\"> ",
|
||||
)
|
||||
}
|
||||
renderBlocks(&o, item.children, tight, options)
|
||||
put(&o, "</li>")
|
||||
cr(&o)
|
||||
}
|
||||
|
||||
fn renderCells(mut o: Out, tag: string, cells: TableCell[], align: Align[], options: Options) {
|
||||
put(&o, "<tr>\n")
|
||||
for [i, cell] of cells.entries() {
|
||||
let a = align.at(i) ?? null
|
||||
put(&o, a != null ? `<${tag} align="${a}">` : `<${tag}>`)
|
||||
renderInlines(&o, cell.children, options)
|
||||
put(&o, `</${tag}>\n`)
|
||||
}
|
||||
put(&o, "</tr>\n")
|
||||
}
|
||||
|
||||
fn renderBlocks(mut o: Out, blocks: Block[], tight: boolean, options: Options): void {
|
||||
for block of blocks {
|
||||
match block {
|
||||
Paragraph { children } => {
|
||||
if tight {
|
||||
renderInlines(&o, children, options)
|
||||
} else {
|
||||
cr(&o)
|
||||
wrap(&o, "p", children, options)
|
||||
cr(&o)
|
||||
}
|
||||
},
|
||||
Heading { level, children } => {
|
||||
let shifted = level + o.offset
|
||||
let shown = shifted > 6 ? 6 : shifted < 1 ? 1 : shifted
|
||||
cr(&o)
|
||||
wrap(&o, `h${shown}`, children, options)
|
||||
cr(&o)
|
||||
},
|
||||
ThematicBreak => {
|
||||
cr(&o)
|
||||
put(&o, "<hr />")
|
||||
cr(&o)
|
||||
},
|
||||
CodeBlock { info, code } => {
|
||||
cr(&o)
|
||||
let language = info.trim().split(" ")[0].split("\t")[0]
|
||||
put(
|
||||
&o,
|
||||
language != "" ? `<pre><code class="language-${escapeHtml(language)}">` : "<pre><code>",
|
||||
)
|
||||
put(&o, escapeHtml(code))
|
||||
put(&o, "</code></pre>")
|
||||
cr(&o)
|
||||
},
|
||||
HtmlBlock { html } => {
|
||||
cr(&o)
|
||||
if isEscaping(o) {
|
||||
put(&o, `<p>${escapeHtml(html)}</p>`)
|
||||
} else {
|
||||
put(&o, html)
|
||||
}
|
||||
cr(&o)
|
||||
},
|
||||
Blockquote { children } => {
|
||||
cr(&o)
|
||||
put(&o, "<blockquote>\n")
|
||||
renderBlocks(&o, children, false, options)
|
||||
cr(&o)
|
||||
put(&o, "</blockquote>")
|
||||
cr(&o)
|
||||
},
|
||||
BulletList { tight: itemsTight, items } => {
|
||||
cr(&o)
|
||||
put(&o, "<ul>\n")
|
||||
for item of items {
|
||||
renderItem(&o, item, itemsTight, options)
|
||||
}
|
||||
put(&o, "</ul>")
|
||||
cr(&o)
|
||||
},
|
||||
OrderedList { tight: itemsTight, start, items } => {
|
||||
cr(&o)
|
||||
put(&o, start != 1 ? `<ol start="${start}">\n` : "<ol>\n")
|
||||
for item of items {
|
||||
renderItem(&o, item, itemsTight, options)
|
||||
}
|
||||
put(&o, "</ol>")
|
||||
cr(&o)
|
||||
},
|
||||
Table { align, head, rows } => {
|
||||
cr(&o)
|
||||
put(&o, "<table>\n<thead>\n")
|
||||
renderCells(&o, "th", head, align, options)
|
||||
put(&o, "</thead>\n")
|
||||
if rows.length > 0 {
|
||||
put(&o, "<tbody>\n")
|
||||
for row of rows {
|
||||
renderCells(&o, "td", row, align, options)
|
||||
}
|
||||
put(&o, "</tbody>\n")
|
||||
}
|
||||
put(&o, "</table>")
|
||||
cr(&o)
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Renders a document to HTML.
|
||||
export fn renderDocument(doc: Document, options: Options): string {
|
||||
let mode = options.html ?? "escape"
|
||||
let mut o: Out = {
|
||||
parts: [],
|
||||
atLineStart: true,
|
||||
mode,
|
||||
schemes: options.schemes ?? DEFAULT_SCHEMES,
|
||||
offset: options.headingOffset ?? 0,
|
||||
target: options.linkTarget,
|
||||
}
|
||||
renderBlocks(&o, doc.children, false, options)
|
||||
let html = o.parts.join("")
|
||||
return mode == "sanitize" ? sanitize(html, options.policy) : html
|
||||
}
|
||||
403
markdown/lib/sanitize.weft
Normal file
403
markdown/lib/sanitize.weft
Normal file
|
|
@ -0,0 +1,403 @@
|
|||
// The HTML sanitiser: an allowlist over a tokenizer. What comes out is well formed and holds only allowed tags and
|
||||
// attributes, every text escaped, so nothing a browser parses differently can bring a removed element back.
|
||||
import type { Policy } from "../ast"
|
||||
import { escapeHtml, isAlpha, scanEntity } from "./chars"
|
||||
|
||||
let DEFAULT_TAGS = [
|
||||
"a",
|
||||
"abbr",
|
||||
"b",
|
||||
"blockquote",
|
||||
"br",
|
||||
"code",
|
||||
"dd",
|
||||
"del",
|
||||
"details",
|
||||
"div",
|
||||
"dl",
|
||||
"dt",
|
||||
"em",
|
||||
"figcaption",
|
||||
"figure",
|
||||
"h1",
|
||||
"h2",
|
||||
"h3",
|
||||
"h4",
|
||||
"h5",
|
||||
"h6",
|
||||
"hr",
|
||||
"i",
|
||||
"img",
|
||||
"input",
|
||||
"ins",
|
||||
"kbd",
|
||||
"li",
|
||||
"mark",
|
||||
"ol",
|
||||
"p",
|
||||
"pre",
|
||||
"q",
|
||||
"s",
|
||||
"samp",
|
||||
"small",
|
||||
"span",
|
||||
"strong",
|
||||
"sub",
|
||||
"summary",
|
||||
"sup",
|
||||
"table",
|
||||
"tbody",
|
||||
"td",
|
||||
"tfoot",
|
||||
"th",
|
||||
"thead",
|
||||
"tr",
|
||||
"u",
|
||||
"ul",
|
||||
"var",
|
||||
]
|
||||
|
||||
let DEFAULT_ATTRIBUTES = Map<string, string[]>(
|
||||
[
|
||||
["a", ["href", "title", "target"]],
|
||||
["img", ["src", "alt", "title", "width", "height"]],
|
||||
["ol", ["start"]],
|
||||
["code", ["class"]],
|
||||
["td", ["align", "colspan", "rowspan"]],
|
||||
["th", ["align", "colspan", "rowspan", "scope"]],
|
||||
["input", ["type", "checked", "disabled"]],
|
||||
["details", ["open"]],
|
||||
],
|
||||
)
|
||||
|
||||
/// URL schemes allowed unless the options say otherwise.
|
||||
export let DEFAULT_SCHEMES = ["http", "https", "mailto", "tel"]
|
||||
|
||||
let VOID_TAGS = Set(
|
||||
[
|
||||
"br",
|
||||
"hr",
|
||||
"img",
|
||||
"input",
|
||||
"wbr",
|
||||
"area",
|
||||
"base",
|
||||
"col",
|
||||
"embed",
|
||||
"link",
|
||||
"meta",
|
||||
"param",
|
||||
"source",
|
||||
"track",
|
||||
],
|
||||
)
|
||||
|
||||
/// Elements removed with everything inside them.
|
||||
let DROPPED = Set(
|
||||
[
|
||||
"script",
|
||||
"style",
|
||||
"template",
|
||||
"iframe",
|
||||
"object",
|
||||
"embed",
|
||||
"applet",
|
||||
"noscript",
|
||||
"noembed",
|
||||
"noframes",
|
||||
"title",
|
||||
"textarea",
|
||||
"xmp",
|
||||
"plaintext",
|
||||
"svg",
|
||||
"math",
|
||||
"select",
|
||||
"frameset",
|
||||
"head",
|
||||
],
|
||||
)
|
||||
|
||||
fn isDataImage(url: string): boolean {
|
||||
let lower = url.slice(0, 16).toLowerCase()
|
||||
return lower.startsWith("data:image/png") ||
|
||||
lower.startsWith("data:image/gif") ||
|
||||
lower.startsWith("data:image/jpeg") ||
|
||||
lower.startsWith("data:image/jpg") ||
|
||||
lower.startsWith("data:image/webp")
|
||||
}
|
||||
|
||||
/// Whether a URL is relative or uses one of `schemes`. Browsers ignore tabs, line breaks and leading control characters
|
||||
/// in a scheme, so they are ignored here too.
|
||||
export fn isSafeUrl(url: string, schemes: string[], image: boolean): boolean {
|
||||
let mut compact: string[] = []
|
||||
for ch of url {
|
||||
let c = ch.codeUnitAt(0) ?? 0
|
||||
if c > 32 && c != 127 {
|
||||
compact.push(ch)
|
||||
}
|
||||
}
|
||||
let text = compact.join("")
|
||||
let mut colon = -1
|
||||
for i of 0..text.length {
|
||||
let c = text.codeUnitAt(i) ?? 0
|
||||
if c == 58 {
|
||||
colon = i
|
||||
break
|
||||
}
|
||||
if c == 47 || c == 63 || c == 35 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if colon < 0 {
|
||||
return true
|
||||
}
|
||||
if image && isDataImage(text) {
|
||||
return true
|
||||
}
|
||||
return schemes.includes(text.slice(0, colon).toLowerCase())
|
||||
}
|
||||
|
||||
/// Text decoded from an attribute value: its character references replaced.
|
||||
fn decodeAttribute(value: string): string {
|
||||
if !value.includes("&") {
|
||||
return value
|
||||
}
|
||||
let mut parts: string[] = []
|
||||
let mut from = 0
|
||||
let mut i = value.indexOf("&")
|
||||
while i >= 0 {
|
||||
let found = scanEntity(value, i)
|
||||
if found != null {
|
||||
parts.push(value.slice(from, i))
|
||||
parts.push(found[0])
|
||||
from = found[1]
|
||||
i = value.indexOf("&", from)
|
||||
} else {
|
||||
i = value.indexOf("&", i + 1)
|
||||
}
|
||||
}
|
||||
parts.push(value.slice(from))
|
||||
return parts.join("")
|
||||
}
|
||||
|
||||
/// Text as it may appear between tags: a character reference stays, anything else that means something is escaped.
|
||||
fn cleanText(text: string): string {
|
||||
if !text.includes("&") && !text.includes("<") && !text.includes(">") {
|
||||
return text
|
||||
}
|
||||
let mut parts: string[] = []
|
||||
let mut from = 0
|
||||
let mut i = 0
|
||||
while i < text.length {
|
||||
let c = text.codeUnitAt(i) ?? 0
|
||||
if c == 38 {
|
||||
let found = scanEntity(text, i)
|
||||
if found != null {
|
||||
i = found[1]
|
||||
} else {
|
||||
parts.push(text.slice(from, i))
|
||||
parts.push("&")
|
||||
i += 1
|
||||
from = i
|
||||
}
|
||||
} else if c == 60 || c == 62 {
|
||||
parts.push(text.slice(from, i))
|
||||
parts.push(c == 60 ? "<" : ">")
|
||||
i += 1
|
||||
from = i
|
||||
} else {
|
||||
i += 1
|
||||
}
|
||||
}
|
||||
parts.push(text.slice(from))
|
||||
return parts.join("")
|
||||
}
|
||||
|
||||
type Attr = { name: string, value: string }
|
||||
|
||||
type Tag = { name: string, attrs: Attr[], end: int }
|
||||
|
||||
fn isSpace(c: int): boolean {
|
||||
return c == 32 || c == 9 || c == 10 || c == 12 || c == 13
|
||||
}
|
||||
|
||||
/// Reads the tag starting at `<` at `pos` (its name begins at `start`).
|
||||
fn readTag(html: string, start: int): Tag {
|
||||
let mut i = start
|
||||
while i < html.length {
|
||||
let c = html.codeUnitAt(i) ?? 32
|
||||
if isSpace(c) || c == 62 || c == 47 {
|
||||
break
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
let name = html.slice(start, i).toLowerCase()
|
||||
let mut attrs: Attr[] = []
|
||||
while i < html.length {
|
||||
let c = html.codeUnitAt(i) ?? 32
|
||||
if c == 62 {
|
||||
return { name, attrs, end: i + 1 }
|
||||
}
|
||||
if isSpace(c) || c == 47 {
|
||||
i += 1
|
||||
continue
|
||||
}
|
||||
let nameStart = i
|
||||
i += 1
|
||||
while i < html.length {
|
||||
let u = html.codeUnitAt(i) ?? 32
|
||||
if isSpace(u) || u == 61 || u == 62 || u == 47 {
|
||||
break
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
let attr = html.slice(nameStart, i).toLowerCase()
|
||||
while isSpace(html.codeUnitAt(i) ?? 0) {
|
||||
i += 1
|
||||
}
|
||||
let mut value = ""
|
||||
if html.codeUnitAt(i) == 61 {
|
||||
i += 1
|
||||
while isSpace(html.codeUnitAt(i) ?? 0) {
|
||||
i += 1
|
||||
}
|
||||
let q = html.codeUnitAt(i) ?? -1
|
||||
if q == 34 || q == 39 {
|
||||
let close = html.indexOf(String.fromCharCode(q), i + 1)
|
||||
let stop = close < 0 ? html.length : close
|
||||
value = html.slice(i + 1, stop)
|
||||
i = close < 0 ? html.length : close + 1
|
||||
} else {
|
||||
let from = i
|
||||
while i < html.length {
|
||||
let u = html.codeUnitAt(i) ?? 32
|
||||
if isSpace(u) || u == 62 {
|
||||
break
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
value = html.slice(from, i)
|
||||
}
|
||||
}
|
||||
attrs.push({ name: attr, value: decodeAttribute(value) })
|
||||
}
|
||||
return { name, attrs, end: html.length }
|
||||
}
|
||||
|
||||
/// Where the content of the dropped element `name` ends: after its closing tag, or at the end of the text.
|
||||
fn skipElement(html: string, name: string, from: int): int {
|
||||
let mut i = html.indexOf("</", from)
|
||||
while i >= 0 {
|
||||
if html.slice(i + 2, i + 2 + name.length).toLowerCase() == name {
|
||||
let after = html.codeUnitAt(i + 2 + name.length) ?? 62
|
||||
if isSpace(after) || after == 62 || after == 47 {
|
||||
let close = html.indexOf(">", i)
|
||||
return close < 0 ? html.length : close + 1
|
||||
}
|
||||
}
|
||||
i = html.indexOf("</", i + 2)
|
||||
}
|
||||
return html.length
|
||||
}
|
||||
|
||||
fn keepAttribute(tag: string, a: Attr, schemes: string[]): boolean {
|
||||
if a.name == "href" || a.name == "src" {
|
||||
return isSafeUrl(a.value, schemes, tag == "img")
|
||||
}
|
||||
if a.name == "target" {
|
||||
return a.value == "_blank"
|
||||
}
|
||||
if a.name == "type" && tag == "input" {
|
||||
return a.value.toLowerCase() == "checkbox"
|
||||
}
|
||||
if a.name == "class" {
|
||||
return a.value.startsWith("language-") && !a.value.includes(" ")
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
/// The only input kept is a task list's checkbox, always disabled: a text field in rendered content can phish.
|
||||
fn isCheckbox(tag: Tag): boolean {
|
||||
return tag.attrs.some(a => a.name == "type" && a.value.toLowerCase() == "checkbox")
|
||||
}
|
||||
|
||||
fn openTag(tag: Tag, allowed: string[], schemes: string[]): string {
|
||||
let mut out: string[] = ["<", tag.name]
|
||||
let mut blank = false
|
||||
let mut seen = Set<string>()
|
||||
for a of tag.attrs {
|
||||
if allowed.includes(a.name) && !seen.has(a.name) && keepAttribute(tag.name, a, schemes) {
|
||||
seen.add(a.name)
|
||||
out.push(` ${a.name}="${escapeHtml(a.value)}"`)
|
||||
blank = blank || (a.name == "target" && a.value == "_blank")
|
||||
}
|
||||
}
|
||||
if blank {
|
||||
out.push(" rel=\"noopener noreferrer\"")
|
||||
}
|
||||
if tag.name == "input" && !seen.has("disabled") {
|
||||
out.push(" disabled=\"\"")
|
||||
}
|
||||
out.push(VOID_TAGS.has(tag.name) ? " />" : ">")
|
||||
return out.join("")
|
||||
}
|
||||
|
||||
/// Removes everything `policy` doesn't allow from `html`: scripts, styles, event handlers and unsafe URLs by default.
|
||||
/// The result is well formed: stray closing tags are dropped and open ones closed.
|
||||
///
|
||||
/// ```weft
|
||||
/// sanitize("<p onclick=\"x()\">hi<script>bad()</script></p>") // "<p>hi</p>"
|
||||
/// ```
|
||||
export fn sanitize(html: string, policy?: Policy): string {
|
||||
let tags = Set(policy?.tags ?? DEFAULT_TAGS)
|
||||
let attributes = policy?.attributes ?? DEFAULT_ATTRIBUTES
|
||||
let schemes = policy?.schemes ?? DEFAULT_SCHEMES
|
||||
let mut out: string[] = []
|
||||
let mut open: string[] = []
|
||||
let mut i = 0
|
||||
while i < html.length {
|
||||
let lt = html.indexOf("<", i)
|
||||
if lt < 0 {
|
||||
out.push(cleanText(html.slice(i)))
|
||||
break
|
||||
}
|
||||
if lt > i {
|
||||
out.push(cleanText(html.slice(i, lt)))
|
||||
}
|
||||
let next = html.codeUnitAt(lt + 1) ?? -1
|
||||
if html.startsWith("<!--", lt) {
|
||||
let end = html.indexOf("-->", lt + 4)
|
||||
i = end < 0 ? html.length : end + 3
|
||||
} else if next == 33 || next == 63 {
|
||||
let end = html.indexOf(">", lt + 2)
|
||||
i = end < 0 ? html.length : end + 1
|
||||
} else if next == 47 && isAlpha(html.codeUnitAt(lt + 2) ?? -1) {
|
||||
let tag = readTag(html, lt + 2)
|
||||
i = tag.end
|
||||
let at = open.lastIndexOf(tag.name)
|
||||
if at >= 0 {
|
||||
while open.length > at {
|
||||
out.push(`</${open.pop() ?? ""}>`)
|
||||
}
|
||||
}
|
||||
} else if isAlpha(next) {
|
||||
let tag = readTag(html, lt + 1)
|
||||
i = tag.end
|
||||
if DROPPED.has(tag.name) {
|
||||
i = VOID_TAGS.has(tag.name) ? i : skipElement(html, tag.name, i)
|
||||
} else if tags.has(tag.name) && (tag.name != "input" || isCheckbox(tag)) {
|
||||
out.push(openTag(tag, attributes.get(tag.name) ?? [], schemes))
|
||||
if !VOID_TAGS.has(tag.name) {
|
||||
open.push(tag.name)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
out.push("<")
|
||||
i = lt + 1
|
||||
}
|
||||
}
|
||||
while open.length > 0 {
|
||||
out.push(`</${open.pop() ?? ""}>`)
|
||||
}
|
||||
return out.join("")
|
||||
}
|
||||
50
markdown/lib/unicode.weft
Normal file
50
markdown/lib/unicode.weft
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
// Generated by scripts/gen-tables.mjs from the engine's Unicode tables (general categories P and S). Don't edit it.
|
||||
|
||||
/// Sorted, inclusive ranges of code points: start, end, start, end, …
|
||||
export let PUNCTUATION: int[] = [
|
||||
33, 47, 58, 64, 91, 96, 123, 126, 161, 169, 171, 172, 174, 177, 180, 180,
|
||||
182, 184, 187, 187, 191, 191, 215, 215, 247, 247, 706, 709, 722, 735, 741, 747,
|
||||
749, 749, 751, 767, 885, 885, 894, 894, 900, 901, 903, 903, 1014, 1014, 1154, 1154,
|
||||
1370, 1375, 1417, 1418, 1421, 1423, 1470, 1470, 1472, 1472, 1475, 1475, 1478, 1478, 1523, 1524,
|
||||
1542, 1551, 1563, 1563, 1565, 1567, 1642, 1645, 1748, 1748, 1758, 1758, 1769, 1769, 1789, 1790,
|
||||
1792, 1805, 2038, 2041, 2046, 2047, 2096, 2110, 2142, 2142, 2184, 2184, 2404, 2405, 2416, 2416,
|
||||
2546, 2547, 2554, 2555, 2557, 2557, 2678, 2678, 2800, 2801, 2928, 2928, 3059, 3066, 3191, 3191,
|
||||
3199, 3199, 3204, 3204, 3407, 3407, 3449, 3449, 3572, 3572, 3647, 3647, 3663, 3663, 3674, 3675,
|
||||
3841, 3863, 3866, 3871, 3892, 3892, 3894, 3894, 3896, 3896, 3898, 3901, 3973, 3973, 4030, 4037,
|
||||
4039, 4044, 4046, 4058, 4170, 4175, 4254, 4255, 4347, 4347, 4960, 4968, 5008, 5017, 5120, 5120,
|
||||
5741, 5742, 5787, 5788, 5867, 5869, 5941, 5942, 6100, 6102, 6104, 6107, 6144, 6154, 6464, 6464,
|
||||
6468, 6469, 6622, 6655, 6686, 6687, 6816, 6822, 6824, 6829, 6990, 6991, 7002, 7018, 7028, 7039,
|
||||
7164, 7167, 7227, 7231, 7294, 7295, 7360, 7367, 7379, 7379, 8125, 8125, 8127, 8129, 8141, 8143,
|
||||
8157, 8159, 8173, 8175, 8189, 8190, 8208, 8231, 8240, 8286, 8314, 8318, 8330, 8334, 8352, 8385,
|
||||
8448, 8449, 8451, 8454, 8456, 8457, 8468, 8468, 8470, 8472, 8478, 8483, 8485, 8485, 8487, 8487,
|
||||
8489, 8489, 8494, 8494, 8506, 8507, 8512, 8516, 8522, 8525, 8527, 8527, 8586, 8587, 8592, 9257,
|
||||
9280, 9290, 9372, 9449, 9472, 10101, 10132, 11123, 11126, 11263, 11493, 11498, 11513, 11516, 11518, 11519,
|
||||
11632, 11632, 11776, 11822, 11824, 11869, 11904, 11929, 11931, 12019, 12032, 12245, 12272, 12287, 12289, 12292,
|
||||
12296, 12320, 12336, 12336, 12342, 12343, 12349, 12351, 12443, 12444, 12448, 12448, 12539, 12539, 12688, 12689,
|
||||
12694, 12703, 12736, 12773, 12783, 12783, 12800, 12830, 12842, 12871, 12880, 12880, 12896, 12927, 12938, 12976,
|
||||
12992, 13311, 19904, 19967, 42128, 42182, 42238, 42239, 42509, 42511, 42611, 42611, 42622, 42622, 42738, 42743,
|
||||
42752, 42774, 42784, 42785, 42889, 42890, 43048, 43051, 43062, 43065, 43124, 43127, 43214, 43215, 43256, 43258,
|
||||
43260, 43260, 43310, 43311, 43359, 43359, 43457, 43469, 43486, 43487, 43612, 43615, 43639, 43641, 43742, 43743,
|
||||
43760, 43761, 43867, 43867, 43882, 43883, 44011, 44011, 64297, 64297, 64434, 64466, 64830, 64847, 64912, 64913,
|
||||
64968, 64975, 65020, 65023, 65040, 65049, 65072, 65106, 65108, 65126, 65128, 65131, 65281, 65295, 65306, 65312,
|
||||
65339, 65344, 65371, 65381, 65504, 65510, 65512, 65518, 65532, 65533, 65792, 65794, 65847, 65855, 65913, 65929,
|
||||
65932, 65934, 65936, 65948, 65952, 65952, 66000, 66044, 66463, 66463, 66512, 66512, 66927, 66927, 67671, 67671,
|
||||
67703, 67704, 67871, 67871, 67903, 67903, 68176, 68184, 68223, 68223, 68296, 68296, 68336, 68342, 68409, 68415,
|
||||
68505, 68508, 68974, 68974, 69006, 69007, 69293, 69293, 69328, 69336, 69461, 69465, 69510, 69513, 69703, 69709,
|
||||
69819, 69820, 69822, 69825, 69952, 69955, 70004, 70005, 70085, 70088, 70093, 70093, 70107, 70107, 70109, 70111,
|
||||
70200, 70205, 70313, 70313, 70612, 70613, 70615, 70616, 70731, 70735, 70746, 70747, 70749, 70749, 70854, 70854,
|
||||
71105, 71127, 71233, 71235, 71264, 71276, 71353, 71353, 71484, 71487, 71739, 71739, 72004, 72006, 72162, 72162,
|
||||
72255, 72262, 72346, 72348, 72350, 72354, 72448, 72457, 72673, 72673, 72769, 72773, 72816, 72817, 73463, 73464,
|
||||
73539, 73551, 73685, 73713, 73727, 73727, 74864, 74868, 77809, 77810, 92782, 92783, 92917, 92917, 92983, 92991,
|
||||
92996, 92997, 93549, 93551, 93847, 93850, 94178, 94178, 113820, 113820, 113823, 113823, 117760, 117999, 118010, 118012,
|
||||
118016, 118451, 118458, 118480, 118496, 118512, 118608, 118723, 118784, 119029, 119040, 119078, 119081, 119140, 119146, 119148,
|
||||
119171, 119172, 119180, 119209, 119214, 119274, 119296, 119361, 119365, 119365, 119552, 119638, 120513, 120513, 120539, 120539,
|
||||
120571, 120571, 120597, 120597, 120629, 120629, 120655, 120655, 120687, 120687, 120713, 120713, 120745, 120745, 120771, 120771,
|
||||
120832, 121343, 121399, 121402, 121453, 121460, 121462, 121475, 121477, 121483, 123215, 123215, 123647, 123647, 124415, 124415,
|
||||
125278, 125279, 126124, 126124, 126128, 126128, 126254, 126254, 126704, 126705, 126976, 127019, 127024, 127123, 127136, 127150,
|
||||
127153, 127167, 127169, 127183, 127185, 127221, 127245, 127405, 127462, 127490, 127504, 127547, 127552, 127560, 127568, 127569,
|
||||
127584, 127589, 127744, 128728, 128732, 128748, 128752, 128764, 128768, 128985, 128992, 129003, 129008, 129008, 129024, 129035,
|
||||
129040, 129095, 129104, 129113, 129120, 129159, 129168, 129197, 129200, 129211, 129216, 129217, 129232, 129240, 129280, 129623,
|
||||
129632, 129645, 129648, 129660, 129664, 129674, 129678, 129734, 129736, 129736, 129741, 129756, 129759, 129770, 129775, 129784,
|
||||
129792, 129938, 129940, 130031, 130042, 130042,
|
||||
]
|
||||
156
markdown/markdown.weft
Normal file
156
markdown/markdown.weft
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
// `@weft/markdown`: CommonMark and GitHub Flavored Markdown.
|
||||
import type { Block, Document, Inline, Options, Policy } from "./ast"
|
||||
import { parseDocument } from "./lib/blocks"
|
||||
import { renderDocument } from "./lib/render"
|
||||
import { sanitize as clean } from "./lib/sanitize"
|
||||
|
||||
/// Parses Markdown into its syntax tree. Parsing never fails: any text is a document.
|
||||
///
|
||||
/// ```weft
|
||||
/// let doc = parse("# Hello\n\nSome *text*.")
|
||||
/// doc.children.length // 2
|
||||
/// ```
|
||||
export fn parse(source: string, options: Options = {}): Document {
|
||||
return parseDocument(source, options.gfm ?? true)
|
||||
}
|
||||
|
||||
/// Renders a syntax tree to HTML. Raw HTML is escaped and unsafe URLs are removed unless `options` say otherwise.
|
||||
export fn render(doc: Document, options: Options = {}): string {
|
||||
return renderDocument(doc, options)
|
||||
}
|
||||
|
||||
/// Markdown to HTML, safe for untrusted input by default.
|
||||
///
|
||||
/// ```weft
|
||||
/// toHtml("**bold** <script>alert(1)</script>")
|
||||
/// // "<p><strong>bold</strong> <script>alert(1)</script></p>\n"
|
||||
///
|
||||
/// toHtml("# Title", { headingOffset: 1, linkTarget: "_blank" })
|
||||
/// // "<h2>Title</h2>\n"
|
||||
/// ```
|
||||
export fn toHtml(source: string, options: Options = {}): string {
|
||||
return renderDocument(parseDocument(source, options.gfm ?? true), options)
|
||||
}
|
||||
|
||||
/// Removes everything `policy` doesn't allow from HTML, whatever produced it. See `Policy` for the defaults.
|
||||
///
|
||||
/// ```weft
|
||||
/// sanitize("<a href=\"javascript:x()\" title=\"t\">hi</a>") // "<a title=\"t\">hi</a>"
|
||||
/// ```
|
||||
export fn sanitize(html: string, policy?: Policy): string {
|
||||
return clean(html, policy)
|
||||
}
|
||||
|
||||
fn inlineText(inlines: Inline[]): string {
|
||||
let mut out: string[] = []
|
||||
for n of inlines {
|
||||
match n {
|
||||
Str { text } => out.push(text),
|
||||
SoftBreak => out.push(" "),
|
||||
HardBreak => out.push("\n"),
|
||||
CodeSpan { code } => out.push(code),
|
||||
Emphasis { children } | Strong { children } | Strikethrough { children } => out.push(
|
||||
inlineText(children),
|
||||
),
|
||||
Link { url, children } => {
|
||||
let label = inlineText(children)
|
||||
out.push(label == url || `mailto:${label}` == url ? label : `${label} (${url})`)
|
||||
},
|
||||
Picture { alt } => {
|
||||
let label = inlineText(alt)
|
||||
out.push(label == "" ? "[image]" : `[image: ${label}]`)
|
||||
},
|
||||
HtmlInline => {},
|
||||
}
|
||||
}
|
||||
return out.join("")
|
||||
}
|
||||
|
||||
fn blockText(blocks: Block[], mut out: string[]): void {
|
||||
for block of blocks {
|
||||
match block {
|
||||
Paragraph { children } | Heading { children } => out.push(inlineText(children)),
|
||||
ThematicBreak => out.push("---"),
|
||||
CodeBlock { code } => out.push(code.trimEnd()),
|
||||
HtmlBlock => {},
|
||||
Blockquote { children } => blockText(children, &out),
|
||||
BulletList { items } | OrderedList { items } => {
|
||||
for item of items {
|
||||
let mut inner: string[] = []
|
||||
blockText(item.children, &inner)
|
||||
let mark = item.checked == null ? "- " : item.checked ? "- [x] " : "- [ ] "
|
||||
out.push(mark + inner.join("\n "))
|
||||
}
|
||||
},
|
||||
Table { head, rows } => {
|
||||
out.push(head.map(c => inlineText(c.children)).join(" | "))
|
||||
for row of rows {
|
||||
out.push(row.map(c => inlineText(c.children)).join(" | "))
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The document as plain text, for previews, search and the text part of an email. Links keep their address and images
|
||||
/// become `[image: caption]`.
|
||||
///
|
||||
/// ```weft
|
||||
/// toText(parse("Read [the guide](https://weft.dev)."))
|
||||
/// // "Read the guide (https://weft.dev)."
|
||||
/// ```
|
||||
export fn toText(doc: Document): string {
|
||||
let mut out: string[] = []
|
||||
blockText(doc.children, &out)
|
||||
return out.join("\n\n")
|
||||
}
|
||||
|
||||
fn visitInlines(inlines: Inline[], f: (node: Inline) => void): void {
|
||||
for n of inlines {
|
||||
f(n)
|
||||
match n {
|
||||
Emphasis { children } | Strong { children } | Strikethrough { children } | Link {
|
||||
children,
|
||||
} => visitInlines(children, f),
|
||||
Picture { alt } => visitInlines(alt, f),
|
||||
_ => {},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn visitBlocks(blocks: Block[], f: (node: Inline) => void): void {
|
||||
for block of blocks {
|
||||
match block {
|
||||
Paragraph { children } | Heading { children } => visitInlines(children, f),
|
||||
Blockquote { children } => visitBlocks(children, f),
|
||||
BulletList { items } | OrderedList { items } => {
|
||||
for item of items {
|
||||
visitBlocks(item.children, f)
|
||||
}
|
||||
},
|
||||
Table { head, rows } => {
|
||||
for cell of head {
|
||||
visitInlines(cell.children, f)
|
||||
}
|
||||
for row of rows {
|
||||
for cell of row {
|
||||
visitInlines(cell.children, f)
|
||||
}
|
||||
}
|
||||
},
|
||||
_ => {},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Calls `f` with every inline of the document, in order, outermost first.
|
||||
///
|
||||
/// ```weft
|
||||
/// let mut images: string[] = []
|
||||
/// visit(parse(source), n => {
|
||||
/// if let Picture { url } = n { images.push(url) }
|
||||
/// })
|
||||
/// ```
|
||||
export fn visit(doc: Document, f: (node: Inline) => void) {
|
||||
visitBlocks(doc.children, f)
|
||||
}
|
||||
49
markdown/scripts/bench.mjs
Normal file
49
markdown/scripts/bench.mjs
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// Times @weft/markdown on the VM against markdown-it on Node (JIT and --jitless), on the same text: the CommonMark
|
||||
// spec's examples, joined and repeated. Needs markdown-it: MARKDOWN_IT=<path to its folder>.
|
||||
//
|
||||
// node packages/markdown/scripts/bench.mjs commonmark.json 1,10,50
|
||||
import { spawnSync } from "node:child_process"
|
||||
import { readFileSync, writeFileSync } from "node:fs"
|
||||
import { dirname, join } from "node:path"
|
||||
import { fileURLToPath } from "node:url"
|
||||
|
||||
const root = join(dirname(fileURLToPath(import.meta.url)), "..")
|
||||
const weft = process.env.WEFT_BIN ?? join(root, "../../core/target/release/weft")
|
||||
const spec = process.argv[2]
|
||||
const sizes = (process.argv[3] ?? "1,10,50").split(",").map(Number)
|
||||
const lib = process.env.MARKDOWN_IT
|
||||
|
||||
const script = `
|
||||
const MarkdownIt = require(${JSON.stringify(lib)})
|
||||
const fs = require("fs")
|
||||
const cases = JSON.parse(fs.readFileSync(${JSON.stringify(spec)}, "utf8"))
|
||||
const text = cases.map((c) => c.markdown).join("\\n\\n").repeat(Number(process.argv[2]))
|
||||
const md = new MarkdownIt({ html: false, linkify: true })
|
||||
const start = process.hrtime.bigint()
|
||||
const html = md.render(text)
|
||||
console.log(Number(process.hrtime.bigint() - start) / 1e6, text.length, html.length)
|
||||
`
|
||||
const file = join(process.env.TMPDIR ?? "/workspace/core/target/tools", "bench-markdown-it.cjs")
|
||||
writeFileSync(file, script)
|
||||
|
||||
function ms(cmd, args, env = {}) {
|
||||
const start = process.hrtime.bigint()
|
||||
const r = spawnSync(cmd, args, { cwd: root, encoding: "utf8", env: { ...process.env, ...env } })
|
||||
if (r.status !== 0) throw new Error(r.stderr)
|
||||
return { ms: Number(process.hrtime.bigint() - start) / 1e6, out: r.stdout.trim() }
|
||||
}
|
||||
|
||||
const limits = { WEFT_FUEL: "100000000000", WEFT_MEMORY_MB: "4000" }
|
||||
const base = Math.min(...[0, 1, 2].map(() => ms(weft, ["run", "bench/bench.weft", "0"], limits).ms))
|
||||
console.log(`VM start-up (loading the package and the spec's text): ${base.toFixed(0)} ms`)
|
||||
console.log("size KB | weft VM ms | markdown-it JIT ms | markdown-it --jitless ms | VM / JIT | VM / jitless")
|
||||
for (const n of sizes) {
|
||||
const vm = Math.min(...[0, 1, 2].map(() => ms(weft, ["run", "bench/bench.weft", String(n)], limits).ms)) - base
|
||||
const best = (flags) => {
|
||||
const runs = [0, 1, 2].map(() => ms("node", [...flags, file, String(n)]).out.split(" ").map(Number))
|
||||
return { ms: Math.min(...runs.map((r) => r[0])), size: runs[0][1] }
|
||||
}
|
||||
const jit = best([])
|
||||
const nojit = best(["--jitless"])
|
||||
console.log(`${(jit.size / 1024).toFixed(0).padStart(7)} | ${vm.toFixed(0).padStart(10)} | ${jit.ms.toFixed(1).padStart(18)} | ${nojit.ms.toFixed(1).padStart(24)} | ${(vm / jit.ms).toFixed(1).padStart(8)} | ${(vm / nojit.ms).toFixed(1).padStart(12)}`)
|
||||
}
|
||||
69
markdown/scripts/gen-spec.mjs
Normal file
69
markdown/scripts/gen-spec.mjs
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
// Generates tests/cases.weft from the CommonMark spec's examples and cmark-gfm's spec text.
|
||||
//
|
||||
// curl -o commonmark.json https://spec.commonmark.org/0.31.2/spec.json
|
||||
// curl -o gfm.txt https://raw.githubusercontent.com/github/cmark-gfm/master/test/spec.txt
|
||||
// node packages/markdown/scripts/gen-spec.mjs commonmark.json gfm.txt
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs"
|
||||
import { dirname, join } from "node:path"
|
||||
import { fileURLToPath } from "node:url"
|
||||
|
||||
const tests = join(dirname(fileURLToPath(import.meta.url)), "..", "tests")
|
||||
mkdirSync(tests, { recursive: true })
|
||||
|
||||
const commonmark = JSON.parse(readFileSync(process.argv[2], "utf8")).map((c) => ({
|
||||
id: c.example,
|
||||
section: c.section,
|
||||
markdown: c.markdown,
|
||||
html: c.html,
|
||||
}))
|
||||
|
||||
// cmark-gfm's spec.txt: examples are fenced by 32 backticks, `example` then the extension's name.
|
||||
const gfm = []
|
||||
let section = ""
|
||||
let number = 0
|
||||
const lines = readFileSync(process.argv[3], "utf8").split("\n")
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const heading = lines[i].match(/^#{1,6} (.*)$/)
|
||||
if (heading) section = heading[1]
|
||||
const open = lines[i].match(/^`{32} example(.*)$/)
|
||||
if (!open) continue
|
||||
number++
|
||||
const extensions = open[1].trim().split(/\s+/).filter(Boolean)
|
||||
let j = i + 1
|
||||
const source = []
|
||||
while (lines[j] !== ".") source.push(lines[j++])
|
||||
j++
|
||||
const html = []
|
||||
while (!lines[j].startsWith("`".repeat(32))) html.push(lines[j++])
|
||||
i = j
|
||||
// `disabled` examples describe behaviour cmark-gfm itself doesn't have. `tagfilter` is for pages that embed raw HTML:
|
||||
// here `html: "escape"` and `"sanitize"` already remove those tags, and `"trust"` means what it says.
|
||||
if (extensions.includes("disabled") || extensions.includes("tagfilter")) continue
|
||||
const text = (l) => l.map((x) => x + "\n").join("").replaceAll("→", "\t")
|
||||
gfm.push({ id: number, section, markdown: text(source), html: text(html), extensions })
|
||||
}
|
||||
|
||||
const str = (s) => JSON.stringify(s).replaceAll("\u2028", "\\u2028").replaceAll("\u2029", "\\u2029").replaceAll("$", "\\$")
|
||||
|
||||
const row = (c) =>
|
||||
` { id: ${c.id}, section: ${str(c.section)}, markdown: ${str(c.markdown)}, html: ${str(c.html)}, gfm: ${(c.extensions ?? []).length > 0} },`
|
||||
|
||||
writeFileSync(
|
||||
join(tests, "cases.weft"),
|
||||
`// Generated by scripts/gen-spec.mjs. Don't edit it.
|
||||
|
||||
/// One example. With gfm set it needs GFM's extensions; cmark-gfm runs the others without them.
|
||||
export type Case = { id: number, section: string, markdown: string, html: string, gfm: boolean }
|
||||
|
||||
/// The examples of the CommonMark spec, version 0.31.2.
|
||||
export let COMMONMARK: Case[] = [
|
||||
${commonmark.map(row).join("\n")}
|
||||
]
|
||||
|
||||
/// The examples of the GitHub Flavored Markdown spec.
|
||||
export let GFM: Case[] = [
|
||||
${gfm.map(row).join("\n")}
|
||||
]
|
||||
`,
|
||||
)
|
||||
console.log(`${commonmark.length} CommonMark examples, ${gfm.length} GFM examples`)
|
||||
73
markdown/scripts/gen-tables.mjs
Normal file
73
markdown/scripts/gen-tables.mjs
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
// Generates lib/entities.weft (the HTML5 named character references) and lib/unicode.weft (Unicode
|
||||
// punctuation and symbols, what CommonMark's delimiter rules call punctuation).
|
||||
//
|
||||
// curl -o entities.json https://html.spec.whatwg.org/entities.json
|
||||
// node packages/markdown/scripts/gen-tables.mjs entities.json
|
||||
import { readFileSync, writeFileSync } from "node:fs"
|
||||
import { dirname, join } from "node:path"
|
||||
import { fileURLToPath } from "node:url"
|
||||
|
||||
const lib = join(dirname(fileURLToPath(import.meta.url)), "..", "lib")
|
||||
|
||||
const entities = JSON.parse(readFileSync(process.argv[2], "utf8"))
|
||||
const pairs = Object.entries(entities)
|
||||
.filter(([name]) => name.endsWith(";"))
|
||||
.map(([name, v]) => `${name.slice(1, -1)}=${v.codepoints.map((c) => c.toString(16)).join(",")}`)
|
||||
.sort()
|
||||
|
||||
writeFileSync(
|
||||
join(lib, "entities.weft"),
|
||||
`// Generated by scripts/gen-tables.mjs from https://html.spec.whatwg.org/entities.json. Don't edit it.
|
||||
import { parseInt } from "std/math"
|
||||
|
||||
let DATA = "${pairs.join(" ")}"
|
||||
|
||||
fn build(): Map<string, string> {
|
||||
let mut table = Map<string, string>()
|
||||
for entry of DATA.split(" ") {
|
||||
let at = entry.indexOf("=")
|
||||
let mut text = ""
|
||||
for hex of entry.slice(at + 1).split(",") {
|
||||
text += String.fromCodePoint(try parseInt(hex, 16) else 0xfffd)
|
||||
}
|
||||
table.set(entry.slice(0, at), text)
|
||||
}
|
||||
return table
|
||||
}
|
||||
|
||||
let ENTITIES = build()
|
||||
|
||||
/// The text of a named character reference (\`amp\` for \`&\`), or \`null\` when HTML has no such name.
|
||||
export fn entity(name: string): string? {
|
||||
return ENTITIES.get(name)
|
||||
}
|
||||
`,
|
||||
)
|
||||
|
||||
const punct = /[\p{P}\p{S}]/u
|
||||
const ranges = []
|
||||
let start = -1
|
||||
for (let c = 0; c <= 0x110000; c++) {
|
||||
const is = c < 0x110000 && !(c >= 0xd800 && c <= 0xdfff) && punct.test(String.fromCodePoint(c))
|
||||
if (is && start < 0) start = c
|
||||
if (!is && start >= 0) {
|
||||
ranges.push(start, c - 1)
|
||||
start = -1
|
||||
}
|
||||
}
|
||||
|
||||
const rows = []
|
||||
for (let i = 0; i < ranges.length; i += 16) rows.push(" " + ranges.slice(i, i + 16).join(", ") + ",")
|
||||
|
||||
writeFileSync(
|
||||
join(lib, "unicode.weft"),
|
||||
`// Generated by scripts/gen-tables.mjs from the engine's Unicode tables (general categories P and S). Don't edit it.
|
||||
|
||||
/// Sorted, inclusive ranges of code points: start, end, start, end, …
|
||||
export let PUNCTUATION: number[] = [
|
||||
${rows.join("\n")}
|
||||
]
|
||||
`,
|
||||
)
|
||||
|
||||
console.log(`${pairs.length} entities, ${ranges.length / 2} punctuation ranges`)
|
||||
39
markdown/scripts/hostile.mjs
Normal file
39
markdown/scripts/hostile.mjs
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
// Runs tests/hostile.weft: each input at n and 2n, timed from outside the VM (the clock doesn't move within a turn).
|
||||
// A size that doubles must not take much more than twice as long.
|
||||
//
|
||||
// node packages/markdown/scripts/hostile.mjs [n]
|
||||
import { spawnSync } from "node:child_process"
|
||||
import { dirname, join } from "node:path"
|
||||
import { fileURLToPath } from "node:url"
|
||||
|
||||
const root = join(dirname(fileURLToPath(import.meta.url)), "..")
|
||||
const weft = process.env.WEFT_BIN ?? join(root, "../../core/target/release/weft")
|
||||
const n = Number(process.argv[2] ?? 8000)
|
||||
// The limits are lifted so that a slow path shows up as time rather than as a stop.
|
||||
const env = { ...process.env, WEFT_FUEL: "20000000000", WEFT_MEMORY_MB: "1500" }
|
||||
|
||||
function run(args) {
|
||||
const start = process.hrtime.bigint()
|
||||
const r = spawnSync(weft, ["run", "tests/hostile.weft", ...args], { cwd: root, env, encoding: "utf8", timeout: 120000 })
|
||||
return { ms: Number(process.hrtime.bigint() - start) / 1e6, out: r.stdout.trim(), err: r.stderr.trim(), status: r.status }
|
||||
}
|
||||
|
||||
const names = run(["--list"]).out.split("\n")
|
||||
const base = run(["lines", "0"]).ms
|
||||
let bad = 0
|
||||
for (const name of names) {
|
||||
const a = run([name, String(n)])
|
||||
const b = run([name, String(n * 2)])
|
||||
const [ta, tb] = [Math.max(a.ms - base, 1), Math.max(b.ms - base, 1)]
|
||||
const stopped = a.status !== 0 || b.status !== 0
|
||||
const quadratic = tb > 1500 && tb / ta > 3.2
|
||||
if (stopped || quadratic) bad++
|
||||
const note = stopped ? ` STOPPED: ${(b.err || a.err).split("\n")[0]}` : quadratic ? " QUADRATIC" : ""
|
||||
console.log(`${name.padEnd(24)} ${ta.toFixed(0).padStart(7)} ms ${tb.toFixed(0).padStart(7)} ms${note}`)
|
||||
}
|
||||
const expansion = run(["expansion"])
|
||||
const size = Number(expansion.out)
|
||||
if (!(size > 0 && size < 1000000)) bad++
|
||||
console.log(`reference expansion: ${size} bytes of HTML from 50 KB of Markdown`)
|
||||
console.log(bad === 0 ? "ok" : `${bad} failed`)
|
||||
process.exit(bad === 0 ? 0 : 1)
|
||||
1333
markdown/tests/cases.weft
Normal file
1333
markdown/tests/cases.weft
Normal file
File diff suppressed because it is too large
Load diff
47
markdown/tests/hostile.weft
Normal file
47
markdown/tests/hostile.weft
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
// Hostile input, one per run: `weft run tests/hostile.weft <name> <n>` renders it and prints the HTML's length.
|
||||
// scripts/hostile.mjs runs them all and checks that the time each takes grows in proportion to its size.
|
||||
import { log } from "std/log"
|
||||
import { args } from "std/process"
|
||||
import { toHtml } from "../markdown"
|
||||
|
||||
type Input = { name: string, make: (n: int) => string }
|
||||
|
||||
let inputs: Input[] = [
|
||||
{ name: "nested quotes", make: n => ">".repeat(n) + " x" },
|
||||
{ name: "nested lists", make: n => "- ".repeat(n) + "x" },
|
||||
{ name: "nested emphasis", make: n => "*a **b ".repeat(n) + "c" + "** d*".repeat(n) },
|
||||
{ name: "nested links", make: n => "[a ".repeat(n) + "](/x)".repeat(n) },
|
||||
{ name: "nested images", make: n => "".repeat(n) },
|
||||
{ name: "open emphasis", make: n => "*a ".repeat(n) },
|
||||
{ name: "closed emphasis", make: n => "_a_ ".repeat(n) },
|
||||
{ name: "open brackets", make: n => "[".repeat(n) },
|
||||
{ name: "close brackets", make: n => "]".repeat(n) },
|
||||
{ name: "unfinished links", make: n => "[a](".repeat(n) },
|
||||
{ name: "backtick runs", make: n => "`a ``b ".repeat(n) },
|
||||
{ name: "open tags", make: n => "<a ".repeat(n) },
|
||||
{ name: "open comments", make: n => "a <!-- ".repeat(n) },
|
||||
{ name: "open cdata", make: n => "a <![CDATA[ ".repeat(n) },
|
||||
{ name: "open instructions", make: n => "a <? ".repeat(n) },
|
||||
{ name: "entities", make: n => "&".repeat(n) },
|
||||
{ name: "backslashes", make: n => "\\".repeat(n) },
|
||||
{ name: "lines", make: n => "a\n".repeat(n) },
|
||||
{ name: "reference definitions", make: n => "[a]: /b\n".repeat(n) },
|
||||
{ name: "list items", make: n => "- a\n".repeat(n) },
|
||||
{ name: "trailing parens", make: n => "http://a.dev/" + ")".repeat(n) },
|
||||
{ name: "at signs", make: n => "a@".repeat(n) },
|
||||
{ name: "table rows", make: n => "| a | b |\n|---|---|\n" + "| 1 | 2 |\n".repeat(n) },
|
||||
{ name: "wide table", make: n => "| a ".repeat(n) + "\n" + "|-".repeat(n) },
|
||||
]
|
||||
|
||||
let name = args().at(0) ?? "--list"
|
||||
if name == "--list" {
|
||||
log(inputs.map(i => i.name).join("\n"))
|
||||
} else if name == "expansion" {
|
||||
// A definition used many times can't multiply the output.
|
||||
let url = "/" + "x".repeat(10000)
|
||||
log(toHtml(`[a]: ${url}\n\n` + "[a] ".repeat(10000)).length)
|
||||
} else {
|
||||
let n = try int(args().at(1) ?? "1000") else 1000
|
||||
let input = inputs.find(i => i.name == name)
|
||||
log(input == null ? -1 : toHtml(input.make(n)).length)
|
||||
}
|
||||
220
markdown/tests/markdown.test.weft
Normal file
220
markdown/tests/markdown.test.weft
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
// `weft test`: the package's API and its safety. The spec's examples are in tests/spec.weft, hostile input in
|
||||
// tests/hostile.weft (both are programs: they run past a test's deadline).
|
||||
import { assert, assertEq } from "std/testing"
|
||||
import { parse, render, sanitize, toHtml, toText, visit } from "../markdown"
|
||||
|
||||
// ---- safe by default ----
|
||||
test "raw HTML is shown as text" {
|
||||
assertEq(toHtml("<script>alert(1)</script>"), "<p><script>alert(1)</script></p>\n")
|
||||
assertEq(
|
||||
toHtml("a <b onclick=\"x()\">b</b>"),
|
||||
"<p>a <b onclick="x()">b</b></p>\n",
|
||||
)
|
||||
}
|
||||
|
||||
test "unsafe link schemes render as their text" {
|
||||
let attacks = [
|
||||
"[x](javascript:alert(1))",
|
||||
"[x](JaVaScRiPt:alert(1))",
|
||||
"[x](vbscript:msgbox(1))",
|
||||
"[x](data:text/html;base64,PHNjcmlwdD4=)",
|
||||
"[x](javascript:alert(1))",
|
||||
"[x](javascript:alert(1))",
|
||||
"[x](file:///etc/passwd)",
|
||||
]
|
||||
for a of attacks {
|
||||
assertEq(toHtml(a), "<p>x</p>\n")
|
||||
}
|
||||
assertEq(toHtml("<javascript:alert(1)>"), "<p>javascript:alert(1)</p>\n")
|
||||
}
|
||||
|
||||
test "unsafe images render as their alt text" {
|
||||
assertEq(toHtml(")"), "<p>cat</p>\n")
|
||||
assertEq(toHtml(""), "<p>cat</p>\n")
|
||||
assertEq(
|
||||
toHtml(""),
|
||||
"<p><img src=\"data:image/png;base64,AAAA\" alt=\"cat\" /></p>\n",
|
||||
)
|
||||
}
|
||||
|
||||
test "safe URLs are kept" {
|
||||
assertEq(
|
||||
toHtml("[a](https://x.dev/?q=1&r=2)"),
|
||||
"<p><a href=\"https://x.dev/?q=1&r=2\">a</a></p>\n",
|
||||
)
|
||||
assertEq(toHtml("[a](/docs/x:y)"), "<p><a href=\"/docs/x:y\">a</a></p>\n")
|
||||
assertEq(toHtml("[a](#top)"), "<p><a href=\"#top\">a</a></p>\n")
|
||||
assertEq(toHtml("[a](mailto:me@x.dev)"), "<p><a href=\"mailto:me@x.dev\">a</a></p>\n")
|
||||
assertEq(toHtml("[a](tel:+38640123456)"), "<p><a href=\"tel:+38640123456\">a</a></p>\n")
|
||||
}
|
||||
|
||||
test "titles, code and attributes are escaped" {
|
||||
assertEq(
|
||||
toHtml("[a](/x \"\\\"><script>\")"),
|
||||
"<p><a href=\"/x\" title=\""><script>\">a</a></p>\n",
|
||||
)
|
||||
assertEq(toHtml("`<b>`"), "<p><code><b></code></p>\n")
|
||||
assertEq(
|
||||
toHtml("```\"><script>\nx\n```"),
|
||||
"<pre><code class=\"language-"><script>\">x\n</code></pre>\n",
|
||||
)
|
||||
}
|
||||
|
||||
// ---- the sanitiser ----
|
||||
test "sanitize removes scripts, handlers and unsafe URLs" {
|
||||
assertEq(sanitize("<p onclick=\"x()\">hi<script>bad()</script></p>"), "<p>hi</p>")
|
||||
assertEq(sanitize("<a href=\"javascript:x()\" title=\"t\">hi</a>"), "<a title=\"t\">hi</a>")
|
||||
assertEq(sanitize("<a href=\"javascript:x()\">hi</a>"), "<a>hi</a>")
|
||||
assertEq(sanitize("<a href=\"java	script:x()\">hi</a>"), "<a>hi</a>")
|
||||
assertEq(sanitize("<a href=\"  javascript:x()\">hi</a>"), "<a>hi</a>")
|
||||
assertEq(sanitize("<img src=x onerror=alert(1)>"), "<img src=\"x\" />")
|
||||
assertEq(sanitize("<IMG SRC=\"JAVASCRIPT:x\">"), "<img />")
|
||||
assertEq(sanitize("<svg><script>x</script></svg>ok"), "ok")
|
||||
assertEq(sanitize("<math><mi>x</mi></math>ok"), "ok")
|
||||
assertEq(sanitize("<style>*{display:none}</style>ok"), "ok")
|
||||
assertEq(sanitize("<iframe src=\"https://evil\"></iframe>ok"), "ok")
|
||||
assertEq(sanitize("<ScRiPt >x</sCrIpT >ok"), "ok")
|
||||
assertEq(sanitize("<script>never closed"), "")
|
||||
assertEq(sanitize("<!-- <script>x</script> -->ok"), "ok")
|
||||
assertEq(sanitize("<form action=\"/x\"><button>go</button></form>"), "go")
|
||||
}
|
||||
|
||||
test "sanitize keeps only a disabled checkbox" {
|
||||
assertEq(sanitize("<input type=\"text\" value=\"pw\">"), "")
|
||||
assertEq(sanitize("<input>"), "")
|
||||
assertEq(
|
||||
sanitize("<input type=\"checkbox\" checked>"),
|
||||
"<input type=\"checkbox\" checked=\"\" disabled=\"\" />",
|
||||
)
|
||||
}
|
||||
|
||||
test "sanitize gives well-formed output" {
|
||||
assertEq(sanitize("<b><i>x</b>y"), "<b><i>x</i></b>y")
|
||||
assertEq(sanitize("</div>x"), "x")
|
||||
assertEq(sanitize("<p>a"), "<p>a</p>")
|
||||
assertEq(sanitize("1 < 2 & 3 > 2 & ok"), "1 < 2 & 3 > 2 & ok")
|
||||
assertEq(sanitize("<a title=\"a"b\">x</a>"), "<a title=\"a"b\">x</a>")
|
||||
assertEq(
|
||||
sanitize("<a target=\"_blank\" href=\"/x\">x</a>"),
|
||||
"<a target=\"_blank\" href=\"/x\" rel=\"noopener noreferrer\">x</a>",
|
||||
)
|
||||
assertEq(sanitize("<a target=\"evil\" href=\"/x\">x</a>"), "<a href=\"/x\">x</a>")
|
||||
}
|
||||
|
||||
test "a policy narrows what is kept" {
|
||||
assertEq(sanitize("<b>x</b><i>y</i>", { tags: ["b"] }), "<b>x</b>y")
|
||||
assertEq(sanitize("<a href=\"ftp://x\">y</a>", { schemes: ["ftp"] }), "<a href=\"ftp://x\">y</a>")
|
||||
assertEq(sanitize("<a href=\"https://x\">y</a>", { schemes: ["ftp"] }), "<a>y</a>")
|
||||
let attributes = Map<string, string[]>([["b", ["title"]]])
|
||||
assertEq(
|
||||
sanitize("<b title=\"t\" id=\"i\">x</b>", { tags: ["b"], attributes }),
|
||||
"<b title=\"t\">x</b>",
|
||||
)
|
||||
}
|
||||
|
||||
test "html: sanitize keeps safe raw HTML in Markdown" {
|
||||
assertEq(
|
||||
toHtml("<div onclick=\"x()\">\n*hi*\n</div>", { html: "sanitize" }),
|
||||
"<div>\n*hi*\n</div>\n",
|
||||
)
|
||||
assertEq(
|
||||
toHtml("a <kbd>b</kbd> <script>c</script>", { html: "sanitize" }),
|
||||
"<p>a <kbd>b</kbd> </p>\n",
|
||||
)
|
||||
assertEq(toHtml("<script>\nbad()\n\n</script>\n\nok", { html: "sanitize" }), "\n<p>ok</p>\n")
|
||||
assertEq(
|
||||
toHtml("- [x] done", { html: "sanitize" }),
|
||||
"<ul>\n<li><input checked=\"\" disabled=\"\" type=\"checkbox\" /> done</li>\n</ul>\n",
|
||||
)
|
||||
}
|
||||
|
||||
test "html: trust passes raw HTML through" {
|
||||
assertEq(
|
||||
toHtml("<div onclick=\"x()\">a</div>", { html: "trust" }),
|
||||
"<div onclick=\"x()\">a</div>\n",
|
||||
)
|
||||
}
|
||||
|
||||
// ---- GFM ----
|
||||
test "tables" {
|
||||
let html = toHtml("| a | b |\n|:--|--:|\n| 1 | `x\\|y` |")
|
||||
assertEq(
|
||||
html,
|
||||
"<table>\n<thead>\n<tr>\n<th align=\"left\">a</th>\n<th align=\"right\">b</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td align=\"left\">1</td>\n<td align=\"right\"><code>x|y</code></td>\n</tr>\n</tbody>\n</table>\n",
|
||||
)
|
||||
}
|
||||
|
||||
test "task lists" {
|
||||
let doc = parse("- [ ] todo\n- [x] done\n- plain")
|
||||
match doc.children[0] {
|
||||
BulletList { items } => {
|
||||
let checked: boolean?[] = items.map(i => i.checked)
|
||||
let expected: boolean?[] = [false, true, null]
|
||||
assertEq(checked, expected)
|
||||
},
|
||||
_ => assert(false, "a list"),
|
||||
}
|
||||
}
|
||||
|
||||
test "strikethrough and bare links" {
|
||||
assertEq(toHtml("~~gone~~"), "<p><del>gone</del></p>\n")
|
||||
assertEq(
|
||||
toHtml("see https://x.dev/a_b."),
|
||||
"<p>see <a href=\"https://x.dev/a_b\">https://x.dev/a_b</a>.</p>\n",
|
||||
)
|
||||
assertEq(toHtml("www.x.dev"), "<p><a href=\"http://www.x.dev\">www.x.dev</a></p>\n")
|
||||
assertEq(toHtml("me@x.dev"), "<p><a href=\"mailto:me@x.dev\">me@x.dev</a></p>\n")
|
||||
}
|
||||
|
||||
test "gfm: false is plain CommonMark" {
|
||||
assertEq(toHtml("~~a~~ https://x.dev", { gfm: false }), "<p>~~a~~ https://x.dev</p>\n")
|
||||
assertEq(toHtml("| a |\n|---|", { gfm: false }), "<p>| a |\n|---|</p>\n")
|
||||
}
|
||||
|
||||
// ---- what Vidni's briefs need ----
|
||||
test "heading offset, link target and URL rewriting" {
|
||||
assertEq(toHtml("# T", { headingOffset: 1 }), "<h2>T</h2>\n")
|
||||
assertEq(toHtml("###### T", { headingOffset: 3 }), "<h6>T</h6>\n")
|
||||
assertEq(
|
||||
toHtml("[a](https://x.dev)", { linkTarget: "_blank" }),
|
||||
"<p><a href=\"https://x.dev\" target=\"_blank\" rel=\"noopener noreferrer\">a</a></p>\n",
|
||||
)
|
||||
let resolve = (url: string, kind: "link" | "image") => url.startsWith("upload:")
|
||||
? `https://cdn.x.dev/${url.slice(7)}`
|
||||
: null
|
||||
assertEq(
|
||||
toHtml("", { rewriteUrl: resolve }),
|
||||
"<p><img src=\"https://cdn.x.dev/abc\" alt=\"c\" /></p>\n",
|
||||
)
|
||||
// Not resolved: `upload:` isn't an allowed scheme, so the caption stays.
|
||||
assertEq(toHtml(""), "<p>c</p>\n")
|
||||
}
|
||||
|
||||
test "toText for previews and emails" {
|
||||
assertEq(
|
||||
toText(parse("# Hi\n\nRead **this** [guide](https://x.dev).\n\n\n\n- one\n- two")),
|
||||
"Hi\n\nRead this guide (https://x.dev).\n\n[image: cat]\n\n- one\n\n- two",
|
||||
)
|
||||
}
|
||||
|
||||
test "visit finds the images of a document" {
|
||||
let mut urls: string[] = []
|
||||
visit(parse("\n\n> **\n\n|  |\n|---|"), n => {
|
||||
if let Picture { url } = n {
|
||||
urls.push(url)
|
||||
}
|
||||
})
|
||||
assertEq(urls, ["upload:1", "upload:2", "upload:3"])
|
||||
}
|
||||
|
||||
test "render uses a tree that was changed" {
|
||||
let doc = parse("hello")
|
||||
assertEq(render({ children: [...doc.children, ...doc.children] }), "<p>hello</p>\n<p>hello</p>\n")
|
||||
}
|
||||
|
||||
test "strange characters survive" {
|
||||
assertEq(toHtml("a\u{0}b"), "<p>a\u{fffd}b</p>\n")
|
||||
assertEq(toHtml("line\r\nnext\rlast"), "<p>line\nnext\nlast</p>\n")
|
||||
assertEq(toHtml("😀 *é*"), "<p>😀 <em>é</em></p>\n")
|
||||
assertEq(toHtml(""), "")
|
||||
}
|
||||
54
markdown/tests/spec.weft
Normal file
54
markdown/tests/spec.weft
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
// Runs the spec's examples: `weft run tests/spec.weft [commonmark|gfm] [-v]` prints every failure and a summary.
|
||||
import { args, exit } from "std/process"
|
||||
import { log } from "std/log"
|
||||
import { toHtml } from "../markdown"
|
||||
import { COMMONMARK, GFM } from "./cases"
|
||||
import type { Case } from "./cases"
|
||||
|
||||
/// Where the GFM spec (written against CommonMark 0.29) and CommonMark 0.31.2 give different HTML for the same text,
|
||||
/// CommonMark wins: cmark-gfm's renderer folds nested `<strong>`, which CommonMark's examples keep.
|
||||
fn followsCommonMark(c: Case, got: string): boolean {
|
||||
return COMMONMARK.some(o => o.markdown == c.markdown && o.html == got)
|
||||
}
|
||||
|
||||
fn runAll(name: string, cases: Case[], verbose: boolean): int {
|
||||
let mut failed = 0
|
||||
let mut disagreements = 0
|
||||
let mut bySection = Map<string, int>()
|
||||
for c of cases {
|
||||
let got = toHtml(c.markdown, { gfm: c.gfm, html: "trust" })
|
||||
if got != c.html && followsCommonMark(c, got) {
|
||||
disagreements += 1
|
||||
} else if got != c.html {
|
||||
failed += 1
|
||||
bySection.set(c.section, (bySection.get(c.section) ?? 0) + 1)
|
||||
if verbose {
|
||||
log(`--- ${name} example ${c.id} (${c.section})`)
|
||||
log(`markdown: ${c.markdown}`)
|
||||
log(`expected: ${c.html}`)
|
||||
log(`got: ${got}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
log(
|
||||
`${name}: ${cases.length - failed - disagreements} of ${cases.length} pass, ${disagreements} follow CommonMark 0.31.2 where the specs disagree, ${failed} fail`,
|
||||
)
|
||||
for [section, n] of bySection {
|
||||
log(` ${n} failing in ${section}`)
|
||||
}
|
||||
return failed
|
||||
}
|
||||
|
||||
let argv = args()
|
||||
let verbose = argv.includes("-v")
|
||||
let only = argv.filter(a => a != "-v").at(0)
|
||||
let mut failures = 0
|
||||
if only == null || only == "commonmark" {
|
||||
failures += runAll("CommonMark", COMMONMARK, verbose)
|
||||
}
|
||||
if only == null || only == "gfm" {
|
||||
failures += runAll("GFM", GFM, verbose)
|
||||
}
|
||||
if failures > 0 {
|
||||
exit(1)
|
||||
}
|
||||
10
markdown/weft.toml
Normal file
10
markdown/weft.toml
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[package]
|
||||
name = "weft/markdown"
|
||||
version = "0.1.0"
|
||||
description = "CommonMark and GitHub Flavored Markdown: an AST, HTML and a sanitiser, safe by default"
|
||||
license = "MIT OR Apache-2.0"
|
||||
# A placeholder until the public repository exists (packages/README.md, Publishing for real).
|
||||
repository = "https://git.weft-lang.dev/weft/packages"
|
||||
exports = ["markdown", "ast"]
|
||||
main = "markdown"
|
||||
include = ["markdown.weft", "ast.weft", "lib", "README.md"]
|
||||
Loading…
Reference in a new issue