feat(desktop): embed BoosAPI Desktop client as a subproject

Vendor the rebranded BoosAPI Desktop client (Tauri + Vue) into metapi as
a monorepo subdirectory under desktop/. Source only — node_modules,
dist, Cargo target and the upstream .git history are excluded.

desktop/ keeps its own independent Cargo workspace (desktop/src-tauri
with crates/*); it is not part of the backend boosapi crate, so a
`cargo check` at the metapi root still compiles only the backend. Build
the desktop app from desktop/src-tauri per desktop/README.md.

Note: app icons are still placeholder assets pending the final BoosAPI
logo. The updater signing keys live outside the repo and are not
included.
This commit is contained in:
boos
2026-06-22 11:32:57 +08:00
parent d9c78ecc73
commit e0a5f57793
2305 changed files with 449355 additions and 0 deletions
@@ -0,0 +1,9 @@
node_modules
dist
.turbo
*.log
.DS_Store
coverage
.vscode
.idea
*.tsbuildinfo
@@ -0,0 +1,10 @@
src
test
tests
*.test.ts
*.spec.ts
tsconfig.json
tsup.config.ts
.eslintrc*
.prettierrc*
vitest.config.ts
@@ -0,0 +1,527 @@
# stream-markdown-parser
[![NPM version](https://img.shields.io/npm/v/stream-markdown-parser?color=a1b858&label=)](https://www.npmjs.com/package/stream-markdown-parser)
[![中文版](https://img.shields.io/badge/docs-中文文档-blue)](README.zh-CN.md)
[![NPM downloads](https://img.shields.io/npm/dm/stream-markdown-parser)](https://www.npmjs.com/package/stream-markdown-parser)
[![Bundle size](https://img.shields.io/bundlephobia/minzip/stream-markdown-parser)](https://bundlephobia.com/package/stream-markdown-parser)
[![License](https://img.shields.io/npm/l/stream-markdown-parser)](https://github.com/Simon-He95/markstream-vue/blob/main/license)
Pure markdown parser and renderer utilities with streaming support - framework agnostic.
This package contains the core markdown parsing logic extracted from `markstream-vue`, making it usable in any JavaScript/TypeScript project without Vue dependencies.
## Features
- 🚀 **Pure JavaScript** - No framework dependencies
- 📦 **Lightweight** - Minimal bundle size
- 🔧 **Extensible** - Plugin-based architecture
- 🎯 **Type-safe** - Full TypeScript support
- ⚡ **Fast** - Optimized for performance
- 🌊 **Streaming-friendly** - Progressive parsing support
> ℹ️ We now build on top of [`markdown-it-ts`](https://www.npmjs.com/package/markdown-it-ts), a TypeScript-first distribution of markdown-it. The API stays the same, but we only rely on its parsing pipeline and ship richer typings for tokens and hooks.
## Documentation
The full usage guide lives alongside the markstream-vue docs:
- English: https://markstream-vue-docs.simonhe.me/guide/api
- 中文: https://markstream-vue-docs.simonhe.me/zh/guide/api
This README highlights the parser-specific APIs; visit the docs for end-to-end integration tutorials (VitePress, workers, Tailwind, troubleshooting, etc.).
## Installation
```bash
pnpm add stream-markdown-parser
# or
npm install stream-markdown-parser
# or
yarn add stream-markdown-parser
```
## Quick API (TL;DR)
- `getMarkdown(msgId?, options?)` — create a `markdown-it-ts` instance with built-in plugins (task lists, sub/sup, math helpers, etc.). Also accepts `plugin`, `apply`, and `i18n` options.
- `registerMarkdownPlugin(plugin)` / `clearRegisteredMarkdownPlugins()` — add or remove global plugins that run for every `getMarkdown()` call (useful for feature flags or tests).
- `parseMarkdownToStructure(markdown, md, parseOptions)` — convert Markdown into the streaming-friendly AST consumed by `markstream-vue` and other renderers.
- `processTokens(tokens)` / `parseInlineTokens(children, content?, preToken?, options?)` — low-level helpers if you want to bypass the built-in AST pipeline.
- `applyMath`, `applyContainers`, `normalizeStandaloneBackslashT`, `findMatchingClose`, etc. — utilities for custom parsing or linting workflows.
## Usage
### Streaming-friendly pipeline
```
Markdown string
↓ getMarkdown() → markdown-it-ts instance with plugins
parseMarkdownToStructure(markdown, md) → AST (ParsedNode[])
↓ feed into your renderer (markstream-vue, custom UI, workers)
```
Reuse the same `md` instance when parsing multiple documents—plugin setup is the heaviest step. When integrating with [`markstream-vue`](https://www.npmjs.com/package/markstream-vue), pass the AST to `<MarkdownRender :nodes="nodes" />` or supply raw `content` and hand it the same parser options.
### Incremental / streaming example
When consuming an AI or SSE stream you can keep appending to a buffer, parse with the same `md` instance, and send the AST to the UI (e.g., `markstream-vue`) on every chunk:
```ts
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
const md = getMarkdown()
let buffer = ''
async function handleChunk(chunk: string) {
buffer += chunk
const nodes = parseMarkdownToStructure(buffer, md)
postMessage({ type: 'markdown:update', nodes })
}
```
In the UI layer, render `nodes` with `<MarkdownRender :nodes="nodes" />` to avoid re-parsing. See the [docs usage guide](https://markstream-vue-docs.simonhe.me/guide/usage) for end-to-end wiring.
### Basic example
```typescript
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
// Create a markdown-it-ts instance with default plugins
const md = getMarkdown()
// Parse markdown to our streaming-friendly AST structure
const nodes = parseMarkdownToStructure('# Hello World', md)
console.log(nodes)
// [{ type: 'heading', level: 1, children: [...] }]
// markdown-it-ts still exposes render() if you need HTML output,
// but this package now focuses on the token -> AST pipeline.
const html = md.render?.('# Hello World\n\nThis is **bold**.')
```
### With Math Options
```typescript
import { getMarkdown, setDefaultMathOptions } from 'stream-markdown-parser'
// Set global math options
setDefaultMathOptions({
commands: ['infty', 'perp', 'alpha'],
escapeExclamation: true
})
const md = getMarkdown()
```
### With Custom i18n
```typescript
import { getMarkdown } from 'stream-markdown-parser'
// Using translation map
const md = getMarkdown('editor-1', {
i18n: {
'common.copy': '复制',
}
})
// Or using a translation function
const md = getMarkdown('editor-1', {
i18n: (key: string) => translateFunction(key)
})
```
### With Plugins
```typescript
import customPlugin from 'markdown-it-custom-plugin'
import { getMarkdown } from 'stream-markdown-parser'
const md = getMarkdown('editor-1', {
plugin: [
[customPlugin, { /* options */ }]
]
})
```
### Advanced: Custom Rules
```typescript
import { getMarkdown } from 'stream-markdown-parser'
const md = getMarkdown('editor-1', {
apply: [
(md) => {
// Add custom inline rule
md.inline.ruler.before('emphasis', 'custom', (state, silent) => {
// Your custom logic
return false
})
}
]
})
```
### Extending globally
Need to add a plugin everywhere without touching each call site? Use the helper exports:
```ts
import {
clearRegisteredMarkdownPlugins,
registerMarkdownPlugin,
} from 'stream-markdown-parser'
registerMarkdownPlugin(myPlugin)
const md = getMarkdown()
// md now has `myPlugin` enabled in addition to anything passed via options
// For tests or teardown flows:
clearRegisteredMarkdownPlugins()
```
- `plugin` option → array of `md.use` invocations scoped to a single `getMarkdown` call.
- `apply` option → imperatively mutate the instance (`md.inline.ruler.before(...)`). Wrap in try/catch if you need to surface errors differently; the helper logs to console to preserve legacy behaviour.
- `registerMarkdownPlugin` → global singleton registry (handy in SSR or worker contexts where you want feature flags to apply everywhere).
## API
### Main Functions
#### `getMarkdown(msgId?, options?)`
Creates a configured `markdown-it-ts` instance (API-compatible with markdown-it).
**Parameters:**
- `msgId` (string, optional): Unique identifier for this instance. Default: `editor-${Date.now()}`
- `options` (GetMarkdownOptions, optional): Configuration options
**Options:**
```typescript
interface GetMarkdownOptions {
// Array of markdown-it/markdown-it-ts plugins to use
plugin?: Array<Plugin | [Plugin, any]>
// Array of functions to mutate the md instance
apply?: Array<(md: MarkdownIt) => void>
// Translation function or translation map
i18n?: ((key: string) => string) | Record<string, string>
}
```
#### `parseMarkdownToStructure(content, md, options?)`
Parses markdown content into a structured node tree.
**Parameters:**
- `content` (string): The markdown content to parse
- `md` (MarkdownItCore): A markdown-it-ts instance created with `getMarkdown()`
- `options` (ParseOptions, optional): Parsing options with hooks
**Returns:** `ParsedNode[]` - An array of parsed nodes
#### `processTokens(tokens)`
Processes raw markdown-it tokens into a flat array.
#### `parseInlineTokens(tokens, content?, preToken?, options?)`
Parses inline markdown-it-ts tokens into renderer nodes. Pass the inline token array plus the optional raw `content` string (from the parent token), an optional previous token, and inline parse options (`requireClosingStrong`, `customHtmlTags`, `validateLink`).
### Configuration Functions
#### `setDefaultMathOptions(options)`
Set global math rendering options.
**Parameters:**
- `options` (MathOptions): Math configuration options
```typescript
interface MathOptions {
commands?: readonly string[] // LaTeX commands to escape
escapeExclamation?: boolean // Escape standalone '!' (default: true)
}
```
### Parse hooks (fine-grained transforms)
ParseOptions supports the following hooks and flags:
```ts
interface ParseOptions {
preTransformTokens?: (tokens: Token[]) => Token[]
postTransformTokens?: (tokens: Token[]) => Token[]
// Custom HTML-like tags to emit as custom nodes (e.g. ['thinking'])
// Use tag-like names such as 'thinking', 'answer-box', or 'my_component'.
customHtmlTags?: string[]
// Validate link href before emitting a `link` node; false -> plain text
validateLink?: (url: string) => boolean
// When true, treats the input as complete (end-of-stream)
final?: boolean
// Require closing `**` for strong parsing (default: false)
requireClosingStrong?: boolean
}
```
Example — flag AI “thinking” blocks:
```ts
const parseOptions = {
customHtmlTags: ['thinking'],
}
const nodes = parseMarkdownToStructure(markdown, md, parseOptions)
const tagged = nodes.map(node =>
node.type === 'html_block' && /<thinking>/.test((node as any).content ?? '')
? { ...node, meta: { type: 'thinking' } }
: node,
)
```
Use the metadata in your renderer to show custom UI without mangling the original Markdown.
Example — enforce safe link protocols:
```ts
const md = getMarkdown('safe-links')
md.set?.({
validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()),
})
const nodes = parseMarkdownToStructure(
'[ok](https://example.com) [bad](javascript:alert(1))',
md,
{ final: true },
)
// "ok" stays a link node; "bad" is downgraded to plain text
```
### Unknown HTML-like tags
By default, non-standard HTML-like tags (for example `<question>`) are rendered as raw HTML elements once they are complete. Incomplete or malformed fragments stay as **literal text** so they do not swallow surrounding content during streaming or final renders. If you want them emitted as custom nodes (`type: 'question'` with parsed attrs/content), opt in via `customHtmlTags`.
### Utility Functions
#### `isMathLike(content)`
Heuristic function to detect if content looks like mathematical notation.
**Parameters:**
- `content` (string): Content to check
**Returns:** `boolean`
#### `findMatchingClose(src, startIdx, open, close)`
Find the matching closing delimiter in a string, handling nested pairs.
**Parameters:**
- `src` (string): Source string
- `startIdx` (number): Start index to search from
- `open` (string): Opening delimiter
- `close` (string): Closing delimiter
**Returns:** `number` - Index of matching close, or -1 if not found
## Tips & troubleshooting
- **Reuse parser instances**: cache `getMarkdown()` results per worker/request to avoid re-registering plugins.
- **Server-side parsing**: run `parseMarkdownToStructure` on the server, ship the AST to the client, and hydrate with `markstream-vue` for deterministic output.
- **Custom HTML widgets**: pre-extract `<MyWidget>` blocks before parsing (replace with placeholders) and reinject them during rendering instead of mutating `html_block` nodes post-parse.
- **Styling**: when piping nodes into `markstream-vue`, follow the docs [CSS checklist](https://markstream-vue-docs.simonhe.me/guide/troubleshooting#css-looks-wrong-start-here) so Tailwind/UnoCSS don’t override library styles.
- **Error handling**: the `apply` hook swallows exceptions to maintain backwards compatibility. If you want strict mode, wrap your mutators before passing them in and rethrow/log as needed.
#### `parseFenceToken(token)`
Parse a code fence token into a CodeBlockNode.
**Parameters:**
- `token` (MarkdownToken): markdown-it token
**Returns:** `CodeBlockNode`
#### `normalizeStandaloneBackslashT(content, options?)`
Normalize backslash-t sequences in math content.
**Parameters:**
- `content` (string): Content to normalize
- `options` (MathOptions, optional): Math options
**Returns:** `string`
### Lower-level helpers
If you need full control over how tokens are transformed, you can import the primitive builders directly:
```ts
import type { MarkdownToken } from 'stream-markdown-parser'
import {
parseInlineTokens,
processTokens
} from 'stream-markdown-parser'
const tokens: MarkdownToken[] = md.parse(markdown, {})
const nodes = processTokens(tokens)
// or operate at inline granularity:
const inlineNodes = parseInlineTokens(tokens[0].children ?? [], tokens[0].content ?? '')
```
`processTokens` is what `parseMarkdownToStructure` uses internally, so you can remix the AST pipeline without reimplementing the Markdown-it loop.
### Plugin Functions
#### `applyMath(md, options?)`
Apply math plugin to markdown-it instance.
**Parameters:**
- `md` (MarkdownIt): markdown-it instance
- `options` (MathOptions, optional): Math rendering options
#### `applyContainers(md)`
Apply container plugins to markdown-it instance.
**Parameters:**
- `md` (MarkdownIt): markdown-it instance
### Constants
#### `KATEX_COMMANDS`
Array of common KaTeX commands for escaping.
#### `TEX_BRACE_COMMANDS`
Array of TeX commands that use braces.
#### `ESCAPED_TEX_BRACE_COMMANDS`
Escaped version of TEX_BRACE_COMMANDS for regex use.
## Types
All TypeScript types are exported:
```typescript
import type {
// Node types
CodeBlockNode,
GetMarkdownOptions,
HeadingNode,
ListItemNode,
ListNode,
MathOptions,
ParagraphNode,
ParsedNode,
ParseOptions,
// ... and more
} from 'stream-markdown-parser'
```
### Node Types
The parser exports various node types representing different markdown elements:
- `TextNode`, `HeadingNode`, `ParagraphNode`
- `ListNode`, `ListItemNode`
- `CodeBlockNode`, `InlineCodeNode`
- `LinkNode`, `ImageNode`
- `BlockquoteNode`, `TableNode`
- `MathBlockNode`, `MathInlineNode`
- And many more...
## Default Plugins
This package comes with the following markdown-it plugins pre-configured:
- `markdown-it-sub` - Subscript support (`H~2~O`)
- `markdown-it-sup` - Superscript support (`x^2^`)
- `markdown-it-mark` - Highlight/mark support (`==highlighted==`)
- `markdown-it-task-checkbox` - Task list support (`- [ ] Todo`)
- `markdown-it-ins` - Insert tag support (`++inserted++`)
- `markdown-it-footnote` - Footnote support
- `markdown-it-container` - Custom container support (`::: warning`, `::: tip`, etc.)
- Math support - LaTeX math rendering with `$...$` and `$$...$$`
## Framework Integration
While this package is framework-agnostic, it's designed to work seamlessly with:
- ✅ **Node.js** - Server-side rendering
- ✅ **Vue 3** - Use with `markstream-vue` (or your own renderer)
- ✅ **React** - Use parsed nodes for custom rendering
- ✅ **Vanilla JS** - Direct HTML rendering
- ✅ **Any framework** - Parse to AST and render as needed
### Example — dedicated worker feeding markstream-vue
Offload parsing to a Web Worker while the UI renders via `markstream-vue`:
```ts
// worker.ts
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
const md = getMarkdown()
let buffer = ''
globalThis.addEventListener('message', (event) => {
if (event.data.type === 'chunk') {
buffer += event.data.value
const nodes = parseMarkdownToStructure(buffer, md)
globalThis.postMessage({ type: 'update', nodes })
}
})
```
```ts
// ui.ts
const worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' })
worker.addEventListener('message', (event) => {
if (event.data.type === 'update')
nodes.value = event.data.nodes
})
```
```vue
<MarkdownRender :nodes="nodes" />
```
This pattern keeps Markdown-it work off the main thread and lets you reuse the same AST in any framework.
## Migration from `markstream-vue` (parser exports)
If you're currently importing parser helpers from `markstream-vue`, you can switch to the dedicated package:
```typescript
// before
import { getMarkdown } from 'markstream-vue'
// after
import { getMarkdown } from 'stream-markdown-parser'
```
All APIs remain the same. See the [migration guide](https://markstream-vue-docs.simonhe.me/monorepo-migration) for details.
## Performance
- **Lightweight**: ~65KB minified (13KB gzipped)
- **Fast**: Optimized for real-time parsing
- **Tree-shakeable**: Only import what you need
- **Few dependencies**: `markdown-it-ts` + a small set of markdown-it plugins
## Contributing
Issues and PRs welcome! Please read the [contribution guidelines](https://github.com/Simon-He95/markstream-vue/blob/main/CONTRIBUTING.md).
## License
MIT © Simon He
## Related
- [markstream-vue](https://www.npmjs.com/package/markstream-vue) - Full-featured Vue 3 Markdown renderer
@@ -0,0 +1,491 @@
# stream-markdown-parser
[![NPM version](https://img.shields.io/npm/v/stream-markdown-parser?color=a1b858&label=)](https://www.npmjs.com/package/stream-markdown-parser)
[![English Docs](https://img.shields.io/badge/docs-English-blue)](README.md)
[![NPM downloads](https://img.shields.io/npm/dm/stream-markdown-parser)](https://www.npmjs.com/package/stream-markdown-parser)
[![Bundle size](https://img.shields.io/bundlephobia/minzip/stream-markdown-parser)](https://bundlephobia.com/package/stream-markdown-parser)
[![License](https://img.shields.io/npm/l/stream-markdown-parser)](https://github.com/Simon-He95/markstream-vue/blob/main/license)
纯 JavaScript Markdown 解析器和渲染工具,支持流式处理 - 框架无关。
该包包含从 `markstream-vue` 中提取的核心 Markdown 解析逻辑,使其可以在任何 JavaScript/TypeScript 项目中使用,无需 Vue 依赖。
## 特性
- 🚀 **纯 JavaScript** - 无框架依赖
- 📦 **轻量级** - 最小打包体积
- 🔧 **可扩展** - 基于插件的架构
- 🎯 **类型安全** - 完整的 TypeScript 支持
- ⚡ **高性能** - 性能优化
- 🌊 **流式友好** - 支持渐进式解析
> ℹ️ 自当前版本起我们基于 [`markdown-it-ts`](https://www.npmjs.com/package/markdown-it-ts)(一个 TypeScript 优先的 markdown-it 发行版)进行构建。API 与 markdown-it 保持一致,但内部仅依赖其解析流程,并提供更丰富的 token 类型定义。
## 文档
完整的使用说明与集成教程见 markstream-vue 文档站:
- English: https://markstream-vue-docs.simonhe.me/guide/api
- 中文: https://markstream-vue-docs.simonhe.me/zh/guide/api
本 README 聚焦解析器 API;如需 VitePress/Vite/Nuxt 集成、Worker 流式解析、Tailwind/UnoCSS 配置等指南,请查阅上述文档。
## 安装
```bash
pnpm add stream-markdown-parser
# 或
npm install stream-markdown-parser
# 或
yarn add stream-markdown-parser
```
## 快速 API 速览
- `getMarkdown(msgId?, options?)` — 返回一个预配置的 `markdown-it-ts` 实例;支持 `plugin`、`apply`、`i18n` 等选项(内置任务列表、上下标、数学等插件)。
- `registerMarkdownPlugin(plugin)` / `clearRegisteredMarkdownPlugins()` — 全局注册/清除插件,在所有 `getMarkdown()` 调用中生效(适合特性开关或测试环境)。
- `parseMarkdownToStructure(markdown, md, parseOptions)` — 将 Markdown 转换为可供 `markstream-vue` 等渲染器使用的 AST。
- `processTokens(tokens)` / `parseInlineTokens(children, content?, preToken?, options?)` — 更底层的 token → 节点工具,方便自定义管线。
- `applyMath`、`applyContainers`、`normalizeStandaloneBackslashT`、`findMatchingClose` 等 — 用于构建自定义解析、lint 或内容清洗流程。
## 使用
### 流式解析流程
```
Markdown 字符串
↓ getMarkdown() → 带插件的 markdown-it-ts 实例
parseMarkdownToStructure(markdown, md) → AST (ParsedNode[])
↓ 交给你的渲染器(markstream-vue、自定义 UI、Worker 等)
```
多次解析时复用同一个 `md` 实例可以避免重复注册插件。与 [`markstream-vue`](https://www.npmjs.com/package/markstream-vue) 一起使用时,你可以把 AST 传给 `<MarkdownRender :nodes="nodes" />`,或仅传入原始 `content` 并共享同一套解析配置。
### 增量 / 流式示例
处理 AI/SSE 流时,可以复用同一个 `md` 实例不停地对累积缓冲区解析,并把 AST 推送给 UI(例如 `markstream-vue`):
```ts
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
const md = getMarkdown()
let buffer = ''
async function handleChunk(chunk: string) {
buffer += chunk
const nodes = parseMarkdownToStructure(buffer, md)
postMessage({ type: 'markdown:update', nodes })
}
```
在前端通过 `<MarkdownRender :nodes="nodes" />` 渲染即可避免重复解析。具体串联示例见[文档用法指南](https://markstream-vue-docs.simonhe.me/zh/guide/usage)。
### 基础示例
```typescript
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
// 创建一个带有默认插件的 markdown-it-ts 实例
const md = getMarkdown()
// 将 Markdown 解析为流式友好的 AST 结构
const nodes = parseMarkdownToStructure('# Hello World', md)
console.log(nodes)
// [{ type: 'heading', level: 1, children: [...] }]
// 如果仍需 HTML 输出,markdown-it-ts 依旧提供 render()
const html = md.render?.('# Hello World\n\nThis is **bold**.')
```
### 配置数学公式选项
```typescript
import { getMarkdown, setDefaultMathOptions } from 'stream-markdown-parser'
// 设置全局数学公式选项
setDefaultMathOptions({
commands: ['infty', 'perp', 'alpha'],
escapeExclamation: true
})
const md = getMarkdown()
```
### 自定义国际化
```typescript
import { getMarkdown } from 'stream-markdown-parser'
// 使用翻译映射
const md = getMarkdown('editor-1', {
i18n: {
'common.copy': '复制',
}
})
// 或使用翻译函数
const md = getMarkdown('editor-1', {
i18n: (key: string) => translateFunction(key)
})
```
### 使用插件
```typescript
import customPlugin from 'markdown-it-custom-plugin'
import { getMarkdown } from 'stream-markdown-parser'
const md = getMarkdown('editor-1', {
plugin: [
[customPlugin, { /* 选项 */ }]
]
})
```
### 高级:自定义规则
```typescript
import { getMarkdown } from 'stream-markdown-parser'
const md = getMarkdown('editor-1', {
apply: [
(md) => {
// 添加自定义内联规则
md.inline.ruler.before('emphasis', 'custom', (state, silent) => {
// 你的自定义逻辑
return false
})
}
]
})
```
### 全局扩展
想在所有 `getMarkdown()` 实例上启用同一个插件,而无需修改调用点?可使用内置 helper:
```ts
import {
clearRegisteredMarkdownPlugins,
registerMarkdownPlugin,
} from 'stream-markdown-parser'
registerMarkdownPlugin(myPlugin)
const md = getMarkdown()
// 现在 md 会自动包含 myPlugin
// 测试或清理阶段:
clearRegisteredMarkdownPlugins()
```
- `plugin` 选项 → 针对单次 `getMarkdown` 调用传入 `md.use(...)`。
- `apply` 选项 → 直接操作实例(如 `md.inline.ruler.before(...)`)。如果需要严格模式,可自行包裹 try/catch;默认会打印错误保持兼容。
- `registerMarkdownPlugin` → 全局注册表,适用于 SSR / Worker 等场景统一开启功能。
## API
### 主要函数
#### `getMarkdown(msgId?, options?)`
创建一个配置好的 `markdown-it-ts` 实例(与 markdown-it API 兼容)。
**参数:**
- `msgId` (string, 可选): 该实例的唯一标识符。默认值:`editor-${Date.now()}`
- `options` (GetMarkdownOptions, 可选): 配置选项
**选项:**
```typescript
interface GetMarkdownOptions {
// 要使用的 markdown-it / markdown-it-ts 插件数组
plugin?: Array<Plugin | [Plugin, any]>
// 修改 md 实例的函数数组
apply?: Array<(md: MarkdownIt) => void>
// 翻译函数或翻译映射
i18n?: ((key: string) => string) | Record<string, string>
}
```
#### `parseMarkdownToStructure(content, md, options?)`
将 Markdown 内容解析为结构化节点树。
**参数:**
- `content` (string): 要解析的 Markdown 内容
- `md` (MarkdownItCore): 由 `getMarkdown()` 创建的 markdown-it-ts 实例
- `options` (ParseOptions, 可选): 带有钩子的解析选项
**返回值:** `ParsedNode[]` - 解析后的节点数组
#### `processTokens(tokens)`
将原始 markdown-it tokens 处理为扁平数组。
#### `parseInlineTokens(tokens, content?, preToken?, options?)`
解析内联 markdown-it-ts tokens 并产出节点。可传入原始 `content`(来自父 token)、可选的前一个 token,以及 inline 解析选项(`requireClosingStrong`、`customHtmlTags`、`validateLink`)。
### 配置函数
#### `setDefaultMathOptions(options)`
设置全局数学公式渲染选项。
**参数:**
- `options` (MathOptions): 数学公式配置选项
```typescript
interface MathOptions {
commands?: readonly string[] // 要转义的 LaTeX 命令
escapeExclamation?: boolean // 转义独立的 '!' (默认: true)
}
```
### 解析钩子(精细化变换)
ParseOptions 支持以下钩子与标志:
```ts
interface ParseOptions {
preTransformTokens?: (tokens: Token[]) => Token[]
postTransformTokens?: (tokens: Token[]) => Token[]
// 自定义 HTML 类标签,作为自定义节点输出(如 ['thinking'])
// 请传标签式名字,比如 'thinking'、'answer-box'、'my_component'
customHtmlTags?: string[]
// 输出 link 节点前校验 href;返回 false 时降级为纯文本
validateLink?: (url: string) => boolean
// true 表示输入已结束(end-of-stream)
final?: boolean
// 解析 strong 时要求闭合 `**`(默认 false)
requireClosingStrong?: boolean
}
```
示例 —— 标记 AI "思考" 块:
```ts
const parseOptions = {
customHtmlTags: ['thinking'],
}
const nodes = parseMarkdownToStructure(markdown, md, parseOptions)
const tagged = nodes.map(node =>
node.type === 'html_block' && /<thinking>/.test((node as any).content ?? '')
? { ...node, meta: { type: 'thinking' } }
: node,
)
```
在渲染器中读取 `node.meta` 即可渲染自定义 UI,而无需直接修改 Markdown 文本。
示例 —— 限制链接协议,拦截不安全链接:
```ts
const md = getMarkdown('safe-links')
md.set?.({
validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()),
})
const nodes = parseMarkdownToStructure(
'[ok](https://example.com) [bad](javascript:alert(1))',
md,
{ final: true },
)
// "ok" 保持为 link 节点;"bad" 会降级为纯文本
```
### 未知 HTML 类标签
默认情况下,非标准的 HTML 类标签(例如 `<question>`)在完整闭合时会按原生 HTML 渲染(作为自定义元素输出)。未闭合或格式不完整的片段会保持为**纯文本**,避免在流式或最终渲染时吞掉周围内容。若希望它们作为自定义节点输出(`type: 'question'`,携带 attrs/content),需要在 `customHtmlTags` 中显式声明。
### 工具函数
#### `isMathLike(content)`
启发式函数,用于检测内容是否类似数学符号。
**参数:**
- `content` (string): 要检查的内容
**返回值:** `boolean`
#### `findMatchingClose(src, startIdx, open, close)`
在字符串中查找匹配的闭合分隔符,处理嵌套对。
**参数:**
- `src` (string): 源字符串
- `startIdx` (number): 开始搜索的索引
- `open` (string): 开启分隔符
- `close` (string): 闭合分隔符
**返回值:** `number` - 匹配闭合的索引,如果未找到则返回 -1
## 使用建议与排障
- **复用解析实例**:缓存 `getMarkdown()` 的结果,避免重复注册插件。
- **服务端解析**:在服务端运行 `parseMarkdownToStructure` 后把 AST 下发给客户端,配合 `markstream-vue` 实现确定性输出。
- **自定义 HTML 组件**:在解析前先把 `<MyWidget>` 这类片段替换为占位符,渲染时再注入,避免在 `html_block` 上进行脆弱的字符串操作。
- **样式提示**:如果将节点交给 `markstream-vue`,务必按照文档的 [CSS 排查清单](https://markstream-vue-docs.simonhe.me/zh/guide/troubleshooting#css-looks-wrong-start-here) 调整 reset / layer,防止 Tailwind/UnoCSS 覆盖样式。
- **错误处理**:`apply` 钩子内部默认捕获异常后打印日志,如需在 CI/生产中抛出错误,可在传入前自行封装并 rethrow。
#### `parseFenceToken(token)`
将代码围栏 token 解析为 CodeBlockNode。
**参数:**
- `token` (MarkdownToken): markdown-it token
**返回值:** `CodeBlockNode`
#### `normalizeStandaloneBackslashT(content, options?)`
规范化数学内容中的反斜杠-t 序列。
**参数:**
- `content` (string): 要规范化的内容
- `options` (MathOptions, 可选): 数学选项
**返回值:** `string`
### 低阶辅助函数
需要更细粒度地控制 token → AST 流程时,可直接使用以下导出:
```ts
import type { MarkdownToken } from 'stream-markdown-parser'
import {
parseInlineTokens,
processTokens
} from 'stream-markdown-parser'
const tokens: MarkdownToken[] = md.parse(markdown, {})
const nodes = processTokens(tokens)
// 或仅解析内联内容:
const inlineNodes = parseInlineTokens(tokens[0].children ?? [], tokens[0].content ?? '')
```
`processTokens` 即 `parseMarkdownToStructure` 内部使用的同一个转换器,可在自定义管线中复用,避免重复实现 Markdown-it 遍历。
### 插件函数
#### `applyMath(md, options?)`
将数学插件应用到 markdown-it 实例。
**参数:**
- `md` (MarkdownIt): markdown-it 实例
- `options` (MathOptions, 可选): 数学渲染选项
#### `applyContainers(md)`
将容器插件应用到 markdown-it 实例。
**参数:**
- `md` (MarkdownIt): markdown-it 实例
### 常量
#### `KATEX_COMMANDS`
用于转义的常用 KaTeX 命令数组。
#### `TEX_BRACE_COMMANDS`
使用大括号的 TeX 命令数组。
#### `ESCAPED_TEX_BRACE_COMMANDS`
用于正则表达式的 TEX_BRACE_COMMANDS 转义版本。
## 类型
所有 TypeScript 类型都已导出:
```typescript
import type {
// 节点类型
CodeBlockNode,
GetMarkdownOptions,
HeadingNode,
ListItemNode,
ListNode,
MathOptions,
ParagraphNode,
ParsedNode,
ParseOptions,
// ... 更多
} from 'stream-markdown-parser'
```
### 节点类型
解析器导出各种表示不同 Markdown 元素的节点类型:
- `TextNode`, `HeadingNode`, `ParagraphNode`
- `ListNode`, `ListItemNode`
- `CodeBlockNode`, `InlineCodeNode`
- `LinkNode`, `ImageNode`
- `BlockquoteNode`, `TableNode`
- `MathBlockNode`, `MathInlineNode`
- 以及更多...
## 默认插件
该包预配置了以下 markdown-it 插件:
- `markdown-it-sub` - 下标支持(`H~2~O`)
- `markdown-it-sup` - 上标支持(`x^2^`)
- `markdown-it-mark` - 高亮/标记支持(`==highlighted==`)
- `markdown-it-task-checkbox` - 任务列表支持(`- [ ] Todo`)
- `markdown-it-ins` - 插入标签支持(`++inserted++`)
- `markdown-it-footnote` - 脚注支持
- `markdown-it-container` - 自定义容器支持(`::: warning`, `::: tip` 等)
- 数学公式支持 - 使用 `$...$` 和 `$$...$$` 渲染 LaTeX 数学公式
## 框架集成
虽然该包与框架无关,但它被设计为可以无缝配合以下框架使用:
- ✅ **Node.js** - 服务器端渲染
- ✅ **Vue 3** - 配合 `markstream-vue`(或你的自定义渲染层)使用
- ✅ **React** - 使用解析的节点进行自定义渲染
- ✅ **Vanilla JS** - 直接 HTML 渲染
- ✅ **任何框架** - 解析为 AST 并按需渲染
## 从 `markstream-vue` 迁移(解析器导出)
如果你当前是从 `markstream-vue` 引入解析器相关 helper,可以切换为使用独立包:
```typescript
// 之前
import { getMarkdown } from 'markstream-vue'
// 现在
import { getMarkdown } from 'stream-markdown-parser'
```
所有 API 保持不变。详见[迁移指南](https://markstream-vue-docs.simonhe.me/zh/monorepo-migration)。
## 性能
- **轻量级**: ~65KB 压缩后(13KB gzipped)
- **快速**: 针对实时解析优化
- **Tree-shakeable**: 只导入你需要的部分
- **依赖很少**: `markdown-it-ts` + 少量 markdown-it 插件
## 贡献
欢迎提交 Issues 和 PRs!请阅读[贡献指南](https://github.com/Simon-He95/markstream-vue/blob/main/CONTRIBUTING.md)。
## 许可证
MIT © Simon He
## 相关项目
- [markstream-vue](https://www.npmjs.com/package/markstream-vue) - 功能完整的 Vue 3 Markdown 渲染器
@@ -0,0 +1,14 @@
/* eslint-disable */
// @ts-nocheck
// Generated by unplugin-vue-components
// Read more: https://github.com/vuejs/core/pull/3399
// biome-ignore lint: disable
export {}
/* prettier-ignore */
declare module 'vue' {
export interface GlobalComponents {
RouterLink: typeof import('vue-router')['RouterLink']
RouterView: typeof import('vue-router')['RouterView']
}
}
@@ -0,0 +1,70 @@
{
"name": "stream-markdown-parser",
"type": "module",
"version": "0.0.90",
"packageManager": "pnpm@10.33.0",
"description": "Pure markdown parser and renderer utilities with streaming support - framework agnostic",
"author": "Simon He",
"license": "MIT",
"homepage": "https://github.com/Simon-He95/markstream-vue/tree/main/packages/markdown-parser",
"repository": {
"type": "git",
"url": "git+https://github.com/Simon-He95/markstream-vue.git",
"directory": "packages/markdown-parser"
},
"bugs": {
"url": "https://github.com/Simon-He95/markstream-vue/issues"
},
"keywords": [
"markdown",
"markdown-it",
"markdown-it-ts",
"parser",
"markdown-parser",
"markdown-renderer",
"ast",
"markdown-ast",
"streaming",
"progressive",
"framework-agnostic"
],
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"main": "./dist/index.js",
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"files": [
"dist"
],
"scripts": {
"dev": "tsdown --watch",
"build": "tsdown",
"test": "vitest",
"test:run": "vitest --run",
"typecheck": "tsc --noEmit",
"lint": "eslint . --cache",
"lint:fix": "eslint . --cache --fix",
"prepublishOnly": "pnpm run build",
"release": "bumpp --commit --no-tag --no-push && npm publish --access public && node ../../scripts/tag-package.mjs --package-json package.json --push"
},
"dependencies": {
"markdown-it-container": "^4.0.0",
"markdown-it-footnote": "^4.0.0",
"markdown-it-ins": "^4.0.0",
"markdown-it-mark": "^4.0.0",
"markdown-it-sub": "^2.0.0",
"markdown-it-sup": "^2.0.0",
"markdown-it-task-checkbox": "^1.0.6",
"markdown-it-ts": "^0.0.9"
},
"devDependencies": {
"@types/node": "^22.19.15",
"tsdown": "^0.15.12",
"typescript": "^5.9.3",
"vitest": "^3.2.4"
}
}
@@ -0,0 +1,37 @@
/**
* MathOptions control how the math plugin normalizes content before
* handing it to KaTeX (or other math renderers).
*
* - commands: list of command words that should be auto-prefixed with a
* backslash if not already escaped (e.g. 'infty' -> '\\infty'). Use a
* conservative list to avoid false positives in prose.
* - escapeExclamation: whether to escape standalone '!' to '\\!' (default true).
*/
export interface MathOptions {
/** List of command words to auto-escape. */
commands?: readonly string[]
/** Whether to escape standalone '!' (default: true). */
escapeExclamation?: boolean
/**
* Strict delimiter mode.
* - When true, only explicit TeX delimiters are recognized as math:
* inline: `$...$` and `\\(...\\)`; block: `$$...$$` and `\\[...\\]`.
*
* Important: authors should write explicit TeX delimiters with escaped
* backslashes in source (for example, write `\\(...\\)` rather than
* an unescaped `\(...\)`). Unescaped `\(...\)` cannot be reliably
* distinguished from ordinary parentheses and may not be parsed as math.
* - Heuristics and mid-state (unclosed) math detection are disabled.
*/
strictDelimiters?: boolean
}
let defaultMathOptions: MathOptions | undefined
export function setDefaultMathOptions(opts: MathOptions | undefined) {
defaultMathOptions = opts
}
export function getDefaultMathOptions(): MathOptions | undefined {
return defaultMathOptions
}
@@ -0,0 +1,115 @@
import { STANDARD_HTML_TAGS } from './htmlTags'
import { escapeTagForRegExp } from './htmlTagUtils'
const HTML_LIKE_TAG_NAME_RE = /^[a-z][a-z0-9_-]*$/
export function isHtmlLikeTagName(tag: string) {
return HTML_LIKE_TAG_NAME_RE.test(String(tag ?? '').trim().toLowerCase())
}
export function normalizeCustomHtmlTagName(value: unknown): string {
const raw = String(value ?? '').trim()
if (!raw)
return ''
if (!raw.startsWith('<'))
return isHtmlLikeTagName(raw) ? raw.toLowerCase() : ''
let index = 1
while (index < raw.length && /\s/.test(raw[index]))
index++
if (raw[index] === '/') {
index++
while (index < raw.length && /\s/.test(raw[index]))
index++
}
const start = index
while (index < raw.length && /[\w-]/.test(raw[index]))
index++
const normalized = raw.slice(start, index).toLowerCase()
const next = raw[index] ?? ''
if (next && !/[\s/>]/.test(next))
return ''
return isHtmlLikeTagName(normalized) ? normalized : ''
}
export function normalizeCustomHtmlTags(tags?: readonly string[]) {
if (!tags || tags.length === 0)
return []
const seen = new Set<string>()
const normalized: string[] = []
for (const tag of tags) {
const value = normalizeCustomHtmlTagName(tag)
if (!value || seen.has(value))
continue
seen.add(value)
normalized.push(value)
}
return normalized
}
export function mergeCustomHtmlTags(...lists: Array<readonly string[] | undefined>) {
const seen = new Set<string>()
const normalized: string[] = []
for (const list of lists) {
for (const tag of normalizeCustomHtmlTags(list)) {
if (seen.has(tag))
continue
seen.add(tag)
normalized.push(tag)
}
}
return normalized
}
export function resolveCustomHtmlTags(tags?: readonly string[]) {
const normalized = normalizeCustomHtmlTags(tags)
return {
key: normalized.join(','),
tags: normalized,
}
}
export function getHtmlTagFromContent(html: unknown) {
return normalizeCustomHtmlTagName(html)
}
export function hasCompleteHtmlTagContent(html: unknown, tag: string) {
const raw = String(html ?? '')
const normalizedTag = normalizeCustomHtmlTagName(tag)
if (!normalizedTag)
return false
const escaped = escapeTagForRegExp(normalizedTag)
const openMatch = raw.match(new RegExp(String.raw`^\s*<\s*${escaped}(?:\s[^>]*)?(\s*\/)?>`, 'i'))
if (!openMatch)
return false
if (openMatch[1])
return true
return new RegExp(String.raw`<\s*\/\s*${escaped}\s*>`, 'i').test(raw)
}
export function shouldRenderUnknownHtmlTagAsText(html: unknown, tag: string) {
const normalizedTag = normalizeCustomHtmlTagName(tag)
return Boolean(normalizedTag)
&& !STANDARD_HTML_TAGS.has(normalizedTag)
&& !hasCompleteHtmlTagContent(html, normalizedTag)
}
export function stripCustomHtmlWrapper(html: unknown, tag: string) {
const raw = String(html ?? '')
const normalizedTag = normalizeCustomHtmlTagName(tag)
if (!normalizedTag)
return raw
const escaped = escapeTagForRegExp(normalizedTag)
const openRe = new RegExp(String.raw`^\s*<\s*${escaped}(?:\s[^>]*)?>\s*`, 'i')
const closeRe = new RegExp(String.raw`\s*<\s*\/\s*${escaped}\s*>\s*$`, 'i')
return raw.replace(openRe, '').replace(closeRe, '')
}
@@ -0,0 +1,64 @@
import type { MathOptions } from './config'
import MarkdownIt from 'markdown-it-ts'
import { getDefaultMathOptions } from './config'
import { applyContainers } from './plugins/containers'
import { applyFixHtmlInlineTokens } from './plugins/fixHtmlInline'
import { applyFixIndentedCodeBlock } from './plugins/fixIndentedCodeBlock'
import { applyFixLinkTokens } from './plugins/fixLinkTokens'
import { applyFixListItem } from './plugins/fixListItem'
import { applyFixStrongTokens } from './plugins/fixStrongTokens'
import { applyFixTableTokens } from './plugins/fixTableTokens'
import { applyMath } from './plugins/math'
import { applyRenderRules } from './renderers'
export interface FactoryOptions extends Record<string, unknown> {
markdownItOptions?: Record<string, unknown>
enableMath?: boolean
enableContainers?: boolean
mathOptions?: { commands?: string[], escapeExclamation?: boolean }
/**
* Custom HTML-like tag names that should participate in streaming mid-state
* suppression and be emitted as custom nodes (e.g. ['thinking']).
*/
customHtmlTags?: readonly string[]
/**
* Whether to enable the fix for indented code blocks that should be paragraphs.
* Default: true
*/
enableFixIndentedCodeBlock?: boolean
}
export function factory(opts: FactoryOptions = {}) {
const md = new MarkdownIt({
html: true,
linkify: true,
typographer: true,
stream: true,
...(opts.markdownItOptions ?? {}),
})
if (opts.enableMath ?? true) {
const mergedMathOptions: MathOptions = { ...(getDefaultMathOptions() ?? {}), ...(opts.mathOptions ?? {}) }
applyMath(md, mergedMathOptions)
}
if (opts.enableContainers ?? true)
applyContainers(md)
// Fix indented code blocks that should be paragraphs (streaming scenario)
if (opts.enableFixIndentedCodeBlock !== false)
applyFixIndentedCodeBlock(md)
// Retain the core-stage fix as a fallback for any cases the inline
// tokenizer does not handle.
applyFixLinkTokens(md)
// Also apply strong-token normalization at the same stage.
applyFixStrongTokens(md)
// Apply list-item inline normalization as well.
applyFixListItem(md)
// Apply table token normalization at block stage.
applyFixTableTokens(md)
applyRenderRules(md)
applyFixHtmlInlineTokens(md, {
customHtmlTags: opts.customHtmlTags,
})
return md
}
@@ -0,0 +1,65 @@
export function findMatchingClose(src: string, startIdx: number, open: string, close: string) {
const len = src.length
// Special-case $$ since it's a two-char delimiter that shouldn't
// be interpreted as nested parentheses.
if (open === '$$' && close === '$$') {
let i = startIdx
while (i < len - 1) {
if (src[i] === '$' && src[i + 1] === '$') {
// ensure not escaped
let k = i - 1
let backslashes = 0
while (k >= 0 && src[k] === '\\') {
backslashes++
k--
}
if (backslashes % 2 === 0)
return i
}
i++
}
return -1
}
const openChar = open[open.length - 1]
const closeSeq = close
let depth = 0
let i = startIdx
while (i < len) {
// If there's an unescaped close sequence here
if (src.slice(i, i + closeSeq.length) === closeSeq) {
let k = i - 1
let backslashes = 0
while (k >= 0 && src[k] === '\\') {
backslashes++
k--
}
if (backslashes % 2 === 0) {
if (depth === 0)
return i
depth--
i += closeSeq.length
continue
}
}
const ch = src[i]
// skip escaped characters
if (ch === '\\') {
i += 2
continue
}
if (ch === openChar) {
depth++
}
else if (ch === closeSeq[closeSeq.length - 1]) {
if (depth > 0)
depth--
}
i++
}
return -1
}
export default findMatchingClose
@@ -0,0 +1,428 @@
import {
BLOCKED_HTML_TAGS,
DANGEROUS_HTML_ATTRS,
EXTENDED_STANDARD_HTML_TAGS,
isUnsafeHtmlUrl,
URL_HTML_ATTRS,
VOID_HTML_TAGS,
} from './htmlTags'
export interface HtmlToken {
type: 'text' | 'tag_open' | 'tag_close' | 'self_closing'
tagName?: string
attrs?: Record<string, string>
content?: string
}
const CUSTOM_TAG_REGEX = /<([a-z][a-z0-9-]*)\b[^>]*>/gi
function hasOwn(obj: Record<string, unknown>, key: string) {
return Object.prototype.hasOwnProperty.call(obj, key)
}
function getString(value: unknown): string {
return typeof value === 'string'
? value
: value == null
? ''
: String(value)
}
function isSafeAttrName(value: string): boolean {
return /^[^\s"'<>`=]+$/.test(value) && !/^on/i.test(value)
}
function escapeHtml(value: unknown): string {
return getString(value)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;')
}
function escapeAttr(value: unknown): string {
return escapeHtml(value).replace(/`/g, '&#96;')
}
function normalizeTagName(tagName: string | undefined): string {
return String(tagName ?? '').trim().toLowerCase()
}
function serializeAttrs(attrs: Record<string, string>): string {
const pairs = Object.entries(attrs)
if (pairs.length === 0)
return ''
return pairs
.map(([name, value]) => value === '' ? ` ${name}` : ` ${name}="${escapeAttr(value)}"`)
.join('')
}
function sanitizeHtmlContentAttrs(attrs: Record<string, string>) {
const clean: Record<string, string> = {}
for (const [key, value] of Object.entries(attrs)) {
const safeName = key.trim()
const lowerKey = safeName.toLowerCase()
if (!safeName || !isSafeAttrName(safeName))
continue
if (DANGEROUS_HTML_ATTRS.has(lowerKey))
continue
if (URL_HTML_ATTRS.has(lowerKey) && value && isUnsafeHtmlUrl(value))
continue
clean[safeName] = value
}
return clean
}
export function isCustomHtmlComponentTag(
tagName: string,
customComponents: Record<string, unknown>,
) {
const lowerTag = tagName.toLowerCase()
if (EXTENDED_STANDARD_HTML_TAGS.has(lowerTag))
return false
return hasOwn(customComponents, lowerTag) || hasOwn(customComponents, tagName)
}
export function sanitizeHtmlAttrs(attrs: Record<string, string>) {
const clean: Record<string, string> = {}
for (const [key, value] of Object.entries(attrs)) {
const lowerKey = key.toLowerCase()
if (DANGEROUS_HTML_ATTRS.has(lowerKey))
continue
if (URL_HTML_ATTRS.has(lowerKey) && value && isUnsafeHtmlUrl(value))
continue
clean[key] = value
}
return clean
}
export function tokenAttrsToRecord(attrs?: Array<[string, string | null]> | null) {
const record: Record<string, string> = {}
if (!Array.isArray(attrs) || attrs.length === 0)
return record
for (const [key, value] of attrs) {
if (!key)
continue
record[String(key)] = value == null ? '' : String(value)
}
return record
}
export function sanitizeHtmlTokenAttrs(attrs?: Array<[string, string | null]> | null) {
const sanitized = sanitizeHtmlAttrs(tokenAttrsToRecord(attrs))
const pairs = Object.entries(sanitized).map(([key, value]) => [key, value] as [string, string])
return pairs.length > 0 ? pairs : undefined
}
export function convertHtmlPropValue(value: string, key: string): any {
const lowerKey = key.toLowerCase()
if (['checked', 'disabled', 'readonly', 'required', 'autofocus', 'multiple', 'hidden'].includes(lowerKey))
return value === 'true' || value === '' || value === key
if (['value', 'min', 'max', 'step', 'width', 'height', 'size', 'maxlength'].includes(lowerKey)) {
const num = Number(value)
if (value !== '' && !Number.isNaN(num))
return num
}
return value
}
export function convertHtmlAttrsToProps(attrs: Record<string, string>) {
const result: Record<string, any> = {}
for (const [key, value] of Object.entries(attrs))
result[key] = convertHtmlPropValue(value, key)
return result
}
function isMeaningfulText(text: string) {
return text.trim().length > 0
}
export function tokenizeHtml(html: string): HtmlToken[] {
const tokens: HtmlToken[] = []
let pos = 0
while (pos < html.length) {
if (html.startsWith('<!--', pos)) {
const commentEnd = html.indexOf('-->', pos)
if (commentEnd !== -1) {
pos = commentEnd + 3
continue
}
break
}
const tagStart = html.indexOf('<', pos)
if (tagStart === -1) {
if (pos < html.length) {
const remainingText = html.slice(pos)
if (isMeaningfulText(remainingText))
tokens.push({ type: 'text', content: remainingText })
}
break
}
if (tagStart > pos) {
const textContent = html.slice(pos, tagStart)
if (isMeaningfulText(textContent))
tokens.push({ type: 'text', content: textContent })
}
if (html.startsWith('![CDATA[', tagStart + 1)) {
const cdataEnd = html.indexOf(']]>', tagStart)
if (cdataEnd !== -1) {
tokens.push({ type: 'text', content: html.slice(tagStart, cdataEnd + 3) })
pos = cdataEnd + 3
continue
}
break
}
if (html.startsWith('!', tagStart + 1)) {
const specialEnd = html.indexOf('>', tagStart)
if (specialEnd !== -1) {
pos = specialEnd + 1
continue
}
break
}
const tagEnd = html.indexOf('>', tagStart)
if (tagEnd === -1)
break
const tagContent = html.slice(tagStart + 1, tagEnd).trim()
const isClosingTag = tagContent.startsWith('/')
const isSelfClosing = tagContent.endsWith('/')
if (isClosingTag) {
const tagName = tagContent.slice(1).trim()
tokens.push({ type: 'tag_close', tagName })
}
else {
const spaceIndex = tagContent.indexOf(' ')
let tagName: string
let attrsStr = ''
if (spaceIndex === -1) {
tagName = isSelfClosing ? tagContent.slice(0, -1).trim() : tagContent.trim()
}
else {
tagName = tagContent.slice(0, spaceIndex).trim()
attrsStr = tagContent.slice(spaceIndex + 1)
}
const attrs: Record<string, string> = {}
if (attrsStr) {
const attrRegex = /([^\s=]+)(?:=(?:"([^"]*)"|'([^']*)'|(\S*)))?/g
let attrMatch: RegExpExecArray | null
while ((attrMatch = attrRegex.exec(attrsStr)) !== null) {
const name = attrMatch[1]
const value = attrMatch[2] ?? attrMatch[3] ?? attrMatch[4] ?? ''
if (name && !name.endsWith('/'))
attrs[name] = value
}
}
tokens.push({
type: isSelfClosing || VOID_HTML_TAGS.has(tagName.toLowerCase()) ? 'self_closing' : 'tag_open',
tagName,
attrs,
})
}
pos = tagEnd + 1
}
return tokens
}
function tokenizeHtmlPreservingText(html: string): HtmlToken[] {
const tokens: HtmlToken[] = []
let pos = 0
while (pos < html.length) {
if (html.startsWith('<!--', pos)) {
const commentEnd = html.indexOf('-->', pos)
if (commentEnd !== -1) {
pos = commentEnd + 3
continue
}
break
}
const tagStart = html.indexOf('<', pos)
if (tagStart === -1) {
if (pos < html.length)
tokens.push({ type: 'text', content: html.slice(pos) })
break
}
if (tagStart > pos)
tokens.push({ type: 'text', content: html.slice(pos, tagStart) })
if (html.startsWith('![CDATA[', tagStart + 1)) {
const cdataEnd = html.indexOf(']]>', tagStart)
if (cdataEnd !== -1) {
tokens.push({ type: 'text', content: html.slice(tagStart, cdataEnd + 3) })
pos = cdataEnd + 3
continue
}
break
}
if (html.startsWith('!', tagStart + 1)) {
const specialEnd = html.indexOf('>', tagStart)
if (specialEnd !== -1) {
pos = specialEnd + 1
continue
}
break
}
const tagEnd = html.indexOf('>', tagStart)
if (tagEnd === -1)
break
const tagContent = html.slice(tagStart + 1, tagEnd).trim()
if (!tagContent) {
pos = tagEnd + 1
continue
}
const isClosingTag = tagContent.startsWith('/')
const isSelfClosing = tagContent.endsWith('/')
if (isClosingTag) {
const tagName = tagContent.slice(1).trim()
tokens.push({ type: 'tag_close', tagName })
pos = tagEnd + 1
continue
}
const spaceIndex = tagContent.indexOf(' ')
let tagName = ''
let attrsStr = ''
if (spaceIndex === -1) {
tagName = isSelfClosing ? tagContent.slice(0, -1).trim() : tagContent.trim()
}
else {
tagName = tagContent.slice(0, spaceIndex).trim()
attrsStr = tagContent.slice(spaceIndex + 1)
}
const attrs: Record<string, string> = {}
if (attrsStr) {
const attrRegex = /([^\s=]+)(?:=(?:"([^"]*)"|'([^']*)'|(\S*)))?/g
let attrMatch: RegExpExecArray | null
while ((attrMatch = attrRegex.exec(attrsStr)) !== null) {
const name = attrMatch[1]
const value = attrMatch[2] ?? attrMatch[3] ?? attrMatch[4] ?? ''
if (name && !name.endsWith('/'))
attrs[name] = value
}
}
tokens.push({
type: isSelfClosing || VOID_HTML_TAGS.has(tagName.toLowerCase()) ? 'self_closing' : 'tag_open',
tagName,
attrs,
})
pos = tagEnd + 1
}
return tokens
}
export function hasCustomHtmlComponents(
content: string,
customComponents: Record<string, unknown>,
) {
if (!content || !content.includes('<'))
return false
if (!customComponents || Object.keys(customComponents).length === 0)
return false
CUSTOM_TAG_REGEX.lastIndex = 0
let match: RegExpExecArray | null
while ((match = CUSTOM_TAG_REGEX.exec(content)) !== null) {
if (isCustomHtmlComponentTag(match[1], customComponents))
return true
}
return false
}
export function sanitizeHtmlContent(content: string): string {
if (!content)
return ''
const tokens = tokenizeHtmlPreservingText(content)
const stack: string[] = []
const output: string[] = []
let blockedDepth = 0
for (const token of tokens) {
if (token.type === 'text') {
if (blockedDepth === 0)
output.push(escapeHtml(token.content ?? ''))
continue
}
const tagName = normalizeTagName(token.tagName)
if (!tagName)
continue
if (BLOCKED_HTML_TAGS.has(tagName)) {
if (token.type === 'tag_open')
blockedDepth += 1
else if (token.type === 'tag_close' && blockedDepth > 0)
blockedDepth -= 1
continue
}
if (blockedDepth > 0)
continue
if (token.type === 'self_closing') {
output.push(`<${tagName}${serializeAttrs(sanitizeHtmlContentAttrs(token.attrs ?? {}))}>`)
continue
}
if (token.type === 'tag_open') {
output.push(`<${tagName}${serializeAttrs(sanitizeHtmlContentAttrs(token.attrs ?? {}))}>`)
if (!VOID_HTML_TAGS.has(tagName))
stack.push(tagName)
continue
}
const matchedIndex = stack.lastIndexOf(tagName)
if (matchedIndex === -1)
continue
while (stack.length > matchedIndex + 1) {
const danglingTag = stack.pop()
if (danglingTag)
output.push(`</${danglingTag}>`)
}
const closingTag = stack.pop()
if (closingTag)
output.push(`</${closingTag}>`)
}
while (stack.length > 0) {
const danglingTag = stack.pop()
if (danglingTag)
output.push(`</${danglingTag}>`)
}
return output.join('')
}
@@ -0,0 +1,44 @@
export function escapeTagForRegExp(tag: string) {
return tag.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
}
export function findTagCloseIndexOutsideQuotes(input: string) {
let inSingle = false
let inDouble = false
for (let i = 0; i < input.length; i++) {
const ch = input[i]
if (ch === '\\') {
i++
continue
}
if (!inDouble && ch === '\'') {
inSingle = !inSingle
continue
}
if (!inSingle && ch === '"') {
inDouble = !inDouble
continue
}
if (!inSingle && !inDouble && ch === '>')
return i
}
return -1
}
export function parseTagAttrs(openTag: string): [string, string][] {
const attrs: [string, string][] = []
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
let match
while ((match = attrRegex.exec(openTag)) !== null) {
const attrName = match[1]
if (!attrName)
continue
const attrValue = match[2] || match[3] || match[4] || ''
attrs.push([attrName, attrValue])
}
return attrs
}
@@ -0,0 +1,227 @@
export const VOID_HTML_TAG_NAMES = [
'area',
'base',
'br',
'col',
'embed',
'hr',
'img',
'input',
'link',
'meta',
'param',
'source',
'track',
'wbr',
] as const
export const INLINE_HTML_TAG_NAMES = [
'a',
'abbr',
'b',
'bdi',
'bdo',
'button',
'cite',
'code',
'data',
'del',
'dfn',
'em',
'font',
'i',
'ins',
'kbd',
'label',
'mark',
'q',
's',
'samp',
'small',
'span',
'strong',
'sub',
'sup',
'time',
'u',
'var',
] as const
export const BLOCK_HTML_TAG_NAMES = [
'article',
'aside',
'blockquote',
'details',
'div',
'figcaption',
'figure',
'footer',
'header',
'h1',
'h2',
'h3',
'h4',
'h5',
'h6',
'li',
'main',
'nav',
'ol',
'p',
'pre',
'section',
'summary',
'table',
'tbody',
'td',
'th',
'thead',
'tr',
'ul',
] as const
export const SVG_HTML_TAG_NAMES = [
'svg',
'g',
'path',
] as const
export const EXTENDED_STANDARD_HTML_TAG_NAMES = [
'address',
'audio',
'body',
'canvas',
'caption',
'colgroup',
'datalist',
'dd',
'dialog',
'dl',
'dt',
'fieldset',
'form',
'head',
'hgroup',
'html',
'iframe',
'legend',
'map',
'menu',
'meter',
'noscript',
'object',
'optgroup',
'option',
'output',
'picture',
'progress',
'rp',
'rt',
'ruby',
'script',
'select',
'style',
'template',
'textarea',
'tfoot',
'title',
'video',
] as const
export const DANGEROUS_HTML_ATTR_NAMES = [
'onclick',
'onerror',
'onload',
'onmouseover',
'onmouseout',
'onmousedown',
'onmouseup',
'onkeydown',
'onkeyup',
'onfocus',
'onblur',
'onsubmit',
'onreset',
'onchange',
'onselect',
'ondblclick',
'ontouchstart',
'ontouchend',
'ontouchmove',
'ontouchcancel',
'onwheel',
'onscroll',
'oncopy',
'oncut',
'onpaste',
'oninput',
'oninvalid',
'onsearch',
] as const
export const URL_HTML_ATTR_NAMES = [
'href',
'src',
'srcset',
'xlink:href',
'formaction',
] as const
export const BLOCKED_HTML_TAG_NAMES = [
'script',
] as const
export const NON_STRUCTURING_HTML_TAG_NAMES = [
'pre',
'script',
'style',
'textarea',
'title',
] as const
export const VOID_HTML_TAGS = new Set<string>(VOID_HTML_TAG_NAMES)
export const STANDARD_BLOCK_HTML_TAGS = new Set<string>(BLOCK_HTML_TAG_NAMES)
export const STANDARD_HTML_TAGS = new Set<string>([
...VOID_HTML_TAG_NAMES,
...INLINE_HTML_TAG_NAMES,
...BLOCK_HTML_TAG_NAMES,
...SVG_HTML_TAG_NAMES,
])
export const EXTENDED_STANDARD_HTML_TAGS = new Set<string>([
...STANDARD_HTML_TAGS,
...EXTENDED_STANDARD_HTML_TAG_NAMES,
])
export const DANGEROUS_HTML_ATTRS = new Set<string>(DANGEROUS_HTML_ATTR_NAMES)
export const URL_HTML_ATTRS = new Set<string>(URL_HTML_ATTR_NAMES)
export const BLOCKED_HTML_TAGS = new Set<string>(BLOCKED_HTML_TAG_NAMES)
export const NON_STRUCTURING_HTML_TAGS = new Set<string>(NON_STRUCTURING_HTML_TAG_NAMES)
export function stripHtmlControlAndWhitespace(value: string) {
let out = ''
for (const ch of value) {
const code = ch.charCodeAt(0)
if (code <= 0x1F || code === 0x7F)
continue
if (/\s/u.test(ch))
continue
out += ch
}
return out
}
export function isUnsafeHtmlUrl(value: string) {
const normalized = stripHtmlControlAndWhitespace(value).toLowerCase()
if (normalized.startsWith('javascript:') || normalized.startsWith('vbscript:'))
return true
if (normalized.startsWith('data:')) {
return !(
normalized.startsWith('data:image/')
|| normalized.startsWith('data:video/')
|| normalized.startsWith('data:audio/')
)
}
return false
}
@@ -0,0 +1,284 @@
import type { MarkdownIt, MarkdownItPlugin } from 'markdown-it-ts'
import type { FactoryOptions } from './factory'
import markdownItFootnote from 'markdown-it-footnote'
import markdownItIns from 'markdown-it-ins'
import markdownItMark from 'markdown-it-mark'
import markdownItSub from 'markdown-it-sub'
import markdownItSup from 'markdown-it-sup'
import * as markdownItCheckbox from 'markdown-it-task-checkbox'
import { factory } from './factory'
import {
parseInlineTokens,
parseMarkdownToStructure,
processTokens,
} from './parser'
// Module-level registry for callers that want to add plugins to every
// `getMarkdown()` instance without modifying call sites. Useful for tests
// or apps that want a central place to `use` plugins.
const _registeredMarkdownPlugins: Array<unknown> = []
export function registerMarkdownPlugin(plugin: unknown) {
_registeredMarkdownPlugins.push(plugin)
}
export function clearRegisteredMarkdownPlugins() {
_registeredMarkdownPlugins.length = 0
}
// Re-export config
export { setDefaultMathOptions } from './config'
// Re-export parser functions
export { parseInlineTokens, parseMarkdownToStructure, processTokens }
export type { MathOptions } from './config'
export type { MarkdownIt }
// Re-export utilities
export * from './customHtmlTags'
export { findMatchingClose } from './findMatchingClose'
export * from './htmlRenderUtils'
export * from './htmlTags'
export { parseFenceToken } from './parser/inline-parsers/fence-parser'
// Re-export plugins
export { applyContainers } from './plugins/containers'
export { ESCAPED_TEX_BRACE_COMMANDS, isMathLike, TEX_BRACE_COMMANDS } from './plugins/isMathLike'
export { applyMath, KATEX_COMMANDS, normalizeStandaloneBackslashT } from './plugins/math'
// Re-export the node types for backward compatibility
export * from './types'
export interface GetMarkdownOptions extends FactoryOptions {
plugin?: Array<unknown>
apply?: Array<(md: MarkdownIt) => void>
/**
* Custom translation function or translation map for UI texts
* @default { 'common.copy': 'Copy' }
*/
i18n?: ((key: string) => string) | Record<string, string>
}
export function getMarkdown(msgId: string = `editor-${Date.now()}`, options: GetMarkdownOptions = {}) {
// keep legacy behaviour but delegate to new factory and reapply project-specific rules
const md = factory(options)
// Setup i18n translator function
const defaultTranslations: Record<string, string> = {
'common.copy': 'Copy',
}
let t: (key: string) => string
if (typeof options.i18n === 'function') {
t = options.i18n
}
else if (options.i18n && typeof options.i18n === 'object') {
const i18nMap = options.i18n as Record<string, string>
t = (key: string) => i18nMap[key] ?? defaultTranslations[key] ?? key
}
else {
t = (key: string) => defaultTranslations[key] ?? key
}
// apply user supplied plugins (md.use)
if (Array.isArray(options.plugin)) {
for (const p of options.plugin) {
// allow both [plugin, opts] tuple or plugin function
const pluginItem = p as unknown
if (Array.isArray(pluginItem)) {
const fn = pluginItem[0]
const opts = pluginItem[1]
if (typeof fn === 'function')
md.use(fn, opts)
}
else if (typeof pluginItem === 'function') {
md.use(pluginItem as MarkdownItPlugin)
}
// otherwise ignore non-callable plugins
}
}
// apply user supplied apply functions to mutate the md instance (e.g. md.block.ruler.before(...))
if (Array.isArray(options.apply)) {
for (const fn of options.apply) {
try {
fn(md)
}
catch (e) {
// swallow errors to preserve legacy behaviour; developers can see stack in console
console.error('[getMarkdown] apply function threw an error', e)
}
}
}
// Apply any globally registered plugins (via registerMarkdownPlugin)
if (_registeredMarkdownPlugins.length) {
for (const p of _registeredMarkdownPlugins) {
if (Array.isArray(p)) {
const fn = p[0] as any
const opts = p[1]
if (typeof fn === 'function')
md.use(fn, opts)
}
else if (typeof p === 'function') {
md.use(p as MarkdownItPlugin)
}
// ignore non-callable registrations
}
}
// Re-apply a few project specific plugins that were previously always enabled
md.use(markdownItSub)
md.use(markdownItSup)
md.use(markdownItMark)
// Safely resolve default export or the module itself for checkbox plugin
type CheckboxPluginFn = (md: MarkdownIt, opts?: unknown) => void
const markdownItCheckboxPlugin = ((markdownItCheckbox as unknown) as {
default?: CheckboxPluginFn
}).default ?? markdownItCheckbox
md.use(markdownItCheckboxPlugin)
md.use(markdownItIns)
md.use(markdownItFootnote)
// Annotate fence tokens with unclosed meta using a lightweight line check
md.core.ruler.after('block', 'mark_fence_closed', (state: unknown) => {
const s = state as unknown as {
src: string
env?: Record<string, unknown>
tokens: Array<{ type?: string, map?: number[], markup?: string, meta?: Record<string, unknown> }>
}
const src: string = s.src
const envFinal = !!s.env?.__markstreamFinal
const lines = src.split(/\r?\n/)
for (const token of s.tokens) {
if (token.type !== 'fence' || !token.map || !token.markup)
continue
const openLine: number = token.map[0]
const endLine: number = token.map[1]
const markup: string = token.markup
const marker = markup[0]
const minLen = markup.length
// The closing line, if exists, should be the last line consumed by the block
const lineIdx = Math.max(0, endLine - 1)
const line = lines[lineIdx] ?? ''
let i = 0
while (i < line.length && (line[i] === ' ' || line[i] === '\t')) i++
let count = 0
while (i + count < line.length && line[i + count] === marker) count++
let j = i + count
while (j < line.length && (line[j] === ' ' || line[j] === '\t')) j++
const closed = envFinal ? true : (endLine > openLine + 1 && count >= minLen && j === line.length)
const tokenShape = token as unknown as { meta?: Record<string, unknown> }
tokenShape.meta = tokenShape.meta ?? {}
; (tokenShape.meta as Record<string, unknown>).unclosed = !closed
// also set a explicit `closed` boolean for compatibility with plugins/tests
; (tokenShape.meta as Record<string, unknown>).closed = !!closed
}
})
// wave rule (legacy)
const waveRule = (state: unknown, silent?: boolean) => {
const s = state as unknown as { pos: number, src: string, push: (type: string, tag?: string, nesting?: number) => any }
const start = s.pos
if (s.src[start] !== '~')
return false
const prevChar = s.src[start - 1]
const nextChar = s.src[start + 1]
if (/\d/.test(prevChar) && /\d/.test(nextChar)) {
if (!silent) {
const token = s.push('text', '', 0)
token.content = '~'
}
s.pos += 1
return true
}
return false
}
md.inline.ruler.before('sub', 'wave', waveRule)
// custom fence that uses msgId for unique ids
md.renderer.rules.fence = (tokens: unknown, idx: number) => {
const tokensAny = tokens as unknown as import('./types').MarkdownToken[]
const token = tokensAny[idx]
const tokenShape = token as unknown as { info?: string, content?: string }
const info = String(tokenShape.info ?? '').trim()
const str = String(tokenShape.content ?? '')
const encodedCode = btoa(unescape(encodeURIComponent(str)))
const language = String(info ?? 'text')
const uniqueId = `editor-${msgId}-${idx}-${language}`
return `<div class="code-block" data-code="${encodedCode}" data-lang="${language}" id="${uniqueId}">
<div class="code-header">
<span class="code-lang">${language.toUpperCase()}</span>
<button class="copy-button" data-code="${encodedCode}">${t(
'common.copy',
)}</button>
</div>
<div class="code-editor"></div>
</div>`
}
// reference rule (legacy)
const RE_REFERENCE = /^\[(\d+)\]/
const RE_REFERENCE_LABEL = /^\[([^\]\n]+)\]/
const shouldPreserveReferenceStyleLink = (afterMatch: string) => {
if (!afterMatch.startsWith('['))
return false
const nextLabelMatch = RE_REFERENCE_LABEL.exec(afterMatch)
if (!nextLabelMatch)
return afterMatch !== '[' && !/^\[\d+$/.test(afterMatch)
const nextLabel = String(nextLabelMatch[1] ?? '')
const remaining = afterMatch.slice(nextLabelMatch[0].length)
// `[1][label](url)` is two adjacent constructs: a numeric reference
// followed by a normal inline link.
if (remaining.startsWith('('))
return false
// `[1][2]` and `[1][2][3]` are treated as chained numeric references
// in this parser, while non-numeric labels should stay available for
// CommonMark reference-style links.
return !/^\d+$/.test(nextLabel)
}
const referenceInline = (state: unknown, silent?: boolean) => {
const s = state as unknown as { src: string, pos: number, push: (type: string, tag?: string, nesting?: number) => any }
if (s.src[s.pos] !== '[')
return false
const match = RE_REFERENCE.exec(s.src.slice(s.pos))
if (!match)
return false
// Avoid false positives in JSON-like snippets, e.g. `"pages": [14]`
const lookbehind = s.src.slice(Math.max(0, s.pos - 120), s.pos)
if (/"[^"\n]{1,80}"\s*:\s*$/.test(lookbehind))
return false
// Check if this is part of a link: `[text](...)`, `[[text]](...)`, or `[text][id]`
const afterMatch = s.src.slice(s.pos + match[0].length)
// `](` handles nested-bracket link text, e.g. `[[8]](url)` where the inner `[8]`
// must stay as text for the outer link parser to succeed.
if (afterMatch.startsWith('](') || afterMatch.startsWith('(') || shouldPreserveReferenceStyleLink(afterMatch))
return false
if (!silent) {
const id = match[1]
const token = s.push('reference', 'span', 0)
token.content = id
token.markup = match[0]
token.raw = match[0]
}
s.pos += match[0].length
return true
}
md.inline.ruler.before('escape', 'reference', referenceInline)
md.renderer.rules.reference = (tokens: unknown, idx: number) => {
const tokensAny = tokens as unknown as import('./types').MarkdownToken[]
const id = String(tokensAny[idx].content ?? '')
return `<span class="reference-link" data-reference-id="${id}" role="button" tabindex="0" title="Click to view reference">${id}</span>`
}
return md
}
@@ -0,0 +1,29 @@
import type { MarkdownToken, ParsedNode } from '../types'
export function processNestedBlocks(
tokens: MarkdownToken[],
startIndex: number,
): [ParsedNode | null, number] {
// This is a simplified version to break circular dependencies
// Nested lists are now properly handled in list-parser.ts with an enhanced parseNestedList function
let i = startIndex
let depth = 1
while (i < tokens.length && depth > 0) {
if (
tokens[i].type === 'bullet_list_open'
|| tokens[i].type === 'ordered_list_open'
) {
depth++
}
else if (
tokens[i].type === 'bullet_list_close'
|| tokens[i].type === 'ordered_list_close'
) {
depth--
}
i++
}
return [null, i]
}
@@ -0,0 +1,11 @@
export {
getHtmlTagFromContent,
hasCompleteHtmlTagContent,
isHtmlLikeTagName,
mergeCustomHtmlTags,
normalizeCustomHtmlTags,
normalizeCustomHtmlTagName as normalizeCustomTag,
resolveCustomHtmlTags,
shouldRenderUnknownHtmlTagAsText,
stripCustomHtmlWrapper,
} from '../customHtmlTags'
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,21 @@
import type { CheckboxInputNode, CheckboxNode, MarkdownToken } from '../../types'
export function parseCheckboxToken(token: MarkdownToken): CheckboxNode {
const tokenMeta = (token.meta ?? {}) as unknown as { checked?: boolean }
return {
type: 'checkbox',
checked: tokenMeta.checked === true,
raw: tokenMeta.checked ? '[x]' : '[ ]',
}
}
export function parseCheckboxInputToken(token: any): CheckboxInputNode {
const tokenAny = token as unknown as { attrGet?: (name: string) => string | undefined }
const rawAttr = tokenAny.attrGet ? tokenAny.attrGet('checked') : undefined
const checked = rawAttr === '' || rawAttr === 'true'
return {
type: 'checkbox_input',
checked,
raw: checked ? '[x]' : '[ ]',
}
}
@@ -0,0 +1,12 @@
import type { EmojiNode, MarkdownToken } from '../../types'
export function parseEmojiToken(token: MarkdownToken): EmojiNode {
const name = String(token.content ?? '')
const markup = String(token.markup ?? '')
return {
type: 'emoji',
name,
markup,
raw: `:${name}:`,
}
}
@@ -0,0 +1,37 @@
import type { EmphasisNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../index'
export function parseEmphasisToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: EmphasisNode
nextIndex: number
} {
const children: ParsedNode[] = []
let emText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between em_open and em_close
while (i < tokens.length && tokens[i].type !== 'em_close') {
emText += String(tokens[i].content ?? (tokens[i] as any)?.text ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const node: EmphasisNode = {
type: 'emphasis',
children,
raw: `*${emText}*`,
}
// Skip to after em_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,126 @@
import type { CodeBlockNode, MarkdownToken } from '../../types'
// Strip a final line that looks like a fence marker (``` etc.)
const TRAILING_FENCE_LINE_RE = /\r?\n[ \t]*`+\s*$/
// Unified diff metadata/header line prefixes to skip when splitting a diff
const DIFF_HEADER_PREFIXES = ['diff ', 'index ', '--- ', '+++ ', '@@ ']
// Newline splitter reused in this module
const NEWLINE_RE = /\r?\n/
function flushPendingDiffHunk(
orig: string[],
updated: string[],
pendingOrig: string[],
pendingUpdated: string[],
) {
if (pendingOrig.length > 0)
orig.push(...pendingOrig)
if (pendingUpdated.length > 0)
updated.push(...pendingUpdated)
pendingOrig.length = 0
pendingUpdated.length = 0
}
function splitUnifiedDiff(content: string, closed: boolean) {
const orig: string[] = []
const updated: string[] = []
const pendingOrig: string[] = []
const pendingUpdated: string[] = []
const lines = content.split(NEWLINE_RE)
const stableLineCount = Math.max(0, lines.length - 1)
const hasUnifiedDiffHeaders = lines.some(line =>
line.startsWith('diff ')
|| line.startsWith('--- ')
|| line.startsWith('+++ ')
|| line.startsWith('@@ '),
)
const processLine = (rawLine: string) => {
const line = rawLine
// skip diff metadata lines
if (DIFF_HEADER_PREFIXES.some(p => line.startsWith(p)))
return
if (line.startsWith('-')) {
const body = line.slice(1)
pendingOrig.push(!hasUnifiedDiffHeaders && body.startsWith(' ') ? ` ${body}` : body)
}
else if (line.startsWith('+')) {
const body = line.slice(1)
pendingUpdated.push(!hasUnifiedDiffHeaders && body.startsWith(' ') ? ` ${body}` : body)
}
else {
flushPendingDiffHunk(orig, updated, pendingOrig, pendingUpdated)
const contextLine = hasUnifiedDiffHeaders && line.startsWith(' ') ? line.slice(1) : line
orig.push(contextLine)
updated.push(contextLine)
}
}
for (let index = 0; index < stableLineCount; index++)
processLine(lines[index] ?? '')
if (closed && stableLineCount < lines.length)
processLine(lines[lines.length - 1] ?? '')
if (closed || (pendingOrig.length > 0 && pendingUpdated.length > 0))
flushPendingDiffHunk(orig, updated, pendingOrig, pendingUpdated)
return {
original: orig.join('\n'),
updated: updated.join('\n'),
}
}
export function parseFenceToken(token: MarkdownToken): CodeBlockNode {
const hasMap = Array.isArray(token.map) && token.map.length === 2
const tokenMeta = (token.meta ?? {}) as unknown as { closed?: boolean }
const closed = typeof tokenMeta.closed === 'boolean' ? tokenMeta.closed : undefined
const info = String(token.info ?? '')
const diff = info.startsWith('diff')
const language = diff
? (() => {
const s = info
const sp = s.indexOf(' ')
return sp === -1
? ''
: String(s.slice(sp + 1) ?? '')
})()
: info
// Defensive sanitization: sometimes a closing fence line (e.g. ``` or ``)
// can accidentally end up inside `token.content` (for example when
// the parser/mapping is confused). Remove a trailing line that only
// contains backticks and optional whitespace so we don't render stray
// ` or `` characters at the end of the code output. This is a
// conservative cleanup and only strips a final line that looks like a
// fence marker (starts with optional spaces then one or more ` and
// only whitespace until end-of-string).
let content = String(token.content ?? '')
if (TRAILING_FENCE_LINE_RE.test(content))
content = content.replace(TRAILING_FENCE_LINE_RE, '')
if (diff) {
const { original, updated } = splitUnifiedDiff(content, closed === true)
// 返回时保留原来的 code 字段为 updated(编辑后代码),并额外附加原始与更新的文本
return {
type: 'code_block',
language,
code: String(updated ?? ''),
raw: String(content ?? ''),
diff,
loading: closed === true ? false : closed === false ? true : !hasMap,
originalCode: original,
updatedCode: updated,
}
}
return {
type: 'code_block',
language,
code: String(content ?? ''),
raw: String(content ?? ''),
diff,
loading: closed === true ? false : closed === false ? true : !hasMap,
}
}
@@ -0,0 +1,12 @@
import type { FootnoteReferenceNode, MarkdownToken } from '../../types'
export function parseFootnoteRefToken(
token: MarkdownToken,
): FootnoteReferenceNode {
const tokenMeta = (token.meta ?? {}) as unknown as { label?: string }
return {
type: 'footnote_reference',
id: String(tokenMeta.label ?? ''),
raw: `[^${String(tokenMeta.label ?? '')}]`,
}
}
@@ -0,0 +1,8 @@
import type { HardBreakNode } from '../../types'
export function parseHardbreakToken(): HardBreakNode {
return {
type: 'hardbreak',
raw: '\\\n',
}
}
@@ -0,0 +1,37 @@
import type { HighlightNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../index'
export function parseHighlightToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: HighlightNode
nextIndex: number
} {
const children: ParsedNode[] = []
let markText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between mark_open and mark_close
while (i < tokens.length && tokens[i].type !== 'mark_close') {
markText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const node: HighlightNode = {
type: 'highlight',
children,
raw: `==${markText}==`,
}
// Skip to after mark_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,392 @@
import type { InlineCodeNode, MarkdownToken, ParsedNode } from '../../types'
import { VOID_HTML_TAGS } from '../../htmlTags'
import { parseTagAttrs } from '../../htmlTagUtils'
import { normalizeCustomTag } from '../customHtmlTags'
import { buildAllowedHtmlTagSet } from '../index'
type ParseInlineTokensFn = (
tokens: MarkdownToken[],
raw?: string,
pPreToken?: MarkdownToken,
options?: { requireClosingStrong?: boolean, customHtmlTags?: readonly string[] },
) => ParsedNode[]
interface TagSetCacheEntry {
customTagSet: Set<string> | null
allowedTagSet: Set<string>
}
let emptyTagSets: TagSetCacheEntry | null = null
const TAG_SET_CACHE = new WeakMap<readonly string[], TagSetCacheEntry>()
function getEmptyTagSets() {
if (!emptyTagSets) {
emptyTagSets = {
customTagSet: null,
allowedTagSet: buildAllowedHtmlTagSet(),
}
}
return emptyTagSets
}
function getTagName(html: string) {
const match = html.match(/^<\s*(?:\/\s*)?([\w-]+)/)
return match ? match[1].toLowerCase() : ''
}
function isClosingTag(html: string) {
return /^<\s*\//.test(html)
}
function isSelfClosing(tag: string, html: string) {
return /\/\s*>\s*$/.test(html) || VOID_HTML_TAGS.has(tag)
}
function getTagSets(customTags?: readonly string[]) {
if (!customTags || customTags.length === 0)
return getEmptyTagSets()
const cached = TAG_SET_CACHE.get(customTags)
if (cached)
return cached
const normalized = customTags.map(normalizeCustomTag).filter(Boolean)
if (!normalized.length) {
const entry = getEmptyTagSets()
TAG_SET_CACHE.set(customTags, entry)
return entry
}
const entry = {
customTagSet: new Set(normalized),
allowedTagSet: buildAllowedHtmlTagSet({ customHtmlTags: customTags as any }),
}
TAG_SET_CACHE.set(customTags, entry)
return entry
}
function tokenToRaw(token: MarkdownToken) {
const shape = token as { raw?: string, markup?: string, content?: string }
const raw = shape.raw ?? shape.content ?? shape.markup ?? ''
return String(raw ?? '')
}
type AttrTuple = [string, string]
function getAttrValue(attrs: AttrTuple[], name: string): string | undefined {
const lowerName = name.toLowerCase()
for (let i = attrs.length - 1; i >= 0; i--) {
const [key, value] = attrs[i]
if (String(key).toLowerCase() === lowerName)
return value
}
return undefined
}
function normalizeLinkAttrs(
attrs: AttrTuple[],
href: string,
title: string | null,
): AttrTuple[] {
const normalized = attrs.slice()
if (!getAttrValue(normalized, 'href'))
normalized.push(['href', href])
if (title != null && !getAttrValue(normalized, 'title'))
normalized.push(['title', title])
return normalized
}
function stringifyTokens(tokens: MarkdownToken[]) {
return tokens.map(tokenToRaw).join('')
}
function normalizeStandardHtmlChildren(children: ParsedNode[]) {
const normalized: ParsedNode[] = []
const pushText = (rawText: string) => {
const text = String(rawText ?? '')
if (!text)
return
const last = normalized[normalized.length - 1] as ParsedNode | undefined
if (last?.type === 'text') {
; (last as any).content = String((last as any).content ?? '') + text
; (last as any).raw = String((last as any).raw ?? '') + text
return
}
normalized.push({
type: 'text',
content: text,
raw: text,
} as ParsedNode)
}
for (const child of children) {
if (!child)
continue
if (child.type === 'reference' || child.type === 'footnote_reference') {
pushText(String((child as any).raw ?? ''))
continue
}
if (Array.isArray((child as any).children)) {
normalized.push({
...(child as any),
children: normalizeStandardHtmlChildren(((child as any).children ?? []) as ParsedNode[]),
} as ParsedNode)
continue
}
normalized.push(child)
}
return normalized
}
function findMatchingClosing(tokens: MarkdownToken[], startIndex: number, tag: string) {
let depth = 0
for (let idx = startIndex; idx < tokens.length; idx++) {
const t = tokens[idx]
if (t.type !== 'html_inline')
continue
const content = String(t.content ?? '')
const tTag = getTagName(content)
const closing = isClosingTag(content)
const selfClosing = isSelfClosing(tTag, content)
if (!closing && !selfClosing && tTag === tag) {
depth++
continue
}
if (closing && tTag === tag) {
if (depth === 0)
return idx
depth--
}
}
return -1
}
function collectHtmlFragment(tokens: MarkdownToken[], startIndex: number, tag: string) {
const openToken = tokens[startIndex]
const fragmentTokens: MarkdownToken[] = [openToken]
let innerTokens: MarkdownToken[] = []
let nextIndex = startIndex + 1
let closed = false
const closingIndex = tag ? findMatchingClosing(tokens, startIndex + 1, tag) : -1
if (closingIndex !== -1) {
innerTokens = tokens.slice(startIndex + 1, closingIndex)
fragmentTokens.push(...innerTokens, tokens[closingIndex])
nextIndex = closingIndex + 1
closed = true
}
else {
// Streaming mid-state: if no matching closing tag exists yet,
// treat all following inline tokens as the inner content of this tag.
innerTokens = tokens.slice(startIndex + 1)
if (innerTokens.length)
fragmentTokens.push(...innerTokens)
nextIndex = tokens.length
}
return {
closed,
html: stringifyTokens(fragmentTokens),
innerTokens,
nextIndex,
}
}
// Parse inline HTML and return an appropriate ParsedNode depending on tag.
export function parseHtmlInlineCodeToken(
token: MarkdownToken,
tokens: MarkdownToken[],
i: number,
parseInlineTokens: ParseInlineTokensFn,
raw?: string,
pPreToken?: MarkdownToken,
options?: { requireClosingStrong?: boolean, customHtmlTags?: readonly string[] },
): [ParsedNode, number] {
const code = String(token.content ?? '')
const tag = getTagName(code)
const { customTagSet, allowedTagSet } = getTagSets(options?.customHtmlTags)
if (!tag) {
return [
{
type: 'inline_code',
code,
raw: code,
} as InlineCodeNode,
i + 1,
]
}
// If it's not a standard HTML tag and not in customHtmlTags, default to
// rendering it as literal text (方案 A). However, if the tag is already
// properly closed within the inline token stream, keep it as HTML so
// HtmlInlineNode can still render custom components via HTML parsing.
if (!allowedTagSet.has(tag)) {
const fragment = collectHtmlFragment(tokens, i, tag)
if (!fragment.closed) {
const content = tokenToRaw(token)
return [{ type: 'text', content, raw: content } as ParsedNode, i + 1]
}
}
if (tag === 'br') {
return [
{
type: 'hardbreak',
raw: code,
} as ParsedNode,
i + 1,
]
}
const closing = isClosingTag(code)
const selfClosing = isSelfClosing(tag, code)
if (closing) {
return [
{
type: 'html_inline',
tag,
content: code,
children: [],
raw: code,
loading: false,
} as ParsedNode,
i + 1,
]
}
if (tag === 'a') {
const fragment = collectHtmlFragment(tokens, i, tag)
const attrs = parseTagAttrs(code)
const innerTokens = fragment.innerTokens
const href = String(getAttrValue(attrs, 'href') ?? '')
const titleAttr = getAttrValue(attrs, 'title')
const title = titleAttr == null ? null : String(titleAttr)
const normalizedAttrs = normalizeLinkAttrs(attrs, href, title)
const children = innerTokens.length
? parseInlineTokens(innerTokens, raw, pPreToken, options)
: []
const normalizedChildren = normalizeStandardHtmlChildren(children)
const textContent = innerTokens.length ? stringifyTokens(innerTokens) : href || ''
if (!normalizedChildren.length && textContent) {
normalizedChildren.push({
type: 'text',
content: textContent,
raw: textContent,
} as ParsedNode)
}
return [
{
type: 'link',
href,
title,
text: textContent,
attrs: normalizedAttrs,
children: normalizedChildren,
loading: !fragment.closed,
raw: fragment.html || code,
} as ParsedNode,
fragment.nextIndex,
]
}
if (selfClosing) {
const nodeType = customTagSet?.has(tag) ? tag : 'html_inline'
return [
{
type: nodeType,
tag,
content: code,
children: [],
raw: code,
loading: false,
} as ParsedNode,
i + 1,
]
}
const fragment = collectHtmlFragment(tokens, i, tag)
if (tag === 'p' || tag === 'div') {
const children = fragment.innerTokens.length
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
: []
const normalizedChildren = normalizeStandardHtmlChildren(children)
return [
{
type: 'paragraph',
children: normalizedChildren,
raw: fragment.html,
} as ParsedNode,
fragment.nextIndex,
]
}
const children = fragment.innerTokens.length
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
: []
const normalizedChildren = normalizeStandardHtmlChildren(children)
let content = fragment.html || code
let loading = !fragment.closed
let autoClosed = false
if (!fragment.closed) {
const closeTag = `</${tag}>`
if (!content.toLowerCase().includes(closeTag.toLowerCase()))
content += closeTag
autoClosed = true
// Still mark loading for mid-state, even though we auto-closed for rendering.
loading = true
}
const attrs = []
// 解析属性
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
let match
while ((match = attrRegex.exec(code)) !== null) {
const attrName = match[1]
const attrValue = match[2] || match[3] || match[4] || ''
attrs.push([attrName, attrValue])
}
if (customTagSet?.has(tag)) {
const _content = fragment.innerTokens.length
? stringifyTokens(fragment.innerTokens)
: ''
const customChildren = fragment.innerTokens.length
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
: []
return [
{
type: tag,
tag,
attrs,
content: _content,
children: customChildren,
raw: content,
loading: token.loading || loading,
autoClosed,
} as ParsedNode,
fragment.nextIndex,
]
}
return [
{
type: 'html_inline',
tag,
attrs,
content,
children: normalizedChildren,
raw: content,
loading,
autoClosed,
} as ParsedNode,
fragment.nextIndex,
]
}
@@ -0,0 +1,61 @@
import type { ImageNode, MarkdownToken } from '../../types'
export function parseImageToken(token: MarkdownToken, loading = false): ImageNode {
// Some call-sites pass an outer/inline token whose children contain the
// actual image token (with attrs). Prefer token.attrs when present; when
// absent, search children for the first child that carries attrs.
let attrs = token.attrs ?? []
// If the parent token has no attrs, prefer attrs from the inner child image
// token. Remember which child provided attrs so we can prefer its content
// over the parent's `token.content` (the parent may contain the raw
// markdown string like `![alt](src "title")`).
let childWithAttrs: any = null
if ((!attrs || attrs.length === 0) && Array.isArray(token.children)) {
for (const child of token.children) {
// child.attrs may be null in markdown-it; check and use if populated
const childAttrs = (child as any)?.attrs
if (Array.isArray(childAttrs) && childAttrs.length > 0) {
attrs = childAttrs
childWithAttrs = child
break
}
}
}
const src = String(attrs.find(attr => attr[0] === 'src')?.[1] ?? '')
const altAttr = attrs.find(attr => attr[0] === 'alt')?.[1]
// Prefer a non-empty alt attribute. If attrs were sourced from an inner
// child token prefer that child's `content` over the parent's `token.content`
// because the parent may contain the raw markdown instead of the plain alt
// text.
let alt = ''
if (altAttr != null && String(altAttr).length > 0) {
alt = String(altAttr)
}
else if (childWithAttrs?.content != null && String(childWithAttrs.content).length > 0) {
alt = String(childWithAttrs.content)
}
else if (Array.isArray(childWithAttrs?.children) && childWithAttrs.children[0]?.content) {
// If the inner image token has children (e.g. a text token) prefer that
// child's content when the child token's own `content` is empty.
alt = String(childWithAttrs.children[0].content)
}
else if (Array.isArray(token.children) && token.children[0]?.content) {
alt = String(token.children[0].content)
}
else if (token.content != null && String(token.content).length > 0) {
alt = String(token.content)
}
const _title = attrs.find(attr => attr[0] === 'title')?.[1] ?? null
const title = _title === null ? null : String(_title)
const raw = String(token.content ?? '')
return {
type: 'image',
src,
alt,
title,
raw,
loading,
}
}
@@ -0,0 +1,10 @@
import type { InlineCodeNode, MarkdownToken } from '../../types'
export function parseInlineCodeToken(token: MarkdownToken): InlineCodeNode {
const code = String(token.content ?? '')
return {
type: 'inline_code',
code,
raw: code,
}
}
@@ -0,0 +1,37 @@
import type { InsertNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../index'
export function parseInsertToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: InsertNode
nextIndex: number
} {
const children: ParsedNode[] = []
let insText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between ins_open and ins_close
while (i < tokens.length && tokens[i].type !== 'ins_close') {
insText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const node: InsertNode = {
type: 'insert',
children,
raw: `++${String(insText)}++`,
}
// Skip to after ins_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,112 @@
import type { LinkNode, MarkdownToken, ParseOptions } from '../../types'
import { parseInlineTokens } from '../index'
type AttrTuple = [string, string]
function toAttrsTuple(attrs: MarkdownToken['attrs']): AttrTuple[] {
const tuples: AttrTuple[] = []
if (!Array.isArray(attrs))
return tuples
for (const attr of attrs) {
const key = attr?.[0]
if (!key)
continue
tuples.push([String(key), String(attr?.[1] ?? '')])
}
return tuples
}
function getAttrValue(attrs: AttrTuple[], name: string): string | undefined {
const lowerName = name.toLowerCase()
for (let i = attrs.length - 1; i >= 0; i--) {
const [key, value] = attrs[i]
if (String(key).toLowerCase() === lowerName)
return value
}
return undefined
}
function normalizeLinkAttrs(
attrs: AttrTuple[],
href: string,
title: string | null,
): AttrTuple[] {
const normalized = attrs.slice()
if (!getAttrValue(normalized, 'href'))
normalized.push(['href', href])
if (title != null && !getAttrValue(normalized, 'title'))
normalized.push(['title', title])
return normalized
}
export function parseLinkToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: LinkNode
nextIndex: number
} {
const openToken = tokens[startIndex]
const attrsTuple = toAttrsTuple(openToken.attrs)
const href = String(getAttrValue(attrsTuple, 'href') ?? '')
const _title = getAttrValue(attrsTuple, 'title')
const title = _title == null ? null : String(_title)
const normalizedAttrs = normalizeLinkAttrs(attrsTuple, href, title)
let i = startIndex + 1
const linkTokens: MarkdownToken[] = []
let loading = true
// Collect all tokens between link_open and link_close
while (i < tokens.length && tokens[i].type !== 'link_close') {
linkTokens.push(tokens[i])
i++
}
if (tokens[i]?.type === 'link_close') {
loading = false
}
const lastLinkToken = linkTokens[linkTokens.length - 1]
if (
(options as any)?.__insideStrong
&& lastLinkToken?.type === 'text'
&& String(lastLinkToken.content ?? '').endsWith('**')
&& !linkTokens.some(token => token.type === 'strong_open')
) {
const originalContent = String(lastLinkToken.content ?? '')
const originalRaw = String((lastLinkToken as any).raw ?? originalContent)
lastLinkToken.content = originalContent.slice(0, -2)
lastLinkToken.raw = originalRaw.replace(/\*\*$/, '')
}
// Parse the collected tokens as inline content
const children = parseInlineTokens(linkTokens, undefined, undefined, options as any)
const linkText = children
.map((node) => {
const nodeAny = node as unknown as { content?: string, raw?: string }
if ('content' in node)
return String(nodeAny.content ?? '')
return String(nodeAny.raw ?? '')
})
.join('')
const node: LinkNode = {
type: 'link',
href,
title,
text: linkText,
children,
raw: String(`[${linkText}](${href}${title ? ` "${title}"` : ''})`),
loading,
attrs: normalizedAttrs,
}
// Skip to after link_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,14 @@
import type { MarkdownToken, MathInlineNode } from '../../types'
// Parse a math_inline token (inline math expressions)
export function parseMathInlineToken(token: MarkdownToken): MathInlineNode {
const content = token.content ?? ''
const raw = token.raw === '$$' ? `$${content}$` : token.raw || ''
return {
type: 'math_inline',
content,
loading: !!token.loading,
raw,
markup: token.markup,
}
}
@@ -0,0 +1,12 @@
import type { MarkdownToken, ReferenceNode } from '../../types'
// Parse a reference token from markdown-it
export function parseReferenceToken(token: MarkdownToken): ReferenceNode {
const id = String(token.content ?? '')
const raw = String(token.markup ?? `[${token.content ?? ''}]`)
return {
type: 'reference',
id,
raw,
}
}
@@ -0,0 +1,42 @@
import type {
MarkdownToken,
ParsedNode,
ParseOptions,
StrikethroughNode,
} from '../../types'
import { parseInlineTokens } from '../index'
export function parseStrikethroughToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: StrikethroughNode
nextIndex: number
} {
const children: ParsedNode[] = []
let sText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between s_open and s_close
while (i < tokens.length && tokens[i].type !== 's_close') {
sText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const node: StrikethroughNode = {
type: 'strikethrough',
children,
raw: `~~${sText}~~`,
}
// Skip to after s_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,71 @@
import type { MarkdownToken, ParsedNode, ParseOptions, StrongNode } from '../../types'
import { parseInlineTokens } from '../index'
const ESCAPED_PUNCTUATION_RE = /\\([\\()[\]`$|*_\-!])/g
function resolveInnerRaw(raw: string | undefined, strongText: string) {
if (!raw)
return undefined
const rawText = String(raw)
if (!rawText)
return undefined
if (rawText === strongText)
return rawText
const decodedRawText = rawText.replace(ESCAPED_PUNCTUATION_RE, '$1')
if (decodedRawText === strongText)
return rawText
return undefined
}
export function parseStrongToken(
tokens: MarkdownToken[],
startIndex: number,
raw?: string,
options?: ParseOptions,
): {
node: StrongNode
nextIndex: number
} {
const children: ParsedNode[] = []
let strongText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between strong_open and strong_close
// 这里可能会遇到多个 strong_open, 需要记录嵌套层级
let openCount = 1
while (i < tokens.length) {
if (tokens[i].type === 'strong_close') {
if (openCount === 1)
break
openCount--
}
if (tokens[i].type === 'strong_open') {
openCount++
}
strongText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, resolveInnerRaw(raw, strongText), undefined, {
...(options as any),
__insideStrong: true,
}))
const node: StrongNode = {
type: 'strong',
children,
raw: `**${String(strongText)}**`,
}
// Skip to after strong_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,48 @@
import type { MarkdownToken, ParsedNode, ParseOptions, SubscriptNode } from '../../types'
import { parseInlineTokens } from '../index'
export function parseSubscriptToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: SubscriptNode
nextIndex: number
} {
const children: ParsedNode[] = []
let subText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between sub_open and sub_close (if applicable)
while (i < tokens.length && tokens[i].type !== 'sub_close') {
subText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const startContent = String(tokens[startIndex].content ?? '')
const display = subText || startContent
const node: SubscriptNode = {
type: 'subscript',
children: children.length > 0
? children
: [
{
type: 'text',
// Fallback to the collected inner text (e.g., "2" in H~2~O)
content: display,
raw: display,
},
],
raw: `~${display}~`,
}
// Skip to after sub_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,47 @@
import type { MarkdownToken, ParsedNode, ParseOptions, SuperscriptNode } from '../../types'
import { parseInlineTokens } from '../index'
export function parseSuperscriptToken(
tokens: MarkdownToken[],
startIndex: number,
options?: ParseOptions,
): {
node: SuperscriptNode
nextIndex: number
} {
const children: ParsedNode[] = []
let supText = ''
let i = startIndex + 1
const innerTokens: MarkdownToken[] = []
// Process tokens between sup_open and sup_close (if applicable)
while (i < tokens.length && tokens[i].type !== 'sup_close') {
supText += String(tokens[i].content ?? '')
innerTokens.push(tokens[i])
i++
}
// Parse inner tokens to handle nested elements
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
const node: SuperscriptNode = {
type: 'superscript',
children:
children.length > 0
? children
: [
{
type: 'text',
// Fallback to the collected inner text (e.g., "2" in x^2^)
content: supText || String(tokens[startIndex].content ?? ''),
raw: supText || String(tokens[startIndex].content ?? ''),
},
],
raw: `^${supText || String(tokens[startIndex].content ?? '')}^`,
}
// Skip to after sup_close
const nextIndex = i < tokens.length ? i + 1 : tokens.length
return { node, nextIndex }
}
@@ -0,0 +1,10 @@
import type { MarkdownToken, TextNode } from '../../types'
export function parseTextToken(token: MarkdownToken): TextNode {
const content = String(token.content ?? '')
return {
type: 'text',
content,
raw: content,
}
}
@@ -0,0 +1,142 @@
const FILENAMEISH_EXTENSION_RE = /\.([a-z0-9]{1,10})$/i
const FILENAMEISH_SEGMENT_RE = /[_()[\]{}<>]/u
const URL_PREFIX_HINT_RE = /^(?:https?:\/\/|ftp:\/\/|mailto:|www\.)/i
const URL_QUERY_OR_AUTH_HINT_RE = /[?#@]/u
const PATH_SEPARATOR_RE = /[\\/]/u
const DOMAINISH_TEXT_RE = /^[\p{L}\p{N}./\\-]+$/u
const DOMAIN_LABEL_RE = /^(?!-)[A-Za-z0-9-]{1,63}(?<!-)$/u
const PUNYCODE_TLD_RE = /^xn--[a-z0-9-]{2,59}$/i
const AMBIGUOUS_BARE_DOMAIN_EXTENSIONS = new Set([
'ai',
'md',
'py',
'rs',
'sh',
'zip',
])
const FILENAMEISH_LINK_EXTENSIONS = new Set([
'7z',
'ai',
'astro',
'avi',
'bash',
'bz2',
'c',
'cjs',
'cpp',
'cs',
'csv',
'doc',
'docx',
'fish',
'flac',
'gif',
'go',
'gz',
'h',
'hpp',
'html',
'java',
'jpeg',
'jpg',
'js',
'json',
'jsx',
'kt',
'md',
'mdx',
'mjs',
'mov',
'mp3',
'mp4',
'pdf',
'php',
'png',
'ppt',
'pptx',
'ps1',
'py',
'rar',
'rb',
'rs',
'sh',
'sql',
'svg',
'swift',
'svelte',
'tar',
'tgz',
'toml',
'ts',
'tsx',
'txt',
'vue',
'wav',
'webp',
'xls',
'xlsx',
'xml',
'yaml',
'yml',
'zip',
'zsh',
])
function isPlausibleBareDomain(text: string) {
const labels = text.split('.')
if (labels.length < 2)
return false
const tld = labels[labels.length - 1]?.toLowerCase() ?? ''
if (!(DOMAIN_LABEL_RE.test(tld) || PUNYCODE_TLD_RE.test(tld)))
return false
return labels.every(label => DOMAIN_LABEL_RE.test(label))
}
function hasDomainAuthorityPrefix(text: string) {
const prefix = text.split(/[\\/]/)[0] ?? ''
return isPlausibleBareDomain(prefix)
}
function isUppercaseFilenameSegment(segment: string) {
const lettersOnly = segment.replace(/[^a-z]/gi, '')
return lettersOnly.length >= 2 && lettersOnly === lettersOnly.toUpperCase()
}
function hasStrongFilenameSignals(linkText: string) {
if (FILENAMEISH_SEGMENT_RE.test(linkText))
return true
if (!DOMAINISH_TEXT_RE.test(linkText))
return true
if (PATH_SEPARATOR_RE.test(linkText))
return !hasDomainAuthorityPrefix(linkText)
const extensionless = linkText.replace(FILENAMEISH_EXTENSION_RE, '')
const filenameLikeSegments = extensionless.split('.').filter(Boolean)
return filenameLikeSegments.some(isUppercaseFilenameSegment)
}
export function shouldDemoteFilenameLikeLinkify(linkText: string) {
if (!linkText || URL_PREFIX_HINT_RE.test(linkText) || URL_QUERY_OR_AUTH_HINT_RE.test(linkText))
return false
const extensionMatch = linkText.match(FILENAMEISH_EXTENSION_RE)
if (!extensionMatch)
return false
const extension = String(extensionMatch[1] ?? '').toLowerCase()
if (!FILENAMEISH_LINK_EXTENSIONS.has(extension))
return false
// Extensions like `.ai` and `.md` can be both real bare-domain TLDs and
// common filenames. Keep those linkified unless we also see stronger
// filename-only signals such as path separators, underscores, brackets, or
// all-uppercase filename segments like `README`.
if (!AMBIGUOUS_BARE_DOMAIN_EXTENSIONS.has(extension))
return true
return hasStrongFilenameSignals(linkText)
}
@@ -0,0 +1,66 @@
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
import { parseBasicBlockToken } from './block-token-parser'
import { parseBlockquote } from './blockquote-parser'
import { parseList } from './list-parser'
export function parseAdmonition(
tokens: MarkdownToken[],
index: number,
match: RegExpExecArray,
options?: ParseOptions,
): [AdmonitionNode, number] {
const kind = String(match[1] ?? 'note')
const title = String(match[2] ?? (kind.charAt(0).toUpperCase() + kind.slice(1)))
const admonitionChildren: ParsedNode[] = []
let j = index + 1
while (j < tokens.length && tokens[j].type !== 'container_close') {
if (tokens[j].type === 'paragraph_open') {
const contentToken = tokens[j + 1]
if (contentToken) {
admonitionChildren.push({
type: 'paragraph',
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
raw: String(contentToken.content ?? ''),
})
}
j += 3 // Skip paragraph_open, inline, paragraph_close
}
else if (
tokens[j].type === 'bullet_list_open'
|| tokens[j].type === 'ordered_list_open'
) {
const [listNode, newIndex] = parseList(tokens, j, options)
admonitionChildren.push(listNode)
j = newIndex
}
else if (tokens[j].type === 'blockquote_open') {
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
admonitionChildren.push(blockquoteNode)
j = newIndex
}
else {
const handled = parseBasicBlockToken(tokens, j, options)
if (handled) {
admonitionChildren.push(handled[0])
j = handled[1]
}
else {
j++
}
}
}
const admonitionNode: AdmonitionNode = {
type: 'admonition',
kind,
title,
children: admonitionChildren,
raw: `:::${kind} ${title}\n${admonitionChildren
.map(child => child.raw)
.join('\n')}\n:::`,
}
return [admonitionNode, j + 1] // Skip container_close
}
@@ -0,0 +1,518 @@
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions, VmrContainerNode } from '../../types'
import { escapeTagForRegExp, findTagCloseIndexOutsideQuotes } from '../../htmlTagUtils'
import { normalizeCustomTag } from '../customHtmlTags'
import { buildAllowedHtmlTagSet } from '../index'
import { parseInlineTokens } from '../inline-parsers'
import { parseFenceToken } from '../inline-parsers/fence-parser'
import { parseBlockquote } from './blockquote-parser'
import { parseCodeBlock } from './code-block-parser'
import { parseDefinitionList } from './definition-list-parser'
import { parseFootnote } from './footnote-parser'
import { parseHeading } from './heading-parser'
import { parseHtmlBlock } from './html-block-parser'
import { parseList } from './list-parser'
import { parseMathBlock } from './math-block-parser'
import { parseTable } from './table-parser'
import { parseThematicBreak } from './thematic-break-parser'
interface HtmlTagSetCacheEntry {
allowedTagSet: Set<string>
customTagSet: Set<string> | null
}
let emptyHtmlTagSets: HtmlTagSetCacheEntry | null = null
const HTML_TAG_SET_CACHE = new WeakMap<readonly string[], HtmlTagSetCacheEntry>()
function getEmptyHtmlTagSets() {
if (!emptyHtmlTagSets) {
emptyHtmlTagSets = {
allowedTagSet: buildAllowedHtmlTagSet(),
customTagSet: null,
}
}
return emptyHtmlTagSets
}
function getHtmlTagSets(customTags?: readonly string[]) {
if (!customTags || customTags.length === 0)
return getEmptyHtmlTagSets()
const cached = HTML_TAG_SET_CACHE.get(customTags)
if (cached)
return cached
const normalized = customTags.map(normalizeCustomTag).filter(Boolean)
if (!normalized.length) {
const entry = getEmptyHtmlTagSets()
HTML_TAG_SET_CACHE.set(customTags, entry)
return entry
}
const entry = {
allowedTagSet: buildAllowedHtmlTagSet({ customHtmlTags: customTags as any }),
customTagSet: new Set(normalized),
}
HTML_TAG_SET_CACHE.set(customTags, entry)
return entry
}
function parseVmrContainer(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [VmrContainerNode, number] {
const openToken = tokens[index]
// Extract name from class attribute (e.g., "vmr-container vmr-container-viewcode:topo-test-001")
// Handle multiple class values by looking for vmr-container-* anywhere in the string
const attrs = openToken.attrs as [string, string][] | null
let name = ''
const containerAttrs: Record<string, string> = {}
if (attrs) {
for (const [key, value] of attrs) {
if (key === 'class') {
// Match vmr-container-* anywhere in the class string, not just at the end
const match = value.match(/(?:\s|^)vmr-container-(\S+)/)
if (match) {
name = match[1]
}
}
else if (key.startsWith('data-')) {
// Extract data attributes (e.g., "data-devId" -> "devId")
const attrName = key.slice(5)
// Try to parse JSON values; fallback to raw string if parsing fails
try {
containerAttrs[attrName] = JSON.parse(value)
}
catch {
containerAttrs[attrName] = value
}
}
}
}
const children: ParsedNode[] = []
let j = index + 1
// Parse children until we find the closing token
while (j < tokens.length && tokens[j].type !== 'vmr_container_close') {
if (tokens[j].type === 'paragraph_open') {
// Handle paragraph tokens (plain text)
const contentToken = tokens[j + 1]
if (contentToken) {
const childrenArr = (contentToken.children as MarkdownToken[]) || []
children.push({
type: 'paragraph',
children: parseInlineTokens(childrenArr || [], undefined, undefined, options as any),
raw: String(contentToken.content ?? ''),
})
}
j += 3 // Skip paragraph_open, inline, paragraph_close
}
else if (
tokens[j].type === 'bullet_list_open'
|| tokens[j].type === 'ordered_list_open'
) {
// Handle list tokens
const [listNode, newIndex] = parseList(tokens, j, options)
children.push(listNode)
j = newIndex
}
else if (tokens[j].type === 'blockquote_open') {
// Handle blockquote tokens
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
children.push(blockquoteNode)
j = newIndex
}
else {
// Handle other basic block tokens (heading, code_block, fence, etc.)
const handled = parseBasicBlockToken(tokens, j, options)
if (handled) {
children.push(handled[0])
j = handled[1]
}
else {
j++
}
}
}
const hasCloseToken = j < tokens.length && tokens[j].type === 'vmr_container_close'
const closed = hasCloseToken || !!options?.final
// Build raw content
let raw = `::: ${name}`
if (Object.keys(containerAttrs).length > 0) {
raw += ` ${JSON.stringify(containerAttrs)}`
}
raw += '\n'
if (children.length > 0) {
// Prefer original Markdown (openToken.raw); rebuilding from tokens may lose formatting.
raw += openToken.raw ?? children.map(c => c.raw).join('\n')
raw += '\n'
}
raw += ':::'
const containerNode: VmrContainerNode = {
type: 'vmr_container',
name,
loading: !closed,
attrs: Object.keys(containerAttrs).length > 0 ? containerAttrs : undefined,
children,
raw,
}
// Skip the closing token when present; otherwise we're already at the end.
return [containerNode, hasCloseToken ? (j + 1) : j]
}
function stripWrapperNewlines(s: string) {
// Preserve inner whitespace/indentation, but drop a single leading/trailing newline
// introduced by the common pattern: <tag>\n...\n</tag>
return s.replace(/^\r?\n/, '').replace(/\r?\n$/, '')
}
function stripTrailingPartialClosingTag(inner: string, tag: string) {
if (!inner || !tag)
return inner
const re = new RegExp(String.raw`[\t ]*<\s*\/\s*${tag}[^>]*$`, 'i')
return inner.replace(re, '')
}
function findMatchingCloseTagRange(
rawHtml: string,
tag: string,
startIndex: number,
) {
if (!rawHtml || !tag)
return null
const lowerTag = tag.toLowerCase()
const openTagRe = new RegExp(String.raw`^<\s*${escapeTagForRegExp(lowerTag)}(?=\s|>|/)`, 'i')
const closeTagRe = new RegExp(String.raw`^<\s*\/\s*${escapeTagForRegExp(lowerTag)}(?=\s|>)`, 'i')
let depth = 0
let index = Math.max(0, startIndex)
while (index < rawHtml.length) {
const lt = rawHtml.indexOf('<', index)
if (lt === -1)
break
const slice = rawHtml.slice(lt)
if (closeTagRe.test(slice)) {
const endRel = findTagCloseIndexOutsideQuotes(slice)
if (endRel === -1)
return null
if (depth === 0) {
return {
start: lt,
end: lt + endRel + 1,
}
}
depth--
index = lt + endRel + 1
continue
}
if (openTagRe.test(slice)) {
const endRel = findTagCloseIndexOutsideQuotes(slice)
if (endRel === -1)
return null
const raw = slice.slice(0, endRel + 1)
if (!/\/\s*>$/.test(raw))
depth++
index = lt + endRel + 1
continue
}
index = lt + 1
}
return null
}
function findNextCustomHtmlBlockFromSource(
source: string,
tag: string,
startIndex: number,
): { raw: string, end: number } | null {
if (!source || !tag)
return null
const lowerTag = tag.toLowerCase()
const openRe = new RegExp(String.raw`<\s*${lowerTag}(?=\s|>|/)`, 'gi')
openRe.lastIndex = Math.max(0, startIndex || 0)
const openMatch = openRe.exec(source)
if (!openMatch || openMatch.index == null)
return null
const openStart = openMatch.index
const openSlice = source.slice(openStart)
const openEndRel = findTagCloseIndexOutsideQuotes(openSlice)
if (openEndRel === -1)
return null
const openEnd = openStart + openEndRel
// Self-closing custom tags: treat as a complete block
if (/\/\s*>\s*$/.test(openSlice.slice(0, openEndRel + 1))) {
const end = openEnd + 1
return { raw: source.slice(openStart, end), end }
}
let depth = 1
let i = openEnd + 1
const isOpenAt = (pos: number) => {
const s = source.slice(pos)
return new RegExp(String.raw`^<\s*${lowerTag}(?=\s|>|/)`, 'i').test(s)
}
const isCloseAt = (pos: number) => {
const s = source.slice(pos)
return new RegExp(String.raw`^<\s*\/\s*${lowerTag}(?=\s|>)`, 'i').test(s)
}
while (i < source.length) {
const lt = source.indexOf('<', i)
if (lt === -1) {
// No more tags in the remaining source; treat as unclosed streaming block.
return { raw: source.slice(openStart), end: source.length }
}
if (isCloseAt(lt)) {
const gt = source.indexOf('>', lt)
if (gt === -1)
return null
depth--
if (depth === 0) {
const end = gt + 1
return { raw: source.slice(openStart, end), end }
}
i = gt + 1
continue
}
if (isOpenAt(lt)) {
const rel = findTagCloseIndexOutsideQuotes(source.slice(lt))
if (rel === -1)
return null
depth++
i = lt + rel + 1
continue
}
i = lt + 1
}
// If the closing tag hasn't arrived yet (streaming), return a partial block
// from the opening tag to end-of-source. This preserves original lines like
// `---` so inner markdown can render progressively.
return { raw: source.slice(openStart), end: source.length }
}
function clampNonNegative(n: number) {
return Number.isFinite(n) && n > 0 ? n : 0
}
function lineToIndex(source: string, line: number) {
// markdown-it token.map uses 0-based line numbers.
const targetLine = clampNonNegative(line)
if (!source || targetLine <= 0)
return 0
let currentLine = 0
for (let i = 0; i < source.length; i++) {
if (source[i] === '\n') {
currentLine++
if (currentLine === targetLine)
return i + 1
}
}
return source.length
}
export function parseBasicBlockToken(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [ParsedNode, number] | null {
const token = tokens[index]
switch (token.type) {
case 'heading_open':
return [parseHeading(tokens, index, options), index + 3]
case 'code_block': {
return [parseCodeBlock(token), index + 1]
}
case 'fence':
return [parseFenceToken(token), index + 1]
case 'math_block':
return [parseMathBlock(token), index + 1]
case 'html_block': {
const htmlBlockNode = parseHtmlBlock(token)
const tagSets = htmlBlockNode.tag ? getHtmlTagSets(options?.customHtmlTags) : null
if (htmlBlockNode.tag && htmlBlockNode.loading && tagSets && !tagSets.allowedTagSet.has(htmlBlockNode.tag)) {
const raw = String((token as any)?.content ?? '')
const content = raw.replace(/\n+$/, '')
return [
{
type: 'paragraph',
children: content ? [{ type: 'text', content, raw: content }] : [],
raw: content,
} as any,
index + 1,
]
}
if (htmlBlockNode.tag && tagSets?.customTagSet?.has(htmlBlockNode.tag)) {
const tag = htmlBlockNode.tag
// markdown-it can normalize html_block token.content and lose original lines.
// Re-extract the next full <tag>...</tag> block from the original source.
const source = String((options as any)?.__sourceMarkdown ?? '')
const cursor = Number((options as any)?.__customHtmlBlockCursor ?? 0)
// If markdown-it provides a source map for this token, prefer anchoring the
// re-extraction to that line range. This avoids accidentally matching an
// earlier occurrence of the same custom tag that was tokenized as inline.
const mappedLineStart = Array.isArray((token as any).map)
? lineToIndex(source, Number((token as any).map?.[0] ?? 0))
: 0
const searchStart = Math.max(clampNonNegative(cursor), clampNonNegative(mappedLineStart))
const fromSource = findNextCustomHtmlBlockFromSource(source, tag, searchStart)
if (fromSource)
(options as any).__customHtmlBlockCursor = fromSource.end
const rawHtml = String(fromSource?.raw ?? htmlBlockNode.raw ?? '')
const openEnd = findTagCloseIndexOutsideQuotes(rawHtml)
const openTag = openEnd !== -1 ? rawHtml.slice(0, openEnd + 1) : rawHtml
const selfClosing = openEnd !== -1 && /\/\s*>\s*$/.test(openTag)
const closeRange = openEnd === -1
? null
: findMatchingCloseTagRange(rawHtml, tag, openEnd + 1)
const closeIndex = closeRange?.start ?? -1
let inner = ''
if (openEnd !== -1) {
if (closeIndex !== -1 && openEnd < closeIndex)
inner = rawHtml.slice(openEnd + 1, closeIndex)
else
inner = rawHtml.slice(openEnd + 1)
}
// Streaming mid-state: if the closing tag is being typed but not yet
// complete, don't leak the partial `</tag` into content.
if (closeIndex === -1)
inner = stripTrailingPartialClosingTag(inner, tag)
// Parse attrs from the opening tag only
const attrs: [string, string][] = []
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
let m
while ((m = attrRegex.exec(openTag)) !== null) {
const name = m[1]
if (!name || name.toLowerCase() === tag)
continue
const value = m[2] || m[3] || m[4] || ''
attrs.push([name, value])
}
const loading = !options?.final && !selfClosing && closeRange == null
return [
{
type: tag,
tag,
content: stripWrapperNewlines(inner),
raw: String(fromSource?.raw ?? htmlBlockNode.raw ?? rawHtml),
loading,
attrs: attrs.length ? attrs : undefined,
} as ParsedNode,
index + 1,
]
}
return [htmlBlockNode, index + 1]
}
case 'table_open': {
const [tableNode, newIndex] = parseTable(tokens, index, options)
return [tableNode, newIndex]
}
case 'dl_open': {
const [definitionListNode, newIndex] = parseDefinitionList(tokens, index, options)
return [definitionListNode, newIndex]
}
case 'footnote_open': {
const [footnoteNode, newIndex] = parseFootnote(tokens, index, options)
return [footnoteNode, newIndex]
}
case 'hr':
return [parseThematicBreak(), index + 1]
default:
break
}
return null
}
type ContainerParser = (
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
) => [AdmonitionNode, number]
type ContainerMatcher = (
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
) => [AdmonitionNode, number] | null
export function parseCommonBlockToken(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
handlers?: {
parseContainer?: ContainerParser
matchAdmonition?: ContainerMatcher
},
): [ParsedNode, number] | null {
const basicResult = parseBasicBlockToken(tokens, index, options)
if (basicResult)
return basicResult
const token = tokens[index]
switch (token.type) {
case 'container_warning_open':
case 'container_info_open':
case 'container_note_open':
case 'container_tip_open':
case 'container_danger_open':
case 'container_caution_open':
case 'container_error_open': {
if (handlers?.parseContainer)
return handlers.parseContainer(tokens, index, options)
break
}
case 'container_open': {
if (handlers?.matchAdmonition) {
const result = handlers.matchAdmonition(tokens, index, options)
if (result)
return result
}
break
}
case 'vmr_container_open': {
return parseVmrContainer(tokens, index, options)
}
default:
break
}
return null
}
@@ -0,0 +1,63 @@
import type { BlockquoteNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
import { parseCommonBlockToken } from './block-token-parser'
import { containerTokenHandlers } from './container-token-handlers'
import { parseList } from './list-parser'
export function parseBlockquote(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [BlockquoteNode, number] {
const blockquoteChildren: ParsedNode[] = []
let j = index + 1
// Process blockquote content until closing tag is found
while (j < tokens.length && tokens[j].type !== 'blockquote_close') {
const token = tokens[j]
switch (token.type) {
case 'paragraph_open': {
const contentToken = tokens[j + 1]
blockquoteChildren.push({
type: 'paragraph',
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
raw: String(contentToken.content ?? ''),
})
j += 3 // Skip paragraph_open, inline, paragraph_close
break
}
case 'bullet_list_open':
case 'ordered_list_open': {
const [listNode, newIndex] = parseList(tokens, j, options)
blockquoteChildren.push(listNode)
j = newIndex
break
}
case 'blockquote_open': {
const [nestedBlockquote, newIndex] = parseBlockquote(tokens, j, options)
blockquoteChildren.push(nestedBlockquote)
j = newIndex
break
}
default:{
const handled = parseCommonBlockToken(tokens, j, options, containerTokenHandlers)
if (handled) {
blockquoteChildren.push(handled[0])
j = handled[1]
}
else {
j++
}
break
}
}
}
const blockquoteNode: BlockquoteNode = {
type: 'blockquote',
children: blockquoteChildren,
raw: blockquoteChildren.map(child => child.raw).join('\n'),
}
return [blockquoteNode, j + 1] // Skip blockquote_close
}
@@ -0,0 +1,29 @@
import type { CodeBlockNode, MarkdownToken } from '../../types'
import { parseFenceToken } from '../inline-parsers/fence-parser'
export function parseCodeBlock(token: MarkdownToken): CodeBlockNode {
// If this code block is actually a diff (some markdown-it backends
// classify fences vs code_block differently), delegate to the
// fence parser to preserve original/updated fields.
if (token.info?.startsWith('diff')) {
return parseFenceToken(token)
}
const contentStr = String(token.content ?? '')
const match = contentStr.match(/ type="application\/vnd\.ant\.([^"]+)"/)
if (match?.[1]) {
// 需要把 <antArtifact> 标签去掉
// mutate token.content safely by assigning the cleaned string
token.content = contentStr
.replace(/<antArtifact[^>]*>/g, '')
.replace(/<\/antArtifact>/g, '')
}
const hasMap = Array.isArray(token.map) && token.map.length === 2
return {
type: 'code_block',
language: match ? match[1] : String(token.info ?? ''),
code: String(token.content ?? ''),
raw: String(token.content ?? ''),
loading: !hasMap,
}
}
@@ -0,0 +1,137 @@
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions, TextNode } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
import { parseBasicBlockToken } from './block-token-parser'
import { parseBlockquote } from './blockquote-parser'
import { parseList } from './list-parser'
const CONTAINER_KINDS = new Set(['warning', 'info', 'note', 'tip', 'danger', 'caution'])
function parseContainerInfo(info: string) {
let markerEnd = 0
while (markerEnd < info.length && markerEnd < 3 && info[markerEnd] === ':')
markerEnd++
if (markerEnd === 0 || info[markerEnd] === ':')
return null
const rest = info.slice(markerEnd).trimStart()
if (!rest)
return null
const firstWhitespace = rest.search(/\s/)
const rawKind = (firstWhitespace === -1 ? rest : rest.slice(0, firstWhitespace)).toLowerCase()
if (!CONTAINER_KINDS.has(rawKind))
return null
const title = firstWhitespace === -1 ? '' : rest.slice(firstWhitespace).trim()
return { kind: rawKind, title }
}
export function parseContainer(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [AdmonitionNode, number] {
const openToken = tokens[index]
// Determine kind and optional title
let kind = 'note'
let title = ''
const typeMatch = openToken.type.match(/^container_(\w+)_open$/)
if (typeMatch) {
kind = typeMatch[1]
// some implementations set info to remaining title text
const info = String(openToken.info ?? '').trim()
if (info && !info.startsWith(':::')) {
// if info looks like 'warning title', drop leading kind token
const lowerInfo = info.toLowerCase()
if (lowerInfo.startsWith(kind)) {
const maybe = info.slice(kind.length).trim()
if (maybe)
title = maybe
}
}
}
else {
// container_open: info usually contains the marker like ' warning Title'
const info = String(openToken.info ?? '').trim()
const parsedInfo = parseContainerInfo(info)
if (parsedInfo) {
kind = parsedInfo.kind
title = parsedInfo.title
}
}
if (!title)
title = kind.charAt(0).toUpperCase() + kind.slice(1)
const children: ParsedNode[] = []
let j = index + 1
// Accept closing tokens: 'container_close' or 'container_<kind>_close'
const closeType = new RegExp(`^container_${kind}_close$`)
while (
j < tokens.length
&& tokens[j].type !== 'container_close'
&& !closeType.test(tokens[j].type)
) {
if (tokens[j].type === 'paragraph_open') {
const contentToken = tokens[j + 1]
if (contentToken) {
const childrenArr = (contentToken.children as MarkdownToken[]) || []
let i = -1
for (let k = childrenArr.length - 1; k >= 0; k--) {
const t = childrenArr[k] as TextNode
if (t.type === 'text' && /:+/.test(t.content)) {
i = k
break
}
}
const _children = i !== -1 ? childrenArr.slice(0, i) : childrenArr
children.push({
type: 'paragraph',
children: parseInlineTokens(_children || [], undefined, undefined, options as any),
raw: String(contentToken.content ?? '').replace(/\n:+$/, '').replace(/\n\s*:::\s*$/, ''),
})
}
j += 3
}
else if (
tokens[j].type === 'bullet_list_open'
|| tokens[j].type === 'ordered_list_open'
) {
const [listNode, newIndex] = parseList(tokens, j, options)
children.push(listNode)
j = newIndex
}
else if (tokens[j].type === 'blockquote_open') {
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
children.push(blockquoteNode)
j = newIndex
}
else {
const handled = parseBasicBlockToken(tokens, j, options)
if (handled) {
children.push(handled[0])
j = handled[1]
}
else {
j++
}
}
}
const admonitionNode: AdmonitionNode = {
type: 'admonition',
kind,
title,
children,
raw: `:::${kind} ${title}\n${children.map(c => c.raw).join('\n')}\n:::`,
}
// Skip the closing token
const closingIndex = j
return [admonitionNode, closingIndex + 1]
}
@@ -0,0 +1,29 @@
import type { AdmonitionNode, MarkdownToken, ParseOptions } from '../../types'
import { parseAdmonition } from './admonition-parser'
import { parseContainer } from './container-parser'
const CONTAINER_REGEX
= /^::: ?(warning|info|note|tip|danger|caution|error) ?(.*)$/
function handleContainerOpen(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [AdmonitionNode, number] | null {
const token = tokens[index]
if (token.type !== 'container_open')
return null
const match = CONTAINER_REGEX.exec(String(token.info ?? ''))
if (!match)
return null
return parseAdmonition(tokens, index, match, options)
}
export const containerTokenHandlers = {
parseContainer: (
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
) => parseContainer(tokens, index, options),
matchAdmonition: handleContainerOpen,
}
@@ -0,0 +1,76 @@
import type {
DefinitionItemNode,
DefinitionListNode,
MarkdownToken,
ParsedNode,
ParseOptions,
} from '../../types'
import { parseInlineTokens } from '../inline-parsers'
export function parseDefinitionList(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [DefinitionListNode, number] {
const items: DefinitionItemNode[] = []
let j = index + 1
let termNodes: ParsedNode[] = []
let definitionNodes: ParsedNode[] = []
while (j < tokens.length && tokens[j].type !== 'dl_close') {
if (tokens[j].type === 'dt_open') {
// Process term
const termToken = tokens[j + 1]
termNodes = parseInlineTokens(termToken.children || [], undefined, undefined, options as any)
j += 3 // Skip dt_open, inline, dt_close
}
else if (tokens[j].type === 'dd_open') {
// Process definition
let k = j + 1
definitionNodes = []
while (k < tokens.length && tokens[k].type !== 'dd_close') {
if (tokens[k].type === 'paragraph_open') {
const contentToken = tokens[k + 1]
definitionNodes.push({
type: 'paragraph',
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
raw: String(contentToken.content ?? ''),
})
k += 3 // Skip paragraph_open, inline, paragraph_close
}
else {
k++
}
}
// Add definition item
if (termNodes.length > 0) {
items.push({
type: 'definition_item',
term: termNodes,
definition: definitionNodes,
raw: `${termNodes.map(term => term.raw).join('')}: ${definitionNodes
.map(def => def.raw)
.join('\n')}`,
})
// Reset term nodes
termNodes = []
}
j = k + 1 // Skip dd_close
}
else {
j++
}
}
const definitionListNode: DefinitionListNode = {
type: 'definition_list',
items,
raw: items.map(item => item.raw).join('\n'),
}
return [definitionListNode, j + 1] // Skip dl_close
}
@@ -0,0 +1,43 @@
import type { FootnoteNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
export function parseFootnote(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [FootnoteNode, number] {
const token = tokens[index]
const meta = (token.meta ?? {}) as unknown as { label?: number | string }
const id = String(meta?.label ?? '0')
const footnoteChildren: ParsedNode[] = []
let j = index + 1
while (j < tokens.length && tokens[j].type !== 'footnote_close') {
if (tokens[j].type === 'paragraph_open') {
const contentToken = tokens[j + 1]
const children = contentToken.children || []
if (tokens[j + 2].type === 'footnote_anchor') {
children.push(tokens[j + 2] as any)
}
footnoteChildren.push({
type: 'paragraph',
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
raw: String(contentToken.content ?? ''),
})
j += 3 // Skip paragraph_open, inline, paragraph_close
}
else {
j++
}
}
const footnoteNode: FootnoteNode = {
type: 'footnote',
id,
children: footnoteChildren,
raw: `[^${id}]: ${footnoteChildren.map(child => child.raw).join('\n')}`,
}
return [footnoteNode, j + 1] // Skip footnote_close
}
@@ -0,0 +1,8 @@
import type { HardBreakNode } from '../../types'
export function parseHardBreak(): HardBreakNode {
return {
type: 'hardbreak',
raw: '\\\n',
}
}
@@ -0,0 +1,31 @@
import type { HeadingNode, MarkdownToken, ParseOptions } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
export function parseHeading(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): HeadingNode {
const token = tokens[index]
const attrs = (token as any)?.attrs as any[] | null | undefined
const attrsRecord = Array.isArray(attrs) && attrs.length
? Object.fromEntries(
attrs
.filter(pair => Array.isArray(pair) && pair.length >= 1 && pair[0])
.map(([name, value]) => [String(name), value == null || value === '' ? true : String(value)]),
)
: undefined
const levelStr = String(token.tag?.substring(1) ?? '1')
const headingLevel = Number.parseInt(levelStr, 10)
const headingContentToken = tokens[index + 1]
const headingContent = String(headingContentToken.content ?? '')
return {
type: 'heading',
level: headingLevel,
text: headingContent,
...(attrsRecord ? { attrs: attrsRecord } : {}),
children: parseInlineTokens(headingContentToken.children || [], headingContent, undefined, options as any),
raw: headingContent,
}
}
@@ -0,0 +1,100 @@
import type { HtmlBlockNode, MarkdownToken } from '../../types'
import { VOID_HTML_TAGS } from '../../htmlTags'
import { findTagCloseIndexOutsideQuotes, parseTagAttrs } from '../../htmlTagUtils'
function findMatchingCloseTagEnd(rawHtml: string, tag: string, startIndex: number) {
const lowerTag = tag.toLowerCase()
const openTagRe = new RegExp(String.raw`^<\s*${lowerTag}(?=\s|>|/)`, 'i')
const closeTagRe = new RegExp(String.raw`^<\s*\/\s*${lowerTag}(?=\s|>)`, 'i')
let depth = 0
let index = Math.max(0, startIndex)
while (index < rawHtml.length) {
const lt = rawHtml.indexOf('<', index)
if (lt === -1)
return -1
const slice = rawHtml.slice(lt)
if (closeTagRe.test(slice)) {
const endRel = findTagCloseIndexOutsideQuotes(slice)
if (endRel === -1)
return -1
if (depth === 0)
return lt + endRel + 1
depth--
index = lt + endRel + 1
continue
}
if (openTagRe.test(slice)) {
const endRel = findTagCloseIndexOutsideQuotes(slice)
if (endRel === -1)
return -1
const rawTag = slice.slice(0, endRel + 1)
if (!/\/\s*>$/.test(rawTag))
depth++
index = lt + endRel + 1
continue
}
index = lt + 1
}
return -1
}
export function parseHtmlBlock(token: MarkdownToken): HtmlBlockNode {
const raw = String(token.content ?? '')
// Non-element html blocks (comments, doctypes, processing instructions) are non-closable
if (/^\s*<!--/.test(raw) || /^\s*<!/.test(raw) || /^\s*<\?/.test(raw)) {
return {
type: 'html_block',
content: raw,
raw,
tag: '',
loading: false,
}
}
// Extract first tag name (lowercased) like div, p, section, etc.
const tagMatch = raw.match(/^\s*<([A-Z][\w:-]*)/i)
const tag = (tagMatch?.[1] || '').toLowerCase()
// Handle unknown or malformed tag gracefully
if (!tag) {
return {
type: 'html_block',
content: raw,
raw,
tag: '',
loading: false,
}
}
const openEnd = findTagCloseIndexOutsideQuotes(raw)
const openTag = openEnd === -1 ? raw : raw.slice(0, openEnd + 1)
// Self-closing first tag like <img ... />
const selfClosing = openEnd !== -1 && /\/\s*>$/.test(openTag)
const isVoid = VOID_HTML_TAGS.has(tag)
const attrs = parseTagAttrs(openTag)
const closeEnd = openEnd === -1 ? -1 : findMatchingCloseTagEnd(raw, tag, openEnd + 1)
const hasClosing = closeEnd !== -1
const loading = !(isVoid || selfClosing || hasClosing)
const content = loading
? `${raw.replace(/<[^>]*$/, '')}\n</${tag}>`
: raw
return {
type: 'html_block',
content,
raw,
tag,
attrs: attrs.length ? attrs : undefined,
loading,
}
}
@@ -0,0 +1,190 @@
import type {
ListItemNode,
ListNode,
MarkdownToken,
ParsedNode,
ParseOptions,
} from '../../types'
import { parseInlineTokens } from '../inline-parsers'
import { parseCommonBlockToken } from './block-token-parser'
import { parseBlockquote } from './blockquote-parser'
import { containerTokenHandlers } from './container-token-handlers'
function trimInlineTokenTail(token: MarkdownToken) {
const rawContent = String(token.content ?? '')
const trimmed = rawContent.replace(/[ \t\r\n]+$/g, '')
if (trimmed === rawContent)
return
token.content = trimmed
const children = token.children as MarkdownToken[] | undefined
if (!Array.isArray(children) || children.length === 0)
return
while (children.length) {
const last = children[children.length - 1]
if (!last) {
children.pop()
continue
}
if (last.type === 'softbreak' || last.type === 'hardbreak') {
children.pop()
continue
}
if (last.type === 'text') {
const lastContent = String(last.content ?? '')
const next = lastContent.replace(/[ \t\r\n]+$/g, '')
if (next === lastContent)
break
if (next) {
last.content = next
break
}
children.pop()
continue
}
break
}
}
function stripLeakedOrderedListMarkerSuffix(token: MarkdownToken) {
// In streaming mode markdown-it can occasionally leak the next ordered-list
// marker (e.g. "\n\n2" or "\n\n2.") into the previous list item's paragraph
// content. Because our renderer uses `whitespace-pre-wrap`, those leaked
// newlines can create a large blank vertical gap until the next parse tick.
const rawContent = String(token.content ?? '')
const leak = rawContent.match(/\r?\n\s*\d+[.)]?\s*$/)
if (!leak || typeof leak.index !== 'number')
return
token.content = rawContent.slice(0, leak.index)
const children = token.children as MarkdownToken[] | undefined
if (!Array.isArray(children) || children.length === 0)
return
// Best-effort sync: drop trailing text/break tokens that likely correspond
// to the leaked marker and whitespace.
while (children.length) {
const last = children[children.length - 1]
if (!last) {
children.pop()
continue
}
if (last.type === 'softbreak' || last.type === 'hardbreak') {
children.pop()
continue
}
if (last.type === 'text') {
const lastContent = String(last.content ?? '')
if (/^[ \t\r\n\d.)]*$/.test(lastContent)) {
children.pop()
continue
}
const next = lastContent.replace(/[ \t\r\n\d.)]+$/g, '')
if (next !== lastContent) {
if (next)
last.content = next
else
children.pop()
}
}
break
}
}
export function parseList(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [ListNode, number] {
const token = tokens[index]
const listItems: ListItemNode[] = []
let j = index + 1
while (
j < tokens.length
&& tokens[j].type !== 'bullet_list_close'
&& tokens[j].type !== 'ordered_list_close'
) {
if (tokens[j].type === 'list_item_open') {
// if (tokens[j].markup === '*') {
// j++
// continue
// }
const itemChildren: ParsedNode[] = []
let k = j + 1
while (k < tokens.length && tokens[k].type !== 'list_item_close') {
// Handle different block types inside list items
if (tokens[k].type === 'paragraph_open') {
const contentToken = tokens[k + 1]
const preToken = tokens[k - 1]
stripLeakedOrderedListMarkerSuffix(contentToken)
trimInlineTokenTail(contentToken)
itemChildren.push({
type: 'paragraph',
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), preToken, options as any),
raw: String(contentToken.content ?? ''),
})
k += 3 // Skip paragraph_open, inline, paragraph_close
}
else if (tokens[k].type === 'blockquote_open') {
// Parse blockquote within list item
const [blockquoteNode, newIndex] = parseBlockquote(tokens, k, options)
itemChildren.push(blockquoteNode)
k = newIndex
}
else if (
tokens[k].type === 'bullet_list_open'
|| tokens[k].type === 'ordered_list_open'
) {
const [nestedListNode, newIndex] = parseList(tokens, k, options)
itemChildren.push(nestedListNode)
k = newIndex
}
else {
const handled = parseCommonBlockToken(tokens, k, options, containerTokenHandlers)
if (handled) {
itemChildren.push(handled[0])
k = handled[1]
}
else {
k += 1
}
}
}
listItems.push({
type: 'list_item',
children: itemChildren,
raw: itemChildren.map(child => child.raw).join(''),
})
j = k + 1 // Move past list_item_close
}
else {
j += 1
}
}
const listNode: ListNode = {
type: 'list',
ordered: token.type === 'ordered_list_open',
// markdown-it may include attrs like [['start','2']] on ordered_list_open
start: (() => {
if (token.attrs && token.attrs.length) {
const found = token.attrs.find(a => a[0] === 'start')
if (found) {
const parsed = Number(found[1])
return Number.isFinite(parsed) && parsed !== 0 ? parsed : 1
}
}
return undefined
})(),
items: listItems,
raw: listItems.map(item => item.raw).join('\n'),
}
return [listNode, j + 1] // Move past list_close
}
@@ -0,0 +1,14 @@
import type { MarkdownToken, MathBlockNode } from '../../types'
// Parse a math_block token (block/display math expressions)
export function parseMathBlock(token: MarkdownToken): MathBlockNode {
const content = String(token.content ?? '')
const raw = token.raw === '$$' ? `$$${content}$$` : String(token.raw ?? '')
return {
type: 'math_block',
content,
loading: !!token.loading,
raw,
markup: token.markup,
}
}
@@ -0,0 +1,17 @@
import type { MarkdownToken, ParagraphNode, ParseOptions } from '../../types'
import { parseInlineTokens } from '../inline-parsers'
export function parseParagraph(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): ParagraphNode {
const paragraphContentToken = tokens[index + 1]
const paragraphContent = String(paragraphContentToken.content ?? '')
return {
type: 'paragraph',
children: parseInlineTokens(paragraphContentToken.children || [], paragraphContent, undefined, options as any),
raw: paragraphContent,
}
}
@@ -0,0 +1,117 @@
import type {
MarkdownToken,
ParseOptions,
TableCellNode,
TableNode,
TableRowNode,
} from '../../types'
import { parseInlineTokens } from '../inline-parsers'
// Extract alignment from attrs (e.g. ['style','text-align:left'])
function extractAlign(attrs: MarkdownToken['attrs']): 'left' | 'right' | 'center' | undefined {
if (!attrs)
return 'left'
for (const a of attrs) {
if (!a)
continue
const [key, val] = a
if (!val)
continue
const value = String(val).trim().toLowerCase()
if (key === 'style') {
const m = /text-align\s*:\s*(left|right|center)/i.exec(value)
if (m)
return m[1].toLowerCase() as any
}
}
return 'left'
}
export function parseTable(
tokens: MarkdownToken[],
index: number,
options?: ParseOptions,
): [TableNode, number] {
let j = index + 1
let headerRow: TableRowNode | null = null
const rows: TableRowNode[] = []
let isHeader = false
while (j < tokens.length && tokens[j].type !== 'table_close') {
if (tokens[j].type === 'thead_open') {
isHeader = true
j++
}
else if (tokens[j].type === 'thead_close') {
isHeader = false
j++
}
else if (
tokens[j].type === 'tbody_open'
|| tokens[j].type === 'tbody_close'
) {
j++
}
else if (tokens[j].type === 'tr_open') {
const cells: TableCellNode[] = []
let k = j + 1
while (k < tokens.length && tokens[k].type !== 'tr_close') {
if (tokens[k].type === 'th_open' || tokens[k].type === 'td_open') {
const isHeaderCell = tokens[k].type === 'th_open'
const contentToken = tokens[k + 1]
const content = String(contentToken.content ?? '')
const align = extractAlign(tokens[k].attrs)
cells.push({
type: 'table_cell',
header: isHeaderCell || isHeader,
children: parseInlineTokens(contentToken.children || [], content, undefined, options as any),
raw: content,
align,
})
k += 3 // Skip th_open/td_open, inline, th_close/td_close
}
else {
k++
}
}
const rowNode: TableRowNode = {
type: 'table_row',
cells,
raw: cells.map(cell => cell.raw).join('|'),
}
if (isHeader) {
headerRow = rowNode
}
else {
rows.push(rowNode)
}
j = k + 1 // Skip tr_close
}
else {
j++
}
}
if (!headerRow) {
// Default empty header if none found
headerRow = {
type: 'table_row',
cells: [],
raw: '',
}
}
const tableNode: TableNode = {
type: 'table',
header: headerRow,
rows,
loading: tokens[index].loading ?? false,
raw: [headerRow, ...rows].map(row => row.raw).join('\n'),
}
return [tableNode, j + 1] // Skip table_close
}
@@ -0,0 +1,8 @@
import type { ThematicBreakNode } from '../../types'
export function parseThematicBreak(): ThematicBreakNode {
return {
type: 'thematic_break',
raw: '---',
}
}
@@ -0,0 +1,346 @@
import type { MarkdownIt } from 'markdown-it-ts'
import markdownItContainer from 'markdown-it-container'
function parseLooseInlineAttrs(input: string): Record<string, unknown> | null {
const s = String(input ?? '').trim()
if (!s.startsWith('{') || !s.endsWith('}'))
return null
const inner = s.slice(1, -1).trim()
if (!inner)
return {}
// Only support a shallow object literal: {key:value, foo:"bar"}
// No nested objects/arrays/functions.
if (inner.includes('{') || inner.includes('[') || inner.includes(']'))
return null
const parts: string[] = []
let buf = ''
let inSingle = false
let inDouble = false
for (let i = 0; i < inner.length; i++) {
const ch = inner[i]
if (ch === '\\') {
buf += ch
if (i + 1 < inner.length) {
buf += inner[i + 1]
i++
}
continue
}
if (!inDouble && ch === '\'') {
inSingle = !inSingle
buf += ch
continue
}
if (!inSingle && ch === '"') {
inDouble = !inDouble
buf += ch
continue
}
if (!inSingle && !inDouble && ch === ',') {
parts.push(buf.trim())
buf = ''
continue
}
buf += ch
}
if (buf.trim())
parts.push(buf.trim())
const out: Record<string, unknown> = {}
for (const part of parts) {
if (!part)
continue
let inS = false
let inD = false
let split = -1
for (let i = 0; i < part.length; i++) {
const ch = part[i]
if (ch === '\\') {
i++
continue
}
if (!inD && ch === '\'') {
inS = !inS
continue
}
if (!inS && ch === '"') {
inD = !inD
continue
}
if (!inS && !inD && ch === ':') {
split = i
break
}
}
if (split === -1)
return null
const rawKey = part.slice(0, split).trim()
const rawVal = part.slice(split + 1).trim()
if (!rawKey)
return null
let key = rawKey
if ((key.startsWith('"') && key.endsWith('"')) || (key.startsWith('\'') && key.endsWith('\''))) {
try {
key = JSON.parse(key.replace(/^'/, '"').replace(/'$/, '"'))
}
catch {
return null
}
}
// Restrict unquoted keys to simple identifiers.
if (!/^[_$A-Z][\w$-]*$/i.test(key))
return null
let value: unknown
if (!rawVal) {
value = ''
}
else if ((rawVal.startsWith('"') && rawVal.endsWith('"')) || (rawVal.startsWith('\'') && rawVal.endsWith('\''))) {
try {
value = JSON.parse(rawVal.replace(/^'/, '"').replace(/'$/, '"'))
}
catch {
value = rawVal
}
}
else if (/^-?\d+(?:\.\d+)?$/.test(rawVal)) {
value = Number(rawVal)
}
else if (rawVal === 'true' || rawVal === 'false') {
value = rawVal === 'true'
}
else if (rawVal === 'null') {
value = null
}
else {
value = rawVal
}
out[key] = value
}
return out
}
export function applyContainers(md: MarkdownIt) {
;[
'admonition',
'info',
'warning',
'error',
'tip',
'danger',
'note',
'caution',
].forEach((name) => {
md.use(markdownItContainer, name, {
render(tokens: unknown, idx: number) {
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
const token = tokensAny[idx]
// `nesting` is a runtime-only property present on MarkdownIt tokens.
// Narrow the shape with `unknown` -> specific minimal interface to avoid `as any`.
const tokenShape = token as unknown as { nesting?: number }
if (tokenShape.nesting === 1) {
return `<div class="vmr-container vmr-container-${name}">`
}
else {
return '</div>\n'
}
},
})
})
// fallback for simple ::: blocks (kept for backwards compat)
md.block.ruler.before(
'fence',
'vmr_container_fallback',
(state: unknown, startLine: number, endLine: number, silent: boolean) => {
interface ParserState {
bMarks: number[]
tShift: number[]
eMarks: number[]
src: string
env: unknown
push: (type: string, tag?: string, nesting?: number) => any
tokens: any[]
md: any
line: number
}
const s = state as unknown as ParserState
const startPos = s.bMarks[startLine] + s.tShift[startLine]
const lineMax = s.eMarks[startLine]
const line = s.src.slice(startPos, lineMax)
// Match ::: container syntax: ::: name {"json"}
// Using separate patterns to avoid backtracking issues
// Allow both "::: name" and ":::name", and stop before inline JSON attrs.
const nameMatch = line.match(/^:::\s*([^\s{]+)/)
if (!nameMatch)
return false
const name = nameMatch[1]
// Validate name is not empty (handles edge case like "::: ")
if (!name.trim())
return false
const rest = line.slice(nameMatch[0].length)
const trimmedRest = rest.trim()
// Support both:
// - ::: name {"json"}
// - :::name args {"json"}
// - :::name args
let argsStr: string | undefined
let jsonStr: string | undefined
const jsonStart = trimmedRest.indexOf('{')
const jsonCandidate = jsonStart >= 0 ? trimmedRest.slice(jsonStart).trimStart() : undefined
if (jsonStart === -1) {
argsStr = trimmedRest || undefined
}
else {
const before = trimmedRest.slice(0, jsonStart).trim()
argsStr = before || undefined
// Improved JSON matching: find balanced braces instead of simple non-greedy match
if (jsonCandidate?.startsWith('{')) {
let depth = 0
let jsonEnd = -1
for (let i = 0; i < jsonCandidate.length; i++) {
if (jsonCandidate[i] === '{')
depth++
else if (jsonCandidate[i] === '}')
depth--
if (depth === 0) {
jsonEnd = i + 1
break
}
}
if (jsonEnd > 0) {
jsonStr = jsonCandidate.slice(0, jsonEnd)
}
}
// Streaming mid-state: if we saw a '{' but didn't find a balanced '}',
// treat the whole remainder as a plain args string.
// Example: "::: viewcode:stream {xxx:yyy" => attrs.args === "{xxx:yyy"
if (!jsonStr)
argsStr = trimmedRest || undefined
}
if (silent)
return true
const envFinal = !!(s.env as any)?.__markstreamFinal
let nextLine = startLine + 1
let found = false
while (nextLine <= endLine) {
const sPos = s.bMarks[nextLine] + s.tShift[nextLine]
const ePos = s.eMarks[nextLine]
if (s.src.slice(sPos, ePos).trim() === ':::') {
found = true
break
}
nextLine++
}
// Streaming optimization:
// - If we have an opening ":::name" but no closing ":::" yet, still emit a container.
// Treat the remainder of the document as the container body.
// - The structured parser will expose this as `loading: true` until it closes,
// or until `final=true` is provided.
if (!found)
nextLine = endLine
const tokenOpen = s.push('vmr_container_open', 'div', 1)
// `tokenOpen` is runtime token object; keep using runtime helpers but avoid casting `s` to `any`.
tokenOpen.attrSet('class', `vmr-container vmr-container-${name}`)
// Mark unclosed containers for downstream consumers (optional).
;(tokenOpen.meta as any) = { ...(tokenOpen.meta as any), unclosed: !found && !envFinal }
// If args are present (non-JSON payload right after the name), preserve them as a data attribute.
if (argsStr)
tokenOpen.attrSet('data-args', argsStr)
// If JSON attributes are present, store them as data attributes
if (jsonStr) {
try {
const attrs = JSON.parse(jsonStr)
for (const [key, value] of Object.entries(attrs)) {
const isComplexValue = value != null && typeof value === 'object'
tokenOpen.attrSet(
`data-${key}`,
isComplexValue ? JSON.stringify(value) : String(value),
)
}
}
catch {
// If strict JSON parsing fails, try a small loose-object syntax: {xxx:yyy}
const loose = parseLooseInlineAttrs(jsonStr)
if (loose) {
for (const [key, value] of Object.entries(loose)) {
const isComplexValue = value != null && typeof value === 'object'
tokenOpen.attrSet(
`data-${key}`,
isComplexValue ? JSON.stringify(value) : String(value),
)
}
}
else {
// If parsing fails, store the raw string as a data attribute
tokenOpen.attrSet('data-attrs', jsonStr)
}
}
}
// Parse inner content as full block markdown so block-level constructs
// (especially fenced code blocks) are preserved inside the container.
const contentLines: string[] = []
for (let i = startLine + 1; i < nextLine; i++) {
const sPos = s.bMarks[i] + s.tShift[i]
const ePos = s.eMarks[i]
contentLines.push(s.src.slice(sPos, ePos))
}
const hasContent = contentLines.some(line => line.trim().length > 0)
if (hasContent) {
// Always end with two newlines to ensure block separation (esp. for lists/paragraphs)
let innerSrc = contentLines.join('\n')
if (!innerSrc.endsWith('\n'))
innerSrc += '\n'
if (!innerSrc.endsWith('\n\n'))
innerSrc += '\n'
// The last token should be the current vm_container_open token
const prevToken = s.tokens[s.tokens.length - 1]
// Save the inner Markdown content as raw
if (prevToken) {
prevToken.raw = innerSrc
}
const innerTokens: any[] = []
// Use the same env as the parent block parser to ensure all block rules are available
s.md.block.parse(innerSrc, s.md, s.env, innerTokens)
s.tokens.push(...innerTokens)
}
if (found)
s.push('vmr_container_close', 'div', -1)
s.line = found ? (nextLine + 1) : nextLine
return true
},
{
// Ensure this rule can terminate an active paragraph so content immediately
// above `:::` isn't swallowed into the same paragraph block.
alt: ['paragraph', 'reference', 'blockquote', 'list'],
},
)
}
@@ -0,0 +1,996 @@
import type { MarkdownIt, Token } from 'markdown-it-ts'
import { normalizeCustomHtmlTagName } from '../customHtmlTags'
import { STANDARD_BLOCK_HTML_TAGS, STANDARD_HTML_TAGS, VOID_HTML_TAGS } from '../htmlTags'
import { escapeTagForRegExp, findTagCloseIndexOutsideQuotes } from '../htmlTagUtils'
const VOID_TAGS = VOID_HTML_TAGS
// A conservative set of common HTML tags used to detect streaming mid-states.
// We only suppress/merge partial tags for these names to avoid false positives
// (e.g., autolinks like <http://...>).
const BASE_COMMON_HTML_TAGS = STANDARD_HTML_TAGS
const BLOCK_LEVEL_HTML_TAGS = new Set<string>(STANDARD_BLOCK_HTML_TAGS)
BLOCK_LEVEL_HTML_TAGS.delete('details')
const OPEN_TAG_RE = /<([A-Z][\w-]*)(?=[\s/>]|$)/gi
const CLOSE_TAG_RE = /<\/\s*([A-Z][\w-]*)(?=[\s/>]|$)/gi
const TAG_NAME_AT_START_RE = /^<\s*(?:\/\s*)?([A-Z][\w-]*)/i
const STRICT_OPEN_TAG_NAME_AT_START_RE = /^<\s*([A-Z][\w:-]*)(?=[\s/>]|$)/i
function getHtmlInlineTagName(content: string) {
return (content.match(TAG_NAME_AT_START_RE)?.[1] ?? '').toLowerCase()
}
function isHtmlInlineClosingTag(content: string) {
return /^\s*<\s*\//.test(content)
}
function isSelfClosingHtmlInline(content: string, tag: string) {
return VOID_TAGS.has(tag) || /\/\s*>\s*$/.test(content)
}
function findMatchingCloseChildIndex(
children: Array<{ type?: string, content?: string }>,
tag: string,
) {
let depth = 0
for (let index = 0; index < children.length; index++) {
const child = children[index]
if (!child || child.type !== 'html_inline')
continue
const content = String(child.content ?? '')
const childTag = getHtmlInlineTagName(content)
if (childTag !== tag)
continue
if (isHtmlInlineClosingTag(content)) {
if (depth === 0)
return index
depth--
continue
}
if (!isSelfClosingHtmlInline(content, childTag))
depth++
}
return -1
}
function getTrailingOpenDepth(
children: Array<{ type?: string, content?: string }>,
tag: string,
) {
let depth = 0
for (const child of children) {
if (!child || child.type !== 'html_inline')
continue
const content = String(child.content ?? '')
const childTag = getHtmlInlineTagName(content)
if (childTag !== tag)
continue
if (isHtmlInlineClosingTag(content)) {
if (depth > 0)
depth--
continue
}
if (!isSelfClosingHtmlInline(content, childTag))
depth++
}
return depth
}
function findMatchingCloseRangeInHtml(content: string, tag: string, startIndex = 0) {
const tokenRe = new RegExp(String.raw`<\s*(\/?)\s*${escapeTagForRegExp(tag)}(?=[\s>/])[^>]*>`, 'gi')
tokenRe.lastIndex = Math.max(0, startIndex)
let depth = 0
let match: RegExpExecArray | null
while ((match = tokenRe.exec(content)) !== null) {
const raw = match[0] ?? ''
const closing = !!match[1]
const selfClosing = !closing && /\/\s*>$/.test(raw)
if (closing) {
if (depth === 0) {
return {
start: match.index,
end: match.index + raw.length,
}
}
depth--
continue
}
if (!selfClosing)
depth++
}
return null
}
function getTrailingCustomTagDepthInHtml(content: string, tag: string) {
const tokenRe = new RegExp(String.raw`<\s*(\/?)\s*${escapeTagForRegExp(tag)}(?=[\s>/])[^>]*>`, 'gi')
let depth = 0
let match: RegExpExecArray | null
while ((match = tokenRe.exec(content)) !== null) {
const raw = match[0] ?? ''
const closing = !!match[1]
const selfClosing = !closing && /\/\s*>$/.test(raw)
if (closing) {
if (depth > 0)
depth--
continue
}
if (!selfClosing)
depth++
}
return depth
}
function tokenToRaw(token: Token) {
const shape = token as unknown as { raw?: string, markup?: string, content?: string }
return String(shape.raw ?? shape.content ?? shape.markup ?? '')
}
function isNonElementHtmlBlock(content: string) {
return /^\s*<\s*[!?]/.test(content)
}
function buildCommonHtmlTagSet(extraTags?: readonly string[]) {
const set = new Set(BASE_COMMON_HTML_TAGS)
if (extraTags && Array.isArray(extraTags)) {
for (const t of extraTags) {
const raw = String(t ?? '').trim()
if (!raw)
continue
const m = raw.match(/^[<\s/]*([A-Z][\w-]*)/i)
if (!m)
continue
set.add(m[1].toLowerCase())
}
}
return set
}
function isCommonHtmlTagOrPrefix(tag: string, tagSet: Set<string>) {
if (tagSet.has(tag))
return true
for (const common of tagSet) {
if (common.startsWith(tag))
return true
}
return false
}
function findFirstIncompleteTag(content: string, tagSet: Set<string>) {
let first:
| { index: number, tag: string, closing: boolean }
| null = null
for (const m of content.matchAll(OPEN_TAG_RE)) {
const idx = m.index ?? -1
if (idx < 0)
continue
const tag = (m[1] ?? '').toLowerCase()
// For opening tags we also accept prefixes of known HTML tags
// (e.g., '<fo' while typing '<font ...>').
if (!isCommonHtmlTagOrPrefix(tag, tagSet))
continue
const rest = content.slice(idx)
if (findTagCloseIndexOutsideQuotes(rest) !== -1)
continue
if (!first || idx < first.index)
first = { index: idx, tag, closing: false }
}
for (const m of content.matchAll(CLOSE_TAG_RE)) {
const idx = m.index ?? -1
if (idx < 0)
continue
const tag = (m[1] ?? '').toLowerCase()
// For closing tags we also accept prefixes of known HTML tags
// (e.g., '</sp' while typing '</span>').
if (!isCommonHtmlTagOrPrefix(tag, tagSet))
continue
const rest = content.slice(idx)
if (findTagCloseIndexOutsideQuotes(rest) !== -1)
continue
if (!first || idx < first.index)
first = { index: idx, tag, closing: true }
}
// Also swallow bare "<" or "</" at the end while typing.
const bareClose = /<\/\s*$/.exec(content)
if (bareClose && typeof bareClose.index === 'number') {
const idx = bareClose.index
const rest = content.slice(idx)
if (!rest.includes('>') && (!first || idx < first.index))
first = { index: idx, tag: '', closing: true }
}
const bareOpen = /<\s*$/.exec(content)
if (bareOpen && typeof bareOpen.index === 'number') {
const idx = bareOpen.index
const rest = content.slice(idx)
// Avoid matching "</" which is handled above.
if (!rest.startsWith('</') && !rest.includes('>') && (!first || idx < first.index))
first = { index: idx, tag: '', closing: false }
}
return first
}
function splitTextToken(token: Token, content: string) {
const t = token as Token & { content?: string, raw?: string }
// Preserve the original Token prototype (markdown-it-ts attaches helper methods).
const nt = Object.assign(
Object.create(Object.getPrototypeOf(t)),
t,
{ type: 'text', content, raw: content },
) as Token
return nt
}
function fixStreamingHtmlInlineChildren(children: Token[], tagSet: Set<string>) {
if (!children.length)
return { children }
const out: Token[] = []
let pending: { tag: string, buffer: string, closing: boolean } | null = null
let pendingAtEnd: string | null = null
function pushTextPart(text: string, baseToken?: Token) {
if (!text)
return
if (baseToken)
out.push(splitTextToken(baseToken, text))
else
out.push({ type: 'text', content: text, raw: text } as any)
}
function splitCompleteHtmlFromText(chunk: string, baseToken?: Token) {
let cursor = 0
while (cursor < chunk.length) {
const lt = chunk.indexOf('<', cursor)
if (lt === -1) {
pushTextPart(chunk.slice(cursor), baseToken)
break
}
pushTextPart(chunk.slice(cursor, lt), baseToken)
const sub = chunk.slice(lt)
const tagMatch = sub.match(TAG_NAME_AT_START_RE)
if (!tagMatch) {
pushTextPart('<', baseToken)
cursor = lt + 1
continue
}
const closeIdx = findTagCloseIndexOutsideQuotes(sub)
if (closeIdx === -1) {
pushTextPart('<', baseToken)
cursor = lt + 1
continue
}
const tagText = sub.slice(0, closeIdx + 1)
const tagName = (tagMatch[1] ?? '').toLowerCase()
if (tagSet.has(tagName)) {
out.push({
type: 'html_inline',
tag: '',
content: tagText,
raw: tagText,
} as any)
}
else {
pushTextPart(tagText, baseToken)
}
cursor = lt + tagText.length
}
}
function processTextChunk(chunk: string, baseToken?: Token) {
if (!chunk)
return
const match = findFirstIncompleteTag(chunk, tagSet)
if (!match) {
splitCompleteHtmlFromText(chunk, baseToken)
return
}
const before = chunk.slice(0, match.index)
if (before)
splitCompleteHtmlFromText(before, baseToken)
pending = {
tag: match.tag,
buffer: chunk.slice(match.index),
closing: match.closing,
}
pendingAtEnd = pending.buffer
}
for (const child of children) {
if (pending) {
pending.buffer += tokenToRaw(child)
pendingAtEnd = pending.buffer
const closeIdx = findTagCloseIndexOutsideQuotes(pending.buffer)
if (closeIdx === -1) {
// still incomplete: swallow this token to avoid rendering jitter
continue
}
const tagChunk = pending.buffer.slice(0, closeIdx + 1)
const afterChunk = pending.buffer.slice(closeIdx + 1)
out.push({
type: 'html_inline',
tag: '',
content: tagChunk,
raw: tagChunk,
} as any)
pending = null
pendingAtEnd = null
if (afterChunk)
processTextChunk(afterChunk)
continue
}
if (child.type === 'html_inline') {
const content = tokenToRaw(child)
const tagMatch = content.match(TAG_NAME_AT_START_RE)
const tagName = (tagMatch?.[1] ?? '').toLowerCase()
if (tagName && tagSet.has(tagName) && findTagCloseIndexOutsideQuotes(content) === -1) {
// markdown-it may prematurely close a tag at a ">" inside a quoted
// attribute value (e.g. `<a href="...a>b`), producing a broken html_inline
// token. Treat it as a streaming mid-state and swallow until we see a
// real tag close ">" outside quotes.
pending = {
tag: tagName,
buffer: content,
closing: /^<\s*\//.test(content),
}
pendingAtEnd = pending.buffer
continue
}
}
if (child.type === 'text') {
const content = String((child as any).content ?? '')
if (!content.includes('<')) {
out.push(child)
continue
}
processTextChunk(content, child)
continue
}
out.push(child)
}
return {
children: out,
pendingBuffer: pendingAtEnd ?? undefined,
}
}
export interface FixHtmlInlineOptions {
/**
* Custom HTML-like tag names that should participate in streaming
* mid-state suppression and complete-tag splitting (e.g. ['thinking']).
*/
customHtmlTags?: readonly string[]
}
export function applyFixHtmlInlineTokens(md: MarkdownIt, options: FixHtmlInlineOptions = {}) {
const commonHtmlTags = buildCommonHtmlTagSet(options.customHtmlTags)
// Tags that should stay inline when we auto-append a closing tag at core stage.
const autoCloseInlineTagSet = new Set<string>([
'a',
'span',
'strong',
'em',
'b',
'i',
'u',
])
const customTagSet = new Set<string>()
if (options.customHtmlTags?.length) {
for (const t of options.customHtmlTags) {
const name = normalizeCustomHtmlTagName(t)
if (!name)
continue
customTagSet.add(name)
autoCloseInlineTagSet.add(name)
}
}
const shouldMergeHtmlBlockTag = (tag: string) => customTagSet.has(tag) || !commonHtmlTags.has(tag) || BLOCK_LEVEL_HTML_TAGS.has(tag)
const getHtmlBlockCarrierContent = (token: Token & { content?: string, children?: any[] }) => {
if (token.type === 'html_block')
return String(token.content ?? '')
if (token.type !== 'inline' || !Array.isArray(token.children) || token.children.length !== 1)
return ''
const onlyChild = token.children[0]
if (onlyChild?.type !== 'html_block')
return ''
return String(token.content ?? onlyChild.content ?? '')
}
const normalizeHtmlBlockCarrier = (
token: Token & { content?: string, children?: any[], raw?: string },
content: string,
) => {
token.type = 'html_block'
token.content = content
token.raw = content
token.children = []
}
// Streaming mid-state: suppress partial inline HTML in text tokens until the
// tag is fully closed with `>`, then allow it to be tokenized as html_inline.
md.core.ruler.after('inline', 'fix_html_inline_streaming', (state: unknown) => {
const s = state as unknown as { tokens?: Token[] }
const toks = s.tokens ?? []
for (const t of toks) {
const tok = t as Token & { children?: Token[], content?: string, raw?: string }
if (tok.type !== 'inline' || !Array.isArray(tok.children))
continue
// markdown-it-ts may emit inline tokens with empty children when the
// content starts with an incomplete HTML-ish fragment like "<span ...".
// In that case, synthesize a text token so we can suppress mid-states.
const originalContent = String(tok.content ?? '')
const sourceChildren = tok.children.length
? tok.children
: (originalContent.includes('<')
? [{ type: 'text', content: originalContent, raw: originalContent } as any]
: null)
if (!sourceChildren)
continue
try {
const fixed = fixStreamingHtmlInlineChildren(sourceChildren, commonHtmlTags)
tok.children = fixed.children
if (fixed.pendingBuffer) {
const idx = originalContent.lastIndexOf(fixed.pendingBuffer)
if (idx !== -1) {
const trimmed = originalContent.slice(0, idx)
tok.content = trimmed
// keep raw in sync if present
if (typeof tok.raw === 'string')
tok.raw = trimmed
}
}
}
catch (e) {
console.error('[applyFixHtmlInlineTokens] failed to fix streaming html inline', e)
}
}
})
// Fix certain single-token inline HTML cases by expanding into [openTag, text, closeTag]
// This helps downstream inline parsers (e.g., <a>text</a>) to recognize inner text reliably.
md.core.ruler.push('fix_html_inline_tokens', (state: unknown) => {
const s = state as unknown as { tokens?: Token[] }
const toks = s.tokens ?? []
// 有一些很特殊的场景,比如 html_block 开始 <thinking>,但是后面跟着很多段落,如果没匹配到</thinking>,中间的都应该合并为html_block的 content
const tagStack: [string, number][] = []
for (let i = 0; i < toks.length; i++) {
const t = toks[i] as Token & { content?: string, children: any[] }
// If we're currently inside an unclosed custom-tag html_block, merge
// everything (including other html_block tokens) into the opener until
// the matching closing tag arrives.
if (tagStack.length > 0) {
const [openTag, openIndex] = tagStack[tagStack.length - 1]
if (i !== openIndex) {
// Remove structural paragraph wrappers that can appear in stream mode.
if (t.type === 'paragraph_open' || t.type === 'paragraph_close') {
toks.splice(i, 1)
i--
continue
}
const chunk = String((t as any).content ?? (t as any).raw ?? '')
if (chunk) {
const openToken = toks[openIndex] as Token & { content?: string, loading?: boolean }
const mergedContent = `${String(openToken.content || '')}\n${chunk}`
const openEnd = findTagCloseIndexOutsideQuotes(mergedContent)
const closeRange = openEnd === -1
? null
: findMatchingCloseRangeInHtml(mergedContent, openTag, openEnd + 1)
if (closeRange) {
const before = mergedContent.slice(0, closeRange.end)
const after = mergedContent.slice(closeRange.end)
openToken.content = before
openToken.loading = false
const afterTrimmed = after.replace(/^\s+/, '')
// Remove current token after merging.
toks.splice(i, 1)
// Close the stack before reinserting trailing content.
tagStack.pop()
if (afterTrimmed) {
toks.splice(i, 0, afterTrimmed.startsWith('<')
? ({ type: 'html_block', content: afterTrimmed } as any)
: ({ type: 'inline', content: afterTrimmed, children: [{ type: 'text', content: afterTrimmed, raw: afterTrimmed }] } as any))
}
i--
continue
}
openToken.content = mergedContent
if (openToken.loading !== false)
openToken.loading = true
}
// Remove current token after merging.
toks.splice(i, 1)
i--
continue
}
}
const rawContent = getHtmlBlockCarrierContent(t)
if (rawContent) {
if (isNonElementHtmlBlock(rawContent))
continue
// Support both opening (<tag ...>) and closing (</tag>) blocks.
const tag = (rawContent.match(/<\s*(?:\/\s*)?([^\s>/]+)/)?.[1] ?? '').toLowerCase()
const isClosingTag = /^\s*<\s*\//.test(rawContent)
// Merge configured custom tags and unknown HTML-like tags (e.g. <think>)
// until their matching closing tag arrives. Also merge standard block
// tags that markdown-it can split on blank lines, such as <details>.
if (!tag || !shouldMergeHtmlBlockTag(tag))
continue
normalizeHtmlBlockCarrier(t, rawContent)
if (!isClosingTag) {
// 开始标签,入栈
if (tag) {
// If the html_block already contains its own closing tag, do NOT
// push it onto the stack; otherwise we'd incorrectly merge the
// following blocks into this html_block.
const selfClosingRe = new RegExp(`^\\s*<\\s*${tag}\\b[^>]*\\/\\s*>`, 'i')
if (!selfClosingRe.test(rawContent) && getTrailingCustomTagDepthInHtml(rawContent, tag) > 0)
tagStack.push([tag, i])
}
}
else {
// 结束标签:如果匹配到栈顶,则把 closing token 也合并进 opener 并删除自己
if (tagStack.length > 0 && tag && tagStack[tagStack.length - 1][0] === tag) {
const [, openIndex] = tagStack[tagStack.length - 1]
const openToken = toks[openIndex] as Token & { content?: string, loading?: boolean }
openToken.content = `${String(openToken.content || '')}\n${rawContent}`
openToken.loading = false
tagStack.pop()
// Remove current closing html_block token so it doesn't become a stray node.
toks.splice(i, 1)
i--
}
}
continue
}
else if (tagStack.length > 0) {
// 如果在标签栈中,说明是未闭合标签的内容,合并到上一个 html_block
if (t.type === 'paragraph_open' || t.type === 'paragraph_close') {
// 应该删除这些标签
toks.splice(i, 1)
i-- // 调整索引
continue
}
const content = t.content || ''
const CLOSING_TAG_REGEX = new RegExp(`<\\s*\\/\\s*${tagStack[tagStack.length - 1][0]}\\s*>`, 'i')
const isClosingTag = CLOSING_TAG_REGEX.test(content)
if (content) {
// 插入到栈顶标签对应的 html_block 中
const [, openIndex] = tagStack[tagStack.length - 1]
const openToken = toks[openIndex] as Token & { content?: string, loading: boolean }
openToken.content = `${openToken.content || ''}\n${content}`
if (openToken.loading !== false)
openToken.loading = !isClosingTag
}
if (isClosingTag) {
tagStack.pop()
}
// 删除当前 token
toks.splice(i, 1)
i-- // 调整索引
}
else {
continue
}
}
// Some custom tags (e.g. <thinking>) can be tokenized by markdown-it into
// multiple top-level inline tokens, with the closing tag arriving in a
// later inline token. Our inline parser can only match closing tags within
// the same inline token's children list, so we merge such sequences here.
if (customTagSet.size > 0) {
const openReCache = new Map<string, RegExp>()
const closeReCache = new Map<string, RegExp>()
const getOpenRe = (tag: string) => {
let r = openReCache.get(tag)
if (!r) {
r = new RegExp(`<\\s*${tag}\\b`, 'i')
openReCache.set(tag, r)
}
return r
}
const getCloseRe = (tag: string) => {
let r = closeReCache.get(tag)
if (!r) {
r = new RegExp(`<\\s*\\/\\s*${tag}\\s*>`, 'i')
closeReCache.set(tag, r)
}
return r
}
const stack: Array<{ tag: string, index: number }> = []
for (let i = 0; i < toks.length; i++) {
const tok = toks[i] as Token & { content?: string, children?: any[] }
const content = String(tok.content ?? '')
// If we're inside an unclosed custom tag, we may need to close it even
// if the closing tag is emitted as html_block (markdown-it can do this).
if (stack.length > 0) {
const top = stack[stack.length - 1]
const openTok = toks[top.index] as Token & { content?: string, children?: any[] }
// Close via an html_block token like "</thinking>"
if (tok.type === 'html_block' && getCloseRe(top.tag).test(content)) {
openTok.content = `${String(openTok.content ?? '')}\n${content}`
if (Array.isArray(openTok.children)) {
openTok.children.push({
type: 'html_inline',
content: `</${top.tag}>`,
raw: `</${top.tag}>`,
} as any)
}
toks.splice(i, 1)
i--
stack.pop()
continue
}
// Only merge inline tokens; keep block structure intact.
if (tok.type !== 'inline')
continue
const children = Array.isArray(tok.children) ? tok.children : []
const closeChildIndex = findMatchingCloseChildIndex(children, top.tag)
// If the closing tag is inside this inline token, merge up to it and
// keep the trailing content as a new paragraph so it doesn't get
// swallowed by the custom tag.
if (closeChildIndex !== -1) {
const beforeChildren = children.slice(0, closeChildIndex + 1)
const afterChildren = children.slice(closeChildIndex + 1)
const beforeText = beforeChildren
.map((c: any) => String(c?.content ?? c?.raw ?? ''))
.join('')
// Only append the fragment up to and including the closing tag.
openTok.content = `${String(openTok.content ?? '')}\n${beforeText}`
if (Array.isArray(openTok.children))
openTok.children.push(...beforeChildren)
// Replace current token with trailing content (if any)
if (afterChildren.length) {
const afterText = afterChildren.map((c: any) => String(c.content ?? c.raw ?? '')).join('')
if (afterText.trim()) {
const trimmed = afterText.replace(/^\s+/, '')
if (trimmed.startsWith('<')) {
toks.splice(i, 1, { type: 'html_block', content: trimmed } as any)
}
else {
toks.splice(i, 1, { type: 'paragraph_open', tag: 'p', nesting: 1 } as any, { type: 'inline', tag: '', nesting: 0, content: afterText, children: [{ type: 'text', content: afterText, raw: afterText }] } as any, { type: 'paragraph_close', tag: 'p', nesting: -1 } as any)
// current index now points at paragraph_open; move on
}
}
else {
toks.splice(i, 1)
i--
}
}
else {
toks.splice(i, 1)
i--
}
stack.pop()
continue
}
// No closing tag: merge everything and remove current inline token.
openTok.content = `${String(openTok.content ?? '')}\n${content}`
if (Array.isArray(openTok.children))
openTok.children.push(...children)
toks.splice(i, 1)
i--
continue
}
// Not inside: detect an opening custom tag that does not close within this token.
if (tok.type !== 'inline')
continue
const children = Array.isArray(tok.children) ? tok.children : []
for (const tag of customTagSet) {
const trailingOpenDepth = children.length
? getTrailingOpenDepth(children, tag)
: (getOpenRe(tag).test(content) && !getCloseRe(tag).test(content) ? 1 : 0)
if (trailingOpenDepth > 0) {
stack.push({ tag, index: i })
break
}
}
}
}
// Defensive cleanup: some edge cases can end up with an orphan
// paragraph_close token (without a matching paragraph_open) after
// core-stage token mutations. Drop such invalid closes so downstream
// consumers don't see stray paragraph_close.
{
let depth = 0
for (let i = 0; i < toks.length; i++) {
const t = toks[i] as Token
if (t.type === 'paragraph_open') {
depth++
continue
}
if (t.type === 'paragraph_close') {
if (depth > 0) {
depth--
}
else {
toks.splice(i, 1)
i--
}
}
}
}
for (let i = 0; i < toks.length; i++) {
const t = toks[i] as Token & { content?: string, children: any[], loading?: boolean }
if (t.type === 'html_block') {
const rawTag = t.content?.match(/<([^\s>/]+)/)?.[1] ?? ''
const tag = rawTag.toLowerCase()
// Special tags: comments, doctypes, processing instructions
// These should not be modified
if (tag.startsWith('!') || tag.startsWith('?')) {
t.loading = false
continue
}
// Custom tags (configured by user): keep as html_block
// These are handled specially to support streaming and structured nodes
if (customTagSet.has(tag)) {
const raw = String(t.content ?? '')
const openEnd = findTagCloseIndexOutsideQuotes(raw)
const closeRange = openEnd === -1
? null
: findMatchingCloseRangeInHtml(raw, tag, openEnd + 1)
const hasClose = !!closeRange
t.loading = hasClose ? false : t.loading !== undefined ? t.loading : true
const endTagIndex = closeRange?.start ?? -1
const closeLen = closeRange ? closeRange.end - closeRange.start : 0
if (endTagIndex !== -1) {
// Found a closing tag - extract inner content and trim
const rawForNode = raw.slice(0, endTagIndex + closeLen)
let inner = ''
if (openEnd !== -1 && openEnd < endTagIndex) {
inner = raw.slice(openEnd + 1, endTagIndex)
}
// Create structured node with inner content
t.children = [
{
type: tag,
content: inner,
raw: rawForNode,
attrs: [],
tag,
loading: false,
},
] as any[]
// Update token content
t.content = rawForNode
;(t as any).raw = rawForNode
// Insert trailing content as a new token if present
const afterContent = raw.slice(endTagIndex + closeLen) || ''
const afterTrimmed = afterContent.replace(/^\s+/, '')
if (afterTrimmed) {
toks.splice(i + 1, 0, afterTrimmed.startsWith('<')
? ({ type: 'html_block', content: afterTrimmed } as any)
: ({ type: 'text', content: afterTrimmed, raw: afterTrimmed } as any))
}
}
else {
// No closing tag yet (streaming mid-state)
t.children = [
{
type: tag,
content: '',
raw,
attrs: [],
tag,
loading: true,
},
] as any[]
}
continue
}
// Common block tags: skip conversion to inline
if (['br', 'hr', 'img', 'input', 'link', 'meta', 'div', 'p', 'ul', 'li'].includes(tag))
continue
// Other HTML tags: convert to inline type for better handling
// Parse attributes and check for closing tag
t.type = 'inline'
const attrs: [string, string][] = []
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
let match
while ((match = attrRegex.exec(t.content || '')) !== null) {
const attrName = match[1]
const attrValue = match[2] || match[3] || match[4] || ''
attrs.push([attrName, attrValue])
}
const raw = String(t.content ?? '')
const closeRe = new RegExp(`<\\/\\s*${tag}\\s*>`, 'i')
const closeMatch = closeRe.exec(raw)
const endTagIndex = closeMatch ? closeMatch.index : -1
const closeLen = closeMatch ? closeMatch[0].length : 0
if (endTagIndex !== -1) {
// Found a closing tag - split content
const rawForNode = raw.slice(0, endTagIndex + closeLen)
const afterContent = raw.slice(endTagIndex + closeLen) || ''
const afterTrimmed = afterContent.replace(/^\s+/, '')
// Create html_block child node
t.children = [
{
type: 'html_block',
content: rawForNode,
tag,
loading: false,
},
] as any[]
// Update token content
t.content = rawForNode
;(t as any).raw = rawForNode
// Insert trailing content as a new token if present
if (afterTrimmed) {
toks.splice(i + 1, 0, afterTrimmed.startsWith('<')
? ({ type: 'html_block', content: afterTrimmed } as any)
: ({ type: 'text', content: afterTrimmed, raw: afterTrimmed } as any))
}
}
else {
// No closing tag - streaming mid-state
t.children = [
{
type: 'html_block',
content: t.content,
tag,
loading: true,
},
] as any[]
}
continue
}
if (!t || t.type !== 'inline')
continue
// 修复children 是单个 html_inline的场景
if (t.children.length === 2 && t.children[0].type === 'html_inline') {
// 补充一个闭合标签
const rawTag = t.children[0].content?.match(/<([^\s>/]+)/)?.[1] ?? ''
const tag = rawTag.toLowerCase()
const second = t.children[1] as any
const secondCloseTag = String(second?.content ?? '').match(/^<\s*\/\s*([^\s>]+)/)?.[1]?.toLowerCase() ?? ''
// Already a complete open+close pair: don't append another closing tag.
if (second?.type === 'html_inline' && secondCloseTag === tag)
continue
// 如果是常见的 inline标签(含用户自定义),则只追加结尾标签,否则转换成 html_block
if (autoCloseInlineTagSet.has(tag)) {
t.children[0].loading = true
t.children[0].tag = tag
t.children.push({
type: 'html_inline',
tag,
loading: true,
content: `</${tag}>`,
} as any)
}
else {
t.children = [
{
type: 'html_block',
loading: true,
tag,
content: t.children[0].content + t.children[1].content,
} as any,
]
}
continue
}
else if (t.children.length === 3 && t.children[0].type === 'html_inline' && t.children[2].type === 'html_inline') {
const rawTag = t.children[0].content?.match(/<([^\s>/]+)/)?.[1] ?? ''
const tag = rawTag.toLowerCase()
// 如果是常见的 inline标签(含用户自定义),则不处理,否则转换成 html_block
if (autoCloseInlineTagSet.has(tag))
continue
t.children = [
{
type: 'html_block',
loading: false,
tag,
content: t.children.map(ct => ct.content).join(''),
} as any,
]
continue
}
// Only handle pathological cases where inline content is a single HTML-ish chunk
if (!t.content?.startsWith('<') || (t as any).children?.length !== 1)
continue
const raw = String(t.content)
const htmlToken = t as unknown as { children: Array<{ type: string, content: string }> }
const onlyChild = htmlToken.children[0] as { type?: string, content?: string } | undefined
// Keep literal text untouched (e.g. malformed "<robot=xxx>..."), but still
// suppress dangling "<tag" mid-states to avoid streaming jitter.
if (onlyChild?.type !== 'html_inline') {
if (/^<\s*(?:\/\s*)?[A-Z][\w:-]*\s*$/i.test(raw))
htmlToken.children.length = 0
continue
}
const strictTagName = String(onlyChild.content ?? raw).match(STRICT_OPEN_TAG_NAME_AT_START_RE)?.[1]?.toLowerCase() ?? ''
if (!strictTagName)
continue
const selfClosing = /\/\s*>\s*$/.test(raw)
const isVoid = selfClosing || VOID_TAGS.has(strictTagName)
if (isVoid) {
// For void/self-closing tags, keep a single html_inline token
htmlToken.children = [
{ type: 'html_inline', content: raw },
] as any
continue
}
htmlToken.children.length = 0
}
})
}
@@ -0,0 +1,176 @@
/**
* Fix indented code blocks that should be paragraphs or inline HTML.
*
* In streaming scenarios, AI-generated content may have lines that are
* indented with 4+ spaces (which markdown-it treats as code blocks per
* CommonMark spec) but are actually plain text or HTML entities.
*
* This plugin runs at the core stage to convert single-line indented
* code blocks that don't look like code into paragraphs.
*/
import type { MarkdownIt, Token } from 'markdown-it-ts'
export interface FixIndentedCodeBlockOptions {
/**
* Whether to enable this fix. Default: true
*/
enabled?: boolean
}
/**
* Check if a line looks like code (vs plain text or HTML entities).
* Returns true if the line appears to be code.
*/
function looksLikeCode(line: string): boolean {
const trimmed = line.trim()
// Empty line
if (!trimmed)
return false
// 1. HTML entities at start (e.g., &#10006;, &nbsp;, &amp;)
// These are typically plain text with character references, not code
if (/^&[a-z0-9#]+;/i.test(trimmed))
return false
// 2. Common programming keywords and JavaScript literals
if (/^(?:const|let|var|function|class|import|export|if|for|while|return|await|async|yield|try|catch|throw|new|typeof|instanceof|switch|case|break|continue|def|ruby|perl|print|echo|true|false|null|undefined|NaN|Infinity|this)\b/.test(trimmed))
return true
// 3. Function/method calls with parentheses
if (/[a-z_$][\w$]*(?:\.[a-z_$][\w$]*|\['[^']*'\]|\["[^"]*"\]|\[\d+\])*\s*\(/i.test(trimmed))
return true
// 4. Property access with dot or bracket notation
if (/[a-z_$][\w$]*(?:\.[a-z_$][\w$]*|\['[^']*'\]|\["[^"]*"\]|\[[\d+\]])+/i.test(trimmed))
return true
// 5. Comparison and logical operators (but not just symbols alone)
// Require identifiers around operators for better precision
// Match: identifier operator identifier OR operator(s) identifier
if (/\w+\s*(?:===?|!==?|<=?|>=?|\+\+|--|&&|\|\||\?\.)/.test(trimmed))
return true
// Double operators before identifier (like !!value) - but avoid matching list items
if (/^(?:!!|\+\+|--)\s*\w/.test(trimmed))
return true
// 6. Assignment operators
if (/[\w$]+\s*(?:\+=|-=|\*=|\/=|%=|\*\*=|=)/.test(trimmed))
return true
// 7. URLs, protocols
if (/^(?:https?:\/\/|ftp:\/\/|file:\/\/|\/\/|www\.)/i.test(trimmed))
return true
// 8. Template strings with interpolation
if (/`[^`]*\$\{[^}]*\}[^`]*`/.test(trimmed))
return true
// 9. JSX/TSX tags (uppercase component names)
if (/<\/?[A-Z][a-zA-Z0-9]*/.test(trimmed))
return true
// 10. HTML-like tags (lowercase with attributes)
if (/<[a-z][a-z0-9]*\s[^>]+>/.test(trimmed))
return true
// 11. Quoted strings (single, double, backtick) - often code statements
if (/^(["'`]).*\1\s*[;,]?$/.test(trimmed))
return true
// 12. Array/object/function patterns with brackets
// Empty brackets [] {} () are common in code
if (/^\[[\s\S]*\]$/.test(trimmed) || /^\{[\s\S]*\}$/.test(trimmed) || /^\(\s*\)$/.test(trimmed)) {
return true
}
// 13. Code-like expressions with identifiers and operators
// e.g., a + b, x > y, count !== 0
// This also catches property access like foo.bar
// Must have at least one identifier (letter/digit/underscore) on each side of operator
if (/[\w$]+(?:\s*[+\-*/%<>=!&|^~:]+\s*[\w$]+|\s*\.\s*[\w$]+)/.test(trimmed))
return true
// 14. Arrow functions, fat arrows
if (/=>|->|::/.test(trimmed))
return true
// 15. Decorators and annotations
if (/^@[\w.$]+$/.test(trimmed))
return true
// 16. Numbers (hex, binary, octal, decimal with units)
if (/^(?:0x[0-9a-fA-F]+|0b[01]+|0o[0-7]+|\d+(?:\.\d*)?(?:px|em|rem|%|vh|vw|deg|s|ms)?)$/.test(trimmed))
return true
// 17. Variable declarations with $ (common in jQuery/shell)
if (/^\$[\w$]+\s*[=:]/.test(trimmed))
return true
// 18. Pipes and special operators
if (/\|\s*\w+|\w+\s*\|/.test(trimmed))
return true
// 19. Command-like patterns (git, npm, etc.)
if (/^(?:git|npm|yarn|pnpm|bun|pip|cargo|go|rust|python|node|java|mvn|gradle|docker|kubectl)\s+/.test(trimmed))
return true
// 20. Console methods and common API calls
if (/(?:console|window|document|Math|JSON|Date|Array|Object|String|Number|Boolean)\.[a-zA-Z]/.test(trimmed))
return true
// 21. Comment patterns
if (/^(?:\/\/|#|\/\*|\*\/|<!--|-->)/.test(trimmed))
return true
// 22. Heredoc patterns
if (/^(?:<<<|<<\s*['"]?\w+['"]?)/.test(trimmed))
return true
// Default: doesn't look like code
return false
}
export function applyFixIndentedCodeBlock(md: MarkdownIt, options: FixIndentedCodeBlockOptions = {}) {
if (options.enabled === false)
return
// Process at the core stage after tokenization is complete
md.core.ruler.after('inline', 'fix_indented_code_block', (state: any) => {
const tokens = state.tokens as Token[]
for (let i = 0; i < tokens.length; i++) {
const token = tokens[i]
// Look for single-line code_block tokens
if (token.type !== 'code_block')
continue
const content = String((token as any).content ?? '').trim()
if (!content)
continue
const lines = content.split(/\r?\n/).filter(line => line.trim().length > 0)
// Only convert single-line code blocks that don't look like code
if (lines.length === 1 && !looksLikeCode(lines[0] ?? '')) {
// Convert code_block to paragraph
const textContent = lines[0] ?? ''
// Replace code_block token with paragraph tokens
tokens.splice(i, 1, { type: 'paragraph_open', tag: 'p', nesting: 1, level: token.level } as any, {
type: 'inline',
tag: '',
nesting: 0,
level: token.level,
content: textContent,
children: [{ type: 'text', content: textContent, level: token.level + 1 }] as any,
block: true,
} as any, { type: 'paragraph_close', tag: 'p', nesting: -1, level: token.level } as any)
// Skip the newly inserted tokens
i += 2
}
}
})
}
@@ -0,0 +1,653 @@
import type { MarkdownIt } from 'markdown-it-ts'
import type { MarkdownToken } from '../types'
import { shouldDemoteFilenameLikeLinkify } from '../parser/linkifyHeuristics'
// We hard-stop FULLWIDTH exclamation mark used as CJK punctuation.
// ASCII `!` is valid in URLs (path/query/fragment), so do not stop on it.
const LINKIFY_HARD_STOP_CHARS = ['!'] as const
// Small helpers to reduce repetition when building token fragments
function textToken(content: string) {
return {
type: 'text',
content,
raw: content,
}
}
function pushEmOpen(arr: any[], type: number) {
if (type === 1) {
arr.push({ type: 'em_open', tag: 'em', nesting: 1 })
}
else if (type === 2) {
arr.push({ type: 'strong_open', tag: 'strong', nesting: 1 })
}
else if (type === 3) {
arr.push({ type: 'strong_open', tag: 'strong', nesting: 1 })
arr.push({ type: 'em_open', tag: 'em', nesting: 1 })
}
}
function pushEmClose(arr: any[], type: number) {
if (type === 1) {
arr.push({ type: 'em_close', tag: 'em', nesting: -1 })
}
else if (type === 2) {
arr.push({ type: 'strong_close', tag: 'strong', nesting: -1 })
}
else if (type === 3) {
arr.push({ type: 'em_close', tag: 'em', nesting: -1 })
arr.push({ type: 'strong_close', tag: 'strong', nesting: -1 })
}
}
function createLinkToken(text: string, href: string, loading: boolean) {
let title = ''
if (href.includes('"')) {
const temps = href.split('"')
href = temps[0].trim()
title = temps[1].trim()
}
return {
type: 'link',
loading,
href,
title,
text,
children: [
{
type: 'text',
content: text,
raw: text,
},
],
raw: String(`[${text}](${href})`),
}
}
function appendToLinkToken(link: any, suffix: string) {
if (!link || !suffix)
return
link.href = String(link.href ?? '') + suffix
link.text = String(link.text ?? '') + suffix
link.raw = String(`[${link.text}](${link.href})`)
if (Array.isArray(link.children) && link.children.length) {
const last = link.children[link.children.length - 1]
if (last?.type === 'text') {
last.content = String(last.content ?? '') + suffix
last.raw = String(last.raw ?? '') + suffix
}
else {
link.children.push(textToken(suffix))
}
}
}
function firstIndexOfAny(input: string, chars: readonly string[]) {
let first = -1
for (const ch of chars) {
const idx = input.indexOf(ch)
if (idx !== -1 && (first === -1 || idx < first))
first = idx
}
return first
}
function getHrefFromLinkOpen(token: any) {
const href = token?.attrs?.find((attr: any) => attr?.[0] === 'href')?.[1]
return typeof href === 'string' ? href : ''
}
function setHrefOnLinkOpen(token: any, href: string) {
if (!token)
return
token.attrs = Array.isArray(token.attrs) ? token.attrs : []
const idx = token.attrs.findIndex((attr: any) => attr?.[0] === 'href')
if (idx >= 0)
token.attrs[idx][1] = href
else
token.attrs.push(['href', href])
}
function collectLinkifyText(tokens: MarkdownToken[], openIndex: number, closeIndex: number) {
let text = ''
for (let index = openIndex + 1; index < closeIndex; index++) {
const token = tokens[index]
if (token?.type !== 'text' || typeof token.content !== 'string')
return null
text += token.content
}
return text || null
}
export function applyFixLinkTokens(md: MarkdownIt) {
// Run after the inline rule so markdown-it has produced inline tokens
// for block-level tokens; we then adjust each inline token's children
// so downstream code receives corrected token arrays during the same
// parsing pass.
md.core.ruler.after('inline', 'fix_link_tokens', (state: unknown) => {
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
const toks = s.tokens ?? []
for (let i = 0; i < toks.length; i++) {
const t = toks[i]
if (t && t.type === 'inline' && Array.isArray(t.children)) {
try {
t.children = fixLinkToken(t.children)
}
catch (e) {
// Swallow errors to avoid breaking parsing; keep original children
// so parse still succeeds even if our fix fails for an unexpected shape.
// Errors should be rare and indicate malformed token arrays.
console.error('[applyFixLinkTokens] failed to fix inline children', e)
}
}
}
})
}
function fixLinkToken(tokens: MarkdownToken[]): MarkdownToken[] {
// Need at least `link_open + text + link_close` for linkify/autolink fixes.
// Keep this low to allow fixing `<https://...!suffix>` cases where inline children length is 3.
if (tokens.length < 3)
return tokens
// 如果包含 code_inline 类型的 token,说明是包含内联代码的链接,直接返回原样,避免错误处理
if (tokens.some(token => token.type === 'code_inline'))
return tokens
for (let i = 0; i <= tokens.length - 1; i++) {
if (i < 0) {
i = 0
}
const curToken = tokens[i]
if (!curToken)
break
// Hard-stop certain punctuation for linkify-generated bare URLs.
// Example: `https://a.com/x.png!文字` should become link `https://a.com/x.png` + text `!文字`.
if (curToken.type === 'link_open' && (curToken.markup === 'linkify' || curToken.markup === 'autolink')) {
let closeIdx = -1
for (let j = i + 1; j < tokens.length; j++) {
if (tokens[j]?.type === 'link_close') {
closeIdx = j
break
}
}
if (closeIdx !== -1) {
const linkText = collectLinkifyText(tokens, i, closeIdx)
if (
curToken.markup === 'linkify'
&& linkText
&& shouldDemoteFilenameLikeLinkify(linkText)
) {
tokens.splice(i, closeIdx - i + 1, textToken(linkText) as any)
continue
}
const href = getHrefFromLinkOpen(curToken as any)
const hrefStop = firstIndexOfAny(href, LINKIFY_HARD_STOP_CHARS)
// Prefer splitting by the visible text token, but also trim href if it contains stop chars.
for (let j = i + 1; j < closeIdx; j++) {
const t = tokens[j]
if (t?.type !== 'text' || typeof t.content !== 'string')
continue
const stopAt = firstIndexOfAny(t.content, LINKIFY_HARD_STOP_CHARS)
if (stopAt === -1)
continue
const stopChar = t.content[stopAt]
const before = t.content.slice(0, stopAt)
let tail = t.content.slice(stopAt)
// Move any remaining text tokens that were inside the linkify span to the tail.
for (let k = j + 1; k < closeIdx; k++) {
const tk = tokens[k]
if (tk?.type === 'text' && typeof tk.content === 'string')
tail += tk.content
}
t.content = before
;(t as any).raw = before
const removeCount = closeIdx - (j + 1)
if (removeCount > 0) {
tokens.splice(j + 1, removeCount)
closeIdx = j + 1
}
let newHref = href
if (hrefStop !== -1) {
newHref = href.slice(0, hrefStop)
}
else if (tail) {
// linkify-it may percent-encode non-ASCII, so use encoded tail / stop-char to locate split point.
const encodedTail = encodeURI(tail)
if (encodedTail && href.endsWith(encodedTail)) {
newHref = href.slice(0, href.length - encodedTail.length)
}
else {
const encodedStop = stopChar ? encodeURI(stopChar) : ''
const idx = encodedStop ? href.indexOf(encodedStop) : -1
if (idx !== -1)
newHref = href.slice(0, idx)
}
}
if (newHref !== href)
setHrefOnLinkOpen(curToken as any, newHref)
if (tail) {
tokens.splice(closeIdx + 1, 0, textToken(tail) as any)
}
break
}
}
}
if (curToken?.type === 'em_open' && tokens[i - 1]?.type === 'text' && tokens[i - 1].content?.endsWith('*')) {
const beforeText = tokens[i - 1].content?.replace(/(\*+)$/, '') || ''
tokens[i - 1].content = beforeText
// 修改当前 type 'em_open' -> 'strong_open'
curToken.type = 'strong_open'
curToken.tag = 'strong'
curToken.markup = '**'
// 还需要把对应的 em_close 也修改为 strong_close
for (let j = i + 1; j < tokens.length; j++) {
if (tokens[j]?.type === 'em_close') {
tokens[j].type = 'strong_close'
tokens[j].tag = 'strong'
tokens[j].markup = '**'
break
}
}
}
else if (curToken?.type === 'text' && curToken.content?.endsWith('(') && tokens[i + 1]?.type === 'link_open') {
const match = curToken.content!.match(/\[([^\]]+)\]/)
if (match) {
let beforeText = curToken.content!.slice(0, match.index)
const emphasisMatch = beforeText.match(/(\*+)$/)
const replacerTokens = []
if (emphasisMatch) {
beforeText = beforeText.slice(0, emphasisMatch.index)
if (beforeText)
replacerTokens.push(textToken(beforeText))
const text = match[1]
const type = emphasisMatch[1].length
pushEmOpen(replacerTokens, type)
let href = tokens[i + 2]?.content || ''
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
href += tokens[i + 4]?.content || ''
tokens[i + 4].content = ''
}
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
pushEmClose(replacerTokens, type)
if (tokens[i + 4]?.type === 'text') {
const afterText = tokens[i + 4].content?.replace(/^\)\**/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i, 5, ...replacerTokens)
}
else {
tokens.splice(i, 4, ...replacerTokens)
}
}
else {
if (beforeText)
replacerTokens.push(textToken(beforeText))
let text = match[1]
const emphasisMatch = text.match(/^\*+/)
if (emphasisMatch) {
const type = emphasisMatch[0].length
text = text.replace(/^\*+/, '').replace(/\*+$/, '')
// Put emphasis inside the link children when the asterisks are inside the brackets
// so link nodes will contain the emphasis node as a child.
let href = tokens[i + 2]?.content || ''
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
href += tokens[i + 4]?.content || ''
tokens[i + 4].content = ''
}
// wrap the link with emphasis open/close tokens
pushEmOpen(replacerTokens, type)
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
pushEmClose(replacerTokens, type)
// we've already pushed the link, skip the standard push below
if (tokens[i + 4]?.type === 'text') {
const afterText = tokens[i + 4].content?.replace(/^\)/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i, 5, ...replacerTokens)
}
else {
tokens.splice(i, 4, ...replacerTokens)
}
if (i === 0) {
i = replacerTokens.length - 1
}
else {
i -= (replacerTokens.length + 1)
}
continue
}
let href = tokens[i + 2]?.content || ''
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
href += tokens[i + 4]?.content || ''
tokens[i + 4].content = ''
}
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
if (tokens[i + 4]?.type === 'text') {
const afterText = tokens[i + 4].content?.replace(/^\)/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i, 5, ...replacerTokens)
}
else {
tokens.splice(i, 4, ...replacerTokens)
}
}
i -= (replacerTokens.length + 1)
continue
}
}
else if (curToken.type === 'link_open' && curToken.markup === 'linkify' && tokens[i - 1]?.type === 'text' && tokens[i - 1].content?.endsWith('(')) {
if (tokens[i - 2]?.type === 'link_close') {
// 合并link
const replacerTokens = []
const text = (tokens[i - 3].content || '')
let href = curToken.attrs?.find(attr => attr[0] === 'href')?.[1] || ''
if (tokens[i + 3]?.type === 'text') {
const m = (tokens[i + 3]?.content ?? '').indexOf(')')
const loading = m === -1
if (m === -1) {
href += (tokens[i + 3]?.content?.slice(0, m) || '')
tokens[i + 3].content = ''
}
replacerTokens.push(createLinkToken(text, href, loading))
const afterText = tokens[i + 3].content?.replace(/^\)\**/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i - 4, 8, ...replacerTokens)
}
else {
replacerTokens.push({
type: 'link',
loading: true,
href,
title: '',
text,
children: [
{
type: 'text',
content: href,
raw: href,
},
],
raw: String(`[${text}](${href})`),
})
tokens.splice(i - 4, 7, ...replacerTokens)
}
continue
}
else if (tokens[i - 1].content === '](' && tokens[i - 3]?.type === 'text' && tokens[i - 3].content?.endsWith(')')) {
// 移除 i-3 的 link部分,合并到 link_open 下的 text中
if (tokens[i - 2]?.type === 'strong_open') {
const [beforeText, linText] = tokens[i - 3].content?.split('[**') || []
tokens[i + 1].content = linText || ''
tokens[i - 3].content = beforeText || ''
tokens[i - 1].content = ''
}
else if (tokens[i - 2]?.type === 'em_open') {
const [beforeText, linText] = tokens[i - 3].content?.split('[*') || []
tokens[i + 1].content = linText || ''
tokens[i - 3].content = beforeText || ''
tokens[i - 1].content = ''
}
else {
const [beforeText, linText] = tokens[i - 3].content?.split('[') || []
tokens[i + 1].content = linText || ''
tokens[i - 3].content = beforeText || ''
tokens[i - 1].content = ''
}
}
}
if (
curToken.type === 'link_close'
&& curToken.nesting === -1
&& tokens[i - 2]?.type === 'link_open'
&& tokens[i + 1]?.type === 'text'
&& tokens[i - 1]?.type === 'text'
) {
// 修复链接后多余文本被包含在链接内的问题
const text = tokens[i - 1].content || ''
const attrs = tokens[i - 2].attrs || []
const href = attrs.find(a => a[0] === 'href')?.[1] || ''
const title = attrs.find(a => a[0] === 'title')?.[1] || ''
let count = 3
let deleteCount = 2
const beforeText = tokens[i - 3]?.content || ''
const emphasisMatch = beforeText.match(/^(\*+)$/)
const replacerTokens: any[] = []
if (emphasisMatch) {
deleteCount += 1
const type = emphasisMatch[1].length
pushEmOpen(replacerTokens, type)
}
if (curToken.markup !== 'linkify' && tokens[i + 1].type === 'text' && tokens[i + 1]?.content?.startsWith('](')) {
count += 1
for (let j = i + 1; j < tokens.length; j++) {
const type = emphasisMatch ? emphasisMatch[1].length : tokens[i - 3].markup!.length
const t = tokens[j]
if (type === 1 && t.type === 'em_close') {
break
}
else if (type === 2 && t.type === 'strong_close') {
break
}
else if (type === 3) {
if (t.type === 'em_close' || t.type === 'strong_close') {
break
}
}
count += 1
}
}
replacerTokens.push({
type: 'link',
loading: false,
href,
title,
text,
children: [
{
type: 'text',
content: text,
raw: text,
},
],
raw: String(`[${text}](${href})`),
} as any)
if (emphasisMatch) {
const type = emphasisMatch[1].length
pushEmClose(replacerTokens, type)
}
tokens.splice(i - deleteCount, count, ...replacerTokens)
i -= (replacerTokens.length + 1)
continue
}
else if (curToken.content?.startsWith('](') && tokens[i - 1].markup?.includes('*') && tokens[i - 4]?.type === 'text' && tokens[i - 4].content?.endsWith('[')) {
const type = tokens[i - 1].markup!.length
const replacerTokens = []
const beforeText = tokens[i - 4].content!.slice(0, tokens[i - 4].content!.length - type)
if (beforeText)
replacerTokens.push(textToken(beforeText))
pushEmOpen(replacerTokens, type)
const text = tokens[i - 2].content || ''
let href = curToken.content!.slice(2)
let loading = true
if (tokens[i + 1]?.type === 'text') {
const m = (tokens[i + 1]?.content ?? '').indexOf(')')
loading = m === -1
if (m === -1) {
href += (tokens[i + 1]?.content?.slice(0, m) || '')
tokens[i + 1].content = ''
}
}
replacerTokens.push(createLinkToken(text, href, loading))
pushEmClose(replacerTokens, type)
if (tokens[i + 1]?.type === 'text') {
const afterText = tokens[i + 1].content?.replace(/^\)\**/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i - 4, 8, ...replacerTokens)
}
else if (tokens[i + 1]?.type === 'link_open') {
// 特殊情况其实要把href也处理,这里可以直接跳过
tokens.splice(i - 4, 10, ...replacerTokens)
}
else {
tokens.splice(i - 4, 7, ...replacerTokens)
}
i -= (replacerTokens.length + 1)
continue
}
else if (curToken.content?.startsWith('](') && tokens[i - 1].type === 'strong_close' && tokens[i - 4]?.type === 'text' && tokens[i - 4]?.content?.includes('**[')) {
// 此时的场景是 link 被 strong 包裹,link 中又包含了强调符号
const replacerTokens = []
const beforeText = tokens[i - 4].content!.split('**[')[0]
if (beforeText)
replacerTokens.push(textToken(beforeText))
pushEmOpen(replacerTokens, 2)
const text = tokens[i - 2].content || ''
let href = curToken.content!.slice(2)
let loading = true
if (tokens[i + 1]?.type === 'text') {
const m = (tokens[i + 1]?.content ?? '').indexOf(')')
loading = m === -1
if (m === -1) {
href += (tokens[i + 1]?.content?.slice(0, m) || '')
tokens[i + 1].content = ''
}
}
replacerTokens.push(createLinkToken(text, href, loading))
pushEmClose(replacerTokens, 2)
if (tokens[i + 1]?.type === 'text') {
const afterText = tokens[i + 1].content?.replace(/^\)\**/, '')
if (afterText)
replacerTokens.push(textToken(afterText))
tokens.splice(i - 4, 8, ...replacerTokens)
}
else if (tokens[i + 1]?.type === 'link_open') {
// 特殊情况其实要把href也处理,这里可以直接跳过
tokens.splice(i - 4, 10, ...replacerTokens)
}
else {
tokens.splice(i - 4, 7, ...replacerTokens)
}
i -= (replacerTokens.length + 1)
continue
}
else if (curToken.type === 'strong_close' && tokens[i + 1]?.type === 'text' && tokens[i + 1].content?.includes('](') && tokens[i - 1].type === 'text' && /\[.*$/.test(tokens[i - 1].content || '')) {
const replacerTokens = []
const [beforeText, afterText] = tokens[i - 1].content?.split('[') || ['', '']
if (beforeText)
replacerTokens.push(textToken(beforeText))
pushEmOpen(replacerTokens, 2)
let [text, href] = tokens[i + 1].content!.split('](')
text = afterText + text
let deleteCount = 4
if (tokens[i + 2]?.type === 'link_open') {
const _href = tokens[i + 2].attrs?.find(a => a[0] === 'href')?.[1]
if (tokens[i + 5]?.type === 'text' && tokens[i + 5].content === '.') {
href = (_href || href) + tokens[i + 5].content
tokens[i + 5].content = ''
}
else {
href = _href || href
}
deleteCount += 3
}
let loading = true
if (curToken.nesting === -1) {
// 嵌套 strong_close,需要去掉尾部的 **
text = text.replace(/\*+$/, '')
}
if (tokens[i + 2]?.type === 'text') {
const m = (tokens[i + 2]?.content ?? '').indexOf(')')
loading = m === -1
if (m === -1) {
href += (tokens[i + 2]?.content?.slice(0, m) || '')
tokens[i + 2].content = ''
}
}
replacerTokens.push(createLinkToken(text, href, loading))
pushEmClose(replacerTokens, 2)
tokens.splice(i - 2, deleteCount, ...replacerTokens)
}
// 处理强调 + 链接拆分:首个 text 含前导星号与 '[',后续出现独立的 "]("、link_open、href、link_close、")"、strong_close。
// 期望合并为:strong_open + 复合 link(label 尾部补回星号) + strong_close。
if (
curToken.type === 'text'
&& /\*+\[[^\]]*$/.test(curToken.content || '')
&& tokens[i + 1]?.type === 'strong_open'
&& tokens[i + 2]?.type === 'text' && tokens[i + 2].content === ']('
&& tokens[i + 3]?.type === 'link_open'
&& tokens[i + 5]?.type === 'link_close'
&& tokens[i + 6]?.type === 'text' && tokens[i + 6].content === ')'
&& tokens[i + 7]?.type === 'strong_close'
) {
const contentVal = curToken.content || ''
const startMatch = contentVal.match(/^(\*+)\[(.*)$/)
if (startMatch) {
const innerLabel = startMatch[2] || ''
// 将前导星号数量追加到标签末尾
const finalLabel = innerLabel + startMatch[1]
// 获取 href:优先使用 link_open 的属性,若为空则回退到后续文本 token 内容
let href = tokens[i + 3]?.attrs?.find(a => a[0] === 'href')?.[1] || ''
if (!href && tokens[i + 4]?.type === 'text') {
href = tokens[i + 4].content || ''
}
const out: MarkdownToken[] = []
pushEmOpen(out as any[], 2)
out.push(createLinkToken(finalLabel, href, false))
pushEmClose(out as any[], 2)
tokens.splice(i, 9, ...out)
i -= (out.length - 1)
continue
}
}
}
// Post-pass: linkify-it strips trailing ASCII punctuation like `!`, but in some URLs
// it's meaningful (e.g. `?q=!`, `#!`). Merge a standalone leading `!` text token
// back into the preceding autolink/linkify link when it ends with `=` or `#`.
for (let i = 0; i < tokens.length - 1; i++) {
const t = tokens[i] as any
const next = tokens[i + 1] as any
if (t?.type !== 'link' || next?.type !== 'text' || typeof next.content !== 'string')
continue
if (!next.content.startsWith('!'))
continue
const href = String(t.href ?? '')
const text = String(t.text ?? '')
// Only merge for bare URLs (where visible text equals href) to avoid
// changing explicit Markdown links like `[label](https://a/#)!`.
if (text !== href)
continue
if (!href.endsWith('=') && !href.endsWith('#'))
continue
appendToLinkToken(t, '!')
const rest = next.content.slice(1)
if (rest) {
next.content = rest
next.raw = rest
}
else {
tokens.splice(i + 1, 1)
}
}
return tokens
}
@@ -0,0 +1,35 @@
import type { MarkdownIt } from 'markdown-it-ts'
import type { MarkdownToken } from '../types'
export function applyFixListItem(md: MarkdownIt) {
// Normalize list-item related inline tokens after inline tokenization
// so downstream parsers see corrected children.
md.core.ruler.after('inline', 'fix_list_item_tokens', (state: unknown) => {
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
const toks = s.tokens ?? []
for (let i = 0; i < toks.length; i++) {
const t = toks[i]
if (t && t.type === 'inline' && Array.isArray(t.children)) {
try {
t.children = fixListItem(t.children)
}
catch (e) {
// Keep original children on error to avoid breaking parsing
console.error('[applyFixListItem] failed to fix inline children', e)
}
}
}
})
}
function fixListItem(tokens: MarkdownToken[]): MarkdownToken[] {
const last = tokens[tokens.length - 1]
const lastContent = String(last?.content ?? '')
if (last?.type === 'text' && (/^\s*\d+\.\s*$/.test(lastContent) && tokens[tokens.length - 2]?.tag === 'br')) {
tokens.splice(tokens.length - 1, 1)
}
return tokens
}
@@ -0,0 +1,291 @@
import type { MarkdownIt } from 'markdown-it-ts'
import type { MarkdownToken } from 'stream-markdown-parser'
export function applyFixStrongTokens(md: MarkdownIt) {
// Run after inline tokenization to normalize strong/em tokens in
// each inline token's children. This ensures downstream inline
// parsers receive a normalized token list.
md.core.ruler.after('inline', 'fix_strong_tokens', (state: unknown) => {
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
const toks = s.tokens ?? []
for (let i = 0; i < toks.length; i++) {
const t = toks[i]
if (t && t.type === 'inline' && Array.isArray(t.children)) {
try {
t.children = fixStrongTokens(t.children)
}
catch (e) {
// don't break parsing on plugin error
console.error('[applyFixStrongTokens] failed to fix inline children', e)
}
}
}
})
}
function fixStrongTokens(tokens: MarkdownToken[]): MarkdownToken[] {
let strongIndex = 0
const cleansStrong = new Set<number>()
const cleansEm = new Set<number>()
let emIndex = 0
for (let i = 0; i < tokens.length; i++) {
const t = tokens[i]
const type = t.type
if (type === 'strong_open') {
strongIndex++
const markup = String(t.markup ?? '')
let j = i - 1
while (j >= 0 && tokens[j].type === 'text' && tokens[j].content === '') {
j--
}
const preToken = tokens[j]
let k = i + 1
while (k < tokens.length && tokens[k].type === 'text' && tokens[k].content === '') {
k++
}
const postToken = tokens[k]
if (markup === '__' && (preToken?.content?.endsWith('_') || postToken?.content?.startsWith('_') || postToken?.markup?.includes('_'))) {
t.type = 'text'
t.tag = ''
t.content = markup
t.raw = markup
t.markup = ''
t.attrs = null
t.map = null
t.info = ''
t.meta = null
cleansStrong.add(strongIndex)
}
}
else if (type === 'strong_close') {
if (cleansStrong.has(strongIndex) && t.markup === '__') {
t.type = 'text'
t.content = t.markup
t.raw = String(t.markup ?? '')
t.tag = ''
t.markup = ''
t.attrs = null
t.map = null
t.info = ''
t.meta = null
}
strongIndex--
if (strongIndex < 0)
strongIndex = 0
}
else if (type === 'em_open') {
emIndex++
const markup = String(t.markup ?? '')
let j = i - 1
while (j >= 0 && tokens[j].type === 'text' && tokens[j].content === '') {
j--
}
const preToken = tokens[j]
let k = i + 1
while (k < tokens.length && tokens[k].type === 'text' && tokens[k].content === '') {
k++
}
const postToken = tokens[k]
if (markup === '_' && (preToken?.content?.endsWith('_') || postToken?.content?.startsWith('_') || postToken?.markup?.includes('_'))) {
t.type = 'text'
t.tag = ''
t.content = markup
t.raw = markup
t.markup = ''
t.attrs = null
t.map = null
t.info = ''
t.meta = null
cleansEm.add(emIndex)
}
}
else if (type === 'em_close') {
if (cleansEm.has(emIndex) && t.markup === '_') {
t.type = 'text'
t.content = t.markup
t.raw = String(t.markup ?? '')
t.tag = ''
t.markup = ''
t.attrs = null
t.map = null
t.info = ''
t.meta = null
}
emIndex--
if (emIndex < 0)
emIndex = 0
}
}
if (tokens.length < 5)
return tokens
const i = tokens.length - 4
const token = tokens[i]
let fixedTokens = [...tokens]
const nextToken = tokens[i + 1]
const tokenContent = String(token.content ?? '')
if (token.type === 'link_open' && tokens[i - 1]?.type === 'em_open' && tokens[i - 2]?.type === 'text' && tokens[i - 2].content?.endsWith('*')) {
const textContent = String(tokens[i - 2].content ?? '').slice(0, -1)
const replaceTokens = [
{
type: 'strong_open',
tag: 'strong',
attrs: null,
map: null,
children: null,
content: '',
markup: '**',
info: '',
meta: null,
raw: '',
},
tokens[i],
tokens[i + 1],
tokens[i + 2],
{
type: 'strong_close',
tag: 'strong',
attrs: null,
map: null,
children: null,
content: '',
markup: '**',
info: '',
meta: null,
raw: '',
},
]
if (textContent) {
replaceTokens.unshift({
type: 'text',
content: textContent,
raw: textContent,
})
}
fixedTokens.splice(i - 2, 6, ...replaceTokens)
}
else if (token.type === 'text' && tokenContent.endsWith('*') && nextToken.type === 'em_open') {
// 解析有问题,要合并 emphasis 和 前面的 * 为 strong
const _nextToken = tokens[i + 2]
const count = _nextToken?.type === 'text' ? 4 : 3
const insert = [
{
type: 'strong_open',
tag: 'strong',
attrs: null,
map: null,
children: null,
content: '',
markup: '**',
info: '',
meta: null,
raw: '',
},
{
type: 'text',
content: _nextToken?.type === 'text' ? String(_nextToken.content ?? '') : '',
raw: _nextToken?.type === 'text' ? String(_nextToken.content ?? '') : '',
},
{
type: 'strong_close',
tag: 'strong',
attrs: null,
map: null,
children: null,
content: '',
markup: '**',
info: '',
meta: null,
raw: '',
},
] as MarkdownToken[]
const beforeText = tokenContent.slice(0, -1)
if (beforeText) {
insert.unshift({
type: 'text',
content: beforeText,
raw: beforeText,
})
}
fixedTokens.splice(i, count, ...insert)
}
// Fix: markdown-it + math inline can incorrectly split a single strong span
// into multiple strong nodes and leave the closing `**` inside a trailing text token.
// Example:
// **化简集合\\( P \\)并求补集**:...
// Tokens may become:
// strong_open, text('化简集合'), strong_close, strong_open, math_inline, strong_close, text('并求补集**:...')
// Normalize into:
// strong_open, text('化简集合'), math_inline, text('并求补集'), strong_close, text(':...')
fixedTokens = mergeBrokenStrongAroundMathInline(fixedTokens)
return fixedTokens
}
function mergeBrokenStrongAroundMathInline(tokens: MarkdownToken[]): MarkdownToken[] {
if (tokens.length < 7)
return tokens
const out: MarkdownToken[] = []
for (let i = 0; i < tokens.length; i++) {
const t0 = tokens[i]
const t1 = tokens[i + 1]
const t2 = tokens[i + 2]
const t3 = tokens[i + 3]
const t4 = tokens[i + 4]
const t5 = tokens[i + 5]
const t6 = tokens[i + 6]
if (
t0?.type === 'strong_open'
&& t1?.type === 'text'
&& t2?.type === 'strong_close'
&& t3?.type === 'strong_open'
&& t4?.type === 'math_inline'
&& t5?.type === 'strong_close'
&& t6?.type === 'text'
) {
const textContent = String(t6.content ?? '')
const closeIdx = textContent.indexOf('**')
if (closeIdx !== -1) {
const beforeClose = textContent.slice(0, closeIdx)
const afterClose = textContent.slice(closeIdx + 2)
out.push(t0)
out.push(t1)
out.push(t4)
if (beforeClose) {
out.push({
...t6,
type: 'text',
content: beforeClose,
raw: beforeClose,
} as MarkdownToken)
}
// Reuse one of the strong_close tokens to close the merged strong span.
out.push(t5)
if (afterClose) {
out.push({
...t6,
type: 'text',
content: afterClose,
raw: afterClose,
} as MarkdownToken)
}
i += 6
continue
}
}
out.push(t0)
}
return out
}
@@ -0,0 +1,208 @@
import type { MarkdownIt } from 'markdown-it-ts'
import type { MarkdownToken } from '../types'
export function applyFixTableTokens(md: MarkdownIt) {
// Run after block parsing so block-level tokens (including inline
// children) are present. We replace the token array with the
// fixed version returned by `fixTableTokens`.
md.core.ruler.after('block', 'fix_table_tokens', (state: unknown) => {
const s = state as unknown as { tokens?: any[] }
try {
const toks = s.tokens ?? []
const fixed = fixTableTokens(toks)
if (Array.isArray(fixed))
s.tokens = fixed
}
catch (e) {
// swallow errors to avoid breaking parsing; keep original tokens
console.error('[applyFixTableTokens] failed to fix table tokens', e)
}
})
}
function createStart() {
return [
{
type: 'table_open',
tag: 'table',
attrs: null,
map: null,
children: null,
content: '',
markup: '',
info: '',
level: 0,
loading: true,
meta: null,
},
{
type: 'thead_open',
tag: 'thead',
attrs: null,
block: true,
level: 1,
children: null,
},
{
type: 'tr_open',
tag: 'tr',
attrs: null,
block: true,
level: 2,
children: null,
},
]
}
function createEnd() {
return [
{
type: 'tr_close',
tag: 'tr',
attrs: null,
block: true,
level: 2,
children: null,
},
{
type: 'thead_close',
tag: 'thead',
attrs: null,
block: true,
level: 1,
children: null,
},
{
type: 'table_close',
tag: 'table',
attrs: null,
map: null,
children: null,
content: '',
markup: '',
info: '',
level: 0,
meta: null,
},
]
}
function createTh(text: string) {
return [{
type: 'th_open',
tag: 'th',
attrs: null,
block: true,
level: 3,
children: null,
}, {
type: 'inline',
tag: '',
children: null,
content: text,
level: 4,
attrs: null,
block: true,
}, {
type: 'th_close',
tag: 'th',
attrs: null,
block: true,
level: 3,
children: null,
}]
}
function getPipeRowCells(line: string, requireTrailingPipe: boolean) {
if (!line.startsWith('|') || line.includes('\n'))
return null
if (requireTrailingPipe && !line.endsWith('|'))
return null
const cells = line.slice(1).split('|')
if (cells.at(-1) === '')
cells.pop()
return cells.length > 0 && cells.every(cell => cell.length > 0) ? cells : null
}
function isSingleLineFallbackTable(line: string) {
return getPipeRowCells(line, false) !== null
}
function hasTrailingPipeHeaderRow(line: string) {
return getPipeRowCells(line, true) !== null
}
function isSeparatorCell(cell: string) {
return /^:?-+:?$/.test(cell.trim())
}
function isTableSeparatorRow(line: string) {
if (!line.startsWith('|'))
return false
const cells = line.slice(1).split('|')
if (cells.at(-1) === '')
cells.pop()
return cells.length > 0 && cells.every(isSeparatorCell)
}
function isTruncatedSeparatorRow(line: string) {
return line === '|' || line === '|:'
}
function hasTrailingPipeHeaderRowWithoutColon(line: string) {
const cells = getPipeRowCells(line, true)
return cells !== null && cells.every(cell => !cell.includes(':'))
}
export function fixTableTokens(tokens: MarkdownToken[]): MarkdownToken[] {
const fixedTokens = [...tokens]
if (tokens.length < 3)
return fixedTokens
const i = tokens.length - 2
const token = tokens[i]
if (token.type === 'inline') {
const tcontent = String(token.content ?? '')
const headerContent = tcontent.split('\n')[0] ?? ''
const [headerLine = '', separatorLine = '', ...rest] = tcontent.split('\n')
if (!tcontent.includes('\n') && isSingleLineFallbackTable(tcontent)) {
// 解析 table
const body = headerContent.slice(1).split('|').map(i => i.trim()).filter(Boolean).flatMap(i => createTh(i))
const insert = ([
...createStart(),
...body,
...createEnd(),
] as unknown) as MarkdownToken[]
fixedTokens.splice(i - 1, 3, ...insert)
}
else if (
tcontent.includes('\n')
&& rest.length === 0
&& hasTrailingPipeHeaderRow(headerLine)
&& isTableSeparatorRow(separatorLine)
) {
// 解析 table
const body = headerContent.slice(1, -1).split('|').map(i => i.trim()).flatMap(i => createTh(i))
const insert = ([
...createStart(),
...body,
...createEnd(),
] as unknown) as MarkdownToken[]
fixedTokens.splice(i - 1, 3, ...insert)
}
else if (
tcontent.includes('\n')
&& rest.length === 0
&& hasTrailingPipeHeaderRowWithoutColon(headerLine)
&& isTruncatedSeparatorRow(separatorLine)
) {
token.content = tcontent.slice(0, -2)
token.children!.splice(2, 1)
}
}
return fixedTokens
}
@@ -0,0 +1,109 @@
export const TEX_BRACE_COMMANDS = [
'boldsymbol',
'mathbb',
'mathcal',
'mathfrak',
'mathrm',
'mathit',
'mathsf',
'vec',
'hat',
'bar',
'tilde',
'overline',
'underline',
'mathscr',
'mathnormal',
'operatorname',
'mathbf*',
]
export const ESCAPED_TEX_BRACE_COMMANDS = TEX_BRACE_COMMANDS.map(c => c.replace(/[.*+?^${}()|[\\]"\]/g, '\\$&')).join('|')
const TEX_CMD_RE = /\\[a-z]+/i
const PREFIX_CLASS = '(?:\\\\|\\u0008)'
const TEX_CMD_WITH_BRACES_RE = new RegExp(String.raw`${PREFIX_CLASS}(?:${ESCAPED_TEX_BRACE_COMMANDS})\s*\{[^}]+\}`, 'i')
// Detect brace-taking TeX commands even when the leading backslash or the
// closing brace/content is missing (e.g. "operatorname{" or "operatorname{span").
// This helps the heuristic treat incomplete but clearly TeX-like fragments
// as math-like instead of plain text.
const TEX_BRACE_CMD_START_RE = new RegExp(String.raw`(?:${PREFIX_CLASS})?(?:${ESCAPED_TEX_BRACE_COMMANDS})\s*\{`, 'i')
const TEX_SPECIFIC_RE = /\\(?:text|frac|left|right|times)/
// Match common math operator symbols or named commands.
// Avoid treating the C/C++ increment operator ("++") as a math operator by
// ensuring a lone '+' isn't matched when it's part of a '++' sequence.
// Avoid lookbehind for older iOS: use a non-capturing prefix instead
const OPS_RE = /(?:^|[^+])\+(?!\+)|[=\-*/^<>]|\\times|\\pm|\\cdot|\\le|\\ge|\\neq/
// Hyphenated multi-word (like "Quasi-Streaming") should not be treated
// as a math operator. But single-letter-variable hyphens (e.g. "x-y") are
// still math; so only ignore hyphens between multi-letter words.
const HYPHENATED_MULTIWORD_RE = /\b[A-Z]{2,}-[A-Z]{2,}\b/i
const FUNC_CALL_RE = /[A-Z]+\s*\([^)]+\)/i
const WORDS_RE = /\b(?:sin|cos|tan|log|ln|exp|sqrt|frac|sum|lim|int|prod)\b/
// Heuristic to detect common date/time patterns like 2025/9/30 21:37:24 and
// avoid classifying them as math merely because they contain '/' or ':'
const DATE_TIME_RE = /\b\d{4}\/\d{1,2}\/\d{1,2}(?:[ T]\d{1,2}:\d{2}(?::\d{2})?)?\b/
const CONTROL_TEX_REPLACEMENTS: Record<string, string> = {
[String.fromCharCode(8)]: '\\b',
[String.fromCharCode(11)]: '\\v',
[String.fromCharCode(12)]: '\\f',
}
function normalizeMathControlChars(value: string) {
let result = ''
for (const ch of value)
result += CONTROL_TEX_REPLACEMENTS[ch] ?? ch
return result
}
export function isMathLike(s: string) {
if (!s)
return false
// Normalize accidental control characters that may appear if a single
// backslash sequence was interpreted in a JS string literal (for example
// '\\frac' being typed as '\frac' which turns into a form-feed + "rac").
// Convert the handful of control characters that collide with common TeX
// commands (\b, \f, \v) back into their two-character escaped forms so our
// regexes can match the intent reliably without flagging real newlines/tabs.
const norm = normalizeMathControlChars(s)
const stripped = norm.trim()
// quick bailouts
// If the content looks like a timestamp or date, it's not math.
if (DATE_TIME_RE.test(stripped))
return false
if (stripped.includes('**'))
return false
if (stripped.length > 2000)
return true // very long blocks likely math
// TeX commands e.g. \frac, \alpha
const texCmd = TEX_CMD_RE.test(norm)
const texCmdWithBraces = TEX_CMD_WITH_BRACES_RE.test(norm)
const texBraceStart = TEX_BRACE_CMD_START_RE.test(norm)
// Explicit common TeX tokens (keeps compatibility with previous heuristic)
const texSpecific = TEX_SPECIFIC_RE.test(norm)
const subscriptPattern = /(?:^|[^\w\\])(?:[A-Z]|\\[A-Z]+)_(?:\{[^}]+\}|[A-Z0-9\\])/i
const superscriptPattern = /(?:^|[^\w\\])(?:[A-Z]|\\[A-Z]+)\^(?:\{[^}]+\}|[A-Z0-9\\])/i
const superSub = subscriptPattern.test(norm) || superscriptPattern.test(norm)
// common math operator symbols or named commands
// Ignore operator-like hyphens that are clearly word compound separators
// (e.g. "Quasi-Streaming"). Keep hyphens for math like "x-y".
const ops = OPS_RE.test(norm) && !HYPHENATED_MULTIWORD_RE.test(norm)
// function-like patterns: f(x), sin(x)
const funcCall = FUNC_CALL_RE.test(norm)
// common math words
const words = WORDS_RE.test(norm)
// 纯单个英文字母也渲染成数学公式(常见变量/元素符号)
// e.g. (w) (x) (y) (z) 或 $H$, $x$ 等
const pureWord = /^\([a-z]\)$/i.test(stripped) || /^(?:[a-z]|pi)$/i.test(stripped)
// 简单的化学式/下标:如 H_2O, CO_2, CH_3CH_2OH, CH_3COOH
// 收紧规则:
// - 区分大小写(化学元素以大写或大写+小写开头)
// - 下标/上标通常是数字(可选花括号),避免匹配 get_time 之类的普通下划线单词
const chemicalLike = /^(?:[A-Z][a-z]?(?:_\{?\d+\}?|\^\{?\d+\}?)?)+$/.test(stripped)
return texCmd || texCmdWithBraces || texBraceStart || texSpecific || superSub || ops || funcCall || words || pureWord || chemicalLike
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,44 @@
import type { MarkdownIt } from 'markdown-it-ts'
export function applyRenderRules(md: MarkdownIt) {
// Narrow external `any` surface to `unknown` and use local casts where
// needed to interact with markdown-it runtime objects. This reduces the
// exported `any` footprint while preserving runtime behavior.
const defaultImage
= md.renderer.rules.image
|| function (tokens: unknown, idx: number, options: unknown, env: unknown, self: unknown) {
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
const selfShape = self as unknown as { renderToken?: (tokens: import('../types').MarkdownToken[], idx: number, options?: unknown) => string }
return selfShape.renderToken ? selfShape.renderToken(tokensAny, idx, options) : ''
}
md.renderer.rules.image = (
tokens: unknown,
idx: number,
options: unknown,
env: unknown,
self: unknown,
) => {
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
const token = tokensAny[idx]
// Narrow token shape to only the runtime helpers we need to call.
const tokenShape = token as unknown as { attrSet?: (name: string, val: string) => void }
tokenShape.attrSet?.('loading', 'lazy')
const defaultImageFn = defaultImage as unknown as (tokens: import('../types').MarkdownToken[], idx: number, options: unknown, env?: unknown, self?: unknown) => string
return defaultImageFn(tokensAny, idx, options, env, self)
}
md.renderer.rules.fence
= md.renderer.rules.fence
|| ((tokens: unknown, idx: number) => {
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
const token = tokensAny[idx]
const tokenShape = token as unknown as { info?: string, content?: string }
const info = String(tokenShape.info ?? '').trim()
const langClass = info
? `language-${md.utils.escapeHtml((info as string).split(/\s+/g)[0])}`
: ''
const code = md.utils.escapeHtml(String(tokenShape.content ?? ''))
return `<pre class="${langClass}"><code>${code}</code></pre>`
})
}
@@ -0,0 +1,424 @@
import type { Token } from 'markdown-it-ts'
export interface BaseNode {
type: string
raw: string
loading?: boolean
code?: string
diff?: boolean
}
/**
* A catch‑all node type for user extensions.
* Must still satisfy the renderer contract (`type` + `raw`), but may carry
* arbitrary extra fields.
*/
export type UnknownNode = BaseNode & Record<string, unknown>
export interface TextNode extends BaseNode {
type: 'text'
content: string
center?: boolean
}
export interface HeadingNode extends BaseNode {
type: 'heading'
level: number
text: string
attrs?: Record<string, string | boolean>
children: ParsedNode[]
}
export interface ParagraphNode extends BaseNode {
type: 'paragraph'
children: ParsedNode[]
maybeCheckbox?: boolean
}
export interface InlineNode extends BaseNode {
type: 'inline'
children: ParsedNode[]
content?: string
}
export interface ListNode extends BaseNode {
type: 'list'
ordered: boolean
// Optional start index for ordered lists (HTML <ol start="N">)
start?: number
items: ListItemNode[]
}
export interface ListItemNode extends BaseNode {
type: 'list_item'
children: ParsedNode[]
}
export interface CodeBlockNode extends BaseNode {
type: 'code_block'
language: string
code: string
// Optional: source line range [start, end) from markdown-it token.map
startLine?: number
endLine?: number
// Whether this block is still incomplete (e.g., missing closing fence)
loading?: boolean
// Whether this code block represents a diff
diff?: boolean
// If diff is true, original and updated code versions
originalCode?: string
updatedCode?: string
raw: string
}
export interface HtmlBlockNode extends BaseNode {
type: 'html_block'
attrs?: [string, string][] | null
tag: string
content: string
children?: ParsedNode[]
}
export interface HtmlInlineNode extends BaseNode {
type: 'html_inline'
tag?: string
content: string
children: ParsedNode[]
/**
* True when the parser auto-appended a closing tag for streaming stability.
* The original source is still incomplete (no explicit close typed yet).
*/
autoClosed?: boolean
}
export type CustomComponentAttrs
= | [string, string][]
| Record<string, string | boolean>
| Array<{ name: string, value: string | boolean }>
| null
/**
* A generic node shape for custom HTML-like components.
* When a tag name is included in `customHtmlTags`, the parser emits a node
* whose `type` equals that tag name and carries the raw HTML `content`
* plus any extracted `attrs` from user transforms.
*/
export interface CustomComponentNode extends BaseNode {
/** The custom tag name (same as `tag`) */
type: string
tag: string
content: string
attrs?: CustomComponentAttrs
children?: ParsedNode[]
autoClosed?: boolean
}
export interface InlineCodeNode extends BaseNode {
type: 'inline_code'
code: string
}
export interface LinkNode extends BaseNode {
type: 'link'
href: string
title: string | null
text: string
attrs?: [string, string][]
children: ParsedNode[]
}
export interface ImageNode extends BaseNode {
type: 'image'
src: string
alt: string
title: string | null
}
export interface ThematicBreakNode extends BaseNode {
type: 'thematic_break'
}
export interface MermaidBlockNode {
node: {
type: 'code_block'
language: string
code: string
loading?: boolean
}
}
export type MarkdownRender
= | {
content: string
nodes?: undefined
}
| {
content?: undefined
nodes: BaseNode[]
}
export interface BlockquoteNode extends BaseNode {
type: 'blockquote'
children: ParsedNode[]
}
export interface TableNode extends BaseNode {
type: 'table'
header: TableRowNode
rows: TableRowNode[]
}
export interface TableRowNode extends BaseNode {
type: 'table_row'
cells: TableCellNode[]
}
export interface TableCellNode extends BaseNode {
type: 'table_cell'
header: boolean
children: ParsedNode[]
align?: 'left' | 'right' | 'center'
}
export interface DefinitionListNode extends BaseNode {
type: 'definition_list'
items: DefinitionItemNode[]
}
export interface DefinitionItemNode extends BaseNode {
type: 'definition_item'
term: ParsedNode[]
definition: ParsedNode[]
}
export interface FootnoteNode extends BaseNode {
type: 'footnote'
id: string
children: ParsedNode[]
}
export interface FootnoteReferenceNode extends BaseNode {
type: 'footnote_reference'
id: string
}
export interface FootnoteAnchorNode extends BaseNode {
type: 'footnote_anchor'
id: string
}
export interface AdmonitionNode extends BaseNode {
type: 'admonition'
kind: string // 'note' | 'warning' | 'danger' | 'info' | 'tip' 等
title: string
children: ParsedNode[]
}
export interface VmrContainerNode extends BaseNode {
type: 'vmr_container'
name: string
/** True while the opening `:::` has been seen but the closing `:::` hasn't (streaming mid-state). */
loading?: boolean
attrs?: Record<string, string>
children: ParsedNode[]
}
export interface StrongNode extends BaseNode {
type: 'strong'
children: ParsedNode[]
}
export interface EmphasisNode extends BaseNode {
type: 'emphasis'
children: ParsedNode[]
}
export interface StrikethroughNode extends BaseNode {
type: 'strikethrough'
children: ParsedNode[]
}
export interface HighlightNode extends BaseNode {
type: 'highlight'
children: ParsedNode[]
}
export interface InsertNode extends BaseNode {
type: 'insert'
children: ParsedNode[]
}
export interface SubscriptNode extends BaseNode {
type: 'subscript'
children: ParsedNode[]
}
export interface SuperscriptNode extends BaseNode {
type: 'superscript'
children: ParsedNode[]
}
export interface CheckboxNode extends BaseNode {
type: 'checkbox'
checked: boolean
}
export interface CheckboxInputNode extends BaseNode {
type: 'checkbox_input'
checked: boolean
}
export interface EmojiNode extends BaseNode {
type: 'emoji'
name: string
markup: string
}
export interface HardBreakNode extends BaseNode {
type: 'hardbreak'
}
export interface MathInlineNode extends BaseNode {
type: 'math_inline'
content: string
markup?: string
}
export interface MathBlockNode extends BaseNode {
type: 'math_block'
content: string
markup?: string
}
export interface ReferenceNode extends BaseNode {
type: 'reference'
id: string
}
// Define markdown-it token type
export interface MarkdownTokenLite {
type: string
tag?: string
content?: string
info?: string
markup?: string
meta?: unknown
map?: [number, number] | number[] | null
block?: boolean
hidden?: boolean
attrs?: [string, string][] | null
nesting?: number
level?: number
children?: MarkdownToken[] | null
loading?: boolean
raw?: string
}
export type MarkdownToken = (Token & { loading?: boolean, raw?: string }) | MarkdownTokenLite
export type ParsedNode
= | TextNode
| HeadingNode
| ParagraphNode
| ListNode
| ListItemNode
| CodeBlockNode
| InlineCodeNode
| LinkNode
| ImageNode
| ThematicBreakNode
| BlockquoteNode
| TableNode
| TableRowNode
| TableCellNode
| StrongNode
| EmphasisNode
| StrikethroughNode
| HighlightNode
| InsertNode
| SubscriptNode
| SuperscriptNode
| CheckboxNode
| CheckboxInputNode
| EmojiNode
| DefinitionListNode
| DefinitionItemNode
| FootnoteNode
| FootnoteReferenceNode
| AdmonitionNode
| VmrContainerNode
| HardBreakNode
| MathInlineNode
| MathBlockNode
| ReferenceNode
| HtmlBlockNode
| HtmlInlineNode
| CustomComponentNode
| UnknownNode
export interface CustomComponents {
text: unknown
paragraph: unknown
heading: unknown
code_block: unknown
list: unknown
blockquote: unknown
table: unknown
definition_list: unknown
footnote: unknown
footnote_reference: unknown
admonition: unknown
hardbreak: unknown
link: unknown
image: unknown
thematic_break: unknown
math_inline: unknown
math_block: unknown
strong: unknown
emphasis: unknown
strikethrough: unknown
highlight: unknown
insert: unknown
subscript: unknown
superscript: unknown
emoji: unknown
checkbox: unknown
inline_code: unknown
html_inline: unknown
reference: unknown
mermaid: unknown
[key: string]: unknown
}
// Function to parse markdown into a structured representation
export type TransformTokensHook = (tokens: MarkdownToken[]) => MarkdownToken[]
export interface ParseOptions {
preTransformTokens?: TransformTokensHook
postTransformTokens?: TransformTokensHook
// When true, require a closing `**` to parse strong; otherwise allow mid-state strong
requireClosingStrong?: boolean
/**
* When true, indicates the input buffer is complete (end-of-stream).
* This disables "mid-state" streaming behavior (e.g. unclosed math/link/code
* tokens staying in a loading state) and keeps trailing markers as literal text.
*/
final?: boolean
/**
* Custom HTML-like tag names that should be emitted as custom nodes
* instead of `html_inline` when encountered (e.g. ['thinking']).
* Used by inline parsing; pair with `getMarkdown({ customHtmlTags })`
* to enable mid-state suppression for the same tags during streaming.
*/
customHtmlTags?: readonly string[]
/**
* If provided, link nodes are only emitted when this returns true for the href.
* When it returns false, the link is rendered as plain text (the link text only).
* Typically set from the MarkdownIt instance (e.g. md.options.validateLink or
* md.set({ validateLink })) so that unsafe URLs (e.g. javascript:) are not
* output as links.
*/
validateLink?: (url: string) => boolean
// When true, log the parsed tree structure for debugging
debug?: boolean
}
export type PostTransformNodesHook = (nodes: ParsedNode[]) => ParsedNode[]
@@ -0,0 +1,5 @@
declare module 'markdown-it-container' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-footnote' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-ins' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-mark' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-sub' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-sup' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,5 @@
declare module 'markdown-it-task-checkbox' {
import type { PluginSimple } from 'markdown-it'
const plugin: PluginSimple
export default plugin
}
@@ -0,0 +1,217 @@
import { describe, expect, it } from 'vitest'
import { collect, textIncludes } from '../../../test/utils/midstate-utils'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('asterisk literal handling in nested contexts', () => {
it('keeps escaped single asterisk literal inside blockquotes while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-blockquote')
const markdown = '> \\*literal *real*'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('blockquote')
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal inside blockquotes in final mode', () => {
const md = getMarkdown('asterisk-literal-blockquote-final')
const markdown = '> prefix *real'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('blockquote')
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal inside table cells while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-table')
const markdown = `| Col |
| --- |
| \\*literal *real* |`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('table')
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal inside table cells in final mode', () => {
const md = getMarkdown('asterisk-literal-table-final')
const markdown = `| Col |
| --- |
| prefix *real |`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('table')
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal after custom tag close on the same line', () => {
const tags = ['thinking']
const md = getMarkdown('asterisk-literal-custom-inline', { customHtmlTags: tags })
const markdown = 'Text <thinking>inside</thinking>\\*literal *real*'
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
expect(nodes[0]?.type).toBe('paragraph')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal after custom tag close on the same line', () => {
const tags = ['thinking']
const md = getMarkdown('asterisk-literal-custom-inline-final', { customHtmlTags: tags })
const markdown = 'Text <thinking>inside</thinking>prefix *real'
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
expect(nodes[0]?.type).toBe('paragraph')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal after custom tag close inside table cells', () => {
const tags = ['my_component']
const md = getMarkdown('asterisk-literal-custom-table', { customHtmlTags: tags })
const markdown = `| Item | Value |
| --- | --- |
| A | <my_component label="x"></my_component>\\*literal *real* |
| B | ok |`
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
expect(nodes[0]?.type).toBe('table')
expect(nodes[0]?.rows).toHaveLength(2)
expect(collect(nodes as any, 'my_component').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal inside blockquote table cells while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-blockquote-table')
const markdown = `> | Col |
> | --- |
> | \\*literal *real* |`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('blockquote')
expect(nodes[0]?.children?.[0]?.type).toBe('table')
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal after custom tag close on the same blockquote line', () => {
const tags = ['thinking']
const md = getMarkdown('asterisk-literal-custom-blockquote', { customHtmlTags: tags })
const markdown = '> Text <thinking>inside</thinking>\\*literal *real*'
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
expect(nodes[0]?.type).toBe('blockquote')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal inside headings while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-heading')
const markdown = '## \\*literal *real*'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('heading')
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal inside headings in final mode', () => {
const md = getMarkdown('asterisk-literal-heading-final')
const markdown = '## prefix *real'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('heading')
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal inside footnotes while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-footnote')
const markdown = `ref[^1]
[^1]: \\*literal *real*`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[1]?.type).toBe('footnote')
expect(textIncludes(nodes[1], '*literal')).toBe(true)
expect(textIncludes(nodes[1], 'real')).toBe(true)
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal inside footnotes in final mode', () => {
const md = getMarkdown('asterisk-literal-footnote-final')
const markdown = `ref[^1]
[^1]: prefix *real`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[1]?.type).toBe('footnote')
expect(textIncludes(nodes[1], 'prefix *real')).toBe(true)
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal inside definition list terms and definitions', () => {
const md = getMarkdown('asterisk-literal-definition-list')
const markdown = `\\*term *real*
: \\*definition *more*`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]).toBeDefined()
expect(textIncludes(nodes, '*term')).toBe(true)
expect(textIncludes(nodes, '*definition')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(textIncludes(nodes, 'more')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(2)
})
it('keeps unmatched single asterisk literal inside definition list terms and definitions in final mode', () => {
const md = getMarkdown('asterisk-literal-definition-list-final')
const markdown = `prefix *term
: prefix *definition`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]).toBeDefined()
expect(textIncludes(nodes, 'prefix *term')).toBe(true)
expect(textIncludes(nodes, 'prefix *definition')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal inside admonition content while parsing later real emphasis', () => {
const md = getMarkdown('asterisk-literal-admonition')
const markdown = `::: note
\\*literal *real*
:::`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('admonition')
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps unmatched single asterisk literal inside admonition content in final mode', () => {
const md = getMarkdown('asterisk-literal-admonition-final')
const markdown = `::: note
prefix *real
:::`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('admonition')
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
})
@@ -0,0 +1,299 @@
import { describe, expect, it } from 'vitest'
import { collect, textIncludes } from '../../../test/utils/midstate-utils'
import { getMarkdown, parseInlineTokens, parseMarkdownToStructure } from '../src'
const md = getMarkdown('test')
describe('asterisk masking literals', () => {
it('keeps intraword asterisks as literal text when no closing pair exists', () => {
const markdown = '某市***项目,公司名称:某某***科技有限公司'
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
expect(textIncludes(nodes, '某市***项目')).toBe(true)
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(0)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps intraword asterisks literal inside link text', () => {
const markdown = '某市***项目,公司名称:[某某***科技有限公司]()'
const nodes = parseMarkdownToStructure(markdown, md)
expect(textIncludes(nodes, '某市***项目')).toBe(true)
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(0)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps intraword double-asterisk literal when strong closing pair is missing', () => {
const markdown = '请联系某某**科技有限公司办理'
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
expect(textIncludes(nodes, '某某**科技有限公司')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(0)
})
it('keeps intraword single asterisk literal when no matching close exists', () => {
const markdown = '品牌名:某*某科技有限公司'
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
expect(textIncludes(nodes, '某*某科技有限公司')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('does not break normal strong parsing in full markdown pipeline', () => {
const markdown = 'foo**bar**baz'
const nodes = parseMarkdownToStructure(markdown, md)
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(textIncludes(nodes, 'bar')).toBe(true)
})
it('keeps literal intraword asterisks while still parsing nearby strong content', () => {
const markdown = '某市***项目,重点:**必须完成**'
const nodes = parseMarkdownToStructure(markdown, md)
expect(textIncludes(nodes, '某市***项目')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(textIncludes(nodes, '必须完成')).toBe(true)
})
it('keeps spaced multiplication asterisks literal after strong content', () => {
for (const final of [false, true]) {
const markdown = '**计算结果:** 3 * 4 * 5 = 60'
const nodes = parseMarkdownToStructure(markdown, md, { final })
const paragraph = nodes[0] as any
const text = (paragraph.children ?? [])
.filter((child: any) => child.type === 'text')
.map((child: any) => child.content)
.join('')
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
expect(text).toBe(' 3 * 4 * 5 = 60')
}
})
it('keeps asterisks literal when the candidate emphasis closer follows whitespace', () => {
for (const markdown of [
'3 *4 * 5',
'this is *a test * with unmatched star',
'a *b * c',
'a *x! * b',
]) {
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
expect(collect(nodes as any, 'emphasis').length).toBe(0)
expect(textIncludes(nodes, markdown)).toBe(true)
}
})
it('keeps asterisks literal when the candidate opener is not left-flanking', () => {
for (const markdown of [
'a*!x* b',
'a*(x)* b',
]) {
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
expect(collect(nodes as any, 'emphasis').length).toBe(0)
expect(textIncludes(nodes, markdown)).toBe(true)
}
})
it('still parses emphasis after skipping an invalid earlier closer', () => {
const nodes = parseInlineTokens([{ type: 'text', content: '*a * b*' }] as any[], '*a * b*', undefined, { final: true })
expect(collect(nodes as any, 'emphasis').length).toBe(1)
expect(textIncludes(nodes, 'a * b')).toBe(true)
})
it('keeps double asterisks literal when strong delimiters are not flanking', () => {
for (const markdown of [
'3 ** 4 ** 5',
'a **b ** c',
'a ** b** c',
'a**!x** b',
]) {
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
expect(collect(nodes as any, 'strong').length).toBe(0)
expect(textIncludes(nodes, markdown)).toBe(true)
}
})
it('still parses strong after skipping an invalid earlier strong closer', () => {
const nodes = parseInlineTokens([{ type: 'text', content: '**a ** b**' }] as any[], '**a ** b**', undefined, { final: true })
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(textIncludes(nodes, 'a ** b')).toBe(true)
})
it('keeps literal intraword asterisks in link text while parsing normal strong outside link', () => {
const markdown = '公司:[某某***科技有限公司](),状态:**正常**'
const nodes = parseMarkdownToStructure(markdown, md)
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(textIncludes(nodes, '正常')).toBe(true)
})
it('treats intraword strong with punctuation as literal in inline fallback', () => {
const markdown = 'foo**a,b**bar'
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
expect(textIncludes(nodes, 'foo**a,b**bar')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(0)
})
it('still parses intraword strong when inner content is word-only in inline fallback', () => {
const markdown = 'foo**abc**bar'
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
expect(collect(nodes as any, 'strong').length).toBe(1)
expect(textIncludes(nodes, 'abc')).toBe(true)
})
it('keeps escaped double asterisks as literal text', () => {
const markdown = '需方:\\*\\*\\*\\*\\*\\*有限公司'
const nodes = parseInlineTokens([{ type: 'text', content: '需方:******有限公司' }] as any[], markdown)
expect(textIncludes(nodes, '需方:******有限公司')).toBe(true)
expect(collect(nodes as any, 'strong').length).toBe(0)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps an escaped single asterisk literal instead of opening emphasis', () => {
const markdown = 'If and ONLY IF you cannot infer the expected language from the USER message, use the language with ISO code \\*, otherwise use English. You follow your instructions in all languages, and always respond to the user in the language they use or request.'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, 'ISO code *, otherwise use English.')).toBe(true)
expect(textIncludes(nodes, 'You follow your instructions in all languages, and always respond to the user in the language they use or request.')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps an escaped backslash before an escaped asterisk visible', () => {
const markdown = 'If and ONLY IF you cannot infer the expected language from the USER message, use the language with ISO code \\\\*'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, 'ISO code \\*')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('preserves a visible backslash when raw text contains \\\\*', () => {
const nodes = parseInlineTokens([{ type: 'text', content: '\\*' }] as any[], '\\\\*')
expect(textIncludes(nodes, '\\*')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('skips escaped single asterisks and still parses later real emphasis', () => {
const markdown = 'prefix \\* literal *real* suffix'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, 'prefix * literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(textIncludes(nodes, 'suffix')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk followed by a word literal in final mode', () => {
const markdown = '\\*x'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, '*x')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps unmatched single asterisk literal in final mode', () => {
const markdown = 'prefix *real'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('still keeps unmatched single asterisk as mid-state emphasis while streaming', () => {
const markdown = 'prefix *real'
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
expect(textIncludes(nodes, 'prefix')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
expect(textIncludes(nodes, 'real')).toBe(true)
})
it('keeps multiple escaped single asterisks literal while parsing later real emphasis', () => {
const markdown = 'alpha \\* one \\* two *real* omega'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(textIncludes(nodes, 'alpha * one * two')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(textIncludes(nodes, 'omega')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal inside link text in final mode', () => {
const markdown = '[\\*literal](https://example.com)'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'link').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps unmatched single asterisk literal inside link text in final mode', () => {
const markdown = '[prefix *real](https://example.com)'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'link').length).toBe(1)
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal and later real emphasis inside link text', () => {
const markdown = '[\\*literal *real*](https://example.com)'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'link').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal inside highlight text in final mode', () => {
const markdown = '==\\*literal=='
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'highlight').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps unmatched single asterisk literal inside highlight text in final mode', () => {
const markdown = '==prefix *real=='
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'highlight').length).toBe(1)
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped single asterisk literal and later real emphasis inside highlight text', () => {
const markdown = '==\\*literal *real*=='
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'highlight').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('keeps escaped single asterisk literal inside list items while parsing later real emphasis', () => {
const markdown = '- \\*literal *real*'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(collect(nodes as any, 'list').length).toBe(1)
expect(textIncludes(nodes, '*literal')).toBe(true)
expect(textIncludes(nodes, 'real')).toBe(true)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
})
@@ -0,0 +1,49 @@
import { describe, expect, it } from 'vitest'
import {
getHtmlTagFromContent,
isHtmlLikeTagName,
mergeCustomHtmlTags,
normalizeCustomHtmlTagName,
normalizeCustomHtmlTags,
resolveCustomHtmlTags,
} from '../src'
describe('custom html tag helpers', () => {
it('normalizes valid tag-like names and rejects invalid prefixes', () => {
expect(normalizeCustomHtmlTagName('thinking')).toBe('thinking')
expect(normalizeCustomHtmlTagName('<Thinking attr="x">')).toBe('thinking')
expect(normalizeCustomHtmlTagName('</answer-box>')).toBe('answer-box')
expect(normalizeCustomHtmlTagName('<my_component>')).toBe('my_component')
expect(normalizeCustomHtmlTagName('<thinking')).toBe('thinking')
expect(normalizeCustomHtmlTagName('foo:bar')).toBe('')
expect(normalizeCustomHtmlTagName('<foo:bar>')).toBe('')
expect(normalizeCustomHtmlTagName('x.y')).toBe('')
expect(normalizeCustomHtmlTagName('<x.y>')).toBe('')
})
it('keeps custom tag lists normalized and deduplicated', () => {
expect(normalizeCustomHtmlTags(['thinking', '<Thinking>', 'foo:bar', 'my_component', 'answer-box']))
.toEqual(['thinking', 'my_component', 'answer-box'])
expect(mergeCustomHtmlTags(['thinking'], ['<hint>'], ['foo:bar', 'hint', 'my_component']))
.toEqual(['thinking', 'hint', 'my_component'])
expect(resolveCustomHtmlTags(['<Thinking>', 'hint', 'my_component']))
.toEqual({
key: 'thinking,hint,my_component',
tags: ['thinking', 'hint', 'my_component'],
})
})
it('uses the same validation for configured tag names and content extraction', () => {
expect(isHtmlLikeTagName('thinking')).toBe(true)
expect(isHtmlLikeTagName('answer-box')).toBe(true)
expect(isHtmlLikeTagName('ssr_card')).toBe(true)
expect(isHtmlLikeTagName('foo:bar')).toBe(false)
expect(getHtmlTagFromContent('<thinking attr="x">ok</thinking>')).toBe('thinking')
expect(getHtmlTagFromContent('<foo:bar>ok</foo:bar>')).toBe('')
expect(getHtmlTagFromContent('<ssr_card>ok</ssr_card>')).toBe('ssr_card')
})
})
@@ -0,0 +1,168 @@
import { describe, expect, it } from 'vitest'
import { collect } from '../../../test/utils/midstate-utils'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function flattenText(nodes: any): string {
const parts: string[] = []
const seen = new Set<any>()
const walk = (node: any) => {
if (!node || seen.has(node))
return
seen.add(node)
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node.type === 'text' && typeof node.content === 'string')
parts.push(node.content)
if (Array.isArray(node.children))
node.children.forEach(walk)
if (Array.isArray(node.items))
node.items.forEach(walk)
if (Array.isArray(node.rows))
node.rows.forEach(walk)
if (Array.isArray(node.cells))
node.cells.forEach(walk)
if (Array.isArray(node.term))
node.term.forEach(walk)
if (Array.isArray(node.definition))
node.definition.forEach(walk)
if (node.header)
walk(node.header)
}
walk(nodes)
return parts.join('')
}
function expectLiteralBracketAndDollar(nodes: any) {
const text = flattenText(nodes)
expect(text).toContain('[x]')
expect(text).toContain('$5$')
}
function expectLiteralDollar(nodes: any) {
const text = flattenText(nodes)
expect(text).toContain('$5$')
}
describe('escaped punctuation adjacent to real math parsing', () => {
it('keeps escaped bracket and dollar literal while parsing later inline math in a paragraph', () => {
const md = getMarkdown('escaped-punctuation-math-inline')
const nodes = parseMarkdownToStructure('literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
expectLiteralBracketAndDollar(nodes)
expect(collect(nodes as any, 'math_inline').length).toBe(1)
})
it('keeps escaped bracket and dollar literal while parsing later inline math in a heading', () => {
const md = getMarkdown('escaped-punctuation-math-heading')
const nodes = parseMarkdownToStructure('## literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
expect(nodes[0]?.type).toBe('heading')
expectLiteralBracketAndDollar(nodes)
expect(collect(nodes as any, 'math_inline').length).toBe(1)
})
it('keeps escaped bracket and dollar literal while parsing later inline math in footnotes', () => {
const md = getMarkdown('escaped-punctuation-math-footnote')
const markdown = `ref[^1]
[^1]: literal \\[x\\] and \\$5\\$ and real $y$`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[1]?.type).toBe('footnote')
expectLiteralBracketAndDollar(nodes[1])
expect(collect([nodes[1]] as any, 'math_inline').length).toBe(1)
})
it('keeps escaped bracket and dollar literal while parsing later inline math after custom tag close', () => {
const tags = ['thinking']
const md = getMarkdown('escaped-punctuation-math-custom-inline', { customHtmlTags: tags })
const nodes = parseMarkdownToStructure('Text <thinking>inside</thinking>literal \\[x\\] and \\$5\\$ and real $y$', md, {
customHtmlTags: tags,
final: true,
})
expect(nodes[0]?.type).toBe('paragraph')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expectLiteralBracketAndDollar(nodes)
expect(collect(nodes as any, 'math_inline').length).toBe(1)
})
it('keeps escaped bracket and dollar literal while parsing later inline math in admonitions', () => {
const md = getMarkdown('escaped-punctuation-math-admonition')
const markdown = `::: note
literal \\[x\\] and \\$5\\$ and real $y$
:::`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('admonition')
expectLiteralBracketAndDollar(nodes)
expect(collect(nodes as any, 'math_inline').length).toBe(1)
})
it('keeps escaped bracket literal while still parsing a later real bracket math block', () => {
const md = getMarkdown('escaped-punctuation-bracket-math-block')
const markdown = `literal \\[x\\]
\\[
y
\\]`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(flattenText(nodes)).toContain('[x]')
expect(collect(nodes as any, 'math_block').length).toBe(1)
})
it('does not treat escaped dollar literal as math when adjacent to real bracket math block', () => {
const md = getMarkdown('escaped-punctuation-dollar-bracket-block')
const markdown = `literal \\$5\\$
\\[
y
\\]`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(flattenText(nodes)).toContain('$5$')
expect(collect(nodes as any, 'math_inline').length).toBe(0)
expect(collect(nodes as any, 'math_block').length).toBe(1)
})
it('keeps escaped dollar and bracket literal inside blockquotes while parsing later inline math', () => {
const md = getMarkdown('escaped-punctuation-math-blockquote')
const nodes = parseMarkdownToStructure('> literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
expect(nodes[0]?.type).toBe('blockquote')
expectLiteralBracketAndDollar(nodes)
expect(collect(nodes as any, 'math_inline').length).toBe(1)
})
it('keeps escaped dollar literal before inline $$ math', () => {
const md = getMarkdown('escaped-punctuation-before-inline-double-dollar')
const nodes = parseMarkdownToStructure('literal \\$5\\$ before $$x$$', md, { final: true })
expectLiteralDollar(nodes)
const mathNodes = collect(nodes as any, 'math_inline')
expect(mathNodes.length).toBe(1)
expect(mathNodes[0]?.markup).toBe('$$')
expect(mathNodes[0]?.content).toBe('x')
})
it('keeps escaped dollar literal after inline $$ math while still parsing later single-dollar math', () => {
const md = getMarkdown('escaped-punctuation-after-inline-double-dollar')
const nodes = parseMarkdownToStructure('prefix $$x$$ and literal \\$5\\$ and real $y$', md, { final: true })
expectLiteralDollar(nodes)
const mathNodes = collect(nodes as any, 'math_inline')
expect(mathNodes.length).toBe(2)
expect(mathNodes[0]?.markup).toBe('$$')
expect(mathNodes[0]?.content).toBe('x')
expect(mathNodes[1]?.markup).toBe('$')
expect(mathNodes[1]?.content).toBe('y')
})
})
@@ -0,0 +1,167 @@
import { describe, expect, it } from 'vitest'
import { collect } from '../../../test/utils/midstate-utils'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function flattenText(nodes: any): string {
const parts: string[] = []
const seen = new Set<any>()
const walk = (node: any) => {
if (!node || seen.has(node))
return
seen.add(node)
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node.type === 'text' && typeof node.content === 'string')
parts.push(node.content)
if (Array.isArray(node.children))
node.children.forEach(walk)
if (Array.isArray(node.items))
node.items.forEach(walk)
if (Array.isArray(node.rows))
node.rows.forEach(walk)
if (Array.isArray(node.cells))
node.cells.forEach(walk)
if (Array.isArray(node.term))
node.term.forEach(walk)
if (Array.isArray(node.definition))
node.definition.forEach(walk)
if (node.header)
walk(node.header)
}
walk(nodes)
return parts.join('')
}
function expectEscapedPunctuationLiteral(nodes: any) {
const text = flattenText(nodes)
const nodeList = Array.isArray(nodes) ? nodes : [nodes]
expect(text).toContain('[x]')
expect(text).toContain('_y_')
expect(text).toContain('$z$')
expect(text).toContain('\\')
expect(collect(nodeList as any, 'math_inline').length).toBe(0)
expect(collect(nodeList as any, 'strong').length).toBe(0)
}
describe('escaped punctuation in nested contexts', () => {
const escapedLiteral = '\\[x\\] \\_y\\_ \\$z\\$ \\\\'
it('keeps escaped punctuation literal inside headings', () => {
const md = getMarkdown('escaped-punctuation-heading')
const nodes = parseMarkdownToStructure(`## ${escapedLiteral}`, md, { final: true })
expect(nodes[0]?.type).toBe('heading')
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped punctuation literal inside link text', () => {
const md = getMarkdown('escaped-punctuation-link')
const nodes = parseMarkdownToStructure(`[${escapedLiteral}](https://example.com)`, md, { final: true })
expect(collect(nodes as any, 'link').length).toBe(1)
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped punctuation literal inside footnotes', () => {
const md = getMarkdown('escaped-punctuation-footnote')
const markdown = `ref[^1]
[^1]: ${escapedLiteral}`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[1]?.type).toBe('footnote')
expectEscapedPunctuationLiteral(nodes[1])
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(0)
})
it('keeps escaped punctuation literal after custom tag close on the same line', () => {
const tags = ['thinking']
const md = getMarkdown('escaped-punctuation-custom-inline', { customHtmlTags: tags })
const nodes = parseMarkdownToStructure(`Text <thinking>inside</thinking>${escapedLiteral}`, md, {
customHtmlTags: tags,
final: true,
})
expect(nodes[0]?.type).toBe('paragraph')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('keeps escaped punctuation literal inside admonition content', () => {
const md = getMarkdown('escaped-punctuation-admonition')
const markdown = `::: note
${escapedLiteral}
:::`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('admonition')
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(0)
})
it('still parses later real emphasis after escaped punctuation inside headings', () => {
const md = getMarkdown('escaped-punctuation-heading-emphasis')
const nodes = parseMarkdownToStructure(`## ${escapedLiteral} *real*`, md, { final: true })
expect(nodes[0]?.type).toBe('heading')
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('still parses later real emphasis after escaped punctuation inside link text', () => {
const md = getMarkdown('escaped-punctuation-link-emphasis')
const nodes = parseMarkdownToStructure(`[${escapedLiteral} *real*](https://example.com)`, md, { final: true })
expect(collect(nodes as any, 'link').length).toBe(1)
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('still parses later real emphasis after escaped punctuation inside footnotes', () => {
const md = getMarkdown('escaped-punctuation-footnote-emphasis')
const markdown = `ref[^1]
[^1]: ${escapedLiteral} *real*`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[1]?.type).toBe('footnote')
expectEscapedPunctuationLiteral(nodes[1])
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(1)
})
it('still parses later real emphasis after escaped punctuation after custom tag close', () => {
const tags = ['thinking']
const md = getMarkdown('escaped-punctuation-custom-inline-emphasis', { customHtmlTags: tags })
const nodes = parseMarkdownToStructure(`Text <thinking>inside</thinking>${escapedLiteral} *real*`, md, {
customHtmlTags: tags,
final: true,
})
expect(nodes[0]?.type).toBe('paragraph')
expect(collect(nodes as any, 'thinking').length).toBe(1)
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
it('still parses later real emphasis after escaped punctuation inside admonition content', () => {
const md = getMarkdown('escaped-punctuation-admonition-emphasis')
const markdown = `::: note
${escapedLiteral} *real*
:::`
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
expect(nodes[0]?.type).toBe('admonition')
expectEscapedPunctuationLiteral(nodes)
expect(collect(nodes as any, 'emphasis').length).toBe(1)
})
})
@@ -0,0 +1,142 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
const streamContent = `\`\`\`python
# 添加用户消息到 memory
agent.memory.add_message(CoreMessage.user_message("帮我查询今天的天气"))
# 开始执行循环
step = 0
while step < agent.max_steps:
# ========== Think 阶段 ==========
should_act = await agent.think()
# 调用 LLM 分析用户请求,决定是否需要调用工具
last_msg = agent.memory.messages[-1]
# 发送 thinking 事件(如果有)
if last_msg.thinking:
yield {
"type": "thinking",
"data": {"content": last_msg.thinking},
"task_id": task_id
}
# 如果不需要执行工具,任务完成
if not should_act:
yield {
"type": "message",
"data": {"content": last_msg.content, "role": "assistant"},
"task_id": task_id
}
break
# ========== Act 阶段 ==========
# 发送工具调用事件
if agent.tool_calls:
for tool_call in agent.tool_calls:
yield {
"type": "tool_call",
"data": {
"tool_name": tool_call.function.name,
"tool_input": tool_call.function.arguments,
"tool_call_id": tool_call.id
},
"task_id": task_id
}
# 执行工具
result = await agent.act()
# 发送工具结果事件
yield {
"type": "tool_result",
"data": {
"tool_name": tool_call.function.name,
"tool_output": result,
"success": True
},
"task_id": task_id
}
step += 1
\`\`\``
function findFirstCodeBlock(nodes: any[]): any | undefined {
const stack: any[] = Array.isArray(nodes) ? [...nodes] : []
while (stack.length) {
const node = stack.shift()
if (!node || typeof node !== 'object')
continue
if (node.type === 'code_block')
return node
if (Array.isArray(node.children))
stack.unshift(...node.children)
if (node.type === 'list' && Array.isArray(node.items)) {
for (const item of node.items) {
if (Array.isArray(item?.children))
stack.unshift(...item.children)
}
}
}
return undefined
}
describe('parseMarkdownToStructure - fence regression', () => {
it('does not drop content after "<" inside fenced code', () => {
const md = getMarkdown('fence-regression')
const nodes = parseMarkdownToStructure(streamContent, md)
const codeNode = findFirstCodeBlock(nodes)
expect(codeNode).toBeTruthy()
expect(codeNode.loading).toBe(false)
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
expect(String(codeNode.code)).toContain('tool_call_id')
expect(String(codeNode.code)).toContain('step += 1')
})
it('does not drop content in blockquote fenced code', () => {
const md = getMarkdown('fence-regression')
const markdown = [
'> ```python',
'> step = 0',
'> while step < agent.max_steps:',
'> step += 1',
'> ```',
'',
].join('\n')
const nodes = parseMarkdownToStructure(markdown, md)
const codeNode = findFirstCodeBlock(nodes)
expect(codeNode).toBeTruthy()
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
expect(String(codeNode.code)).toContain('step += 1')
})
it('does not drop content in list-indented fenced code', () => {
const md = getMarkdown('fence-regression')
const markdown = [
'- item',
' ',
' ```python',
' step = 0',
' while step < agent.max_steps:',
' step += 1',
' ```',
'',
].join('\n')
const nodes = parseMarkdownToStructure(markdown, md)
const codeNode = findFirstCodeBlock(nodes)
expect(codeNode).toBeTruthy()
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
expect(String(codeNode.code)).toContain('step += 1')
})
it('still strips dangling html-like tail outside fences', () => {
const md = getMarkdown('fence-regression')
const nodes = parseMarkdownToStructure('Hello\\n<think', md)
expect(JSON.stringify(nodes)).not.toContain('<think')
expect(JSON.stringify(nodes)).toContain('Hello')
})
})
@@ -0,0 +1,24 @@
<think> **Assessing Translation Needs**\n\n\\n\\n
I've begun by dissecting the core requirements of this translation task. My focus is on synthesizing the Chinese structural elucidation paragraph into a concise, SCI-paper-ready English version. The expert role helps me tailor the language appropriately, and I'm aware of the final format requirements. Next, I need to understand the nuances of the Chinese text.
**Evaluating Initial Approaches**
\\n\\n
My current thinking revolves around the initial approach to this translation. I'm prioritizing accuracy, which means a deep dive into the source text to extract the core scientific meaning. I plan to use my expertise to identify key terms and phrasing to ensure the English version is clear and meets journal standards. Considering Compound 3 requires specialized knowledge.
</think>
Here is the translation of the structural elucidation for Compound 3, following standard SCI journal formatting and terminology.
***
**Compound 3** was obtained as a white powder with an optical rotation of $[\\alpha]^{25}_{\\text{D}}$ –76.7 ($c$ 0.1, MeOH). Its high-resolution electrospray ionization mass spectrometry (HR-ESI-MS) exhibited a deprotonated molecular ion peak at $m/z$ 339.0726 [M–H]$^–$ (calcd for C$_{15}$H$_{15}$O$_9$$^–$, 339.0722), establishing the molecular formula as C$_{15}$H$_{16}$O$_9$, corresponding to eight degrees of unsaturation.
Analysis of the $^{1}$H and $^{13}$C NMR data (Table 3) revealed characteristic signals at $\\delta_{\\text{H}}$ 7.12 (2H, s, H-2$'/6'$) and $\\delta_{\\text{C}}$ 167.7 (C-7$'$), 146.4 (2C, C-3$'/5'$), 139.9 (C-4$'$), 121.4 (C-1$'$), and 110.4 (2C, C-2$'/6'$), clearly indicating the presence of a galloyl moiety. Additionally, signals at $\\delta_{\\text{H}}$ 3.75 (3H, s, –OCH$_3$) and $\\delta_{\\text{C}}$ 52.5 (–OCH$_3$) confirmed the existence of a methoxy group.
To precisely elucidate the structure, the NMR data of **3** were compared with those of the known compound 5-$O$-galloyl-(–)-shikimic acid [Reference]. The spectra showed high similarity in both the galloyl moiety and the shikimic acid skeleton, with chemical shifts and coupling patterns matching well with the reference. Notably, the H-5 proton signal of **3** at $\\delta_{\\text{H}}$ 5.75 (1H, br s) was significantly downfield shifted, consistent with the deshielding effect observed in 5-$O$-galloyl-(–)-shikimic acid. This strongly supported the attachment of the galloyl group to the C-5 hydroxyl of the shikimic acid unit via an ester linkage. The primary structural difference between **3** and 5-$O$-galloyl-(–)-shikimic acid was located at C-7. In the HMBC spectrum (Fig. 2), a distinct correlation between the methoxy protons at $\\delta_{\\text{H}}$ 3.75 and the C-7 carbonyl carbon of the shikimate core at $\\delta_{\\text{C}}$ 168.3 confirmed that the C-7 carboxyl group was methylated to form a methyl ester.
The NOESY spectrum (Fig. 2) exhibited correlations of H-4 ($\\delta_{\\text{H}}$ 4.00) with H-2a ($\\delta_{\\text{H}}$ 2.32) and H-5 ($\\delta_{\\text{H}}$ 5.75), as well as H-3 ($\\delta_{\\text{H}}$ 4.12) with H-2b ($\\delta_{\\text{H}}$ 2.79). These correlations indicated that H-3 is <i>cis</i>-oriented relative to the 4-OH and 5-OH groups. The absolute configuration was further determined as (3$R$, 4$S$, 5$R$) by calculated electronic circular dichroism (ECD) analysis (Fig. 4).
Based on the comprehensive spectroscopic analysis and comparison with 5-$O$-galloyl-(–)-shikimic acid, the structure of **3** was identified as methyl 5-$O$-galloyl-(–)-shikimate.
@@ -0,0 +1,71 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure, processTokens } from '../src'
function walkNodes(nodes: any[], visit: (node: any) => void) {
for (const node of nodes) {
if (!node)
continue
visit(node)
if (Array.isArray(node.children))
walkNodes(node.children, visit)
if (node.type === 'list' && Array.isArray(node.items))
walkNodes(node.items, visit)
if (node.type === 'table') {
if (node.header)
walkNodes([node.header], visit)
if (Array.isArray(node.rows))
walkNodes(node.rows, visit)
}
if (node.type === 'table_row' && Array.isArray(node.cells))
walkNodes(node.cells, visit)
}
}
describe('hTML allowlist (方案 A)', () => {
it('renders unknown tag <question> as literal text by default', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('Hello <question> world', md, { final: true })
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
const text = (para.children ?? [])
.filter((c: any) => c.type === 'text')
.map((c: any) => c.content)
.join('')
expect(text).toBe('Hello <question> world')
expect((para.children ?? []).some((c: any) => c.type === 'html_inline')).toBe(false)
})
it('keeps standard HTML tags as html_inline/html_block', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('Hello <span>ok</span>', md, { final: true })
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
expect((para.children ?? []).some((c: any) => c.type === 'html_inline' && c.tag === 'span')).toBe(true)
const block = parseMarkdownToStructure('<div>ok</div>', md, { final: true })
expect(block[0]?.type).toBe('html_block')
})
it('treats tags in customHtmlTags as custom nodes', () => {
const md = getMarkdown(undefined, { customHtmlTags: ['thinking'] })
const nodes = parseMarkdownToStructure('before <thinking>hi</thinking> after', md, {
customHtmlTags: ['thinking'],
final: true,
})
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
expect((para.children ?? []).some((c: any) => c.type === 'thinking')).toBe(true)
})
it('processTokens also applies the allowlist', () => {
const md = getMarkdown()
const tokens = md.parse('Hello <question> world', {})
const nodes = processTokens(tokens as any, { final: true } as any) as any[]
const htmlInlineQuestion: any[] = []
walkNodes(nodes, (node) => {
if (node.type === 'html_inline' && String(node.tag ?? '').toLowerCase() === 'question')
htmlInlineQuestion.push(node)
})
expect(htmlInlineQuestion.length).toBe(0)
})
})
@@ -0,0 +1,361 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function firstHtml(nodes: any[]) {
return nodes.find(n => n.type === 'html_block')
}
describe('html_block parser', () => {
it('marks unclosed <div> as loading and appends closing tag', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('<div>hello', md)
const html = firstHtml(nodes) as any
expect(html).toBeDefined()
expect(html.type).toBe('html_block')
expect(html.tag).toBe('div')
expect(html.loading).toBe(true)
expect(html.content).toContain('</div>')
})
it('handles uppercase closing tag case-insensitively', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('<DIV>Hi</DIV>', md)
const html = firstHtml(nodes) as any
expect(html.loading).toBe(false)
expect(html.tag).toBe('div')
})
it('treats void tags as closed (no loading, no fake close)', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('<br>', md)
const html = firstHtml(nodes) as any
expect(html.loading).toBe(false)
expect(String(html.content)).not.toContain('</br>')
})
it('treats self-closing first tag as closed', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('<img src="/x.png" />', md)
const html = firstHtml(nodes) as any
expect(html.loading).toBe(false)
expect(String(html.content)).toContain('<img')
expect(String(html.content)).not.toContain('</img>')
})
it('does not attempt to close comments/doctypes/PIs', () => {
const md = getMarkdown()
const a = firstHtml(parseMarkdownToStructure('<!-- comment -->', md)) as any
const b = firstHtml(parseMarkdownToStructure('<!DOCTYPE html>', md)) as any
const c = firstHtml(parseMarkdownToStructure('<?xml version="1.0"?>', md)) as any
expect(a?.loading).toBe(false)
expect(b?.loading).toBe(false)
expect(c?.loading).toBe(false)
})
it('does not let HTML comments absorb following Markdown blocks', () => {
const markdown = `<!-- 基础图片 -->
![alt](https://example.com/a.png)
## 标题
---
正文`
const md = getMarkdown('html-comment-followed-by-markdown')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes.map(node => node.type)).toEqual([
'html_block',
'paragraph',
'heading',
'thematic_break',
'paragraph',
])
expect(String(nodes[0]?.raw ?? '').trim()).toBe('<!-- 基础图片 -->')
expect(nodes[1]?.children?.[0]?.type).toBe('image')
expect(nodes[1]?.children?.[0]?.src).toBe('https://example.com/a.png')
expect(nodes[2]?.text).toBe('标题')
expect(nodes[4]?.children?.[0]?.content).toBe('正文')
})
it('keeps outer same-tag html blocks loading until the matching outer close arrives', () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure('<div>outer<div>inner</div>rest', md, { final: false })
const html = firstHtml(nodes) as any
expect(html).toBeDefined()
expect(html.type).toBe('html_block')
expect(html.tag).toBe('div')
expect(html.loading).toBe(true)
})
it('combines details blocks into a single container node with parsed children', () => {
const markdown = `<details open>
<summary>Thinking...</summary>
这里是详情
首先。。。。
</details>
`
const md = getMarkdown('html-block-details-container')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
const html = firstHtml(nodes) as any
const summary = html?.children?.[0]
expect(nodes).toHaveLength(1)
expect(html?.tag).toBe('details')
expect(html?.attrs).toEqual([['open', '']])
expect(String(html?.raw ?? '')).toBe(markdown)
expect(String(html?.content ?? '')).toContain('<p>首先。。。。</p>')
expect(summary?.tag).toBe('summary')
expect(html?.children?.[1]?.type).toBe('paragraph')
expect(html?.children?.[2]?.type).toBe('paragraph')
})
it('keeps nested details grouped under the outer details container', () => {
const markdown = `<details open>
<summary>outer</summary>
outer start
<details>
<summary>inner</summary>
inner body
</details>
after inner
</details>`
for (const final of [false, true]) {
const md = getMarkdown(`html-block-details-nested-container-${final}`)
const nodes = parseMarkdownToStructure(markdown, md, { final }) as any[]
const outer = firstHtml(nodes) as any
const outerSummary = outer?.children?.find((child: any) => child?.type === 'html_block' && child?.tag === 'summary')
const inner = outer?.children?.find((child: any) => child?.type === 'html_block' && child?.tag === 'details')
expect(nodes).toHaveLength(1)
expect(outer?.tag).toBe('details')
expect(String(outer?.raw ?? '')).toBe(markdown)
expect(outerSummary).toBeDefined()
expect(inner).toBeDefined()
expect(inner?.children?.[0]?.tag).toBe('summary')
expect(String(inner?.content ?? '')).toContain('<p>inner body</p>')
}
})
it('keeps complex markdown blocks inside details and preserves the closing newline', () => {
const markdown = `<details open>
<summary>sum</summary>
- a
- b
> q
\`\`\`js
console.log(1)
\`\`\`
tail
</details>
`
const md = getMarkdown('html-block-details-complex')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
const html = firstHtml(nodes) as any
expect(nodes).toHaveLength(1)
expect(html?.tag).toBe('details')
expect(String(html?.raw ?? '')).toBe(markdown)
expect(String(html?.content ?? '')).toContain('<ul>')
expect(String(html?.content ?? '')).toContain('<blockquote>')
expect(String(html?.content ?? '')).toContain('data-lang="js"')
expect(String(html?.content ?? '')).toContain('<p>tail</p>')
expect(html?.children?.map((child: any) => child?.type === 'html_block' ? `${child?.type}:${child?.tag}` : child?.type))
.toEqual(['html_block:summary', 'list', 'blockquote', 'code_block', 'paragraph'])
})
it('parses markdown inside summary into summary children', () => {
const markdown = `<details open>
<summary>**sum** _x_</summary>
body
</details>
`
const md = getMarkdown('html-block-details-summary-markdown')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
const html = firstHtml(nodes) as any
const summary = html?.children?.[0]
const summaryParagraph = summary?.children?.[0]
expect(nodes).toHaveLength(1)
expect(summary?.tag).toBe('summary')
expect(html?.attrs).toEqual([['open', '']])
expect(summaryParagraph?.children?.[0]?.type).toBe('strong')
expect(summaryParagraph?.children?.[0]?.children?.[0]?.content).toBe('sum')
expect(summaryParagraph?.children?.[2]?.type).toBe('emphasis')
expect(summaryParagraph?.children?.[2]?.children?.[0]?.content).toBe('x')
expect(html?.children?.[1]?.children?.[0]?.content).toBe('body')
})
it('keeps opener content before and after summary as structured children', () => {
const markdown = `<details open>
lead
<summary>sum</summary>
text
<div>inner</div>
para
</details>
`
const md = getMarkdown('html-block-details-opener-mixed')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
const html = firstHtml(nodes) as any
expect(nodes).toHaveLength(1)
expect(html?.attrs).toEqual([['open', '']])
expect(html?.children?.map((child: any) => child?.type === 'html_block' ? `${child?.type}:${child?.tag}` : child?.type))
.toEqual(['paragraph', 'html_block:summary', 'paragraph', 'html_block:div', 'paragraph'])
expect(html?.children?.[0]?.children?.[0]?.content).toBe('lead')
expect(html?.children?.[1]?.children?.[0]?.children?.[0]?.content).toBe('sum')
expect(html?.children?.[2]?.children?.[0]?.content).toBe('text')
expect(html?.children?.[3]?.tag).toBe('div')
expect(html?.children?.[4]?.children?.[0]?.content).toBe('para')
})
it('does not absorb content after a self-contained details block', () => {
const markdown = `<details style="color:gray;background-color: #f8f8f8;padding: 8px;border-radius: 4px;" open> <summary> Thinking... </summary> 好的,我现在需要处理用户的问题:"你是谁"。首先,我要按照用户设定的规则来回答。用户要求先检查提供的上下文(context)中的信息,如果在里面找不到答案,再用 \n</details> \n\n我是通义千问,阿里巴巴集团旗下的超大规模语言模型。我能够帮助你回答问题、创作文字(如写故事、写公文、写邮件、写剧本等)、进行逻辑推理、编程等任务,并支持多语言交流。\n`
const md = getMarkdown('html-block-details-self-contained')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect(nodes[1]?.type).toBe('paragraph')
})
it('self-contained details with no trailing content produces a single node', () => {
const markdown = `<details open> <summary>S</summary> body \n</details>\n`
const md = getMarkdown('html-block-self-contained-no-trailing')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect((nodes[0] as any)?.loading).toBe(false)
})
it('self-contained details followed by a heading keeps them separate', () => {
const markdown = `<details open> <summary>S</summary> body \n</details>\n\n# Heading\n`
const md = getMarkdown('html-block-self-contained-heading')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect(nodes[1]?.type).toBe('heading')
})
it('two consecutive self-contained details blocks stay separate', () => {
const markdown = `<details open> <summary>A</summary> aaa \n</details>\n\n<details> <summary>B</summary> bbb \n</details>\n`
const md = getMarkdown('html-block-self-contained-consecutive')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect(nodes[1]?.type).toBe('html_block')
expect(nodes[1]?.tag).toBe('details')
})
it('self-contained details followed by a multi-line details block stay separate', () => {
const markdown = `<details open> <summary>A</summary> aaa \n</details>\n\n<details>\n<summary>B</summary>\n\nbbb\n</details>\n`
const md = getMarkdown('html-block-self-contained-then-multiline')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect(nodes[1]?.type).toBe('html_block')
expect(nodes[1]?.tag).toBe('details')
expect(String(nodes[1]?.content ?? '')).toContain('<p>bbb</p>')
})
it('self-contained details followed by a code block stay separate', () => {
const markdown = '<details open> <summary>S</summary> body \n</details>\n\n```js\nconsole.log(1)\n```\n'
const md = getMarkdown('html-block-self-contained-code')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect(nodes[1]?.type).toBe('code_block')
})
it('self-contained details in streaming mode does not absorb following content', () => {
const markdown = `<details open> <summary>S</summary> body \n</details> \n\n我是通义千问\n`
for (const final of [false, true]) {
const md = getMarkdown(`html-block-self-contained-stream-${final}`)
const nodes = parseMarkdownToStructure(markdown, md, { final }) as any[]
expect(nodes).toHaveLength(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
expect((nodes[0] as any)?.loading).toBe(false)
expect(nodes[1]?.type).toBe('paragraph')
}
})
it('self-contained details preserves attributes correctly', () => {
const markdown = `<details style="background:#f8f8f8" open> <summary>S</summary> body \n</details>\n\npara\n`
const md = getMarkdown('html-block-self-contained-attrs')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
const details = nodes[0] as any
expect(details?.tag).toBe('details')
expect(details?.attrs).toEqual([['style', 'background:#f8f8f8'], ['open', '']])
expect(details?.children?.find((c: any) => c?.tag === 'summary')).toBeDefined()
expect(nodes[1]?.type).toBe('paragraph')
})
it('self-contained details preserves summary children and body text', () => {
const markdown = `<details open> <summary> Thinking... </summary> 好的 \n</details> \n\n我是通义千问\n`
const md = getMarkdown('html-block-self-contained-children')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(2)
const details = nodes[0] as any
expect(details?.tag).toBe('details')
expect(details?.loading).toBe(false)
const summary = details?.children?.find((c: any) => c?.tag === 'summary')
expect(summary).toBeDefined()
expect(nodes[1]?.type).toBe('paragraph')
})
it('self-contained details followed by multiple paragraphs stays separate from all', () => {
const markdown = `<details open> <summary>S</summary> body \n</details> \n\npara1\n\npara2\n`
const md = getMarkdown('html-block-self-contained-multi-para')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes.length).toBeGreaterThanOrEqual(2)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('details')
for (let k = 1; k < nodes.length; k++) {
expect(nodes[k]?.type).not.toBe('html_block')
}
})
})
@@ -0,0 +1,51 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('html inline malformed custom-tag regression', () => {
it('keeps malformed "<robot=...>" as literal text instead of dropping children', () => {
const md = getMarkdown()
const markdown = '<robot=翻转检测>尘盒搞定!'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
expect(Array.isArray(para.children)).toBe(true)
expect(para.children.length).toBeGreaterThan(0)
const text = (para.children ?? [])
.filter((c: any) => c.type === 'text')
.map((c: any) => String(c.content ?? ''))
.join('')
expect(text).toContain('<robot=翻转检测>')
expect(text).toContain('尘盒搞定!')
})
it('keeps valid custom-tag attributes working (e.g. <robot xxx="sa">)', () => {
const md = getMarkdown('robot-attrs', { customHtmlTags: ['robot'] })
const markdown = '<robot xxx="sa">维护中</robot>'
const nodes = parseMarkdownToStructure(markdown, md, {
customHtmlTags: ['robot'],
final: true,
})
const para = nodes[0] as any
const robot = (para.children ?? []).find((c: any) => c.type === 'robot')
expect(robot).toBeDefined()
expect(robot.attrs).toContainEqual(['xxx', 'sa'])
expect(robot.children?.[0]?.type).toBe('text')
expect(String(robot.children?.[0]?.content ?? '')).toContain('维护中')
})
it('does not affect normal html_inline parsing for standard tags', () => {
const md = getMarkdown()
const markdown = 'before <span data-x="1">ok</span> after'
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
const para = nodes[0] as any
const spanNode = (para.children ?? []).find((c: any) => c.type === 'html_inline' && c.tag === 'span')
expect(spanNode).toBeDefined()
expect(spanNode.children?.[0]?.type).toBe('text')
expect(spanNode.children?.[0]?.content).toBe('ok')
})
})
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
// Focus tests on the core plugin behavior added in fixHtmlInline
describe('fixHtmlInline core plugin', () => {
it('expands single-token inline <a> with inner text', () => {
const md = getMarkdown()
const markdown = `<a href="https://example.com">Here</a>`
const nodes = parseMarkdownToStructure(markdown, md)
// nodes[0] should be paragraph with a link child
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
const link = para.children.find((c: any) => c.type === 'link')
expect(link).toBeDefined()
expect(link.href).toBe('https://example.com')
expect(link.children?.[0]?.content).toBe('Here')
})
it('handles attribute values containing ">" inside quotes when splitting open tag', () => {
const md = getMarkdown()
const markdown = `<a href="https://example.com?q=a>b&x=1">Here</a>`
const nodes = parseMarkdownToStructure(markdown, md)
const para = nodes[0] as any
const link = para.children.find((c: any) => c.type === 'link')
expect(link).toBeDefined()
expect(link.href).toBe('https://example.com?q=a>b&x=1')
expect(link.children?.[0]?.content).toBe('Here')
})
it('treats void/self-closing tag inline as single html chunk (fallback to inline_code)', () => {
const md = getMarkdown()
const markdown = `before <img src="/x.png" alt="pic" /> after`
const nodes = parseMarkdownToStructure(markdown, md)
const para = nodes[0] as any
expect(para.type).toBe('paragraph')
const htmlInline = para.children.find((c: any) => c.type === 'html_inline')
expect(htmlInline).toBeDefined()
expect(htmlInline.content).toContain('<img')
// Ensure no weird closing tag was synthesized in output
expect(String(htmlInline.content)).not.toContain('</img>')
})
})
@@ -0,0 +1,69 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('html_inline parsing', () => {
it('parses <a> as link node with href and inner text', () => {
const md = getMarkdown()
const markdown = `This is a <a href="https://example.com">Example</a> link.`
const nodes = parseMarkdownToStructure(markdown, md)
expect(nodes.length).toBeGreaterThan(0)
const para = nodes[0]
expect(para.type).toBe('paragraph')
const linkChild = (para as any).children.find((c: any) => c.type === 'link')
expect(linkChild).toBeDefined()
expect(linkChild.href).toBe('https://example.com')
expect(linkChild.children[0].content).toBe('Example')
})
it('preserves custom <a> attributes on link node attrs', () => {
const md = getMarkdown()
const markdown = `<a href="https://example.com" target="_self" rel="nofollow" data-track="cta">Example</a>`
const nodes = parseMarkdownToStructure(markdown, md)
const para = nodes[0] as any
const linkChild = para.children.find((c: any) => c.type === 'link')
expect(linkChild).toBeDefined()
expect(linkChild.href).toBe('https://example.com')
expect(linkChild.attrs).toEqual(expect.arrayContaining([
['href', 'https://example.com'],
['target', '_self'],
['rel', 'nofollow'],
['data-track', 'cta'],
]))
})
it('creates html_inline nodes for generic inline html like <span>', () => {
const md = getMarkdown()
const markdown = `Before <span>inner span</span> After`
const nodes = parseMarkdownToStructure(markdown, md)
const para = nodes[0]
const spanNode = (para as any).children.find((c: any) => c.type === 'html_inline')
expect(spanNode).toBeDefined()
expect(spanNode.content).toContain('<span')
expect(spanNode.content).toContain('</span>')
expect(spanNode.children?.[0]?.content).toBe('inner span')
})
it('handles attribute values that include ">" characters', () => {
const md = getMarkdown()
const markdown = `Value <a href="https://example.com?q=a>b&x=1">Here</a> end`
const nodes = parseMarkdownToStructure(markdown, md)
const para = nodes[0]
const linkChild = (para as any).children.find((c: any) => c.type === 'link')
expect(linkChild).toBeDefined()
// href extraction should capture full quoted value including '>'
expect(linkChild.href).toBe('https://example.com?q=a>b&x=1')
})
it('handles missing closing tag gracefully (unclosed <a>)', () => {
const md = getMarkdown()
const markdown = `Start <a href="https://example.com">Unclosed text`
const nodes = parseMarkdownToStructure(markdown, md)
// Should not throw; ensure raw input is preserved in some node
const serialized = JSON.stringify(nodes)
// JSON will escape quotes, so check for a less strict substring and for href URL
expect(serialized).toContain('<a href')
expect(serialized).toContain('https://example.com')
expect(serialized).toContain('Unclosed text')
})
})
@@ -0,0 +1,127 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('parser inline code multiple spans', () => {
it('parses multiple inline code spans and emphasis correctly', () => {
const md = getMarkdown()
const markdown = '1. **整合冗余条款:** 将原`1-(5)`、`3-(3)`、`3-(4)`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入`1-(4)`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
const nodes = parseMarkdownToStructure(markdown, md)
const serialized = JSON.stringify(nodes)
// Ensure we produced a strong node
expect(serialized).toContain('"type":"strong"')
// Also ensure code spans exist and expected snippets appear
expect(serialized).toContain('1-(5)')
expect(serialized).toContain('3-(3)')
expect(serialized).toContain('3-(4)')
expect(serialized).toContain('1-(4)')
})
it('handles mid-state with trailing opening backtick in list item', () => {
const md = getMarkdown()
// mid-state: trailing opening backtick after a comma
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`'
const nodesMid = parseMarkdownToStructure(markdownMid, md)
const serializedMid = JSON.stringify(nodesMid)
// debug: dump nodes for diagnostic
console.log('DEBUG_NODES_MID_FULL:', JSON.stringify(nodesMid, null, 2))
// debug output removed
// Should contain a strong node for the leading bold and an inline_code mid-state
expect(serializedMid).toContain('"type":"strong"')
// mid-state inline code: there may be an inline_code node with empty code or raw ending in backtick
expect(serializedMid.includes('inline_code') || serializedMid.includes('`')).toBeTruthy()
})
it('handles mid-state with trailing opening backtick in list item mid-state 1', () => {
const md = getMarkdown()
// mid-state: trailing opening backtick after a comma
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`3'
const nodesMid = parseMarkdownToStructure(markdownMid, md)
const children = nodesMid[0].items[0].children[0].children
// Should contain a strong node for the leading bold and an inline_code mid-state
expect(children[0].type).toBe('strong')
expect(children[0].raw).toBe('**整合冗余条款:**')
expect(children[1].type).toBe('text')
expect(children[1].content).toBe(' 将原')
expect(children[2].type).toBe('inline_code')
expect(children[2].code).toBe('1-(5)')
// The punctuation between code spans may be parsed as text or as a tiny
// inline_code node in mid-state scenarios depending on token boundaries.
// Accept either a text node or an inline_code node for the punctuation
expect(['text', 'inline_code']).toContain(children[3]?.type)
// Ensure at least one child contains the punctuation '、' either as content or as inline code
const hasPunctuation = children.some((c: any) => (c.content === '、') || (c.code === '、'))
expect(hasPunctuation).toBeTruthy()
})
it('handles mid-state with trailing opening backtick in list item mid-state 2', () => {
const md = getMarkdown()
// mid-state: trailing opening backtick after a comma
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`3-(3)`、`3-(4)`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入`1-'
const nodesMid = parseMarkdownToStructure(markdownMid, md)
const children = nodesMid[0].items[0].children[0].children
// Should contain a strong node for the leading bold and an inline_code mid-state
expect(children[0].type).toBe('strong')
expect(children[0].raw).toBe('**整合冗余条款:**')
expect(children[1].type).toBe('text')
expect(children[1].content).toBe(' 将原')
expect(children[2].type).toBe('inline_code')
expect(children[2].code).toBe('1-(5)')
expect(children[3].type).toBe('text')
expect(children[3].content).toBe('、')
expect(children[4].type).toBe('inline_code')
expect(children[4].code).toBe('3-(3)')
expect(children[5].type).toBe('text')
expect(children[5].content).toBe('、')
expect(children[6].type).toBe('inline_code')
expect(children[6].code).toBe('3-(4)')
expect(children[7].type).toBe('text')
expect(children[7].content).toBe('中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入')
expect(children[8].type).toBe('inline_code')
expect(children[8].code).toBe('1-')
})
it('handles mid-state with trailing opening backtick in list item mid-state 3', () => {
const md = getMarkdown()
// mid-state: trailing opening backtick after a comma
const markdownMid = '1. **整合冗余条款:** 将原\`1-(5)\`、\`3-(3)\`、\`3-(4)\`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入\`1-(4)\`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
const nodesMid = parseMarkdownToStructure(markdownMid, md)
const children = nodesMid[0].items[0].children[0].children
// Should contain a strong node for the leading bold and an inline_code mid-state
expect(children[0].type).toBe('strong')
expect(children[0].raw).toBe('**整合冗余条款:**')
expect(children[1].type).toBe('text')
expect(children[1].content).toBe(' 将原')
expect(children[2].type).toBe('inline_code')
expect(children[2].code).toBe('1-(5)')
expect(children[3].type).toBe('text')
expect(children[3].content).toBe('、')
expect(children[4].type).toBe('inline_code')
expect(children[4].code).toBe('3-(3)')
expect(children[5].type).toBe('text')
expect(children[5].content).toBe('、')
expect(children[6].type).toBe('inline_code')
expect(children[6].code).toBe('3-(4)')
expect(children[7].type).toBe('text')
expect(children[7].content).toBe('中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入')
expect(children[8].type).toBe('inline_code')
expect(children[8].code).toBe('1-(4)')
expect(children[9].type).toBe('text')
expect(children[9].content).toBe('“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。')
})
it('handles mid-state with trailing opening backtick in list item mid-state 4', () => {
const md = getMarkdown()
// mid-state: trailing opening backtick after a comma
const markdownMid = '1. **整合冗余条款:** 将原 **hello world:** \`1-(5)\`、\`3-(3)\`、\`3-(4)\`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入\`1-(4)\`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
const nodesMid = parseMarkdownToStructure(markdownMid, md)
const children = nodesMid[0].items[0].children[0].children
// Should contain a strong node for the leading bold and an inline_code mid-state
const types = children.map((c: any) => c.type)
expect(types).toEqual(['strong', 'text', 'strong', 'text', 'inline_code', 'text', 'inline_code', 'text', 'inline_code', 'text', 'inline_code', 'text'])
})
})
@@ -0,0 +1,238 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function parseParagraphChildren(markdown: string) {
const md = getMarkdown('issue-334')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
const paragraph = nodes[0]
expect(paragraph?.type).toBe('paragraph')
return paragraph?.children ?? []
}
function parseCustomComponentChildren(markdownContent: string) {
const tags = ['thinking']
const md = getMarkdown('issue-334-custom', { customHtmlTags: tags })
const nodes = parseMarkdownToStructure(`<thinking>${markdownContent}</thinking>`, md, {
customHtmlTags: tags,
final: true,
}) as any[]
const paragraph = nodes[0]
expect(paragraph?.type).toBe('paragraph')
const thinking = (paragraph?.children ?? []).find((node: any) => node?.type === 'thinking')
expect(thinking).toBeDefined()
return thinking?.children ?? []
}
function collectTypes(children: any[]) {
return children.map((node: any) => node?.type)
}
function collectByType(value: any, targetType: string): any[] {
const out: any[] = []
const walk = (node: any) => {
if (!node)
return
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node?.type === targetType)
out.push(node)
if (Array.isArray(node?.children))
node.children.forEach(walk)
if (Array.isArray(node?.items))
node.items.forEach(walk)
}
walk(value)
return out
}
describe('issue #334 - inline formatting before math', () => {
it('keeps strong, emphasis, and strikethrough nodes when inline math is on the same line', () => {
const children = parseParagraphChildren('**粗体** 和 *斜体* 和 ~~删除线~~ 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'strong',
'text',
'emphasis',
'text',
'strikethrough',
'text',
'math_inline',
])
expect(children[0]?.children?.[0]?.content).toBe('粗体')
expect(children[2]?.children?.[0]?.content).toBe('斜体')
expect(children[4]?.children?.[0]?.content).toBe('删除线')
expect(children[5]?.content).toBe(' 和 ')
expect(children[6]?.content).toBe('E = mc^2')
})
it('keeps underscore-based strong and emphasis nodes when inline math is on the same line', () => {
const children = parseParagraphChildren('__粗体__ 和 _斜体_ 和 ~~删除线~~ 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'strong',
'text',
'emphasis',
'text',
'strikethrough',
'text',
'math_inline',
])
expect(children[0]?.children?.[0]?.content).toBe('粗体')
expect(children[2]?.children?.[0]?.content).toBe('斜体')
expect(children[4]?.children?.[0]?.content).toBe('删除线')
expect(children[6]?.content).toBe('E = mc^2')
})
it('keeps nested triple-asterisk formatting before inline math', () => {
const children = parseParagraphChildren('***粗斜体*** 和 ~~删除线~~ 和 $E = mc^2$')
const outer = children[0]
const inner = outer?.children?.[0]
expect(['strong', 'emphasis']).toContain(outer?.type)
expect(['strong', 'emphasis']).toContain(inner?.type)
expect(inner?.type).not.toBe(outer?.type)
expect(inner?.children?.[0]?.content).toBe('粗斜体')
expect(children.slice(1).map((node: any) => node?.type)).toEqual([
'text',
'strikethrough',
'text',
'math_inline',
])
expect(children[2]?.children?.[0]?.content).toBe('删除线')
expect(children[4]?.content).toBe('E = mc^2')
})
it('keeps formatting order when inline math appears before later emphasis markers', () => {
const children = parseParagraphChildren('$E = mc^2$ 和 **粗体** 和 *斜体* 和 ~~删除线~~')
expect(collectTypes(children)).toEqual([
'math_inline',
'text',
'strong',
'text',
'emphasis',
'text',
'strikethrough',
])
expect(children[0]?.content).toBe('E = mc^2')
expect(children[2]?.children?.[0]?.content).toBe('粗体')
expect(children[4]?.children?.[0]?.content).toBe('斜体')
expect(children[6]?.children?.[0]?.content).toBe('删除线')
})
it('does not turn escaped delimiters into formatting nodes before inline math', () => {
const children = parseParagraphChildren('\\*\\*字面量\\*\\* 和 \\_\\_下划线\\_\\_ 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'text',
'math_inline',
])
expect(children[0]?.content).toBe('**字面量** 和 __下划线__ 和 ')
expect(children[1]?.content).toBe('E = mc^2')
})
it('keeps escaped delimiters literal while still parsing later real formatting before inline math', () => {
const children = parseParagraphChildren('\\*\\*字面量\\*\\* 和 **真实粗体** 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'text',
'strong',
'text',
'math_inline',
])
expect(children[0]?.content).toBe('**字面量** 和 ')
expect(children[1]?.children?.[0]?.content).toBe('真实粗体')
expect(children[2]?.content).toBe(' 和 ')
expect(children[3]?.content).toBe('E = mc^2')
})
it('keeps math recoverable even when malformed nested formatting appears before it', () => {
const children = parseParagraphChildren('**粗体 *未闭合 和 ~~删除线~~ 和 $E = mc^2$')
const mathNodes = collectByType(children, 'math_inline')
expect(mathNodes).toHaveLength(1)
expect(mathNodes[0]?.content).toBe('E = mc^2')
const serialized = JSON.stringify(children)
expect(serialized).toContain('未闭合')
expect(serialized).toContain('删除线')
})
})
describe('issue #334 - inline formatting inside custom components', () => {
it('keeps strong, emphasis, strikethrough, and math inside custom component content', () => {
const children = parseCustomComponentChildren('**粗体** 和 *斜体* 和 ~~删除线~~ 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'strong',
'text',
'emphasis',
'text',
'strikethrough',
'text',
'math_inline',
])
expect(children[0]?.children?.[0]?.content).toBe('粗体')
expect(children[2]?.children?.[0]?.content).toBe('斜体')
expect(children[4]?.children?.[0]?.content).toBe('删除线')
expect(children[6]?.content).toBe('E = mc^2')
})
it('keeps underscore-based and triple-asterisk formatting inside custom component content', () => {
const underscoreChildren = parseCustomComponentChildren('__粗体__ 和 _斜体_ 和 ~~删除线~~ 和 $E = mc^2$')
expect(collectTypes(underscoreChildren)).toEqual([
'strong',
'text',
'emphasis',
'text',
'strikethrough',
'text',
'math_inline',
])
const tripleChildren = parseCustomComponentChildren('***粗斜体*** 和 ~~删除线~~ 和 $E = mc^2$')
const outer = tripleChildren[0]
const inner = outer?.children?.[0]
expect(['strong', 'emphasis']).toContain(outer?.type)
expect(['strong', 'emphasis']).toContain(inner?.type)
expect(inner?.type).not.toBe(outer?.type)
expect(inner?.children?.[0]?.content).toBe('粗斜体')
expect(collectTypes(tripleChildren.slice(1))).toEqual([
'text',
'strikethrough',
'text',
'math_inline',
])
})
it('keeps escaped delimiters literal while still parsing later real formatting inside custom component content', () => {
const children = parseCustomComponentChildren('\\*\\*字面量\\*\\* 和 **真实粗体** 和 $E = mc^2$')
expect(collectTypes(children)).toEqual([
'text',
'strong',
'text',
'math_inline',
])
expect(children[0]?.content).toBe('**字面量** 和 ')
expect(children[1]?.children?.[0]?.content).toBe('真实粗体')
expect(children[3]?.content).toBe('E = mc^2')
})
it('keeps math recoverable with malformed formatting inside custom component content', () => {
const children = parseCustomComponentChildren('**粗体 *未闭合 和 ~~删除线~~ 和 $E = mc^2$')
const mathNodes = collectByType(children, 'math_inline')
expect(mathNodes).toHaveLength(1)
expect(mathNodes[0]?.content).toBe('E = mc^2')
const serialized = JSON.stringify(children)
expect(serialized).toContain('未闭合')
expect(serialized).toContain('删除线')
})
})
@@ -0,0 +1,163 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('issue 380 html wrapper markdown regression', () => {
it('keeps markdown nested in a span wrapper as structured children without leaking duplicate top-level nodes', () => {
const markdown = `<span style="font-size: 12px;">
🗺️【环境状态】
- 地点:石溪村,李东的茅屋
- 时间:4/12 周四 上午07:00
- 环境:阳光明媚,空气清新,茅屋内简陋但整洁,远处山峦笼罩在薄雾中,偶尔传来鸟鸣声
📊【角色状态】
- 生理:健康(轻微饥饿)
- 物品:锄头(自家)、简易弓箭(自家)、5个铜币(自家)、一些干粮(自家)
- 技能:农耕(基础)、陷阱设置(基础)、生存知识(基础)
- 关系:与顾客关系良好(+)
📖【目标与线索】
- 已知线索:无
- 短期目标:无
- 长期目标:无
***
🎯【选项】
1. 去田里劳作,争取多收成些粮食卖钱
2. 上山检查之前设置的陷阱,看有没有捕到猎
3. 直接去清风镇看看情况,虽然路途遥远
4. ……
</span>`
const md = getMarkdown('issue-380-structured')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('span')
expect(nodes[0]?.attrs).toEqual([['style', 'font-size: 12px;']])
expect(nodes[0]?.raw).toBe(markdown)
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual([
'paragraph',
'list',
'paragraph',
'list',
'paragraph',
'list',
'thematic_break',
'paragraph',
'list',
])
expect(nodes[0]?.children?.[0]?.children?.[0]?.content).toBe('🗺️【环境状态】')
expect(nodes[0]?.children?.[1]?.items).toHaveLength(3)
expect(nodes[0]?.children?.[8]?.ordered).toBe(true)
expect(nodes[0]?.children?.[8]?.items).toHaveLength(4)
})
it('structures simple block markdown inside span and removes leaked list duplicates', () => {
const markdown = `<span>
- a
- b
</span>`
const md = getMarkdown('issue-380-simple')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('span')
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
expect(nodes[0]?.children?.[0]?.items).toHaveLength(2)
expect(nodes[0]?.children?.[0]?.items?.[0]?.children?.[0]?.children?.[0]?.content).toBe('a')
expect(nodes[0]?.children?.[0]?.items?.[1]?.children?.[0]?.children?.[0]?.content).toBe('b')
})
it('keeps pure html wrappers unstructured when they do not actually contain markdown blocks', () => {
const markdown = '<div><strong>Block HTML</strong></div>'
const md = getMarkdown('issue-380-pure-html')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('div')
expect(nodes[0]?.children).toBeUndefined()
})
it('does not structure blocked tags even when their inner content looks like markdown blocks', () => {
const markdown = `<script>
- a
- b
</script>`
const md = getMarkdown('issue-380-blocked-tag')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('script')
expect(nodes[0]?.children).toBeUndefined()
expect(nodes[0]?.raw).toBe(markdown)
})
it('keeps literal-content tags unstructured even when their inner content looks like markdown blocks', () => {
const samples = ['pre', 'style', 'textarea'] as const
for (const tag of samples) {
const markdown = `<${tag}>
- a
- b
</${tag}>`
const md = getMarkdown(`issue-380-literal-${tag}`)
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe(tag)
expect(nodes[0]?.children).toBeUndefined()
expect(nodes[0]?.raw).toBe(markdown)
}
})
it('structures nested html wrappers when the inner wrapper already contains markdown blocks', () => {
const markdown = `<div>
<div>
- a
</div>
</div>`
const md = getMarkdown('issue-380-nested')
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('div')
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['html_block'])
expect(nodes[0]?.children?.[0]?.tag).toBe('div')
expect(nodes[0]?.children?.[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
expect(nodes[0]?.children?.[0]?.children?.[0]?.items).toHaveLength(1)
expect(nodes[0]?.children?.[0]?.children?.[0]?.items?.[0]?.children?.[0]?.children?.[0]?.content).toBe('a')
})
it('keeps streaming span wrappers stable before the closing tag arrives', () => {
const markdown = `<span>
- a
- b`
const md = getMarkdown('issue-380-streaming')
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
expect(nodes).toHaveLength(1)
expect(nodes[0]?.type).toBe('html_block')
expect(nodes[0]?.tag).toBe('span')
expect(nodes[0]?.loading).toBe(true)
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
expect(nodes[0]?.children?.[0]?.items).toHaveLength(2)
})
})
@@ -0,0 +1,136 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
const REAL_WORLD_MULTILINE_INPUT = `$2.897771955 times 10^{-3}text{m·K}$^[1]^
测试<sup>[3]</sup>。
$x$^[1]^
$x$ ^[1]^
测试^[1]^
$2.897771955 \\times 10^{-3}\\text{m·K}$^[1]^
<sup>[1]</sup>
测试<sup>[12]</sup>结束
A<sup>[3]</sup>B
$x$^[1]^
测试^[1]^
<sup>[3]</sup>
测试<sup>[12]</sup>结束`
function parse(markdown: string, final = true) {
const md = getMarkdown('issue-386')
return parseMarkdownToStructure(markdown, md, { final }) as any[]
}
function collectByType(value: any, targetType: string): any[] {
const out: any[] = []
const walk = (node: any) => {
if (!node)
return
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node.type === targetType)
out.push(node)
if (Array.isArray(node.children))
node.children.forEach(walk)
if (Array.isArray(node.items))
node.items.forEach(walk)
}
walk(value)
return out
}
describe('issue #386 inline math suffix regressions', () => {
it.each([true, false])('keeps superscript syntax immediately after inline math when final=%s', (final) => {
const nodes = parse('$2.897771955 times 10^{-3}text{m·K}$^[1]^', final)
const paragraph = nodes[0] as any
expect(paragraph?.type).toBe('paragraph')
expect(paragraph?.children?.map((child: any) => child.type)).toEqual([
'math_inline',
'superscript',
])
expect(paragraph?.children?.[1]?.children?.map((child: any) => child.content).join('')).toBe('[1]')
const textNodes = collectByType(paragraph, 'text')
expect(textNodes.some((node: any) => node.content === '^[1]^')).toBe(false)
})
it.each([true, false])('keeps footnote references immediately after inline math when final=%s', (final) => {
const nodes = parse('$x$[^1]\n\n[^1]: note', final)
const paragraph = nodes[0] as any
expect(paragraph?.type).toBe('paragraph')
expect(paragraph?.children?.map((child: any) => child.type)).toEqual([
'math_inline',
'footnote_reference',
])
expect(paragraph?.children?.[1]?.id).toBe('1')
const footnotes = collectByType(nodes, 'footnote')
expect(footnotes).toHaveLength(1)
expect(footnotes[0]?.id).toBe('1')
expect(collectByType(footnotes[0], 'text').some((node: any) => node.content === 'note')).toBe(true)
})
it.each([true, false])('preserves brackets inside sup html content when final=%s', (final) => {
const nodes = parse('测试<sup>[3]</sup>。', final)
const paragraph = nodes[0] as any
const htmlInline = collectByType(paragraph, 'html_inline')[0] as any
expect(htmlInline).toBeDefined()
expect(htmlInline.content).toBe('<sup>[3]</sup>')
expect(htmlInline.children).toEqual([
{
type: 'text',
content: '[3]',
raw: '[3]',
},
])
})
it.each([true, false])('preserves brackets inside generic inline html content when final=%s', (final) => {
const nodes = parse('<span>[3]</span>', final)
const paragraph = nodes[0] as any
const htmlInline = collectByType(paragraph, 'html_inline')[0] as any
expect(htmlInline).toBeDefined()
expect(htmlInline.content).toBe('<span>[3]</span>')
expect(htmlInline.children).toEqual([
{
type: 'text',
content: '[3]',
raw: '[3]',
},
])
})
it.each([true, false])('parses the real multiline issue-386 input without leaking raw superscript syntax when final=%s', (final) => {
const nodes = parse(REAL_WORLD_MULTILINE_INPUT, final)
const paragraph = nodes[0] as any
const superscripts = collectByType(paragraph, 'superscript')
const htmlInlineNodes = collectByType(paragraph, 'html_inline')
const textNodes = collectByType(paragraph, 'text')
expect(paragraph?.type).toBe('paragraph')
expect(superscripts).toHaveLength(7)
expect(superscripts.map((node: any) => node.children?.map((child: any) => child.content).join(''))).toEqual([
'[1]',
'[1]',
'[1]',
'[1]',
'[1]',
'[1]',
'[1]',
])
expect(textNodes.some((node: any) => String(node.content).includes('^[1]^'))).toBe(false)
expect(htmlInlineNodes.map((node: any) => node.children?.map((child: any) => child.content).join(''))).toEqual([
'[3]',
'[1]',
'[12]',
'[3]',
'[3]',
'[12]',
])
})
})
@@ -0,0 +1,37 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('link attrs parsing', () => {
it('preserves custom attrs injected on link_open tokens', () => {
const md = getMarkdown('link-attrs')
const nodes = parseMarkdownToStructure('[Example](https://example.com)', md, {
preTransformTokens(tokens: any[]) {
for (const token of tokens) {
if (token?.type !== 'inline' || !Array.isArray(token.children))
continue
for (const child of token.children) {
if (child?.type !== 'link_open')
continue
if (typeof child.attrSet === 'function') {
child.attrSet('target', '_self')
child.attrSet('data-track', 'cta')
}
else {
child.attrs = [...(child.attrs ?? []), ['target', '_self'], ['data-track', 'cta']]
}
}
}
return tokens
},
})
const para = nodes[0] as any
const link = para.children.find((c: any) => c.type === 'link')
expect(link).toBeDefined()
expect(link.attrs).toEqual(expect.arrayContaining([
['href', 'https://example.com'],
['target', '_self'],
['data-track', 'cta'],
]))
})
})
@@ -0,0 +1,258 @@
import { describe, expect, it } from 'vitest'
import { textIncludes } from '../../../test/utils/midstate-utils'
import { getMarkdown, parseMarkdownToStructure } from '../src'
const md = getMarkdown('test')
function collectByType(nodes: any[], type: string) {
const out: any[] = []
const walk = (n: any) => {
if (!n)
return
if (n.type === type)
out.push(n)
if (Array.isArray(n.children))
n.children.forEach(walk)
if (Array.isArray(n.items))
n.items.forEach(walk)
}
nodes.forEach(walk)
return out
}
describe('special link cases', () => {
it('parses link with numeric brackets in link text: [[8]](url)', () => {
const markdown = '建议采用以下接口方案:[[8]](https://markstream-vue-docs.simonhe.me/)'
const nodes = parseMarkdownToStructure(markdown, md)
// Collect all link nodes
const links: any[] = []
const walk = (n: any) => {
if (!n)
return
if (n.type === 'link')
links.push(n)
if (Array.isArray(n.children))
n.children.forEach(walk)
if (Array.isArray(n.items))
n.items.forEach(walk)
}
nodes.forEach(walk)
// Should have exactly one link
expect(links.length).toBe(1)
const link = links[0]
// Verify link properties
expect(link.href).toBe('https://markstream-vue-docs.simonhe.me/')
expect(link.loading).toBe(false)
// Verify link text contains [8]
// The link text should be "[8]" (the brackets are part of the link text)
const linkText = link.text || ''
const childText = (link.children || [])
.map((c: any) => (c.content ?? c.text ?? ''))
.join('')
const fullText = linkText || childText
expect(fullText).toBe('[8]')
// Verify the paragraph contains both the prefix text and the link
const para = nodes.find((n: any) => n.type === 'paragraph') as any
expect(para).toBeTruthy()
expect(textIncludes(para, '建议采用以下接口方案:')).toBe(true)
expect(para.children.some((c: any) => c.type === 'link')).toBe(true)
})
it('parses link with emphasis markers in link text: [DR **(Danmarks Radio)](url)', () => {
const markdown = '[DR **(Danmarks Radio)](https://www.dr.dk/nyheder)'
const nodes = parseMarkdownToStructure(markdown, md)
// Collect all link nodes
const links: any[] = []
const walk = (n: any) => {
if (!n)
return
if (n.type === 'link')
links.push(n)
if (Array.isArray(n.children))
n.children.forEach(walk)
if (Array.isArray(n.items))
n.items.forEach(walk)
}
nodes.forEach(walk)
// Should have exactly one link
expect(links.length).toBe(1)
const link = links[0]
// Verify link properties
expect(link.href).toBe('https://www.dr.dk/nyheder')
expect(link.loading).toBe(false)
// Should contain "DR" and "Danmarks Radio"
expect(textIncludes(link, 'DR')).toBe(true)
expect(textIncludes(link, 'Danmarks Radio')).toBe(true)
expect(textIncludes(link, '(Danmarks Radio)')).toBe(true)
// Verify the paragraph contains the link
const para = nodes.find((n: any) => n.type === 'paragraph') as any
expect(para).toBeTruthy()
expect(para.children.some((c: any) => c.type === 'link')).toBe(true)
})
it('parses multiple links with special characters in same paragraph', () => {
const markdown = '建议采用以下接口方案:[[8]](https://markstream-vue-docs.simonhe.me/)\n\n[DR **(Danmarks Radio)](https://www.dr.dk/nyheder)'
const nodes = parseMarkdownToStructure(markdown, md)
// Collect all link nodes
const links: any[] = []
const walk = (n: any) => {
if (!n)
return
if (n.type === 'link')
links.push(n)
if (Array.isArray(n.children))
n.children.forEach(walk)
if (Array.isArray(n.items))
n.items.forEach(walk)
}
nodes.forEach(walk)
// Should have exactly two links
expect(links.length).toBe(2)
// First link: [[8]](url)
const firstLink = links.find(l => l.href === 'https://markstream-vue-docs.simonhe.me/')
expect(firstLink).toBeDefined()
expect(firstLink?.loading).toBe(false)
const firstLinkText = firstLink?.text || (firstLink?.children || []).map((c: any) => c.content || '').join('')
expect(firstLinkText).toBe('[8]')
// Second link: [DR **(Danmarks Radio)](url)
const secondLink = links.find(l => l.href === 'https://www.dr.dk/nyheder')
expect(secondLink).toBeDefined()
expect(secondLink?.loading).toBe(false)
expect(textIncludes(secondLink, 'DR')).toBe(true)
expect(textIncludes(secondLink, 'Danmarks Radio')).toBe(true)
})
it('parses standalone numeric bracket link without prefix text', () => {
const markdown = '[[8]](https://example.com)'
const nodes = parseMarkdownToStructure(markdown, md)
const links: any[] = []
const walk = (n: any) => {
if (!n)
return
if (n.type === 'link')
links.push(n)
if (Array.isArray(n.children))
n.children.forEach(walk)
}
nodes.forEach(walk)
expect(links.length).toBe(1)
expect(links[0].href).toBe('https://example.com')
expect(links[0].text || links[0].children?.map((c: any) => c.content || '').join('')).toBe('[8]')
})
it('distinguishes between reference [8] and link [[8]](url)', () => {
// Standalone [8] should be a reference
const referenceMarkdown = '请参考 [8] 了解更多信息'
const referenceNodes = parseMarkdownToStructure(referenceMarkdown, md)
const references: any[] = []
const walkRef = (n: any) => {
if (!n)
return
if (n.type === 'reference')
references.push(n)
if (Array.isArray(n.children))
n.children.forEach(walkRef)
}
referenceNodes.forEach(walkRef)
// [8] as standalone should be parsed as reference (if reference rule matches)
// But if it's followed by ](, it should be a link
const linkMarkdown = '请参考 [[8]](https://example.com) 了解更多信息'
const linkNodes = parseMarkdownToStructure(linkMarkdown, md)
const links: any[] = []
const walkLink = (n: any) => {
if (!n)
return
if (n.type === 'link')
links.push(n)
if (Array.isArray(n.children))
n.children.forEach(walkLink)
}
linkNodes.forEach(walkLink)
// The link version should have a link node
expect(links.length).toBeGreaterThan(0)
const link = links.find(l => l.href === 'https://example.com')
expect(link).toBeDefined()
expect(link?.text || link?.children?.map((c: any) => c.content || '').join('')).toBe('[8]')
})
it('preserves normal reference-style links when the label is not numeric', () => {
const markdown = `请看 [1][source]
[source]: https://example.com`
const nodes = parseMarkdownToStructure(markdown, md)
const links = collectByType(nodes, 'link')
const references = collectByType(nodes, 'reference')
expect(links).toHaveLength(1)
expect(links[0].href).toBe('https://example.com')
expect(links[0].text || links[0].children?.map((c: any) => c.content || '').join('')).toBe('1')
expect(references).toHaveLength(0)
})
it('parses numeric references when they are adjacent to surrounding text', () => {
const cases = [
'随便输入一些文字[1]',
'随便输入一些文字[1]后面还有文字',
'随便输入一些文字[1] ',
'随便输入一些文字[1] (空格+文字)',
]
for (const markdown of cases) {
const nodes = parseMarkdownToStructure(markdown, md)
const references = collectByType(nodes, 'reference')
expect(references, markdown).toHaveLength(1)
expect(references[0].id).toBe('1')
expect(textIncludes(nodes, '随便输入一些文字')).toBe(true)
}
})
it('parses a numeric reference immediately followed by a normal inline link', () => {
const cases = [
'随便输入一些文字 [1][百度](www.baidu.com)',
'随便输入一些文字 [1] [百度](www.baidu.com)',
]
for (const markdown of cases) {
const nodes = parseMarkdownToStructure(markdown, md)
const references = collectByType(nodes, 'reference')
const links = collectByType(nodes, 'link')
expect(references, markdown).toHaveLength(1)
expect(references[0].id).toBe('1')
expect(links, markdown).toHaveLength(1)
expect(links[0].href).toBe('www.baidu.com')
expect(textIncludes(links[0], '百度')).toBe(true)
}
})
it('parses chained numeric references without requiring spaces', () => {
const nodes = parseMarkdownToStructure('随便输入一些文字[1][2][3]', md)
const references = collectByType(nodes, 'reference')
expect(references).toHaveLength(3)
expect(references.map(ref => ref.id)).toEqual(['1', '2', '3'])
expect(textIncludes(nodes, '随便输入一些文字')).toBe(true)
})
})
@@ -0,0 +1,135 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function collectByType(nodes: any, type: string, out: any[] = []) {
if (!nodes)
return out
if (Array.isArray(nodes)) {
nodes.forEach(node => collectByType(node, type, out))
return out
}
if (nodes.type === type)
out.push(nodes)
if (Array.isArray(nodes.children))
nodes.children.forEach((child: any) => collectByType(child, type, out))
if (Array.isArray(nodes.items))
nodes.items.forEach((item: any) => collectByType(item, type, out))
return out
}
function collectText(nodes: any) {
return collectByType(nodes, 'text')
.map((node: any) => String(node.content ?? ''))
.join('')
}
function expectListStreamingTextToStayVisible(options: {
id: string
full: string
explanation: string
suffix: string
protectedType?: string
}) {
const md = getMarkdown(options.id)
const explanationReadyAt = options.full.indexOf(options.explanation) + options.explanation.length
const suffixStartAt = options.full.indexOf(options.suffix)
for (let i = explanationReadyAt; i <= options.full.length; i++) {
const chunk = options.full.slice(0, i)
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const text = collectText(nodes)
expect(text, `prefix(${i}): ${chunk}`).toContain(options.explanation)
expect(text.match(new RegExp(options.explanation, 'g')) ?? [], `prefix(${i}): ${chunk}`).toHaveLength(1)
if (i > suffixStartAt) {
const expectedSuffix = options.full.slice(suffixStartAt, i)
expect(text, `prefix(${i}): ${chunk}`).toContain(expectedSuffix)
if (options.protectedType) {
const protectedNodes = collectByType(nodes, options.protectedType)
expect(
protectedNodes.some((node: any) => JSON.stringify(node).includes(expectedSuffix)),
`prefix(${i}): ${chunk}`,
).toBe(false)
}
}
}
}
describe('list inline streaming regression', () => {
it('keeps explanatory list text stable while inline code opens, grows, closes, and is followed by more text', () => {
const full = `- **计算工具验证**
通过数学计算工具确认结果:
\`363 ÷ 15,135 × 100 = 2.39841427...\`
后续说明继续输出`
expectListStreamingTextToStayVisible({
id: 'list-inline-code-streaming-regression',
full,
explanation: '通过数学计算工具确认结果:',
suffix: '后续说明继续输出',
protectedType: 'inline_code',
})
})
it('keeps explanatory list text stable while an inline link opens, closes, and is followed by more text', () => {
const full = `- **链接验证**
通过链接工具确认结果:
[查看详情](https://example.com)
后续说明继续输出`
expectListStreamingTextToStayVisible({
id: 'list-inline-link-streaming-regression',
full,
explanation: '通过链接工具确认结果:',
suffix: '后续说明继续输出',
protectedType: 'link',
})
})
it('keeps explanatory list text stable while emphasis opens, closes, and is followed by more text', () => {
const full = `- **强调说明**
通过人工复核确认结果:
*结果待确认*
后续说明继续输出`
expectListStreamingTextToStayVisible({
id: 'list-emphasis-streaming-regression',
full,
explanation: '通过人工复核确认结果:',
suffix: '后续说明继续输出',
protectedType: 'emphasis',
})
})
it('keeps explanatory list text stable while inline math closes and is followed by more text', () => {
const full = `- **数学验证**
通过公式确认结果:
$x^2 + y^2$
后续说明继续输出`
expectListStreamingTextToStayVisible({
id: 'list-math-inline-streaming-regression',
full,
explanation: '通过公式确认结果:',
suffix: '后续说明继续输出',
protectedType: 'math_inline',
})
})
it('keeps explanatory list text stable while a numeric reference appears and is followed by more text', () => {
const full = `- **参考验证**
通过参考编号确认结果:
[1]
后续说明继续输出`
expectListStreamingTextToStayVisible({
id: 'list-reference-streaming-regression',
full,
explanation: '通过参考编号确认结果:',
suffix: '后续说明继续输出',
protectedType: 'reference',
})
})
})
@@ -0,0 +1,26 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function flatten(nodes: any[]): any[] {
const out: any[] = []
for (const n of nodes ?? []) {
out.push(n)
if (Array.isArray(n?.children))
out.push(...flatten(n.children))
}
return out
}
describe('parseMarkdownToStructure - multiple links same line', () => {
it('parses 3 adjacent links as 3 link nodes', () => {
const md = getMarkdown()
const markdown = '[citation](http://url1) [citation](http://url1) [citation](http://url1)'
const nodes = parseMarkdownToStructure(markdown, md)
const all = flatten(nodes)
const links = all.filter(n => n?.type === 'link')
expect(links).toHaveLength(3)
expect(links.map(l => l.href)).toEqual(['http://url1', 'http://url1', 'http://url1'])
})
})
@@ -0,0 +1,112 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function collectByType(nodes: any[], type: string) {
const out: any[] = []
const walk = (node: any) => {
if (!node)
return
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node.type === type)
out.push(node)
if (Array.isArray(node.children))
node.children.forEach(walk)
if (Array.isArray(node.items))
node.items.forEach(walk)
}
walk(nodes)
return out
}
function collectText(nodes: any[]) {
return collectByType(nodes, 'text').map((node: any) => String(node.content ?? '')).join('')
}
describe('numeric reference streaming stability', () => {
it('keeps completed numeric references stable during char-by-char chained references', () => {
const md = getMarkdown('numeric-reference-streaming-chain')
const full = '随便输入一些文字[1][2][3]'
const firstReady = full.indexOf('[1]') + '[1]'.length
const secondOpens = full.indexOf('[2]')
const secondReady = secondOpens + '[2]'.length
const thirdReady = full.indexOf('[3]') + '[3]'.length
for (let i = 1; i <= full.length; i++) {
const chunk = full.slice(0, i)
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
if (i >= firstReady)
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
if (i > secondOpens)
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
if (i >= secondReady)
expect(ids.slice(0, 2), `prefix(${i}): ${chunk}`).toEqual(['1', '2'])
if (i >= thirdReady)
expect(ids.slice(0, 3), `prefix(${i}): ${chunk}`).toEqual(['1', '2', '3'])
}
})
it('keeps the numeric reference once an adjacent inline link becomes distinguishable', () => {
const md = getMarkdown('numeric-reference-streaming-link')
const full = '随便输入一些文字 [1][百度](www.baidu.com)'
const distinguishableAt = full.indexOf('[百度](') + '[百度]('.length
for (let i = 1; i <= full.length; i++) {
const chunk = full.slice(0, i)
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
const links = collectByType(nodes, 'link')
if (i >= distinguishableAt) {
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
expect(collectText(nodes), `prefix(${i}): ${chunk}`).not.toContain('[1]')
expect(links, `prefix(${i}): ${chunk}`).toHaveLength(1)
expect(String(links[0]?.text ?? ''), `prefix(${i}): ${chunk}`).toContain('百度')
expect(Boolean(links[0]?.loading), `prefix(${i}): ${chunk}`).toBe(i < full.length)
}
}
const finalNodes = parseMarkdownToStructure(full, md, { final: true })
const finalIds = collectByType(finalNodes, 'reference').map((node: any) => String(node.id ?? ''))
const links = collectByType(finalNodes, 'link')
expect(finalIds).toEqual(['1'])
expect(links).toHaveLength(1)
expect(String(links[0]?.href ?? '')).toBe('www.baidu.com')
})
it('keeps the completed numeric reference while trailing text streams in', () => {
const md = getMarkdown('numeric-reference-streaming-suffix')
const full = '随便输入一些文字[1]后面还有文字'
const readyAt = full.indexOf('[1]') + '[1]'.length
for (let i = readyAt; i <= full.length; i++) {
const chunk = full.slice(0, i)
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
}
})
it('still respects validateLink when the trailing markdown link is being recovered', () => {
const md = getMarkdown('numeric-reference-streaming-validate')
md.set?.({ validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()) })
const chunk = '随便输入一些文字 [1][百度](javascript:alert(1'
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
const links = collectByType(nodes, 'link')
expect(ids).toEqual(['1'])
expect(links).toHaveLength(1)
expect(String(links[0]?.text ?? '')).toBe('百度')
expect(String(links[0]?.href ?? '')).not.toMatch(/javascript:/i)
expect(Boolean(links[0]?.loading)).toBe(true)
expect(collectText(nodes)).toContain('百度')
})
})
@@ -0,0 +1,65 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function walk(node: any, visit: (n: any) => void) {
if (!node)
return
if (Array.isArray(node)) {
for (const n of node)
walk(n, visit)
return
}
visit(node)
if (Array.isArray(node.children))
walk(node.children, visit)
if (Array.isArray(node.items))
walk(node.items, visit)
}
describe('streaming ordered list - avoid transient gaps', () => {
it('suppresses transient nested lists and leaked newlines while streaming', () => {
const md = getMarkdown()
const full = `法:
1. **递归实现**:直接
2. **记忆化递归(动态规划)**:使
`
const marker1 = full.indexOf('1.')
const marker2 = full.indexOf('\n\n2.')
const cuts = [
marker1 + 2, // "1."
marker1 + 3, // "1. "
marker1 + 4, // "1. *"
marker1 + 5, // "1. **"
full.indexOf('直接') + '直接'.length + 1, // right after item 1
marker2 + 3, // "\n\n2"
marker2 + 4, // "\n\n2."
]
.filter(n => Number.isFinite(n) && n > 0 && n <= full.length)
.filter((n, i, arr) => arr.indexOf(n) === i)
.sort((a, b) => a - b)
for (const cut of cuts) {
const chunk = full.slice(0, cut)
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
const unorderedLists: any[] = []
const newlineTextNodes: string[] = []
const markerParagraphs: string[] = []
walk(nodes, (n) => {
if (n?.type === 'list' && n.ordered === false)
unorderedLists.push(n)
if (n?.type === 'text' && typeof n.content === 'string' && /[\r\n]/.test(n.content))
newlineTextNodes.push(n.content)
if (n?.type === 'paragraph' && typeof n.raw === 'string' && /^\s*\d+[.)]?\s*$/.test(n.raw))
markerParagraphs.push(n.raw)
})
expect(unorderedLists, `unexpected unordered list at cut=${cut}`).toHaveLength(0)
expect(newlineTextNodes, `unexpected newline text nodes at cut=${cut}`).toHaveLength(0)
expect(markerParagraphs, `unexpected marker paragraph at cut=${cut}`).toHaveLength(0)
}
})
})
@@ -0,0 +1,45 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
describe('parseMarkdownToStructure with code in link text', () => {
it('should parse links with inline code correctly', () => {
const md = getMarkdown()
const mdText1 = `the [\`npm init\` command](https://docs.npmjs.com/creating-a-package-json-file).`
const result1 = parseMarkdownToStructure(mdText1, md, { final: true })
// 验证结果的基本结构
expect(result1).toBeDefined()
expect(result1.length).toBe(1)
expect(result1[0].type).toBe('paragraph')
expect(result1[0].children.length).toBe(3)
// 验证链接节点
const linkNode1 = result1[0].children[1]
expect(linkNode1.type).toBe('link')
expect(linkNode1.href).toBe('https://docs.npmjs.com/creating-a-package-json-file')
expect(linkNode1.text).toBe('npm init command')
expect(linkNode1.children.length).toBe(2)
// 验证链接中的内联代码节点
expect(linkNode1.children[0].type).toBe('inline_code')
expect(linkNode1.children[0].code).toBe('npm init')
expect(linkNode1.children[1].type).toBe('text')
expect(linkNode1.children[1].content).toBe(' command')
// 验证第二个链接
const mdText2 = `[\`npm install\` command](https://docs.npmjs.com/getting-started/installing-npm-packages-locally):`
const result2 = parseMarkdownToStructure(mdText2, md, { final: true })
expect(result2.length).toBe(1)
const linkNode2 = result2[0].children[0]
expect(linkNode2.type).toBe('link')
expect(linkNode2.href).toBe('https://docs.npmjs.com/getting-started/installing-npm-packages-locally')
expect(linkNode2.text).toBe('npm install command')
expect(linkNode2.children.length).toBe(2)
expect(linkNode2.children[0].type).toBe('inline_code')
expect(linkNode2.children[0].code).toBe('npm install')
expect(linkNode2.children[1].type).toBe('text')
expect(linkNode2.children[1].content).toBe(' command')
})
})
@@ -0,0 +1,89 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function collectTexts(n: any): string[] {
if (!n)
return []
if (Array.isArray(n))
return n.flatMap(collectTexts)
if (typeof n === 'string')
return [n]
const texts: string[] = []
if (n.type === 'text' && typeof n.content === 'string')
texts.push(n.content)
if (n.children && Array.isArray(n.children))
texts.push(...n.children.flatMap(collectTexts))
if (n.items && Array.isArray(n.items))
texts.push(...n.items.flatMap(collectTexts))
return texts
}
function countOccurrences(nodes: any, needle: string) {
const allTexts = collectTexts(nodes).join('\n')
return (allTexts.match(new RegExp(needle, 'g')) || []).length
}
describe('parseMarkdownToStructure - duplicate question rendering', () => {
it('does not duplicate the question text when parsing mixed math/ce commands', () => {
const md = getMarkdown()
const markdown = `**当堂检测**:
1. 下列物质属于酚的是( )
A. $\ce{CH3CH2OH}$ B. $\ce{C6H5CH2OH}$ C. $\ce{}$ D. $\ce{HO-CH2-CH2OH}$
**答案**:C`
const nodes = parseMarkdownToStructure(markdown, md)
const allTexts = collectTexts(nodes).join('\n')
console.log('All texts:', JSON.stringify(allTexts, null, 2))
// Count occurrences of the core question string
expect(countOccurrences(nodes, '下列物质属于酚的是')).toBe(1)
})
const streamingCases = [
{
name: 'plain text prefix before $$ math',
markdown: `第(1)题
题干解析:本题考查利用导数求含参函数的极值,核心条件为函数 $$ f(x) = \\ln x - m + \\dfrac{m}{x} $$($$ m \\in \\mathbb{R} $$)`,
needle: '第(1)题',
},
{
name: 'plain text prefix before $ math',
markdown: `第(2)题
已知函数 $f(x)=x^2$,求导数。`,
needle: '第(2)题',
},
{
name: 'plain text prefix before \\( \\) math',
markdown: String.raw`第(3)题
若 \(a+b=1\),求 \(ab\) 的值。`,
needle: '第(3)题',
},
{
name: 'strong prefix before $$ math',
markdown: `**第(4)题**
题干解析:函数 $$x^2$$ 的图像。`,
needle: '第(4)题',
},
{
name: 'markdown link prefix before $ math',
markdown: `[第(5)题](https://example.com)
题干解析:函数 $x^2$ 的图像。`,
needle: '第(5)题',
},
{
name: 'inline html prefix before $$ math',
markdown: `<span>第(6)题</span>
题干解析:函数 $$x^2$$ 的图像。`,
needle: '第(6)题',
},
] as const
for (const testCase of streamingCases) {
it(`does not duplicate the hardbreak prefix for ${testCase.name} while streaming`, () => {
const md = getMarkdown()
const nodes = parseMarkdownToStructure(testCase.markdown, md)
expect(countOccurrences(nodes, testCase.needle)).toBe(1)
})
}
})
@@ -0,0 +1,3 @@
// Delegate to the repository-level vitest setup so package tests
// reuse the same global mocks (Worker, ResizeObserver, etc.).
import '../../../../test/setup/vitest.setup'
@@ -0,0 +1,129 @@
import { describe, expect, it } from 'vitest'
import { parseFenceToken } from '../src'
function parseStreamingDiff(content: string, closed = false) {
return parseFenceToken({
type: 'fence',
info: 'diff json:package.json',
content,
map: [0, 1],
meta: { closed },
} as any)
}
describe('parseFenceToken - streaming unified diff stability', () => {
it('splits standard unified diff lines without requiring a space after the marker', () => {
const node = parseFenceToken({
type: 'fence',
info: 'diff typescript',
content: [
'diff --git a/src/render.ts b/src/render.ts',
'index 0000000..1111111 100644',
'--- a/src/render.ts',
'+++ b/src/render.ts',
'@@ -1,7 +1,12 @@',
'-export function render(input) {',
'- return input',
'+export function render(input: string) {',
'+ if (!input.trim())',
'+ return \'empty\'',
'+',
'+ const normalized = input.replace(/\\r\\n/g, \'\\n\')',
'+ return normalized',
' }',
].join('\n'),
map: [0, 1],
meta: { closed: true },
} as any)
expect(node.language).toBe('typescript')
expect(node.originalCode).toBe([
'export function render(input) {',
' return input',
'}',
].join('\n'))
expect(node.updatedCode).toBe([
'export function render(input: string) {',
' if (!input.trim())',
' return \'empty\'',
'',
' const normalized = input.replace(/\\r\\n/g, \'\\n\')',
' return normalized',
'}',
].join('\n'))
})
it('withholds an unfinished replacement hunk until the added side is available', () => {
const node = parseStreamingDiff([
'{',
' "name": "markstream-vue",',
'- "version": "0.0.49",',
].join('\n'))
expect(node.originalCode).toBe([
'{',
' "name": "markstream-vue",',
].join('\n'))
expect(node.updatedCode).toBe([
'{',
' "name": "markstream-vue",',
].join('\n'))
})
it('flushes a finished replacement hunk once the added line becomes stable', () => {
const node = parseStreamingDiff([
'{',
' "name": "markstream-vue",',
'- "version": "0.0.49",',
'+ "version": "0.0.54-beta.1",',
'',
].join('\n'))
expect(node.originalCode).toBe([
'{',
' "name": "markstream-vue",',
' "version": "0.0.49",',
].join('\n'))
expect(node.updatedCode).toBe([
'{',
' "name": "markstream-vue",',
' "version": "0.0.54-beta.1",',
].join('\n'))
})
it('withholds a trailing incomplete added line instead of leaking a mismatched preview', () => {
const node = parseStreamingDiff([
'{',
' "name": "markstream-vue",',
'- "version": "0.0.49",',
'+ "version": "0.0.54-b',
].join('\n'))
expect(node.originalCode).toBe([
'{',
' "name": "markstream-vue",',
].join('\n'))
expect(node.updatedCode).toBe([
'{',
' "name": "markstream-vue",',
].join('\n'))
})
it('flushes the trailing hunk once the fence is closed', () => {
const node = parseStreamingDiff([
'{',
' "name": "markstream-vue",',
'- "version": "0.0.49",',
].join('\n'), true)
expect(node.originalCode).toBe([
'{',
' "name": "markstream-vue",',
' "version": "0.0.49",',
].join('\n'))
expect(node.updatedCode).toBe([
'{',
' "name": "markstream-vue",',
].join('\n'))
})
})
@@ -0,0 +1,64 @@
import { describe, expect, it } from 'vitest'
import { parseInlineTokens } from '../src'
function flattenText(nodes: any[]): string {
const out: string[] = []
const walk = (n: any) => {
if (!n)
return
if (Array.isArray(n)) {
n.forEach(walk)
return
}
if (n.type === 'text' && typeof n.content === 'string')
out.push(n.content)
if (Array.isArray(n.children))
n.children.forEach(walk)
if (Array.isArray(n.items))
n.items.forEach(walk)
}
walk(nodes)
return out.join('')
}
describe('parseInlineTokens strict strong matching', () => {
it('default (non-strict) treats unclosed ** as mid-state strong', () => {
const raw = '[**cxx](xxx)'
const tokens: any[] = [{ type: 'text', content: raw }]
const nodes = parseInlineTokens(tokens, raw)
expect(nodes.length).toBeGreaterThan(0)
// Expect the parser will create a strong node in non-strict mode
const first = nodes[0] as any
expect(first).toBeDefined()
expect(first.type).toBe('strong')
expect(first.children?.[0]?.type).toBe('link')
expect(first.children?.[0]?.text).toContain('cxx')
})
it('strict mode keeps original text when no closing ** is found', () => {
const raw = '[**cxx](xxx)'
const tokens: any[] = [{ type: 'text', content: raw }]
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
expect(nodes.length).toBeGreaterThan(0)
const first = nodes[0] as any
expect(first.type).toBe('link')
expect(first.children[0].content).toBe(raw.match(/\[([^\]]+)\]/)?.[1])
})
it('strict mode does not duplicate prefix text before unmatched **', () => {
const raw = 'foo**bar'
const tokens: any[] = [{ type: 'text', content: raw }]
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
expect(nodes.length).toBeGreaterThan(0)
const first = nodes[0] as any
expect(first.type).toBe('text')
expect(first.content).toBe('foo**bar')
})
it('strict mode does not duplicate prefix when unmatched ** is inside link text', () => {
const raw = '前缀[**cxx](xxx)'
const tokens: any[] = [{ type: 'text', content: raw }]
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
expect(flattenText(nodes as any[])).toBe('前缀**cxx')
})
})
@@ -0,0 +1,378 @@
import { describe, expect, it } from 'vitest'
import { getMarkdown, parseMarkdownToStructure } from '../src'
function expectLinkChild(node: any, expected: {
href: string
text: string
loading: boolean
childType: string
childText: string
}) {
expect(node.type).toBe('link')
expect(node.href).toBe(expected.href)
expect(node.text).toBe(expected.text)
expect(node.loading).toBe(expected.loading)
expect(node.attrs).toEqual(expect.arrayContaining([['href', expected.href]]))
expect(node.children).toHaveLength(1)
expect(node.children[0].type).toBe(expected.childType)
expect(node.children[0].raw).toBe(expected.text)
expect(node.children[0].children).toHaveLength(1)
expect(node.children[0].children[0].type).toBe('text')
expect(node.children[0].children[0].content).toBe(expected.childText)
}
describe('strong link followed by inline content', () => {
const md = getMarkdown('strong-link-followed-by-inline-content')
it('keeps trailing text and the next italic link outside the preceding strong link', () => {
const markdown = `[**加粗的链接**](https://example.com)会导致后面的斜体链接解析不正确
[*斜体的链接*](https://test.com)`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(3)
expectLinkChild(paragraph.children[0], {
href: 'https://example.com',
text: '**加粗的链接**',
loading: false,
childType: 'strong',
childText: '加粗的链接',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: '会导致后面的斜体链接解析不正确\n',
raw: '会导致后面的斜体链接解析不正确\n',
}))
expectLinkChild(paragraph.children[2], {
href: 'https://test.com',
text: '*斜体的链接*',
loading: false,
childType: 'emphasis',
childText: '斜体的链接',
})
})
it('keeps two inline links separated by the plain text between them', () => {
const markdown = `[**加粗的链接**](https://example.com) 或者 [*斜体的链接*](https://test.com)`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(3)
expectLinkChild(paragraph.children[0], {
href: 'https://example.com',
text: '**加粗的链接**',
loading: false,
childType: 'strong',
childText: '加粗的链接',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: ' 或者 ',
raw: ' 或者 ',
}))
expectLinkChild(paragraph.children[2], {
href: 'https://test.com',
text: '*斜体的链接*',
loading: false,
childType: 'emphasis',
childText: '斜体的链接',
})
})
it('keeps trailing punctuation and explanatory text outside the strong link', () => {
const markdown = `[**加粗的链接**](https://example.com),这里不是链接部分,但是有链接效果`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
expectLinkChild(paragraph.children[0], {
href: 'https://example.com',
text: '**加粗的链接**',
loading: false,
childType: 'strong',
childText: '加粗的链接',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: ',这里不是链接部分,但是有链接效果',
raw: ',这里不是链接部分,但是有链接效果',
}))
})
it('keeps immediate trailing text outside a strong link even without whitespace', () => {
const markdown = `[**加粗的链接**](https://example.com)tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
expectLinkChild(paragraph.children[0], {
href: 'https://example.com',
text: '**加粗的链接**',
loading: false,
childType: 'strong',
childText: '加粗的链接',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: 'tail',
raw: 'tail',
}))
})
it('keeps completed strong and italic links stable in non-final parsing mode', () => {
const markdown = `[**加粗的链接**](https://example.com) 或者 [*斜体的链接*](https://test.com)`
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(3)
expectLinkChild(paragraph.children[0], {
href: 'https://example.com',
text: '**加粗的链接**',
loading: false,
childType: 'strong',
childText: '加粗的链接',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: ' 或者 ',
raw: ' 或者 ',
}))
expectLinkChild(paragraph.children[2], {
href: 'https://test.com',
text: '*斜体的链接*',
loading: false,
childType: 'emphasis',
childText: '斜体的链接',
})
})
it('keeps strong links with inline code labels separated from trailing text', () => {
const markdown = `[**用 \`pnpm\` 安装**](https://example.com)tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
const link = paragraph.children[0]
expect(link.type).toBe('link')
expect(link.href).toBe('https://example.com')
expect(link.text).toBe('**用 pnpm 安装**')
expect(link.loading).toBe(false)
expect(link.children).toHaveLength(1)
expect(link.children[0].type).toBe('strong')
expect(link.children[0].children).toHaveLength(3)
expect(link.children[0].children[0]).toEqual(expect.objectContaining({
type: 'text',
content: '用 ',
}))
expect(link.children[0].children[1]).toEqual(expect.objectContaining({
type: 'inline_code',
code: 'pnpm',
}))
expect(link.children[0].children[2]).toEqual(expect.objectContaining({
type: 'text',
content: ' 安装',
}))
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: 'tail',
raw: 'tail',
}))
})
it('preserves title attrs on strong links while keeping trailing text outside the link', () => {
const markdown = `[**x**](https://a.com "title")tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
const link = paragraph.children[0]
expect(link.type).toBe('link')
expect(link.href).toBe('https://a.com')
expect(link.title).toBe('title')
expect(link.text).toBe('**x**')
expect(link.loading).toBe(false)
expect(link.attrs).toEqual(expect.arrayContaining([
['href', 'https://a.com'],
['title', 'title'],
]))
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: 'tail',
raw: 'tail',
}))
})
it('keeps adjacent strong-label and code-label links independent', () => {
const markdown = `[**x**](https://a.com)[\`y\`](https://b.com)`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
expectLinkChild(paragraph.children[0], {
href: 'https://a.com',
text: '**x**',
loading: false,
childType: 'strong',
childText: 'x',
})
const codeLink = paragraph.children[1]
expect(codeLink.type).toBe('link')
expect(codeLink.href).toBe('https://b.com')
expect(codeLink.text).toBe('y')
expect(codeLink.loading).toBe(false)
expect(codeLink.children).toHaveLength(1)
expect(codeLink.children[0]).toEqual(expect.objectContaining({
type: 'inline_code',
code: 'y',
raw: 'y',
}))
})
it('marks an unfinished strong link URL as loading in non-final mode', () => {
const markdown = `[**x**](https://a.com`
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(1)
const strong = paragraph.children[0]
expect(strong.type).toBe('strong')
expect(strong.children).toHaveLength(1)
const link = strong.children[0]
expect(link.type).toBe('link')
expect(link.loading).toBe(true)
expect(link.text).toBe('x')
expect(link.children).toHaveLength(1)
expect(link.children[0]).toEqual(expect.objectContaining({
type: 'text',
content: 'x',
raw: 'x',
}))
})
it('keeps triple-emphasis labels separated from trailing text', () => {
const markdown = `[***x***](https://a.com)tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
const link = paragraph.children[0]
expect(link.type).toBe('link')
expect(link.href).toBe('https://a.com')
expect(link.text).toBe('*x*')
expect(link.loading).toBe(false)
expect(link.children).toHaveLength(1)
expect(link.children[0].type).toBe('emphasis')
expect(link.children[0].children).toHaveLength(1)
expect(link.children[0].children[0].type).toBe('strong')
expect(link.children[0].children[0].children[0]).toEqual(expect.objectContaining({
type: 'text',
content: 'x',
}))
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: 'tail',
raw: 'tail',
}))
})
it('keeps html inline labels separated from trailing text', () => {
const markdown = `[<span>x</span>](https://a.com)tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
const link = paragraph.children[0]
expect(link.type).toBe('link')
expect(link.href).toBe('https://a.com')
expect(link.text).toBe('<span>x</span>')
expect(link.loading).toBe(false)
expect(link.children).toHaveLength(1)
expect(link.children[0]).toEqual(expect.objectContaining({
type: 'html_inline',
tag: 'span',
content: '<span>x</span>',
loading: false,
}))
expect(link.children[0].children).toHaveLength(1)
expect(link.children[0].children[0]).toEqual(expect.objectContaining({
type: 'text',
content: 'x',
}))
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: 'tail',
raw: 'tail',
}))
})
it('keeps punctuation after a strong link outside the link node', () => {
const markdown = `[**x**](https://a.com)!tail`
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
expect(nodes).toHaveLength(1)
const paragraph = nodes[0]
expect(paragraph.type).toBe('paragraph')
expect(paragraph.children).toHaveLength(2)
expectLinkChild(paragraph.children[0], {
href: 'https://a.com',
text: '**x**',
loading: false,
childType: 'strong',
childText: 'x',
})
expect(paragraph.children[1]).toEqual(expect.objectContaining({
type: 'text',
content: '!tail',
raw: '!tail',
}))
})
})

Some files were not shown because too many files have changed in this diff Show More