feat(desktop): embed BoosAPI Desktop client as a subproject
Vendor the rebranded BoosAPI Desktop client (Tauri + Vue) into metapi as a monorepo subdirectory under desktop/. Source only — node_modules, dist, Cargo target and the upstream .git history are excluded. desktop/ keeps its own independent Cargo workspace (desktop/src-tauri with crates/*); it is not part of the backend boosapi crate, so a `cargo check` at the metapi root still compiles only the backend. Build the desktop app from desktop/src-tauri per desktop/README.md. Note: app icons are still placeholder assets pending the final BoosAPI logo. The updater signing keys live outside the repo and are not included.
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
node_modules
|
||||
dist
|
||||
.turbo
|
||||
*.log
|
||||
.DS_Store
|
||||
coverage
|
||||
.vscode
|
||||
.idea
|
||||
*.tsbuildinfo
|
||||
@@ -0,0 +1,10 @@
|
||||
src
|
||||
test
|
||||
tests
|
||||
*.test.ts
|
||||
*.spec.ts
|
||||
tsconfig.json
|
||||
tsup.config.ts
|
||||
.eslintrc*
|
||||
.prettierrc*
|
||||
vitest.config.ts
|
||||
@@ -0,0 +1,527 @@
|
||||
# stream-markdown-parser
|
||||
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](README.zh-CN.md)
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](https://bundlephobia.com/package/stream-markdown-parser)
|
||||
[](https://github.com/Simon-He95/markstream-vue/blob/main/license)
|
||||
|
||||
Pure markdown parser and renderer utilities with streaming support - framework agnostic.
|
||||
|
||||
This package contains the core markdown parsing logic extracted from `markstream-vue`, making it usable in any JavaScript/TypeScript project without Vue dependencies.
|
||||
|
||||
## Features
|
||||
|
||||
- 🚀 **Pure JavaScript** - No framework dependencies
|
||||
- 📦 **Lightweight** - Minimal bundle size
|
||||
- 🔧 **Extensible** - Plugin-based architecture
|
||||
- 🎯 **Type-safe** - Full TypeScript support
|
||||
- ⚡ **Fast** - Optimized for performance
|
||||
- 🌊 **Streaming-friendly** - Progressive parsing support
|
||||
|
||||
> ℹ️ We now build on top of [`markdown-it-ts`](https://www.npmjs.com/package/markdown-it-ts), a TypeScript-first distribution of markdown-it. The API stays the same, but we only rely on its parsing pipeline and ship richer typings for tokens and hooks.
|
||||
|
||||
## Documentation
|
||||
|
||||
The full usage guide lives alongside the markstream-vue docs:
|
||||
|
||||
- English: https://markstream-vue-docs.simonhe.me/guide/api
|
||||
- 中文: https://markstream-vue-docs.simonhe.me/zh/guide/api
|
||||
|
||||
This README highlights the parser-specific APIs; visit the docs for end-to-end integration tutorials (VitePress, workers, Tailwind, troubleshooting, etc.).
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pnpm add stream-markdown-parser
|
||||
# or
|
||||
npm install stream-markdown-parser
|
||||
# or
|
||||
yarn add stream-markdown-parser
|
||||
```
|
||||
|
||||
## Quick API (TL;DR)
|
||||
|
||||
- `getMarkdown(msgId?, options?)` — create a `markdown-it-ts` instance with built-in plugins (task lists, sub/sup, math helpers, etc.). Also accepts `plugin`, `apply`, and `i18n` options.
|
||||
- `registerMarkdownPlugin(plugin)` / `clearRegisteredMarkdownPlugins()` — add or remove global plugins that run for every `getMarkdown()` call (useful for feature flags or tests).
|
||||
- `parseMarkdownToStructure(markdown, md, parseOptions)` — convert Markdown into the streaming-friendly AST consumed by `markstream-vue` and other renderers.
|
||||
- `processTokens(tokens)` / `parseInlineTokens(children, content?, preToken?, options?)` — low-level helpers if you want to bypass the built-in AST pipeline.
|
||||
- `applyMath`, `applyContainers`, `normalizeStandaloneBackslashT`, `findMatchingClose`, etc. — utilities for custom parsing or linting workflows.
|
||||
|
||||
## Usage
|
||||
|
||||
### Streaming-friendly pipeline
|
||||
|
||||
```
|
||||
Markdown string
|
||||
↓ getMarkdown() → markdown-it-ts instance with plugins
|
||||
parseMarkdownToStructure(markdown, md) → AST (ParsedNode[])
|
||||
↓ feed into your renderer (markstream-vue, custom UI, workers)
|
||||
```
|
||||
|
||||
Reuse the same `md` instance when parsing multiple documents—plugin setup is the heaviest step. When integrating with [`markstream-vue`](https://www.npmjs.com/package/markstream-vue), pass the AST to `<MarkdownRender :nodes="nodes" />` or supply raw `content` and hand it the same parser options.
|
||||
|
||||
### Incremental / streaming example
|
||||
|
||||
When consuming an AI or SSE stream you can keep appending to a buffer, parse with the same `md` instance, and send the AST to the UI (e.g., `markstream-vue`) on every chunk:
|
||||
|
||||
```ts
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown()
|
||||
let buffer = ''
|
||||
|
||||
async function handleChunk(chunk: string) {
|
||||
buffer += chunk
|
||||
const nodes = parseMarkdownToStructure(buffer, md)
|
||||
postMessage({ type: 'markdown:update', nodes })
|
||||
}
|
||||
```
|
||||
|
||||
In the UI layer, render `nodes` with `<MarkdownRender :nodes="nodes" />` to avoid re-parsing. See the [docs usage guide](https://markstream-vue-docs.simonhe.me/guide/usage) for end-to-end wiring.
|
||||
|
||||
### Basic example
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
// Create a markdown-it-ts instance with default plugins
|
||||
const md = getMarkdown()
|
||||
|
||||
// Parse markdown to our streaming-friendly AST structure
|
||||
const nodes = parseMarkdownToStructure('# Hello World', md)
|
||||
console.log(nodes)
|
||||
// [{ type: 'heading', level: 1, children: [...] }]
|
||||
|
||||
// markdown-it-ts still exposes render() if you need HTML output,
|
||||
// but this package now focuses on the token -> AST pipeline.
|
||||
const html = md.render?.('# Hello World\n\nThis is **bold**.')
|
||||
```
|
||||
|
||||
### With Math Options
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, setDefaultMathOptions } from 'stream-markdown-parser'
|
||||
|
||||
// Set global math options
|
||||
setDefaultMathOptions({
|
||||
commands: ['infty', 'perp', 'alpha'],
|
||||
escapeExclamation: true
|
||||
})
|
||||
|
||||
const md = getMarkdown()
|
||||
```
|
||||
|
||||
### With Custom i18n
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
// Using translation map
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: {
|
||||
'common.copy': '复制',
|
||||
}
|
||||
})
|
||||
|
||||
// Or using a translation function
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: (key: string) => translateFunction(key)
|
||||
})
|
||||
```
|
||||
|
||||
### With Plugins
|
||||
|
||||
```typescript
|
||||
import customPlugin from 'markdown-it-custom-plugin'
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
plugin: [
|
||||
[customPlugin, { /* options */ }]
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### Advanced: Custom Rules
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
apply: [
|
||||
(md) => {
|
||||
// Add custom inline rule
|
||||
md.inline.ruler.before('emphasis', 'custom', (state, silent) => {
|
||||
// Your custom logic
|
||||
return false
|
||||
})
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### Extending globally
|
||||
|
||||
Need to add a plugin everywhere without touching each call site? Use the helper exports:
|
||||
|
||||
```ts
|
||||
import {
|
||||
clearRegisteredMarkdownPlugins,
|
||||
registerMarkdownPlugin,
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
registerMarkdownPlugin(myPlugin)
|
||||
|
||||
const md = getMarkdown()
|
||||
// md now has `myPlugin` enabled in addition to anything passed via options
|
||||
|
||||
// For tests or teardown flows:
|
||||
clearRegisteredMarkdownPlugins()
|
||||
```
|
||||
|
||||
- `plugin` option → array of `md.use` invocations scoped to a single `getMarkdown` call.
|
||||
- `apply` option → imperatively mutate the instance (`md.inline.ruler.before(...)`). Wrap in try/catch if you need to surface errors differently; the helper logs to console to preserve legacy behaviour.
|
||||
- `registerMarkdownPlugin` → global singleton registry (handy in SSR or worker contexts where you want feature flags to apply everywhere).
|
||||
|
||||
## API
|
||||
|
||||
### Main Functions
|
||||
|
||||
#### `getMarkdown(msgId?, options?)`
|
||||
|
||||
Creates a configured `markdown-it-ts` instance (API-compatible with markdown-it).
|
||||
|
||||
**Parameters:**
|
||||
- `msgId` (string, optional): Unique identifier for this instance. Default: `editor-${Date.now()}`
|
||||
- `options` (GetMarkdownOptions, optional): Configuration options
|
||||
|
||||
**Options:**
|
||||
```typescript
|
||||
interface GetMarkdownOptions {
|
||||
// Array of markdown-it/markdown-it-ts plugins to use
|
||||
plugin?: Array<Plugin | [Plugin, any]>
|
||||
|
||||
// Array of functions to mutate the md instance
|
||||
apply?: Array<(md: MarkdownIt) => void>
|
||||
|
||||
// Translation function or translation map
|
||||
i18n?: ((key: string) => string) | Record<string, string>
|
||||
}
|
||||
```
|
||||
|
||||
#### `parseMarkdownToStructure(content, md, options?)`
|
||||
|
||||
Parses markdown content into a structured node tree.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): The markdown content to parse
|
||||
- `md` (MarkdownItCore): A markdown-it-ts instance created with `getMarkdown()`
|
||||
- `options` (ParseOptions, optional): Parsing options with hooks
|
||||
|
||||
**Returns:** `ParsedNode[]` - An array of parsed nodes
|
||||
|
||||
#### `processTokens(tokens)`
|
||||
|
||||
Processes raw markdown-it tokens into a flat array.
|
||||
|
||||
#### `parseInlineTokens(tokens, content?, preToken?, options?)`
|
||||
|
||||
Parses inline markdown-it-ts tokens into renderer nodes. Pass the inline token array plus the optional raw `content` string (from the parent token), an optional previous token, and inline parse options (`requireClosingStrong`, `customHtmlTags`, `validateLink`).
|
||||
|
||||
### Configuration Functions
|
||||
|
||||
#### `setDefaultMathOptions(options)`
|
||||
|
||||
Set global math rendering options.
|
||||
|
||||
**Parameters:**
|
||||
- `options` (MathOptions): Math configuration options
|
||||
|
||||
```typescript
|
||||
interface MathOptions {
|
||||
commands?: readonly string[] // LaTeX commands to escape
|
||||
escapeExclamation?: boolean // Escape standalone '!' (default: true)
|
||||
}
|
||||
```
|
||||
|
||||
### Parse hooks (fine-grained transforms)
|
||||
|
||||
ParseOptions supports the following hooks and flags:
|
||||
|
||||
```ts
|
||||
interface ParseOptions {
|
||||
preTransformTokens?: (tokens: Token[]) => Token[]
|
||||
postTransformTokens?: (tokens: Token[]) => Token[]
|
||||
// Custom HTML-like tags to emit as custom nodes (e.g. ['thinking'])
|
||||
// Use tag-like names such as 'thinking', 'answer-box', or 'my_component'.
|
||||
customHtmlTags?: string[]
|
||||
// Validate link href before emitting a `link` node; false -> plain text
|
||||
validateLink?: (url: string) => boolean
|
||||
// When true, treats the input as complete (end-of-stream)
|
||||
final?: boolean
|
||||
// Require closing `**` for strong parsing (default: false)
|
||||
requireClosingStrong?: boolean
|
||||
}
|
||||
```
|
||||
|
||||
Example — flag AI “thinking” blocks:
|
||||
|
||||
```ts
|
||||
const parseOptions = {
|
||||
customHtmlTags: ['thinking'],
|
||||
}
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md, parseOptions)
|
||||
const tagged = nodes.map(node =>
|
||||
node.type === 'html_block' && /<thinking>/.test((node as any).content ?? '')
|
||||
? { ...node, meta: { type: 'thinking' } }
|
||||
: node,
|
||||
)
|
||||
```
|
||||
|
||||
Use the metadata in your renderer to show custom UI without mangling the original Markdown.
|
||||
|
||||
Example — enforce safe link protocols:
|
||||
|
||||
```ts
|
||||
const md = getMarkdown('safe-links')
|
||||
md.set?.({
|
||||
validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()),
|
||||
})
|
||||
|
||||
const nodes = parseMarkdownToStructure(
|
||||
'[ok](https://example.com) [bad](javascript:alert(1))',
|
||||
md,
|
||||
{ final: true },
|
||||
)
|
||||
// "ok" stays a link node; "bad" is downgraded to plain text
|
||||
```
|
||||
|
||||
### Unknown HTML-like tags
|
||||
|
||||
By default, non-standard HTML-like tags (for example `<question>`) are rendered as raw HTML elements once they are complete. Incomplete or malformed fragments stay as **literal text** so they do not swallow surrounding content during streaming or final renders. If you want them emitted as custom nodes (`type: 'question'` with parsed attrs/content), opt in via `customHtmlTags`.
|
||||
|
||||
### Utility Functions
|
||||
|
||||
#### `isMathLike(content)`
|
||||
|
||||
Heuristic function to detect if content looks like mathematical notation.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): Content to check
|
||||
|
||||
**Returns:** `boolean`
|
||||
|
||||
#### `findMatchingClose(src, startIdx, open, close)`
|
||||
|
||||
Find the matching closing delimiter in a string, handling nested pairs.
|
||||
|
||||
**Parameters:**
|
||||
- `src` (string): Source string
|
||||
- `startIdx` (number): Start index to search from
|
||||
- `open` (string): Opening delimiter
|
||||
- `close` (string): Closing delimiter
|
||||
|
||||
**Returns:** `number` - Index of matching close, or -1 if not found
|
||||
|
||||
## Tips & troubleshooting
|
||||
|
||||
- **Reuse parser instances**: cache `getMarkdown()` results per worker/request to avoid re-registering plugins.
|
||||
- **Server-side parsing**: run `parseMarkdownToStructure` on the server, ship the AST to the client, and hydrate with `markstream-vue` for deterministic output.
|
||||
- **Custom HTML widgets**: pre-extract `<MyWidget>` blocks before parsing (replace with placeholders) and reinject them during rendering instead of mutating `html_block` nodes post-parse.
|
||||
- **Styling**: when piping nodes into `markstream-vue`, follow the docs [CSS checklist](https://markstream-vue-docs.simonhe.me/guide/troubleshooting#css-looks-wrong-start-here) so Tailwind/UnoCSS don’t override library styles.
|
||||
- **Error handling**: the `apply` hook swallows exceptions to maintain backwards compatibility. If you want strict mode, wrap your mutators before passing them in and rethrow/log as needed.
|
||||
|
||||
#### `parseFenceToken(token)`
|
||||
|
||||
Parse a code fence token into a CodeBlockNode.
|
||||
|
||||
**Parameters:**
|
||||
- `token` (MarkdownToken): markdown-it token
|
||||
|
||||
**Returns:** `CodeBlockNode`
|
||||
|
||||
#### `normalizeStandaloneBackslashT(content, options?)`
|
||||
|
||||
Normalize backslash-t sequences in math content.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): Content to normalize
|
||||
- `options` (MathOptions, optional): Math options
|
||||
|
||||
**Returns:** `string`
|
||||
|
||||
### Lower-level helpers
|
||||
|
||||
If you need full control over how tokens are transformed, you can import the primitive builders directly:
|
||||
|
||||
```ts
|
||||
import type { MarkdownToken } from 'stream-markdown-parser'
|
||||
import {
|
||||
|
||||
parseInlineTokens,
|
||||
processTokens
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
const tokens: MarkdownToken[] = md.parse(markdown, {})
|
||||
const nodes = processTokens(tokens)
|
||||
// or operate at inline granularity:
|
||||
const inlineNodes = parseInlineTokens(tokens[0].children ?? [], tokens[0].content ?? '')
|
||||
```
|
||||
|
||||
`processTokens` is what `parseMarkdownToStructure` uses internally, so you can remix the AST pipeline without reimplementing the Markdown-it loop.
|
||||
|
||||
### Plugin Functions
|
||||
|
||||
#### `applyMath(md, options?)`
|
||||
|
||||
Apply math plugin to markdown-it instance.
|
||||
|
||||
**Parameters:**
|
||||
- `md` (MarkdownIt): markdown-it instance
|
||||
- `options` (MathOptions, optional): Math rendering options
|
||||
|
||||
#### `applyContainers(md)`
|
||||
|
||||
Apply container plugins to markdown-it instance.
|
||||
|
||||
**Parameters:**
|
||||
- `md` (MarkdownIt): markdown-it instance
|
||||
|
||||
### Constants
|
||||
|
||||
#### `KATEX_COMMANDS`
|
||||
|
||||
Array of common KaTeX commands for escaping.
|
||||
|
||||
#### `TEX_BRACE_COMMANDS`
|
||||
|
||||
Array of TeX commands that use braces.
|
||||
|
||||
#### `ESCAPED_TEX_BRACE_COMMANDS`
|
||||
|
||||
Escaped version of TEX_BRACE_COMMANDS for regex use.
|
||||
|
||||
## Types
|
||||
|
||||
All TypeScript types are exported:
|
||||
|
||||
```typescript
|
||||
import type {
|
||||
// Node types
|
||||
CodeBlockNode,
|
||||
GetMarkdownOptions,
|
||||
HeadingNode,
|
||||
ListItemNode,
|
||||
ListNode,
|
||||
MathOptions,
|
||||
ParagraphNode,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
// ... and more
|
||||
} from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
### Node Types
|
||||
|
||||
The parser exports various node types representing different markdown elements:
|
||||
|
||||
- `TextNode`, `HeadingNode`, `ParagraphNode`
|
||||
- `ListNode`, `ListItemNode`
|
||||
- `CodeBlockNode`, `InlineCodeNode`
|
||||
- `LinkNode`, `ImageNode`
|
||||
- `BlockquoteNode`, `TableNode`
|
||||
- `MathBlockNode`, `MathInlineNode`
|
||||
- And many more...
|
||||
|
||||
## Default Plugins
|
||||
|
||||
This package comes with the following markdown-it plugins pre-configured:
|
||||
|
||||
- `markdown-it-sub` - Subscript support (`H~2~O`)
|
||||
- `markdown-it-sup` - Superscript support (`x^2^`)
|
||||
- `markdown-it-mark` - Highlight/mark support (`==highlighted==`)
|
||||
- `markdown-it-task-checkbox` - Task list support (`- [ ] Todo`)
|
||||
- `markdown-it-ins` - Insert tag support (`++inserted++`)
|
||||
- `markdown-it-footnote` - Footnote support
|
||||
- `markdown-it-container` - Custom container support (`::: warning`, `::: tip`, etc.)
|
||||
- Math support - LaTeX math rendering with `$...$` and `$$...$$`
|
||||
|
||||
## Framework Integration
|
||||
|
||||
While this package is framework-agnostic, it's designed to work seamlessly with:
|
||||
|
||||
- ✅ **Node.js** - Server-side rendering
|
||||
- ✅ **Vue 3** - Use with `markstream-vue` (or your own renderer)
|
||||
- ✅ **React** - Use parsed nodes for custom rendering
|
||||
- ✅ **Vanilla JS** - Direct HTML rendering
|
||||
- ✅ **Any framework** - Parse to AST and render as needed
|
||||
|
||||
### Example — dedicated worker feeding markstream-vue
|
||||
|
||||
Offload parsing to a Web Worker while the UI renders via `markstream-vue`:
|
||||
|
||||
```ts
|
||||
// worker.ts
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown()
|
||||
let buffer = ''
|
||||
|
||||
globalThis.addEventListener('message', (event) => {
|
||||
if (event.data.type === 'chunk') {
|
||||
buffer += event.data.value
|
||||
const nodes = parseMarkdownToStructure(buffer, md)
|
||||
globalThis.postMessage({ type: 'update', nodes })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
```ts
|
||||
// ui.ts
|
||||
const worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' })
|
||||
worker.addEventListener('message', (event) => {
|
||||
if (event.data.type === 'update')
|
||||
nodes.value = event.data.nodes
|
||||
})
|
||||
```
|
||||
|
||||
```vue
|
||||
<MarkdownRender :nodes="nodes" />
|
||||
```
|
||||
|
||||
This pattern keeps Markdown-it work off the main thread and lets you reuse the same AST in any framework.
|
||||
|
||||
## Migration from `markstream-vue` (parser exports)
|
||||
|
||||
If you're currently importing parser helpers from `markstream-vue`, you can switch to the dedicated package:
|
||||
|
||||
```typescript
|
||||
// before
|
||||
import { getMarkdown } from 'markstream-vue'
|
||||
|
||||
// after
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
All APIs remain the same. See the [migration guide](https://markstream-vue-docs.simonhe.me/monorepo-migration) for details.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Lightweight**: ~65KB minified (13KB gzipped)
|
||||
- **Fast**: Optimized for real-time parsing
|
||||
- **Tree-shakeable**: Only import what you need
|
||||
- **Few dependencies**: `markdown-it-ts` + a small set of markdown-it plugins
|
||||
|
||||
## Contributing
|
||||
|
||||
Issues and PRs welcome! Please read the [contribution guidelines](https://github.com/Simon-He95/markstream-vue/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT © Simon He
|
||||
|
||||
## Related
|
||||
|
||||
- [markstream-vue](https://www.npmjs.com/package/markstream-vue) - Full-featured Vue 3 Markdown renderer
|
||||
@@ -0,0 +1,491 @@
|
||||
# stream-markdown-parser
|
||||
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](README.md)
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](https://bundlephobia.com/package/stream-markdown-parser)
|
||||
[](https://github.com/Simon-He95/markstream-vue/blob/main/license)
|
||||
|
||||
纯 JavaScript Markdown 解析器和渲染工具,支持流式处理 - 框架无关。
|
||||
|
||||
该包包含从 `markstream-vue` 中提取的核心 Markdown 解析逻辑,使其可以在任何 JavaScript/TypeScript 项目中使用,无需 Vue 依赖。
|
||||
|
||||
## 特性
|
||||
|
||||
- 🚀 **纯 JavaScript** - 无框架依赖
|
||||
- 📦 **轻量级** - 最小打包体积
|
||||
- 🔧 **可扩展** - 基于插件的架构
|
||||
- 🎯 **类型安全** - 完整的 TypeScript 支持
|
||||
- ⚡ **高性能** - 性能优化
|
||||
- 🌊 **流式友好** - 支持渐进式解析
|
||||
|
||||
> ℹ️ 自当前版本起我们基于 [`markdown-it-ts`](https://www.npmjs.com/package/markdown-it-ts)(一个 TypeScript 优先的 markdown-it 发行版)进行构建。API 与 markdown-it 保持一致,但内部仅依赖其解析流程,并提供更丰富的 token 类型定义。
|
||||
|
||||
## 文档
|
||||
|
||||
完整的使用说明与集成教程见 markstream-vue 文档站:
|
||||
|
||||
- English: https://markstream-vue-docs.simonhe.me/guide/api
|
||||
- 中文: https://markstream-vue-docs.simonhe.me/zh/guide/api
|
||||
|
||||
本 README 聚焦解析器 API;如需 VitePress/Vite/Nuxt 集成、Worker 流式解析、Tailwind/UnoCSS 配置等指南,请查阅上述文档。
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
pnpm add stream-markdown-parser
|
||||
# 或
|
||||
npm install stream-markdown-parser
|
||||
# 或
|
||||
yarn add stream-markdown-parser
|
||||
```
|
||||
|
||||
## 快速 API 速览
|
||||
|
||||
- `getMarkdown(msgId?, options?)` — 返回一个预配置的 `markdown-it-ts` 实例;支持 `plugin`、`apply`、`i18n` 等选项(内置任务列表、上下标、数学等插件)。
|
||||
- `registerMarkdownPlugin(plugin)` / `clearRegisteredMarkdownPlugins()` — 全局注册/清除插件,在所有 `getMarkdown()` 调用中生效(适合特性开关或测试环境)。
|
||||
- `parseMarkdownToStructure(markdown, md, parseOptions)` — 将 Markdown 转换为可供 `markstream-vue` 等渲染器使用的 AST。
|
||||
- `processTokens(tokens)` / `parseInlineTokens(children, content?, preToken?, options?)` — 更底层的 token → 节点工具,方便自定义管线。
|
||||
- `applyMath`、`applyContainers`、`normalizeStandaloneBackslashT`、`findMatchingClose` 等 — 用于构建自定义解析、lint 或内容清洗流程。
|
||||
|
||||
## 使用
|
||||
|
||||
### 流式解析流程
|
||||
|
||||
```
|
||||
Markdown 字符串
|
||||
↓ getMarkdown() → 带插件的 markdown-it-ts 实例
|
||||
parseMarkdownToStructure(markdown, md) → AST (ParsedNode[])
|
||||
↓ 交给你的渲染器(markstream-vue、自定义 UI、Worker 等)
|
||||
```
|
||||
|
||||
多次解析时复用同一个 `md` 实例可以避免重复注册插件。与 [`markstream-vue`](https://www.npmjs.com/package/markstream-vue) 一起使用时,你可以把 AST 传给 `<MarkdownRender :nodes="nodes" />`,或仅传入原始 `content` 并共享同一套解析配置。
|
||||
|
||||
### 增量 / 流式示例
|
||||
|
||||
处理 AI/SSE 流时,可以复用同一个 `md` 实例不停地对累积缓冲区解析,并把 AST 推送给 UI(例如 `markstream-vue`):
|
||||
|
||||
```ts
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown()
|
||||
let buffer = ''
|
||||
|
||||
async function handleChunk(chunk: string) {
|
||||
buffer += chunk
|
||||
const nodes = parseMarkdownToStructure(buffer, md)
|
||||
postMessage({ type: 'markdown:update', nodes })
|
||||
}
|
||||
```
|
||||
|
||||
在前端通过 `<MarkdownRender :nodes="nodes" />` 渲染即可避免重复解析。具体串联示例见[文档用法指南](https://markstream-vue-docs.simonhe.me/zh/guide/usage)。
|
||||
|
||||
### 基础示例
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
// 创建一个带有默认插件的 markdown-it-ts 实例
|
||||
const md = getMarkdown()
|
||||
|
||||
// 将 Markdown 解析为流式友好的 AST 结构
|
||||
const nodes = parseMarkdownToStructure('# Hello World', md)
|
||||
console.log(nodes)
|
||||
// [{ type: 'heading', level: 1, children: [...] }]
|
||||
|
||||
// 如果仍需 HTML 输出,markdown-it-ts 依旧提供 render()
|
||||
const html = md.render?.('# Hello World\n\nThis is **bold**.')
|
||||
```
|
||||
|
||||
### 配置数学公式选项
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, setDefaultMathOptions } from 'stream-markdown-parser'
|
||||
|
||||
// 设置全局数学公式选项
|
||||
setDefaultMathOptions({
|
||||
commands: ['infty', 'perp', 'alpha'],
|
||||
escapeExclamation: true
|
||||
})
|
||||
|
||||
const md = getMarkdown()
|
||||
```
|
||||
|
||||
### 自定义国际化
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
// 使用翻译映射
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: {
|
||||
'common.copy': '复制',
|
||||
}
|
||||
})
|
||||
|
||||
// 或使用翻译函数
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: (key: string) => translateFunction(key)
|
||||
})
|
||||
```
|
||||
|
||||
### 使用插件
|
||||
|
||||
```typescript
|
||||
import customPlugin from 'markdown-it-custom-plugin'
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
plugin: [
|
||||
[customPlugin, { /* 选项 */ }]
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### 高级:自定义规则
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
apply: [
|
||||
(md) => {
|
||||
// 添加自定义内联规则
|
||||
md.inline.ruler.before('emphasis', 'custom', (state, silent) => {
|
||||
// 你的自定义逻辑
|
||||
return false
|
||||
})
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### 全局扩展
|
||||
|
||||
想在所有 `getMarkdown()` 实例上启用同一个插件,而无需修改调用点?可使用内置 helper:
|
||||
|
||||
```ts
|
||||
import {
|
||||
clearRegisteredMarkdownPlugins,
|
||||
registerMarkdownPlugin,
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
registerMarkdownPlugin(myPlugin)
|
||||
|
||||
const md = getMarkdown()
|
||||
// 现在 md 会自动包含 myPlugin
|
||||
|
||||
// 测试或清理阶段:
|
||||
clearRegisteredMarkdownPlugins()
|
||||
```
|
||||
|
||||
- `plugin` 选项 → 针对单次 `getMarkdown` 调用传入 `md.use(...)`。
|
||||
- `apply` 选项 → 直接操作实例(如 `md.inline.ruler.before(...)`)。如果需要严格模式,可自行包裹 try/catch;默认会打印错误保持兼容。
|
||||
- `registerMarkdownPlugin` → 全局注册表,适用于 SSR / Worker 等场景统一开启功能。
|
||||
|
||||
## API
|
||||
|
||||
### 主要函数
|
||||
|
||||
#### `getMarkdown(msgId?, options?)`
|
||||
|
||||
创建一个配置好的 `markdown-it-ts` 实例(与 markdown-it API 兼容)。
|
||||
|
||||
**参数:**
|
||||
- `msgId` (string, 可选): 该实例的唯一标识符。默认值:`editor-${Date.now()}`
|
||||
- `options` (GetMarkdownOptions, 可选): 配置选项
|
||||
|
||||
**选项:**
|
||||
```typescript
|
||||
interface GetMarkdownOptions {
|
||||
// 要使用的 markdown-it / markdown-it-ts 插件数组
|
||||
plugin?: Array<Plugin | [Plugin, any]>
|
||||
|
||||
// 修改 md 实例的函数数组
|
||||
apply?: Array<(md: MarkdownIt) => void>
|
||||
|
||||
// 翻译函数或翻译映射
|
||||
i18n?: ((key: string) => string) | Record<string, string>
|
||||
}
|
||||
```
|
||||
|
||||
#### `parseMarkdownToStructure(content, md, options?)`
|
||||
|
||||
将 Markdown 内容解析为结构化节点树。
|
||||
|
||||
**参数:**
|
||||
- `content` (string): 要解析的 Markdown 内容
|
||||
- `md` (MarkdownItCore): 由 `getMarkdown()` 创建的 markdown-it-ts 实例
|
||||
- `options` (ParseOptions, 可选): 带有钩子的解析选项
|
||||
|
||||
**返回值:** `ParsedNode[]` - 解析后的节点数组
|
||||
|
||||
#### `processTokens(tokens)`
|
||||
|
||||
将原始 markdown-it tokens 处理为扁平数组。
|
||||
|
||||
#### `parseInlineTokens(tokens, content?, preToken?, options?)`
|
||||
|
||||
解析内联 markdown-it-ts tokens 并产出节点。可传入原始 `content`(来自父 token)、可选的前一个 token,以及 inline 解析选项(`requireClosingStrong`、`customHtmlTags`、`validateLink`)。
|
||||
|
||||
### 配置函数
|
||||
|
||||
#### `setDefaultMathOptions(options)`
|
||||
|
||||
设置全局数学公式渲染选项。
|
||||
|
||||
**参数:**
|
||||
- `options` (MathOptions): 数学公式配置选项
|
||||
|
||||
```typescript
|
||||
interface MathOptions {
|
||||
commands?: readonly string[] // 要转义的 LaTeX 命令
|
||||
escapeExclamation?: boolean // 转义独立的 '!' (默认: true)
|
||||
}
|
||||
```
|
||||
|
||||
### 解析钩子(精细化变换)
|
||||
|
||||
ParseOptions 支持以下钩子与标志:
|
||||
|
||||
```ts
|
||||
interface ParseOptions {
|
||||
preTransformTokens?: (tokens: Token[]) => Token[]
|
||||
postTransformTokens?: (tokens: Token[]) => Token[]
|
||||
// 自定义 HTML 类标签,作为自定义节点输出(如 ['thinking'])
|
||||
// 请传标签式名字,比如 'thinking'、'answer-box'、'my_component'
|
||||
customHtmlTags?: string[]
|
||||
// 输出 link 节点前校验 href;返回 false 时降级为纯文本
|
||||
validateLink?: (url: string) => boolean
|
||||
// true 表示输入已结束(end-of-stream)
|
||||
final?: boolean
|
||||
// 解析 strong 时要求闭合 `**`(默认 false)
|
||||
requireClosingStrong?: boolean
|
||||
}
|
||||
```
|
||||
|
||||
示例 —— 标记 AI "思考" 块:
|
||||
|
||||
```ts
|
||||
const parseOptions = {
|
||||
customHtmlTags: ['thinking'],
|
||||
}
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md, parseOptions)
|
||||
const tagged = nodes.map(node =>
|
||||
node.type === 'html_block' && /<thinking>/.test((node as any).content ?? '')
|
||||
? { ...node, meta: { type: 'thinking' } }
|
||||
: node,
|
||||
)
|
||||
```
|
||||
|
||||
在渲染器中读取 `node.meta` 即可渲染自定义 UI,而无需直接修改 Markdown 文本。
|
||||
|
||||
示例 —— 限制链接协议,拦截不安全链接:
|
||||
|
||||
```ts
|
||||
const md = getMarkdown('safe-links')
|
||||
md.set?.({
|
||||
validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()),
|
||||
})
|
||||
|
||||
const nodes = parseMarkdownToStructure(
|
||||
'[ok](https://example.com) [bad](javascript:alert(1))',
|
||||
md,
|
||||
{ final: true },
|
||||
)
|
||||
// "ok" 保持为 link 节点;"bad" 会降级为纯文本
|
||||
```
|
||||
|
||||
### 未知 HTML 类标签
|
||||
|
||||
默认情况下,非标准的 HTML 类标签(例如 `<question>`)在完整闭合时会按原生 HTML 渲染(作为自定义元素输出)。未闭合或格式不完整的片段会保持为**纯文本**,避免在流式或最终渲染时吞掉周围内容。若希望它们作为自定义节点输出(`type: 'question'`,携带 attrs/content),需要在 `customHtmlTags` 中显式声明。
|
||||
|
||||
### 工具函数
|
||||
|
||||
#### `isMathLike(content)`
|
||||
|
||||
启发式函数,用于检测内容是否类似数学符号。
|
||||
|
||||
**参数:**
|
||||
- `content` (string): 要检查的内容
|
||||
|
||||
**返回值:** `boolean`
|
||||
|
||||
#### `findMatchingClose(src, startIdx, open, close)`
|
||||
|
||||
在字符串中查找匹配的闭合分隔符,处理嵌套对。
|
||||
|
||||
**参数:**
|
||||
- `src` (string): 源字符串
|
||||
- `startIdx` (number): 开始搜索的索引
|
||||
- `open` (string): 开启分隔符
|
||||
- `close` (string): 闭合分隔符
|
||||
|
||||
**返回值:** `number` - 匹配闭合的索引,如果未找到则返回 -1
|
||||
|
||||
## 使用建议与排障
|
||||
|
||||
- **复用解析实例**:缓存 `getMarkdown()` 的结果,避免重复注册插件。
|
||||
- **服务端解析**:在服务端运行 `parseMarkdownToStructure` 后把 AST 下发给客户端,配合 `markstream-vue` 实现确定性输出。
|
||||
- **自定义 HTML 组件**:在解析前先把 `<MyWidget>` 这类片段替换为占位符,渲染时再注入,避免在 `html_block` 上进行脆弱的字符串操作。
|
||||
- **样式提示**:如果将节点交给 `markstream-vue`,务必按照文档的 [CSS 排查清单](https://markstream-vue-docs.simonhe.me/zh/guide/troubleshooting#css-looks-wrong-start-here) 调整 reset / layer,防止 Tailwind/UnoCSS 覆盖样式。
|
||||
- **错误处理**:`apply` 钩子内部默认捕获异常后打印日志,如需在 CI/生产中抛出错误,可在传入前自行封装并 rethrow。
|
||||
|
||||
#### `parseFenceToken(token)`
|
||||
|
||||
将代码围栏 token 解析为 CodeBlockNode。
|
||||
|
||||
**参数:**
|
||||
- `token` (MarkdownToken): markdown-it token
|
||||
|
||||
**返回值:** `CodeBlockNode`
|
||||
|
||||
#### `normalizeStandaloneBackslashT(content, options?)`
|
||||
|
||||
规范化数学内容中的反斜杠-t 序列。
|
||||
|
||||
**参数:**
|
||||
- `content` (string): 要规范化的内容
|
||||
- `options` (MathOptions, 可选): 数学选项
|
||||
|
||||
**返回值:** `string`
|
||||
|
||||
### 低阶辅助函数
|
||||
|
||||
需要更细粒度地控制 token → AST 流程时,可直接使用以下导出:
|
||||
|
||||
```ts
|
||||
import type { MarkdownToken } from 'stream-markdown-parser'
|
||||
import {
|
||||
|
||||
parseInlineTokens,
|
||||
processTokens
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
const tokens: MarkdownToken[] = md.parse(markdown, {})
|
||||
const nodes = processTokens(tokens)
|
||||
// 或仅解析内联内容:
|
||||
const inlineNodes = parseInlineTokens(tokens[0].children ?? [], tokens[0].content ?? '')
|
||||
```
|
||||
|
||||
`processTokens` 即 `parseMarkdownToStructure` 内部使用的同一个转换器,可在自定义管线中复用,避免重复实现 Markdown-it 遍历。
|
||||
|
||||
### 插件函数
|
||||
|
||||
#### `applyMath(md, options?)`
|
||||
|
||||
将数学插件应用到 markdown-it 实例。
|
||||
|
||||
**参数:**
|
||||
- `md` (MarkdownIt): markdown-it 实例
|
||||
- `options` (MathOptions, 可选): 数学渲染选项
|
||||
|
||||
#### `applyContainers(md)`
|
||||
|
||||
将容器插件应用到 markdown-it 实例。
|
||||
|
||||
**参数:**
|
||||
- `md` (MarkdownIt): markdown-it 实例
|
||||
|
||||
### 常量
|
||||
|
||||
#### `KATEX_COMMANDS`
|
||||
|
||||
用于转义的常用 KaTeX 命令数组。
|
||||
|
||||
#### `TEX_BRACE_COMMANDS`
|
||||
|
||||
使用大括号的 TeX 命令数组。
|
||||
|
||||
#### `ESCAPED_TEX_BRACE_COMMANDS`
|
||||
|
||||
用于正则表达式的 TEX_BRACE_COMMANDS 转义版本。
|
||||
|
||||
## 类型
|
||||
|
||||
所有 TypeScript 类型都已导出:
|
||||
|
||||
```typescript
|
||||
import type {
|
||||
// 节点类型
|
||||
CodeBlockNode,
|
||||
GetMarkdownOptions,
|
||||
HeadingNode,
|
||||
ListItemNode,
|
||||
ListNode,
|
||||
MathOptions,
|
||||
ParagraphNode,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
// ... 更多
|
||||
} from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
### 节点类型
|
||||
|
||||
解析器导出各种表示不同 Markdown 元素的节点类型:
|
||||
|
||||
- `TextNode`, `HeadingNode`, `ParagraphNode`
|
||||
- `ListNode`, `ListItemNode`
|
||||
- `CodeBlockNode`, `InlineCodeNode`
|
||||
- `LinkNode`, `ImageNode`
|
||||
- `BlockquoteNode`, `TableNode`
|
||||
- `MathBlockNode`, `MathInlineNode`
|
||||
- 以及更多...
|
||||
|
||||
## 默认插件
|
||||
|
||||
该包预配置了以下 markdown-it 插件:
|
||||
|
||||
- `markdown-it-sub` - 下标支持(`H~2~O`)
|
||||
- `markdown-it-sup` - 上标支持(`x^2^`)
|
||||
- `markdown-it-mark` - 高亮/标记支持(`==highlighted==`)
|
||||
- `markdown-it-task-checkbox` - 任务列表支持(`- [ ] Todo`)
|
||||
- `markdown-it-ins` - 插入标签支持(`++inserted++`)
|
||||
- `markdown-it-footnote` - 脚注支持
|
||||
- `markdown-it-container` - 自定义容器支持(`::: warning`, `::: tip` 等)
|
||||
- 数学公式支持 - 使用 `$...$` 和 `$$...$$` 渲染 LaTeX 数学公式
|
||||
|
||||
## 框架集成
|
||||
|
||||
虽然该包与框架无关,但它被设计为可以无缝配合以下框架使用:
|
||||
|
||||
- ✅ **Node.js** - 服务器端渲染
|
||||
- ✅ **Vue 3** - 配合 `markstream-vue`(或你的自定义渲染层)使用
|
||||
- ✅ **React** - 使用解析的节点进行自定义渲染
|
||||
- ✅ **Vanilla JS** - 直接 HTML 渲染
|
||||
- ✅ **任何框架** - 解析为 AST 并按需渲染
|
||||
|
||||
## 从 `markstream-vue` 迁移(解析器导出)
|
||||
|
||||
如果你当前是从 `markstream-vue` 引入解析器相关 helper,可以切换为使用独立包:
|
||||
|
||||
```typescript
|
||||
// 之前
|
||||
import { getMarkdown } from 'markstream-vue'
|
||||
|
||||
// 现在
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
所有 API 保持不变。详见[迁移指南](https://markstream-vue-docs.simonhe.me/zh/monorepo-migration)。
|
||||
|
||||
## 性能
|
||||
|
||||
- **轻量级**: ~65KB 压缩后(13KB gzipped)
|
||||
- **快速**: 针对实时解析优化
|
||||
- **Tree-shakeable**: 只导入你需要的部分
|
||||
- **依赖很少**: `markdown-it-ts` + 少量 markdown-it 插件
|
||||
|
||||
## 贡献
|
||||
|
||||
欢迎提交 Issues 和 PRs!请阅读[贡献指南](https://github.com/Simon-He95/markstream-vue/blob/main/CONTRIBUTING.md)。
|
||||
|
||||
## 许可证
|
||||
|
||||
MIT © Simon He
|
||||
|
||||
## 相关项目
|
||||
|
||||
- [markstream-vue](https://www.npmjs.com/package/markstream-vue) - 功能完整的 Vue 3 Markdown 渲染器
|
||||
@@ -0,0 +1,14 @@
|
||||
/* eslint-disable */
|
||||
// @ts-nocheck
|
||||
// Generated by unplugin-vue-components
|
||||
// Read more: https://github.com/vuejs/core/pull/3399
|
||||
// biome-ignore lint: disable
|
||||
export {}
|
||||
|
||||
/* prettier-ignore */
|
||||
declare module 'vue' {
|
||||
export interface GlobalComponents {
|
||||
RouterLink: typeof import('vue-router')['RouterLink']
|
||||
RouterView: typeof import('vue-router')['RouterView']
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
{
|
||||
"name": "stream-markdown-parser",
|
||||
"type": "module",
|
||||
"version": "0.0.90",
|
||||
"packageManager": "pnpm@10.33.0",
|
||||
"description": "Pure markdown parser and renderer utilities with streaming support - framework agnostic",
|
||||
"author": "Simon He",
|
||||
"license": "MIT",
|
||||
"homepage": "https://github.com/Simon-He95/markstream-vue/tree/main/packages/markdown-parser",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/Simon-He95/markstream-vue.git",
|
||||
"directory": "packages/markdown-parser"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/Simon-He95/markstream-vue/issues"
|
||||
},
|
||||
"keywords": [
|
||||
"markdown",
|
||||
"markdown-it",
|
||||
"markdown-it-ts",
|
||||
"parser",
|
||||
"markdown-parser",
|
||||
"markdown-renderer",
|
||||
"ast",
|
||||
"markdown-ast",
|
||||
"streaming",
|
||||
"progressive",
|
||||
"framework-agnostic"
|
||||
],
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js"
|
||||
}
|
||||
},
|
||||
"main": "./dist/index.js",
|
||||
"module": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"scripts": {
|
||||
"dev": "tsdown --watch",
|
||||
"build": "tsdown",
|
||||
"test": "vitest",
|
||||
"test:run": "vitest --run",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"lint": "eslint . --cache",
|
||||
"lint:fix": "eslint . --cache --fix",
|
||||
"prepublishOnly": "pnpm run build",
|
||||
"release": "bumpp --commit --no-tag --no-push && npm publish --access public && node ../../scripts/tag-package.mjs --package-json package.json --push"
|
||||
},
|
||||
"dependencies": {
|
||||
"markdown-it-container": "^4.0.0",
|
||||
"markdown-it-footnote": "^4.0.0",
|
||||
"markdown-it-ins": "^4.0.0",
|
||||
"markdown-it-mark": "^4.0.0",
|
||||
"markdown-it-sub": "^2.0.0",
|
||||
"markdown-it-sup": "^2.0.0",
|
||||
"markdown-it-task-checkbox": "^1.0.6",
|
||||
"markdown-it-ts": "^0.0.9"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.19.15",
|
||||
"tsdown": "^0.15.12",
|
||||
"typescript": "^5.9.3",
|
||||
"vitest": "^3.2.4"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
/**
|
||||
* MathOptions control how the math plugin normalizes content before
|
||||
* handing it to KaTeX (or other math renderers).
|
||||
*
|
||||
* - commands: list of command words that should be auto-prefixed with a
|
||||
* backslash if not already escaped (e.g. 'infty' -> '\\infty'). Use a
|
||||
* conservative list to avoid false positives in prose.
|
||||
* - escapeExclamation: whether to escape standalone '!' to '\\!' (default true).
|
||||
*/
|
||||
export interface MathOptions {
|
||||
/** List of command words to auto-escape. */
|
||||
commands?: readonly string[]
|
||||
/** Whether to escape standalone '!' (default: true). */
|
||||
escapeExclamation?: boolean
|
||||
/**
|
||||
* Strict delimiter mode.
|
||||
* - When true, only explicit TeX delimiters are recognized as math:
|
||||
* inline: `$...$` and `\\(...\\)`; block: `$$...$$` and `\\[...\\]`.
|
||||
*
|
||||
* Important: authors should write explicit TeX delimiters with escaped
|
||||
* backslashes in source (for example, write `\\(...\\)` rather than
|
||||
* an unescaped `\(...\)`). Unescaped `\(...\)` cannot be reliably
|
||||
* distinguished from ordinary parentheses and may not be parsed as math.
|
||||
* - Heuristics and mid-state (unclosed) math detection are disabled.
|
||||
*/
|
||||
strictDelimiters?: boolean
|
||||
}
|
||||
|
||||
let defaultMathOptions: MathOptions | undefined
|
||||
|
||||
export function setDefaultMathOptions(opts: MathOptions | undefined) {
|
||||
defaultMathOptions = opts
|
||||
}
|
||||
|
||||
export function getDefaultMathOptions(): MathOptions | undefined {
|
||||
return defaultMathOptions
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
import { STANDARD_HTML_TAGS } from './htmlTags'
|
||||
import { escapeTagForRegExp } from './htmlTagUtils'
|
||||
|
||||
const HTML_LIKE_TAG_NAME_RE = /^[a-z][a-z0-9_-]*$/
|
||||
|
||||
export function isHtmlLikeTagName(tag: string) {
|
||||
return HTML_LIKE_TAG_NAME_RE.test(String(tag ?? '').trim().toLowerCase())
|
||||
}
|
||||
|
||||
export function normalizeCustomHtmlTagName(value: unknown): string {
|
||||
const raw = String(value ?? '').trim()
|
||||
if (!raw)
|
||||
return ''
|
||||
|
||||
if (!raw.startsWith('<'))
|
||||
return isHtmlLikeTagName(raw) ? raw.toLowerCase() : ''
|
||||
|
||||
let index = 1
|
||||
while (index < raw.length && /\s/.test(raw[index]))
|
||||
index++
|
||||
if (raw[index] === '/') {
|
||||
index++
|
||||
while (index < raw.length && /\s/.test(raw[index]))
|
||||
index++
|
||||
}
|
||||
|
||||
const start = index
|
||||
while (index < raw.length && /[\w-]/.test(raw[index]))
|
||||
index++
|
||||
|
||||
const normalized = raw.slice(start, index).toLowerCase()
|
||||
const next = raw[index] ?? ''
|
||||
if (next && !/[\s/>]/.test(next))
|
||||
return ''
|
||||
return isHtmlLikeTagName(normalized) ? normalized : ''
|
||||
}
|
||||
|
||||
export function normalizeCustomHtmlTags(tags?: readonly string[]) {
|
||||
if (!tags || tags.length === 0)
|
||||
return []
|
||||
|
||||
const seen = new Set<string>()
|
||||
const normalized: string[] = []
|
||||
for (const tag of tags) {
|
||||
const value = normalizeCustomHtmlTagName(tag)
|
||||
if (!value || seen.has(value))
|
||||
continue
|
||||
seen.add(value)
|
||||
normalized.push(value)
|
||||
}
|
||||
return normalized
|
||||
}
|
||||
|
||||
export function mergeCustomHtmlTags(...lists: Array<readonly string[] | undefined>) {
|
||||
const seen = new Set<string>()
|
||||
const normalized: string[] = []
|
||||
|
||||
for (const list of lists) {
|
||||
for (const tag of normalizeCustomHtmlTags(list)) {
|
||||
if (seen.has(tag))
|
||||
continue
|
||||
seen.add(tag)
|
||||
normalized.push(tag)
|
||||
}
|
||||
}
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
export function resolveCustomHtmlTags(tags?: readonly string[]) {
|
||||
const normalized = normalizeCustomHtmlTags(tags)
|
||||
return {
|
||||
key: normalized.join(','),
|
||||
tags: normalized,
|
||||
}
|
||||
}
|
||||
|
||||
export function getHtmlTagFromContent(html: unknown) {
|
||||
return normalizeCustomHtmlTagName(html)
|
||||
}
|
||||
|
||||
export function hasCompleteHtmlTagContent(html: unknown, tag: string) {
|
||||
const raw = String(html ?? '')
|
||||
const normalizedTag = normalizeCustomHtmlTagName(tag)
|
||||
if (!normalizedTag)
|
||||
return false
|
||||
|
||||
const escaped = escapeTagForRegExp(normalizedTag)
|
||||
const openMatch = raw.match(new RegExp(String.raw`^\s*<\s*${escaped}(?:\s[^>]*)?(\s*\/)?>`, 'i'))
|
||||
if (!openMatch)
|
||||
return false
|
||||
if (openMatch[1])
|
||||
return true
|
||||
|
||||
return new RegExp(String.raw`<\s*\/\s*${escaped}\s*>`, 'i').test(raw)
|
||||
}
|
||||
|
||||
export function shouldRenderUnknownHtmlTagAsText(html: unknown, tag: string) {
|
||||
const normalizedTag = normalizeCustomHtmlTagName(tag)
|
||||
return Boolean(normalizedTag)
|
||||
&& !STANDARD_HTML_TAGS.has(normalizedTag)
|
||||
&& !hasCompleteHtmlTagContent(html, normalizedTag)
|
||||
}
|
||||
|
||||
export function stripCustomHtmlWrapper(html: unknown, tag: string) {
|
||||
const raw = String(html ?? '')
|
||||
const normalizedTag = normalizeCustomHtmlTagName(tag)
|
||||
if (!normalizedTag)
|
||||
return raw
|
||||
|
||||
const escaped = escapeTagForRegExp(normalizedTag)
|
||||
const openRe = new RegExp(String.raw`^\s*<\s*${escaped}(?:\s[^>]*)?>\s*`, 'i')
|
||||
const closeRe = new RegExp(String.raw`\s*<\s*\/\s*${escaped}\s*>\s*$`, 'i')
|
||||
return raw.replace(openRe, '').replace(closeRe, '')
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
import type { MathOptions } from './config'
|
||||
import MarkdownIt from 'markdown-it-ts'
|
||||
import { getDefaultMathOptions } from './config'
|
||||
import { applyContainers } from './plugins/containers'
|
||||
import { applyFixHtmlInlineTokens } from './plugins/fixHtmlInline'
|
||||
import { applyFixIndentedCodeBlock } from './plugins/fixIndentedCodeBlock'
|
||||
import { applyFixLinkTokens } from './plugins/fixLinkTokens'
|
||||
import { applyFixListItem } from './plugins/fixListItem'
|
||||
import { applyFixStrongTokens } from './plugins/fixStrongTokens'
|
||||
import { applyFixTableTokens } from './plugins/fixTableTokens'
|
||||
import { applyMath } from './plugins/math'
|
||||
import { applyRenderRules } from './renderers'
|
||||
|
||||
export interface FactoryOptions extends Record<string, unknown> {
|
||||
markdownItOptions?: Record<string, unknown>
|
||||
enableMath?: boolean
|
||||
enableContainers?: boolean
|
||||
mathOptions?: { commands?: string[], escapeExclamation?: boolean }
|
||||
/**
|
||||
* Custom HTML-like tag names that should participate in streaming mid-state
|
||||
* suppression and be emitted as custom nodes (e.g. ['thinking']).
|
||||
*/
|
||||
customHtmlTags?: readonly string[]
|
||||
/**
|
||||
* Whether to enable the fix for indented code blocks that should be paragraphs.
|
||||
* Default: true
|
||||
*/
|
||||
enableFixIndentedCodeBlock?: boolean
|
||||
}
|
||||
|
||||
export function factory(opts: FactoryOptions = {}) {
|
||||
const md = new MarkdownIt({
|
||||
html: true,
|
||||
linkify: true,
|
||||
typographer: true,
|
||||
stream: true,
|
||||
...(opts.markdownItOptions ?? {}),
|
||||
})
|
||||
|
||||
if (opts.enableMath ?? true) {
|
||||
const mergedMathOptions: MathOptions = { ...(getDefaultMathOptions() ?? {}), ...(opts.mathOptions ?? {}) }
|
||||
applyMath(md, mergedMathOptions)
|
||||
}
|
||||
if (opts.enableContainers ?? true)
|
||||
applyContainers(md)
|
||||
// Fix indented code blocks that should be paragraphs (streaming scenario)
|
||||
if (opts.enableFixIndentedCodeBlock !== false)
|
||||
applyFixIndentedCodeBlock(md)
|
||||
// Retain the core-stage fix as a fallback for any cases the inline
|
||||
// tokenizer does not handle.
|
||||
applyFixLinkTokens(md)
|
||||
// Also apply strong-token normalization at the same stage.
|
||||
applyFixStrongTokens(md)
|
||||
// Apply list-item inline normalization as well.
|
||||
applyFixListItem(md)
|
||||
// Apply table token normalization at block stage.
|
||||
applyFixTableTokens(md)
|
||||
applyRenderRules(md)
|
||||
applyFixHtmlInlineTokens(md, {
|
||||
customHtmlTags: opts.customHtmlTags,
|
||||
})
|
||||
|
||||
return md
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
export function findMatchingClose(src: string, startIdx: number, open: string, close: string) {
|
||||
const len = src.length
|
||||
// Special-case $$ since it's a two-char delimiter that shouldn't
|
||||
// be interpreted as nested parentheses.
|
||||
if (open === '$$' && close === '$$') {
|
||||
let i = startIdx
|
||||
while (i < len - 1) {
|
||||
if (src[i] === '$' && src[i + 1] === '$') {
|
||||
// ensure not escaped
|
||||
let k = i - 1
|
||||
let backslashes = 0
|
||||
while (k >= 0 && src[k] === '\\') {
|
||||
backslashes++
|
||||
k--
|
||||
}
|
||||
if (backslashes % 2 === 0)
|
||||
return i
|
||||
}
|
||||
i++
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
const openChar = open[open.length - 1]
|
||||
const closeSeq = close
|
||||
let depth = 0
|
||||
let i = startIdx
|
||||
while (i < len) {
|
||||
// If there's an unescaped close sequence here
|
||||
if (src.slice(i, i + closeSeq.length) === closeSeq) {
|
||||
let k = i - 1
|
||||
let backslashes = 0
|
||||
while (k >= 0 && src[k] === '\\') {
|
||||
backslashes++
|
||||
k--
|
||||
}
|
||||
if (backslashes % 2 === 0) {
|
||||
if (depth === 0)
|
||||
return i
|
||||
depth--
|
||||
i += closeSeq.length
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
const ch = src[i]
|
||||
// skip escaped characters
|
||||
if (ch === '\\') {
|
||||
i += 2
|
||||
continue
|
||||
}
|
||||
|
||||
if (ch === openChar) {
|
||||
depth++
|
||||
}
|
||||
else if (ch === closeSeq[closeSeq.length - 1]) {
|
||||
if (depth > 0)
|
||||
depth--
|
||||
}
|
||||
i++
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
export default findMatchingClose
|
||||
@@ -0,0 +1,428 @@
|
||||
import {
|
||||
BLOCKED_HTML_TAGS,
|
||||
DANGEROUS_HTML_ATTRS,
|
||||
EXTENDED_STANDARD_HTML_TAGS,
|
||||
isUnsafeHtmlUrl,
|
||||
URL_HTML_ATTRS,
|
||||
VOID_HTML_TAGS,
|
||||
} from './htmlTags'
|
||||
|
||||
export interface HtmlToken {
|
||||
type: 'text' | 'tag_open' | 'tag_close' | 'self_closing'
|
||||
tagName?: string
|
||||
attrs?: Record<string, string>
|
||||
content?: string
|
||||
}
|
||||
|
||||
const CUSTOM_TAG_REGEX = /<([a-z][a-z0-9-]*)\b[^>]*>/gi
|
||||
|
||||
function hasOwn(obj: Record<string, unknown>, key: string) {
|
||||
return Object.prototype.hasOwnProperty.call(obj, key)
|
||||
}
|
||||
|
||||
function getString(value: unknown): string {
|
||||
return typeof value === 'string'
|
||||
? value
|
||||
: value == null
|
||||
? ''
|
||||
: String(value)
|
||||
}
|
||||
|
||||
function isSafeAttrName(value: string): boolean {
|
||||
return /^[^\s"'<>`=]+$/.test(value) && !/^on/i.test(value)
|
||||
}
|
||||
|
||||
function escapeHtml(value: unknown): string {
|
||||
return getString(value)
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>')
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, ''')
|
||||
}
|
||||
|
||||
function escapeAttr(value: unknown): string {
|
||||
return escapeHtml(value).replace(/`/g, '`')
|
||||
}
|
||||
|
||||
function normalizeTagName(tagName: string | undefined): string {
|
||||
return String(tagName ?? '').trim().toLowerCase()
|
||||
}
|
||||
|
||||
function serializeAttrs(attrs: Record<string, string>): string {
|
||||
const pairs = Object.entries(attrs)
|
||||
if (pairs.length === 0)
|
||||
return ''
|
||||
|
||||
return pairs
|
||||
.map(([name, value]) => value === '' ? ` ${name}` : ` ${name}="${escapeAttr(value)}"`)
|
||||
.join('')
|
||||
}
|
||||
|
||||
function sanitizeHtmlContentAttrs(attrs: Record<string, string>) {
|
||||
const clean: Record<string, string> = {}
|
||||
|
||||
for (const [key, value] of Object.entries(attrs)) {
|
||||
const safeName = key.trim()
|
||||
const lowerKey = safeName.toLowerCase()
|
||||
if (!safeName || !isSafeAttrName(safeName))
|
||||
continue
|
||||
if (DANGEROUS_HTML_ATTRS.has(lowerKey))
|
||||
continue
|
||||
if (URL_HTML_ATTRS.has(lowerKey) && value && isUnsafeHtmlUrl(value))
|
||||
continue
|
||||
clean[safeName] = value
|
||||
}
|
||||
|
||||
return clean
|
||||
}
|
||||
|
||||
export function isCustomHtmlComponentTag(
|
||||
tagName: string,
|
||||
customComponents: Record<string, unknown>,
|
||||
) {
|
||||
const lowerTag = tagName.toLowerCase()
|
||||
if (EXTENDED_STANDARD_HTML_TAGS.has(lowerTag))
|
||||
return false
|
||||
return hasOwn(customComponents, lowerTag) || hasOwn(customComponents, tagName)
|
||||
}
|
||||
|
||||
export function sanitizeHtmlAttrs(attrs: Record<string, string>) {
|
||||
const clean: Record<string, string> = {}
|
||||
for (const [key, value] of Object.entries(attrs)) {
|
||||
const lowerKey = key.toLowerCase()
|
||||
if (DANGEROUS_HTML_ATTRS.has(lowerKey))
|
||||
continue
|
||||
if (URL_HTML_ATTRS.has(lowerKey) && value && isUnsafeHtmlUrl(value))
|
||||
continue
|
||||
clean[key] = value
|
||||
}
|
||||
return clean
|
||||
}
|
||||
|
||||
export function tokenAttrsToRecord(attrs?: Array<[string, string | null]> | null) {
|
||||
const record: Record<string, string> = {}
|
||||
if (!Array.isArray(attrs) || attrs.length === 0)
|
||||
return record
|
||||
|
||||
for (const [key, value] of attrs) {
|
||||
if (!key)
|
||||
continue
|
||||
record[String(key)] = value == null ? '' : String(value)
|
||||
}
|
||||
|
||||
return record
|
||||
}
|
||||
|
||||
export function sanitizeHtmlTokenAttrs(attrs?: Array<[string, string | null]> | null) {
|
||||
const sanitized = sanitizeHtmlAttrs(tokenAttrsToRecord(attrs))
|
||||
const pairs = Object.entries(sanitized).map(([key, value]) => [key, value] as [string, string])
|
||||
return pairs.length > 0 ? pairs : undefined
|
||||
}
|
||||
|
||||
export function convertHtmlPropValue(value: string, key: string): any {
|
||||
const lowerKey = key.toLowerCase()
|
||||
|
||||
if (['checked', 'disabled', 'readonly', 'required', 'autofocus', 'multiple', 'hidden'].includes(lowerKey))
|
||||
return value === 'true' || value === '' || value === key
|
||||
|
||||
if (['value', 'min', 'max', 'step', 'width', 'height', 'size', 'maxlength'].includes(lowerKey)) {
|
||||
const num = Number(value)
|
||||
if (value !== '' && !Number.isNaN(num))
|
||||
return num
|
||||
}
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
export function convertHtmlAttrsToProps(attrs: Record<string, string>) {
|
||||
const result: Record<string, any> = {}
|
||||
for (const [key, value] of Object.entries(attrs))
|
||||
result[key] = convertHtmlPropValue(value, key)
|
||||
return result
|
||||
}
|
||||
|
||||
function isMeaningfulText(text: string) {
|
||||
return text.trim().length > 0
|
||||
}
|
||||
|
||||
export function tokenizeHtml(html: string): HtmlToken[] {
|
||||
const tokens: HtmlToken[] = []
|
||||
let pos = 0
|
||||
|
||||
while (pos < html.length) {
|
||||
if (html.startsWith('<!--', pos)) {
|
||||
const commentEnd = html.indexOf('-->', pos)
|
||||
if (commentEnd !== -1) {
|
||||
pos = commentEnd + 3
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
const tagStart = html.indexOf('<', pos)
|
||||
if (tagStart === -1) {
|
||||
if (pos < html.length) {
|
||||
const remainingText = html.slice(pos)
|
||||
if (isMeaningfulText(remainingText))
|
||||
tokens.push({ type: 'text', content: remainingText })
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
if (tagStart > pos) {
|
||||
const textContent = html.slice(pos, tagStart)
|
||||
if (isMeaningfulText(textContent))
|
||||
tokens.push({ type: 'text', content: textContent })
|
||||
}
|
||||
|
||||
if (html.startsWith('![CDATA[', tagStart + 1)) {
|
||||
const cdataEnd = html.indexOf(']]>', tagStart)
|
||||
if (cdataEnd !== -1) {
|
||||
tokens.push({ type: 'text', content: html.slice(tagStart, cdataEnd + 3) })
|
||||
pos = cdataEnd + 3
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
if (html.startsWith('!', tagStart + 1)) {
|
||||
const specialEnd = html.indexOf('>', tagStart)
|
||||
if (specialEnd !== -1) {
|
||||
pos = specialEnd + 1
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
const tagEnd = html.indexOf('>', tagStart)
|
||||
if (tagEnd === -1)
|
||||
break
|
||||
|
||||
const tagContent = html.slice(tagStart + 1, tagEnd).trim()
|
||||
const isClosingTag = tagContent.startsWith('/')
|
||||
const isSelfClosing = tagContent.endsWith('/')
|
||||
|
||||
if (isClosingTag) {
|
||||
const tagName = tagContent.slice(1).trim()
|
||||
tokens.push({ type: 'tag_close', tagName })
|
||||
}
|
||||
else {
|
||||
const spaceIndex = tagContent.indexOf(' ')
|
||||
let tagName: string
|
||||
let attrsStr = ''
|
||||
|
||||
if (spaceIndex === -1) {
|
||||
tagName = isSelfClosing ? tagContent.slice(0, -1).trim() : tagContent.trim()
|
||||
}
|
||||
else {
|
||||
tagName = tagContent.slice(0, spaceIndex).trim()
|
||||
attrsStr = tagContent.slice(spaceIndex + 1)
|
||||
}
|
||||
|
||||
const attrs: Record<string, string> = {}
|
||||
if (attrsStr) {
|
||||
const attrRegex = /([^\s=]+)(?:=(?:"([^"]*)"|'([^']*)'|(\S*)))?/g
|
||||
let attrMatch: RegExpExecArray | null
|
||||
while ((attrMatch = attrRegex.exec(attrsStr)) !== null) {
|
||||
const name = attrMatch[1]
|
||||
const value = attrMatch[2] ?? attrMatch[3] ?? attrMatch[4] ?? ''
|
||||
if (name && !name.endsWith('/'))
|
||||
attrs[name] = value
|
||||
}
|
||||
}
|
||||
|
||||
tokens.push({
|
||||
type: isSelfClosing || VOID_HTML_TAGS.has(tagName.toLowerCase()) ? 'self_closing' : 'tag_open',
|
||||
tagName,
|
||||
attrs,
|
||||
})
|
||||
}
|
||||
|
||||
pos = tagEnd + 1
|
||||
}
|
||||
|
||||
return tokens
|
||||
}
|
||||
|
||||
function tokenizeHtmlPreservingText(html: string): HtmlToken[] {
|
||||
const tokens: HtmlToken[] = []
|
||||
let pos = 0
|
||||
|
||||
while (pos < html.length) {
|
||||
if (html.startsWith('<!--', pos)) {
|
||||
const commentEnd = html.indexOf('-->', pos)
|
||||
if (commentEnd !== -1) {
|
||||
pos = commentEnd + 3
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
const tagStart = html.indexOf('<', pos)
|
||||
if (tagStart === -1) {
|
||||
if (pos < html.length)
|
||||
tokens.push({ type: 'text', content: html.slice(pos) })
|
||||
break
|
||||
}
|
||||
|
||||
if (tagStart > pos)
|
||||
tokens.push({ type: 'text', content: html.slice(pos, tagStart) })
|
||||
|
||||
if (html.startsWith('![CDATA[', tagStart + 1)) {
|
||||
const cdataEnd = html.indexOf(']]>', tagStart)
|
||||
if (cdataEnd !== -1) {
|
||||
tokens.push({ type: 'text', content: html.slice(tagStart, cdataEnd + 3) })
|
||||
pos = cdataEnd + 3
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
if (html.startsWith('!', tagStart + 1)) {
|
||||
const specialEnd = html.indexOf('>', tagStart)
|
||||
if (specialEnd !== -1) {
|
||||
pos = specialEnd + 1
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
const tagEnd = html.indexOf('>', tagStart)
|
||||
if (tagEnd === -1)
|
||||
break
|
||||
|
||||
const tagContent = html.slice(tagStart + 1, tagEnd).trim()
|
||||
if (!tagContent) {
|
||||
pos = tagEnd + 1
|
||||
continue
|
||||
}
|
||||
|
||||
const isClosingTag = tagContent.startsWith('/')
|
||||
const isSelfClosing = tagContent.endsWith('/')
|
||||
|
||||
if (isClosingTag) {
|
||||
const tagName = tagContent.slice(1).trim()
|
||||
tokens.push({ type: 'tag_close', tagName })
|
||||
pos = tagEnd + 1
|
||||
continue
|
||||
}
|
||||
|
||||
const spaceIndex = tagContent.indexOf(' ')
|
||||
let tagName = ''
|
||||
let attrsStr = ''
|
||||
if (spaceIndex === -1) {
|
||||
tagName = isSelfClosing ? tagContent.slice(0, -1).trim() : tagContent.trim()
|
||||
}
|
||||
else {
|
||||
tagName = tagContent.slice(0, spaceIndex).trim()
|
||||
attrsStr = tagContent.slice(spaceIndex + 1)
|
||||
}
|
||||
|
||||
const attrs: Record<string, string> = {}
|
||||
if (attrsStr) {
|
||||
const attrRegex = /([^\s=]+)(?:=(?:"([^"]*)"|'([^']*)'|(\S*)))?/g
|
||||
let attrMatch: RegExpExecArray | null
|
||||
while ((attrMatch = attrRegex.exec(attrsStr)) !== null) {
|
||||
const name = attrMatch[1]
|
||||
const value = attrMatch[2] ?? attrMatch[3] ?? attrMatch[4] ?? ''
|
||||
if (name && !name.endsWith('/'))
|
||||
attrs[name] = value
|
||||
}
|
||||
}
|
||||
|
||||
tokens.push({
|
||||
type: isSelfClosing || VOID_HTML_TAGS.has(tagName.toLowerCase()) ? 'self_closing' : 'tag_open',
|
||||
tagName,
|
||||
attrs,
|
||||
})
|
||||
|
||||
pos = tagEnd + 1
|
||||
}
|
||||
|
||||
return tokens
|
||||
}
|
||||
|
||||
export function hasCustomHtmlComponents(
|
||||
content: string,
|
||||
customComponents: Record<string, unknown>,
|
||||
) {
|
||||
if (!content || !content.includes('<'))
|
||||
return false
|
||||
if (!customComponents || Object.keys(customComponents).length === 0)
|
||||
return false
|
||||
CUSTOM_TAG_REGEX.lastIndex = 0
|
||||
let match: RegExpExecArray | null
|
||||
while ((match = CUSTOM_TAG_REGEX.exec(content)) !== null) {
|
||||
if (isCustomHtmlComponentTag(match[1], customComponents))
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
export function sanitizeHtmlContent(content: string): string {
|
||||
if (!content)
|
||||
return ''
|
||||
|
||||
const tokens = tokenizeHtmlPreservingText(content)
|
||||
const stack: string[] = []
|
||||
const output: string[] = []
|
||||
let blockedDepth = 0
|
||||
|
||||
for (const token of tokens) {
|
||||
if (token.type === 'text') {
|
||||
if (blockedDepth === 0)
|
||||
output.push(escapeHtml(token.content ?? ''))
|
||||
continue
|
||||
}
|
||||
|
||||
const tagName = normalizeTagName(token.tagName)
|
||||
if (!tagName)
|
||||
continue
|
||||
|
||||
if (BLOCKED_HTML_TAGS.has(tagName)) {
|
||||
if (token.type === 'tag_open')
|
||||
blockedDepth += 1
|
||||
else if (token.type === 'tag_close' && blockedDepth > 0)
|
||||
blockedDepth -= 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (blockedDepth > 0)
|
||||
continue
|
||||
|
||||
if (token.type === 'self_closing') {
|
||||
output.push(`<${tagName}${serializeAttrs(sanitizeHtmlContentAttrs(token.attrs ?? {}))}>`)
|
||||
continue
|
||||
}
|
||||
|
||||
if (token.type === 'tag_open') {
|
||||
output.push(`<${tagName}${serializeAttrs(sanitizeHtmlContentAttrs(token.attrs ?? {}))}>`)
|
||||
if (!VOID_HTML_TAGS.has(tagName))
|
||||
stack.push(tagName)
|
||||
continue
|
||||
}
|
||||
|
||||
const matchedIndex = stack.lastIndexOf(tagName)
|
||||
if (matchedIndex === -1)
|
||||
continue
|
||||
|
||||
while (stack.length > matchedIndex + 1) {
|
||||
const danglingTag = stack.pop()
|
||||
if (danglingTag)
|
||||
output.push(`</${danglingTag}>`)
|
||||
}
|
||||
|
||||
const closingTag = stack.pop()
|
||||
if (closingTag)
|
||||
output.push(`</${closingTag}>`)
|
||||
}
|
||||
|
||||
while (stack.length > 0) {
|
||||
const danglingTag = stack.pop()
|
||||
if (danglingTag)
|
||||
output.push(`</${danglingTag}>`)
|
||||
}
|
||||
|
||||
return output.join('')
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
export function escapeTagForRegExp(tag: string) {
|
||||
return tag.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
|
||||
}
|
||||
|
||||
export function findTagCloseIndexOutsideQuotes(input: string) {
|
||||
let inSingle = false
|
||||
let inDouble = false
|
||||
|
||||
for (let i = 0; i < input.length; i++) {
|
||||
const ch = input[i]
|
||||
if (ch === '\\') {
|
||||
i++
|
||||
continue
|
||||
}
|
||||
if (!inDouble && ch === '\'') {
|
||||
inSingle = !inSingle
|
||||
continue
|
||||
}
|
||||
if (!inSingle && ch === '"') {
|
||||
inDouble = !inDouble
|
||||
continue
|
||||
}
|
||||
if (!inSingle && !inDouble && ch === '>')
|
||||
return i
|
||||
}
|
||||
|
||||
return -1
|
||||
}
|
||||
|
||||
export function parseTagAttrs(openTag: string): [string, string][] {
|
||||
const attrs: [string, string][] = []
|
||||
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
|
||||
let match
|
||||
|
||||
while ((match = attrRegex.exec(openTag)) !== null) {
|
||||
const attrName = match[1]
|
||||
if (!attrName)
|
||||
continue
|
||||
const attrValue = match[2] || match[3] || match[4] || ''
|
||||
attrs.push([attrName, attrValue])
|
||||
}
|
||||
|
||||
return attrs
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
export const VOID_HTML_TAG_NAMES = [
|
||||
'area',
|
||||
'base',
|
||||
'br',
|
||||
'col',
|
||||
'embed',
|
||||
'hr',
|
||||
'img',
|
||||
'input',
|
||||
'link',
|
||||
'meta',
|
||||
'param',
|
||||
'source',
|
||||
'track',
|
||||
'wbr',
|
||||
] as const
|
||||
|
||||
export const INLINE_HTML_TAG_NAMES = [
|
||||
'a',
|
||||
'abbr',
|
||||
'b',
|
||||
'bdi',
|
||||
'bdo',
|
||||
'button',
|
||||
'cite',
|
||||
'code',
|
||||
'data',
|
||||
'del',
|
||||
'dfn',
|
||||
'em',
|
||||
'font',
|
||||
'i',
|
||||
'ins',
|
||||
'kbd',
|
||||
'label',
|
||||
'mark',
|
||||
'q',
|
||||
's',
|
||||
'samp',
|
||||
'small',
|
||||
'span',
|
||||
'strong',
|
||||
'sub',
|
||||
'sup',
|
||||
'time',
|
||||
'u',
|
||||
'var',
|
||||
] as const
|
||||
|
||||
export const BLOCK_HTML_TAG_NAMES = [
|
||||
'article',
|
||||
'aside',
|
||||
'blockquote',
|
||||
'details',
|
||||
'div',
|
||||
'figcaption',
|
||||
'figure',
|
||||
'footer',
|
||||
'header',
|
||||
'h1',
|
||||
'h2',
|
||||
'h3',
|
||||
'h4',
|
||||
'h5',
|
||||
'h6',
|
||||
'li',
|
||||
'main',
|
||||
'nav',
|
||||
'ol',
|
||||
'p',
|
||||
'pre',
|
||||
'section',
|
||||
'summary',
|
||||
'table',
|
||||
'tbody',
|
||||
'td',
|
||||
'th',
|
||||
'thead',
|
||||
'tr',
|
||||
'ul',
|
||||
] as const
|
||||
|
||||
export const SVG_HTML_TAG_NAMES = [
|
||||
'svg',
|
||||
'g',
|
||||
'path',
|
||||
] as const
|
||||
|
||||
export const EXTENDED_STANDARD_HTML_TAG_NAMES = [
|
||||
'address',
|
||||
'audio',
|
||||
'body',
|
||||
'canvas',
|
||||
'caption',
|
||||
'colgroup',
|
||||
'datalist',
|
||||
'dd',
|
||||
'dialog',
|
||||
'dl',
|
||||
'dt',
|
||||
'fieldset',
|
||||
'form',
|
||||
'head',
|
||||
'hgroup',
|
||||
'html',
|
||||
'iframe',
|
||||
'legend',
|
||||
'map',
|
||||
'menu',
|
||||
'meter',
|
||||
'noscript',
|
||||
'object',
|
||||
'optgroup',
|
||||
'option',
|
||||
'output',
|
||||
'picture',
|
||||
'progress',
|
||||
'rp',
|
||||
'rt',
|
||||
'ruby',
|
||||
'script',
|
||||
'select',
|
||||
'style',
|
||||
'template',
|
||||
'textarea',
|
||||
'tfoot',
|
||||
'title',
|
||||
'video',
|
||||
] as const
|
||||
|
||||
export const DANGEROUS_HTML_ATTR_NAMES = [
|
||||
'onclick',
|
||||
'onerror',
|
||||
'onload',
|
||||
'onmouseover',
|
||||
'onmouseout',
|
||||
'onmousedown',
|
||||
'onmouseup',
|
||||
'onkeydown',
|
||||
'onkeyup',
|
||||
'onfocus',
|
||||
'onblur',
|
||||
'onsubmit',
|
||||
'onreset',
|
||||
'onchange',
|
||||
'onselect',
|
||||
'ondblclick',
|
||||
'ontouchstart',
|
||||
'ontouchend',
|
||||
'ontouchmove',
|
||||
'ontouchcancel',
|
||||
'onwheel',
|
||||
'onscroll',
|
||||
'oncopy',
|
||||
'oncut',
|
||||
'onpaste',
|
||||
'oninput',
|
||||
'oninvalid',
|
||||
'onsearch',
|
||||
] as const
|
||||
|
||||
export const URL_HTML_ATTR_NAMES = [
|
||||
'href',
|
||||
'src',
|
||||
'srcset',
|
||||
'xlink:href',
|
||||
'formaction',
|
||||
] as const
|
||||
|
||||
export const BLOCKED_HTML_TAG_NAMES = [
|
||||
'script',
|
||||
] as const
|
||||
|
||||
export const NON_STRUCTURING_HTML_TAG_NAMES = [
|
||||
'pre',
|
||||
'script',
|
||||
'style',
|
||||
'textarea',
|
||||
'title',
|
||||
] as const
|
||||
|
||||
export const VOID_HTML_TAGS = new Set<string>(VOID_HTML_TAG_NAMES)
|
||||
export const STANDARD_BLOCK_HTML_TAGS = new Set<string>(BLOCK_HTML_TAG_NAMES)
|
||||
export const STANDARD_HTML_TAGS = new Set<string>([
|
||||
...VOID_HTML_TAG_NAMES,
|
||||
...INLINE_HTML_TAG_NAMES,
|
||||
...BLOCK_HTML_TAG_NAMES,
|
||||
...SVG_HTML_TAG_NAMES,
|
||||
])
|
||||
export const EXTENDED_STANDARD_HTML_TAGS = new Set<string>([
|
||||
...STANDARD_HTML_TAGS,
|
||||
...EXTENDED_STANDARD_HTML_TAG_NAMES,
|
||||
])
|
||||
export const DANGEROUS_HTML_ATTRS = new Set<string>(DANGEROUS_HTML_ATTR_NAMES)
|
||||
export const URL_HTML_ATTRS = new Set<string>(URL_HTML_ATTR_NAMES)
|
||||
export const BLOCKED_HTML_TAGS = new Set<string>(BLOCKED_HTML_TAG_NAMES)
|
||||
export const NON_STRUCTURING_HTML_TAGS = new Set<string>(NON_STRUCTURING_HTML_TAG_NAMES)
|
||||
|
||||
export function stripHtmlControlAndWhitespace(value: string) {
|
||||
let out = ''
|
||||
for (const ch of value) {
|
||||
const code = ch.charCodeAt(0)
|
||||
if (code <= 0x1F || code === 0x7F)
|
||||
continue
|
||||
if (/\s/u.test(ch))
|
||||
continue
|
||||
out += ch
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
export function isUnsafeHtmlUrl(value: string) {
|
||||
const normalized = stripHtmlControlAndWhitespace(value).toLowerCase()
|
||||
|
||||
if (normalized.startsWith('javascript:') || normalized.startsWith('vbscript:'))
|
||||
return true
|
||||
|
||||
if (normalized.startsWith('data:')) {
|
||||
return !(
|
||||
normalized.startsWith('data:image/')
|
||||
|| normalized.startsWith('data:video/')
|
||||
|| normalized.startsWith('data:audio/')
|
||||
)
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
import type { MarkdownIt, MarkdownItPlugin } from 'markdown-it-ts'
|
||||
import type { FactoryOptions } from './factory'
|
||||
import markdownItFootnote from 'markdown-it-footnote'
|
||||
import markdownItIns from 'markdown-it-ins'
|
||||
import markdownItMark from 'markdown-it-mark'
|
||||
import markdownItSub from 'markdown-it-sub'
|
||||
|
||||
import markdownItSup from 'markdown-it-sup'
|
||||
import * as markdownItCheckbox from 'markdown-it-task-checkbox'
|
||||
import { factory } from './factory'
|
||||
import {
|
||||
parseInlineTokens,
|
||||
parseMarkdownToStructure,
|
||||
processTokens,
|
||||
} from './parser'
|
||||
|
||||
// Module-level registry for callers that want to add plugins to every
|
||||
// `getMarkdown()` instance without modifying call sites. Useful for tests
|
||||
// or apps that want a central place to `use` plugins.
|
||||
const _registeredMarkdownPlugins: Array<unknown> = []
|
||||
|
||||
export function registerMarkdownPlugin(plugin: unknown) {
|
||||
_registeredMarkdownPlugins.push(plugin)
|
||||
}
|
||||
|
||||
export function clearRegisteredMarkdownPlugins() {
|
||||
_registeredMarkdownPlugins.length = 0
|
||||
}
|
||||
|
||||
// Re-export config
|
||||
export { setDefaultMathOptions } from './config'
|
||||
|
||||
// Re-export parser functions
|
||||
export { parseInlineTokens, parseMarkdownToStructure, processTokens }
|
||||
export type { MathOptions } from './config'
|
||||
export type { MarkdownIt }
|
||||
|
||||
// Re-export utilities
|
||||
export * from './customHtmlTags'
|
||||
export { findMatchingClose } from './findMatchingClose'
|
||||
export * from './htmlRenderUtils'
|
||||
export * from './htmlTags'
|
||||
export { parseFenceToken } from './parser/inline-parsers/fence-parser'
|
||||
|
||||
// Re-export plugins
|
||||
export { applyContainers } from './plugins/containers'
|
||||
export { ESCAPED_TEX_BRACE_COMMANDS, isMathLike, TEX_BRACE_COMMANDS } from './plugins/isMathLike'
|
||||
export { applyMath, KATEX_COMMANDS, normalizeStandaloneBackslashT } from './plugins/math'
|
||||
// Re-export the node types for backward compatibility
|
||||
export * from './types'
|
||||
|
||||
export interface GetMarkdownOptions extends FactoryOptions {
|
||||
plugin?: Array<unknown>
|
||||
apply?: Array<(md: MarkdownIt) => void>
|
||||
/**
|
||||
* Custom translation function or translation map for UI texts
|
||||
* @default { 'common.copy': 'Copy' }
|
||||
*/
|
||||
i18n?: ((key: string) => string) | Record<string, string>
|
||||
}
|
||||
|
||||
export function getMarkdown(msgId: string = `editor-${Date.now()}`, options: GetMarkdownOptions = {}) {
|
||||
// keep legacy behaviour but delegate to new factory and reapply project-specific rules
|
||||
const md = factory(options)
|
||||
|
||||
// Setup i18n translator function
|
||||
const defaultTranslations: Record<string, string> = {
|
||||
'common.copy': 'Copy',
|
||||
}
|
||||
|
||||
let t: (key: string) => string
|
||||
if (typeof options.i18n === 'function') {
|
||||
t = options.i18n
|
||||
}
|
||||
else if (options.i18n && typeof options.i18n === 'object') {
|
||||
const i18nMap = options.i18n as Record<string, string>
|
||||
t = (key: string) => i18nMap[key] ?? defaultTranslations[key] ?? key
|
||||
}
|
||||
else {
|
||||
t = (key: string) => defaultTranslations[key] ?? key
|
||||
}
|
||||
|
||||
// apply user supplied plugins (md.use)
|
||||
if (Array.isArray(options.plugin)) {
|
||||
for (const p of options.plugin) {
|
||||
// allow both [plugin, opts] tuple or plugin function
|
||||
const pluginItem = p as unknown
|
||||
if (Array.isArray(pluginItem)) {
|
||||
const fn = pluginItem[0]
|
||||
const opts = pluginItem[1]
|
||||
if (typeof fn === 'function')
|
||||
md.use(fn, opts)
|
||||
}
|
||||
else if (typeof pluginItem === 'function') {
|
||||
md.use(pluginItem as MarkdownItPlugin)
|
||||
}
|
||||
// otherwise ignore non-callable plugins
|
||||
}
|
||||
}
|
||||
|
||||
// apply user supplied apply functions to mutate the md instance (e.g. md.block.ruler.before(...))
|
||||
if (Array.isArray(options.apply)) {
|
||||
for (const fn of options.apply) {
|
||||
try {
|
||||
fn(md)
|
||||
}
|
||||
catch (e) {
|
||||
// swallow errors to preserve legacy behaviour; developers can see stack in console
|
||||
|
||||
console.error('[getMarkdown] apply function threw an error', e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Apply any globally registered plugins (via registerMarkdownPlugin)
|
||||
if (_registeredMarkdownPlugins.length) {
|
||||
for (const p of _registeredMarkdownPlugins) {
|
||||
if (Array.isArray(p)) {
|
||||
const fn = p[0] as any
|
||||
const opts = p[1]
|
||||
if (typeof fn === 'function')
|
||||
md.use(fn, opts)
|
||||
}
|
||||
else if (typeof p === 'function') {
|
||||
md.use(p as MarkdownItPlugin)
|
||||
}
|
||||
// ignore non-callable registrations
|
||||
}
|
||||
}
|
||||
|
||||
// Re-apply a few project specific plugins that were previously always enabled
|
||||
md.use(markdownItSub)
|
||||
md.use(markdownItSup)
|
||||
md.use(markdownItMark)
|
||||
// Safely resolve default export or the module itself for checkbox plugin
|
||||
type CheckboxPluginFn = (md: MarkdownIt, opts?: unknown) => void
|
||||
const markdownItCheckboxPlugin = ((markdownItCheckbox as unknown) as {
|
||||
default?: CheckboxPluginFn
|
||||
}).default ?? markdownItCheckbox
|
||||
md.use(markdownItCheckboxPlugin)
|
||||
md.use(markdownItIns)
|
||||
md.use(markdownItFootnote)
|
||||
|
||||
// Annotate fence tokens with unclosed meta using a lightweight line check
|
||||
md.core.ruler.after('block', 'mark_fence_closed', (state: unknown) => {
|
||||
const s = state as unknown as {
|
||||
src: string
|
||||
env?: Record<string, unknown>
|
||||
tokens: Array<{ type?: string, map?: number[], markup?: string, meta?: Record<string, unknown> }>
|
||||
}
|
||||
const src: string = s.src
|
||||
const envFinal = !!s.env?.__markstreamFinal
|
||||
const lines = src.split(/\r?\n/)
|
||||
for (const token of s.tokens) {
|
||||
if (token.type !== 'fence' || !token.map || !token.markup)
|
||||
continue
|
||||
const openLine: number = token.map[0]
|
||||
const endLine: number = token.map[1]
|
||||
const markup: string = token.markup
|
||||
const marker = markup[0]
|
||||
const minLen = markup.length
|
||||
// The closing line, if exists, should be the last line consumed by the block
|
||||
const lineIdx = Math.max(0, endLine - 1)
|
||||
const line = lines[lineIdx] ?? ''
|
||||
let i = 0
|
||||
while (i < line.length && (line[i] === ' ' || line[i] === '\t')) i++
|
||||
let count = 0
|
||||
while (i + count < line.length && line[i + count] === marker) count++
|
||||
let j = i + count
|
||||
while (j < line.length && (line[j] === ' ' || line[j] === '\t')) j++
|
||||
const closed = envFinal ? true : (endLine > openLine + 1 && count >= minLen && j === line.length)
|
||||
const tokenShape = token as unknown as { meta?: Record<string, unknown> }
|
||||
tokenShape.meta = tokenShape.meta ?? {}
|
||||
; (tokenShape.meta as Record<string, unknown>).unclosed = !closed
|
||||
// also set a explicit `closed` boolean for compatibility with plugins/tests
|
||||
; (tokenShape.meta as Record<string, unknown>).closed = !!closed
|
||||
}
|
||||
})
|
||||
|
||||
// wave rule (legacy)
|
||||
const waveRule = (state: unknown, silent?: boolean) => {
|
||||
const s = state as unknown as { pos: number, src: string, push: (type: string, tag?: string, nesting?: number) => any }
|
||||
const start = s.pos
|
||||
if (s.src[start] !== '~')
|
||||
return false
|
||||
const prevChar = s.src[start - 1]
|
||||
const nextChar = s.src[start + 1]
|
||||
if (/\d/.test(prevChar) && /\d/.test(nextChar)) {
|
||||
if (!silent) {
|
||||
const token = s.push('text', '', 0)
|
||||
token.content = '~'
|
||||
}
|
||||
s.pos += 1
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
md.inline.ruler.before('sub', 'wave', waveRule)
|
||||
|
||||
// custom fence that uses msgId for unique ids
|
||||
md.renderer.rules.fence = (tokens: unknown, idx: number) => {
|
||||
const tokensAny = tokens as unknown as import('./types').MarkdownToken[]
|
||||
const token = tokensAny[idx]
|
||||
const tokenShape = token as unknown as { info?: string, content?: string }
|
||||
const info = String(tokenShape.info ?? '').trim()
|
||||
const str = String(tokenShape.content ?? '')
|
||||
const encodedCode = btoa(unescape(encodeURIComponent(str)))
|
||||
const language = String(info ?? 'text')
|
||||
const uniqueId = `editor-${msgId}-${idx}-${language}`
|
||||
|
||||
return `<div class="code-block" data-code="${encodedCode}" data-lang="${language}" id="${uniqueId}">
|
||||
<div class="code-header">
|
||||
<span class="code-lang">${language.toUpperCase()}</span>
|
||||
<button class="copy-button" data-code="${encodedCode}">${t(
|
||||
'common.copy',
|
||||
)}</button>
|
||||
</div>
|
||||
<div class="code-editor"></div>
|
||||
</div>`
|
||||
}
|
||||
|
||||
// reference rule (legacy)
|
||||
const RE_REFERENCE = /^\[(\d+)\]/
|
||||
const RE_REFERENCE_LABEL = /^\[([^\]\n]+)\]/
|
||||
const shouldPreserveReferenceStyleLink = (afterMatch: string) => {
|
||||
if (!afterMatch.startsWith('['))
|
||||
return false
|
||||
|
||||
const nextLabelMatch = RE_REFERENCE_LABEL.exec(afterMatch)
|
||||
if (!nextLabelMatch)
|
||||
return afterMatch !== '[' && !/^\[\d+$/.test(afterMatch)
|
||||
|
||||
const nextLabel = String(nextLabelMatch[1] ?? '')
|
||||
const remaining = afterMatch.slice(nextLabelMatch[0].length)
|
||||
|
||||
// `[1][label](url)` is two adjacent constructs: a numeric reference
|
||||
// followed by a normal inline link.
|
||||
if (remaining.startsWith('('))
|
||||
return false
|
||||
|
||||
// `[1][2]` and `[1][2][3]` are treated as chained numeric references
|
||||
// in this parser, while non-numeric labels should stay available for
|
||||
// CommonMark reference-style links.
|
||||
return !/^\d+$/.test(nextLabel)
|
||||
}
|
||||
const referenceInline = (state: unknown, silent?: boolean) => {
|
||||
const s = state as unknown as { src: string, pos: number, push: (type: string, tag?: string, nesting?: number) => any }
|
||||
if (s.src[s.pos] !== '[')
|
||||
return false
|
||||
const match = RE_REFERENCE.exec(s.src.slice(s.pos))
|
||||
if (!match)
|
||||
return false
|
||||
// Avoid false positives in JSON-like snippets, e.g. `"pages": [14]`
|
||||
const lookbehind = s.src.slice(Math.max(0, s.pos - 120), s.pos)
|
||||
if (/"[^"\n]{1,80}"\s*:\s*$/.test(lookbehind))
|
||||
return false
|
||||
|
||||
// Check if this is part of a link: `[text](...)`, `[[text]](...)`, or `[text][id]`
|
||||
const afterMatch = s.src.slice(s.pos + match[0].length)
|
||||
// `](` handles nested-bracket link text, e.g. `[[8]](url)` where the inner `[8]`
|
||||
// must stay as text for the outer link parser to succeed.
|
||||
if (afterMatch.startsWith('](') || afterMatch.startsWith('(') || shouldPreserveReferenceStyleLink(afterMatch))
|
||||
return false
|
||||
if (!silent) {
|
||||
const id = match[1]
|
||||
const token = s.push('reference', 'span', 0)
|
||||
token.content = id
|
||||
token.markup = match[0]
|
||||
token.raw = match[0]
|
||||
}
|
||||
s.pos += match[0].length
|
||||
return true
|
||||
}
|
||||
|
||||
md.inline.ruler.before('escape', 'reference', referenceInline)
|
||||
md.renderer.rules.reference = (tokens: unknown, idx: number) => {
|
||||
const tokensAny = tokens as unknown as import('./types').MarkdownToken[]
|
||||
const id = String(tokensAny[idx].content ?? '')
|
||||
return `<span class="reference-link" data-reference-id="${id}" role="button" tabindex="0" title="Click to view reference">${id}</span>`
|
||||
}
|
||||
|
||||
return md
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
import type { MarkdownToken, ParsedNode } from '../types'
|
||||
|
||||
export function processNestedBlocks(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
): [ParsedNode | null, number] {
|
||||
// This is a simplified version to break circular dependencies
|
||||
// Nested lists are now properly handled in list-parser.ts with an enhanced parseNestedList function
|
||||
let i = startIndex
|
||||
let depth = 1
|
||||
|
||||
while (i < tokens.length && depth > 0) {
|
||||
if (
|
||||
tokens[i].type === 'bullet_list_open'
|
||||
|| tokens[i].type === 'ordered_list_open'
|
||||
) {
|
||||
depth++
|
||||
}
|
||||
else if (
|
||||
tokens[i].type === 'bullet_list_close'
|
||||
|| tokens[i].type === 'ordered_list_close'
|
||||
) {
|
||||
depth--
|
||||
}
|
||||
i++
|
||||
}
|
||||
|
||||
return [null, i]
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
export {
|
||||
getHtmlTagFromContent,
|
||||
hasCompleteHtmlTagContent,
|
||||
isHtmlLikeTagName,
|
||||
mergeCustomHtmlTags,
|
||||
normalizeCustomHtmlTags,
|
||||
normalizeCustomHtmlTagName as normalizeCustomTag,
|
||||
resolveCustomHtmlTags,
|
||||
shouldRenderUnknownHtmlTagAsText,
|
||||
stripCustomHtmlWrapper,
|
||||
} from '../customHtmlTags'
|
||||
File diff suppressed because it is too large
Load Diff
+21
@@ -0,0 +1,21 @@
|
||||
import type { CheckboxInputNode, CheckboxNode, MarkdownToken } from '../../types'
|
||||
|
||||
export function parseCheckboxToken(token: MarkdownToken): CheckboxNode {
|
||||
const tokenMeta = (token.meta ?? {}) as unknown as { checked?: boolean }
|
||||
return {
|
||||
type: 'checkbox',
|
||||
checked: tokenMeta.checked === true,
|
||||
raw: tokenMeta.checked ? '[x]' : '[ ]',
|
||||
}
|
||||
}
|
||||
|
||||
export function parseCheckboxInputToken(token: any): CheckboxInputNode {
|
||||
const tokenAny = token as unknown as { attrGet?: (name: string) => string | undefined }
|
||||
const rawAttr = tokenAny.attrGet ? tokenAny.attrGet('checked') : undefined
|
||||
const checked = rawAttr === '' || rawAttr === 'true'
|
||||
return {
|
||||
type: 'checkbox_input',
|
||||
checked,
|
||||
raw: checked ? '[x]' : '[ ]',
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
import type { EmojiNode, MarkdownToken } from '../../types'
|
||||
|
||||
export function parseEmojiToken(token: MarkdownToken): EmojiNode {
|
||||
const name = String(token.content ?? '')
|
||||
const markup = String(token.markup ?? '')
|
||||
return {
|
||||
type: 'emoji',
|
||||
name,
|
||||
markup,
|
||||
raw: `:${name}:`,
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
import type { EmphasisNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseEmphasisToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: EmphasisNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let emText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between em_open and em_close
|
||||
while (i < tokens.length && tokens[i].type !== 'em_close') {
|
||||
emText += String(tokens[i].content ?? (tokens[i] as any)?.text ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const node: EmphasisNode = {
|
||||
type: 'emphasis',
|
||||
children,
|
||||
raw: `*${emText}*`,
|
||||
}
|
||||
|
||||
// Skip to after em_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
import type { CodeBlockNode, MarkdownToken } from '../../types'
|
||||
|
||||
// Strip a final line that looks like a fence marker (``` etc.)
|
||||
const TRAILING_FENCE_LINE_RE = /\r?\n[ \t]*`+\s*$/
|
||||
// Unified diff metadata/header line prefixes to skip when splitting a diff
|
||||
const DIFF_HEADER_PREFIXES = ['diff ', 'index ', '--- ', '+++ ', '@@ ']
|
||||
// Newline splitter reused in this module
|
||||
const NEWLINE_RE = /\r?\n/
|
||||
|
||||
function flushPendingDiffHunk(
|
||||
orig: string[],
|
||||
updated: string[],
|
||||
pendingOrig: string[],
|
||||
pendingUpdated: string[],
|
||||
) {
|
||||
if (pendingOrig.length > 0)
|
||||
orig.push(...pendingOrig)
|
||||
if (pendingUpdated.length > 0)
|
||||
updated.push(...pendingUpdated)
|
||||
pendingOrig.length = 0
|
||||
pendingUpdated.length = 0
|
||||
}
|
||||
|
||||
function splitUnifiedDiff(content: string, closed: boolean) {
|
||||
const orig: string[] = []
|
||||
const updated: string[] = []
|
||||
const pendingOrig: string[] = []
|
||||
const pendingUpdated: string[] = []
|
||||
const lines = content.split(NEWLINE_RE)
|
||||
const stableLineCount = Math.max(0, lines.length - 1)
|
||||
const hasUnifiedDiffHeaders = lines.some(line =>
|
||||
line.startsWith('diff ')
|
||||
|| line.startsWith('--- ')
|
||||
|| line.startsWith('+++ ')
|
||||
|| line.startsWith('@@ '),
|
||||
)
|
||||
|
||||
const processLine = (rawLine: string) => {
|
||||
const line = rawLine
|
||||
// skip diff metadata lines
|
||||
if (DIFF_HEADER_PREFIXES.some(p => line.startsWith(p)))
|
||||
return
|
||||
|
||||
if (line.startsWith('-')) {
|
||||
const body = line.slice(1)
|
||||
pendingOrig.push(!hasUnifiedDiffHeaders && body.startsWith(' ') ? ` ${body}` : body)
|
||||
}
|
||||
else if (line.startsWith('+')) {
|
||||
const body = line.slice(1)
|
||||
pendingUpdated.push(!hasUnifiedDiffHeaders && body.startsWith(' ') ? ` ${body}` : body)
|
||||
}
|
||||
else {
|
||||
flushPendingDiffHunk(orig, updated, pendingOrig, pendingUpdated)
|
||||
const contextLine = hasUnifiedDiffHeaders && line.startsWith(' ') ? line.slice(1) : line
|
||||
orig.push(contextLine)
|
||||
updated.push(contextLine)
|
||||
}
|
||||
}
|
||||
|
||||
for (let index = 0; index < stableLineCount; index++)
|
||||
processLine(lines[index] ?? '')
|
||||
|
||||
if (closed && stableLineCount < lines.length)
|
||||
processLine(lines[lines.length - 1] ?? '')
|
||||
|
||||
if (closed || (pendingOrig.length > 0 && pendingUpdated.length > 0))
|
||||
flushPendingDiffHunk(orig, updated, pendingOrig, pendingUpdated)
|
||||
|
||||
return {
|
||||
original: orig.join('\n'),
|
||||
updated: updated.join('\n'),
|
||||
}
|
||||
}
|
||||
|
||||
export function parseFenceToken(token: MarkdownToken): CodeBlockNode {
|
||||
const hasMap = Array.isArray(token.map) && token.map.length === 2
|
||||
const tokenMeta = (token.meta ?? {}) as unknown as { closed?: boolean }
|
||||
const closed = typeof tokenMeta.closed === 'boolean' ? tokenMeta.closed : undefined
|
||||
const info = String(token.info ?? '')
|
||||
const diff = info.startsWith('diff')
|
||||
const language = diff
|
||||
? (() => {
|
||||
const s = info
|
||||
const sp = s.indexOf(' ')
|
||||
return sp === -1
|
||||
? ''
|
||||
: String(s.slice(sp + 1) ?? '')
|
||||
})()
|
||||
: info
|
||||
|
||||
// Defensive sanitization: sometimes a closing fence line (e.g. ``` or ``)
|
||||
// can accidentally end up inside `token.content` (for example when
|
||||
// the parser/mapping is confused). Remove a trailing line that only
|
||||
// contains backticks and optional whitespace so we don't render stray
|
||||
// ` or `` characters at the end of the code output. This is a
|
||||
// conservative cleanup and only strips a final line that looks like a
|
||||
// fence marker (starts with optional spaces then one or more ` and
|
||||
// only whitespace until end-of-string).
|
||||
let content = String(token.content ?? '')
|
||||
if (TRAILING_FENCE_LINE_RE.test(content))
|
||||
content = content.replace(TRAILING_FENCE_LINE_RE, '')
|
||||
|
||||
if (diff) {
|
||||
const { original, updated } = splitUnifiedDiff(content, closed === true)
|
||||
// 返回时保留原来的 code 字段为 updated(编辑后代码),并额外附加原始与更新的文本
|
||||
return {
|
||||
type: 'code_block',
|
||||
language,
|
||||
code: String(updated ?? ''),
|
||||
raw: String(content ?? ''),
|
||||
diff,
|
||||
loading: closed === true ? false : closed === false ? true : !hasMap,
|
||||
originalCode: original,
|
||||
updatedCode: updated,
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
type: 'code_block',
|
||||
language,
|
||||
code: String(content ?? ''),
|
||||
raw: String(content ?? ''),
|
||||
diff,
|
||||
loading: closed === true ? false : closed === false ? true : !hasMap,
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
import type { FootnoteReferenceNode, MarkdownToken } from '../../types'
|
||||
|
||||
export function parseFootnoteRefToken(
|
||||
token: MarkdownToken,
|
||||
): FootnoteReferenceNode {
|
||||
const tokenMeta = (token.meta ?? {}) as unknown as { label?: string }
|
||||
return {
|
||||
type: 'footnote_reference',
|
||||
id: String(tokenMeta.label ?? ''),
|
||||
raw: `[^${String(tokenMeta.label ?? '')}]`,
|
||||
}
|
||||
}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
import type { HardBreakNode } from '../../types'
|
||||
|
||||
export function parseHardbreakToken(): HardBreakNode {
|
||||
return {
|
||||
type: 'hardbreak',
|
||||
raw: '\\\n',
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
import type { HighlightNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseHighlightToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: HighlightNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let markText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between mark_open and mark_close
|
||||
while (i < tokens.length && tokens[i].type !== 'mark_close') {
|
||||
markText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const node: HighlightNode = {
|
||||
type: 'highlight',
|
||||
children,
|
||||
raw: `==${markText}==`,
|
||||
}
|
||||
|
||||
// Skip to after mark_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
import type { InlineCodeNode, MarkdownToken, ParsedNode } from '../../types'
|
||||
import { VOID_HTML_TAGS } from '../../htmlTags'
|
||||
import { parseTagAttrs } from '../../htmlTagUtils'
|
||||
import { normalizeCustomTag } from '../customHtmlTags'
|
||||
import { buildAllowedHtmlTagSet } from '../index'
|
||||
|
||||
type ParseInlineTokensFn = (
|
||||
tokens: MarkdownToken[],
|
||||
raw?: string,
|
||||
pPreToken?: MarkdownToken,
|
||||
options?: { requireClosingStrong?: boolean, customHtmlTags?: readonly string[] },
|
||||
) => ParsedNode[]
|
||||
|
||||
interface TagSetCacheEntry {
|
||||
customTagSet: Set<string> | null
|
||||
allowedTagSet: Set<string>
|
||||
}
|
||||
|
||||
let emptyTagSets: TagSetCacheEntry | null = null
|
||||
const TAG_SET_CACHE = new WeakMap<readonly string[], TagSetCacheEntry>()
|
||||
|
||||
function getEmptyTagSets() {
|
||||
if (!emptyTagSets) {
|
||||
emptyTagSets = {
|
||||
customTagSet: null,
|
||||
allowedTagSet: buildAllowedHtmlTagSet(),
|
||||
}
|
||||
}
|
||||
return emptyTagSets
|
||||
}
|
||||
|
||||
function getTagName(html: string) {
|
||||
const match = html.match(/^<\s*(?:\/\s*)?([\w-]+)/)
|
||||
return match ? match[1].toLowerCase() : ''
|
||||
}
|
||||
|
||||
function isClosingTag(html: string) {
|
||||
return /^<\s*\//.test(html)
|
||||
}
|
||||
|
||||
function isSelfClosing(tag: string, html: string) {
|
||||
return /\/\s*>\s*$/.test(html) || VOID_HTML_TAGS.has(tag)
|
||||
}
|
||||
|
||||
function getTagSets(customTags?: readonly string[]) {
|
||||
if (!customTags || customTags.length === 0)
|
||||
return getEmptyTagSets()
|
||||
const cached = TAG_SET_CACHE.get(customTags)
|
||||
if (cached)
|
||||
return cached
|
||||
const normalized = customTags.map(normalizeCustomTag).filter(Boolean)
|
||||
if (!normalized.length) {
|
||||
const entry = getEmptyTagSets()
|
||||
TAG_SET_CACHE.set(customTags, entry)
|
||||
return entry
|
||||
}
|
||||
const entry = {
|
||||
customTagSet: new Set(normalized),
|
||||
allowedTagSet: buildAllowedHtmlTagSet({ customHtmlTags: customTags as any }),
|
||||
}
|
||||
TAG_SET_CACHE.set(customTags, entry)
|
||||
return entry
|
||||
}
|
||||
|
||||
function tokenToRaw(token: MarkdownToken) {
|
||||
const shape = token as { raw?: string, markup?: string, content?: string }
|
||||
const raw = shape.raw ?? shape.content ?? shape.markup ?? ''
|
||||
return String(raw ?? '')
|
||||
}
|
||||
|
||||
type AttrTuple = [string, string]
|
||||
|
||||
function getAttrValue(attrs: AttrTuple[], name: string): string | undefined {
|
||||
const lowerName = name.toLowerCase()
|
||||
for (let i = attrs.length - 1; i >= 0; i--) {
|
||||
const [key, value] = attrs[i]
|
||||
if (String(key).toLowerCase() === lowerName)
|
||||
return value
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function normalizeLinkAttrs(
|
||||
attrs: AttrTuple[],
|
||||
href: string,
|
||||
title: string | null,
|
||||
): AttrTuple[] {
|
||||
const normalized = attrs.slice()
|
||||
|
||||
if (!getAttrValue(normalized, 'href'))
|
||||
normalized.push(['href', href])
|
||||
if (title != null && !getAttrValue(normalized, 'title'))
|
||||
normalized.push(['title', title])
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
function stringifyTokens(tokens: MarkdownToken[]) {
|
||||
return tokens.map(tokenToRaw).join('')
|
||||
}
|
||||
|
||||
function normalizeStandardHtmlChildren(children: ParsedNode[]) {
|
||||
const normalized: ParsedNode[] = []
|
||||
|
||||
const pushText = (rawText: string) => {
|
||||
const text = String(rawText ?? '')
|
||||
if (!text)
|
||||
return
|
||||
const last = normalized[normalized.length - 1] as ParsedNode | undefined
|
||||
if (last?.type === 'text') {
|
||||
; (last as any).content = String((last as any).content ?? '') + text
|
||||
; (last as any).raw = String((last as any).raw ?? '') + text
|
||||
return
|
||||
}
|
||||
normalized.push({
|
||||
type: 'text',
|
||||
content: text,
|
||||
raw: text,
|
||||
} as ParsedNode)
|
||||
}
|
||||
|
||||
for (const child of children) {
|
||||
if (!child)
|
||||
continue
|
||||
|
||||
if (child.type === 'reference' || child.type === 'footnote_reference') {
|
||||
pushText(String((child as any).raw ?? ''))
|
||||
continue
|
||||
}
|
||||
|
||||
if (Array.isArray((child as any).children)) {
|
||||
normalized.push({
|
||||
...(child as any),
|
||||
children: normalizeStandardHtmlChildren(((child as any).children ?? []) as ParsedNode[]),
|
||||
} as ParsedNode)
|
||||
continue
|
||||
}
|
||||
|
||||
normalized.push(child)
|
||||
}
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
function findMatchingClosing(tokens: MarkdownToken[], startIndex: number, tag: string) {
|
||||
let depth = 0
|
||||
for (let idx = startIndex; idx < tokens.length; idx++) {
|
||||
const t = tokens[idx]
|
||||
if (t.type !== 'html_inline')
|
||||
continue
|
||||
const content = String(t.content ?? '')
|
||||
const tTag = getTagName(content)
|
||||
const closing = isClosingTag(content)
|
||||
const selfClosing = isSelfClosing(tTag, content)
|
||||
if (!closing && !selfClosing && tTag === tag) {
|
||||
depth++
|
||||
continue
|
||||
}
|
||||
if (closing && tTag === tag) {
|
||||
if (depth === 0)
|
||||
return idx
|
||||
depth--
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
function collectHtmlFragment(tokens: MarkdownToken[], startIndex: number, tag: string) {
|
||||
const openToken = tokens[startIndex]
|
||||
const fragmentTokens: MarkdownToken[] = [openToken]
|
||||
let innerTokens: MarkdownToken[] = []
|
||||
let nextIndex = startIndex + 1
|
||||
let closed = false
|
||||
|
||||
const closingIndex = tag ? findMatchingClosing(tokens, startIndex + 1, tag) : -1
|
||||
if (closingIndex !== -1) {
|
||||
innerTokens = tokens.slice(startIndex + 1, closingIndex)
|
||||
fragmentTokens.push(...innerTokens, tokens[closingIndex])
|
||||
nextIndex = closingIndex + 1
|
||||
closed = true
|
||||
}
|
||||
else {
|
||||
// Streaming mid-state: if no matching closing tag exists yet,
|
||||
// treat all following inline tokens as the inner content of this tag.
|
||||
innerTokens = tokens.slice(startIndex + 1)
|
||||
if (innerTokens.length)
|
||||
fragmentTokens.push(...innerTokens)
|
||||
nextIndex = tokens.length
|
||||
}
|
||||
|
||||
return {
|
||||
closed,
|
||||
html: stringifyTokens(fragmentTokens),
|
||||
innerTokens,
|
||||
nextIndex,
|
||||
}
|
||||
}
|
||||
|
||||
// Parse inline HTML and return an appropriate ParsedNode depending on tag.
|
||||
export function parseHtmlInlineCodeToken(
|
||||
token: MarkdownToken,
|
||||
tokens: MarkdownToken[],
|
||||
i: number,
|
||||
parseInlineTokens: ParseInlineTokensFn,
|
||||
raw?: string,
|
||||
pPreToken?: MarkdownToken,
|
||||
options?: { requireClosingStrong?: boolean, customHtmlTags?: readonly string[] },
|
||||
): [ParsedNode, number] {
|
||||
const code = String(token.content ?? '')
|
||||
const tag = getTagName(code)
|
||||
const { customTagSet, allowedTagSet } = getTagSets(options?.customHtmlTags)
|
||||
|
||||
if (!tag) {
|
||||
return [
|
||||
{
|
||||
type: 'inline_code',
|
||||
code,
|
||||
raw: code,
|
||||
} as InlineCodeNode,
|
||||
i + 1,
|
||||
]
|
||||
}
|
||||
|
||||
// If it's not a standard HTML tag and not in customHtmlTags, default to
|
||||
// rendering it as literal text (方案 A). However, if the tag is already
|
||||
// properly closed within the inline token stream, keep it as HTML so
|
||||
// HtmlInlineNode can still render custom components via HTML parsing.
|
||||
if (!allowedTagSet.has(tag)) {
|
||||
const fragment = collectHtmlFragment(tokens, i, tag)
|
||||
if (!fragment.closed) {
|
||||
const content = tokenToRaw(token)
|
||||
return [{ type: 'text', content, raw: content } as ParsedNode, i + 1]
|
||||
}
|
||||
}
|
||||
|
||||
if (tag === 'br') {
|
||||
return [
|
||||
{
|
||||
type: 'hardbreak',
|
||||
raw: code,
|
||||
} as ParsedNode,
|
||||
i + 1,
|
||||
]
|
||||
}
|
||||
|
||||
const closing = isClosingTag(code)
|
||||
const selfClosing = isSelfClosing(tag, code)
|
||||
|
||||
if (closing) {
|
||||
return [
|
||||
{
|
||||
type: 'html_inline',
|
||||
tag,
|
||||
content: code,
|
||||
children: [],
|
||||
raw: code,
|
||||
loading: false,
|
||||
} as ParsedNode,
|
||||
i + 1,
|
||||
]
|
||||
}
|
||||
|
||||
if (tag === 'a') {
|
||||
const fragment = collectHtmlFragment(tokens, i, tag)
|
||||
const attrs = parseTagAttrs(code)
|
||||
const innerTokens = fragment.innerTokens
|
||||
const href = String(getAttrValue(attrs, 'href') ?? '')
|
||||
const titleAttr = getAttrValue(attrs, 'title')
|
||||
const title = titleAttr == null ? null : String(titleAttr)
|
||||
const normalizedAttrs = normalizeLinkAttrs(attrs, href, title)
|
||||
const children = innerTokens.length
|
||||
? parseInlineTokens(innerTokens, raw, pPreToken, options)
|
||||
: []
|
||||
const normalizedChildren = normalizeStandardHtmlChildren(children)
|
||||
const textContent = innerTokens.length ? stringifyTokens(innerTokens) : href || ''
|
||||
|
||||
if (!normalizedChildren.length && textContent) {
|
||||
normalizedChildren.push({
|
||||
type: 'text',
|
||||
content: textContent,
|
||||
raw: textContent,
|
||||
} as ParsedNode)
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
type: 'link',
|
||||
href,
|
||||
title,
|
||||
text: textContent,
|
||||
attrs: normalizedAttrs,
|
||||
children: normalizedChildren,
|
||||
loading: !fragment.closed,
|
||||
raw: fragment.html || code,
|
||||
} as ParsedNode,
|
||||
fragment.nextIndex,
|
||||
]
|
||||
}
|
||||
|
||||
if (selfClosing) {
|
||||
const nodeType = customTagSet?.has(tag) ? tag : 'html_inline'
|
||||
return [
|
||||
{
|
||||
type: nodeType,
|
||||
tag,
|
||||
content: code,
|
||||
children: [],
|
||||
raw: code,
|
||||
loading: false,
|
||||
} as ParsedNode,
|
||||
i + 1,
|
||||
]
|
||||
}
|
||||
|
||||
const fragment = collectHtmlFragment(tokens, i, tag)
|
||||
|
||||
if (tag === 'p' || tag === 'div') {
|
||||
const children = fragment.innerTokens.length
|
||||
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
|
||||
: []
|
||||
const normalizedChildren = normalizeStandardHtmlChildren(children)
|
||||
return [
|
||||
{
|
||||
type: 'paragraph',
|
||||
children: normalizedChildren,
|
||||
raw: fragment.html,
|
||||
} as ParsedNode,
|
||||
fragment.nextIndex,
|
||||
]
|
||||
}
|
||||
|
||||
const children = fragment.innerTokens.length
|
||||
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
|
||||
: []
|
||||
const normalizedChildren = normalizeStandardHtmlChildren(children)
|
||||
|
||||
let content = fragment.html || code
|
||||
let loading = !fragment.closed
|
||||
let autoClosed = false
|
||||
if (!fragment.closed) {
|
||||
const closeTag = `</${tag}>`
|
||||
if (!content.toLowerCase().includes(closeTag.toLowerCase()))
|
||||
content += closeTag
|
||||
autoClosed = true
|
||||
// Still mark loading for mid-state, even though we auto-closed for rendering.
|
||||
loading = true
|
||||
}
|
||||
const attrs = []
|
||||
// 解析属性
|
||||
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
|
||||
let match
|
||||
while ((match = attrRegex.exec(code)) !== null) {
|
||||
const attrName = match[1]
|
||||
const attrValue = match[2] || match[3] || match[4] || ''
|
||||
attrs.push([attrName, attrValue])
|
||||
}
|
||||
if (customTagSet?.has(tag)) {
|
||||
const _content = fragment.innerTokens.length
|
||||
? stringifyTokens(fragment.innerTokens)
|
||||
: ''
|
||||
|
||||
const customChildren = fragment.innerTokens.length
|
||||
? parseInlineTokens(fragment.innerTokens, raw, pPreToken, options)
|
||||
: []
|
||||
return [
|
||||
{
|
||||
type: tag,
|
||||
tag,
|
||||
attrs,
|
||||
content: _content,
|
||||
children: customChildren,
|
||||
raw: content,
|
||||
loading: token.loading || loading,
|
||||
autoClosed,
|
||||
} as ParsedNode,
|
||||
fragment.nextIndex,
|
||||
]
|
||||
}
|
||||
return [
|
||||
{
|
||||
type: 'html_inline',
|
||||
tag,
|
||||
attrs,
|
||||
content,
|
||||
children: normalizedChildren,
|
||||
raw: content,
|
||||
loading,
|
||||
autoClosed,
|
||||
} as ParsedNode,
|
||||
fragment.nextIndex,
|
||||
]
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
import type { ImageNode, MarkdownToken } from '../../types'
|
||||
|
||||
export function parseImageToken(token: MarkdownToken, loading = false): ImageNode {
|
||||
// Some call-sites pass an outer/inline token whose children contain the
|
||||
// actual image token (with attrs). Prefer token.attrs when present; when
|
||||
// absent, search children for the first child that carries attrs.
|
||||
let attrs = token.attrs ?? []
|
||||
// If the parent token has no attrs, prefer attrs from the inner child image
|
||||
// token. Remember which child provided attrs so we can prefer its content
|
||||
// over the parent's `token.content` (the parent may contain the raw
|
||||
// markdown string like ``).
|
||||
let childWithAttrs: any = null
|
||||
if ((!attrs || attrs.length === 0) && Array.isArray(token.children)) {
|
||||
for (const child of token.children) {
|
||||
// child.attrs may be null in markdown-it; check and use if populated
|
||||
const childAttrs = (child as any)?.attrs
|
||||
if (Array.isArray(childAttrs) && childAttrs.length > 0) {
|
||||
attrs = childAttrs
|
||||
childWithAttrs = child
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
const src = String(attrs.find(attr => attr[0] === 'src')?.[1] ?? '')
|
||||
const altAttr = attrs.find(attr => attr[0] === 'alt')?.[1]
|
||||
// Prefer a non-empty alt attribute. If attrs were sourced from an inner
|
||||
// child token prefer that child's `content` over the parent's `token.content`
|
||||
// because the parent may contain the raw markdown instead of the plain alt
|
||||
// text.
|
||||
let alt = ''
|
||||
if (altAttr != null && String(altAttr).length > 0) {
|
||||
alt = String(altAttr)
|
||||
}
|
||||
else if (childWithAttrs?.content != null && String(childWithAttrs.content).length > 0) {
|
||||
alt = String(childWithAttrs.content)
|
||||
}
|
||||
else if (Array.isArray(childWithAttrs?.children) && childWithAttrs.children[0]?.content) {
|
||||
// If the inner image token has children (e.g. a text token) prefer that
|
||||
// child's content when the child token's own `content` is empty.
|
||||
alt = String(childWithAttrs.children[0].content)
|
||||
}
|
||||
else if (Array.isArray(token.children) && token.children[0]?.content) {
|
||||
alt = String(token.children[0].content)
|
||||
}
|
||||
else if (token.content != null && String(token.content).length > 0) {
|
||||
alt = String(token.content)
|
||||
}
|
||||
|
||||
const _title = attrs.find(attr => attr[0] === 'title')?.[1] ?? null
|
||||
const title = _title === null ? null : String(_title)
|
||||
const raw = String(token.content ?? '')
|
||||
|
||||
return {
|
||||
type: 'image',
|
||||
src,
|
||||
alt,
|
||||
title,
|
||||
raw,
|
||||
loading,
|
||||
}
|
||||
}
|
||||
+1846
File diff suppressed because it is too large
Load Diff
+10
@@ -0,0 +1,10 @@
|
||||
import type { InlineCodeNode, MarkdownToken } from '../../types'
|
||||
|
||||
export function parseInlineCodeToken(token: MarkdownToken): InlineCodeNode {
|
||||
const code = String(token.content ?? '')
|
||||
return {
|
||||
type: 'inline_code',
|
||||
code,
|
||||
raw: code,
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
import type { InsertNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseInsertToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: InsertNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let insText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between ins_open and ins_close
|
||||
while (i < tokens.length && tokens[i].type !== 'ins_close') {
|
||||
insText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const node: InsertNode = {
|
||||
type: 'insert',
|
||||
children,
|
||||
raw: `++${String(insText)}++`,
|
||||
}
|
||||
|
||||
// Skip to after ins_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
import type { LinkNode, MarkdownToken, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
type AttrTuple = [string, string]
|
||||
|
||||
function toAttrsTuple(attrs: MarkdownToken['attrs']): AttrTuple[] {
|
||||
const tuples: AttrTuple[] = []
|
||||
if (!Array.isArray(attrs))
|
||||
return tuples
|
||||
for (const attr of attrs) {
|
||||
const key = attr?.[0]
|
||||
if (!key)
|
||||
continue
|
||||
tuples.push([String(key), String(attr?.[1] ?? '')])
|
||||
}
|
||||
return tuples
|
||||
}
|
||||
|
||||
function getAttrValue(attrs: AttrTuple[], name: string): string | undefined {
|
||||
const lowerName = name.toLowerCase()
|
||||
for (let i = attrs.length - 1; i >= 0; i--) {
|
||||
const [key, value] = attrs[i]
|
||||
if (String(key).toLowerCase() === lowerName)
|
||||
return value
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function normalizeLinkAttrs(
|
||||
attrs: AttrTuple[],
|
||||
href: string,
|
||||
title: string | null,
|
||||
): AttrTuple[] {
|
||||
const normalized = attrs.slice()
|
||||
|
||||
if (!getAttrValue(normalized, 'href'))
|
||||
normalized.push(['href', href])
|
||||
if (title != null && !getAttrValue(normalized, 'title'))
|
||||
normalized.push(['title', title])
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
export function parseLinkToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: LinkNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const openToken = tokens[startIndex]
|
||||
const attrsTuple = toAttrsTuple(openToken.attrs)
|
||||
const href = String(getAttrValue(attrsTuple, 'href') ?? '')
|
||||
const _title = getAttrValue(attrsTuple, 'title')
|
||||
const title = _title == null ? null : String(_title)
|
||||
const normalizedAttrs = normalizeLinkAttrs(attrsTuple, href, title)
|
||||
|
||||
let i = startIndex + 1
|
||||
const linkTokens: MarkdownToken[] = []
|
||||
let loading = true
|
||||
|
||||
// Collect all tokens between link_open and link_close
|
||||
while (i < tokens.length && tokens[i].type !== 'link_close') {
|
||||
linkTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
if (tokens[i]?.type === 'link_close') {
|
||||
loading = false
|
||||
}
|
||||
|
||||
const lastLinkToken = linkTokens[linkTokens.length - 1]
|
||||
if (
|
||||
(options as any)?.__insideStrong
|
||||
&& lastLinkToken?.type === 'text'
|
||||
&& String(lastLinkToken.content ?? '').endsWith('**')
|
||||
&& !linkTokens.some(token => token.type === 'strong_open')
|
||||
) {
|
||||
const originalContent = String(lastLinkToken.content ?? '')
|
||||
const originalRaw = String((lastLinkToken as any).raw ?? originalContent)
|
||||
lastLinkToken.content = originalContent.slice(0, -2)
|
||||
lastLinkToken.raw = originalRaw.replace(/\*\*$/, '')
|
||||
}
|
||||
|
||||
// Parse the collected tokens as inline content
|
||||
const children = parseInlineTokens(linkTokens, undefined, undefined, options as any)
|
||||
const linkText = children
|
||||
.map((node) => {
|
||||
const nodeAny = node as unknown as { content?: string, raw?: string }
|
||||
if ('content' in node)
|
||||
return String(nodeAny.content ?? '')
|
||||
return String(nodeAny.raw ?? '')
|
||||
})
|
||||
.join('')
|
||||
|
||||
const node: LinkNode = {
|
||||
type: 'link',
|
||||
href,
|
||||
title,
|
||||
text: linkText,
|
||||
children,
|
||||
raw: String(`[${linkText}](${href}${title ? ` "${title}"` : ''})`),
|
||||
loading,
|
||||
attrs: normalizedAttrs,
|
||||
}
|
||||
|
||||
// Skip to after link_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
import type { MarkdownToken, MathInlineNode } from '../../types'
|
||||
|
||||
// Parse a math_inline token (inline math expressions)
|
||||
export function parseMathInlineToken(token: MarkdownToken): MathInlineNode {
|
||||
const content = token.content ?? ''
|
||||
const raw = token.raw === '$$' ? `$${content}$` : token.raw || ''
|
||||
return {
|
||||
type: 'math_inline',
|
||||
content,
|
||||
loading: !!token.loading,
|
||||
raw,
|
||||
markup: token.markup,
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
import type { MarkdownToken, ReferenceNode } from '../../types'
|
||||
|
||||
// Parse a reference token from markdown-it
|
||||
export function parseReferenceToken(token: MarkdownToken): ReferenceNode {
|
||||
const id = String(token.content ?? '')
|
||||
const raw = String(token.markup ?? `[${token.content ?? ''}]`)
|
||||
return {
|
||||
type: 'reference',
|
||||
id,
|
||||
raw,
|
||||
}
|
||||
}
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
import type {
|
||||
MarkdownToken,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
StrikethroughNode,
|
||||
} from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseStrikethroughToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: StrikethroughNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let sText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between s_open and s_close
|
||||
while (i < tokens.length && tokens[i].type !== 's_close') {
|
||||
sText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const node: StrikethroughNode = {
|
||||
type: 'strikethrough',
|
||||
children,
|
||||
raw: `~~${sText}~~`,
|
||||
}
|
||||
|
||||
// Skip to after s_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
import type { MarkdownToken, ParsedNode, ParseOptions, StrongNode } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
const ESCAPED_PUNCTUATION_RE = /\\([\\()[\]`$|*_\-!])/g
|
||||
|
||||
function resolveInnerRaw(raw: string | undefined, strongText: string) {
|
||||
if (!raw)
|
||||
return undefined
|
||||
|
||||
const rawText = String(raw)
|
||||
if (!rawText)
|
||||
return undefined
|
||||
|
||||
if (rawText === strongText)
|
||||
return rawText
|
||||
|
||||
const decodedRawText = rawText.replace(ESCAPED_PUNCTUATION_RE, '$1')
|
||||
if (decodedRawText === strongText)
|
||||
return rawText
|
||||
|
||||
return undefined
|
||||
}
|
||||
|
||||
export function parseStrongToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
raw?: string,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: StrongNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let strongText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between strong_open and strong_close
|
||||
// 这里可能会遇到多个 strong_open, 需要记录嵌套层级
|
||||
let openCount = 1
|
||||
while (i < tokens.length) {
|
||||
if (tokens[i].type === 'strong_close') {
|
||||
if (openCount === 1)
|
||||
break
|
||||
openCount--
|
||||
}
|
||||
if (tokens[i].type === 'strong_open') {
|
||||
openCount++
|
||||
}
|
||||
strongText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, resolveInnerRaw(raw, strongText), undefined, {
|
||||
...(options as any),
|
||||
__insideStrong: true,
|
||||
}))
|
||||
|
||||
const node: StrongNode = {
|
||||
type: 'strong',
|
||||
children,
|
||||
raw: `**${String(strongText)}**`,
|
||||
}
|
||||
|
||||
// Skip to after strong_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
import type { MarkdownToken, ParsedNode, ParseOptions, SubscriptNode } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseSubscriptToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: SubscriptNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let subText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between sub_open and sub_close (if applicable)
|
||||
while (i < tokens.length && tokens[i].type !== 'sub_close') {
|
||||
subText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const startContent = String(tokens[startIndex].content ?? '')
|
||||
const display = subText || startContent
|
||||
const node: SubscriptNode = {
|
||||
type: 'subscript',
|
||||
children: children.length > 0
|
||||
? children
|
||||
: [
|
||||
{
|
||||
type: 'text',
|
||||
// Fallback to the collected inner text (e.g., "2" in H~2~O)
|
||||
content: display,
|
||||
raw: display,
|
||||
},
|
||||
],
|
||||
raw: `~${display}~`,
|
||||
}
|
||||
|
||||
// Skip to after sub_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
import type { MarkdownToken, ParsedNode, ParseOptions, SuperscriptNode } from '../../types'
|
||||
import { parseInlineTokens } from '../index'
|
||||
|
||||
export function parseSuperscriptToken(
|
||||
tokens: MarkdownToken[],
|
||||
startIndex: number,
|
||||
options?: ParseOptions,
|
||||
): {
|
||||
node: SuperscriptNode
|
||||
nextIndex: number
|
||||
} {
|
||||
const children: ParsedNode[] = []
|
||||
let supText = ''
|
||||
let i = startIndex + 1
|
||||
const innerTokens: MarkdownToken[] = []
|
||||
|
||||
// Process tokens between sup_open and sup_close (if applicable)
|
||||
while (i < tokens.length && tokens[i].type !== 'sup_close') {
|
||||
supText += String(tokens[i].content ?? '')
|
||||
innerTokens.push(tokens[i])
|
||||
i++
|
||||
}
|
||||
|
||||
// Parse inner tokens to handle nested elements
|
||||
children.push(...parseInlineTokens(innerTokens, undefined, undefined, options as any))
|
||||
|
||||
const node: SuperscriptNode = {
|
||||
type: 'superscript',
|
||||
children:
|
||||
children.length > 0
|
||||
? children
|
||||
: [
|
||||
{
|
||||
type: 'text',
|
||||
// Fallback to the collected inner text (e.g., "2" in x^2^)
|
||||
content: supText || String(tokens[startIndex].content ?? ''),
|
||||
raw: supText || String(tokens[startIndex].content ?? ''),
|
||||
},
|
||||
],
|
||||
raw: `^${supText || String(tokens[startIndex].content ?? '')}^`,
|
||||
}
|
||||
|
||||
// Skip to after sup_close
|
||||
const nextIndex = i < tokens.length ? i + 1 : tokens.length
|
||||
|
||||
return { node, nextIndex }
|
||||
}
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
import type { MarkdownToken, TextNode } from '../../types'
|
||||
|
||||
export function parseTextToken(token: MarkdownToken): TextNode {
|
||||
const content = String(token.content ?? '')
|
||||
return {
|
||||
type: 'text',
|
||||
content,
|
||||
raw: content,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
const FILENAMEISH_EXTENSION_RE = /\.([a-z0-9]{1,10})$/i
|
||||
const FILENAMEISH_SEGMENT_RE = /[_()[\]{}<>]/u
|
||||
const URL_PREFIX_HINT_RE = /^(?:https?:\/\/|ftp:\/\/|mailto:|www\.)/i
|
||||
const URL_QUERY_OR_AUTH_HINT_RE = /[?#@]/u
|
||||
const PATH_SEPARATOR_RE = /[\\/]/u
|
||||
const DOMAINISH_TEXT_RE = /^[\p{L}\p{N}./\\-]+$/u
|
||||
const DOMAIN_LABEL_RE = /^(?!-)[A-Za-z0-9-]{1,63}(?<!-)$/u
|
||||
const PUNYCODE_TLD_RE = /^xn--[a-z0-9-]{2,59}$/i
|
||||
const AMBIGUOUS_BARE_DOMAIN_EXTENSIONS = new Set([
|
||||
'ai',
|
||||
'md',
|
||||
'py',
|
||||
'rs',
|
||||
'sh',
|
||||
'zip',
|
||||
])
|
||||
const FILENAMEISH_LINK_EXTENSIONS = new Set([
|
||||
'7z',
|
||||
'ai',
|
||||
'astro',
|
||||
'avi',
|
||||
'bash',
|
||||
'bz2',
|
||||
'c',
|
||||
'cjs',
|
||||
'cpp',
|
||||
'cs',
|
||||
'csv',
|
||||
'doc',
|
||||
'docx',
|
||||
'fish',
|
||||
'flac',
|
||||
'gif',
|
||||
'go',
|
||||
'gz',
|
||||
'h',
|
||||
'hpp',
|
||||
'html',
|
||||
'java',
|
||||
'jpeg',
|
||||
'jpg',
|
||||
'js',
|
||||
'json',
|
||||
'jsx',
|
||||
'kt',
|
||||
'md',
|
||||
'mdx',
|
||||
'mjs',
|
||||
'mov',
|
||||
'mp3',
|
||||
'mp4',
|
||||
'pdf',
|
||||
'php',
|
||||
'png',
|
||||
'ppt',
|
||||
'pptx',
|
||||
'ps1',
|
||||
'py',
|
||||
'rar',
|
||||
'rb',
|
||||
'rs',
|
||||
'sh',
|
||||
'sql',
|
||||
'svg',
|
||||
'swift',
|
||||
'svelte',
|
||||
'tar',
|
||||
'tgz',
|
||||
'toml',
|
||||
'ts',
|
||||
'tsx',
|
||||
'txt',
|
||||
'vue',
|
||||
'wav',
|
||||
'webp',
|
||||
'xls',
|
||||
'xlsx',
|
||||
'xml',
|
||||
'yaml',
|
||||
'yml',
|
||||
'zip',
|
||||
'zsh',
|
||||
])
|
||||
|
||||
function isPlausibleBareDomain(text: string) {
|
||||
const labels = text.split('.')
|
||||
if (labels.length < 2)
|
||||
return false
|
||||
|
||||
const tld = labels[labels.length - 1]?.toLowerCase() ?? ''
|
||||
if (!(DOMAIN_LABEL_RE.test(tld) || PUNYCODE_TLD_RE.test(tld)))
|
||||
return false
|
||||
|
||||
return labels.every(label => DOMAIN_LABEL_RE.test(label))
|
||||
}
|
||||
|
||||
function hasDomainAuthorityPrefix(text: string) {
|
||||
const prefix = text.split(/[\\/]/)[0] ?? ''
|
||||
return isPlausibleBareDomain(prefix)
|
||||
}
|
||||
|
||||
function isUppercaseFilenameSegment(segment: string) {
|
||||
const lettersOnly = segment.replace(/[^a-z]/gi, '')
|
||||
return lettersOnly.length >= 2 && lettersOnly === lettersOnly.toUpperCase()
|
||||
}
|
||||
|
||||
function hasStrongFilenameSignals(linkText: string) {
|
||||
if (FILENAMEISH_SEGMENT_RE.test(linkText))
|
||||
return true
|
||||
|
||||
if (!DOMAINISH_TEXT_RE.test(linkText))
|
||||
return true
|
||||
|
||||
if (PATH_SEPARATOR_RE.test(linkText))
|
||||
return !hasDomainAuthorityPrefix(linkText)
|
||||
|
||||
const extensionless = linkText.replace(FILENAMEISH_EXTENSION_RE, '')
|
||||
const filenameLikeSegments = extensionless.split('.').filter(Boolean)
|
||||
return filenameLikeSegments.some(isUppercaseFilenameSegment)
|
||||
}
|
||||
|
||||
export function shouldDemoteFilenameLikeLinkify(linkText: string) {
|
||||
if (!linkText || URL_PREFIX_HINT_RE.test(linkText) || URL_QUERY_OR_AUTH_HINT_RE.test(linkText))
|
||||
return false
|
||||
|
||||
const extensionMatch = linkText.match(FILENAMEISH_EXTENSION_RE)
|
||||
if (!extensionMatch)
|
||||
return false
|
||||
|
||||
const extension = String(extensionMatch[1] ?? '').toLowerCase()
|
||||
if (!FILENAMEISH_LINK_EXTENSIONS.has(extension))
|
||||
return false
|
||||
|
||||
// Extensions like `.ai` and `.md` can be both real bare-domain TLDs and
|
||||
// common filenames. Keep those linkified unless we also see stronger
|
||||
// filename-only signals such as path separators, underscores, brackets, or
|
||||
// all-uppercase filename segments like `README`.
|
||||
if (!AMBIGUOUS_BARE_DOMAIN_EXTENSIONS.has(extension))
|
||||
return true
|
||||
|
||||
return hasStrongFilenameSignals(linkText)
|
||||
}
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
import { parseBasicBlockToken } from './block-token-parser'
|
||||
import { parseBlockquote } from './blockquote-parser'
|
||||
import { parseList } from './list-parser'
|
||||
|
||||
export function parseAdmonition(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
match: RegExpExecArray,
|
||||
options?: ParseOptions,
|
||||
): [AdmonitionNode, number] {
|
||||
const kind = String(match[1] ?? 'note')
|
||||
const title = String(match[2] ?? (kind.charAt(0).toUpperCase() + kind.slice(1)))
|
||||
const admonitionChildren: ParsedNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
while (j < tokens.length && tokens[j].type !== 'container_close') {
|
||||
if (tokens[j].type === 'paragraph_open') {
|
||||
const contentToken = tokens[j + 1]
|
||||
if (contentToken) {
|
||||
admonitionChildren.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
}
|
||||
j += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
}
|
||||
else if (
|
||||
tokens[j].type === 'bullet_list_open'
|
||||
|| tokens[j].type === 'ordered_list_open'
|
||||
) {
|
||||
const [listNode, newIndex] = parseList(tokens, j, options)
|
||||
admonitionChildren.push(listNode)
|
||||
j = newIndex
|
||||
}
|
||||
else if (tokens[j].type === 'blockquote_open') {
|
||||
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
|
||||
admonitionChildren.push(blockquoteNode)
|
||||
j = newIndex
|
||||
}
|
||||
else {
|
||||
const handled = parseBasicBlockToken(tokens, j, options)
|
||||
if (handled) {
|
||||
admonitionChildren.push(handled[0])
|
||||
j = handled[1]
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const admonitionNode: AdmonitionNode = {
|
||||
type: 'admonition',
|
||||
kind,
|
||||
title,
|
||||
children: admonitionChildren,
|
||||
raw: `:::${kind} ${title}\n${admonitionChildren
|
||||
.map(child => child.raw)
|
||||
.join('\n')}\n:::`,
|
||||
}
|
||||
|
||||
return [admonitionNode, j + 1] // Skip container_close
|
||||
}
|
||||
+518
@@ -0,0 +1,518 @@
|
||||
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions, VmrContainerNode } from '../../types'
|
||||
import { escapeTagForRegExp, findTagCloseIndexOutsideQuotes } from '../../htmlTagUtils'
|
||||
import { normalizeCustomTag } from '../customHtmlTags'
|
||||
import { buildAllowedHtmlTagSet } from '../index'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
import { parseFenceToken } from '../inline-parsers/fence-parser'
|
||||
import { parseBlockquote } from './blockquote-parser'
|
||||
import { parseCodeBlock } from './code-block-parser'
|
||||
import { parseDefinitionList } from './definition-list-parser'
|
||||
import { parseFootnote } from './footnote-parser'
|
||||
import { parseHeading } from './heading-parser'
|
||||
import { parseHtmlBlock } from './html-block-parser'
|
||||
import { parseList } from './list-parser'
|
||||
import { parseMathBlock } from './math-block-parser'
|
||||
import { parseTable } from './table-parser'
|
||||
import { parseThematicBreak } from './thematic-break-parser'
|
||||
|
||||
interface HtmlTagSetCacheEntry {
|
||||
allowedTagSet: Set<string>
|
||||
customTagSet: Set<string> | null
|
||||
}
|
||||
|
||||
let emptyHtmlTagSets: HtmlTagSetCacheEntry | null = null
|
||||
const HTML_TAG_SET_CACHE = new WeakMap<readonly string[], HtmlTagSetCacheEntry>()
|
||||
|
||||
function getEmptyHtmlTagSets() {
|
||||
if (!emptyHtmlTagSets) {
|
||||
emptyHtmlTagSets = {
|
||||
allowedTagSet: buildAllowedHtmlTagSet(),
|
||||
customTagSet: null,
|
||||
}
|
||||
}
|
||||
return emptyHtmlTagSets
|
||||
}
|
||||
|
||||
function getHtmlTagSets(customTags?: readonly string[]) {
|
||||
if (!customTags || customTags.length === 0)
|
||||
return getEmptyHtmlTagSets()
|
||||
const cached = HTML_TAG_SET_CACHE.get(customTags)
|
||||
if (cached)
|
||||
return cached
|
||||
const normalized = customTags.map(normalizeCustomTag).filter(Boolean)
|
||||
if (!normalized.length) {
|
||||
const entry = getEmptyHtmlTagSets()
|
||||
HTML_TAG_SET_CACHE.set(customTags, entry)
|
||||
return entry
|
||||
}
|
||||
const entry = {
|
||||
allowedTagSet: buildAllowedHtmlTagSet({ customHtmlTags: customTags as any }),
|
||||
customTagSet: new Set(normalized),
|
||||
}
|
||||
HTML_TAG_SET_CACHE.set(customTags, entry)
|
||||
return entry
|
||||
}
|
||||
|
||||
function parseVmrContainer(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [VmrContainerNode, number] {
|
||||
const openToken = tokens[index]
|
||||
|
||||
// Extract name from class attribute (e.g., "vmr-container vmr-container-viewcode:topo-test-001")
|
||||
// Handle multiple class values by looking for vmr-container-* anywhere in the string
|
||||
const attrs = openToken.attrs as [string, string][] | null
|
||||
let name = ''
|
||||
const containerAttrs: Record<string, string> = {}
|
||||
|
||||
if (attrs) {
|
||||
for (const [key, value] of attrs) {
|
||||
if (key === 'class') {
|
||||
// Match vmr-container-* anywhere in the class string, not just at the end
|
||||
const match = value.match(/(?:\s|^)vmr-container-(\S+)/)
|
||||
if (match) {
|
||||
name = match[1]
|
||||
}
|
||||
}
|
||||
else if (key.startsWith('data-')) {
|
||||
// Extract data attributes (e.g., "data-devId" -> "devId")
|
||||
const attrName = key.slice(5)
|
||||
// Try to parse JSON values; fallback to raw string if parsing fails
|
||||
try {
|
||||
containerAttrs[attrName] = JSON.parse(value)
|
||||
}
|
||||
catch {
|
||||
containerAttrs[attrName] = value
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const children: ParsedNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
// Parse children until we find the closing token
|
||||
while (j < tokens.length && tokens[j].type !== 'vmr_container_close') {
|
||||
if (tokens[j].type === 'paragraph_open') {
|
||||
// Handle paragraph tokens (plain text)
|
||||
const contentToken = tokens[j + 1]
|
||||
if (contentToken) {
|
||||
const childrenArr = (contentToken.children as MarkdownToken[]) || []
|
||||
children.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(childrenArr || [], undefined, undefined, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
}
|
||||
j += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
}
|
||||
else if (
|
||||
tokens[j].type === 'bullet_list_open'
|
||||
|| tokens[j].type === 'ordered_list_open'
|
||||
) {
|
||||
// Handle list tokens
|
||||
const [listNode, newIndex] = parseList(tokens, j, options)
|
||||
children.push(listNode)
|
||||
j = newIndex
|
||||
}
|
||||
else if (tokens[j].type === 'blockquote_open') {
|
||||
// Handle blockquote tokens
|
||||
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
|
||||
children.push(blockquoteNode)
|
||||
j = newIndex
|
||||
}
|
||||
else {
|
||||
// Handle other basic block tokens (heading, code_block, fence, etc.)
|
||||
const handled = parseBasicBlockToken(tokens, j, options)
|
||||
if (handled) {
|
||||
children.push(handled[0])
|
||||
j = handled[1]
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const hasCloseToken = j < tokens.length && tokens[j].type === 'vmr_container_close'
|
||||
const closed = hasCloseToken || !!options?.final
|
||||
|
||||
// Build raw content
|
||||
let raw = `::: ${name}`
|
||||
if (Object.keys(containerAttrs).length > 0) {
|
||||
raw += ` ${JSON.stringify(containerAttrs)}`
|
||||
}
|
||||
raw += '\n'
|
||||
if (children.length > 0) {
|
||||
// Prefer original Markdown (openToken.raw); rebuilding from tokens may lose formatting.
|
||||
raw += openToken.raw ?? children.map(c => c.raw).join('\n')
|
||||
raw += '\n'
|
||||
}
|
||||
raw += ':::'
|
||||
|
||||
const containerNode: VmrContainerNode = {
|
||||
type: 'vmr_container',
|
||||
name,
|
||||
loading: !closed,
|
||||
attrs: Object.keys(containerAttrs).length > 0 ? containerAttrs : undefined,
|
||||
children,
|
||||
raw,
|
||||
}
|
||||
|
||||
// Skip the closing token when present; otherwise we're already at the end.
|
||||
return [containerNode, hasCloseToken ? (j + 1) : j]
|
||||
}
|
||||
|
||||
function stripWrapperNewlines(s: string) {
|
||||
// Preserve inner whitespace/indentation, but drop a single leading/trailing newline
|
||||
// introduced by the common pattern: <tag>\n...\n</tag>
|
||||
return s.replace(/^\r?\n/, '').replace(/\r?\n$/, '')
|
||||
}
|
||||
|
||||
function stripTrailingPartialClosingTag(inner: string, tag: string) {
|
||||
if (!inner || !tag)
|
||||
return inner
|
||||
const re = new RegExp(String.raw`[\t ]*<\s*\/\s*${tag}[^>]*$`, 'i')
|
||||
return inner.replace(re, '')
|
||||
}
|
||||
|
||||
function findMatchingCloseTagRange(
|
||||
rawHtml: string,
|
||||
tag: string,
|
||||
startIndex: number,
|
||||
) {
|
||||
if (!rawHtml || !tag)
|
||||
return null
|
||||
|
||||
const lowerTag = tag.toLowerCase()
|
||||
const openTagRe = new RegExp(String.raw`^<\s*${escapeTagForRegExp(lowerTag)}(?=\s|>|/)`, 'i')
|
||||
const closeTagRe = new RegExp(String.raw`^<\s*\/\s*${escapeTagForRegExp(lowerTag)}(?=\s|>)`, 'i')
|
||||
|
||||
let depth = 0
|
||||
let index = Math.max(0, startIndex)
|
||||
|
||||
while (index < rawHtml.length) {
|
||||
const lt = rawHtml.indexOf('<', index)
|
||||
if (lt === -1)
|
||||
break
|
||||
|
||||
const slice = rawHtml.slice(lt)
|
||||
if (closeTagRe.test(slice)) {
|
||||
const endRel = findTagCloseIndexOutsideQuotes(slice)
|
||||
if (endRel === -1)
|
||||
return null
|
||||
if (depth === 0) {
|
||||
return {
|
||||
start: lt,
|
||||
end: lt + endRel + 1,
|
||||
}
|
||||
}
|
||||
depth--
|
||||
index = lt + endRel + 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (openTagRe.test(slice)) {
|
||||
const endRel = findTagCloseIndexOutsideQuotes(slice)
|
||||
if (endRel === -1)
|
||||
return null
|
||||
const raw = slice.slice(0, endRel + 1)
|
||||
if (!/\/\s*>$/.test(raw))
|
||||
depth++
|
||||
index = lt + endRel + 1
|
||||
continue
|
||||
}
|
||||
|
||||
index = lt + 1
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
function findNextCustomHtmlBlockFromSource(
|
||||
source: string,
|
||||
tag: string,
|
||||
startIndex: number,
|
||||
): { raw: string, end: number } | null {
|
||||
if (!source || !tag)
|
||||
return null
|
||||
|
||||
const lowerTag = tag.toLowerCase()
|
||||
const openRe = new RegExp(String.raw`<\s*${lowerTag}(?=\s|>|/)`, 'gi')
|
||||
openRe.lastIndex = Math.max(0, startIndex || 0)
|
||||
const openMatch = openRe.exec(source)
|
||||
if (!openMatch || openMatch.index == null)
|
||||
return null
|
||||
|
||||
const openStart = openMatch.index
|
||||
const openSlice = source.slice(openStart)
|
||||
const openEndRel = findTagCloseIndexOutsideQuotes(openSlice)
|
||||
if (openEndRel === -1)
|
||||
return null
|
||||
const openEnd = openStart + openEndRel
|
||||
|
||||
// Self-closing custom tags: treat as a complete block
|
||||
if (/\/\s*>\s*$/.test(openSlice.slice(0, openEndRel + 1))) {
|
||||
const end = openEnd + 1
|
||||
return { raw: source.slice(openStart, end), end }
|
||||
}
|
||||
|
||||
let depth = 1
|
||||
let i = openEnd + 1
|
||||
|
||||
const isOpenAt = (pos: number) => {
|
||||
const s = source.slice(pos)
|
||||
return new RegExp(String.raw`^<\s*${lowerTag}(?=\s|>|/)`, 'i').test(s)
|
||||
}
|
||||
const isCloseAt = (pos: number) => {
|
||||
const s = source.slice(pos)
|
||||
return new RegExp(String.raw`^<\s*\/\s*${lowerTag}(?=\s|>)`, 'i').test(s)
|
||||
}
|
||||
|
||||
while (i < source.length) {
|
||||
const lt = source.indexOf('<', i)
|
||||
if (lt === -1) {
|
||||
// No more tags in the remaining source; treat as unclosed streaming block.
|
||||
return { raw: source.slice(openStart), end: source.length }
|
||||
}
|
||||
|
||||
if (isCloseAt(lt)) {
|
||||
const gt = source.indexOf('>', lt)
|
||||
if (gt === -1)
|
||||
return null
|
||||
depth--
|
||||
if (depth === 0) {
|
||||
const end = gt + 1
|
||||
return { raw: source.slice(openStart, end), end }
|
||||
}
|
||||
i = gt + 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (isOpenAt(lt)) {
|
||||
const rel = findTagCloseIndexOutsideQuotes(source.slice(lt))
|
||||
if (rel === -1)
|
||||
return null
|
||||
depth++
|
||||
i = lt + rel + 1
|
||||
continue
|
||||
}
|
||||
|
||||
i = lt + 1
|
||||
}
|
||||
|
||||
// If the closing tag hasn't arrived yet (streaming), return a partial block
|
||||
// from the opening tag to end-of-source. This preserves original lines like
|
||||
// `---` so inner markdown can render progressively.
|
||||
return { raw: source.slice(openStart), end: source.length }
|
||||
}
|
||||
|
||||
function clampNonNegative(n: number) {
|
||||
return Number.isFinite(n) && n > 0 ? n : 0
|
||||
}
|
||||
|
||||
function lineToIndex(source: string, line: number) {
|
||||
// markdown-it token.map uses 0-based line numbers.
|
||||
const targetLine = clampNonNegative(line)
|
||||
if (!source || targetLine <= 0)
|
||||
return 0
|
||||
|
||||
let currentLine = 0
|
||||
for (let i = 0; i < source.length; i++) {
|
||||
if (source[i] === '\n') {
|
||||
currentLine++
|
||||
if (currentLine === targetLine)
|
||||
return i + 1
|
||||
}
|
||||
}
|
||||
return source.length
|
||||
}
|
||||
|
||||
export function parseBasicBlockToken(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [ParsedNode, number] | null {
|
||||
const token = tokens[index]
|
||||
switch (token.type) {
|
||||
case 'heading_open':
|
||||
return [parseHeading(tokens, index, options), index + 3]
|
||||
|
||||
case 'code_block': {
|
||||
return [parseCodeBlock(token), index + 1]
|
||||
}
|
||||
|
||||
case 'fence':
|
||||
return [parseFenceToken(token), index + 1]
|
||||
|
||||
case 'math_block':
|
||||
return [parseMathBlock(token), index + 1]
|
||||
|
||||
case 'html_block': {
|
||||
const htmlBlockNode = parseHtmlBlock(token)
|
||||
const tagSets = htmlBlockNode.tag ? getHtmlTagSets(options?.customHtmlTags) : null
|
||||
if (htmlBlockNode.tag && htmlBlockNode.loading && tagSets && !tagSets.allowedTagSet.has(htmlBlockNode.tag)) {
|
||||
const raw = String((token as any)?.content ?? '')
|
||||
const content = raw.replace(/\n+$/, '')
|
||||
return [
|
||||
{
|
||||
type: 'paragraph',
|
||||
children: content ? [{ type: 'text', content, raw: content }] : [],
|
||||
raw: content,
|
||||
} as any,
|
||||
index + 1,
|
||||
]
|
||||
}
|
||||
if (htmlBlockNode.tag && tagSets?.customTagSet?.has(htmlBlockNode.tag)) {
|
||||
const tag = htmlBlockNode.tag
|
||||
// markdown-it can normalize html_block token.content and lose original lines.
|
||||
// Re-extract the next full <tag>...</tag> block from the original source.
|
||||
const source = String((options as any)?.__sourceMarkdown ?? '')
|
||||
const cursor = Number((options as any)?.__customHtmlBlockCursor ?? 0)
|
||||
|
||||
// If markdown-it provides a source map for this token, prefer anchoring the
|
||||
// re-extraction to that line range. This avoids accidentally matching an
|
||||
// earlier occurrence of the same custom tag that was tokenized as inline.
|
||||
const mappedLineStart = Array.isArray((token as any).map)
|
||||
? lineToIndex(source, Number((token as any).map?.[0] ?? 0))
|
||||
: 0
|
||||
const searchStart = Math.max(clampNonNegative(cursor), clampNonNegative(mappedLineStart))
|
||||
|
||||
const fromSource = findNextCustomHtmlBlockFromSource(source, tag, searchStart)
|
||||
if (fromSource)
|
||||
(options as any).__customHtmlBlockCursor = fromSource.end
|
||||
|
||||
const rawHtml = String(fromSource?.raw ?? htmlBlockNode.raw ?? '')
|
||||
|
||||
const openEnd = findTagCloseIndexOutsideQuotes(rawHtml)
|
||||
const openTag = openEnd !== -1 ? rawHtml.slice(0, openEnd + 1) : rawHtml
|
||||
const selfClosing = openEnd !== -1 && /\/\s*>\s*$/.test(openTag)
|
||||
const closeRange = openEnd === -1
|
||||
? null
|
||||
: findMatchingCloseTagRange(rawHtml, tag, openEnd + 1)
|
||||
const closeIndex = closeRange?.start ?? -1
|
||||
|
||||
let inner = ''
|
||||
if (openEnd !== -1) {
|
||||
if (closeIndex !== -1 && openEnd < closeIndex)
|
||||
inner = rawHtml.slice(openEnd + 1, closeIndex)
|
||||
else
|
||||
inner = rawHtml.slice(openEnd + 1)
|
||||
}
|
||||
|
||||
// Streaming mid-state: if the closing tag is being typed but not yet
|
||||
// complete, don't leak the partial `</tag` into content.
|
||||
if (closeIndex === -1)
|
||||
inner = stripTrailingPartialClosingTag(inner, tag)
|
||||
|
||||
// Parse attrs from the opening tag only
|
||||
const attrs: [string, string][] = []
|
||||
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
|
||||
let m
|
||||
while ((m = attrRegex.exec(openTag)) !== null) {
|
||||
const name = m[1]
|
||||
if (!name || name.toLowerCase() === tag)
|
||||
continue
|
||||
const value = m[2] || m[3] || m[4] || ''
|
||||
attrs.push([name, value])
|
||||
}
|
||||
|
||||
const loading = !options?.final && !selfClosing && closeRange == null
|
||||
|
||||
return [
|
||||
{
|
||||
type: tag,
|
||||
tag,
|
||||
content: stripWrapperNewlines(inner),
|
||||
raw: String(fromSource?.raw ?? htmlBlockNode.raw ?? rawHtml),
|
||||
loading,
|
||||
attrs: attrs.length ? attrs : undefined,
|
||||
} as ParsedNode,
|
||||
index + 1,
|
||||
]
|
||||
}
|
||||
return [htmlBlockNode, index + 1]
|
||||
}
|
||||
|
||||
case 'table_open': {
|
||||
const [tableNode, newIndex] = parseTable(tokens, index, options)
|
||||
return [tableNode, newIndex]
|
||||
}
|
||||
|
||||
case 'dl_open': {
|
||||
const [definitionListNode, newIndex] = parseDefinitionList(tokens, index, options)
|
||||
return [definitionListNode, newIndex]
|
||||
}
|
||||
|
||||
case 'footnote_open': {
|
||||
const [footnoteNode, newIndex] = parseFootnote(tokens, index, options)
|
||||
return [footnoteNode, newIndex]
|
||||
}
|
||||
|
||||
case 'hr':
|
||||
return [parseThematicBreak(), index + 1]
|
||||
default:
|
||||
break
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
type ContainerParser = (
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
) => [AdmonitionNode, number]
|
||||
|
||||
type ContainerMatcher = (
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
) => [AdmonitionNode, number] | null
|
||||
|
||||
export function parseCommonBlockToken(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
handlers?: {
|
||||
parseContainer?: ContainerParser
|
||||
matchAdmonition?: ContainerMatcher
|
||||
},
|
||||
): [ParsedNode, number] | null {
|
||||
const basicResult = parseBasicBlockToken(tokens, index, options)
|
||||
if (basicResult)
|
||||
return basicResult
|
||||
|
||||
const token = tokens[index]
|
||||
switch (token.type) {
|
||||
case 'container_warning_open':
|
||||
case 'container_info_open':
|
||||
case 'container_note_open':
|
||||
case 'container_tip_open':
|
||||
case 'container_danger_open':
|
||||
case 'container_caution_open':
|
||||
case 'container_error_open': {
|
||||
if (handlers?.parseContainer)
|
||||
return handlers.parseContainer(tokens, index, options)
|
||||
break
|
||||
}
|
||||
|
||||
case 'container_open': {
|
||||
if (handlers?.matchAdmonition) {
|
||||
const result = handlers.matchAdmonition(tokens, index, options)
|
||||
if (result)
|
||||
return result
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
case 'vmr_container_open': {
|
||||
return parseVmrContainer(tokens, index, options)
|
||||
}
|
||||
default:
|
||||
break
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
import type { BlockquoteNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
import { parseCommonBlockToken } from './block-token-parser'
|
||||
import { containerTokenHandlers } from './container-token-handlers'
|
||||
import { parseList } from './list-parser'
|
||||
|
||||
export function parseBlockquote(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [BlockquoteNode, number] {
|
||||
const blockquoteChildren: ParsedNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
// Process blockquote content until closing tag is found
|
||||
while (j < tokens.length && tokens[j].type !== 'blockquote_close') {
|
||||
const token = tokens[j]
|
||||
switch (token.type) {
|
||||
case 'paragraph_open': {
|
||||
const contentToken = tokens[j + 1]
|
||||
blockquoteChildren.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
j += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
break
|
||||
}
|
||||
case 'bullet_list_open':
|
||||
case 'ordered_list_open': {
|
||||
const [listNode, newIndex] = parseList(tokens, j, options)
|
||||
blockquoteChildren.push(listNode)
|
||||
j = newIndex
|
||||
break
|
||||
}
|
||||
case 'blockquote_open': {
|
||||
const [nestedBlockquote, newIndex] = parseBlockquote(tokens, j, options)
|
||||
blockquoteChildren.push(nestedBlockquote)
|
||||
j = newIndex
|
||||
break
|
||||
}
|
||||
default:{
|
||||
const handled = parseCommonBlockToken(tokens, j, options, containerTokenHandlers)
|
||||
if (handled) {
|
||||
blockquoteChildren.push(handled[0])
|
||||
j = handled[1]
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const blockquoteNode: BlockquoteNode = {
|
||||
type: 'blockquote',
|
||||
children: blockquoteChildren,
|
||||
raw: blockquoteChildren.map(child => child.raw).join('\n'),
|
||||
}
|
||||
|
||||
return [blockquoteNode, j + 1] // Skip blockquote_close
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
import type { CodeBlockNode, MarkdownToken } from '../../types'
|
||||
import { parseFenceToken } from '../inline-parsers/fence-parser'
|
||||
|
||||
export function parseCodeBlock(token: MarkdownToken): CodeBlockNode {
|
||||
// If this code block is actually a diff (some markdown-it backends
|
||||
// classify fences vs code_block differently), delegate to the
|
||||
// fence parser to preserve original/updated fields.
|
||||
if (token.info?.startsWith('diff')) {
|
||||
return parseFenceToken(token)
|
||||
}
|
||||
|
||||
const contentStr = String(token.content ?? '')
|
||||
const match = contentStr.match(/ type="application\/vnd\.ant\.([^"]+)"/)
|
||||
if (match?.[1]) {
|
||||
// 需要把 <antArtifact> 标签去掉
|
||||
// mutate token.content safely by assigning the cleaned string
|
||||
token.content = contentStr
|
||||
.replace(/<antArtifact[^>]*>/g, '')
|
||||
.replace(/<\/antArtifact>/g, '')
|
||||
}
|
||||
const hasMap = Array.isArray(token.map) && token.map.length === 2
|
||||
return {
|
||||
type: 'code_block',
|
||||
language: match ? match[1] : String(token.info ?? ''),
|
||||
code: String(token.content ?? ''),
|
||||
raw: String(token.content ?? ''),
|
||||
loading: !hasMap,
|
||||
}
|
||||
}
|
||||
+137
@@ -0,0 +1,137 @@
|
||||
import type { AdmonitionNode, MarkdownToken, ParsedNode, ParseOptions, TextNode } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
import { parseBasicBlockToken } from './block-token-parser'
|
||||
import { parseBlockquote } from './blockquote-parser'
|
||||
import { parseList } from './list-parser'
|
||||
|
||||
const CONTAINER_KINDS = new Set(['warning', 'info', 'note', 'tip', 'danger', 'caution'])
|
||||
|
||||
function parseContainerInfo(info: string) {
|
||||
let markerEnd = 0
|
||||
while (markerEnd < info.length && markerEnd < 3 && info[markerEnd] === ':')
|
||||
markerEnd++
|
||||
|
||||
if (markerEnd === 0 || info[markerEnd] === ':')
|
||||
return null
|
||||
|
||||
const rest = info.slice(markerEnd).trimStart()
|
||||
if (!rest)
|
||||
return null
|
||||
|
||||
const firstWhitespace = rest.search(/\s/)
|
||||
const rawKind = (firstWhitespace === -1 ? rest : rest.slice(0, firstWhitespace)).toLowerCase()
|
||||
if (!CONTAINER_KINDS.has(rawKind))
|
||||
return null
|
||||
|
||||
const title = firstWhitespace === -1 ? '' : rest.slice(firstWhitespace).trim()
|
||||
return { kind: rawKind, title }
|
||||
}
|
||||
|
||||
export function parseContainer(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [AdmonitionNode, number] {
|
||||
const openToken = tokens[index]
|
||||
|
||||
// Determine kind and optional title
|
||||
let kind = 'note'
|
||||
let title = ''
|
||||
|
||||
const typeMatch = openToken.type.match(/^container_(\w+)_open$/)
|
||||
if (typeMatch) {
|
||||
kind = typeMatch[1]
|
||||
// some implementations set info to remaining title text
|
||||
const info = String(openToken.info ?? '').trim()
|
||||
if (info && !info.startsWith(':::')) {
|
||||
// if info looks like 'warning title', drop leading kind token
|
||||
const lowerInfo = info.toLowerCase()
|
||||
if (lowerInfo.startsWith(kind)) {
|
||||
const maybe = info.slice(kind.length).trim()
|
||||
if (maybe)
|
||||
title = maybe
|
||||
}
|
||||
}
|
||||
}
|
||||
else {
|
||||
// container_open: info usually contains the marker like ' warning Title'
|
||||
const info = String(openToken.info ?? '').trim()
|
||||
const parsedInfo = parseContainerInfo(info)
|
||||
if (parsedInfo) {
|
||||
kind = parsedInfo.kind
|
||||
title = parsedInfo.title
|
||||
}
|
||||
}
|
||||
|
||||
if (!title)
|
||||
title = kind.charAt(0).toUpperCase() + kind.slice(1)
|
||||
|
||||
const children: ParsedNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
// Accept closing tokens: 'container_close' or 'container_<kind>_close'
|
||||
const closeType = new RegExp(`^container_${kind}_close$`)
|
||||
|
||||
while (
|
||||
j < tokens.length
|
||||
&& tokens[j].type !== 'container_close'
|
||||
&& !closeType.test(tokens[j].type)
|
||||
) {
|
||||
if (tokens[j].type === 'paragraph_open') {
|
||||
const contentToken = tokens[j + 1]
|
||||
if (contentToken) {
|
||||
const childrenArr = (contentToken.children as MarkdownToken[]) || []
|
||||
let i = -1
|
||||
for (let k = childrenArr.length - 1; k >= 0; k--) {
|
||||
const t = childrenArr[k] as TextNode
|
||||
if (t.type === 'text' && /:+/.test(t.content)) {
|
||||
i = k
|
||||
break
|
||||
}
|
||||
}
|
||||
const _children = i !== -1 ? childrenArr.slice(0, i) : childrenArr
|
||||
children.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(_children || [], undefined, undefined, options as any),
|
||||
raw: String(contentToken.content ?? '').replace(/\n:+$/, '').replace(/\n\s*:::\s*$/, ''),
|
||||
})
|
||||
}
|
||||
j += 3
|
||||
}
|
||||
else if (
|
||||
tokens[j].type === 'bullet_list_open'
|
||||
|| tokens[j].type === 'ordered_list_open'
|
||||
) {
|
||||
const [listNode, newIndex] = parseList(tokens, j, options)
|
||||
children.push(listNode)
|
||||
j = newIndex
|
||||
}
|
||||
else if (tokens[j].type === 'blockquote_open') {
|
||||
const [blockquoteNode, newIndex] = parseBlockquote(tokens, j, options)
|
||||
children.push(blockquoteNode)
|
||||
j = newIndex
|
||||
}
|
||||
else {
|
||||
const handled = parseBasicBlockToken(tokens, j, options)
|
||||
if (handled) {
|
||||
children.push(handled[0])
|
||||
j = handled[1]
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const admonitionNode: AdmonitionNode = {
|
||||
type: 'admonition',
|
||||
kind,
|
||||
title,
|
||||
children,
|
||||
raw: `:::${kind} ${title}\n${children.map(c => c.raw).join('\n')}\n:::`,
|
||||
}
|
||||
|
||||
// Skip the closing token
|
||||
const closingIndex = j
|
||||
return [admonitionNode, closingIndex + 1]
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
import type { AdmonitionNode, MarkdownToken, ParseOptions } from '../../types'
|
||||
import { parseAdmonition } from './admonition-parser'
|
||||
import { parseContainer } from './container-parser'
|
||||
|
||||
const CONTAINER_REGEX
|
||||
= /^::: ?(warning|info|note|tip|danger|caution|error) ?(.*)$/
|
||||
|
||||
function handleContainerOpen(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [AdmonitionNode, number] | null {
|
||||
const token = tokens[index]
|
||||
if (token.type !== 'container_open')
|
||||
return null
|
||||
const match = CONTAINER_REGEX.exec(String(token.info ?? ''))
|
||||
if (!match)
|
||||
return null
|
||||
return parseAdmonition(tokens, index, match, options)
|
||||
}
|
||||
|
||||
export const containerTokenHandlers = {
|
||||
parseContainer: (
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
) => parseContainer(tokens, index, options),
|
||||
matchAdmonition: handleContainerOpen,
|
||||
}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
import type {
|
||||
DefinitionItemNode,
|
||||
DefinitionListNode,
|
||||
MarkdownToken,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
} from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
|
||||
export function parseDefinitionList(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [DefinitionListNode, number] {
|
||||
const items: DefinitionItemNode[] = []
|
||||
let j = index + 1
|
||||
let termNodes: ParsedNode[] = []
|
||||
let definitionNodes: ParsedNode[] = []
|
||||
|
||||
while (j < tokens.length && tokens[j].type !== 'dl_close') {
|
||||
if (tokens[j].type === 'dt_open') {
|
||||
// Process term
|
||||
const termToken = tokens[j + 1]
|
||||
termNodes = parseInlineTokens(termToken.children || [], undefined, undefined, options as any)
|
||||
j += 3 // Skip dt_open, inline, dt_close
|
||||
}
|
||||
else if (tokens[j].type === 'dd_open') {
|
||||
// Process definition
|
||||
let k = j + 1
|
||||
definitionNodes = []
|
||||
|
||||
while (k < tokens.length && tokens[k].type !== 'dd_close') {
|
||||
if (tokens[k].type === 'paragraph_open') {
|
||||
const contentToken = tokens[k + 1]
|
||||
definitionNodes.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
k += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
}
|
||||
else {
|
||||
k++
|
||||
}
|
||||
}
|
||||
|
||||
// Add definition item
|
||||
if (termNodes.length > 0) {
|
||||
items.push({
|
||||
type: 'definition_item',
|
||||
term: termNodes,
|
||||
definition: definitionNodes,
|
||||
raw: `${termNodes.map(term => term.raw).join('')}: ${definitionNodes
|
||||
.map(def => def.raw)
|
||||
.join('\n')}`,
|
||||
})
|
||||
|
||||
// Reset term nodes
|
||||
termNodes = []
|
||||
}
|
||||
|
||||
j = k + 1 // Skip dd_close
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
|
||||
const definitionListNode: DefinitionListNode = {
|
||||
type: 'definition_list',
|
||||
items,
|
||||
raw: items.map(item => item.raw).join('\n'),
|
||||
}
|
||||
|
||||
return [definitionListNode, j + 1] // Skip dl_close
|
||||
}
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
import type { FootnoteNode, MarkdownToken, ParsedNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
|
||||
export function parseFootnote(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [FootnoteNode, number] {
|
||||
const token = tokens[index]
|
||||
const meta = (token.meta ?? {}) as unknown as { label?: number | string }
|
||||
const id = String(meta?.label ?? '0')
|
||||
const footnoteChildren: ParsedNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
while (j < tokens.length && tokens[j].type !== 'footnote_close') {
|
||||
if (tokens[j].type === 'paragraph_open') {
|
||||
const contentToken = tokens[j + 1]
|
||||
const children = contentToken.children || []
|
||||
if (tokens[j + 2].type === 'footnote_anchor') {
|
||||
children.push(tokens[j + 2] as any)
|
||||
}
|
||||
footnoteChildren.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), undefined, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
|
||||
j += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
|
||||
const footnoteNode: FootnoteNode = {
|
||||
type: 'footnote',
|
||||
id,
|
||||
children: footnoteChildren,
|
||||
raw: `[^${id}]: ${footnoteChildren.map(child => child.raw).join('\n')}`,
|
||||
}
|
||||
|
||||
return [footnoteNode, j + 1] // Skip footnote_close
|
||||
}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
import type { HardBreakNode } from '../../types'
|
||||
|
||||
export function parseHardBreak(): HardBreakNode {
|
||||
return {
|
||||
type: 'hardbreak',
|
||||
raw: '\\\n',
|
||||
}
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
import type { HeadingNode, MarkdownToken, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
|
||||
export function parseHeading(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): HeadingNode {
|
||||
const token = tokens[index]
|
||||
const attrs = (token as any)?.attrs as any[] | null | undefined
|
||||
const attrsRecord = Array.isArray(attrs) && attrs.length
|
||||
? Object.fromEntries(
|
||||
attrs
|
||||
.filter(pair => Array.isArray(pair) && pair.length >= 1 && pair[0])
|
||||
.map(([name, value]) => [String(name), value == null || value === '' ? true : String(value)]),
|
||||
)
|
||||
: undefined
|
||||
const levelStr = String(token.tag?.substring(1) ?? '1')
|
||||
const headingLevel = Number.parseInt(levelStr, 10)
|
||||
const headingContentToken = tokens[index + 1]
|
||||
const headingContent = String(headingContentToken.content ?? '')
|
||||
|
||||
return {
|
||||
type: 'heading',
|
||||
level: headingLevel,
|
||||
text: headingContent,
|
||||
...(attrsRecord ? { attrs: attrsRecord } : {}),
|
||||
children: parseInlineTokens(headingContentToken.children || [], headingContent, undefined, options as any),
|
||||
raw: headingContent,
|
||||
}
|
||||
}
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
import type { HtmlBlockNode, MarkdownToken } from '../../types'
|
||||
import { VOID_HTML_TAGS } from '../../htmlTags'
|
||||
import { findTagCloseIndexOutsideQuotes, parseTagAttrs } from '../../htmlTagUtils'
|
||||
|
||||
function findMatchingCloseTagEnd(rawHtml: string, tag: string, startIndex: number) {
|
||||
const lowerTag = tag.toLowerCase()
|
||||
const openTagRe = new RegExp(String.raw`^<\s*${lowerTag}(?=\s|>|/)`, 'i')
|
||||
const closeTagRe = new RegExp(String.raw`^<\s*\/\s*${lowerTag}(?=\s|>)`, 'i')
|
||||
|
||||
let depth = 0
|
||||
let index = Math.max(0, startIndex)
|
||||
|
||||
while (index < rawHtml.length) {
|
||||
const lt = rawHtml.indexOf('<', index)
|
||||
if (lt === -1)
|
||||
return -1
|
||||
|
||||
const slice = rawHtml.slice(lt)
|
||||
if (closeTagRe.test(slice)) {
|
||||
const endRel = findTagCloseIndexOutsideQuotes(slice)
|
||||
if (endRel === -1)
|
||||
return -1
|
||||
if (depth === 0)
|
||||
return lt + endRel + 1
|
||||
depth--
|
||||
index = lt + endRel + 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (openTagRe.test(slice)) {
|
||||
const endRel = findTagCloseIndexOutsideQuotes(slice)
|
||||
if (endRel === -1)
|
||||
return -1
|
||||
const rawTag = slice.slice(0, endRel + 1)
|
||||
if (!/\/\s*>$/.test(rawTag))
|
||||
depth++
|
||||
index = lt + endRel + 1
|
||||
continue
|
||||
}
|
||||
|
||||
index = lt + 1
|
||||
}
|
||||
|
||||
return -1
|
||||
}
|
||||
|
||||
export function parseHtmlBlock(token: MarkdownToken): HtmlBlockNode {
|
||||
const raw = String(token.content ?? '')
|
||||
|
||||
// Non-element html blocks (comments, doctypes, processing instructions) are non-closable
|
||||
if (/^\s*<!--/.test(raw) || /^\s*<!/.test(raw) || /^\s*<\?/.test(raw)) {
|
||||
return {
|
||||
type: 'html_block',
|
||||
content: raw,
|
||||
raw,
|
||||
tag: '',
|
||||
loading: false,
|
||||
}
|
||||
}
|
||||
|
||||
// Extract first tag name (lowercased) like div, p, section, etc.
|
||||
const tagMatch = raw.match(/^\s*<([A-Z][\w:-]*)/i)
|
||||
const tag = (tagMatch?.[1] || '').toLowerCase()
|
||||
|
||||
// Handle unknown or malformed tag gracefully
|
||||
if (!tag) {
|
||||
return {
|
||||
type: 'html_block',
|
||||
content: raw,
|
||||
raw,
|
||||
tag: '',
|
||||
loading: false,
|
||||
}
|
||||
}
|
||||
|
||||
const openEnd = findTagCloseIndexOutsideQuotes(raw)
|
||||
const openTag = openEnd === -1 ? raw : raw.slice(0, openEnd + 1)
|
||||
// Self-closing first tag like <img ... />
|
||||
const selfClosing = openEnd !== -1 && /\/\s*>$/.test(openTag)
|
||||
const isVoid = VOID_HTML_TAGS.has(tag)
|
||||
const attrs = parseTagAttrs(openTag)
|
||||
|
||||
const closeEnd = openEnd === -1 ? -1 : findMatchingCloseTagEnd(raw, tag, openEnd + 1)
|
||||
const hasClosing = closeEnd !== -1
|
||||
|
||||
const loading = !(isVoid || selfClosing || hasClosing)
|
||||
|
||||
const content = loading
|
||||
? `${raw.replace(/<[^>]*$/, '')}\n</${tag}>`
|
||||
: raw
|
||||
|
||||
return {
|
||||
type: 'html_block',
|
||||
content,
|
||||
raw,
|
||||
tag,
|
||||
attrs: attrs.length ? attrs : undefined,
|
||||
loading,
|
||||
}
|
||||
}
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
import type {
|
||||
ListItemNode,
|
||||
ListNode,
|
||||
MarkdownToken,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
} from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
import { parseCommonBlockToken } from './block-token-parser'
|
||||
import { parseBlockquote } from './blockquote-parser'
|
||||
import { containerTokenHandlers } from './container-token-handlers'
|
||||
|
||||
function trimInlineTokenTail(token: MarkdownToken) {
|
||||
const rawContent = String(token.content ?? '')
|
||||
const trimmed = rawContent.replace(/[ \t\r\n]+$/g, '')
|
||||
if (trimmed === rawContent)
|
||||
return
|
||||
|
||||
token.content = trimmed
|
||||
|
||||
const children = token.children as MarkdownToken[] | undefined
|
||||
if (!Array.isArray(children) || children.length === 0)
|
||||
return
|
||||
|
||||
while (children.length) {
|
||||
const last = children[children.length - 1]
|
||||
if (!last) {
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
if (last.type === 'softbreak' || last.type === 'hardbreak') {
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
if (last.type === 'text') {
|
||||
const lastContent = String(last.content ?? '')
|
||||
const next = lastContent.replace(/[ \t\r\n]+$/g, '')
|
||||
if (next === lastContent)
|
||||
break
|
||||
if (next) {
|
||||
last.content = next
|
||||
break
|
||||
}
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
function stripLeakedOrderedListMarkerSuffix(token: MarkdownToken) {
|
||||
// In streaming mode markdown-it can occasionally leak the next ordered-list
|
||||
// marker (e.g. "\n\n2" or "\n\n2.") into the previous list item's paragraph
|
||||
// content. Because our renderer uses `whitespace-pre-wrap`, those leaked
|
||||
// newlines can create a large blank vertical gap until the next parse tick.
|
||||
const rawContent = String(token.content ?? '')
|
||||
const leak = rawContent.match(/\r?\n\s*\d+[.)]?\s*$/)
|
||||
if (!leak || typeof leak.index !== 'number')
|
||||
return
|
||||
|
||||
token.content = rawContent.slice(0, leak.index)
|
||||
|
||||
const children = token.children as MarkdownToken[] | undefined
|
||||
if (!Array.isArray(children) || children.length === 0)
|
||||
return
|
||||
|
||||
// Best-effort sync: drop trailing text/break tokens that likely correspond
|
||||
// to the leaked marker and whitespace.
|
||||
while (children.length) {
|
||||
const last = children[children.length - 1]
|
||||
if (!last) {
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
if (last.type === 'softbreak' || last.type === 'hardbreak') {
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
if (last.type === 'text') {
|
||||
const lastContent = String(last.content ?? '')
|
||||
if (/^[ \t\r\n\d.)]*$/.test(lastContent)) {
|
||||
children.pop()
|
||||
continue
|
||||
}
|
||||
const next = lastContent.replace(/[ \t\r\n\d.)]+$/g, '')
|
||||
if (next !== lastContent) {
|
||||
if (next)
|
||||
last.content = next
|
||||
else
|
||||
children.pop()
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
export function parseList(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [ListNode, number] {
|
||||
const token = tokens[index]
|
||||
const listItems: ListItemNode[] = []
|
||||
let j = index + 1
|
||||
|
||||
while (
|
||||
j < tokens.length
|
||||
&& tokens[j].type !== 'bullet_list_close'
|
||||
&& tokens[j].type !== 'ordered_list_close'
|
||||
) {
|
||||
if (tokens[j].type === 'list_item_open') {
|
||||
// if (tokens[j].markup === '*') {
|
||||
// j++
|
||||
// continue
|
||||
// }
|
||||
const itemChildren: ParsedNode[] = []
|
||||
let k = j + 1
|
||||
while (k < tokens.length && tokens[k].type !== 'list_item_close') {
|
||||
// Handle different block types inside list items
|
||||
if (tokens[k].type === 'paragraph_open') {
|
||||
const contentToken = tokens[k + 1]
|
||||
const preToken = tokens[k - 1]
|
||||
stripLeakedOrderedListMarkerSuffix(contentToken)
|
||||
trimInlineTokenTail(contentToken)
|
||||
itemChildren.push({
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(contentToken.children || [], String(contentToken.content ?? ''), preToken, options as any),
|
||||
raw: String(contentToken.content ?? ''),
|
||||
})
|
||||
k += 3 // Skip paragraph_open, inline, paragraph_close
|
||||
}
|
||||
else if (tokens[k].type === 'blockquote_open') {
|
||||
// Parse blockquote within list item
|
||||
const [blockquoteNode, newIndex] = parseBlockquote(tokens, k, options)
|
||||
itemChildren.push(blockquoteNode)
|
||||
k = newIndex
|
||||
}
|
||||
else if (
|
||||
tokens[k].type === 'bullet_list_open'
|
||||
|| tokens[k].type === 'ordered_list_open'
|
||||
) {
|
||||
const [nestedListNode, newIndex] = parseList(tokens, k, options)
|
||||
itemChildren.push(nestedListNode)
|
||||
k = newIndex
|
||||
}
|
||||
else {
|
||||
const handled = parseCommonBlockToken(tokens, k, options, containerTokenHandlers)
|
||||
if (handled) {
|
||||
itemChildren.push(handled[0])
|
||||
k = handled[1]
|
||||
}
|
||||
else {
|
||||
k += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
listItems.push({
|
||||
type: 'list_item',
|
||||
children: itemChildren,
|
||||
raw: itemChildren.map(child => child.raw).join(''),
|
||||
})
|
||||
|
||||
j = k + 1 // Move past list_item_close
|
||||
}
|
||||
else {
|
||||
j += 1
|
||||
}
|
||||
}
|
||||
|
||||
const listNode: ListNode = {
|
||||
type: 'list',
|
||||
ordered: token.type === 'ordered_list_open',
|
||||
// markdown-it may include attrs like [['start','2']] on ordered_list_open
|
||||
start: (() => {
|
||||
if (token.attrs && token.attrs.length) {
|
||||
const found = token.attrs.find(a => a[0] === 'start')
|
||||
if (found) {
|
||||
const parsed = Number(found[1])
|
||||
return Number.isFinite(parsed) && parsed !== 0 ? parsed : 1
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
})(),
|
||||
items: listItems,
|
||||
raw: listItems.map(item => item.raw).join('\n'),
|
||||
}
|
||||
|
||||
return [listNode, j + 1] // Move past list_close
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
import type { MarkdownToken, MathBlockNode } from '../../types'
|
||||
|
||||
// Parse a math_block token (block/display math expressions)
|
||||
export function parseMathBlock(token: MarkdownToken): MathBlockNode {
|
||||
const content = String(token.content ?? '')
|
||||
const raw = token.raw === '$$' ? `$$${content}$$` : String(token.raw ?? '')
|
||||
return {
|
||||
type: 'math_block',
|
||||
content,
|
||||
loading: !!token.loading,
|
||||
raw,
|
||||
markup: token.markup,
|
||||
}
|
||||
}
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
import type { MarkdownToken, ParagraphNode, ParseOptions } from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
|
||||
export function parseParagraph(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): ParagraphNode {
|
||||
const paragraphContentToken = tokens[index + 1]
|
||||
const paragraphContent = String(paragraphContentToken.content ?? '')
|
||||
|
||||
return {
|
||||
type: 'paragraph',
|
||||
children: parseInlineTokens(paragraphContentToken.children || [], paragraphContent, undefined, options as any),
|
||||
raw: paragraphContent,
|
||||
}
|
||||
}
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
import type {
|
||||
MarkdownToken,
|
||||
ParseOptions,
|
||||
TableCellNode,
|
||||
TableNode,
|
||||
TableRowNode,
|
||||
} from '../../types'
|
||||
import { parseInlineTokens } from '../inline-parsers'
|
||||
|
||||
// Extract alignment from attrs (e.g. ['style','text-align:left'])
|
||||
function extractAlign(attrs: MarkdownToken['attrs']): 'left' | 'right' | 'center' | undefined {
|
||||
if (!attrs)
|
||||
return 'left'
|
||||
for (const a of attrs) {
|
||||
if (!a)
|
||||
continue
|
||||
const [key, val] = a
|
||||
if (!val)
|
||||
continue
|
||||
const value = String(val).trim().toLowerCase()
|
||||
if (key === 'style') {
|
||||
const m = /text-align\s*:\s*(left|right|center)/i.exec(value)
|
||||
if (m)
|
||||
return m[1].toLowerCase() as any
|
||||
}
|
||||
}
|
||||
return 'left'
|
||||
}
|
||||
export function parseTable(
|
||||
tokens: MarkdownToken[],
|
||||
index: number,
|
||||
options?: ParseOptions,
|
||||
): [TableNode, number] {
|
||||
let j = index + 1
|
||||
let headerRow: TableRowNode | null = null
|
||||
const rows: TableRowNode[] = []
|
||||
let isHeader = false
|
||||
while (j < tokens.length && tokens[j].type !== 'table_close') {
|
||||
if (tokens[j].type === 'thead_open') {
|
||||
isHeader = true
|
||||
j++
|
||||
}
|
||||
else if (tokens[j].type === 'thead_close') {
|
||||
isHeader = false
|
||||
j++
|
||||
}
|
||||
else if (
|
||||
tokens[j].type === 'tbody_open'
|
||||
|| tokens[j].type === 'tbody_close'
|
||||
) {
|
||||
j++
|
||||
}
|
||||
else if (tokens[j].type === 'tr_open') {
|
||||
const cells: TableCellNode[] = []
|
||||
let k = j + 1
|
||||
|
||||
while (k < tokens.length && tokens[k].type !== 'tr_close') {
|
||||
if (tokens[k].type === 'th_open' || tokens[k].type === 'td_open') {
|
||||
const isHeaderCell = tokens[k].type === 'th_open'
|
||||
const contentToken = tokens[k + 1]
|
||||
const content = String(contentToken.content ?? '')
|
||||
const align = extractAlign(tokens[k].attrs)
|
||||
|
||||
cells.push({
|
||||
type: 'table_cell',
|
||||
header: isHeaderCell || isHeader,
|
||||
children: parseInlineTokens(contentToken.children || [], content, undefined, options as any),
|
||||
raw: content,
|
||||
align,
|
||||
})
|
||||
|
||||
k += 3 // Skip th_open/td_open, inline, th_close/td_close
|
||||
}
|
||||
else {
|
||||
k++
|
||||
}
|
||||
}
|
||||
|
||||
const rowNode: TableRowNode = {
|
||||
type: 'table_row',
|
||||
cells,
|
||||
raw: cells.map(cell => cell.raw).join('|'),
|
||||
}
|
||||
|
||||
if (isHeader) {
|
||||
headerRow = rowNode
|
||||
}
|
||||
else {
|
||||
rows.push(rowNode)
|
||||
}
|
||||
|
||||
j = k + 1 // Skip tr_close
|
||||
}
|
||||
else {
|
||||
j++
|
||||
}
|
||||
}
|
||||
|
||||
if (!headerRow) {
|
||||
// Default empty header if none found
|
||||
headerRow = {
|
||||
type: 'table_row',
|
||||
cells: [],
|
||||
raw: '',
|
||||
}
|
||||
}
|
||||
|
||||
const tableNode: TableNode = {
|
||||
type: 'table',
|
||||
header: headerRow,
|
||||
rows,
|
||||
loading: tokens[index].loading ?? false,
|
||||
raw: [headerRow, ...rows].map(row => row.raw).join('\n'),
|
||||
}
|
||||
|
||||
return [tableNode, j + 1] // Skip table_close
|
||||
}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
import type { ThematicBreakNode } from '../../types'
|
||||
|
||||
export function parseThematicBreak(): ThematicBreakNode {
|
||||
return {
|
||||
type: 'thematic_break',
|
||||
raw: '---',
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
import markdownItContainer from 'markdown-it-container'
|
||||
|
||||
function parseLooseInlineAttrs(input: string): Record<string, unknown> | null {
|
||||
const s = String(input ?? '').trim()
|
||||
if (!s.startsWith('{') || !s.endsWith('}'))
|
||||
return null
|
||||
|
||||
const inner = s.slice(1, -1).trim()
|
||||
if (!inner)
|
||||
return {}
|
||||
|
||||
// Only support a shallow object literal: {key:value, foo:"bar"}
|
||||
// No nested objects/arrays/functions.
|
||||
if (inner.includes('{') || inner.includes('[') || inner.includes(']'))
|
||||
return null
|
||||
|
||||
const parts: string[] = []
|
||||
let buf = ''
|
||||
let inSingle = false
|
||||
let inDouble = false
|
||||
for (let i = 0; i < inner.length; i++) {
|
||||
const ch = inner[i]
|
||||
if (ch === '\\') {
|
||||
buf += ch
|
||||
if (i + 1 < inner.length) {
|
||||
buf += inner[i + 1]
|
||||
i++
|
||||
}
|
||||
continue
|
||||
}
|
||||
if (!inDouble && ch === '\'') {
|
||||
inSingle = !inSingle
|
||||
buf += ch
|
||||
continue
|
||||
}
|
||||
if (!inSingle && ch === '"') {
|
||||
inDouble = !inDouble
|
||||
buf += ch
|
||||
continue
|
||||
}
|
||||
if (!inSingle && !inDouble && ch === ',') {
|
||||
parts.push(buf.trim())
|
||||
buf = ''
|
||||
continue
|
||||
}
|
||||
buf += ch
|
||||
}
|
||||
if (buf.trim())
|
||||
parts.push(buf.trim())
|
||||
|
||||
const out: Record<string, unknown> = {}
|
||||
for (const part of parts) {
|
||||
if (!part)
|
||||
continue
|
||||
let inS = false
|
||||
let inD = false
|
||||
let split = -1
|
||||
for (let i = 0; i < part.length; i++) {
|
||||
const ch = part[i]
|
||||
if (ch === '\\') {
|
||||
i++
|
||||
continue
|
||||
}
|
||||
if (!inD && ch === '\'') {
|
||||
inS = !inS
|
||||
continue
|
||||
}
|
||||
if (!inS && ch === '"') {
|
||||
inD = !inD
|
||||
continue
|
||||
}
|
||||
if (!inS && !inD && ch === ':') {
|
||||
split = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if (split === -1)
|
||||
return null
|
||||
|
||||
const rawKey = part.slice(0, split).trim()
|
||||
const rawVal = part.slice(split + 1).trim()
|
||||
if (!rawKey)
|
||||
return null
|
||||
|
||||
let key = rawKey
|
||||
if ((key.startsWith('"') && key.endsWith('"')) || (key.startsWith('\'') && key.endsWith('\''))) {
|
||||
try {
|
||||
key = JSON.parse(key.replace(/^'/, '"').replace(/'$/, '"'))
|
||||
}
|
||||
catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
// Restrict unquoted keys to simple identifiers.
|
||||
if (!/^[_$A-Z][\w$-]*$/i.test(key))
|
||||
return null
|
||||
|
||||
let value: unknown
|
||||
if (!rawVal) {
|
||||
value = ''
|
||||
}
|
||||
else if ((rawVal.startsWith('"') && rawVal.endsWith('"')) || (rawVal.startsWith('\'') && rawVal.endsWith('\''))) {
|
||||
try {
|
||||
value = JSON.parse(rawVal.replace(/^'/, '"').replace(/'$/, '"'))
|
||||
}
|
||||
catch {
|
||||
value = rawVal
|
||||
}
|
||||
}
|
||||
else if (/^-?\d+(?:\.\d+)?$/.test(rawVal)) {
|
||||
value = Number(rawVal)
|
||||
}
|
||||
else if (rawVal === 'true' || rawVal === 'false') {
|
||||
value = rawVal === 'true'
|
||||
}
|
||||
else if (rawVal === 'null') {
|
||||
value = null
|
||||
}
|
||||
else {
|
||||
value = rawVal
|
||||
}
|
||||
|
||||
out[key] = value
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
export function applyContainers(md: MarkdownIt) {
|
||||
;[
|
||||
'admonition',
|
||||
'info',
|
||||
'warning',
|
||||
'error',
|
||||
'tip',
|
||||
'danger',
|
||||
'note',
|
||||
'caution',
|
||||
].forEach((name) => {
|
||||
md.use(markdownItContainer, name, {
|
||||
render(tokens: unknown, idx: number) {
|
||||
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
|
||||
const token = tokensAny[idx]
|
||||
// `nesting` is a runtime-only property present on MarkdownIt tokens.
|
||||
// Narrow the shape with `unknown` -> specific minimal interface to avoid `as any`.
|
||||
const tokenShape = token as unknown as { nesting?: number }
|
||||
if (tokenShape.nesting === 1) {
|
||||
return `<div class="vmr-container vmr-container-${name}">`
|
||||
}
|
||||
else {
|
||||
return '</div>\n'
|
||||
}
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
// fallback for simple ::: blocks (kept for backwards compat)
|
||||
md.block.ruler.before(
|
||||
'fence',
|
||||
'vmr_container_fallback',
|
||||
(state: unknown, startLine: number, endLine: number, silent: boolean) => {
|
||||
interface ParserState {
|
||||
bMarks: number[]
|
||||
tShift: number[]
|
||||
eMarks: number[]
|
||||
src: string
|
||||
env: unknown
|
||||
push: (type: string, tag?: string, nesting?: number) => any
|
||||
tokens: any[]
|
||||
md: any
|
||||
line: number
|
||||
}
|
||||
const s = state as unknown as ParserState
|
||||
const startPos = s.bMarks[startLine] + s.tShift[startLine]
|
||||
const lineMax = s.eMarks[startLine]
|
||||
const line = s.src.slice(startPos, lineMax)
|
||||
|
||||
// Match ::: container syntax: ::: name {"json"}
|
||||
// Using separate patterns to avoid backtracking issues
|
||||
// Allow both "::: name" and ":::name", and stop before inline JSON attrs.
|
||||
const nameMatch = line.match(/^:::\s*([^\s{]+)/)
|
||||
if (!nameMatch)
|
||||
return false
|
||||
|
||||
const name = nameMatch[1]
|
||||
// Validate name is not empty (handles edge case like "::: ")
|
||||
if (!name.trim())
|
||||
return false
|
||||
|
||||
const rest = line.slice(nameMatch[0].length)
|
||||
const trimmedRest = rest.trim()
|
||||
|
||||
// Support both:
|
||||
// - ::: name {"json"}
|
||||
// - :::name args {"json"}
|
||||
// - :::name args
|
||||
let argsStr: string | undefined
|
||||
let jsonStr: string | undefined
|
||||
|
||||
const jsonStart = trimmedRest.indexOf('{')
|
||||
const jsonCandidate = jsonStart >= 0 ? trimmedRest.slice(jsonStart).trimStart() : undefined
|
||||
|
||||
if (jsonStart === -1) {
|
||||
argsStr = trimmedRest || undefined
|
||||
}
|
||||
else {
|
||||
const before = trimmedRest.slice(0, jsonStart).trim()
|
||||
argsStr = before || undefined
|
||||
|
||||
// Improved JSON matching: find balanced braces instead of simple non-greedy match
|
||||
if (jsonCandidate?.startsWith('{')) {
|
||||
let depth = 0
|
||||
let jsonEnd = -1
|
||||
for (let i = 0; i < jsonCandidate.length; i++) {
|
||||
if (jsonCandidate[i] === '{')
|
||||
depth++
|
||||
else if (jsonCandidate[i] === '}')
|
||||
depth--
|
||||
if (depth === 0) {
|
||||
jsonEnd = i + 1
|
||||
break
|
||||
}
|
||||
}
|
||||
if (jsonEnd > 0) {
|
||||
jsonStr = jsonCandidate.slice(0, jsonEnd)
|
||||
}
|
||||
}
|
||||
|
||||
// Streaming mid-state: if we saw a '{' but didn't find a balanced '}',
|
||||
// treat the whole remainder as a plain args string.
|
||||
// Example: "::: viewcode:stream {xxx:yyy" => attrs.args === "{xxx:yyy"
|
||||
if (!jsonStr)
|
||||
argsStr = trimmedRest || undefined
|
||||
}
|
||||
|
||||
if (silent)
|
||||
return true
|
||||
|
||||
const envFinal = !!(s.env as any)?.__markstreamFinal
|
||||
|
||||
let nextLine = startLine + 1
|
||||
let found = false
|
||||
while (nextLine <= endLine) {
|
||||
const sPos = s.bMarks[nextLine] + s.tShift[nextLine]
|
||||
const ePos = s.eMarks[nextLine]
|
||||
if (s.src.slice(sPos, ePos).trim() === ':::') {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
nextLine++
|
||||
}
|
||||
// Streaming optimization:
|
||||
// - If we have an opening ":::name" but no closing ":::" yet, still emit a container.
|
||||
// Treat the remainder of the document as the container body.
|
||||
// - The structured parser will expose this as `loading: true` until it closes,
|
||||
// or until `final=true` is provided.
|
||||
if (!found)
|
||||
nextLine = endLine
|
||||
|
||||
const tokenOpen = s.push('vmr_container_open', 'div', 1)
|
||||
// `tokenOpen` is runtime token object; keep using runtime helpers but avoid casting `s` to `any`.
|
||||
tokenOpen.attrSet('class', `vmr-container vmr-container-${name}`)
|
||||
|
||||
// Mark unclosed containers for downstream consumers (optional).
|
||||
;(tokenOpen.meta as any) = { ...(tokenOpen.meta as any), unclosed: !found && !envFinal }
|
||||
|
||||
// If args are present (non-JSON payload right after the name), preserve them as a data attribute.
|
||||
if (argsStr)
|
||||
tokenOpen.attrSet('data-args', argsStr)
|
||||
|
||||
// If JSON attributes are present, store them as data attributes
|
||||
if (jsonStr) {
|
||||
try {
|
||||
const attrs = JSON.parse(jsonStr)
|
||||
for (const [key, value] of Object.entries(attrs)) {
|
||||
const isComplexValue = value != null && typeof value === 'object'
|
||||
tokenOpen.attrSet(
|
||||
`data-${key}`,
|
||||
isComplexValue ? JSON.stringify(value) : String(value),
|
||||
)
|
||||
}
|
||||
}
|
||||
catch {
|
||||
// If strict JSON parsing fails, try a small loose-object syntax: {xxx:yyy}
|
||||
const loose = parseLooseInlineAttrs(jsonStr)
|
||||
if (loose) {
|
||||
for (const [key, value] of Object.entries(loose)) {
|
||||
const isComplexValue = value != null && typeof value === 'object'
|
||||
tokenOpen.attrSet(
|
||||
`data-${key}`,
|
||||
isComplexValue ? JSON.stringify(value) : String(value),
|
||||
)
|
||||
}
|
||||
}
|
||||
else {
|
||||
// If parsing fails, store the raw string as a data attribute
|
||||
tokenOpen.attrSet('data-attrs', jsonStr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Parse inner content as full block markdown so block-level constructs
|
||||
// (especially fenced code blocks) are preserved inside the container.
|
||||
const contentLines: string[] = []
|
||||
for (let i = startLine + 1; i < nextLine; i++) {
|
||||
const sPos = s.bMarks[i] + s.tShift[i]
|
||||
const ePos = s.eMarks[i]
|
||||
contentLines.push(s.src.slice(sPos, ePos))
|
||||
}
|
||||
const hasContent = contentLines.some(line => line.trim().length > 0)
|
||||
if (hasContent) {
|
||||
// Always end with two newlines to ensure block separation (esp. for lists/paragraphs)
|
||||
let innerSrc = contentLines.join('\n')
|
||||
if (!innerSrc.endsWith('\n'))
|
||||
innerSrc += '\n'
|
||||
if (!innerSrc.endsWith('\n\n'))
|
||||
innerSrc += '\n'
|
||||
|
||||
// The last token should be the current vm_container_open token
|
||||
const prevToken = s.tokens[s.tokens.length - 1]
|
||||
|
||||
// Save the inner Markdown content as raw
|
||||
if (prevToken) {
|
||||
prevToken.raw = innerSrc
|
||||
}
|
||||
|
||||
const innerTokens: any[] = []
|
||||
// Use the same env as the parent block parser to ensure all block rules are available
|
||||
s.md.block.parse(innerSrc, s.md, s.env, innerTokens)
|
||||
s.tokens.push(...innerTokens)
|
||||
}
|
||||
|
||||
if (found)
|
||||
s.push('vmr_container_close', 'div', -1)
|
||||
|
||||
s.line = found ? (nextLine + 1) : nextLine
|
||||
return true
|
||||
},
|
||||
{
|
||||
// Ensure this rule can terminate an active paragraph so content immediately
|
||||
// above `:::` isn't swallowed into the same paragraph block.
|
||||
alt: ['paragraph', 'reference', 'blockquote', 'list'],
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,996 @@
|
||||
import type { MarkdownIt, Token } from 'markdown-it-ts'
|
||||
import { normalizeCustomHtmlTagName } from '../customHtmlTags'
|
||||
import { STANDARD_BLOCK_HTML_TAGS, STANDARD_HTML_TAGS, VOID_HTML_TAGS } from '../htmlTags'
|
||||
import { escapeTagForRegExp, findTagCloseIndexOutsideQuotes } from '../htmlTagUtils'
|
||||
|
||||
const VOID_TAGS = VOID_HTML_TAGS
|
||||
|
||||
// A conservative set of common HTML tags used to detect streaming mid-states.
|
||||
// We only suppress/merge partial tags for these names to avoid false positives
|
||||
// (e.g., autolinks like <http://...>).
|
||||
const BASE_COMMON_HTML_TAGS = STANDARD_HTML_TAGS
|
||||
|
||||
const BLOCK_LEVEL_HTML_TAGS = new Set<string>(STANDARD_BLOCK_HTML_TAGS)
|
||||
BLOCK_LEVEL_HTML_TAGS.delete('details')
|
||||
|
||||
const OPEN_TAG_RE = /<([A-Z][\w-]*)(?=[\s/>]|$)/gi
|
||||
const CLOSE_TAG_RE = /<\/\s*([A-Z][\w-]*)(?=[\s/>]|$)/gi
|
||||
const TAG_NAME_AT_START_RE = /^<\s*(?:\/\s*)?([A-Z][\w-]*)/i
|
||||
const STRICT_OPEN_TAG_NAME_AT_START_RE = /^<\s*([A-Z][\w:-]*)(?=[\s/>]|$)/i
|
||||
|
||||
function getHtmlInlineTagName(content: string) {
|
||||
return (content.match(TAG_NAME_AT_START_RE)?.[1] ?? '').toLowerCase()
|
||||
}
|
||||
|
||||
function isHtmlInlineClosingTag(content: string) {
|
||||
return /^\s*<\s*\//.test(content)
|
||||
}
|
||||
|
||||
function isSelfClosingHtmlInline(content: string, tag: string) {
|
||||
return VOID_TAGS.has(tag) || /\/\s*>\s*$/.test(content)
|
||||
}
|
||||
|
||||
function findMatchingCloseChildIndex(
|
||||
children: Array<{ type?: string, content?: string }>,
|
||||
tag: string,
|
||||
) {
|
||||
let depth = 0
|
||||
|
||||
for (let index = 0; index < children.length; index++) {
|
||||
const child = children[index]
|
||||
if (!child || child.type !== 'html_inline')
|
||||
continue
|
||||
|
||||
const content = String(child.content ?? '')
|
||||
const childTag = getHtmlInlineTagName(content)
|
||||
if (childTag !== tag)
|
||||
continue
|
||||
|
||||
if (isHtmlInlineClosingTag(content)) {
|
||||
if (depth === 0)
|
||||
return index
|
||||
depth--
|
||||
continue
|
||||
}
|
||||
|
||||
if (!isSelfClosingHtmlInline(content, childTag))
|
||||
depth++
|
||||
}
|
||||
|
||||
return -1
|
||||
}
|
||||
|
||||
function getTrailingOpenDepth(
|
||||
children: Array<{ type?: string, content?: string }>,
|
||||
tag: string,
|
||||
) {
|
||||
let depth = 0
|
||||
|
||||
for (const child of children) {
|
||||
if (!child || child.type !== 'html_inline')
|
||||
continue
|
||||
|
||||
const content = String(child.content ?? '')
|
||||
const childTag = getHtmlInlineTagName(content)
|
||||
if (childTag !== tag)
|
||||
continue
|
||||
|
||||
if (isHtmlInlineClosingTag(content)) {
|
||||
if (depth > 0)
|
||||
depth--
|
||||
continue
|
||||
}
|
||||
|
||||
if (!isSelfClosingHtmlInline(content, childTag))
|
||||
depth++
|
||||
}
|
||||
|
||||
return depth
|
||||
}
|
||||
|
||||
function findMatchingCloseRangeInHtml(content: string, tag: string, startIndex = 0) {
|
||||
const tokenRe = new RegExp(String.raw`<\s*(\/?)\s*${escapeTagForRegExp(tag)}(?=[\s>/])[^>]*>`, 'gi')
|
||||
tokenRe.lastIndex = Math.max(0, startIndex)
|
||||
let depth = 0
|
||||
let match: RegExpExecArray | null
|
||||
|
||||
while ((match = tokenRe.exec(content)) !== null) {
|
||||
const raw = match[0] ?? ''
|
||||
const closing = !!match[1]
|
||||
const selfClosing = !closing && /\/\s*>$/.test(raw)
|
||||
|
||||
if (closing) {
|
||||
if (depth === 0) {
|
||||
return {
|
||||
start: match.index,
|
||||
end: match.index + raw.length,
|
||||
}
|
||||
}
|
||||
depth--
|
||||
continue
|
||||
}
|
||||
|
||||
if (!selfClosing)
|
||||
depth++
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
function getTrailingCustomTagDepthInHtml(content: string, tag: string) {
|
||||
const tokenRe = new RegExp(String.raw`<\s*(\/?)\s*${escapeTagForRegExp(tag)}(?=[\s>/])[^>]*>`, 'gi')
|
||||
let depth = 0
|
||||
let match: RegExpExecArray | null
|
||||
|
||||
while ((match = tokenRe.exec(content)) !== null) {
|
||||
const raw = match[0] ?? ''
|
||||
const closing = !!match[1]
|
||||
const selfClosing = !closing && /\/\s*>$/.test(raw)
|
||||
|
||||
if (closing) {
|
||||
if (depth > 0)
|
||||
depth--
|
||||
continue
|
||||
}
|
||||
|
||||
if (!selfClosing)
|
||||
depth++
|
||||
}
|
||||
|
||||
return depth
|
||||
}
|
||||
|
||||
function tokenToRaw(token: Token) {
|
||||
const shape = token as unknown as { raw?: string, markup?: string, content?: string }
|
||||
return String(shape.raw ?? shape.content ?? shape.markup ?? '')
|
||||
}
|
||||
|
||||
function isNonElementHtmlBlock(content: string) {
|
||||
return /^\s*<\s*[!?]/.test(content)
|
||||
}
|
||||
|
||||
function buildCommonHtmlTagSet(extraTags?: readonly string[]) {
|
||||
const set = new Set(BASE_COMMON_HTML_TAGS)
|
||||
if (extraTags && Array.isArray(extraTags)) {
|
||||
for (const t of extraTags) {
|
||||
const raw = String(t ?? '').trim()
|
||||
if (!raw)
|
||||
continue
|
||||
const m = raw.match(/^[<\s/]*([A-Z][\w-]*)/i)
|
||||
if (!m)
|
||||
continue
|
||||
set.add(m[1].toLowerCase())
|
||||
}
|
||||
}
|
||||
return set
|
||||
}
|
||||
|
||||
function isCommonHtmlTagOrPrefix(tag: string, tagSet: Set<string>) {
|
||||
if (tagSet.has(tag))
|
||||
return true
|
||||
for (const common of tagSet) {
|
||||
if (common.startsWith(tag))
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
function findFirstIncompleteTag(content: string, tagSet: Set<string>) {
|
||||
let first:
|
||||
| { index: number, tag: string, closing: boolean }
|
||||
| null = null
|
||||
|
||||
for (const m of content.matchAll(OPEN_TAG_RE)) {
|
||||
const idx = m.index ?? -1
|
||||
if (idx < 0)
|
||||
continue
|
||||
const tag = (m[1] ?? '').toLowerCase()
|
||||
// For opening tags we also accept prefixes of known HTML tags
|
||||
// (e.g., '<fo' while typing '<font ...>').
|
||||
if (!isCommonHtmlTagOrPrefix(tag, tagSet))
|
||||
continue
|
||||
const rest = content.slice(idx)
|
||||
if (findTagCloseIndexOutsideQuotes(rest) !== -1)
|
||||
continue
|
||||
if (!first || idx < first.index)
|
||||
first = { index: idx, tag, closing: false }
|
||||
}
|
||||
|
||||
for (const m of content.matchAll(CLOSE_TAG_RE)) {
|
||||
const idx = m.index ?? -1
|
||||
if (idx < 0)
|
||||
continue
|
||||
const tag = (m[1] ?? '').toLowerCase()
|
||||
// For closing tags we also accept prefixes of known HTML tags
|
||||
// (e.g., '</sp' while typing '</span>').
|
||||
if (!isCommonHtmlTagOrPrefix(tag, tagSet))
|
||||
continue
|
||||
const rest = content.slice(idx)
|
||||
if (findTagCloseIndexOutsideQuotes(rest) !== -1)
|
||||
continue
|
||||
if (!first || idx < first.index)
|
||||
first = { index: idx, tag, closing: true }
|
||||
}
|
||||
|
||||
// Also swallow bare "<" or "</" at the end while typing.
|
||||
const bareClose = /<\/\s*$/.exec(content)
|
||||
if (bareClose && typeof bareClose.index === 'number') {
|
||||
const idx = bareClose.index
|
||||
const rest = content.slice(idx)
|
||||
if (!rest.includes('>') && (!first || idx < first.index))
|
||||
first = { index: idx, tag: '', closing: true }
|
||||
}
|
||||
|
||||
const bareOpen = /<\s*$/.exec(content)
|
||||
if (bareOpen && typeof bareOpen.index === 'number') {
|
||||
const idx = bareOpen.index
|
||||
const rest = content.slice(idx)
|
||||
// Avoid matching "</" which is handled above.
|
||||
if (!rest.startsWith('</') && !rest.includes('>') && (!first || idx < first.index))
|
||||
first = { index: idx, tag: '', closing: false }
|
||||
}
|
||||
|
||||
return first
|
||||
}
|
||||
|
||||
function splitTextToken(token: Token, content: string) {
|
||||
const t = token as Token & { content?: string, raw?: string }
|
||||
// Preserve the original Token prototype (markdown-it-ts attaches helper methods).
|
||||
const nt = Object.assign(
|
||||
Object.create(Object.getPrototypeOf(t)),
|
||||
t,
|
||||
{ type: 'text', content, raw: content },
|
||||
) as Token
|
||||
return nt
|
||||
}
|
||||
|
||||
function fixStreamingHtmlInlineChildren(children: Token[], tagSet: Set<string>) {
|
||||
if (!children.length)
|
||||
return { children }
|
||||
|
||||
const out: Token[] = []
|
||||
let pending: { tag: string, buffer: string, closing: boolean } | null = null
|
||||
let pendingAtEnd: string | null = null
|
||||
|
||||
function pushTextPart(text: string, baseToken?: Token) {
|
||||
if (!text)
|
||||
return
|
||||
if (baseToken)
|
||||
out.push(splitTextToken(baseToken, text))
|
||||
else
|
||||
out.push({ type: 'text', content: text, raw: text } as any)
|
||||
}
|
||||
|
||||
function splitCompleteHtmlFromText(chunk: string, baseToken?: Token) {
|
||||
let cursor = 0
|
||||
while (cursor < chunk.length) {
|
||||
const lt = chunk.indexOf('<', cursor)
|
||||
if (lt === -1) {
|
||||
pushTextPart(chunk.slice(cursor), baseToken)
|
||||
break
|
||||
}
|
||||
pushTextPart(chunk.slice(cursor, lt), baseToken)
|
||||
const sub = chunk.slice(lt)
|
||||
const tagMatch = sub.match(TAG_NAME_AT_START_RE)
|
||||
if (!tagMatch) {
|
||||
pushTextPart('<', baseToken)
|
||||
cursor = lt + 1
|
||||
continue
|
||||
}
|
||||
const closeIdx = findTagCloseIndexOutsideQuotes(sub)
|
||||
if (closeIdx === -1) {
|
||||
pushTextPart('<', baseToken)
|
||||
cursor = lt + 1
|
||||
continue
|
||||
}
|
||||
|
||||
const tagText = sub.slice(0, closeIdx + 1)
|
||||
const tagName = (tagMatch[1] ?? '').toLowerCase()
|
||||
if (tagSet.has(tagName)) {
|
||||
out.push({
|
||||
type: 'html_inline',
|
||||
tag: '',
|
||||
content: tagText,
|
||||
raw: tagText,
|
||||
} as any)
|
||||
}
|
||||
else {
|
||||
pushTextPart(tagText, baseToken)
|
||||
}
|
||||
cursor = lt + tagText.length
|
||||
}
|
||||
}
|
||||
|
||||
function processTextChunk(chunk: string, baseToken?: Token) {
|
||||
if (!chunk)
|
||||
return
|
||||
const match = findFirstIncompleteTag(chunk, tagSet)
|
||||
if (!match) {
|
||||
splitCompleteHtmlFromText(chunk, baseToken)
|
||||
return
|
||||
}
|
||||
|
||||
const before = chunk.slice(0, match.index)
|
||||
if (before)
|
||||
splitCompleteHtmlFromText(before, baseToken)
|
||||
pending = {
|
||||
tag: match.tag,
|
||||
buffer: chunk.slice(match.index),
|
||||
closing: match.closing,
|
||||
}
|
||||
pendingAtEnd = pending.buffer
|
||||
}
|
||||
|
||||
for (const child of children) {
|
||||
if (pending) {
|
||||
pending.buffer += tokenToRaw(child)
|
||||
pendingAtEnd = pending.buffer
|
||||
const closeIdx = findTagCloseIndexOutsideQuotes(pending.buffer)
|
||||
if (closeIdx === -1) {
|
||||
// still incomplete: swallow this token to avoid rendering jitter
|
||||
continue
|
||||
}
|
||||
|
||||
const tagChunk = pending.buffer.slice(0, closeIdx + 1)
|
||||
const afterChunk = pending.buffer.slice(closeIdx + 1)
|
||||
out.push({
|
||||
type: 'html_inline',
|
||||
tag: '',
|
||||
content: tagChunk,
|
||||
raw: tagChunk,
|
||||
} as any)
|
||||
pending = null
|
||||
pendingAtEnd = null
|
||||
if (afterChunk)
|
||||
processTextChunk(afterChunk)
|
||||
continue
|
||||
}
|
||||
|
||||
if (child.type === 'html_inline') {
|
||||
const content = tokenToRaw(child)
|
||||
const tagMatch = content.match(TAG_NAME_AT_START_RE)
|
||||
const tagName = (tagMatch?.[1] ?? '').toLowerCase()
|
||||
if (tagName && tagSet.has(tagName) && findTagCloseIndexOutsideQuotes(content) === -1) {
|
||||
// markdown-it may prematurely close a tag at a ">" inside a quoted
|
||||
// attribute value (e.g. `<a href="...a>b`), producing a broken html_inline
|
||||
// token. Treat it as a streaming mid-state and swallow until we see a
|
||||
// real tag close ">" outside quotes.
|
||||
pending = {
|
||||
tag: tagName,
|
||||
buffer: content,
|
||||
closing: /^<\s*\//.test(content),
|
||||
}
|
||||
pendingAtEnd = pending.buffer
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if (child.type === 'text') {
|
||||
const content = String((child as any).content ?? '')
|
||||
if (!content.includes('<')) {
|
||||
out.push(child)
|
||||
continue
|
||||
}
|
||||
processTextChunk(content, child)
|
||||
continue
|
||||
}
|
||||
|
||||
out.push(child)
|
||||
}
|
||||
|
||||
return {
|
||||
children: out,
|
||||
pendingBuffer: pendingAtEnd ?? undefined,
|
||||
}
|
||||
}
|
||||
|
||||
export interface FixHtmlInlineOptions {
|
||||
/**
|
||||
* Custom HTML-like tag names that should participate in streaming
|
||||
* mid-state suppression and complete-tag splitting (e.g. ['thinking']).
|
||||
*/
|
||||
customHtmlTags?: readonly string[]
|
||||
}
|
||||
|
||||
export function applyFixHtmlInlineTokens(md: MarkdownIt, options: FixHtmlInlineOptions = {}) {
|
||||
const commonHtmlTags = buildCommonHtmlTagSet(options.customHtmlTags)
|
||||
// Tags that should stay inline when we auto-append a closing tag at core stage.
|
||||
const autoCloseInlineTagSet = new Set<string>([
|
||||
'a',
|
||||
'span',
|
||||
'strong',
|
||||
'em',
|
||||
'b',
|
||||
'i',
|
||||
'u',
|
||||
])
|
||||
const customTagSet = new Set<string>()
|
||||
if (options.customHtmlTags?.length) {
|
||||
for (const t of options.customHtmlTags) {
|
||||
const name = normalizeCustomHtmlTagName(t)
|
||||
if (!name)
|
||||
continue
|
||||
customTagSet.add(name)
|
||||
autoCloseInlineTagSet.add(name)
|
||||
}
|
||||
}
|
||||
const shouldMergeHtmlBlockTag = (tag: string) => customTagSet.has(tag) || !commonHtmlTags.has(tag) || BLOCK_LEVEL_HTML_TAGS.has(tag)
|
||||
const getHtmlBlockCarrierContent = (token: Token & { content?: string, children?: any[] }) => {
|
||||
if (token.type === 'html_block')
|
||||
return String(token.content ?? '')
|
||||
if (token.type !== 'inline' || !Array.isArray(token.children) || token.children.length !== 1)
|
||||
return ''
|
||||
const onlyChild = token.children[0]
|
||||
if (onlyChild?.type !== 'html_block')
|
||||
return ''
|
||||
return String(token.content ?? onlyChild.content ?? '')
|
||||
}
|
||||
const normalizeHtmlBlockCarrier = (
|
||||
token: Token & { content?: string, children?: any[], raw?: string },
|
||||
content: string,
|
||||
) => {
|
||||
token.type = 'html_block'
|
||||
token.content = content
|
||||
token.raw = content
|
||||
token.children = []
|
||||
}
|
||||
// Streaming mid-state: suppress partial inline HTML in text tokens until the
|
||||
// tag is fully closed with `>`, then allow it to be tokenized as html_inline.
|
||||
md.core.ruler.after('inline', 'fix_html_inline_streaming', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: Token[] }
|
||||
const toks = s.tokens ?? []
|
||||
for (const t of toks) {
|
||||
const tok = t as Token & { children?: Token[], content?: string, raw?: string }
|
||||
if (tok.type !== 'inline' || !Array.isArray(tok.children))
|
||||
continue
|
||||
|
||||
// markdown-it-ts may emit inline tokens with empty children when the
|
||||
// content starts with an incomplete HTML-ish fragment like "<span ...".
|
||||
// In that case, synthesize a text token so we can suppress mid-states.
|
||||
const originalContent = String(tok.content ?? '')
|
||||
const sourceChildren = tok.children.length
|
||||
? tok.children
|
||||
: (originalContent.includes('<')
|
||||
? [{ type: 'text', content: originalContent, raw: originalContent } as any]
|
||||
: null)
|
||||
|
||||
if (!sourceChildren)
|
||||
continue
|
||||
|
||||
try {
|
||||
const fixed = fixStreamingHtmlInlineChildren(sourceChildren, commonHtmlTags)
|
||||
tok.children = fixed.children
|
||||
if (fixed.pendingBuffer) {
|
||||
const idx = originalContent.lastIndexOf(fixed.pendingBuffer)
|
||||
if (idx !== -1) {
|
||||
const trimmed = originalContent.slice(0, idx)
|
||||
tok.content = trimmed
|
||||
// keep raw in sync if present
|
||||
if (typeof tok.raw === 'string')
|
||||
tok.raw = trimmed
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (e) {
|
||||
console.error('[applyFixHtmlInlineTokens] failed to fix streaming html inline', e)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Fix certain single-token inline HTML cases by expanding into [openTag, text, closeTag]
|
||||
// This helps downstream inline parsers (e.g., <a>text</a>) to recognize inner text reliably.
|
||||
md.core.ruler.push('fix_html_inline_tokens', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: Token[] }
|
||||
const toks = s.tokens ?? []
|
||||
|
||||
// 有一些很特殊的场景,比如 html_block 开始 <thinking>,但是后面跟着很多段落,如果没匹配到</thinking>,中间的都应该合并为html_block的 content
|
||||
const tagStack: [string, number][] = []
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i] as Token & { content?: string, children: any[] }
|
||||
|
||||
// If we're currently inside an unclosed custom-tag html_block, merge
|
||||
// everything (including other html_block tokens) into the opener until
|
||||
// the matching closing tag arrives.
|
||||
if (tagStack.length > 0) {
|
||||
const [openTag, openIndex] = tagStack[tagStack.length - 1]
|
||||
if (i !== openIndex) {
|
||||
// Remove structural paragraph wrappers that can appear in stream mode.
|
||||
if (t.type === 'paragraph_open' || t.type === 'paragraph_close') {
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
continue
|
||||
}
|
||||
|
||||
const chunk = String((t as any).content ?? (t as any).raw ?? '')
|
||||
|
||||
if (chunk) {
|
||||
const openToken = toks[openIndex] as Token & { content?: string, loading?: boolean }
|
||||
const mergedContent = `${String(openToken.content || '')}\n${chunk}`
|
||||
const openEnd = findTagCloseIndexOutsideQuotes(mergedContent)
|
||||
const closeRange = openEnd === -1
|
||||
? null
|
||||
: findMatchingCloseRangeInHtml(mergedContent, openTag, openEnd + 1)
|
||||
if (closeRange) {
|
||||
const before = mergedContent.slice(0, closeRange.end)
|
||||
const after = mergedContent.slice(closeRange.end)
|
||||
|
||||
openToken.content = before
|
||||
openToken.loading = false
|
||||
|
||||
const afterTrimmed = after.replace(/^\s+/, '')
|
||||
// Remove current token after merging.
|
||||
toks.splice(i, 1)
|
||||
// Close the stack before reinserting trailing content.
|
||||
tagStack.pop()
|
||||
if (afterTrimmed) {
|
||||
toks.splice(i, 0, afterTrimmed.startsWith('<')
|
||||
? ({ type: 'html_block', content: afterTrimmed } as any)
|
||||
: ({ type: 'inline', content: afterTrimmed, children: [{ type: 'text', content: afterTrimmed, raw: afterTrimmed }] } as any))
|
||||
}
|
||||
i--
|
||||
continue
|
||||
}
|
||||
|
||||
openToken.content = mergedContent
|
||||
if (openToken.loading !== false)
|
||||
openToken.loading = true
|
||||
}
|
||||
|
||||
// Remove current token after merging.
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
const rawContent = getHtmlBlockCarrierContent(t)
|
||||
if (rawContent) {
|
||||
if (isNonElementHtmlBlock(rawContent))
|
||||
continue
|
||||
|
||||
// Support both opening (<tag ...>) and closing (</tag>) blocks.
|
||||
const tag = (rawContent.match(/<\s*(?:\/\s*)?([^\s>/]+)/)?.[1] ?? '').toLowerCase()
|
||||
const isClosingTag = /^\s*<\s*\//.test(rawContent)
|
||||
|
||||
// Merge configured custom tags and unknown HTML-like tags (e.g. <think>)
|
||||
// until their matching closing tag arrives. Also merge standard block
|
||||
// tags that markdown-it can split on blank lines, such as <details>.
|
||||
if (!tag || !shouldMergeHtmlBlockTag(tag))
|
||||
continue
|
||||
|
||||
normalizeHtmlBlockCarrier(t, rawContent)
|
||||
|
||||
if (!isClosingTag) {
|
||||
// 开始标签,入栈
|
||||
if (tag) {
|
||||
// If the html_block already contains its own closing tag, do NOT
|
||||
// push it onto the stack; otherwise we'd incorrectly merge the
|
||||
// following blocks into this html_block.
|
||||
const selfClosingRe = new RegExp(`^\\s*<\\s*${tag}\\b[^>]*\\/\\s*>`, 'i')
|
||||
if (!selfClosingRe.test(rawContent) && getTrailingCustomTagDepthInHtml(rawContent, tag) > 0)
|
||||
tagStack.push([tag, i])
|
||||
}
|
||||
}
|
||||
else {
|
||||
// 结束标签:如果匹配到栈顶,则把 closing token 也合并进 opener 并删除自己
|
||||
if (tagStack.length > 0 && tag && tagStack[tagStack.length - 1][0] === tag) {
|
||||
const [, openIndex] = tagStack[tagStack.length - 1]
|
||||
const openToken = toks[openIndex] as Token & { content?: string, loading?: boolean }
|
||||
openToken.content = `${String(openToken.content || '')}\n${rawContent}`
|
||||
openToken.loading = false
|
||||
tagStack.pop()
|
||||
|
||||
// Remove current closing html_block token so it doesn't become a stray node.
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
else if (tagStack.length > 0) {
|
||||
// 如果在标签栈中,说明是未闭合标签的内容,合并到上一个 html_block
|
||||
if (t.type === 'paragraph_open' || t.type === 'paragraph_close') {
|
||||
// 应该删除这些标签
|
||||
toks.splice(i, 1)
|
||||
i-- // 调整索引
|
||||
continue
|
||||
}
|
||||
const content = t.content || ''
|
||||
const CLOSING_TAG_REGEX = new RegExp(`<\\s*\\/\\s*${tagStack[tagStack.length - 1][0]}\\s*>`, 'i')
|
||||
const isClosingTag = CLOSING_TAG_REGEX.test(content)
|
||||
|
||||
if (content) {
|
||||
// 插入到栈顶标签对应的 html_block 中
|
||||
const [, openIndex] = tagStack[tagStack.length - 1]
|
||||
const openToken = toks[openIndex] as Token & { content?: string, loading: boolean }
|
||||
openToken.content = `${openToken.content || ''}\n${content}`
|
||||
if (openToken.loading !== false)
|
||||
openToken.loading = !isClosingTag
|
||||
}
|
||||
if (isClosingTag) {
|
||||
tagStack.pop()
|
||||
}
|
||||
// 删除当前 token
|
||||
toks.splice(i, 1)
|
||||
i-- // 调整索引
|
||||
}
|
||||
else {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Some custom tags (e.g. <thinking>) can be tokenized by markdown-it into
|
||||
// multiple top-level inline tokens, with the closing tag arriving in a
|
||||
// later inline token. Our inline parser can only match closing tags within
|
||||
// the same inline token's children list, so we merge such sequences here.
|
||||
if (customTagSet.size > 0) {
|
||||
const openReCache = new Map<string, RegExp>()
|
||||
const closeReCache = new Map<string, RegExp>()
|
||||
const getOpenRe = (tag: string) => {
|
||||
let r = openReCache.get(tag)
|
||||
if (!r) {
|
||||
r = new RegExp(`<\\s*${tag}\\b`, 'i')
|
||||
openReCache.set(tag, r)
|
||||
}
|
||||
return r
|
||||
}
|
||||
const getCloseRe = (tag: string) => {
|
||||
let r = closeReCache.get(tag)
|
||||
if (!r) {
|
||||
r = new RegExp(`<\\s*\\/\\s*${tag}\\s*>`, 'i')
|
||||
closeReCache.set(tag, r)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
const stack: Array<{ tag: string, index: number }> = []
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const tok = toks[i] as Token & { content?: string, children?: any[] }
|
||||
const content = String(tok.content ?? '')
|
||||
|
||||
// If we're inside an unclosed custom tag, we may need to close it even
|
||||
// if the closing tag is emitted as html_block (markdown-it can do this).
|
||||
if (stack.length > 0) {
|
||||
const top = stack[stack.length - 1]
|
||||
const openTok = toks[top.index] as Token & { content?: string, children?: any[] }
|
||||
|
||||
// Close via an html_block token like "</thinking>"
|
||||
if (tok.type === 'html_block' && getCloseRe(top.tag).test(content)) {
|
||||
openTok.content = `${String(openTok.content ?? '')}\n${content}`
|
||||
if (Array.isArray(openTok.children)) {
|
||||
openTok.children.push({
|
||||
type: 'html_inline',
|
||||
content: `</${top.tag}>`,
|
||||
raw: `</${top.tag}>`,
|
||||
} as any)
|
||||
}
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
stack.pop()
|
||||
continue
|
||||
}
|
||||
|
||||
// Only merge inline tokens; keep block structure intact.
|
||||
if (tok.type !== 'inline')
|
||||
continue
|
||||
|
||||
const children = Array.isArray(tok.children) ? tok.children : []
|
||||
const closeChildIndex = findMatchingCloseChildIndex(children, top.tag)
|
||||
|
||||
// If the closing tag is inside this inline token, merge up to it and
|
||||
// keep the trailing content as a new paragraph so it doesn't get
|
||||
// swallowed by the custom tag.
|
||||
if (closeChildIndex !== -1) {
|
||||
const beforeChildren = children.slice(0, closeChildIndex + 1)
|
||||
const afterChildren = children.slice(closeChildIndex + 1)
|
||||
|
||||
const beforeText = beforeChildren
|
||||
.map((c: any) => String(c?.content ?? c?.raw ?? ''))
|
||||
.join('')
|
||||
|
||||
// Only append the fragment up to and including the closing tag.
|
||||
openTok.content = `${String(openTok.content ?? '')}\n${beforeText}`
|
||||
if (Array.isArray(openTok.children))
|
||||
openTok.children.push(...beforeChildren)
|
||||
|
||||
// Replace current token with trailing content (if any)
|
||||
if (afterChildren.length) {
|
||||
const afterText = afterChildren.map((c: any) => String(c.content ?? c.raw ?? '')).join('')
|
||||
if (afterText.trim()) {
|
||||
const trimmed = afterText.replace(/^\s+/, '')
|
||||
if (trimmed.startsWith('<')) {
|
||||
toks.splice(i, 1, { type: 'html_block', content: trimmed } as any)
|
||||
}
|
||||
else {
|
||||
toks.splice(i, 1, { type: 'paragraph_open', tag: 'p', nesting: 1 } as any, { type: 'inline', tag: '', nesting: 0, content: afterText, children: [{ type: 'text', content: afterText, raw: afterText }] } as any, { type: 'paragraph_close', tag: 'p', nesting: -1 } as any)
|
||||
// current index now points at paragraph_open; move on
|
||||
}
|
||||
}
|
||||
else {
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
}
|
||||
}
|
||||
else {
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
}
|
||||
|
||||
stack.pop()
|
||||
continue
|
||||
}
|
||||
|
||||
// No closing tag: merge everything and remove current inline token.
|
||||
openTok.content = `${String(openTok.content ?? '')}\n${content}`
|
||||
if (Array.isArray(openTok.children))
|
||||
openTok.children.push(...children)
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
continue
|
||||
}
|
||||
|
||||
// Not inside: detect an opening custom tag that does not close within this token.
|
||||
if (tok.type !== 'inline')
|
||||
continue
|
||||
const children = Array.isArray(tok.children) ? tok.children : []
|
||||
for (const tag of customTagSet) {
|
||||
const trailingOpenDepth = children.length
|
||||
? getTrailingOpenDepth(children, tag)
|
||||
: (getOpenRe(tag).test(content) && !getCloseRe(tag).test(content) ? 1 : 0)
|
||||
if (trailingOpenDepth > 0) {
|
||||
stack.push({ tag, index: i })
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Defensive cleanup: some edge cases can end up with an orphan
|
||||
// paragraph_close token (without a matching paragraph_open) after
|
||||
// core-stage token mutations. Drop such invalid closes so downstream
|
||||
// consumers don't see stray paragraph_close.
|
||||
{
|
||||
let depth = 0
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i] as Token
|
||||
if (t.type === 'paragraph_open') {
|
||||
depth++
|
||||
continue
|
||||
}
|
||||
if (t.type === 'paragraph_close') {
|
||||
if (depth > 0) {
|
||||
depth--
|
||||
}
|
||||
else {
|
||||
toks.splice(i, 1)
|
||||
i--
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i] as Token & { content?: string, children: any[], loading?: boolean }
|
||||
if (t.type === 'html_block') {
|
||||
const rawTag = t.content?.match(/<([^\s>/]+)/)?.[1] ?? ''
|
||||
const tag = rawTag.toLowerCase()
|
||||
|
||||
// Special tags: comments, doctypes, processing instructions
|
||||
// These should not be modified
|
||||
if (tag.startsWith('!') || tag.startsWith('?')) {
|
||||
t.loading = false
|
||||
continue
|
||||
}
|
||||
|
||||
// Custom tags (configured by user): keep as html_block
|
||||
// These are handled specially to support streaming and structured nodes
|
||||
if (customTagSet.has(tag)) {
|
||||
const raw = String(t.content ?? '')
|
||||
const openEnd = findTagCloseIndexOutsideQuotes(raw)
|
||||
const closeRange = openEnd === -1
|
||||
? null
|
||||
: findMatchingCloseRangeInHtml(raw, tag, openEnd + 1)
|
||||
const hasClose = !!closeRange
|
||||
t.loading = hasClose ? false : t.loading !== undefined ? t.loading : true
|
||||
|
||||
const endTagIndex = closeRange?.start ?? -1
|
||||
const closeLen = closeRange ? closeRange.end - closeRange.start : 0
|
||||
|
||||
if (endTagIndex !== -1) {
|
||||
// Found a closing tag - extract inner content and trim
|
||||
const rawForNode = raw.slice(0, endTagIndex + closeLen)
|
||||
let inner = ''
|
||||
if (openEnd !== -1 && openEnd < endTagIndex) {
|
||||
inner = raw.slice(openEnd + 1, endTagIndex)
|
||||
}
|
||||
|
||||
// Create structured node with inner content
|
||||
t.children = [
|
||||
{
|
||||
type: tag,
|
||||
content: inner,
|
||||
raw: rawForNode,
|
||||
attrs: [],
|
||||
tag,
|
||||
loading: false,
|
||||
},
|
||||
] as any[]
|
||||
|
||||
// Update token content
|
||||
t.content = rawForNode
|
||||
;(t as any).raw = rawForNode
|
||||
|
||||
// Insert trailing content as a new token if present
|
||||
const afterContent = raw.slice(endTagIndex + closeLen) || ''
|
||||
const afterTrimmed = afterContent.replace(/^\s+/, '')
|
||||
if (afterTrimmed) {
|
||||
toks.splice(i + 1, 0, afterTrimmed.startsWith('<')
|
||||
? ({ type: 'html_block', content: afterTrimmed } as any)
|
||||
: ({ type: 'text', content: afterTrimmed, raw: afterTrimmed } as any))
|
||||
}
|
||||
}
|
||||
else {
|
||||
// No closing tag yet (streaming mid-state)
|
||||
t.children = [
|
||||
{
|
||||
type: tag,
|
||||
content: '',
|
||||
raw,
|
||||
attrs: [],
|
||||
tag,
|
||||
loading: true,
|
||||
},
|
||||
] as any[]
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// Common block tags: skip conversion to inline
|
||||
if (['br', 'hr', 'img', 'input', 'link', 'meta', 'div', 'p', 'ul', 'li'].includes(tag))
|
||||
continue
|
||||
|
||||
// Other HTML tags: convert to inline type for better handling
|
||||
// Parse attributes and check for closing tag
|
||||
t.type = 'inline'
|
||||
const attrs: [string, string][] = []
|
||||
const attrRegex = /\s([\w:-]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g
|
||||
let match
|
||||
while ((match = attrRegex.exec(t.content || '')) !== null) {
|
||||
const attrName = match[1]
|
||||
const attrValue = match[2] || match[3] || match[4] || ''
|
||||
attrs.push([attrName, attrValue])
|
||||
}
|
||||
|
||||
const raw = String(t.content ?? '')
|
||||
const closeRe = new RegExp(`<\\/\\s*${tag}\\s*>`, 'i')
|
||||
const closeMatch = closeRe.exec(raw)
|
||||
const endTagIndex = closeMatch ? closeMatch.index : -1
|
||||
const closeLen = closeMatch ? closeMatch[0].length : 0
|
||||
|
||||
if (endTagIndex !== -1) {
|
||||
// Found a closing tag - split content
|
||||
const rawForNode = raw.slice(0, endTagIndex + closeLen)
|
||||
const afterContent = raw.slice(endTagIndex + closeLen) || ''
|
||||
const afterTrimmed = afterContent.replace(/^\s+/, '')
|
||||
|
||||
// Create html_block child node
|
||||
t.children = [
|
||||
{
|
||||
type: 'html_block',
|
||||
content: rawForNode,
|
||||
tag,
|
||||
loading: false,
|
||||
},
|
||||
] as any[]
|
||||
|
||||
// Update token content
|
||||
t.content = rawForNode
|
||||
;(t as any).raw = rawForNode
|
||||
|
||||
// Insert trailing content as a new token if present
|
||||
if (afterTrimmed) {
|
||||
toks.splice(i + 1, 0, afterTrimmed.startsWith('<')
|
||||
? ({ type: 'html_block', content: afterTrimmed } as any)
|
||||
: ({ type: 'text', content: afterTrimmed, raw: afterTrimmed } as any))
|
||||
}
|
||||
}
|
||||
else {
|
||||
// No closing tag - streaming mid-state
|
||||
t.children = [
|
||||
{
|
||||
type: 'html_block',
|
||||
content: t.content,
|
||||
tag,
|
||||
loading: true,
|
||||
},
|
||||
] as any[]
|
||||
}
|
||||
continue
|
||||
}
|
||||
if (!t || t.type !== 'inline')
|
||||
continue
|
||||
|
||||
// 修复children 是单个 html_inline的场景
|
||||
if (t.children.length === 2 && t.children[0].type === 'html_inline') {
|
||||
// 补充一个闭合标签
|
||||
const rawTag = t.children[0].content?.match(/<([^\s>/]+)/)?.[1] ?? ''
|
||||
const tag = rawTag.toLowerCase()
|
||||
const second = t.children[1] as any
|
||||
const secondCloseTag = String(second?.content ?? '').match(/^<\s*\/\s*([^\s>]+)/)?.[1]?.toLowerCase() ?? ''
|
||||
// Already a complete open+close pair: don't append another closing tag.
|
||||
if (second?.type === 'html_inline' && secondCloseTag === tag)
|
||||
continue
|
||||
// 如果是常见的 inline标签(含用户自定义),则只追加结尾标签,否则转换成 html_block
|
||||
if (autoCloseInlineTagSet.has(tag)) {
|
||||
t.children[0].loading = true
|
||||
t.children[0].tag = tag
|
||||
t.children.push({
|
||||
type: 'html_inline',
|
||||
tag,
|
||||
loading: true,
|
||||
content: `</${tag}>`,
|
||||
} as any)
|
||||
}
|
||||
else {
|
||||
t.children = [
|
||||
{
|
||||
type: 'html_block',
|
||||
loading: true,
|
||||
tag,
|
||||
content: t.children[0].content + t.children[1].content,
|
||||
} as any,
|
||||
]
|
||||
}
|
||||
continue
|
||||
}
|
||||
else if (t.children.length === 3 && t.children[0].type === 'html_inline' && t.children[2].type === 'html_inline') {
|
||||
const rawTag = t.children[0].content?.match(/<([^\s>/]+)/)?.[1] ?? ''
|
||||
const tag = rawTag.toLowerCase()
|
||||
// 如果是常见的 inline标签(含用户自定义),则不处理,否则转换成 html_block
|
||||
if (autoCloseInlineTagSet.has(tag))
|
||||
continue
|
||||
t.children = [
|
||||
{
|
||||
type: 'html_block',
|
||||
loading: false,
|
||||
tag,
|
||||
content: t.children.map(ct => ct.content).join(''),
|
||||
} as any,
|
||||
]
|
||||
continue
|
||||
}
|
||||
// Only handle pathological cases where inline content is a single HTML-ish chunk
|
||||
if (!t.content?.startsWith('<') || (t as any).children?.length !== 1)
|
||||
continue
|
||||
|
||||
const raw = String(t.content)
|
||||
const htmlToken = t as unknown as { children: Array<{ type: string, content: string }> }
|
||||
const onlyChild = htmlToken.children[0] as { type?: string, content?: string } | undefined
|
||||
|
||||
// Keep literal text untouched (e.g. malformed "<robot=xxx>..."), but still
|
||||
// suppress dangling "<tag" mid-states to avoid streaming jitter.
|
||||
if (onlyChild?.type !== 'html_inline') {
|
||||
if (/^<\s*(?:\/\s*)?[A-Z][\w:-]*\s*$/i.test(raw))
|
||||
htmlToken.children.length = 0
|
||||
continue
|
||||
}
|
||||
|
||||
const strictTagName = String(onlyChild.content ?? raw).match(STRICT_OPEN_TAG_NAME_AT_START_RE)?.[1]?.toLowerCase() ?? ''
|
||||
if (!strictTagName)
|
||||
continue
|
||||
|
||||
const selfClosing = /\/\s*>\s*$/.test(raw)
|
||||
const isVoid = selfClosing || VOID_TAGS.has(strictTagName)
|
||||
|
||||
if (isVoid) {
|
||||
// For void/self-closing tags, keep a single html_inline token
|
||||
htmlToken.children = [
|
||||
{ type: 'html_inline', content: raw },
|
||||
] as any
|
||||
continue
|
||||
}
|
||||
htmlToken.children.length = 0
|
||||
}
|
||||
})
|
||||
}
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* Fix indented code blocks that should be paragraphs or inline HTML.
|
||||
*
|
||||
* In streaming scenarios, AI-generated content may have lines that are
|
||||
* indented with 4+ spaces (which markdown-it treats as code blocks per
|
||||
* CommonMark spec) but are actually plain text or HTML entities.
|
||||
*
|
||||
* This plugin runs at the core stage to convert single-line indented
|
||||
* code blocks that don't look like code into paragraphs.
|
||||
*/
|
||||
|
||||
import type { MarkdownIt, Token } from 'markdown-it-ts'
|
||||
|
||||
export interface FixIndentedCodeBlockOptions {
|
||||
/**
|
||||
* Whether to enable this fix. Default: true
|
||||
*/
|
||||
enabled?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a line looks like code (vs plain text or HTML entities).
|
||||
* Returns true if the line appears to be code.
|
||||
*/
|
||||
function looksLikeCode(line: string): boolean {
|
||||
const trimmed = line.trim()
|
||||
|
||||
// Empty line
|
||||
if (!trimmed)
|
||||
return false
|
||||
|
||||
// 1. HTML entities at start (e.g., ✖, , &)
|
||||
// These are typically plain text with character references, not code
|
||||
if (/^&[a-z0-9#]+;/i.test(trimmed))
|
||||
return false
|
||||
|
||||
// 2. Common programming keywords and JavaScript literals
|
||||
if (/^(?:const|let|var|function|class|import|export|if|for|while|return|await|async|yield|try|catch|throw|new|typeof|instanceof|switch|case|break|continue|def|ruby|perl|print|echo|true|false|null|undefined|NaN|Infinity|this)\b/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 3. Function/method calls with parentheses
|
||||
if (/[a-z_$][\w$]*(?:\.[a-z_$][\w$]*|\['[^']*'\]|\["[^"]*"\]|\[\d+\])*\s*\(/i.test(trimmed))
|
||||
return true
|
||||
|
||||
// 4. Property access with dot or bracket notation
|
||||
if (/[a-z_$][\w$]*(?:\.[a-z_$][\w$]*|\['[^']*'\]|\["[^"]*"\]|\[[\d+\]])+/i.test(trimmed))
|
||||
return true
|
||||
|
||||
// 5. Comparison and logical operators (but not just symbols alone)
|
||||
// Require identifiers around operators for better precision
|
||||
// Match: identifier operator identifier OR operator(s) identifier
|
||||
if (/\w+\s*(?:===?|!==?|<=?|>=?|\+\+|--|&&|\|\||\?\.)/.test(trimmed))
|
||||
return true
|
||||
// Double operators before identifier (like !!value) - but avoid matching list items
|
||||
if (/^(?:!!|\+\+|--)\s*\w/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 6. Assignment operators
|
||||
if (/[\w$]+\s*(?:\+=|-=|\*=|\/=|%=|\*\*=|=)/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 7. URLs, protocols
|
||||
if (/^(?:https?:\/\/|ftp:\/\/|file:\/\/|\/\/|www\.)/i.test(trimmed))
|
||||
return true
|
||||
|
||||
// 8. Template strings with interpolation
|
||||
if (/`[^`]*\$\{[^}]*\}[^`]*`/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 9. JSX/TSX tags (uppercase component names)
|
||||
if (/<\/?[A-Z][a-zA-Z0-9]*/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 10. HTML-like tags (lowercase with attributes)
|
||||
if (/<[a-z][a-z0-9]*\s[^>]+>/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 11. Quoted strings (single, double, backtick) - often code statements
|
||||
if (/^(["'`]).*\1\s*[;,]?$/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 12. Array/object/function patterns with brackets
|
||||
// Empty brackets [] {} () are common in code
|
||||
if (/^\[[\s\S]*\]$/.test(trimmed) || /^\{[\s\S]*\}$/.test(trimmed) || /^\(\s*\)$/.test(trimmed)) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 13. Code-like expressions with identifiers and operators
|
||||
// e.g., a + b, x > y, count !== 0
|
||||
// This also catches property access like foo.bar
|
||||
// Must have at least one identifier (letter/digit/underscore) on each side of operator
|
||||
if (/[\w$]+(?:\s*[+\-*/%<>=!&|^~:]+\s*[\w$]+|\s*\.\s*[\w$]+)/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 14. Arrow functions, fat arrows
|
||||
if (/=>|->|::/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 15. Decorators and annotations
|
||||
if (/^@[\w.$]+$/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 16. Numbers (hex, binary, octal, decimal with units)
|
||||
if (/^(?:0x[0-9a-fA-F]+|0b[01]+|0o[0-7]+|\d+(?:\.\d*)?(?:px|em|rem|%|vh|vw|deg|s|ms)?)$/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 17. Variable declarations with $ (common in jQuery/shell)
|
||||
if (/^\$[\w$]+\s*[=:]/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 18. Pipes and special operators
|
||||
if (/\|\s*\w+|\w+\s*\|/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 19. Command-like patterns (git, npm, etc.)
|
||||
if (/^(?:git|npm|yarn|pnpm|bun|pip|cargo|go|rust|python|node|java|mvn|gradle|docker|kubectl)\s+/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 20. Console methods and common API calls
|
||||
if (/(?:console|window|document|Math|JSON|Date|Array|Object|String|Number|Boolean)\.[a-zA-Z]/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 21. Comment patterns
|
||||
if (/^(?:\/\/|#|\/\*|\*\/|<!--|-->)/.test(trimmed))
|
||||
return true
|
||||
|
||||
// 22. Heredoc patterns
|
||||
if (/^(?:<<<|<<\s*['"]?\w+['"]?)/.test(trimmed))
|
||||
return true
|
||||
|
||||
// Default: doesn't look like code
|
||||
return false
|
||||
}
|
||||
|
||||
export function applyFixIndentedCodeBlock(md: MarkdownIt, options: FixIndentedCodeBlockOptions = {}) {
|
||||
if (options.enabled === false)
|
||||
return
|
||||
|
||||
// Process at the core stage after tokenization is complete
|
||||
md.core.ruler.after('inline', 'fix_indented_code_block', (state: any) => {
|
||||
const tokens = state.tokens as Token[]
|
||||
|
||||
for (let i = 0; i < tokens.length; i++) {
|
||||
const token = tokens[i]
|
||||
|
||||
// Look for single-line code_block tokens
|
||||
if (token.type !== 'code_block')
|
||||
continue
|
||||
|
||||
const content = String((token as any).content ?? '').trim()
|
||||
if (!content)
|
||||
continue
|
||||
|
||||
const lines = content.split(/\r?\n/).filter(line => line.trim().length > 0)
|
||||
|
||||
// Only convert single-line code blocks that don't look like code
|
||||
if (lines.length === 1 && !looksLikeCode(lines[0] ?? '')) {
|
||||
// Convert code_block to paragraph
|
||||
const textContent = lines[0] ?? ''
|
||||
|
||||
// Replace code_block token with paragraph tokens
|
||||
tokens.splice(i, 1, { type: 'paragraph_open', tag: 'p', nesting: 1, level: token.level } as any, {
|
||||
type: 'inline',
|
||||
tag: '',
|
||||
nesting: 0,
|
||||
level: token.level,
|
||||
content: textContent,
|
||||
children: [{ type: 'text', content: textContent, level: token.level + 1 }] as any,
|
||||
block: true,
|
||||
} as any, { type: 'paragraph_close', tag: 'p', nesting: -1, level: token.level } as any)
|
||||
// Skip the newly inserted tokens
|
||||
i += 2
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,653 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
import type { MarkdownToken } from '../types'
|
||||
import { shouldDemoteFilenameLikeLinkify } from '../parser/linkifyHeuristics'
|
||||
|
||||
// We hard-stop FULLWIDTH exclamation mark used as CJK punctuation.
|
||||
// ASCII `!` is valid in URLs (path/query/fragment), so do not stop on it.
|
||||
const LINKIFY_HARD_STOP_CHARS = ['!'] as const
|
||||
|
||||
// Small helpers to reduce repetition when building token fragments
|
||||
function textToken(content: string) {
|
||||
return {
|
||||
type: 'text',
|
||||
content,
|
||||
raw: content,
|
||||
}
|
||||
}
|
||||
|
||||
function pushEmOpen(arr: any[], type: number) {
|
||||
if (type === 1) {
|
||||
arr.push({ type: 'em_open', tag: 'em', nesting: 1 })
|
||||
}
|
||||
else if (type === 2) {
|
||||
arr.push({ type: 'strong_open', tag: 'strong', nesting: 1 })
|
||||
}
|
||||
else if (type === 3) {
|
||||
arr.push({ type: 'strong_open', tag: 'strong', nesting: 1 })
|
||||
arr.push({ type: 'em_open', tag: 'em', nesting: 1 })
|
||||
}
|
||||
}
|
||||
|
||||
function pushEmClose(arr: any[], type: number) {
|
||||
if (type === 1) {
|
||||
arr.push({ type: 'em_close', tag: 'em', nesting: -1 })
|
||||
}
|
||||
else if (type === 2) {
|
||||
arr.push({ type: 'strong_close', tag: 'strong', nesting: -1 })
|
||||
}
|
||||
else if (type === 3) {
|
||||
arr.push({ type: 'em_close', tag: 'em', nesting: -1 })
|
||||
arr.push({ type: 'strong_close', tag: 'strong', nesting: -1 })
|
||||
}
|
||||
}
|
||||
|
||||
function createLinkToken(text: string, href: string, loading: boolean) {
|
||||
let title = ''
|
||||
if (href.includes('"')) {
|
||||
const temps = href.split('"')
|
||||
href = temps[0].trim()
|
||||
title = temps[1].trim()
|
||||
}
|
||||
return {
|
||||
type: 'link',
|
||||
loading,
|
||||
href,
|
||||
title,
|
||||
text,
|
||||
children: [
|
||||
{
|
||||
type: 'text',
|
||||
content: text,
|
||||
raw: text,
|
||||
},
|
||||
],
|
||||
raw: String(`[${text}](${href})`),
|
||||
}
|
||||
}
|
||||
|
||||
function appendToLinkToken(link: any, suffix: string) {
|
||||
if (!link || !suffix)
|
||||
return
|
||||
link.href = String(link.href ?? '') + suffix
|
||||
link.text = String(link.text ?? '') + suffix
|
||||
link.raw = String(`[${link.text}](${link.href})`)
|
||||
if (Array.isArray(link.children) && link.children.length) {
|
||||
const last = link.children[link.children.length - 1]
|
||||
if (last?.type === 'text') {
|
||||
last.content = String(last.content ?? '') + suffix
|
||||
last.raw = String(last.raw ?? '') + suffix
|
||||
}
|
||||
else {
|
||||
link.children.push(textToken(suffix))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function firstIndexOfAny(input: string, chars: readonly string[]) {
|
||||
let first = -1
|
||||
for (const ch of chars) {
|
||||
const idx = input.indexOf(ch)
|
||||
if (idx !== -1 && (first === -1 || idx < first))
|
||||
first = idx
|
||||
}
|
||||
return first
|
||||
}
|
||||
|
||||
function getHrefFromLinkOpen(token: any) {
|
||||
const href = token?.attrs?.find((attr: any) => attr?.[0] === 'href')?.[1]
|
||||
return typeof href === 'string' ? href : ''
|
||||
}
|
||||
|
||||
function setHrefOnLinkOpen(token: any, href: string) {
|
||||
if (!token)
|
||||
return
|
||||
token.attrs = Array.isArray(token.attrs) ? token.attrs : []
|
||||
const idx = token.attrs.findIndex((attr: any) => attr?.[0] === 'href')
|
||||
if (idx >= 0)
|
||||
token.attrs[idx][1] = href
|
||||
else
|
||||
token.attrs.push(['href', href])
|
||||
}
|
||||
|
||||
function collectLinkifyText(tokens: MarkdownToken[], openIndex: number, closeIndex: number) {
|
||||
let text = ''
|
||||
for (let index = openIndex + 1; index < closeIndex; index++) {
|
||||
const token = tokens[index]
|
||||
if (token?.type !== 'text' || typeof token.content !== 'string')
|
||||
return null
|
||||
text += token.content
|
||||
}
|
||||
return text || null
|
||||
}
|
||||
|
||||
export function applyFixLinkTokens(md: MarkdownIt) {
|
||||
// Run after the inline rule so markdown-it has produced inline tokens
|
||||
// for block-level tokens; we then adjust each inline token's children
|
||||
// so downstream code receives corrected token arrays during the same
|
||||
// parsing pass.
|
||||
md.core.ruler.after('inline', 'fix_link_tokens', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
|
||||
const toks = s.tokens ?? []
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i]
|
||||
if (t && t.type === 'inline' && Array.isArray(t.children)) {
|
||||
try {
|
||||
t.children = fixLinkToken(t.children)
|
||||
}
|
||||
catch (e) {
|
||||
// Swallow errors to avoid breaking parsing; keep original children
|
||||
// so parse still succeeds even if our fix fails for an unexpected shape.
|
||||
// Errors should be rare and indicate malformed token arrays.
|
||||
|
||||
console.error('[applyFixLinkTokens] failed to fix inline children', e)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function fixLinkToken(tokens: MarkdownToken[]): MarkdownToken[] {
|
||||
// Need at least `link_open + text + link_close` for linkify/autolink fixes.
|
||||
// Keep this low to allow fixing `<https://...!suffix>` cases where inline children length is 3.
|
||||
if (tokens.length < 3)
|
||||
return tokens
|
||||
|
||||
// 如果包含 code_inline 类型的 token,说明是包含内联代码的链接,直接返回原样,避免错误处理
|
||||
if (tokens.some(token => token.type === 'code_inline'))
|
||||
return tokens
|
||||
|
||||
for (let i = 0; i <= tokens.length - 1; i++) {
|
||||
if (i < 0) {
|
||||
i = 0
|
||||
}
|
||||
const curToken = tokens[i]
|
||||
if (!curToken)
|
||||
break
|
||||
// Hard-stop certain punctuation for linkify-generated bare URLs.
|
||||
// Example: `https://a.com/x.png!文字` should become link `https://a.com/x.png` + text `!文字`.
|
||||
if (curToken.type === 'link_open' && (curToken.markup === 'linkify' || curToken.markup === 'autolink')) {
|
||||
let closeIdx = -1
|
||||
for (let j = i + 1; j < tokens.length; j++) {
|
||||
if (tokens[j]?.type === 'link_close') {
|
||||
closeIdx = j
|
||||
break
|
||||
}
|
||||
}
|
||||
if (closeIdx !== -1) {
|
||||
const linkText = collectLinkifyText(tokens, i, closeIdx)
|
||||
if (
|
||||
curToken.markup === 'linkify'
|
||||
&& linkText
|
||||
&& shouldDemoteFilenameLikeLinkify(linkText)
|
||||
) {
|
||||
tokens.splice(i, closeIdx - i + 1, textToken(linkText) as any)
|
||||
continue
|
||||
}
|
||||
|
||||
const href = getHrefFromLinkOpen(curToken as any)
|
||||
const hrefStop = firstIndexOfAny(href, LINKIFY_HARD_STOP_CHARS)
|
||||
// Prefer splitting by the visible text token, but also trim href if it contains stop chars.
|
||||
for (let j = i + 1; j < closeIdx; j++) {
|
||||
const t = tokens[j]
|
||||
if (t?.type !== 'text' || typeof t.content !== 'string')
|
||||
continue
|
||||
const stopAt = firstIndexOfAny(t.content, LINKIFY_HARD_STOP_CHARS)
|
||||
if (stopAt === -1)
|
||||
continue
|
||||
|
||||
const stopChar = t.content[stopAt]
|
||||
const before = t.content.slice(0, stopAt)
|
||||
let tail = t.content.slice(stopAt)
|
||||
// Move any remaining text tokens that were inside the linkify span to the tail.
|
||||
for (let k = j + 1; k < closeIdx; k++) {
|
||||
const tk = tokens[k]
|
||||
if (tk?.type === 'text' && typeof tk.content === 'string')
|
||||
tail += tk.content
|
||||
}
|
||||
|
||||
t.content = before
|
||||
;(t as any).raw = before
|
||||
|
||||
const removeCount = closeIdx - (j + 1)
|
||||
if (removeCount > 0) {
|
||||
tokens.splice(j + 1, removeCount)
|
||||
closeIdx = j + 1
|
||||
}
|
||||
|
||||
let newHref = href
|
||||
if (hrefStop !== -1) {
|
||||
newHref = href.slice(0, hrefStop)
|
||||
}
|
||||
else if (tail) {
|
||||
// linkify-it may percent-encode non-ASCII, so use encoded tail / stop-char to locate split point.
|
||||
const encodedTail = encodeURI(tail)
|
||||
if (encodedTail && href.endsWith(encodedTail)) {
|
||||
newHref = href.slice(0, href.length - encodedTail.length)
|
||||
}
|
||||
else {
|
||||
const encodedStop = stopChar ? encodeURI(stopChar) : ''
|
||||
const idx = encodedStop ? href.indexOf(encodedStop) : -1
|
||||
if (idx !== -1)
|
||||
newHref = href.slice(0, idx)
|
||||
}
|
||||
}
|
||||
if (newHref !== href)
|
||||
setHrefOnLinkOpen(curToken as any, newHref)
|
||||
|
||||
if (tail) {
|
||||
tokens.splice(closeIdx + 1, 0, textToken(tail) as any)
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if (curToken?.type === 'em_open' && tokens[i - 1]?.type === 'text' && tokens[i - 1].content?.endsWith('*')) {
|
||||
const beforeText = tokens[i - 1].content?.replace(/(\*+)$/, '') || ''
|
||||
tokens[i - 1].content = beforeText
|
||||
// 修改当前 type 'em_open' -> 'strong_open'
|
||||
curToken.type = 'strong_open'
|
||||
curToken.tag = 'strong'
|
||||
curToken.markup = '**'
|
||||
// 还需要把对应的 em_close 也修改为 strong_close
|
||||
for (let j = i + 1; j < tokens.length; j++) {
|
||||
if (tokens[j]?.type === 'em_close') {
|
||||
tokens[j].type = 'strong_close'
|
||||
tokens[j].tag = 'strong'
|
||||
tokens[j].markup = '**'
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
else if (curToken?.type === 'text' && curToken.content?.endsWith('(') && tokens[i + 1]?.type === 'link_open') {
|
||||
const match = curToken.content!.match(/\[([^\]]+)\]/)
|
||||
if (match) {
|
||||
let beforeText = curToken.content!.slice(0, match.index)
|
||||
const emphasisMatch = beforeText.match(/(\*+)$/)
|
||||
const replacerTokens = []
|
||||
if (emphasisMatch) {
|
||||
beforeText = beforeText.slice(0, emphasisMatch.index)
|
||||
if (beforeText)
|
||||
replacerTokens.push(textToken(beforeText))
|
||||
const text = match[1]
|
||||
const type = emphasisMatch[1].length
|
||||
pushEmOpen(replacerTokens, type)
|
||||
let href = tokens[i + 2]?.content || ''
|
||||
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
|
||||
href += tokens[i + 4]?.content || ''
|
||||
tokens[i + 4].content = ''
|
||||
}
|
||||
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
|
||||
pushEmClose(replacerTokens, type)
|
||||
if (tokens[i + 4]?.type === 'text') {
|
||||
const afterText = tokens[i + 4].content?.replace(/^\)\**/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i, 5, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
tokens.splice(i, 4, ...replacerTokens)
|
||||
}
|
||||
}
|
||||
else {
|
||||
if (beforeText)
|
||||
replacerTokens.push(textToken(beforeText))
|
||||
let text = match[1]
|
||||
const emphasisMatch = text.match(/^\*+/)
|
||||
if (emphasisMatch) {
|
||||
const type = emphasisMatch[0].length
|
||||
text = text.replace(/^\*+/, '').replace(/\*+$/, '')
|
||||
// Put emphasis inside the link children when the asterisks are inside the brackets
|
||||
// so link nodes will contain the emphasis node as a child.
|
||||
let href = tokens[i + 2]?.content || ''
|
||||
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
|
||||
href += tokens[i + 4]?.content || ''
|
||||
tokens[i + 4].content = ''
|
||||
}
|
||||
// wrap the link with emphasis open/close tokens
|
||||
pushEmOpen(replacerTokens, type)
|
||||
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
|
||||
pushEmClose(replacerTokens, type)
|
||||
// we've already pushed the link, skip the standard push below
|
||||
if (tokens[i + 4]?.type === 'text') {
|
||||
const afterText = tokens[i + 4].content?.replace(/^\)/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i, 5, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
tokens.splice(i, 4, ...replacerTokens)
|
||||
}
|
||||
if (i === 0) {
|
||||
i = replacerTokens.length - 1
|
||||
}
|
||||
else {
|
||||
i -= (replacerTokens.length + 1)
|
||||
}
|
||||
continue
|
||||
}
|
||||
let href = tokens[i + 2]?.content || ''
|
||||
if (tokens[i + 4]?.type === 'text' && !tokens[i + 4].content?.startsWith(')')) {
|
||||
href += tokens[i + 4]?.content || ''
|
||||
tokens[i + 4].content = ''
|
||||
}
|
||||
replacerTokens.push(createLinkToken(text, href, !tokens[i + 4]?.content?.startsWith(')')))
|
||||
if (tokens[i + 4]?.type === 'text') {
|
||||
const afterText = tokens[i + 4].content?.replace(/^\)/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i, 5, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
tokens.splice(i, 4, ...replacerTokens)
|
||||
}
|
||||
}
|
||||
i -= (replacerTokens.length + 1)
|
||||
continue
|
||||
}
|
||||
}
|
||||
else if (curToken.type === 'link_open' && curToken.markup === 'linkify' && tokens[i - 1]?.type === 'text' && tokens[i - 1].content?.endsWith('(')) {
|
||||
if (tokens[i - 2]?.type === 'link_close') {
|
||||
// 合并link
|
||||
const replacerTokens = []
|
||||
const text = (tokens[i - 3].content || '')
|
||||
let href = curToken.attrs?.find(attr => attr[0] === 'href')?.[1] || ''
|
||||
|
||||
if (tokens[i + 3]?.type === 'text') {
|
||||
const m = (tokens[i + 3]?.content ?? '').indexOf(')')
|
||||
const loading = m === -1
|
||||
if (m === -1) {
|
||||
href += (tokens[i + 3]?.content?.slice(0, m) || '')
|
||||
tokens[i + 3].content = ''
|
||||
}
|
||||
|
||||
replacerTokens.push(createLinkToken(text, href, loading))
|
||||
const afterText = tokens[i + 3].content?.replace(/^\)\**/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i - 4, 8, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
replacerTokens.push({
|
||||
type: 'link',
|
||||
loading: true,
|
||||
href,
|
||||
title: '',
|
||||
text,
|
||||
children: [
|
||||
{
|
||||
type: 'text',
|
||||
content: href,
|
||||
raw: href,
|
||||
},
|
||||
],
|
||||
raw: String(`[${text}](${href})`),
|
||||
})
|
||||
tokens.splice(i - 4, 7, ...replacerTokens)
|
||||
}
|
||||
continue
|
||||
}
|
||||
else if (tokens[i - 1].content === '](' && tokens[i - 3]?.type === 'text' && tokens[i - 3].content?.endsWith(')')) {
|
||||
// 移除 i-3 的 link部分,合并到 link_open 下的 text中
|
||||
if (tokens[i - 2]?.type === 'strong_open') {
|
||||
const [beforeText, linText] = tokens[i - 3].content?.split('[**') || []
|
||||
tokens[i + 1].content = linText || ''
|
||||
tokens[i - 3].content = beforeText || ''
|
||||
tokens[i - 1].content = ''
|
||||
}
|
||||
else if (tokens[i - 2]?.type === 'em_open') {
|
||||
const [beforeText, linText] = tokens[i - 3].content?.split('[*') || []
|
||||
tokens[i + 1].content = linText || ''
|
||||
tokens[i - 3].content = beforeText || ''
|
||||
tokens[i - 1].content = ''
|
||||
}
|
||||
else {
|
||||
const [beforeText, linText] = tokens[i - 3].content?.split('[') || []
|
||||
tokens[i + 1].content = linText || ''
|
||||
tokens[i - 3].content = beforeText || ''
|
||||
tokens[i - 1].content = ''
|
||||
}
|
||||
}
|
||||
}
|
||||
if (
|
||||
curToken.type === 'link_close'
|
||||
&& curToken.nesting === -1
|
||||
&& tokens[i - 2]?.type === 'link_open'
|
||||
&& tokens[i + 1]?.type === 'text'
|
||||
&& tokens[i - 1]?.type === 'text'
|
||||
) {
|
||||
// 修复链接后多余文本被包含在链接内的问题
|
||||
const text = tokens[i - 1].content || ''
|
||||
const attrs = tokens[i - 2].attrs || []
|
||||
const href = attrs.find(a => a[0] === 'href')?.[1] || ''
|
||||
const title = attrs.find(a => a[0] === 'title')?.[1] || ''
|
||||
let count = 3
|
||||
let deleteCount = 2
|
||||
const beforeText = tokens[i - 3]?.content || ''
|
||||
const emphasisMatch = beforeText.match(/^(\*+)$/)
|
||||
const replacerTokens: any[] = []
|
||||
if (emphasisMatch) {
|
||||
deleteCount += 1
|
||||
const type = emphasisMatch[1].length
|
||||
pushEmOpen(replacerTokens, type)
|
||||
}
|
||||
|
||||
if (curToken.markup !== 'linkify' && tokens[i + 1].type === 'text' && tokens[i + 1]?.content?.startsWith('](')) {
|
||||
count += 1
|
||||
for (let j = i + 1; j < tokens.length; j++) {
|
||||
const type = emphasisMatch ? emphasisMatch[1].length : tokens[i - 3].markup!.length
|
||||
const t = tokens[j]
|
||||
if (type === 1 && t.type === 'em_close') {
|
||||
break
|
||||
}
|
||||
else if (type === 2 && t.type === 'strong_close') {
|
||||
break
|
||||
}
|
||||
else if (type === 3) {
|
||||
if (t.type === 'em_close' || t.type === 'strong_close') {
|
||||
break
|
||||
}
|
||||
}
|
||||
count += 1
|
||||
}
|
||||
}
|
||||
|
||||
replacerTokens.push({
|
||||
type: 'link',
|
||||
loading: false,
|
||||
href,
|
||||
title,
|
||||
text,
|
||||
children: [
|
||||
{
|
||||
type: 'text',
|
||||
content: text,
|
||||
raw: text,
|
||||
},
|
||||
],
|
||||
raw: String(`[${text}](${href})`),
|
||||
} as any)
|
||||
if (emphasisMatch) {
|
||||
const type = emphasisMatch[1].length
|
||||
pushEmClose(replacerTokens, type)
|
||||
}
|
||||
tokens.splice(i - deleteCount, count, ...replacerTokens)
|
||||
i -= (replacerTokens.length + 1)
|
||||
continue
|
||||
}
|
||||
else if (curToken.content?.startsWith('](') && tokens[i - 1].markup?.includes('*') && tokens[i - 4]?.type === 'text' && tokens[i - 4].content?.endsWith('[')) {
|
||||
const type = tokens[i - 1].markup!.length
|
||||
const replacerTokens = []
|
||||
const beforeText = tokens[i - 4].content!.slice(0, tokens[i - 4].content!.length - type)
|
||||
if (beforeText)
|
||||
replacerTokens.push(textToken(beforeText))
|
||||
pushEmOpen(replacerTokens, type)
|
||||
const text = tokens[i - 2].content || ''
|
||||
let href = curToken.content!.slice(2)
|
||||
let loading = true
|
||||
if (tokens[i + 1]?.type === 'text') {
|
||||
const m = (tokens[i + 1]?.content ?? '').indexOf(')')
|
||||
loading = m === -1
|
||||
if (m === -1) {
|
||||
href += (tokens[i + 1]?.content?.slice(0, m) || '')
|
||||
tokens[i + 1].content = ''
|
||||
}
|
||||
}
|
||||
replacerTokens.push(createLinkToken(text, href, loading))
|
||||
pushEmClose(replacerTokens, type)
|
||||
if (tokens[i + 1]?.type === 'text') {
|
||||
const afterText = tokens[i + 1].content?.replace(/^\)\**/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i - 4, 8, ...replacerTokens)
|
||||
}
|
||||
else if (tokens[i + 1]?.type === 'link_open') {
|
||||
// 特殊情况其实要把href也处理,这里可以直接跳过
|
||||
tokens.splice(i - 4, 10, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
tokens.splice(i - 4, 7, ...replacerTokens)
|
||||
}
|
||||
i -= (replacerTokens.length + 1)
|
||||
continue
|
||||
}
|
||||
else if (curToken.content?.startsWith('](') && tokens[i - 1].type === 'strong_close' && tokens[i - 4]?.type === 'text' && tokens[i - 4]?.content?.includes('**[')) {
|
||||
// 此时的场景是 link 被 strong 包裹,link 中又包含了强调符号
|
||||
const replacerTokens = []
|
||||
const beforeText = tokens[i - 4].content!.split('**[')[0]
|
||||
if (beforeText)
|
||||
replacerTokens.push(textToken(beforeText))
|
||||
pushEmOpen(replacerTokens, 2)
|
||||
const text = tokens[i - 2].content || ''
|
||||
let href = curToken.content!.slice(2)
|
||||
let loading = true
|
||||
if (tokens[i + 1]?.type === 'text') {
|
||||
const m = (tokens[i + 1]?.content ?? '').indexOf(')')
|
||||
loading = m === -1
|
||||
if (m === -1) {
|
||||
href += (tokens[i + 1]?.content?.slice(0, m) || '')
|
||||
tokens[i + 1].content = ''
|
||||
}
|
||||
}
|
||||
replacerTokens.push(createLinkToken(text, href, loading))
|
||||
pushEmClose(replacerTokens, 2)
|
||||
if (tokens[i + 1]?.type === 'text') {
|
||||
const afterText = tokens[i + 1].content?.replace(/^\)\**/, '')
|
||||
if (afterText)
|
||||
replacerTokens.push(textToken(afterText))
|
||||
tokens.splice(i - 4, 8, ...replacerTokens)
|
||||
}
|
||||
else if (tokens[i + 1]?.type === 'link_open') {
|
||||
// 特殊情况其实要把href也处理,这里可以直接跳过
|
||||
tokens.splice(i - 4, 10, ...replacerTokens)
|
||||
}
|
||||
else {
|
||||
tokens.splice(i - 4, 7, ...replacerTokens)
|
||||
}
|
||||
i -= (replacerTokens.length + 1)
|
||||
continue
|
||||
}
|
||||
else if (curToken.type === 'strong_close' && tokens[i + 1]?.type === 'text' && tokens[i + 1].content?.includes('](') && tokens[i - 1].type === 'text' && /\[.*$/.test(tokens[i - 1].content || '')) {
|
||||
const replacerTokens = []
|
||||
const [beforeText, afterText] = tokens[i - 1].content?.split('[') || ['', '']
|
||||
if (beforeText)
|
||||
replacerTokens.push(textToken(beforeText))
|
||||
pushEmOpen(replacerTokens, 2)
|
||||
let [text, href] = tokens[i + 1].content!.split('](')
|
||||
text = afterText + text
|
||||
let deleteCount = 4
|
||||
if (tokens[i + 2]?.type === 'link_open') {
|
||||
const _href = tokens[i + 2].attrs?.find(a => a[0] === 'href')?.[1]
|
||||
if (tokens[i + 5]?.type === 'text' && tokens[i + 5].content === '.') {
|
||||
href = (_href || href) + tokens[i + 5].content
|
||||
tokens[i + 5].content = ''
|
||||
}
|
||||
else {
|
||||
href = _href || href
|
||||
}
|
||||
|
||||
deleteCount += 3
|
||||
}
|
||||
let loading = true
|
||||
if (curToken.nesting === -1) {
|
||||
// 嵌套 strong_close,需要去掉尾部的 **
|
||||
text = text.replace(/\*+$/, '')
|
||||
}
|
||||
if (tokens[i + 2]?.type === 'text') {
|
||||
const m = (tokens[i + 2]?.content ?? '').indexOf(')')
|
||||
loading = m === -1
|
||||
if (m === -1) {
|
||||
href += (tokens[i + 2]?.content?.slice(0, m) || '')
|
||||
tokens[i + 2].content = ''
|
||||
}
|
||||
}
|
||||
replacerTokens.push(createLinkToken(text, href, loading))
|
||||
pushEmClose(replacerTokens, 2)
|
||||
tokens.splice(i - 2, deleteCount, ...replacerTokens)
|
||||
}
|
||||
// 处理强调 + 链接拆分:首个 text 含前导星号与 '[',后续出现独立的 "]("、link_open、href、link_close、")"、strong_close。
|
||||
// 期望合并为:strong_open + 复合 link(label 尾部补回星号) + strong_close。
|
||||
if (
|
||||
curToken.type === 'text'
|
||||
&& /\*+\[[^\]]*$/.test(curToken.content || '')
|
||||
&& tokens[i + 1]?.type === 'strong_open'
|
||||
&& tokens[i + 2]?.type === 'text' && tokens[i + 2].content === ']('
|
||||
&& tokens[i + 3]?.type === 'link_open'
|
||||
&& tokens[i + 5]?.type === 'link_close'
|
||||
&& tokens[i + 6]?.type === 'text' && tokens[i + 6].content === ')'
|
||||
&& tokens[i + 7]?.type === 'strong_close'
|
||||
) {
|
||||
const contentVal = curToken.content || ''
|
||||
const startMatch = contentVal.match(/^(\*+)\[(.*)$/)
|
||||
if (startMatch) {
|
||||
const innerLabel = startMatch[2] || ''
|
||||
// 将前导星号数量追加到标签末尾
|
||||
const finalLabel = innerLabel + startMatch[1]
|
||||
// 获取 href:优先使用 link_open 的属性,若为空则回退到后续文本 token 内容
|
||||
let href = tokens[i + 3]?.attrs?.find(a => a[0] === 'href')?.[1] || ''
|
||||
if (!href && tokens[i + 4]?.type === 'text') {
|
||||
href = tokens[i + 4].content || ''
|
||||
}
|
||||
const out: MarkdownToken[] = []
|
||||
pushEmOpen(out as any[], 2)
|
||||
out.push(createLinkToken(finalLabel, href, false))
|
||||
pushEmClose(out as any[], 2)
|
||||
tokens.splice(i, 9, ...out)
|
||||
i -= (out.length - 1)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Post-pass: linkify-it strips trailing ASCII punctuation like `!`, but in some URLs
|
||||
// it's meaningful (e.g. `?q=!`, `#!`). Merge a standalone leading `!` text token
|
||||
// back into the preceding autolink/linkify link when it ends with `=` or `#`.
|
||||
for (let i = 0; i < tokens.length - 1; i++) {
|
||||
const t = tokens[i] as any
|
||||
const next = tokens[i + 1] as any
|
||||
if (t?.type !== 'link' || next?.type !== 'text' || typeof next.content !== 'string')
|
||||
continue
|
||||
if (!next.content.startsWith('!'))
|
||||
continue
|
||||
const href = String(t.href ?? '')
|
||||
const text = String(t.text ?? '')
|
||||
// Only merge for bare URLs (where visible text equals href) to avoid
|
||||
// changing explicit Markdown links like `[label](https://a/#)!`.
|
||||
if (text !== href)
|
||||
continue
|
||||
if (!href.endsWith('=') && !href.endsWith('#'))
|
||||
continue
|
||||
|
||||
appendToLinkToken(t, '!')
|
||||
const rest = next.content.slice(1)
|
||||
if (rest) {
|
||||
next.content = rest
|
||||
next.raw = rest
|
||||
}
|
||||
else {
|
||||
tokens.splice(i + 1, 1)
|
||||
}
|
||||
}
|
||||
|
||||
return tokens
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
import type { MarkdownToken } from '../types'
|
||||
|
||||
export function applyFixListItem(md: MarkdownIt) {
|
||||
// Normalize list-item related inline tokens after inline tokenization
|
||||
// so downstream parsers see corrected children.
|
||||
md.core.ruler.after('inline', 'fix_list_item_tokens', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
|
||||
const toks = s.tokens ?? []
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i]
|
||||
if (t && t.type === 'inline' && Array.isArray(t.children)) {
|
||||
try {
|
||||
t.children = fixListItem(t.children)
|
||||
}
|
||||
catch (e) {
|
||||
// Keep original children on error to avoid breaking parsing
|
||||
|
||||
console.error('[applyFixListItem] failed to fix inline children', e)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function fixListItem(tokens: MarkdownToken[]): MarkdownToken[] {
|
||||
const last = tokens[tokens.length - 1]
|
||||
const lastContent = String(last?.content ?? '')
|
||||
|
||||
if (last?.type === 'text' && (/^\s*\d+\.\s*$/.test(lastContent) && tokens[tokens.length - 2]?.tag === 'br')) {
|
||||
tokens.splice(tokens.length - 1, 1)
|
||||
}
|
||||
|
||||
return tokens
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
import type { MarkdownToken } from 'stream-markdown-parser'
|
||||
|
||||
export function applyFixStrongTokens(md: MarkdownIt) {
|
||||
// Run after inline tokenization to normalize strong/em tokens in
|
||||
// each inline token's children. This ensures downstream inline
|
||||
// parsers receive a normalized token list.
|
||||
md.core.ruler.after('inline', 'fix_strong_tokens', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: Array<{ type?: string, children?: any[] }> }
|
||||
const toks = s.tokens ?? []
|
||||
for (let i = 0; i < toks.length; i++) {
|
||||
const t = toks[i]
|
||||
if (t && t.type === 'inline' && Array.isArray(t.children)) {
|
||||
try {
|
||||
t.children = fixStrongTokens(t.children)
|
||||
}
|
||||
catch (e) {
|
||||
// don't break parsing on plugin error
|
||||
|
||||
console.error('[applyFixStrongTokens] failed to fix inline children', e)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function fixStrongTokens(tokens: MarkdownToken[]): MarkdownToken[] {
|
||||
let strongIndex = 0
|
||||
const cleansStrong = new Set<number>()
|
||||
const cleansEm = new Set<number>()
|
||||
let emIndex = 0
|
||||
for (let i = 0; i < tokens.length; i++) {
|
||||
const t = tokens[i]
|
||||
const type = t.type
|
||||
if (type === 'strong_open') {
|
||||
strongIndex++
|
||||
const markup = String(t.markup ?? '')
|
||||
let j = i - 1
|
||||
while (j >= 0 && tokens[j].type === 'text' && tokens[j].content === '') {
|
||||
j--
|
||||
}
|
||||
const preToken = tokens[j]
|
||||
let k = i + 1
|
||||
while (k < tokens.length && tokens[k].type === 'text' && tokens[k].content === '') {
|
||||
k++
|
||||
}
|
||||
const postToken = tokens[k]
|
||||
|
||||
if (markup === '__' && (preToken?.content?.endsWith('_') || postToken?.content?.startsWith('_') || postToken?.markup?.includes('_'))) {
|
||||
t.type = 'text'
|
||||
t.tag = ''
|
||||
t.content = markup
|
||||
t.raw = markup
|
||||
t.markup = ''
|
||||
t.attrs = null
|
||||
t.map = null
|
||||
t.info = ''
|
||||
t.meta = null
|
||||
cleansStrong.add(strongIndex)
|
||||
}
|
||||
}
|
||||
else if (type === 'strong_close') {
|
||||
if (cleansStrong.has(strongIndex) && t.markup === '__') {
|
||||
t.type = 'text'
|
||||
t.content = t.markup
|
||||
t.raw = String(t.markup ?? '')
|
||||
t.tag = ''
|
||||
t.markup = ''
|
||||
t.attrs = null
|
||||
t.map = null
|
||||
t.info = ''
|
||||
t.meta = null
|
||||
}
|
||||
strongIndex--
|
||||
if (strongIndex < 0)
|
||||
strongIndex = 0
|
||||
}
|
||||
else if (type === 'em_open') {
|
||||
emIndex++
|
||||
const markup = String(t.markup ?? '')
|
||||
let j = i - 1
|
||||
while (j >= 0 && tokens[j].type === 'text' && tokens[j].content === '') {
|
||||
j--
|
||||
}
|
||||
const preToken = tokens[j]
|
||||
let k = i + 1
|
||||
while (k < tokens.length && tokens[k].type === 'text' && tokens[k].content === '') {
|
||||
k++
|
||||
}
|
||||
const postToken = tokens[k]
|
||||
if (markup === '_' && (preToken?.content?.endsWith('_') || postToken?.content?.startsWith('_') || postToken?.markup?.includes('_'))) {
|
||||
t.type = 'text'
|
||||
t.tag = ''
|
||||
t.content = markup
|
||||
t.raw = markup
|
||||
t.markup = ''
|
||||
t.attrs = null
|
||||
t.map = null
|
||||
t.info = ''
|
||||
t.meta = null
|
||||
cleansEm.add(emIndex)
|
||||
}
|
||||
}
|
||||
else if (type === 'em_close') {
|
||||
if (cleansEm.has(emIndex) && t.markup === '_') {
|
||||
t.type = 'text'
|
||||
t.content = t.markup
|
||||
t.raw = String(t.markup ?? '')
|
||||
t.tag = ''
|
||||
t.markup = ''
|
||||
t.attrs = null
|
||||
t.map = null
|
||||
t.info = ''
|
||||
t.meta = null
|
||||
}
|
||||
emIndex--
|
||||
if (emIndex < 0)
|
||||
emIndex = 0
|
||||
}
|
||||
}
|
||||
if (tokens.length < 5)
|
||||
return tokens
|
||||
const i = tokens.length - 4
|
||||
const token = tokens[i]
|
||||
let fixedTokens = [...tokens]
|
||||
const nextToken = tokens[i + 1]
|
||||
const tokenContent = String(token.content ?? '')
|
||||
if (token.type === 'link_open' && tokens[i - 1]?.type === 'em_open' && tokens[i - 2]?.type === 'text' && tokens[i - 2].content?.endsWith('*')) {
|
||||
const textContent = String(tokens[i - 2].content ?? '').slice(0, -1)
|
||||
|
||||
const replaceTokens = [
|
||||
{
|
||||
type: 'strong_open',
|
||||
tag: 'strong',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '**',
|
||||
info: '',
|
||||
meta: null,
|
||||
raw: '',
|
||||
},
|
||||
tokens[i],
|
||||
tokens[i + 1],
|
||||
tokens[i + 2],
|
||||
{
|
||||
type: 'strong_close',
|
||||
tag: 'strong',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '**',
|
||||
info: '',
|
||||
meta: null,
|
||||
raw: '',
|
||||
},
|
||||
]
|
||||
if (textContent) {
|
||||
replaceTokens.unshift({
|
||||
type: 'text',
|
||||
content: textContent,
|
||||
raw: textContent,
|
||||
})
|
||||
}
|
||||
fixedTokens.splice(i - 2, 6, ...replaceTokens)
|
||||
}
|
||||
else if (token.type === 'text' && tokenContent.endsWith('*') && nextToken.type === 'em_open') {
|
||||
// 解析有问题,要合并 emphasis 和 前面的 * 为 strong
|
||||
const _nextToken = tokens[i + 2]
|
||||
const count = _nextToken?.type === 'text' ? 4 : 3
|
||||
const insert = [
|
||||
{
|
||||
type: 'strong_open',
|
||||
tag: 'strong',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '**',
|
||||
info: '',
|
||||
meta: null,
|
||||
raw: '',
|
||||
},
|
||||
{
|
||||
type: 'text',
|
||||
content: _nextToken?.type === 'text' ? String(_nextToken.content ?? '') : '',
|
||||
raw: _nextToken?.type === 'text' ? String(_nextToken.content ?? '') : '',
|
||||
},
|
||||
{
|
||||
type: 'strong_close',
|
||||
tag: 'strong',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '**',
|
||||
info: '',
|
||||
meta: null,
|
||||
raw: '',
|
||||
},
|
||||
] as MarkdownToken[]
|
||||
const beforeText = tokenContent.slice(0, -1)
|
||||
if (beforeText) {
|
||||
insert.unshift({
|
||||
type: 'text',
|
||||
content: beforeText,
|
||||
raw: beforeText,
|
||||
})
|
||||
}
|
||||
fixedTokens.splice(i, count, ...insert)
|
||||
}
|
||||
|
||||
// Fix: markdown-it + math inline can incorrectly split a single strong span
|
||||
// into multiple strong nodes and leave the closing `**` inside a trailing text token.
|
||||
// Example:
|
||||
// **化简集合\\( P \\)并求补集**:...
|
||||
// Tokens may become:
|
||||
// strong_open, text('化简集合'), strong_close, strong_open, math_inline, strong_close, text('并求补集**:...')
|
||||
// Normalize into:
|
||||
// strong_open, text('化简集合'), math_inline, text('并求补集'), strong_close, text(':...')
|
||||
fixedTokens = mergeBrokenStrongAroundMathInline(fixedTokens)
|
||||
|
||||
return fixedTokens
|
||||
}
|
||||
|
||||
function mergeBrokenStrongAroundMathInline(tokens: MarkdownToken[]): MarkdownToken[] {
|
||||
if (tokens.length < 7)
|
||||
return tokens
|
||||
|
||||
const out: MarkdownToken[] = []
|
||||
for (let i = 0; i < tokens.length; i++) {
|
||||
const t0 = tokens[i]
|
||||
const t1 = tokens[i + 1]
|
||||
const t2 = tokens[i + 2]
|
||||
const t3 = tokens[i + 3]
|
||||
const t4 = tokens[i + 4]
|
||||
const t5 = tokens[i + 5]
|
||||
const t6 = tokens[i + 6]
|
||||
|
||||
if (
|
||||
t0?.type === 'strong_open'
|
||||
&& t1?.type === 'text'
|
||||
&& t2?.type === 'strong_close'
|
||||
&& t3?.type === 'strong_open'
|
||||
&& t4?.type === 'math_inline'
|
||||
&& t5?.type === 'strong_close'
|
||||
&& t6?.type === 'text'
|
||||
) {
|
||||
const textContent = String(t6.content ?? '')
|
||||
const closeIdx = textContent.indexOf('**')
|
||||
if (closeIdx !== -1) {
|
||||
const beforeClose = textContent.slice(0, closeIdx)
|
||||
const afterClose = textContent.slice(closeIdx + 2)
|
||||
|
||||
out.push(t0)
|
||||
out.push(t1)
|
||||
out.push(t4)
|
||||
|
||||
if (beforeClose) {
|
||||
out.push({
|
||||
...t6,
|
||||
type: 'text',
|
||||
content: beforeClose,
|
||||
raw: beforeClose,
|
||||
} as MarkdownToken)
|
||||
}
|
||||
|
||||
// Reuse one of the strong_close tokens to close the merged strong span.
|
||||
out.push(t5)
|
||||
|
||||
if (afterClose) {
|
||||
out.push({
|
||||
...t6,
|
||||
type: 'text',
|
||||
content: afterClose,
|
||||
raw: afterClose,
|
||||
} as MarkdownToken)
|
||||
}
|
||||
|
||||
i += 6
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
out.push(t0)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
import type { MarkdownToken } from '../types'
|
||||
|
||||
export function applyFixTableTokens(md: MarkdownIt) {
|
||||
// Run after block parsing so block-level tokens (including inline
|
||||
// children) are present. We replace the token array with the
|
||||
// fixed version returned by `fixTableTokens`.
|
||||
md.core.ruler.after('block', 'fix_table_tokens', (state: unknown) => {
|
||||
const s = state as unknown as { tokens?: any[] }
|
||||
try {
|
||||
const toks = s.tokens ?? []
|
||||
const fixed = fixTableTokens(toks)
|
||||
if (Array.isArray(fixed))
|
||||
s.tokens = fixed
|
||||
}
|
||||
catch (e) {
|
||||
// swallow errors to avoid breaking parsing; keep original tokens
|
||||
console.error('[applyFixTableTokens] failed to fix table tokens', e)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function createStart() {
|
||||
return [
|
||||
{
|
||||
type: 'table_open',
|
||||
tag: 'table',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '',
|
||||
info: '',
|
||||
level: 0,
|
||||
loading: true,
|
||||
meta: null,
|
||||
},
|
||||
{
|
||||
type: 'thead_open',
|
||||
tag: 'thead',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 1,
|
||||
children: null,
|
||||
},
|
||||
{
|
||||
type: 'tr_open',
|
||||
tag: 'tr',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 2,
|
||||
children: null,
|
||||
},
|
||||
|
||||
]
|
||||
}
|
||||
function createEnd() {
|
||||
return [
|
||||
{
|
||||
type: 'tr_close',
|
||||
tag: 'tr',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 2,
|
||||
children: null,
|
||||
},
|
||||
{
|
||||
type: 'thead_close',
|
||||
tag: 'thead',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 1,
|
||||
children: null,
|
||||
},
|
||||
{
|
||||
type: 'table_close',
|
||||
tag: 'table',
|
||||
attrs: null,
|
||||
map: null,
|
||||
children: null,
|
||||
content: '',
|
||||
markup: '',
|
||||
info: '',
|
||||
level: 0,
|
||||
meta: null,
|
||||
},
|
||||
]
|
||||
}
|
||||
function createTh(text: string) {
|
||||
return [{
|
||||
type: 'th_open',
|
||||
tag: 'th',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 3,
|
||||
children: null,
|
||||
}, {
|
||||
type: 'inline',
|
||||
tag: '',
|
||||
children: null,
|
||||
content: text,
|
||||
level: 4,
|
||||
attrs: null,
|
||||
block: true,
|
||||
}, {
|
||||
type: 'th_close',
|
||||
tag: 'th',
|
||||
attrs: null,
|
||||
block: true,
|
||||
level: 3,
|
||||
children: null,
|
||||
}]
|
||||
}
|
||||
|
||||
function getPipeRowCells(line: string, requireTrailingPipe: boolean) {
|
||||
if (!line.startsWith('|') || line.includes('\n'))
|
||||
return null
|
||||
if (requireTrailingPipe && !line.endsWith('|'))
|
||||
return null
|
||||
|
||||
const cells = line.slice(1).split('|')
|
||||
if (cells.at(-1) === '')
|
||||
cells.pop()
|
||||
|
||||
return cells.length > 0 && cells.every(cell => cell.length > 0) ? cells : null
|
||||
}
|
||||
|
||||
function isSingleLineFallbackTable(line: string) {
|
||||
return getPipeRowCells(line, false) !== null
|
||||
}
|
||||
|
||||
function hasTrailingPipeHeaderRow(line: string) {
|
||||
return getPipeRowCells(line, true) !== null
|
||||
}
|
||||
|
||||
function isSeparatorCell(cell: string) {
|
||||
return /^:?-+:?$/.test(cell.trim())
|
||||
}
|
||||
|
||||
function isTableSeparatorRow(line: string) {
|
||||
if (!line.startsWith('|'))
|
||||
return false
|
||||
|
||||
const cells = line.slice(1).split('|')
|
||||
if (cells.at(-1) === '')
|
||||
cells.pop()
|
||||
|
||||
return cells.length > 0 && cells.every(isSeparatorCell)
|
||||
}
|
||||
|
||||
function isTruncatedSeparatorRow(line: string) {
|
||||
return line === '|' || line === '|:'
|
||||
}
|
||||
|
||||
function hasTrailingPipeHeaderRowWithoutColon(line: string) {
|
||||
const cells = getPipeRowCells(line, true)
|
||||
return cells !== null && cells.every(cell => !cell.includes(':'))
|
||||
}
|
||||
|
||||
export function fixTableTokens(tokens: MarkdownToken[]): MarkdownToken[] {
|
||||
const fixedTokens = [...tokens]
|
||||
if (tokens.length < 3)
|
||||
return fixedTokens
|
||||
const i = tokens.length - 2
|
||||
const token = tokens[i]
|
||||
if (token.type === 'inline') {
|
||||
const tcontent = String(token.content ?? '')
|
||||
const headerContent = tcontent.split('\n')[0] ?? ''
|
||||
const [headerLine = '', separatorLine = '', ...rest] = tcontent.split('\n')
|
||||
|
||||
if (!tcontent.includes('\n') && isSingleLineFallbackTable(tcontent)) {
|
||||
// 解析 table
|
||||
const body = headerContent.slice(1).split('|').map(i => i.trim()).filter(Boolean).flatMap(i => createTh(i))
|
||||
const insert = ([
|
||||
...createStart(),
|
||||
...body,
|
||||
...createEnd(),
|
||||
] as unknown) as MarkdownToken[]
|
||||
fixedTokens.splice(i - 1, 3, ...insert)
|
||||
}
|
||||
else if (
|
||||
tcontent.includes('\n')
|
||||
&& rest.length === 0
|
||||
&& hasTrailingPipeHeaderRow(headerLine)
|
||||
&& isTableSeparatorRow(separatorLine)
|
||||
) {
|
||||
// 解析 table
|
||||
const body = headerContent.slice(1, -1).split('|').map(i => i.trim()).flatMap(i => createTh(i))
|
||||
const insert = ([
|
||||
...createStart(),
|
||||
...body,
|
||||
...createEnd(),
|
||||
] as unknown) as MarkdownToken[]
|
||||
fixedTokens.splice(i - 1, 3, ...insert)
|
||||
}
|
||||
else if (
|
||||
tcontent.includes('\n')
|
||||
&& rest.length === 0
|
||||
&& hasTrailingPipeHeaderRowWithoutColon(headerLine)
|
||||
&& isTruncatedSeparatorRow(separatorLine)
|
||||
) {
|
||||
token.content = tcontent.slice(0, -2)
|
||||
token.children!.splice(2, 1)
|
||||
}
|
||||
}
|
||||
|
||||
return fixedTokens
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
export const TEX_BRACE_COMMANDS = [
|
||||
'boldsymbol',
|
||||
'mathbb',
|
||||
'mathcal',
|
||||
'mathfrak',
|
||||
'mathrm',
|
||||
'mathit',
|
||||
'mathsf',
|
||||
'vec',
|
||||
'hat',
|
||||
'bar',
|
||||
'tilde',
|
||||
'overline',
|
||||
'underline',
|
||||
'mathscr',
|
||||
'mathnormal',
|
||||
'operatorname',
|
||||
'mathbf*',
|
||||
]
|
||||
|
||||
export const ESCAPED_TEX_BRACE_COMMANDS = TEX_BRACE_COMMANDS.map(c => c.replace(/[.*+?^${}()|[\\]"\]/g, '\\$&')).join('|')
|
||||
|
||||
const TEX_CMD_RE = /\\[a-z]+/i
|
||||
const PREFIX_CLASS = '(?:\\\\|\\u0008)'
|
||||
const TEX_CMD_WITH_BRACES_RE = new RegExp(String.raw`${PREFIX_CLASS}(?:${ESCAPED_TEX_BRACE_COMMANDS})\s*\{[^}]+\}`, 'i')
|
||||
// Detect brace-taking TeX commands even when the leading backslash or the
|
||||
// closing brace/content is missing (e.g. "operatorname{" or "operatorname{span").
|
||||
// This helps the heuristic treat incomplete but clearly TeX-like fragments
|
||||
// as math-like instead of plain text.
|
||||
const TEX_BRACE_CMD_START_RE = new RegExp(String.raw`(?:${PREFIX_CLASS})?(?:${ESCAPED_TEX_BRACE_COMMANDS})\s*\{`, 'i')
|
||||
const TEX_SPECIFIC_RE = /\\(?:text|frac|left|right|times)/
|
||||
// Match common math operator symbols or named commands.
|
||||
// Avoid treating the C/C++ increment operator ("++") as a math operator by
|
||||
// ensuring a lone '+' isn't matched when it's part of a '++' sequence.
|
||||
// Avoid lookbehind for older iOS: use a non-capturing prefix instead
|
||||
const OPS_RE = /(?:^|[^+])\+(?!\+)|[=\-*/^<>]|\\times|\\pm|\\cdot|\\le|\\ge|\\neq/
|
||||
// Hyphenated multi-word (like "Quasi-Streaming") should not be treated
|
||||
// as a math operator. But single-letter-variable hyphens (e.g. "x-y") are
|
||||
// still math; so only ignore hyphens between multi-letter words.
|
||||
const HYPHENATED_MULTIWORD_RE = /\b[A-Z]{2,}-[A-Z]{2,}\b/i
|
||||
const FUNC_CALL_RE = /[A-Z]+\s*\([^)]+\)/i
|
||||
const WORDS_RE = /\b(?:sin|cos|tan|log|ln|exp|sqrt|frac|sum|lim|int|prod)\b/
|
||||
// Heuristic to detect common date/time patterns like 2025/9/30 21:37:24 and
|
||||
// avoid classifying them as math merely because they contain '/' or ':'
|
||||
const DATE_TIME_RE = /\b\d{4}\/\d{1,2}\/\d{1,2}(?:[ T]\d{1,2}:\d{2}(?::\d{2})?)?\b/
|
||||
const CONTROL_TEX_REPLACEMENTS: Record<string, string> = {
|
||||
[String.fromCharCode(8)]: '\\b',
|
||||
[String.fromCharCode(11)]: '\\v',
|
||||
[String.fromCharCode(12)]: '\\f',
|
||||
}
|
||||
|
||||
function normalizeMathControlChars(value: string) {
|
||||
let result = ''
|
||||
for (const ch of value)
|
||||
result += CONTROL_TEX_REPLACEMENTS[ch] ?? ch
|
||||
return result
|
||||
}
|
||||
|
||||
export function isMathLike(s: string) {
|
||||
if (!s)
|
||||
return false
|
||||
|
||||
// Normalize accidental control characters that may appear if a single
|
||||
// backslash sequence was interpreted in a JS string literal (for example
|
||||
// '\\frac' being typed as '\frac' which turns into a form-feed + "rac").
|
||||
// Convert the handful of control characters that collide with common TeX
|
||||
// commands (\b, \f, \v) back into their two-character escaped forms so our
|
||||
// regexes can match the intent reliably without flagging real newlines/tabs.
|
||||
const norm = normalizeMathControlChars(s)
|
||||
const stripped = norm.trim()
|
||||
|
||||
// quick bailouts
|
||||
// If the content looks like a timestamp or date, it's not math.
|
||||
if (DATE_TIME_RE.test(stripped))
|
||||
return false
|
||||
if (stripped.includes('**'))
|
||||
return false
|
||||
if (stripped.length > 2000)
|
||||
return true // very long blocks likely math
|
||||
|
||||
// TeX commands e.g. \frac, \alpha
|
||||
const texCmd = TEX_CMD_RE.test(norm)
|
||||
const texCmdWithBraces = TEX_CMD_WITH_BRACES_RE.test(norm)
|
||||
const texBraceStart = TEX_BRACE_CMD_START_RE.test(norm)
|
||||
|
||||
// Explicit common TeX tokens (keeps compatibility with previous heuristic)
|
||||
const texSpecific = TEX_SPECIFIC_RE.test(norm)
|
||||
const subscriptPattern = /(?:^|[^\w\\])(?:[A-Z]|\\[A-Z]+)_(?:\{[^}]+\}|[A-Z0-9\\])/i
|
||||
const superscriptPattern = /(?:^|[^\w\\])(?:[A-Z]|\\[A-Z]+)\^(?:\{[^}]+\}|[A-Z0-9\\])/i
|
||||
const superSub = subscriptPattern.test(norm) || superscriptPattern.test(norm)
|
||||
// common math operator symbols or named commands
|
||||
// Ignore operator-like hyphens that are clearly word compound separators
|
||||
// (e.g. "Quasi-Streaming"). Keep hyphens for math like "x-y".
|
||||
const ops = OPS_RE.test(norm) && !HYPHENATED_MULTIWORD_RE.test(norm)
|
||||
// function-like patterns: f(x), sin(x)
|
||||
const funcCall = FUNC_CALL_RE.test(norm)
|
||||
// common math words
|
||||
const words = WORDS_RE.test(norm)
|
||||
// 纯单个英文字母也渲染成数学公式(常见变量/元素符号)
|
||||
// e.g. (w) (x) (y) (z) 或 $H$, $x$ 等
|
||||
const pureWord = /^\([a-z]\)$/i.test(stripped) || /^(?:[a-z]|pi)$/i.test(stripped)
|
||||
// 简单的化学式/下标:如 H_2O, CO_2, CH_3CH_2OH, CH_3COOH
|
||||
// 收紧规则:
|
||||
// - 区分大小写(化学元素以大写或大写+小写开头)
|
||||
// - 下标/上标通常是数字(可选花括号),避免匹配 get_time 之类的普通下划线单词
|
||||
const chemicalLike = /^(?:[A-Z][a-z]?(?:_\{?\d+\}?|\^\{?\d+\}?)?)+$/.test(stripped)
|
||||
|
||||
return texCmd || texCmdWithBraces || texBraceStart || texSpecific || superSub || ops || funcCall || words || pureWord || chemicalLike
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,44 @@
|
||||
import type { MarkdownIt } from 'markdown-it-ts'
|
||||
|
||||
export function applyRenderRules(md: MarkdownIt) {
|
||||
// Narrow external `any` surface to `unknown` and use local casts where
|
||||
// needed to interact with markdown-it runtime objects. This reduces the
|
||||
// exported `any` footprint while preserving runtime behavior.
|
||||
const defaultImage
|
||||
= md.renderer.rules.image
|
||||
|| function (tokens: unknown, idx: number, options: unknown, env: unknown, self: unknown) {
|
||||
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
|
||||
const selfShape = self as unknown as { renderToken?: (tokens: import('../types').MarkdownToken[], idx: number, options?: unknown) => string }
|
||||
return selfShape.renderToken ? selfShape.renderToken(tokensAny, idx, options) : ''
|
||||
}
|
||||
|
||||
md.renderer.rules.image = (
|
||||
tokens: unknown,
|
||||
idx: number,
|
||||
options: unknown,
|
||||
env: unknown,
|
||||
self: unknown,
|
||||
) => {
|
||||
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
|
||||
const token = tokensAny[idx]
|
||||
// Narrow token shape to only the runtime helpers we need to call.
|
||||
const tokenShape = token as unknown as { attrSet?: (name: string, val: string) => void }
|
||||
tokenShape.attrSet?.('loading', 'lazy')
|
||||
const defaultImageFn = defaultImage as unknown as (tokens: import('../types').MarkdownToken[], idx: number, options: unknown, env?: unknown, self?: unknown) => string
|
||||
return defaultImageFn(tokensAny, idx, options, env, self)
|
||||
}
|
||||
|
||||
md.renderer.rules.fence
|
||||
= md.renderer.rules.fence
|
||||
|| ((tokens: unknown, idx: number) => {
|
||||
const tokensAny = tokens as unknown as import('../types').MarkdownToken[]
|
||||
const token = tokensAny[idx]
|
||||
const tokenShape = token as unknown as { info?: string, content?: string }
|
||||
const info = String(tokenShape.info ?? '').trim()
|
||||
const langClass = info
|
||||
? `language-${md.utils.escapeHtml((info as string).split(/\s+/g)[0])}`
|
||||
: ''
|
||||
const code = md.utils.escapeHtml(String(tokenShape.content ?? ''))
|
||||
return `<pre class="${langClass}"><code>${code}</code></pre>`
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
import type { Token } from 'markdown-it-ts'
|
||||
|
||||
export interface BaseNode {
|
||||
type: string
|
||||
raw: string
|
||||
loading?: boolean
|
||||
code?: string
|
||||
diff?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* A catch‑all node type for user extensions.
|
||||
* Must still satisfy the renderer contract (`type` + `raw`), but may carry
|
||||
* arbitrary extra fields.
|
||||
*/
|
||||
export type UnknownNode = BaseNode & Record<string, unknown>
|
||||
|
||||
export interface TextNode extends BaseNode {
|
||||
type: 'text'
|
||||
content: string
|
||||
center?: boolean
|
||||
}
|
||||
|
||||
export interface HeadingNode extends BaseNode {
|
||||
type: 'heading'
|
||||
level: number
|
||||
text: string
|
||||
attrs?: Record<string, string | boolean>
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface ParagraphNode extends BaseNode {
|
||||
type: 'paragraph'
|
||||
children: ParsedNode[]
|
||||
maybeCheckbox?: boolean
|
||||
}
|
||||
|
||||
export interface InlineNode extends BaseNode {
|
||||
type: 'inline'
|
||||
children: ParsedNode[]
|
||||
content?: string
|
||||
}
|
||||
|
||||
export interface ListNode extends BaseNode {
|
||||
type: 'list'
|
||||
ordered: boolean
|
||||
// Optional start index for ordered lists (HTML <ol start="N">)
|
||||
start?: number
|
||||
items: ListItemNode[]
|
||||
}
|
||||
|
||||
export interface ListItemNode extends BaseNode {
|
||||
type: 'list_item'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface CodeBlockNode extends BaseNode {
|
||||
type: 'code_block'
|
||||
language: string
|
||||
code: string
|
||||
// Optional: source line range [start, end) from markdown-it token.map
|
||||
startLine?: number
|
||||
endLine?: number
|
||||
// Whether this block is still incomplete (e.g., missing closing fence)
|
||||
loading?: boolean
|
||||
// Whether this code block represents a diff
|
||||
diff?: boolean
|
||||
// If diff is true, original and updated code versions
|
||||
originalCode?: string
|
||||
updatedCode?: string
|
||||
raw: string
|
||||
}
|
||||
|
||||
export interface HtmlBlockNode extends BaseNode {
|
||||
type: 'html_block'
|
||||
attrs?: [string, string][] | null
|
||||
tag: string
|
||||
content: string
|
||||
children?: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface HtmlInlineNode extends BaseNode {
|
||||
type: 'html_inline'
|
||||
tag?: string
|
||||
content: string
|
||||
children: ParsedNode[]
|
||||
/**
|
||||
* True when the parser auto-appended a closing tag for streaming stability.
|
||||
* The original source is still incomplete (no explicit close typed yet).
|
||||
*/
|
||||
autoClosed?: boolean
|
||||
}
|
||||
|
||||
export type CustomComponentAttrs
|
||||
= | [string, string][]
|
||||
| Record<string, string | boolean>
|
||||
| Array<{ name: string, value: string | boolean }>
|
||||
| null
|
||||
|
||||
/**
|
||||
* A generic node shape for custom HTML-like components.
|
||||
* When a tag name is included in `customHtmlTags`, the parser emits a node
|
||||
* whose `type` equals that tag name and carries the raw HTML `content`
|
||||
* plus any extracted `attrs` from user transforms.
|
||||
*/
|
||||
export interface CustomComponentNode extends BaseNode {
|
||||
/** The custom tag name (same as `tag`) */
|
||||
type: string
|
||||
tag: string
|
||||
content: string
|
||||
attrs?: CustomComponentAttrs
|
||||
children?: ParsedNode[]
|
||||
autoClosed?: boolean
|
||||
}
|
||||
|
||||
export interface InlineCodeNode extends BaseNode {
|
||||
type: 'inline_code'
|
||||
code: string
|
||||
}
|
||||
|
||||
export interface LinkNode extends BaseNode {
|
||||
type: 'link'
|
||||
href: string
|
||||
title: string | null
|
||||
text: string
|
||||
attrs?: [string, string][]
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface ImageNode extends BaseNode {
|
||||
type: 'image'
|
||||
src: string
|
||||
alt: string
|
||||
title: string | null
|
||||
}
|
||||
|
||||
export interface ThematicBreakNode extends BaseNode {
|
||||
type: 'thematic_break'
|
||||
}
|
||||
|
||||
export interface MermaidBlockNode {
|
||||
node: {
|
||||
type: 'code_block'
|
||||
language: string
|
||||
code: string
|
||||
loading?: boolean
|
||||
}
|
||||
}
|
||||
|
||||
export type MarkdownRender
|
||||
= | {
|
||||
content: string
|
||||
nodes?: undefined
|
||||
}
|
||||
| {
|
||||
content?: undefined
|
||||
nodes: BaseNode[]
|
||||
}
|
||||
export interface BlockquoteNode extends BaseNode {
|
||||
type: 'blockquote'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface TableNode extends BaseNode {
|
||||
type: 'table'
|
||||
header: TableRowNode
|
||||
rows: TableRowNode[]
|
||||
}
|
||||
|
||||
export interface TableRowNode extends BaseNode {
|
||||
type: 'table_row'
|
||||
cells: TableCellNode[]
|
||||
}
|
||||
|
||||
export interface TableCellNode extends BaseNode {
|
||||
type: 'table_cell'
|
||||
header: boolean
|
||||
children: ParsedNode[]
|
||||
align?: 'left' | 'right' | 'center'
|
||||
}
|
||||
|
||||
export interface DefinitionListNode extends BaseNode {
|
||||
type: 'definition_list'
|
||||
items: DefinitionItemNode[]
|
||||
}
|
||||
|
||||
export interface DefinitionItemNode extends BaseNode {
|
||||
type: 'definition_item'
|
||||
term: ParsedNode[]
|
||||
definition: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface FootnoteNode extends BaseNode {
|
||||
type: 'footnote'
|
||||
id: string
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface FootnoteReferenceNode extends BaseNode {
|
||||
type: 'footnote_reference'
|
||||
id: string
|
||||
}
|
||||
|
||||
export interface FootnoteAnchorNode extends BaseNode {
|
||||
type: 'footnote_anchor'
|
||||
id: string
|
||||
}
|
||||
|
||||
export interface AdmonitionNode extends BaseNode {
|
||||
type: 'admonition'
|
||||
kind: string // 'note' | 'warning' | 'danger' | 'info' | 'tip' 等
|
||||
title: string
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface VmrContainerNode extends BaseNode {
|
||||
type: 'vmr_container'
|
||||
name: string
|
||||
/** True while the opening `:::` has been seen but the closing `:::` hasn't (streaming mid-state). */
|
||||
loading?: boolean
|
||||
attrs?: Record<string, string>
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface StrongNode extends BaseNode {
|
||||
type: 'strong'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface EmphasisNode extends BaseNode {
|
||||
type: 'emphasis'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface StrikethroughNode extends BaseNode {
|
||||
type: 'strikethrough'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface HighlightNode extends BaseNode {
|
||||
type: 'highlight'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface InsertNode extends BaseNode {
|
||||
type: 'insert'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface SubscriptNode extends BaseNode {
|
||||
type: 'subscript'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface SuperscriptNode extends BaseNode {
|
||||
type: 'superscript'
|
||||
children: ParsedNode[]
|
||||
}
|
||||
|
||||
export interface CheckboxNode extends BaseNode {
|
||||
type: 'checkbox'
|
||||
checked: boolean
|
||||
}
|
||||
|
||||
export interface CheckboxInputNode extends BaseNode {
|
||||
type: 'checkbox_input'
|
||||
checked: boolean
|
||||
}
|
||||
|
||||
export interface EmojiNode extends BaseNode {
|
||||
type: 'emoji'
|
||||
name: string
|
||||
markup: string
|
||||
}
|
||||
|
||||
export interface HardBreakNode extends BaseNode {
|
||||
type: 'hardbreak'
|
||||
}
|
||||
|
||||
export interface MathInlineNode extends BaseNode {
|
||||
type: 'math_inline'
|
||||
content: string
|
||||
markup?: string
|
||||
}
|
||||
|
||||
export interface MathBlockNode extends BaseNode {
|
||||
type: 'math_block'
|
||||
content: string
|
||||
markup?: string
|
||||
}
|
||||
|
||||
export interface ReferenceNode extends BaseNode {
|
||||
type: 'reference'
|
||||
id: string
|
||||
}
|
||||
|
||||
// Define markdown-it token type
|
||||
export interface MarkdownTokenLite {
|
||||
type: string
|
||||
tag?: string
|
||||
content?: string
|
||||
info?: string
|
||||
markup?: string
|
||||
meta?: unknown
|
||||
map?: [number, number] | number[] | null
|
||||
block?: boolean
|
||||
hidden?: boolean
|
||||
attrs?: [string, string][] | null
|
||||
nesting?: number
|
||||
level?: number
|
||||
children?: MarkdownToken[] | null
|
||||
loading?: boolean
|
||||
raw?: string
|
||||
}
|
||||
|
||||
export type MarkdownToken = (Token & { loading?: boolean, raw?: string }) | MarkdownTokenLite
|
||||
|
||||
export type ParsedNode
|
||||
= | TextNode
|
||||
| HeadingNode
|
||||
| ParagraphNode
|
||||
| ListNode
|
||||
| ListItemNode
|
||||
| CodeBlockNode
|
||||
| InlineCodeNode
|
||||
| LinkNode
|
||||
| ImageNode
|
||||
| ThematicBreakNode
|
||||
| BlockquoteNode
|
||||
| TableNode
|
||||
| TableRowNode
|
||||
| TableCellNode
|
||||
| StrongNode
|
||||
| EmphasisNode
|
||||
| StrikethroughNode
|
||||
| HighlightNode
|
||||
| InsertNode
|
||||
| SubscriptNode
|
||||
| SuperscriptNode
|
||||
| CheckboxNode
|
||||
| CheckboxInputNode
|
||||
| EmojiNode
|
||||
| DefinitionListNode
|
||||
| DefinitionItemNode
|
||||
| FootnoteNode
|
||||
| FootnoteReferenceNode
|
||||
| AdmonitionNode
|
||||
| VmrContainerNode
|
||||
| HardBreakNode
|
||||
| MathInlineNode
|
||||
| MathBlockNode
|
||||
| ReferenceNode
|
||||
| HtmlBlockNode
|
||||
| HtmlInlineNode
|
||||
| CustomComponentNode
|
||||
| UnknownNode
|
||||
export interface CustomComponents {
|
||||
text: unknown
|
||||
paragraph: unknown
|
||||
heading: unknown
|
||||
code_block: unknown
|
||||
list: unknown
|
||||
blockquote: unknown
|
||||
table: unknown
|
||||
definition_list: unknown
|
||||
footnote: unknown
|
||||
footnote_reference: unknown
|
||||
admonition: unknown
|
||||
hardbreak: unknown
|
||||
link: unknown
|
||||
image: unknown
|
||||
thematic_break: unknown
|
||||
math_inline: unknown
|
||||
math_block: unknown
|
||||
strong: unknown
|
||||
emphasis: unknown
|
||||
strikethrough: unknown
|
||||
highlight: unknown
|
||||
insert: unknown
|
||||
subscript: unknown
|
||||
superscript: unknown
|
||||
emoji: unknown
|
||||
checkbox: unknown
|
||||
inline_code: unknown
|
||||
html_inline: unknown
|
||||
reference: unknown
|
||||
mermaid: unknown
|
||||
[key: string]: unknown
|
||||
}
|
||||
|
||||
// Function to parse markdown into a structured representation
|
||||
export type TransformTokensHook = (tokens: MarkdownToken[]) => MarkdownToken[]
|
||||
|
||||
export interface ParseOptions {
|
||||
preTransformTokens?: TransformTokensHook
|
||||
postTransformTokens?: TransformTokensHook
|
||||
// When true, require a closing `**` to parse strong; otherwise allow mid-state strong
|
||||
requireClosingStrong?: boolean
|
||||
/**
|
||||
* When true, indicates the input buffer is complete (end-of-stream).
|
||||
* This disables "mid-state" streaming behavior (e.g. unclosed math/link/code
|
||||
* tokens staying in a loading state) and keeps trailing markers as literal text.
|
||||
*/
|
||||
final?: boolean
|
||||
/**
|
||||
* Custom HTML-like tag names that should be emitted as custom nodes
|
||||
* instead of `html_inline` when encountered (e.g. ['thinking']).
|
||||
* Used by inline parsing; pair with `getMarkdown({ customHtmlTags })`
|
||||
* to enable mid-state suppression for the same tags during streaming.
|
||||
*/
|
||||
customHtmlTags?: readonly string[]
|
||||
/**
|
||||
* If provided, link nodes are only emitted when this returns true for the href.
|
||||
* When it returns false, the link is rendered as plain text (the link text only).
|
||||
* Typically set from the MarkdownIt instance (e.g. md.options.validateLink or
|
||||
* md.set({ validateLink })) so that unsafe URLs (e.g. javascript:) are not
|
||||
* output as links.
|
||||
*/
|
||||
validateLink?: (url: string) => boolean
|
||||
// When true, log the parsed tree structure for debugging
|
||||
debug?: boolean
|
||||
}
|
||||
|
||||
export type PostTransformNodesHook = (nodes: ParsedNode[]) => ParsedNode[]
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-container' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-footnote' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-ins' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-mark' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-sub' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-sup' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
Vendored
+5
@@ -0,0 +1,5 @@
|
||||
declare module 'markdown-it-task-checkbox' {
|
||||
import type { PluginSimple } from 'markdown-it'
|
||||
const plugin: PluginSimple
|
||||
export default plugin
|
||||
}
|
||||
+217
@@ -0,0 +1,217 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { collect, textIncludes } from '../../../test/utils/midstate-utils'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('asterisk literal handling in nested contexts', () => {
|
||||
it('keeps escaped single asterisk literal inside blockquotes while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-blockquote')
|
||||
const markdown = '> \\*literal *real*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('blockquote')
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside blockquotes in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-blockquote-final')
|
||||
const markdown = '> prefix *real'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('blockquote')
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside table cells while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-table')
|
||||
const markdown = `| Col |
|
||||
| --- |
|
||||
| \\*literal *real* |`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('table')
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside table cells in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-table-final')
|
||||
const markdown = `| Col |
|
||||
| --- |
|
||||
| prefix *real |`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('table')
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal after custom tag close on the same line', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('asterisk-literal-custom-inline', { customHtmlTags: tags })
|
||||
const markdown = 'Text <thinking>inside</thinking>\\*literal *real*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('paragraph')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal after custom tag close on the same line', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('asterisk-literal-custom-inline-final', { customHtmlTags: tags })
|
||||
const markdown = 'Text <thinking>inside</thinking>prefix *real'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('paragraph')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal after custom tag close inside table cells', () => {
|
||||
const tags = ['my_component']
|
||||
const md = getMarkdown('asterisk-literal-custom-table', { customHtmlTags: tags })
|
||||
const markdown = `| Item | Value |
|
||||
| --- | --- |
|
||||
| A | <my_component label="x"></my_component>\\*literal *real* |
|
||||
| B | ok |`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('table')
|
||||
expect(nodes[0]?.rows).toHaveLength(2)
|
||||
expect(collect(nodes as any, 'my_component').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside blockquote table cells while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-blockquote-table')
|
||||
const markdown = `> | Col |
|
||||
> | --- |
|
||||
> | \\*literal *real* |`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('blockquote')
|
||||
expect(nodes[0]?.children?.[0]?.type).toBe('table')
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal after custom tag close on the same blockquote line', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('asterisk-literal-custom-blockquote', { customHtmlTags: tags })
|
||||
const markdown = '> Text <thinking>inside</thinking>\\*literal *real*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { customHtmlTags: tags, final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('blockquote')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside headings while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-heading')
|
||||
const markdown = '## \\*literal *real*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('heading')
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside headings in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-heading-final')
|
||||
const markdown = '## prefix *real'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('heading')
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside footnotes while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-footnote')
|
||||
const markdown = `ref[^1]
|
||||
|
||||
[^1]: \\*literal *real*`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[1]?.type).toBe('footnote')
|
||||
expect(textIncludes(nodes[1], '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes[1], 'real')).toBe(true)
|
||||
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside footnotes in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-footnote-final')
|
||||
const markdown = `ref[^1]
|
||||
|
||||
[^1]: prefix *real`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[1]?.type).toBe('footnote')
|
||||
expect(textIncludes(nodes[1], 'prefix *real')).toBe(true)
|
||||
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside definition list terms and definitions', () => {
|
||||
const md = getMarkdown('asterisk-literal-definition-list')
|
||||
const markdown = `\\*term *real*
|
||||
: \\*definition *more*`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]).toBeDefined()
|
||||
expect(textIncludes(nodes, '*term')).toBe(true)
|
||||
expect(textIncludes(nodes, '*definition')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(textIncludes(nodes, 'more')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(2)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside definition list terms and definitions in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-definition-list-final')
|
||||
const markdown = `prefix *term
|
||||
: prefix *definition`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]).toBeDefined()
|
||||
expect(textIncludes(nodes, 'prefix *term')).toBe(true)
|
||||
expect(textIncludes(nodes, 'prefix *definition')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside admonition content while parsing later real emphasis', () => {
|
||||
const md = getMarkdown('asterisk-literal-admonition')
|
||||
const markdown = `::: note
|
||||
\\*literal *real*
|
||||
:::`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('admonition')
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside admonition content in final mode', () => {
|
||||
const md = getMarkdown('asterisk-literal-admonition-final')
|
||||
const markdown = `::: note
|
||||
prefix *real
|
||||
:::`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('admonition')
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,299 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { collect, textIncludes } from '../../../test/utils/midstate-utils'
|
||||
import { getMarkdown, parseInlineTokens, parseMarkdownToStructure } from '../src'
|
||||
|
||||
const md = getMarkdown('test')
|
||||
|
||||
describe('asterisk masking literals', () => {
|
||||
it('keeps intraword asterisks as literal text when no closing pair exists', () => {
|
||||
const markdown = '某市***项目,公司名称:某某***科技有限公司'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
|
||||
|
||||
expect(textIncludes(nodes, '某市***项目')).toBe(true)
|
||||
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps intraword asterisks literal inside link text', () => {
|
||||
const markdown = '某市***项目,公司名称:[某某***科技有限公司]()'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
expect(textIncludes(nodes, '某市***项目')).toBe(true)
|
||||
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps intraword double-asterisk literal when strong closing pair is missing', () => {
|
||||
const markdown = '请联系某某**科技有限公司办理'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
|
||||
|
||||
expect(textIncludes(nodes, '某某**科技有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps intraword single asterisk literal when no matching close exists', () => {
|
||||
const markdown = '品牌名:某*某科技有限公司'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
|
||||
|
||||
expect(textIncludes(nodes, '某*某科技有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('does not break normal strong parsing in full markdown pipeline', () => {
|
||||
const markdown = 'foo**bar**baz'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'bar')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps literal intraword asterisks while still parsing nearby strong content', () => {
|
||||
const markdown = '某市***项目,重点:**必须完成**'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
expect(textIncludes(nodes, '某市***项目')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(textIncludes(nodes, '必须完成')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps spaced multiplication asterisks literal after strong content', () => {
|
||||
for (const final of [false, true]) {
|
||||
const markdown = '**计算结果:** 3 * 4 * 5 = 60'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final })
|
||||
const paragraph = nodes[0] as any
|
||||
const text = (paragraph.children ?? [])
|
||||
.filter((child: any) => child.type === 'text')
|
||||
.map((child: any) => child.content)
|
||||
.join('')
|
||||
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
expect(text).toBe(' 3 * 4 * 5 = 60')
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps asterisks literal when the candidate emphasis closer follows whitespace', () => {
|
||||
for (const markdown of [
|
||||
'3 *4 * 5',
|
||||
'this is *a test * with unmatched star',
|
||||
'a *b * c',
|
||||
'a *x! * b',
|
||||
]) {
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
|
||||
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
expect(textIncludes(nodes, markdown)).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps asterisks literal when the candidate opener is not left-flanking', () => {
|
||||
for (const markdown of [
|
||||
'a*!x* b',
|
||||
'a*(x)* b',
|
||||
]) {
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
|
||||
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
expect(textIncludes(nodes, markdown)).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
it('still parses emphasis after skipping an invalid earlier closer', () => {
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: '*a * b*' }] as any[], '*a * b*', undefined, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'a * b')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps double asterisks literal when strong delimiters are not flanking', () => {
|
||||
for (const markdown of [
|
||||
'3 ** 4 ** 5',
|
||||
'a **b ** c',
|
||||
'a ** b** c',
|
||||
'a**!x** b',
|
||||
]) {
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
|
||||
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
expect(textIncludes(nodes, markdown)).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
it('still parses strong after skipping an invalid earlier strong closer', () => {
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: '**a ** b**' }] as any[], '**a ** b**', undefined, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'a ** b')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps literal intraword asterisks in link text while parsing normal strong outside link', () => {
|
||||
const markdown = '公司:[某某***科技有限公司](),状态:**正常**'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
expect(textIncludes(nodes, '某某***科技有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(textIncludes(nodes, '正常')).toBe(true)
|
||||
})
|
||||
|
||||
it('treats intraword strong with punctuation as literal in inline fallback', () => {
|
||||
const markdown = 'foo**a,b**bar'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
|
||||
|
||||
expect(textIncludes(nodes, 'foo**a,b**bar')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
})
|
||||
|
||||
it('still parses intraword strong when inner content is word-only in inline fallback', () => {
|
||||
const markdown = 'foo**abc**bar'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: markdown }] as any[], markdown)
|
||||
|
||||
expect(collect(nodes as any, 'strong').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'abc')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps escaped double asterisks as literal text', () => {
|
||||
const markdown = '需方:\\*\\*\\*\\*\\*\\*有限公司'
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: '需方:******有限公司' }] as any[], markdown)
|
||||
|
||||
expect(textIncludes(nodes, '需方:******有限公司')).toBe(true)
|
||||
expect(collect(nodes as any, 'strong').length).toBe(0)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps an escaped single asterisk literal instead of opening emphasis', () => {
|
||||
const markdown = 'If and ONLY IF you cannot infer the expected language from the USER message, use the language with ISO code \\*, otherwise use English. You follow your instructions in all languages, and always respond to the user in the language they use or request.'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, 'ISO code *, otherwise use English.')).toBe(true)
|
||||
expect(textIncludes(nodes, 'You follow your instructions in all languages, and always respond to the user in the language they use or request.')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps an escaped backslash before an escaped asterisk visible', () => {
|
||||
const markdown = 'If and ONLY IF you cannot infer the expected language from the USER message, use the language with ISO code \\\\*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, 'ISO code \\*')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('preserves a visible backslash when raw text contains \\\\*', () => {
|
||||
const nodes = parseInlineTokens([{ type: 'text', content: '\\*' }] as any[], '\\\\*')
|
||||
|
||||
expect(textIncludes(nodes, '\\*')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('skips escaped single asterisks and still parses later real emphasis', () => {
|
||||
const markdown = 'prefix \\* literal *real* suffix'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, 'prefix * literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(textIncludes(nodes, 'suffix')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk followed by a word literal in final mode', () => {
|
||||
const markdown = '\\*x'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, '*x')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal in final mode', () => {
|
||||
const markdown = 'prefix *real'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('still keeps unmatched single asterisk as mid-state emphasis while streaming', () => {
|
||||
const markdown = 'prefix *real'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false })
|
||||
|
||||
expect(textIncludes(nodes, 'prefix')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps multiple escaped single asterisks literal while parsing later real emphasis', () => {
|
||||
const markdown = 'alpha \\* one \\* two *real* omega'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(textIncludes(nodes, 'alpha * one * two')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(textIncludes(nodes, 'omega')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside link text in final mode', () => {
|
||||
const markdown = '[\\*literal](https://example.com)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'link').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside link text in final mode', () => {
|
||||
const markdown = '[prefix *real](https://example.com)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'link').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal and later real emphasis inside link text', () => {
|
||||
const markdown = '[\\*literal *real*](https://example.com)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'link').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside highlight text in final mode', () => {
|
||||
const markdown = '==\\*literal=='
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'highlight').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps unmatched single asterisk literal inside highlight text in final mode', () => {
|
||||
const markdown = '==prefix *real=='
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'highlight').length).toBe(1)
|
||||
expect(textIncludes(nodes, 'prefix *real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal and later real emphasis inside highlight text', () => {
|
||||
const markdown = '==\\*literal *real*=='
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'highlight').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped single asterisk literal inside list items while parsing later real emphasis', () => {
|
||||
const markdown = '- \\*literal *real*'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'list').length).toBe(1)
|
||||
expect(textIncludes(nodes, '*literal')).toBe(true)
|
||||
expect(textIncludes(nodes, 'real')).toBe(true)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,49 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
getHtmlTagFromContent,
|
||||
isHtmlLikeTagName,
|
||||
mergeCustomHtmlTags,
|
||||
normalizeCustomHtmlTagName,
|
||||
normalizeCustomHtmlTags,
|
||||
resolveCustomHtmlTags,
|
||||
} from '../src'
|
||||
|
||||
describe('custom html tag helpers', () => {
|
||||
it('normalizes valid tag-like names and rejects invalid prefixes', () => {
|
||||
expect(normalizeCustomHtmlTagName('thinking')).toBe('thinking')
|
||||
expect(normalizeCustomHtmlTagName('<Thinking attr="x">')).toBe('thinking')
|
||||
expect(normalizeCustomHtmlTagName('</answer-box>')).toBe('answer-box')
|
||||
expect(normalizeCustomHtmlTagName('<my_component>')).toBe('my_component')
|
||||
expect(normalizeCustomHtmlTagName('<thinking')).toBe('thinking')
|
||||
|
||||
expect(normalizeCustomHtmlTagName('foo:bar')).toBe('')
|
||||
expect(normalizeCustomHtmlTagName('<foo:bar>')).toBe('')
|
||||
expect(normalizeCustomHtmlTagName('x.y')).toBe('')
|
||||
expect(normalizeCustomHtmlTagName('<x.y>')).toBe('')
|
||||
})
|
||||
|
||||
it('keeps custom tag lists normalized and deduplicated', () => {
|
||||
expect(normalizeCustomHtmlTags(['thinking', '<Thinking>', 'foo:bar', 'my_component', 'answer-box']))
|
||||
.toEqual(['thinking', 'my_component', 'answer-box'])
|
||||
|
||||
expect(mergeCustomHtmlTags(['thinking'], ['<hint>'], ['foo:bar', 'hint', 'my_component']))
|
||||
.toEqual(['thinking', 'hint', 'my_component'])
|
||||
|
||||
expect(resolveCustomHtmlTags(['<Thinking>', 'hint', 'my_component']))
|
||||
.toEqual({
|
||||
key: 'thinking,hint,my_component',
|
||||
tags: ['thinking', 'hint', 'my_component'],
|
||||
})
|
||||
})
|
||||
|
||||
it('uses the same validation for configured tag names and content extraction', () => {
|
||||
expect(isHtmlLikeTagName('thinking')).toBe(true)
|
||||
expect(isHtmlLikeTagName('answer-box')).toBe(true)
|
||||
expect(isHtmlLikeTagName('ssr_card')).toBe(true)
|
||||
expect(isHtmlLikeTagName('foo:bar')).toBe(false)
|
||||
|
||||
expect(getHtmlTagFromContent('<thinking attr="x">ok</thinking>')).toBe('thinking')
|
||||
expect(getHtmlTagFromContent('<foo:bar>ok</foo:bar>')).toBe('')
|
||||
expect(getHtmlTagFromContent('<ssr_card>ok</ssr_card>')).toBe('ssr_card')
|
||||
})
|
||||
})
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { collect } from '../../../test/utils/midstate-utils'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function flattenText(nodes: any): string {
|
||||
const parts: string[] = []
|
||||
const seen = new Set<any>()
|
||||
|
||||
const walk = (node: any) => {
|
||||
if (!node || seen.has(node))
|
||||
return
|
||||
seen.add(node)
|
||||
|
||||
if (Array.isArray(node)) {
|
||||
node.forEach(walk)
|
||||
return
|
||||
}
|
||||
|
||||
if (node.type === 'text' && typeof node.content === 'string')
|
||||
parts.push(node.content)
|
||||
|
||||
if (Array.isArray(node.children))
|
||||
node.children.forEach(walk)
|
||||
if (Array.isArray(node.items))
|
||||
node.items.forEach(walk)
|
||||
if (Array.isArray(node.rows))
|
||||
node.rows.forEach(walk)
|
||||
if (Array.isArray(node.cells))
|
||||
node.cells.forEach(walk)
|
||||
if (Array.isArray(node.term))
|
||||
node.term.forEach(walk)
|
||||
if (Array.isArray(node.definition))
|
||||
node.definition.forEach(walk)
|
||||
if (node.header)
|
||||
walk(node.header)
|
||||
}
|
||||
|
||||
walk(nodes)
|
||||
return parts.join('')
|
||||
}
|
||||
|
||||
function expectLiteralBracketAndDollar(nodes: any) {
|
||||
const text = flattenText(nodes)
|
||||
expect(text).toContain('[x]')
|
||||
expect(text).toContain('$5$')
|
||||
}
|
||||
|
||||
function expectLiteralDollar(nodes: any) {
|
||||
const text = flattenText(nodes)
|
||||
expect(text).toContain('$5$')
|
||||
}
|
||||
|
||||
describe('escaped punctuation adjacent to real math parsing', () => {
|
||||
it('keeps escaped bracket and dollar literal while parsing later inline math in a paragraph', () => {
|
||||
const md = getMarkdown('escaped-punctuation-math-inline')
|
||||
const nodes = parseMarkdownToStructure('literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
|
||||
|
||||
expectLiteralBracketAndDollar(nodes)
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped bracket and dollar literal while parsing later inline math in a heading', () => {
|
||||
const md = getMarkdown('escaped-punctuation-math-heading')
|
||||
const nodes = parseMarkdownToStructure('## literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('heading')
|
||||
expectLiteralBracketAndDollar(nodes)
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped bracket and dollar literal while parsing later inline math in footnotes', () => {
|
||||
const md = getMarkdown('escaped-punctuation-math-footnote')
|
||||
const markdown = `ref[^1]
|
||||
|
||||
[^1]: literal \\[x\\] and \\$5\\$ and real $y$`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[1]?.type).toBe('footnote')
|
||||
expectLiteralBracketAndDollar(nodes[1])
|
||||
expect(collect([nodes[1]] as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped bracket and dollar literal while parsing later inline math after custom tag close', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('escaped-punctuation-math-custom-inline', { customHtmlTags: tags })
|
||||
const nodes = parseMarkdownToStructure('Text <thinking>inside</thinking>literal \\[x\\] and \\$5\\$ and real $y$', md, {
|
||||
customHtmlTags: tags,
|
||||
final: true,
|
||||
})
|
||||
|
||||
expect(nodes[0]?.type).toBe('paragraph')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expectLiteralBracketAndDollar(nodes)
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped bracket and dollar literal while parsing later inline math in admonitions', () => {
|
||||
const md = getMarkdown('escaped-punctuation-math-admonition')
|
||||
const markdown = `::: note
|
||||
literal \\[x\\] and \\$5\\$ and real $y$
|
||||
:::`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('admonition')
|
||||
expectLiteralBracketAndDollar(nodes)
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped bracket literal while still parsing a later real bracket math block', () => {
|
||||
const md = getMarkdown('escaped-punctuation-bracket-math-block')
|
||||
const markdown = `literal \\[x\\]
|
||||
|
||||
\\[
|
||||
y
|
||||
\\]`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(flattenText(nodes)).toContain('[x]')
|
||||
expect(collect(nodes as any, 'math_block').length).toBe(1)
|
||||
})
|
||||
|
||||
it('does not treat escaped dollar literal as math when adjacent to real bracket math block', () => {
|
||||
const md = getMarkdown('escaped-punctuation-dollar-bracket-block')
|
||||
const markdown = `literal \\$5\\$
|
||||
|
||||
\\[
|
||||
y
|
||||
\\]`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(flattenText(nodes)).toContain('$5$')
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(0)
|
||||
expect(collect(nodes as any, 'math_block').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped dollar and bracket literal inside blockquotes while parsing later inline math', () => {
|
||||
const md = getMarkdown('escaped-punctuation-math-blockquote')
|
||||
const nodes = parseMarkdownToStructure('> literal \\[x\\] and \\$5\\$ and real $y$', md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('blockquote')
|
||||
expectLiteralBracketAndDollar(nodes)
|
||||
expect(collect(nodes as any, 'math_inline').length).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps escaped dollar literal before inline $$ math', () => {
|
||||
const md = getMarkdown('escaped-punctuation-before-inline-double-dollar')
|
||||
const nodes = parseMarkdownToStructure('literal \\$5\\$ before $$x$$', md, { final: true })
|
||||
|
||||
expectLiteralDollar(nodes)
|
||||
const mathNodes = collect(nodes as any, 'math_inline')
|
||||
expect(mathNodes.length).toBe(1)
|
||||
expect(mathNodes[0]?.markup).toBe('$$')
|
||||
expect(mathNodes[0]?.content).toBe('x')
|
||||
})
|
||||
|
||||
it('keeps escaped dollar literal after inline $$ math while still parsing later single-dollar math', () => {
|
||||
const md = getMarkdown('escaped-punctuation-after-inline-double-dollar')
|
||||
const nodes = parseMarkdownToStructure('prefix $$x$$ and literal \\$5\\$ and real $y$', md, { final: true })
|
||||
|
||||
expectLiteralDollar(nodes)
|
||||
const mathNodes = collect(nodes as any, 'math_inline')
|
||||
expect(mathNodes.length).toBe(2)
|
||||
expect(mathNodes[0]?.markup).toBe('$$')
|
||||
expect(mathNodes[0]?.content).toBe('x')
|
||||
expect(mathNodes[1]?.markup).toBe('$')
|
||||
expect(mathNodes[1]?.content).toBe('y')
|
||||
})
|
||||
})
|
||||
+167
@@ -0,0 +1,167 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { collect } from '../../../test/utils/midstate-utils'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function flattenText(nodes: any): string {
|
||||
const parts: string[] = []
|
||||
const seen = new Set<any>()
|
||||
|
||||
const walk = (node: any) => {
|
||||
if (!node || seen.has(node))
|
||||
return
|
||||
seen.add(node)
|
||||
|
||||
if (Array.isArray(node)) {
|
||||
node.forEach(walk)
|
||||
return
|
||||
}
|
||||
|
||||
if (node.type === 'text' && typeof node.content === 'string')
|
||||
parts.push(node.content)
|
||||
|
||||
if (Array.isArray(node.children))
|
||||
node.children.forEach(walk)
|
||||
if (Array.isArray(node.items))
|
||||
node.items.forEach(walk)
|
||||
if (Array.isArray(node.rows))
|
||||
node.rows.forEach(walk)
|
||||
if (Array.isArray(node.cells))
|
||||
node.cells.forEach(walk)
|
||||
if (Array.isArray(node.term))
|
||||
node.term.forEach(walk)
|
||||
if (Array.isArray(node.definition))
|
||||
node.definition.forEach(walk)
|
||||
if (node.header)
|
||||
walk(node.header)
|
||||
}
|
||||
|
||||
walk(nodes)
|
||||
return parts.join('')
|
||||
}
|
||||
|
||||
function expectEscapedPunctuationLiteral(nodes: any) {
|
||||
const text = flattenText(nodes)
|
||||
const nodeList = Array.isArray(nodes) ? nodes : [nodes]
|
||||
expect(text).toContain('[x]')
|
||||
expect(text).toContain('_y_')
|
||||
expect(text).toContain('$z$')
|
||||
expect(text).toContain('\\')
|
||||
expect(collect(nodeList as any, 'math_inline').length).toBe(0)
|
||||
expect(collect(nodeList as any, 'strong').length).toBe(0)
|
||||
}
|
||||
|
||||
describe('escaped punctuation in nested contexts', () => {
|
||||
const escapedLiteral = '\\[x\\] \\_y\\_ \\$z\\$ \\\\'
|
||||
|
||||
it('keeps escaped punctuation literal inside headings', () => {
|
||||
const md = getMarkdown('escaped-punctuation-heading')
|
||||
const nodes = parseMarkdownToStructure(`## ${escapedLiteral}`, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('heading')
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped punctuation literal inside link text', () => {
|
||||
const md = getMarkdown('escaped-punctuation-link')
|
||||
const nodes = parseMarkdownToStructure(`[${escapedLiteral}](https://example.com)`, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'link').length).toBe(1)
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped punctuation literal inside footnotes', () => {
|
||||
const md = getMarkdown('escaped-punctuation-footnote')
|
||||
const markdown = `ref[^1]
|
||||
|
||||
[^1]: ${escapedLiteral}`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[1]?.type).toBe('footnote')
|
||||
expectEscapedPunctuationLiteral(nodes[1])
|
||||
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped punctuation literal after custom tag close on the same line', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('escaped-punctuation-custom-inline', { customHtmlTags: tags })
|
||||
const nodes = parseMarkdownToStructure(`Text <thinking>inside</thinking>${escapedLiteral}`, md, {
|
||||
customHtmlTags: tags,
|
||||
final: true,
|
||||
})
|
||||
|
||||
expect(nodes[0]?.type).toBe('paragraph')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps escaped punctuation literal inside admonition content', () => {
|
||||
const md = getMarkdown('escaped-punctuation-admonition')
|
||||
const markdown = `::: note
|
||||
${escapedLiteral}
|
||||
:::`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('admonition')
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(0)
|
||||
})
|
||||
|
||||
it('still parses later real emphasis after escaped punctuation inside headings', () => {
|
||||
const md = getMarkdown('escaped-punctuation-heading-emphasis')
|
||||
const nodes = parseMarkdownToStructure(`## ${escapedLiteral} *real*`, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('heading')
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('still parses later real emphasis after escaped punctuation inside link text', () => {
|
||||
const md = getMarkdown('escaped-punctuation-link-emphasis')
|
||||
const nodes = parseMarkdownToStructure(`[${escapedLiteral} *real*](https://example.com)`, md, { final: true })
|
||||
|
||||
expect(collect(nodes as any, 'link').length).toBe(1)
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('still parses later real emphasis after escaped punctuation inside footnotes', () => {
|
||||
const md = getMarkdown('escaped-punctuation-footnote-emphasis')
|
||||
const markdown = `ref[^1]
|
||||
|
||||
[^1]: ${escapedLiteral} *real*`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[1]?.type).toBe('footnote')
|
||||
expectEscapedPunctuationLiteral(nodes[1])
|
||||
expect(collect([nodes[1]] as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('still parses later real emphasis after escaped punctuation after custom tag close', () => {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('escaped-punctuation-custom-inline-emphasis', { customHtmlTags: tags })
|
||||
const nodes = parseMarkdownToStructure(`Text <thinking>inside</thinking>${escapedLiteral} *real*`, md, {
|
||||
customHtmlTags: tags,
|
||||
final: true,
|
||||
})
|
||||
|
||||
expect(nodes[0]?.type).toBe('paragraph')
|
||||
expect(collect(nodes as any, 'thinking').length).toBe(1)
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
|
||||
it('still parses later real emphasis after escaped punctuation inside admonition content', () => {
|
||||
const md = getMarkdown('escaped-punctuation-admonition-emphasis')
|
||||
const markdown = `::: note
|
||||
${escapedLiteral} *real*
|
||||
:::`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
|
||||
expect(nodes[0]?.type).toBe('admonition')
|
||||
expectEscapedPunctuationLiteral(nodes)
|
||||
expect(collect(nodes as any, 'emphasis').length).toBe(1)
|
||||
})
|
||||
})
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
const streamContent = `\`\`\`python
|
||||
# 添加用户消息到 memory
|
||||
agent.memory.add_message(CoreMessage.user_message("帮我查询今天的天气"))
|
||||
|
||||
# 开始执行循环
|
||||
step = 0
|
||||
while step < agent.max_steps:
|
||||
# ========== Think 阶段 ==========
|
||||
should_act = await agent.think()
|
||||
# 调用 LLM 分析用户请求,决定是否需要调用工具
|
||||
|
||||
last_msg = agent.memory.messages[-1]
|
||||
|
||||
# 发送 thinking 事件(如果有)
|
||||
if last_msg.thinking:
|
||||
yield {
|
||||
"type": "thinking",
|
||||
"data": {"content": last_msg.thinking},
|
||||
"task_id": task_id
|
||||
}
|
||||
|
||||
# 如果不需要执行工具,任务完成
|
||||
if not should_act:
|
||||
yield {
|
||||
"type": "message",
|
||||
"data": {"content": last_msg.content, "role": "assistant"},
|
||||
"task_id": task_id
|
||||
}
|
||||
break
|
||||
|
||||
# ========== Act 阶段 ==========
|
||||
# 发送工具调用事件
|
||||
if agent.tool_calls:
|
||||
for tool_call in agent.tool_calls:
|
||||
yield {
|
||||
"type": "tool_call",
|
||||
"data": {
|
||||
"tool_name": tool_call.function.name,
|
||||
"tool_input": tool_call.function.arguments,
|
||||
"tool_call_id": tool_call.id
|
||||
},
|
||||
"task_id": task_id
|
||||
}
|
||||
|
||||
# 执行工具
|
||||
result = await agent.act()
|
||||
|
||||
# 发送工具结果事件
|
||||
yield {
|
||||
"type": "tool_result",
|
||||
"data": {
|
||||
"tool_name": tool_call.function.name,
|
||||
"tool_output": result,
|
||||
"success": True
|
||||
},
|
||||
"task_id": task_id
|
||||
}
|
||||
|
||||
step += 1
|
||||
\`\`\``
|
||||
|
||||
function findFirstCodeBlock(nodes: any[]): any | undefined {
|
||||
const stack: any[] = Array.isArray(nodes) ? [...nodes] : []
|
||||
while (stack.length) {
|
||||
const node = stack.shift()
|
||||
if (!node || typeof node !== 'object')
|
||||
continue
|
||||
if (node.type === 'code_block')
|
||||
return node
|
||||
if (Array.isArray(node.children))
|
||||
stack.unshift(...node.children)
|
||||
if (node.type === 'list' && Array.isArray(node.items)) {
|
||||
for (const item of node.items) {
|
||||
if (Array.isArray(item?.children))
|
||||
stack.unshift(...item.children)
|
||||
}
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
describe('parseMarkdownToStructure - fence regression', () => {
|
||||
it('does not drop content after "<" inside fenced code', () => {
|
||||
const md = getMarkdown('fence-regression')
|
||||
const nodes = parseMarkdownToStructure(streamContent, md)
|
||||
|
||||
const codeNode = findFirstCodeBlock(nodes)
|
||||
expect(codeNode).toBeTruthy()
|
||||
expect(codeNode.loading).toBe(false)
|
||||
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
|
||||
expect(String(codeNode.code)).toContain('tool_call_id')
|
||||
expect(String(codeNode.code)).toContain('step += 1')
|
||||
})
|
||||
|
||||
it('does not drop content in blockquote fenced code', () => {
|
||||
const md = getMarkdown('fence-regression')
|
||||
const markdown = [
|
||||
'> ```python',
|
||||
'> step = 0',
|
||||
'> while step < agent.max_steps:',
|
||||
'> step += 1',
|
||||
'> ```',
|
||||
'',
|
||||
].join('\n')
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const codeNode = findFirstCodeBlock(nodes)
|
||||
expect(codeNode).toBeTruthy()
|
||||
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
|
||||
expect(String(codeNode.code)).toContain('step += 1')
|
||||
})
|
||||
|
||||
it('does not drop content in list-indented fenced code', () => {
|
||||
const md = getMarkdown('fence-regression')
|
||||
const markdown = [
|
||||
'- item',
|
||||
' ',
|
||||
' ```python',
|
||||
' step = 0',
|
||||
' while step < agent.max_steps:',
|
||||
' step += 1',
|
||||
' ```',
|
||||
'',
|
||||
].join('\n')
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const codeNode = findFirstCodeBlock(nodes)
|
||||
expect(codeNode).toBeTruthy()
|
||||
expect(String(codeNode.code)).toContain('while step < agent.max_steps:')
|
||||
expect(String(codeNode.code)).toContain('step += 1')
|
||||
})
|
||||
|
||||
it('still strips dangling html-like tail outside fences', () => {
|
||||
const md = getMarkdown('fence-regression')
|
||||
const nodes = parseMarkdownToStructure('Hello\\n<think', md)
|
||||
expect(JSON.stringify(nodes)).not.toContain('<think')
|
||||
expect(JSON.stringify(nodes)).toContain('Hello')
|
||||
})
|
||||
})
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
<think> **Assessing Translation Needs**\n\n\\n\\n
|
||||
|
||||
I've begun by dissecting the core requirements of this translation task. My focus is on synthesizing the Chinese structural elucidation paragraph into a concise, SCI-paper-ready English version. The expert role helps me tailor the language appropriately, and I'm aware of the final format requirements. Next, I need to understand the nuances of the Chinese text.
|
||||
|
||||
**Evaluating Initial Approaches**
|
||||
|
||||
\\n\\n
|
||||
|
||||
My current thinking revolves around the initial approach to this translation. I'm prioritizing accuracy, which means a deep dive into the source text to extract the core scientific meaning. I plan to use my expertise to identify key terms and phrasing to ensure the English version is clear and meets journal standards. Considering Compound 3 requires specialized knowledge.
|
||||
|
||||
</think>
|
||||
Here is the translation of the structural elucidation for Compound 3, following standard SCI journal formatting and terminology.
|
||||
|
||||
***
|
||||
|
||||
**Compound 3** was obtained as a white powder with an optical rotation of $[\\alpha]^{25}_{\\text{D}}$ –76.7 ($c$ 0.1, MeOH). Its high-resolution electrospray ionization mass spectrometry (HR-ESI-MS) exhibited a deprotonated molecular ion peak at $m/z$ 339.0726 [M–H]$^–$ (calcd for C$_{15}$H$_{15}$O$_9$$^–$, 339.0722), establishing the molecular formula as C$_{15}$H$_{16}$O$_9$, corresponding to eight degrees of unsaturation.
|
||||
|
||||
Analysis of the $^{1}$H and $^{13}$C NMR data (Table 3) revealed characteristic signals at $\\delta_{\\text{H}}$ 7.12 (2H, s, H-2$'/6'$) and $\\delta_{\\text{C}}$ 167.7 (C-7$'$), 146.4 (2C, C-3$'/5'$), 139.9 (C-4$'$), 121.4 (C-1$'$), and 110.4 (2C, C-2$'/6'$), clearly indicating the presence of a galloyl moiety. Additionally, signals at $\\delta_{\\text{H}}$ 3.75 (3H, s, –OCH$_3$) and $\\delta_{\\text{C}}$ 52.5 (–OCH$_3$) confirmed the existence of a methoxy group.
|
||||
|
||||
To precisely elucidate the structure, the NMR data of **3** were compared with those of the known compound 5-$O$-galloyl-(–)-shikimic acid [Reference]. The spectra showed high similarity in both the galloyl moiety and the shikimic acid skeleton, with chemical shifts and coupling patterns matching well with the reference. Notably, the H-5 proton signal of **3** at $\\delta_{\\text{H}}$ 5.75 (1H, br s) was significantly downfield shifted, consistent with the deshielding effect observed in 5-$O$-galloyl-(–)-shikimic acid. This strongly supported the attachment of the galloyl group to the C-5 hydroxyl of the shikimic acid unit via an ester linkage. The primary structural difference between **3** and 5-$O$-galloyl-(–)-shikimic acid was located at C-7. In the HMBC spectrum (Fig. 2), a distinct correlation between the methoxy protons at $\\delta_{\\text{H}}$ 3.75 and the C-7 carbonyl carbon of the shikimate core at $\\delta_{\\text{C}}$ 168.3 confirmed that the C-7 carboxyl group was methylated to form a methyl ester.
|
||||
|
||||
The NOESY spectrum (Fig. 2) exhibited correlations of H-4 ($\\delta_{\\text{H}}$ 4.00) with H-2a ($\\delta_{\\text{H}}$ 2.32) and H-5 ($\\delta_{\\text{H}}$ 5.75), as well as H-3 ($\\delta_{\\text{H}}$ 4.12) with H-2b ($\\delta_{\\text{H}}$ 2.79). These correlations indicated that H-3 is <i>cis</i>-oriented relative to the 4-OH and 5-OH groups. The absolute configuration was further determined as (3$R$, 4$S$, 5$R$) by calculated electronic circular dichroism (ECD) analysis (Fig. 4).
|
||||
|
||||
Based on the comprehensive spectroscopic analysis and comparison with 5-$O$-galloyl-(–)-shikimic acid, the structure of **3** was identified as methyl 5-$O$-galloyl-(–)-shikimate.
|
||||
@@ -0,0 +1,71 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure, processTokens } from '../src'
|
||||
|
||||
function walkNodes(nodes: any[], visit: (node: any) => void) {
|
||||
for (const node of nodes) {
|
||||
if (!node)
|
||||
continue
|
||||
visit(node)
|
||||
if (Array.isArray(node.children))
|
||||
walkNodes(node.children, visit)
|
||||
if (node.type === 'list' && Array.isArray(node.items))
|
||||
walkNodes(node.items, visit)
|
||||
if (node.type === 'table') {
|
||||
if (node.header)
|
||||
walkNodes([node.header], visit)
|
||||
if (Array.isArray(node.rows))
|
||||
walkNodes(node.rows, visit)
|
||||
}
|
||||
if (node.type === 'table_row' && Array.isArray(node.cells))
|
||||
walkNodes(node.cells, visit)
|
||||
}
|
||||
}
|
||||
|
||||
describe('hTML allowlist (方案 A)', () => {
|
||||
it('renders unknown tag <question> as literal text by default', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('Hello <question> world', md, { final: true })
|
||||
const para = nodes[0] as any
|
||||
expect(para.type).toBe('paragraph')
|
||||
const text = (para.children ?? [])
|
||||
.filter((c: any) => c.type === 'text')
|
||||
.map((c: any) => c.content)
|
||||
.join('')
|
||||
expect(text).toBe('Hello <question> world')
|
||||
expect((para.children ?? []).some((c: any) => c.type === 'html_inline')).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps standard HTML tags as html_inline/html_block', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('Hello <span>ok</span>', md, { final: true })
|
||||
const para = nodes[0] as any
|
||||
expect(para.type).toBe('paragraph')
|
||||
expect((para.children ?? []).some((c: any) => c.type === 'html_inline' && c.tag === 'span')).toBe(true)
|
||||
|
||||
const block = parseMarkdownToStructure('<div>ok</div>', md, { final: true })
|
||||
expect(block[0]?.type).toBe('html_block')
|
||||
})
|
||||
|
||||
it('treats tags in customHtmlTags as custom nodes', () => {
|
||||
const md = getMarkdown(undefined, { customHtmlTags: ['thinking'] })
|
||||
const nodes = parseMarkdownToStructure('before <thinking>hi</thinking> after', md, {
|
||||
customHtmlTags: ['thinking'],
|
||||
final: true,
|
||||
})
|
||||
const para = nodes[0] as any
|
||||
expect(para.type).toBe('paragraph')
|
||||
expect((para.children ?? []).some((c: any) => c.type === 'thinking')).toBe(true)
|
||||
})
|
||||
|
||||
it('processTokens also applies the allowlist', () => {
|
||||
const md = getMarkdown()
|
||||
const tokens = md.parse('Hello <question> world', {})
|
||||
const nodes = processTokens(tokens as any, { final: true } as any) as any[]
|
||||
const htmlInlineQuestion: any[] = []
|
||||
walkNodes(nodes, (node) => {
|
||||
if (node.type === 'html_inline' && String(node.tag ?? '').toLowerCase() === 'question')
|
||||
htmlInlineQuestion.push(node)
|
||||
})
|
||||
expect(htmlInlineQuestion.length).toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,361 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function firstHtml(nodes: any[]) {
|
||||
return nodes.find(n => n.type === 'html_block')
|
||||
}
|
||||
|
||||
describe('html_block parser', () => {
|
||||
it('marks unclosed <div> as loading and appends closing tag', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('<div>hello', md)
|
||||
const html = firstHtml(nodes) as any
|
||||
expect(html).toBeDefined()
|
||||
expect(html.type).toBe('html_block')
|
||||
expect(html.tag).toBe('div')
|
||||
expect(html.loading).toBe(true)
|
||||
expect(html.content).toContain('</div>')
|
||||
})
|
||||
|
||||
it('handles uppercase closing tag case-insensitively', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('<DIV>Hi</DIV>', md)
|
||||
const html = firstHtml(nodes) as any
|
||||
expect(html.loading).toBe(false)
|
||||
expect(html.tag).toBe('div')
|
||||
})
|
||||
|
||||
it('treats void tags as closed (no loading, no fake close)', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('<br>', md)
|
||||
const html = firstHtml(nodes) as any
|
||||
expect(html.loading).toBe(false)
|
||||
expect(String(html.content)).not.toContain('</br>')
|
||||
})
|
||||
|
||||
it('treats self-closing first tag as closed', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('<img src="/x.png" />', md)
|
||||
const html = firstHtml(nodes) as any
|
||||
expect(html.loading).toBe(false)
|
||||
expect(String(html.content)).toContain('<img')
|
||||
expect(String(html.content)).not.toContain('</img>')
|
||||
})
|
||||
|
||||
it('does not attempt to close comments/doctypes/PIs', () => {
|
||||
const md = getMarkdown()
|
||||
const a = firstHtml(parseMarkdownToStructure('<!-- comment -->', md)) as any
|
||||
const b = firstHtml(parseMarkdownToStructure('<!DOCTYPE html>', md)) as any
|
||||
const c = firstHtml(parseMarkdownToStructure('<?xml version="1.0"?>', md)) as any
|
||||
expect(a?.loading).toBe(false)
|
||||
expect(b?.loading).toBe(false)
|
||||
expect(c?.loading).toBe(false)
|
||||
})
|
||||
|
||||
it('does not let HTML comments absorb following Markdown blocks', () => {
|
||||
const markdown = `<!-- 基础图片 -->
|
||||

|
||||
|
||||
## 标题
|
||||
|
||||
---
|
||||
|
||||
正文`
|
||||
|
||||
const md = getMarkdown('html-comment-followed-by-markdown')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes.map(node => node.type)).toEqual([
|
||||
'html_block',
|
||||
'paragraph',
|
||||
'heading',
|
||||
'thematic_break',
|
||||
'paragraph',
|
||||
])
|
||||
expect(String(nodes[0]?.raw ?? '').trim()).toBe('<!-- 基础图片 -->')
|
||||
expect(nodes[1]?.children?.[0]?.type).toBe('image')
|
||||
expect(nodes[1]?.children?.[0]?.src).toBe('https://example.com/a.png')
|
||||
expect(nodes[2]?.text).toBe('标题')
|
||||
expect(nodes[4]?.children?.[0]?.content).toBe('正文')
|
||||
})
|
||||
|
||||
it('keeps outer same-tag html blocks loading until the matching outer close arrives', () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure('<div>outer<div>inner</div>rest', md, { final: false })
|
||||
const html = firstHtml(nodes) as any
|
||||
expect(html).toBeDefined()
|
||||
expect(html.type).toBe('html_block')
|
||||
expect(html.tag).toBe('div')
|
||||
expect(html.loading).toBe(true)
|
||||
})
|
||||
|
||||
it('combines details blocks into a single container node with parsed children', () => {
|
||||
const markdown = `<details open>
|
||||
<summary>Thinking...</summary>
|
||||
这里是详情
|
||||
|
||||
首先。。。。
|
||||
</details>
|
||||
`
|
||||
|
||||
const md = getMarkdown('html-block-details-container')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
const html = firstHtml(nodes) as any
|
||||
const summary = html?.children?.[0]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(html?.tag).toBe('details')
|
||||
expect(html?.attrs).toEqual([['open', '']])
|
||||
expect(String(html?.raw ?? '')).toBe(markdown)
|
||||
expect(String(html?.content ?? '')).toContain('<p>首先。。。。</p>')
|
||||
expect(summary?.tag).toBe('summary')
|
||||
expect(html?.children?.[1]?.type).toBe('paragraph')
|
||||
expect(html?.children?.[2]?.type).toBe('paragraph')
|
||||
})
|
||||
|
||||
it('keeps nested details grouped under the outer details container', () => {
|
||||
const markdown = `<details open>
|
||||
<summary>outer</summary>
|
||||
outer start
|
||||
|
||||
<details>
|
||||
<summary>inner</summary>
|
||||
|
||||
inner body
|
||||
</details>
|
||||
|
||||
after inner
|
||||
</details>`
|
||||
|
||||
for (const final of [false, true]) {
|
||||
const md = getMarkdown(`html-block-details-nested-container-${final}`)
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final }) as any[]
|
||||
const outer = firstHtml(nodes) as any
|
||||
const outerSummary = outer?.children?.find((child: any) => child?.type === 'html_block' && child?.tag === 'summary')
|
||||
const inner = outer?.children?.find((child: any) => child?.type === 'html_block' && child?.tag === 'details')
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(outer?.tag).toBe('details')
|
||||
expect(String(outer?.raw ?? '')).toBe(markdown)
|
||||
expect(outerSummary).toBeDefined()
|
||||
expect(inner).toBeDefined()
|
||||
expect(inner?.children?.[0]?.tag).toBe('summary')
|
||||
expect(String(inner?.content ?? '')).toContain('<p>inner body</p>')
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps complex markdown blocks inside details and preserves the closing newline', () => {
|
||||
const markdown = `<details open>
|
||||
<summary>sum</summary>
|
||||
|
||||
- a
|
||||
- b
|
||||
|
||||
> q
|
||||
|
||||
\`\`\`js
|
||||
console.log(1)
|
||||
\`\`\`
|
||||
|
||||
tail
|
||||
</details>
|
||||
`
|
||||
|
||||
const md = getMarkdown('html-block-details-complex')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
const html = firstHtml(nodes) as any
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(html?.tag).toBe('details')
|
||||
expect(String(html?.raw ?? '')).toBe(markdown)
|
||||
expect(String(html?.content ?? '')).toContain('<ul>')
|
||||
expect(String(html?.content ?? '')).toContain('<blockquote>')
|
||||
expect(String(html?.content ?? '')).toContain('data-lang="js"')
|
||||
expect(String(html?.content ?? '')).toContain('<p>tail</p>')
|
||||
expect(html?.children?.map((child: any) => child?.type === 'html_block' ? `${child?.type}:${child?.tag}` : child?.type))
|
||||
.toEqual(['html_block:summary', 'list', 'blockquote', 'code_block', 'paragraph'])
|
||||
})
|
||||
|
||||
it('parses markdown inside summary into summary children', () => {
|
||||
const markdown = `<details open>
|
||||
<summary>**sum** _x_</summary>
|
||||
|
||||
body
|
||||
</details>
|
||||
`
|
||||
|
||||
const md = getMarkdown('html-block-details-summary-markdown')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
const html = firstHtml(nodes) as any
|
||||
const summary = html?.children?.[0]
|
||||
const summaryParagraph = summary?.children?.[0]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(summary?.tag).toBe('summary')
|
||||
expect(html?.attrs).toEqual([['open', '']])
|
||||
expect(summaryParagraph?.children?.[0]?.type).toBe('strong')
|
||||
expect(summaryParagraph?.children?.[0]?.children?.[0]?.content).toBe('sum')
|
||||
expect(summaryParagraph?.children?.[2]?.type).toBe('emphasis')
|
||||
expect(summaryParagraph?.children?.[2]?.children?.[0]?.content).toBe('x')
|
||||
expect(html?.children?.[1]?.children?.[0]?.content).toBe('body')
|
||||
})
|
||||
|
||||
it('keeps opener content before and after summary as structured children', () => {
|
||||
const markdown = `<details open>
|
||||
lead
|
||||
<summary>sum</summary>
|
||||
text
|
||||
<div>inner</div>
|
||||
|
||||
para
|
||||
</details>
|
||||
`
|
||||
|
||||
const md = getMarkdown('html-block-details-opener-mixed')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
const html = firstHtml(nodes) as any
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(html?.attrs).toEqual([['open', '']])
|
||||
expect(html?.children?.map((child: any) => child?.type === 'html_block' ? `${child?.type}:${child?.tag}` : child?.type))
|
||||
.toEqual(['paragraph', 'html_block:summary', 'paragraph', 'html_block:div', 'paragraph'])
|
||||
expect(html?.children?.[0]?.children?.[0]?.content).toBe('lead')
|
||||
expect(html?.children?.[1]?.children?.[0]?.children?.[0]?.content).toBe('sum')
|
||||
expect(html?.children?.[2]?.children?.[0]?.content).toBe('text')
|
||||
expect(html?.children?.[3]?.tag).toBe('div')
|
||||
expect(html?.children?.[4]?.children?.[0]?.content).toBe('para')
|
||||
})
|
||||
|
||||
it('does not absorb content after a self-contained details block', () => {
|
||||
const markdown = `<details style="color:gray;background-color: #f8f8f8;padding: 8px;border-radius: 4px;" open> <summary> Thinking... </summary> 好的,我现在需要处理用户的问题:"你是谁"。首先,我要按照用户设定的规则来回答。用户要求先检查提供的上下文(context)中的信息,如果在里面找不到答案,再用 \n</details> \n\n我是通义千问,阿里巴巴集团旗下的超大规模语言模型。我能够帮助你回答问题、创作文字(如写故事、写公文、写邮件、写剧本等)、进行逻辑推理、编程等任务,并支持多语言交流。\n`
|
||||
|
||||
const md = getMarkdown('html-block-details-self-contained')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect(nodes[1]?.type).toBe('paragraph')
|
||||
})
|
||||
|
||||
it('self-contained details with no trailing content produces a single node', () => {
|
||||
const markdown = `<details open> <summary>S</summary> body \n</details>\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-no-trailing')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect((nodes[0] as any)?.loading).toBe(false)
|
||||
})
|
||||
|
||||
it('self-contained details followed by a heading keeps them separate', () => {
|
||||
const markdown = `<details open> <summary>S</summary> body \n</details>\n\n# Heading\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-heading')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect(nodes[1]?.type).toBe('heading')
|
||||
})
|
||||
|
||||
it('two consecutive self-contained details blocks stay separate', () => {
|
||||
const markdown = `<details open> <summary>A</summary> aaa \n</details>\n\n<details> <summary>B</summary> bbb \n</details>\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-consecutive')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect(nodes[1]?.type).toBe('html_block')
|
||||
expect(nodes[1]?.tag).toBe('details')
|
||||
})
|
||||
|
||||
it('self-contained details followed by a multi-line details block stay separate', () => {
|
||||
const markdown = `<details open> <summary>A</summary> aaa \n</details>\n\n<details>\n<summary>B</summary>\n\nbbb\n</details>\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-then-multiline')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect(nodes[1]?.type).toBe('html_block')
|
||||
expect(nodes[1]?.tag).toBe('details')
|
||||
expect(String(nodes[1]?.content ?? '')).toContain('<p>bbb</p>')
|
||||
})
|
||||
|
||||
it('self-contained details followed by a code block stay separate', () => {
|
||||
const markdown = '<details open> <summary>S</summary> body \n</details>\n\n```js\nconsole.log(1)\n```\n'
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-code')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect(nodes[1]?.type).toBe('code_block')
|
||||
})
|
||||
|
||||
it('self-contained details in streaming mode does not absorb following content', () => {
|
||||
const markdown = `<details open> <summary>S</summary> body \n</details> \n\n我是通义千问\n`
|
||||
|
||||
for (const final of [false, true]) {
|
||||
const md = getMarkdown(`html-block-self-contained-stream-${final}`)
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
expect((nodes[0] as any)?.loading).toBe(false)
|
||||
expect(nodes[1]?.type).toBe('paragraph')
|
||||
}
|
||||
})
|
||||
|
||||
it('self-contained details preserves attributes correctly', () => {
|
||||
const markdown = `<details style="background:#f8f8f8" open> <summary>S</summary> body \n</details>\n\npara\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-attrs')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
const details = nodes[0] as any
|
||||
expect(details?.tag).toBe('details')
|
||||
expect(details?.attrs).toEqual([['style', 'background:#f8f8f8'], ['open', '']])
|
||||
expect(details?.children?.find((c: any) => c?.tag === 'summary')).toBeDefined()
|
||||
expect(nodes[1]?.type).toBe('paragraph')
|
||||
})
|
||||
|
||||
it('self-contained details preserves summary children and body text', () => {
|
||||
const markdown = `<details open> <summary> Thinking... </summary> 好的 \n</details> \n\n我是通义千问\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-children')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(2)
|
||||
const details = nodes[0] as any
|
||||
expect(details?.tag).toBe('details')
|
||||
expect(details?.loading).toBe(false)
|
||||
const summary = details?.children?.find((c: any) => c?.tag === 'summary')
|
||||
expect(summary).toBeDefined()
|
||||
expect(nodes[1]?.type).toBe('paragraph')
|
||||
})
|
||||
|
||||
it('self-contained details followed by multiple paragraphs stays separate from all', () => {
|
||||
const markdown = `<details open> <summary>S</summary> body \n</details> \n\npara1\n\npara2\n`
|
||||
|
||||
const md = getMarkdown('html-block-self-contained-multi-para')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes.length).toBeGreaterThanOrEqual(2)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('details')
|
||||
for (let k = 1; k < nodes.length; k++) {
|
||||
expect(nodes[k]?.type).not.toBe('html_block')
|
||||
}
|
||||
})
|
||||
})
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('html inline malformed custom-tag regression', () => {
|
||||
it('keeps malformed "<robot=...>" as literal text instead of dropping children', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = '<robot=翻转检测>尘盒搞定!'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
const para = nodes[0] as any
|
||||
|
||||
expect(para.type).toBe('paragraph')
|
||||
expect(Array.isArray(para.children)).toBe(true)
|
||||
expect(para.children.length).toBeGreaterThan(0)
|
||||
|
||||
const text = (para.children ?? [])
|
||||
.filter((c: any) => c.type === 'text')
|
||||
.map((c: any) => String(c.content ?? ''))
|
||||
.join('')
|
||||
|
||||
expect(text).toContain('<robot=翻转检测>')
|
||||
expect(text).toContain('尘盒搞定!')
|
||||
})
|
||||
|
||||
it('keeps valid custom-tag attributes working (e.g. <robot xxx="sa">)', () => {
|
||||
const md = getMarkdown('robot-attrs', { customHtmlTags: ['robot'] })
|
||||
const markdown = '<robot xxx="sa">维护中</robot>'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, {
|
||||
customHtmlTags: ['robot'],
|
||||
final: true,
|
||||
})
|
||||
const para = nodes[0] as any
|
||||
const robot = (para.children ?? []).find((c: any) => c.type === 'robot')
|
||||
|
||||
expect(robot).toBeDefined()
|
||||
expect(robot.attrs).toContainEqual(['xxx', 'sa'])
|
||||
expect(robot.children?.[0]?.type).toBe('text')
|
||||
expect(String(robot.children?.[0]?.content ?? '')).toContain('维护中')
|
||||
})
|
||||
|
||||
it('does not affect normal html_inline parsing for standard tags', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = 'before <span data-x="1">ok</span> after'
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true })
|
||||
const para = nodes[0] as any
|
||||
const spanNode = (para.children ?? []).find((c: any) => c.type === 'html_inline' && c.tag === 'span')
|
||||
|
||||
expect(spanNode).toBeDefined()
|
||||
expect(spanNode.children?.[0]?.type).toBe('text')
|
||||
expect(spanNode.children?.[0]?.content).toBe('ok')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,43 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
// Focus tests on the core plugin behavior added in fixHtmlInline
|
||||
|
||||
describe('fixHtmlInline core plugin', () => {
|
||||
it('expands single-token inline <a> with inner text', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `<a href="https://example.com">Here</a>`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
// nodes[0] should be paragraph with a link child
|
||||
const para = nodes[0] as any
|
||||
expect(para.type).toBe('paragraph')
|
||||
const link = para.children.find((c: any) => c.type === 'link')
|
||||
expect(link).toBeDefined()
|
||||
expect(link.href).toBe('https://example.com')
|
||||
expect(link.children?.[0]?.content).toBe('Here')
|
||||
})
|
||||
|
||||
it('handles attribute values containing ">" inside quotes when splitting open tag', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `<a href="https://example.com?q=a>b&x=1">Here</a>`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const para = nodes[0] as any
|
||||
const link = para.children.find((c: any) => c.type === 'link')
|
||||
expect(link).toBeDefined()
|
||||
expect(link.href).toBe('https://example.com?q=a>b&x=1')
|
||||
expect(link.children?.[0]?.content).toBe('Here')
|
||||
})
|
||||
|
||||
it('treats void/self-closing tag inline as single html chunk (fallback to inline_code)', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `before <img src="/x.png" alt="pic" /> after`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const para = nodes[0] as any
|
||||
expect(para.type).toBe('paragraph')
|
||||
const htmlInline = para.children.find((c: any) => c.type === 'html_inline')
|
||||
expect(htmlInline).toBeDefined()
|
||||
expect(htmlInline.content).toContain('<img')
|
||||
// Ensure no weird closing tag was synthesized in output
|
||||
expect(String(htmlInline.content)).not.toContain('</img>')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('html_inline parsing', () => {
|
||||
it('parses <a> as link node with href and inner text', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `This is a <a href="https://example.com">Example</a> link.`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
expect(nodes.length).toBeGreaterThan(0)
|
||||
const para = nodes[0]
|
||||
expect(para.type).toBe('paragraph')
|
||||
const linkChild = (para as any).children.find((c: any) => c.type === 'link')
|
||||
expect(linkChild).toBeDefined()
|
||||
expect(linkChild.href).toBe('https://example.com')
|
||||
expect(linkChild.children[0].content).toBe('Example')
|
||||
})
|
||||
|
||||
it('preserves custom <a> attributes on link node attrs', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `<a href="https://example.com" target="_self" rel="nofollow" data-track="cta">Example</a>`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const para = nodes[0] as any
|
||||
const linkChild = para.children.find((c: any) => c.type === 'link')
|
||||
|
||||
expect(linkChild).toBeDefined()
|
||||
expect(linkChild.href).toBe('https://example.com')
|
||||
expect(linkChild.attrs).toEqual(expect.arrayContaining([
|
||||
['href', 'https://example.com'],
|
||||
['target', '_self'],
|
||||
['rel', 'nofollow'],
|
||||
['data-track', 'cta'],
|
||||
]))
|
||||
})
|
||||
|
||||
it('creates html_inline nodes for generic inline html like <span>', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `Before <span>inner span</span> After`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const para = nodes[0]
|
||||
const spanNode = (para as any).children.find((c: any) => c.type === 'html_inline')
|
||||
expect(spanNode).toBeDefined()
|
||||
expect(spanNode.content).toContain('<span')
|
||||
expect(spanNode.content).toContain('</span>')
|
||||
expect(spanNode.children?.[0]?.content).toBe('inner span')
|
||||
})
|
||||
|
||||
it('handles attribute values that include ">" characters', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `Value <a href="https://example.com?q=a>b&x=1">Here</a> end`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const para = nodes[0]
|
||||
const linkChild = (para as any).children.find((c: any) => c.type === 'link')
|
||||
expect(linkChild).toBeDefined()
|
||||
// href extraction should capture full quoted value including '>'
|
||||
expect(linkChild.href).toBe('https://example.com?q=a>b&x=1')
|
||||
})
|
||||
|
||||
it('handles missing closing tag gracefully (unclosed <a>)', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = `Start <a href="https://example.com">Unclosed text`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
// Should not throw; ensure raw input is preserved in some node
|
||||
const serialized = JSON.stringify(nodes)
|
||||
// JSON will escape quotes, so check for a less strict substring and for href URL
|
||||
expect(serialized).toContain('<a href')
|
||||
expect(serialized).toContain('https://example.com')
|
||||
expect(serialized).toContain('Unclosed text')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,127 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('parser inline code multiple spans', () => {
|
||||
it('parses multiple inline code spans and emphasis correctly', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = '1. **整合冗余条款:** 将原`1-(5)`、`3-(3)`、`3-(4)`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入`1-(4)`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
const serialized = JSON.stringify(nodes)
|
||||
// Ensure we produced a strong node
|
||||
expect(serialized).toContain('"type":"strong"')
|
||||
|
||||
// Also ensure code spans exist and expected snippets appear
|
||||
expect(serialized).toContain('1-(5)')
|
||||
expect(serialized).toContain('3-(3)')
|
||||
expect(serialized).toContain('3-(4)')
|
||||
expect(serialized).toContain('1-(4)')
|
||||
})
|
||||
|
||||
it('handles mid-state with trailing opening backtick in list item', () => {
|
||||
const md = getMarkdown()
|
||||
// mid-state: trailing opening backtick after a comma
|
||||
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`'
|
||||
const nodesMid = parseMarkdownToStructure(markdownMid, md)
|
||||
const serializedMid = JSON.stringify(nodesMid)
|
||||
// debug: dump nodes for diagnostic
|
||||
|
||||
console.log('DEBUG_NODES_MID_FULL:', JSON.stringify(nodesMid, null, 2))
|
||||
// debug output removed
|
||||
|
||||
// Should contain a strong node for the leading bold and an inline_code mid-state
|
||||
expect(serializedMid).toContain('"type":"strong"')
|
||||
// mid-state inline code: there may be an inline_code node with empty code or raw ending in backtick
|
||||
expect(serializedMid.includes('inline_code') || serializedMid.includes('`')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('handles mid-state with trailing opening backtick in list item mid-state 1', () => {
|
||||
const md = getMarkdown()
|
||||
// mid-state: trailing opening backtick after a comma
|
||||
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`3'
|
||||
const nodesMid = parseMarkdownToStructure(markdownMid, md)
|
||||
const children = nodesMid[0].items[0].children[0].children
|
||||
// Should contain a strong node for the leading bold and an inline_code mid-state
|
||||
expect(children[0].type).toBe('strong')
|
||||
expect(children[0].raw).toBe('**整合冗余条款:**')
|
||||
expect(children[1].type).toBe('text')
|
||||
expect(children[1].content).toBe(' 将原')
|
||||
expect(children[2].type).toBe('inline_code')
|
||||
expect(children[2].code).toBe('1-(5)')
|
||||
// The punctuation between code spans may be parsed as text or as a tiny
|
||||
// inline_code node in mid-state scenarios depending on token boundaries.
|
||||
// Accept either a text node or an inline_code node for the punctuation
|
||||
expect(['text', 'inline_code']).toContain(children[3]?.type)
|
||||
// Ensure at least one child contains the punctuation '、' either as content or as inline code
|
||||
const hasPunctuation = children.some((c: any) => (c.content === '、') || (c.code === '、'))
|
||||
expect(hasPunctuation).toBeTruthy()
|
||||
})
|
||||
|
||||
it('handles mid-state with trailing opening backtick in list item mid-state 2', () => {
|
||||
const md = getMarkdown()
|
||||
// mid-state: trailing opening backtick after a comma
|
||||
const markdownMid = '1. **整合冗余条款:** 将原`1-(5)`、`3-(3)`、`3-(4)`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入`1-'
|
||||
const nodesMid = parseMarkdownToStructure(markdownMid, md)
|
||||
const children = nodesMid[0].items[0].children[0].children
|
||||
// Should contain a strong node for the leading bold and an inline_code mid-state
|
||||
expect(children[0].type).toBe('strong')
|
||||
expect(children[0].raw).toBe('**整合冗余条款:**')
|
||||
expect(children[1].type).toBe('text')
|
||||
expect(children[1].content).toBe(' 将原')
|
||||
expect(children[2].type).toBe('inline_code')
|
||||
expect(children[2].code).toBe('1-(5)')
|
||||
expect(children[3].type).toBe('text')
|
||||
expect(children[3].content).toBe('、')
|
||||
expect(children[4].type).toBe('inline_code')
|
||||
expect(children[4].code).toBe('3-(3)')
|
||||
expect(children[5].type).toBe('text')
|
||||
expect(children[5].content).toBe('、')
|
||||
expect(children[6].type).toBe('inline_code')
|
||||
expect(children[6].code).toBe('3-(4)')
|
||||
expect(children[7].type).toBe('text')
|
||||
expect(children[7].content).toBe('中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入')
|
||||
expect(children[8].type).toBe('inline_code')
|
||||
expect(children[8].code).toBe('1-')
|
||||
})
|
||||
|
||||
it('handles mid-state with trailing opening backtick in list item mid-state 3', () => {
|
||||
const md = getMarkdown()
|
||||
// mid-state: trailing opening backtick after a comma
|
||||
const markdownMid = '1. **整合冗余条款:** 将原\`1-(5)\`、\`3-(3)\`、\`3-(4)\`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入\`1-(4)\`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
|
||||
const nodesMid = parseMarkdownToStructure(markdownMid, md)
|
||||
const children = nodesMid[0].items[0].children[0].children
|
||||
// Should contain a strong node for the leading bold and an inline_code mid-state
|
||||
expect(children[0].type).toBe('strong')
|
||||
expect(children[0].raw).toBe('**整合冗余条款:**')
|
||||
expect(children[1].type).toBe('text')
|
||||
expect(children[1].content).toBe(' 将原')
|
||||
expect(children[2].type).toBe('inline_code')
|
||||
expect(children[2].code).toBe('1-(5)')
|
||||
expect(children[3].type).toBe('text')
|
||||
expect(children[3].content).toBe('、')
|
||||
expect(children[4].type).toBe('inline_code')
|
||||
expect(children[4].code).toBe('3-(3)')
|
||||
expect(children[5].type).toBe('text')
|
||||
expect(children[5].content).toBe('、')
|
||||
expect(children[6].type).toBe('inline_code')
|
||||
expect(children[6].code).toBe('3-(4)')
|
||||
expect(children[7].type).toBe('text')
|
||||
expect(children[7].content).toBe('中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入')
|
||||
expect(children[8].type).toBe('inline_code')
|
||||
expect(children[8].code).toBe('1-(4)')
|
||||
expect(children[9].type).toBe('text')
|
||||
expect(children[9].content).toBe('“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。')
|
||||
})
|
||||
|
||||
it('handles mid-state with trailing opening backtick in list item mid-state 4', () => {
|
||||
const md = getMarkdown()
|
||||
// mid-state: trailing opening backtick after a comma
|
||||
const markdownMid = '1. **整合冗余条款:** 将原 **hello world:** \`1-(5)\`、\`3-(3)\`、\`3-(4)\`中关于“缴纳费用”和“告知法律状态变化”的核心义务合并,统一放入\`1-(4)\`“法律状态维护与通知义务”中,使其逻辑更集中,避免了在不同部分重复提及相似义务。'
|
||||
const nodesMid = parseMarkdownToStructure(markdownMid, md)
|
||||
const children = nodesMid[0].items[0].children[0].children
|
||||
// Should contain a strong node for the leading bold and an inline_code mid-state
|
||||
const types = children.map((c: any) => c.type)
|
||||
expect(types).toEqual(['strong', 'text', 'strong', 'text', 'inline_code', 'text', 'inline_code', 'text', 'inline_code', 'text', 'inline_code', 'text'])
|
||||
})
|
||||
})
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function parseParagraphChildren(markdown: string) {
|
||||
const md = getMarkdown('issue-334')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph?.type).toBe('paragraph')
|
||||
return paragraph?.children ?? []
|
||||
}
|
||||
|
||||
function parseCustomComponentChildren(markdownContent: string) {
|
||||
const tags = ['thinking']
|
||||
const md = getMarkdown('issue-334-custom', { customHtmlTags: tags })
|
||||
const nodes = parseMarkdownToStructure(`<thinking>${markdownContent}</thinking>`, md, {
|
||||
customHtmlTags: tags,
|
||||
final: true,
|
||||
}) as any[]
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph?.type).toBe('paragraph')
|
||||
const thinking = (paragraph?.children ?? []).find((node: any) => node?.type === 'thinking')
|
||||
expect(thinking).toBeDefined()
|
||||
return thinking?.children ?? []
|
||||
}
|
||||
|
||||
function collectTypes(children: any[]) {
|
||||
return children.map((node: any) => node?.type)
|
||||
}
|
||||
|
||||
function collectByType(value: any, targetType: string): any[] {
|
||||
const out: any[] = []
|
||||
const walk = (node: any) => {
|
||||
if (!node)
|
||||
return
|
||||
if (Array.isArray(node)) {
|
||||
node.forEach(walk)
|
||||
return
|
||||
}
|
||||
if (node?.type === targetType)
|
||||
out.push(node)
|
||||
if (Array.isArray(node?.children))
|
||||
node.children.forEach(walk)
|
||||
if (Array.isArray(node?.items))
|
||||
node.items.forEach(walk)
|
||||
}
|
||||
walk(value)
|
||||
return out
|
||||
}
|
||||
|
||||
describe('issue #334 - inline formatting before math', () => {
|
||||
it('keeps strong, emphasis, and strikethrough nodes when inline math is on the same line', () => {
|
||||
const children = parseParagraphChildren('**粗体** 和 *斜体* 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'strong',
|
||||
'text',
|
||||
'emphasis',
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
|
||||
expect(children[0]?.children?.[0]?.content).toBe('粗体')
|
||||
expect(children[2]?.children?.[0]?.content).toBe('斜体')
|
||||
expect(children[4]?.children?.[0]?.content).toBe('删除线')
|
||||
expect(children[5]?.content).toBe(' 和 ')
|
||||
expect(children[6]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps underscore-based strong and emphasis nodes when inline math is on the same line', () => {
|
||||
const children = parseParagraphChildren('__粗体__ 和 _斜体_ 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'strong',
|
||||
'text',
|
||||
'emphasis',
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
|
||||
expect(children[0]?.children?.[0]?.content).toBe('粗体')
|
||||
expect(children[2]?.children?.[0]?.content).toBe('斜体')
|
||||
expect(children[4]?.children?.[0]?.content).toBe('删除线')
|
||||
expect(children[6]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps nested triple-asterisk formatting before inline math', () => {
|
||||
const children = parseParagraphChildren('***粗斜体*** 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
|
||||
const outer = children[0]
|
||||
const inner = outer?.children?.[0]
|
||||
expect(['strong', 'emphasis']).toContain(outer?.type)
|
||||
expect(['strong', 'emphasis']).toContain(inner?.type)
|
||||
expect(inner?.type).not.toBe(outer?.type)
|
||||
expect(inner?.children?.[0]?.content).toBe('粗斜体')
|
||||
|
||||
expect(children.slice(1).map((node: any) => node?.type)).toEqual([
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
expect(children[2]?.children?.[0]?.content).toBe('删除线')
|
||||
expect(children[4]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps formatting order when inline math appears before later emphasis markers', () => {
|
||||
const children = parseParagraphChildren('$E = mc^2$ 和 **粗体** 和 *斜体* 和 ~~删除线~~')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'math_inline',
|
||||
'text',
|
||||
'strong',
|
||||
'text',
|
||||
'emphasis',
|
||||
'text',
|
||||
'strikethrough',
|
||||
])
|
||||
|
||||
expect(children[0]?.content).toBe('E = mc^2')
|
||||
expect(children[2]?.children?.[0]?.content).toBe('粗体')
|
||||
expect(children[4]?.children?.[0]?.content).toBe('斜体')
|
||||
expect(children[6]?.children?.[0]?.content).toBe('删除线')
|
||||
})
|
||||
|
||||
it('does not turn escaped delimiters into formatting nodes before inline math', () => {
|
||||
const children = parseParagraphChildren('\\*\\*字面量\\*\\* 和 \\_\\_下划线\\_\\_ 和 $E = mc^2$')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
expect(children[0]?.content).toBe('**字面量** 和 __下划线__ 和 ')
|
||||
expect(children[1]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps escaped delimiters literal while still parsing later real formatting before inline math', () => {
|
||||
const children = parseParagraphChildren('\\*\\*字面量\\*\\* 和 **真实粗体** 和 $E = mc^2$')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'text',
|
||||
'strong',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
expect(children[0]?.content).toBe('**字面量** 和 ')
|
||||
expect(children[1]?.children?.[0]?.content).toBe('真实粗体')
|
||||
expect(children[2]?.content).toBe(' 和 ')
|
||||
expect(children[3]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps math recoverable even when malformed nested formatting appears before it', () => {
|
||||
const children = parseParagraphChildren('**粗体 *未闭合 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
|
||||
const mathNodes = collectByType(children, 'math_inline')
|
||||
expect(mathNodes).toHaveLength(1)
|
||||
expect(mathNodes[0]?.content).toBe('E = mc^2')
|
||||
|
||||
const serialized = JSON.stringify(children)
|
||||
expect(serialized).toContain('未闭合')
|
||||
expect(serialized).toContain('删除线')
|
||||
})
|
||||
})
|
||||
|
||||
describe('issue #334 - inline formatting inside custom components', () => {
|
||||
it('keeps strong, emphasis, strikethrough, and math inside custom component content', () => {
|
||||
const children = parseCustomComponentChildren('**粗体** 和 *斜体* 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'strong',
|
||||
'text',
|
||||
'emphasis',
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
expect(children[0]?.children?.[0]?.content).toBe('粗体')
|
||||
expect(children[2]?.children?.[0]?.content).toBe('斜体')
|
||||
expect(children[4]?.children?.[0]?.content).toBe('删除线')
|
||||
expect(children[6]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps underscore-based and triple-asterisk formatting inside custom component content', () => {
|
||||
const underscoreChildren = parseCustomComponentChildren('__粗体__ 和 _斜体_ 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
expect(collectTypes(underscoreChildren)).toEqual([
|
||||
'strong',
|
||||
'text',
|
||||
'emphasis',
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
|
||||
const tripleChildren = parseCustomComponentChildren('***粗斜体*** 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
const outer = tripleChildren[0]
|
||||
const inner = outer?.children?.[0]
|
||||
expect(['strong', 'emphasis']).toContain(outer?.type)
|
||||
expect(['strong', 'emphasis']).toContain(inner?.type)
|
||||
expect(inner?.type).not.toBe(outer?.type)
|
||||
expect(inner?.children?.[0]?.content).toBe('粗斜体')
|
||||
expect(collectTypes(tripleChildren.slice(1))).toEqual([
|
||||
'text',
|
||||
'strikethrough',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps escaped delimiters literal while still parsing later real formatting inside custom component content', () => {
|
||||
const children = parseCustomComponentChildren('\\*\\*字面量\\*\\* 和 **真实粗体** 和 $E = mc^2$')
|
||||
|
||||
expect(collectTypes(children)).toEqual([
|
||||
'text',
|
||||
'strong',
|
||||
'text',
|
||||
'math_inline',
|
||||
])
|
||||
expect(children[0]?.content).toBe('**字面量** 和 ')
|
||||
expect(children[1]?.children?.[0]?.content).toBe('真实粗体')
|
||||
expect(children[3]?.content).toBe('E = mc^2')
|
||||
})
|
||||
|
||||
it('keeps math recoverable with malformed formatting inside custom component content', () => {
|
||||
const children = parseCustomComponentChildren('**粗体 *未闭合 和 ~~删除线~~ 和 $E = mc^2$')
|
||||
|
||||
const mathNodes = collectByType(children, 'math_inline')
|
||||
expect(mathNodes).toHaveLength(1)
|
||||
expect(mathNodes[0]?.content).toBe('E = mc^2')
|
||||
|
||||
const serialized = JSON.stringify(children)
|
||||
expect(serialized).toContain('未闭合')
|
||||
expect(serialized).toContain('删除线')
|
||||
})
|
||||
})
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('issue 380 html wrapper markdown regression', () => {
|
||||
it('keeps markdown nested in a span wrapper as structured children without leaking duplicate top-level nodes', () => {
|
||||
const markdown = `<span style="font-size: 12px;">
|
||||
|
||||
🗺️【环境状态】
|
||||
- 地点:石溪村,李东的茅屋
|
||||
- 时间:4/12 周四 上午07:00
|
||||
- 环境:阳光明媚,空气清新,茅屋内简陋但整洁,远处山峦笼罩在薄雾中,偶尔传来鸟鸣声
|
||||
|
||||
📊【角色状态】
|
||||
- 生理:健康(轻微饥饿)
|
||||
- 物品:锄头(自家)、简易弓箭(自家)、5个铜币(自家)、一些干粮(自家)
|
||||
- 技能:农耕(基础)、陷阱设置(基础)、生存知识(基础)
|
||||
- 关系:与顾客关系良好(+)
|
||||
|
||||
📖【目标与线索】
|
||||
- 已知线索:无
|
||||
- 短期目标:无
|
||||
- 长期目标:无
|
||||
***
|
||||
|
||||
|
||||
🎯【选项】
|
||||
1. 去田里劳作,争取多收成些粮食卖钱
|
||||
2. 上山检查之前设置的陷阱,看有没有捕到猎
|
||||
3. 直接去清风镇看看情况,虽然路途遥远
|
||||
4. ……
|
||||
|
||||
</span>`
|
||||
|
||||
const md = getMarkdown('issue-380-structured')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('span')
|
||||
expect(nodes[0]?.attrs).toEqual([['style', 'font-size: 12px;']])
|
||||
expect(nodes[0]?.raw).toBe(markdown)
|
||||
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual([
|
||||
'paragraph',
|
||||
'list',
|
||||
'paragraph',
|
||||
'list',
|
||||
'paragraph',
|
||||
'list',
|
||||
'thematic_break',
|
||||
'paragraph',
|
||||
'list',
|
||||
])
|
||||
expect(nodes[0]?.children?.[0]?.children?.[0]?.content).toBe('🗺️【环境状态】')
|
||||
expect(nodes[0]?.children?.[1]?.items).toHaveLength(3)
|
||||
expect(nodes[0]?.children?.[8]?.ordered).toBe(true)
|
||||
expect(nodes[0]?.children?.[8]?.items).toHaveLength(4)
|
||||
})
|
||||
|
||||
it('structures simple block markdown inside span and removes leaked list duplicates', () => {
|
||||
const markdown = `<span>
|
||||
|
||||
- a
|
||||
- b
|
||||
|
||||
</span>`
|
||||
const md = getMarkdown('issue-380-simple')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('span')
|
||||
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
|
||||
expect(nodes[0]?.children?.[0]?.items).toHaveLength(2)
|
||||
expect(nodes[0]?.children?.[0]?.items?.[0]?.children?.[0]?.children?.[0]?.content).toBe('a')
|
||||
expect(nodes[0]?.children?.[0]?.items?.[1]?.children?.[0]?.children?.[0]?.content).toBe('b')
|
||||
})
|
||||
|
||||
it('keeps pure html wrappers unstructured when they do not actually contain markdown blocks', () => {
|
||||
const markdown = '<div><strong>Block HTML</strong></div>'
|
||||
const md = getMarkdown('issue-380-pure-html')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('div')
|
||||
expect(nodes[0]?.children).toBeUndefined()
|
||||
})
|
||||
|
||||
it('does not structure blocked tags even when their inner content looks like markdown blocks', () => {
|
||||
const markdown = `<script>
|
||||
|
||||
- a
|
||||
- b
|
||||
|
||||
</script>`
|
||||
const md = getMarkdown('issue-380-blocked-tag')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('script')
|
||||
expect(nodes[0]?.children).toBeUndefined()
|
||||
expect(nodes[0]?.raw).toBe(markdown)
|
||||
})
|
||||
|
||||
it('keeps literal-content tags unstructured even when their inner content looks like markdown blocks', () => {
|
||||
const samples = ['pre', 'style', 'textarea'] as const
|
||||
|
||||
for (const tag of samples) {
|
||||
const markdown = `<${tag}>
|
||||
|
||||
- a
|
||||
- b
|
||||
|
||||
</${tag}>`
|
||||
const md = getMarkdown(`issue-380-literal-${tag}`)
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe(tag)
|
||||
expect(nodes[0]?.children).toBeUndefined()
|
||||
expect(nodes[0]?.raw).toBe(markdown)
|
||||
}
|
||||
})
|
||||
|
||||
it('structures nested html wrappers when the inner wrapper already contains markdown blocks', () => {
|
||||
const markdown = `<div>
|
||||
<div>
|
||||
|
||||
- a
|
||||
|
||||
</div>
|
||||
</div>`
|
||||
const md = getMarkdown('issue-380-nested')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('div')
|
||||
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['html_block'])
|
||||
expect(nodes[0]?.children?.[0]?.tag).toBe('div')
|
||||
expect(nodes[0]?.children?.[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
|
||||
expect(nodes[0]?.children?.[0]?.children?.[0]?.items).toHaveLength(1)
|
||||
expect(nodes[0]?.children?.[0]?.children?.[0]?.items?.[0]?.children?.[0]?.children?.[0]?.content).toBe('a')
|
||||
})
|
||||
|
||||
it('keeps streaming span wrappers stable before the closing tag arrives', () => {
|
||||
const markdown = `<span>
|
||||
|
||||
- a
|
||||
- b`
|
||||
const md = getMarkdown('issue-380-streaming')
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
expect(nodes[0]?.type).toBe('html_block')
|
||||
expect(nodes[0]?.tag).toBe('span')
|
||||
expect(nodes[0]?.loading).toBe(true)
|
||||
expect(nodes[0]?.children?.map((child: any) => child?.type)).toEqual(['list'])
|
||||
expect(nodes[0]?.children?.[0]?.items).toHaveLength(2)
|
||||
})
|
||||
})
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
const REAL_WORLD_MULTILINE_INPUT = `$2.897771955 times 10^{-3}text{m·K}$^[1]^
|
||||
测试<sup>[3]</sup>。
|
||||
$x$^[1]^
|
||||
$x$ ^[1]^
|
||||
测试^[1]^
|
||||
$2.897771955 \\times 10^{-3}\\text{m·K}$^[1]^
|
||||
<sup>[1]</sup>
|
||||
测试<sup>[12]</sup>结束
|
||||
A<sup>[3]</sup>B
|
||||
$x$^[1]^
|
||||
测试^[1]^
|
||||
<sup>[3]</sup>
|
||||
测试<sup>[12]</sup>结束`
|
||||
|
||||
function parse(markdown: string, final = true) {
|
||||
const md = getMarkdown('issue-386')
|
||||
return parseMarkdownToStructure(markdown, md, { final }) as any[]
|
||||
}
|
||||
|
||||
function collectByType(value: any, targetType: string): any[] {
|
||||
const out: any[] = []
|
||||
const walk = (node: any) => {
|
||||
if (!node)
|
||||
return
|
||||
if (Array.isArray(node)) {
|
||||
node.forEach(walk)
|
||||
return
|
||||
}
|
||||
if (node.type === targetType)
|
||||
out.push(node)
|
||||
if (Array.isArray(node.children))
|
||||
node.children.forEach(walk)
|
||||
if (Array.isArray(node.items))
|
||||
node.items.forEach(walk)
|
||||
}
|
||||
walk(value)
|
||||
return out
|
||||
}
|
||||
|
||||
describe('issue #386 inline math suffix regressions', () => {
|
||||
it.each([true, false])('keeps superscript syntax immediately after inline math when final=%s', (final) => {
|
||||
const nodes = parse('$2.897771955 times 10^{-3}text{m·K}$^[1]^', final)
|
||||
const paragraph = nodes[0] as any
|
||||
|
||||
expect(paragraph?.type).toBe('paragraph')
|
||||
expect(paragraph?.children?.map((child: any) => child.type)).toEqual([
|
||||
'math_inline',
|
||||
'superscript',
|
||||
])
|
||||
expect(paragraph?.children?.[1]?.children?.map((child: any) => child.content).join('')).toBe('[1]')
|
||||
|
||||
const textNodes = collectByType(paragraph, 'text')
|
||||
expect(textNodes.some((node: any) => node.content === '^[1]^')).toBe(false)
|
||||
})
|
||||
|
||||
it.each([true, false])('keeps footnote references immediately after inline math when final=%s', (final) => {
|
||||
const nodes = parse('$x$[^1]\n\n[^1]: note', final)
|
||||
const paragraph = nodes[0] as any
|
||||
|
||||
expect(paragraph?.type).toBe('paragraph')
|
||||
expect(paragraph?.children?.map((child: any) => child.type)).toEqual([
|
||||
'math_inline',
|
||||
'footnote_reference',
|
||||
])
|
||||
expect(paragraph?.children?.[1]?.id).toBe('1')
|
||||
|
||||
const footnotes = collectByType(nodes, 'footnote')
|
||||
expect(footnotes).toHaveLength(1)
|
||||
expect(footnotes[0]?.id).toBe('1')
|
||||
expect(collectByType(footnotes[0], 'text').some((node: any) => node.content === 'note')).toBe(true)
|
||||
})
|
||||
|
||||
it.each([true, false])('preserves brackets inside sup html content when final=%s', (final) => {
|
||||
const nodes = parse('测试<sup>[3]</sup>。', final)
|
||||
const paragraph = nodes[0] as any
|
||||
const htmlInline = collectByType(paragraph, 'html_inline')[0] as any
|
||||
|
||||
expect(htmlInline).toBeDefined()
|
||||
expect(htmlInline.content).toBe('<sup>[3]</sup>')
|
||||
expect(htmlInline.children).toEqual([
|
||||
{
|
||||
type: 'text',
|
||||
content: '[3]',
|
||||
raw: '[3]',
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
it.each([true, false])('preserves brackets inside generic inline html content when final=%s', (final) => {
|
||||
const nodes = parse('<span>[3]</span>', final)
|
||||
const paragraph = nodes[0] as any
|
||||
const htmlInline = collectByType(paragraph, 'html_inline')[0] as any
|
||||
|
||||
expect(htmlInline).toBeDefined()
|
||||
expect(htmlInline.content).toBe('<span>[3]</span>')
|
||||
expect(htmlInline.children).toEqual([
|
||||
{
|
||||
type: 'text',
|
||||
content: '[3]',
|
||||
raw: '[3]',
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
it.each([true, false])('parses the real multiline issue-386 input without leaking raw superscript syntax when final=%s', (final) => {
|
||||
const nodes = parse(REAL_WORLD_MULTILINE_INPUT, final)
|
||||
const paragraph = nodes[0] as any
|
||||
const superscripts = collectByType(paragraph, 'superscript')
|
||||
const htmlInlineNodes = collectByType(paragraph, 'html_inline')
|
||||
const textNodes = collectByType(paragraph, 'text')
|
||||
|
||||
expect(paragraph?.type).toBe('paragraph')
|
||||
expect(superscripts).toHaveLength(7)
|
||||
expect(superscripts.map((node: any) => node.children?.map((child: any) => child.content).join(''))).toEqual([
|
||||
'[1]',
|
||||
'[1]',
|
||||
'[1]',
|
||||
'[1]',
|
||||
'[1]',
|
||||
'[1]',
|
||||
'[1]',
|
||||
])
|
||||
expect(textNodes.some((node: any) => String(node.content).includes('^[1]^'))).toBe(false)
|
||||
expect(htmlInlineNodes.map((node: any) => node.children?.map((child: any) => child.content).join(''))).toEqual([
|
||||
'[3]',
|
||||
'[1]',
|
||||
'[12]',
|
||||
'[3]',
|
||||
'[3]',
|
||||
'[12]',
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,37 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('link attrs parsing', () => {
|
||||
it('preserves custom attrs injected on link_open tokens', () => {
|
||||
const md = getMarkdown('link-attrs')
|
||||
const nodes = parseMarkdownToStructure('[Example](https://example.com)', md, {
|
||||
preTransformTokens(tokens: any[]) {
|
||||
for (const token of tokens) {
|
||||
if (token?.type !== 'inline' || !Array.isArray(token.children))
|
||||
continue
|
||||
for (const child of token.children) {
|
||||
if (child?.type !== 'link_open')
|
||||
continue
|
||||
if (typeof child.attrSet === 'function') {
|
||||
child.attrSet('target', '_self')
|
||||
child.attrSet('data-track', 'cta')
|
||||
}
|
||||
else {
|
||||
child.attrs = [...(child.attrs ?? []), ['target', '_self'], ['data-track', 'cta']]
|
||||
}
|
||||
}
|
||||
}
|
||||
return tokens
|
||||
},
|
||||
})
|
||||
|
||||
const para = nodes[0] as any
|
||||
const link = para.children.find((c: any) => c.type === 'link')
|
||||
expect(link).toBeDefined()
|
||||
expect(link.attrs).toEqual(expect.arrayContaining([
|
||||
['href', 'https://example.com'],
|
||||
['target', '_self'],
|
||||
['data-track', 'cta'],
|
||||
]))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,258 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { textIncludes } from '../../../test/utils/midstate-utils'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
const md = getMarkdown('test')
|
||||
|
||||
function collectByType(nodes: any[], type: string) {
|
||||
const out: any[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === type)
|
||||
out.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
if (Array.isArray(n.items))
|
||||
n.items.forEach(walk)
|
||||
}
|
||||
nodes.forEach(walk)
|
||||
return out
|
||||
}
|
||||
|
||||
describe('special link cases', () => {
|
||||
it('parses link with numeric brackets in link text: [[8]](url)', () => {
|
||||
const markdown = '建议采用以下接口方案:[[8]](https://markstream-vue-docs.simonhe.me/)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
// Collect all link nodes
|
||||
const links: any[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'link')
|
||||
links.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
if (Array.isArray(n.items))
|
||||
n.items.forEach(walk)
|
||||
}
|
||||
nodes.forEach(walk)
|
||||
|
||||
// Should have exactly one link
|
||||
expect(links.length).toBe(1)
|
||||
const link = links[0]
|
||||
|
||||
// Verify link properties
|
||||
expect(link.href).toBe('https://markstream-vue-docs.simonhe.me/')
|
||||
expect(link.loading).toBe(false)
|
||||
|
||||
// Verify link text contains [8]
|
||||
// The link text should be "[8]" (the brackets are part of the link text)
|
||||
const linkText = link.text || ''
|
||||
const childText = (link.children || [])
|
||||
.map((c: any) => (c.content ?? c.text ?? ''))
|
||||
.join('')
|
||||
const fullText = linkText || childText
|
||||
|
||||
expect(fullText).toBe('[8]')
|
||||
|
||||
// Verify the paragraph contains both the prefix text and the link
|
||||
const para = nodes.find((n: any) => n.type === 'paragraph') as any
|
||||
expect(para).toBeTruthy()
|
||||
expect(textIncludes(para, '建议采用以下接口方案:')).toBe(true)
|
||||
expect(para.children.some((c: any) => c.type === 'link')).toBe(true)
|
||||
})
|
||||
|
||||
it('parses link with emphasis markers in link text: [DR **(Danmarks Radio)](url)', () => {
|
||||
const markdown = '[DR **(Danmarks Radio)](https://www.dr.dk/nyheder)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
// Collect all link nodes
|
||||
const links: any[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'link')
|
||||
links.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
if (Array.isArray(n.items))
|
||||
n.items.forEach(walk)
|
||||
}
|
||||
nodes.forEach(walk)
|
||||
|
||||
// Should have exactly one link
|
||||
expect(links.length).toBe(1)
|
||||
const link = links[0]
|
||||
|
||||
// Verify link properties
|
||||
expect(link.href).toBe('https://www.dr.dk/nyheder')
|
||||
expect(link.loading).toBe(false)
|
||||
|
||||
// Should contain "DR" and "Danmarks Radio"
|
||||
expect(textIncludes(link, 'DR')).toBe(true)
|
||||
expect(textIncludes(link, 'Danmarks Radio')).toBe(true)
|
||||
expect(textIncludes(link, '(Danmarks Radio)')).toBe(true)
|
||||
|
||||
// Verify the paragraph contains the link
|
||||
const para = nodes.find((n: any) => n.type === 'paragraph') as any
|
||||
expect(para).toBeTruthy()
|
||||
expect(para.children.some((c: any) => c.type === 'link')).toBe(true)
|
||||
})
|
||||
|
||||
it('parses multiple links with special characters in same paragraph', () => {
|
||||
const markdown = '建议采用以下接口方案:[[8]](https://markstream-vue-docs.simonhe.me/)\n\n[DR **(Danmarks Radio)](https://www.dr.dk/nyheder)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
// Collect all link nodes
|
||||
const links: any[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'link')
|
||||
links.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
if (Array.isArray(n.items))
|
||||
n.items.forEach(walk)
|
||||
}
|
||||
nodes.forEach(walk)
|
||||
|
||||
// Should have exactly two links
|
||||
expect(links.length).toBe(2)
|
||||
|
||||
// First link: [[8]](url)
|
||||
const firstLink = links.find(l => l.href === 'https://markstream-vue-docs.simonhe.me/')
|
||||
expect(firstLink).toBeDefined()
|
||||
expect(firstLink?.loading).toBe(false)
|
||||
const firstLinkText = firstLink?.text || (firstLink?.children || []).map((c: any) => c.content || '').join('')
|
||||
expect(firstLinkText).toBe('[8]')
|
||||
|
||||
// Second link: [DR **(Danmarks Radio)](url)
|
||||
const secondLink = links.find(l => l.href === 'https://www.dr.dk/nyheder')
|
||||
expect(secondLink).toBeDefined()
|
||||
expect(secondLink?.loading).toBe(false)
|
||||
expect(textIncludes(secondLink, 'DR')).toBe(true)
|
||||
expect(textIncludes(secondLink, 'Danmarks Radio')).toBe(true)
|
||||
})
|
||||
|
||||
it('parses standalone numeric bracket link without prefix text', () => {
|
||||
const markdown = '[[8]](https://example.com)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
const links: any[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'link')
|
||||
links.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
}
|
||||
nodes.forEach(walk)
|
||||
|
||||
expect(links.length).toBe(1)
|
||||
expect(links[0].href).toBe('https://example.com')
|
||||
expect(links[0].text || links[0].children?.map((c: any) => c.content || '').join('')).toBe('[8]')
|
||||
})
|
||||
|
||||
it('distinguishes between reference [8] and link [[8]](url)', () => {
|
||||
// Standalone [8] should be a reference
|
||||
const referenceMarkdown = '请参考 [8] 了解更多信息'
|
||||
const referenceNodes = parseMarkdownToStructure(referenceMarkdown, md)
|
||||
|
||||
const references: any[] = []
|
||||
const walkRef = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'reference')
|
||||
references.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walkRef)
|
||||
}
|
||||
referenceNodes.forEach(walkRef)
|
||||
|
||||
// [8] as standalone should be parsed as reference (if reference rule matches)
|
||||
// But if it's followed by ](, it should be a link
|
||||
const linkMarkdown = '请参考 [[8]](https://example.com) 了解更多信息'
|
||||
const linkNodes = parseMarkdownToStructure(linkMarkdown, md)
|
||||
|
||||
const links: any[] = []
|
||||
const walkLink = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (n.type === 'link')
|
||||
links.push(n)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walkLink)
|
||||
}
|
||||
linkNodes.forEach(walkLink)
|
||||
|
||||
// The link version should have a link node
|
||||
expect(links.length).toBeGreaterThan(0)
|
||||
const link = links.find(l => l.href === 'https://example.com')
|
||||
expect(link).toBeDefined()
|
||||
expect(link?.text || link?.children?.map((c: any) => c.content || '').join('')).toBe('[8]')
|
||||
})
|
||||
|
||||
it('preserves normal reference-style links when the label is not numeric', () => {
|
||||
const markdown = `请看 [1][source]
|
||||
|
||||
[source]: https://example.com`
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const links = collectByType(nodes, 'link')
|
||||
const references = collectByType(nodes, 'reference')
|
||||
|
||||
expect(links).toHaveLength(1)
|
||||
expect(links[0].href).toBe('https://example.com')
|
||||
expect(links[0].text || links[0].children?.map((c: any) => c.content || '').join('')).toBe('1')
|
||||
expect(references).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('parses numeric references when they are adjacent to surrounding text', () => {
|
||||
const cases = [
|
||||
'随便输入一些文字[1]',
|
||||
'随便输入一些文字[1]后面还有文字',
|
||||
'随便输入一些文字[1] ',
|
||||
'随便输入一些文字[1] (空格+文字)',
|
||||
]
|
||||
|
||||
for (const markdown of cases) {
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const references = collectByType(nodes, 'reference')
|
||||
|
||||
expect(references, markdown).toHaveLength(1)
|
||||
expect(references[0].id).toBe('1')
|
||||
expect(textIncludes(nodes, '随便输入一些文字')).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
it('parses a numeric reference immediately followed by a normal inline link', () => {
|
||||
const cases = [
|
||||
'随便输入一些文字 [1][百度](www.baidu.com)',
|
||||
'随便输入一些文字 [1] [百度](www.baidu.com)',
|
||||
]
|
||||
|
||||
for (const markdown of cases) {
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const references = collectByType(nodes, 'reference')
|
||||
const links = collectByType(nodes, 'link')
|
||||
|
||||
expect(references, markdown).toHaveLength(1)
|
||||
expect(references[0].id).toBe('1')
|
||||
expect(links, markdown).toHaveLength(1)
|
||||
expect(links[0].href).toBe('www.baidu.com')
|
||||
expect(textIncludes(links[0], '百度')).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
it('parses chained numeric references without requiring spaces', () => {
|
||||
const nodes = parseMarkdownToStructure('随便输入一些文字[1][2][3]', md)
|
||||
const references = collectByType(nodes, 'reference')
|
||||
|
||||
expect(references).toHaveLength(3)
|
||||
expect(references.map(ref => ref.id)).toEqual(['1', '2', '3'])
|
||||
expect(textIncludes(nodes, '随便输入一些文字')).toBe(true)
|
||||
})
|
||||
})
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function collectByType(nodes: any, type: string, out: any[] = []) {
|
||||
if (!nodes)
|
||||
return out
|
||||
if (Array.isArray(nodes)) {
|
||||
nodes.forEach(node => collectByType(node, type, out))
|
||||
return out
|
||||
}
|
||||
if (nodes.type === type)
|
||||
out.push(nodes)
|
||||
if (Array.isArray(nodes.children))
|
||||
nodes.children.forEach((child: any) => collectByType(child, type, out))
|
||||
if (Array.isArray(nodes.items))
|
||||
nodes.items.forEach((item: any) => collectByType(item, type, out))
|
||||
return out
|
||||
}
|
||||
|
||||
function collectText(nodes: any) {
|
||||
return collectByType(nodes, 'text')
|
||||
.map((node: any) => String(node.content ?? ''))
|
||||
.join('')
|
||||
}
|
||||
|
||||
function expectListStreamingTextToStayVisible(options: {
|
||||
id: string
|
||||
full: string
|
||||
explanation: string
|
||||
suffix: string
|
||||
protectedType?: string
|
||||
}) {
|
||||
const md = getMarkdown(options.id)
|
||||
const explanationReadyAt = options.full.indexOf(options.explanation) + options.explanation.length
|
||||
const suffixStartAt = options.full.indexOf(options.suffix)
|
||||
|
||||
for (let i = explanationReadyAt; i <= options.full.length; i++) {
|
||||
const chunk = options.full.slice(0, i)
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
const text = collectText(nodes)
|
||||
|
||||
expect(text, `prefix(${i}): ${chunk}`).toContain(options.explanation)
|
||||
expect(text.match(new RegExp(options.explanation, 'g')) ?? [], `prefix(${i}): ${chunk}`).toHaveLength(1)
|
||||
|
||||
if (i > suffixStartAt) {
|
||||
const expectedSuffix = options.full.slice(suffixStartAt, i)
|
||||
expect(text, `prefix(${i}): ${chunk}`).toContain(expectedSuffix)
|
||||
|
||||
if (options.protectedType) {
|
||||
const protectedNodes = collectByType(nodes, options.protectedType)
|
||||
expect(
|
||||
protectedNodes.some((node: any) => JSON.stringify(node).includes(expectedSuffix)),
|
||||
`prefix(${i}): ${chunk}`,
|
||||
).toBe(false)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
describe('list inline streaming regression', () => {
|
||||
it('keeps explanatory list text stable while inline code opens, grows, closes, and is followed by more text', () => {
|
||||
const full = `- **计算工具验证**
|
||||
通过数学计算工具确认结果:
|
||||
\`363 ÷ 15,135 × 100 = 2.39841427...\`
|
||||
后续说明继续输出`
|
||||
|
||||
expectListStreamingTextToStayVisible({
|
||||
id: 'list-inline-code-streaming-regression',
|
||||
full,
|
||||
explanation: '通过数学计算工具确认结果:',
|
||||
suffix: '后续说明继续输出',
|
||||
protectedType: 'inline_code',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps explanatory list text stable while an inline link opens, closes, and is followed by more text', () => {
|
||||
const full = `- **链接验证**
|
||||
通过链接工具确认结果:
|
||||
[查看详情](https://example.com)
|
||||
后续说明继续输出`
|
||||
|
||||
expectListStreamingTextToStayVisible({
|
||||
id: 'list-inline-link-streaming-regression',
|
||||
full,
|
||||
explanation: '通过链接工具确认结果:',
|
||||
suffix: '后续说明继续输出',
|
||||
protectedType: 'link',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps explanatory list text stable while emphasis opens, closes, and is followed by more text', () => {
|
||||
const full = `- **强调说明**
|
||||
通过人工复核确认结果:
|
||||
*结果待确认*
|
||||
后续说明继续输出`
|
||||
|
||||
expectListStreamingTextToStayVisible({
|
||||
id: 'list-emphasis-streaming-regression',
|
||||
full,
|
||||
explanation: '通过人工复核确认结果:',
|
||||
suffix: '后续说明继续输出',
|
||||
protectedType: 'emphasis',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps explanatory list text stable while inline math closes and is followed by more text', () => {
|
||||
const full = `- **数学验证**
|
||||
通过公式确认结果:
|
||||
$x^2 + y^2$
|
||||
后续说明继续输出`
|
||||
|
||||
expectListStreamingTextToStayVisible({
|
||||
id: 'list-math-inline-streaming-regression',
|
||||
full,
|
||||
explanation: '通过公式确认结果:',
|
||||
suffix: '后续说明继续输出',
|
||||
protectedType: 'math_inline',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps explanatory list text stable while a numeric reference appears and is followed by more text', () => {
|
||||
const full = `- **参考验证**
|
||||
通过参考编号确认结果:
|
||||
[1]
|
||||
后续说明继续输出`
|
||||
|
||||
expectListStreamingTextToStayVisible({
|
||||
id: 'list-reference-streaming-regression',
|
||||
full,
|
||||
explanation: '通过参考编号确认结果:',
|
||||
suffix: '后续说明继续输出',
|
||||
protectedType: 'reference',
|
||||
})
|
||||
})
|
||||
})
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function flatten(nodes: any[]): any[] {
|
||||
const out: any[] = []
|
||||
for (const n of nodes ?? []) {
|
||||
out.push(n)
|
||||
if (Array.isArray(n?.children))
|
||||
out.push(...flatten(n.children))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
describe('parseMarkdownToStructure - multiple links same line', () => {
|
||||
it('parses 3 adjacent links as 3 link nodes', () => {
|
||||
const md = getMarkdown()
|
||||
const markdown = '[citation](http://url1) [citation](http://url1) [citation](http://url1)'
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
|
||||
const all = flatten(nodes)
|
||||
const links = all.filter(n => n?.type === 'link')
|
||||
|
||||
expect(links).toHaveLength(3)
|
||||
expect(links.map(l => l.href)).toEqual(['http://url1', 'http://url1', 'http://url1'])
|
||||
})
|
||||
})
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function collectByType(nodes: any[], type: string) {
|
||||
const out: any[] = []
|
||||
const walk = (node: any) => {
|
||||
if (!node)
|
||||
return
|
||||
if (Array.isArray(node)) {
|
||||
node.forEach(walk)
|
||||
return
|
||||
}
|
||||
if (node.type === type)
|
||||
out.push(node)
|
||||
if (Array.isArray(node.children))
|
||||
node.children.forEach(walk)
|
||||
if (Array.isArray(node.items))
|
||||
node.items.forEach(walk)
|
||||
}
|
||||
walk(nodes)
|
||||
return out
|
||||
}
|
||||
|
||||
function collectText(nodes: any[]) {
|
||||
return collectByType(nodes, 'text').map((node: any) => String(node.content ?? '')).join('')
|
||||
}
|
||||
|
||||
describe('numeric reference streaming stability', () => {
|
||||
it('keeps completed numeric references stable during char-by-char chained references', () => {
|
||||
const md = getMarkdown('numeric-reference-streaming-chain')
|
||||
const full = '随便输入一些文字[1][2][3]'
|
||||
const firstReady = full.indexOf('[1]') + '[1]'.length
|
||||
const secondOpens = full.indexOf('[2]')
|
||||
const secondReady = secondOpens + '[2]'.length
|
||||
const thirdReady = full.indexOf('[3]') + '[3]'.length
|
||||
|
||||
for (let i = 1; i <= full.length; i++) {
|
||||
const chunk = full.slice(0, i)
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
|
||||
|
||||
if (i >= firstReady)
|
||||
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
|
||||
if (i > secondOpens)
|
||||
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
|
||||
if (i >= secondReady)
|
||||
expect(ids.slice(0, 2), `prefix(${i}): ${chunk}`).toEqual(['1', '2'])
|
||||
if (i >= thirdReady)
|
||||
expect(ids.slice(0, 3), `prefix(${i}): ${chunk}`).toEqual(['1', '2', '3'])
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps the numeric reference once an adjacent inline link becomes distinguishable', () => {
|
||||
const md = getMarkdown('numeric-reference-streaming-link')
|
||||
const full = '随便输入一些文字 [1][百度](www.baidu.com)'
|
||||
const distinguishableAt = full.indexOf('[百度](') + '[百度]('.length
|
||||
|
||||
for (let i = 1; i <= full.length; i++) {
|
||||
const chunk = full.slice(0, i)
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
|
||||
const links = collectByType(nodes, 'link')
|
||||
|
||||
if (i >= distinguishableAt) {
|
||||
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
|
||||
expect(collectText(nodes), `prefix(${i}): ${chunk}`).not.toContain('[1]')
|
||||
expect(links, `prefix(${i}): ${chunk}`).toHaveLength(1)
|
||||
expect(String(links[0]?.text ?? ''), `prefix(${i}): ${chunk}`).toContain('百度')
|
||||
expect(Boolean(links[0]?.loading), `prefix(${i}): ${chunk}`).toBe(i < full.length)
|
||||
}
|
||||
}
|
||||
|
||||
const finalNodes = parseMarkdownToStructure(full, md, { final: true })
|
||||
const finalIds = collectByType(finalNodes, 'reference').map((node: any) => String(node.id ?? ''))
|
||||
const links = collectByType(finalNodes, 'link')
|
||||
|
||||
expect(finalIds).toEqual(['1'])
|
||||
expect(links).toHaveLength(1)
|
||||
expect(String(links[0]?.href ?? '')).toBe('www.baidu.com')
|
||||
})
|
||||
|
||||
it('keeps the completed numeric reference while trailing text streams in', () => {
|
||||
const md = getMarkdown('numeric-reference-streaming-suffix')
|
||||
const full = '随便输入一些文字[1]后面还有文字'
|
||||
const readyAt = full.indexOf('[1]') + '[1]'.length
|
||||
|
||||
for (let i = readyAt; i <= full.length; i++) {
|
||||
const chunk = full.slice(0, i)
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
|
||||
|
||||
expect(ids[0], `prefix(${i}): ${chunk}`).toBe('1')
|
||||
}
|
||||
})
|
||||
|
||||
it('still respects validateLink when the trailing markdown link is being recovered', () => {
|
||||
const md = getMarkdown('numeric-reference-streaming-validate')
|
||||
md.set?.({ validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()) })
|
||||
|
||||
const chunk = '随便输入一些文字 [1][百度](javascript:alert(1'
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
const ids = collectByType(nodes, 'reference').map((node: any) => String(node.id ?? ''))
|
||||
const links = collectByType(nodes, 'link')
|
||||
|
||||
expect(ids).toEqual(['1'])
|
||||
expect(links).toHaveLength(1)
|
||||
expect(String(links[0]?.text ?? '')).toBe('百度')
|
||||
expect(String(links[0]?.href ?? '')).not.toMatch(/javascript:/i)
|
||||
expect(Boolean(links[0]?.loading)).toBe(true)
|
||||
expect(collectText(nodes)).toContain('百度')
|
||||
})
|
||||
})
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function walk(node: any, visit: (n: any) => void) {
|
||||
if (!node)
|
||||
return
|
||||
if (Array.isArray(node)) {
|
||||
for (const n of node)
|
||||
walk(n, visit)
|
||||
return
|
||||
}
|
||||
visit(node)
|
||||
if (Array.isArray(node.children))
|
||||
walk(node.children, visit)
|
||||
if (Array.isArray(node.items))
|
||||
walk(node.items, visit)
|
||||
}
|
||||
|
||||
describe('streaming ordered list - avoid transient gaps', () => {
|
||||
it('suppresses transient nested lists and leaked newlines while streaming', () => {
|
||||
const md = getMarkdown()
|
||||
const full = `法:
|
||||
|
||||
1. **递归实现**:直接
|
||||
|
||||
2. **记忆化递归(动态规划)**:使
|
||||
`
|
||||
|
||||
const marker1 = full.indexOf('1.')
|
||||
const marker2 = full.indexOf('\n\n2.')
|
||||
const cuts = [
|
||||
marker1 + 2, // "1."
|
||||
marker1 + 3, // "1. "
|
||||
marker1 + 4, // "1. *"
|
||||
marker1 + 5, // "1. **"
|
||||
full.indexOf('直接') + '直接'.length + 1, // right after item 1
|
||||
marker2 + 3, // "\n\n2"
|
||||
marker2 + 4, // "\n\n2."
|
||||
]
|
||||
.filter(n => Number.isFinite(n) && n > 0 && n <= full.length)
|
||||
.filter((n, i, arr) => arr.indexOf(n) === i)
|
||||
.sort((a, b) => a - b)
|
||||
|
||||
for (const cut of cuts) {
|
||||
const chunk = full.slice(0, cut)
|
||||
const nodes = parseMarkdownToStructure(chunk, md, { final: false })
|
||||
|
||||
const unorderedLists: any[] = []
|
||||
const newlineTextNodes: string[] = []
|
||||
const markerParagraphs: string[] = []
|
||||
walk(nodes, (n) => {
|
||||
if (n?.type === 'list' && n.ordered === false)
|
||||
unorderedLists.push(n)
|
||||
if (n?.type === 'text' && typeof n.content === 'string' && /[\r\n]/.test(n.content))
|
||||
newlineTextNodes.push(n.content)
|
||||
if (n?.type === 'paragraph' && typeof n.raw === 'string' && /^\s*\d+[.)]?\s*$/.test(n.raw))
|
||||
markerParagraphs.push(n.raw)
|
||||
})
|
||||
|
||||
expect(unorderedLists, `unexpected unordered list at cut=${cut}`).toHaveLength(0)
|
||||
expect(newlineTextNodes, `unexpected newline text nodes at cut=${cut}`).toHaveLength(0)
|
||||
expect(markerParagraphs, `unexpected marker paragraph at cut=${cut}`).toHaveLength(0)
|
||||
}
|
||||
})
|
||||
})
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
describe('parseMarkdownToStructure with code in link text', () => {
|
||||
it('should parse links with inline code correctly', () => {
|
||||
const md = getMarkdown()
|
||||
|
||||
const mdText1 = `the [\`npm init\` command](https://docs.npmjs.com/creating-a-package-json-file).`
|
||||
const result1 = parseMarkdownToStructure(mdText1, md, { final: true })
|
||||
// 验证结果的基本结构
|
||||
expect(result1).toBeDefined()
|
||||
expect(result1.length).toBe(1)
|
||||
expect(result1[0].type).toBe('paragraph')
|
||||
expect(result1[0].children.length).toBe(3)
|
||||
|
||||
// 验证链接节点
|
||||
const linkNode1 = result1[0].children[1]
|
||||
expect(linkNode1.type).toBe('link')
|
||||
expect(linkNode1.href).toBe('https://docs.npmjs.com/creating-a-package-json-file')
|
||||
expect(linkNode1.text).toBe('npm init command')
|
||||
expect(linkNode1.children.length).toBe(2)
|
||||
|
||||
// 验证链接中的内联代码节点
|
||||
expect(linkNode1.children[0].type).toBe('inline_code')
|
||||
expect(linkNode1.children[0].code).toBe('npm init')
|
||||
expect(linkNode1.children[1].type).toBe('text')
|
||||
expect(linkNode1.children[1].content).toBe(' command')
|
||||
|
||||
// 验证第二个链接
|
||||
const mdText2 = `[\`npm install\` command](https://docs.npmjs.com/getting-started/installing-npm-packages-locally):`
|
||||
const result2 = parseMarkdownToStructure(mdText2, md, { final: true })
|
||||
|
||||
expect(result2.length).toBe(1)
|
||||
const linkNode2 = result2[0].children[0]
|
||||
expect(linkNode2.type).toBe('link')
|
||||
expect(linkNode2.href).toBe('https://docs.npmjs.com/getting-started/installing-npm-packages-locally')
|
||||
expect(linkNode2.text).toBe('npm install command')
|
||||
expect(linkNode2.children.length).toBe(2)
|
||||
|
||||
expect(linkNode2.children[0].type).toBe('inline_code')
|
||||
expect(linkNode2.children[0].code).toBe('npm install')
|
||||
expect(linkNode2.children[1].type).toBe('text')
|
||||
expect(linkNode2.children[1].content).toBe(' command')
|
||||
})
|
||||
})
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function collectTexts(n: any): string[] {
|
||||
if (!n)
|
||||
return []
|
||||
if (Array.isArray(n))
|
||||
return n.flatMap(collectTexts)
|
||||
if (typeof n === 'string')
|
||||
return [n]
|
||||
const texts: string[] = []
|
||||
if (n.type === 'text' && typeof n.content === 'string')
|
||||
texts.push(n.content)
|
||||
if (n.children && Array.isArray(n.children))
|
||||
texts.push(...n.children.flatMap(collectTexts))
|
||||
if (n.items && Array.isArray(n.items))
|
||||
texts.push(...n.items.flatMap(collectTexts))
|
||||
return texts
|
||||
}
|
||||
|
||||
function countOccurrences(nodes: any, needle: string) {
|
||||
const allTexts = collectTexts(nodes).join('\n')
|
||||
return (allTexts.match(new RegExp(needle, 'g')) || []).length
|
||||
}
|
||||
|
||||
describe('parseMarkdownToStructure - duplicate question rendering', () => {
|
||||
it('does not duplicate the question text when parsing mixed math/ce commands', () => {
|
||||
const md = getMarkdown()
|
||||
|
||||
const markdown = `**当堂检测**:
|
||||
1. 下列物质属于酚的是( )
|
||||
A. $\ce{CH3CH2OH}$ B. $\ce{C6H5CH2OH}$ C. $\ce{}$ D. $\ce{HO-CH2-CH2OH}$
|
||||
**答案**:C`
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md)
|
||||
const allTexts = collectTexts(nodes).join('\n')
|
||||
console.log('All texts:', JSON.stringify(allTexts, null, 2))
|
||||
// Count occurrences of the core question string
|
||||
expect(countOccurrences(nodes, '下列物质属于酚的是')).toBe(1)
|
||||
})
|
||||
|
||||
const streamingCases = [
|
||||
{
|
||||
name: 'plain text prefix before $$ math',
|
||||
markdown: `第(1)题
|
||||
题干解析:本题考查利用导数求含参函数的极值,核心条件为函数 $$ f(x) = \\ln x - m + \\dfrac{m}{x} $$($$ m \\in \\mathbb{R} $$)`,
|
||||
needle: '第(1)题',
|
||||
},
|
||||
{
|
||||
name: 'plain text prefix before $ math',
|
||||
markdown: `第(2)题
|
||||
已知函数 $f(x)=x^2$,求导数。`,
|
||||
needle: '第(2)题',
|
||||
},
|
||||
{
|
||||
name: 'plain text prefix before \\( \\) math',
|
||||
markdown: String.raw`第(3)题
|
||||
若 \(a+b=1\),求 \(ab\) 的值。`,
|
||||
needle: '第(3)题',
|
||||
},
|
||||
{
|
||||
name: 'strong prefix before $$ math',
|
||||
markdown: `**第(4)题**
|
||||
题干解析:函数 $$x^2$$ 的图像。`,
|
||||
needle: '第(4)题',
|
||||
},
|
||||
{
|
||||
name: 'markdown link prefix before $ math',
|
||||
markdown: `[第(5)题](https://example.com)
|
||||
题干解析:函数 $x^2$ 的图像。`,
|
||||
needle: '第(5)题',
|
||||
},
|
||||
{
|
||||
name: 'inline html prefix before $$ math',
|
||||
markdown: `<span>第(6)题</span>
|
||||
题干解析:函数 $$x^2$$ 的图像。`,
|
||||
needle: '第(6)题',
|
||||
},
|
||||
] as const
|
||||
|
||||
for (const testCase of streamingCases) {
|
||||
it(`does not duplicate the hardbreak prefix for ${testCase.name} while streaming`, () => {
|
||||
const md = getMarkdown()
|
||||
const nodes = parseMarkdownToStructure(testCase.markdown, md)
|
||||
|
||||
expect(countOccurrences(nodes, testCase.needle)).toBe(1)
|
||||
})
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,3 @@
|
||||
// Delegate to the repository-level vitest setup so package tests
|
||||
// reuse the same global mocks (Worker, ResizeObserver, etc.).
|
||||
import '../../../../test/setup/vitest.setup'
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { parseFenceToken } from '../src'
|
||||
|
||||
function parseStreamingDiff(content: string, closed = false) {
|
||||
return parseFenceToken({
|
||||
type: 'fence',
|
||||
info: 'diff json:package.json',
|
||||
content,
|
||||
map: [0, 1],
|
||||
meta: { closed },
|
||||
} as any)
|
||||
}
|
||||
|
||||
describe('parseFenceToken - streaming unified diff stability', () => {
|
||||
it('splits standard unified diff lines without requiring a space after the marker', () => {
|
||||
const node = parseFenceToken({
|
||||
type: 'fence',
|
||||
info: 'diff typescript',
|
||||
content: [
|
||||
'diff --git a/src/render.ts b/src/render.ts',
|
||||
'index 0000000..1111111 100644',
|
||||
'--- a/src/render.ts',
|
||||
'+++ b/src/render.ts',
|
||||
'@@ -1,7 +1,12 @@',
|
||||
'-export function render(input) {',
|
||||
'- return input',
|
||||
'+export function render(input: string) {',
|
||||
'+ if (!input.trim())',
|
||||
'+ return \'empty\'',
|
||||
'+',
|
||||
'+ const normalized = input.replace(/\\r\\n/g, \'\\n\')',
|
||||
'+ return normalized',
|
||||
' }',
|
||||
].join('\n'),
|
||||
map: [0, 1],
|
||||
meta: { closed: true },
|
||||
} as any)
|
||||
|
||||
expect(node.language).toBe('typescript')
|
||||
expect(node.originalCode).toBe([
|
||||
'export function render(input) {',
|
||||
' return input',
|
||||
'}',
|
||||
].join('\n'))
|
||||
expect(node.updatedCode).toBe([
|
||||
'export function render(input: string) {',
|
||||
' if (!input.trim())',
|
||||
' return \'empty\'',
|
||||
'',
|
||||
' const normalized = input.replace(/\\r\\n/g, \'\\n\')',
|
||||
' return normalized',
|
||||
'}',
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('withholds an unfinished replacement hunk until the added side is available', () => {
|
||||
const node = parseStreamingDiff([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
'- "version": "0.0.49",',
|
||||
].join('\n'))
|
||||
|
||||
expect(node.originalCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
].join('\n'))
|
||||
expect(node.updatedCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('flushes a finished replacement hunk once the added line becomes stable', () => {
|
||||
const node = parseStreamingDiff([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
'- "version": "0.0.49",',
|
||||
'+ "version": "0.0.54-beta.1",',
|
||||
'',
|
||||
].join('\n'))
|
||||
|
||||
expect(node.originalCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
' "version": "0.0.49",',
|
||||
].join('\n'))
|
||||
expect(node.updatedCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
' "version": "0.0.54-beta.1",',
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('withholds a trailing incomplete added line instead of leaking a mismatched preview', () => {
|
||||
const node = parseStreamingDiff([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
'- "version": "0.0.49",',
|
||||
'+ "version": "0.0.54-b',
|
||||
].join('\n'))
|
||||
|
||||
expect(node.originalCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
].join('\n'))
|
||||
expect(node.updatedCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('flushes the trailing hunk once the fence is closed', () => {
|
||||
const node = parseStreamingDiff([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
'- "version": "0.0.49",',
|
||||
].join('\n'), true)
|
||||
|
||||
expect(node.originalCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
' "version": "0.0.49",',
|
||||
].join('\n'))
|
||||
expect(node.updatedCode).toBe([
|
||||
'{',
|
||||
' "name": "markstream-vue",',
|
||||
].join('\n'))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,64 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { parseInlineTokens } from '../src'
|
||||
|
||||
function flattenText(nodes: any[]): string {
|
||||
const out: string[] = []
|
||||
const walk = (n: any) => {
|
||||
if (!n)
|
||||
return
|
||||
if (Array.isArray(n)) {
|
||||
n.forEach(walk)
|
||||
return
|
||||
}
|
||||
if (n.type === 'text' && typeof n.content === 'string')
|
||||
out.push(n.content)
|
||||
if (Array.isArray(n.children))
|
||||
n.children.forEach(walk)
|
||||
if (Array.isArray(n.items))
|
||||
n.items.forEach(walk)
|
||||
}
|
||||
walk(nodes)
|
||||
return out.join('')
|
||||
}
|
||||
|
||||
describe('parseInlineTokens strict strong matching', () => {
|
||||
it('default (non-strict) treats unclosed ** as mid-state strong', () => {
|
||||
const raw = '[**cxx](xxx)'
|
||||
const tokens: any[] = [{ type: 'text', content: raw }]
|
||||
const nodes = parseInlineTokens(tokens, raw)
|
||||
expect(nodes.length).toBeGreaterThan(0)
|
||||
// Expect the parser will create a strong node in non-strict mode
|
||||
const first = nodes[0] as any
|
||||
expect(first).toBeDefined()
|
||||
expect(first.type).toBe('strong')
|
||||
expect(first.children?.[0]?.type).toBe('link')
|
||||
expect(first.children?.[0]?.text).toContain('cxx')
|
||||
})
|
||||
|
||||
it('strict mode keeps original text when no closing ** is found', () => {
|
||||
const raw = '[**cxx](xxx)'
|
||||
const tokens: any[] = [{ type: 'text', content: raw }]
|
||||
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
|
||||
expect(nodes.length).toBeGreaterThan(0)
|
||||
const first = nodes[0] as any
|
||||
expect(first.type).toBe('link')
|
||||
expect(first.children[0].content).toBe(raw.match(/\[([^\]]+)\]/)?.[1])
|
||||
})
|
||||
|
||||
it('strict mode does not duplicate prefix text before unmatched **', () => {
|
||||
const raw = 'foo**bar'
|
||||
const tokens: any[] = [{ type: 'text', content: raw }]
|
||||
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
|
||||
expect(nodes.length).toBeGreaterThan(0)
|
||||
const first = nodes[0] as any
|
||||
expect(first.type).toBe('text')
|
||||
expect(first.content).toBe('foo**bar')
|
||||
})
|
||||
|
||||
it('strict mode does not duplicate prefix when unmatched ** is inside link text', () => {
|
||||
const raw = '前缀[**cxx](xxx)'
|
||||
const tokens: any[] = [{ type: 'text', content: raw }]
|
||||
const nodes = parseInlineTokens(tokens, raw, undefined, { requireClosingStrong: true })
|
||||
expect(flattenText(nodes as any[])).toBe('前缀**cxx')
|
||||
})
|
||||
})
|
||||
+378
@@ -0,0 +1,378 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getMarkdown, parseMarkdownToStructure } from '../src'
|
||||
|
||||
function expectLinkChild(node: any, expected: {
|
||||
href: string
|
||||
text: string
|
||||
loading: boolean
|
||||
childType: string
|
||||
childText: string
|
||||
}) {
|
||||
expect(node.type).toBe('link')
|
||||
expect(node.href).toBe(expected.href)
|
||||
expect(node.text).toBe(expected.text)
|
||||
expect(node.loading).toBe(expected.loading)
|
||||
expect(node.attrs).toEqual(expect.arrayContaining([['href', expected.href]]))
|
||||
expect(node.children).toHaveLength(1)
|
||||
expect(node.children[0].type).toBe(expected.childType)
|
||||
expect(node.children[0].raw).toBe(expected.text)
|
||||
expect(node.children[0].children).toHaveLength(1)
|
||||
expect(node.children[0].children[0].type).toBe('text')
|
||||
expect(node.children[0].children[0].content).toBe(expected.childText)
|
||||
}
|
||||
|
||||
describe('strong link followed by inline content', () => {
|
||||
const md = getMarkdown('strong-link-followed-by-inline-content')
|
||||
|
||||
it('keeps trailing text and the next italic link outside the preceding strong link', () => {
|
||||
const markdown = `[**加粗的链接**](https://example.com)会导致后面的斜体链接解析不正确
|
||||
[*斜体的链接*](https://test.com)`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(3)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://example.com',
|
||||
text: '**加粗的链接**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: '加粗的链接',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: '会导致后面的斜体链接解析不正确\n',
|
||||
raw: '会导致后面的斜体链接解析不正确\n',
|
||||
}))
|
||||
|
||||
expectLinkChild(paragraph.children[2], {
|
||||
href: 'https://test.com',
|
||||
text: '*斜体的链接*',
|
||||
loading: false,
|
||||
childType: 'emphasis',
|
||||
childText: '斜体的链接',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps two inline links separated by the plain text between them', () => {
|
||||
const markdown = `[**加粗的链接**](https://example.com) 或者 [*斜体的链接*](https://test.com)`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(3)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://example.com',
|
||||
text: '**加粗的链接**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: '加粗的链接',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: ' 或者 ',
|
||||
raw: ' 或者 ',
|
||||
}))
|
||||
|
||||
expectLinkChild(paragraph.children[2], {
|
||||
href: 'https://test.com',
|
||||
text: '*斜体的链接*',
|
||||
loading: false,
|
||||
childType: 'emphasis',
|
||||
childText: '斜体的链接',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps trailing punctuation and explanatory text outside the strong link', () => {
|
||||
const markdown = `[**加粗的链接**](https://example.com),这里不是链接部分,但是有链接效果`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://example.com',
|
||||
text: '**加粗的链接**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: '加粗的链接',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: ',这里不是链接部分,但是有链接效果',
|
||||
raw: ',这里不是链接部分,但是有链接效果',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps immediate trailing text outside a strong link even without whitespace', () => {
|
||||
const markdown = `[**加粗的链接**](https://example.com)tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://example.com',
|
||||
text: '**加粗的链接**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: '加粗的链接',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'tail',
|
||||
raw: 'tail',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps completed strong and italic links stable in non-final parsing mode', () => {
|
||||
const markdown = `[**加粗的链接**](https://example.com) 或者 [*斜体的链接*](https://test.com)`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(3)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://example.com',
|
||||
text: '**加粗的链接**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: '加粗的链接',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: ' 或者 ',
|
||||
raw: ' 或者 ',
|
||||
}))
|
||||
|
||||
expectLinkChild(paragraph.children[2], {
|
||||
href: 'https://test.com',
|
||||
text: '*斜体的链接*',
|
||||
loading: false,
|
||||
childType: 'emphasis',
|
||||
childText: '斜体的链接',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps strong links with inline code labels separated from trailing text', () => {
|
||||
const markdown = `[**用 \`pnpm\` 安装**](https://example.com)tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
const link = paragraph.children[0]
|
||||
expect(link.type).toBe('link')
|
||||
expect(link.href).toBe('https://example.com')
|
||||
expect(link.text).toBe('**用 pnpm 安装**')
|
||||
expect(link.loading).toBe(false)
|
||||
expect(link.children).toHaveLength(1)
|
||||
expect(link.children[0].type).toBe('strong')
|
||||
expect(link.children[0].children).toHaveLength(3)
|
||||
expect(link.children[0].children[0]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: '用 ',
|
||||
}))
|
||||
expect(link.children[0].children[1]).toEqual(expect.objectContaining({
|
||||
type: 'inline_code',
|
||||
code: 'pnpm',
|
||||
}))
|
||||
expect(link.children[0].children[2]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: ' 安装',
|
||||
}))
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'tail',
|
||||
raw: 'tail',
|
||||
}))
|
||||
})
|
||||
|
||||
it('preserves title attrs on strong links while keeping trailing text outside the link', () => {
|
||||
const markdown = `[**x**](https://a.com "title")tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
const link = paragraph.children[0]
|
||||
expect(link.type).toBe('link')
|
||||
expect(link.href).toBe('https://a.com')
|
||||
expect(link.title).toBe('title')
|
||||
expect(link.text).toBe('**x**')
|
||||
expect(link.loading).toBe(false)
|
||||
expect(link.attrs).toEqual(expect.arrayContaining([
|
||||
['href', 'https://a.com'],
|
||||
['title', 'title'],
|
||||
]))
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'tail',
|
||||
raw: 'tail',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps adjacent strong-label and code-label links independent', () => {
|
||||
const markdown = `[**x**](https://a.com)[\`y\`](https://b.com)`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://a.com',
|
||||
text: '**x**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: 'x',
|
||||
})
|
||||
|
||||
const codeLink = paragraph.children[1]
|
||||
expect(codeLink.type).toBe('link')
|
||||
expect(codeLink.href).toBe('https://b.com')
|
||||
expect(codeLink.text).toBe('y')
|
||||
expect(codeLink.loading).toBe(false)
|
||||
expect(codeLink.children).toHaveLength(1)
|
||||
expect(codeLink.children[0]).toEqual(expect.objectContaining({
|
||||
type: 'inline_code',
|
||||
code: 'y',
|
||||
raw: 'y',
|
||||
}))
|
||||
})
|
||||
|
||||
it('marks an unfinished strong link URL as loading in non-final mode', () => {
|
||||
const markdown = `[**x**](https://a.com`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: false }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(1)
|
||||
|
||||
const strong = paragraph.children[0]
|
||||
expect(strong.type).toBe('strong')
|
||||
expect(strong.children).toHaveLength(1)
|
||||
|
||||
const link = strong.children[0]
|
||||
expect(link.type).toBe('link')
|
||||
expect(link.loading).toBe(true)
|
||||
expect(link.text).toBe('x')
|
||||
expect(link.children).toHaveLength(1)
|
||||
expect(link.children[0]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'x',
|
||||
raw: 'x',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps triple-emphasis labels separated from trailing text', () => {
|
||||
const markdown = `[***x***](https://a.com)tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
const link = paragraph.children[0]
|
||||
expect(link.type).toBe('link')
|
||||
expect(link.href).toBe('https://a.com')
|
||||
expect(link.text).toBe('*x*')
|
||||
expect(link.loading).toBe(false)
|
||||
expect(link.children).toHaveLength(1)
|
||||
expect(link.children[0].type).toBe('emphasis')
|
||||
expect(link.children[0].children).toHaveLength(1)
|
||||
expect(link.children[0].children[0].type).toBe('strong')
|
||||
expect(link.children[0].children[0].children[0]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'x',
|
||||
}))
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'tail',
|
||||
raw: 'tail',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps html inline labels separated from trailing text', () => {
|
||||
const markdown = `[<span>x</span>](https://a.com)tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
const link = paragraph.children[0]
|
||||
expect(link.type).toBe('link')
|
||||
expect(link.href).toBe('https://a.com')
|
||||
expect(link.text).toBe('<span>x</span>')
|
||||
expect(link.loading).toBe(false)
|
||||
expect(link.children).toHaveLength(1)
|
||||
expect(link.children[0]).toEqual(expect.objectContaining({
|
||||
type: 'html_inline',
|
||||
tag: 'span',
|
||||
content: '<span>x</span>',
|
||||
loading: false,
|
||||
}))
|
||||
expect(link.children[0].children).toHaveLength(1)
|
||||
expect(link.children[0].children[0]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'x',
|
||||
}))
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: 'tail',
|
||||
raw: 'tail',
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps punctuation after a strong link outside the link node', () => {
|
||||
const markdown = `[**x**](https://a.com)!tail`
|
||||
const nodes = parseMarkdownToStructure(markdown, md, { final: true }) as any[]
|
||||
|
||||
expect(nodes).toHaveLength(1)
|
||||
const paragraph = nodes[0]
|
||||
expect(paragraph.type).toBe('paragraph')
|
||||
expect(paragraph.children).toHaveLength(2)
|
||||
|
||||
expectLinkChild(paragraph.children[0], {
|
||||
href: 'https://a.com',
|
||||
text: '**x**',
|
||||
loading: false,
|
||||
childType: 'strong',
|
||||
childText: 'x',
|
||||
})
|
||||
|
||||
expect(paragraph.children[1]).toEqual(expect.objectContaining({
|
||||
type: 'text',
|
||||
content: '!tail',
|
||||
raw: '!tail',
|
||||
}))
|
||||
})
|
||||
})
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user