feat(desktop): embed BoosAPI Desktop client as a subproject
Vendor the rebranded BoosAPI Desktop client (Tauri + Vue) into metapi as a monorepo subdirectory under desktop/. Source only — node_modules, dist, Cargo target and the upstream .git history are excluded. desktop/ keeps its own independent Cargo workspace (desktop/src-tauri with crates/*); it is not part of the backend boosapi crate, so a `cargo check` at the metapi root still compiles only the backend. Build the desktop app from desktop/src-tauri per desktop/README.md. Note: app icons are still placeholder assets pending the final BoosAPI logo. The updater signing keys live outside the repo and are not included.
This commit is contained in:
boos
@@ -0,0 +1,527 @@
|
||||
# stream-markdown-parser
|
||||
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](README.zh-CN.md)
|
||||
[](https://www.npmjs.com/package/stream-markdown-parser)
|
||||
[](https://bundlephobia.com/package/stream-markdown-parser)
|
||||
[](https://github.com/Simon-He95/markstream-vue/blob/main/license)
|
||||
|
||||
Pure markdown parser and renderer utilities with streaming support - framework agnostic.
|
||||
|
||||
This package contains the core markdown parsing logic extracted from `markstream-vue`, making it usable in any JavaScript/TypeScript project without Vue dependencies.
|
||||
|
||||
## Features
|
||||
|
||||
- 🚀 **Pure JavaScript** - No framework dependencies
|
||||
- 📦 **Lightweight** - Minimal bundle size
|
||||
- 🔧 **Extensible** - Plugin-based architecture
|
||||
- 🎯 **Type-safe** - Full TypeScript support
|
||||
- ⚡ **Fast** - Optimized for performance
|
||||
- 🌊 **Streaming-friendly** - Progressive parsing support
|
||||
|
||||
> ℹ️ We now build on top of [`markdown-it-ts`](https://www.npmjs.com/package/markdown-it-ts), a TypeScript-first distribution of markdown-it. The API stays the same, but we only rely on its parsing pipeline and ship richer typings for tokens and hooks.
|
||||
|
||||
## Documentation
|
||||
|
||||
The full usage guide lives alongside the markstream-vue docs:
|
||||
|
||||
- English: https://markstream-vue-docs.simonhe.me/guide/api
|
||||
- 中文: https://markstream-vue-docs.simonhe.me/zh/guide/api
|
||||
|
||||
This README highlights the parser-specific APIs; visit the docs for end-to-end integration tutorials (VitePress, workers, Tailwind, troubleshooting, etc.).
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pnpm add stream-markdown-parser
|
||||
# or
|
||||
npm install stream-markdown-parser
|
||||
# or
|
||||
yarn add stream-markdown-parser
|
||||
```
|
||||
|
||||
## Quick API (TL;DR)
|
||||
|
||||
- `getMarkdown(msgId?, options?)` — create a `markdown-it-ts` instance with built-in plugins (task lists, sub/sup, math helpers, etc.). Also accepts `plugin`, `apply`, and `i18n` options.
|
||||
- `registerMarkdownPlugin(plugin)` / `clearRegisteredMarkdownPlugins()` — add or remove global plugins that run for every `getMarkdown()` call (useful for feature flags or tests).
|
||||
- `parseMarkdownToStructure(markdown, md, parseOptions)` — convert Markdown into the streaming-friendly AST consumed by `markstream-vue` and other renderers.
|
||||
- `processTokens(tokens)` / `parseInlineTokens(children, content?, preToken?, options?)` — low-level helpers if you want to bypass the built-in AST pipeline.
|
||||
- `applyMath`, `applyContainers`, `normalizeStandaloneBackslashT`, `findMatchingClose`, etc. — utilities for custom parsing or linting workflows.
|
||||
|
||||
## Usage
|
||||
|
||||
### Streaming-friendly pipeline
|
||||
|
||||
```
|
||||
Markdown string
|
||||
↓ getMarkdown() → markdown-it-ts instance with plugins
|
||||
parseMarkdownToStructure(markdown, md) → AST (ParsedNode[])
|
||||
↓ feed into your renderer (markstream-vue, custom UI, workers)
|
||||
```
|
||||
|
||||
Reuse the same `md` instance when parsing multiple documents—plugin setup is the heaviest step. When integrating with [`markstream-vue`](https://www.npmjs.com/package/markstream-vue), pass the AST to `<MarkdownRender :nodes="nodes" />` or supply raw `content` and hand it the same parser options.
|
||||
|
||||
### Incremental / streaming example
|
||||
|
||||
When consuming an AI or SSE stream you can keep appending to a buffer, parse with the same `md` instance, and send the AST to the UI (e.g., `markstream-vue`) on every chunk:
|
||||
|
||||
```ts
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown()
|
||||
let buffer = ''
|
||||
|
||||
async function handleChunk(chunk: string) {
|
||||
buffer += chunk
|
||||
const nodes = parseMarkdownToStructure(buffer, md)
|
||||
postMessage({ type: 'markdown:update', nodes })
|
||||
}
|
||||
```
|
||||
|
||||
In the UI layer, render `nodes` with `<MarkdownRender :nodes="nodes" />` to avoid re-parsing. See the [docs usage guide](https://markstream-vue-docs.simonhe.me/guide/usage) for end-to-end wiring.
|
||||
|
||||
### Basic example
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
// Create a markdown-it-ts instance with default plugins
|
||||
const md = getMarkdown()
|
||||
|
||||
// Parse markdown to our streaming-friendly AST structure
|
||||
const nodes = parseMarkdownToStructure('# Hello World', md)
|
||||
console.log(nodes)
|
||||
// [{ type: 'heading', level: 1, children: [...] }]
|
||||
|
||||
// markdown-it-ts still exposes render() if you need HTML output,
|
||||
// but this package now focuses on the token -> AST pipeline.
|
||||
const html = md.render?.('# Hello World\n\nThis is **bold**.')
|
||||
```
|
||||
|
||||
### With Math Options
|
||||
|
||||
```typescript
|
||||
import { getMarkdown, setDefaultMathOptions } from 'stream-markdown-parser'
|
||||
|
||||
// Set global math options
|
||||
setDefaultMathOptions({
|
||||
commands: ['infty', 'perp', 'alpha'],
|
||||
escapeExclamation: true
|
||||
})
|
||||
|
||||
const md = getMarkdown()
|
||||
```
|
||||
|
||||
### With Custom i18n
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
// Using translation map
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: {
|
||||
'common.copy': '复制',
|
||||
}
|
||||
})
|
||||
|
||||
// Or using a translation function
|
||||
const md = getMarkdown('editor-1', {
|
||||
i18n: (key: string) => translateFunction(key)
|
||||
})
|
||||
```
|
||||
|
||||
### With Plugins
|
||||
|
||||
```typescript
|
||||
import customPlugin from 'markdown-it-custom-plugin'
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
plugin: [
|
||||
[customPlugin, { /* options */ }]
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### Advanced: Custom Rules
|
||||
|
||||
```typescript
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown('editor-1', {
|
||||
apply: [
|
||||
(md) => {
|
||||
// Add custom inline rule
|
||||
md.inline.ruler.before('emphasis', 'custom', (state, silent) => {
|
||||
// Your custom logic
|
||||
return false
|
||||
})
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### Extending globally
|
||||
|
||||
Need to add a plugin everywhere without touching each call site? Use the helper exports:
|
||||
|
||||
```ts
|
||||
import {
|
||||
clearRegisteredMarkdownPlugins,
|
||||
registerMarkdownPlugin,
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
registerMarkdownPlugin(myPlugin)
|
||||
|
||||
const md = getMarkdown()
|
||||
// md now has `myPlugin` enabled in addition to anything passed via options
|
||||
|
||||
// For tests or teardown flows:
|
||||
clearRegisteredMarkdownPlugins()
|
||||
```
|
||||
|
||||
- `plugin` option → array of `md.use` invocations scoped to a single `getMarkdown` call.
|
||||
- `apply` option → imperatively mutate the instance (`md.inline.ruler.before(...)`). Wrap in try/catch if you need to surface errors differently; the helper logs to console to preserve legacy behaviour.
|
||||
- `registerMarkdownPlugin` → global singleton registry (handy in SSR or worker contexts where you want feature flags to apply everywhere).
|
||||
|
||||
## API
|
||||
|
||||
### Main Functions
|
||||
|
||||
#### `getMarkdown(msgId?, options?)`
|
||||
|
||||
Creates a configured `markdown-it-ts` instance (API-compatible with markdown-it).
|
||||
|
||||
**Parameters:**
|
||||
- `msgId` (string, optional): Unique identifier for this instance. Default: `editor-${Date.now()}`
|
||||
- `options` (GetMarkdownOptions, optional): Configuration options
|
||||
|
||||
**Options:**
|
||||
```typescript
|
||||
interface GetMarkdownOptions {
|
||||
// Array of markdown-it/markdown-it-ts plugins to use
|
||||
plugin?: Array<Plugin | [Plugin, any]>
|
||||
|
||||
// Array of functions to mutate the md instance
|
||||
apply?: Array<(md: MarkdownIt) => void>
|
||||
|
||||
// Translation function or translation map
|
||||
i18n?: ((key: string) => string) | Record<string, string>
|
||||
}
|
||||
```
|
||||
|
||||
#### `parseMarkdownToStructure(content, md, options?)`
|
||||
|
||||
Parses markdown content into a structured node tree.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): The markdown content to parse
|
||||
- `md` (MarkdownItCore): A markdown-it-ts instance created with `getMarkdown()`
|
||||
- `options` (ParseOptions, optional): Parsing options with hooks
|
||||
|
||||
**Returns:** `ParsedNode[]` - An array of parsed nodes
|
||||
|
||||
#### `processTokens(tokens)`
|
||||
|
||||
Processes raw markdown-it tokens into a flat array.
|
||||
|
||||
#### `parseInlineTokens(tokens, content?, preToken?, options?)`
|
||||
|
||||
Parses inline markdown-it-ts tokens into renderer nodes. Pass the inline token array plus the optional raw `content` string (from the parent token), an optional previous token, and inline parse options (`requireClosingStrong`, `customHtmlTags`, `validateLink`).
|
||||
|
||||
### Configuration Functions
|
||||
|
||||
#### `setDefaultMathOptions(options)`
|
||||
|
||||
Set global math rendering options.
|
||||
|
||||
**Parameters:**
|
||||
- `options` (MathOptions): Math configuration options
|
||||
|
||||
```typescript
|
||||
interface MathOptions {
|
||||
commands?: readonly string[] // LaTeX commands to escape
|
||||
escapeExclamation?: boolean // Escape standalone '!' (default: true)
|
||||
}
|
||||
```
|
||||
|
||||
### Parse hooks (fine-grained transforms)
|
||||
|
||||
ParseOptions supports the following hooks and flags:
|
||||
|
||||
```ts
|
||||
interface ParseOptions {
|
||||
preTransformTokens?: (tokens: Token[]) => Token[]
|
||||
postTransformTokens?: (tokens: Token[]) => Token[]
|
||||
// Custom HTML-like tags to emit as custom nodes (e.g. ['thinking'])
|
||||
// Use tag-like names such as 'thinking', 'answer-box', or 'my_component'.
|
||||
customHtmlTags?: string[]
|
||||
// Validate link href before emitting a `link` node; false -> plain text
|
||||
validateLink?: (url: string) => boolean
|
||||
// When true, treats the input as complete (end-of-stream)
|
||||
final?: boolean
|
||||
// Require closing `**` for strong parsing (default: false)
|
||||
requireClosingStrong?: boolean
|
||||
}
|
||||
```
|
||||
|
||||
Example — flag AI “thinking” blocks:
|
||||
|
||||
```ts
|
||||
const parseOptions = {
|
||||
customHtmlTags: ['thinking'],
|
||||
}
|
||||
|
||||
const nodes = parseMarkdownToStructure(markdown, md, parseOptions)
|
||||
const tagged = nodes.map(node =>
|
||||
node.type === 'html_block' && /<thinking>/.test((node as any).content ?? '')
|
||||
? { ...node, meta: { type: 'thinking' } }
|
||||
: node,
|
||||
)
|
||||
```
|
||||
|
||||
Use the metadata in your renderer to show custom UI without mangling the original Markdown.
|
||||
|
||||
Example — enforce safe link protocols:
|
||||
|
||||
```ts
|
||||
const md = getMarkdown('safe-links')
|
||||
md.set?.({
|
||||
validateLink: (url: string) => !/^\s*javascript:/i.test(url.trim()),
|
||||
})
|
||||
|
||||
const nodes = parseMarkdownToStructure(
|
||||
'[ok](https://example.com) [bad](javascript:alert(1))',
|
||||
md,
|
||||
{ final: true },
|
||||
)
|
||||
// "ok" stays a link node; "bad" is downgraded to plain text
|
||||
```
|
||||
|
||||
### Unknown HTML-like tags
|
||||
|
||||
By default, non-standard HTML-like tags (for example `<question>`) are rendered as raw HTML elements once they are complete. Incomplete or malformed fragments stay as **literal text** so they do not swallow surrounding content during streaming or final renders. If you want them emitted as custom nodes (`type: 'question'` with parsed attrs/content), opt in via `customHtmlTags`.
|
||||
|
||||
### Utility Functions
|
||||
|
||||
#### `isMathLike(content)`
|
||||
|
||||
Heuristic function to detect if content looks like mathematical notation.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): Content to check
|
||||
|
||||
**Returns:** `boolean`
|
||||
|
||||
#### `findMatchingClose(src, startIdx, open, close)`
|
||||
|
||||
Find the matching closing delimiter in a string, handling nested pairs.
|
||||
|
||||
**Parameters:**
|
||||
- `src` (string): Source string
|
||||
- `startIdx` (number): Start index to search from
|
||||
- `open` (string): Opening delimiter
|
||||
- `close` (string): Closing delimiter
|
||||
|
||||
**Returns:** `number` - Index of matching close, or -1 if not found
|
||||
|
||||
## Tips & troubleshooting
|
||||
|
||||
- **Reuse parser instances**: cache `getMarkdown()` results per worker/request to avoid re-registering plugins.
|
||||
- **Server-side parsing**: run `parseMarkdownToStructure` on the server, ship the AST to the client, and hydrate with `markstream-vue` for deterministic output.
|
||||
- **Custom HTML widgets**: pre-extract `<MyWidget>` blocks before parsing (replace with placeholders) and reinject them during rendering instead of mutating `html_block` nodes post-parse.
|
||||
- **Styling**: when piping nodes into `markstream-vue`, follow the docs [CSS checklist](https://markstream-vue-docs.simonhe.me/guide/troubleshooting#css-looks-wrong-start-here) so Tailwind/UnoCSS don’t override library styles.
|
||||
- **Error handling**: the `apply` hook swallows exceptions to maintain backwards compatibility. If you want strict mode, wrap your mutators before passing them in and rethrow/log as needed.
|
||||
|
||||
#### `parseFenceToken(token)`
|
||||
|
||||
Parse a code fence token into a CodeBlockNode.
|
||||
|
||||
**Parameters:**
|
||||
- `token` (MarkdownToken): markdown-it token
|
||||
|
||||
**Returns:** `CodeBlockNode`
|
||||
|
||||
#### `normalizeStandaloneBackslashT(content, options?)`
|
||||
|
||||
Normalize backslash-t sequences in math content.
|
||||
|
||||
**Parameters:**
|
||||
- `content` (string): Content to normalize
|
||||
- `options` (MathOptions, optional): Math options
|
||||
|
||||
**Returns:** `string`
|
||||
|
||||
### Lower-level helpers
|
||||
|
||||
If you need full control over how tokens are transformed, you can import the primitive builders directly:
|
||||
|
||||
```ts
|
||||
import type { MarkdownToken } from 'stream-markdown-parser'
|
||||
import {
|
||||
|
||||
parseInlineTokens,
|
||||
processTokens
|
||||
} from 'stream-markdown-parser'
|
||||
|
||||
const tokens: MarkdownToken[] = md.parse(markdown, {})
|
||||
const nodes = processTokens(tokens)
|
||||
// or operate at inline granularity:
|
||||
const inlineNodes = parseInlineTokens(tokens[0].children ?? [], tokens[0].content ?? '')
|
||||
```
|
||||
|
||||
`processTokens` is what `parseMarkdownToStructure` uses internally, so you can remix the AST pipeline without reimplementing the Markdown-it loop.
|
||||
|
||||
### Plugin Functions
|
||||
|
||||
#### `applyMath(md, options?)`
|
||||
|
||||
Apply math plugin to markdown-it instance.
|
||||
|
||||
**Parameters:**
|
||||
- `md` (MarkdownIt): markdown-it instance
|
||||
- `options` (MathOptions, optional): Math rendering options
|
||||
|
||||
#### `applyContainers(md)`
|
||||
|
||||
Apply container plugins to markdown-it instance.
|
||||
|
||||
**Parameters:**
|
||||
- `md` (MarkdownIt): markdown-it instance
|
||||
|
||||
### Constants
|
||||
|
||||
#### `KATEX_COMMANDS`
|
||||
|
||||
Array of common KaTeX commands for escaping.
|
||||
|
||||
#### `TEX_BRACE_COMMANDS`
|
||||
|
||||
Array of TeX commands that use braces.
|
||||
|
||||
#### `ESCAPED_TEX_BRACE_COMMANDS`
|
||||
|
||||
Escaped version of TEX_BRACE_COMMANDS for regex use.
|
||||
|
||||
## Types
|
||||
|
||||
All TypeScript types are exported:
|
||||
|
||||
```typescript
|
||||
import type {
|
||||
// Node types
|
||||
CodeBlockNode,
|
||||
GetMarkdownOptions,
|
||||
HeadingNode,
|
||||
ListItemNode,
|
||||
ListNode,
|
||||
MathOptions,
|
||||
ParagraphNode,
|
||||
ParsedNode,
|
||||
ParseOptions,
|
||||
// ... and more
|
||||
} from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
### Node Types
|
||||
|
||||
The parser exports various node types representing different markdown elements:
|
||||
|
||||
- `TextNode`, `HeadingNode`, `ParagraphNode`
|
||||
- `ListNode`, `ListItemNode`
|
||||
- `CodeBlockNode`, `InlineCodeNode`
|
||||
- `LinkNode`, `ImageNode`
|
||||
- `BlockquoteNode`, `TableNode`
|
||||
- `MathBlockNode`, `MathInlineNode`
|
||||
- And many more...
|
||||
|
||||
## Default Plugins
|
||||
|
||||
This package comes with the following markdown-it plugins pre-configured:
|
||||
|
||||
- `markdown-it-sub` - Subscript support (`H~2~O`)
|
||||
- `markdown-it-sup` - Superscript support (`x^2^`)
|
||||
- `markdown-it-mark` - Highlight/mark support (`==highlighted==`)
|
||||
- `markdown-it-task-checkbox` - Task list support (`- [ ] Todo`)
|
||||
- `markdown-it-ins` - Insert tag support (`++inserted++`)
|
||||
- `markdown-it-footnote` - Footnote support
|
||||
- `markdown-it-container` - Custom container support (`::: warning`, `::: tip`, etc.)
|
||||
- Math support - LaTeX math rendering with `$...$` and `$$...$$`
|
||||
|
||||
## Framework Integration
|
||||
|
||||
While this package is framework-agnostic, it's designed to work seamlessly with:
|
||||
|
||||
- ✅ **Node.js** - Server-side rendering
|
||||
- ✅ **Vue 3** - Use with `markstream-vue` (or your own renderer)
|
||||
- ✅ **React** - Use parsed nodes for custom rendering
|
||||
- ✅ **Vanilla JS** - Direct HTML rendering
|
||||
- ✅ **Any framework** - Parse to AST and render as needed
|
||||
|
||||
### Example — dedicated worker feeding markstream-vue
|
||||
|
||||
Offload parsing to a Web Worker while the UI renders via `markstream-vue`:
|
||||
|
||||
```ts
|
||||
// worker.ts
|
||||
import { getMarkdown, parseMarkdownToStructure } from 'stream-markdown-parser'
|
||||
|
||||
const md = getMarkdown()
|
||||
let buffer = ''
|
||||
|
||||
globalThis.addEventListener('message', (event) => {
|
||||
if (event.data.type === 'chunk') {
|
||||
buffer += event.data.value
|
||||
const nodes = parseMarkdownToStructure(buffer, md)
|
||||
globalThis.postMessage({ type: 'update', nodes })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
```ts
|
||||
// ui.ts
|
||||
const worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' })
|
||||
worker.addEventListener('message', (event) => {
|
||||
if (event.data.type === 'update')
|
||||
nodes.value = event.data.nodes
|
||||
})
|
||||
```
|
||||
|
||||
```vue
|
||||
<MarkdownRender :nodes="nodes" />
|
||||
```
|
||||
|
||||
This pattern keeps Markdown-it work off the main thread and lets you reuse the same AST in any framework.
|
||||
|
||||
## Migration from `markstream-vue` (parser exports)
|
||||
|
||||
If you're currently importing parser helpers from `markstream-vue`, you can switch to the dedicated package:
|
||||
|
||||
```typescript
|
||||
// before
|
||||
import { getMarkdown } from 'markstream-vue'
|
||||
|
||||
// after
|
||||
import { getMarkdown } from 'stream-markdown-parser'
|
||||
```
|
||||
|
||||
All APIs remain the same. See the [migration guide](https://markstream-vue-docs.simonhe.me/monorepo-migration) for details.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Lightweight**: ~65KB minified (13KB gzipped)
|
||||
- **Fast**: Optimized for real-time parsing
|
||||
- **Tree-shakeable**: Only import what you need
|
||||
- **Few dependencies**: `markdown-it-ts` + a small set of markdown-it plugins
|
||||
|
||||
## Contributing
|
||||
|
||||
Issues and PRs welcome! Please read the [contribution guidelines](https://github.com/Simon-He95/markstream-vue/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT © Simon He
|
||||
|
||||
## Related
|
||||
|
||||
- [markstream-vue](https://www.npmjs.com/package/markstream-vue) - Full-featured Vue 3 Markdown renderer
|
||||
Reference in New Issue
Block a user