跳转至

Blog Wiki Schema#

Pattern credit: this wiki implements Andrej Karpathy's LLM Wiki — an LLM incrementally builds and maintains a persistent wiki over raw sources.

Instructions for the LLM maintaining this wiki. The wiki is built from the owner's blog archive (blog/WordPress.*.xml, exported to sources/).

Structure#

  • blog/ — original WordPress WXR export. Immutable.
  • sources/posts/<YYYY>/ — one markdown file per post, converted by tools/wxr_to_md.py. Immutable. Do not edit. Frontmatter: title, date, categories, source URL.
  • sources/_catalog.tsv — tab-separated catalog of all posts (date, year, title, categories, words, path, post_id). Regenerated by the converter script. Invariant: one row per file — _catalog.tsv row count must equal the number of sources/posts/**/*.md files. Both gen_index.py and lint_wiki.py fail loudly if a duplicate path appears, because a duplicate row silently inflates every count derived from it (the post total, the per-year table, and overview.md word counts). If a duplicate shows up, re-run tools/wxr_to_md.py rather than hand-editing the TSV.
  • wiki/ — LLM-owned knowledge layer:
  • overview.md — blog-wide stats (posts/year, categories). Auto-generated by tools/gen_index.py.
  • index.md — curated wiki index (manually maintained, links topic pages + key docs). gen_index.py no longer overwrites it.
  • sources.md — auto-generated full catalog of all ingested posts by year. (Previously this file was index.md; split out so index.md can be hand-curated.) Regenerate with python3 tools/gen_index.py.
  • books.md — auto-generated 书目索引: every rsywx.net/books link found in post bodies (HTML anchors and markdown links), with links back to each mentioning post. Data table: sources/_books.tsv. Regenerate with python3 tools/extract_books.py after each sync. （2026-09-19 之前 digest 工具剥掉全部 HTML，书链从未进入综合层。）
  • 生成物一律不入 git：overview.md、sources.md、books.md、 _catalog.tsv、_books.tsv 均在 .gitignore 中，只存在于本机； 本机跑 tools/gen_index.py / tools/extract_books.py（及转换器）再生。 git 仓库只跟踪手工/LLM 维护层（pages/、index.md、schema.md、 log.md）与语料（sources/posts/）、工具、配置。
  • schema.md — this file.
  • 第二语料源：作者的 Obsidian 库（2026-10-04 登记，不在 sources/ 内）
    • 路径：/mnt/ugreen-sync/My Books/Working/My Own Writings Managed by Obsidian （CIFS 网络挂载 //192.168.150.245/Sync，不在 git 里，也不随本仓库分发）。
    • 内容：博客文字/<年>/（博文原稿）、书评/<年>/（书评）、<年>/（当年博文）。
    • 引用规矩与 sources/** 同款：只写纯文本路径，绝不可写成可点击链接， 否则发布后在公开站点上必然 404。这条对库内文件与图片同样适用。
    • 何时该去查它：sources/posts/** 只收已发表的博文， 而只写在库里、没有发成博文的书评（例如 2025 年的 9 篇）不在语料内。 年度书单、#50本书 项目归属一类问题，只能回库核对。
    • 可靠性分级（2026-10-04 实测）：
    • #50本书 标签是最可靠的成员判据。库内共 71 个文件带此标签 （书评/ 67 + 2026年/ 4）。但标签只表「属该项目」，不表「读于当年」—— 例：《奥德赛和史诗之惑》读于 2025、成文于 2026。
    • 图片里的数据取不到：年末读书时间线是 Visme 生成的图片（2021、2022 两篇）， 书单与本数只在图上；文字语料里只剩 <img>。这类内容必须回库开图看。
    • 图片是精选，不是全量：2022 正文自述 33 本、图上仅 8 本。 图上书单不能当年度总数。
    • 书名 grep 不可用于计数：2025 年的《这本书叫什么？》在博文里出现过， 但只是被引作逻辑谜题的出处，并非重读。必须判断有无真实阅读行为。
    • 注意「重看」类标题：2025-11-02《重看安徒生童话》标题写「重看」， 正文却是「新购入」的《八：老槲树的梦》——标题会骗人，正文不会。
  • log.md — chronological record of all wiki operations, 最新条目在最上 （2026-09-19 起改为倒序；新条目插到头部说明之后，旧条目不动）。 条目只写最终完成的结果（改了什么、验证结果、待办）；讨论过程、备选方案、 用户的原始措辞一律不入 log。
  • index.md（仓库根目录）— 对外首页，手工维护层，任何脚本都不得生成或覆盖它。 gen_index.py 只写 overview.md 与 sources.md；build_site.py 只复制此文件。 该页曾载一段献给作者先严、先慈的题献，已于 2026-09-18 应作者要求撤下； 原先配套的两道哨兵（lint_wiki.py 第 7 项、build_site.py 发布前复核）已一并移除。 缘由见 wiki/log.md（2026-09-18）。
  • assets/favicon.ico — 站点图标，取自 https://rsywx.net（字节完全相同的副本， md5 33d080e7c2c8ba4d382f697f501dcddd）。build_site.py 会把它复制到 docs/assets/images/favicon.ico，mkdocs.yml 的 theme.favicon 指向它。 它是发布层的一部分（不是 sources/ 那种本地层）。
  • tools/ — helper scripts (gen_index.py, build_site.py, lint_wiki.py, wxr_to_md.py, sync_posts.py, digest.py).

Conventions for synthesis pages#

  • Filename: wiki/pages/<topic>.md, one topic per page, kebab-case or Chinese titles allowed.
  • Every claim sourced from a post must cite the source as a relative link to the file under sources/posts/... (with post date).
  • YAML frontmatter on every page: title, created, updated, sources.
  • 日期纪律：新建页面时同时写 created 与 updated（两者相同）；日后任何 编辑只更新 updated，created 永不再改。漏 bump 会被 lint_wiki.py （mtime vs updated）拦住。
  • sources = the number of corpus posts in the page's cluster, i.e. how many catalog rows match its category keyword — not the number of citation links in the body. Compute it, don't guess: python3 -c "import csv;rows=list(csv.DictReader(open('sources/_catalog.tsv'),delimiter='\t'));print(sum(1 for r in rows if '游记' in r['categories']))"
  • 分类只是博客的便利标签，不是归属判据（作者，2026-09-20）。一篇文落在哪个主题页 由内容决定，不要被 WordPress 分类牵着走（例：post 9968「我一直是我自己。QED」 分类是「编程、软件、技术」，实质是三公理的自陈与检验，故归入 三公理检视.md / 三公理溯源.md，并从 编程总览 的引文列表中移出）。由此产生一处刻意的不一致： sources: 仍按分类关键字计数（lint 闸门与阶段 B 触发依赖它），而引文列表与正文 归属按内容判断 —— 两者不必相等，这不是疏漏。
  • 同一原则的另一面：分类会漏计。一日一得 系列按标题/内容计 24 篇，但博客 上只有 22 篇带该分类（2008-09-20 厄里斯魔镜...、2008-11-26 上海俚语汇编 两篇被标为 Uncategorized）。该页取 24（内容口径），这是对的。
  • Reference values (2026-10-02, 按分类关键字): 读书 262 · 编程 274 · 游记 52 · 太宗本纪 64 · 电影、音乐、游戏、娱乐 81；一日一得 24（内容口径，见上）。 Pages synthesizing several categories (e.g. 生活 / 人生的下半场 / 蛋壳与利他) list the count of posts actually cited.
  • A hub for the combined category 电影、音乐、游戏、娱乐 must be titled with that full string (电影、音乐、游戏、娱乐总览) so the check has a keyword to match — titling it 音乐总览 would wrongly match all 81 posts, since the blog stores music/film/games as one category rather than three.
  • A "local citations" block lists the posts cited on the page; build_site.py rewrites those links to blog URLs at publish time.
  • 书目类事实只能来自书目库，不能来自分类或常识。 博主有全量藏书目录 https://rsywx.net/books/<id>.html：著者 / 译者 / 出版社 / 出版日期 / ISBN 一律以该页为准。书号不必等人给 —— 博文正文里本来就带着链接， tools/extract_books.py 已把「书目 → 书号 → 提及该书的日志」汇总为 sources/_books.tsv 与 wiki/books.md（374 条；生成物，不入 git）。 默认动作是：先用索引定位书号，再开书页核对。
  • 两个来源不可互代：引号里的原文只能回查语料；书目元数据只能回查书目库。 语料没写某本书的出版社，不等于可以照常识补。实例（2026-09-20）： 《凯恩斯传：一个利他主义者的七面人生》被照常识记为 Skidelsky（中信）， 而书目库与博文均指明为 Richard Davenport-Hines（电子工业出版社）。 「读起来完全通顺」的错句最难发现，书目表格尤甚。
  • 书目库没有「原名」字段（英文原版记录也只列标题与著者）：原文书名只能取自 博文，博文没写就留空，不要推。
  • _books.tsv 只收 rsywx.net/books/<id>.html 形式的链接。博客早期用的是 rsywx.com/doku.php 页面（如《911调查委员会报告》《上帝创造了整数》）—— 不在索引里不是错误，只是没有书目记录。
  • Numeric claims must be verified, not estimated. Word counts, per-year post counts and totals are all in sources/_catalog.tsv — check them before publishing (e.g. grep '编程' sources/_catalog.tsv | cut -f2 | uniq -c). Do not invent titles: every post named in a timeline must exist in the catalog.
  • 「可枚举」内容必须逐条核，不得凭印象写（2026-09-20，作者指出，本 wiki 最严重的一次 失败）：对我影响最大的十本书 的书单表整张是编的 —— 10 行里只有 3 行对得上原文， 没有时间的世界 两篇原文根本没出现过却占了第 1 与第 7 位，基地长老会大设计 甚至不是书名（是一篇日志的标题）。每一行都带着可用的引文链接，所以 lint_wiki.py 全绿、mkdocs --strict 全绿、句子读起来通顺 —— 链接能通、语义全假。
  • 适用对象：书单、时间线、案例清单、卷次、人物表 —— 凡可以逐条清点的东西。
  • 口径：回到原文自己的小标题／段落逐条比对条目与顺序，不能靠"读着像"； 引号里的话必须能在语料里逐字 grep 到（tools/audit_quotes.py）。
  • 扩散性：同一批伪内容会顺着引文链长到别的页面上（本次波及 5 页：此书单页之外的 哥德尔-数学-时间、科幻与公理体系、卡尔维诺、读书）。发现一处即顺链全查。
  • Cross-link related pages with markdown links.
  • Citations to source posts use a page-relative link to the markdown file, e.g. https://blog.rsywx.net/2007/04/14/central-kingdom/ from wiki/pages/. tools/build_site.py rewrites every such link to the post's original blog URL on publish (regex (?:\.\./)+sources/(posts/[^)]+\.md)); locally the markdown link still resolves to the repo file.

Workflows#

Ingest (batch)#

  1. Run python3 tools/gen_index.py to refresh overview/sources.md after new posts are converted into sources/posts/.
  2. Read the catalog slice assigned (e.g., one year or one category).
  3. Read the actual post files; write/update synthesis pages:
  4. Year pages wiki/pages/year-<YYYY>.md
  5. Category/topic pages (读书, 编程, 游记, 太宗本纪, canada, ...)
  6. Recurring entity pages (books, technologies, people, places) as they emerge.
  7. Update wiki/index.md links where needed; append an entry to log.md: ## [YYYY-MM-DD] ingest | <scope>.

Sync (incremental) — two stages, and the split matters#

The blog is live at blog.rsywx.net (WordPress, a different machine from the wiki VPS). New posts arrive over time, so the archive is kept current by tools/sync_posts.py, which is deliberately split:

Stage A — python3 tools/sync_posts.py (automatable, cron-safe). Fetches only. It: 1. reads the date cursor from sources/_sync_state.json (falling back to the newest catalog date) and queries /wp-json/wp/v2/posts?after=<cursor - window>; 2. skips any post_id already in the catalog (idempotent — safe to re-run); 3. writes new posts through wxr_to_md.render_markdown(), so the REST and WXR ingest paths emit byte-identical markdown. Never fork that renderer; 4. appends to _catalog.tsv, updates the state file, writes a worklist to inbox/<YYYY-MM-DD>.md, then runs the validation gate.

Flags: --dry-run (change nothing), --no-build, --window N (re-scan window, default 30 days, so back-dated publishing isn't missed).

Stage B — synthesis (agent only, never cron). The tool does not write synthesis prose. Automation is safe for bookkeeping and unsafe for synthesis: lint catches broken links and known typos, and the 引文门禁 (audit_quotes.py --check) catches quotations that are nowhere in the corpus — but neither can catch a fluent, plausible, false claim. This wiki has shipped an invented timeline entry, a page describing content its source does not contain, and (2026-09-20) a wholly invented book list whose every row carried a working citation link; that last failure is why the quote gate exists. So stage B is a session that reads inbox/*.md, updates the affected hub pages, verifies every numeric claim against _catalog.tsv, and — for any quotation or any enumerable (list, table, timeline) — re-reads the source posts and checks item by item, then logs and rebuilds.

Expected friction: when a new post lands in a hub's category, that hub's sources: count is immediately stale, so lint fails and the gate stops before the site build. That is working as designed — fix the hub (stage B) rather than suppressing the check. Deploy is withheld meanwhile so the published wiki never contradicts itself.

Conventions specific to incremental ingest:

  • New files are named from the WordPress slug, not the title: <date>-<slug>.md, falling back to the title slug if the WP slug is percent-encoded/non-ASCII, and to -<post_id> on collision. (The WP slug is already unique per post, which is exactly why the historical title-slug collision — see "Ingest dedupe" — cannot recur for new posts.) Existing files are not renamed.
  • sources/ stays append-only. Upstream edits are reported in the worklist, never written back.
  • sources/_sync_state.json — date cursor plus a post_id -> modified_gmt map. The map is the only way to notice that a post was edited upstream (after= filters on publication date, so an old post edited today never appears in that query; modified_after= is queried separately for this). Baselines are learned lazily: a post backfilled from the WXR has no recorded modified_gmt, so the first time it shows up as modified it is reported as "moved (baseline recorded)" and only the next change to it can be called confirmed drift. Do not bulk-seed the map — paginating all 1650+ posts is slow and was observed to hang mid-run.
  • inbox/ — not published; it is a machine work queue, not knowledge.

Daily cron (adjust paths; the venv python must be absolute):

17 6 * * * cd /home/tr/projects/llm-wiki && \
  /home/tr/projects/llm-wiki/.venv/bin/python3 tools/sync_posts.py \
  >> /home/tr/projects/llm-wiki/inbox/cron.log 2>&1

A non-zero exit means the wiki needs attention (either the fetch failed or a hub is stale), which is what you want a cron job to tell you.

Query#

  1. Read wiki/index.md (and overview.md) to locate relevant pages.
  2. Read those pages, drill into sources when needed, answer with citations.
  3. If the answer is worth keeping (comparisons, analyses, timelines), file it as a new page under wiki/pages/ and log it: ## [YYYY-MM-DD] query | <question>.

Lint#

Periodically check: orphan pages, missing cross-references, stale claims, uncategorized posts worth classifying, contradictions. Log as ## [YYYY-MM-DD] lint | <notes>.

Automated checks that must pass before finishing any batch of page edits:

  1. python3 tools/gen_index.py — refresh overview.md / sources.md.
  2. python3 tools/build_site.py — runs mkdocs build --strict; must exit 0 and report 0 left unresolved citation links. Also verifies every wiki/*.md path referenced in mkdocs.yml nav exists.
  3. python3 tools/lint_wiki.py — must exit 0. Checks that every sources/posts/... citation and every *.md internal link resolves on disk, that nav targets exist, that no {{ / TODO / placeholder markers are left behind, that each <category>总览 hub page's sources: count matches the catalog, and that _catalog.tsv has no duplicate or dangling paths, and that no known homoglyph / transcription typos (TYPO_PATTERNS in the script, e.g. 书名《没有时间的世界》被误写成 含日文假名"の"的错例) appear in the synthesis layer — new recurring typos should be appended there.
  4. python3 tools/audit_quotes.py --check — must exit 0. 引文来源门禁：页面上每一处 “…”／「…」 都要能在 sources/posts/** 里找到（比对前去标点、去 HTML 标签与链接标记， …… 截断的多段各自命中即可）。这是唯一能抓到“链接全通、句子全假”的自动检查。 结论非零有两种情形：出现未豁免的可疑引文，或 allowlist 里出现过期条目。 豁免写在 tools/audit_allowlist.txt，只收两类：作者在审阅对话中给出的口径／数字 （按作者裁定保持原样），以及界面文案与构建日志；新增豁免前必须先把该引文回原文核一遍， 确属伪造的改页面而不是加清单。

四个门禁互不替代，都要跑#

四个检查各自针对不同层次，各能抓到另两个抓不到的错误（2026-09-18 实测， 其中两条是我自己改出来的；第四道是 2026-09-20 补上的）：

门禁 检查对象 抓不到什么
lint_wiki.py 仓库内部一致性（链接、标记、计数） 指向仓库专用文件的链接（本地存在，发布后 404）；句子真假
audit_quotes.py --check 引文的可回查性（每处引言都能在语料里找到） 非引文断言（数字、年份、谁写了什么）；轻微改写与张冠李戴（需回原文读）
mkdocs build --strict 发布产物的可达性 正文内容问题；且不校验 theme.logo/theme.favicon 是否存在
build_site.py 的资产引用检查 mkdocs.yml 与暂存树是否一致 内容问题

为什么需要第三个：MkDocs 在 theme.logo 指向不存在的文件时不报错， 照样输出一个指向 404 的 <img>（实测：--strict 退出码仍为 0）。 该检查会解析 mkdocs.yml 的 theme.favicon、theme.logo 与 extra_css， 逐个断言其已存在于 docs/，否则报错中止。

硬规则：凡指向仓库专用文件（sources/**、tools/**、blog/**、 _catalog.tsv 等）的路径，一律写成纯文本（反引号），不要写成可点击链接。 只 stage wiki/ + 根 index.md + assets/，所以这些路径在公开站点上必然 404； 而 build_site.py 只改写 sources/posts/*.md 形式的引文链接。

资产路径约定：仓库里的 assets/<x> 会镜像复制到 docs/assets/<x>， 因此 mkdocs.yml 里直接写 assets/<x>（不要再写 assets/images/...）。 新增资产时要同时改两处：build_site.py 的资产清单与 mkdocs.yml 的引用； 漏改会被资产引用检查当场拦住。

相对路径：从 wiki/pages/<页>.md 指向 wiki/ 下的文件（index.md、 overview.md）要写 ../index.md；指向同级主题页写 读书.md（不带路径）。

Ingest dedupe (why 1652, not 1653)#

The converter derives each output filename from wp:post_date_gmt + a slug of the title, so two distinct WXR items can collide on one path. The blog has one such case: 《老彼得五上奖状》 was published twice (post_id 2941 at 2012/01/04/, post_id 2943 at 2012/01/05/...-2/, identical 464-char body) and both land on GMT 2012-01-04. wxr_to_md.py resolves collisions explicitly:

  • identical body → the duplicate is dropped, the first (earliest) item kept. This is the 1652 case, and the kept file's source: is the original URL, not the -2 duplicate;
  • different body → the file is disambiguated with a -<post_id> suffix and never silently overwritten.

So the honest post count is the file count, which was 1652 before the first incremental sync and 1654 after it (verified 2026-09-18). Never quote a count taken from _catalog.tsv line count without checking that it agrees with find sources/posts -name '*.md' | wc -l.

Notes#

  • Posts are mostly Chinese; write synthesis pages in Chinese unless asked otherwise.
  • ~950 posts are "Uncategorized" — infer topics from content when synthesizing.
  • Content is raw WordPress HTML; strip/skip markup noise when summarizing.
  • Ingest in batches (one year or one category at a time). Stay incremental; the wiki compounds.