All pull requests

Pull requests

• perf(word_cache): implement the shrn trick for bitmask

  #2496·SBrandeis opened Oct 7, 2026Collaborator·

  ·
  1

• chore(security) pin dtolnay/rust-toolchain to v1

  #2495·SBrandeis opened Oct 7, 2026Collaborator·

  ·
  1

• chore(deps): bump the actions group across 1 directory with 5 updates

  dependenciesgithub_actions
  #2494·dependabot[bot] opened Oct 7, 2026Bot·

  ·
  1

• (wip) feat(tk-train): refactor and unify

  #2487·SBrandeis opened Oct 2, 2026Collaborator·

  ·
  1

• Allow huggingface_hub 2.0

  #2486·musicinmybrain opened Oct 1, 2026Contributor·

  ·
  3

• fix(added-vocab): keep the original content of normalized added tokens

  #2485·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  5

• fix(bpe): apply end_of_word_suffix to single-character words

  #2484·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  4

• docs: using the crate and its features

  #2481·SBrandeis opened Sep 30, 2026Collaborator·11/11·

  ·
  3

• chore(refactor): move progress bars from tk-encode to tk-train

  #2480·SBrandeis opened Sep 30, 2026Collaborator·10/11·

  ·
  2

• feat(umbrella): expose tk-train through a train feature

  #2479·SBrandeis opened Sep 30, 2026Collaborator·9/11·

  ·
  1

• chore(bindings): use the umbrella tokenizers crate instead of the tk-* sub crates

  #2478·SBrandeis opened Sep 30, 2026Collaborator·8/11·

  ·
  2

• feat(umbrella): rework umbrella crate feature set

  #2477·SBrandeis opened Sep 30, 2026Collaborator·7/11·

  ·
  1

• chore(dx): rework tk-train feature set

  #2476·SBrandeis opened Sep 30, 2026Collaborator·6/11·

  ·
  2

• chore(dx): add tk-train to the cargo workspace

  #2475·SBrandeis opened Sep 30, 2026Collaborator·5/11·

  ·
  2

• fix(tk-train): ParityBpeTrainer: train into Vocab and Merges

  #2474·SBrandeis opened Sep 30, 2026Collaborator·4/11·

  ·
  2

• chore(tk-train): fmt, remove dead code, stale docs, and unused tk-convert dependency

  #2473·SBrandeis opened Sep 30, 2026Collaborator·3/11·

  ·
  4

• fix(tk-encode): remove unicode-normalization-alignments dependency

  #2472·SBrandeis opened Sep 30, 2026Collaborator·2/11·

  ·
  3

• chore(tk-encode): drop itertools, paste and unstable_wasm

  #2471·SBrandeis opened Sep 30, 2026Collaborator·1/11·

  ·
  1

• Support ByteLevel add_prefix_space during canonicalization

  #2462·ducanhnguyen223 opened Sep 27, 2026·

  ·
  1

  4

• Load byte-fallback BPE vocabs that lack some <0xNN> codes

  #2461·daaain opened Sep 26, 2026·

  ·
  2

• chore(cleanup): Remove dead code, rename, simplify

  #2455·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  1

• feat(python): role_to_token

  #2454·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  1

• feat(python): Support encoding pairs of text

  #2453·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  1

• chore(python): Remove unecessary encoding copy at the rust-python boundary

  #2452·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  3

• Load sparse byte-level BPE vocabularies without fallback policy

  #2451·dlowzzxx opened Sep 23, 2026·

  ·