What's Changed

• chore: Update email by @Pringled in #43
• Migrate crate to Rust 2024 edition by @seuros in #44
• Avoid serializing tokenizer to read unknown-token metadata by @seuros in #46
• Share immutable model state across clones by @seuros in #47
• Make Clap CLI opt-in with a cli feature by @seuros in #48
• Deduplicate model construction and reuse library in CLI by @seuros in #49
• Deduplicate float decoding and model layout candidates by @seuros in #50
• fix: Drop unknown token for Unigram tokenizers, prepare release by @Pringled in #52

New Contributors

• @seuros made their first contribution in #44

Breaking changes

• The CLI is now opt-in. Clap is no longer a default dependency. To install the binary, use:

  cargo install model2vec-rs --features cli
  

Full Changelog: v0.2.1...v0.3.0

What's Changed

• chore: Update citation, exclude assets by @Pringled in #39
• Support I64 token mappings in safetensors model loading by @maxprogrammer007 in #41
• chore: Bump version by @Pringled in #42

New Contributors

• @maxprogrammer007 made their first contribution in #41

Full Changelog: v0.2.0...v0.2.1

What's Changed

• feat: Add from_borrowed() constructor by @zharinov in #33
• Add in-memory model loading API by @jolestar in #35
• Split optional hf-hub loading feature by @jolestar in #36
• Add minimal wasm and local-only feature support by @jolestar in #37
• fix: Cover local-only and wasm feature matrix in CI by @Pringled in #38
• feat: Improve loading, support more formats by @Pringled in #34

New Contributors

• @zharinov made their first contribution in #33
• @jolestar made their first contribution in #35

Full Changelog: v0.1.4...v0.2.0

What's Changed

• feat: add clone and debug to struct by @ZimboPro in #31
• feat: Add vocabulary quantization support by @Pringled in #30
• chore: Bumped version by @Pringled in #32

New Contributors

• @ZimboPro made their first contribution in #31

Full Changelog: v0.1.3...v0.1.4

What's Changed

• chore: Allow downstream users to opt out of onig and use fancy-regex instead by @huevosabio in #27
• chore: Added CI by @Pringled in #28
• chore: Bumped version by @Pringled in #29

New Contributors

• @huevosabio made their first contribution in #27

Full Changelog: v0.1.2...v0.1.3

What's Changed

• feat: Updated hf-hub dependency by @Pringled in #23
• chore: Bumped version by @Pringled in #24

Full Changelog: v0.1.1...v0.1.2

What's Changed

• docs: Updated readme by @Pringled in #11
• Add compilation steps for 10x faster inference by @do-me in #12
• Update models in README.md by @Pringled in #13
• fix: Fixed bug with unk token by @Pringled in #16
• fix: Fixed design issues by @Pringled in #17
• feat: Added encode_single function by @Pringled in #18
• feat: Ran clippy and formatter by @Pringled in #19
• Bumped version by @Pringled in #20
• feat: Updated lockfile by @Pringled in #21

New Contributors

• @do-me made their first contribution in #12

Full Changelog: v0.1.0...v0.1.1

What's Changed

• feat: Added initial model2vec-rs code by @Pringled in #1
• fix: Fixed name mismatch by @Pringled in #2
• feat: Added optional arguments for encode and from_pretrained by @Pringled in #3
• docs: Updated readme by @Pringled in #4
• feat: Optimized performance by @Pringled in #5
• feat: Change batch_encode to batch_encode_fast by @Pringled in #6
• feat: Updated toml by @Pringled in #7
• fix: Correctly remove unk token by @Pringled in #8
• docs: Updated readme by @Pringled in #9
• fix: Updated slugs by @Pringled in #10

New Contributors

• @Pringled made their first contribution in #1

Full Changelog: https://github.com/MinishLab/model2vec-rs/commits/v0.1.0