Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: T4ras123/SmolBPE
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.0.1
Choose a base ref
...
head repository: T4ras123/SmolBPE
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0.3.2
Choose a head ref
  • 20 commits
  • 13 files changed
  • 1 contributor

Commits on Oct 11, 2024

  1. Update README.md

    @T4ras123
    T4ras123 authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    9d38dfc View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Update README.md

    @T4ras123
    T4ras123 authored Oct 13, 2024
    Configuration menu
    Copy the full SHA
    223c3bd View commit details
    Browse the repository at this point in the history

  2. Update README.md

    @T4ras123
    T4ras123 authored Oct 13, 2024
    Configuration menu
    Copy the full SHA
    d20ad44 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2024

  1. feat: enhance GPT4Tokenizer with vocab loading/saving and CLI support

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    ef3ac4d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/T4ras123/SmolBPE

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    a5f0260 View commit details
    Browse the repository at this point in the history

  3. feat: initialize smolbpe module

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    88a0805 View commit details
    Browse the repository at this point in the history

  4. feat: add initial implementation of GPT-4 compatible tokenizer and se… 

    …tup configuration
    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    d952b4a View commit details
    Browse the repository at this point in the history

  5. fix: update author email and repository URL in setup.py; adjust impor… 

    …t path in character_tokenizing.py
    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    5c74ff8 View commit details
    Browse the repository at this point in the history

  6. fix: rename package from gpt4tokenizer to smolbpe in setup.py

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    0ac5f14 View commit details
    Browse the repository at this point in the history

  7. chore: update README.md with installation instructions and usage exam… 

    …ples; modify setup.py for long description and version bump; include README.md in MANIFEST.in; remove unused test files and refactor GPT4Tokenizer class
    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    fb785de View commit details
    Browse the repository at this point in the history

  8. chore: include LICENSE in MANIFEST.in and update setup.py for UTF-8 e… 

    …ncoding and markdown content type
    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    dbbf7bf View commit details
    Browse the repository at this point in the history

  9. feat: add badges for version, downloads, stars, license, Python versi… 

    …ons, and sponsorship to README.md
    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    c358f02 View commit details
    Browse the repository at this point in the history

  10. fix: reorder badges in README.md for improved clarity

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    11035c4 View commit details
    Browse the repository at this point in the history

  11. fix: update Twitter badge link in README.md for correct username

    @T4ras123
    T4ras123 committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    dd20e82 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. feat: enhance GPT4Tokenizer to support special tokens and improve tra… 

    …ining logic
    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    d6ab197 View commit details
    Browse the repository at this point in the history

  2. chore: bump version to 0.3.0 in setup.py

    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    3a18947 View commit details
    Browse the repository at this point in the history

  3. refactor: clean up comments and whitespace in GPT4Tokenizer

    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    503c4ef View commit details
    Browse the repository at this point in the history

  4. feat: migrate GPT4Tokenizer implementation to a new file and enhance … 

    …structure
    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    eb6a3f8 View commit details
    Browse the repository at this point in the history

  5. feat: update GPT4Tokenizer constructor to set default regex pattern a… 

    …nd improve argument order
    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    1120285 View commit details
    Browse the repository at this point in the history

  6. feat: rename GPT4Tokenizer to Tokenizer and update README and setup.py

    @T4ras123
    T4ras123 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    fba88fa View commit details
    Browse the repository at this point in the history
Loading