A PHP implementation of OpenAI's BPE tokenizer tiktoken.
-
Updated
Jan 25, 2025 - PHP
#
bpe-tokenizer
Here are 11 public repositories matching this topic...
Byte-Pair Encoding tokenizer for training large language models on huge datasets
python machine-learning deep-learning tokenizer chunking from-scratch bpe byte-pair-encoding large-language-models llm bpe-tokenizer
-
Updated
Jun 4, 2024 - Python
(1) Train large language models to help people with automatic essay scoring. (2) Extract essay features and train new tokenizer to build tree models for score prediction.
regression transformer classification vectorizer awp pooling huggingface treemodel kfold llm deberta-v3-large bpe-tokenizer
-
Updated
Jul 3, 2024 - Python
processing de LANguage NATural
natural-language-processing named-entity-recognition bpe-tokenizer bigram-language-model bio-encoding
-
Updated
Apr 3, 2024 - Jupyter Notebook
The basic BPE-tokenizer in NLP
-
Updated
Oct 8, 2024 - Jupyter Notebook
Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.
nlp pytorch alibi transformer-models rotary-position-embedding grouped-query-attention swiglu bpe-tokenizer
-
Updated
Mar 10, 2025 - Jupyter Notebook
A modular Byte Pair Encoding (BPE) tokenizer implemented in Python
-
Updated
Jan 15, 2025 - Python
Naive implementation of byte pair encoding (BPE) tokenization algorithm in golang
-
Updated
Mar 15, 2025 - Go
This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.
scala logback word2vec scalatest hadoop-mapreduce deeplearning4j nd4j apache-hadoop llm bpe-tokenizer jtokkit
-
Updated
Nov 2, 2024 - Scala
self made byte-pair-encoding tokenizer
-
Updated
Jan 20, 2025 - Python
Improve this page
Add a description, image, and links to the bpe-tokenizer topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the bpe-tokenizer topic, visit your repo's landing page and select "manage topics."