Train tokenizer

This commit is contained in:
2025-10-16 06:11:40 -07:00
parent 2352cc9976
commit b584ad48f5
12 changed files with 4537 additions and 1 deletions

5
.gitignore vendored
View File

@@ -1,2 +1,5 @@
/target
*.ignore
*.ignore
/data
/tokenizer.json