bigcode
/

tokenizer-the-stack-march-sample-v3

Model card Files Files and versions

RaymondLi commited on Mar 9, 2023

Commit

339010c

·

1 Parent(s): 1b4233d

fix pretokenizer

Files changed (1) hide show

tokenizer.json +1 -1

tokenizer.json CHANGED Viewed

@@ -185,7 +185,7 @@
       },
       {
         "type": "ByteLevel",
-        "add_prefix_space": true,
         "trim_offsets": true,
         "use_regex": true
       }

       },
       {
         "type": "ByteLevel",
+        "add_prefix_space": false,
         "trim_offsets": true,
         "use_regex": true
       }