tokenizer.py File Reference

Go to the source code of this file.

Namespaces
namespace	oc_chat
namespace	oc_chat.core
namespace	oc_chat.core.tokenizer

Functions
	oc_chat.core.tokenizer._get_encoding ()
int	oc_chat.core.tokenizer.count_tokens (str text)
int	oc_chat.core.tokenizer.count_message_tokens (list[dict[str, Any]] messages)

Variables
	oc_chat.core.tokenizer._encoding = None