#کتابخانه #پردازش_متن #تنسورفلو
TF.Text - Text processing in Tensorflow
کتابخانه مخصوص پردازش متن در نسخه جدید Tensorflow
TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0. The library can perform the preprocessing regularly required by text-based models, and includes other features useful for sequence modeling not provided by core TensorFlow.
وقتی از سرگروه تیم TF.Text پرسیدند که کراس هم یک API پردازش متن داره که روی تنسورفلو اجرا میشه،پس TF.Text چه فرقی با اون داره؟ گفت:
"Keras has a subset, but not the breadth of TF.Text. We are actively talking with them to fill in gaps we believe language engineers want, but are not provided in the core Keras API, and I wouldn't be surprised if additional Keras layers are provided by TF.Text in the future."
ویژگی های این کتابخانه از زبان سرگروه تیم TF.Text:
- Focusing on the new tools for tokenizing text strings
- Tools for pattern-matching, n-gram creation, unicode normalization, and sequence constraints
- The code is designed to operate on RaggedTensors: Variable-length tensors which are better-suited for processing textual sequences.
- Pre-processing steps are now first-class citizens of the TensorFlow compute graph, which gives them all the advantages of that system. In particular, according to the documentation, "You do not need to worry about tokenization in training being different than the tokenization at inference...."
صفحه گیت هاب این کتابخانه:
https://github.com/tensorflow/text
نوت بوک شروع:
https://storage.googleapis.com/tensorflow_docs/text/examples/intro.ipynb
TF.Text - Text processing in Tensorflow
کتابخانه مخصوص پردازش متن در نسخه جدید Tensorflow
TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0. The library can perform the preprocessing regularly required by text-based models, and includes other features useful for sequence modeling not provided by core TensorFlow.
وقتی از سرگروه تیم TF.Text پرسیدند که کراس هم یک API پردازش متن داره که روی تنسورفلو اجرا میشه،پس TF.Text چه فرقی با اون داره؟ گفت:
"Keras has a subset, but not the breadth of TF.Text. We are actively talking with them to fill in gaps we believe language engineers want, but are not provided in the core Keras API, and I wouldn't be surprised if additional Keras layers are provided by TF.Text in the future."
ویژگی های این کتابخانه از زبان سرگروه تیم TF.Text:
- Focusing on the new tools for tokenizing text strings
- Tools for pattern-matching, n-gram creation, unicode normalization, and sequence constraints
- The code is designed to operate on RaggedTensors: Variable-length tensors which are better-suited for processing textual sequences.
- Pre-processing steps are now first-class citizens of the TensorFlow compute graph, which gives them all the advantages of that system. In particular, according to the documentation, "You do not need to worry about tokenization in training being different than the tokenization at inference...."
صفحه گیت هاب این کتابخانه:
https://github.com/tensorflow/text
نوت بوک شروع:
https://storage.googleapis.com/tensorflow_docs/text/examples/intro.ipynb
GitHub
GitHub - tensorflow/text: Making text a first-class citizen in TensorFlow.
Making text a first-class citizen in TensorFlow. Contribute to tensorflow/text development by creating an account on GitHub.