A text generation plugin using local LLMs or other text generation methods. Builds on top of transformers by Hugging Face.

• Generate text from a prompt using a local LLM
• Access all LLMs from the Hugging Face Model Hub

Since we depend on transformers, please make sure that you fulfill their requirements. You also need Pytorch installed on your system, please refer to the pytorch installation instructions.

1. Install Nendo
2. pip install nendo-plugin-textgen

If you have a CUDA GPU you can install the following library for an additional speedup:

pip install flash-attn --no-build-isolation

Then set ATTN_IMPLEMENTATION=flash_attention_2 in your environment variables.

Take a look at a basic usage example below. For more detailed information, please refer to the documentation.

>>> from nendo import Nendo
>>> nd = Nendo(plugins=["nendo_plugin_textgen"])

>>> nd.plugins.textgen(prompt=["Tell me about your favorite song."])[0]

Visit our docs to learn all about how to contribute to Nendo: Contributing

Nendo: MIT License