# Install the latest release version
pip install runpod-llm
# or
# Install the latest development version (main branch)
pip install git+https://https://github.com/tsangwailam/langchain-runpod-llm
Create a new API key by click the "+ API Key" button.
Usage
fromrunpod_llmimportRunpodLlama2llm=RunpodLlama2(
apikey="YOU_RUNPOD_API_KEY",
llm_type="7b|13b",
config={
"max_tokens": 500,
#Maximum number of tokens to generate per output sequence."n": 1, # Number of output sequences to return for the given prompt."best_of": 1, # Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This is treated as the beam width when use_beam_search is True. By default, best_of is set to n."Presence penalty": 0.2, # Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens."Frequency penalty": 0.5, # Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens."temperature": 0.3, # Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling."top_p": 1, # Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens."top_k": -1, # Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens."use_beam_search": False, # Whether to use beam search instead of sampling.
},
verbose=True, # verbose output
)
some_prompt_template=xxxxxoutput_chain=some_prompt_template|llmoutput_chain.invoke({"input":"some input to prompt template"})
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.