With longer context lengths for LLMs. It is increasingly difficult to test if fine tuned models attend to all depths of the context.
The needle in haystack is a popular approach. However since the LLMs can also answer about the needle instead of the needle. Tests have shown that a "Lie" works well in this context 😊
lie: "Picasso painted the Mona Lisa"
retrieve: "Who painted the Mona Lisa?"
pip install liah
# update OPENAI_API_KEY in the env with your token.
# If you need Open AI models for the final evaluation
from liah import Liah
from vllm import LLM, SamplingParams
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=4096)
llm = LLM(model="meta-llama/Llama-2-70b-hf", tensor_parallel_size=4, max_model_len=1500) # need 4 A100s 40GB
#Create Liah
liah = Liah(max_context_length=2000)
#Get a sample from different depths and context_lengths
for i, sample in enumerate(liah.getSample()):
# test the sample text with your model
output = llm.generate([sample["prompt"]], sampling_params)[0]
#Update liah with the response
liah.update(sample, output.outputs[0].text)
#Contains the plot file from Liah
plotFilePath = liah.evaluate()
bash
pip install pre-commit
then (in the repository, just once)
bash
pre-commit install
bash
pre-commit run --all-files
