llm_change_agent.evaluations package

Submodules

llm_change_agent.evaluations.evaluator module

Evaluation script for the LLM Change Agent.

llm_change_agent.evaluations.evaluator.compare_changes(expected_dir, output_dir)

Compare the actual changes with the predicted changes.

llm_change_agent.evaluations.evaluator.download_document(url, input_dir)

Download the document from the URL.

llm_change_agent.evaluations.evaluator.generate_changes_via_llm(eval_dir, output_dir, provider, model)

Generate changes via the LLM Change Agent.

llm_change_agent.evaluations.evaluator.get_comparison_metrics(expected_changes, output_changes)

Compare the expected changes with the output changes.

llm_change_agent.evaluations.evaluator.prepare_eval_and_expected_yamls(input_dir)

Prepare the evaluation and expected YAMLs for the input documents.

llm_change_agent.evaluations.evaluator.run_evaluate(model, provider)

Evaluate the LLM Change Agent.

llm_change_agent.evaluations.evaluator.run_llm_change_agent(prompt, provider, model, docs=None)

Run the LLM Change Agent.

Return type:

List

Module contents

Evaluate the LLM Change Agent.