Usage

The CLI provides several commands to interact with the language models.

List Available Models

To list all available models from supported providers:

llm-change list-models

OpenAI models:
  gpt-4o-2024-08-06
  gpt-4o-mini
  gpt-4o-mini-2024-07-18
  gpt-4o-2024-05-13
  gpt-4o
  gpt-4-turbo-2024-04-09
  gpt-4-turbo
  gpt-4-turbo-preview
Anthropic models:
  claude-3-5-sonnet-20240620
  claude-3-opus-20240229
  claude-3-sonnet-20240229
  claude-3-haiku-20240307
Ollama models:
  llama3.1
LBL-CBORG models:
  lbl/cborg-chat:latest
  lbl/cborg-chat-nano:latest
  openai/gpt-3.5-turbo
  openai/gpt-4o
  openai/gpt-4o-mini
  anthropic/claude-haiku
  anthropic/claude-sonnet
  anthropic/claude-opus
  google/gemini-pro
  google/gemini-flash

Generate Text

To generate text using a specified model and provider:

llm-change execute --model <MODEL_NAME> --provider <PROVIDER_NAME> --prompt "<YOUR_PROMPT>"

Replace <MODEL_NAME>, <PROVIDER_NAME>, and <YOUR_PROMPT> with your desired values.

Examples

Generate text using CBORG’s lbl/llama-3 model with a custom prompt:

llm-change execute --model lbl/llama-3 --prompt "I want to change the definition of class ABC:123 to 'foo bar' and also create a new class labelled 'bar foo' with the curie DEF:123."

OR

llm-change execute --provider cborg --prompt "I want to change the definition of class ABC:123 to 'foo bar' and also create a new class labelled 'bar foo' with the curie DEF:123."

generates

Final Answer:

[
  "change definition of ABC:123 to 'foo bar'",
  "create class DEF:123 'bar foo'"
]

Evaluations

Input

The project also contains input data for evaluations in the form of YAML files for the following ontologies:

Expected changes

The expected changes for specific pull requests are listed ontologywise below:

Actual results

The actual results for some of the LLM models evaluated can be found here and corresponding metrics here

Development

To run the project locally, clone the repository and navigate to the project directory:

git clone https://github.com/yourusername/llm-change-agent.git
cd llm-change-agent
poetry install

Make sure you have poetry installed in your system.