Scripting Quarkus AI with Large Language Models

Max Andersen

1 year ago

Artificial Intelligence is all the craze lately. With the recent Quarkiverse extension you can can get to work with Large Language Models (LLM) using LangChain4j. I thought it would be nice to demystify and show the simplest way you can get to have an AI agent to help you understand and update your own code.

What is LLM

LLMs, or Large Language Models, are advanced AI systems trained on vast text data that excel in understanding and generating human language. The basics of the system is a neural network known from the 20th century which with more access to data and compute power now is really good at classifying and generating content. Java developers can utilize such LLMs to enhance their applications by generating code, providing natural language interfaces, and improving language-related tasks, all through easy API integration.

Why use Java to work with LLM?

The java ecosystem is huge – not only is there a massive ecosystem of reusable libraries, there are also 9 million developers using it in existing and new Java applications. With LLM rolling over the planet, it is critical to enable these application developers to use it and help shape this new approach to software.

Why use Quarkus to work with LLM?

By using Quarkus we not only get a simple extendable programming mode. It enables your existing or new applications to have LLM integrated all while getting all the enterprise features, such as the following:

Seamless integration with the Quarkus programming model
- CDI beans for the Langchain4j models
- Standard configuration properties for configuring said models
Declarative AI Services
Built-in observability
- Metrics
- Tracing
- Auditing
Build time wiring
- Reduced footprint of the library
- Feedback about misuse at build time
Leverage runtime Quarkus components
- REST calls and JSON handling are performed using the libraries used throughout Quarkus
  - Results in reduces library footprint
  - Enables GraalVM native image compilation
Dev UI features
- View table with information about AI services and tools
- Add embeddings into the embedding store
- Search for relevant embeddings in the embedding store

Enough with the context – lets write some code!

Demo Usecase

In this example, we’ll make a script that gives you a simple chat interface where an AI can help answering questions about your project by reading the filesystem and do updates and deletes if needed.

To do this, we will make a prompt to the AI asking for help and give it access to a set of methods annotated with @Tool which the AI is allowed to call. It will then do those operations and ask/inform about what it does.

In this, we also add some basic security measures to protect against a possible malicious attack or error.

The source code is available on github.com/maxandersen/javaadvent-2023-quarkus-ai-scripting

Keeping it Simple

For this holiday example, we are going to keep it simple and just make a Java script that we can run directly using JBang. If you prefer you can put this into a full-blown Maven or Gradle project.

First, we need the right setup. We will use Java 17 or higher and have -parameters enabled to have Java compiler retain parameter names metadata.

//JAVA 17+
//JAVAC_OPTIONS -parameters

Now lets setup the dependencies we use: Quarkus Platform for dependency management, LangChain4j OpenAI (could be any other model) and then PicoCli for easy parsing of command line options.

//DEPS io.quarkus.platform:quarkus-bom:3.6.3@pom
//DEPS io.quarkiverse.langchain4j:quarkus-langchain4j-openai:0.4.0
//DEPS io.quarkus:quarkus-picocli

And a little bit of Quarkus config properties. In this case, we are going to increase the OpenAI timeout as sometimes the LLM needs more than just a few seconds to process. We are also enabling use of “gpt-4” as it is better at assisting but if your account does not have that API access you can remove the line and it will use the default gpt-3.5 model.

//Q:CONFIG quarkus.langchain4j-openai.timeout=60s
//Q:CONFIG quarkus.langchain4j.openai.chat-model.model-name=gpt-4

FileManager Tool

To have the AI be able to read, update, and delete files we need to make a class available that has those operations. In this example, we make a FileManager class that has methods like getFiles, getFile, createOrUpdateFile and removeFile to perform those operations.

They are straightforward standard Java code using java.io.File’s API annotated with some natural language hints for the AI to understand. Below is the getFiles to illustrate it:

@Tool("""
	Get the files in a directory.
	The list of files recursively found in this directory.
	Will by default return all files in the root directory.
  """)
public List<String> getFiles(
		@P("""
		    The name of the directory relatively to the root of the project.
		    Is a simple string. Use '/' to get the root directory.
		   """) String directory) throws IOException {
			
   directory = handleDir(directory);
   info("Getting files in directory " + directory);

   var files = Files.list(Paths.get(directory)).map(p -> p.toString()).toList();

   return files;
}

Securing the tool

The tool is going to be executed by Quarkus LangChain4j based on what you tell the AI. Just like if you expose an API to a human or some other system it can perform malicious or accidental problematic actions. For example ask to remove a file in the parent directory or your root folder.

To mitigate that all the methods taking a path calls out to this handleDir() method to check if a potential issue.


// Check for '..' in the path to avoid accessing files outside of the project.
// Make request to / be the same as a request to . (root of current directory)
private String handleDir(String directory) {
	if (directory.contains("..")) {
		throw new IllegalArgumentException(
				"The path cannot contain '..' as it would allow to access files outside of the project.");
	}
	directory = (directory == null || directory.isBlank())? "/" : directory;
	return (directory.startsWith("/"))? directory.substring(1) : directory;
}

In addition, the script also ask the user if he is okey to perform any action that updates or remove content.

Talking to the AI

To communicate with the AI we add an interface called ProjectHelper that is annotated with @RegisterAiService and some message annotations to give the AI its initial prompts.

The “tools = FileManager.class” tells the AI it can use it to perform additional operations.

@RegisterAiService(tools = FileManager.class)
public interface ProjectHelper {

    @SystemMessage("""
	You are to help a developer understand his project. 
        You can ask him questions or query the files in his project to get more info.
	If user asks you to modify or create a file, you can do so by calling the 'updateFile' tool.
        """)
	   @UserMessage("{question}")
	   String ask(String question);
}

Question and Answers

To handle questions and answers we make a simple loop to have a conversation with the ProjectHelper that gets injected. The run() method has @ActivateRequestContext as a hint to Quarkus that this method wants to have request scope available.

@Inject
ProjectHelper ai;

@ActivateRequestContext
public void run() {

	while (true) {
		var scanner = new Scanner(System.in);
		if (question == null || question.isBlank()) {
			out.println("Please enter your question:");
			question = scanner.nextLine();
			if (Set.of("exit", "quit").contains(question.toLowerCase())) {
				break;
			}
		}
		out.println("Thinking...");
		String answer = ai.ask(question);
		out.println("Answer: " + answer);
		question = null;
	}
}

Running it all

To use OpenAI we need a token setup. Easiest is to put a .env file with the following key and put your own secret token.

QUARKUS_LANGCHAIN4J_OPENAI_API_KEY=sk-a-secret-1234</code></pre>

Now you can run the script in an existing project directory. In this example we do it on the code itself using jbang devhelper.java.

The initial question is “What can you tell me about this project?”.

Remember, what is shown below is a unique run – the run(s) you make will be different.

Thinking...
Answer: Your project contains the following files:
- devhelper.java
- .gitignore
- .env
- .git

The main file in your project is `devhelper.java`. It is a Java file that uses Quarkus LangChain4j to help understand and update code. The file contains a command-line interface for asking questions and getting answers related to the project.

The `devhelper.java` file includes several dependencies and configuration settings related to Quarkus and LangChain4j. It also defines an interface `ProjectHelper` with a method `ask` for asking questions.

Within the `devhelper.java` file, there is a nested `FileManager` class that provides several tools to interact with files in the project. These tools include:
- `getFiles`: Retrieves the files in a directory.
- `getFile`: Retrieves the content of a file.
- `createOrUpdateFile`: Allows creating or updating the content of a file.
- `removeFile`: Removes a file.

The `devhelper.java` file also includes a `run()` method that continuously prompts for user input (`question`) and calls the `ask` method to get answers. The answers are then printed to the console.

To run the project in development mode, you can use the command: `jbang --fresh -Dquarkus.dev -Dquarkus.console.enabled=false devhelper.java`. This command ensures a fresh build, enables development mode, and doesn't grab the console so that you can ask questions in the terminal.

Note: The code provided here is a truncated version of the `devhelper.java` file for readability purposes. If you need more detailed information, please let me know.</code></pre>

It knows about the files present thus it used the FileManager to get the directory content. It also know details about the devhelper.java so it fetched the content too.

Let’s make the AI do work for us by making a readme file:

Please make a readme.md to explain how to run this using jbang.
Thinking...
Update file 'readme.md' with content:# Running the Project with JBang

This project can be run using &#091;JBang](https://www.jbang.dev/), which is a lightweight scripting tool for Java.

To run the project, follow these steps:

1. Install JBang by following the instructions &#091;here](https://www.jbang.dev/download).

2. Clone the project repository:

```
git clone <repository-url>
cd <project-folder>
```

3. Open a terminal and navigate to the project folder.

4. Execute the following command to run the project with JBang:

```
jbang --fresh -Dquarkus.dev -Dquarkus.console.enabled=false devhelper.java
```

The `--fresh` option ensures a fresh build, while `-Dquarkus.dev` and `-Dquarkus.console.enabled=false` enable development mode and disable the console in order to interact through the terminal.

5. The project will start running and prompt you to enter your question. You can ask any question related to your project, and the Quarkus LangChain4j will provide an answer.

Example:

```
Please enter your question:
What can you tell me about my project?
Thinking...
Answer: Your project contains several files including 'devhelper.java', '.gitignore', '.env', and '.git'. The main file in your project is 'devhelper.java' which uses Quarkus LangChain4j to help understand and update code.
```

6. You can continue asking questions and interacting with the project as needed. To exit the program, you can enter 'exit' or 'quit' when prompted for a question.

Are you sure you want to update the content of readme.md? (yes/no)
yes
Answer: I have created the `readme.md` file with instructions on how to run the project using JBang. You can find the file in your project repository.

And just like that it made a brilliant readme.md for the project.

You can continue interacting with it and ask things like “Add a Apache license header to all the files”, “Please refactor to separate classes”, etc.

Note: it is not perfect and can fail – so don’t let it loose on your project without having a backup or be okey to loose data.

Conclusion

In very few lines of code we made a runnable chatbot that interacted with our own local API and allow the AI to help us on our project.

Now imagine you added more tools to the AI like your ToDo list, Calendar and issue tracker – it could help you do all sorts of things so you can get a more relaxed holiday.

Similarly if this was in an enterprise application setting that exposed tool API could be anything from there – you rest endpoints, other Quarkus extensions, Camel routes, etc. The possibilities are endless – try it out but do ensure you take security into considerations.

Hope you liked it and do give it a try from github.com/maxandersen/javaadvent-2023-quarkus-ai-scripting