Building Your First AI Service with Quarkus, LangChain4j, and Ollama

The real value of AI in the enterprise lies in what an application can do. Integrating Large Language Models (LLMs) into Java applications is no longer a complex task. With Quarkus and LangChain4j, you can build a production-ready AI service that runs entirely on your local hardware using Ollama.

1. Introduction

For Java developers, the focus has shifted from what an LLM knows to how we can use that knowledge within our systems. This tutorial guides you through creating a local AI service that uses Retrieval-Augmented Generation (RAG) to answer questions based on a custom dataset.

2. Prerequisites

• JDK 21+: Required for the latest Quarkus features.

• Maven 3.9+: For project builds and dependency management.

• Ollama: Download and install from ollama.com.

• Cloud Model: Run ollama run gpt-oss:120b-cloud so you can establish the connection with this model on Ollama Cloud.

3. Project Setup

Use the Quarkus Maven plugin to generate a project named first-chatbot with the necessary extensions :

mvn "io.quarkus.platform:quarkus-maven-plugin:create" \
    -DprojectGroupId=com.eldermoraes \
    -DprojectArtifactId=first-chatbot \
    -Dextensions="langchain4j-ollama,langchain4j-easy-rag,quarkus-rest,quarkus-smallrye-openapi"

Key Extensions

• quarkus-rest: Provides the modern Jakarta REST implementation.

• langchain4j-ollama: Connects your application to the Ollama API.

• langchain4j-easy-rag: Handles document ingestion and retrieval automatically.

• quarkus-smallrye-openapi: Generates Swagger UI for easy testing.

4. Exploring Quarkus, LangChain4j, and Ollama

These three components form a powerful stack :

• Quarkus: The engine that handles dependency injection and lifecycle management.

• LangChain4j: The orchestrator that allows you to define AI behavior via Java interfaces.

• Ollama: The brain that runs the LLM locally on your machine.

5. Implementing Your First AI Service

Step 1: Prepare Your Knowledge Base

Create a directory at src/main/resources/rag. You need data files for the RAG process to work. For this tutorial, download the Star Wars dataset (JSON files) from this repository: swapi-app data.

Place the downloaded files into your src/main/resources/rag folder.

Step 2: Configure the Application

Edit src/main/resources/application.properties to wire the components together.

# Ollama Model Configuration
quarkus.langchain4j.ollama.chat-model.model-id=gpt-oss:120b-cloud
quarkus.langchain4j.timeout=60s

# Easy RAG Path
quarkus.langchain4j.easy-rag.path=src/main/resources/rag

package com.eldermoraes;

import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import io.quarkiverse.langchain4j.RegisterAiService;
import jakarta.enterprise.context.ApplicationScoped;

@RegisterAiService
@ApplicationScoped
public interface StarWarsAssistant {

    @SystemMessage("""
        You are an expert on the Star Wars universe. 
        Use the provided information to answer questions concisely.
        """)
    String chat(@UserMessage String question);
}

package com.eldermoraes;

import jakarta.inject.Inject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.QueryParam;

@Path("/star-wars")
public class StarWarsResource {

    @Inject
    StarWarsAssistant assistant;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String ask(@QueryParam("question") String question) {
        return assistant.chat(question);
    }
}

./mvnw quarkus:dev