A lightweight ontology stores simple facts about a domain in a form that a computer can use. Typical facts say that one type of object is a more specific type of another object, or that two types are connected by a relationship. For example, an ontology may state that every penguin is a bird, or that every university course is taught by a person.
When an LLM is used to help construct an ontology, the same statement can be presented in different ways. It may be written as an ordinary sentence or using a formal notation. Consider the following three versions:
• Every penguin is a bird.
• Everything that is a penguin is also a bird.
• Penguin SubClassOf Bird
These versions express the same idea, but an LLM may not answer them in the same way. The choice of words and notation can therefore affect the result. Longer statements and statements involving relationships may also be more difficult for the model.
This thesis turns that problem into a controlled "prompt laboratory". You will automatically create ontology questions, ask each question in several clearly defined ways and compare the answers. The ontology itself will not be included in the LLM prompt. It will be used to generate test questions and determine the expected answers. The LLM will answer each question using the knowledge acquired during its training. The goal is to discover which ways of asking are reliable across different models, ontologies and types of statements.
Main question
How much do small changes in wording affect LLM answers to simple ontology questions, and can we design question templates that work reliably in a tool that builds knowledge bases?
Main tasks
No prior knowledge of description logics is required. You will first learn the basic ideas needed to work with simple "is-a" statements and relationships between objects. You will:
• Read introductory material about lightweight ontologies and existing work that uses LLMs to construct or query them.
• Select a few small public ontologies and use a reasoning tool to generate statements for which it is known whether they follow from the ontology. Use the ontologies to determine reference answers, but do not include them in the prompts sent to the LLM.
• Design several ways of asking each type of question. Check every version carefully to ensure that it asks the same thing.
• Build a program that sends the questions to a few LLMs whose exact versions and settings can be recorded and reused. Record the answers, response format, response time and token use.
• Measure answer accuracy against the references and how often two phrasings of the same question receive different answers.
• Investigate which features make a question difficult, such as its length, the presence of a relationship or the use of technical notation.
• Create a visual "scorecard" or heat map showing which templates work best for each kind of statement.
• Use the best performing templates in a tool that builds knowledge bases and test whether they improve the result compared with the templates used in earlier experiments.
The core contribution is a public collection of test questions, a catalogue of tested question templates and an analysis of when wording matters. Combining several phrasings into one final answer or testing a commercial model can be treated as optional extensions.