Open WebUI: Build and Verify a Document Workspace

Start with one document and three known answers

Open WebUI can provide a shared interface for models and document-based conversations, but an uploaded file is not proof that the assistant can answer from it. A useful first workspace should demonstrate something narrower: a selected model can find a fact in a small document, show the relevant evidence, and decline to invent an answer when the document does not contain it.

Imagine a fictional office handbook with opening hours, an equipment-return procedure and a named contact. Those facts are easy to inspect manually. They make a better first test than a folder of unrelated PDFs whose contents nobody has checked recently. If the small test fails, you can investigate the processing path without wondering whether the answer was somewhere in the thousandth page.

This guide follows that practical path. You will establish the application and provider connection, create a tiny test collection, inspect document processing, ask controlled questions, and keep a record for future changes. The workflow is an instructional example rather than a claim that a particular installation or model was benchmarked here.

Key takeaways

• Start with a small fictional document and save the passages that answer your test questions.

• Check extraction, retrieval and answer accuracy separately, including a question the document cannot answer.

• Rerun the same checks after changing documents, embeddings, model providers or access permissions.

Write the acceptance questions before uploading

Prepare three questions with known answers in the document and one question that cannot be answered from it. For example: “What time does the office open?”, “Where is equipment returned?”, and “Who handles access requests?” The negative question could ask about a travel reimbursement rate that the handbook never mentions.

For each answerable question, save the original passage you expect the assistant to use. Your test now checks both the response and its relationship to the source. A confident answer that happens to be plausible should not pass when the cited passage says something else.

Keep the document fictional or public during setup. That makes the experiment easy to share and avoids confusing application configuration with the handling rules for real internal material. Once the small test is defined, you can build the workspace around evidence you already understand.

Establish the application and the model connection

Open WebUI is an application layer; the selected model provider performs inference. Document extraction and embeddings may involve additional components. Write down those roles before you troubleshoot, because a working chat connection does not establish that document processing is configured correctly.

Use the official quick-start guide for the installation method that matches your environment. The guide includes different deployment paths and persistent-data requirements. Choose one path, record the version or image you install, and complete account creation before adding documents. Avoid mixing an old installation command with current instructions for a different package variant.

Make persistence part of the first setup

Identify where the application's data is stored and how that storage survives a restart or container replacement. A named container is not the same thing as a verified backup. Keep the installation configuration, relevant secrets and persistent-data location documented according to your environment's normal practices.

For a workstation trial, restrict access to the intended local environment. A multi-user deployment adds authentication, access and operational questions that deserve their own review. You can learn the document workflow without exposing a new service broadly.

Verify a plain chat before testing retrieval

Connect a model provider and send a short message without a file. Record which model responded. If you use a local provider, confirm that the application can reach it from the environment where Open WebUI actually runs.

The provider-connection guide distinguishes local and cloud model connections. Container networking is a common source of confusion: a loopback address inside a container does not automatically point to a service on the host. Use the provider's documented connection path and check the host listener as well as the hostname.

A successful plain chat gives you one useful baseline. Save that result, then leave the model connection alone while you test the document pipeline. If a later question fails, you can repeat the plain chat to determine whether the problem affects inference generally or only the document path.

Create an Open WebUI knowledge test you can inspect

Use a short text file for the first collection. You can introduce a scanned PDF or a complex layout after basic ingestion works. Starting with plain text removes OCR and page-order problems from the first experiment.

Here is an example of harmless source material. It is fictional and exists only to make the test easy to evaluate. You can copy it into a text file and change the details, provided you update the expected answers too.

# Fictional office handbook for a retrieval test.
Office hours: Monday to Friday, 09:00 to 17:00.
Equipment returns: Put the device in the returns cabinet beside reception.
Access requests: Contact the office coordinator, Morgan Lee.
Document revision: Test handbook A.

The knowledge-base guide describes creating a collection under Workspace > Knowledge, uploading a small file and waiting for processing. It also explains referencing a knowledge base in chat with the hash selector or attaching it to a model. Follow the path supported by your installed version, and verify that the intended collection is selected in the conversation.

Do not confuse attachment with successful retrieval

The current guide explains that focused retrieval in native mode depends on model tool support and enabled knowledge tools. That means the presence of an attached collection does not necessarily prove that a model searched it. Check the configured mode and capabilities when the assistant behaves as though no file exists.

After attaching the collection, ask the model to find the office-hours passage and inspect the evidence it returns. If it answers from general knowledge without locating the file, check the retrieval mode and tool configuration before adding more documents.

Name the test collection clearly and keep unrelated documents out. Record the file's revision and processing status. If an upload is still being processed, wait until it completes rather than immediately changing retrieval settings. Once the collection is ready, inspect the content that the pipeline actually produced.

Inspect extraction and embeddings before tuning answers

Document processing has several stages. The file must yield usable text, that text must be represented and stored for the configured retrieval path, relevant content must be found, and the model must use it. A failure at the beginning can look like poor reasoning at the end.

The official RAG troubleshooting guide recommends previewing extracted content when a model cannot see a document. It also notes that processing can happen asynchronously and that changing the embedding model requires reindexing existing documents. Those are useful diagnostic checks, not reasons to reset an entire library at the first unusual answer.

Check the source text first

Look for the office hours, equipment-return sentence and coordinator's name in the processed content. If a passage is missing or unreadable, do not try to solve that problem by making the system prompt more forceful. The model cannot reliably quote a passage that never entered the usable document text.

For a PDF, compare the extracted sequence with the original. Columns, tables, scanned pages and unusual layouts can require a different extraction approach. Introduce one representative file at a time so that you can identify the document type responsible for a failure.

If you later test OCR, keep a manually checked transcription of a small passage. That provides an independent reference. Merely receiving a long block of extracted text does not prove that names, dates or table relationships survived correctly.

Treat embedding changes as data-processing changes

An embedding model has a different job from the chat model. Changing the one selected for generation does not necessarily rebuild the representations used by the knowledge collection. Record the embedding provider and model separately from the model that writes the answer.

Before changing embeddings on a real collection, preserve the relevant configuration and follow the documented reindexing workflow. Test the change on a small collection first. If you change extraction, chunking, embeddings and generation together, even a better result will be difficult to explain.

Keep a processing note with the file revision and configuration. That note connects the source material to the state you are about to evaluate, making the next question test far more meaningful.

Evaluate retrieval and answer quality separately

Ask the three prepared questions in fresh conversations with the intended knowledge collection selected. Request the relevant passage and compare the response with the original. Do not award a pass merely because a source chip is visible: the source needs to support the specific claim being made.

For the office-hours question, an answer that adds weekend availability should fail unless the document supports it. For the equipment-return question, citing a passage about access requests is also a failure, even if the answer sounds plausible. Evidence and response quality are related checks, not interchangeable ones.

Use a small evaluation record

Keep the questions, expected passages, responses and observations together. The table below is a template for your own test, not a report of measured results.

Repeat one question with different wording after the exact version works. That helps distinguish a fragile keyword coincidence from a useful question-answering path. Then add a second small document with a clearly different topic and check that the evidence still comes from the right file.

For a broader evaluation workflow, see Artifilog's guide to evals for agents. Use its distinction between outcome checks and execution traces when recording document-chat failures.

Investigate the first failed stage

If the correct passage is absent from extracted text, return to ingestion. If it exists but is not retrieved, inspect collection selection, retrieval mode, tool behavior and relevant settings. If it is retrieved but the answer misstates it, examine the generation instruction and the selected model's behavior.

Avoid immediately increasing every retrieval limit. More context can introduce irrelevant material and consume the model's input budget. First determine what evidence is missing and whether the system is finding it. A smaller, explainable correction is easier to validate than a broad setting change.

Also preserve failures. An unsuccessful answer is useful when it includes the source revision and configuration that produced it. Once the small test passes consistently, you can consider what changes when the workspace becomes real and shared.

Expand the workspace without losing the evidence trail

A working handbook test is the beginning of a document workspace, not proof that every future file will behave equally well. Expand by document type and use case. Add a representative text PDF, then a document with a table, then a larger collection whose expected answers you have checked. Keep the original tiny collection available as a regression test.

For recurring documents, define which revision is authoritative and how obsolete versions will be handled. An assistant may retrieve a perfectly legible but outdated policy if the collection contains both versions without a clear process. Naming helps people inspect a result, but a filename alone is not an access-control or correctness guarantee.

Verify the local boundary across all components

The community-contributed offline-operation guide explains that local document chat requires the relevant models and processing components to be prepared locally. The tutorial is hosted in the project documentation but is not a supported deployment guarantee. Review extraction, embeddings, generation and any optional tools separately. A locally hosted interface can still call an external provider if that is how the workspace is configured.

Make offline operation an observable test when it is a requirement. Prepare the needed resources, restart under the intended conditions and repeat the small document questions. Record which component fails if the test cannot complete. This is more informative than calling the whole installation private simply because its browser address is local.

Add a document-change test

Replace the fictional handbook with a clearly labeled revision that changes the office opening time. Follow your chosen update process, wait for processing and ask the original question in a new conversation. Check both the answer and the cited revision. This tests a different requirement from initial ingestion: whether the workspace can move from old evidence to current evidence without leaving users unsure which document was consulted.

If both revisions are intentionally retained, ask a date-specific question and inspect the source selection. If only the new revision should remain authoritative, verify that your content-management process reflects that decision. Keep the old expected answer in the test record, but mark it obsolete so that a future reviewer does not mistake it for a current requirement. A document assistant needs a maintenance habit as much as a successful first upload.

Test with the account that will actually use it

If you add other users, review collection permissions and test using an ordinary user account rather than only the administrator. Confirm that the intended files can be accessed and that another collection is unavailable where required. Do not rely on prompt instructions to enforce separation between teams' documents.

Keep backups and a recovery procedure appropriate to the deployment. Reindexing is not a substitute for a backup, and a collection that can be reconstructed still needs its source files and configuration preserved. Before an update, save the current configuration and rerun the small acceptance set afterward.

The useful result is a workspace whose answers can be traced back to known material. You know which provider writes the response, how files become searchable and what evidence counts as a pass. Start with one document, preserve the failures as well as the successes, and Open WebUI becomes a system you can improve deliberately rather than a chat window you have to trust blindly.