Authors

Salma Alkhalily, Shundarevskiy Nazarii, Mandee Botson, Elbedweihy Ammar

Abstract

PolicyGap is a server-based, AI-assisted system for examining an organisation’s policydocuments against compliance framework controls. It is aimed particularly at small businessesand start-ups seeking a more accessible way to prepare for policy review and auditing. Thesystem retrieves textual evidence, classifies each control as implemented, partiallyimplemented, or not implemented, and presents gaps for human confirmation. Its purpose isto accelerate routine evidence checking and remediation planning, not to replace expertjudgment with an independently validated audit decision.

After account creation, a user creates a workspace with company and workspace names,uploads policy documents, and runs an assessment. The demonstrated interface moves theassessment from “running” to “completed” and then opens a dashboard. The system isconceived as support for an auditor or reviewer: it reduces a long source document to controllevelfindings while retaining the original sentence used as evidence.

On ingestion, the backend extracts raw text and divides it into overlapping chunks intended topreserve evidence across boundaries. Sentence Transformers converts each chunk into a384-dimensional embedding. ChromaDB stores embeddings and text in a separate collectionfor each workspace, while SQLite retains document and chunk metadata. Framework controlsare embedded once and cached. A batch vector query retrieves the five most similar chunksfor each control. If keyword overlap is below 20%, the control bypasses the language model and is marked not implemented; remaining controls are submitted to the LLM concurrently inbatches of 50.

Each finding receives one of three values: 1 for implemented, 0.5 for partially implemented,and 0 for not implemented. Their weighted average is converted to a percentage. Thedashboard shows coverage by function, status counts, priority gaps, confidence, evidence,and gap summaries. Confidence was described as combining five signals: cosine similarity,the number of matching chunks, term overlap, model-reported certainty, and whether theevidence excerpt is verifiable. Reviewers can accept or reject findings and rescore theassessment. A remediation roadmap orders gaps by priority and supplies recommendations.Reports classify risk as low at 70% or above, medium at 40–69%, and high below 40%, andcan be exported in report, spreadsheet, or Word-document form.

license

Creative Commons BY-SA 4.0