Every place an AI system touches personal data
Most AI privacy discussion is about training data. Training data is one of seven places, and usually not the one that causes the problem.
1. Training and fine-tuning data. The obvious one. If it relates to identifiable people it is personal data, and that does not stop being true because the model is "only learning patterns".
2. Prompts and context. What users type, plus everything the application silently attaches: the customer record, the transcript, the document. In practice this is where the most sensitive data flows, because it is operational rather than curated, and nobody reviewed it.
3. Retrieved content. Whatever the retrieval layer pulls in. The index was often built over a document store nobody mapped for personal data.
4. Outputs. A generated summary of a person is personal data about that person. So is an inference — a risk score, a propensity, a guess at health or income. An inferred attribute is personal data even when it is wrong, and being wrong creates an accuracy obligation rather than excusing one.
5. Logs and telemetry. The quiet one. Prompt and response logging for debugging, evaluation and quality monitoring routinely captures everything in items 2 to 4, retains it far longer than the source system would, and sits in a store the data team never inventoried. In most organisations this is the largest unacknowledged personal-data store created by an AI project.
6. Embeddings and vector stores. An embedding derived from personal data is still linked to that person and can often be used to retrieve or approximate the original text. Treating a vector as anonymous because it looks like numbers is a mistake several organisations have made publicly.
7. Memory and derived state. Summaries, profiles and preferences an assistant keeps between sessions. Created by the system rather than collected from the person, which is exactly why nobody has a retention rule for it.
The practical consequence. A data map for an AI system that lists only "training data" is wrong by five stores. The first task on any such project is to enumerate all seven, name an owner and a retention period for each, and discover which of them nobody had thought about. In most reviews, that is the logs.
And the three words used to escape all seven. Anonymisation, pseudonymisation and synthetic data are each offered as a reason the rules do not apply. Only one of them works, and only sometimes.
- Pseudonymised data is still personal data. Replacing a name with a customer number protects it and changes nothing about its status, because the link still exists somewhere. It reduces risk; it does not remove the obligation.
- Anonymised data is outside the regime only if re-identification is not reasonably possible — judged against the other data available to anyone who might try, not against the good intentions of the person holding it. A table of a hundred people with age, postcode and diagnosis is not anonymous, however carefully the names were stripped.
- Synthetic data generated from personal data inherits the question rather than settling it. A generator trained on real records can reproduce them, and the more faithful the synthetic data is to the original — which is the whole point of it — the more plausible that becomes.
In each case the status is a conclusion you have to be able to defend, with a test behind it. It is not a label you apply to a dataset to move it out of scope.
Lawful basis, purpose and the special categories
The basis for training is not the basis for inference. These are different processing operations with different purposes, different data subjects and often different bases. An organisation may have a perfectly good basis for using customer records to run a service, and no basis at all for using those records to train a model that will serve other customers.
Purpose limitation is where most AI projects actually fail. Data was collected to provide a service, administer an account or meet a legal obligation. Training a model is a new purpose. The question is not "do we hold this data" — it is "is this use compatible with why we collected it, and would the person expect it". Where it is not compatible, the lawful route is a fresh basis, not a broader privacy notice written afterwards.
On consent. Consent must be freely given, specific, informed and withdrawable. It is a poor fit for model training, because withdrawal cannot easily be honoured once training has happened — which is a reason to be careful about relying on it, not a reason to ignore the withdrawal.
On legitimate interests, where the regime offers it: it requires a real assessment — the interest, the necessity, and the balance against the person's rights and reasonable expectations — written down before processing, not reconstructed afterwards. "We have a legitimate interest in innovation" is not an assessment.
Special categories need their own basis. Health, biometrics, race, religion, sexual orientation, trade union membership, political opinion. Two traps specific to AI:
- Biometric data for identification is special-category in most regimes. Face matching in an onboarding flow is squarely in scope, and both the UK Information Commissioner's Office and Kenya's Office of the Data Protection Commissioner have issued biometric-specific guidance.
- Inferred special-category data. A model that predicts health status, ethnicity or sexuality from non-special data is generating special-category data. The output's nature governs, not the input's — which is why "we never collected health data" is not an answer when the system infers it.
Children's data, where it may be present, attracts stricter treatment in most regimes, and an AI system that cannot tell whether a data subject is a child is a system that cannot apply those rules.
Automated decisions. Where a decision is made solely by automated means and has legal or similarly significant effects, most regimes give rights to be informed, to obtain human intervention and to contest. Kenya's Data Protection Act addresses this directly, as do the European regimes. An institution deploying an AI decision has to be able to say which decisions are solely automated — and many organisations have never written that list down.
Erasure, access and the rights that are hard
Access. A person is entitled to know what personal data you hold about them. For an AI system that means the prompts, the retrieved context, the outputs, the logs and the memory — not only the row in the customer database. Answering an access request without the stores from lesson one produces an answer that is incomplete, and the incompleteness is discoverable.
Rectification. If an inference is wrong, the obligation is to correct it. For a system that regenerates inferences on each run, correcting the source is usually enough; where the wrong inference was written to memory, a profile or a downstream record, it has to be corrected there too.
Erasure, and the honest answer. A request to delete reaches training data, prompts, retrieved documents, logs, embeddings and memory straightforwardly. It does not straightforwardly reach the model's weights. Machine unlearning is an active research area rather than an operational capability, and retraining a large model to remove one person's data is usually disproportionate.
So the defensible position is to do what is actually possible and say so plainly:
- Delete from every store you control, including the ones in lesson one that nobody listed.
- Remove the person from the retrieval index so that nothing about them is returned.
- Suppress at inference where the model may still reproduce something.
- Record, in the response and in the file, what has been done and what has not.
What is not defensible is claiming deletion that did not happen, or treating the model as outside the request because it is technically inconvenient.
The design conclusion that follows. Because erasure against weights is hard, what you put into training is close to irreversible. That raises the stakes on the basis and the minimisation decisions made at the start, and it is the strongest practical argument for not training on operational personal data when a less sensitive route exists.
Explanation. Where a right to meaningful information about the logic exists, it is not satisfied by naming the algorithm. It means being able to say what factors drove this decision for this person, in terms they can contest. A system whose output cannot be explained to the person it affects is a system that cannot lawfully make certain decisions about them, whatever its accuracy.
Vendors, transfers and the terms that matter
Most organisations do not train models; they send data to someone who does. That is a processing arrangement and usually a cross-border transfer, and both have requirements.
The vendor is a processor, and the contract has to say so. Instructions, confidentiality, security, sub-processors, assistance with rights requests, deletion or return at the end, and audit. A click-through terms-of-service page is not a processing agreement.
The questions to put to a model provider, in order of how often the answer is unsatisfactory:
- Do you train on our data, including prompts and outputs? Get it in the contract, not the marketing page, and check whether it differs by plan.
- How long do you retain prompts and outputs, and where? Many providers retain for abuse monitoring by default, which is a legitimate purpose and still a retention you must account for.
- Which sub-processors, and in which countries? Including the inference host, which is frequently a different company from the model developer.
- What happens on deletion? Including backups and any copies held for safety review.
- Can we get logs of what was sent? Needed for access requests and incident response.
- Where is the processing physically performed, and does that change with load?
Transfers. Sending personal data to a model hosted in another country is a transfer and needs whatever mechanism the regime requires — adequacy, standard contractual clauses, or the regime's own route. Kenya's Office of the Data Protection Commissioner has published guidance notes specifically on cross-border transfers, and the obligation applies to prompt content exactly as it does to a database export, which is the point most often missed: a prompt is a transfer.
The retention trap. The source system deletes a record after seven years. The prompt that contained it sits in an evaluation set indefinitely, the embedding sits in a vector store with no lifecycle, and the log sits in an observability platform with a different retention policy set by a different team. Retention has to be set per store, and the AI stores are the ones nobody set.
Shadow usage. Staff pasting customer data into consumer AI tools is the most common real-world incident in this area. It is a policy, training and tooling problem rather than a legal one: give people an approved tool that is good enough to use, or they will use one that is not.
And decide now what a breach looks like here, because the clock does not wait for the argument. Three shapes recur, and none resembles a stolen laptop:
- a model reproducing training data in an output — someone else's record appearing in an answer;
- a prompt or response log exposed, which is the biggest store in lesson one and the one with the weakest access control;
- a retrieval index returning documents to the wrong user, because permissions were enforced in the application and not at retrieval.
Each is a personal data breach. How fast you must report, to whom, and whether you must also tell the people affected is set by national law, and the deadlines are short. Some laws fix them in hours. Kenya's Data Protection Act, for example, requires the controller to notify the data protection authority without delay and within seventy-two hours of becoming aware, where there is a real risk of harm, and requires a processor to tell the controller within forty-eight hours where reasonably practicable. Other laws leave the timing to subordinate rules: India's Digital Personal Data Protection Act, 2023 (section 8(6)) requires the data fiduciary to give intimation of a breach to the Board and to each affected person, in the form and manner prescribed by rules made under the Act. Find three things in your own law before an incident rather than during one: the deadline, who the regulator is, and whether the people affected must be told as well.
Read those two periods together and the vendor question above becomes concrete: your seventy-two hours start when you become aware, your provider has forty-eight to tell you, and if the contract says nothing about notification you will learn about it from a customer.
Designing for it, and what a regulator will ask
Start with the impact assessment, and use it as a design tool. Most regimes require one before high-risk processing, and AI involving personal data usually qualifies — particularly where decisions affect people, where special categories are involved, or where processing is large-scale. Kenya's ODPC has published a guidance note on data protection impact assessments; the European regimes have their own. Done at the end it is paperwork. Done at the start it changes the design, which is the only reason it exists.
Minimise at the prompt, not just at the database. The largest single reduction in exposure on most AI projects is sending less: the fields the task needs rather than the whole record, redaction or tokenisation of identifiers that the model does not need to see, and retrieval scoped to what the requesting user may access.
Set retention per store. Training sets, evaluation sets, prompt logs, response logs, embeddings, memory, traces. Each gets an owner and a period. Logs are the common failure, because they are created by engineers for debugging and never enter the retention register.
Enforce access control outside the model. Retrieval filtered by the requesting user's permissions before the model sees anything — the same conclusion the security course reaches from a different direction, which is a good sign.
Write down which decisions are solely automated, and build the human intervention route before anyone asks for it.
Test for the harms, not just the accuracy. Disparate performance across groups is both a fairness problem and, in many regimes, a lawfulness problem. NIST's AI Risk Management Framework and its generative AI profile give a structure for this that maps onto what supervisors ask.
What a regulator actually asks, and the honest state of most answers:
- What personal data does this system process, in every store? Usually incomplete.
- What is your lawful basis for each processing operation, including training? Often one basis claimed for several operations.
- Show me the impact assessment and what changed because of it. Frequently written after the build.
- How do you answer an access or erasure request for this system? Rarely tested.
- Which decisions are solely automated, and how does someone contest one? Often never listed.
- Who is your processor, where is the data, and what do your terms say about training? The terms are often unread.
The judgement to take away. Nothing here is exotic. It is ordinary data protection applied to a system whose data stores are unusually numerous, unusually operational and unusually easy to forget — and where one of them, the model itself, is unusually hard to take data back out of. That last point is why the decisions made before training matter more than the controls applied afterwards.
Twenty questions
Answer all twenty questions, then check your answers. You need 18 out of 20 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Standards and frameworks covered
The published frameworks this course teaches toward. Saolix is not affiliated with, endorsed by or accredited by any of these bodies; naming a standard is a citation, not a claim about its author.
National detail, by market
This course teaches the international standard and one worked example. The national detail for a market sits in its own pack, so it can be kept current without reopening the course: India, Japan.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- Data Protection Act, 2019 (No. 24 of 2019) · Kenya Law
- Guidance Note on Data Protection Impact Assessment · Office of the Data Protection Commissioner (Kenya)
- Guidance notes, including cross-border transfers and biometric data · Office of the Data Protection Commissioner (Kenya)
- The Digital Personal Data Protection Act, 2023 (section 8(6), breach intimation) · Ministry of Electronics and Information Technology, India
- Laws and policies on the protection of personal information · Personal Information Protection Commission, Japan
- Legal framework of EU data protection · European Commission
- Biometric data guidance (biometric recognition) · Information Commissioner's Office (UK)
- Artificial Intelligence Risk Management Framework (AI RMF 1.0) · US National Institute of Standards and Technology
- NIST AI 600-1: Generative Artificial Intelligence Profile · National Institute of Standards and Technology
- NIST SP 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence · US National Institute of Standards and Technology
- Personal Data Protection Act overview · Personal Data Protection Commission, Singapore



