When a college or university considers using an AI tool, the conversation quickly turns from use cases to security and privacy. Let’s discuss how these providers handle security and compliance and what to ask before sending student data to a model. We’ll start with foundational knowledge in how AI works to better prepare for the security conversation with any given vendor. 

What is an LLM? 

An LLM (large language model) is a type of AI that learns statistical patterns in text and predicts what comes next. It doesn’t “think” like a person — it looks at the words you give it, turns them into numbers the model understands, and predicts what comes next. It’s simply a natural language prediction algorithm. For more information on what AI is and different flavors of AI, check out our previous post AI Primer for Higher Education. 

To develop an LLM, it requires a significantly large amount of training data. It needs to map patterns in language to an algorithm. This makes it advantageous for a company such as OpenAI or Google to store every chat interaction and use that to train the next ChatGPT or Gemini, in a sense “learning from past mistakes.” 

The word “provider” is often thrown around a lot, and it can mean two different things. The model developer is the company that built and trained the model. The host/operator is the company that hosts these models and serves it to users (this might be the same company or a cloud partner). For example, even though a company such as Anthropic or OpenAI develops the model, they may give it to Google to host due to the massive hardware requirements. It’s important to evaluate both sides of this coin when considering any AI tool because both vendors may have access to your data depending on the contracts you’ve signed. 

Security and Compliance Considerations 

The first thing to investigate when considering AI is state mandates revolving around usage. While most of these currently affect public institutions, it’s important to look at consumer privacy laws that will also encompass private institutions. Currently, many states around the U.S. have state specific mandates regarding how AI can and cannot be utilized. For example, Illinois signed into law a policy that prevents courses being taught solely by AI. In most states this appears to be mainly K-12 mandates, but some if not most resonate into higher education as well. Every college or university should include Legal and Academic Affairs in this decision-making process to effectively account for state mandates that may affect model usage. With so much change overnight, it’s important for these departments to understand campus usage, and steer towards proper usage if mandates were to change tomorrow. 

When evaluating a model developer and host, ensure that they are not using your chat history to train their future models . This is an evaluation that needs to be done for every model. It’s important because if they’re using your data to train future models, they are storing that data and it could be found in the newer model release. Confirm these companies have security measures in place for the APIs that will send and receive your data. Most business/education licenses with these vendors will conform to the above. This information can typically be found on the model developers’ website. For example, OpenAI has a landing page for privacy and security  while Anthropic has a Trust Center. 

Even though providers will publish security and privacy articles, it’s also important to use these tools properly. This boils down to sending AI the relevant details it needs to give you the answer you’re looking for. If a student’s personal information isn’t relevant to the answer, ensure end users aren’t supplying it in the chat window. FERPA (Family Educational Rights and Privacy Act) means schools should say when staff are acting as “school officials” with a legitimate reason to see student data, and when data is being handled by an outside vendor — which needs a formal agreement. Educate employees to ensure they know the difference between viewing data as part of their job versus sending it to a vendor. 

Here are some example questions regarding security and privacy when evaluating vendors in the space: 

1. Data Retention & Training

• • Do you retain customer inputs or outputs for training or model improvement? If yes, can we opt out contractually? 

• • How long do you retain inputs, API logs, and backups? Are deletion requests actionable and auditable? 

2. Data Residency & Storage

• • Where is customer data stored (region/country)? Can we require data residency or geographic controls? 

• • Is data encrypted at rest and in transit? Who controls the encryption keys (vendor vs. customer)? 

3. Access Controls & Personnel

• • What access controls and role-based permissions exist for vendor staff? Are there privileged access audits and least-privilege controls? 

• • Do you use sub-processors? Provide a current list and your notification process for changes. 

4. Compliance & Certifications

• • What certifications do you hold (SOC 2, ISO 27001, FedRAMP)? Can you provide recent audit reports or attestation letters? 

• • How do you handle FERPA/HIPAA-covered data? Do you sign a Data Processing Agreement (DPA) that addresses these laws and any contractual safeguards? 

5. Deployment Options & Controls

• • Can we use private deployment options, customer-managed keys (CMKs), or on-prem hosting to isolate data from shared multi-tenant systems? 

• • Are there options for disabling logging or ensuring customer data is not used to improve shared models? 

6. Incident Response & Legal

• • What is your incident response plan and breach notification timeline? Provide recent examples and SLAs for notification. 

• • Under what legal process will you disclose customer data to governments or third parties? How will you notify customers? 

Any vague answers should be flagged for further review. Send them to your legal and compliance teams for contractual language. The core items to be concerned with on this list are encrypted endpoints, not training using customer data, and compliance certifications. If a vendor fails in any of these areas, you likely should not use their solution. 

AI tools can bring real benefits to teaching, advising, and administration, but they can also introduce new data-handling risks. Treat each vendor as a separate decision by understanding how the model is hosted and will be used and then verifying contractual and technical protections. It’s crucial to distinguish public data from school data from confidential data. Map the data you’ll want to process, classify the sensitivity of this data, and consider what mandates are in place. By including security and compliance early in the process, you ensure your institution is adequately protected and ready to use these models to their full capacity, enabling efficiencies across the departments requesting use.