AI Red Team Engineering notes, research references, and field notes on AI systems, responsible AI, governance, and compliance.
Global search

Search AI Red Team

Search across the complete knowledge base.

Enter at least two characters to begin.
↑↓ select Enter open Browse the full research archive
AI Engineer · June 24, 2025

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

video AI Engineering
RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot video thumbnail
Why it matters

AI Engineer session on RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot. It adds practical context for how teams are building and operating AI systems in production.

My takeaway: RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot is a model-evaluation signal. The practical read is to tie capability claims to evidence, launch criteria, and regression tests rather than relying on demos or benchmark headlines.
Keep exploring

More curated notes connected through AI Engineering.

OpenAI News · guide

The Defender’s Window

OpenAI describes a staged program for AI-assisted defense: use agents to review code and infrastructure, triage alerts, enumerate attack paths, and validate security invariants while retaining strong isolation and least privilege. Its recommended rollout starts with internet-facing services and vulnerability backlogs, moves security review into CI, requires focused fixes and regression tests, and expands from read-only triage to narrowly bounded automation only after teams build evidence and confidence.

Google Cloud Security Blog · tool

Now in preview: Find and fix software vulnerabilities with CodeMender

Google opened a preview of CodeMender, an AI code-security agent delivered through Gemini Enterprise Agent Platform and AI Threat Defense. It is designed to inspect code, identify and validate potentially exploitable defects, and produce targeted fixes, with Google’s specialized Gemini 3.5 Flash Cyber model initially restricted to governments and trusted partners.

AWS Security Blog · guide

Transforming Bedrock Guardrails events into OCSF with CloudWatch

AWS provides an implementation guide for a Lambda pipeline that converts Bedrock Guardrails intervention logs into OCSF Detection Findings in the CloudWatch unified data store. It includes field mapping and queries that correlate guardrail events with identity and network activity.