V1 on Systems & Engineering

01 Starting Sequence

Mon, 01 Jan 0001 00:00:00 +0000

Based on the attached files, we have different levels of startup definition for the three loops. The material describes methods and interfaces; it does not establish that the corresponding product, tooling, datasets, or processes have already been implemented.

10 User Inputs

Mon, 01 Jan 0001 00:00:00 +0000

Building a Starting Set of User Inputs for AI Evaluation

This guide addresses building a representative set of user inputs. These inputs form one component of the evaluation dataset and define the portion of the query space that should cover the important ways users may interact with the application.

11 Building Balanced Set

Mon, 01 Jan 0001 00:00:00 +0000

Building a balanced starting evaluation set

A balanced evaluation set represents the parts of the product that matter, in proportions appropriate to their importance and risk. Here we’re building 10 User Inputs user inputs for evaluation set, but also the methods and principles could be applied to other or more broad concepts.

20 Error Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Failure Understanding: Discovering and Structuring How LLM Systems Fail

Intro

LLM applications rarely fail in only one visible place. A poor final response may originate in an earlier misunderstanding, an incorrect tool call, missing context, stale environment state, or a failure to preserve a user constraint. Evaluating only the final response can therefore conceal both the first visible problem and the way it propagates through the execution.

AI Evaluation as an Engineering Learning System

Mon, 01 Jan 0001 00:00:00 +0000

AI evaluation connects product intent with observed system behaviour. It turns human and automated judgment into reusable knowledge and uses that knowledge to produce evidence for product decisions.

AI Evaluation as an Iterative Engineering Practice

Mon, 01 Jan 0001 00:00:00 +0000

AI evaluation connects product intent, observed system behaviour, and a defined decision or knowledge need. It uses deliberate probes and operational observations to produce evidence, develops explicit Quality Understanding, and applies that understanding to product decisions.

Conceptualization

Mon, 01 Jan 0001 00:00:00 +0000

I want to build a conceptual movel–or environment–within which our evaluations (evals) subsystem will be designed and will operate. Currently, we use evals for specific tasks, and these tasks exist within a broader context. We can use this framework to guide and validate the design of our evals.

Conceptualization 2

Mon, 01 Jan 0001 00:00:00 +0000

Evaluation Conceptualization

1. Purpose

An evaluation system exists because the behavior of an AI product cannot be inferred reliably from its specification or implementation alone.

Guide

Mon, 01 Jan 0001 00:00:00 +0000

Where to start?

It could depend on different factors, but first - get to know most base / ambiguous concepts at Taxonomy. For example, what is trace, span, …. It gives us alignment these required concepts and tools to investigate the domain further.

Evaluation of LLM Workflows and Coding Agents

Mon, 01 Jan 0001 00:00:00 +0000

This page covers evaluation for LLM-based workflows and coding agents.

The goal is to make agent behavior observable, repeatable, and comparable across prompts, tools, models, orchestration strategies, and workflow designs.