Where to start?

It could depend on different factors, but first - get to know most base / ambiguous concepts at Taxonomy. For example, what is trace, span, …. It gives us alignment these required concepts and tools to investigate the domain further.

If you need to start now. Sequence:

0. problem framing and broader concepts from AI Problem Cycle. More data: [[outer-cycle]]
1. get product understanding - mvp, guarantess, JTBDs. These two sections provide a basis to reason about and introduce evaluations, like understand if this failure is critical or not.
2. sdf

Then if you have time - read descriptive knowledge about eval framework. For example, the structure, .. Why I’ve introduced several loops here. Offline/online Loops are just additional views where we better understand some moments.