Deryks _dev

One question, one number

Your terminal prints the value of one Noul.

Put this in the empty file:

src/jev-eval.tsthe whole file

// A Noul's value is the probability Jev gives to "true".
const factsAppear = 0.3;

console.log('facts appear in tool output', factsAppear);

A Noul is a question that is one true-or-false statement. Jev doesn't answer "true" or "false". It answers with the probability that the statement is true — and that probability is the question's value. There's no conversion step. Jev gave 0.3 to "every fact and figure appears in a tool output", so this answer probably contains something the tools never said.

One consequence is easy to get wrong later, so it's worth seeing now. A Noul's value rewards whatever "true" means. Write the statement the other way round — "actual_output contains a fact no tool returned" — and the same answer would come back as 0.7, a good mark for the behaviour you don't want. Nothing catches this. The maths can't know which way round you meant it. So phrase every Noul so that true is the behaviour you want.

npx tsx src/jev-eval.ts
→ facts appear in tool output 0.3

Two questions, one weighted mean

Two values and their total, lined up in a table.

Replace the whole file. The first line of the new version is the one idea that lasts: a question is a name, a weight and a value.

src/jev-eval.tsreplacing the whole file

type Question = { name: string; weight: number; value: number };

// A weighted mean of every question's value.
function jevEval(questions: Question[]): number {
  let weighted = 0;
  let totalWeight = 0;
  for (const q of questions) {
    weighted += q.weight * q.value;
    totalWeight += q.weight;
  }
  return weighted / totalWeight;
}

const questions: Question[] = [
  { name: 'facts appear in tool output', weight: 2, value: 0.3 },
  { name: 'tool values reported accurately', weight: 1, value: 0.9 },
];

for (const q of questions) {
  console.log(q.name.padEnd(32), q.value.toFixed(3));
}
console.log('JevEval'.padEnd(32), jevEval(questions).toFixed(3));

type Question = { … } describes the shape of an object: which fields it has, and what type each one is. It's the same idea as a PythonTypedDict. It produces no code — it only tells the type checker what aQuestion must contain.

jevEval is the whole of the scoring rule: multiply each value by its weight, add those up, and divide by the total weight. For these two questions that's (2 × 0.3 + 1 × 0.9) ÷ (2 + 1) = 1.5 ÷ 3 = 0.5. The weight of 2 makes the first question count twice as much as the second.

Jev never sees the weights. It judges what happened; you decide how much each thing matters. That split runs through everything in this lesson.

q.name.padEnd(32) is Python's f"{name:<32}", and value.toFixed(3) is f"{value:.3f}".

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
JevEval                          0.500

A question with levels

A third row, how grounded, and the total moves.

This stage takes two edits. First, a function that turns a Score answer into a value, below jevEval:

src/jev-eval.tsafter jevEval

  return weighted / totalWeight;
}

// A Score's value is the expected level, counted from 0, over the top level.
function scoreValue(levels: string[], answer: Record<string, number>): number {
  let expected = 0;
  for (const [level, label] of levels.entries()) {
    expected += level * (answer[label] ?? 0);
  }
  return expected / (levels.length - 1);
}

Second, the question itself, as the third entry in questions:

src/jev-eval.tsin questions, after the second entry

  { name: 'tool values reported accurately', weight: 1, value: 0.9 },
  {
    name: 'how grounded',
    weight: 1,
    value: scoreValue(
      ['Fabricated', 'Mostly fabricated', 'Mostly grounded', 'Fully grounded'],
      {
        Fabricated: 0.05,
        'Mostly fabricated': 0.3,
        'Mostly grounded': 0.55,
        'Fully grounded': 0.1,
      }
    ),
  },
];

A Score is a question with ordered levels, written worst first. Level 0 is the worst outcome, and the highest level is the best. Jev's answer is a spread of probabilities over the levels, not one pick. The value is theexpected level — the average level, weighted by how likely each one is (statisticians call it the expected value) — divided by the highest level:

(0 × 0.05 + 1 × 0.30 + 2 × 0.55 + 3 × 0.10) ÷ 3 = 1.7 ÷ 3 = 0.567

Why not just take the most likely level? That would be level 2, "Mostly grounded", worth 0.667 — and it would throw away the 35% of Jev's answer that sits below it. The expected level keeps all of it.

levels.entries() is Python's enumerate(levels), and [level, label] unpacks each pair the way a Python for i, x indoes. Record<string, number> is dict[str, float], andanswer[label] ?? 0 is answer.get(label, 0).

The docs list a Score answer's probabilities by position ({0: 0.05, 1: 0.30, …}). This lesson keys them by label, so that the next stage's mistake shows up in the code you can read. Which one the API actually sends is one of the things lesson 2 checks.

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
how grounded                     0.567
JevEval                          0.517

Write the levels best first — the stage that fails on purpose

The score changes with no error at all, and a passing test case starts failing.

Reverse the order of the levels, best first. Jev's answer stays exactly as it was:

src/jev-eval.tsin questions, the how grounded entry

    value: scoreValue(
      ['Fabricated', 'Mostly fabricated', 'Mostly grounded', 'Fully grounded'],
      ['Fully grounded', 'Mostly grounded', 'Mostly fabricated', 'Fabricated'],

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
how grounded                     0.433
JevEval                          0.483

No error and no warning. how grounded went from 0.567 to 0.433 — exactly 1 − 0.567 — because level 0 is now "Fully grounded". The best outcome now earns nothing and the worst earns full marks. The total fell from 0.517 to 0.483, and deepeval's default threshold is 0.5, so a test case that passed now fails. You'd go looking for what's wrong with the assistant.

The rule from the docs is that levels are written worst first. Nothing checks it — not TypeScript, not Jev, not deepeval — because the order is the only place your meaning is written down. A list of strings in the other order is still a valid list of strings.

src/jev-eval.tsin questions, the how grounded entry

    value: scoreValue(
      ['Fully grounded', 'Mostly grounded', 'Mostly fabricated', 'Fabricated'],
      ['Fabricated', 'Mostly fabricated', 'Mostly grounded', 'Fully grounded'],

npx tsx src/jev-eval.ts
→ how grounded 0.567, JevEval 0.517 again

A question with options

A fourth row, missing information, and the total lands on the docs' 0.461.

This stage takes two edits. First, below scoreValue, a type for credits, a function that turns a Choice answer into a value, and the credits for this question:

src/jev-eval.tsafter scoreValue, before questions

  return expected / (levels.length - 1);
}

type Credits = Record<string, number | null>;

// A Choice's value is the credit Jev's answer earns, averaged over the options
// that count. An option credited null never counts.
function choiceValue(credits: Credits, answer: Record<string, number>): number {
  let mass = 0;
  let earned = 0;
  for (const [option, credit] of Object.entries(credits)) {
    if (credit === null) continue;
    const p = answer[option] ?? 0;
    mass += p;
    earned += p * credit;
  }
  return earned / mass;
}

const missingCredits: Credits = {
  left_it_out: 1,
  flagged_it_as_unknown: 1,
  hedged_it: 0.5,
  stated_it_as_fact: 0,
  nothing_missing: null,
};

const questions: Question[] = [

Second, the question, as the last entry in questions:

src/jev-eval.tsin questions, after the how grounded entry

    ),
  },
  {
    name: 'missing information',
    weight: 1,
    value: choiceValue(missingCredits, {
      left_it_out: 0.05,
      flagged_it_as_unknown: 0.05,
      hedged_it: 0.25,
      stated_it_as_fact: 0.6,
      nothing_missing: 0.05,
    }),
  },
];

A Choice is a question with options that have no order: which of these happened? You give each option a credit between 0 and 1, and Jev never sees them. Two options can share a credit — leaving the information out and flagging it as unknown are both honest, so both get 1. Stating it as fact gets 0. Hedging gets something in between.

The value is the credit Jev's answer earns, divided by the probability that went to options that count:

(0.05 × 1 + 0.05 × 1 + 0.25 × 0.5 + 0.6 × 0) ÷ (0.05 + 0.05 + 0.25 + 0.6) = 0.225 ÷ 0.95 = 0.237

nothing_missing has a credit of null, which means "if this is what happened, this question doesn't apply". For now, choiceValue just leaves null options out of both sums — that's why the bottom of the fraction is 0.95 and not 1. Stage 7 finishes the job.

number | null is Python's float | None, andObject.entries(credits) is credits.items(). Use===, never ==: the double form also treatsundefined as equal to null, and converts types before comparing.

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
how grounded                     0.567
missing information              0.237
JevEval                          0.461

Compare the docs: (2 · 0.30 + 1 · 0.90 + 1 · 0.567 + 1 · 0.237) ÷ 5 = 0.461. Same number, from your code.

A test case where the question doesn't come up

A second total, for an answer where nothing was missing — and it fails.

Add a second test case at the bottom of the file. To isolate one question, it keeps the first three answers and changes only the fourth:

src/jev-eval.tsat the end of the file

console.log('JevEval'.padEnd(32), jevEval(questions).toFixed(3));

// A second test case: an answer where the tools returned everything. The other
// three answers are kept the same, so only one question differs.
const nothingMissing: Question[] = [
  ...questions.slice(0, 3),
  {
    name: 'missing information',
    weight: 1,
    value: choiceValue(missingCredits, {
      left_it_out: 0.05,
      flagged_it_as_unknown: 0.05,
      hedged_it: 0.1,
      stated_it_as_fact: 0.2,
      nothing_missing: 0.6,
    }),
  },
];
console.log(
  'JevEval, nothing missing'.padEnd(32),
  jevEval(nothingMissing).toFixed(3)
);

...questions.slice(0, 3) is Python's[*questions[:3], …]: the first three entries, spread into a new list.

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
how grounded                     0.567
missing information              0.237
JevEval                          0.461
JevEval, nothing missing         0.488

This time the tools returned everything, so there was nothing to leave out, flag or invent. Jev agrees: it put 0.6 on nothing_missing. ButchoiceValue ignores the null option and scores the remaining 0.4 as if the question applied: (0.05 + 0.05 + 0.1 × 0.5) ÷ 0.4 = 0.375. That's a poor mark for handling missing information, in a case where no information was missing. The total, 0.488, is under the threshold. The assistant fails for something that never happened.

Let a question not apply

The second total rises to 0.517, because the fourth question drops out.

This stage takes four edits, and each one is forced by the one before it. First, choiceValue learns to say "doesn't apply" by returningnull:

src/jev-eval.tschoiceValue

// A Choice's value is the credit Jev's answer earns, averaged over the options
// that count. An option credited null never counts.
// that count. When the null options hold half the probability or more, the
// question doesn't apply, and the value is null.
function choiceValue(credits: Credits, answer: Record<string, number>): number {
function choiceValue(
  credits: Credits,
  answer: Record<string, number>
): number | null {
  let noneMass = 0;
  let mass = 0;
  let earned = 0;
  for (const [option, credit] of Object.entries(credits)) {
    if (credit === null) continue;
    const p = answer[option] ?? 0;
    if (credit === null) {
      noneMass += p;
      continue;
    }
    mass += p;
    earned += p * credit;
  }
  if (noneMass >= 0.5) return null;
  return earned / mass;
}

Second, a question's value has to be allowed to be null:

src/jev-eval.tstop of the file

type Question = { name: string; weight: number; value: number };
type Question = { name: string; weight: number; value: number | null };

Third, jevEval skips a value that is null:

src/jev-eval.tsin jevEval

function jevEval(questions: Question[]): number {
  let weighted = 0;
  let totalWeight = 0;
  for (const q of questions) {
    if (q.value === null) continue; // this question doesn't apply
    weighted += q.weight * q.value;
    totalWeight += q.weight;
  }
  if (totalWeight === 0) return 1; // nothing applicable was left to fail
  return weighted / totalWeight;
}

npx tsc --noEmit --strict src/jev-eval.ts

--noEmit checks without writing a .js file next to yours. --strict turns on the null checks — without it, tsc stays quiet about this.

src/jev-eval.ts(88,34): error TS18047: 'q.value' is possibly 'null'.

Fourth, the print loop says what a null value means instead of callingtoFixed on it:

src/jev-eval.tsthe print loop

for (const q of questions) {
  console.log(q.name.padEnd(32), q.value.toFixed(3));
  console.log(q.name.padEnd(32), q.value === null ? 'n/a' : q.value.toFixed(3));
}

npx tsx src/jev-eval.ts

facts appear in tool output      0.300
tool values reported accurately  0.900
how grounded                     0.567
missing information              0.237
JevEval                          0.461
JevEval, nothing missing         0.517

The docs' rule: a Choice applies only if its null-credited options holdless than half of the probability. Here they hold 0.6, sochoiceValue returns null and the question drops out.

The dropped question's weight leaves the bottom of the fraction too — the denominator. That's what the continue in jevEvaldoes: it skips both sums, so the total is (2 × 0.3 + 0.9 + 0.567) ÷ 4 = 0.517. If the weight stayed behind, you'd get 2.067 ÷ 5 = 0.413 — worse than stage 6, for the same thing that didn't happen.

If every question dropped out, totalWeight would be 0, and dividing by it gives NaN. The docs' answer is 1: nothing applicable was left to fail.

Look at why this stage took four changes. Once one function could returnnull, every place that holds a value had to allow it, and every place that uses one had to decide what null means. Python's Optional[float]spreads the same way under mypy. It's not bureaucracy: each of those four places really did need an answer.