Reliable LLM evaluation pipeline

Question

# Question

# OpenAI Interview Question: Reliable LLM Evaluation Pipeline

You are building an internal evaluation pipeline for comparing multiple LLMs.

The pipeline has three components:

1. Running model inference with timeout and restart.
2. Computing entropy from model logits under strict memory constraints.
3. Assigning human annotation tasks while balancing coverage.

---

## Part 1: Inference Time and Restart Policy

An LLM inference job has random running time `T`.

You only know that:

```text
E[T] = 1
```

You do not know the distribution of `T`.

### 1.1

Give the best general upper bound you can for:

```text
P(T > 5)
```

using only `E[T] = 1`.

### 1.2

Suppose you are allowed to restart the inference job if it has not finished after 5 seconds.

You have a total time budget of 10 seconds.

Assume every restart is independent and has the same distribution as the original run.

Using the bound from 1.1, lower-bound the probability that the job succeeds within 10 seconds.

### 1.3

Generalize the result.

If each attempt is killed after `K` seconds, and you can try `N` independent attempts, what lower bound can you give for success probability?

### 1.4

Is there always

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.