What the lexical baseline missed

Q3: Broad topic matches outrank the labeled answer

How much current can this charger deliver to the battery?

Top five pages: 25, 26, 24, 10, 11. Hit@5: 0; recall@5: 0%; nDCG@5: 0.0000.

Observed evidence

Interpretation, not proven cause

The leading chunks and all three labeled pages share battery, charger and current. The first labeled page is only one position outside the cutoff. Token overlap alone does not distinguish a charging-state discussion from the requested current capability.

Next experiment — not yet run

Test a semantic retriever against this lexical baseline on development data, then use fresh questions to check whether improvements generalize.

Q10: Stages and states are different tokens

What are the different charging stages?

Top five pages: 37, 10, 14, 13, 12. Hit@5: 0; recall@5: 0%; nDCG@5: 0.0000.

Observed evidence

Interpretation, not proven cause

The query contains stages, while labeled chunk titles include Charger States and INIT State. Those best-scoring labeled chunks match only charging; the leading chunks also match different. This is consistent with vocabulary mismatch, but is not a causal ablation.

Next experiment — not yet run

Compare a small, documented query-expansion rule with dense retrieval. Inspect the source page before assuming every retrieved chunk title accurately describes its full contents.

Q13: Order does not match ordering

Where do I find the part numbers to order this chip?

Top five pages: 1, 21, 33, 24, 36. Hit@5: 0; recall@5: 0%; nDCG@5: 0.0000.

Observed evidence

Interpretation, not proven cause

The labeled page is headed Ordering Information, but its best-scoring chunk matches only part from the query. The tokenizer has no stemming. Pages matching numbers or the unrelated use of order rank above it.

Next experiment — not yet run

Test stemming or a targeted order/ordering expansion on development cases; measure regressions as well as gains.

Q15: A hit hides missing primary evidence

What protections does the charger provide?

Top five pages: 36, 21, 33, 17, 26. Hit@5: 1; recall@5: 25%; nDCG@5: 0.1086.

Observed evidence

Interpretation, not proven cause

Only supporting page 26 appears in the first five. The grade-2 page 20 is at rank 32. The best-scoring chunks on all labeled pages match charger, while the leading pages also match provide.

Next experiment — not yet run

Review singular/plural protection vocabulary and the source labels, then test coverage-aware retrieval. A positive hit metric alone is insufficient for this question.

Q5: Shared USB vocabulary retrieves partial evidence

How do I make the charger produce 5V for a USB accessory?

Top five pages: 13, 10, 17, 33, 8. Hit@5: 1; recall@5: 25%; nDCG@5: 0.1193.

Observed evidence

Interpretation, not proven cause

Supporting page 17 is third, but complete-answer pages 29 and 27 are ninth and thirty-sixth. The leading pages share 5v, charger and usb. This establishes a coverage failure against the labels; it does not establish an extraction defect.

Next experiment — not yet run

Inspect all chunks on the missing pages and the PDF before changing extraction. Then compare dense and hybrid retrieval under the same page-level scoring.

What this changes

The baseline makes three complete top-five misses and two partial-evidence failures inspectable. The next step is a controlled experiment followed by evaluation on fresh documents. No fix or performance gain is claimed here.