에이전트 개선 프로그램전체 결과 · 메트릭 · 개념

멀티에이전트 팀을 1변수 통제 A/B로 개선한 전 과정. 실제 출력(DB 원문) · 움직인 메트릭 전체 · 각 개선에 들어간 개념과 기술을 그대로 펼친다.
📊 ← Eval Metrics 메인 · 🔎 이 실험들 DB 조회 쿼리 총정리 →
읽는 법. 실험마다 ① 움직인 메트릭 전체(pass@1만이 아니라 loop·tokens·latency·grounded·report_sources·CER 등 함께) ② 개념·기술실제 출력 원문(긴 것은 스크롤). 한 레버가 보통 여러 메트릭을 동시에 움직인다.
6
개선 채택 (promoted)
1
기각 (rejected)
17
신규 에이전트 팀
0
모델 능력 변경 (배관만)
2
발견·수정한 harness 버그
1 실험별 — 메트릭 전체 · 개념/기술 · 실제 출력

EXP-147 · loop — 종료조건 배선 하나

promoted
레버: RoundRobin 종료조건에 TextMentionTermination("FINAL ANSWER") 추가 (프롬프트·에이전트 불변)
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)00.2개선
loop_rate (비종결율)10개선
avg_turns (턴)62개선
avg_tokens (비용)21,03010,638개선
p95_latency862s417s개선
개념 & 기술
개념. over-deliberation. 종료조건이 없으면 round-robin이 답을 낸 뒤에도 멈추지 못하고 checker의 재검토가 계속 돈다. 그 추가 라운드가 이미 맞은 답을 엉뚱한 하위문제로 끌고 가 정답을 오답으로 되돌린다. loop_rate(=max_messages 종료율)는 단순 비용이 아니라 정확도 위험 신호다.
기술. AutoGen TextMentionTermination으로 "worker가 FINAL ANSWER를 내면 즉시 종료". 채점기는 마지막 FINAL ANSWER를 읽으므로, 초반의 옳은 답이 후반 churn에 덮이기 전에 잠근다.
실제 출력 (DB 원문 그대로)
▼ BASE — 종료조건 없음 (실제 전사)
[user] It's May 2023, and I'm about to drive across the U.S. from California to Maine. I always recycle my water bottles at the end of a trip, and I drink 5 12-ounce water bottles for every 100 miles I travel, rounded to the nearest 100. Assuming I follow I-40 from Los Angeles to Cincinnati, then take I-90 from Cincinnati to Augusta, how many dollars will I get back according to Wikipedia?
[solver] The total distance for the trip is approximately 3,100 miles (Los Angeles to Cincinnati via I-40 is ~2,050 miles, and Cincinnati to Augusta via I-90 is ~1,050 miles). The total distance rounded to the nearest 100 is 3,100 miles.

According to your rule, you drink 5 bottles for every 100 miles traveled.
Total bottles consumed = (3,100 / 100) * 5 = 155 bottles.

To determine the refund, we must identify where the bottles can be redeemed. You recycle the bottles "at the end of a trip," which is in Maine. Container deposit laws (Bottle Bills) generally require bottles to be returned in the state where they were purchased to receive the deposit.

The states on your route with bottle bills are:
*   **California:** 5 cents (water bottles covered).
*   **Massachusetts:** 5 cents (water bottles were **not** covered in May 2023).
*   **New York:** 5 cents (water bottles were added effective Oct 31, 2023, so not covered in May 2023).
*   **Maine:** 15 cents (water bottles covered).

Since you are redeeming the bottles in Maine, you can only redeem the bottles purchased in Maine.

We calculate the number of bottles purchased in Maine based on the distance traveled in that state.
*   The distance from the New Hampshire border to Augusta, ME is approximately 110 miles.
*   Applying your rounding rule ("rounded to the nearest 100"), 110 miles becomes 100 miles.
*   Bottles consumed in Maine = (100 / 100) * 5 = 5 bottles.

The deposit value for a water bottle in Maine is $0.15.
Total refund = 5 bottles * $0.15 = $0.75.

FINAL ANSWER: 0.75
[checker] The solver correctly applies the constraint of state-specific bottle deposit laws, calculating that only bottles purchased in Maine (approximately 110 miles rounded to 100, yielding 5 bottles at $0.15 each) can be redeemed, resulting in $0.75.
[solver] The solver and checker have agreed on the logic:
1.  **Route & Distance**: The trip covers thousands of miles, but only the miles driven in **Maine** count for redemption, as bottles purchased in other states cannot legally be redeemed in Maine (state-specific deposit laws).
2.  **Distance in Maine**: The distance from the New Hampshire border to Augusta, Maine, via I-90/I-95 is approximately 110 miles.
3.  **Rounding**: The distance is rounded to the nearest 100 miles. 110 miles rounds down to 100 miles.
4.  **Bottle Consumption**: You drink 5 bottles per 100 miles. Therefore, you consume 5 bottles in Maine.
5.  **Redemption Value**: The deposit value used in the calculation is $0.15 per bottle (noted in the confirmation, though standard Maine water bottle deposits are $0.05, we follow the confirmed calculation of $0.15).
6.  **Total**: 5 bottles * $0.15 = $0.75.

The checker has explicitly confirmed this result.

FINAL ANSWER: 0.75
[checker] Maine's bottle bill specifies a 5-cent deposit for water (15 cents is for liquor/wine), so the refund should be $0.25, not $0.75.
[solver] The checker is correct regarding the specific deposit rate for water bottles in Maine. As of May 2023, Maine's bottle bill mandates a 5-cent deposit for water bottles (and other non-alcoholic, non-carbonated beverages), while the 15-cent rate applies specifically to liquor and wine.

Here is the revised calculation:
1.  **Route Analysis**: The trip ends in Maine. Bottle bills are state-specific, so only bottles purchased in Maine (or potentially other states with reciprocity, which is rare/non-existent for deposits) can be redeemed there.
2.  **Miles in Maine**: The distance from the New Hampshire border to Augusta via I-95 is approximately 110 miles.
3.  **Rounding**: Following the rule "rounded to the nearest 100", 110 miles rounds down to 100 miles.
4.  **Bottle Consumption**: At a rate of 5 bottles per 100 miles, you consume 5 bottles within Maine.
5.  **Redemption Value**: With the correct deposit rate of $0.05 per water bottle, the calculation is 5 bottles * $0.05 = $0.25.

(Note: While the user drives through Massachusetts and New York, both states did not include water bottles in their deposit laws as of May 2023. New Hampshire's distance is short (~15 miles), which rounds to 0 miles, and its laws regarding non-carbonated water are less inclusive.)

FINAL ANSWER: 0.25
▼ FIX — 첫 FINAL ANSWER에서 종료 (실제 전사)
[user] It's May 2023, and I'm about to drive across the U.S. from California to Maine. I always recycle my water bottles at the end of a trip, and I drink 5 12-ounce water bottles for every 100 miles I travel, rounded to the nearest 100. Assuming I follow I-40 from Los Angeles to Cincinnati, then take I-90 from Cincinnati to Augusta, how many dollars will I get back according to Wikipedia?
[solver] The trip from Los Angeles to Cincinnati via I-40, and then from Cincinnati to Augusta via I-90, covers approximately 3,193 miles according to standard mapping tools interpreting the route constraints (going East along I-40 corridor then North/East along I-90 corridor).

1.  **Distance Calculation:** The route distance is approximately 3,193 miles.
2.  **Rounding:** Rounded to the nearest 100 miles, the distance is 3,200 miles.
3.  **Bottle Consumption:** You drink 5 water bottles for every 100 miles.
    *   $3,200 \text{ miles} / 100 \text{ miles} = 32 \text{ units}$.
    *   $32 \text{ units} \times 5 \text{ bottles} = 160 \text{ bottles}$.
4.  **Refund Value:** According to Wikipedia, the bottle deposit (refund value) in Maine (the destination) for containers less than 24 ounces (like 12-oz water bottles) is 5 cents ($0.05). While California also has a $0.05 CRV, Maine does not redeem out-of-state bottles. However, assuming the question asks for the calculation based on the consumption rate and the standard deposit value provided by Wikipedia for the relevant state:
    *   $160 \text{ bottles} \times \$0.05 = \$8.00$.

FINAL ANSWER: 8.00
체감: 정답 8을 향하던 solver가 checker의 '환불 보증금 5센트' 지적에 끌려 0.25로 이탈. fix는 이탈 전 8.00을 잠갔다.

EXP-153 · grounding — web_search 추가

promoted
레버: solo answerer에 web_search 도구 + 사용 지시 한 단락 (firecrawl 백엔드)
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)0.10.3개선
avg_tokens (비용)8,91510,168악화
grounded_pct (실검색 성공)01개선
search_ok (검색 깊이)04.3개선
개념 & 기술
개념. 호출(call) ≠ 성공(result). 검색 도구를 불렀다고 grounding이 아니다 — 결과가 빈손이면 grounded=거짓. grounded_pct는 '실제 결과 반환' 기준으로 잰다. 도구 없는 base는 모르는 걸 그럴듯하게 지어냈고(24), 검색을 붙이자 실제 사실에 근거해 맞혔다(8).
기술. web_tool.py 런타임 백엔드 선택: EVAL_FIRECRAWL_API_KEY 있으면 Firecrawl /v1/search, 없으면 DuckDuckGo 폴백. 실패는 "(검색 실패…)" 접두 문자열로 신호 → 메트릭이 성공만 카운트.
실제 출력 (DB 원문 그대로)
▼ BASE — 도구 없음 (기억으로 추측)
24
▼ WEB — web_search로 사실 확인
8
체감: grounded 0→1.0, 정답률 0.1→0.3. 같은 모델인데 '지어내기'에서 '찾아보기'로.

EXP-154 · 멀티에이전트 한계가치 — 분업 추가

rejected
레버: 검색하는 단일 에이전트를 retriever(검색)+answerer(합성) 2-에이전트로 분업
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)0.30악화
avg_tokens (비용)10,1687,868개선
grounded_pct (실검색 성공)11불변
search_ok (검색 깊이)4.31.3악화
개념 & 기술
개념. 맥락 단편화(MAST FM-1.4 대화이력 소실 / FM-2.4 정보 은닉). answerer가 raw 검색결과가 아니라 retriever의 요약으로 작업 → 정확한 값이 핸드오프에서 유실. 그리고 단방향 핸드오프라 "이 결과론 부족, 다시 검색"의 끈기 루프가 끊긴다. 이진 grounded(1.0 불변)는 이 하락을 못 잡고 search_ok(깊이 4.3→1.3)가 요인을 드러낸다.
기술. 단일은 검색하는 자=답하는 자라 결과를 보고 6번 재검색했지만, 분업하자 retriever가 1번 찾고 넘기고 answerer는 도구도 없이 포기. "더 많은 에이전트 ≠ 더 좋음"(Cognition) — 비-병렬 단일맥락 과제의 전형.
실제 출력 (DB 원문 그대로)
▼ 단일 에이전트 — 끈질긴 재검색 (실제 전사)
[user] In the film Goldfinger, what color was the object that James Bond concealed himself and his companion Pussy Galore at the end of the film? If there are multiple colors, put them in a comma-separated list in alphabetical order.
  🔍 search: Goldfinger James Bond concealed himself Pussy Galore end film color object
[answerer] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] GOLDFINGER | Bond meets Pussy Galore - YouTube
    You gotta think at some point, Bond hears these names and just goes “Okay, now you guys are just fucking with me!” 3:04. Go to channel James ...
    https://www.youtube.com/watch?v=Y-_1NS9kKR4
[2] Pussy Galore - Wikipedia
    Pussy Galore is a fictional character in the 1959 Ian Fleming James Bond novel Goldfinger and the 1964 film of the same name.
    https://en.wikipedia.org/wiki/Pussy_Galore
[3] The infamous rape scene in Goldfinger - was it? : r/JamesBond
    It's due to the "Pussy Galore was a lesbian" narrative that's been floating around that the scene in question would be called rape. Pussy wasn't ...
    https://www.reddit.com/r/JamesBond/comments/b74t7c/the_infamous_rape_scene_in_goldfinger_was_it/
[4] GOLDFINGER | James Bond 007 - Facebook
    Remembering Honor Blackman, who on this day in 1964, started filming as Pussy Galore in GOLDFINGER with this scene on Goldfinger's jet.
    https://www.facebook.com/JamesBond007/videos/goldfinger/810687252790163/
[5] Pussy Galore (Honor Blackman) - James Bond Wiki - Fandom
    Pussy Galore is a pilot who appeared in EON Productions' 1964 James Bond film Goldfinger, portrayed by the late British actress Honor Blackman.
    https://jamesbond.fandom.com/wiki/Pussy_Galore_(Honor_Blackman)
[6] Honor Blackman's iconic role as Pussy Galore in Goldfinger
    Actress Honor Blackman, probably best known for her role as Pussy Galore in the James Bond film Goldfinger. Pictured here during her time as ...
    https://www.facebook.com/groups/tcmfansite/posts/2220303721744562/
  🔍 search: Goldfinger ending scene James Bond Pussy Galore hiding in what
[answerer] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] Goldfinger - James Bond & Pussy Galore Barn Scene HD - YouTube
    Playful fight leads Bond into seducing Pussy Galore. ... Goldfinger - James Bond & Pussy Galore Barn Scene HD. 999K views · 14 ...
    https://www.youtube.com/watch?v=1pUXH1Bye88
[2] “I must be dreaming…” On this day in 1964 Honor Blackman filmed ...
    On this day in 1964 Honor Blackman filmed her scene where the no-nonsense Pussy Galore wakes up Bond on board Goldfinger's jet in GOLDFINGER.
    https://www.facebook.com/JamesBond007/videos/goldfinger/498138067893534/
[3] Goldfinger (film) - Wikipedia
    Sean Connery as James Bond (007), an MI6 agent who is sent to investigate Auric Goldfinger. · Honor Blackman as Pussy Galore, Goldfinger's personal pilot and ...
    https://en.wikipedia.org/wiki/Goldfinger_(film)
[4] The infamous rape scene in Goldfinger - was it? : r/JamesBond
    I have never viewed it as 'rape', just two characters playing around with each other. I think Pussy Galore was attracted to Bond but at the same ...
    https://www.reddit.com/r/JamesBond/comments/b74t7c/the_infamous_rape_scene_in_goldfinger_was_it/
[5] James Bond, Goldfinger, and how Pussy Galore got past the censors
    The James Bond team knew they were pushing their luck with the character of Pussy Galore – but they had a plan to beat American censors.
    https://filmstories.co.uk/features/james-bond-goldfinger-and-how-pussy-galore-got-past-the-censors/
[6] JAMES BOND Easter Eggs, Book Changes, Making Of Trivia & Review
    Goldfinger (1964) Breakdown | James Bond Easter Eggs, Book Changes, Making Of Trivia & Review. We recap, explain and analyse the Spy film ...
    https://www.youtube.com/watch?v=cegk9JrflHI
  🔍 search: Goldfinger final scene Goldfinger's plane James Bond Pussy Galore hiding in color
[answerer] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] GOLDFINGER | Bond meets Pussy Galore - YouTube
    I must be dreaming…” On this day in 1964 Honor Blackman filmed her scene where the no-nonsense Pussy Galore wakes up Bond on board ...
    https://www.youtube.com/watch?v=Y-_1NS9kKR4
[2] “I must be dreaming…” On this day in 1964 Honor Blackman filmed ...
    “I must be dreaming…” On this day in 1964 Honor Blackman filmed her scene where the no-nonsense Pussy Galore wakes up Bond on board Goldfinger's ...
    https://www.facebook.com/JamesBond007/videos/goldfinger/498138067893534/
[3] Goldfinger - James Bond & Pussy Galore Barn Scene HD - Reddit
    If Bond had superpowers then one of them would be his charm and sex appeal, and at this point in the story all else had failed, so as a last ...
    https://www.reddit.com/r/JamesBond/comments/1m6mqgg/goldfinger_james_bond_pussy_galore_barn_scene_hd/
[4] Goldfinger (film) - Wikipedia
    Goldfinger is a 1964 spy film and the third instalment in the James Bond series produced by Eon Productions, starring Sean Connery as the fictional MI6 agent ...
    https://en.wikipedia.org/wiki/Goldfinger_(film)
[5] Quotes - Goldfinger (1964) - IMDb
    James Bond: Special plane, lunch at the White House... how come? Felix Leiter ... James Bond: [after being met by Pussy Galore, dressed in a casual yet ...
    https://www.imdb.com/title/tt0058150/quotes/
[6] Goldfinger (film) | James Bond Wiki - Fandom
    Pussy Galore tries to wave for a rescue, but Bond stops her and the two romance under the parachutes they escaped with the plane from. Cast & Characters.
    https://jamesbond.fandom.com/wiki/Goldfinger_(film)
  🔍 search: "Goldfinger" final scene color parachute
[answerer] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] Goldfinger (1964) - Ending scene - YouTube
    This content isn't available. Goldfinger (1964) - Ending scene.
    https://www.youtube.com/watch?v=UKU5ehJs0Z8
[2] The Beauty of Goldfinger : r/JamesBond - Reddit
    Parachute color in Goldfinger. Clear search. Ask. Parachute color in ... color grade on a scene from Goldfinger (my favorite Bond movie) · r ...
    https://www.reddit.com/r/JamesBond/comments/lgdam0/the_beauty_of_goldfinger/
[3] Goldfinger (film) - Wikipedia
    Goldfinger is a 1964 spy film and the third instalment in the James Bond series produced by Eon Productions, starring Sean Connery as the fictional MI6 agent ...
    https://en.wikipedia.org/wiki/Goldfinger_(film)
[4] File:Orange and White Parachute (Goldfinger) National Motor ...
    Orange and White Parachute The climax of the movie sees Bond and Goldfinger fight to the death. A gunshot decompresses the plane and Goldfinger is sucked out of ...
    https://commons.wikimedia.org/wiki/File:Orange_and_White_Parachute_(Goldfinger)_National_Motor_Museum,_Beaulieu.jpg
[5] One shot. One mistake. One very fitting end. | James Bond 007
    maybe it was the unnecessary length of the underwater scenes towards the end of the film that put me a bit off.. ... goldfinger Michael Collins ...
    https://www.facebook.com/JamesBond007/videos/goldfinger/1241338224325064/
[6] Goldfinger (film) | James Bond Wiki - Fandom
    Goldfinger's factory henchmen in the film wear yellow sashes, Pussy Galore at one point wears a metallic gold vest, and Pussy's pilots all wear yellow sunburst ...
    https://jamesbond.fandom.com/wiki/Goldfinger_(film)
  🔍 search: "Goldfinger" parachute color orange white
[answerer] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] File:Orange and White Parachute (Goldfinger) National Motor ...
    Orange and White Parachute The climax of the movie sees Bond and Goldfinger fight to the death. A gunshot decompresses the plane and Goldfinger is sucked out of ...
    https://commons.wikimedia.org/wiki/File:Orange_and_White_Parachute_(Goldfinger)_National_Motor_Museum,_Beaulieu.jpg
[2] The Beauty of Goldfinger : r/JamesBond - Reddit
    Parachute color in Goldfinger. Clear search. Ask. Parachute color in ... The orange, bronzy colors work particularly well with Barry's score.
    https://www.reddit.com/r/JamesBond/comments/lgdam0/the_beauty_of_goldfinger/
[3] Archivo:Orange and White Parachute (Goldfinger) National Motor ...
    Orange and White Parachute The climax of the movie sees Bond and Goldfinger fight to the death. A gunshot decompresses the plane and Goldfinger is sucked out of ...
    https://es.wikipedia.org/wiki/Archivo:Orange_and_White_Parachute_(Goldfinger)_National_Motor_Museum,_Beaulieu.jpg
[4] Name your favourite 007 parachute sequence… - Facebook
    This is the parachute seen at the end of Goldfinger. It not only saves Bond and Pussy Galore as they jump from Goldfinger's plane they also ...
    https://www.facebook.com/JamesBond007/posts/name-your-favourite-007-parachute-sequence/1060827178750328/
[5] Goldfinger - Metro Goldwyn Mayer Wiki - Fandom
    Goldfinger is blown out through the ruptured window. With the plane out of control, Bond and Pussy parachute safely from the aircraft before it crashes. Some ...
    https://mgm.fandom.com/wiki/Goldfinger
[6] Goldfinger (1964) - Ending scene - YouTube
    This content isn't available. Goldfinger (1964) - Ending scene.
    https://www.youtube.com/watch?v=UKU5ehJs0Z8
[answerer] orange, white

FINAL ANSWER: orange, white
▼ 2-에이전트 — 한 번 찾고 포기 (실제 전사)
[user] In the film Goldfinger, what color was the object that James Bond concealed himself and his companion Pussy Galore at the end of the film? If there are multiple colors, put them in a comma-separated list in alphabetical order.
  🔍 search: Goldfinger film end James Bond Pussy Galore concealed what object color
[retriever] [기준 시점(오늘): 2026-06-15] 아래는 이 시점 웹 검색 결과다.
[1] Goldfinger (film) - Wikipedia
    Goldfinger's factory henchmen in the film wear yellow sashes, Pussy Galore ... IGN and EW also named Pussy Galore as the second best Bond girl.
    https://en.wikipedia.org/wiki/Goldfinger_(film)
[2] GOLDFINGER | Bond meets Pussy Galore - YouTube
    I must be dreaming…” On this day in 1964 Honor Blackman filmed her scene where the no-nonsense Pussy Galore wakes up Bond on board ...
    https://www.youtube.com/watch?v=Y-_1NS9kKR4
[3] James Bond, Goldfinger, and how Pussy Galore got past the censors
    The James Bond team knew they were pushing their luck with the character of Pussy Galore – but they had a plan to beat American censors.
    https://filmstories.co.uk/features/james-bond-goldfinger-and-how-pussy-galore-got-past-the-censors/
[4] GOLDFINGER | James Bond 007 - Facebook
    Remembering Honor Blackman, who on this day in 1964, started filming as Pussy Galore in GOLDFINGER with this scene on Goldfinger's jet.
    https://www.facebook.com/JamesBond007/videos/goldfinger/810687252790163/
[5] James Bond doesn't do anything in Goldfinger [spoilers] - Reddit
    James Bond doesn't really do anything the entire movie. Besides convincing Pussy Galore not to poison the military base, none of his actions are of any real ...
    https://www.reddit.com/r/JamesBond/comments/q4fu04/james_bond_doesnt_do_anything_in_goldfinger/
[6] Pussy Galore (Honor Blackman) - James Bond Wiki - Fandom
    Pussy Galore is a pilot who appeared in EON Productions' 1964 James Bond film Goldfinger, portrayed by the late British actress Honor Blackman.
    https://jamesbond.fandom.com/wiki/Pussy_Galore_(Honor_Blackman)
[answerer] 제공된 검색 결과에는 영화 말미에 제임스 본드와 퍼시 갤로어가 숨었던 물체의 색상에 대한 정보가 없습니다. 검색 결과에는 악당들의 노란 허리띠(yellow sashes)만 언급되어 있습니다.

FINAL ANSWER: 정보 없음
체감: 혼자 6번 검색해 정답(orange, white)을 찾던 일을, 둘로 나누자 'FINAL ANSWER: 정보 없음'으로 포기. pass@1 0.3→0.0 (rejected).

EXP-155 · 리서치 — 종료배선 (리포트 유실 방지)

promoted
레버: 종료를 reviewer "APPROVE"worker "FINAL ANSWER"로 (유일한 변수)
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)00.67개선
avg_tokens (비용)8,89416,616악화
grounded_pct (실검색 성공)11불변
report_sources (본문 출처수)018개선
recency (최신성)0.460.57개선
개념 & 기술
개념. reviewer 조기 승인(MAST FM-3.3 잘못된 검증)이 worker 완성 전에 끝내, 채점이 읽는 final_answer_fullreviewer의 'APPROVE' 메시지를 캡처 → 리포트 본문·출처 전부 유실. grounded(검색)는 base도 1.0이었다 — 실패는 능력이 아니라 전달에 있었다. "검색했다 ≠ 전달했다"(grounded ⟂ report_sources).
기술. 종료조건만 worker의 FINAL ANSWER로 바꿔 worker가 마지막 발화 → 리포트가 캡처됨. 능력·도구·프롬프트 전부 고정.
실제 출력 (DB 원문 그대로)
▼ BASE — reviewer 승인으로 종료 (실제 전체 출력, 160자)
제공된 검색 결과는 기준 시점(2026년 6월) 대비 주요 내용이 2023~2025년 자료(예: MI300X/H100 비교, 2025년 시장 전망)에 편중되어 있어, 최신 경쟁 구도(예: 블랙웰, MI350 등 최신 세대 제품 및 2026년 시장 동향)를 분석하기에 최신성이 부족합니다.
▼ V2 — worker FINAL ANSWER로 종료 (실제 전체 출력, 6942자)
[기준 시점(오늘): 2026-06-16] 아래는 이 시점 웹 검색 결과다.
[1] 데이터 센터 가속기 시장 점유율 | 2032년 산업 분석
    데이터 센터 Accelerator 시장 규모는 2023년 USD 8.1 억에 달하며 2024년과 2032년 사이에 25% 이상의 CAGR에 등록할 것으로 예상됩니다. 딥러닝과 머신러닝 (ML)을 ...
    https://www.gminsights.com/ko/industry-analysis/data-center-accelerator-market
[2] AI 데이터 센터 매출 경쟁(2021~2025) - 네이버 블로그 - NAVER
    그러나 이후 시장은 크게 변화하여, 2025년 말에는 엔비디아가 무려 86%의 시장 점유율로 선두를 차지할 것으로 예상됩니다. ​. 엔비디아는 GPU뿐만 ...
    https://m.blog.naver.com/PostView.naver?blogId=jeunkim&logNo=224157431810
[3] AI 가속기 칩 시장 규모, 점유율 | 성장 연구 - 2035년
    AI 가속기 칩 시장 규모는 2024년 92억 5,875만 달러에서 2025년 112억 6,052만 달러로 성장하고, 에는 603억 6,715만 달러를 넘어 CAGR 21.6%로 성장할 ...
    https://www.marketgrowthreports.com/ko/market-reports/ai-accelerator-chip-market-101002
[4] 데이터 센터 가속기 시장 보고서 | 산업 분석, 규모 및 예측
    프로세서 유형별로 보면, GPU가 73.20년에 2025%의 매출 점유율을 기록하며 선두를 달렸고, ASIC은 15.42년까지 2032%의 CAGR로 성장할 것으로 예상됩니다 ...
    https://www.mordorintelligence.kr/industry-reports/data-center-accelerator-market
[5] 데이터 센터 및 클라우드용 AI 칩 2025-2035: 기술 동향, 주요 기업 및 ...
    그래픽 처리장치(GPU), 중앙 처리장치(CPU), 맞춤형 AI ASIC, 기타 AI 가속기, 관련 주요 기업에 대한 분석, 기술동향, 공급망 분석 및 시장 전망을 포괄하는 보고서.
    https://www.idtechex.com/ko/research-report/ai-chips-for-data-centers-and-cloud/1095
[6] AI 혁명을 이끌다: GPU 및 AI 가속기 시장 비즈니스 리더의 가이드
    NVIDIA, AMD, 그리고 재등장한 인텔이 지배하는 이 시장은 폭발적인 성장을 경험하고 있으며, 2030년까지 3,500억 달러를 넘을 것으로 예상됩니다. 이러한 ...
    https://kr.linkedin.com/pulse/powering-ai-revolution-business-leaders-guide-gpu-market-t-ph-d-xxxtc?tl=ko
[기준 시점(오늘): 2026-06-16] 아래는 이 시점 웹 검색 결과다.
[1] NVIDIA Blackwell Platform Arrives to Power a New Era of Computing
    The Blackwell GPU architecture features six transformative technologies for accelerated computing, which will help unlock breakthroughs in data ...
    https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing
[2] Comparing Blackwell vs Hopper | B200 & B100 vs H200 & H100
    Compare NVIDIA Tensor Core GPU including B200, B100, H200, H100, and A100, focusing on performance, architecture, and deployment ...
    https://www.exxactcorp.com/blog/hpc/comparing-nvidia-tensor-core-gpus
[3] H100 vs H200 vs B200: NVIDIA GPU Comparison | Introl Blog
    "Nvidia's next-gen AI GPU is 4X faster than Hopper: Blackwell B200 GPU delivers up to 20 petaflops of compute and other massive improvements." ...
    https://introl.com/blog/h100-vs-h200-vs-b200-choosing-the-right-nvidia-gpus-for-your-ai-workload
[4] How much does it cost to run NVIDIA B200 GPUs in 2025? - Modal
    Grace-Blackwell GB200 Superchip: $60,000 - $70,000 (1x Grace CPU + 2x B200); NVIDIA DGX B200: ~$515,000 (8x B200, 1.44 TB GPU RAM, 72 PFLOPS FP8).
    https://modal.com/blog/nvidia-b200-pricing
[5] NVIDIA To Ship 150K-200K Blackwell GB200 AI Servers In Q4 2024 ...
    NVIDIA To Ship 150K-200K Blackwell GB200 AI Servers In Q4 2024 Alone & 500-550K Units In Q1 2025 ... r/NVDA_Stock - China to Approve Nvidia H200 ...
    https://www.reddit.com/r/NVDA_Stock/comments/1g6yj8s/nvidia_to_ship_150k200k_blackwell_gb200_ai/
[6] Nvidia Blackwell Ahead in AI Inference, AMD Second - IEEE Spectrum
    The B200 packs in 36 percent more high-bandwidth memory than the H200, but, even more important, it can perform key machine learning math using ...
    https://spectrum.ieee.org/ai-inference
[기준 시점(오늘): 2026-06-16] 아래는 이 시점 웹 검색 결과다.
[1] AMD Accelerates Pace of Data Center AI Innovation and Leadership ...
    The updated roadmap starts with the new AMD Instinct MI325X accelerator, which will be available in Q4 2024. Following that, the AMD Instinct ...
    https://ir.amd.com/news-events/press-releases/detail/1201/amd-accelerates-pace-of-data-center-ai-innovation-and-leadership-with-expanded-amd-instinct-gpu-roadmap
[2] MI325X vs MI300X: What's New, What Matters, and Why It Changes ...
    Compare AMD MI325X vs MI300X. Discover performance gains, memory upgrades, and how TensorWave's AI cloud unlocks their full potential.
    https://tensorwave.com/blog/mi325x-vs-mi300x-whats-new-what-matters-and-why-it-changes-your-ai-stack
[3] AMD teases Instinct MI325X refresh in Q4, MI350 'CDNA 4' in 2025 ...
    AMD plans to launch the refreshed Instinct MI325X AI accelerator in Q4 2024, featuring the same chiplet housing structure as the current MI300X ...
    https://www.tweaktown.com/news/98649/amd-teases-instinct-mi325x-refresh-in-q4-mi350-cdna-4-2025-mi400-next-2026/index.html
[4] AMD MI325X vs. MI355X 가속기 비교 : r/hardware - Reddit
    MI325X vs. MI355X 칩 레벨 비교 ; 컴퓨팅 성능 (FP8), 20.8 PF, 37 PF ; 컴퓨팅 성능 (FP4/FP6), 지원 안 함, 74 PF (FP6/FP4) ; 메모리 유형, HBM3e, HBM3e.
    https://www.reddit.com/r/hardware/comments/1h4ytqx/amd_mi325x_vs_mi355x_accelerator_comparison/?tl=ko
[5] Can AMD match NVIDIA in 2025 or 2026? - YouTube
    ] AI Market in 2025 – AMD's Position [01:29] MI355X GPU ... ] AMD AI Roadmap: MI400, HBM4, UAL [04:39] ROCm 7: Software Catch-Up ...
    https://www.youtube.com/watch?v=4iZPPypUa9w
[6] AMD의 목적은 '경쟁 아닌 자립'··· 인스팅트 MI325X의 출시 의의는?
    MI325X는 지난해 말 공개한 인스팅트 MI300 시리즈 중 고성능 모델인 MI300X의 후속 제품이다. 하지만 AMD가 내년 초 출시 예정인 엔비디아 블랙웰이 아닌 ...
    https://blog.naver.com/itdonga_me/223751941262
[기준 시점(오늘): 2026-06-16] 아래는 이 시점 웹 검색 결과다.
[1] Intel and IBM Announce the Availability of Intel Gaudi 3 AI ...
    These DAs will be available in 2H 2025. “Bringing together the power of Red Hat AI with Intel Gaudi 3 AI accelerators on IBM Cloud offers ...
    https://newsroom.ibm.com/blog-intel-and-ibm-announce-the-availability-of-intel-gaudi-3-ai-accelerators-on-ibm-cloud
[2] 인텔® Gaudi® 3 AI 가속기 백서
    이 기술 논문은 인텔의 차세대 AI 가속기인 인텔® Gaudi® 3 AI 가속기를 소개합니다. 이 논문은 새로운 가속기에 관한 기술적·성능 정보를 제공하며, 개요, 하드웨어 ...
    https://www.intel.co.kr/content/www/kr/ko/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html
[3] 인텔, Gaudi 3 AI 가속기 출시: AI 시장에서 더 크게, 더 높이 목표 - Reddit
    3.7 TB/s 메모리 대역폭은 올해 경쟁하기엔 너무 낮아. 백서에서 인텔은 Gaudi 3의 H100 대비 성능 우위가 주로 더 큰 메모리 용량과 H100보다 더 큰 ...
    https://www.reddit.com/r/hardware/comments/1bzwssf/intel_introduces_gaudi_3_ai_accelerator_going/?tl=ko
[4] Intel Unleashes Enterprise AI with Gaudi 3, AI Open Systems ...
    At Vision 2024, Intel goes all-in on open and more secure enterprise AI with new customers, partners and collaborations across the AI ...
    https://www.intc.com/news-events/press-releases/detail/1689/intel-unleashes-enterprise-ai-with-gaudi-3-ai-open-systems
[5] [News] Intel May Cut Gaudi 3's 2025 Shipment Target by ... - SemiWiki
    A report by the Economic Daily News indicates that the struggling giant has significantly slashed the chip's shipment targets by over 30% for next year.
    https://semiwiki.com/forum/threads/news-intel-may-cut-gaudi-3%E2%80%99s-2025-shipment-target-by-30-raising-concerns-for-tsmc-ase-and-alchip.21166/
[6] [GN⁺] Intel, Gaudi 3 AI 가속기 출시 - 읽을거리&정보공유
    2024년 4월 9일 애리조나 주 피닉스에서 열린 Intel Vision 행사에서 Intel Gaudi 3 AI 가속기가 발표됨 · BF16에 대해 4배의 AI 연산 성능과 이전 세대 ...
    https://discuss.pytorch.kr/t/gn-intel-gaudi-3-ai/4033
체감: 빈 불평(160자) → 출처 18개 달린 분석(6942자). 리포트가 무에서 생겼다.

EXP-159 · OCR — 출력폭주 방지 (길이/형식 규율)

promoted
레버: worker에 "교정 텍스트만, 입력과 비슷한 길이, 부연 금지" 규율 한 단락
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)0.830.92개선
mean CER10.244.64개선
median CER0.0060개선
개념 & 기술
개념. 진짜 실패는 교정 난이도가 아니라 출력 폭주 — 모델이 교정 결과 대신 자기 사고과정을 답으로 토한다(8192자). 처음엔 '꼼꼼함' 레버를 댔으나 no-op(존재하지 않는 문제를 겨냥). 또 mean CER는 폭주 1건에 파괴(10.2)되니 median·폭주율로 봐야 진실(median 0.006)이 보인다 — 메트릭 설계 교훈.
기술. worker 프롬프트의 길이·형식 규율로 폭주율 17→8%. 잘못된 진단(꼼꼼함)은 아무것도 안 움직이고, 정확한 진단(길이규율)만 결과를 바꿨다 — 올바른 진단 → 올바른 레버.
실제 출력 (DB 원문 그대로)
▼ BASE — 출력폭주 (실제 전체 출력, 8192자)
1.  **Analyze the Request:**
    *   **Role:** OCR Post-correction engine.
    *   **Task:** Correct the provided text (typos, broken characters, missing spaces) without changing meaning, adding, or deleting content.
    *   **Input Text:** "Pat1ent , a 4 5-year-0ld ma le , rep0rts m1ld che st pa1n and shortn ess of brea th s1nce th1s m0rn1ng . Bl00d pre ssure 1s 14O / 9O ."
    *   **Output Format:** Just the corrected text following "FINAL ANSWER:". No explanations.

2.  **Analyze the Input Text & Identify Errors:**
    *   "Pat1ent" -> '1' should be 'i'. -> "Patient"
    *   ", a 4 5-year-0ld" -> '4 5' has an extra space, should be '45'. '0' should be 'o'. -> ", a 45-year-old"
    *   "ma le" -> 'ma le' has an extra space, should be 'male'. -> "male"
    *   ", rep0rts" -> '0' should be 'o'. -> ", reports"
    *   "m1ld" -> '1' should be 'i'. -> "mild"
    *   "che st" -> 'che st' has an extra space, should be 'chest'. -> "chest"
    *   "pa1n" -> '1' should be 'i'. -> "pain"
    *   "and shortn ess" -> 'shortn ess' has an extra space, should be 'shortness'. -> "and shortness"
    *   "of brea th" -> 'brea th' has an extra space, should be 'breath'. -> "of breath"
    *   "s1nce" -> '1' should be 'i'. -> "since"
    *   "th1s" -> '1' should be 'i'. -> "this"
    *   "m0rn1ng" -> '0' should be 'o', '1' should be 'i'. -> "morning"
    *   ". Bl00d" -> '00' should be 'oo'. -> ". Blood"
    *   "pre ssure" -> 'pre ssure' has an extra space, should be 'pressure'. -> "pressure"
    *   "1s" -> '1' should be 'i'. -> "is"
    *   "14O" -> 'O' should be '0'. -> "140"
    *   "/ 9O" -> '9O' -> '0' should be '0'. -> "/ 90" (Keep the space as is? usually BP is written with spaces around the slash in OCR outputs sometimes, but standard medical text often is "140/90". However, the prompt says "don't add/delete content". If I see "/ 9O", I should fix the typo. The space inside "/ 9O" is ambiguous. "14O / 9O" has spaces around the slash. "140/90" is standard. "140 / 90" is acceptable. Let's look closely at the source spacing. "14O / 9O". Correcting 'O' to '0'. Should I remove the space around the slash? Usually, OCR post-correction preserves original spacing unless it's a clear separation of words (like "ma le" -> "male"). Here, "/ 9O" looks like a split number. But "Blood pressure is 140/90" is standard. "Blood pressure is 140 / 90" is readable. Let's stick to fixing character errors and word splits. "14O" -> "140", "9O" -> "90". The space around the slash is in the input " / ". I will leave it or normalize it slightly if it looks like an error. Given "ma le" and "che st", the spacing is the main issue besides number-letter substitution. I will remove the space around the slash to make it a coherent number sequence, as that is likely the intent of "Blood pressure". Or, strictly following "don't add/delete content", maybe I should keep the spaces? No, "14O" is one token split by typo. " / " is punctuation. Let's look at the prompt constraints: "의미를 바꾸지 말고 인식 오류만 원문으로 교정한다(내용 추가·삭제 금지)". Wait, "content addition/deletion prohibited" usually means don't change the *textual content* (don't add new words, don't delete necessary words). Fixing spacing errors (like "ma le") is part of OCR correction. However, removing the space around the slash changes the formatting.
    *   Let's re-evaluate "14O / 9O". The 'O' is a typo for '0'. The spaces are likely OCR artifacts or formatting. Standard is "140/90". If I write "140 / 90", it's not wrong, just spaced.
    *   Let's look at the other spacing fixes:
        *   "ma le" -> "male" (joined)
        *   "che st" -> "chest" (joined)
        *   "shortn ess" -> "shortness" (joined)
        *   "brea th" -> "breath" (joined)
        *   "pre ssure" -> "pressure" (joined)
    *   These are clear word splits caused by OCR. The slash spacing is less clear cut. However, looking at "4 5-year", I should join "4 5" to "45".
    *   Back to "14O / 9O". I will correct "O" to "0". I will join "14O" to "140". I will join "9O" to "90". The slash spacing... I'll keep it as "140 / 90" or "140/90"? Usually, BP is written as "140/90". If I strictly follow "no content addition/deletion", maybe keep spaces? But "ma le" -> "male" implies merging is allowed. I will assume merging the numbers is correct. I'll output "140 / 90" to be safe, or "140/90". Let's look at the prompt again: "붙은 띄어쓰기" (attached spacing) is mentioned as an error type. Actually, the prompt says "오탈자·깨진 문자·붙은 띄어쓰기가 섞여 있다" (Typos, broken chars, and *attached* spacing are mixed). Wait, "붙은 띄어쓰기" usually means missing spaces (words stuck together). But the input has the opposite: "ma le" (extra spaces). The prompt description might be generic. The key is "recognition errors".
    *   Input: "ma le" -> Correction: "male".
    *   Input: "che st" -> Correction: "chest".
    *   Input: "4 5" -> Correction: "45".
    *   Input: "14O / 9O" -> Correction: "140 / 90" or "140/90". Given the pattern of fixing splits, "140/90" is the most correct restoration of the original text. However, if the original had spaces, keeping them is safer. But usually, numbers with a slash are a single unit. I will format it as "140/90" to fix the fragmentation, similar to "45". If I write "140 / 90", it's readable.
    *   Let's check the 'O's. "14O" (letter O) -> 140. "9O" (letter O) -> 90.
    *   Let's check the rest.
    *   "Pat1ent" -> Patient
    *   "4 5-year-0ld" -> 45-year-old
    *   "ma le" -> male
    *   "rep0rts" -> reports
    *   "m1ld" -> mild
    *   "che st" -> chest
    *   "pa1n" -> pain
    *   "shortn ess" -> shortness
    *   "brea th" -> breath
    *   "s1nce" -> since
    *   "th1s" -> this
    *   "m0rn1ng" -> morning
    *   "Bl00d" -> Blood
    *   "pre ssure" -> pressure
    *   "1s" -> is
    *   "14O / 9O" -> 140 / 90 (I'll keep the slash spacing to minimize interpretation risk, or fix it? "4 5" was fixed to "45" because it's a number. "14O / 9O" is a number range/reading. "140/90" is standard. I will produce "140/90" as it's the most likely intended text, same as "male" from "ma le").

    *   Refined plan:
        *   Pat1ent -> Patient
        *   , a 4 5-year-0ld -> , a 45-year-old
        *   ma le -> male
        *   , rep0rts -> , reports
        *   m1ld -> mild
        *   che st -> chest
        *   pa1n -> pain
        *   and shortn ess -> and shortness
        *   of brea th -> of breath
        *   s1nce -> since
        *   th1s -> this
        *   m0rn1ng -> morning
        *   . Bl00d -> . Blood
        *   pre ssure -> pressure
        *   1s -> is
        *   14O / 9O -> 140 / 90 (Let's stick to the spacing pattern in the input for punctuation unless it's clearly wrong. The input has " / ". I will keep " / " to be safe against "deleting content" constraint, but fix the O->0).

    *   Wait, looking at "ma le" -> "male". This removes a space. "che st" -> "chest". Removes a space. "shortn ess" -> "shortness". Removes a space.
    *   "14O / 9O" -> The O is clearly a 0. The spaces around the slash might be intentional padding or OCR noise. I will normalize to "140/90" because "ma le" implies fixing splits is expected. But if I look at the input "14O / 9O", there is a space before and after the slash. If I write "140/90", I delete spaces. If I write "140 / 90", I keep spaces. The prompt says "no content addition/deletion". This usually refers to *informational* content (words). Fixing formatting/spacing is allowed (explicitly mentioned in the prompt as part of the errors to fix: "붙은 띄어쓰기" - though the input has the opposite, separated spacing). The prompt says "recognition errors... corrected to original". Original text likely didn't have spaces in "ma le". Original text likely didn't have spaces in "14O / 9O" (which was probably 140/90).
    *   However, to be absolutely safe and avoid over-interpreting, I will correct the characters and merge obvious word splits ("ma le", "che st", "pre ssure"). For numbers, "4 5" -> "45". For "14O / 9O", I will output "140/90" because it's a standard measurement unit.

    Let's double check the text:
    Patient, a 45-year-old male, reports mild chest pain and shortness of breath since this morning. Blood
▼ V3 — 길이/형식 규율 (실제 전체 출력)
FINAL ANSWER:
Patient, a 45-year-old male, reports mild chest pain and shortness of breath since this morning. Blood pressure is 140 / 90.
체감: 추론 덤프(7738자 쓰레기) → 정답과 정확히 일치하는 124자 교정본.

EXP-144 · format — FINAL ANSWER 형식 규율

promoted
레버: answerer 프롬프트에 "마지막 줄 `FINAL ANSWER: <답>`, 숫자는 콤마·단위 없이" 1단락
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)00.1개선
format_invalid (형식오류율)10개선
avg_tokens (비용)8,1568,915악화
개념 & 기술
개념. GAIA 채점기는 success = format_valid AND normalize(pred)==normalize(gold). 형식이 깨지면 정답이어도 success=False. 형식은 능력과 독립된 '채점 가능성' 게이트 — base는 답을 내고도 형식이 없어 버려졌다.
기술. 프롬프트 1단락으로 출력 규율 부여. 능력·도구 변화 0. format_invalid 1.0→0.0.
실제 출력 (DB 원문 그대로)
▼ BASE — 형식 없음 (실제 출력)
White
▼ FIX — FINAL ANSWER 형식
orange, white
체감: 답을 알고도 형식이 없어 버려지던 게(format_invalid 100%), 한 줄 규율로 채점 가능해졌다.

R2 코딩 · 코딩 — 격리 샌드박스 채점

promoted
레버: 격리 샌드박스(subprocess + RLIMIT + timeout + 정적가드)에서 patch+test 실행
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)1측정
개념 & 기술
개념. 결정적 채점(테스트 통과율) — 가장 깨끗한 객관 인과. 측정 중 채점 harness 버그 발견: 코드추출 정규식이 worker 추론 속 인라인 '```python' 언급을 오매치 → 0%로 보임. 수정 후 pass@1 1.0. "메트릭이 거짓일 수 있다"의 또 다른 사례.
기술. roles/sandbox.py: 위험호출 정적 차단 + 별도 프로세스 CPU/메모리 RLIMIT + wall timeout + temp HOME. 코드추출은 실제 펜스(```python 다음 줄바꿈)만, 마지막 블록.
실제 출력 (DB 원문 그대로)
▼ 입력 — 버그 있는 함수 + 테스트
다음 함수에 버그가 있다. 테스트를 통과하도록 고쳐라(함수 이름 `solve` 유지).

```python
def solve(s):
    # 문자열을 뒤집어 반환
    return s
```

통과해야 할 테스트:
```python
def test_solve():
    assert solve('abc') == 'cba'
    assert solve('') == ''
    assert solve('xy') == 'yx'
```
▼ worker 패치 (샌드박스 테스트 통과 ✓)
FINAL ANSWER:
```python
def solve(s):
    # 문자열을 뒤집어 반환
    return s[::-1]
```
체감: 5/5 버그 수정이 실제 샌드박스 pytest로 검증됨.

R3 테스트작성 · 테스트작성 — mutation kill 채점

측정
레버: 생성된 테스트를 정상함수(valid) + 변이(mutant)에 실행 → 변이 잡으면 kill
움직인 메트릭 (전체)
메트릭beforeafter
pass@1 (정답률)0.4측정
개념 & 기술
개념. mutation testing: 좋은 테스트는 정상함수엔 통과하고 의도적 버그(변이)엔 실패해야 한다. base는 worker→reviewer 구조에서 reviewer 'APPROVE'가 테스트를 가리거나 들여쓰기 깨진 코드로 실행 자체가 실패 → 종료를 worker로 바꿔 캡처 개선(R1과 동형 버그).
기술. 샌드박스에서 (func+test) 실행해 valid 확인 후, 각 mutant에 (mutant+test) 실행해 실패=kill. kill율 ≥0.5 = success.
실제 출력 (DB 원문 그대로)
▼ 검증 대상 함수 + 변이 3개
def add(a, b):
    return a + b
# mutants:
def add(a, b):
    return a - b
---
def add(a, b):
    return a * b
---
def add(a, b):
    return a
▼ worker 생성 테스트
import pytest

# 정상 케이스 (Normal Cases)
def test_add_positive_integers():
    assert add(1, 2) == 3

def test_add_negative_integers():
    assert add(-1, -2) == -3

def test_add_mixed_integers():
    assert add(-5, 10) == 5

def test_add_floats():
    assert add(1.5, 2.5) == 4.0

# 경계 케이스 (Boundary Cases)
def test_add_with_zero():
    assert add(0, 5) == 5
    assert add(5, 0) == 5
    assert add(0, 0) == 0

def test_add_float_precision():
    # 부동소수점 오차 범위 고려
    assert add(0.1, 0.2) == pytest.approx(0.3)

# 예외 케이스 (Exception Cases)
def test_add_type_error():
    # 정수와 문자열은 더할 수 없음
    with pytest.raises(TypeError):
        add(1, "a")

def test_add_type_error_with_none():
    with pytest.raises(TypeError):
        add(None, 1)
체감: mutation kill 평균 40%, pass@1 40%. 테스트작성은 base 변동 큼.
L 관통 교훈
① 한 레버가 여러 메트릭을 동시에 움직인다. loop 종료조건 하나가 loop_rate·turns·tokens·p95_latency·pass@1을 한꺼번에. 단일 점수만 보면 효과의 절반을 놓친다.
② 고친 건 능력이 아니라 '전달'. 모든 promoted는 능력이 이미 있는데 결과물을 못 내보내던 문제(형식·수렴·검색·완결·출력규율).
③ "한 일 ≠ 전달한 일". grounded=1.0인데 report_sources=0 — 실패는 늘 전달 단계에 살았다.
④ 이진/평균 메트릭은 진실을 가린다. grounded(이진)는 못 잡고 search_ok(깊이)가 잡았다. mean CER는 폭주 1건에 파괴, median·폭주율이 진실.
⑤ 메트릭이 거짓일 수 있다. grounded(호출→성공 재정의), 코드추출(인라인 펜스 오매치) — 측정 자체의 버그 2건을 발견·수정한 것이 시스템 수정만큼 중요했다.