PE Civil Bench: Advancing Agentic Retrieval-Augmented Generation Performance of Large Language Models on Exam Problems and Automated Component Design

Saurav Silwal1, Joshua Gao2, Subin Varghese2, Vedhus Hoskere1,2*
1Dept. of Civil & Environmental Engineering, University of Houston 2Dept. of Electrical & Computer Engineering, University of Houston *Corresponding author: vhoskere@Central.UH.EDU

Published in Computer-Aided Civil and Infrastructure Engineering

Paper Code Data

Abstract

Large language models are increasingly being piloted in civil engineering firms to automate workflows and help engineers navigate dense design codes and specifications. However, safety-critical use remains limited by reliability concerns, inconsistent reasoning, and the need for auditable, code-grounded outputs suitable for quality control. While retrieval augmentation and agentic workflows have improved factuality in other domains, their effectiveness in civil engineering has not been systematically established, and it remains unclear how to scale them into transparent, code-compliant design pipelines with reliable evaluation.

To address these limitations, we develop the first benchmark, PE Civil Bench, consisting of Fundamentals of Engineering and Professional Engineering Civil examination questions and produce baseline scores of thirteen frontier and open-weight large language models, including the state-of-the-art GPT-5.5, under baseline prompting, conventional retrieval-augmented generation (RAG), and agentic RAG. We then develop a retrieval-augmented multi-agent framework for fully automated concrete element design, demonstrated on reinforced concrete beams and columns, that infers design steps from worked examples and performs code retrieval, calculation, validation, and iterative self-correction with transparent traces. Across 33 beam configurations it produced code-compliant designs in all cases and closely matched ETABS finite-element outputs (r ≥ 0.90). Requiring only a new reference example and no reprogramming, the same architecture extended to column design. Finally, we introduce an autonomous RAG-based evaluator that extracts assessment criteria from reference solutions and agrees strongly with a traditional rule-based scorer (r = 0.976).

Three Contributions

✓
01

PE Civil Bench

The first open benchmark of 150 FE- and PE-style civil engineering problems spanning 8 subdisciplines, 3 difficulty tiers, and multiple question types, enabling systematic evaluation of base, RAG-augmented, and agentic LLMs.

  • 110 MCQ (FE-style) + 40 open-ended numerical (PE-style)
  • Structural, geotechnical, water resources, transportation & more
  • Easy / Medium / Hard difficulty taxonomy
02

Agentic Design Pipeline

A self-correcting multi-agent framework for fully automated, code-compliant structural design. Design logic is inferred dynamically from reference examples: no hard-coded rules, no reprogramming when codes change.

  • 100% code-compliant across 33 RC beam configurations
  • Extends to column design with only a new reference example
  • Under $2 per complete design run using GPT-4o
03

RAG-Based Evaluator

An autonomous evaluation framework that extracts assessment criteria from retrieved reference solutions and verifies design accuracy without manual rubric construction, scaling quality control alongside the design pipeline.

  • r = 0.976 agreement with traditional hardcoded scorer
  • R² = 0.953 across 66 RC beam designs
  • Learns criteria from examples, not hand-coded rules
150FE / PE Civil Questions
13LLMs Evaluated
33RC Beam Designs
r ≥ 0.90Pearson correlation (r), agreement with ETABS
<$2Cost per Design Run

Three-Phase Methodology

From strategy benchmarking through automated design to quality verification: a unified pipeline for AI-assisted civil engineering.

Phase 1

Systematic Benchmarking

Three augmentation strategies of increasing sophistication (Base LLM, Vector RAG, and Agentic RAG) are evaluated on PE Civil Bench using LangChain with consistent hyperparameters across all 13 models.

Base LLM: intrinsic reasoning, no retrieval
Vector RAG: single-pass semantic retrieval from the FE Reference Handbook
Agentic RAG: iterative retrieval loop with formula injection and self-reflection (ReAct-inspired, LangGraph)
→
Phase 2

Agentic Design Generation

The best-performing strategy drives a self-correcting multi-agent framework for code-compliant structural design. Design logic is inferred dynamically: no hard-coded procedures.

Planner: extracts design steps from reference examples
Executor: agentic RAG retrieval + Python REPL for calculations
Validator: hierarchical error detection (local fix or full restart)
Report Generator: traceable, auditable output with ACI 318 references
→
Phase 3

Autonomous Evaluation

A RAG-based evaluation framework learns assessment criteria from reference solutions and scores designs without manual rubric construction.

Retriever: finds relevant reference designs by semantic search
Rubric Generator: extracts methodology and converts to scoring criteria
Evaluator: LLM-based agent assigns weighted scores with justifications

PE Civil Bench Composition

150 problems curated by professional engineers at the Structures and AI Lab, University of Houston, covering all major FE/PE Civil subdisciplines.

Benchmark Breakdown (N = 150)
DimensionCategoryn (%)
SubdisciplineStructural50 (33%)
Water Resources27 (18%)
Mathematics22 (15%)
Geotechnical21 (14%)
Transportation13 (9%)
Construction12 (8%)
Ethics3 (2%)
Surveying2 (1%)
DifficultyEasy47 (31%)
Medium63 (42%)
Hard40 (27%)
Question TypeNumerical114 (76%)
Conceptual26 (17%)
Code Interpretation10 (7%)
Reasoning DemandMulti-step111 (74%)
Single-step39 (26%)
Response FormatMultiple Choice (MCQ)110 (73%)
Open-Ended Numerical40 (27%)

Sample Questions

EASY  ·  FE-Style MCQ  ·  Structural

The elastic modulus of concrete with f'c = 3 ksi and a unit weight of 145 pcf is almost nearly:

A. 3,155 ksi    B. 3,834 ksi    C. 3,484 ksi    D. 3,562 ksi

MEDIUM  ·  FE-Style MCQ  ·  Geotechnical

A sand sample was tested in a triaxial device. At the beginning of the test, a confining pressure of 50 psi was applied, and deviatoric pressure was then applied. The sample failed when the applied external pressure reached 320 psi. The angle of internal friction of the sand is most nearly:

A. 32°    B. 37°    C. 43°    D. 47°

HARD  ·  PE-Style Open-Ended  ·  Transportation

Consider a four-legged intersection with a single lane for each approach. The intersection is currently signalized with simple two-phase actuated control and a 50-second phase duration for the northbound/southbound approaches, with 4-second change and 1.5-second clearance intervals. The southbound approach has a saturation flow rate of 1,615 vehicles per hour and a queue of 14 vehicles. Assuming a green extension time of 4.5 seconds, the arrival flow rate during the effective green time for the southbound approach is most nearly:

Note: Numerical answer required, no answer choices provided.

Benchmark Results

Overall accuracy across 13 LLMs under three augmentation strategies on PE Civil Bench (N = 150). Agentic RAG improved every evaluable model with no degradations.

Model Base LLM Vector RAG Agentic RAG
MCQOEOverall MCQOEOverall MCQOEOverall
Reasoning-Optimized Models
GPT-5.5 93.6%52.5%82.7% 94.5%65.0%86.7% 94.5%67.5%87.3%
GPT-5.4 91.0%42.5%78.0% 95.5%45.0%82.0% 97.3%50.0%84.7%
o4-mini 87.0%22.5%70.0% 87.0%22.5%70.0% 87.0%32.5%72.0%
o3-mini 84.0%15.0%65.3% 84.0%30.0%69.3% 86.3%30.0%71.3%
Gemini 2.5 Pro 63.0%12.5%49.3% 73.0%32.5%57.3% 93.6%35.0%78.0% +28.7pp
DeepSeek R1 7B 40.0%0.0%29.3% 39.0%0.0%28.7% † No tool-calling support
General-Purpose Flagship Models
GPT-4o ★ selected 66.3%2.5%49.3% 71.7%25.0%59.3% 86.4%27.5%70.7% +21.4pp
GPT-4o-mini 54.5%2.5%40.7% 58.1%10.0%45.3% 74.0%12.5%52.7%
Open-Source Models
Llama 3.3 70B 61.8%2.5%46.0% 60.0%12.5%43.3% 61.0%17.5%49.3%
Llama 3.1 405B 57.3%2.5%42.7% 62.0%10.0%48.0% 74.0%25.0%60.7%
Llama 3.1 8B 37.2%2.5%28.0% 34.5%7.5%27.3% 36.0%10.0%29.3%
Mistral 7B 40.0%0.0%29.3% 32.0%0.0%23.3% −6.0pp 43.0%2.5%32.0%
Gemma2 9B 29.0%0.0%21.3% 32.0%5.0%24.7% † No tool-calling support

† DeepSeek R1 7B and Gemma2 9B do not support tool calling and could not be evaluated under Agentic RAG. MCQ = Multiple Choice (n=110); OE = Open-Ended numerical (n=40). ■ Best per model   ■ Degraded vs. baseline

📈
Agentic RAG: zero degradations

Unlike Vector RAG (which hurt Mistral by 6pp and Llama 3.3 70B by 2.7pp), Agentic RAG improved every evaluable model.

🚀
Biggest jump: Gemini 2.5 Pro

+28.7 percentage points with Agentic RAG (49.3% → 78.0%), the largest gain among all models evaluated.

⚙️
Iterative retrieval is the key

Ablation shows removing multi-turn retrieval causes the largest drops: −8.7pp for Gemini, −16.0pp for GPT-4o.

🎯
Residual bottleneck: grounding

After augmentation, 68.6% of remaining failures are grounding or conceptual errors: selecting the right principle, not computing it.

Automated RC Beam Design Results

A self-correcting multi-agent framework produced code-compliant reinforced concrete beam designs across all 33 configurations, closely matching ETABS finite-element outputs with no hard-coded design rules.

Pipeline Performance Summary

33/33Code-compliant designs
r = 0.99Moment & shear vs ETABS
r = 0.976RAG vs hardcoded evaluator
<$2Per beam design (GPT-4o)
0 changesTo extend to column design

The pipeline executed 60 LLM calls consuming ~887K tokens for a single representative beam design, producing ~15,200 words of step-by-step documentation with Python execution logs and ACI 318 references at every decision point.

Evaluator Correlation: RAG-Based vs. Hardcoded Scoring

The autonomous RAG-based evaluator was validated against a traditional hardcoded scoring script across 66 RC beam designs (33 full-pipeline + 33 ablated without RAG). The near-perfect agreement confirms that retrieval-augmented evaluation can recover the expert judgment embedded in hand-crafted rubrics, without any manual rule construction. The Pearson correlation coefficient is r = 0.976, with R² = 0.953 indicating that 95.3% of variance in RAG-based scores is explained by the hardcoded scores. The RMSE of 4.69 represents under 8% of the 36–98 point scale, confirming practical accuracy.

Scatter plot showing correlation between RAG-based and hardcoded evaluation scores for 66 RC beam designs. Green points (Regular Beam Design) cluster in the upper right; red points (Ablated Pipeline Design) cluster in the lower left. Pearson r = 0.976, R-squared = 0.953, RMSE = 4.69.
r = 0.976
Pearson correlation between RAG evaluator and hardcoded scorer
R² = 0.953
Variance in RAG scores explained by hardcoded scores
RMSE = 4.69
Average score deviation: under 8% of the 36–98 point range
66
RC beam designs evaluated (33 full pipeline + 33 ablated)

Figure 12: Correlation between RAG-based and hardcoded evaluation scores for RC beam designs. Green markers = full agentic pipeline (Regular Beam Design); red markers = ablated pipeline (no RAG retrieval). The two populations separate cleanly across both evaluation methods, confirming the RAG evaluator captures design quality with the same fidelity as traditional rubrics.

BibTeX

@article{SILWAL2026100126,
  title    = {PE Civil Bench: Advancing Agentic Retrieval-Augmented Generation
              Performance of Large Language Models on Exam Problems and
              Automated Component Design},
  journal  = {Computer-Aided Civil and Infrastructure Engineering},
  pages    = {100126},
  year     = {2026},
  issn     = {1093-9687},
  doi      = {https://doi.org/10.1016/j.cacaie.2026.100126},
  url      = {https://www.sciencedirect.com/science/article/pii/S1093968726031129},
  author   = {Saurav Silwal and Joshua Gao and Subin Varghese and Vedhus Hoskere},
  keywords = {Large Language Models (LLMs), Retrieval-Augmented Generation (RAG)}
}