Test an Agent with Eval Studio

Eval Studio measures how well a specific agent answers a fixed set of questions, using an AI system called an automated judge. The judge compares the agent's response to your expected answer and scores it. A case passes when the judge score is 0.5 or higher.

• Availability: Eval Studio is available in Procore Explore for Procore AI Digital Coworker accounts in the Closed Beta.

• Permissions: To use Eval Studio, you must have the Owner or Admin role on the project where you’re using it.

• Prerequisite: You (or someone on your team) have already created the agent you want to test, with its tools and instructions configured and stable for your scenarios.

  • Once you start building an eval set for an agent, freeze that agent’s tools and instructions until you finish testing against the set. If you change them partway through, you introduce a second variable. A later score cannot tell you which part of the agent got better or worse, and a change that helps one scenario can silently hurt another you already ran. 

• Credits: Running an eval, whether manually or on a schedule, uses credits.

Phase 1: Identify a Project

Choose a project that's indexed and populated with data that matches what your eval questions will ask about. Some guidelines for the project you choose: 

• Review the best practices in Phase 2, so you know how projects and the eval questions should be designed to optimize results. For example, you should either choose a project with good placeholder records or create them.

• Finish the agent’s tools and instructions before you start running the set. Changing the agent between runs makes score changes uninterpretable.  

Phase 2: Create an eval set

Each eval set tests one agent.

1. Open any Agent’s configuration panel in Procore AI. It does not need to be an agent you plan to test. How to open the panel: When chatting with just one agent, click the … more menu next to the agent’s name above one of its responses to you. 

2. Click Eval Studio. This opens the list of evaluations for the whole project.

3. Click + Create Eval Set. 

4. Fill out the ‘Eval Details’ section. This sets up your eval’s basics (eval name, description, etc.), connects it to an Agent of your choice, and gives you an opportunity to add labels.
   Tips:

   • Choosing an agent: If you do not select a different agent from the dropdown, the test will target whichever happens to be first in the list. 

   • Choosing a Name: 

     • Frontload the distinguishing elements in the name, like: RFI Agent - retrieval (read-only)

     • The list already shows last run time, so dates in the name aren’t usually necessary.

5. Review the ‘Best Practices’ below.

6. Choose an authoring path (A or B) below and follow those steps for adding your test cases. 

Best practices

These apply whether you type test cases in Eval Studio or generate them with another LLM and import the file. 

1. Repeatability: Design your eval questions so the project's data looks the same after a run as it did before the run started. That way every run begins from an identical baseline, so if a score changes, you know it's more likely to be the agent — not because the underlying data drifted.

2. Testing create actions: If you need to test the agent’s ability to create items, such as an RFI or a daily log, use an isolated eval project, design the test question so that the agent gives the created item a unique eval-only name, and then delete it yourself after the run. Do not ask the agent to delete it in the same test case unless delete is the skill you are scoring. 

   • Isolated eval project: A dedicated test project, not a live job people are working in. This is best for create, update, or delete tests. Make sure the project is indexed in Procore AI, and put only the placeholder data you need there. However, read-only tests can use a real project. 

3. Testing update and delete actions: In your project, consider using placeholder records you created for testing, with known titles/IDs. Reset those placeholder records after the run so the next run sees the same starting data. 

4. Single-turn: The Eval Studio UI is built for single-turn cases today. A single-turn case is a single question followed by a single answer. Each test case is a new single-turn conversation. The agent does not see earlier cases, and you cannot add a follow-up turn to check whether it remembered something from the message before. If a scenario needs prior facts (a record title, a project, a constraint), put those facts in the question itself. 

5. Completion: Every case needs both a Question and an Expected Answer. When you click Save or Export, any question without an expected answer will be removed. 

6. Maximum: Each eval set can hold up to 100 test cases. Rows past 100 are not kept and will not Save, Export, or run. If you need more coverage, split the work into another eval set.

7. Write conclusions, not scripts: Expected answers should state material facts and required conclusions, not exact wording or a full script of how the agent should talk, unless exact wording is the actual point of the test. Only include facts you actually require: If you list optional follow-ups or nice-to-have phrasing (a polite wrap-up, a suggested next step), the judge doesn't require them, so the score won’t be affected. 

   • Equivalent identifiers are fine. "Alex," "owner: Alex," and an ID paired with the same name should all pass, so state the fact you need, not the exact label or format it has to appear under.
     Example: Must include projects Main Street Office and Riverside Clinic by name.

8. Group by type: It’s useful to test only one capability (such as read-only actions) per eval set. A test run produces one aggregate score for the whole eval set. If retrieval actions improve 10%, while create actions regress by that same amount, in a mixed eval set the number barely moves and you learn nothing. And since you can't filter by label within a set, separate sets are the only way to get per-capability signals. 

9. Group by performance: There are benefits to keeping a set aimed at areas you want the agent to improve on (expecting 60–80% and room to climb) and a separate set of cases that usually pass, to catch backsliding.

   • As a case improves, graduate it into the regression set. 

10. Don't over-split: 20–50 cases per set is the commonly recommended starting range (saves credits and effort). If your set has below roughly 10, a single flaky case swings the score a lot. 

11. Balance positive and negative cases: If every case rewards the agent for taking an action, you train yourself toward an agent that always acts. Include cases where the right answer is "that information isn't available" or "I can't create that item type." 

12. Duplicate sets: The Duplicate action in the … more menu copies the name with "(copy)" appended, plus the agent, description, labels, schedule, and all test cases. Rename the copy, point it at your second agent, and you get identical questions with two independent score histories you can compare side by side. 

    • This is better than swapping the agent on one set and re-running. It works but the set’s run history then blends two different configurations and you lose the ability to tell which run belonged to which setup. 

Note: Scoring works as an automated judge on the agent's response in each case. A case passes when the judge score is 0.5 or higher. The judge grades substance, not presentation — formatting and extra correct metadata usually don't affect the score, but wrong entities, contradictions, and omitted material facts do.

Authoring Path A: Author directly in Eval Studio

1. Enter each sample query (question).

2. Enter an expected answer, or use the sparkle control to generate a first draft from the selected agent.
   Note: If you haven’t clicked ‘Save Changes’ after establishing the eval set’s name and info, the sparkle control is unavailable until you do so. 

3. Repeat until you've entered all your cases.

4. Click Save Changes.

Tip: Review AI-generated drafts carefully. Since they come from the same agent under test, they can bake in that agent's current bugs or quirks. Treat them as a starting point, not as ground truth.

Authoring Path B: Generate with a third-party LLM, then import

If you import an existing set into Eval Studio, it overwrites any current questions and expected answers and settings for that current eval set, and this cannot be undone. 

1. Export an existing Eval Studio set (or create a small sample set and export it) to use as your format example.

2. Give an LLM of your choice that export, along with the source material that should drive your new questions (documentation, placeholder records, known issues).

3. Have the LLM output a new set in the same export format (formatVersion: 1, with conversations and turns).

4. Import the file into Eval Studio, review it, and click Save.

Phase 3: Run a Test

1. In ‘Run Schedule’, click Run Eval Now.
   Note: If this button is disabled, you may have unsaved edits. Click Save Changes.  

2. Watch the ‘Run History’ until status is ‘done’ or ‘failed’. Then open the View Report in the Run History.

3. Inspect any failures, and tighten or loosen expected answers as needed. Review the Best Practices in Phase 2 and Phase 5, as needed. 

4. Reset or delete leftover records (as needed) before the next run. See the ‘Best Practices’ section in Phase 2 of this tutorial for more information. 

5. Re-run until you're satisfied with the results.

What score should you aim for?

Different scores are expected for what you consider a capability set versus a regression set. 

• A capability set measures skills the agent is still learning — looking up the right record, creating an item, refusing an unsupported action. You expect some failures. A mid-range score is useful because it can go up (or down) when you change tools, instructions, or models.

• A regression set is a safety net: cases the agent already handles well. You run it to make sure a later change didn’t break something that used to work. A high score is the goal; a drop is the signal.

Authoring guidelines for capability sets

These are guidelines, not product SLAs.

• Success rate: Aim for roughly 60–80% for a capability set. This leaves headroom to see improvement (or regression) when models, prompts, or tools change. 

  • A success rate near 100% for a capability set often means your expected answers are too loose or self-confirming. If almost every case scores 1.0, the expected answers are probably too thin — they aren’t requiring enough material facts to catch mistakes 

  • A success rate near 0% often means your set up needs improvement. Check that the project has the data the questions ask about, data is syncing, the agent has the right tools and instructions, and that the expected answers match what’s actually in the project.

• Set size: Most authors start with 20–50 cases and grow from there. Cover a range of capabilities and failure modes (discovery, list, get-by-id, multi-entity), not just happy paths.

Authoring guidelines for regression sets

These are guidelines, not product SLAs.

• Regression set: Aim near 100%. That isn’t “expected answers are too thin” — it means these cases are supposed to keep passing.

• A drop on a regression set is the alarm. Inspect View Report before changing expected answers.

• If a regression set sits around 60–80%, those cases aren’t stable yet. Leave them in the capability set (or move them back).

Phase 4: Schedule Recurring Runs

Once your eval set is stable, you can schedule recurring runs to stay informed of regressions. 

1. On the eval set page, scroll to Run Schedule.

2. In Run, choose how often it should run. Use Weekly for most cases.

   • Set the day (if asked), time (at), and timezone. A low-traffic window, such as a weekend morning, is a good default.

3. Set Schedule Status to Active. Use Paused to stop automatic runs without clearing the schedule.
   Note: You won’t see this toggle if you selected ‘Manual’ as the frequency. 

4. Click Save Changes. Confirm ‘Next Run’ looks right

Scheduled runs use credits the same way manual runs do, but because they run automatically, it's easier to lose track of how often they're consuming credits.

Phase 5: Improve and Expand Your Eval Set Over Time

• False negatives: If the agent's answer is acceptably equivalent to the expected answer but still fails the judge, update the expected answer and re-run. Only loosen an expected answer when the agent's response is genuinely acceptable — don't adjust it just to match a wrong response, or you'll hide real regressions.

• Bug fixes: When you fix a problem with the agent’s instructions or tools, add a case that covers it so future regressions are caught.

• Drift: Revisit expected answers whenever the underlying product or agent changes in a way that affects what a correct answer should look like. For example, if a new feature changes how the agent should respond, or the agent's instructions are intentionally updated, older expected answers may no longer reflect the current correct behavior, so you should update them to match.

• Graduate cases: When a capability case passes reliably, move it into the regression set. That keeps the capability set focused on skills that can still improve, and puts the stable case on the safety net so a later change can’t break it unnoticed. 

  • New bug-fix cases belong in the regression set so that bug can’t return unnoticed. 

Get Help

Questions, feedback, or requirements? Reach out to your Procore point of contact.