🔍 Read the full analysis: OpenAI Agent Training And Ironclad: What The Fine Print May Tell You on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI says it trained GPT-6 Astra in hosted copies of Ironclad’s contract-management software using 11 legal, commercial and procurement tasks. Astra met an average of 55% of task criteria; the reported time estimates were simulated, and OpenAI says human oversight remains necessary.
OpenAI says it trained its GPT-6 Astra model on legal, commercial and procurement workflows inside hosted copies of Ironclad’s contract-management software, reporting that Astra met an average of 55% of the criteria set for 11 tasks. The work matters because it tests whether advanced AI models can handle professional processes inside real business software, but OpenAI’s figures do not show that the workflows are ready to run without human review.
OpenAI’s October 6 post, “Advancing computer use with Ironclad,” describes a collaboration with Ironclad, a contract-management software company. The post says Ironclad staff and OpenAI employees who use the product selected 11 tasks spanning legal, commercial and procurement work. Examples included creating nondisclosure agreements, building procurement approval processes and updating reusable contract language to reflect a requester’s jurisdiction. OpenAI estimates an experienced user would need 30 to 40 minutes for each task.
Tasks were assessed against rubrics containing 8 to 50 criteria, depending on complexity. OpenAI reports that GPT-6 Astra met an average of 55% of those criteria, compared with 41.6% for GPT-5.6 Sol at the high setting. An internal OpenAI model used during Astra’s development scored 63.7%. On one showcased task, Astra met about 94% of the criteria. These are rubric results, not the share of tasks completed successfully: the average indicates how many listed requirements were met across evaluations.
For training, Ironclad supplied hosted product copies in which models could practise. OpenAI says it generated synthetic tasks using publicly filed contracts from the U.S. Securities and Exchange Commission’s EDGAR database, filtered to remove personal information. The company says it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data. OpenAI also reports estimated time per attempt of 19.2 minutes for Astra and 37 minutes for GPT-5.6 Sol, while specifying that these are simulated estimates based on assumed processing and generation speeds—not measured customer time savings.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Partial Workflow Scores Matter
The results point to a practical test for AI agents: can they follow linked business rules inside specialised software, not just produce plausible text? A procurement workflow may require Finance approval above a spending threshold, Security review for certain requests and Legal review when terms depart from a standard form. Missing even one condition can create a control failure. A score showing that an agent met more than half the criteria does not establish that it completed the workflow safely or correctly.
That distinction matters to companies considering agents for contracts, finance or customer records. OpenAI’s post says an agent may lose track of a business rule during a multistep task and that human oversight remains necessary. Ironclad CTO Sunita Verma likewise stressed that agents must preserve “the controls teams rely on.” The reported improvement may be relevant to model development, but the available results do not establish that businesses can replace review by experienced staff.
The partnership model also has implications for software providers. OpenAI is asking a small number of companies to contribute concrete examples of tasks agents cannot reliably complete, along with subject-matter expertise, secure test environments and data suitable for research. If agents improve inside a vendor’s product, customers may increasingly interact through the agent rather than the product’s screens. That makes the software’s underlying business rules, records, audit trails and safeguards central to its value—and increases the need to define who is responsible when an agent misses a requirement.
Products Worth Considering
Samsung 27-inch Odyssey G5 QHD Gaming Monitor (G51F)
Logitech MX Keys S Wireless Keyboard, Graphite
Samsung Odyssey G4 25-inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ironclad Became a Test Environment
The OpenAI post describes a specific research setup, not a general deployment announcement. Ironclad provided hosted copies of its product for model practice, and the selected tasks were drawn from work associated with legal, commercial and procurement processes. The approach differs from evaluating a model only through written questions: the agent is tested on actions and requirements within specialised software.
OpenAI presents GPT-6 Astra as its first frontier model trained in this way. The source material also notes that some AI trackers mistook “Ironclad” for the name of a hardened agent framework; in the post, it refers to the software company. OpenAI’s account frames the collaboration as a way to learn where agents fail on real workflows and says a full contracting platform remains essential. The described work is research on selected tasks, not evidence that the company has rolled out autonomous contract processing across Ironclad customers.
What the Evaluation Leaves Open
The post’s reported averages do not show which criteria Astra missed across all 11 tasks, how often an error affected a required approval, or whether any task was completed end to end without correction. The source describes one showcase task at about 94% of criteria met, but that single result does not establish performance across the wider set. It also does not provide a customer deployment study or measured productivity outcomes.
OpenAI’s time figures are simulations, and the source says they apply to the 11 research tasks rather than Ironclad workflows generally. The evaluation’s broader applicability, the frequency of failures in ordinary use and the level of review needed for different types of contracts remain unclear. OpenAI says it did not use specified categories of customer or confidential data, but the source does not detail all data-governance arrangements or describe a commercial release timeline.
The Next Test for Software Partners
OpenAI says it is inviting a small number of software companies to partner on tasks that current agents cannot reliably complete. Prospective partners are asked to bring a concrete failure example, people with deep knowledge of the work, a secure test environment and data that can safely be used for research. The post does not name other partners or give a schedule for further collaborations.
For companies weighing whether to let agents operate in business systems, the immediate next step is to ask vendors for more than an overall score: which requirements were missed, how exceptions are handled, what human review is required, and what records capture the agent’s actions. Further evaluations would need to show performance across varied tasks and clarify whether reported improvements translate into accurate, controlled work in customer settings. Until those details are available, OpenAI’s results are best read as evidence of progress in a research test—not proof that agents can independently run contract workflows.
Key Questions
What is Ironclad in this OpenAI announcement?
Ironclad is a contract-management software company. OpenAI’s post describes using hosted copies of its product to train and evaluate models on selected legal, commercial and procurement tasks.
Does Astra’s 55% score mean it completed 55% of the tasks?
No. OpenAI says 55% is the average share of rubric criteria met across the evaluation, not the percentage of tasks completed successfully.
Were the reported time savings measured with customers?
No. OpenAI describes the 19.2-minute and 37-minute figures as simulated estimates based on assumed processing and generation speeds. They are not measured customer time savings.
Did OpenAI say the agent can run contract workflows without review?
No. The source says human oversight still matters, particularly when an agent may lose track of a business rule. The reported evaluation does not establish readiness for unsupervised use.
What data did OpenAI say it used for training?
OpenAI says it generated synthetic tasks from publicly filed contracts in the SEC’s EDGAR database, after filtering for personal information. It says it did not use OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.
Source: ThorstenMeyerAI.com
