OpenAI、Anthropic等在金融智能体上如何权衡自主性与安全

本周 AI Engineer New York 大会 的核心主题之一是 如何在智能体自主性与安全之间取得平衡。前沿实验室和金融公司本周的演讲都聚焦于这一问题。

主要经验是：安全的自主性取决于 你把控制和人类判断放在哪里，而不是你施加了多少限制。

赞助商：用 shop.dev 将购物功能集成到你的智能体中

为智能体添加商务功能通常意味着从头重建搜索和结账流程。shop.dev 提供单一 API，可在任何智能体或应用中实现购物功能。一次接入即可访问超过十亿件商品的实时价格和库存，支持 Shop 即时登录，以及在对话中直接完成的一键 Shop Pay 结账。它对任何开发者开放，无需账户或密钥即可试用。→ 立即试用 shop.dev

OpenAI 金融服务 ChatGPT 负责人 Joseph Kim 今天在 AIE New York 发表了主题演讲。（我们还与 Kim 录制了一期特别播客，即将发布。）他谈到了可靠性以及 “有用性门槛”，他说这是“模型变得足够强大和可靠、能够为你节省时间的那个临界点。”

Kim 认为，不断增强的模型能力很快将使金融分析师跨过这一门槛，从而能够将更多工作委托给智能体。

他对 2027 年 AI 在金融领域的作用做出了一些具体预测。他声称，明年金融行业不仅会减少对 Excel 和 PowerPoint 的依赖，而且 智能体将足够强大，能够完成“投资想法生成和交易”。

模型能力的“涨潮”也带来了更好的 可追溯性。

“在金融领域，你需要能够核查一个数字的来源并理解其背景，”Kim 说。“我们希望这种审查成为 [ChatGPT for Financial Services] 体验的自然组成部分。正是这类产品细节 帮助将模型的能力转化为可用的工作成果。”

至于何时需要人类参与其中，Kim 将此视为一个产品设计决策。

“界面会如何变化？用户仍然需要控制、审查或批准什么？这些都是你现在就可以开始着手处理的产品设计问题。”

在对冲基金中平衡 AI 安全与创新

对冲基金 Two Sigma 的架构高级副总裁 John Feminella 正在处理 Kim 所提到的那类产品设计问题。在主题演讲中，他介绍了他们实施的五项控制措施，旨在约束智能体的同时不限制模型能力。

Feminella 认为，安全与创新不应被视为光谱的两端。他特别批评了 “审批疲劳”，即一个人批准所有事情不仅会阻碍创新——而且如果他们没有全神贯注于所有必须做出的决定，还会引入风险。

在 Two Sigma，他们实施了 一系列控制措施，他将其定义为 “护栏和共享能力，让团队能够以不妨碍其目标的方式开展工作。”

这些控制措施是可组合的，意味着“每一项各司其职，它们共同 让我们能够在适当的安全水平下以真正的自主性运行智能体。”

One of the controls is to use auto-mode classifiers, which Feminella said “make decisions about routine calls and then send a person only the requests that are most important, the ones that are most likely to make a difference.”

Classifiers can be used for ambiguous cases, whereas deterministic rules are better for clear cases.

Anthropic: Give agents context, then define the boundaries of trust

Anthropic’s presentation outlined a similar framework, in its case with the goal of building “trust” into financial agentic workflows.

Last month, Anthropic announced Claude for Financial Advisors, allowing financial advisors to connect their systems to Claude and a set of workflow skills.

At AIE this week, Belinda Neal, Managing Director of Financial Services at Anthropic, explained why Anthropic has partnered with financial firms on the new product.

“Every so often, a model crosses a threshold. And when it does, a wave of working changes.”

Neal’s colleague Keon Shahab, who works on applied AI at Anthropic, then talked about “building the trust layer for the agents.” He said this involves three steps:

“Who the agent is acting as with an identity and permissions layer. How we can sandbox and determine the approval layer of the agent, so that someone is signing off before it acts. And lastly, how we know the agent will still work once we adjust any variables over time.”

Shahab illustrated the boundary between proposing and acting with a trade break investigation agent, which would recommend an order amendment for human approval.

“We want it [the agent] to just propose new amendments to our orders data, not actually make those amendments,” he said.

“By building once with a foundational layer that you trust, you can work at speed to deploy many agents at scale,” Shahab concluded.

Bloomberg: Enforce security outside the model

As Head of Architecture & Emerging Technologies in the Office of the CTO at Bloomberg, Andrey Rybka has the responsibility to balance autonomy and scale at the company.

In a presentation today at AIE, Rybka talked about a series of real-life events where agents caused havoc — and the defenses that can help prevent it. These include scoped identities and credentials, controls over tool calls, outbound traffic restrictions, audit trails, budgets, and kill switches.

In one example, an agent treated instructions embedded in a marketplace review as an instruction to stock a product. Rybka recommended input validation, prompt injection defenses, provenance tracking, and human approval for high-value actions.

“We want deterministic controls outside the model,” Rybka said. “Check every call, log it, and keep a kill switch.”

Go fast and stop safely

Even though frontier labs like OpenAI and Anthropic are building a ‘trust framework’ for financial agents, ultimately it’s up to financial firms to put controls around their agents. Two Sigma’s John Feminella hit the nail on the head: implement controls that enable autonomy to happen, but don’t put up roadblocks.

Or as he put it in his conclusion: “Why do cars have brakes? It’s not just so that you can stop, it’s so that you can go fast and stop safely.”