When an organisation offers predictions through an API, repeated queries can reveal enough about a model's behaviour for someone to train a substitute. This is called model extraction. The substitute may imitate some of the original model's outputs, but that does not mean the attacker has recovered its exact weights or training data.
What is model stealing?
A model is exposed when an external user can send inputs and receive predictions. In a model extraction attack, the attacker collects many such input-output examples and uses them to train a separate model. The aim is usually to reproduce the target's behaviour on relevant inputs, not to obtain the original model file.
How model stealing works
At a high level, an attacker identifies a prediction interface, submits selected inputs, and records the returned labels, scores or other outputs. Those examples can then be used to train a substitute model. The feasibility and fidelity of extraction depend on the model, the information returned, query limits and the attacker's resources.
Business impact
A substitute model may reduce the commercial advantage of a proprietary prediction service, enable competing functionality, or make it easier to study a model's behaviour offline. The actual business impact depends on what the model does, the value of its outputs, and how closely a substitute can reproduce them. Extraction does not by itself bypass the target service's authentication or access controls.
Security controls
Reduce unnecessary exposure: authenticate callers where appropriate, apply per-account quotas, monitor unusual query patterns, and return only the output detail the product needs. These measures can raise the cost or improve detection, but no single control guarantees that extraction is impossible. Choose controls in light of the model, interface and business use.
Conclusion
Model extraction is a practical risk for some systems that expose prediction interfaces. Teams can assess how much information their interface returns, who can query it, and whether usage controls match the value of the model. The right mitigations depend on the system; the research below provides technical detail and examples.