Design Model Weight Distribution

Question

Design a system for distributing large machine learning model weight files to a fleet of inference workers.
Context:
Model weights may be tens to hundreds of GB and may be split into multiple shards.
A new model version can be published several times per day.
Thousands of GPU inference workers across multiple regions need to receive the correct version.
The system must support staged rollout, rollback, integrity verification, access control, and minimal serving downtime.
Discuss:
Functional requirements and non-functional requirements.
High-level architecture.
Storage and metadata design.
APIs for publishing, discovering, downloading, and activating model versions.
How workers fetch and cache weights efficiently.
Versioning, consistency, and rollout strategy.
Failure handling, security, monitoring, and scalability tradeoffs.

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.