What started as a simple migration project three months ago had now become a major roadblock. According to the SOW and project plan, the data migration should have been completed by now, clearing the way for a much larger transformation initiative. Instead, less than ten percent of the data had moved. Schedules were constantly being revised, senior management was frustrated, and the outsourced migration team insisted the bottleneck was the old storage system. They said they had completed many successful file system migrations before, but this one was different. It was the slowest they had ever seen, and nothing they tried made a meaningful dent in performance.

The roots of the problem stretched back five years, when the customer embraced the promise of low‑cost, highly scalable, durable S3 object storage. Their content management system, however, only supported SMB access, so they deployed a file system gateway to bridge SMB to S3 and provide a caching layer for recently accessed data. The cache was sized to keep 30 days of data in a hot state.

Four months ago, we were invited to pitch our solution. Ultimately, the customer chose not to pay for migration software when a free open‑source option existed, and they selected another professional services team. I tried to explain the technical challenges of migrating tiered data, but my reasoning was overshadowed by assumptions about how the system “should” work.

Now I was back—this time explaining the difference between online and offline data, and how the rehydration of files back into cache during migration becomes a performance killer. The gateway cache is only a fraction of the size of the underlying object store, which magnifies the problem. These two factors alone explained why the open‑source approach was failing to deliver.

There are two ways to accelerate this type of migration by orders of magnitude. The first is an intelligent method that aggressively manages cache rehydration and clearing. We pioneered this approach over 15 years ago with DiskXtender and Centera, and it typically delivers a 4–5× speed improvement. The second approach identifies which files are online and which are offline, then migrates them independently. The “special sauce” is our ability to extract file contents directly from the S3 objects and write them straight to the new target storage—bypassing the gateway entirely. This requires reconstructing the original file names, directory structure, timestamps, and ACLs, then validating each file with a content‑based cryptographic hash to ensure a perfect migration.

The customer asked us to run a small 20 TB proof of concept to validate our process and performance. Our software migrated the data in less than a day with 100% accuracy. Within a few weeks, nearly a petabyte had been moved. In the end, a modest investment in migration software saved the customer months of time and frustration.