Missing or corrupt files in a migration run are expensive – they often crash the migration run and a lot of time (and money) is spent tracking them down … one by one as they come up. Yet it doesn’t take long (with the right tools) to check all the files in your file store and prevent a lot of later pain.
With the ECM File Store Validator and Copier we typically find a few hundred corrupt or missing files for every million documents! But in the initial run we often find a lot more that just look to be missing (see the real life examples below).
In two recent data migrations using the ECM File Store Validator and Copier we found:
- 10s of thousands of “missing” files.
- The reason: they were invisible as some users had been removing access to everyone except their own work group.
- 10s of thousands of “missing” files:
- The reason: files for a branch office had not been copied to head office
- when copying files to the location the migration tool was working from a few files that appeared to be present were not able to be copied.
- The reason: a subtle security issue.
- a few hundred zero byte files (corrupt) and a few hundred missing files per million documents:
- The reason: they really were missing, a few hundred per million is a typical result.
By dealing to all those issues BEFORE doing the migration runs much pain (and some embarrassment) was avoided, and the truly missing or zero byte files were replaced with placeholders by the ECM File Store Validator and Copier allowing the migration tool to smoothly do its job.
Of course you can’t find every problem, but you can prevent most of them affecting the process..
Other uses for the ECM File Store Validator and Copier
- after the analysis it was used to work in-synch with a migration tool by copying files from the data store to the “pick-up” location. It would start about the same time but being faster than a migration tool would always be ahead.
- do a regular quality check on your file store to check for any missing or zero byte files
Download the free trial
- Main screen showing results of a Copy action
- Main screen showing results of a Checking action
- Error Log showing Errors from the Checking action
File Checker and Copier – results of copying files
- 532 documents copied
- 1 zero byte file
- 489 MB in all
- 1 placeholder used for the missing file
File Checker and Copier – results of checking files
This is checking the same files and of course gets the same results.
File Checker and Copier – How the errors are displayed
On the Errors tab screen shot (I’ve merged some of the “off screen” content into the one screen shot) we see the original file name, the batch run date, and most important, the Error type.
While this screen will only show errors, when the files are copied a record is created in the log table for each file including the full path of the new location of the file. This can then be used by the migration software.
- identify problem files and offer replacement with placeholders so migration runs don’t crash or have other problems due to these problem files
- copy out just the files needed for a migration run. Minimise space requirements for your working copy.
ECM File Store Validator
- check file store for missing or zero byte files
- report on number of files and total size of selection being analysed
- analyse entire file store or just the subset you are interested in
- it logs results to an SQL Server table and has a simple log viewer which you can copy to Excel if needed.
- it’s fast – we’ve seen it analyse a million documents in about half an hour
ECM File Store Validator and Copier
- all File Validator features (it’s included). It also reports on files that can’t be copied.
- copy a subset of the file store for the current migration run
- so your migration tool can be run against a copy of the file store instead of the live store (risk management)
- so you don’t need to find the space to store a full copy of the file store
- replace problem files with placeholders
- add file extensions
- rename files if needed
- the Logging table includes the full file path of the copied file so you can link this to your migration software if required.
Both can use substitutions to the file path as read from the database if they are to look at a copy of the file store rather than the master copy
How they work
- You provide the SQL string which extracts document and version information and the file path.
[File path can be altered by the FVC if needed].
- Set up a table to hold the log data.
- Set the parameters
- Click on Start
For more detail, or to download the free trial …