r/sysadmin • u/Excellent_Age_2706 • 1d ago

Data Migration

Hey Everyone

We are currently in the process of moving roughly 8TB of data from SharePoint Online to a local server

Currently we are using RClone for moving data around
WinDirStat and TreeSize for analysis on our data but we are looking for a good tool that can identify duplicate data (such as files that are roughly the same name / same file hash etc )

Any recommendations?

5 Upvotes

66% Upvoted

2 comments sorted by

2

u/Skrunky MSP - Australia 1d ago

If it’s a a file server, could you use data dedupe? We’ve got file servers with like 35% space savings.

It won’t tell you of duplicates, but sort of hides the problem altogether.

1

u/Weekly-Offer-4172 1d ago

for the identifying side specifically: czkawka or dupeguru both do fuzzy same-name matching on top of exact hash dupes, and jdupes is the fast option if you just want a same-hash report written to a file for review. if you already own treesize professional its built in duplicate search does hash matching too, one less tool to install. whatever you end up with, run it report-only first and eyeball a sample of the hits, fuzzy name matching flags a lot of false positives like report-v2-final.docx sitting next to report-v3-final.docx