Copying contents of one hard drive to another

Topic

New Reply

I have two external hard drives attached to my Dell XPS 8700 desktop PC. One is a Western Digital USB 2.0 1TB drive, and the other is a new Seagate USB 3.0 2TB drive. I’ve used the Western Digital (old) drive as a file archive. My aim is to copy all the files from the old drive to the new, larger, faster drive. The PC is running Windows 7 Home Premium with all available updates as of today.

The copy process will involve a total of 83,679 files in 4,738 folders, taking up about 155 GB of hard drive space.

I’ve worked with PCs for almost 30 years, but I’ve never copied that many files at one time. Can I just use Windows Explorer to grab all the files on one drive and drag-and-drop them (or copy and paste them) from the old drive to the new drive, or am I asking for trouble? If such a massive file copy is problematic, is there a better way? The files are in 18 directories (and lots of subdirectories within those), so I could copy the files in chunks, but it would be more convenient just to do it all in one big (and probably hours-long) process.

–Larry

Reply | Quote

Replies

Whatever you do don’t copy all that data at once, do it in manageable chunks of 10GB or less, and stay near
so that you can monitor the entire process. You can graduate the process up to higher GB transfers, but keep it reasonable.

YES, it IS going to take a few hours at least, so plan on doing little else.

*There is no need for any other software or processes other than simple copy and paste.
*Checkdisk both drive to ensure they are error free.
*Don’t do much of anything else with the system.
*You might start off with a freshly booted system so that the memory is fresh. (optional)

I’ve attempted similar backups with large video files, as long as the above is done you’ll be fine.
If I can do it with 1-2TB worth of data you can do it with just 155GB. (smaller than my music collection)

Reply | Quote

Whatever you do don’t copy all that data at once, do it in manageable chunks of 10GB or less, and stay near so that you can monitor the entire process.

Why? I’ve done over 500GB of data in a single copy. Just started it running Friday night.

Does help if both devices are USB 3

Reply | Quote

Why? I’ve done over 500GB of data in a single copy. Just started it running Friday night.

Just because it CAN be done does not mean that it SHOULD be done.

I’ve outlined the safest way to do it, the OP has the option of graduating the process as he sees fit.

Reply | Quote

I’ve outlined the safest way to do it, the OP has the option of graduating the process as he sees fit.

Why is it the safest? What’s wrong with XCopy or RoboCopy or HoboCopy or EIEIOCopy of the entire thing in one chunk?

Reply | Quote

I also recommend the copy process, regardless of how much chunk or all-at-once you do. Pooblematic recovery can be the effect if something goes wrong during said Move process. With Copy, you can quick-compare both source and destination HDs before deleting source material.

"Take care of thy backups and thy restores shall take care of thee." Ben Franklin, revisted

Reply | Quote

I’d recommend RoboCopy (builtin Win 7+ you can download it for XP) as it is more efficient than standard copy. At least use a command window and XCopy. Just my opinion of course. 😆

HTH :cheers:

May the Forces of good computing be with you!

RG

PowerShell & VBA Rule!
Computer Specs

Reply | Quote

I wondered if you’d chip in with that RG! I concur, having started using it, it’s quite simple and deals with a lot of data easily.

Eliminate spare time: start programming PowerShell

Reply | Quote

Thanks, everyone, for the suggestions so far. Here are some initial thoughts after reading your posts:

I asked my question partly because of an experience I had probably 10 years ago. I tried dragging-and-dropping a few thousand files from one hard drive to another (copying, not moving), and somewhere in the process Windows threw up an error message and just stopped the process. The error message wasn’t that helpful in telling me what had been copied and what hadn’t, and it took me quite some time to figure out how to resume the process without omitting some files. I’m guessing I was using Windows XP at the time, but I don’t remember exactly. In any case, that adventure made me a little shy about copying files en masse. I was thinking that Windows 7 might be safer for this sort of task, but based on the bulk of responses so far, I’m not so sure.

I will definitely be copying and not moving. I plan to take the old drive off-line and store it off-site as a backup of all the files I’ve accumulated so far.

CLiNT, Thanks for the suggestion regarding doing the copy right after booting the PC, and not doing much of anything else with the PC while the files are being copied. I’m guessing that anything else I do will just slow down the process. I figure I’ll grab a good book or watch a good movie in the same room so I can keep an eye on it. 🙂

RetiredGeek, I’ve never used RoboCopy, but it certainly sounds promising. Would I be able to do the sort of mass copying I’m contemplating with it? I’ve done some command-line stuff, but it’s been a while (I’m retired, and I’ve probably forgotten most of what I used to know). Is there a “getting started” guide to using RoboCopy?

–Larry

Reply | Quote

RetiredGeek, I’ve never used RoboCopy, but it certainly sounds promising. Would I be able to do the sort of mass copying I’m contemplating with it? I’ve done some command-line stuff, but it’s been a while (I’m retired, and I’ve probably forgotten most of what I used to know). Is there a “getting started” guide to using RoboCopy?

–Larry

Larry,

Click on the RoboCopy link in my post for the syntax. HTH :cheers:

May the Forces of good computing be with you!

RG

PowerShell & VBA Rule!
Computer Specs

Reply | Quote

TechNet Magazine recommend RichCopy over RoboCopy.

If you are going to use RoboCopy then it may help to use the Microsoft GUI to it. The article doesn’t make it particularly clear but the GUI can be downloaded using the UtilitySpotlight2006_11.exe link at the top of the page.

Hope this helps…

Reply | Quote

I’ve migrated plenty of file servers and I always use Robocopy. Never had an issue.

Here’s a little batch file to help you on your way.

cheers, Paul

Code:

@echo off
rem *****************************************************************************
rem Script Robocopies the File Server files to a new Location
rem 
rem *****************************************************************************

setlocal

REM CHANGE the following lines to a new folder ; make sure there is enough space on the destination server.
set Orig_Server=D:
set Dest_server=E:

Rem Additional options if required.
set _opt=/quit
::set _opt=/XD RECYCLER /XF pagefile.sys
set _opt=/XD RECYCLER /XD $RECYCLE.BIN /XD “System Volume Information”

Rem Collect date and set up log file
call %0..GetDate.cmd
Rem Set log file location and name. Create directory if required.
set _logdir=%~dp0Logs
if not exist %_logdir% md %_logdir%
set _log=%_logdir%MigrateData%yy%%mm%%dd%%hr%%min%.log

Rem Edit the file for whichever command you want to use. 
Rem Use steps in order for efficient copying.

Rem STEP 1 – RUN MANUALLY
Rem Create file/directory structure only, do NOT copy security.
Rem Prevents directory fragmentation and improves file system performance.
Rem Verbose output.
:::robocopy %Orig_Server% %Dest_server% /NDL /NFL /NP /S /E /COPY:DAT /CREATE /V /TEE /R:1 /W:1 /log+:%_log% %_opt%

Rem STEP 2 – SCHEDULE TO RUN NIGHTLY
Rem Update, only copy files that are newer than the destination. Do NOT include offline files.
Rem Delete files not in source.
Rem Do NOT copy security.
Rem Quiet, log summary and errors only.
::robocopy %Orig_Server% %Dest_server% /XA:O /NDL /NFL /NP /MIR /COPY:DAT /R:1 /W:1 /log+:%_log% %_opt%

Rem STEP 3 – RUN MANUALLY AFTER THE LAST RUN OF STEP 2
Rem Copy Security, ownership and auditing data only. Do NOT include offline files.
Rem Quiet, log summary and errors only.
::robocopy %Orig_Server% %Dest_server% /XA:O /NDL /NFL /NP /S /E /IS /COPY:SOU /TEE /R:1 /W:1 /log+:%_log% %_opt%

Rem STEP 4 – UPDATE ANY FILES THAT WERE IN USE DURING THE LAST SYNC – RUN MANUALLY
Rem Copy all newer files, do NOT delete files not in source. Do NOT include offline files.
Rem Quiet, log summary and errors only.
::robocopy %Orig_Server% %Dest_server% /XA:O /NDL /NFL /NP /S /E /COPYALL /TEE /R:0 /W:1 /log+:%_log% %_opt%

Rem STEP 5 – RESTORE OFFLINE FILES FROM BACKUP.
Rem Robocopy is not suited to migrating offline files.

Rem ALTERNATE COMMAND – PERFORM FIRST 3 STEPS AT ONCE
Rem Copy all data including security, delete files not in source. Do NOT include offline files.
Rem Verbose output.
robocopy %Orig_Server% %Dest_server% /S /E /XA:O /NDL /NFL /NP /R:1 /W:1 /log+:%_log% %_opt%
Rem RESTORE OFFLINE FILES FROM BACKUP.

Rem errorlevel checking
if errorlevel 16  echo ERROR: ***FATAL ERROR***  >>%_log% & goto end
if errorlevel 15  echo ERROR: FAIL MISM XTRA COPY >>%_log% & goto end
if errorlevel 14  echo ERROR: FAIL MISM XTRA      >>%_log% & goto end
if errorlevel 13  echo ERROR: FAIL MISM      COPY >>%_log% & goto end
if errorlevel 12  echo ERROR: FAIL MISM           >>%_log% & goto end
if errorlevel 11  echo ERROR: FAIL      XTRA COPY >>%_log% & goto end
if errorlevel 10  echo ERROR: FAIL      XTRA      >>%_log% & goto end
if errorlevel  9  echo ERROR: FAIL           COPY >>%_log% & goto end
if errorlevel  8  echo ERROR: FAIL                >>%_log% & goto end
if errorlevel  7  echo OK:      MISM XTRA COPY >>%_log% & goto end
if errorlevel  6  echo OK:      MISM XTRA      >>%_log% & goto end
if errorlevel  5  echo OK:      MISM      COPY >>%_log% & goto end
if errorlevel  4  echo OK:      MISM           >>%_log% & goto end
if errorlevel  3  echo OK:           XTRA COPY >>%_log% & goto end
if errorlevel  2  echo OK:           XTRA      >>%_log% & goto end
if errorlevel  1  echo * SUCCESSFUL COPY * >>%_log% & goto end
if errorlevel  0  echo Notice:    –no change, no data copied–    >>%_log% & goto end
:end

Reply | Quote

I have two external hard drives attached to my Dell XPS 8700 desktop PC…My aim is to copy all the files from the old drive to the new, larger, faster drive.

Hello Larry,

I had a similar task recently, copying nearly 3Tb of data from an existing 4Tb external drive to another, new one. My backup software, SyncBack Pro, did the work for me, with the old drive as the source and the new drive the destination. Did it one go (slowly, of course) but it was a start and forget process.

The software has error reporting, so had there been any problems I feel I could have dived in at the right place to fix them.

Roger

Reply | Quote

Doing our own copies in one block is OK, we (should) know our own PC’s are stable and cool enough to cope, what we don’t know is anything about the state of other people’s PCs.

Reply | Quote

My recommendation is to use Backup software that can do a file and folder backup. Just set it up to copy everything you want from the source drive to the target drive and start it. Most of these programs can do a file verify and error reporting.

Reply | Quote

One thing that seems to have been omitted is any discussion of the structure of the data. Big files, like backup files, are just big (and simple to transfer), but when you are in a directory tree (sorry, folder tree) with sub-sub-sub-sub-sub-sub-sub folders, or when filenames are extremely long, problems can arise that will at minimum slow things down, and possibly bring them to a halt. I find that GoodSync is exceptional in handling complex structures.

Cloning is of interest, although it would not apply in this case.

Reply | Quote

Never seen copying halt with lots of files / folders, examples please?

cheers, Paul

Reply | Quote

Never seen copying halt with lots of files / folders…

Neither have I, and I regularly have to copy customers’ data when setting up a new PC for them. Usually anything from 20GB to hundreds of GBs.

Reply | Quote

Neither have I, and I regularly have to copy customers’ data when setting up a new PC for them. Usually anything from 20GB to hundreds of GBs.

Same. Unless you are copying into existing folders where you might get a conflict or you have very long file names, you should be fine.

I’ve done thousands of files and hundreds of GB in one go with no issues. I also use my PC when it’s copying and have had no issues with that other than it slows down if I’m copying across my WiFi connection and accessing the ‘net at the same time.

The only thing I would add is to expect it to take longer than you thought it would!

Reply | Quote

My thanks to everyone for the discussion and suggestions. I wound up using FastCopy based on several reviews I read about file-copying utilities. After I posted my question, I purchased another USB 3.0 external drive to use as an additional file backup. I used FastCopy to basically “clone” all the data from the old drive to both of the new drives (one at a time). The process went smoothly and flawlessly, and I’m a happy camper.

–Larry

Reply | Quote

Sorry I never got around to sending a reply to Paul T, Coochin, and dogknees, complete with reference.

As it happened, I had my mind on one program when I should have cited a different program that I use viz. TeraCopy.

If you read the documentation for TeraCopy, you will see that it chokes on certain files that have a problem and leaves them behind:
“TeraCopy is designed to copy and move files at the maximum possible speed. It skips bad files during the copying process, and then displays them at the end of the transfer so that you can see which ones need attention.”

That is the behaviour that I was trying to describe, quite poorly I will admit, and before referring to the wrong program on top of that, documented for the program that I do use regularly. I recommend that you give TeraCopy a try: it is PDQ.

Reply | Quote

As the object was to copy one hard drive to another which was, presumably, the same size or larger, the logical solution would be to clone it. I just cloned one external mechanical hard drive to another via USB3 (using DriveClone software) as an experiment, resulting in a perfect copy. Data transfer speed was about 200Gb per hour. Using SSDs more than doubled the speed.

Reply | Quote

As the object was to copy one hard drive to another which was, presumably, the same size or larger, the logical solution would be to clone it. I just cloned one external mechanical hard drive to another via USB3 (using DriveClone software) as an experiment, resulting in a perfect copy. Data transfer speed was about 200Gb per hour. Using SSDs more than doubled the speed.

Cloning is a risky procedure to recommend because the drive being copied to is a large new drive, which typically has pre-installed software, and cloning will overwrite the entire drive. I have a new WD 4 TB drive that has a collection of extremely valuable goodies installed on it, which I admittedly copied as soon as I began, but to simply clone to the drive without warning of the consequences will devalue the drive by the true value of the pre-installed software.

Reply | Quote

In 1997 I worked in a small, private, computer business. It was one of those times in Computer History, when small and very slow Hard Drives were giving way to larger and faster Hard Drives, as OS’s and Programs got bigger and bigger.

My boss, a technician himself, found a new program from New Zealand, called simply “Ghost”. Every month he would download a FREE 30 day trial copy and provide copies to his tech’s. I used Ghost almost daily, to CLONE small HD’s to larger HD’s and it worked perfectly, every time.
And of course, since it ran from a DOS boot disk, it was totally independent of any OS.

As time went on, Ghost was purchased by Symantec Corp. and upgraded to run from within Windows. However up to 2005 there was still a DOS version.
Ghost 11.5 was the last DOS version and had been revised to work with the latest NTFS disk format.

I still use that version today, and I still run it from a DOS boot disk, (either a CD or Flash Drive) so it runs totally independent of the OS, or I can even copy disks with NO OS at all, or a corrupted OS that will not even boot the system. My software GURU tells me that Ghost 11.5 will back up Linux, Windows NT and Windows Server.

Yes, I’m fully aware that there are other Backup/Restore programs that claim to be able to CLONE one HD to another HD, and I’ve tested many of them, but I have found every one of them, seriously lacking in some respect. So I continue to use Ghost 11.5. Since it runs in DOS, it has 100% use of RAM and CPU time.
Programs running in Windows, don’t have that luxury.

Cheers mates!
The Doctor 😎

Reply | Quote

And just where would one find a copy of that ghost??

:cheers:

🍻

Just because you don't know where you are going doesn't mean any road will get you there.

Reply | Quote

And just where would one find a copy of that ghost??

:cheers:

Buy an SSD.

With every (boxed) SSD comes one or two CDs and (possibly) additional hardware.

Mine has Acronis in one form or another that includes the Clone function, and it definitely works.

That is either ‘buy hardware to get the bundled software’, or ‘buy (what there is of) the software to get the bundled hardware’, or, a win-win purchase. Ask what is in the box before you buy.

The Acronis CD is re-usable to the end of time or the end of CD readers, whichever comes first.

I have three Verbatim (complete with Acronis and hardware for installation in a desktop) and one Samsung (with disk still in the envelope and no hardware that I am aware of). All are installed in laptops and cloning them was a breeze.

Edited to add: one particularly convenient piece of hardware that comes with the Verbatim drives that I forgot to include is an external enclosure to accommodate a 2-1/2″ displaced hard drive. It’s just USB2, but it lets you use the old drive as an external drive.

Reply | Quote

A different opinion:

Ghost version 11/12 is unable to interpret the 100MB FAT32 EFI partition so if an Image of an entire GPT Windows disk is captured and then applied to another system, it will not be bootable regardless of the ghost switches specified.

See Imaging an EFI/UEFI System with Windows 8 and Symantec Ghost from USB for more info.

Reply | Quote

“and cloning will overwrite the entire drive”
That’s the point of cloning. You want an exact bit for bit copy of the original. Buying a drive for this purpose with software already installed is pointless therefore, and you would never clone to a drive containing data you want to keep because, obviously, you would destroy that data. Its also a waste of money as bare OEM drives are cheaper, even after you’ve included the cost of the necessary USB adapter (cheaper than an enclosure) if its to be used externally. If you want to copy data to a drive with data on it that you want to retain, that’s what drag ‘n drop is for.:)

Reply | Quote

used the, clone drive method recently, cloning the whole contents of the partition of an old hhd to a brand new one, intended as a replacement. i was suprised how soon the process completed.

I installed the new hdd and booted unsuccessfully, but I soon got it sorted with the windows disc, which resolved the boot.ini file. its been great ever since. :0

Reply | Quote

The OP specified that both drives are external hard drives. Such drives normally have valuable utilities pre-installed. I think it best to install them from the drive to the computers from which the drive will be running while the drive is new, because there may be updates to apply on top of that, as I discovered with my new drive. The OP also specified that his new drive is faster, but we don’t know if it’s faster by RPM or by having USB3.

My ‘mature’ Seagates have Memeo as premium software, especially for sync and backup purposes, and the software is still available and updateable. My new WD has a slightly crippled version of Acronis as a premium, as well as other clever tricks. Most of the functionality of Acronis is retained, including Clone, but incremental images and the Acronis version of a bare metal install are excluded. There may be other bits and pieces missing, but it’s good for three computers, and it provides standard Acronis files.

As for the transfer, and assuming that it will not be a clone, we might consider sorting the data in transferring it. Transfer the largest files first, for example, or the oldest, or the newest, or the most important, or do it by file type (convenient for libraries). Scan for drive errors before copying.

Think of the file managers so many of us use, and forget when they might really be useful. Among the freebies, FreeCommander seems to be the favourite. The advantage of this or any other utility with a sync capability is that you can see whether you have missed anything, and you can sort the files. If you are new to file managers, you may have some learning to do before they become useful.

The biggest files may have the fastest transfer speed, and be the easiest to confirm that they have transferred successfully, and they can then be deleted on the old drive if confidence allows. They will presumably be backups, and they might take the lion’s share of the space on the drive. If you are doing this in stages, that makes a comfortable first stage.

If you assign a confidence level to the integrity of files, you might consider the older files to be more likely to be corrupt, but if you assign a dollar value to the files, you might want to grab the latest and greatest etc.

Reply | Quote

Where would I begin with a new drive?

Postpone copying anything from the old drive, and perform a number of operations that are confined to the computer or computers you intend to use it with first. You want the computer married to the drive, and you would like everything in place, complete with some understanding of what you have, before you start messing with transfers.

Install the bundled software that comes with the drive on the computer – mine required updates right from the start – and try to figure out the basics. Incidentally, both of the major manufacturers call their device ‘My Book’, which is designed to drive users with both brands crazy, and certain drives have better bundles, which may be usable with lesser drives by the same manufacturer.

Perform a normal backup of your computer to the new drive. The new drive is faster, and the faster speed will show itself in the backup time. Time it and see. You may prefer to back up the computer both before and after installing the software on the computer.

This backup is newer than any on the old drive, and backups on the old drive can be deleted once you are confident the new setup works. You don’t have to transfer them – they’re obsolete. They’re also insurance, so don’t delete them prematurely. Do this for every computer that is backed up to the old drive.

Reply | Quote

dogberry — Thanks for FreeCommander recommendation! I just now installed the newest version; I’m going to be using FCxe a lot — in place of explorer.exe 🙂

"Take care of thy backups and thy restores shall take care of thee." Ben Franklin, revisted

Reply | Quote