I came across a need to replicate a large amount of data between 2 relatively low bandwidth sites. Now I know if I was running Linux I could simply use rsync, but no such luck. I’m also aware that rsync exists for Windows but these were highly sensitive production boxes and I didn’t want to introduce any issues. Actually at the time I think you needed to install Cygwin first (or part there of) and then rsync.

After various tests I found that the data would benefit enormously from compression and the actual file changes daily were a small percentage of the entire file ste. With this in mind I wrote the following Windows batch script. Why batch? Well at the time I’d been heavily using vbscript and felt a need for a retro feel, but seriously the script does the job efficiently. So I needed a way to replicate only the changed data to the DR server and of course verify the transfer. The tools I ended up using were robocopy – for the incremental file transfers, 7za for compression, psexec because I needed to stop services during the the file transfer on the DR server, and blat to email the results. I also ended up using ftp for the file transfer as SMB copies were running way to slow. (That’s another story).

The steps are as follows:

1. Stop the Required Services on the remote DR Server
2. Replicate the data to the DR server
2.1. Changed and new files at the source are compressed.
2.2. The compressed files are sent to the destination.
2.3. The compressed files are extracted at the destination.
2.4. Extra files on the destination are removed.
2.5. Verification that the source and destination are identical.
3. Start the Required Services on the remote DR server

And it also maintains 7 copies of the data set of the DR server.

The end process is a file copy which would take 16 hours using standard methods is reduced to 2. I’m pretty happy with the results. Here’s the script in full, enjoy.

@echo off
SET VERSION=v1.0
SET AUTHOR=Matt Marschall, Red Mars Consulting
SET MODIFYDATE=29/11/2010
REM Repl_DR.bat
REM Repl_DR.bat controls the process for updating the remote DR server
REM with the file set on the production server. It replicates data from a source
REM to a destination. It is intended for use on WAN environments where
REM sending large amounts of data across the network benefits from compression.
REM The script follows the following process:
REM 1. Stop the Required Services on the remote DR Server
REM 2. Replicate the data to the DR server
REM  2.1. Changed and new files at the source are compressed.
REM  2.2. The compressed files are sent to the destination.
REM  2.3. The compressed files are extracted at the destination.
REM  2.4. Extra files on the destination are removed.
REM  2.5. Verification that the source and destination are identical.
REM 3. Start the Required Services on the remote DR server
REM 4. Email DR results log
REM Set the parameters of the replication
SET DRSERVER=\SERVERB
SET FTPSERVER=SERVERB
SET LOGFILE=DRlog.html
SET SOURCE=d:pathtofilesbackupdat
SET REMOTESHARE=%DRSERVER%d$pathtofiles
for %%a in ("%SOURCE%") DO set DESTINATION=%REMOTESHARE%%%~na
SET HISTORYLEVEL=6
echo ^<HEAD^> > %LOGFILE%
echo ^<style type="text/css"^> >> %LOGFILE%
echo div {color: red} >>%LOGFILE%
echo p {color: green} >>%LOGFILE%
echo ^</style^> >> %LOGFILE%
echo ^</HEAD^> >> %LOGFILE%
echo ^<H1^>Required DR Replication Log^</H1^> >> %LOGFILE%
echo %DATE% %TIME% Required DR process commencing^<BR^> >> %LOGFILE%
REM Check for required utils
set utilsdir=%cd%
set util=7za.exe
for /f %%a in ("%util%") do if exist %%~$PATH:a (set COMPRESS=%%~$PATH:a) else (if exist %utilsdir%%%a (set COMPRESS=%utilsdir%%%a) else (echo ^<div^> %DATE% %TIME% ERROR: The required utility %util% cannot be found <^/div^>>> %LOGFILE% & goto :error))
set util=robocopy.exe
for /f %%a in ("%util%") do if exist %%~$PATH:a (set ROBOCOPY=%%~$PATH:a) else (if exist %utilsdir%%%a (set ROBOCOPY=%utilsdir%%%a) else (echo ^<div^> %DATE% %TIME% ERROR: The required utility %util% cannot be found ^</div^>>> %LOGFILE% & goto :error))
set util=psexec.exe
for /f %%a in ("%util%") do if exist %%~$PATH:a (set PSEXEC=%%~$PATH:a) else (if exist %utilsdir%%%a (set PSEXEC=%utilsdir%%%a) else (echo ^<div^> %DATE% %TIME% ERROR: The required utility %util% cannot be found ^</div^>>> %LOGFILE% & goto :error))
echo ^<HR^> >>%LOGFILE%
echo Replication Settings: ^<BR^>>>%LOGFILE%
echo Source: %SOURCE% ^<BR^>>>%LOGFILE%
echo DR Server: %DRSERVER% ^<BR^>>>%LOGFILE%
echo Destination: %REMOTESHARE% ^<BR^>>>%LOGFILE%
echo Replication Destination: %DESTINATION% ^<BR^>>>%LOGFILE%
echo Replica History Level: %HISTORYLEVEL% ^<BR^>>>%LOGFILE%
echo ^<HR^> >>%LOGFILE%
echo Location of Dependancies: ^<BR^>>> %LOGFILE%
echo Robocopy: %ROBOCOPY% ^<BR^>>> %LOGFILE%
echo 7za: %COMPRESS% ^<BR^>>> %LOGFILE%
echo PsExec: %PSEXEC% ^<BR^>>> %LOGFILE%
echo ^<HR^> ^<BR^>>>%LOGFILE%
REM Stop the services on the DR server
echo %DATE% %TIME% Stopping Required services on %DRSERVER%^<BR^>>>%LOGFILE%
REM check access to DR Server Service Control
for  /F "tokens=1,2,3" %%a in ('sc %DRSERVER% query ') do if "%%c"=="FAILED" ( echo ^<div^> %DATE% %TIME% ERROR: Unable to connect the the service control manager on %DRSERVER% ^</div^> >>%LOGFILE% & goto :error)
set SERVICE=aservice
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE"  call :stopservice "%%b"
set SERVICE=bservice
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE"  call :stopservice "%%b"
REM replicate the data. Maintain today's and yesterday's data set
echo %DATE% %TIME% Commencing the replication ^<BR^>>>%LOGFILE%
echo %DATE% %TIME% Removing old replicate.7z from source^<BR^>>>%LOGFILE%
if exist "replicate.7z" del replicate.7z
echo %DATE% %TIME% Removing old replicate.7z from destination^<BR^>>>%LOGFILE%
if exist "%REMOTESHARE%replicate.7z" del "%REMOTESHARE%replicate.7z"
REM Test for changed files and add them to the compressed replication file
echo %DATE% %TIME% Creating new replicate.7z^<BR^>>>%LOGFILE%
echo ^<pre^>>>%LOGFILE%
for /f "tokens=1-3* delims= " %%i in ('"%ROBOCOPY%" %SOURCE% %DESTINATION% /MIR /njh /njs /np /ndl /fp /l') do call :zipit %%i "%%k"
echo ^</pre^>>>%LOGFILE%
REM copy zipped file to destination
if exist replicate.7z (echo %DATE% %TIME% FTP replicate.7z to %REMOTESHARE%^<BR^>^<pre^>>>%LOGFILE% & ftp -n -s:ftp.dat %FTPSERVER% 1>2>>%LOGFILE% & echo ^</pre^> >>%LOGFILE%) ELSE (echo %DATE% %TIME% No new files at source^<BR^>>>%LOGFILE%& goto verify)
echo %DATE% %TIME% Starting remote process to expand replicate.7z^<BR^>>>%LOGFILE%
echo ^<pre^> >>%LOGFILE%
"%PSEXEC%" %DRSERVER% 7za x -y %REMOTESHARE%replicate.7z -o%DESTINATION% 1>2>> %LOGFILE%
echo ^</pre^> >>%LOGFILE%
if NOT %ERRORLEVEL%==0 (echo ^<div^>%DATE% %TIME% ERROR: Remote expansion process on %DRSERVER% has failed!!!!! ^</div^>>>%LOGFILE% & goto :error)
:verify
echo %DATE% %TIME% Removing extra files at Destination^<BR^>>>%LOGFILE%
echo ^<pre^> >>%LOGFILE%
"%ROBOCOPY%" %SOURCE% %DESTINATION% /PURGE /NOCOPY /njs /njh /ndl /fp >>%LOGFILE%
echo ^</pre^> >>%LOGFILE%
echo %DATE% %TIME% Fixing Timestamps, ownership, security^<BR^>>>%LOGFILE%
echo ^<pre^> >>%LOGFILE%
"%ROBOCOPY%" %SOURCE% %DESTINATION% /MIR /njh /njs /np /ndl /fp /XC /XN /XL /COPY:ATSOU >>%LOGFILE%
echo ^</pre^> >>%LOGFILE%
echo %DATE% %TIME% Verifying replication^<BR^>>>%LOGFILE%
echo ^<pre^> >>%LOGFILE%
"%ROBOCOPY%" %SOURCE% %DESTINATION% /MIR /njh /njs /np /ndl /fp /l >>%LOGFILE%
echo ^</pre^> >>%LOGFILE%
if NOT %ERRORLEVEL%==0 echo ^<div^>%DATE% %TIME% ERROR: Source and Destination are out of sync!!!!! ^</div^>>>%LOGFILE% & goto error
echo ^<p^>%DATE% %TIME% Verified Replication has completed successfully ^</p^>>>%LOGFILE%
echo %DATE% %TIME% Maintaining History Replicas^<BR^>>>%LOGFILE%
echo %DATE% %TIME% Removing oldest replica based on selected History Level^<BR^>>>%LOGFILE%
if exist "%DESTINATION%-%HISTORYLEVEL%" (echo %DATE% %TIME% Attempting to remove %DESTINATION%-%HISTORYLEVEL% ^<BR^>>>%LOGFILE% & rmdir "%DESTINATION%-%HISTORYLEVEL%" /S /Q)
if NOT %errorlevel%==0 (^<div^>%DATE% %TIME% Error encountered removing replica. History Replica update aborted!!!!^</div^> >>%LOGFILE% & goto error)
echo %DATE% %TIME% Aging existing replicas^<BR^> >>%LOGFILE%
set /a COUNT=%HISTORYLEVEL% - 1
set /a MOVE=%HISTORYLEVEL%
:loop
if "%COUNT%" LEQ "0" goto loopend
if exist "%DESTINATION%-%COUNT%" (echo %DATE% %TIME% Attempting to age replica %DESTINATION%-%COUNT% to %DESTINATION%-%MOVE% ^<BR^>>>%LOGFILE% & move /y "%DESTINATION%-%COUNT%" "%DESTINATION%-%MOVE%")
if NOT %errorlevel%==0 (echo ^<div^>%DATE% %TIME% Error encountered aging replica. History Replica update aborted!!!! ^</div^>>>%LOGFILE% & goto error)
set /a MOVE=%COUNT%
set /a COUNT=%COUNT% - 1
goto loop
:loopend
echo %DATE% %TIME% Creating replica of current file set^<BR^> >>%LOGFILE%
echo ^<pre^> >>%LOGFILE%
"%PSEXEC%" %DRSERVER% xcopy %DESTINATION% %DESTINATION%-1 /E /S /V /I /Q /O /X /K /Y 1>2>> %LOGFILE%
SET _ERR=%errorlevel%
echo ^</pre^> >>%LOGFILE%
if NOT "%_ERR%"=="0" echo ^<div^>%DATE% %TIME% Error encountered creating replica of current file set!!!! %_ERR%^</div^>>> %LOGFILE% & goto error
REM Start the services
echo Starting Required services on %DRSERVER%^<BR^>>>%LOGFILE%
set SERVICE=aservice
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE"  call :startservice "%%b"
set SERVICE=bservice
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE"  call :startservice "%%b"
echo ^<p^>%DATE% %TIME% Replication has completed successfully ^</p^>^<BR^>>>%LOGFILE%
SET SUBJECT="Required DR Replication SUCCESSFUL"
goto :email
:zipit
echo %1 %2 %3 >>%LOGFILE%
if "%1"=="New" ("%COMPRESS%" a replicate.7z %3 1>2>> nul
) ELSE (if "%1"=="Newer" ("%COMPRESS%" a replicate.7z %2 1>2>> nul) ELSE (if "%1"=="Changed" ("%COMPRESS%" a replicate.7z %2 1>2>> nul)))
goto :EOF
:startservice
if %1=="4  RUNNING " echo ^<p^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% is already started. Hope this is OK^</p^> >>%LOGFILE% & goto :EOF
echo %DATE% %TIME% Attempting to start Service: %SERVICE% on %DRSERVER% ^<BR^>>>%LOGFILE%
echo ^<pre^>>>%LOGFILE%
sc %DRSERVER% start %SERVICE% >>%LOGFILE%
REM The following ping will force the script to wait 30secs before checking service state
ping -n 30 localhost >NUL
echo ^</pre^>>>%LOGFILE%
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE" call :checkstarted "%%b"
echo ^<p^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% is started.^</p^>^<BR^> >>%LOGFILE%
goto :EOF
:stopservice
if %1=="1  STOPPED " echo ^<p^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% is already stopped. Hope this is OK.^</p^> >>%LOGFILE% & goto :EOF
echo %DATE% %TIME% Attempting to stop Service: %SERVICE% on %DRSERVER% ^<BR^>>>%LOGFILE%
echo ^<pre^>>>%LOGFILE%
sc %DRSERVER% stop %SERVICE% >> %LOGFILE%
REM The following ping will force the script to wait 30secs before checking service state
ping -n 30 localhost >NUL
echo ^</pre^>>>%LOGFILE%
for  /F "tokens=1* delims=: " %%a in ('sc %DRSERVER% query %SERVICE%') do if "%%a"=="STATE" call :checkstopped "%%b"
echo ^<p^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% is stopped.^</p^>^<BR^> >>%LOGFILE%
goto :EOF
:checkstarted
if %1=="2  START_PENDING " for /F "tokens=1* delims=: " %%a in ('sc query %1') do if "%%a"=="STATE" call :checkstarted "%%b" & GOTO :EOF
if %1=="4  RUNNING " goto :EOF
echo ^<div^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% has failed to start cleanly.^</div^> >> %LOGFILE%
echo ^<div^>%DATE% %TIME% ERROR MESSAGE: %1 ^</div^>>>%LOGFILE%
goto :error
:checkstopped
if %1=="3  STOP_PENDING " for  /F "tokens=1* delims=: " %%a in ('sc query %1') do if "%%a"=="STATE" call :checkstopped "%%b" & GOTO :EOF
if %1=="1  STOPPED " goto :EOF
echo ^<div^>%DATE% %TIME% Service: %SERVICE% on %DRSERVER% has failed to stop cleanly.^</div^> >> %LOGFILE%
echo ^<div^>%DATE% %TIME% ERROR MESSAGE: %1 ^</div^>>> %LOGFILE%
goto :error
:error
set SUBJECT="Required DR Replication FAILED"
echo ^<div^>%DATE% %TIME% A non recoverable error has occured. Replication has failed!!!!!^</div^> >>%LOGFILE%
goto :email
:email
REM Email the Log File
blat %LOGFILE% -to admin@a.domain.com -subject %SUBJECT% -server smtp.a.domain.com -f DRMAILER@a.domain.com -html
exit