Attix5 Pro FAQ
Search:     Advanced search

Backing up very large file counts - suggested configuration

Article ID: 95
Last updated: 18 Feb, 2013

Attix5 Pro can be configured to back up large file counts in a relatively short time period by changing the way that small files are handled. The ability to achieve these backup speeds is dependent on the environment, and is easier to achieve within a LAN speed environment where compression is not essential. 

Before changing any settings, it is recommended to first increase the service memory as detailed here: http://v6.attix5.com/kb/?View=entry&EntryID=146.

Tests were conducted on two servers with the following specifications:

  • SERVER1 - Server 2003 x86 - Physical Server - 4GB Ram - 4 Core Processor - Static Data
  • SERVER2 - Server 2008 x64 - VMware - 8GB Ram - 4 Core Processor - User Data

SERVER1 contained 6 267 660 files and SERVER2 contained 1 748 432 files. Before the settings were modified for large file counts, backups would take approximately 24 hours (and sometimes many more) to complete. After the settings were optimized for large file counts, the backup times dropped significantly, which can be seen in the table below.

Account

Backup Time

Version

Type

Prep. Time

Data Protected

Files Protected

Transferred

OS

 SERVER1

 30 Sep 2011 05:50

 6.1.0.4

 SE

 10 hours 50 min

 938.21 GB

 6 267 660

 2 407.13 MB

 Windows 2003

 SERVER2

 29 Sep 2011 21:58

 6.1.0.4

 SE

 2 hours 57 min

 991.01 GB

 1 748 432

 2 544.69 MB

 Windows Server 2008


Settings used:

The settings used to achieve this massive improvement can either be configured by profiling, or added into the a5backup.properties file for individual Backup Clients.

Setting

Profiling

a5backup.properties

Note

 Disk Usage  Minimize Disk Usage=True  backup.minimize.diskusage=True  True or false
 Background Transmission Queue Capacity  Background Transmission Queue Capacity=50  backup.txqueue.capacity=50  Number of files
 Backup Transmission Queue In-memory Limit  Backup Transmission Queue In-memory Limit=3072  backup.txqueue.inmemory.maxfilesize=3145728  File size in bytes in the properties file and KB in profiling
 Small File Patching  Disable small file patching=102400  backup.patch.minfilesize=102400  File size in bytes in the properties file and profiling
 Small File Compression  Disable small file compression=102400  backup.compression.minfilesize=102400  File size in bytes in the properties file and profiling
 Retry Attempts  Retry Attempts=50  backup.max_retry=50  Number of connection retries
 Retry Sleep  Retry Sleep=360  connection.retries.delay=360000  Sleep period in milliseconds in the properties file and seconds in profiling 
 Connection Timeout  Connection Timeout=999  connection.timeout=999000  Timeout in milliseconds in the properties file and seconds in profiling

Note: Backup Transmission Queue In-memory Limit, Background Transmission Queue Capacity and Disk Usage have been deprecated from the V7 SP Console. This functionality can still be used on V6 Backup Clients backing up to V7 Storage Platforms by configuring the options on the Backup Client side. V7 Backup Clients feature Streaming Backup functionality which effectively replaces the background transmission queue. Please enable Streaming Backup to use this functionality.

Screenshot of profiling:

 

Further Suggestions:

You may also want to investigate the NTFS change journal as using this decreases the amount of time taken to build the backup selection list (after the first backup, and if the selection has not changed). Details of setting this can be found here: http://v6.attix5.com/kb/?View=entry&EntryID=24.

Article ID: 95
Last updated: 18 Feb, 2013
Revision: 3
Views: 4818
Comments: 0
Posted: 08 Feb, 2012 by Flood A.
Updated: 18 Feb, 2013 by -- .
This article was:  
Prev   Next
Article 92 - Storage Platform Console still showing the old version after an...     Article 150 - Backing up the ReportServer database to the StorageServer and...