CrashPlan Memory Utilization

I’ve been using CrashPlan as an online backup solution for quite some time, and it works really well.

I like the fact that I can subscribe to the consumer plan, with almost 3.5TB of data backed up, and that the backup client installs on a server OS. Many of the other “unlimited” backup providers I tested have restrictions in place that makes such a setup impossible.

CrashPlan sends email notifications about backup status, and I noticed that something was wrong with the backup:
CrashPlan.Email

I logged onto the machine, opened the main UI, and after a few seconds the UI just closed. opened it again, same thing, after about 15s the UI closed.

My initial thoughts were that it is a crash, but on attaching a debugger, the exit call stack showed that the process was cleanly terminated after receiving a signal.

On looking at the NT eventlog I could see that the service was restarting about every 15s:

The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.
The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.
The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.

The service wasn’t crashing, it was externally being stopped and restarted. I looked in the CrashPlan directory, and I found several log files with a naming like restart_1342296082496.log. The contents of these files looked like this:

Sat 07/14/2012 13:01:22.53 : "C:\Program Files\CrashPlan\bin\restart.bat"
ECHO is off.
Sat 07/14/2012 13:01:22.53 : APP_BASE_NAME=CrashPlan
Sat 07/14/2012 13:01:22.53 : APP_DIR=C:\Program Files\CrashPlan
ECHO is off.
Sat 07/14/2012 13:01:22.53 : Stopping CrashPlanService
The CrashPlan Backup Service service is stopping.
The CrashPlan Backup Service service was stopped successfully.

Sat 07/14/2012 13:01:25.05 : Sleeing 15 seconds...

Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

Ping statistics for 127.0.0.1:
Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Sat 07/14/2012 13:01:39.08 : Starting CrashPlanService

The CrashPlan Backup Service service was started successfully.

ECHO is off.
Sat 07/14/2012 13:01:39.13 : Exiting...

I looked for a newer version, but 3.2.1 was the latest version. I logged a support ticket with CrashPlan, but I continued my investigation. I found a log file service.log.0, several MB in size, and inside it I found this:

[07.14.12 12:32:39.480 ERROR   QPub-BackupMgr       backup42.service.backup.BackupController] OutOfMemoryError occurred...RESTARTING! message=OutOfMemoryError in BackupQueue!

So it seems that the service is running out of memory. I now had a few good keywords to search on, and I found this post of a user with the same problem. At about the same time I received a reply from CrashPlan support, not bad for weekend service, with the same solution.

The CrashPlan backup service and desktop applications are Java apps, and as such the maximum amount of memory they use are capped by configuration. I have had similar problems with other memory hungry Java apps, like Jaikoz, that simply fail unless you increase the memory limit.

To fix the problem, shutdown the service, open the CrashPlanService.ini file in the program directory, and increase the maximum memory utilization parameter to 2GB, the default is 512MB, and restart the service:

Virtual Machine Parameters=-Xrs -Xms15M –Xmx2048M

After upping the memory all seemed well, and the service has been running for more than a day. But, I wanted to know just how much memory is CrashPlan using, and it turns out to be insane.

Here are the current stats for the amount of data I backup, as well as the resource utilization by the backup service and desktop app:

CrashPlan.Size
CrashPlan.Memory.Desktop
CrashPlan.Memory.Service

As you can see, the desktop app’s peak private bytes exceed 250MB, and the service exceeds 1.3GB, that’s right 1.3GB of memory!

Those numbers are simply outrageous.

Unlimited online backup providers becoming extinct

I just received an email from ElephantDrive informing me that my legacy unlimited storage account will be terminated in 30 days, and that I must select a new plan.

In July 2009 ElephantDrive announced that they are no longer offering their $100 per year unlimited storage plan. ElephantDrive is now offering a $200 per year for 500GB plan.
In February 2011 Mozy announced that they are no longer offering their $55 per year unlimited storage plan. Mozy is now offering a $120 per year for 125GB plan.
In February 2011 Trend Micro SafeSync announced that they are bandwidth throttling large accounts. In March 2011 they announced that they are no longer offering their $35 per year unlimited storage plan. SafeSync is now offering a $150 per year for 150GB plan.
Carbonite offers a $55 per year for unlimited storage plan, but they are bandwidth throttling accounts over 35GB to 512Kbps and accounts over 200GB to 100Kbps access speeds.
AVG LiveKive offers a $80 per year for unlimited storage plan, but the terms of service defines unlimited as 500GB.
BackBlaze offers a $60 per year for unlimited storage plan.
CrashPlan offers a $50 per year for unlimited storage plan.
Neither BackBlaze nor CrashPlan supports their unlimited plan on server class machines.

I currently have 2.1TB of data backed up online with ElephantDrive running on my Windows Server 2008 R2 machine. Needles to say, none of their new plans are affordable for that amount of storage. I either need to significantly trim down what I backup, or I need to find a new unlimited storage provider, that also allows installs on Windows Server.
For now, I’m uninstalling ElephantDrive.

[Update]
CrashPlan’s new v3 software installs and runs fine on Windows Server 2008 R2, and I have switched to using CrashPlan for my backup needs.

Here is an example snippet of the status emails I receive from CrashPlan:

Source → Target Selected Files Backed
Up %
Last
Connected
Last
Backup
VM-STORAGE → CrashPlan Central 2.1TB ↑1KB 423k 0 100.0% 2.5 hrs 4.3 hrs