I’ve been using CrashPlan as an online backup solution for quite some time, and it works really well.
I like the fact that I can subscribe to the consumer plan, with almost 3.5TB of data backed up, and that the backup client installs on a server OS. Many of the other “unlimited” backup providers I tested have restrictions in place that makes such a setup impossible.
CrashPlan sends email notifications about backup status, and I noticed that something was wrong with the backup:
I logged onto the machine, opened the main UI, and after a few seconds the UI just closed. opened it again, same thing, after about 15s the UI closed.
My initial thoughts were that it is a crash, but on attaching a debugger, the exit call stack showed that the process was cleanly terminated after receiving a signal.
On looking at the NT eventlog I could see that the service was restarting about every 15s:
The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.
The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.
The CrashPlan Backup Service service entered the stopped state.
The CrashPlan Backup Service service entered the running state.
The service wasn’t crashing, it was externally being stopped and restarted. I looked in the CrashPlan directory, and I found several log files with a naming like restart_1342296082496.log. The contents of these files looked like this:
Sat 07/14/2012 13:01:22.53 : "C:\Program Files\CrashPlan\bin\restart.bat"
ECHO is off.
Sat 07/14/2012 13:01:22.53 : APP_BASE_NAME=CrashPlan
Sat 07/14/2012 13:01:22.53 : APP_DIR=C:\Program Files\CrashPlan
ECHO is off.
Sat 07/14/2012 13:01:22.53 : Stopping CrashPlanService
The CrashPlan Backup Service service is stopping.
The CrashPlan Backup Service service was stopped successfully.
Sat 07/14/2012 13:01:25.05 : Sleeing 15 seconds...
Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Ping statistics for 127.0.0.1:
Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Sat 07/14/2012 13:01:39.08 : Starting CrashPlanService
The CrashPlan Backup Service service was started successfully.
ECHO is off.
Sat 07/14/2012 13:01:39.13 : Exiting...
I looked for a newer version, but 3.2.1 was the latest version. I logged a support ticket with CrashPlan, but I continued my investigation. I found a log file service.log.0, several MB in size, and inside it I found this:
[07.14.12 12:32:39.480 ERROR QPub-BackupMgr backup42.service.backup.BackupController] OutOfMemoryError occurred...RESTARTING! message=OutOfMemoryError in BackupQueue!
So it seems that the service is running out of memory. I now had a few good keywords to search on, and I found this post of a user with the same problem. At about the same time I received a reply from CrashPlan support, not bad for weekend service, with the same solution.
The CrashPlan backup service and desktop applications are Java apps, and as such the maximum amount of memory they use are capped by configuration. I have had similar problems with other memory hungry Java apps, like Jaikoz, that simply fail unless you increase the memory limit.
To fix the problem, shutdown the service, open the CrashPlanService.ini file in the program directory, and increase the maximum memory utilization parameter to 2GB, the default is 512MB, and restart the service:
Virtual Machine Parameters=-Xrs -Xms15M –Xmx2048M
After upping the memory all seemed well, and the service has been running for more than a day. But, I wanted to know just how much memory is CrashPlan using, and it turns out to be insane.
Here are the current stats for the amount of data I backup, as well as the resource utilization by the backup service and desktop app:
As you can see, the desktop app’s peak private bytes exceed 250MB, and the service exceeds 1.3GB, that’s right 1.3GB of memory!
Those numbers are simply outrageous.
Pieter Viljoen's Blog 
It is not surprising or that atypical for such backup solution software. They set up change notification events on every folder they monitor. This, in itself, is a big task. Then sync’ing all that data. Still, you’re right, that is quite the peak working set! Similarly problematic I’ve found to be WD’s backup software, which tries to ‘categorize’ all your files first, then watches for each and every change. Talk about some massive resource utilization, even with its own memory leak at one time (which I reported, then they fixed). Hopefully this company will work on their virtual memory utilization.
LikeLike
Jeremy, what makes it worse is that I disable realtime backup, and instead use scheduled backup. From the Java out of memory stack trace it looks like the app is inserting all items it finds in some kind of hashed container, and that this is where it ran out of memory. If this is truly how they built it, it is totally not scalable as memory utilization is a function of total files to back up.
LikeLike
Indeed, that does make it worse. SImilar with WD Smartware. You couldn’t disable its ‘classification’ and monitoring of the entire FS – even if backups were COMPLETELY disabled. Fortunately, it was isolated into a single process (WDFME.EXE=WD File Monitor Engine?) which made a good disallowed process for Process Lasso 😉
LikeLike
Pretty outrageous resource utilization – looks like a poorly written client.
LikeLike
Thanks for the info. Came in handy!
LikeLike
3.4TB of data? 1.3GB of RAM? not really using that much RAM for that amount of Data.. if you have 3.4TB of data and don’t have 1.3GB of RAM to spare then its the user who should be blamed for an imbalanced implementation and management of data..
LikeLike
There are definitely shortcomings in their client, but if you come across another service that provides user-controlled encryption (448 bit blowfish), free offsite backups (to friends running crashplan), and cross-platform compatibility (windows, mac, linux) then please be sure to share.
LikeLike
1.3 GB backed up here and the service takes about 1 GB of memory. Sort of annoying, but like Harvey Knight says, it’s a good deal.
I’ve been in contact with their customer support and it seems like there are better native clients on the way, i.e. not Java. At least for Windows, if I understood correctly. So I’ll continue to use it and hopefully one day soon the memory usage will drop 🙂
LikeLike
Peter, I’m very grateful for this post. I don’t think I would have figured this out. I’m now backing up my important VM. It’s not a lot of data, but it’s still very important. Thank you.
LikeLike
And I’m sorry about misspelling your name, Pieter!
LikeLike
I have a backup set of 13 partitions, totalling at 1.5M files and 7.7TB. The CrashPlanService.exe reaches over 4 GB of memory use easily. I’m still in my initial backup phase, and I’ve heard that CPU and memory consumption go down after the initial upload has been completed – I’m hopeful.
I really hope the other commenter’s report of planned native clients is true, although it’s been over two years ago and nothing has happened yet. It’s time everybody realises that, except for the most simple applications light on resource requirements, Java is never a sensible choice (especially in enterprise applications). The truth is that, if it wasn’t for CrashPlan, I would’ve long banished from my system the atrocity that is the Java Runtime Environment.
LikeLike