CrashPlan throws in the towel … for home users

Today CrashPlan, my current online backup provider, announced on Facebook of all places, that they threw in the towel, and will no longer provide service to home users. The backlash was heated, and I found the CEO’s video message on the blog post rather condescending.

I’ve been a long time user of online backup providers, and many have thrown in the towel, especially when free file sync from Google and Microsoft offers ever expanding capabilities and more and more free storage. Eventually even the cheapest backup storage implementation becomes expensive, when compared to a cloud provider, and not profitable as a primary business.

I’ve been using CrashPlan’s unlimited home plan for quite some time now, they were one of a few, today none, that were reasonably priced, allowed unlimited storage, and supported server class OS’s. But, I could sense the writing was on the wall; they split the home and business Facebook account, they split the website, the home support site has not seen activity in ages, they made major improvements to the enterprise backup agent, switching to a much leaner and faster C++ agent, while the home agent remained the old Java app with its many shortcomings, and there were some vague rumors on the street of a home business selloff attempt.

The transition offered a free switch to the small business plan, for the remaining duration of the home subscription, plus 3 months, and then a 75% discount on next year’s plan. For my account, this means free CrashPlan Pro until 12 June 2018, then $2.50 per month until 12 June 2019, and then $10.00 per month.

I’ve switched to the Pro plan, as they promised the agent updated itself, going from the old Java to the new C++ agent, the already backed up data was retained without needing to backup again, and all seems well, for now…

Razer BSOD When Driver Verifier is Enabled

I am done with Razer, exciting promises for technology on paper, great looking hardware, terrible support, terrible software.

Not too long ago I complained about Razer’s poor UX and Support, this time it is a BSOD in one of their drivers, and forever crashing Razer Stargazer camera software.

I’ve been looking for a Windows Hello capable webcam, and the Razer Stargazer, based on Intel RealSense technology, looked promising. The device is all metal and tactical looking, but the software experience is so buggy, install this, install that, then crash after crash after crash. I ended up returning it for a refund, and got a Logitech BRIO instead, the BRIO is cheaper, and works great.

A couple days ago I was greeted with a BSOD on one of my test machines, a crash in the RZUDD.SYS “Razer Rzudd Engine” driver, part of the Razer Synapse software. What makes this interesting, is that the issue seems to be triggered by having Driver Verifier enabled.

20170416_201259779_iOS

One may be tempted to say do not enable Driver Verifier, but, the point of driver verifier is to help detect bugs in drivers, and is a basic requirement for driver certification. Per the WinDbg analysis, this appears to be a memory corruption bug. After some searching, I found that the Driver Verifier BSOD has been reported by other users, with no acknowledgement, and no fix forthcoming. I contacted Razer support, and not surprisingly, they suggested uninstall and reinstall. I tried the community forums, and I was just pointed back to support.

FAULTING_IP:
rzudd+28c80
...
DEFAULT_BUCKET_ID:  CODE_CORRUPTION
...
PROCESS_NAME:  RzSynapse.exe
...
STACK_TEXT:
nt!KeBugCheckEx
nt!MiSystemFault+0x12e69c
nt!MmAccessFault+0xae6
nt!KiPageFault+0x132
rzudd+0x28c80
rzudd+0x218d4
rzudd+0x7a9f
Wdf01000!FxIoQueue::DispatchRequestToDriver+0x1bf [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3325]
Wdf01000!FxIoQueue::DispatchEvents+0x3bf [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3125]
Wdf01000!FxPkgIo::DispatchStep1+0x53e [minkernel\wdf\framework\shared\irphandlers\io\fxpkgio.cpp @ 324]
Wdf01000!FxDevice::DispatchWithLock+0x5a5 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1430]
nt!IovCallDriver+0x245
...
FAILURE_BUCKET_ID:  MEMORY_CORRUPTION_LARGE

I am done with Razer, exciting promises for technology on paper, great looking hardware, terrible support, terrible software.

Razer Shoddy Support and Bad Software UX

This post is just me venting my frustration at Razer’s poor software user experience, and their shoddy support practices. I’m writing this after I just had to go and find a working mouse, so I could click a button on a dialog that had no keyboard navigation support.

I’ve been using Razer keyboards and mice for some time, love them, their software not so much. I had to replace an aging ThinkPad, and the newly released Razer Blade Stealth looked like a great candidate, small and fast, reasonably priced, should be perfect, well, not so much.

I keep my monitors color calibrated, and I cringe whenever I see side-by-side monitors that clearly don’t match, or when somebody creates graphic content (yes you graphic artists using MacBooks to create content for PC software without proper color profiles) that looks like shades of vomit on a projector or a cheap screen, but I digress. My monitor of choice is NEC and their native SpectraView color calibration software. Unfortunately, the Blade with its lower end Intel graphics processor, and HDMI port, does not support DDC/CI, so no ability to color calibrate my monitor. My main monitor is a NEC MultiSync EA275UHD 4K monitor, and the internal Intel graphics processor is frustratingly slow on this high resolution display. And, the HDMI connectivity would drop out whenever the monitor went into power saving mode. Why not use a more standard mini-DisplayPort connector, would not solve the speed problem, but at least would have resolved the connection reliability and allowed for proper color calibration.

To solve the problem, I decided to get a Razer Core with an EVGA GeForce GTX 1070 graphics adapter. The Core is an external USB and network dock, with a PSU and PCIe connector for a graphics card, all connected to the notebook by Thunderbolt 3 over a, too short, USB-C cable. I connected my monitor to the GTX 1070 DisplayPort connector, connectivity was fine, I could color calibrate my monitor, and the display performance with the GTX 1070 was fast, great. By the way, JayzTwoCents has a great video on the performance of external graphic cards.

But, my USB devices connected to the dock kept on dropping out. I found several threads on the Razer support forum complaining about the same USB problems, and the threads are promptly closed with a contact support message. I contacted Razer support and they told me they are working on the problem, and closed my ticket. I contacted them again stating that closing my ticket did not resolve the problem, and they said my choice is RMA the device, with no known solution, or wait, and then they closed my ticket again. To this day this issue has not been resolved, and I have to connect my USB devices directly the notebook, defeating the purpose of a dock. They did publish a FAQ advising users to not use 2.4GHz WiFi, but to stick with 5GHz due to interference issues, so much for their hardware testing.

Now, let’s talk about their Razer Synapse software, the real topic of this post. The software is used to configure all the Razer devices, and sync the device preferences across computers with a cloud account, neat idea. The color scheme and custom drawn controls of this software matches their edgy “brand”, but their choice of thin grey font on a dark background fails in my usability book when used in a brightly lit office space.

Synapse.1

Whenever Windows 10 updates, the stupid Synapse software pops up while the install is still going, if you say yes, install now, then as expected the install fails due to Windows still installing. I logged the issue with Razer support, and they told me it is behaving as designed, really, designed to fail.

Synapse.2

So, today the Synapse software, again, prompts me to update, a frequent occurrence, and my mouse dies during the update, presumably because they updated the mouse driver, but this time I am prompted with a reboot required dialog. Dead mouse, no problem, have keyboard, tab over, wait, no keyboard navigation on the stupid owner drawn custom control dialog, no way to interact with the dialog without a mouse, just fail.

Synapse.3

Moral of the story, UX is important people, and I should just stick with ThinkPad or Microsoft Surface Book hardware, costs more, but never disappoints.

Circumventing ThinkPad’s WiFi Card Whitelisting

What started as a simple Mini PCI Express WiFi card swap on a ThinkPad T61 notebook, turned into deploying a custom BIOS in order to get the card to work.

I love ThinkPad notebooks, they are workhorses that keep on going and going. I always keep my older models around for testing, and one of my old T61’s had an Intel 4965AGN card, that worked fine with Windows 10, until the release of the Anniversary / Redstone 1 update. After the RS1 update, WiFi would either fail to connect, or randomly drop out. The 4965AGN card is not supported by Intel on Win10, and the internet is full of problem reports of Win10 and 4965AGN cards.

Ok, no problem, I’ll just get a cheap, reasonably new, with support for Win10, Mini PCIe WiFi card, and swap the card. I got an Intel 3160 dual band 802.11AC card and mounting bracket for about $20. The 3160 is a circa 2013 card with Win10 support. I installed the card, booted, and got a BIOS error 1802: Unauthorized network card is plugged in.

This lead me to the discovery of ThinkPad hardware whitelisting, where the BIOS only allows specific cards to be used, which lead me to Middleton’s BIOS, a custom T61 BIOS, that removes the hardware whitelisting, and enables SATA-2 support. I found working download links to the v2.29-1.08 Middleton BIOS here.

The BIOS update is packaged as a Win7 x86 executable or DOS bootable ISO image. As I’m running Win10 x64, and I could not find any CD-R discs around, I used Rufus to create a bootable DOS USB key, and I extracted the ISO contents using 7-Zip to a directory on the USB key. The ISO is created using a bootable 1.44MB DOS floppy image, and AUTOEXEC.BAT launches “FLASH2.EXE /U”, I created a batch file that does the same.

I removed the WiFi card, booted from USB, ran the flash, and got an error 1, complaining that flashing over the LAN is disabled. Ok, I enabled flashing the BIOS over the LAN in the BIOS, and rebooted.

I ran the update again, and this time I got error 99, complaining that BitLocker is enabled, and to temporarily disable BitLocker. I did not have BitLocker enabled, so I removed the hard drive and tried again, same error. Must be something in the BIOS, I disabled the security chip in the BIOS, tried again, and the update starts, but a minute or so later the screen goes crazy with INVALID OPCODE messages.

Hmm, maybe the updater does not like the FreeDOS boot image used by Rufus. Ok, let me create a MS-DOS USB key, uhh, on Win10, that turned out to be near impossible. Win10 does not include MS-DOS files, Rufus does not support custom locations for MS-DOS files, nor does it support getting them from floppy or CD images (readily available for download), the HP USB Disk utility complains my USB drive is locked, and writing raw images to USB result in a FAT12 disk structure that is too small to use. I say near impossible because I gave up, and instead went looking for an existing MS-DOS USB key I had made a long time ago. I am sure with a bit more persistence I could have found a way to create MS-DOS bootable USB keys on Win10, but that is an exercise of another day.

Trying again with a MS-DOS USB key, and voilà, BIOS flashed, and WiFi working.

I am annoyed that I had to go to this much trouble to get the new WiFi card working, but the best part of the exercise turns out to be the SATA-2 speed increase. This machine had a SSD drive, that I always found to be slow, but with the SATA-2 speed bump in Middleton’s BIOS, the machine is noticeably snappier.

A couple hours later, my curiosity got the better of me, and I made my own version of Rufus that will allow formatting of MS-DOS USB drives on Win10. In the process I engaged in an interesting discussion with the author of Rufus. I say interesting, but it was rather frustrating, Microsoft removed the MS-DOS files from Win10, and Rufus refuses to add support for sourcing of MS-DOS files from a user specified location, citing legal reasons, and my reluctance to first report the issue to FreeDOS. Anyway, can code, have compiler, if have time, will solve problem.

UPS Battery Replacement Turns Into Unrecoverable Firmware Update

Two lessons learned; do not trust scheduled battery tests, and leave working firmware be!

As the saying goes, if it is not broken do not fix it, especially when it comes to firmware.

I have a couple APC Smart-UPS‘s at my house, same as the models I like to use at the office. I use the SMT750 models with AP9631 Network Monitoring Cards. The problem started when we had a short power outage, and the UPS that powers the home network switch, cell repeater, alarm internet connection, and PoE IP cameras, unexpectedly died. A battery replacement led to the opportunity to do a UPS firmware update, which led to an unrecoverable firmware update.

It started when I woke up one morning and it was obvious the power had been out, first indicator is the kitchen appliances have blinking clocks, second are the numerous power failure email notifications, and the emails that stood out were from the alarm system that says it lost power and internet connectivity. The alarm has it’s own backup battery, the network switches and FiOS internet have their own battery backups, and the outage was only about 4 minutes.  So how is it that the UPS died, killing the switch, disconnecting the internet, especially when the outage was only 4 minutes, and typical runtimes on the UPS should be about an hour?

Here is the UPS outage log produced by the NMC card:

10/11/2016 06:53:05 Device UPS: A discharged battery condition no longer exists. 0x0108
10/11/2016 06:12:27 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 06:12:24 Device UPS: Restored the local network management interface-to-UPS communication. 0x0101
10/11/2016 06:12:12 System Network service started. IPv6 address FE80::2C0:B7FF:FE98:9BAF assigned by link-local autoconfiguration. 0x0007
10/11/2016 06:12:10 Device Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication. 0x0344
10/11/2016 06:12:09 System Network service started. System IP is 192.168.1.11 from manually configured settings. 0x0007
10/11/2016 06:12:02 System Network Interface coldstarted. 0x0001
10/11/2016 05:45:36 Device UPS: A low battery condition no longer exists. 0x0110
10/11/2016 05:45:36 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 05:45:35 Device UPS: The output power is turned off. 0x0114
10/11/2016 05:45:35 Device UPS: The graceful shutdown period has ended. 0x014F
10/11/2016 05:45:35 Device UPS: No longer on battery power. 0x010A
10/11/2016 05:45:35 Device UPS: Main outlet group, UPS Outlets, has been commanded to shutdown with on delay. 0x0174
10/11/2016 05:45:35 Device UPS: The power for the main outlet group, UPS Outlets, is now turned off. 0x0135
10/11/2016 05:45:18 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
10/11/2016 05:41:36 Device UPS: On battery power in response to rapid change of input. 0x0109

I could see from the log that the UPS battery power ran out within 4 minutes, 05:41:36 on battery, 05:45:18 battery too low, 05:45:35 output turned off. The UPS status page was equally puzzling, load was at 9.7%, yet reported runtime was only 5 minutes, impossible.

Here is the status screenshot:

apc-1

I ran a manual battery test, the test passed, but from the log it was clear the battery failed. I have bi-weekly scheduled battery tests for all UPS’s, never received a failure report. So what is the point of a battery test if the test comes back no problem yet it is clear to me from the logs that the battery failed?

Here is the log:

10/11/2016 18:07:06 Device UPS: A low battery condition no longer exists. 0x0110
10/11/2016 18:07:06 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 18:07:05 Device UPS: Self-Test passed. 0x0105
10/11/2016 18:06:59 Device UPS: A discharged battery condition no longer exists. 0x0108
10/11/2016 18:06:59 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
10/11/2016 18:06:58 Device UPS: Self-Test started by management device. 0x0137
10/11/2016 18:06:56 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107

I asked for advice on the APC forum, no reply yet, and I ordered a replacement RBC48 battery. I received the battery, installed it, and the reported runtime is back to normal, 1 hour 48 minutes.

Here is a status screenshot with the new battery:

apc-2

Here is where I should have stopped and called it a day, but no. I knew that the UPS’s were on old firmware, and I decided to use this opportunity to update the firmware. I’d normally let firmware be, unless I have a good reason to update, but I convinced myself that the new firmware readme had some fixes that may help with the false pass on the battery test:

Release Notes (UPS09.3):
========================
2. Improved self-test logging for PCBE / NMC.
11. Repaired an occasional math error in the battery replacement date algorithm that resulted in incorrect dates.

I update the UPS, where I just replaced the battery, instructions are pretty simple. Only hassle is I have to bypass the network equipment to be mains powered so I can turn the UPS outputs off while updating the firmware, while maintaining network connectivity.

I did the same for my office UPS, PC and office switch on mains power, and when I power down the output, I made sure to not notify PowerChute Network Shutdown (PCNS) clients, as my PC had the PowerChute client installed to receive power state via the network. I start the firmware update over the network, and a few seconds later I get a Windows message that shutdown had been initiated by PCNS, what? I sit there in frustration, nothing to do but watch my PC shutdown while it is still delivering the firmware update.

On rebooting my PC, NMC comes up, but reports the UPS has stopped communicating. I pull AC power from the UPS, no change, I also pull the batteries, and when I plug the batteries and mains back on, beeeeeeeeeep. NMC now reports no UPS found, the UPS LCD panel reports all is fine. And still beeeeeeeeeep, and no way to stop the beeeeeeeeeep.

Here is the NMC status page:

apc-3

I try to do Firmware Upgrade Wizard update via USB, plug a USB cable in, PC sees UPS, reports critical condition, but the upgrade wizard reports no UPS found on USB.

Here is the wizard error page:

apc-4

So, here I am, stuck with a bricked UPS, lesson learned, actually two lessons learned; do not trust scheduled battery tests, and leave working firmware be!

Amazon Associate’s Account Closed

Amazon just notified me in email that my Associate’s account was closed due to not being in compliance with their operating agreement:

“You are not in compliance with Participation Requirement Number 29 because purchases resulting from Special Links on your site have been used for resale or commercial use.”

I have no idea how or why this happened.

 

A couple of years ago I moved my blog from the free Blogger platform to a paid WordPress.com hosted site. About the same time I signed up for an Amazon Associate’s account, profiting from any Amazon links resulting in sales, hoping that the proceeds would cover the costs of WordPress hosting and domain registration.

A quick calculation shows Amazon payouts of $669.57 between 2 August 2012 and 30 August 2016, that is about $167.39 per year, less the $99.00 for WordPress hosting, less $36.00 for Akismet blog spam filtering, less $19.00 for domain registration, leaves a profit of $13.39. Less $99.00 for bulk domain registration fees, not really fair to charge this fee to one domain, leaves a loss of $85.61 per year.

 

I do not know why I was suddenly out of compliance, I made no changes to either my Amazon Associates or WordPress accounts, and I’ve not posted any new content in a number of months. The WordPress stats show typical traffic (ignore the last two days), but the Amazon Associates report does show a marked increase in traffic:

 

I sent an email to Amazon support to clarify the violation, and to request my account be reinstated, but based on similar reports from other low traffic users,  I do not expect a resolution.

Instead, I opted-in to use WordPress’s own WordAds advertising platform, I still need to adjust the blog theme and settings to not interfere with reading, and I have no idea what the monetization would be, but at least I no longer have to bother with making special Amazon links.

Please comment and let me know if you find the ads to be intrusive, and I’ll consider funding the site without advertising assistance.

 

[Update: 1 September 2016]

A day after sending Amazon a request asking for an explanation, I received the following in email:

“This message is to advise you that your pievilsblo-20 account and your August 2016 Advertising Fees have been reinstated. Please accept our apologies for the closure.”

Looks like my account has been reinstated, no explanation of what happened.

Ecobee3 WiFi Connection Troubles

Ecobee3 thermostat WiFi troubles

In a previous post I wrote about my transition from Nest to Ecobee3 thermostats, and how the biggest benefit of the E3 was the use of remote sensors.

After several months of use, winter and summer, I find the remote sensors really do work very well, and our bedrooms remain at the desired temperature, while the areas around the thermostats can be warmer or colder.

But, the E3 is not perfect, there are two recurring problems; the remote sensors would report offline, and the units would lose network connectivity.

 

I’ve received sensor offline alerts a couple of times, typically happens early mornings, maybe interference, don’t know, the sensors never move from where they are placed.

Every time a I get a sensors offline report, the sensor already restored connectivity. This behavior is very annoying, I will get two emails a minute apart, and the E3 UI will have an alert saying sensor offline, I click ok, and then immediately an alert saying sensor online.

E3.Sensor.Email

I expect the E3 to have some sort of grace period before it deems a problem so important that it needs to notify me. As is, it is just an annoyance as there is no remediation action to take.

 

 

The second problem is the E3 loses network connectivity, this is a real problem, as the units remain offline until power cycled, and to power cycle the E3 has to be removed from the wall bracket, i.e. there is no reboot menu option.

I reported this problem to Ecobee support in October, and on the SmartHomeHub community forum. Yes, it is a bit pathetic that Ecobee does not have their own support community forum. Ecobee had me reserve static IP’s in the DHCP server, setup a dedicated 2.4GHz SSID, still disconnects. By December the problem was still happening, and Ecobee support escalated the problem to their development team, it is two months later, and still no updates from Ecobee support on the problem.

E3.Log

Through my own research and experimentation I suspect the problem to be that the E3 is unable to handle a WiFi channel change, and unable to roam between access points. There are some conditions that trigger the problem that I cannot explain.

I have multiple access points in my house, same SSID, different channels, 2.4GHz and 5GHz bands. I tested with Ubiquity UniFi AC and with Ruckus/Xclaim Xi-3. And in case you’re wondering, no other devices in my house have any problems with WiFi, even with dynamic channel selection.

I can make the E3 fail by either changing the AP channel, or by making it roam to a different AP, also changing channels. If I configure the AP’s to use auto channel selection, then the E3 will fail as soon as the AP chooses to change channels (UniFi does this on startup, Xclaim does this dynamically). If I manually change the AP channel, the E3 will fail. If I take one AP offline, the E3 will fail to roam to a different AP (on a different channel).

Even with my AP’s configured with static non-overlapping channels, the E3 would still sometimes fail, requiring a power cycle. I do not know why this would happen, as signal strength by the E3’s are perfect.

On the plus side, the E3 units remember the schedule, and even when offline, they continue to operate.

 

Bottom line is E3 WiFi is not reliable, and E3 support/dev is not responsive.

Ecobee3 Thermostat with Remote Sensors

We’ve had a particularly warm summer, for our very moderate area, and between my wife and my parent’s in-law, they were constantly changing the thermostat temperature, leaving nobody particularly happy, and our electricity consumption skyhigh. I needed a better solution, I found one, but it has some quirks.

I was an early adopter of the Nest Generation 1 thermostat, and when we moved to our new house, Nest was still the best available, and I installed two Nest Generation 2 thermostats, one upstairs, and one downstairs.

Nest used to be an innovator and leader in the home thermostat space, and then two things happened; they were acquired by Google, and competitors like Ecobee and Honeywell released very competitive products. The just released Nest Generation 3 thermostat has no new notable features, it is simply thinner, not unlike its competitors.

The one feature I, and many other users, asked for was remote temperature sensors. In many homes, like ours, where there are two HVAC units, one for upstairs and one for downstairs, with no room specific dampers or temperature control, the upstairs and downstairs air mixes and causes large temperature differentials between closed rooms and open spaces. Adding to that warm air rises and cold air falls, so in the summer the upstairs pumps cold air downstairs, and in the winter the downstairs pumps warm air upstairs, this again leaves bedrooms too hot or too cold.

Ecobee solved this problem, to a large degree, with the Ecobee3 thermostat, that comes with one remote sensing unit, and extra sensors can be purchased at $35 per sensor. The latest version of their thermostat is also Apple HomeKit compatible, allowing Siri to control the thermostat.

There are 3rd party integrations that can control the Nest temperature, like Wally at an additional $299, or SmartThings at an additional $139, but these are integration solutions, not integrated solutions, and makes the Nest solution much more expensive, especially considering the Ecobee3 (with one extra sensor included) and the Nest Gen3 are both $249.

There are alternate solutions like EcoVent that controls the individual vents per room, but that adds an additional $499 minimum for two rooms.

The Ecobee3 solution cannot control the temperature in each room, for that you really need a split AC unit per room, but it does allow the temperature sensing logic to take input from any number of rooms. And in my case, I am specifically interested in the temperatures in the bedrooms, not the general open areas.

Replacing the Nest with an Ecobee3 was easy, the Ecobee3 is slightly thinner than the Nest Gen2, and slightly larger, with that extra size used for a multi-color touch display.

Size Comparison

The remote sensors come with stands, wall screw mounts, and wall sticky tape mounts, they are pretty small and unobtrusive.

Sensor Size

The WiFi setup was really easy, and the first time I’ve seen this particular scheme in action, I believe it is called Wireless Accessory Configuration (WAC), not sure, Apple documentation is as always in short supply. Basically it worked like this; install the EB3 app on my iPhone, EB3 told me to connect my phone to the EB3 SSID, my phone asked me if I want to connect the device to WiFi, select yes, and the EB3 was automatically connected to my home WiFi, no passwords, no hassles, easy.

For each thermostat install the EB3 app asked me details about my house, address, size, construction, etc. This was annoying, one house, one set of details, multiple thermostats, why do I need to configure this for every thermostat. The Nest config was always very easy, one set of options per house, multiple thermostats. A call to EB support told me I need to create a group, then add the thermostats to the same group, then select what options I want to share between thermostats in the same group, and this can only be done from the web portal. This was a setup and first experience usability fail.

The second problem I ran into was the time configuration, the EB3 correctly selected Los Angeles as the time zone, and synced the time, but the displayed time was 3 hours ahead, East Coast time instead of Pacific Time. I fixed this by manually changing the timezone to something other than LA, saving, then changing it back to LA, and the time was correct. I did send EB support an email about this, and they replied that I need to make sure I have the correct time zone selected, right.

Time Zone

Pairing the remote sensors was really easy, stand in front of the EB3 and pull the plastic tab to let the sensor battery make contact, the EB3 detects the sensor, and lets you pair it, and enter a sensor name. The EB3 thermostat UI allowed me to use non-alpha characters, e.g. “Childs Room”, note the apostrophe, but when I later renamed the sensors, the mobile app and the web app restricted names to numbers and letters only, a slight inconsistency fail.

Sensor Naming

Configuring the schedule and comfort zones was easy and intuitive. I particularly like the concept of the comfort zones, and reusing them in the schedule, vs. the Nest’s more primitive setting of desired temperatures at times of days. This is where I configured the night comfort zones to use only the sensors in the bedrooms, and ignore the temperature at the main thermostat. Made a huge difference in comfort and AC runtime.

After a couple days of use I noticed something weird, the AC would turn on, but the thermostats would show the current temperature is still below the set temperature. This happened with both thermostats, and only in the afternoon. The schedule only called for 74F after 8pm during the night time comfort zone, this was around 7pm, when the set temperature was 78F.

Something is clearly wrong with the scheduler and the schedule. My bet was the scheduler is still using East Coast time (turned out I was wrong), Ecobee phone support was already closed, so I had to wait for the following day.

While I was looking at the System Monitor feature, very neat, I noticed gaps in the data. After a bit of research I found that Ecobee is having scaling problems, and their backend cannot handle the load, may be exacerbated by the Apple store kicking out Nest and now selling Ecobee3, or HomeKit integration, or poor planning. It was also weird that Ecobee does not run their own support forums, the Ecobee community supports themselves at the SmartHomeHub forums. One intrepid forum user created an availability graph based on his data gaps, clearly shows the recent problems.

I called Ecobee support, and they explained what was going on with the schedule; the thermostat has a feature called Smart Recovery Mode, in this mode the AC starts running before a schedule change in an attempt to reach the desired temperature when the schedule starts. And that this prediction takes a week or so to become more accurate, and that it can be impacted by fluctuations in the weather. Ok, makes sense, but usability fail by not making this behavior clear in the status UI.

Smart Recovery

As for the gaps in data, Ecobee support said they are busy migrating data, that this impacts backend performance, and that all systems should be operational in a week, and no data should be lost, slight contradiction to the post on the SmartHomeHub forum, but at least acknowledged.

I am pretty happy with the E3, the remote sensors really do make a big difference in efficiency.

 

[Update: WiFi troubles with the E3]

MR16 12V Halogen to LED retrofit

This post is about my research into finding suitable MR16 LED’s for replacing the 50W recessed halogen lights in our house. In summary, I’ve found “ok” bulbs, not great bulbs, and you can read about the details below.

Our house is about 3 years old, new construction, and one of the many decisions we made during planning was recessed halogen vs. recessed LED lighting. At the time my calculations showed the additional cost for LED lights would only be recovered in electricity cost savings after about 12 years, not worth the cost at the time. Another problem was the optical quality of the products, the near halogen optical quality LED products were ridiculously expensive, and the mainstream LED’s were of poor optical quality, and had poor dimmability.

Given the situation we opted for recessed Elco MR16 low voltage 12V AC magnetic transformer halogens, and planned on retrofitting them with LED’s as the technology improved and costs came down.

Now, 3 years later; our electricity cost is way higher than originally estimated, we installed solar that gave us a 50% reduction in cost, some of the recessed reflectors are showing signs of heat damage from the halogen bulbs, and 12V MR16 LED’s have entered the mainstream.

I’ve been looking for MR16 LED’s for some time now, same problem as 3 years ago, dimmable good optical quality bulbs are very expensive, ~$20 per, while eBay and Amazon sourced Chinese manufactured no-name brands are ~$4.

During my research I’ve made a few important observations:

  • US electrical code requires the use of GU10 bi-pin twist-lock lamp bases for new construction, and GU10 will eventually replace all E26 style screw in bases. In the past months I found that there is a much wider supply of 110V GU10 base MR16 dimmable LED bulbs compared to GU5.3 12V bulbs. This is especially true for the no-name brand Chinese suppliers on eBay. I am assuming that the electronic circuitry used is similar to that used in the widely available regular E26 / A21 110V dimmable LED bulbs, and that the only difference is the MR16 housing construction. Keeping in mind that most installed AC dimmers are forward phase, and support a large variety of load types, while an LED is a constant current device that typically uses pulse width modulation for dimming. Thus a line voltage forward phase dimmer to pulse width modulated LED driver circuit is non-trivial, adding a 12V AC transformer in the mix, and supporting both 12V AC and 12V DC loads further complicates the circuitry, especially when dimming is required.
  • Just like there is an expanding variety of dimmable GU10 line voltage MR16’s, there is an equal growing number of line voltage dimmable retrofit LED housings, that are near the cost of a MR16 bulb. These housings replace the old recessed can with an integrated AC LED driver and LED bulb array.
  • Today, as was the case 3 years ago, commercial and residential LED recessed lights separate the enclosure from the LED driver, allowing for different color and brightness LED’s to be used, and to optimize the electronic circuitry for the type of dimmer being used. The prices are much lower compared to 3 years ago, but still high comparing a $40 halogen enclosure to a similar style $150 LED enclosure. It is possible to replace the entire enclosure, but it is a big job requiring ripping out ceiling drywall.
  • 12V MR16’s can be powered by 12V AC electromagnetic transformers or electronic low voltage drivers. ELV drivers offer much higher efficiencies, but require compatible dimmers, and sometimes dimmers specifically designed for ELV drivers. Halogens are almost always powered by electromagnetic transformers due to the reduced cost and complexity. It is possible to replace the electromagnetic transformers in the enclosure with an ELV transformer, I’ve seen the electrician replace a blown transformer, he had to bring in the “small hand guy” from his crew and even then it took a lot of blind finger fiddling.
  • I use a Vantage Controls InFusion home automation lighting control system in my house. The system supports line-voltage forward phase and reverse phase dimmer modules, 0-10V control, and PWM control LED dimming. All loads in my installation are dimmed using forward phase dimmers. The recommended LED control setup is to use the 0-10V or PWM circuits, i.e. the dimming control and power lines are separate. The 0-10V / PWM control modules are about the same cost per line as AC dimmer modules, but the real cost is again in LED driver circuitry.

So what are my realistic choices:

  • The best quality option is to replace the halogen housings and line voltage control circuitry with native PWM control and native LED drivers. But, same as during construction, this is not a cost effective solution.
  • I can remove the transformers and convert the enclosure to line voltage, and use GU10 type MR16’s. But, tricky to remove the transformer, and the safety and legal state of the enclosure would be unknown after being converted to a line voltage receptacle.
  • I can remove the transformers and convert the enclosure to line voltage as above, but instead of using MR16 form factor bulbs, I can replace the insert with LED retrofit inserts.
  • I can replace the electromagnetic transformers with ELV’s to improve efficiency and dimmability. But again, a tricky job with marginal cost savings, and I still end up using 12V LED MR16’s.
  • I’ve opted to replace the halogen bulbs with LED’s on an as needed basis, i.e. when I need to replace a burnt trim, or blown bulb, I will replace the entire zone of lights with the same model of LED’s. My hope is that there will be ongoing improvements in product quality and performance, and ongoing reduction in costs as availability goes mainstream.

Here then is my review, more an exploration, greatly skewed by my subjective opinion vs. scientific fact, of the products I could find and test.

I initially tested the bulbs by replacing the halogens in my office, but this quickly became cumbersome, so instead I created a test bed for evaluation, trying to simulate the various dimmer and transformer types available.

I bought the the following items to match what I use in my house:

Here are some pictures of the Elco enclosure, this will give you an idea of how to go about swapping the transformer, and how tight a squeeze it is:

The DV-603P is a vanilly halogen and incandescent dimmer, it works just fine with the magnetic transformer and halogen bulbs in my house, but the MR16 LED manufacturer’s compatibility guide require the use of specific low voltage magnetic or electronic low voltage dimmers. So I also bought:

I considered a more elaborate test setup, but I don’t have access to the required equipment, and the measurements would be interesting from a scientific perspective, not so much a subjective perspective. So I opted for a simpler test setup, attached to a piece of hobby board, capturing waveforms using my Rigol DS4022 scope and a Rigol RP1050D high voltage differential probe and the UltraScope software.

For transformers, I used the magnetic transformer from the Elco enclosure, and I bought three ELV’s from eBay, two from a known brand, and one unknown brand:

For bulbs, I bought a variety of models from Amazon, eBay, and 1000bulbs:

  • Sylvania 58327: 50W Halogen MR16, 3000K, 35 Degree, 1450 CBCP.
    These are the halogen bulbs I currently use, about $2.20 per bulb.
  • Torchstar TS010: Dimmable, 12V 4W MR16 LED, 6000K Daylight, 50 Watt Equivalent, 330 Lumen, 60 Degree Beam Angle.
    I ordered a 10-pack from Amazon, the price worked out at about $5.50 per bulb. The packaging is generic, with a black marker dot indicating this to be a “pure white” variant. The bulb itself contains no markings, other than a small Torchstar sticker on the base. The bulb color is very blueish, like that of a daylight compact fluorescent bulb. I found the color to be very displeasing and distracting in my office environment, it made my color calibrated monitor screen appear yellow.
  • Torchstar TS010: Dimmable, 12V 4W MR16 LED, 3200K Warm White, 50 Watt Equivalent, 330 Lumen, 60 Degree Beam Angle
    I ordered a 10-pack from Amazon, the price worked out to about $5.70 per bulb. Like the daylight version, the packaging is generic, with a black marker dot indicating this to be a “warm white” variant. The bulb itself contains no markings, other than a small Torchstar sticker on the base. The bulb color is pleasing, pretty close to the halogen.
  • Soraa Brilliant 00965: Dimmable, 12V 9W MR16 LED, 75 Watt Equivalent, 3000K, CRI 80, CBCP 1540, 590 Lumen
    I ordered the bulbs from 1000bulbs, the price is about $28 per bulb. The color is pleasing but it appears to be ever so slightly bluer, more noticeable when dimmed. This bulb is bright, at 75W equivalent, almost too bright for my office as one of the bulbs is right above my head.
    Soraa specializes in high color quality products, and this model is from the older Brilliant Series, while I was really looking for the new Vivid Series bulbs, like the 00943, but it seems these bulbs are not yet available. I hope to find and test some when they do become available. At the price point of near $30 they are definitely specialty use, but I am interested in the supposed dimmability improvements.
  • Soraa Outdoor 00107: Dimmable, 12V 9.8W MR16 LED, 2700K
    This is a 36W equivalent LED for outdoor use, I bought them for about $24 more than a year ago, the line has since been discontinued.
  • Soraa Premium 2 00249: Dimmable, 12V 11.5W MR16 LED, 3000K
    I bought these more than a year ago for about $34 each,  the line has since been discontinued.
  • Architectural LED MR16-DIM-12V: 2700K 45deg
    I received samples of these MR16 LED’s from my electrician, I could not find any info on them.
  • eBay Dimmable CREE LED COB MR16: 6W MR16
    I bought a batch of 10 warm white and 10 daylight 6W bulbs, and a 9W and a 12W. The 6W bulbs are about $3 per bulb. These bulbs worked surprisingly well and the color was good. Note that the 9W and 12W variants are longer than standard MR16’s.

I tested the transformer response by monitoring the high voltage AC input and low voltage AC output sides using the oscilloscope. I controlled the ELV transformers using the ELV dimmer and the magnetic transformer using the magnetic dimmer. I attached a 12ohm resistor for a purely resistive load, the halogen bulb, and the Torchstar LED bulb. I captured oscilloscope screenshots at full, half, and lowest dimming settings.

Here are the results for the magnetic transformer:

Here are the results for the HATCH RS12-60M-LED ELV transformer:

Here are the results for the HATCH RL12-60A ELV transformer:

Here are the results for the Advance Lite TC60W ELV transformer:

Looking at the results we can see that the response waveforms for the halogen bulb is, not surprisingly, near that of the resistor. We can see that the magnetic transformer and LED load has all sorts of inductive goodness going on. And we can see that the RL12-60W and TC60W ELV transformers are not nearly as well behaved as the RS12-60M-LED ELV that is specifically designed for LED loads.

I then proceeded to test the dimmability of the various LED bulbs, I summarize my subjective findings below:

Halogen:

Magnetic: Good dimming range
RS12-60M-LED: Good dimming range, slight transformer buzzing
RL12-60A: Good dimming range
TC60W: Good dimming range

Torchstar:

Magnetic: Good dimming range, flicker at low end
RS12-60M-LED: Limited dimming range, no flicker, slight transformer buzzing
RL12-60A: Good dimming range, continuous flicker
TC60W: Good dimming range, flicker at low end

Soraa Premium 2:

Magnetic: Good dimming range, flicker at low end, very loud transformer buzzing
RS12-60M-LED: Good dimming range, flicker at low end, slight transformer buzzing

Soraa Brilliant:

Magnetic: Good dimming range, flicker at low end, slight transformer buzzing
RS12-60M-LED: Good dimming range, slight transformer buzzing

eBay CREE COB:

Magnetic: Good dimming range, switches off before end of dim range
RS12-60M-LED: Good dimming range, slight transformer buzzing

I was surprised that the cheap $3 eBay CREE COB MR16 LED bulbs worked as well as they did. Only downside is they switch off at around 20% when using the magnetic transformer, but dim down well. I don’t know if they really contain CREE COB LED’s, but the COB array arrangement of LED’s provide an even light source.

The Torchstar bulbs have a slight flicker at the low end, but dims down all the way, a bit more expensive compared to the eBay bulbs, but US based Torchstar support may be worth the extra 1$ per bulb.

The RS12-60M-LED ELV transformer performed well with halogen and LED loads, but the buzzing sound with or without load was a disappointment. I tested with two units, both buzz. I contacted the manufacturer to find out if this is normal, or if the units I bought on eBay are faulty.

I have yet to find a MR16 LED that can be driven by a magnetic transformer that performs like halogens, my search continues.

Nest Protect False Alarms

2AM, beep, smoke alarm low battery warning, and when one beeps, all the interconnected ones beep, now it is impossible to find which one has a low battery. As for how smoke alarms look, I’ve always wondered who made those terrible aesthetic design choices, maybe it is some kind of industry insider competition to see who can design the ugliest unit with the most obnoxious markings, and still get them sold.

I was thrilled when Nest announced the Nest Protect combination smoke and CO alarm, finally usability and technology catching up with smoke alarms, and an attractive looking unit. I’ve been a long time fan and user of the Nest thermostats, first one v1 unit, and later two v2 units, and I hoped the Nest Protect would do for smoke alarms what Nest did for thermostats.

I pre-ordered ten alarms from Amazon in October 2013, delivered in December 2013. Installation was easy, but I do wish there was a way to get more spoken locations, e.g. “smoke in kids bedroom”, which kid’s bedroom, wait, let me get my phone to see, not.

A week or two after installation we are having friends over for a barbecue, I show the alarm units, I show the mobile app, I explain how great the wave to silence alarm feature is, and how it will warn you before the alarm sounds, everybody is very impressed. Until a few hours later when one of the units go off, “smoke in the guest bedroom”, what smoke. I wave at it, nothing, I press the button, “this alarm cannot be silenced”. Keep in mind they are all interconnected with a mesh wireless network, so all ten units are screaming. After the kids stopped crying and we moved the party outside, I get a ladder and remove the unit, still screaming, I take it to fresh air, still screaming, I get a screwdriver open it up and remove the batteries, silence, but the rest of the units are still screaming, and pressing the button on those units still say “this alarm cannot be silenced”. About 5 minutes after removing the battery from the failed alarm the the other alarms stop. Egg on my face.

Nest support exchanged the unit and sent out a replacement.

As I was browsing the Nest support forums I noticed many other users reporting false alarms, some reporting that replacement units resolved the problem, some reporting repeat problems. Things got worse for Nest when they issued a recall, offering refunds, disabling the wave feature with a firmware update, and stopped selling units until they swapped stock for units with the newer firmware before re-releasing at a reduced price.

October 2014 early AM the alarm goes off, false alarm again, at least this time the alarm silenced itself after a minute. After some back and forth, and an escalation, Nest support agreed to replace all units. The new units have September 2014 manufacturing dates, so I hope these new units are less buggy.

January 2015 early AM the alarm goes off, false alarm again, this time the alarm stopped after only a few seconds. I’ve had enough, my kids are scared, my wife is mad, Nest, you’re out.

Nest support agreed to issue a refund for all ten units, we’ll see how long it takes to receive the refund. And now I’m in the market for combination smoke and CO alarms again, and there are not many choices, if you want something that is functional and good looking.

I was tempted to wait for the First Alert Wi-Fi enabled combination smoke and CO alarm, available for pre-order on Amazon, and although this unit is from a well established manufacturer, hopefully no false alarms, I’m not making the same mistake I made with Nest. Regardless of the pre-order option, it still leaves me unprotected, and I need something now. I could simply not find a decent looking, combination smoke and CO, interconnectable, and hardwired unit, big problem being decent looking.

In the end I opted for the First Alert PC910V units, they are low profile voice enabled combination smoke and CO units with a built-in 10 year battery, sold at Lowes or Amazon. Not interconnected, not hardwired, but at least they look half decent.

Installing these units turned out to be a bit more tricky than I anticipated. The install base is so small that the round ceiling junction boxes are barely hidden, and the instructions specifically call out that they are not to be installed on junction boxes due to air flow concerns.

Smoke

Below are some pictures showing the size differences between the Nest base (left, bottom), First Alert base (center, middle), and a round cover plate (right, top):

Base Size Comparison

Base Size Comparison

To account for the junction box ventilation warning I sealed between the junction boxes and the ceiling drywall, and between the cover plate and the ceiling. The alarm bases were mounted on the cover plates, see pics below.

Junction box

Sealed around junction box

Sealed around cover

Base on cover

Installed

Due to the small footprint of the alarm, the cover plate and imperfections around the hole in the ceiling can be seen when looking up at an angle. (Sorry for the crappy pictures, iPhone in low night not so great)

Drywall marks

Let’s hope I never hear them peep, at least not for ten years if we can trust the battery life, and at least not without a real emergency.