Razer BSOD When Driver Verifier is Enabled

I am done with Razer, exciting promises for technology on paper, great looking hardware, terrible support, terrible software.

Not too long ago I complained about Razer’s poor UX and Support, this time it is a BSOD in one of their drivers, and forever crashing Razer Stargazer camera software.

I’ve been looking for a Windows Hello capable webcam, and the Razer Stargazer, based on Intel RealSense technology, looked promising. The device is all metal and tactical looking, but the software experience is so buggy, install this, install that, then crash after crash after crash. I ended up returning it for a refund, and got a Logitech BRIO instead, the BRIO is cheaper, and works great.

A couple days ago I was greeted with a BSOD on one of my test machines, a crash in the RZUDD.SYS “Razer Rzudd Engine” driver, part of the Razer Synapse software. What makes this interesting, is that the issue seems to be triggered by having Driver Verifier enabled.

20170416_201259779_iOS

One may be tempted to say do not enable Driver Verifier, but, the point of driver verifier is to help detect bugs in drivers, and is a basic requirement for driver certification. Per the WinDbg analysis, this appears to be a memory corruption bug. After some searching, I found that the Driver Verifier BSOD has been reported by other users, with no acknowledgement, and no fix forthcoming. I contacted Razer support, and not surprisingly, they suggested uninstall and reinstall. I tried the community forums, and I was just pointed back to support.

FAULTING_IP:
rzudd+28c80
...
DEFAULT_BUCKET_ID:  CODE_CORRUPTION
...
PROCESS_NAME:  RzSynapse.exe
...
STACK_TEXT:
nt!KeBugCheckEx
nt!MiSystemFault+0x12e69c
nt!MmAccessFault+0xae6
nt!KiPageFault+0x132
rzudd+0x28c80
rzudd+0x218d4
rzudd+0x7a9f
Wdf01000!FxIoQueue::DispatchRequestToDriver+0x1bf [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3325]
Wdf01000!FxIoQueue::DispatchEvents+0x3bf [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3125]
Wdf01000!FxPkgIo::DispatchStep1+0x53e [minkernel\wdf\framework\shared\irphandlers\io\fxpkgio.cpp @ 324]
Wdf01000!FxDevice::DispatchWithLock+0x5a5 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1430]
nt!IovCallDriver+0x245
...
FAILURE_BUCKET_ID:  MEMORY_CORRUPTION_LARGE

I am done with Razer, exciting promises for technology on paper, great looking hardware, terrible support, terrible software.

Razer Shoddy Support and Bad Software UX

This post is just me venting my frustration at Razer’s poor software user experience, and their shoddy support practices. I’m writing this after I just had to go and find a working mouse, so I could click a button on a dialog that had no keyboard navigation support.

I’ve been using Razer keyboards and mice for some time, love them, their software not so much. I had to replace an aging ThinkPad, and the newly released Razer Blade Stealth looked like a great candidate, small and fast, reasonably priced, should be perfect, well, not so much.

I keep my monitors color calibrated, and I cringe whenever I see side-by-side monitors that clearly don’t match, or when somebody creates graphic content (yes you graphic artists using MacBooks to create content for PC software without proper color profiles) that looks like shades of vomit on a projector or a cheap screen, but I digress. My monitor of choice is NEC and their native SpectraView color calibration software. Unfortunately, the Blade with its lower end Intel graphics processor, and HDMI port, does not support DDC/CI, so no ability to color calibrate my monitor. My main monitor is a NEC MultiSync EA275UHD 4K monitor, and the internal Intel graphics processor is frustratingly slow on this high resolution display. And, the HDMI connectivity would drop out whenever the monitor went into power saving mode. Why not use a more standard mini-DisplayPort connector, would not solve the speed problem, but at least would have resolved the connection reliability and allowed for proper color calibration.

To solve the problem, I decided to get a Razer Core with an EVGA GeForce GTX 1070 graphics adapter. The Core is an external USB and network dock, with a PSU and PCIe connector for a graphics card, all connected to the notebook by Thunderbolt 3 over a, too short, USB-C cable. I connected my monitor to the GTX 1070 DisplayPort connector, connectivity was fine, I could color calibrate my monitor, and the display performance with the GTX 1070 was fast, great. By the way, JayzTwoCents has a great video on the performance of external graphic cards.

But, my USB devices connected to the dock kept on dropping out. I found several threads on the Razer support forum complaining about the same USB problems, and the threads are promptly closed with a contact support message. I contacted Razer support and they told me they are working on the problem, and closed my ticket. I contacted them again stating that closing my ticket did not resolve the problem, and they said my choice is RMA the device, with no known solution, or wait, and then they closed my ticket again. To this day this issue has not been resolved, and I have to connect my USB devices directly the notebook, defeating the purpose of a dock. They did publish a FAQ advising users to not use 2.4GHz WiFi, but to stick with 5GHz due to interference issues, so much for their hardware testing.

Now, let’s talk about their Razer Synapse software, the real topic of this post. The software is used to configure all the Razer devices, and sync the device preferences across computers with a cloud account, neat idea. The color scheme and custom drawn controls of this software matches their edgy “brand”, but their choice of thin grey font on a dark background fails in my usability book when used in a brightly lit office space.

Synapse.1

Whenever Windows 10 updates, the stupid Synapse software pops up while the install is still going, if you say yes, install now, then as expected the install fails due to Windows still installing. I logged the issue with Razer support, and they told me it is behaving as designed, really, designed to fail.

Synapse.2

So, today the Synapse software, again, prompts me to update, a frequent occurrence, and my mouse dies during the update, presumably because they updated the mouse driver, but this time I am prompted with a reboot required dialog. Dead mouse, no problem, have keyboard, tab over, wait, no keyboard navigation on the stupid owner drawn custom control dialog, no way to interact with the dialog without a mouse, just fail.

Synapse.3

Moral of the story, UX is important people, and I should just stick with ThinkPad or Microsoft Surface Book hardware, costs more, but never disappoints.

Circumventing ThinkPad’s WiFi Card Whitelisting

What started as a simple Mini PCI Express WiFi card swap on a ThinkPad T61 notebook, turned into deploying a custom BIOS in order to get the card to work.

I love ThinkPad notebooks, they are workhorses that keep on going and going. I always keep my older models around for testing, and one of my old T61’s had an Intel 4965AGN card, that worked fine with Windows 10, until the release of the Anniversary / Redstone 1 update. After the RS1 update, WiFi would either fail to connect, or randomly drop out. The 4965AGN card is not supported by Intel on Win10, and the internet is full of problem reports of Win10 and 4965AGN cards.

Ok, no problem, I’ll just get a cheap, reasonably new, with support for Win10, Mini PCIe WiFi card, and swap the card. I got an Intel 3160 dual band 802.11AC card and mounting bracket for about $20. The 3160 is a circa 2013 card with Win10 support. I installed the card, booted, and got a BIOS error 1802: Unauthorized network card is plugged in.

This lead me to the discovery of ThinkPad hardware whitelisting, where the BIOS only allows specific cards to be used, which lead me to Middleton’s BIOS, a custom T61 BIOS, that removes the hardware whitelisting, and enables SATA-2 support. I found working download links to the v2.29-1.08 Middleton BIOS here.

The BIOS update is packaged as a Win7 x86 executable or DOS bootable ISO image. As I’m running Win10 x64, and I could not find any CD-R discs around, I used Rufus to create a bootable DOS USB key, and I extracted the ISO contents using 7-Zip to a directory on the USB key. The ISO is created using a bootable 1.44MB DOS floppy image, and AUTOEXEC.BAT launches “FLASH2.EXE /U”, I created a batch file that does the same.

I removed the WiFi card, booted from USB, ran the flash, and got an error 1, complaining that flashing over the LAN is disabled. Ok, I enabled flashing the BIOS over the LAN in the BIOS, and rebooted.

I ran the update again, and this time I got error 99, complaining that BitLocker is enabled, and to temporarily disable BitLocker. I did not have BitLocker enabled, so I removed the hard drive and tried again, same error. Must be something in the BIOS, I disabled the security chip in the BIOS, tried again, and the update starts, but a minute or so later the screen goes crazy with INVALID OPCODE messages.

Hmm, maybe the updater does not like the FreeDOS boot image used by Rufus. Ok, let me create a MS-DOS USB key, uhh, on Win10, that turned out to be near impossible. Win10 does not include MS-DOS files, Rufus does not support custom locations for MS-DOS files, nor does it support getting them from floppy or CD images (readily available for download), the HP USB Disk utility complains my USB drive is locked, and writing raw images to USB result in a FAT12 disk structure that is too small to use. I say near impossible because I gave up, and instead went looking for an existing MS-DOS USB key I had made a long time ago. I am sure with a bit more persistence I could have found a way to create MS-DOS bootable USB keys on Win10, but that is an exercise of another day.

Trying again with a MS-DOS USB key, and voilà, BIOS flashed, and WiFi working.

I am annoyed that I had to go to this much trouble to get the new WiFi card working, but the best part of the exercise turns out to be the SATA-2 speed increase. This machine had a SSD drive, that I always found to be slow, but with the SATA-2 speed bump in Middleton’s BIOS, the machine is noticeably snappier.

A couple hours later, my curiosity got the better of me, and I made my own version of Rufus that will allow formatting of MS-DOS USB drives on Win10. In the process I engaged in an interesting discussion with the author of Rufus. I say interesting, but it was rather frustrating, Microsoft removed the MS-DOS files from Win10, and Rufus refuses to add support for sourcing of MS-DOS files from a user specified location, citing legal reasons, and my reluctance to first report the issue to FreeDOS. Anyway, can code, have compiler, if have time, will solve problem.

UPS Battery Replacement Turns Into Unrecoverable Firmware Update

Two lessons learned; do not trust scheduled battery tests, and leave working firmware be!

As the saying goes, if it is not broken do not fix it, especially when it comes to firmware.

I have a couple APC Smart-UPS‘s at my house, same as the models I like to use at the office. I use the SMT750 models with AP9631 Network Monitoring Cards. The problem started when we had a short power outage, and the UPS that powers the home network switch, cell repeater, alarm internet connection, and PoE IP cameras, unexpectedly died. A battery replacement led to the opportunity to do a UPS firmware update, which led to an unrecoverable firmware update.

It started when I woke up one morning and it was obvious the power had been out, first indicator is the kitchen appliances have blinking clocks, second are the numerous power failure email notifications, and the emails that stood out were from the alarm system that says it lost power and internet connectivity. The alarm has it’s own backup battery, the network switches and FiOS internet have their own battery backups, and the outage was only about 4 minutes.  So how is it that the UPS died, killing the switch, disconnecting the internet, especially when the outage was only 4 minutes, and typical runtimes on the UPS should be about an hour?

Here is the UPS outage log produced by the NMC card:

10/11/2016 06:53:05 Device UPS: A discharged battery condition no longer exists. 0x0108
10/11/2016 06:12:27 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 06:12:24 Device UPS: Restored the local network management interface-to-UPS communication. 0x0101
10/11/2016 06:12:12 System Network service started. IPv6 address FE80::2C0:B7FF:FE98:9BAF assigned by link-local autoconfiguration. 0x0007
10/11/2016 06:12:10 Device Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication. 0x0344
10/11/2016 06:12:09 System Network service started. System IP is 192.168.1.11 from manually configured settings. 0x0007
10/11/2016 06:12:02 System Network Interface coldstarted. 0x0001
10/11/2016 05:45:36 Device UPS: A low battery condition no longer exists. 0x0110
10/11/2016 05:45:36 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 05:45:35 Device UPS: The output power is turned off. 0x0114
10/11/2016 05:45:35 Device UPS: The graceful shutdown period has ended. 0x014F
10/11/2016 05:45:35 Device UPS: No longer on battery power. 0x010A
10/11/2016 05:45:35 Device UPS: Main outlet group, UPS Outlets, has been commanded to shutdown with on delay. 0x0174
10/11/2016 05:45:35 Device UPS: The power for the main outlet group, UPS Outlets, is now turned off. 0x0135
10/11/2016 05:45:18 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
10/11/2016 05:41:36 Device UPS: On battery power in response to rapid change of input. 0x0109

I could see from the log that the UPS battery power ran out within 4 minutes, 05:41:36 on battery, 05:45:18 battery too low, 05:45:35 output turned off. The UPS status page was equally puzzling, load was at 9.7%, yet reported runtime was only 5 minutes, impossible.

Here is the status screenshot:

apc-1

I ran a manual battery test, the test passed, but from the log it was clear the battery failed. I have bi-weekly scheduled battery tests for all UPS’s, never received a failure report. So what is the point of a battery test if the test comes back no problem yet it is clear to me from the logs that the battery failed?

Here is the log:

10/11/2016 18:07:06 Device UPS: A low battery condition no longer exists. 0x0110
10/11/2016 18:07:06 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107
10/11/2016 18:07:05 Device UPS: Self-Test passed. 0x0105
10/11/2016 18:06:59 Device UPS: A discharged battery condition no longer exists. 0x0108
10/11/2016 18:06:59 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
10/11/2016 18:06:58 Device UPS: Self-Test started by management device. 0x0137
10/11/2016 18:06:56 Device UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately. 0x0107

I asked for advice on the APC forum, no reply yet, and I ordered a replacement RBC48 battery. I received the battery, installed it, and the reported runtime is back to normal, 1 hour 48 minutes.

Here is a status screenshot with the new battery:

apc-2

Here is where I should have stopped and called it a day, but no. I knew that the UPS’s were on old firmware, and I decided to use this opportunity to update the firmware. I’d normally let firmware be, unless I have a good reason to update, but I convinced myself that the new firmware readme had some fixes that may help with the false pass on the battery test:

Release Notes (UPS09.3):
========================
2. Improved self-test logging for PCBE / NMC.
11. Repaired an occasional math error in the battery replacement date algorithm that resulted in incorrect dates.

I update the UPS, where I just replaced the battery, instructions are pretty simple. Only hassle is I have to bypass the network equipment to be mains powered so I can turn the UPS outputs off while updating the firmware, while maintaining network connectivity.

I did the same for my office UPS, PC and office switch on mains power, and when I power down the output, I made sure to not notify PowerChute Network Shutdown (PCNS) clients, as my PC had the PowerChute client installed to receive power state via the network. I start the firmware update over the network, and a few seconds later I get a Windows message that shutdown had been initiated by PCNS, what? I sit there in frustration, nothing to do but watch my PC shutdown while it is still delivering the firmware update.

On rebooting my PC, NMC comes up, but reports the UPS has stopped communicating. I pull AC power from the UPS, no change, I also pull the batteries, and when I plug the batteries and mains back on, beeeeeeeeeep. NMC now reports no UPS found, the UPS LCD panel reports all is fine. And still beeeeeeeeeep, and no way to stop the beeeeeeeeeep.

Here is the NMC status page:

apc-3

I try to do Firmware Upgrade Wizard update via USB, plug a USB cable in, PC sees UPS, reports critical condition, but the upgrade wizard reports no UPS found on USB.

Here is the wizard error page:

apc-4

So, here I am, stuck with a bricked UPS, lesson learned, actually two lessons learned; do not trust scheduled battery tests, and leave working firmware be!

Amazon Associate’s Account Closed

Amazon just notified me in email that my Associate’s account was closed due to not being in compliance with their operating agreement:

“You are not in compliance with Participation Requirement Number 29 because purchases resulting from Special Links on your site have been used for resale or commercial use.”

I have no idea how or why this happened.

 

A couple of years ago I moved my blog from the free Blogger platform to a paid WordPress.com hosted site. About the same time I signed up for an Amazon Associate’s account, profiting from any Amazon links resulting in sales, hoping that the proceeds would cover the costs of WordPress hosting and domain registration.

A quick calculation shows Amazon payouts of $669.57 between 2 August 2012 and 30 August 2016, that is about $167.39 per year, less the $99.00 for WordPress hosting, less $36.00 for Akismet blog spam filtering, less $19.00 for domain registration, leaves a profit of $13.39. Less $99.00 for bulk domain registration fees, not really fair to charge this fee to one domain, leaves a loss of $85.61 per year.

 

I do not know why I was suddenly out of compliance, I made no changes to either my Amazon Associates or WordPress accounts, and I’ve not posted any new content in a number of months. The WordPress stats show typical traffic (ignore the last two days), but the Amazon Associates report does show a marked increase in traffic:

 

I sent an email to Amazon support to clarify the violation, and to request my account be reinstated, but based on similar reports from other low traffic users,  I do not expect a resolution.

Instead, I opted-in to use WordPress’s own WordAds advertising platform, I still need to adjust the blog theme and settings to not interfere with reading, and I have no idea what the monetization would be, but at least I no longer have to bother with making special Amazon links.

Please comment and let me know if you find the ads to be intrusive, and I’ll consider funding the site without advertising assistance.

 

[Update: 1 September 2016]

A day after sending Amazon a request asking for an explanation, I received the following in email:

“This message is to advise you that your pievilsblo-20 account and your August 2016 Advertising Fees have been reinstated. Please accept our apologies for the closure.”

Looks like my account has been reinstated, no explanation of what happened.

Nest Protect False Alarms

2AM, beep, smoke alarm low battery warning, and when one beeps, all the interconnected ones beep, now it is impossible to find which one has a low battery. As for how smoke alarms look, I’ve always wondered who made those terrible aesthetic design choices, maybe it is some kind of industry insider competition to see who can design the ugliest unit with the most obnoxious markings, and still get them sold.

I was thrilled when Nest announced the Nest Protect combination smoke and CO alarm, finally usability and technology catching up with smoke alarms, and an attractive looking unit. I’ve been a long time fan and user of the Nest thermostats, first one v1 unit, and later two v2 units, and I hoped the Nest Protect would do for smoke alarms what Nest did for thermostats.

I pre-ordered ten alarms from Amazon in October 2013, delivered in December 2013. Installation was easy, but I do wish there was a way to get more spoken locations, e.g. “smoke in kids bedroom”, which kid’s bedroom, wait, let me get my phone to see, not.

A week or two after installation we are having friends over for a barbecue, I show the alarm units, I show the mobile app, I explain how great the wave to silence alarm feature is, and how it will warn you before the alarm sounds, everybody is very impressed. Until a few hours later when one of the units go off, “smoke in the guest bedroom”, what smoke. I wave at it, nothing, I press the button, “this alarm cannot be silenced”. Keep in mind they are all interconnected with a mesh wireless network, so all ten units are screaming. After the kids stopped crying and we moved the party outside, I get a ladder and remove the unit, still screaming, I take it to fresh air, still screaming, I get a screwdriver open it up and remove the batteries, silence, but the rest of the units are still screaming, and pressing the button on those units still say “this alarm cannot be silenced”. About 5 minutes after removing the battery from the failed alarm the the other alarms stop. Egg on my face.

Nest support exchanged the unit and sent out a replacement.

As I was browsing the Nest support forums I noticed many other users reporting false alarms, some reporting that replacement units resolved the problem, some reporting repeat problems. Things got worse for Nest when they issued a recall, offering refunds, disabling the wave feature with a firmware update, and stopped selling units until they swapped stock for units with the newer firmware before re-releasing at a reduced price.

October 2014 early AM the alarm goes off, false alarm again, at least this time the alarm silenced itself after a minute. After some back and forth, and an escalation, Nest support agreed to replace all units. The new units have September 2014 manufacturing dates, so I hope these new units are less buggy.

January 2015 early AM the alarm goes off, false alarm again, this time the alarm stopped after only a few seconds. I’ve had enough, my kids are scared, my wife is mad, Nest, you’re out.

Nest support agreed to issue a refund for all ten units, we’ll see how long it takes to receive the refund. And now I’m in the market for combination smoke and CO alarms again, and there are not many choices, if you want something that is functional and good looking.

I was tempted to wait for the First Alert Wi-Fi enabled combination smoke and CO alarm, available for pre-order on Amazon, and although this unit is from a well established manufacturer, hopefully no false alarms, I’m not making the same mistake I made with Nest. Regardless of the pre-order option, it still leaves me unprotected, and I need something now. I could simply not find a decent looking, combination smoke and CO, interconnectable, and hardwired unit, big problem being decent looking.

In the end I opted for the First Alert PC910V units, they are low profile voice enabled combination smoke and CO units with a built-in 10 year battery, sold at Lowes or Amazon. Not interconnected, not hardwired, but at least they look half decent.

Installing these units turned out to be a bit more tricky than I anticipated. The install base is so small that the round ceiling junction boxes are barely hidden, and the instructions specifically call out that they are not to be installed on junction boxes due to air flow concerns.

Smoke

Below are some pictures showing the size differences between the Nest base (left, bottom), First Alert base (center, middle), and a round cover plate (right, top):

Base Size Comparison

Base Size Comparison

To account for the junction box ventilation warning I sealed between the junction boxes and the ceiling drywall, and between the cover plate and the ceiling. The alarm bases were mounted on the cover plates, see pics below.

Junction box

Sealed around junction box

Sealed around cover

Base on cover

Installed

Due to the small footprint of the alarm, the cover plate and imperfections around the hole in the ceiling can be seen when looking up at an angle. (Sorry for the crappy pictures, iPhone in low night not so great)

Drywall marks

Let’s hope I never hear them peep, at least not for ten years if we can trust the battery life, and at least not without a real emergency.

Electrical Power Quality

Earlier this year we moved a couple miles from Redondo Beach to Manhattan Beach, bigger house, better school district.
As far as the house and area is concerned, it is definitely an upgrade, but not so for the utilities.

Monthly utilities are a lot more expensive, not so much the per unit fees, but the base service fees, not just a couple $, but three of four times what we paid in Redondo Beach. Now, if it came with better offerings, or better service, or higher quality, ok, but the opposite.
Water quality is worse, specifically hardness, MB supplies its own water, RB gets water from LADWP, and that unsightly water tower that no longer serves any practical purpose, with efforts to demolish it always being thwarted.
As a new resident trash collection makes me pay almost thirty $ extra per month for an extra trash can, while grandfathered-in residents keep extras for free. Now, I know it is unfair to judge a service by their employee’s actions, or is it, but the trash collection guy is a jerk, if a little dust and having to get out of the truck is going to get you agitated, you are in the wrong business, especially when compared with the pack of trash collection men in RB that were always friendly and willing to give a hand.
But, I really digress, I want to discuss electrical power quality problems.

In the six plus years we lived in RB, I think we had one scheduled power outage, and maybe two short unplanned outages. Since moving to MB earlier this year, we’ve had two scheduled outages, one lasting an entire day, and several unscheduled outages.
The power is unreliable, SCE knows it, the city knows it, there are some plans addressing it, see here here here here.

My concern is not really power being on or off, it is power being on but of poor quality; an electronic equipment killer.

When we moved in, the first signs of electrical problems were flickering lights. At first I thought it was a problem with the Vantage light control system, but even lights directly on utility power flickered. As soon as I hooked up UPS’s to my servers and the signal distribution system, the UPS’s started complaining about power quality. Occasionally during the day I would get a notification from the UPS’s that it detected a distorted input, and every night the UPS’s would complain about low input voltage.
It may be coincidental, but I’ve also had two astronomical clock light timers fail at the same time, the casings were scorched in what appears to be signs of electrical damage.

UPS Event Log:
APC Event Log

In order to quantify the problem, I used a Fluke VR1710 Voltage Quality Recorder. The device plugs into a mains outlet, and records events, and a USB port is used to configure the device, and download recorded data.

As I am not a power quality expert, I referred to Wikipedia to and Power Quality In Electrical Systems for information and reference material. To further simplify the analysis, I opted to compare my office power with my home power, this allowed me to easily visualize the quality differences, granted, I am assuming my office power is good.

I configured the VR1710 to take measurements every 10s, and to record exceptional events, about 10 days worth of data. I set the dip threshold to 106V, the swell threshold to 127V, and the transient sensitivity to 5V.

VR1710 Settings:
VR1710 Settings

Below are reports detailing the recorded events, click graphs to view full resolution:

Home Voltage:
Home - Voltage

There is a clear pattern of voltage drops below 102V every evening, these drops are also observed in the UPS logs showing low voltage warnings around 7:30PM every evening.

Office Voltage:
Office - Voltage

The office voltage is very stable.

Home Flicker:
Home - Flicker

According to Wikipedia and PQW short term flicker (Pst) is noticeable at values exceeding 1.0, and long term flicker (Plt) is noticeable at values exceeding 0.65. These results would explain why we observe lights flickering.

Office Flicker:
Office - Flicker

Office flicker values are well within acceptable ranges.

Home Statistics:
Home - Statistics

From this distribution we can see the wide spread in voltages, well below the 120V theoretical norm. This chart does not show it, but the 95% distribution is 115.5V, and the 5% distribution is 106.1V.

Office Statistics:
Office - Statistics

The office voltage distribution is nicely clustered around 119V, with the 95% distribution at 119.6V, and the 5% distribution at 117.4V.

Home Dips And Swells:
Home - Dips Swells

ITIC and CBEMA are standards for acceptable power quality, see here for a detailed description.
To describe the graph, I quote from the Fluke Power Log software manual:
Dips and swells are shown on a CBEMA (Computer Business Equipment Manufacturers Association) and ITIC (Information Technology Industry Council) plot classification table according to EN50160. On the CBEMA (blue) and ITIC (red), curve markers are plotted for each dip and swell. The height on the vertical axis shows the severity of the dip or swell relative to the nominal voltage. The horizontal position shows the duration of the dip or swell. These curves show an ac input voltage envelope which typically can be tolerated (no interruption in function) by most Information Technology Equipment (ITE).

Based on the graph we can see a large number of events exceeding the acceptable ranges. Since there were no dips at the office, there is no graph for the office.

Home Transients:
Home - Transients

I only show the transients graph for home, as the wave forms all look different, and the only difference between home and office is 87 events were recorded at home while 10 events were recorded at the office for the same approximate time duration. See PQW for an explanation of transients.

We can clearly see that the power quality at my house is significantly worse compared to the power at my office.

I am speculating, but I wonder if the old transformer across the road can supply sufficient power, given that it used to supply power to three small very old houses on four lots, demolished to make room for four new larger houses?

I just opened a support ticket with SCE, let’s hope they can do something about the problem.

LSI turns their back on Green

I previously blogged here and here on my research into finding a power saving RAID controllers.

I have been using LSI MegaRAID SAS 9280-4i4e controllers in my Windows 7 workstations and LSI MegaRAID SAS 9280-8e controllers Windows Server 2008 R2 servers. These controllers work great, my workstations go to sleep and wake up, and in workstations and servers drives spin down when not in use.

I am testing a new set of workstation and server systems running Windows 8 and Server 2012, and using the “2nd generation” PCIe 3.0 based LSI RAID controllers. I’m using LSI MegaRAID SAS 9271-8i with CacheVault and LSI MegaRAID SAS 9286CV-8eCC controllers.

I am unable to get any of the configured drives to spin down on either of the controllers, nor in Windows 8 or Windows Server 2012.

LSI has not yet published any Windows 8 or Server 2012 drivers on their support site. In September 2012, after the public release of Windows Server 2012, LSI support told me drivers would ship in November, and now they tell me drivers will ship in December. All is not lost as the 9271 and 9286 cards are detected by the default in-box drivers, and appear to be functional.

I had hoped the no spin-down problem was a driver issue, and that it would be corrected by updated drivers, but that appears to be wishful thinking.

I contacted LSI support about the drive spin-down issue, and was referred to this August 2011 KB 16563, pointing to KB 16385 stating:

newer versions of firmware no longer support DS3; the newest version of firmware to support DS3 was 12.12.0-0045_SAS_2108_FW_Image_APP-2.120.33-1197

When I objected to the removal, support replied with this canned quote:

In some cases, when Dimmer Switch with DS3 spins down the volume, the volume cannot spin up in time when I/O access is requested by the operating system.  This can cause the volume to go offline, requiring a reboot to access the volume again.

LSI basically turned their back on green by disabling drive spin-down on all new controllers and new firmware versions.

I have not had any issues with this functionality on my systems, and spinning down unused drives to save power and reduce heat is a basic operational requirement. Maybe there are issues with some systems, but at least give me the choice of enabling it in my environment.

A little bit of searching shows I am not alone in my complaint, see here and here.

And from Intel a November 2012 KB 033877 that they have disabled drive power save on all their RAID controllers, maybe not that surprising given that Intel uses rebranded LSI controllers.

After a series of overheating batteries and S3 failures, I have long ago given up on Adaptec RAID controllers, but this situation with LSI is making me take another look at them.

Adaptec is advertising Intelligent Power Management as a feature of their controllers, I ordered a 7805Q controller, and will report my findings in a future post.

RIP Boxee Box

After nearly six months of no software updates for the Boxee Box, Boxee announced the Boxee TV, and, as far as I’m concerned, the death of the Boxee Box.

Boxee is releasing an updated hardware platform, but they are abandoning all local media playback and cataloging capabilities, and instead focusing on a, US only, cloud storage DVR device.

I have no need for such a device, and based on the Boxee community forum posts, the blog comments, and even comments from their XBMC roots, I am not alone in expressing my disappointment.

I suspected this may happen, but I had always hoped that Boxee would eventually make good on their empty promises and fix the issues. If not fix it, then release an updated hardware platform that corrects the problems that plagued the first version, and I’d still be willing to pay for it.

 

I am one of the many users that is plagued by the HD audio playback dropout issues introduced in a firmware update almost two years ago. A problem Boxee blamed on the Intel CE4100 SDK, and promised to fix in March, but then backtracked saying that fixing it would incur too much testing overhead. Yes, break a feature that worked, then claim it is Intel’s fault, but refuse to correct it because it is too much trouble to test.

The Boxee Box will get a last update to fix an issue with Flash playback, but the HD audio issue will not be fixed.

 

I have already transitioned one of my Boxee Boxes to XBMC based OpenELEC 2.0 running on a Zotac ZBOX Nano XS ID11 Plus. It still has a few rough edges, but XBMC is actively being developed for a variety of exciting platforms.

The one thing about Boxee I will miss the most is the standalone D-Link DSM-22 Boxee remote, best remote for XBMC ever. If I had known they will be discontinued, and impossible to buy, I would have bought a couple spares. If you know where to buy DSM-22’s, please let me know.

 

Rest In Peace Boxee Box.

Dyslexic Intel RSTe Driver

I encounter one problem after another running Windows 8 and Server 2012 on the dual Xeon E5 Intel C600 chipset based SuperMicro 7047A-T and 7047A-73 SuperWorkstation machines. I will say that this is really not representative of my Windows 8 experience in general, as all other machines I installed on worked fine with the in-box drivers.

The C602 includes the Intel Storage Controller Unit (SCU) SATA / SAS controller. Windows 8 and Server 2012 do not include in-box drivers for the SCU. The SCU drivers are part of the Intel Rapid Storage Technology Enterprise (RSTe) driver set. Note that the RSTe and RST drivers are different and not compatible with one another. When you install the full RSTe package, it includes SCU drivers for the SCU RAID controller, AHCI drivers for the SATA controller, and the Windows RST management application.

A clean install of Windows 8 will use the in-box drivers for the SATA controller. In the image below you can see the Intel 520 Series 480GB SSD drive show up with the correct model number:

Device.Manager.Win8

After installing RSTe (3.2.0.1132, 3.2.0.1134), the 4TB Hitachi drives attached to the SCU show up, but the model numbers of the drives, including the SSD drive attached to the SATA port, are now messed up:

Device.Manager.RSTe

The drive hardware identifiers are correct, but the friendly name is not:

Intel.SSD.Hardware

Intel.SSD.Friendly

It appears that the text BYTE’s are WORD swapped, i.e. ABCD becomes BADC.

The driver is also not functional, attempting to create a storage spaces pool using the Hitachi drives hangs forever, with no drive activity, requiring a hard power cycle:

Storage.Pool

And lastly, the Intel SSD Toolbox 3.0.3 is not compatible with Windows 8:

SSD.Toolbox

The clock is ticking for Windows Server 2012 (4 September, 1 day left) and Windows 8 (26 October, 7 weeks left) general availability, I can only hope compatible drivers, firmware, and utilities are forthcoming.

 

[Update: 4 September 2012]
SuperMicro posted updated RSTe drivers (package v3.5.0.1101, driver v3.5.0.1096). This driver set resolves the hang during storage space creation, but the drive names are still messed up.