TiVo Roamio DVR vs. Verizon FiOS DVR

I just replaced the last of my four Verizon FiOS DVR boxes with a TiVo Roamio DVR box.

The last time I used a TiVo was about seven years ago, it was a DirecTV TiVo box, commonly referred to as a DirecTiVo. At that time DirectTV was my only option for television service, TiVo the only option for a DVR, and AT&T DSL the only option for internet and phone.

I then moved to an area where I could only get Time Warner Cable for internet and TV, and only Verizon for phone service. I used TWC for internet and TV, and Vonage VoIP for phone service. The TWC DVR sucked compared the TiVo that I was used, and so did their internet reliability, and so did their customer service.

Let me just say how much I hate it that city councils, property owners, and service providers negotiate exclusive service provider agreements. Competition is good, and exclusivity and long service contracts leads to substandard offerings and poor service levels.

A year later Verizon FiOS became available in my area, and I switched to Verizon FiOS for TV, internet, and phone. The Verizon DVR was the same Motorola hardware used by TWC, but different software. The Verizon DVR software was slightly better than the TWC software, but only slightly. The HD channel lineup and internet speed was the primary reasons for switching.

About two years ago Verizon released new DVR hardware and software, a significant improvement over the old. The current DVR UI is all HD all the time, supports multi-room features, and offers mobile and web series management apps. All-in-all I was very happy with the DVR software, the only thing lacking was the number of tuners and the storage capacity.

I’ve often considered switching to non-Verizon DVR solutions, like the Moxi or the Ceton Q. I tried a Window MCE box with a Ceton infiniTV6 ETH CableCard decoder, worked ok, but the form factor was not very practical, and it required constant maintenance. The Moxi is no longer available for purchase, and the Ceton Q appears to be vaporware. The TiVo Premiere was an option, but I found the cost prohibitive, and it did not offer substantially more than my Verizon FiOS DVR box.

TiVo recently released the new Roamio range of DVR’s. They offer three variants, the Roamio, Romio Plus, and Roamio Pro, differences and details are here. These new devices offer 4 to 6 tuners, 75 to 450 hours of HD recording capacity, and pricing ranges from $200 to $600. You still have to pay TiVo $15 per month or a $500 lifetime TiVo service fee per device, as well as the $5 Verizon per-cablecard fee, a total of $20 per month. Compared to $20 per month for a Verizon multi-room HD DVR offering 50 hours of HD recording and 2 tuners, TiVo is equally priced on a month-to-month basis, costs more upfront, but offers significantly more capability and capacity. Amazon occasionally offers the Roamio units cheaper than TiVo direct, and weaKnees offers units with upgraded capacity.

The TiVo Mini is a playback only device that lets you watch live or recorded TV. You need to pair it with a Roamio, and it acts like an extender for the Roamio UI. What makes no sense is that you need to buy it at $100, and then still pay a monthly $6 or lifetime $150 service fee. I am already paying a subscription fee for the main Roamio, why do I need to pay twice?

I tested three TiVo devices, a Mini, a standard Roamio, and a Roamio Pro.

The screenshots below shows the relative sizes of the devices.

Roamio Size Front Roamio Size Back

The Mini is very small, the Pro is thinner but wider than the FiOS DVR box, the standard Roamio is reasonable thin and small. The Mini and the standard use small wall mount external power supplies.

The screenshot below shows the relative sizes of the remote controls.

Roamio Remotes

The Roamio remote is smaller than the Mini remote, and supports both IR and RF, making it work through cabinet doors. The standard Verizon remote is ugly and bulky. I use URC MX-890 programmable IR/RF remotes with RF to IR base stations.

I installed the Roamio Pro first, hardware setup was easy, the out of box experience was rather disappointing. The setup took a really long time, and one of the first screens you are greeted with tells you that the install will take more than an hour. The Pro setup, including reboots for updates, took about an hour. The Mini setup was even longer at about 1 hour 20 minutes, at times the Mini UI reported 99% completed for almost half an hour without any movement. I find it fascinating that a company that used to pride itself on user experience could come up with such a crappy first customer interaction experience.

TiVo.Hour

The cablecard activation required a call to Verizon. New cablecards can be activated online or over the automated phone system, but previously activated cablecards that are moved between devices needs to be re-activated by Verizon tech support. This was the only device with a previously activated cablecard, the other three boxes had new cablecards, and I activated them using the automated phone system.

I immediately ran into network connectivity problems; the box would randomly report C130, C133, C501, and UI-113 errors.

TiVo.C130 TiVo.C133 TiVo.C501 TiVo.UI113

Manually verifying internet connectivity or TiVo service connectivity always succeeded.

I contacted TiVo support, who told me the problem is my internet connection, and that network switches are not supported, and that I need to connect the TiVo directly to my router, what? My house is wired with CAT6 connected to a HP ProCurve gigabit switch, and each AV closet has a TrendNET TEG-S8g 8-port gigabit switch. Nothing wrong with my network, and nothing wrong with my internet access, the TiVo is the only device with issues. No point arguing with support, they suggested I disconnect ethernet and instead use the built-in MoCA bridge for internet connectivity. This worked, no more connectivity issues.

After some more research I found similar issues reported on the TiVo Community forum, siting incompatibility with “green” or 802.11az compliant switches. TiVo acknowledged the problem, and stated a future update would resolve the issue.

I started out with a Pro and a Mini, while keeping my main FiOS DVR active for use by the family.

I have become used to the slick HD UI of my FiOS DVR, compared to the TiVo, the TiVo UI really does look old, and ignoring the addition of internet video services, still looked about the same as it did seven years ago when I last used a TiVo. Not to say that they need to change something that works, but really, no changes, no color scheme updates, no graphics refresh, no stop button support?

Unlike the FiOS box where you have to pick one output resolution, 720p or 1080i, and all input resolutions are scaled to that output resolution, the Roamio lets you select multiple supported output resolutions, 480p, 720p, 1080i, 1080p60 and 1080p24, this has pros and cons. If you enable multiple resolutions in the Roamio, and the menu you are navigating to does not have HD support, like settings, the screen will switch resolutions as you navigate between menu’s, causing a temporary delay and black screen as the resolution changes. On the positive side, if the video source resolution matches a supported output resolution, the video will be passed through without scaling, allowing the destination device to perform the scaling. SD channels are typically 480i, and HD channels are 720p or 1080i. I performed non-scientific visual appearance tests using Disney Junior and Nick Junior 480i SD channels. Setting the Roamio to one of 720p, 1080i, or 1080p only, resulted in significant artifacts very noticeable around the channel logos. The FiOS box set to 720p or 1080i showed no such artifacts. Setting the Roamio to output at all resolutions (except 480i that is listed as not supported), it produced output at 480p for the 480i material, and resulted in reasonably good quality, equal to or better than the FiOS box. This leads me to believe that the Roamio is not good at scaling video material, or at least not as good as the FiOS box. Preferring image quality, I enable all supported resolutions.

I found the following to be negative aspects of Roamio:

  • Setup is tedious and time consuming.
  • When pressing the TiVo button, there is a short pause in the playback stream before it is displayed in a window on the main UI, and when returning to the full screen view there is a gain a short pause. Compared to the FiOS box where the video remains smooth moving from full screen to windowed and back never skipping a beat.
  • On the Mini, as soon as you press the TiVo button it stops playing whatever you were watching. Unlike the regular Roamio boxes, and the FiOS box, the Mini never displays the current watching video in the main UI. This is a big negative for the Mini, I often switch to the menu to search for and add shows to watch as trailers air on the current show. With the embedded video I can keep watching and still use the menu system, on the Mini it is one or the other.
  • On the Roamio, while playing live TV, and pressing the TiVo button, the TV stream will continue to display in an embedded window in the main UI, but if you were playing recorded content, the feed would switch to live TV, and there is no way to return the playback to full screen. The FiOS DVR does this seamlessly, you can navigate between the menu system and back to full screen without interrupting the feed, be that live TV or a recording on this or another device.
  • The FiOS box menu and UI system is all HD all the time. On the Roamio and the Mini, only the main window and main functionality is in HD, many of the settings and other menus are in SD. When navigating around the UI, and the resolution switches from HD to SD, there is a noticeable delay and black screen as the video equipment adjusts to the new resolution. It looks as if TiVo ran out of time to convert all UI to HD, or maybe they don’t care about looks?
  • On the FiOS box the skip forward and skip back buttons skip the same amount of time. On the TiVo the skip forward skips 30s and skip backwards / replay skips 8s only. Being used to using the skip buttons to go forward or backwards, I find the 8s replay functionality mostly useless, and instead end up using the less convenient rewind most of the time.
  • When navigating the UI, specifically when adding a season pass, there is often a few seconds of delay where the UI is not responsive showing no indication of accepting and processing the input command, again, poor UI responsiveness leads to poor usability.
  • I already mentioned this, I find it preposterous to have to pay a monthly service fee for the Mini when I’m already paying for the main DVR service.
  • The basic Roamio does not support MoCA, meaning many users will need to purchase a MoCA bridge for internet connectivity. The Mini supports MoCA, why not the basic Roamio?
  • Combining the cablecard fee and the TiVo service fee, there is no financial incentive to switch to a Roamio.
  • No stop button, I really don’t know why they don’t include a stop button. In order delete this show, or go back to the menu, without loosing location context, you need to press the left button.
  • Power on to fully functional takes much longer than the FiOS box.
  • Amazon video does not support Amazon Prime free content.
  • Netflix UI navigation not as good as Roku.
  • As already described, video up-scaling results in artifacts.

I found the following to be positive aspects of Roamio:

  • 6 Tuners compared to FiOS 2, no more conflicting recordings and live TV.
  • 3TB storage and 450 hours recording compared to FiOS 50 hours, no more shows getting deleted before they can be watched.
  • The TiVo iPhone and iPad apps are much better than the equivalent Verizon apps.
  • The Watch Now functionality, no more guide surfing or channel hopping, see everything related to your watching habit on one page.
  • Automatic recording of related shows, nice to explore previously unknown shows or just to watch something new.
  • Pressing skip forward multiple times aggregates the skip distance, making it easy to skip in well known fixed increments, i.e. skip over fixed length ads.

In the end the 6 tuners and 450 hours of capacity were the most important reasons for switching.

If the Mini’s supported displaying the current play feed while using the menu, I would have used one Pro, upgraded to 8TB, and three Mini’s. But since the basic Roamio is only $100 more than Mini, there is little point in buying a Mini.

I ended up installing two Pro’s and two basic’s, the Pro’s in the game and living rooms, and the basic’s in the guest room and master bedroom. Enough space and enough tuners to keep everybody happy.

Advertisements

Panasonic Lumix WiFi Dissapoints

I’ve been a long time fan of Panasonic Lumix digital cameras, and I tend to update them every couple of years. My current cameras are the DMC-ZS20 for travel, and the DMZ-ZS7 for pocket use.

 

My workflow typically entails taking lots of pictures, and then using a Transcend USB3 card reader and ImageIngester Pro to copy and rename them to my PC, and finally import them into Adobe Photoshop Lightroom. This is a tedious process, but with the release of the Lumix WiFi capable handheld cameras, an opportunity to make my life simpler by uploading directly using WiFi.

 

I bought the WiFi enabled DMC-ZS30 to replace my DMC-Z20, and a DMC-TS5 to replace my DMC-ZS7. The TS5 is not really an upgrade to the ZS7, but it is waterproof, shockproof, and kids-proof.

 

The cameras were typical great Lumix quality, but my interest was the WiFi capability.

 

The camera offers several modes of connecting, I setup the PC WiFi connected option. During the setup process you have to select the access point SSID, in my case I have a couple roaming enabled access points around me, and the camera showed multiple access points with the same SSID. I’ve seen this behavior in some devices that bind to a specific access point, and these devices would refuse to roam, requiring a new setup for every single access point.

It did not really matter as the camera would not connect, it would ask for my password, but fail to connect. My camera was running firmware 1.0, and there was a v1.2 available, I upgraded, and the WiFi connection succeeded. The firmware notes say nothing about WiFi connectivity, but for whatever reason it helped.

 

First big frustration; every time anything in the connection setup flow fails, you have to re-enter the WiFi password, the login username, and the password, very painful when using a navigation only keyboard, and long complicated passwords. It would have been much more convenient to configure WiFi in one place, save the WiFi connection, and then configure transfer profiles, or at least always remember the previously entered values.

 

PC connectivity requires a SMB network share and a named host, i.e. no support for any protocol other than SMB, and no support for entering servers by IP address.

I could not get the camera to connect to my server, looking at the security logs, I could see that the connection was using “WORKGROUP” as the domain, and using the DOMAIN portion of  [DOMAIN\UserName] specification as part of the literal username. The camera has no provisioning for changing the workgroup or domain, and I had to resort to creating a machine local account in order to connect. Once I had my local account created, and again re-entered all information, I could finally connect.

 

I took some pictures, pressed the WiFi button, navigated to the profiles, and uploaded the pictures, it took forever, around 90s per 5MB average picture, or around 400Kbps. Using my USB3 card reader the same pictures all transferred in a few seconds. Yes, USB3 is much faster than WiFi 802.11n, but 400 kilo bits per second is super slow, not near the capability of the WiFi network.

 

I also tried the Panasonic Lumix Club Cloud Sync service, what a joke. You setup the account from the camera, it reports your a username in the form of aaaa-bbbb-cccc-dddd, then you enter your own password, but when you try to login at the website, that you need to find using google, you need to use aaaabbbbccccdddd. I only discovered this through trial and error.

On logging in, you need to supply an email address, and then validate the email address, by clicking a link emailed to you. On clicking the link, it takes you to the service page, and it displays an error message, any action more errors. If you manually login again, you are in.

From there nothing works. From what I can tell from FAQ’s, you are supposed to be able to create a drop that allows you to pull the data from the cloud to your PC. But when you go to configure the devices and services, none of the links work.

 

What Panasonic should have done is build a very simple get connected flow, many small complicated devices today use a USB connection and an app, PC or mobile, to get the device provisioned, there are many other novel ways of simplifying this process, e.g. QR codes, BT, P2P, SD card data file, etc. Not supporting FTP, or IP addresses, or domain credentials, or workgroup configurations would be forgiven if connecting was easy, and transfer speed was at WiFi capacity, sadly, it is not. 

 

From my perspective the WiFi capability on these cameras are a waste of good silicon and battery power.

XBMC on NUC’s and Pi’s

I’m still looking for the perfect XBMC hardware; must be small, silent, low power, low heat, 1080p, HD audio, and play anything I throw at it without a hiccup. The number of options are increasing, but no clear winner.

 

 

I previously tested a XIOS DS running XBMC on Android, and XBMC on Linux. At that time the builds were pretty unstable. I retested the latest Linux builds, that also include XBMC 12 Frodo RC2.

I tested using the 121512 release, after rebooting, I just saw a black screen. I could see that the AVR had negotiated HDMI audio, but the screen remained black. Reading the forum thread there were many reports of similar problems, same symptoms, leave the system up, and after 15 minutes XBMC loaded. The bug has been identified, but not yet fixed in official firmware. I used a community build that included the fix, and the system booted normally.

I noticed that there are now two hardware variants of the DS, a M1 version, that I have, and a new M3 version, that apparently includes a faster processor and more memory, and is currently only shipped in the EU and UK. This seems to be consistent with the AMLogic AML8726-M SoC device containing an ARM Cortex-A9 and a Mali-400 graphic processor.

The playback results were rather disappointing, no HD audio pass-through, high bitrate content would stutter, and I would get frequent network re-buffering. This device still shows promise, but not in its current state.

 

 

I tested XBMC on a Raspberry Pi. The Pi devices are pretty cheap at $35, but the units at this price have very long lead times. Instead I opted to buy an in-stock Model B Revision 2 unit from Amazon, and also a case.

The Pi Model B Revision 2 uses the Broadcom BCM2835 SoC device containing an ARM1176JZ-F with VideoCore IV graphic processor.

Deploying XBMC to a Pi is rather more involved compared to the DS, and I opted to use the Raspbmc distribution that includes easy to use tools for Windows. The deployment tool creates a bootable SD card, that then retrieves and installs the latest builds over the internet, similar to many Linux network boot disk installers.

The playback results were rather disappointing, no HD audio support, high bitrate content would stutter, and I would get very frequent network re-buffering.

Similar to openELEC that provides a XBMC plugin for OS configuration, Raspbmc configuration in XBMC is done using the Raspbmc plugin. When I first clicked the plugin I thought it did nothing, and after several more remote clicks it suddenly displayed and did whatever my remote clicks did, causing a restart. The plugin provides lots of configuration options, including switching of XBMC versions, downloading and running nightly builds, and advanced configuration, but really it is super slow to load up.

XBMC on the DS supported HD audio passthrough, but Raspbmc did not include HD audio support. The plugin allowed me to enable the XBMC AudioEngine, with a warning that it may not work. After restarting XBMC with AE enabled, there were options for HD audio, but AE did not detect the HDMI audio output device and only offered audio output over analog or SPDIF.

MPEG2 and VC-1 codecs have to be purchased for the Pi, but as my test results were disappointing, I did not bother purchasing the codecs.

 

 

I tested one of the new Intel Next Unit of Computing devices, specifically the DC3217IYE. The device is barebones, and I used Kingston KVR16S11K2/16 16GB memory and a Kingston SMS100S2/64G 64GB mSATA card. Oh, and you need your own power cable, I happened to have a spare Monoprice 7687 3-prong power cable lying around that fit the PSU.

I don’t know what to make of it, but Intel included a gadget in the box, that plays the Intel jingle every time you open the box. I’m inclined to think that they could have included a power cable instead of the jingle gadget, but my kids do enjoy playing with the box, so it may have some marketing value.

Here are a few unboxing pictures:
IMG_1384_DxOIMG_1385_DxOIMG_1386_DxOIMG_1388_DxOIMG_1389_DxOIMG_1390_DxOIMG_1391_DxOIMG_1392_DxOIMG_1393_DxOIMG_1394_DxOIMG_1395_DxOIMG_1396_DxOIMG_1397_DxOIMG_1398_DxOIMG_1399_DxO

 

I installed openELEC v3 Beta 6, that includes XBMC 12 Frodo RC2.

Most things worked fine, audio output device was automatically detected and set to HDMI, but HD audio passthrough did not work, and several videos showed artifacts during playback, even worse, some videos caused lots of artifacts and caused the device to hang. I assume the video issue is a problem with the Intel HD graphics driver being picked up by openELEC.

I am using a D-Link DSM-22 RF remote (I wish I can find more for sale), and I found that the key presses were erratic, after moving the RF dongle from a rear USB port to the front USB port, everything worked fine. I assume there is some interference near the back of the unit.

Physical size wise the NUC compares well against a Zotac ZBox Nano XS AD11 Plus, but price wise the NUC is more expensive once memory and flash storage is added.

The Nano XS is a Fusion based device, which means it will never get HD audio passthrough (AMD drivers lack HD audio support on Linux), so if openELEC and Intel can resolve the video corruption on the NUC, and XBMC can resolve the HD passthrough problem with my setup, the NUC would be a good contender.

 

I am still running openELEC on my Zotac ZBOX ID84 system with a NVidia GeForce GT520M GPU. This GPU supports HD audio passthrough, but as with my other devices, it does not work on my setup. The problem appears to be related to how XBMC AudioEngine targets audio, and that instead of sending the audio to the AVR, it sends it to the television, but this is speculation on my part. I logged a ticket with openELEC and XBMC, and there is a forum thread at openELEC with other Yamaha and Onkyo AVR users reporting similar problems, but nobody from openELEC or XBMC has yet responded 😦

 

Here is a comparison of device sizes, top is Raspberry Pi, then XIOS DS, then ZBOX AD11, then Intel NUC, and ZBOX ID84 at the bottom:
PV_20121226_5060_DxOPV_20121226_5061_DxO

 

My quest continues.

LSI turns their back on Green

I previously blogged here and here on my research into finding a power saving RAID controllers.

I have been using LSI MegaRAID SAS 9280-4i4e controllers in my Windows 7 workstations and LSI MegaRAID SAS 9280-8e controllers Windows Server 2008 R2 servers. These controllers work great, my workstations go to sleep and wake up, and in workstations and servers drives spin down when not in use.

I am testing a new set of workstation and server systems running Windows 8 and Server 2012, and using the “2nd generation” PCIe 3.0 based LSI RAID controllers. I’m using LSI MegaRAID SAS 9271-8i with CacheVault and LSI MegaRAID SAS 9286CV-8eCC controllers.

I am unable to get any of the configured drives to spin down on either of the controllers, nor in Windows 8 or Windows Server 2012.

LSI has not yet published any Windows 8 or Server 2012 drivers on their support site. In September 2012, after the public release of Windows Server 2012, LSI support told me drivers would ship in November, and now they tell me drivers will ship in December. All is not lost as the 9271 and 9286 cards are detected by the default in-box drivers, and appear to be functional.

I had hoped the no spin-down problem was a driver issue, and that it would be corrected by updated drivers, but that appears to be wishful thinking.

I contacted LSI support about the drive spin-down issue, and was referred to this August 2011 KB 16563, pointing to KB 16385 stating:

newer versions of firmware no longer support DS3; the newest version of firmware to support DS3 was 12.12.0-0045_SAS_2108_FW_Image_APP-2.120.33-1197

When I objected to the removal, support replied with this canned quote:

In some cases, when Dimmer Switch with DS3 spins down the volume, the volume cannot spin up in time when I/O access is requested by the operating system.  This can cause the volume to go offline, requiring a reboot to access the volume again.

LSI basically turned their back on green by disabling drive spin-down on all new controllers and new firmware versions.

I have not had any issues with this functionality on my systems, and spinning down unused drives to save power and reduce heat is a basic operational requirement. Maybe there are issues with some systems, but at least give me the choice of enabling it in my environment.

A little bit of searching shows I am not alone in my complaint, see here and here.

And from Intel a November 2012 KB 033877 that they have disabled drive power save on all their RAID controllers, maybe not that surprising given that Intel uses rebranded LSI controllers.

After a series of overheating batteries and S3 failures, I have long ago given up on Adaptec RAID controllers, but this situation with LSI is making me take another look at them.

Adaptec is advertising Intelligent Power Management as a feature of their controllers, I ordered a 7805Q controller, and will report my findings in a future post.

RIP Boxee Box

After nearly six months of no software updates for the Boxee Box, Boxee announced the Boxee TV, and, as far as I’m concerned, the death of the Boxee Box.

Boxee is releasing an updated hardware platform, but they are abandoning all local media playback and cataloging capabilities, and instead focusing on a, US only, cloud storage DVR device.

I have no need for such a device, and based on the Boxee community forum posts, the blog comments, and even comments from their XBMC roots, I am not alone in expressing my disappointment.

I suspected this may happen, but I had always hoped that Boxee would eventually make good on their empty promises and fix the issues. If not fix it, then release an updated hardware platform that corrects the problems that plagued the first version, and I’d still be willing to pay for it.

 

I am one of the many users that is plagued by the HD audio playback dropout issues introduced in a firmware update almost two years ago. A problem Boxee blamed on the Intel CE4100 SDK, and promised to fix in March, but then backtracked saying that fixing it would incur too much testing overhead. Yes, break a feature that worked, then claim it is Intel’s fault, but refuse to correct it because it is too much trouble to test.

The Boxee Box will get a last update to fix an issue with Flash playback, but the HD audio issue will not be fixed.

 

I have already transitioned one of my Boxee Boxes to XBMC based OpenELEC 2.0 running on a Zotac ZBOX Nano XS ID11 Plus. It still has a few rough edges, but XBMC is actively being developed for a variety of exciting platforms.

The one thing about Boxee I will miss the most is the standalone D-Link DSM-22 Boxee remote, best remote for XBMC ever. If I had known they will be discontinued, and impossible to buy, I would have bought a couple spares. If you know where to buy DSM-22’s, please let me know.

 

Rest In Peace Boxee Box.

Storage Spaces Leaves Me Empty

I was very intrigued when I found out about Storage Spaces and ReFS being introduced in Windows Server 2012 and Windows 8. But now that I’ve spent some time with it, I’m left disappointed, and I will not be trusting my precious data with either of these features, just yet.

 

Microsoft publicly announced Storage Spaces and ReFS in early Windows 8 blog posts. Storage Spaces was of special interest to the Windows Home Server community in light of Microsoft first dropping support for Drive Extender in Windows Home Server 2011, and then completely dropping Windows Home Server, and replacing it with Windows Server 2012 Essentials. My personal interest was more geared towards expanding my home storage capacity in a cost effective and energy efficient way, without tying myself to proprietary hardware solutions.

 

I archive all my CD’s, DVD’s, and BD discs, and store the media files on a Synology DS2411+ with 12 x 3TB drives in a RAID6 volume, giving me approximately 27TB of usable storage. Seems like a lot of space, but I’ve run out of space, and I have a backlog of BD discs that need to be archived. In general I have been very happy with Synology (except for an ongoing problem with “Local UPS was plugged out” errors), and they do offer devices capable of more storage, specifically the RS2212+ with the RX1211 expansion unit offering up to 22 combined drive bays. But, at $2300 plus $1700, this is expensive, capped at 22 drives, and further ties me in with Synology. Compare that with $1400 for a Norco DS24-E or $1700 for a SansDigital ES424X6+BS 24 bay 4U storage unit, an inexpensive LSI OEM branded SAS HBA from eBay, or a LSI SAS 9207-8e if you like the real thing, connected to Windows Server 2012, running Storage Spaces and ReFS, and things look promising.

Arguable I am swapping one proprietary technology for another, but with native Windows support, I have many more choices for expansion. One could make the same argument for the use of ZFS on Linux, and if I was a Linux expert, that may have been my choice, but I’m not.

 

I tested using a SuperMicro SuperWorkstation 7047A-73, with dual Xeon E5-2660 processors and 32GB RAM. The 7047A-73 uses a X9DA7 motherboard, that includes a LSI SAS2308 6Gb/s SAS2 HBA, connected to 8 hot-swap drive bays.

For comparison with a hardware RAID solution I also tested using a LSI MegaRAID SAS 9286CV-8e 6Gb/s SAS2 RAID adapter, with the CacheCade 2.0 option, and a Norco DS12-E 12 bay SAS2 2U expander.

For drives I used Hitachi Deskstar 7K4000 4TB SATA3 desktop drives and Intel 520 series 480GB SATA3 SSD drives. I did not test with enterprise class drives, 4TB models are still excessively expensive, and defeats the purpose of cost effective home use storage.

 

I previously reported that the Windows Server 2012 and Windows 8 install will hang when trying to install on a SSD connected to the SAS2308. As such I installed Server 2012 Datacenter on an Intel 480GB SSD connected to the onboard SATA3 controller.

Windows automatically installed the drivers for the LSI SAS2308 controller.

I had to manually install the drivers for the C600 chipset RSTe controller, and as reported before, the driver works, but suffers from dyslexia.

The SAS2308 controller firmware was updated to the latest released SuperMicro v13.0.57.0.

 

Since LSI already released v14.0.0.0 firmware for their own SAS2308 based boards like the SAS 9207-8e, I asked SuperMicro support for their v14 version, and they provided me with an as yet unreleased v14.0.0.0 firmware version for test purposes. Doing a binary compare between the LSI version and the SuperMicro version, the differences appear to be limited to descriptive model numbers, and a few one byte differences that are probably configuration or default parameters. It is possible to cross-flash between some LSI and OEM adapters, but since I had a SuperMicro version of the firmware, this was not necessary.

SuperMicro publishes a v2.0.58.0 LSI driver that lists Windows 8 support, but LSI has not yet released Windows 8 or Server 2012 drivers for their own SAS2308 based products. I contacted LSI support, and their Windows 8 and Server 2012 drivers are scheduled for release in the P15 November 2012 update.

I tested the SuperMicro v14.0.0.0 firmware with the SuperMicro v2.0.58.0 driver, the SuperMicro v14.0.0.0 firmware with the Windows v2.0.55.84 driver, and the SuperMicro v2.0.58.0 driver with the SuperMicro v13.0.57.0 firmware. Any combination that included the SuperMicro v2.0.58.0 driver or the SuperMicro v14.0.0.0 firmware resulted in problems with the drives or controller not responding. The in-box Windows v2.0.55.84 driver and the released SuperMicro v13.0.57.0 firmware was the only stable combination.

Below are some screenshots of the driver versions and errors:

LSI.2.0.55.84LSI.2.0.58.0

Eventlog.Controller.ErrorEventlog.IO.RetriedEventlog.Reset.DeviceFormat.Failed

 

One of the reasons I am not yet prepared to use Storage Spaces or ReFS is because of the complete lack of decent documentation, best practice guides, or deployment recommendations. As an example, the only documentation on SSD journal drive configuration is in TechNet forum post from a Microsoft employee, requiring the use of PowerShell, and even then there is no mention of scaling or size ratio requirements. Yes, the actual PowerShell commandlet parameters are documented on MSDN, but not the use or the meaning.

PowerShell is very powerful and Server 2012 is completely manageable using PowerShell, but an appeal of Windows has always been the management user interface, especially important for adoption by SMB’s that do not have a dedicated IT staff. With Windows Home Server being replaced by Windows Server 2012 Essentials, the lack of storage management via the UI will require regular users to become PowerShell experts, or maybe Microsoft anticipates that configuration UI’s will be developed by hardware OEM’s deploying Windows Storage Server 2012 or Windows Server 2012 Essentials based systems.

My feeling is that Storage Spaces will be one of those technologies that matures and becomes generally usable after one or two releases or service packs post the initial release.

 

I tested disk performance using ATTO Disk Benchmark 2.47, and CrystalDiskMark 3.01c.

I ran each test twice, back to back, and report the average. I realize two runs are not statistically significant, but with just two runs it took several days to complete the testing in between regular work activities. I opted to only publish the CrystalDiskMark data as the ATTO Disk Benchmark results varied greatly between runs, while the CrystalDiskMark results were consistent.

Consider the values useful for relative comparison under my test conditions, but not useful for absolute comparison with other systems.

 

Before we get to the results, a word on the tests.

The JBOD tests were performed using the C600 SATA3 controller.
The Simple, Mirror, Triple, and RAID0 tests were performed using the SAS 2308 SAS2 controller.
The Parity, RAID5, RAID6, and CacheCade tests were performed using the SAS 9286CV-8e controller.

The Simple test created a simple storage pool.
The Mirror test created a 2-way mirrored storage pool.
The Triple test created a 3-way mirrored storage pool.
The Parity test created a parity storage pool.
The Journal test created a parity storage pool, with SSD drives used for the journal disks.
The CacheCade test created RAID sets, with SSD drives used for caching.

 

As I mentioned earlier, there is next to no documentation on how to use Storage Spaces. In order to use SSD drives as journal drives, I followed information provided in a TechNet forum post.

Create the parity storage pool using PowerShell or the GUI. Then associate the SSD drives as journal drives with the pool.

Windows PowerShell
Copyright (C) 2012 Microsoft Corporation. All rights reserved.

PS C:\Users\Administrator> Get-PhysicalDisk -CanPool $True

FriendlyName CanPool OperationalStatus HealthStatus Usage Size
------------ ------- ----------------- ------------ ----- ----
PhysicalDisk4 True OK Healthy Auto-Select 447.13 GB
PhysicalDisk5 True OK Healthy Auto-Select 447.13 GB

PS C:\Users\Administrator> $PDToAdd = Get-PhysicalDisk -CanPool $True
PS C:\Users\Administrator>
PS C:\Users\Administrator> Add-PhysicalDisk -StoragePoolFriendlyName "Pool" -PhysicalDisks $PDToAdd -Usage Journal
PS C:\Users\Administrator>
PS C:\Users\Administrator>
PS C:\Users\Administrator> Get-VirtualDisk

FriendlyName ResiliencySettingNa OperationalStatus HealthStatus IsManualAttach Size
me
------------ ------------------- ----------------- ------------ -------------- ----
Pool Parity OK Healthy False 18.18 TB

PS C:\Users\Administrator> Get-PhysicalDisk

FriendlyName CanPool OperationalStatus HealthStatus Usage Size
------------ ------- ----------------- ------------ ----- ----
PhysicalDisk0 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk1 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk2 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk3 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk4 False OK Healthy Journal 446.5 GB
PhysicalDisk5 False OK Healthy Journal 446.5 GB
PhysicalDisk6 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk7 False OK Healthy Auto-Select 3.64 TB
PhysicalDisk8 False OK Healthy Auto-Select 447.13 GB
PhysicalDisk10 False OK Healthy Auto-Select 14.9 GB

PS C:\Users\Administrator>

I initially added the journal drives after the virtual drive was already created, but that would not use the journal drives. I had to delete the virtual drive, recreate it, and then the journal drives kicked in. There must be some way to manage this after virtual drives already exist, but again, no documentation.

 

In order to test Storage Spaces using the SAS 9286CV-8e RAID controller I had to switch it to JBOD mode using the commandline MegaCli utility.


D:\Install>MegaCli64.exe AdpSetProp EnableJBOD 1 a0

Adapter 0: Set JBOD to Enable success.

Exit Code: 0x00

D:\Install>MegaCli64.exe AdpSetProp EnableJBOD 0 a0

Adapter 0: Set JBOD to Disable success.

Exit Code: 0x00

D:\Install>

 

The RAID and CacheCade disk sets were created using the LSI MegaRAID Storage Manager GUI utility.

 

Below is a summary of the throughput results:

ReadWriteKBPS

ReadWriteIOPS

 

Not surprisingly the SSD drives had very good scores all around for JBOD, Simple, and RAID0. I only had two drives to test with, but I expect more drives to further improve performance.

The Simple, Mirror, and Triple test results speak for themselves, performance halving, and halving again.

The Parity test shows good read performance, and bad write performance. The write performance approaches that of a single disk.

The Parity with SSD Journal disks shows about the same read performance as without journal disks, and the write performance double that of a single disk.

The RAID0 and Simple throughput results are close, but the RAID0 write IOPS doubling that of the Simple volume.

The RAID5 and RAID6 read performance is close to Parity, but the write performance almost ten fold that of Parity. It appears that the SLI card writes to all drives in parallel, while Storage Spaces parity writes to one drive only.

The CacheCade read and write performance is less than without CacheCade, but the IOPS ten fold higher.

The ReFS performance is about 30% less than the equivalent NTFS performance.

 

 

Until Storage Spaces gets thoroughly documented and improves performance, I’m sticking with hardware RAID solutions.

XBMC for Linux on Pivos XIOS DS

Pivos released a XBMC build for Linux, and I tried it out.

The Pivos XIOS DS is very small (less than 5” x 5” x 1”) HTPC supporting hardware accelerated 1080p video and HD audio playback. The XIOS DS supports XBMC for Android, and XBMC for Linux, with native hardware acceleration. I reviewed the Android port of XBMC in a previous post.

The XIOS DS is available for $115 at Amazon, placing it, price wise, between the $98 Roku 2 XS and the $178 Boxee Box.

 

I downloaded the 09/07/12 firmware release, and installed it using the system update procedure; extract update.img to MicroSD, hold reset button on back of unit, plug in power, release reset button when update screen displays.

Firmware.Update.Reset

Firmware.Update.Linux

 

XBMC launched immediately on reboot, very similar to the XBMC for Linux OpenELEC experience.

XIOS.XBMC

XIOS.XBMC.1

XBMC.System

 

A quick zoom adjustment and the UI fits on the screen without the need to adjust resolution.

XBMC.Zoom

 

Unlike the Android version where I had to use a mouse and keyboard, I could use the included IR remote to perform all operations. And unlike the Android version, where I had to create special guest access SMB shares because NFS was not supported, the Linux version supported NFS shares with no problems.

I did encounter the same problem as current OpenELEC builds, where some addons are reported as broken in the repository, but as with OpenELEC, this did not prevent movie and series media from being correctly identified, or played.

 

I tested a variety of media formats, all in MKV containers, and all played without issue. I did not test DTS, DTS-HD, AC3, and TrueHD passthrough, as this build of XBMC is based on v11 Eden that does not support HD audio (included in the unreleased v12 Frodo), and I had the box directly connected to a television over HDMI, so all audio was downmixed to two channels.

 

All in all the Linux port of XBMC on the XIOS DS worked much better than the Android port, but as the Android port is classified as Alpha and the Linux port classified as Beta, that is expected.

The XIOS DS running Linux XBMC is not up to Boxee Box standards yet, but it may be a contender.

Koubachi Wi-Fi Plant Sensor

Last week several tech and gadget news outlets reported that the Koubachi Wi-Fi Plant Sensor has been released and is available for order (Amazon or SmartHome). I remembered reading about this device some time ago, and I decided to try it out.

Our houseplants are under my care, and they are generally happy and healthy. A simple water moisture indicator would not get my attention, and certainly would not be worthy of a place in my collection of semi-useful/useless Wi-Fi enabled devices (of which the Nest Learning Thermostat is the most useful), but the Koubachi promises more than just moisture monitoring:

Thanks to the unique Plant Care Engine (PCE), Koubachi is able to advise you about everything your plant needs: water, fertilizer, hu­mi­di­ty, temperature and light! Koubachi not only tells you WHEN to care for your plants, but also gives you specific instructions HOW.

 

The device is about the size and shape of a golf club driver head:

Koubachi.Box Koubachi.Battery

 

The installation and configuration process is interesting. There is only one button on the device, no USB plugs for direct configuration. In order to configure the device to connect to your Wi-Fi network, you first place it in Ad-Hoc mode, then directly connect to it using your computer Wi-Fi adapter, then access the device settings using a web browser, and configure the Wi-Fi settings, after which the device will connect to your home network.

Below are screenshots of the configuration process, starting with online account creation:

Create.AccountSetup.1Setup.2Setup.3Setup.4Setup.5Setup.6Setup.7Setup.8Setup.9Setup.10Setup.11Setup.12

 

At this point everything appeared to be setup and working, except the “To my plants” button was not working. On clicking the “Plants” link, I got to a screen where I can add my first plant, but nothing happened when I clicked on or dragged the pot icon. I tried using Chrome and Internet Explorer, same thing.

The following day I logged in from the office, and now the plants link worked and dragging the pot to the canvas let me create my first plant. Maybe the sensor had to check in with the backend before the backend allowed me to add a plant?

After adding the plant to the canvas, you select the type of plant and pot from the online database. Only 538 types seems a bit limited, but my corn plant was easy enough to find.

Below are screenshots of the plant selection process:

Plants.8Plants.2Plants.3Plants.4Plants.5Plants.6Plants.7

 

After adding the plant, you have to associate the sensor with the plant.  But, that required that I press the button on the sensor, and since I was in the office, I had to wait until the next day at home to continue the setup.

 

Once configured, the sensor enters a calibration period that will take about a week. In the mean time it is displaying semi-interesting information:

Info.1Info.2Info.3Info.4

 

The iPhone app shows similar information to the website, and includes push notifications of events. So far nothing exciting has happened, no email notifications, no push notification, I guess I’ll have to wait until the sensor completes the calibration procedure, or until I sacrifice a plant for the sake of curiosity.

In the mean time I’ll install some more sensors in the other house plants. I was planning on installing sensors in the patio plants, but this sensor is apparently not quite rain proof, and an outdoor sensor has been announced, to be released in October.

 

Oh, and in case you were concerned about effect of the Wi-Fi radiation on your plants, there is a FAQ entry for that Smile

Does the Wi-Fi radiation affect my plant?
No. According to the current state of knowledge there is neither a positive nor a negative effect of Wi-Fi radiation on plants. Note that the Koubachi Wi-Fi Plant Sensor usually transmits the data only once a day and that the transmission lasts only ca. 5 seconds. Therefore, the radiation exposure is many times smaller compared to other wireless devices.

Dyslexic Intel RSTe Driver

I encounter one problem after another running Windows 8 and Server 2012 on the dual Xeon E5 Intel C600 chipset based SuperMicro 7047A-T and 7047A-73 SuperWorkstation machines. I will say that this is really not representative of my Windows 8 experience in general, as all other machines I installed on worked fine with the in-box drivers.

The C602 includes the Intel Storage Controller Unit (SCU) SATA / SAS controller. Windows 8 and Server 2012 do not include in-box drivers for the SCU. The SCU drivers are part of the Intel Rapid Storage Technology Enterprise (RSTe) driver set. Note that the RSTe and RST drivers are different and not compatible with one another. When you install the full RSTe package, it includes SCU drivers for the SCU RAID controller, AHCI drivers for the SATA controller, and the Windows RST management application.

A clean install of Windows 8 will use the in-box drivers for the SATA controller. In the image below you can see the Intel 520 Series 480GB SSD drive show up with the correct model number:

Device.Manager.Win8

After installing RSTe (3.2.0.1132, 3.2.0.1134), the 4TB Hitachi drives attached to the SCU show up, but the model numbers of the drives, including the SSD drive attached to the SATA port, are now messed up:

Device.Manager.RSTe

The drive hardware identifiers are correct, but the friendly name is not:

Intel.SSD.Hardware

Intel.SSD.Friendly

It appears that the text BYTE’s are WORD swapped, i.e. ABCD becomes BADC.

The driver is also not functional, attempting to create a storage spaces pool using the Hitachi drives hangs forever, with no drive activity, requiring a hard power cycle:

Storage.Pool

And lastly, the Intel SSD Toolbox 3.0.3 is not compatible with Windows 8:

SSD.Toolbox

The clock is ticking for Windows Server 2012 (4 September, 1 day left) and Windows 8 (26 October, 7 weeks left) general availability, I can only hope compatible drivers, firmware, and utilities are forthcoming.

 

[Update: 4 September 2012]
SuperMicro posted updated RSTe drivers (package v3.5.0.1101, driver v3.5.0.1096). This driver set resolves the hang during storage space creation, but the drive names are still messed up.

Windows 8 VIDEO_TDR_FAILURE Madness

I finally figured out why I kept on getting VIDEO_TDR_FAILURE BSOD’s when installing Windows 8 on my SuperMicro workstations. It turns out that the problem goes away when I use a PCIe slot associated with CPU #1, instead of a slot associated with CPU #2.

Some history on my adventures with Windows 8 and SuperMicro SuperWorkstations:
I got ACPI_BIOS_ERROR BSOD’s while installing Windows 8, SuperMicro provided a Beta BIOS that resolved the problem.
The Windows 8 install hangs if installing to a SSD drive on a LSI 2308 SAS controller, that issue is still unresolved, but can be worked around by connecting the SSD to the Intel SATA controller.
I got VIDEO_TDR_ERROR BSOD’s while installing Windows 8 with a NVidia Quadro 5000 graphic card, same with an ATI FirePro V7900 or a NVidia GeForce GTX 680 or an ATI HD 7970. And this post is about resolving that problem.

 

SuperMicro released v1.0a BIOS updates for the X9DAi and X9DA7 motherboards used in the 7470A-T and 7470A-73 SuperWorkstations. I was hoping this will resolve the VIDEO_TDR_FAILURE BSOD’s, but no.

The X9DA7 BIOS updated without issue, but the X9DAi update reported an error at the end of the update process; “Error when sending Enable Message to ME”.

I contacted SuperMicro support, and they asked me to make sure that there is no jumper on JPME1. There is no mention of JPME1 in the motherboard manual, but it is located next to JIPMB1, next to PCIe slot #1. The header had a jumper on pins 2 and 3, where the same header on the X9DA7 motherboard had a jumper between 1 and 2. I removed the jumper, and the BIOS update succeeded.

JPME1

 

Unlike the ACPI_BIOS_ERROR BSOD that happens during the WinPE phase of the install, the VIDEO_TDR_FAILURE BSOD happens on the first boot after the install, during the hardware detection and driver install phase. This means that the technique I used to kernel debug the initial boot phase will not work, as the second boot is using the BCD already deployed to the target hard drive. I had to modify the BCD of the already installed image, prior to the install continuing after the reboot.

 

I tested many permutations of graphic cards and configurations, and it quickly became very annoying to have to type my Win8 product key every single time I boot and install. To avoid this I created configuration files in the sources directory on the install media, and this bypassed the key question. You can read more about the meaning of the file contents here:

EI.cfg:

[EditionID]
Professional
[Channel]
Retail
[VL]
0

PID.txt:

[PID]
Value=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

 

To modify the BCD of the installed image, and be able to easily repeat the second phase of install testing, I installed a second hard drive, and deployed WinPE to the second drive. By using F11 during boot to choose the boot drive, I could select booting from the second drive at any time.

 

I have a variety WinPE v3 (Win7) based utility images, and I updated them to use WinPE v4 (Win8). In the process I lost the boot menu, and the first image in the menu automatically started booting. After some trial and error, I found the bootmenupolicy BCD option, and when set to legacy mode, the old style menu is back:

bcdedit /set {default} bootmenupolicy legacy

 

I installed Win8 on the primary drive, and during the reboot, instead of booting to the installed Win8 drive, I used F11 and booted to my secondary WinPE drive. From WinPE I modified the boot BCD to enable kernel debugging over the network:

bcdedit -store c:\boot\bcd /set {default} nocrashautoreboot yes
bcdedit -store c:\boot\bcd /set {default} debugtype net
bcdedit -store c:\boot\bcd /set {default} hostip 3232235876
bcdedit -store c:\boot\bcd /set {default} port 50000
bcdedit -store c:\boot\bcd /set {default} key my.secret.debug.key
bcdedit -store c:\boot\bcd /debug {default} yes

This is equivalent to:

bcdedit /dbgsettings net host:192.168.1.100 port:50000 key:my.secret.debug.key

But unlike the dbgsettings command, this allows me to specify a BCD store. Also note that the IP address is stored as a single numeric value instead of the dotted IP format.

 

While still in WinPE, I captured the state of the primary Win8 drive by making a drive image using Symantec Ghost, the real Ghost, currently sold as Symantec Ghost Solution Suite, not the same named but volume snapshot based Norton Ghost or Symantec System Recovery. By saving a drive image, I can easily change hardware or configurations, test the install starting at the second phase, reboot to the secondary WinPE drive using F11, restore the entire drive image, and try again, while leaving the kernel debug options intact.

 

I tested with following hardware configurations in various permutations:

 

With the kernel debugger attached, I captured the following crash details in WinDbg for NVidia based cards:

VIDEO_TDR_FAILURE (116)
Attempt to reset the display driver and recover from timeout failed.
Arguments:
Arg1: fffffa80211cd010, Optional pointer to internal TDR recovery context (TDR_RECOVERY_CONTEXT).
Arg2: fffff8800782d0d8, The pointer into responsible device driver module (e.g. owner tag).
Arg3: 0000000000000000, Optional error code (NTSTATUS) of the last failed operation.
Arg4: 0000000000000002, Optional internal context dependent data.

Debugging Details:
------------------

FAULTING_IP:
nvlddmkm+1ae0d8
fffff880`0782d0d8 4055 push rbp

DEFAULT_BUCKET_ID: GRAPHICS_DRIVER_TDR_FAULT

BUGCHECK_STR: 0x116

PROCESS_NAME: System

CURRENT_IRQL: 0

STACK_TEXT:
fffff880`12c76078 fffff801`66fef0ea : 00000000`00000000 00000000`00000116 fffff880`12c761e0 fffff801`66f734b8 : nt!DbgBreakPointWithStatus
fffff880`12c76080 fffff801`66fee742 : 00000000`00000003 fffff880`12c761e0 fffff801`66f73e90 00000000`00000116 : nt!KiBugCheckDebugBreak+0x12
fffff880`12c760e0 fffff801`66ef4144 : fffffa80`2094b100 fffff880`021ee9c0 fffffa80`1f54e400 00000000`00000000 : nt!KeBugCheck2+0x79f
fffff880`12c76800 fffff880`04b33dcb : 00000000`00000116 fffffa80`211cd010 fffff880`0782d0d8 00000000`00000000 : nt!KeBugCheckEx+0x104
fffff880`12c76840 fffff880`04b32518 : fffff880`0782d0d8 fffffa80`211cd010 fffff880`12c76949 00000000`000000c7 : dxgkrnl!TdrBugcheckOnTimeout+0xef
fffff880`12c76880 fffff880`04a1e608 : fffffa80`211cd010 fffff880`12c76949 00000000`00000000 00000000`00000002 : dxgkrnl!TdrIsRecoveryRequired+0x168
fffff880`12c768b0 fffff880`04a4d539 : 00000000`00000000 fffff780`00000320 00000000`00000000 fffffa80`1f54e400 : dxgmms1!VidSchiReportHwHang+0x438
fffff880`12c769b0 fffff880`04a4ba49 : fffffa80`00000002 fffffa80`1f54e400 fffffa80`1f54e840 fffffa80`1f54e840 : dxgmms1!VidSchiCheckHwProgress+0xe5
fffff880`12c76a00 fffff880`04a16fe5 : ffffffff`ff676980 00000000`00000001 fffff880`12c76b69 fffffa80`1f54e400 : dxgmms1!VidSchiWaitForSchedulerEvents+0x20d
fffff880`12c76aa0 fffff880`04a4b646 : 00000000`00000000 00000000`0000000f fffffa80`1f54e400 fffffa80`1f54e400 : dxgmms1!VidSchiScheduleCommandToRun+0x289
fffff880`12c76bd0 fffff801`66e9b521 : fffffa80`1f5abb00 fffffa80`1f54e400 fffff880`03b01140 00000000`06a21e1e : dxgmms1!VidSchiWorkerThread+0xca
fffff880`12c76c10 fffff801`66ed9dd6 : fffff880`03af5180 fffffa80`1f5abb00 fffff880`03b01140 fffffa80`19aac040 : nt!PspSystemThreadStartup+0x59
fffff880`12c76c60 00000000`00000000 : fffff880`12c77000 fffff880`12c71000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

STACK_COMMAND: .bugcheck ; kb

FOLLOWUP_IP:
nvlddmkm+1ae0d8
fffff880`0782d0d8 4055 push rbp

SYMBOL_NAME: nvlddmkm+1ae0d8

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nvlddmkm

IMAGE_NAME: nvlddmkm.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4fdf93d7

FAILURE_BUCKET_ID: 0x116_IMAGE_nvlddmkm.sys

BUCKET_ID: 0x116_IMAGE_nvlddmkm.sys

 

With the kernel debugger attached, I captured the following crash details in WinDbg for ATI based cards:

VIDEO_TDR_FAILURE (116)
Attempt to reset the display driver and recover from timeout failed.
Arguments:
Arg1: fffffa801ed114d0, Optional pointer to internal TDR recovery context (TDR_RECOVERY_CONTEXT).
Arg2: fffff8800725cefc, The pointer into responsible device driver module (e.g. owner tag).
Arg3: 0000000000000000, Optional error code (NTSTATUS) of the last failed operation.
Arg4: 000000000000000d, Optional internal context dependent data.

Debugging Details:
------------------

FAULTING_IP:
atikmpag+8efc
fffff880`0725cefc 4055 push rbp

DEFAULT_BUCKET_ID: GRAPHICS_DRIVER_TDR_FAULT

BUGCHECK_STR: 0x116

PROCESS_NAME: System

CURRENT_IRQL: 0

STACK_TEXT:
fffff880`06fa9ee8 fffff803`e6ff20ea : 00000000`00000000 00000000`00000116 fffff880`06faa050 fffff803`e6f764b8 : nt!DbgBreakPointWithStatus
fffff880`06fa9ef0 fffff803`e6ff1742 : 00000000`00000003 fffff880`06faa050 fffff803`e6f76e90 00000000`00000116 : nt!KiBugCheckDebugBreak+0x12
fffff880`06fa9f50 fffff803`e6ef7144 : fffffa80`1e2df4e0 fffff880`020b99c0 fffffa80`1d31f010 00000000`00000000 : nt!KeBugCheck2+0x79f
fffff880`06faa670 fffff880`04d31dcb : 00000000`00000116 fffffa80`1ed114d0 fffff880`0725cefc 00000000`00000000 : nt!KeBugCheckEx+0x104
fffff880`06faa6b0 fffff880`04d30548 : fffff880`0725cefc fffffa80`1ed114d0 fffff880`06faa7b9 00000000`00000180 : dxgkrnl!TdrBugcheckOnTimeout+0xef
fffff880`06faa6f0 fffff880`04c11608 : fffffa80`1ed114d0 fffff880`06faa7b9 00000000`0000000f fffffa80`1d31f8f8 : dxgkrnl!TdrIsRecoveryRequired+0x198
fffff880`06faa720 fffff880`04c459f9 : 00000000`00000001 fffff880`06faa8a0 fffff880`06faa920 00000000`00000000 : dxgmms1!VidSchiReportHwHang+0x438
fffff880`06faa820 fffff880`04c3ff72 : fffffa80`1d31f010 fffff780`00000320 fffffa80`1d31f770 fffffa80`1d31f010 : dxgmms1!VidSchWaitForCompletionEvent+0x411
fffff880`06faa8e0 fffff880`04c4206c : fffffa80`1d31f010 fffffa80`1d31f450 fffffa80`1d31f450 00000000`00000000 : dxgmms1!VidSchiWaitForEmptyHwQueue+0x9a
fffff880`06faa9d0 fffff880`04c3ea85 : 00000000`00000000 fffffa80`1d31f010 fffffa80`1d31f450 00000000`00000000 : dxgmms1!VidSchiSuspend+0x74
fffff880`06faaa00 fffff880`04c09fe5 : ffffffff`ff676980 00000000`00000001 fffff880`06faab69 fffffa80`1d31f010 : dxgmms1!VidSchiWaitForSchedulerEvents+0x249
fffff880`06faaaa0 fffff880`04c3e646 : 00000000`00000000 fffffa80`1d585660 fffffa80`1d44d7f0 fffffa80`1d31f010 : dxgmms1!VidSchiScheduleCommandToRun+0x289
fffff880`06faabd0 fffff803`e6e9e521 : fffffa80`1d6b9b00 fffffa80`1d31f010 fffff880`03932140 00000000`04d91ecb : dxgmms1!VidSchiWorkerThread+0xca
fffff880`06faac10 fffff803`e6edcdd6 : fffff880`03926180 fffffa80`1d6b9b00 fffff880`03932140 fffffa80`19ac7500 : nt!PspSystemThreadStartup+0x59
fffff880`06faac60 00000000`00000000 : fffff880`06fab000 fffff880`06fa5000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

STACK_COMMAND: .bugcheck ; kb

FOLLOWUP_IP:
atikmpag+8efc
fffff880`0725cefc 4055 push rbp

SYMBOL_NAME: atikmpag+8efc

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: atikmpag

IMAGE_NAME: atikmpag.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4fdf9279

FAILURE_BUCKET_ID: 0x116_IMAGE_atikmpag.sys

BUCKET_ID: 0x116_IMAGE_atikmpag.sys

 

This was not really helping me much, and I decided to repeat the tests but use the checked build of Windows 8 to help troubleshoot.

With the kernel debugger attached, I captured the following ASSERT during the boot:

Windows 8 Kernel Version 9200 MP (1 procs) Checked x64
Built by: 9200.16384.amd64chk.win8_rtm.120725-1247
Machine Name:
Kernel base = 0xfffff802`0e01d000 PsLoadedModuleList = 0xfffff802`0e760ac0
System Uptime: 0 days 0:00:06.228 (checked kernels begin at 49 days)
Assertion: The BIOS has reported inconsistent resources (_CRS). Please upgrade your BIOS.
ACPI!PnpBiosGetDeviceResourceList+0x15e:
fffff880`012c3c2a cd2c int 2Ch
...
Unknown bugcheck code (0)
Unknown bugcheck description
Arguments:
Arg1: 0000000000000000
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000

Debugging Details:
------------------

PROCESS_NAME: System

FAULTING_IP:
ACPI!PnpBiosGetDeviceResourceList+15e
fffff880`012c3c2a cd2c int 2Ch

ERROR_CODE: (NTSTATUS) 0xc0000420 - An assertion failure has occurred.

EXCEPTION_CODE: (NTSTATUS) 0xc0000420 - An assertion failure has occurred.

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: 0x0

CURRENT_IRQL: 0

LOCK_ADDRESS: fffff8020e7c5d60 -- (!locks fffff8020e7c5d60)

Resource @ nt!PiEngineLock (0xfffff8020e7c5d60) Exclusively owned
Threads: fffffa8019a36040-01<*>
1 total locks, 1 locks currently held

PNP_TRIAGE:
Lock address : 0xfffff8020e7c5d60
Thread Count : 1
Thread address: 0xfffffa8019a36040
Thread wait : 0x105eccd4

LAST_CONTROL_TRANSFER: from fffff880012b736f to fffff880012c3c2a

STACK_TEXT:
fffff880`009b4b30 fffff880`012b736f : fffffa80`23a9e900 fffff880`012a7e01 fffff880`009b4c08 fffff880`012a7e70 : ACPI!PnpBiosGetDeviceResourceList+0x15e
fffff880`009b4bd0 fffff880`0125acba : fffffa80`23a9e900 fffffa80`19ac54c0 fffff880`012a7e70 fffffa80`1f477010 : ACPI!ACPIBusIrpQueryResourceRequirements+0x8b
fffff880`009b4c50 fffff802`0e91b6a4 : fffffa80`23a9e900 fffffa80`19ac54c0 fffff880`009b4db0 fffffa80`23a9e900 : ACPI!ACPIDispatchIrp+0x2a6
fffff880`009b4cf0 fffff802`0e91cd1b : fffffa80`23a9e900 fffff880`009b4db0 00000001`c00000bb 00000000`00000000 : nt!IopSynchronousCall+0x10c
fffff880`009b4d80 fffff802`0e915bdb : fffffa80`23a9e900 fffff880`009b4e50 fffffa80`23a4f850 00000000`0000001e : nt!PpIrpQueryResourceRequirements+0x5f
fffff880`009b4e10 fffff802`0e91748d : fffffa80`23a9b8e0 00000000`00000000 ffffffff`80000218 fffffa80`23a9b8e0 : nt!PiQueryResourceRequirements+0x47
fffff880`009b4ea0 fffff802`0e91a1f2 : fffffa80`23a9b8e0 fffffa80`23a9b8e0 00000000`00000001 00000000`00000000 : nt!PiProcessNewDeviceNode+0x159d
fffff880`009b5070 fffff802`0e08feb5 : fffffa80`19adcd20 00000000`00000000 fffff880`009b5358 00000000`00000000 : nt!PipProcessDevNodeTree+0x1fe
fffff880`009b5310 fffff802`0e08fb59 : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa80`37e19cc0 : nt!PnpDeviceActionWorker+0x345
fffff880`009b53d0 fffff802`0ed4010d : 00000000`00000000 fffff8a0`00000007 fffff8a0`00f08c00 00000000`00000000 : nt!PnpRequestDeviceAction+0x2ed
fffff880`009b5420 fffff802`0ed3b39d : fffff802`0d536800 fffff802`0e7c83c0 00000000`00000006 fffff802`0d536800 : nt!IopInitializeBootDrivers+0x905
fffff880`009b5650 fffff802`0ed2deb5 : fffff802`0d536800 00000000`00000000 fffff802`0d536800 fffff802`0d51ebf0 : nt!IoInitSystem+0xb5d
fffff880`009b59b0 fffff802`0e82d013 : fffff802`0d536800 fffffa80`19a36040 00000000`00000000 fffffa80`19ab3040 : nt!Phase1InitializationDiscard+0x1899
fffff880`009b5bc0 fffff802`0e1b289e : fffff802`0d536800 fffff802`0d536800 00000000`00000000 00000000`00000000 : nt!Phase1Initialization+0x13
fffff880`009b5bf0 fffff802`0e24ef96 : fffff802`0e82d000 fffff802`0d536800 fffff802`0e6c6180 00000000`f8ffffff : nt!PspSystemThreadStartup+0x1a2
fffff880`009b5c60 00000000`00000000 : fffff880`009b6000 fffff880`009b0000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
ACPI!PnpBiosGetDeviceResourceList+15e
fffff880`012c3c2a cd2c int 2Ch

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: ACPI!PnpBiosGetDeviceResourceList+15e

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ACPI

IMAGE_NAME: ACPI.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 50109dd0

BUCKET_ID_FUNC_OFFSET: 15e

FAILURE_BUCKET_ID: 0x0_ACPI!PnpBiosGetDeviceResourceList

BUCKET_ID: 0x0_ACPI!PnpBiosGetDeviceResourceList

 

This is interesting, the kernel ASSERT’s on a problem reported by the BIOS.

I contacted SuperMicro support, they said they will investigate the BIOS failure, and they suggested I try to use PCIe slot #3 instead of slot #5. The motherboard manual mentions that slots #1, #2, and #3 are to be used if CPU #1 is installed, and slots #4, #5, and #6 to be used only if CPU #2 is installed.

PCIe

I have both processors installed, so not using the more conveniently located slot #5 never came to mind. I moved the graphic card to CPU #1 slot #3, and voila, install succeeded and Windows 8 was up and running!

 

I repeated the checked build test with the graphic card in slot #3, and the same BIOS ASSERT error was reported, so the BIOS ASSERT seems to be unrelated to the ACPI_TDR_FAILURE error.

 

This was a very frustrating problem, and I still don’t understand the root cause, but I am happy to be able to finally switch both workstations to Windows 8.