Few things in collaboration will get people more worked up than a user experience change, especially when a contact center is involved. (Protip: do NOT anger the contact center gods). Here is a little story about firmware release 9.4(2SR1.1) (or 9-4-2SR1-1S or 9.4(2)SR1 depending on where you see the version referenced) for the Cisco 7900 series IP Phones and how it has caused grief for not only us, but other customers as well.
A Little Background
First let me begin by saying, my team and I really work hard when it comes to doing software upgrades and a CUCM upgrade is no exception. I have helped maintain the care and feeding of our production CUCM cluster for 33 upgrades now (yes we track the history of every upgrade/patch). There isn’t an UC application more important to a Cisco Collaboration deployment than CUCM, so it is important that all associated pieces and parts are working well.
Part of our evaluation of a potential CUCM software release is reviewing the new firmware and any changes associated with those releases. Generally we do not deploy standalone firmware updates to IP Phones unless there is a major issue that needs fixing or a new feature required and, quite honestly, there is more than enough stuff to do around here then to mess around with ad-hoc IP Phone firmware updates. Just say’n.
Now I still believe that it is generally safe to assume that if Cisco has checked in new default firmware into a CUCM release: 1) the code is mostly stable. 2) It has probably had a little soak time in the field. One would also hope that maybe a little bit of Quality Assurance (QA) has been done on the release. Historically this has been a fairly safe assumption. Enter firmware 9-4-2SR1-1.
The Red Headed Step-Child
At this point, I should point out that our phones were already running firmware 9-4-2-1S, so when reviewing the release notes, I wasn’t too concerned only moving up to a Service Release. Nothing jumped out in the release notes. All was good until 30 minutes after the contact center opened the following Monday and suddenly the wrath of angry contact center agents filled our trouble ticket queue with the power of a thousand burning suns. Ok, so maybe it wasn’t that dramatic, but clearly there was an issue.
The reports were basically describing the agents still being able to hear themselves talk even when not on a call. We had performed quite a bit of testing after the update and nothing odd had jumped out. However, there is usually some type of odd test case not validated during testing and this was no exception. It’s important to adapt your testing plans as you do these CUCM upgrades to make sure history doesn’t repeat itself. Our test plans has evolved for 33 upgrades now and it seems we always add something new.
Sure enough, I took a Cisco 7965 IP Phone and connected a simple wired headset and made a call. When I ended the call, the headset light was still on and I could hear myself in the headset. So knowing there was a change, I tested reverting the code to 9-4-2-1S and the issue went away.
Here are the steps to easily reproduce this issue on a Cisco 7900 Series IP Phone running 9-4-2SR1-1:
Connect up a wired headset (Doesn’t matter what brand. I tested with: VXI, Plantronics, and Sennheiser headsets)
Make a call to another phone and press the “EndCall” softkey
Headset light will stay on and if you speak you will hear yourself talking through the headset.
If you use the headset button to end the call it will disable the “hot mic” on the headset, but the side-effect is the default audio path gets returned to the handset. Sure the “hot mic” is fixed now, but with the default audio path changed, the “Auto Answer with Headset” feature no longer works. This feature is pretty common in contact centers where most prefer to use auto answer for the high volume of calls. At this point, we opted to back out the code to our previous release 9-4-2-1S which stopped the sunburn from the agents. This was going to be a fun one to track down.
“Hi ho, hi ho, it’s off to TAC I go.”
My Service Request gets initially assigned to the lovely and infamous Costa Rica TAC. The call taker, er, “Engineer”, began to take a look. I provided him the exact steps to reproduce the issue. He said he couldn’t reproduce it in his lab — liar. I will spare you the next 3 weeks of back and forth drama. Clearly this guy wasn’t testing squat because after my SR got escalated up to a team lead in the Boxborough TAC he reproduced the issue in under 5 minutes. I digress…
The crux of our issue was enhancement defect CSCus97871. This enhancement was requested by one Cisco customer earlier this year, yet the way it was implemented is, in my opinion, embarrassing and changed the default headset behavior going forward.
First of all, when you look at the public portion of this bug, it is more than a little vague. (spoiler alert, it’s currently one line). Basically the confusion came from documentation discrepancy. The end-user guides said the headset light should say “on” if the headset is the selected talk path. This behavior did change awhile back but the documentation likely didn’t get updated. So instead of submitting a documentation defect, Cisco appears to have changed the default phone behavior to match the documentation.
Now, to be fair, CSCus97871, does get called out in the Cisco Unified IP Phone 7900 Series for Firmware Release 9.4(2)SR1 but even that documentation is a little misleading. Consider this phrase from the release notes:
…Enable the feature using the Customer Support Use field, and entering CSCus97871.
Seems harmless, however, what is really in the firmware code is that you need to enter CSCus97871 in the CUCM Customer Support Use field on the device configuration to restore the original behavior. Here is the field shown in the CUCM Admin GUI on the device configuration screen:
How this will fully shake out is yet to be seen. The TAC team lead indicated he was embarrassed that this even made it out into the wild with the lack of understanding of the impact. He’s currently working with the BU and the DEs to figure out next steps. This one customer must have had a big stick to pull off this change because so far the BU is putting up a fight.
So What Now?
Here is what we know:
- Any version before and exclusive of 9.3.1SR4 –> Headset button light remains lit up
- Any version after and inclusive of 9.4.2SR1 –> Headset button light remains lit up
- Any version in between 9.3.1SR4 (inclusive) and 9.4.2SR1 (exclusive) –> Headset button light does NOT light up
The configurable default audio path feature was first integrated into 9.4.2SR1 FCS release.
The expected behavior in 9.4.2SR1 release is as follows:
- Customer Support Use = <blank> –> Headset button light remains lit up
- Customer Support Use = CSCus97871 –> Headset button light remains off
I must point out that Cisco has tried to keep end-user experience consistent across their IP Phone portfolio. I must also point out that the current default state of this firmware is different from any other IP Phone Cisco manufactures. Now there is a viable work around with the “Customer Support Use” field, but I would have used this field for the exception, not the default; however, they didn’t ask me.
I will keep you posted on any further developments.
Cracking post, and really useful!