Ok, it is another post from the network engineering voice trenches. We have been working the past 19 months (longest project ever) with a major carrier to get their SIP trunking solution in place to eventually replace our tons of standalone PRI and NFAS T-1 Circuits. We have had more than our fair share of problems along the way and, I promise, some day I plan to share some of our horror stories experiences, but I will save that for later.

First, a little background. Our reasons for invetigating SIP trunking was not one of cost savings — which is what most carriers try to push when they come to talk with you — but rather one of redundancy. Redundancy for our high value phone number blocks. These not only include the toll-free numbers that route into our contact center for our customers (which are already very redundant thanks to advanced feature capabilities not available on normal PRIs), but more so for our DID (or DDI if you prefer) phone numbers that power outside communications for our back office employees.

Our company has about 11,000 DID numbers all over globe. Most of these are delivered on PRI circuits that are anchored to some LEC’s CO someplace near our various offices. We’ve tried to do a pretty good job of having the carriers provision last mile diversity, but what happens if that local CO has an “event”? What happens to those numbers? Well, they go away temporarily. How does SIP help us? Well if you pick-up those numbers and port them into a SIP provider’s network or “cloud” (I hate that term), we can begin to engineer some redundancy, gain some economies of scale, and if we do it right, hopefully save a little bit of money in the process.

So how did I break your voicemail again? In the process of provisioning these new SIP circuits our carrier required us to attach a Diversion Header (DH) to all outbound calls due to our requirement of the flexibility to control our outbound calling party number. Why the diversion header? First, don’t get me started on why on earth this carrier chose to use the Diversion Header (DH) field to accomplish this. The PAI field, I could buy, but DH is a bad idea and voice folks might see the foreshadowing here. I argued against the use of DH for hours with various folks at the carrier about this, but they all droned the same line, “this is the way it is, there is nothing I can do about it.”

Why did the carrier require us to use DH to start with? We have a requirement that on occasion, we want to send a valid calling party number that is not provisioned on the SIP trunks to say a toll-free number if a contact center agent was calling back a customer. The reason for this is if one of our customers misses the call and tries to call the number back, we don’t want those calls to route to the contact center agent’s DID extension, but rather flow through our normal contact center infrastructure so we can tier and handle that customer as best we can — it’s all about customer experience. It’s possible that original calling agent has gone to lunch or is on a different “skill” when the customer calls back. Anyway, in order for us to do this, we had to send the DH populated with a number that was on the service, then the “FROM” SIP field could be whatever we wanted and what would display to the called party. What did the config look like? I used a SIP Profile on the CUBE to make this happen (remember you need to “apply” this SIP profile to your dial-peer(s) or globally in your ‘voice service voip’ ‘sip’ config section:

voice class sip-profiles 1
 request INVITE sip-header Diversion add "Diversion: <sip:+12125551212@10.1.1.1>"

Initially there was no major issues as we did testing on the SIP trunks. One thing that we noticed right away is that the DH value was leaving the SIP Carriers network, which they initially informed us would not happen. On BlackBerry devices, you would see a “Redirected by” field show up on the screen with the DH phone number under the larger Calling Party number. Only BlackBerry phones displayed this additional information but we didn’t think much of it. It didn’t seem to be causing any other issues until I was working with Cisco TAC one day on a seperate issue. For the preceding few weeks, everyone in the network group was calling outbound over these new SIP trunks to “kick the tires” before we redirected all enterprise outbound calling over them. When I called TAC, I could not get the engineers voicemail box, it would go to Cisco’s generic auto attendant. At the time, I knew that Cisco was in the process of changing over from Unity to Unity Connection internally so I thought maybe the guy just hadn’t setup his mailbox yet or something. First time I let it slide, the next time it happened with another TAC engineer I started to get suspicious; then it hit me. That dang Diversion Header! If I called the engineer over our legacy PSTN trunks, I got his voicemail message just fine. I was breaking their voicemail coverage!

At first I was irate at our carrier (which happens frequently when you are in the voice realm of networking) but after I calmed down I did more testing to “ammo up” my communication back to the carrier on this. I placed an outbound call over the SIP trunks back into our unported DID blocks with “debug isdn q931” enabled. Sure enough, there was that dang DH coming ingress to my network even on an ISDN circuit! So what exactly was going on?

The Diversion Header field was being used at our carriers request, but that field is also used in SIP to tell voicemail systems which voicemail box greeting to play. Depending on how you have your voicemail system configured it can look at the first or the last diversion header, if there is more than one. Since my carrier’s requested DH was populated and was leaking into the called party, when their phone system added another diversion header their voicemail system it ignored their DH and used mine. By default with Unity Connection it will use the first diversion header to play the original called party voicemail box — again, this can be changed if desired. If the DH is populated with data specific to my carrier and our install, it likely doesn’t mean anything to your voicemail system — thus the generic auto attendant gets played to the caller. So now what?

Well we removed the DH on the CUBEs and restored order in the world. Ultimately, the carrier provided a bunch of PRIs (at their expense) until this issue can get fixed because my contact center agents need to send toll-free numbers as their calling party number. They continue to use these TDM circuits while the rest of  organization is using the SIP trunks. The latest update we have is that the carrier will have this fixed in June — 2013! Yet another year of waiting. In working with telco companies, you just come to expect this level of “service.”