outbound calls dropping out

[davidav]

We are suffering a lot of pain with outbound calls dropping out - happens pretty regularly but not with every call - probably about 30-50% of outbound calls.

I have no real idea why this is happening - we have tried all sorts of things to improve but generally 1 step forward, 1 or 2 steps back. Looking at a trace for a dropped call today (a normal VOIP call to a PSTN landline), there are two things that look suspicious:

 

1. We always seem to get the following "falling back to transcoding" messages when the call is initiated:

 

[7] 2010/10/19 14:55:58: Set packet length to 20

[6] 2010/10/19 14:55:58: Sending RTP for 4f3c17cb@pbx#1193819909 to 58.96.1.2:59256

[7] 2010/10/19 14:55:58: 4f3c17cb@pbx#1193819909: RTP pass-through mode

[7] 2010/10/19 14:55:58: 60f323f4-684b9d07@10.0.1.21#510adaefc6: RTP pass-through mode

[7] 2010/10/19 14:55:58: Cannot pass through on 60f323f4-684b9d07@10.0.1.21#510adaefc6, falling back to transcoding

[7] 2010/10/19 14:55:58: Cannot pass through on 4f3c17cb@pbx#1193819909, falling back to transcoding

[7] 2010/10/19 14:56:40: SIP Rx udp:10.0.1.21:5060:

 

2. Just before the call dropped out, there is the following mysterious entry in the log. Why is it trying to connect to 173.166.177.221?? A quick whois search suggests this is something to do with commcast.

 

[7] 2010/10/19 14:57:00: Last message repeated 7 times

[3] 2010/10/19 14:57:00: Could not connect to 173.166.77.221:443

[1] 2010/10/19 14:57:00: Could not send via TCP: 0 bytes

[7] 2010/10/19 14:57:03: SIP Tr udp:58.96.1.2:5060:

 

Any ideas on either of these issues?

[Vodia PBX]

We are suffering a lot of pain with outbound calls dropping out - happens pretty regularly but not with every call - probably about 30-50% of outbound calls.

 

 

I guess for this we need the SIP messages, can you turn SIP Message logging on? Transcoding does not have to be a problem. Are you using a PSTN gateway or a ITSP? 30 second timeout smells like a problem with the ACK message, maybe a problem with the routing of the SIP messages.

 

 

[7] 2010/10/19 14:57:00: Last message repeated 7 times

[3] 2010/10/19 14:57:00: Could not connect to 173.166.77.221:443

[1] 2010/10/19 14:57:00: Could not send via TCP: 0 bytes

[7] 2010/10/19 14:57:03: SIP Tr udp:58.96.1.2:5060:

 

# host license.pbxnsip.com
license.pbxnsip.com has address 173.166.77.221

 

This is the license server, obviously down right now . Not beautiful, but that should not have any effect for the calls.

[davidav]

Call logs attached for two dropped calls. Today it is happening on every single outbound call. The two logs are for calls made on each of the two trunks we have setup - same result on both.

DroppedCall1.rtf

DroppedCall2.rtf

[Vodia PBX]

Call logs attached for two dropped calls. Today it is happening on every single outbound call. The two logs are for calls made on each of the two trunks we have setup - same result on both.

 

As far as I can see the phone cancels the call:

 

2010/10/22 08:44:55: SIP Rx udp:10.0.1.20:5060:

 

CANCEL sip:0419275642@10.0.1.9:5060 SIP/2.0

Via: SIP/2.0/UDP 10.0.1.20:5060;branch=z9hG4bK-bec56812

From: "Study" <sip:10@10.0.1.9>;tag=2ee313253e138a52o0

To: "David Mobile" <sip:0419275642@10.0.1.9>

Call-ID: cb32e5e1-aab976c6@10.0.1.20

CSeq: 102 CANCEL

Max-Forwards: 70

Authorization: Digest username="10",realm="10.0.1.9",nonce="98778396f04a6faf7c8fc9510728eec9",uri="sip:0419275642@10.0.1.9:5060",algorithm=MD5,response="de8ea57e799ccaf0ead067704fc28557"\

User-Agent: Linksys/SPA942-6.1.5(a)

Content-Length: 0

 

The call starts at 2010/10/22 08:44:18 (37 seconds), that smells like another problem: The call was never connected!

 

What do we get from the service provider?

 

2010/10/22 08:44:18: SIP Rx udp:58.96.1.2:5060

SIP/2.0 100 Giving a try

Via: SIP/2.0/UDP 10.0.1.9:5060;branch=z9hG4bK-788da3e8c08e13808dd46012212dc33e;rport=38116;received=220.233.7.137

From: "Silvereye" <sip:0280905395@58.96.1.2>;tag=599761217

To: <sip:0419275642@58.96.1.2;user=phone>

Call-ID: 3b4df2ec@pbx

CSeq: 17597 INVITE

Server: Exetel Voip

Content-Length: 0

 

and then

 

2010/10/22 08:44:23: SIP Rx udp:58.96.1.2:5060

 

SIP/2.0 183 Session Progress

Via: SIP/2.0/UDP 10.0.1.9:5060;received=220.233.7.137;branch=z9hG4bK-788da3e8c08e13808dd46012212dc33e;rport=38116

Record-Route: <sip:58.96.1.2;lr=on;ftag=599761217;nat=yes>

From: "Silvereye" <sip:0280905395@58.96.1.2>;tag=599761217

To: <sip:0419275642@58.96.1.2;user=phone>;tag=as59355012

Call-ID: 3b4df2ec@pbx

CSeq: 17597 INVITE

User-Agent: ExetelVoip

Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY

Supported: replaces

Contact: <sip:0419275642@58.96.1.7:5091>

Content-Type: application/sdp

Content-Length: 303

 

v=0

o=root 1279 1279 IN IP4 58.96.1.7

s=session

c=IN IP4 58.96.1.2

t=0 0

m=audio 56164 RTP/AVP 0 8 18 101

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:18 G729/8000

a=fmtp:18 annexb=no

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-16

a=silenceSupp:off - - - -

a=ptime:20

a=sendrecv

 

No 200 Okay after that in the log. Maybe the service provider wants to be nice and dont want to start the billing period... But the PBX does not receive a 200 Ok.

 

I would say either the provider has a problem or the NAT timeout is very short and your firewall closes the UDP ports very quickly, so that the 200 Ok gets blocked. Are you able to receive inbound calls (SIP messages from the provider to the PBX after a relatively long time of port inactivity)? That would be an indidation that the firewall is not the problem.

[davidav]

I have discovered that we seem to get more success with outbound calls when I change the "Strict RTP Routing:" setting for the trunk to No. This had defaulted to Yes but I read in the Admin Guide that it recommends setting it to No. The attached GoodCall trace shows that there is more chatter immediately after the call is established - in the dropped calls we never saw all the chatter around "Clear last INVITE" and "Determine pass-through mode after receiving response".

 

Problem now though is that the price of getting outbound calls working seems to be that inbound callers now hear no ring tone.

The behaviour of inbound calls seems to be changed by changing the "Remote Party/Privacy Indication:" setting for the trunk:

- if it is set to "No Indication" (the setting we have always used) then the caller hears no ring and nothing at all once answered although the person answering can hear them talk;

- if it is set to "Remote-Party-ID" then the conversation is audible at both ends - but there is still no ring tone for the caller so they will assume that the number is incorrect/not working unless its picked up immediately.

- none of the other options work as well (!!) - with some neither party can hear each other and with others the call only rings once or twice before being dropped - and the ring tone is still inaudible to the caller in all of these settings.

 

See siplogs attached for more information.

GoodCall.rtf

InboundNoRing.rtf

[Vodia PBX]

There are two topics: Media and signalling (SIP).

 

As for media, it is usually safe to use to use loose RTP routing. The IETF defined something very NAT unfriendly with the usage of the RTP ports and usually no vendor is so strict. You can do this on trunk leven, but also for the whole system (in the admin/ports section).

 

As for signalling, yea the other big topic is on how to "say who you are" when using trunks. The conflict is the indication of the caller-ID and the other thing the indication who pays the bill. This is a big mess when it comes to SIP trunking. IMHO the RFC says that the From-header contains the caller-ID that should be displayed on the screen, and the P-Preferred-Identity (or Asserted-Identity) says who will pay the bill. Unfortunaly there are is a lot of free software out there that thinks the From is the one who pays and then other headers like Remote-Party-ID will say what the caller-ID is. It gets even worse when service providers don't use session border controllers, because then it mixes even more when the SIP proxy and the PSTN gateway use different interpretations! That's why we added this drop down to give you several choices. I know it is bad, but you have to play with this setting until it works. The next release will make it even more flexible, because this is really a problematic point with the SIP trunking today.

[davidav]

There are two topics: Media and signalling (SIP).

 

As for media, it is usually safe to use to use loose RTP routing. The IETF defined something very NAT unfriendly with the usage of the RTP ports and usually no vendor is so strict. You can do this on trunk leven, but also for the whole system (in the admin/ports section).

 

As for signalling, yea the other big topic is on how to "say who you are" when using trunks. The conflict is the indication of the caller-ID and the other thing the indication who pays the bill. This is a big mess when it comes to SIP trunking. IMHO the RFC says that the From-header contains the caller-ID that should be displayed on the screen, and the P-Preferred-Identity (or Asserted-Identity) says who will pay the bill. Unfortunaly there are is a lot of free software out there that thinks the From is the one who pays and then other headers like Remote-Party-ID will say what the caller-ID is. It gets even worse when service providers don't use session border controllers, because then it mixes even more when the SIP proxy and the PSTN gateway use different interpretations! That's why we added this drop down to give you several choices. I know it is bad, but you have to play with this setting until it works. The next release will make it even more flexible, because this is really a problematic point with the SIP trunking today.

 

The problem is that it doesn't work. I have not been able to find any combination of settings that will allow us to have inbound and outbound calls on the same trunk. Surely this is a pretty basic and common requirement?

 

It seems like we need the Strict RTP Routing setting to be ON in order for incoming callers to hear a ring tone - is this what you expect or is there some other setting we need to change to overcome this?

 

Because it seems that we need the Strict RTP Routing setting to be OFF in order to make outbound calls which don't drop out after approx 30 secs - is this what you expect or is there some other setting we need to change to overcome this?

[pbx support]

The problem is that it doesn't work. I have not been able to find any combination of settings that will allow us to have inbound and outbound calls on the same trunk. Surely this is a pretty basic and common requirement?

 

It seems like we need the Strict RTP Routing setting to be ON in order for incoming callers to hear a ring tone - is this what you expect or is there some other setting we need to change to overcome this?

 

Because it seems that we need the Strict RTP Routing setting to be OFF in order to make outbound calls which don't drop out after approx 30 secs - is this what you expect or is there some other setting we need to change to overcome this?

Looked at the log file and seems like the call is being connected on the hunt group party/agent 11. The issue may be that the caller is doing PCMA and the PBX is doing PCMU. If you think your system should be doing PCMA (mostly outside the USA), then please setup the system accordingly (Admin->System->Ports page change the order of codecs and/or the trunk page change the order of codec. This change generally does not require a restart of the PBX. But just start with a clean slate restart the PBX after the changes are made)

[Vodia PBX]

Disconnects after 30 secs usually happen when there is a SIP routing problem. The ACK message probably does not make it. THis is typically the case if the service provider expects that the customers are running their clients on a public (routable) IP address and/or don't use a session border controller. In this case you either need to get a routable IP address for the PBX or switch to another service provider.

[davidav]

Looked at the log file and seems like the call is being connected on the hunt group party/agent 11. The issue may be that the caller is doing PCMA and the PBX is doing PCMU. If you think your system should be doing PCMA (mostly outside the USA), then please setup the system accordingly (Admin->System->Ports page change the order of codecs and/or the trunk page change the order of codec. This change generally does not require a restart of the PBX. But just start with a clean slate restart the PBX after the changes are made)

I have tried PCMA and G729A as alternatives but neither seem to work any better. I have made PCMU the first choice because this seems to be the codec most often selected by inbound callers - most calls now seem to have PCMU at both ends and this seems to have eliminated the "transcoding" messages we were getting on most/all calls before. I have assumed that RTP Pass Through is better than transcoding?

[Vodia PBX]

I have tried PCMA and G729A as alternatives but neither seem to work any better. I have made PCMU the first choice because this seems to be the codec most often selected by inbound callers - most calls now seem to have PCMU at both ends and this seems to have eliminated the "transcoding" messages we were getting on most/all calls before. I have assumed that RTP Pass Through is better than transcoding?

 

Is there any chance you can try another service provider or a PSTN gateway, just for the sake of nailing down where the problem is? I dont think that hangup depends on the codec type.

[davidav]

Is there any chance you can try another service provider or a PSTN gateway, just for the sake of nailing down where the problem is? I dont think that hangup depends on the codec type.

Maybe you were onto something with PCMA - I changed so this is preferred codec for the trunk (1st in list) and inbound calls can hear ring tone and only had one dropped outbound call yesterday (which could have been because phone was set to PCMU as preferred - now changed to PCMA). Too early to say that its working because it was a very quiet day in the office here but we'll monitor closely over next couple of days and report back on status.

I thought I'd tried all possible combinations of these three settings (codec, strict RTP routing & remote party/privacy indication) but must be that I missed this one. None of this anyway makes a lot of sense to me - I can see that PCMU is still being chosen for some/most of calls anyway but some calls are connected with PCMA. If yesterday's success holds for next few days I will analyse to see if there is any obvious pattern or logic to actual codec selection.

[pbx support]

Maybe you were onto something with PCMA - I changed so this is preferred codec for the trunk (1st in list) and inbound calls can hear ring tone and only had one dropped outbound call yesterday (which could have been because phone was set to PCMU as preferred - now changed to PCMA). Too early to say that its working because it was a very quiet day in the office here but we'll monitor closely over next couple of days and report back on status.

I thought I'd tried all possible combinations of these three settings (codec, strict RTP routing & remote party/privacy indication) but must be that I missed this one. None of this anyway makes a lot of sense to me - I can see that PCMU is still being chosen for some/most of calls anyway but some calls are connected with PCMA. If yesterday's success holds for next few days I will analyse to see if there is any obvious pattern or logic to actual codec selection.

Codec negotiation can be a headache in the some deployments. The issue is that many vendors/providers advertise that they can do many codecs for a call. But only few really switch to the codec that was on the RTP stream if there is a codec change during the call setup. That is why we see these issues.

[davidav]

Codec negotiation can be a headache in the some deployments. The issue is that many vendors/providers advertise that they can do many codecs for a call. But only few really switch to the codec that was on the RTP stream if there is a codec change during the call setup. That is why we see these issues.

Thought it was too good to be true. We are now back to a few good outbound calls but many dropping out. I have attached the log for a typical sequence. Three outbound calls were made to the same number in quick succession: the first two dropped out and the third time the call was OK. The log has the trace for the second and third calls, ie. one good, one bad. Hopefully the comparison of trace for a good and bad call to the same number will help shed some light on this mystery.

Good_BadCalls.rtf

[Vodia PBX]

Thought it was too good to be true. We are now back to a few good outbound calls but many dropping out. I have attached the log for a typical sequence. Three outbound calls were made to the same number in quick succession: the first two dropped out and the third time the call was OK. The log has the trace for the second and third calls, ie. one good, one bad. Hopefully the comparison of trace for a good and bad call to the same number will help shed some light on this mystery.

 

Well, the PBX received a BYE after 37 seconds:

 

[7] 2010/11/05 13:10:00: SIP Rx udp:10.0.1.21:5060:

BYE sip:11@10.0.1.9:5060 SIP/2.0

Via: SIP/2.0/UDP 10.0.1.21:5060;branch=z9hG4bK-bae30358

From: "Office" <sip:11@10.0.1.9>;tag=6b78760386b6280ao0

To: "Study" <sip:10@10.0.1.9>;tag=3a2a4e313b

Call-ID: 1f329437-b0ed7f66@10.0.1.21

CSeq: 103 BYE

Max-Forwards: 70

Authorization: Digest username="11",realm="10.0.1.9",nonce="719cfa3acbaaf7b9c38bb0ce467a6112",uri="sip:11@10.0.1.9:5060",algorithm=MD5,response="7bc669e8becca87f166b785d11fdafbf"

User-Agent: Linksys/SPA942-6.1.5(a)

Content-Length: 0

 

I am not sure why the phone would initiate a hangup. It looks very "beautiful" from a signalling perspective. Maybe the hook switch is instable or some other hardware defect causing this issue? Is the problem related to a specific phone?

[davidav]

Well, the PBX received a BYE after 37 seconds:

 

[7] 2010/11/05 13:10:00: SIP Rx udp:10.0.1.21:5060:

BYE sip:11@10.0.1.9:5060 SIP/2.0

Via: SIP/2.0/UDP 10.0.1.21:5060;branch=z9hG4bK-bae30358

From: "Office" <sip:11@10.0.1.9>;tag=6b78760386b6280ao0

To: "Study" <sip:10@10.0.1.9>;tag=3a2a4e313b

Call-ID: 1f329437-b0ed7f66@10.0.1.21

CSeq: 103 BYE

Max-Forwards: 70

Authorization: Digest username="11",realm="10.0.1.9",nonce="719cfa3acbaaf7b9c38bb0ce467a6112",uri="sip:11@10.0.1.9:5060",algorithm=MD5,response="7bc669e8becca87f166b785d11fdafbf"

User-Agent: Linksys/SPA942-6.1.5(a)

Content-Length: 0

 

I am not sure why the phone would initiate a hangup. It looks very "beautiful" from a signalling perspective. Maybe the hook switch is instable or some other hardware defect causing this issue? Is the problem related to a specific phone?

I don't believe the problem is related to a specific phone - we have experienced outbound call dropouts on both of our Linksys SPA942 phones (very common phone) and also using a softphone. I can switch the phones to verify and/or try again using the softphone. Not sure what a "hook switch" is?

[Vodia PBX]

I don't believe the problem is related to a specific phone - we have experienced outbound call dropouts on both of our Linksys SPA942 phones (very common phone) and also using a softphone. I can switch the phones to verify and/or try again using the softphone. Not sure what a "hook switch" is?

 

Well, from PBX perspective it looks that "someone" hangs up... A hook switch is that thing that detects when the user puts the handset down. But if it is not related to a specific handset then this should be not the problem.

 

Are you getting emails on disconnect events? The PBX sends out emails (if you set up the admin email account) when the PBX disconnects a call for example because of one-way audio. These emails contain the SIP messages for the call, which should make it easier to find out what is going on. You can test this feature by establishing a regular call and then pull the Ethernet plug out of the phone. If you get those emails, then you should also get emails on other events where the PBX disconnects the call.

 

The other source for trouble that you can check is the trunk. We had cases where the PSTN termination was buggy and the gateway detected a busy tone where there was no busy tone. Maybe there is a similar problem with the provider here. The way to narrow this down is to vary the PSTN termination (e.g. different provider, different gateway). You should also log the SIP messages associated with the IP address of the trunk and write them to a log file, even if it gets very big (make sure you are using the $ in the log file name to have a seperate file for each day).

 

BTW make sure that you dont have the hangup tone detection turned on on the PBX trunk. This could be another reason for trouble.