Audio quality is suffering

[Tom Waterman]

Hello All. I am looking for some help with an on going audio issue that we are having here at out office. We are currently running PBXnSIP 4.2.0.3958 (Win32. We have a dedicated server that has 2 network connections. One is connected directly to the Internet via a 3meg dedicated fiber pipe. The second connection connects to a Ethernet switch. All of my phones plug into the same switch. The entire PBX is on its own network. Yet still I get audio loss. I am not talking about 1 way audio. Let me give an example. Last week I have 10 people in the conference room at out office who are dialed into a PBX conference extension via a single pod phone. I then have 4 folks from our Denver office dial into the same conference room. They are calling into the Auto Attendant like any normal person would do. During the meeting while the new Director of Operations is speaking the sound cuts out or becomes distorted. It is her voice that no one can here for a few seconds and then the sound comes back. This is happening more and more often. It happens in all types of situations this is just a single example. I had previously gathered some wireshark captures and I could see the audio was breaking up leaving the PBX to the phones but I am sure this is not the only case. Can someone please give me some assistance. I have already read all the stuff in the old wiki and most of it does not apply to our setup. One last thing. Our provider is Callcentric. Thank you for your time.

 

tom

[Vodia PBX]

Here is a old checklist that you can use: http://kiwi.pbxnsip.com/index.php/Troubleshooting_SIP_Trunk_Problems#Poor_Audio_Quality and also http://kiwi.pbxnsip.com/index.php/One-way_Audio. For example we had a tricky case where a (stupid) IP address conflict was causing such problems. But it could also be a router that temporarily runs out of CPU resources and introduces a lot of jitter and complete temporary loss of media. We also had such cases.

 

IMHO it is with installing a analytical tool like Wireshark and try to catch a call where the call quality is not okay. Then you can nail the problem at least from the PBX perspective.

 

Also, the MOS score graph and the email CDR (which include QoS measurement attachments) can give you some indication where the problem might be. But I find it easier in such cases to use Wireshark to find the problem.

[Tom Waterman]

I have used both of these troubleshooting guides with no luck. The audio dropping happens everyday in all configurations. It happens on an extension to extension call, on a Callcentric to extension call. I have to figure out what is going on before people get more upset. I am thinking about moving the PBX from Windows to Linux as I have exhausted other options. I do have a support ticket open but have not heard anything back for 2 days. :-(

 

Tom

[mattlandis]

Tom,

 

Want our team to give a crack at it?

 

I doubt it is an inherently Windows issue.

But:

-any virtualization going on?

-any chance switch is finicky?

-are the 10 people coming into the conference call on LAN phones?

-are the ext. to ext. calls that are breakin up on same lan?

 

just some ideas to get you going in the meantime.

 

Matt

[Vodia PBX]

I remember there was a case with a USB driver that would drive the server mad. You never know! It would really be an option to try either a fresh Windows or something else like Linux.

[Tom Waterman]

Tom,

 

Want our team to give a crack at it?

 

I doubt it is an inherently Windows issue.

But:

-any virtualization going on?

-any chance switch is finicky?

-are the 10 people coming into the conference call on LAN phones?

-are the ext. to ext. calls that are breakin up on same lan?

 

just some ideas to get you going in the meantime.

 

Matt

 

 

Matt,

here are some of the issues I have found. We do have issues with ext to ext calls on the same lan. When the conference calls happen they are not always on the lan. Some dial in from outside sources. This is a virtualized server with plenty of juice to run pbxnsip. The switches are new linksys by cisco switches. I know Cisco switches would have been better but we needed 8 and I could not get the company to swallow that bill. I have looked at the CPU usage and it is very very low. I do have some low MOS scores so I know the PBX knows the quality is bad. So I am now running wireshark captures on both interfaces and sending them to a shared drive. I spend about 6 hours a day in wireshark.I have made one change to the pbx since Monday and that has been to increase the ram from 2 gigs to 4 gigs.

 

Any thoughts?

 

Thank you!

Tom

[mattlandis]

Tom,

 

I would watch the lan to lan calls. This is the simplest.

 

Any chance you could put the pbx server on its own server hardware just to eliminate virtualization as an issue?

[Vodia PBX]

Ethernet switch will be okay. I would also say with 50 % probably this has to do with the virtualization setup and 40 % something with the Windows setup inside the VM. 10 % "something else"

[Tom Waterman]

On thing I have noticed is that on the trunk between PBXnSIP and Callcentric the are using G729A and we want G711u because the audio quality is better and we have the bandwidth. We have a dedicated 3 meg pipe for the pbx. Could this be part of my problem? And how do I get the trunk between Callcentric and the PBX to stay at G711u? Let's hope this is a step in the right direction.

 

Tom

[mattlandis]

"Override Codec Preference:" in the trunk.

[Tom Waterman]

Matt, I thought about that and checked it. Here is my config. I believe this locks it but obviosly it does not.

 

 

Tom

[mattlandis]

shouldn't you be clicking "g711" and clicking add?

[Tom Waterman]

shouldn't you be clicking "g711" and clicking add?

 

I was able to get the codec locked at g711u and I am still having packet loss. Any other ideas? I am taking a beating here.

[mattlandis]

did you move the pbx to it's own hardware?

[andrewgroup]

Tom,

 

Sorry to me late in commenting, but your problem wreaks of having PNP devices on the LAN, perhaps the gateway router supports PNP...

 

I'll go out on a limb and assume that your setup is not running 100% PNP on the phone configs and you have phones manually registered against the local IP of the PBX?

 

It's very common for popular L2 switches to have a featured called ICMP snooping enabled/disabled and this prevents the switch to deliver broadcast packets to all ports.

 

I also suggest that if you L2 switch supports ports sniffing, then use it to capture the packets going to the PBX.

I also suggest getting some basic tools like PRTG to track latency to the devices during calls.

 

Getting QOS correct is important... What brand switches are you running?

 

We have grown to use PORT based QOS assignments for the PBX port and the PUBLIC IP port. You assign the port to a Queue level, normally 1,2,3,4 in priorities, then you also assign the DSCP values used by the phones to the same Queue.

 

You said these problems are occurring on the local LAN, so creating a scalable test environment should be easy and establishing a repeatable experiment should net some results.

 

If you have the stomach, expertise, or time to learn, check this tool out http://sipp.sourceforge.net/

[hosted]

This is easy to troubleshoot. load wireshard on the windows box capture the RTP and it will tell you exactly what devices are causing issues.

 

If you find ALL devices are lossing packets you have 2 places to look your switch or your server. Most likely your server.

 

I would not run VM. you might have "juice" but you will probably have "jitter" which is much worse