Trunk - Send email on status change - frequent emails, no info

[mcbsys]

Hi,

I'm using a registered Telnyx trunk on 68.0.28. I'd like an email notification if it goes offline for some reason. In the trunk page, I set Send email on status change to "Only for status changes." I started getting emails like this I think it was maybe every hour:

Since it doesn't say what status changed, I don't know why it was sending the email, maybe on re-REGISTER events?

I don't want emails that often, so I disabled the alert. Then the trunk did go down and I didn't know until the customer called.

1. How can I update the alert email to tell me what status changed?

2. How can I set it to only send an alert if it needs my attention, e.g. if the trunk goes offline?

[mcbsys]

Let me expand this question a bit:  how do I customize the admin emails? I've started receiving occasional emails informing me that "The IP address 122.104.112.222 has been blacklisted for 60 minutes because there were 100 unsuccessful connections." Good to know, but I might like to add a link to an IP location service so I can see where the request came from. Also, I'd really like to know what port they are trying to hit--I've got 5060 and 5061 restricted at the firewall level.

[Vodia PBX]

Those admin emails are not customizable, actually they are even hardcoded to English (who cares about the administrator ). It might be somewhere on the todo list, however it's usually not a big deal and most administrators on the planet can understand the content. But we are always interested in revealing more interesting information. For example the local address is indeed interesting. 

We did include a hardcoded link to a location service, however it changed and became useless. 

We do not send email on re-register. If you get an email every hour, you do have a problem that need to investigate. 

[mcbsys]

Thank you for your reply.

21 hours ago, Vodia PBX said:

Those admin emails are not customizable, actually they are even hardcoded to English (who cares about the administrator ). It might be somewhere on the todo list, however it's usually not a big deal and most administrators on the planet can understand the content.

Agreed. English is fine for most administrators. Nur die ganz speziellen können auch Deutsch .

21 hours ago, Vodia PBX said:

But we are always interested in revealing more interesting information. For example the local address is indeed interesting. 

I think you mean the local port? Ideally the email would show a log (table) of the attempts: source IP and port, destination URL or IP with port, username and password attempted. That would quickly give me an indication of how seriously to take the hack attempt. Maybe something like this is possible in IPTables, but I'm not a Linux pro.

21 hours ago, Vodia PBX said:

We did include a hardcoded link to a location service, however it changed and became useless. 

My point exactly. This needs to be in a customizable template.

21 hours ago, Vodia PBX said:

We do not send email on re-register. If you get an email every hour, you do have a problem that need to investigate. 

If only I knew what to investigate. "Status change" is the only notice. What Status? Reachability? IP address? Registration? The message as is doesn't tell me much!

[Vodia PBX]

5 hours ago, mcbsys said:

I think you mean the local port? Ideally the email would show a log (table) of the attempts: source IP and port, destination URL or IP with port, username and password attempted. That would quickly give me an indication of how seriously to take the hack attempt. Maybe something like this is possible in IPTables, but I'm not a Linux pro.

The port, and with it we also easily get the local IP address that was used. However the PBX SBC does not keep too much detail about each access (this would make DoS a real nightmare), so it essentially logs what has been stored with the associated address.

5 hours ago, mcbsys said:

If only I knew what to investigate. "Status change" is the only notice. What Status? Reachability? IP address? Registration? The message as is doesn't tell me much!

The status is the SIP status for the trunk. 200 means "Ok", and other status depend. For example 408 means that the REGISTER that the PBX has sent did not return anything and timed out. Usually this is because of Internet connectivity problems. 

[mcbsys]

On 3/31/2023 at 10:13 PM, Vodia PBX said:

The status is the SIP status for the trunk. 200 means "Ok", and other status depend. For example 408 means that the REGISTER that the PBX has sent did not return anything and timed out. Usually this is because of Internet connectivity problems. 

Is there a way to generate a trunk-level PCAP inside the PBX, one that tracks _all_traffic (not just subsets attached to individual calls)?

This PBX is running on an Azure virtual machine so the Internet connection is probably pretty good.

[Vodia PBX]

On 4/5/2023 at 8:01 AM, mcbsys said:

Is there a way to generate a trunk-level PCAP inside the PBX, one that tracks _all_traffic (not just subsets attached to individual calls)?

This PBX is running on an Azure virtual machine so the Internet connection is probably pretty good.

You can enable PCAP on trunk level as well. However it tracks only traffic for calls, registration traffic is not being recorded. But what you can do is to filter the SIP packets by IP address, then you get a good picture on that is going back and forth with the trunk.

[mcbsys]

I needed a PCAP to see all traffic on the trunk, so I captured at the Linux machine level

sudo tcpdump -i any -nn host 192.76.120.10 -v -w telnyx.pcap

The traffic is TLS-encrypted, so I need the private key, I assume from the Let's Encrypt certificate, to decrypt it.

https://www.zoiper.com/en/support/home/article/162/How to decode SIP over TLS with Wireshark and Decrypting SDES Protected SRTP Stream

The Vodia UI does not show the private key and the Let's Encrypt certificates are not in the usual place (/etc/letsencrypt/live). I finally found  them in /usr/local/pbx/certificates/136.xml. Alas, the private key is encrypted, so I won't be able to decrypt the PCAPs.

How can I get a full, decrypted trace of all PBX traffic?

 

[Vodia PBX]

You can actually write the private key, use the "TLS key log file" in the security settings for this. Wireshark is able to read this.

[mcbsys]

Very helpful, thanks. Not the original private certificate key that I was seeking, but the private session keys, which are more useful as they can be shared with a third party who needs to review the PCAP.

This allowed me to move forward in diagnosing a connection problem:

I may come back to this email notification issue later.

[mcbsys]

On 3/30/2023 at 6:48 PM, Vodia PBX said:

We do not send email on re-register. If you get an email every hour, you do have a problem that need to investigate. 

Circling back to this. For the record, I'm on 68.0.28.

Yesterday, all inbound calls to the PBX suddenly started dropping. Outbound worked okay. Rebooting the PBX solved the problem, but Telnyx asked if the trunk was still registered. I didn't think to check at the time, but no one is in the customer office today, so I thought I would try some debugging. I enabled Send email on status change :

and I set up a packet trace at the Debian machine level:

sudo tcpdump -i any -nn host 192.76.120.10 -v -w telnyx.pcap

Results: 

1. Every 29-30 minutes, the trace has four packets:  REGISTER, 401 Unauthorized, REGISTER, 200 OK. Looks like normal traffic to me. There were no calls in the 6.5 hours I tested, so I have exactly 52 packets (13 registrations).

2. Every 29-30 minutes, matching the REGISTER events, I get two emails from the PBX (total 26 emails, 13 registrations):

Subject:  Phone System: Status change on trunk Telynx - Registered
Body:  Trunk . This is a notification email. Do not reply.

Subject:  Phone System: Registration Metrics for trunk Telynx - Registered
Body: 
x-RegData: RPN=sip.telnyx.com
x-RegMetrics: RRD=217 RRS=200  [the RRD number varies; RRS is always 200]

So if the PBX doesn't send emails on re-register, what's going on here?

And more importantly, is there a way to only be notified when the system registration fails, i.e. when it is offline?

Thanks for your help.

[Vodia PBX]

The trunk status change emails are (supposed to be) independent from the registration metrics emails. They are only sent out when something has changed. Are the subjects exactly the same for the status change? 

[mcbsys]

18 minutes ago, Vodia PBX said:

The trunk status change emails are (supposed to be) independent from the registration metrics emails. They are only sent out when something has changed. Are the subjects exactly the same for the status change? 

Yes. Here are my deleted emails, sorted by Subject and Time.

[Vodia PBX]

As far as I can see it might be because the associated IP address has changed, even though the registration status stays the same.

[mcbsys]

The PBX has a static IP address.

sip.telnyx.com is consistently 192.76.120.10 in the U.S. (and in the PCAP, if I recall correctly), though they do allow 64.16.250.10 as a secondary. https://sip.telnyx.com/ The current DNS status is below.

As mentioned, I saw four packets every half hour, REGISTER, 401 Unauthorized, REGISTER, 200 OK. Does the "401 Unauthorized" challenge make the system think it lost registration, then the 200 OK that it re-gained registration, thus triggering a "status change" email?

 

[Vodia PBX]

I have no doubt that the registration is stable. It's just the glitch with the notifications. 

sip-anycast1.telnyx.com and sip-anycast1.telnyx.com seems to have a random TTL between 1 and 60 seconds, and maybe we see those glitches because it changes and then that address is not on the IP address whitelist any more. I would assume that the glitch goes away if you add the 64.16.250.10 and 192.76.120.10.

[mcbsys]

2 hours ago, Vodia PBX said:

I have no doubt that the registration is stable. It's just the glitch with the notifications. 

sip-anycast1.telnyx.com and sip-anycast1.telnyx.com seems to have a random TTL between 1 and 60 seconds, and maybe we see those glitches because it changes and then that address is not on the IP address whitelist any more. I would assume that the glitch goes away if you add the 64.16.250.10 and 192.76.120.10.

I think you are suggesting that I add those two IPs to a whitelist somewhere, but where? Most references to "whitelist" in the docs seem to be about what IP has permission to administer Vodia:

https://doc.vodia.com/docs/search?query=whitelist&searchType=articles

Does the SRV record sip-anycast1/2.telnyx.com even matter when I am registering directly to sip.telnyx.com? Here's the trunk status page; we're talking about the second, registered connection.

[Vodia PBX]

The name is "Explicitly list addresses for inbound traffic".

The whole logic of figuring out where packets can come from is tricky because of the DNS NAPTR, SRV, A, AAAA records that are in play. Its also tricky to figure out if something is changing!

[mcbsys]

Okay I set the trunk to email on status change only, with the two explicit IP addresses:

I'm still getting an email every 30 minutes:

I expanded my tcpdump to include both IPs:

sudo tcpdump -i any -nn '(host 192.76.120.10 or host 64.16.250.10)' -v -w telnyx.pcap

I captured the last re-registration at 12:03pm. Here's a screenshot of of the PCAP:

Now what? I could capture DNS traffic, but it doesn't seem confused about the IP address.

P.S. Have I ever mentioned that this forum software is pretty awesome? Rarely is it this easy to insert inline images in a forum!

[Vodia PBX]

I still believe that the 401 does not matter... Anyhow in the screenshot the flag to check DNS every time is on—is there any particular reason for that? It adds another point of failure, and who knows, maybe that is causing our glitch?

On 6/6/2023 at 9:22 PM, mcbsys said:

P.S. Have I ever mentioned that this forum software is pretty awesome? Rarely is it this easy to insert inline images in a forum!

Well its not Vodia software but HTML5 rocks!

[mcbsys]

11 hours ago, Vodia PBX said:

I still believe that the 401 does not matter... Anyhow in the screenshot the flag to check DNS every time is on—is there any particular reason for that? It adds another point of failure, and who knows, maybe that is causing our glitch?

Years ago I had and provider that had a different IP for each of their locations. They would use DNS to re-route to a different IP when one location went down. So I probably got in the habit then. Telnyx seems pretty stable with just the one IP (hopefully with redundant routing) so it's probably not necessary. And sure enough, with that turned off, Vodia stopped sending an email every 30 minutes. So why does Vodia consider it a status change when it refreshes the DNS, assuming the DNS doesn't change every 30 minutes?

[Vodia PBX]

57 minutes ago, mcbsys said:

So why does Vodia consider it a status change when it refreshes the DNS, assuming the DNS doesn't change every 30 minutes?

Changing the DNS address for TCP/TLS-based registrations IMHO make no sense. In the old times when the SIP standard was written, there was not such an abundant choice of TCP-based load balancers so they thought UDP would be the answer (but it was not). So why bother looking up the address again and again when the TCP connection has not dropped. Dropping it would leave a small window of non-registration where inbound calls are not sure to terminate. 

The other real life problem is that providers tend to set DNS durations very short, which causes a lot of extra steps that can fail and cost time, especially when going through the while NAPTR, SRV, A and AAAA chain. That is another reason to keep the TCP connection when the trunk is still registered. 

[mcbsys]

1 hour ago, Vodia PBX said:

Changing the DNS address for TCP/TLS-based registrations IMHO make no sense.

 Okay but this issue is happening on a classic, old-fashioned, non-TCP, non-TLS, registered UDP trunk. Seems odd that the DNS lookup is considered a trunk change, worthy of an email alert, when the DNS results do not change (except maybe current TTL).

[Vodia PBX]

We'll try to suppress it in the next build. 

[mcbsys]

14 minutes ago, Vodia PBX said:

We'll try to suppress it in the next build. 

Great, thanks!