IP-based authentication not working with Telnyx TLS / FQDN trunk

[mcbsys]

I've set up a TLS / FQDN Telnyx trunk pretty much following https://doc.vodia.com/docs/telnyx-secure-sip-trunking. However I'm trying to use IP-based rather than REGISTER-based authentication:

What happens is that the first outbound call works fine, but a second call to the same number five minutes later fails.

Machine-level tracing shows me the first call has a full TLS handshake, but the next call just sends an INVITE with no handshake, which Telnyx ignores. According to Telnyx support, "As far as I know Every calls requires a new TLS handshake which is failing here."

I've tried this with the trunk configured as a SIP Proxy and a SIP Gateway and still see the same behavior.

Is there some way to get Vodia to start a TLS handshake on each call? Or do I just need to use REGISTER-based authentication with TLS?

Thanks.

 

[Vodia PBX]

The PBX generally pools connections, which is for example also necessary for Teams. Usually there is some keep-alive traffic, so that the one TLS connection stays up for days, weeks and hopefully many months and then there is only one TLS handshake in the beginning. This is also the only way to keep TLS working behind NAT. 

5 minutes sound to me like the connection idles and then gets torn down. If there is no REGISTER, what would still be an option would be — OPTION. This would generate keep alive traffic if there is no REGISTER in use.

[mcbsys]

This is a single-tenant install with 5-15 calls per day, mostly one at a time. There isn't much to pool!

So is OPTION something I can add to a trunk and if so, how? The documentation says that the Keep-alive time setting applies to registration, which is not in use here.

I guess I can switch to registration; I just didn't think it was necessary if I have a static IP.

[Vodia PBX]

On 4/14/2023 at 5:10 PM, mcbsys said:

So is OPTION something I can add to a trunk and if so, how? The documentation says that the Keep-alive time setting applies to registration, which is not in use here.

I guess I can switch to registration; I just didn't think it was necessary if I have a static IP.

In the type from down for the trunk, there is an option for "option". You can try that. 

But even if there are only few calls and the amount of keep-alive might look ludicrous it makes sense to use the registration model. If you compare with how much traffic is being generated by watching a video, the keep-alive again looks more like a rounding error again. Actually depending on the firewall setup or if there is no firewall, you might be good with refreshing the connection every 5 minutes or so, which would really keep the overall overhead small. 

[mcbsys]

Came back to this issue in the last few days.

I believe I've uncovered the main issue:  Azure closes idle connections after 4 minutes without sending a TCP Reset (TCP RST) to let the other party know. The timeout can be extended up to 30 minutes on a static IP address, but sending a TCP RST is only possible if you add another layer, a load balancer:

https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout

So without TCP Reset, we need some kind of keep-alive in under four minutes to keep the connection open. There’s a pretty good article on that here:  https://www.asterisk.org/wanted-dead-or-alive/. I tried the network-level keep-alives suggested here https://serverfault.com/a/851251/166311 and documented here https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html, but I had an issue with the standard registered UDP trunk where inbound calls were failing. It may have been unrelated, but I backed out the Linux changes.

Also, I’m thinking that the TLS connection is established in each direction, so even if I kept the outbound connection open from the PBX to Telnyx, unless Telnyx is also sending keep-alives, inbound calls will fail after four idle minutes. Telynx support says their TCP timeout is 3604 seconds, so just over an hour, after which I assume they would re-do the TLS handshake.

I tried the Vodia trunk of type OPTIONS, but I didn’t see an OPTIONS traffic. Is there some way to set its frequency?

Last I tried using TLS with registration:

• If I left inbound as FQDN (as defined at Telnyx) but did registration for outbound, outbound worked after a five-minute delay, but inbound failed after a delay—confirming that Telnyx didn’t restart the TLS conversation.
• If I set up the connection at Telnyx for registration in both directions, and set the proposed duration in Vodia to 6 minutes with re-registrations at 50% of expiry time, outbound worked but inbound failed immediately. I then realized that Telnyx doesn’t allow setting the inbound SIP Transport Protocol to TLS on a registered connection—that’s only available on FQDN connections.

I'm slowly coming to the conclusion that TLS won't work with Azure hosting the virtual machine unless I set up a load balancer and turn on TCP RST. Theoretically that would gracefully close the connection so the PBX would re-handshake for outbound and Telnyx would re-handshake for inbound.

 

[Vodia PBX]

Thanks for the great analysis. 

We had a similar problem many years ago with UDP and missing SBC on the SIP trunk provider side. OPTIONS work only for SIP trunks that don't register. If the SIP trunk does register, you can instead the "Keep-alive time" and force the PBX to re-register every 3.5 minutes or so.

[mcbsys]

5 hours ago, Vodia PBX said:

OPTIONS work only for SIP trunks that don't register.

When I tried OPTIONS, it was without registration, with TLS. But it looked the same as SIP Proxy--I never saw an OPTIONS packet sent in the machine-level trace. How often are the OPTIONS packets supposed to go out and can I control that?

5 hours ago, Vodia PBX said:

If the SIP trunk does register, you can instead the "Keep-alive time" and force the PBX to re-register every 3.5 minutes or so.

Unfortunately, it seems Telnyx doesn't allow TLS on a registered connection. I may need the "keep-alive time" if I try TCP without TLS. UDP seems fine with the default 1-hour timeout, I guess because it doesn't depend on keeping the connection open.

[Vodia PBX]

You would definitively see the OPTIONS packets, e.g. by filtering SIP packets by the IP address. If you don't see them—something is wrong. Originally we did that for Teams (not sure why, probably just the "Microsoft way"), but other vendors are also starting to use OPTIONS as well and this seems to work fine. 

[mcbsys]

5 hours ago, Vodia PBX said:

You would definitively see the OPTIONS packets, e.g. by filtering SIP packets by the IP address. If you don't see them—something is wrong. Originally we did that for Teams (not sure why, probably just the "Microsoft way"), but other vendors are also starting to use OPTIONS as well and this seems to work fine. 

I changed the FQDN/TLS trunk from SIP Proxy to Options and traced it for 2.5 hours, decrypting with the keylogfile. Not a single OPTIONS packet. I tried one outbound call, which failed. That shows up as one encrypted packet (in spite of the keylogfile) and 15 failed retransmissions. Maybe it's waiting on the other side to send OPTIONS first?

I was struggling to understand this setting of Options as a trunk type instead of Registration, Gateway, or Proxy. Then the mention of Teams took me to this article:

https://learn.microsoft.com/en-us/microsoftteams/direct-routing-protocols-sip

"Before an incoming or outbound call can be processed, OPTIONS messages are exchanged between SIP Proxy and the SBC. These OPTIONS messages allow SIP Proxy to provide the allowed capabilities to SBC."

That does, in fact, sound like Microsoft's approach to SIP conversations. It may be a legitimate use of OPTIONS (to discover capabilities of the other party). In that case, maybe the registration type should be called "Teams TLS with Options".

The rest of the world seems to use OPTIONS basically as a SIP "ping." Like the Asterisk article says, "OPTIONS messages are full SIP messages that a user agent can send to a peer and expect to get a standard SIP response. You can configure Asterisk to send these messages to a peer by setting the 'qualify_frequency' parameter in the peer’s aor object.   At that interval, Asterisk will send the OPTIONS and will mark the peer’s contact as available and record the round trip time if it gets a response.  If not, the contact is marked as unavailable..."

From what I can tell, the standard doesn't require TLS or even TCP for OPTIONS. To solve this kind of issue, where we need a keep-alive, we would need to be an setting on any trunk, "Send OPTIONS every x seconds" and then, if there's no response and TCP/TLS was active, close the connection (TCP RST).

What I still don't quite understand is whether a TLS conversation, once opened from the PBX for outbound calls, can (and will) be used by the vendor (Telnyx) for inbound calls. Can Telnyx "find" the open connection and put the inbound call on it? If so, a recurring OPTIONS would help. If not, it doesn't matter how many OPTIONS the PBX sends if the vendor always tries to start their own TLS handshake.

[mcbsys]

Good news:  I was mistaken. Telnyx does support TLS over a registered connection. You don't specify it; you just send the REGISTER over TLS and it accepts it. It does allow specifying that the media be encrypted and I've confirmed that that works with SRTP.

By setting registration to 6 minutes with a 50% renewal, I think I’m maintaining the connection okay. (I’ll want to do some more testing to confirm.) The one thing I don’t get is why some of the packets in my trace are still encrypted even after applying the keylogfile.

[Vodia PBX]

Ok that makes sense. Another provider supporting SRTP! 

[mcbsys]

You do have to manually specify SRTP (or ZRTP) in the Telnyx connection for Inbound, then Outbound matches it. I guess that then tells the PBX to send encrypted media?

[Vodia PBX]

It's all about SDP offer/answer. The PBX will answer a SRTP offer over TLS, and it will offer SRTP over TLS. 

There was also some support for ZRTP in the PBX some time ago, but it fizzled probably because DTLS became mainstream. 

[mcbsys]

Going back to my longer post above:  am I seeing this correctly, that Vodia will not let me set up OPTIONS as a kind of keep-alive SIP "ping"? That would seem preferable over re-registering every three minutes, but I'd need to be able to set the frequency that OPTIONS are sent.

[Vodia PBX]

There is no mixing of REGISTER with OPTIONS in Vodia SIP trunks. But IMHO re-register is not as heavy as it might look in the logs; it's actually relatively small bandwidth. 

[mcbsys]

Right. I was thinking of SIP Proxy with the option of adding OPTIONS every x seconds. I never got Options to work as a trunk type (no OPTIONS are sent in 2.5 hours). I was guessing that that trunk type may be specific to Teams, or at least to exchange OPTIONS for information purposes, not keep-alive purposes.

I can probably live with re-registering for TLS, just trying to understand all the options, so to speak.

 

[Vodia PBX]

Other providers are also starting to use OPTIONS, for example Amazon Chime is using OPTIONS. It seems to become "fashion", maybe just to make it very hard to register a VoIP phone there to keep pesky consumers out .