Strange CDR logging format ?

[rudik]

Hi all,

 

I am currently implementing a new Snom One Blue PBX (2011-4.3.0.5021 Win64)and have a strange (out of the box ?) CDR logging format.

Instead of starting the file with <?xml version="1.0" encoding="utf-8"?> according to XML standard the example file called "93.xml" in the cdre-folder contains the following data ?? :

 

=== Start of file ===

TLVB c 1349258449.548 cid ,YjdlMWEzNzI4ZDVhMzhmNDhkYTkyM2NjMmIxNTM5YWQ. ct d d 1 e 1349258454.688 f +"Rudi Test" <sip:800@vijf> i ,YjdlMWEzNzI4ZDVhMzhmNDhkYTkyM2NjMmIxNTM5YWQ. o I p udp:192.168.20.142:36108 r "Rudi" <sip:061111111@vijf> rec /recordings/20121003/120049-o-061111111-800.wav s

1349258441.81 t "Rudi" <sip:061111111@vijf> u 44 vq VQSessionReport: CallTerm

LocalMetrics:

Timestamps:START=2012-10-03T10:00:49Z STOP=2012-10-03T10:00:54Z

CallID:YjdlMWEzNzI4ZDVhMzhmNDhkYTkyM2NjMmIxNTM5YWQ.

FromID:"Test 5 New"<sip:800@192.168.20.200>;tag=a817667b

ToID:"061111111" <sip:01111111@192.168.20.200>;tag=eb71e45556

SessionDesc:PT=8 PD=pcma SR=8000 FD=20 FO=160 FPP=1 PPS=50 PLC=3

LocalAddr:IP=192.168.20.200 PORT=52162 SSRC=0x48763f50

RemoteAddr:IP=192.168.20.142 PORT=37286 SSRC=0xe6cb2e02

x-UserAgent:snom-PBX/2011-4.3.0.5021

x-SIPterm:SDC=OK SDR=AN

PacketLoss:NLR=0.0 JDR=0.0

BurstGapLoss:BLD=0.0 BD=0 GLD=0.0 GD=0 GMIN=16

Delay:RTD=0 ESD=0 IAJ=0

QualityEst:MOSLQ=4.1 MOSCQ=4.1

RemoteMetrics:

JitterBuffer:JBA=3 JBR=0 JBN=40 JBM=40 JBX=80

PacketLoss:NLR=0.0 JDR=0.0

BurstGapLoss:BLD=100.0 BD=40 GLD=0.0 GD=5920 GMIN=16

Delay:RTD=0 ESD=0 IAJ=15

QualityEst:MOSLQ=4.1 MOSCQ=4.1

x-UserAgent:eyeBeam release 1009l stamp 37965

y extcall

=== End of file ===

 

Clearly this does not have a XML header but some other (unknown to me ) header. As currently the CDR XML files should be analyzed by a seperate control-system these files are useless because the control system expects XML format. I have been reading the documentation and 'normal' XML should be default for Snom One? Unfortunately in this case it is clearly not default ? What settings should be changed to 'reset' the logging to the 'good old' XML ?

 

The control-system runs great on the old version (3.4.0.3201 Win32), if possible the same format would be ideal.

 

Thanks,

Rudi

[Vodia PBX]

No that's a feature. Because there are so many CDR, there is a special binary format for those entries. Background is that XML parsing cost a lot of CPU and you cannot just copy the contents because characters need to be un-escaped. TLV stands for Tag-Length-Value, and this is a primitive format to parse the contents of a stream.

[ndemou]

From what I can understand the format of the new cdr files is as folows:

First 4 bytes is the string "TLVB"

After that follow 1 or more records like this:

<ll> <string of length ll> <LLLL> <string of length LLLL>

 

where:

ll = 2-bytes-integer

LLLL = 4-bytes-integer

 

I've tested the above in a few dozen cdrs

[Vodia PBX]

Yepp. It is not too complicated and it saves the PBX the time to escape and un-escape all those forbidden XML characters.

[ndemou]

Here's a python parser I came up with for the CDRs. It's been tested on a few hundreds CDRs (i,e and t) without any apparent problem but I can give no warranties.

 

def tlvb_parser(a):
   '''PBXnSIP v4 CDRs parser 
Acts as a generator so call it like: for key,value in tlvb_parser(data):

The CDR files have an .xml extension but are obviously not xml
They also start with TLV but they are not Type-Length-Value formated either 
From what I've seen in a few hundrend files their format is as folows:

First 4 bytes is the string "TLVB"
After that follow 1 or more records like this:
<ll> <string of length ll> <LLLL> <string of length LLLL>

where:
ll = 2-bytes-integer
LLLL = 4-bytes-integer
   '''
   # TODO: the case of 0 length for the value may be either:
   #   1) expected indicating an empty string   --or--
   #   2) unexpected indicating a corrupt file
   # Since I'm not sure I make the behaviour configurable:
   zero_len_for_value_is_expected = True
   # I also configure this sanity limit for the length of values
   maximum_expected_len_for_value = 8192

   import struct
   if a[:4]<>'TLVB':
       raise ValueError, "Invalid data (they don't start with TLVB)"
   else:
       a=a[4:]
       state='LOOKING_FOR_KEY' # initial state
       while a:
           if state=='LOOKING_FOR_KEY':
               # read key 
               l=struct.unpack('>H',a[:2])[0] # assume 1st 2 bytes are an integer representing the length of data to follow
               if l==0:
                   raise ValueError, 'Was expecting the key length but instead found 00h 00h'
               else:
                   if l>len(a)-2:
                       raise ValueError, 'Length for key points after EOF'
                   if l>255:
                       raise ValueError, 'Length for key>255 (%s) which is suspiciously large' % l
                   a=a[2:]
                   key=a[:l]
                   a=a[l:]
                   state='LOOKING_FOR_VALUE'
           else:
               # read value 
               l=struct.unpack('>I',a[:4])[0] # assume 1st 4 bytes are an integer representing the length of data to follow
               if l>len(a)-4:
                   raise ValueError, 'Length for value points after EOF'
               if l>maximum_expected_len_for_value:
                   raise ValueError, 'Length for value>%s (%s) which is suspiciously large' % (maximum_expected_len_for_value, l)
               if l==0 :
                   if not zero_len_for_value_is_expected:
                       raise ValueError, 'Was expecting the value length but instead found 00h 00h 00h 00h'
               a=a[4:]
               if l: 
                   value=a[:l] 
                   a=a[l:]
               else: 
                   value=''
               state='LOOKING_FOR_KEY' 
               yield key,value