Overview
--------

This is a simple tool that scrubs all personally identifiable information (PII)
from the log files produced by Mediant 500/800/1K/4K/9K/VE/CE/SE SBCs.

The tool can be run on any laptop/PC/server that has Python 3 installed.
Refer to https://www.python.org/downloads/ for detailed installation instructions.

The same tool may also be used as an rsyslog filter - see detailed instructions below.

Usage:
    python log_scrub.py <in_filename> [<out_filename>] [--tag]
    python log_scrub.py --stdin [--tag]
    python log_scrub.py --rsyslog [--tag]

Arguments:
    in_filename  - name of the input file
    out_filename - name of the output file
    --stdin      - take input from stdin
    --rsyslog    - work as rsyslog filter
    --tag        - tag PII elements instead of removing them

Default vs "Tag" Mode
---------------------

In default mode the tool removes PII elements and puts corresponding tag names instead.
For example, instead of:
    To: sip:1111@test.local;tag=1c2029910788  
You will get
    To: sip:NUM@HOST;tag=1c2029910788

If you specify --tag argument PII elements will be preserved but surrounded with XML-like tags:
    To: sip:<NUM>1111</NUM>@<HOST>test.local</HOST>;tag=1c2029910788  
This mode may be useful if you need to preserve PII elements for internal debugging but need to 
remove them prior to passing logs to the 3rd parties.

Sample Output
-------------

default mode:
    08:47:51.971  0.0.0.0  local3.notice [S=2850846] [SID=f124e3:49:83278]  (N  1062346) ---- Outgoing SIP Message to 0.0.0.0:61694 from SIPInterface #0 (IPP) TLS TO(#0) SocketID(2669) ---- [Time:09-05@08:47:51.214]
    08:47:51.971  0.0.0.0  local3.notice [S=2850847] [SID=f124e3:49:83278]  SIP/2.0 200 OK
                                     Via: SIP/2.0/TLS 0.0.0.0:61694;branch=z9hG4bK64e15480A617E7F9
                                     From: NAME sip:NUM@HOST;tag=8CD3F3E0-11A66CD9
                                     To: sip:NUM@HOST;tag=1c2029910788
                                     Call-ID: a916eeaec1214e3192dfcb30976b34b9
                                     CSeq: 304 REGISTER
                                     Contact: sip:NUM@HOST:61694;transport=tls;expires=60;methods="INVITE,ACK,BYE,CANCEL,OPTIONS,INFO,MESSAGE,SUBSCRIBE,NOTIFY,PRACK,UPDATE,REFER"
                                     Expires: 60
                                     Content-Length: 0
                                     [Time:09-05@08:47:51.214]


"tag" mode:
    08:47:51.971  0.0.0.0  local3.notice [S=2850846] [SID=f124e3:49:83278]  (N  1062346) ---- Outgoing SIP Message to <IP>10.2.3.4</IP>:61694 from SIPInterface #0 (IPP) TLS TO(#0) SocketID(2669) ---- [Time:09-05@08:47:51.214]
    08:47:51.971  0.0.0.0  local3.notice [S=2850847] [SID=f124e3:49:83278]  SIP/2.0 200 OK
                                     Via: SIP/2.0/TLS <IP>10.2.3.4</IP>:61694;branch=z9hG4bK64e15480A617E7F9
                                     From: <NAME>John</NAME> sip:<NUM>1000</NUM>@<HOST>test.local</HOST>;tag=8CD3F3E0-11A66CD9
                                     To: sip:<NUM>1000</NUM>@<HOST>test.local</HOST>;tag=1c2029910788
                                     Call-ID: a916eeaec1214e3192dfcb30976b34b9
                                     CSeq: 304 REGISTER
                                     Contact: sip:<NUM>1000</NUM>@<HOST>test.local</HOST>:61694;transport=tls;expires=60;methods="INVITE,ACK,BYE,CANCEL,OPTIONS,INFO,MESSAGE,SUBSCRIBE,NOTIFY,PRACK,UPDATE,REFER"
                                     Expires: 60
                                     Content-Length: 0
                                     [Time:09-05@08:47:51.214]

Rsyslog Filter Mode
-------------------

The same tool may be used as rsyslog filter to scrub received messages in real-time.
This, for example, may be used to scrub SBC messages prior to injecting them into
Azure Log Analytics.

In order to use the tool in this mode do the following:

    1. Install Linux VM (e.g., ubuntu 18.04 or 20.04)
 
    2. Install Log Analytics agent on the VM - as per https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux

    3. Connect the Log Analytics workspace to the Linux VM

    4. Upload log_scrub.py script to /opt folder

    5. Set proper permissions for log_scrub.py script

        chmod 755 /opt/log_scrub.py

    6. Add the following lines to /etc/rsyslog.conf configuration file

        template(name="forwardFormat" type="string"
                 string="<%PRI%>%TIMESTAMP% %HOSTNAME% sbc: %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
                )
        module(load="imudp")
        input(type="imudp" port="514" ruleset="sbc" )
        module(load="mmexternal")
        ruleset(name="sbc") {
            action(type="mmexternal" binary="/opt/log_scrub.py --rsyslog --tag" interface.input="msg")
            action(type="omfwd" Target="127.0.0.1" Port="25224" Protocol="udp" template="forwardFormat")
        }

    7. Restart the rsyslog service

        sudo systemctl restart rsyslog
    
    8. Configure the Mediant SBC to send logs to this Linux server

    9. Check logs on the Log Analytics.


Version History
---------------

1.5
  - add support for processing logs from stdin
  - add rsyslog filter mode
  - add support for tagging PII elements instead of removing them

1.4
  - fix some bugs in NUM, HOST and NAME tags regexes

1.3
  - add support for registration logs
    - AOR
    - URI
    - User-part
    - XML elements phoneNumber, extension and alias

1.2
  - add support for tel: URIs
  - improve support for gateway logs

1.1
  - add support for PII elements in gateway logs
    - caller and callee numbers
    - collected digits
    - dialed digits

1.0
  - initial version
  - the following PII elements are supported:
    - IP addresses
    - SIP URIs:
      - hostnames
      - numbers
      - display names
    - urn:uuid
    - hostnames in Host: header
    - usernames in Authentication: header
    - mac addresses