01 RA41141EN08GLA0 MRBTS Troubleshooting

Nokia AcademyLTE MRBTS Troubleshooting • RL70 / RL55 Release RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use Copyright and confidentiality The contents of this document are proprietary and confidential property of Nokia Solutions and Networks. This document is provided subject to confidentiality obligations of the applicable agreement(s). This document is intended for use of Nokia Solutions and Networks customers and collaborators only for the purpose for which this document is submitted by Nokia Solutions and Networks. No part of this document may be reproduced or made available to the public or to any third party in any form or means without the prior written permission of Nokia Solutions and Networks. This document is to be used by properly trained professional personnel. Any use of the contents in this document is limited strictly to the use(s) specifically created in the applicable agreement(s) under which the document is submitted. The user of this document may voluntarily provide suggestions, comments or other feedback to Nokia Solutions and Networks in respect of the contents of this document ("Feedback"). Such Feedback may be used in Nokia Solutions and Networks products and related specifications or other documentation. Accordingly, if the user of this document gives Nokia Solutions and Networks feedback on the contents of this document, Nokia Solutions and Networks may freely use, disclose, reproduce, license, distribute and otherwise commercialize the feedback in any Nokia Solutions and Networks product, technology, service, specification or other documentation. Nokia Solutions and Networks operates a policy of ongoing development. Nokia Solutions and Networks reserves the right to make changes and improvements to any of the products and/or services described in this document or withdraw this document at any time without prior notice. The contents of this document are provided "as is". Except as required by applicable law, no warranties of any kind, either express or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose, are made in relation to the accuracy, reliability or contents of this document. NOKIA SOLUTIONS AND NETWORKS SHALL NOT BE RESPONSIBLE IN ANY EVENT FOR ERRORS IN THIS DOCUMENT or for any loss of data or income or any special, incidental, consequential, indirect or direct damages howsoever caused, that might arise from the use of this document or any contents of this document. This document and the product(s) it describes are protected by copyright according to the applicable laws. Nokia is a registered trademark of Nokia Corporation. Other product and company names mentioned herein may be trademarks or trade names of their respective owners. © Nokia Solutions and Networks 2015 2 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 Objectives • Review key concepts regarding eNB troubleshooting • General troubleshooting guidelines • Alarms vs Faults • Importance of fault ID • Alarm history • Site Commissioning File Tools • Explain BTS Snapshot or Technical Report log description and contents and decoder and analysis tool. • Describe the Automatic Error Symptom Saving (LTE1099) feature. • Describe the Local and Remote IP Traffic Capturing (LTE1460) feature. • Describe the Remote Syslog (LTE1408) feature • Explain the Flexi MRBTS LTE eNB HW Architecture • Explain iOMS troubleshooting and error symptom log collection. 4 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 General hints for troubleshooting The first and most important step in troubleshooting is to collect all relevant information: • when did the problem start? • if the starting point cannot be determined, when was the last confirmed time when the system was working without the problem? • what was the first symptom of the problem? • which KPIs are affected • which activities were performed (shortly) before the problem started? - SW updates on eNB - configuration changes on the eNB - SW updates on other NEs (MME, SGW, SecGW, …) - configuration changes on other NEs (MME, SGW, SecGW, …) - eNB reset • if the problem starts after a long “uptime” of the eNB and if it is cured by eNB reset, observe the behavior on other eNBs having similar “uptime” (this is to understand if resource leaks are causing the problem) • have there been other changes in the customer network? - launch of new UE type? - special event with high traffic? • how many sites are affected and when did the problem start for each site? • have any recovery actions already been executed? • whether the problem happens just to specific subscribers (e.g., for roaming users) or not 5 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 record all test steps and observations in a logbook (call setup. achieved data rates) with accurate time information .General hints for troubleshooting • If the problem is NOT visible for all eNBs.collect the same set of logs from both a bad and a good site for comparison . mobility events.cover both good and bad sites for comparison • If a dependency on another network element is suspected collect detailed information about the other network element (vendor. try to understand the differences between “good” and “bad” sites: . call termination. HW. SW level) 6 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .compare configuration (both HW and commissioning) of good and bad sites • If drive testing is done . including Emil. BTSLog and port mirroring for wireshark. • Document is updated by SW release to reflect new features and changes to R&D tools.nokiasolutionsandnetworks.inside.com/livelink/livelink? func=ll&objId=409795709&objAction=browse&sort=name&viewType=1 RL70 / RL55 Error Symptom Saving 7 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . MACTTI Uploader. • Here you will find links to documents on how to setup tools to help capture error symptoms traces and logs. • https://sharenet-ims.LTE Error Symptom Saving Guidelines • The MBB Technical Support group maintains troubleshooting and error symptom saving guidelines. This severity level can should be taken to prevent a be reported. The following severity levels are used in the system: Critical Major Minor • The Critical severity level • The Major severity level • The Minor severity level indicates that a service-affecting indicates that a service-affecting indicates that there is a condition has occurred and an condition has developed and an condition that does not affect immediate corrective action is urgent corrective action is services. when more serious fault. when be reported. totally out of service and its the capability of the managed Such a severity can be capability must be restored. for example. a managed object becomes there is a severe degradation in a fault that affects a service. for example. Corrective action required. This severity level can required. for example.Alarm Severity The severity levels of the alarms indicate the default and perceived severity of the alarms. detected alarm condition is not currently degrading the capacity of the managed object. object and its full capability reported. for example. when the must be restored. 8 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . BTS alarms may be triggered by a number of different faults. the faults are mapped to the alarm based on the severity of the fault. 9 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . BTS Site Manager alarm view shows BTS Fault and will include the BTS Fault ID in the fault description. Supplementary information of alarm (reported to NetAct) will have description of the fault.Alarms vs. BTS Faults BTS alarms indicate severity and scope of a fault. BTS diagnostic logging is based on the BTS fault ID. triggers triggers the the alarm alarm 10 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .BTS Site Manager Alarms vs. OSS. Faults Alarm Alarm reported reported to to OSS. depends depends on on severity severity and and scope scope BTS BTS fault fault name name (id). (id). RISE Viewer Link: Contains details fault information. 11 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . rather than the fault description. The alarm description will list all potential faults that may be the trigger for the alarm. BTS Site Manager Fault Descriptions When selecting an alarm via NetAct or iOMS the help shows the alarm description. • 3 Gbps can handle up to 40MHz combined antenna line BW (e. 12 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . 2x2 MIMO for one 20 MHz cell is 40MHz).g.Failure in RP-3 Interface Example RP-3 interface is OBSAI interface between baseband and RF unit. For FD-LTE. click on ? icon to get fault details to better understand root cause. Could indicate problem with SFP or optical cable. RP-3 may use 3 Gbps or 6 Gbps SFP. • Beyond the limit additional optical cable is required. Same Fault Name appears for several different faults. Indicates communication problem between system module and RF unit. Fault ID is important. Broken Optical Cable • Internal fault name is EFaultId_NoConnectionToUnitAl • Indicates no communication to RF unit • Check optical cable and connectors 13 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Bad or Missing SFP • Fault ID indicates EFaultId_OIC_PresAl which indicates SFP malfunction or no electrical connection to SFP • Check SFP is properly installed or replace SFP 14 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Wrong RP-3 Link Speed or Not Enough Optical Interfaces • Fault ID 4064 is EFaultId_MissingRp301LinkAl. which indicates not enough RP-3 links are connected (not enough bandwidth) • Check configuration and make sure enough RP-3 links are connected between system module and RF units 15 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . connecting.Status of Links to Adjacent Network Elements Search for MO to find details such as IP address and eNB ID. •• X2X2 interface interface may may be be blacklisted blacklisted by by adjacent adjacent eNB.eNB. IfIf X2 X2 link link is is Not Not Connected Connected there there is is not not always always aa fault fault generated: generated: •• IfIf LNADJ LNADJ is is eNB eNB controlled controlled SCTP SCTP association may time out before association may time out before connecting. 16 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Alarm History 17 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Alarm History Current 10s / Last 10 s Current min / Last min Current hour / Last hour Current day / Last day All 18 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Site Commissioning File Tools RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . inside.nokiasolutionsandnetworks. SCF tools • For parameter troubleshooting comparing the site commissioning file to another reference or the golden configuration file may help isolate differences or problems.nokiasolutionsandnetworks. • Normal side by side comparison is difficult. but there are tools that can help to make the comparison easier: • Flattener / conversion to CSV or XLSX to a single line per parameter https://twiki.com/bin/view/LTETechSup/ComparisonSCF 20 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .inside.com/bin/view/LTETechSup/SCFtoCSV • Compare SCF with a structured comparison that reports differences in: • Objects that exist only in one file • Parameters that exist only in one file • Parameters with different values • https://twiki. Golden Configuration via MINT tool GMC can be accessed from main MINT page 21 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Golden Configuration via MINT Tool • Select correct sw level and branch information. 22 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • Get SCF . • Select correct bandwidth for configuration to be compared. to download xml SCF file. BTS Snapshot (aka Technical Report) RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . BTS Snapshot or Tech Report Contents 24 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • LTEBTS\BTSLogFiles is only captured if the snapshot is saved with most important data. or all data options. • Log contents for each depend on which type of system module and transport module are used. • BTSLogFiles contains the most useful data for troubleshooting. 25 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Snapshot File Structure • Logs are split up into separate folders for transport (FTRS) and BTS (LTEBTS). TechReport Log Structure 26 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . inside.Tech Report (Snapshot) Tools and Info https://wft.nsn.inside.com/bin/view /LTETechSup/TechnicalLogDecoder TRBL Log List 27 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Workflow Tool .nsn.com TLDA information and online decoder can be found at: https://twiki. 0x130D.0x120D.0 Details information Shows which units Shows which log regarding file produce this file.gz 0 200000 0x1011.TRBL Log List Example core. file. level required for contents or size. 28 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Technical Log Decoder / Analyzer ########################################################## • Python must be installed in order ## TECHNICAL LOG DECODER ANALYZER ## to use the tool. ########################################################## • Instructions and URL for python Description: installation can be found in the This tool automatically decompresses, decodes and analyzes snapshot packages. README.txt under the Docs folder. Mandatory arguments: • This tool can decode / analyze --techlog_path : path to the compressed snapshot file or folder with already decompressed snapshot Snapshots and TechReport log Optional arguments: files. --output : specifies folder, where results should be saved Folder should be empty, otherwise execution will be terminated. --timestamp_merge : The option allows to merge and sort several files based on timestamps. Recognized timestamp format: <DD.MM hh:mm:ss.uuuuuu> - standard AaSyslog, <DD.MM hh:mm:ss.mmm> - present in some decoded TTi Traces Timestamp string of above defined format may appear anywhere in the line. By default, result is named: merged_files.log, but output file might be specified by using --output_file option. Please note: timestamp_merge functionality is independent of techlog decoded and analyzer functionality. --output_file : when --timestamp_merge is used, you can define output filename with this switch. --only : specifies action, which should be performed on snapshot package (default: all) CHOICES : ['decompress_outfiles', 'decompress', 'analyze', 'decode', 'merge', 'btsinfo', 'plugininfo'] --components : specifies components whose files should be decoded (default: all components) CHOICES : ['CPLANE', 'BTSOAM', 'RFSW', 'DSPHWAPI', 'LFS', 'CCS', 'HWAPI', 'TRSW', 'DLPHY', 'ULPHY', 'MAC_DCM_PS', 'MAC_WMP_PS', 'MAC_DATA', 'TUPU', 'DCM_UM', 'SMC', 'TDD_DLPHY', 'TDD_ULPHY', 'TDD_MAC', 'LOM', 'BTSSM', 'BM', 'DSP_COMMON'] --filter : specifies from which node files should be decoded and analyzed (default: all nodes) Sample usage: > python tlda.py --techlog_path D:\path\to\snapshot\file.zip > python tlda.py --techlog_path D:\path\to\snapshot\file.zip --output result_folder > python tlda.py --techlog_path D:\path\to\snapshot\file.zip --output result_folder --only decompress decode --components TUPU MAC_DATA --filter 1232 > python tlda.py --timestamp_merge file1.log file3.log 29 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 Basic Structure Snapshot files FSMr3 BTS FSMr2 BTS • For FSMr2 the transport module is a separate subsystem and the extended NE logs are saved under FTRS. • For FSMr3 the transport function is integrated with system module and the extended logs can be found under LTEBTS. 30 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 Important Transport Log Files Most import transport snapshot logs are in the file ExtendedNeLogs.tar - FtmDump.bin -> FTM runtimelogs (Log Reader tool needed to decode) - Config.xml -> FTM configuration file - Counters.log.gz -> FTM counters - staticLog.txt -> Information about file system, memory usage, running processes, interface settings, ARP/IP tables, IPSec keys FTLB, etc - ExtendedSystemInfo.log -> serial number, production information, MAC addresses, complete file system listing, IPSec keys FTIB,etc - Traces.log -> basic log level traces logreader.exe 31 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 Contents of Transport ExtendedNeLogs.tar (FSMr2) 32 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 Contents of Transport ExtendedNeLogs.tar (FSMr3) 33 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . 190.190.190.108.164 dst 10.108.164 proto esp spi 0xc57c9e4e reqid 16384 mode tunnel replay-window 32 flag 24 auth hmac(sha1) 0x12cec0c36f5baf3064d1afbd6309d7bf0e54385e enc ecb(cipher_null) 0x src 10.108.164 dst 10.IPSec Keys • IPSec Keys can be found in staticLog.108.164 proto esp spi 0x1118182b reqid 16384 mode tunnel replay-window 32 flag 24 auth hmac(sha1) 0x67bbb6e53729092ccb73ee77b12bd0253f0eaa41 enc ecb(cipher_null) 0x src 10.232.108.164 dst 10.164 proto esp spi 0xc51f0e73 reqid 16384 mode tunnel replay-window 32 flag 24 auth hmac(sha1) 0xd1e8a56f3faa626366b0e6a6ff9e1378fb324458 enc ecb(cipher_null) 0x 34 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .108.log or directly via the Octeon processor CLI via SSH: ********************************************** * * * ip x s * * * ********************************************** src 10.164 proto esp spi 0x113657b9 reqid 16384 mode tunnel replay-window 32 flag 24 auth hmac(sha1) 0x334d213af490e39f04fdd2f412259c60006d6d29 enc ecb(cipher_null) 0x src 10.232.232.232.164 dst 10.108.190.txt.108. ExtendedSystemInfo. LTE1099 Event-triggered Error Symptom Saving RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . Event-triggered Error Symptom Saving and Provisioning 36 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . received file in the archive file. Users is seeing the fault for which logs eNB collects the first file Each piece of the data is will be collected transferred automatically to the OMS as separate file using BTS O&M interface file upload BTS O&M interface: File upload procedure eNB collects procedure. BTS fault that triggers log collection When a trigger fault is noticed. See A1 in figure on slide 5 the next file The OMS packs and compresses all files in one troubleshooting data OMS creates an archive file and save file or BTS Technical log. (TRSH_id is included) Alarm event for the fault buffers / takes (TRSH_id is included) snapshot BTSOM starts uploading of troubleshooting data to OMS. the BTSOM freezes buffers and takes Alarm event for the fault eNB freezes snapshots of data. Event-triggered Error Symptom Saving Procedure NetAct iOMS eNB Techlog is triggered by the BTS when a raw alarm is raised. for the first file. 37 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . procedure for the last file. OMS will indicate that file. OMS sets 71106 alarm indicating that a New troubleshooting data file is available On receiving a notification about available troubleshooting data. 38 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . management of troubleshooting data is PIFileTransferred quite similar to Performance Management data handling. When troubleshooting data have been uploaded from an NE and stored in their OMS adds file to the archive eNB recovery reset temporary location. (optional) data are ready for upload via a FilesReadyEvent_v1 notification. TROUBLESHOOTING DATA RECEIVED in eNB to for the next file. NetAct will use its AlarmEvent (TRS data received) Performance Data Transfer Management Service to retrieve these from OMS. FilesReadyEvent_V1 File upload from iOMS From the overall handling perspective. Loop as long BTS O&M interface: File upload eNB collects The OMS sets 71106 as there is file procedure the next file. Event-triggered Error Symptom Saving Procedure NetAct iOMS eNB The BTS deletes frozen data immediately after successful transfer to the OMS. alarm to inform the user that OMS received collect new troubleshooting data file. OMS indicates to NetAct via alarm (71106) that Troubleshooting Data will become BTS O&M interface: file upload available to NetAct User. iOMS adds file to the archive file. Provisioning and Parameters 39 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . technical log collection.Provisioning and Parameters • New MOC under MRBTS allows up to 20 • iOMS does not require any parameter fault IDs to be configured as triggers for changes for feature to be enabled. • New parameters are under OMS • Online modifiable changes – does not System parameters. require restart. 40 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . log • For verification of file transfer of technical log to NetAct: /var/log/auth.Location of Technical Logs • In iOMS techlogs are stored in directory: /var/opt/OMSftproot/NE/TroubleshootingData/ • For details regarding BTS to iOMS communication and file transfer: /var/log/oms/NELogManager_trace.txt • For details related to alarm 71106 forwarding to NetAct: /var/log/nwi3adapter. Can also be found in the alarm details. technical report can be retrieved from following directory on NetAct CS server: /var/opt/nokia/oss/global/ltemei/lte_ne_symptom_file File naming convention: TSUPL_eNB_<eNBid>_<TRSHid>.txt • For details regarding notification to NetAct regarding technical log file available /var/log/oms/genericpmctrlSrv_trace.log • After transfer to NetAct log file is deleted from iOMS (similar to PM data) • After transfer to NetAct. 41 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .tar <TRSHId> identifier of the troubleshooting task in the eNB.<date-time>. How to Retrieve Technical Report Log from iOMS • If log has not been deleted from iOMS you can also fetch the log via OMS Element Manager web interface. using Log Viewer tool. 42 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . “Loading” indicates file transfer is ongoing.Example Alarm detail from eNB includes TSHId to indicate log collection is triggered. 43 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . automatically clears after 15 minutes. • Temporary alarm.Example • Alarm 71106 shows up after file transfer complete. 44 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . then can be found in alarm history. • Identifies filename of complete technical report log. LTE1460 Local and Remote IP Traffic Capturing RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . ARP) are NOT captured. 46 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .g. The NE implementation prevents malfunction of the IP traffic capturing session in case of mutliple Web sessions at the same time. • Layer 2 information and protocols (e.Introduction • The feature extends the troubleshooting capabilities of a BTS. • The IP traffic capturing functionality is configured via Web interface. Capturing is possible locally and remotely connected from a Network Operations Center. Ethernet headers. Ethernet slow protocols. The access to the Web interface is secured with operator-configurable credentials. The NE supports a single IP traffic capturing session at the same time even though the Web interface may support multiple Web sessions at the same time. • Requires FSMr3 system module. for remote connection use https://management_plane_IP 47 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Functional Description Login for LMP shown. 48 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . uplink traffic has already passed it. At this capture point downlink traffic is not filtered by the system firewall. A IP traffic to/from the BTS.Functional Description B IP traffic to/from the BTS and SSE captured at the transport network interfaces. TLS-encrypted M-plane is still encrypted. At this capture point the traffic is captured fragmented and encrypted (if in use). M-plane is contained unencrypted or decrypted (in case of IPsec in use). At this capture point the traffic is captured non-fragmented and IPsec decrypted (if in use). If TLS is not in use. Wireshark) on the tracing PC 7. NOTE: If LMP port is used as output. Infinite streaming begins when clicking the START button. Select the interface through which the trace PC is receiving the stream. it is limited to 100Mbps as it is a FastEthernet port. Select Capture Output option: Streaming 4.g. Start an IP tracer (e. Select the Capture Point A or B 2.Activation of the feature 1/3 Capture Output: Streaming 1. Enter the trace PCs MAC address (get it through ipconfig –all) 6. 5. Select whether User-Plan shall be included in the streamed messages 3. 49 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . When clicking the START button. 50 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Select the Capture Point A or B 2.. When clicking the GEN&DOWNLOAD_PCAP button. the starts creating the message file and stores it in its RAM memory 4.ZIP-file.. Select whether User-Plan shall be included in the streamed messages 3.. the message capturing stops and the messages will be converted to PCAP format and compressed to a .Activation of the feature 2/3 Capture Output: File 1/2 1. It is a ring buffer. 40MB. so older packets will be overwritten once the max is reached. 51 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . the File Download window pops up and lets you browse for the storage location. NOTE: The raw trace file is limited to 150MB.Activation of the feature 3/3 Capture Output: File 2/2 5. When the zipped PCAP file is ready for download. • Compressed file uploaded is expected to be approx. • Header information allows analysis of user traffic by identifying applications and QoS mapping.User Plane Capture • The user plane capture option captures user plane IP and transport headers (UE IP and transport in addition to LTE Interface). 52 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • User payload is not included. • Alarm will indicate if it is local streaming (with MAC) or file capture IP Capture to file is ongoing (max of 150MB) IP Capture to LMP port is being streamed to MAC address shown.Related Alarms • When IP traffic capture is started. 53 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . an alarm is triggered to notify others. LTE1408 Remote Syslog RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . error correction times and service O&M DCN effort from customer and Nokia Base Station with syslog client 55 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Remote Syslog Main trigger:  Limited memory for syslog storage in eNB. logs frequently Remote syslog overwritten and not useful for problem investigation server Solution:  Syslog messages from different sources within the eNB are aggregated by TRSW and pushed continously to the remote syslog server  eNB acts as one source (client) towards the remote syslog server Benefit: Syslog  Logs without suffering from the impact on the logfile size  Reduced outage times.LTE1408 . BTSEM.LTE1408 – Syslog Client Key Functionality • Remote syslog client is a functionality implemented in BTSOM. CC&S and TRSW • Remote syslog client is disabled (not commissioned) by default • Remote syslog client may be enabled by customer through BTSEM (described later) or optionaly via NetAct (CM Editor + CM Operation Manager) • Remote syslog client is started without a service impact (eNB reset is not required) • Remote syslog transmission is not disabled by eNB restart • Remote syslog transmission is automatically stopped after a defined period of time • eNBs sends syslog prints to the remote syslog server in addition to storing the logs in its own buffers 56 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . LTE1408 – Syslog Server Key Functionality Syslog server RL70 iOMS* Customer’s server  3rd party application Rsyslog Linux Windows  CLI tool fsremotesyslog  Application to receive syslog:  Application to receive syslog:  RFC 5424 compliant syslog Rsyslog pyRemoteLogServer or only  RFC 5424 compliant syslog only Rsyslog + Rsyslog Windows  Limited number of Agent simoultnaous remote syslog  For TLS the customer’s CA signs  RFC 5424 and Prioprietary connections the certificate for the Linux PC (just syslog formats possible  TLS supported like it does for the eNB & iOMS)  TLS mode may be additionally simulated in pyRemoteLogServer * The remote syslog server shall primarily be a platform owned by customer. In emergency situations iOMS can serve as the remote syslog server but only for a small number of eNBs 57 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • Syslog IP packets have fixed DSCP=0 and they are scheduled according to common configured DSCP to PHB mapping at the transmitting network IP interface. Certificate generation is customer’s responsibility) 58 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • Supported file transfer protocols (configurable) for syslog traffic are: • UDP/IP (CC&S sends UDP packets towards TRSW up the maximum packet size of 2048 Octets) • UDP/IPSec (IPSec is transparent from feature p. .packets are sent via IPSec tunnel if IPSec is configured for eNB) • TLS/TCP/IP (iOMS and eNB authenticates each other with certificates during establishment of TLS connection for remote syslog transfer.o.v.LTE1408 – TRS Implementation • Syslog IP packets are transmitted with eNB Management plane IP address as a source IP address • Syslog IP packets are transmitted with a remote syslog server IP address (configured in commissioning) as a destination IP address. Flexi 10 FDD LBTS7.LTE1408 – Supported Platforms System SW Release: RL70 and RL55TD onwards Supported eNB platforms: . FZ Micro FDD LBTS7.0.0 (FSMF).0 .Flexi 10 TD-LTE TD-LBTS5.Flexi FDD LBTS7.0 (FSMD/E) support removed from content Supported iOMS platforms: .0 (Flexi Zone) .iOMS hardware with a disk capacity of 300GB and higher [OMS_EFS_LTE1408_252] .At least 50 GB for storing remote syslog files [OMS_ENFS_2311] 59 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Flexi 10 Indoor TD-LTE TD-LBTS5. LTE1408 Client Feature Activation Feature is activated. NOTE: Remote syslog transmission is also stopped in the following cases: • Specified duration (Remote syslog transmit duration) ends • CPU load alarm (6500 or 6501) is raised • OMS switchover happens (discussed later) 60 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . This indicates the syslog transmission has completed the transmission duration and is not longer sending syslog data. but status is false. TLS requires proper signed certificate to match eNB. • Format is Proprietary for Nokia tools (including BTSLog) • Remote syslog server IP may be OMS (if supported) or customer syslog server. • Protocol can be UDP or TLS. can also be set to 51000 for R&D BTSLog tool. or R&D BTSLog tool PC (if connected via LMP then IP should be set to 192.126). • Remote syslog server port is 514 for UDP or 6514 for TLS.255.168.LTE1408 Client Feature Activation • Format is RFC5424 for OMS or 3rd Party server. 61 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Changing Syslog Print Levels • It is possible to control the print level of syslog output using the BTS Log tool from BTS Site Manager.txt file. 62 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • BTS Log print level of Full is equal to setting the flags 0x1003F=1 and 0x10040=1 which is DEBUG. • It is still possible to change the print level via R&D parameters or swconfig. • BTS Log print level of Normal is equal to the default value with print level INFO. fsremotesyslog disable <udp | tcp >-listener .LTE1408 – OMS Syslog Server Configuration Use fsremotesyslog CLI tool To enable UDP connection type the following commands: .fsremotesyslog commit Check the status Remote syslog server state .fsremotesyslog commit To enable TLS-connection use: .fsremotesyslog add source udp:<address ip | address/mask | any> .fsremotesyslog enable udp-listener .fsremotesyslog commit 63 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .fsremotesyslog enable tcp-listener .fsremotesyslog status To disable the following commands: .fsremotesyslog add source tcp:<address ip | address/mask | any> . Flexi MRBTS LTE eNB HW Architecture RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . One FSP Control MCU for cell level ASIC CTRL Ethernet Faraday CP tasks MCU Switch #4 .Basic eNB system knowledge – HW architecture: FSMr2 RF Module #3 RF Module #2 Extension Module RF Module #1 Extension Module • FSM Rel2 consists of FlexiBTS Control and Multiplexing unit (FCM) and 1-3 Flexi Signal Processing units (FSP) • FSME contains 3 FSP units OBSAI RP3/01 - • FCM .7 Faraday DSP for cell UP related task Faraday #1 Faraday #2 Faraday #3 . IXP2350 To/from DAC GW GPS Transport Block 65 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . NPU Block FPGA.One FSP for up to 2 cells MUKSU .3 Faraday Cores under each Faraday for separating UP processes Timing CTRL Ethernet Block MCU Switch GEthernet OCXO.One MUKSU ASIC for routing data to FSPC BB Card #3 RF Modules via optical RP3/01 FSPC BB Card #2 interface FCM RP3 bus FSPC BB Card #1 3072 Mbps • FSP Faraday #7 Faraday Faraday #6 #5 .One FCM Control MCU for eNB level CP tasks Flexi System Module (FSME) . e. IIC 3V3_EXT buffers ETH_SEL IIC_SEL RP3 RP3 RP3 RP3_L3_TX_p/n RP3_L3_RX_p/n RP3_L5_TX_p/n RP3_L5_RX_p/n Faraday Faraday Faraday Faraday SRIO_L0_TX_p/n #7 #6 #5 #4 SRIO_L0_RX_p/n SRIO CLOCK BLOCK SRIO SRIO SRIO CLK_STATUS Clock IIC RP1_CLK_p/n drivers Frame burst DDR2 DDR2 DDR2 DDR2 RP1_BURST_p/n RP1 clock and SDRAM SDRAM SDRAM SDRAM RP3 clock sources Local clocks FARADAY FARADAY FARADAY FARADAY BLOCK#7 BLOCK#6 BLOCK#5 BLOCK#4 66 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Transmission Module is always needed FSPRST 3V3_EXT Ctrl JTAG_Int 1V1_adj 1V1 S G M I I_6 SG M II_7 1V2 48VDC_POS Bus Conv 1V8 48VDC_NEG conv 1V8_MCU POWER _OFF 2V5 3V3_INT VCNTL_n[3:0] FNA[3:0] EXT_IF BLOCK ( from all Faradays) I2C_Int I2C_Int I2C I2C McBSP_TDM BSCAN JTAG_Int FNA[3:0] devices.Basic eNB system knowledge Single-ended signal FARADAY FARADAY FARADAY Differential signal BLOCK#1 BLOCK#2 BLOCK#3 Single-ended bus DDR2 DDR2 DDR2 Power supply input – HW architecture: FSMr2 SDRAM SDRAM SDRAM Local power plane SRIO_L1_TX_p/n SRIO SRIO SRIO SRIO SRIO_L1_RX_p/n RP3 RP3 RP3 RP3_L0_TX_p/n RP3_L0_RX_p/n Faraday Faraday Faraday RP3_L1_TX_p/n #1 #2 #3 RP3_L1_RX_p/n RP3_L2_TX_p/n RP3_L2_RX_p/n MCU BLOCK • Faradays are connected DDR2 FLASH FNA[3:0] McBSP_TDM SDRAM I2C_Int • Together via SRIO bus ring architecture 256MB 64MB Resets& L2_CONFIGs • To FSP and FCM MCU via Ethernet SMI GMII FNA[3:0] • To RF modules via RP3 and MUKSU ASIC MPC SG M II_ 1 SG M II_ 2 SFCLK_p/n IIC Flash_prog 8548E SWITCH ETH_SEL • FCM MCU is connected to UART FSIB IIC_SEL CLK_STATUS BLOCK POR INT_ETHERNET SGMII_3 • Transmission interfaces via Ethernet and RP2_RX_p/n CodeWarrior port SGMII_0 RP2_TX_p/n SGMII_4 Ethernet Transmission Module POWER BLOCK IIC INT_ETHERNET Switch SGMII_5 • I. since it contains an Octeon processor with 8 cores • FTIB IPSec implementation is SW based • FTLB can handle fragmented IPSec packets. FTIB for FSMr2 • FTLB has much higher IPSec performance. Basic eNB system knowledge – HW architecture: Transport Modules FTLB. FTIB does not due to the HW limitations (might change in the future) FTLB FTIB 67 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . SRIO RF Modules via optical RP3/01 interface • FSP Flexi System Module .One FSP for up to 6 cells (FSMF) .4 Nyquist Cores under each Nyquist for Nyquist Nyquist Nyquist separating UP processes #1 #2 #3 • More BB capacity can be allocated with FBBA/FBBC (Flexi Baseband Extension Module).3 Nyquist + 3 Turbo Nyquist DSP for 3072 Mbps Nyquist Nyquist Nyquist cell UP related task #6 #5 #4 Turbo Nyquist has additional FFT and MURKKU ASIC Decoder processors suitable for certain Octeon MCU SRIO Switch UP tasks . RP3.One FSP 6 Core Octeon MCU for cell level CP tasks FCT RP3 bus FSP .One MURKKU ASIC for routing data to OBSAI RP3/01 - RP3/01.One FCT 6 Core Octeon MCU for eNB level CP tasks and Transmission tasks . FBBA/FBBC contains one FSP Timing Octeon Ethernet Block MCU Switch OCXO SRIO Switch GPS 68 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Basic eNB system knowledge – HW architecture: FSMr3 RF Module #3 • FSM Rel3 (FSMF) consists of FlexiBTS Control RF Module #2 FBBA #2 and Transport unit (FCT) and one Flexi Signal RF Module #1 FBBA #1 Processing units (FSP) • FCT FSP . Basic eNB system knowledge – HW architecture: FSMr3 • Nyquists are connected • Together via SRIO bus switch architecture • To FSP and FCT MCU via SRIO bus switch architecture • To RF modules via RP3 and MURKKU ASIC • FCM MCU is connected to • Transmission interfaces directly via Ethernet • optional transport module (FTIF) is needed for supporting • Synchronization from E1/T1 interface • eNB transport chaining with optical (2x SFP) or electrical connection • Synchronization Hub function based on Synchronous Ethernet input or output • Transport link redundancy with optical (2x SFP) or electrical connection • CESoPSN (Circuit Emulation Service over Packet Switched Network ) for legacy BTS co-location 69 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . iOMS Troubleshooting RA41141EN08GLA0 © Nokia For Solutions and Networks 2015 internal use . iOMS Alarms 71 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Useful Commands for iOMS CLI 72 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Useful Commands for iOMS CLI 73 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Operational Status 74 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Contains all logs with severity set to debug except for the AUTH and AUTHPRIV facilities.log The authorization log. Creating the debug is not enabled by default. auth. which is one of the security logs. Log Files Log file Description syslog Contains all logs with severity set between info and emerg except for the AUTH and AUTHPRIV facilities.conf 75 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Log files are stored on iOMS filesytem under /var/log To enable or disable debug trace logs: # ztracecli –on # ztracecli –off # ztracecli –p (to print current status) NWI3 traces can be enabled or disabled (log file saved to /tmp/trace.pcap It is advised to clean the log file before problem reproduction.log file. For this purpose please run logrotate -f /etc/nwi3Adapter_logrotate. The system writes to it information on user actions and accesses to the system.log): # fsnwi3ctl -l debug # fsnwi3ctl -l off Wireshark traces can be enabled via: # tshark -i bond0 -f "host <host ip>" -w /tmp/<filename>. debug Belongs to troubleshooting logs. Contains logs of any severity with the AUTH and AUTHPRIV facilities. Only the root user can view the auth. Collected data is stored for example into file named: '/var/log/oms/TroubleshootingData/TSUPL_OMS_OMS- 1.gz' .21832. you will have to copy /tmp/trace.20110308110340.tar. MEMory and filesystem status • Process status • Active alarms Data collected in full mode in addition to default data collection: • Also older (all) log files generated by the Flexi SW Platform • Also older (all) log files generated by applications • Analyze from core files generated by process crashes • Cluster Environment information (FlexiPlatform environment camera tool) .tar.21832. NWI3 Trace logs will NOT be collected.20110308110340. --help : show detailed help --full : collect more troubleshooting data (full mode) --bzip2 : use bzip2 instead of gzip -o <file> : store troubleshooting data to <file> instead of default (/var/log/oms/TroubleshootingData/TSUPL_OMS_OMS-1.Collecting Log Files To collect the most common log files: zcollectlogs [options] -h.log separately 76 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .435007000.435007000.gz ) Data collected in default mode: • Most recent log files generated by the Flexi SW Platform • Most recent log files generated by applications • OMS installation log files • Static HW and SW configuration of the OMS • OMS local license and feature information • CPU. Enable Performance Measurements Backup • By default PM data is deleted from iOMS after sending files to NetAct • It is possible to enable PM data backup for storage for up to 7 days • Change the iOMS LDAP configuration settings .after the file backup is enabled and next measurement period has finished.Open the Parameter Editor from the iOMS Web GUI . .double-click the BackupEnabled parameter and value and set it to “copying enabled”. PM files can be found in /var/opt/OMSftproot/xmlfilesBU 77 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Select the Performance Management tab . ztopocli -r MRBTS-2 Example: # ztopocli -lc List Connected OMS topology database contains following connected network elements: 1.g. MRBTS-1 3. MRBTS-3 2.Topology Ztopocli is used to list or remove network elements from the OMS topology database. MRBTS-2 78 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . E. Usage: ztopocli OPTION [NE] -l list network elements in the OMS topology database -lc list connected network elements in the OMS topology database -ld list disconnected network elements in the OMS topology database -r NE remove network element from the OMS topology. has detailed procedure descriptions for administrative tasks and troubleshooting guides. available via NOLS ( https://online. Operating Documentation.portal.Additional Information The LTE Radio Access. Rel.com/nolshome/homepage). • Installation and Commissioning procedures • Software management • User management • Backup and Restore • Certificate management • Time and Log management 79 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . RL70 (RL55).nokiasolutionsandnetworks. iOMS Debug Logs via Parameter Tool Set to trace level via Parameter Editor or via CLI interface with ztracecli command 80 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . • Logs can be viewed via web interface https://<nortbound_ip> then select OMS Log Viewer 81 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .Viewing iOMS Logs • Logs via command line /var/log for system logs and /var/log/oms for debug traces. Disable TLS Security on iOMS 82 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 . Questions? 83 RA41141EN08GLA0 © Nokia Solutions For internal use and Networks 2015 .