UCCX TroubleshootingAngelique De Vos Customer Support Engineer March 2017 Agenda Problem Isolation UCCX Engine - CUCM Interaction UCCX Database and Reporting Live Data HA Failover System Monitoring Tools – CUIC & RTMT Finesse Supervisor Call Monitoring Post Call Treatment Call Recording with Mediasense Single Sign-On (SSO) Problem isolation Clarify – Ask the right questions • What is the exact issue? • Who is facing this issue? • When did this issue start? • What else? Isolate • Break the problem down • Exclude as many factors as possible to pin it down to a single suspect/component • Do not jump to conclusions! • Identify the evidence needed to confirm the root cause UCCX – CUCM interaction UCCX– CUCM integration overview PSTN Router/GW UCCX Communications Manager Cluster Agent Stations SIP (5060) SIP/H323/MGCP (5060/1720/2427) LDAP (389 | 3268) CTI/JTAPI (2748) AD/LDAP Recording AXL (443 | 8443) CTI (12028) SCCP | SIP (2000 | 5060) Inbound PSTN call flow 6 2 1 PSTN 4 3 6 Router/GW 23 5 5 4 5 Agent Stations UCCX 1. Inbound PSTN Call to VGW. CallManager 2. VoIP leg to CUCM. Supervisor Cluster 3. CUCM routes call to RP registered by UCCX. Stations 4. UCCX responds with redirect request to available CTI Port. 5. UCCX triggers application, instantiates and executes script. Call is Agent ICD Extension: 6000 answered upon Accept step executing. CTI Route Point 8222 6. UCCX provides scripted queuing treatment or identifies an available CTI Ports: 10001-9988-9990 resource and consult transfers the call to the agents device. Check Engine status – Partial service? Sample call analysis //Call is offered to CTI RP 2872713: Mar 28 09:15:31.264 IST %MIVR-SS_TEL-7-UNK:Route Connection=[8222::1/(P1-abhi_1) GCID=(1,7057)->ACTIVE]->OFFERED, reason=1, Event= (P1-abhi_1) 7057/1 CallCtlConnOfferedEv 8222::1 [#827] Cause:100 CallCtlCause:100 CiscoCause:0 FeatReason:12, cause=100, metacode=129, isMaster=true //CTI Port selected 2872720: Mar 28 09:15:31.266 IST %MIVR-SS_TEL-7-UNK:Route Connection: [8222::1/(P1-abhi_1) GCID=(1,7057)->ACTIVE]->OFFERED, CTI Port selected: TP[id=1,implId=9989,state=IN_USE] //Call disconnected at the CTI RP since it’s REDIRECTED 2872734: Mar 28 09:15:31.299 IST %MIVR-SS_TEL-7-UNK:RP[num=8222], conn=[8222::1/(P1-abhi_1) GCID=(1,7057)->ACTIVE]->DISCONNECTED, event=(P1-abhi_1) 7057/1 CallCtlConnDisconnectedEv 8222::1 [#842] Cause:100 CallCtlCause:210 CiscoCause:0 FeatReason:6, cause=CAUSE_NORMAL[100], meta=META_CALL_REMOVING_PARTY[131] Common call failure scenario - #1 Partition and CSS issue: it is important to note that the original calling number should be able to reach BOTH THE CTI ROUTE POINT AND THE CTI PORTS. In the MIVR logs, the following error message is seen: 302849: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-ROUTE_FAILED:Route failed: All Call ids=JTAPICallContact[id=10,implId=1481/1,state=STATE_RECEIVED_IDX,inbound=true,App name=sample_1,task=null,session=1000000011,seq num=0,cn=7420,dn=7420,cgn=1009,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null,ocn=7420,route=RP[num=7420],TP=null,List of Active Connections=[7420::1/(P1-sydney_cti_1) GCID=(1,1481)->ACTIVE]- >OFFERED,Extension=7410,Exception=com.cisco.jtapi.InvalidPartyExceptionImpl: Attempt to redirect to an unknown destination,Failure reason= call will be rejected, CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION=0x8ccc0034::Attempt to redirect to an unknown destination,Contact.Reject.reason=TRIGGER_FAIL,(SelectRouteTime,ObtainingIdleChannelTime,RedirectTime=0,0,16) 302850: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION:com.cisco.jtapi.InvalidPartyExceptionImpl: Attempt to redirect to an unknown destination 302851: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION: at com.cisco.jtapi.ConnectionImpl.redirect(ConnectionImpl.java:1378) 302852: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION: at com.cisco.jtapi.ConnectionImpl.redirect(ConnectionImpl.java:1261) Common call failure scenario - #2 Codec problem between caller and the CTI port: Use the reference callctl to identify the issue with the establishment of media due to codec related issue. In the MIVR logs, the following error is seen: 485624: Sep 25 12:29:14.958 EEST %MIVR-SS_TEL-7-UNK:CallID:763 MediaId:48650/1 Task:21000000978, CallCtlConnFailed, Inbound call, callctl cause:107, [5519:VZD_CC/(P1-CTIAdmin_1) GCID=(1,48650)->ACTIVE]->FAILED Solution: Check the codec requirements and if needed, add transcoders so that the call leg can be established. Look at UCCX JTAPI cause codes on the web to get a list of all possible failure codes. UCCX Database and reporting Logical overview of databases UCCX Configuration Datastore Historical Datastore CCX DB Repository Datastore CM Platform Finesse CUIC Platform Platform database instance Database What does it store? How to query it? ccm cluster and platform run sql select name,nodeid from processnode info, configuration cuic_data CUIC application data run sql select name from cuic_data:cuicuser db_phx_config Finesse config No connect permissions via CLI, this needs TAC remote account • Can be found as “A Cisco DB” when you list the services • To check replication use “utils dbreplication status” UCCX database instance • Can be found as “Cisco Unified CCX Database” when you list the services Database What does it store? How to query it? db_cra Historical datastore – run uccx sql db_cra select count(*) from resource historical data where active (COUNT(*)) Configuration datastore – configuration data from CCX admin (CSQ’s, applications) db_cra_repository Repository datastore run uccx sql db_cra_repository select * from – scripts, prompts, DocumentsFiletbl documents,… UCCX database instance • To check replication use CLI command “utils uccx dbreplication status” or the UCCX serviceability pages. Active/Connected is a good replication status, Dropped/Timeout indicates a replication issue DBReplication commands to check configuration • utils ntp status • utils diagnose test • utils network connectivity <secondary node> • utils uccx dbreplication dump configfiles Configuration changes when subscriber is down • Both nodes must be available to commit changes to the CDS (Configuration Datastore) CCX Administration (MADM) Logs: %MADM-ADM_CFG-7-UNK:ICDServlet :: Exception occurred The SUBSCRIBER node of the HA is not available %MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception: Exception=java.lang.RuntimeException: The SUBSCRIBER node of the HA is not available • You can temporarily disable CDS on the subscriber to make changes on the publisher UCCX DB replication reset • Replication can be broken if Subscriber is unavailable for too long and send queues buffer is exceeded • Typically 3-4 days (*can vary with load) Alert Raised! DBReplicationStopped • Alert will be raised Tools-> Datastore Control Center-> Replication Servers Issuing a Reset Replication causes the following to occur: 1. Remove database replication utils uccx dbreplication teardown 2. Setup database replication utils uccx dbreplication setup 3. Perform data repair process utils uccx dbreplication repair all Force database sync - CLI Remote DB CDS Overwrite Target = Local! CDS HDS HDS RDS RDS Cluster Reboot + Replication Reset required! Reporting – CUIC datasources CUIC Data Sources: Publisher CUIC uses Subscriber’s db_cra Subscriber’s CUIC Also uses Subscriber’s db_cra Historical reporting – no DB available • If no Database is available, historical data records are written to flat file and alert is raised Primary Server Secondary Server M Replication S ! won’t occur ! Alert Raised! DB DB HistoricalDataWrittenToFiles If both DBs are down Engine File System File System writes records directly to file • Fix Database issue. • Then, manually import records from file system with Tools-> Historical Reporting -> File Restore menu from CCX Admin Database issues – further analysis • From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Unified CCX Database DB online.uccx.log file for issues related to database service DB REPLICATION uccx_repl_output_util.log file for issues related to database replication or at a minimum gather these logs for TAC Reporting and database performance • Historical reports are running slow • Overall high CPU Looking into processes, you will find that ‘uccxoninit ‘ is using the highest percentage of CPU time Reporting and database performance • Review the configured CUIC reports and their frequency • Check the historical tables size Reporting and database performance • If tables have millions of rows, review purge config: Default of 90 months can be reduced Live Data Live data design overview Live data logs • Engine logs (MIVR) –ICD_RTDM, SS_RMCM, upto Xdebugging5 • SocketIO Service logs – DEBUG mode Socket IO debugs 1. The Socket IO Server receives messages from the JMS bus for the Topic. 0000001076: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530: %CCBU_Camel (camel-3) thread #3 - JmsConsumer[ResourceIAQStats]-6- MessageProducer: %[message={"id":"agent1","operation":"UPDATE","ResourceIAQStats": {"resourceId":"agent1","resourceName": "agent1","resourceState":3,................. .. room_name=ResourceIAQStats*agent1][room_prefix=ResourceIAQStats][sta tus=RECEIVED][topic=jms:topic:ResourceIAQStats]: Event Detail Trace Socket IO debugs 2. Socket IO Server processes the messages and maps them to the appropriate rooms. 0000005789: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530: %CCBU_pool-19-thread-1-6-MessageDispatcher: %[client_count=- 1][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats": {"resourceId":"agent1","resourceName":"agent1","..... .. room_name=ResourceIAQStats*agent1][socket_io_server_type=WS/WSS][status=EN QUEUED]: Event Detail Trace Socket IO debugs 3. SocketIO message dispatcher sending the update to clients 0000005791: 10.106.87.133: Jan 26 2016 18:57:27.008 +0530: %CCBU_pool-19-thread-1-6-MessageDispatcher: %[client_count=2][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resour ceId":"agent1","resourceName":"agent1","resourceState":3, "durationInStateMillis":426719,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDurati on":1165394, "longestTalkDuration":2266049,"avgHoldDuration":0,................... ..... [room_name=ResourceIAQStats*agent1][socket_io_server_type=WSS][status=SENT]: Event Detail Trace Socket IO debugs 4. On the Browser Side logs (F12) we see the following: Message received on socket::{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceId":"a gent1","resourceName":"agent1","resourceState":3,"durationInStateMillis":264720 ,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDuration":1165394,"longest TalkDuration":2266049,"avgHoldDuration":0,"longestHoldDuration":0,"avgHandleDur ation":0,"avgWorkDuration":0,"totalTalkTime":6992368,"totalHoldTime":0,"maxRead yTime":10199923,"avgReadyTime":1156099, HA Failover High Availability CCX Component Redundancy Reporting Primary CCX Server Secondary CCX Server • CUIC reporting Engine always queries M Engine Engine S • First started the Slave DB for becomes Master reporting. Database Database • This is to • Primary is preferred Master Reporting Reporting prevent load on • Other node will the current be Standby Finesse Finesse Master DB for large queries. Databases • Comprised of Historical, Agent, Repository Finesse and Config Datastores (HDS, RDS and CDS) • Finesse always talks to the Master •DB mastership prefers Engine Master (unless Engine. DB is down on Engine Master) • Finesse Failover follows Engine • Data is read from / written to DB Master Failover • HDS/RDS has Publisher/Subscriber and • Finesse service Restart does not syncs using Informix Enterprise Replication initiate FAILOVER • CDS uses distributed transactions (both databases must be operational for updates) HA – Failover detection • Cluster View Daemon (CVD) sends and monitors heartbeats between CCX nodes HA over LAN HA over WAN Heartbeat sent every 500 Heartbeat sent every 1 ms second TCP Heartbeats Failover after 5 missed Failover after 10 missed heartbeats Primary CCX Server Secondary CCX Server heartbeats Failover initiated within 5 Port Failover initiated within 10 5900 seconds CVD CVD seconds Port 5900 TCP Heartbeats HA – Lost Connectivity Secondary CVD detects missing keep-alives from Primary CVD %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT: CVD does not receive heartbeat from node for a long period: Primary CCX Server Secondary CCX Server nodeId=1,dt=2049 ..... %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT: CVD M ? ? S ? M CVD CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10245 %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash: Engine ! Engine state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5 %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash: Databases Databases state=Convergence State,nodeInfo=Node id=1 ip=192.168.12.5 Reporting Reporting Master Election initiated on Secondary CVD %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: CONVERGENCE_STARTED, name=Cisco Unified CCX Engine Finesse Finesse 7631: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:JavaService66: Cisco Unified CCX Engine on node 1 change master from true to false 7632: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1 • Primary lost connectivity Secondary CVD elects Secondary as Master %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on node 2 change master from false to true %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED, • Master Election initiated name=Cisco Unified CCX Engine, node=2 %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine • Master Election completed • Failover completed HAoWAN – Island mode Primary CVD detects missing Heartbeats Primary CCX Server Secondary CCX Server and assumes secondaryhas failed %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT: CVD does not receive heartbeat from node for a long period: nodeId=2,dt=10197 CVD M M S ? M CVD %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash: state=Convergence State,nodeInfo=Node id=2 ip=192.168.13.5 Engine Engine ! Secondary CVD detects missing Heartbeats Databases Databases and assumes Primary has failed %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT: CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10242 Reporting Reporting %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash: state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5 Finesse Finesse Secondary CVD performs Master Election %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: CONVERGENCE_STARTED, name=Cisco Unified CCX Engine %MCVD-CLUSTER_MGR-7-UNK:JavaService66: • WAN network failure Cisco Unified CCX Engine on node 1 change master from true to false %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1 • Missing Heartbeats detected Secondary CVD elects Secondary as Master %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on node 2 change master from false to true • Master Election performed %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED, name=Cisco Unified CCX Engine, node=2 %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine • Failover completed HAoWAN – Network restored Primary CVD detects Secondary and keeps Master %MCVD-CVD-7-UNK:Split after network partition is detected, new nodeId=2 Primary CCX Server Secondary CCX Server %MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1 because primaryEngineComputerName=UC85CCXPRI CVD M M M ? S CVD %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine conditional- Secondary CVD detects Primary and drops Master Keep-LocalMaster from localServiceCisco Unified %MCVD-CVD-7-UNK:Split CCX Engine after network on node partition 1 is detected, new Engine Engine ! nodeId=1 %MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1 Databases Databases because primaryEngineComputerName=UC85CCXPRI %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine Reporting Reporting Secondary CVDfrom DropLocalMaster performs Master election and drops localServiceCisco Unified CCX Engine on node 2, Master conditional=true %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: Finesse Finesse CONVERGENCE_STARTED, name=Cisco Unified CCX Engine %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on • WAN network restored node 2 change master from true to false %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED, • CVDs detect dual Masters name=Cisco Unified CCX Engine, node=2 Secondary CVD completes Master election %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event: • Master Election performed CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine • Failover completed HA Failover issue – further analysis From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Unified CCX Cluster View Daemon System monitoring tools CUIC – Utilization monitoring CUIC provides stock reports which lets admins view license utilization and monitor system usage: • License Utilization Hourly report: hourly breakdown about number of inbound and outbound ports used, as well as agent seats • Application Performance Analysis report: information about calls presented, calls handled, abandon rate per hour and also the average call duration. • Aborted and rejected report: The Aborted and Rejected Call Detail Report provides information about each call that is aborted or rejected by the system CUIC – Aborted and rejected report • Incoming Call Contacts are Classified as Aborted if an exception occurs in the workflow that is processing a call. • Incoming Call Contacts are Classified as Rejected when a UCCX Application system resources reach maximum capacity • Database Contains a Cause Code for Abort/Reject that Can Alert Admins to Problems • Reject - TRIGGER_MAX_SESSION • Reject - REMOTE_TIMEOUT • Reject - CHANNELS BUSY • Aborted – Too many transfer failures • Aborted – Max Steps Executed Reports give helpful contact details and exact timestamps! RTMT – System summary RTMT – Alert Central Catch Problems Before Users or Customers Do! RTMT – Schedule log collection Intermittent Issue? Finesse Finesse health • Finesse depends on the following services for its normal functioning : • Cisco Unified CCX Engine Service • Cisco Unified CCX Notification Service • Cisco Finesse Tomcat (Tomcat Service exclusive to Finesse) • If any of the above services is not running or in partial service state (viewed from the CCX Serviceability UI or using CLI), Finesse will not function properly. • The service state on GUI shows the state of the Finesse Tomcat only and is not an indication of the state of Cisco Finesse. Finesse health can be checked using the URL: https://<UCCX IP>:8445/finesse/api/SystemInfo Finesse failover Scenario CCX HA Behavior Finesse Server HA Finesse Client Recovery Behavior Behavior CCX Engine Failure Failover to HA node Failover to HA node Failover to HA node Finesse follows Engine mastership CCX Notification No Failover Finesse goes Out of Sessions closed Finesse unavailable Service Failure Service until Notification Service comes online Finesse Tomcat No Failover Finesse goes Out of Sessions closed Finesse unavailable Failure Service until Tomcat Service comes online Finesse Service OOS No Failover Finesse goes Out of Sessions closed Finesse unavailable Service until issue fixed Island Mode Both HA nodes Both Finesse servers Clients connect to Clients reconnect to become Master In Service either Master node Finesse issues – further analysis • From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Finesse, Cisco Unified CCX Engine, Cisco Unified CCX Notification Service Finesse Logs Notification Service Logs webservices, client-logs, Needs to be explicitly realm log file for issues enabled from CLI. related to Finesse For all login issues, state change issues on Finesse Engine Logs MIVR logs, Axlclient logs, for login issues Supervisor Call Monitoring Monitoring overview Supervisor phone places Finesse monitoring call to agent phone Agent Phone Desktop Supervisor Phone V (Browser) Agent phone connected with caller UCCX Server Finesse Server (Tomcat) SIP/SCCP SUPERVISE_CALL_REQ with Supervisory Action field of SUPERVISOR_MONITOR CTI RmCm JTAPI startMonitor method call with agent device info CUCM Server UCCX Engine Monitoring overview Cisco Finesse Server Receives Silent Monitor request from Finesse client browser Finesse Web Services log 15:35:34.122 -0500: %CCBU_http-bio-8082-exec-9-6-API_REQUEST: %[REQUEST_URL=User/agent1/Dialogs][agent_id=agent1][requestId=569bdab5-56a0-4964-b233- 32f515a86cc3][request_method=user.POST][request_parameters= fromAddress:2001 toAddress:2002 requestedAction:SILENT_MONITOR]: Request from client to webservice api Monitoring overview Cisco Finesse Server Sends Silent Monitoring Request to UCCX Engine Finesse Web Services log 15:35:34.123 -0500: %CCBU_pool-12-thread-21-6- MESSAGE_TO_CTI_SERVER: %[cti_message=invokeID=6576, agentConnectionCallId=-1, supervisorConnectionCallID=-1, supervisoryAction=1, agentExt=2002, supervisorExt=2001] [cti_message_name=SuperviseCallReq]: Message going to the backend cti server Monitoring overview UCCX Engine Receives Silent Monitor Request from Finesse Server UCCX Engine (MIVR) Log 15:35:34.126 EST %MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35684,localport=12028] invokeID:6576 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=6576 AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR, AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice = 2002, SupervisorDevice = 2001 Monitoring overview UCCX Engine Begins Silent Monitoring Request to CUCM via JTAPI UCCX Engine (MIVR) Log 1221494: Jan 14 15:35:34.129 EST %MIVR-SS_RM-3-Initiating silent monitor request to JTAPI server...:Undefined mnemonic 'Initiating silent monitor request to JTAPI server...' Monitoring overview JTAPI Calls CallStartMonitoring Request Method to Invoke Monitoring on CUCM JTAPI Log 15:35:34.131 EST %JTAPI-PROTOCOL-7-UNK:(P2-14.10.191.3) [MIVR_SS_RM_TPCCPROCESSOR-219-12-SupervisorMonitorReqMsgHandler] sending: com.cisco.cti.protocol.LineCallStartMonitoringRequest { sequenceNumber = 95 lineCallManagerID = 1 lineID = 190 globalCallManagerID = 1 globalCallID = 1105 callingAddress = 2001 targetAddress = 2002 targetdeviceName = SEP9CAFCAFFCBD5 targetCallID = 29117566 monitorMode =1 playToneDirection = 3 } Monitoring overview JTAPI Shows Monitoring Call Setup Events JTAPI Log 15:35:34.510 EST %JTAPI-PROTOCOL-7-UNK:(P2-14.10.191.3) received Event: com.cisco.cti.protocol.CallMonitoringStartedEvent { eventSequence = 543 lineCallManagerID = 1 lineID = 194 callCallManagerID = 1 callLegID = 29117566 monitorMode = 1 activeToneDirection = 3 } Monitoring overview UCCX Engine Sends Confirmation to Finesse Server that Monitoring has Started UCCX Engine Log 1221618: Jan 14 15:35:34.573 EST %MIVR-ICD_CTI-7-UNK:MsgHandler : Sent : { SUPERVISOR_CALL_CONF to Socket[addr=127.0.0.1,port=35684,localport=12028] } Monitoring overview Finesse Server Receives Monitoring Confirmation Back from UCCX Engine Finesse WebServices Log 15:35:34.684 -0500: %CCBU_CTIMessageEventExecutor-0 -6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=com.cisco.ccbu.finesse.adapter.acmi.message.CTISuperviseCallConf@163c08b [connectionCallId=16778321,connectionDeviceIdType=0,connectionDeviceId=2002,invokeID=6576 ,msgID=125,timeTracker= {"id":"SuperviseCallConf","CTI_MSG_NOTIFIED":1452803734684,"CTI_MSG_RECEIVED":14528037346 83},msgName=SuperviseCallConf,deploymentType=CCX]][cti_response_time=1]: Decoded Message to Finesse from backend cti server Monitoring troubleshooting #1 Problem: Start Monitoring Button Is Not Enabled • Agent to be monitored needs to be in a Talking state (Not Ready on a call doesn’t work) • Supervisor’s phone must be on hook and supervisor must be in the Not Ready state Monitoring troubleshooting #2 Problem: Monitoring Fails to Start CCX Engine Receives Request to Silent Monitor agent106 CCX Engine log: %MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3081 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3081 AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR, AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice = 2002, SupervisorDevice = 2001 Monitoring troubleshooting Problem: Monitoring Fails to Start #2 CCX Engine Runs a Check to see if the agent to be monitored is already being monitored CCX Engine log: %MIVR-SS_RM-7-UNK:thisAgent.isSilentMonitored() is true. returning... Monitoring troubleshooting Problem: Monitoring Fails to Start #2 CCX Engine sends error code and text to Finesse CCX Engine log: %MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1 type=CONTROL_FAILURE_CONF,invokeId=3081,failureCode=CF_RESOURCE_BUSY,errorCode=88056, text=Agent with ID: agent106 is already in another silent monitoring. Monitoring troubleshooting Problem: Monitoring Fails to Start #3 CCX Engine Receives Request to Silent Monitor agent106 CCX Engine log: %MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3164 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3164 AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR, AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 1004, AgentID = null, AgentDevice = 2002, SupervisorDevice = 1004 Monitoring troubleshooting Problem: Monitoring Fails to Start #3 CCX Engine Invokes Monitoring Request via JTAPI but displays error CCX Engine log: %MIVR-SS_RM-3-Initiating silemt monitor request to JTAPI server...:Undefined mnemonic 'Initiating silent monitor request to JTAPI server...‘ ….. %MIVR-SS_RM-7-UNK: JTAPI error code:88046 Monitoring troubleshooting Problem: Monitoring Fails to Start #3 CCX Engine Reports Error Back to Finesse Server CCX Engine log: 39332: Feb 23 15:05:41.144 EST %MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1 type=CONTROL_FAILURE_CONF,invokeId=3175,failureCode=CF_GENERIC_UNSPECIFIED,errorCode=88046, text=Error from Supervisor Monitor request. Description: ICDJtapiCallControlChannel (monitor): Agent Phone has no BiB Configured. to socket: Socket[addr=127.0.0.1,port=35064,localport=12028] Supervisor Barge In Barge In Overview • Allows a supervisor to join a call between an agent and a caller • Supervisor must first be succesfully monitoring agent call • Supervisor clicks the barge-in button enabled on the monitoring call • UCCX Engine instructs Agent Phone via JTAPI to place caller on hold and dial the supervisor phone • On answer (auto), the supervisor is conferenced into the agent call. Barge In - Troubleshooting Supervisor is monitoring agent call and clicks Barge In A delay is observed, A delayby followed is observed, Error followed by Error The supervisor later goes to a Reserved State Barge In - Troubleshooting Supervisor is monitoring agent call and clicks Barge In A delay is observed, Finesse Server Requests Barge-In to UCCX Engine followed by Error Finesse Desktop WebServices log and UCCX Engine log: %CCBU_http-bio-8082-exec-11-6-API_REQUEST: %[REQUEST_URL=User/agent3/Dialogs][agent_id=agent3][requestId=d4bd0140-6be5- 4e75-9e00-7a20e47e2e1d][request_method=user.POST][request_parameters= fromAddress:1004 toAddress:2002 requestedAction:BARGE_CALL associatedDialogUri: /finesse/api/Dialog/16779268]: Request from client to webservice api %MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3404 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3404 AgentConnectionCallID = 16779279, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_BARGE_IN, AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice = 2002, SupervisorDevice = 2001 Barge In - Troubleshooting Supervisor is monitoring agent call and clicks Barge In Finesse Server Requests Barge-In to UCCX Engine A delay is observed, UCCX Engine log and Finesse WebServices log: followed by Error %MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1 type=CONTROL_FAILURE_CONF,invokeId=3404,failureCode=CF_GENERIC_UNSPECIFIED,errorCode=0, text=Error from Supervisor Barge-In request. State of the request during failue: WAIT_FOR_SESSION_OFFERED Failed to get notification from Supervisor agent after Consult to socket: Socket[addr=127.0.0.1,port=35064,localport=12028] %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIControlFailureConf [failureCode=0, peripheralErrorCode=0, text=Error from Supervisor Barge-In request. State of the request during failue: WAIT_FOR_SESSION_OFFERED Failed to get notification from Supervisor agent after Consult]CTIMessageBean [invokeID=3404, msgID=35, timeTracker={"id":"ControlFailureConf","CTI_MSG_NOTIFIED":1456263587009,"CTI_MSG_RECEIVED":1456263587009}, msgName=ControlFailureConf, deploymentType=CCX]][cti_response_time=0]: Decoded Message to Finesse from backend cti server Barge In - Troubleshooting A delay is observed, followed by Error 10 seconds later, Finesse Server Notifies Client of Failure via XMPP (OpenFire) A delay is observed, Finesse WebServices log: followed by Error %CCBU_pool-12-thread-5-3-API_ERROR: %[error_code=1][error_message=Generic Barge Call Error]: Error processing REST API request %CCBU_pool-12-thread-5-6-XMPP_PUBLISH_ASYNCHRONOUS: %[NodeId=/finesse/api/User/agent1/Dialogs][Payload=<Update> <data> <apiErrors> <apiError> <errorData>1</errorData> <errorMessage>Generic Barge Call Error</errorMessage> <errorType>Generic Error</errorType> </apiError> </apiErrors> </data> <event>post</event> <requestId>ae5bf830-0713-45c6-b085-5810da44ab94</requestId> <source>/finesse/api/User/agent1/Dialogs</source></Update>]: Publishing XMPP Message Asynchronously Barge In - Troubleshooting A delay is observed, followed by Error 5 seconds later, CCX Engine Notifies Finesse Server of New Incoming Call to Supervisor A delay is observed, Finesse WebServices log: followed by Error %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=com.cisco.ccbu.finesse.adapter.acmi.message.CTICallOriginatedEvent@dd1de[peripheralId=1,connectionDeviceIdType=0,callId=167 79284,skillGroupNumber=1,callingDeviceType=76,calledDeviceType=76,localConnectionState=1,eventCause=- 1,connectionDeviceId=2002,callingDeviceId=2002,calledDeviceId=2001,fltReqMaskNumMasks=1,fltReqMaskInstrumentList=[2002],fltReqMaskExtensi onList=[2002],fltReqMaskCallIDList=[16779284],fltReqMaskFlagsList=[0000000000000000],fltReqCallMaskList=[0804000000000000],fltReqAgentMas kList=[0000000000000000],invokeID=<null>,msgID=15,timeTracker={"id":"CallOriginatedEvent","CTI_MSG_RECEIVED":1456263592134,"CTI_MSG_ DISPATCH":1456263592135},msgName=CallOriginatedEvent,deploymentType=CCX]][cti_response_time=1]: Decoded Message to Finesse from backend cti server Barge In - Troubleshooting The supervisor later goes to a Reserved State Finesse Server Notifies Finesse Client that Supervisor is receiving a new call A delay is observed, Finesse WebServices log: followed by Error %CCBU_pool-12-thread-12-6-XMPP_PUBLISH_ASYNCHRONOUS: %[NodeId=/finesse/api/User/agent1][Payload=<Update> <data> <user> <dialogs>/finesse/api/User/agent1/Dialogs</dialogs> <extension>2001</extension> <firstName>Agent</firstName> <lastName>One</lastName> <loginId>agent1</loginId> <loginName>agent1</loginName> <pendingState></pendingState> <roles> <role>Agent</role> <role>Supervisor</role> </roles> <settings> <wrapUpOnIncoming></wrapUpOnIncoming> </settings> <state>RESERVED</state> <stateChangeTime>2016-02-23T21:39:52.232Z</stateChangeTime> <teamId>1</teamId> <teamName>Default</teamName> <teams> <Team> <id>1</id> <name>Default</name> <uri>/finesse/api/Team/1</uri> </Team> </teams> <uri>/finesse/api/User/agent1</uri> </user> </data> <event>PUT</event> <requestId></requestId> <source>/finesse/api/User/agent1</source></Update>]: Publishing XMPP Message Asynchronously Post Call Treatment Post Call Treatment Overview How it works: 1. Agent clicks End Call via Finesse 2. When CCX Engine Receives this CTI Request to end call, it first runs a check for a call variable attached to the call called PostCallTreatment 3. If it finds this variable, it then checks to make sure the agent ending the call is the only agent connected to the caller 4. If a DN is found in this variable, it performs a transfer to that DN rather than ending the call Post Call Treatment – Script review • 1. Verify the script has a variable of type int named something like PostCallTreatment with the correct Value 2. Verify that this variable is mapped to an ECC variable named exactly --PostCallTreatment-- Post Call Treatment - Logs UCCX Engine Script Debugging Shows Script Variable Set Correctly UCCX Engine Log (MIVR): An object of com.cisco.wfframework.obj.WFWorkflowContext { {resourceID, java.lang.String, ""} {CSQ, java.lang.String, ""} {DelayWhileQueued, java.lang.Integer, 30} {WelcomePrompt, com.cisco.prompt.Playable, SP[ICD\ICDWelcome.wav]} {QueuePrompt, com.cisco.prompt.Playable, SP[ICD\ICDQueue.wav]} {SRS_TempResourceSelectedVar2, com.cisco.user.User, null} {PostCallTreatment, java.lang.Integer, 1071} } Most importantly, UCCX Engine Shows the PostCallTreatment variable attached to the call is set correctly UCCX Engine Log: %MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_DELIVERED_EVENT: Socket:Socket: null monitoredDeviceDN:2082, connectionCallID: 16779415, lineType: 3, skillGroupNumber: 1, serviceNumber: 0, alertingDeviceType: 73, callingDeviceType: 0, calledDeviceType: 73, lastRedirectDeviceType: 76, localConnectionState: 2, eventCause: 22, connectionDeviceID: 2082, alertingDeviceID: 2082, callingDeviceID: 1003, calledDeviceID: 2082, lastRedirectDeviceID: 2002, secondaryCallID: 0, ani: 1003, dnis: 1071, userToUserInfo: null, dialedNumber: 1071, callerEnteredDigits: null, callVar1: voice1, callVar2: null, callVar3: null, callVar4: null, callVar5: null, callVar6: null, callVar7: null, callVar8: null, callVar9: null, callVar10: null, wrapupData: null, PostCallTreatment: 1071, RemaskField1 : CRACTIRemaskField [reqMaskInstrument=2002, reqMaskExtension=2002, reqMaskCallID=16779415, reqMaskFlags=[0, 0, 0, 0, 0, 0, 0, 0], reqCallMask=[0, 0, 0, 0, 0, 0, 4, 88], reqAgentMask=[0, 0, 0, 0, 0, 0, 0, 0]] } to outboundQ Post Call Treatment - Logs UCCX Engine Receives Agent Hangup Request from Finesse Server UCCX Engine Log: %MIVR-SS_RM-7-UNK:Processing msg: CTIClearConnectionReqMsg CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:6310, peripheralId:1, connectionCallId:16779415, connectionDeviceIdType: 65535, connectionDeviceId: 2002, agentInstrument: null Instead of immediately going to JTAPI to disconnect the call on the agent phone, run PostCallSurvey checks: UCCX Engine Log: %MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler - isPostCallSurveyEnabled postCallSurveyDN: 1071728797: Feb 24 16:27:53.283 EST %MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler:runHandler connectedAgents.size: 1728798: Feb 24 16:27:53.283 EST %MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler - isPostCallSurveyEnabled. Only agent. Transferring the call to survey Call Recording with Mediasense Subscribed Application MediaSense recording – CUCM 1. Caller dials in, CUCM sets up call 4. CUCM Sends 2 Invites to MediaSense across SIP Trunk 6.Call Control Service sends RTP ports in 11. API Service Sends SIP 200 OK Reply SESSION_STARTED_EVENT CUCM Subscribed App Call Control 3.CUCM Knows that Phone Has a API Service Recording Profile Attached Service 9. Call Control Service sends metadata to API Service 10. API Service writes to DB 7.CUCM communicates RTP ports for forking stream to the phone Media Service Database Service 5.Call Control Service notifies 8. Phone establishes RTP with Media Media Service of incoming Service recording. MediaService 2. RTP established between caller and phone chooses 2 RTP Ports. Mediasense recording – Finesse Supervisor Agent Customer Voice RTP Stream Desktop Desktop Agent Voice RTP Stream REST API Calls JTAPI Finesse UCCX CTI Requests (ACMI) Server ACMI UCCX Engine CTI RmCm CUCM Server SIP MGCP/H323 SIP/SCCP BIB Subscription PSTN/WAN Customer Agent Phone MediaSense Voice Gateway Mediasense Troubleshooting Is CUCM Automatic recording Yes CLOSED_ERROR working? status No recording found Test call Review Check MediaSe Finesse network nse Success workflow Busy Verify Verify device and Review MS Analyze MS logs line configuration UCCX service licensing status Verify Check UCCX + CUCM Finesse logs config Analyze CUCM + MS logs Is CUCM Automatic Call Recording working? • Isolate the issue by taking UCCX and Finesse out of the flow, on the CUCM agent line configuration, enable automatic call recording: 1. Closed_Error status 2. Recordings cannot be found Mediasense Troubleshooting Is CUCM Automatic recording Yes CLOSED_ERROR working? status No recording found Test call Review Check MediaSe Finesse network nse Success workflow Busy Verify Verify device and Review MS Analyze MS logs line configuration UCCX service licensing status Verify Check UCCX + CUCM Finesse logs config Analyze CUCM + MS logs CLOSED_ERROR Cause can be identified from logs or metadata Error Detail Code in metadata and Closed_Error Reason in Description logs Search and Play MEDIA_SERVER_ERROR Record Response Fail Call control service got an error response from the Media (recording) server for the open or close request. MEDIA_SERVER_TIMEOUT Record Response Timeout Call control service timed out waiting for response from the Media (recording) server for the open or close request. SIP_SIGNALING_ERROR SIP Signal Error Call control service detected a SIP signaling error, for example a missing ACK. SIP_CANCEL_RECEIVED Record Cancelled Recording was canceled by call control service, such as CANCEL or premature BYE received from CUCM. CLOSED_ERROR Cause can be identified from logs or metadata Error Detail Code in metadata Closed Error Reason in Description and logs Search and Play NO_MEDIA_RECEIVED Zero Size Tracks Session was successfully closed, but ALL tracks have zero size. ORPHANED Orphaned Session Session was orphaned, e.g., forcibly closed after service restart. UNSUPPORTED_CODEC Unsupported Codec Unsupported Codec used to record session. Check MediaSense logs Mediasense Troubleshooting Is CUCM Automatic recording Yes CLOSED_ERROR working? status No recording found Test call Review Check MediaSe Finesse network nse Success workflow Busy Verify Verify device and Review MS Analyze MS logs line configuration UCCX service licensing status Verify Check UCCX + CUCM Finesse logs config Analyze CUCM + MS logs Test call to MediaSense • Dial the Route Pattern that points to to the MediaSense SIP trunk from an IP Phone • A valid recording, with just a single participant (the calling phone) should be successfully created: Verify MS service status • Check if all services are running from Control Center – Feature services Verify MS service status Show Tech in CLI Use ‘show tech call_control_service’ from each node in the CLI Shows Useful System Info • System Info • Status of All Adapters • Extensive Recording Stats • Local Storage Info Verify CUCM configuration Route List MediaSense Route group Route pattern Route SIPgroup Trunk MediaSense SIP Trunk Recording profile Mediasense Troubleshooting Is CUCM Automatic recording Yes CLOSED_ERROR working? status No recording found Test call Review Check MediaSe Finesse network nse Success workflow Busy Verify Verify device and Review MS Analyze MS logs line configuration UCCX service licensing status Verify Check UCCX + CUCM Finesse logs config Analyze CUCM + MS logs Verify device and line configuration • Ensure that a supported device is being used and that Builtin Bridge is enabled on the device configuration • Ensure Line has the correct Recording Profile Mediasense Troubleshooting Is CUCM Automatic recording Yes CLOSED_ERROR working? status No recording found Test call Review Check MediaSe Finesse network nse Success workflow Busy Verify Verify device and Review MS Analyze MS logs line configuration UCCX service licensing status Verify Check UCCX + CUCM Finesse logs config Analyze CUCM + MS logs Review Finesse workflow • Verify if CUCM has selective recording selected • Verify action configuration on Finesse admin pages Review Finesse workflow • Verify workflow configuration Review UCCX licensing • Recording ports in MediaSense are not enforced in MS software • Instead, UCCX enforces simultaneous recordings Single Sign On SSO overview Applications/logs for troubleshooting Application/log Details Where to find Cisco IdS log Cisco IdS logger will log any RTMT, collect “Cisco Identifity error that happened in Cisco IdS Service” logs Fedlet logs Fedlet logs will give more details RTMT, same location as IdS about any SAML errors that logs, they have the suffix “fedlet- happens in Cisco IdS ” Cisco IdS API metrics API metrics can be used to look RTMT – Under Cisco Identity into and validate any errors that Service you’ll find a folder Cisco IdS APIs may have ”metrics”, returned and number of requests saml_metrics.csv and authorize_ that are processed by Cisco idS metrics.csv are the relevant metrics for this document. Applications/logs for troubleshooting Application/log Details Where to find Event Viewer in AD FS Allows users to view In AD FS machine, navigate to Event Viewer the event logs in the system. Applications and Services Logs Any error in AD FS while AdDFS 2.0 Admin processing the SAML request/sending the SAML response will be logged here. SAML Viewer A SAML Viewer will help in These are some suggested SAML viewers looking at the SAML Request that you can use for looking at the SAML and Response that are sent request and response from/to Cisco IdS. Fiddler This browser application is How to use fiddler with AD FS very useful for the analysis of Fiddler Chrome Plugin SAML Request/Response. SAML Tracer - Firefox SAML Chrome Panel Troubleshoot flow diagram Auth code request processing by IdS • Link • Error message: {"error":"invalid_redirectUri","error_description":"Invalid Redirect URI."} • Cause: User accesses application using IP Address/ Alternate Host Name. In SSO mode, if the application is accessed using IP, it does not work. Applications should be accessed by the hostname by which they are registered in Cisco IdS. This issue can happen if user accessed an alternate host name that is not registered with Cisco IdS. https://uccx1.ccteam.bru.lab:8553/ids/v1/oauth/authorize?redirect_uri=https%3A%2F%2Fuccx- angelique.ccteam.bru.lab%3A443%2Fappadmin%2Fsso%2Fauthcode&client_id=12bc07eb55dc7495 609bdfdd2e1fa3b083bfeacd&state=aHR0cHM6Ly91Y2N4LWFuZ2VsaXF1ZS5jY3RlYW0uYnJ1LmxhYi 9hcHBhZG1pbi9tYWluCWFwcGxvZ2lu&response_type=code SAML request initiation by IdS • Error Message: {"error":"service_unavailable","error_description":"SAML Metadata is not initialized"} • Possible Cause: Idp Metadata is not available in Cisco IdS. Trust establishment between Cisco IdS and AD FS is not complete. • Recommended action: Navigate to Cisco IdS Management console and see if the IdS is in Not Configured state. Confirm if IdP metadata is uploaded or not. If not, upload the IdP metadata downloaded from AD FS. SAML request processing by ADFS • Error Message: • Cause: From ADFS Event viewer log: Encountered error during federation passive request. Additional Data Exception details: Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service URL is not configured. ---> Microsoft.IdentityServer.PolicyModel.Client.PolicyStoreConnectionException: ADMIN0017: An exception occurred while connecting to the configuration service. The configuration service URL 'net.tcp://localhost:1500/policy' may be incorrect or the AD FS 2.0 Windows Service is not running. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:1500/policy. The connection attempt lasted for a time span of 00:00:02.0001144. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:1500. --- > System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:1500… SAML request processing by ADFS #2 • Error Message: From AD FS Event Viewer The Federation Service encountered an error while processing the SAML authentication request. Additional Data Exception details: Microsoft.IdentityModel.Protocols.XmlSignature.SignatureVerificationFailedException: MSIS0038: SAML Message has wrong signature. Issuer: 'myuccx.cisco.com'. at Microsoft.IdentityServer.Protocols.Saml.Contract.SamlContractUtility.CreateSamlMessage(MSISSamlBindingMessage message) at Microsoft.IdentityServer.Service.SamlProtocol.SamlProtocolService.CreateErrorMessage(CreateErrorMessageRequest createErrorMessageRequest) at Microsoft.IdentityServer.Service.SamlProtocol.SamlProtocolService.ProcessRequest(Message requestMessage) • Possible cause: Relying party trust is not established or Cisco IdS certificate has changed, but the same is not uploaded to the AD FS. SAML request processing by ADFS #2 • Recommended action: Establish trust between AD FS and Cisco IdS with the latest Cisco IdS certificate. Ensure that the Cisco IdS Certificate is not expired. You can see the status dashboard in Cisco Identity Service Management. If so, regenerate the certificate in the Settings page. SAML Response Sending by ADFS • Problem: Browser shows NTLM login, but after successful log in, it fails with many redirects. • Possible cause: Cisco IdS supports only form based authentication, Form authentication is not enabled in AD FS. • Recommended action: For more details on how to enable Form authentication see: ADFS 2.0 Form Authentication Setting ADFS 3.0 Form Authentication Setting SAML response processing by Cisco IdS • Error Message: SAML response processing by Cisco IdS • ADFS event viewer • IdS logs: 2017-03-13 21:37:53.189 CET(+0100) [IdSEndPoints-SAML-54] ERROR com.cisco.ccbu.ids IdSSAMLAsyncServlet.java:299 - SAML response processing failed with exception com.sun.identity.saml2.common.SAML2Exception: Invalid Status code in Response…. SAML response processing by Cisco IdS • Possible cause: AD FS is configured to use SHA-256. • Recommended action: 1. RDP to the AD FS system. 2. Open AD FS console. 3. Select the Relying Party Trust and click Properties 4. Select the Advanced tab. 5. Select SHA-1 from the drop-down list. SAML response processing by Cisco IdS #2 • Error Message: SAML response processing by Cisco IdS #2 • ADFS Event Viewer: SAML response processing by Cisco IdS #2 • IdS logs: 2017-03-13 21:52:50.161 CET(+0100) [IdSEndPoints-SAML-55] INFO com.cisco.ccbu.ids SAML2SPAdapter.java:76 - SSO failed with code: 1. Response status: <samlp:Status> <samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Responder"> </samlp:StatusCode> </samlp:Status> for AuthnRequest: n/a 2017-03-13 21:52:50.161 CET(+0100) [IdSEndPoints-SAML-55] ERROR com.cisco.ccbu.ids IdSSAMLAsyncServlet.java:299 - SAML response processing failed with exception com.sun.identity.saml2.common.SAML2Exception: Invalid Status code in Response. SAML response processing by Cisco IdS #2 • Possible cause: Custom claim rule is not configured correctly. • Recommended action: 1. Under AD FS claim rules, ensure that attributes mapping for "user_principal" and "uid" are defined correctly 2. RDP to AD FS system. 3. Edit the Claim Rules for custom claim rules. 4. Verify that the AD FS and Cisco IdS fully qualified domain names are given. SAML response processing by Cisco IdS #2 Useful links • Configure the Identity Provider for UCCX based on SSO http://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-contact-center- express/200612-Configure-the-Identity-Provider-for-UCCX.html • ADFS/IdS Troubleshooting and Common Problems http://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-contact-center- express/200662-ADFS-IdS-Troubleshooting-and-Common-Prob.html • Troubleshooting docwiki http://docwiki.cisco.com/wiki/Troubleshooting_Tips_for_Unified_CCX_11.5 • Troubleshooting Tech Notes UCCX http://www.cisco.com/c/en/us/support/customer-collaboration/unified-contact-center- express/products-tech-notes-list.html If all of the above fails… Open a TAC case Information that is useful to add from the beginning already: • Versions • Topology • Known changes • Exact error (with screenshots/logs and your analysis) • Steps you’ve tried already • Impact Questions?