Troubleshooting the SAPHanaSR python hook

SUSE Linux Enterprise Server for SAP Applications 15 SP3
SUSE Linux Enterprise Server for SAP Applications 15 SP2
SUSE Linux Enterprise Server for SAP Applications 15 SP1
SUSE Linux Enterprise Server for SAP Applications 12 SP5
SUSE Linux Enterprise Server for SAP Applications 12 SP4
SAPHanaSR-0.154.1-4.14.1.noarch

How to verify the SAPHanaSR.py hook is working properly after installing and  implementing the hook provided by SAPHanaSR RPM package and following directions provided in the Best Practices Guide .

1. SAPHanaSR python hooks are only supported on SAP HANA 2.0.
2. SAP HANA 1.0 does not support SAPHanaSR python hooks (srHook).
For HANA 1.0 the current recommendation is to stay with the former configuration (without the HA/DR provider hook). End of maintenance for SAP HANA v1.0 is May 2021, customer need to plan migration to SAP HANA v2.0.

If it's not working properly,  reference the Troubleshooting: section below.

1.  Check the SAP logs after starting the databases back up and look for entries with "ha_dr_" in SAP's trace directory logged in as the <sidadm> user. 
     Example:  <sidadm> = sleadm

hana2 # su - sleadm

sleadm@hana2:/usr/sap/SLE/HDB00> cdtrace

sleadm@hana2:/usr/sap/SLE/HDB00/hana2/trace> grep ha_dr_ nameserver_*

nameserver_hana2.30001.066.trc:[86754]{-1}[-1/-1] 2021-02-01 14:17:07.863502 i ha_dr_provider   PythonProxyImpl.cpp(01096) : calling HA/DR provider SAPHanaSR.hookDRConnectionChanged(hostname=hana2, port=30007, volume=2, service_name=xsengine, database=SLE, status=15, database_status=13, system_status=13, timestamp=2021-02-01T14:17:07.863451-07:00, is_in_sync=1, reason=, siteName=SiteA)
nameserver_hana2.30001.066.trc:[86754]{-1}[-1/-1] 2021-02-01 14:17:07.863845 i ha_dr_SAPHanaSR  SAPHanaSR.py(00086) : SAPHanaSR (0.162.0) SAPHanaSR.srConnectionChanged method called with Dict={'status': 15, 'is_in_sync': True, 'timestamp': '2021-02-01T14:17:07.863451-07:00', 'database': 'SLE', 'siteName': 'SiteA', 'service_name': 'xsengine', 'hostname': 'hana2', 'volume': 2, 'system_status': 13, 'reason': '', 'database_status': 13, 'port': '30007'}

2.  A new cluster property called "SAPHanaSR" is also created.   
To see this, use the following command as root user. 

hana2:/ # crm configure show SAPHanaSR
property SAPHanaSR: \
	hana_sle_site_srHook_SiteB=PRIM \
	hana_sle_site_srHook_SiteA=SOK

Note: If a failover or srTakeover has not taken place, you may only see one of the attributes populated.

3. The utility SAPHanaSR-showAttr should also show information about the srHook.

hana2:/ # SAPHanaSR-showAttr 
Global cib-time                 maintenance 
--------------------------------------------
global Fri Feb  5 11:16:45 2021 false       

Resource              maintenance 
----------------------------------
msl_SAPHana_SLE_HDB00 false       

Sites srHook 
-------------
SiteA SOK    
SiteB PRIM   

Hosts clone_state lpa_sle_lpt maintenance node_state op_mode   remoteHost roles                            score site  srmode  standby sync_state version                vhost 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hana1 DEMOTED     30          off         online     logreplay hana2      4:S:master1:master:worker:master 100   SiteA syncmem off     SOK        2.00.042.00.1564994110 hana1 
hana2 PROMOTED    1612549005  off         online     logreplay hana1      4:P:master1:master:worker:master 150   SiteB syncmem off     PRIM       2.00.042.00.1564994110 hana2 

Note: If a failover or srTakeover has not taken place, you may only see one of the attributes populated.

4. The SAPHana resource monitor now also populates the SRHOOK1= <value> instead of being blank. 
This can be seen in the /var/log/messages on each node. 

primary node: 
hana2 SAPHana(rsc_SAPHana_SLE_HDB00)[134236]: INFO: RA: SRHOOK1=PRIM
hana2 SAPHana(rsc_SAPHana_SLE_HDB00)[134236]: INFO: RA: SRHOOK3=PRIM
or results on secondary node: 
hana1 SAPHana(rsc_SAPHana_SLE_HDB00)[45054]: INFO: RA: SRHOOK1=SOK
hana1 SAPHana(rsc_SAPHana_SLE_HDB00)[45054]: INFO: RA: SRHOOK3=SOK

Before implementing the SAPHanaSR.py hook.
hana1 SAPHana(rsc_SAPHana_SLE_HDB00)[20948]: INFO: RA: SRHOOK1=
hana1 SAPHana(rsc_SAPHana_SLE_HDB00)[20948]: INFO: RA: SRHOOK2=SOK
hana1 SAPHana(rsc_SAPHana_SLE_HDB00)[20948]: INFO: RA: SRHOOK3=SOK

Troubleshooting: 
1. Verify the visudo entry on both nodes is correct.  Verify the <sidadm> and <sid> variables match your environment. 

# SAPHanaSR-ScaleUp entries for writing srHook cluster attribute
<sidadm> ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_*

Example:  <sidadm>=nw1adm   <sid>=nw1
nw1adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_nw1_site_srHook_*

2. Verify that correct entry is added to the global.ini on both nodes and the path= points to correct location.

[ha_dr_provider_SAPHanaSR]
provider = SAPHanaSR
path = /usr/share/SAPHanaSR
execution_order = 1

[trace]
ha_dr_saphanasr = info

3. Verify the SAPHanaSR.py has the correct owner and permissions.

hana01:~ # ls -l /usr/share/SAPHanaSR/SAPHanaSR.py
-rw-r--r-- 1 root root 4869 Jul 16  2020 /usr/share/SAPHanaSR/SAPHanaSR.py

4. Configuring SAPHanaSR Hook has an error similar to:

SAPHanaSR/SAPHanaSR:srConnectionChanged() failed with python error: 'siteName' 

SAP HA/DR provider for HANA Version 1.0 SPS12 doesn't provide the site name. This means SAP HANA 1.0 HA/DR PAI does not cover the feature needed by SUSE resource agent (multi-site-awareness). SAPHanaSR Hook designed only to works with SAP Hana 2.0 beginning with SPS03. SAPHanaSR Hook does not work on SAP HANA 1.0. See also TID#000019754

Documentation References: 
Best Practices Guide --> Implementing the Python Hook SAPHanaSR
man page for srHook --> man SAPHanaSR.py (7)