Hi again,

My name is Man-Ted Chan and I’m from the SAP HANA product support team. This is part 2 to my High Availability/System Replication blog, part 1 can be found here.

This will continue where the last blog left off

How to turn off replication

First we will unregister the secondary server, this means no more data from the primary will go to this server:

After this have been unregistered we can check the hdbnsutil –sr_state to confirm this:

However, if you check the primary node you will see that the replication is still enabled, but no server for the replication is listed.

Next we can disable the replication on the primary

Once this is done you can check the replication tab and hdbnsutil –sr_state

As a test, I stopped the primary to see what happens on

Other things tested during this phase

As a test I stopped the primary to see what happen to the replication. No automated takeover will occur, but we will see the following network communication errors in the trace files

e Stream NetworkChannelCompletion.cpp(00524) : NetworkChannelCompletionThread #2 NetworkChannel FD 28 [0x00007fc028072818] {refCnt=3, idx=2}> ConnectWait,[—c]

: Error in asynchronous stream event: exception  1: no.2110001 (Basis/IO/Stream/impl/NetworkChannelCompletion.cpp:450)

    Generic stream error: getsockopt, Event=EPOLLERR – , rc=111: Connection refused

Please note that if you stop the replication server the primary server will throw the following alerts

ReplicationError with state INFO with event ID  1 occurred at on xxxx36f509:30007. Additional info: Communication channel closed

Associated with Alert ID 78

The following error will be found in the trace files

e TNS TNSClient.cpp(00671) : sendRequest dr_getremotereplicationinfo to xxxx301545c:30001 failed with NetException. data=(I)drsender=1|

e sr_nameserver TNSClient.cpp(06880) : error when sending request ‘dr_getremotereplicationinfo’ to xxxx301545c:30102: connection refused,location=xxxx301545c:30001

i EventHandler EventManagerImpl.cpp(00602) : acknowledge: ReplicationEvent(): Communication channel closed

If you run into this alert in your own system you should check to see if the secondary node is down (can you start it or was there a crash?)

How to perform a takeover

*Please note that performing a takeover should be done only if there is an issue if the primary or if you would like zero down during a HANA upgrade
Right click on the secondary node and open the “Configure System Replication”

At an OS level you will see the takeover process

To perform the takeover via the command prompt you would run the following on the secondary server:

Hdbnsutil –sr_takeover

*After the takeover a new server needed to be made so the server name is different from 301545c to 59e3753f1

Please note on your replication server you will now be able to open the admin panel and not just the diagnosis mode (in the diagnosis mode only ‘Processes’, ‘Diagnosis Files’, and ‘Emergency Information’ tabs are available)

On the old primary server and old replication we can check the Landscape->System Replication and see there is no replication

Since the replication hasn’t been disabled we will see the communication errors again on the original primary

i EventHandler EventManagerImpl.cpp(00780) : –removeAllEvents: ReplicationEvent(): Communication channel closed

On the old replication server the nameserver trace will show the following during the takeover if it was successful

i sr_nameserver TREXNameServer.cpp(15647) : re-assign for databaseId 2 volume 2 returned successfully

i sr_nameserver TREXNameServer.cpp(15647) : re-assign for databaseId 2 volume 4 returned successfully

i sr_nameserver TREXNameServer.cpp(15647) : re-assign for databaseId 2 volume 3 returned successfully

i sr_nameserver TREXNameServer.cpp(15703) : issueing “/usr/sap/MV1/SYS/global/hdb/install/bin/hdbupdrep -s MV1 –user_store_key=SRTAKEOVER -b”

i sr_nameserver TREXNameServer.cpp(15686) : reconfiguring all services

Check the global.ini and nameserver.ini on the secondary node (the primary will not change)

/usr/sap/MV1/global/hdb/custom/config> cat global.ini


site_id = 2

mode = sync

actual_mode = primary

site_name = rep

mo-59e3753f1:/usr/sap/MV1/global/hdb/custom/config> cat nameserver.ini


id = 55de6934-1b45-7f0a-e100-00000a6116ac

master = mo-59e3753f1:30001

worker = mo-59e3753f1

active_master = mo-59e3753f1:30001

idsr = 55f36543-7352-8161-e100-00000a61131b

roles_mo-59e3753f1 = worker


In order to minimize memory consumption, the following parameters should be set in the secondary system:

1) global.ini/[system_replication]/preload_column_tables = false

2) global.ini/[memorymanager]/global_allocation_limit =

If the parameter “preload_column_tables” is set to “true” on the secondary side, the secondary system will dynamically load tables into memory according to the preload information shipped from the primary side.

During the takeover procedure, the “global_allocation_limit” should be increased on the secondary side to the same value as on the primary side.

Memory on the primary can be consumed in async mode there is a logbuffer that gets loaded and then sent over to the secondary, the amount of memory this takes up is set by

  1. global.ini -> [system_replication] -> logshipping_async_buffer_size =


For additional information during a takeover please run the following

alter system alter configuration (‘nameserver.ini’,’SYSTEM’) SET (‘trace’,’failover’)=’debug’ with reconfigure;

alter system alter configuration (‘nameserver.ini’,’SYSTEM’) SET (‘trace’,’ha_provider’)=’debug’ with reconfigure;

Perform failover test. Once done you can turne off this tracing

alter system alter configuration (‘nameserver.ini’,’SYSTEM’) UNSET (‘trace’,’failover’) with reconfigure;

alter system alter configuration (‘nameserver.ini’,’SYSTEM’) UNSET (‘trace’,’ha_provider’) with reconfigure;

For general tracing during the replication you can go edit in the SAP HANA studio global.ini-> trace-> sr_dataaccess = debug and studio global.ini-> trace->stream= debug. This will add additional tracing in the indexserver trace.


System Replication Configuration Parameters


Issues Encountered


-After SP9 users ran into Alert 79, Configuration Parameter Mismatch, to resolve this you can edit global.ini->system_replication->keep_old_style_alert = false

The ini’s will still be mismatched, but the alert will stop appearing. User can manually check the mismatches, or can go to /usr/sap//global/hdb/customer/config and copy from the primary and paste it to the secondary, but do not overwrite global.ini->system_replication and nameserver.ini->landscape section as this will break replication. Another option you can do is run the SQL script to find the differences:


Network Related

-‘Communication Channel Closed’ errors, the replication server is either down or there is a networking error. (Check to see if the HANA services are running, if they are talk to your networking team about blocked ports)

-(DataAccess/impl/DisasterRecoveryProtocol.cpp:3478) Asynchronous Replication Buffer is Overloaded exception throw location:

This error occurs only if you choose ASYNC replication, this can occur if there is a slowness in the network. You can check your network statistics on with the following table



If you need to resolve this issue prior to looking into you network you can do one of the following:

1) Change the replication mode, -sr_change mode –mode= sync|syncmem

2) Change global.ini->system_replication->logshipping_async_wait_on_buffer_full = false, this will temporarily decouple the synchronization.

Registration fails


Unable to contact primary site error: at 30001


Check the host name you have entered, something’s to check:

The hostnames are unique

The secondary host name is not a substring of the primary

Do not use the IP address


f sr_nameserver TREXNameServer.cpp(10651) : remoteHost does not match with any host of the source site. Please ensure that all hosts of source and target site

Can’t resolve all hostnames of both sites correctly.


Run the following query and

select name from m_topology_tree where path = ‘/host/’

Startup of secondary fails


Secondary nameserver starup fails after registration of secondary to primary: TREXNameServer.cpp(02876) : source site is not active, cannot start secondary site. Please run hdbnsutil -sr_takeover in case of a disaster or start primary site first. -> stopping instance ..


Do not use secondary hostnames that are substring of primary hostnames.


nameserver server:30001 not responding.

collecting information …

error: source system and target system have overlapping logical

hostnames; each site must have a unique set of logical hostnames.

hdbrename can be used to change names;



This is caused by connection timeouts, but if you see it only for a few services check to see if the landscape are the same.

MultiDB issue


“unhandled ltt exception: exception 1000003:

Index 1 out of range [0, 0)” when i check the sr_state after running


Resolved in 97.01 and 102



i LogReplay RowStoreTransactionCallback.cc(00226) : starting master-slave DTX consistency check

e LogReplay RowStoreTransactionCallback.cc(00264) : Slave volume 3 is not available


Resolved in rev 74.04 and 82

Work around:

1) Add following INI parameters as ‘false’ in indexserver.ini and statisticserver.ini


check_slave_on_master_restart = false

check_global_trans_consistency = false

2) The, restart your system.


From time to time the takeover process hangs

w Backup BackupMonitor_TransferQueue.cpp(00048) : Master index server not available! Following trace Entries are in written to the trace file, and there is a time gap in the trace of 30m: [11596]{-1}[-1/-

i PersistenceManag PersistenceManagerImpl.cpp(02359) : Activating periodic savepoint, frequency 300 e TrexNet Channel.cpp(00362) : active channel 33 from 53223 to reading failed with timeout error; timeout=1800000ms elapsed


There is no work around, this issue is fixed in 85.02 and 90


If a takeover is performed on a secondary system where not all tenants could be taken over (e.g. because they were not initialized yet) then the takeover flag is not removed from the topolgy (/topology/datacenters/takeover/*)


Resolved in HANA 10.1

Crash on secondary

indexserver crash at DataRecovery::LoggerImpl::IsSecondaryBackupHistoryComplete on the secondary system.

The bug is fixed as of revision 90 so a permanent solution is available via an upgrade.

In the interim the workaround to the issue is the setting of the parameter [system_replication] ensure_backup_history = false within the global.ini file.

The setting of this parameter disables the maintenance of the backup history.  The takeover process is not affected by this parameter but full recovery scenarios after takeover (using old primary data/log backups with new primary log backups) may be impacted.

SAP Notes

1995412 – Secondary site of System Replication runs out of disk space due to closed data shipping connection

1945676 – Correct usage of hdbnsutil -sr_unregister

2057595 – FAQ: SAP HANA High Availability

2100052 – How to disable parameter mismatch alert for system replication

2050830 – Registering a secondary system via HANA Studio fails with error ‘remoteHost does not match with any host of the source site’

2021186 – Garbage collection takes a long time during HANA service restart

2075771 – SAP HANA DB: System Replication – Possible persistence corruption on secondary site

1852017 – Error 10061 when connecting SAP Instances to failed over HANA nodes

2063657 – HANA System Replication takeover decision guideline

2062631 – high availability limitation for SAN storage

2129651 – Indexserver crash caused by inconsistent log position when startup

1681092 – Multiple SAP HANA DBMSs (SIDs) on one SAP HANA system

2033624 -System replication: Secondary system hangs during takeover

2081563 – secondary system’s replication mode and replication status changed to “UNKNOWN”

2135107 – Log segment for backup history is still missing after reconnect with log shipping

New NetWeaver Information at SAP.com

Very Helpfull

User Rating: Be the first one !