Exchange 2013/2016/2019 – EventID 22004 – MSExchangeTransport: The periodic heartbeat to primary server ex-002.domain.local failed.
On some morning we experienced following eventlog records on a primary/active MS-Exchange 2019 server. The server is the primary server in a redundant DAG (database availability group).
Log Name: Application
Source: MSExchangeTransport
Date: 17.05.2023 09:13:06
Event ID: 22004
Task Category: ShadowRedundancy
Level: Warnung
Keywords: Klassisch
User: N/A
Computer: EX-001.domain.local
Description:
The periodic heartbeat to primary server ex-002.ad.sehag.ch.
Event Xml:
22004 3 22 0x80000000000000 15988867 Application EX-001.domain.local ex-002.domain.localFollowing toubleshooting steps has been executed via Exchange Management Shell:
1. List all message queues
[PS] C:\Windows\system32>Get-Queue
Identity DeliveryType Status MessageCount Velocity RiskLevel OutboundIPPool NextHopDomain
-------- ------------ ------ ------------ -------- --------- -------------- -------------
EX-001\4 SmtpDeliveryToMailbox Ready 0 0 Normal 0 db01
EX-001\Submission Undefined Ready 0 0 Normal 0 Übermittlung
EX-001\Shadow\3 ShadowRedundancy Ready 16 0 Normal 0 ex-002.domain.local
The interesting queue is EX-001\Shadow\3
2. List the transport configuration of the related queue
[PS] C:\Windows\system32>Get-TransportConfig | FL Identity,*Shadow*
Identity : Transport Settings
ShadowRedundancyEnabled : True
ShadowHeartbeatTimeoutInterval : 00:15:00
ShadowHeartbeatRetryCount : 12
ShadowHeartbeatFrequency : 00:02:00
ShadowResubmitTimeSpan : 03:00:00
ShadowMessageAutoDiscardInterval : 2.00:00:00
RejectMessageOnShadowFailure : False
ShadowMessagePreferenceSetting : PreferRemote
MaxRetriesForLocalSiteShadow : 2
MaxRetriesForRemoteSiteShadow : 43. List the stucked messages in the queue
[PS] C:\Windows\system32>Get-Queue
Identity DeliveryType Status MessageCount Velocity RiskLevel OutboundIPPool NextHopDomain
-------- ------------ ------ ------------ -------- --------- -------------- -------------
EX-001\4 SmtpDeliveryToMailbox Ready 0 0 Normal 0 db01
EX-001\Submission Undefined Ready 0 0 Normal 0 Übermittlung
EX-001\Shadow\3 ShadowRedundancy Ready 16 0 Normal 0 ex-002.domain.local4. Resubmit all messages in EX-001\Shadow\3
[PS] C:\Windows\system32>Retry-Queue "EX-001\Shadow\3" -Resubmit $True5. Recheck the queue
[PS] C:\Windows\system32>Get-Queue
Identity DeliveryType Status MessageCount Velocity RiskLevel OutboundIPPool NextHopDomain
-------- ------------ ------ ------------ -------- --------- -------------- -------------
EX-001\4 SmtpDeliveryToMailbox Ready 0 0 Normal 0 db01
EX-001\7 SmartHostConnectorDelivery Ready 0 0 Normal 0 [172.25.10.35]
EX-001\Submission Undefined Ready 0 0 Normal 0 Übermittlung
EX-001\Shadow\3 ShadowRedundancy Ready 0 0 Normal 0 ex-002.domain.localIn case the queue does not clean up, take following actions:
- Check all receive connectors and make sure default connectors are enabled on the primary server.
- Proceed with troubleshooting on receive connectors.