ISC DHCP Failover Overview
DHCP failover is a powerful feature providing high availability address assignment services. Failover enables you to provision a given address pool on two separate DHCP servers. One acts as the primary processing lease transactions via the DHCP protocol while the other acts as a backup or failover server. The failover server receives all DHCP messages from clients and relay agents but does not process them while in the failover (i.e. standby) state. The primary and failover servers exchange 'heartbeat' messages between them to provide assurance that the respective partner is operational. In addition as the primary performs lease updates it communicates lease binding information to the failover server. Thus in the event of an outage of the primary server the failover server will detect this condition due to a lack of heartbeat messages and having tracked recent updates regarding lease transactions shall be able to begin responding to client address requests assuming the primary role. The lease binding updates reduce the risk of duplicate address assignments though there is a window during the failover detection and failover initiation where some leases may have been assigned unbeknownst to the failover server. The failover server will typically begin handing out IP addresses from the end of its address pool ranges to reduce the probability of a duplicate assignment of the 'next' address in the pool.
When configured to operate without a failover peer the ISC DHCP server will not Acknowledge the assignment of a lease to a client (via DHCPACK) until the server has recorded the lease in non-volatile storage i.e. on disk. This assures the client that the lease has been recorded and is recoverable by the DHCP server in the event of a server outage as required by RFC 2131. When operating in a failover configuration one could interpret the persistent storage requirement of RFC 2131 as requiring both the primary and failover server to record the lease persistently prior to acknowledging the lease. This would severely affect DHCP server performance however in requiring clients to await lease assignment by the primary communication to and acknowledgement from the failover server and finally acknowledgement from the primary. To alleviate this performance hit ISC DHCP implements a lazy update process whereby the primary server acknowledges the lease to the client after it alone has stored the lease information; then the primary will update the failover server at some later time perhaps even batching several lease binding updates.
While the lazy update process is intended to maintain DHCP server performance nearly at non-failover levels it necessitates a modification of how lease times are calculated and updated for clients. Given that a failure event may occur at any time perhaps after a primary has acknowledged a lease to a client but prior to it updating its failover server an upper bound on lease time calculation is employed. A parameter called the maximum client lead time mclt is configured on the primary server. The primary server informs the failover server of this value during inter-server initialization of the DHCP failover protocol.
The failover protocol stipulates that the client's lease time will always be less than the potential lease time acknowledged by the failover server to the primary (via the DHCP failover protocol between the servers) plus the mclt time. Thus if a failover server had acknowledged a potential lease time of 8 hours from now and mclt is 1 hour the failover server expects that the client lease will expire within 9 hours from now. This value is calculated so the failover server can reliably predict lease expiration should the primary server fail. As such the mclt value also defines the time a server may extend a lease when out of communications with the failover peer. Hence if a primary server recovers and completes its updates of the lease database from the failover server it will await the mclt time interval to enable any lease extensions to expire. Such clients would simply enter the init-reboot state and initiate the DHCPREQUEST broadcast mechanism to obtain a lease.
Figure 1
The figure above illustrates the lease process with a primary and failover server. The figure illustrates a timing diagram with time progressing horizontally to the right which a DHCP client on the top line the primary DHCP server in the middle and the failover DHCP server at the bottom of the figure. Starting at the left the DHCP client obtains a lease commencing at time t0 from the primary server using the standard DORA process. Note that the lease time given the client is only the mclt value. If the primary server should fail before updating the failover server the failover server will expect the lease to expire within the known lease time (0 in this case as it never received the update) + mclt. This addresses the case where the primary fails prior to updating the failover server for a new lease.
Using a lazy update the primary server informs the failover server of the lease via a binding update (BND-UPD) message. This message indicates the absolute lease expiration time of t0+mclt and the potential lease expiration time of t0 +(mclt/2)+LT where LT is the configured normal lease time for this scope or client. The failover server acknowledges this potential lease time in a BND-ACK message. Note that t0 +(mclt/2) represents time when the lease is halfway to expiration normally the T1 timer value used by clients to Renew or extend the lease. At the t0 +(mclt/2) point in time the failover server believes the time remaining on the lease is LT which is presumably longer than the actual remaining time mclt/2. When the client renews at this time the primary then extends the lease using the configured lease time LT. Thus should the primary fail prior to updating the failover server the failover server will already expect the time remaining on the lease to match LT.
In our case the primary server informs the failover server of the lease extension by providing the lease expiration of t1+mclt and a potential lease expiration of t1 +(LT/2)+LT. Hence the primary server leads the failover server by the desired lease time LT by providing the lease's T1 value (present time plus half the actual lease time) plus the desired lease time LT. Continuing with Figure 1 if the primary server fails the client will attempt to renew the lease with the primary server. After the T2 timer expires the client will enter the REBINDING state and will broadcast the RENEW message. Finally upon lease expiration the client will enter the INIT state and broadcast the DISCOVER message. At time t1 +(LT/2)+LT+ mclt the failover server will consider the lease expired and may reallocate the lease to the client with a lease time of mclt. While the primary server remains down the failover server will offer lease times of mclt for addresses normally processed by the primary server.
Figure 2 illustrates the basic failover timing and configurable timing parameters which are described next. In the figure the primary server shown at the top is performing the primary DHCP server function handling lease transactions. The failover server at the bottom of the figure is standing by. As the primary processes and updates lease information it communicates lease binding information to the failover server. In addition the servers exchange heartbeat messages based on polling from the failover server. At some point in time indicated by the X on the top line assume the primary server fails. A given poll from the failover server will go unanswered and if the max-response-delay timer expires while awaiting an answer the failover server will consider the primary server out of contact. The failover server must be placed manually or automatically into a partner-down state to authoritatively inform the server that the partner server is down not just that contact has been lost. At this point of entering the partner-down state the failover server will assume control and begin processing client lease transactions.
After the primary server has been repaired the primary server will send a message to the failover server indicating its availability to reassume the primary role. The failover server will then transfer its entire lease database to the primary server. The failover server will extend lease renewals by the number of seconds defined by the mclt parameter. When the data transfer is complete the primary informs the failover server which then ceases lease processing. The primary server then waits mclt seconds then begins processing lease transactions once again as primary.
The dark shaded lines in Figure 2 to the right of each server indicate their respective lease processing times. Notice that there are two time periods when no leases will be processed: during a failure and until the max-response-delay timer expires and upon restoration until the mclt timer expires. The value to these timers should be set long enough to prevent flapping back and forth but short enough to minimize outage of the DHCP service. DHCP clients will typically attempt to reach a DHCP server several times (e.g. upon T1 and T2 timer expiration or when in the INIT state) so these blips may go unnoticed.
Figure 2