after reboot, keepalived state will always be fault, though the check_script run successed.

[ddhq]

just like the title, it really confused me, if anyone come into this problem?

[pqarmitage]

If we are to be able to try and understand what is happening, we will need to following information:

1. Output of keepavelid -v
2. Your keepalived configuration file
3. The full logs of keepalived from when it starts.
4. When keepalived is running execute kill -USR1 $(cat /run/keepalived.pid). This should produce some files /tmp/keepalived*.data which should also be posted.

Once you have provided these, it will either tell us what further information we need, or we may be able to identify the cause of the problem.

[ddhq]

just like this issue,#2237,
thanks,i will try the vrrp_startup_delay

[ddhq]

version info

Keepalived v2.2.7 (01/16,2022)

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 4.19.90
Running on Linux 4.19.90-2212.2.0.0181.u89.fos22.x86_64 #1 SMP Tue Jan 3 11:09:40 UTC 2023
Distro: FusionOS 22

configure options: --prefix=/release/usr --with-init=SYSV

Config options:  LVS VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=SYSV

System options:  VSYSLOG MEMFD_CREATE IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

keepalived.conf

! Configuration File for keepalived
instance sds_keepalived
global_defs {
router_id LVS_DEVEL
vrrp_startup_delay 60
}

vrrp_script check_dasd
{
    script "/check_health_for_ftp.sh dasd"
    interval 5
    timeout 8
    rise 1
    fall 2
}


vrrp_instance dasd {
    state BACKUP
    interface ens192
    virtual_router_id 56
    priority 180
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    unicast_src_ip 192.168.102.144
    unicast_peer {
        192.168.102.145
        192.168.102.146
    }
    virtual_ipaddress {
        192.168.102.141 dev ens192 label ens192:26177356
    }
    track_script
    {
        check_dasd
    }
    notify_master "/lvs_virtual_router_notify.sh master dasd__ftp"
    notify_backup "/lvs_virtual_router_notify.sh backup dasd__ftp"
    notify_fault "/lvs_virtual_router_notify.sh fault dasd__ftp"
    notify_stop "/lvs_virtual_router_notify.sh stop dasd__ftp"
}

check_health_for_ftp.sh

#!/bin/bash

LOCAL_SHELL_PATH="$(dirname $0)"
LOCAL_SHELL_ABSOLUTE_PATH="$(cd ${LOCAL_SHELL_PATH}; pwd )"
IP='127.0.0.1'
AK='xxxx'
SK='xxxxxxx'
LOG_DIR=${LOCAL_SHELL_ABSOLUTE_PATH}'/log'
CONNECT_TIMEOUT=4
MAX_TIME=6
keepalived_name=$1

if [ ! -d ${LOG_DIR} ]; then
    mkdir -p ${LOG_DIR}
fi
LOG_FILE=${LOG_DIR}'/check_ftp_'$keepalived_name'_'$(date "+%F").log
echo "`date` [check $keepalived_name ftp service] check start" >> ${LOG_FILE}
msg=$(curl --connect-timeout ${CONNECT_TIMEOUT} -m ${MAX_TIME} ftp://${AK}:${SK}@${IP} 2>&1)
ret=$?

if [ $ret -eq 0 ];then
  echo "`date` [check ftp service] check successed : $msg" >> ${LOG_FILE}
  exit 0
else
  echo "`date` [check ftp service] check failed : code=$ret, msg=$msg" >> ${LOG_FILE}
  exit 1
fi

keepalived.log

Mar 07 16:21:51 localhost.localdomain systemd[1]: Starting LVS and VRRP High Availability Monitor...
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Starting Keepalived v2.2.7 (01/16,2022)
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Running on Linux 4.19.90-2212.2.0.0181.u89.fos22.x86_64 #1 SMP Tue Jan 3 11:09:40 UTC 2023 (built for Linux 4.19.90)
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Command line: '/usr/sbin/keepalived' '-D'
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: '--pid=/var/run/keepalived_ftp.pid'
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: '--vrrp_pid=/var/run/vrrp_ftp.pid' '-f'
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: '/etc/keepalived/keepalived_ftp.conf'
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Opening file '/var/lib/sdsom/etc/keepalived/keepalived_ftp.conf'.
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Configuration file /etc/keepalived/keepalived_ftp.conf
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: The vrrp_startup_delay is very large - 60 seconds
Mar 07 16:21:51 localhost.localdomain Keepalived[32957]: Changing syslog ident to Keepalived_sds_keepalived
Mar 07 16:21:51 localhost.localdomain Keepalived_sds_keepalived[32968]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Mar 07 16:21:51 localhost.localdomain systemd[1]: Started LVS and VRRP High Availability Monitor.
Mar 07 16:21:51 localhost.localdomain Keepalived_sds_keepalived[32968]: Starting VRRP child process, pid=32969
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Registering Kernel netlink reflector
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Registering Kernel netlink command channel
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Script user 'keepalived_script' does not exist
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Delaying startup for 60 seconds
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/check_health_for_ftp.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/check_keepalived_node.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/check_loss_package.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/lvs_virtual_router_notify.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/lvs_virtual_router_notify.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/lvs_virtual_router_notify.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Unsafe permissions found for script '/var/lib/sdsom/tools/configure/lvs_virtual_router_notify.sh'.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: SECURITY VIOLATION - scripts are being executed but script_security not enabled. There are insecure scripts.
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Assigned address 192.168.102.144 for interface ens192
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Assigned address fe80::250:56ff:feae:5626 for interface ens192
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Registering gratuitous ARP shared channel
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: (ftp) removing VIPs.
Mar 07 16:21:51 localhost.localdomain Keepalived_sds_keepalived[32968]: Startup complete
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: VRRP sockpool: [ifindex( 2), family(IPv4), proto(112), fd(12,13) , unicast, address(192.168.102.144)]
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Script check_ftp now returning 1
Mar 07 16:21:51 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: VRRP_Script(check_ftp) failed (exited with status 1)
Mar 07 16:22:01 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: Script check_ftp now returning 0
Mar 07 16:22:01 localhost.localdomain Keepalived_vrrp_sds_keepalived[32969]: VRRP_Script(check_ftp) succeeded

NetworkManager log

Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6124] manager: (ens192): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6135] device (ens192): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6311] device (ens192): carrier: link connected
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6337] manager: (ens224): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6349] device (ens224): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6520] device (ens224): carrier: link connected
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6542] manager: (ens256): new Ethernet device (/org/freedesktop/NetworkManager/Devices/4)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6549] device (ens256): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6740] device (ens256): carrier: link connected
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6911] ovsdb: Could not connect: No such file or directory
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6917] device (ens192): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6924] device (ens224): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6930] device (ens256): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6935] policy: auto-activating connection 'ens192' (03da7500-2101-c722-2438-d0d006c28c73)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6938] policy: auto-activating connection 'ens224' (1fce788a-5235-4f22-bba6-93995a54db07)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6941] device (ens192): Activation: starting connection 'ens192' (03da7500-2101-c722-2438-d0d006c28c73)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6943] device (ens224): Activation: starting connection 'ens224' (1fce788a-5235-4f22-bba6-93995a54db07)
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6943] device (ens192): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6946] manager: NetworkManager state is now CONNECTING
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6947] device (ens192): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6950] device (ens224): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6953] device (ens224): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6957] device (ens192): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6964] device (ens224): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6976] device (ens192): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.6993] device (ens224): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7049] device (ens192): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7051] device (ens192): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7062] manager: NetworkManager state is now CONNECTED_SITE
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7063] policy: set 'ens192' (ens192) as default for IPv4 routing and DNS
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7359] device (ens192): Activation: successful, device activated.
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7364] manager: NetworkManager state is now CONNECTED_GLOBAL
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7376] device (ens224): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7379] device (ens224): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7394] device (ens224): Activation: successful, device activated.
Mar 07 16:20:04 localhost.localdomain NetworkManager[1431]: [1741335604.7400] manager: startup complete

[pqarmitage]

See also issue #2528 although this isn't relevant if you don't have commit b48f005 included in your code (it was merged after v2.3.2).

[ddhq]

vrrp_startup_delay 300 still not work for me, but systemctl reload or restart keepalived will help

[pqarmitage]

Restarting keepalived was also found as a workaround to issue #2237, but it is not one that we can live with.

What we need to see is the keepalived logs produced when keepalived first starts up after a reboot, and not following restarting keepalived, since we need to see what is not being set up that keepalived needs, before keepalived starts up. We also need to see the /tmp/keepalived*.data files produced be sending SIGUSR1 to the parent keepalived process. It would be helpful to see those files produced both by the keepalived that is started after a reboot, and also the same files produced by a keepalived that has been restarted, so we can compare the difference. Without these files, there is really very little that we can do.

vrrp_startup_delay will not solve the problem you are experiencing. vrrp_startup_delay delays vrrp instances transitioning to master state, but it doesn't delay the vrrp process reading all the network configuration etc, which is where I believe the problem lies.

One option to try is that you change the SYSV init script for keepalived so that rather than directly executing keepalived it executes a script that does a sleep 60 before it then runs keepalived. That would allow time for all the network configuration to settle down.

[pqarmitage]

My recollection of how SYSV init scripts work is somewhat lost in the mists of time, but a quick search online suggests that you could simply add a sleep 60 in the init script itself before keepalived is run.

[ddhq]

o, i misunderstand the ability of vrrp_startup_delay, i originally think i has the same effect just like sleep, and then start to init keepalived;

the keepalived log above is the first time starting output after reboot, i add the required of network-online.target in systemd service

[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
Type=forking
EnvironmentFile=/etc/xxx
PIDFile=/var/lib/sdsom/var/run/keepalived_%I.pid
ExecStart=/bin/bash -c '${INSTALLDIR}/usr/sbin/keepalived -D --pid=${INSTALLDIR}/var/run/keepalived_%I.pid -f ${INSTALLDIR}/etc/keepalived/keepalived_%I.conf'
ExecReload=/bin/bash -c '/bin/kill -HUP $(cat ${INSTALLDIR}/var/run/keepalived_%I.pid)'

but the final sleep can help, i might be something need to be ok before keepalived,

thanks for the help again -U-

[ddhq]

#2277 the scene sounds just mine, persistent FAULT state, but the networkManager log above shows that the base interface is already ok.
oh, it is about ipv6, but i am using v4

[pqarmitage]

Can you please try keepalived v2.3.3. It has a number of fixes for persistent fault states.

[pqarmitage]

It is possible that v2.3.3 may not solve the problem for you since you are using such an old kernel. Some of the fixes rely on using IP_FREEBIND socket option on raw sockets, which wasn't supported until kernel 5.17.