The slave failed with the error that the relay binlog is corrupt. It had copied close to 12 binlogs from the master and they were yet to be applied. Unfortunately those binlogs have been purged on the master. Now to sync up cleanly we might have to refresh data from the master which can be costly since it was a 290 GB database. We had the option of shutting down the server. We thought we can try our luck with a crazy hack. We shutdown the server. Tried reading the binlog using mysqlbinlog utility from the corrupt position. It failed as expected. Then we tried reading from the next immediate position and it went through fine. Now we had a proof that our hack might work. We opened the relay-log.info and incremented the second row by a value of one. Then we started the server. Boom, the slave started running and we were saved from a great pain of resyncing the slave.

PS : We might have missed one transaction in this hack, but that was ok for our use case.

The following one liner helps to sync a slave that is facing duplicate entry errors.

while [ 1 ]; do if [ `mysql -u root -e "show slave status \G;" | grep "Duplicate entry" | wc -l` -eq 1 ] ; then mysql -u root -e “stop slave; set global sql_slave_skip_counter=1; start slave;”; fi; sleep 1; mysql -u root -e “show slave status\G”; done

One of our peer groups had a badly configured setup where they were writing to 2 masters. Both the masters were setup with circular replication. They were also having a corrupted binlog event in the relay log.

To fix that, had to run a change master command to use the logfile listed under relay_master_log_file and the log position under exec_master_log_pos in show slave status.