backport duplicate checking fixes to 1.4