Rushden nearly replied with an equaliser on 71 minutes when Hinton's weak back-header was seized upon by Hall who rounded Brock but could not direct his shot on target from an acute angle.
Excellent play from Stamps down the left led to a perfect cross which Bennett headed back on 77 minutes.
Keeper Turley tipped the ball away and Henriksen followed up only for Hunter to block his drive.
Manager's son Talbot replaced Hall eight minutes from time but Harriers confirmed their victory on 84 minutes with a tremendous header from Blake who stormed in to nod home Stamps's left-wing cross. Mark Blake
Rushden gave the ball away straight from their resulting kick-off and Larkin raced clear to slip the ball past Turley.
Harriers made a triple substitution on 87 minutes with Shilton, Ducros and Corbett replacing Larkin, Appleby and Henriksen.
Partridge was booked in the last minute for a foul on midfielder Williams.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article