Implementing video change detection by frame

I have a ML project I am working on that blurs objects in the frame. It currently does this frame by frame.

In order to increase accuracy and speed, I was thinking of maybe creating a heatmap of changed areas and if the area from the previous frame has not changed much then the box for the blur would remain regardless of whether it was detected again.

Obviously the model needs improvement, but it's not mine and I don't have the hardware to improve and retrain it yet.

Are there any libraries out there that could achieve this?

I'd love for it to be able to predict motion of moving objects and adjust the blur location but that may be asking too much.