Please DM or email me your recent sync reports and log files—this seems like something at the very end of the asset finalization pipeline is failing. If the finalization isn’t successful we don’t mark the asset as “shown”, and the library metrics only count shown assets.
I dug into it: the odd numbers are our bug, not your library: sync was re-counting the same files over and over, which inflated the totals on the Activity screen and makes it look like old files are being re-added. This will be fixed in the next release. If you can still send those logs, I’d like to double-check that your actual photo count is unaffected.