New author here. Being the narcissistic bastard that I am, I've sometimes noticed the number of votes for my works decreasing. I haven't lost all that many votes, I'm just curious. Can you unvote? What causes this?
The site regularly performs "sweeps" that remove votes considered questionable. The only known criteria is multiple votes from the same ID(s)/IP(s) Beyond that, it is a secret to prevent voters with less than honorable intentions from avoiding the sweeps.
I've noticed it too on one of my stories. The votes are steadily going down, at least 9 or 10 gone now, and the rating is steadily going up.
May be happening with others of my stories, too, I don't know. The only reason I noticed it on this one is that if two more leave there won't be enough votes left for it to make the toplist.
I found that to be always the case with my scoring. Every time vote totals sink, my score improves. As for fearing your loss on the Top List, you probably shouldn't. Votes come in all the time.
Have you noticed your external/public numbers don't match your internal ones? The only thing I have up that pretty much always matches is the stroker that so ticked (or just struck them as a piece of sh**!) folks off it sits deep, deep in the lower level of the Lit stacks and no one ever sees it or votes on it. That puppy keeps at solid 4.18 and 111 votes.
So, don't fret too much. Also, when a new work is submitted, you'll notice an increase of votes on all your pieces, so that'll pull that low vote up, too.