You're not losing the right. I liken it to Handy's example of he chooses not to assemble, then does he lose the right to assemble. My answer was no, but if he has a permit to assemble, it doesn't last forever. If he doesn't use it the municipality can revoke it after a certain amount of time and require that he get a new one. That's not trampling on the Constitution.
Regarding why in general the voter rolls are updated I would think it has less to do with mythical voter fraud that knuckledraggers are always wheeling out and more to do with more mundane issues such as the cost of contacting voters for mailers and other such updates. I get quite a few of those from the town each year, but if 3 times as many people are on the rolls as who actually live in the town that's a lot of wasted $$$ for people who are no longer around. So the boring question remains, how long do some of you want to keep non-voters on the rolls? Like if someone's a Civil War veteran, can we take them off?