Don't link it to your Google account. It's been such a problem for me too until I just unlinked my Google account from contacts. There is a sync issue and I don't know why it affects some users but not everyone. It has been a major issue before that had to be addressed by Google, but if it still persists for you, I'd recommend unlinking your Google account or switch your Samsung contact app to the Google one.
Try storing it on your samsung account or phone or as google account storage can be easily hacked with python json especially if someone knows their analytics.
