Ideally, you would approach the person in a low key manner and explain your desire to not have someone on your property unannounced and ask for his cooperation in not tresspassing your land in the future. If you approach in a hostile or too confrontational manner you may be motivating this person to retaliate in some other way later.
Yes, they've already demonstrated by this point that they don't respect private property, but it may be such a thing that they commonly do this on other land and aren't aware of your preferences. I have found that many people "think" they can tresspass just because no one has confronted them previously or they knew the previous landowner, etc. So sometimes, all that is necessary is a simple request to get compliance.
If the gentlemanly approach does not work, then repeat offenses should be reported to the local CO, but I would try the low key method first. Good luck.
