All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
There is no blame upon you if you distance yourself from this person if it is in your interest; however, this should not be by means of severing the relation. In other words, you should speak with him and greet him when you meet with each other – in which case you would not be labeled as “forsaker”, and thus it is not included in the Hadith you quoted. In his Fat'h Al-Baari, Ibn Hajar said, “The majority of scholars are of the opinion that forsaking ends with just saying salaam and replying to it.”
If his way of speech is to speak privately in the presence of a third party in a way that might sadden him, then it is called At-Tanaajee (i.e. conversing with each other in secret) that is prohibited in the Sharee'ah. On that, Al-Bukhaari and Muslim narrated on the authority of Ibn Masooʿd that the Messenger of Allah said, “When three of you are together, two of you must not converse privately ignoring the third till the number increases, lest the third should be grieved.”
Allah knows best.