Preventing the not invited from adding themselves to my meetings

[Doublej]

For a meeting that I have proposed is there any way to prevent individuals
who were not on the invite list from adding themselves to my meeting?

I think what is occurring is one of the invited guests is forwarding the
email which is then accepted by the not invited individual.