Dennis, this happened to me on my first new home. They (the neighbor) were 4 inches into my property with the wall that was constructed by the builders.
Give your neighbor the six inches verbally and keep the wall. Everything they do on the other 6 inches belongs to you! This is a secret. Shhh