You need to consider any applicable zoning laws. In some cases there is a "set back" -- the fence or wall must be x feet from the property line. Before you build you need to get up to speed on this subject -- or you may have to take the fence out if it doesn't conform.