This is one of those situations where using the same representation (i.e. column matrices) for both points and vectors is confusing.
You have to keep track of what type of object each matrix represents.