Mother is the physical source of a house (Jews follow the maternal line) while fathers are the master and protector of the house.
Referring back to Christ earlier references to a house not being divided against itself, Christ is using this opportunity ask what the real meaning of a "house" is under the universal rule. He is asking us if we can see beyond our physical relationships. Just as we are joined into a physical house by the information we share in our DNA, we are also bound into a "house of ideas" with those with whom we share ideas and beliefs.
"Mother" is from mêtêr (meter), which means "mother," "grandmother," "mother hen," "source," and "origin." "Brethren" is from adelphos (adelphos),which means "son of the same mother," "kinsman," "colleague," "associate," and "brother."