Is what the questioner did—taking a quarter of his father's house instead of his brother's debt, and paying 17,0 to his brother when he got married—unlawful or lawful?

The father is a guarantor for his minor son's debt. The liability of the guarantor and the guaranteed party are one and the same in terms of claim, so it is permissible to demand the debt from the guarantor. If the value of a quarter of the house is equal to the value of the debt, there is no harm in selling it to the son to settle the debt. If its value is greater, and the son (the questioner) pays the difference, there is also no harm. The permission of the brothers is not required for the sale to be valid, because the father is disposing of his own property due to the guarantee he is bound by. As for giving money to the youngest brother, it is a gift from the father, and singling out one of the sons for a gift is permissible if there is a legitimate reason, such as poverty and need, or assisting him with marriage, provided that fairness is observed among the children in giving, by assisting his brothers with the equivalent of what he assisted him with.