Scholars question whether or not the actual father was Joseph because of the way the traditions in Matthew and Luke are recorded. They suggest that Mary's pregnancy happened outside of wedlock, although within her betrothal period and Joseph seems surprised, needing a vision from an angel to convince him to marry Mary.
Of course this is very much a hot button topic, since Mary has become in the religious tradition the Queen of Heaven and Holy Mother, perpetual Virgin, and so on.
For my own reasoning, I don't trust any of the information given in the virgin birth stories, not even the references to Mary's pregnancy outside of wedlock. These references all appear to me to be part of a growing story to portray Mary as non-sexual, a virgin whom Joseph didn't even touch, so that she can properly birth a god. Paul doesn't yet know this, only that God's son was "born of woman." This phrase is idiomatic and means something like "born a human being." His reference to "son" may be a reference to his status as Angel, since angels were known in the tradition to be sons of God.
What I do know is that the genealogies which both Matthew and Luke preserve (despite their very different versions of Jesus' virgin birth) trace Jesus through Joseph's line. The early teachings from Jerusalem also agree that Joseph is Jesus' real dad. So the earliest traditions appear to me to be that Joseph was Jesus' father. This gets overlaid with the virgin birth stories when they develop.
More to come...