"Late have I loved you, beauty so ancient and so new, late have I loved you!" is the sentence in which Augustine sums up his search: God was within him, and he was outside, seeking him among the beautiful things God made (Book X, ch. 27).
The sentence comes at the end of the investigation of memory. Augustine has concluded that he did not find God in any region of memory, but above it, and then turns to God and confesses: "behold, you were within, and I was outside, and there I was seeking you." Deformed, he cast himself upon the beautiful things God had created, and these, which would not exist without God, kept him far from him (Book X, ch. 27).
The "late" sets the tone: it is not a lament over lost time, but the recognition that what he sought all his life was already deeper in him than he himself was. Augustine had already said something similar about the years among the Manichees: God was deeper within him than he was himself (Book III, ch. 6). And what memory has inventoried through the senses now serves to tell how God reached him: God called, cried out and broke his deafness, shone and dispelled his blindness, and he breathes, hungers and thirsts (Book X, ch. 27).
That is why the two most quoted sentences of the book complement each other: the restless heart of the beginning and the "late have I loved you" of Book X tell the same story, one as lack, the other as encounter.