Paying attention is the most critical skill for college learning. Assuming that you have an adequate vocabulary to understand what you are hearing or reading, attending to the words and processing the information they convey is the essence of verbal learning. Doing so is hard mental work, and we are all afflicted with the tendency for our minds to wander to easier, more pleasant thoughts. It is therefore important to learn to recognize when you are paying attention and when your mind is wandering.
The tendency for a student's mind to wander off from academic subjects is largely a result of word associations. As I like to put it, your mind has a mind of its own...namely, that part of your mind that is unconscious. It seems that your unconscious is always alert for any word that may occur in a book or a lecture that is in some way associated with any non(tm)academic topic that may be troubling you. If you are concerned about money, health, sex, loneliness, inter(tm)personal relations, etc., your unconscious keeps trying to bring these to the fore. Unless you are alert to the temptation, your mind can easily wander from physics to your love life.
There is no cure for your mind's wanderlust. It may help if you set aside some time in your daily schedule to think about your troubles. By satisfying the need to worry, to reminisce, to day-dream at other times, you can better resist those thoughts when you are studying. Otherwise, the best you can do is to learn to recognize when your mind starts to wander so you can bring it back on course.