Will I lose my retainer if I don’t want to work with my attorney anymore?

Not necessarily. The retainer you paid up front should be held by your attorney and used to pay for your legal work as incurred. If you haven’t used it all up yet, the remaining balance should be refunded to you.