As someone who went to college for the first time at 50 (I couldn't afford it earlier), I also would advise getting your degree. However, I think you should consider that it might not be college you want to give up so much as your course. Perhaps it just doesn't suit you? If this is the case, could you change to one you are more passionate about? It doesn't matter what others think, it's important that the course is the right one for you.
