Read popular science books! There are lots of great popular science books out there that do an excellent job of explaining complex, cutting edge physics in a way that anyone can understand (and with no maths involved!). Just because you find physics difficult, doesn’t mean you can’t find out about all the weird and wonderful physics there is out there!
There are also lots of great documentaries as well that you can watch to find out more about physics – BBC Four/iPlayer is a good place to look for these.
You shouldn’t worry about not being particularly attracted to a certain subject, but it is useful to have an understanding at some level of most subjects. Alice makes good suggestions, so I just want to add that you will find that the physics you are taught changes over the years, in both content and how it is taught. So stick with it, you might find you grow into it, and it changes to your liking!