Depending on your companys policy, fraternization can include romantic relations between managers and subordinates and relationships between co-workers. You may get fired if the fraternization interferes with your work or goes against company policy.

When should I go to HR?

If your manager is discriminating against you because of your race or national origin or some other protected area -- you should go to HR and file an official complaint. HR is legally bound to investigate the situation. If your complaint is found to be valid, they are required to act.