The Parliamentary Women and Equalities Committee has released a new report on the issue, in which it says it was "astonished to find that the HSE does not see tackling or investigating [sexual harassment at work] as part of its remit".
It also criticised employers for "ignoring" their responsibilities, and said it is time the government "put sexual harassment at the top of the agenda".
The committee carried out a six-month inquiry after a BBC survey last November found that 40% of women and 18% of men had experienced unwanted sexual behaviour in the workplace.
The report of the inquiry, entitled Sexual Harassment in the Workplace, says: "The HSE told us that there is no specific duty under health and safety legislation regarding sexual harassment, and that law on sexual harassment was for the EHRC and the police to enforce."
"We understand that the HSE must prioritise its use of resources, but we cannot accept that sexual harassment is not sufficiently serious to be worthy of its attention. We note that HSE guidance on work-related violence lists sexual harassment as a potential form of verbal abuse but not as a form of physical violence."
It continued: "We are deeply concerned that the HSE's analysis of the potential for harm caused by sexual harassment appears to be cursory and ill-informed. We suspect that this issue has simply been ignored, as it has been by employers themselves, but we are perplexed that it continues to reject the suggestion that it should now be taking action."
The Women and Equalities Committee has published several recommendations for the government to prevent harassment, including requiring regulators to do more to actively deal with the problem. Bodies should draw up an action plan stating how they will ensure the employers they monitor tackle workplace sexual harassment, and the enforcement action they will take, it said.
The report concluded: "The HSE must take up its share of the burden of holding employers to account if they fail to take reasonable steps to protect workers from sexual harassment. This could include issuing guidance on the actions that employers could take, including undertaking specific risk assessments, and investigating reports of particularly poor practice."
A spokesperson for the HSE told IOSH Magazine it will contribute to the government's response after considering the findings and recommendations. "The Equality and Human Rights Commission continues to be the most appropriate regulator on matters of sexual harassment," it added.