Many small companies and individuals are desperate to reduce the burden on themselves, e.g. From CDM regs.
However, if they look up and smell the roses, most of the serious recent custodial sentences, have gone to small companies where the directors "flout the law" in the pusuance of profit.
When you walk down the street you see lots of breaches of HS laws on small construction sites.......
If I was running one of these small sites I'd be very keen to ensure that I knew exactly what was expected. After all it isn't really the HSE you need to worry about. Its the new sentencing guidelines pushed by the legal brigade who have helped to establish a "gold plated" prosecution scheme, way in excess and anti-business in the UK.
This is going to remain, regardless of any post Brexit law tinkering.
My advice. Learn the law and use the resources of HSE and CITB. Pushing back is quite frankly, really stupid. Protect thyself!