There are lots of articles about bad use cases of ChatGPT that Google already provided for decades.
Want to get bad medical advice for the weird pain in your belly? Google can tell you it’s cancer, no problem.
Do you want to know how to make drugs without a lab? Google even gives you links to stores where you can buy the materials for it.
Want some racism/misogyny/other evil content? Google is your ever helpful friend and garbage dump.
What’s the difference apart from ChatGPT’s inability to link to existing sources?
But that was possible before. Just that these bots would just copy content, add random words and run it through Google Translate, translating it to a different language and back again. That already did the trick for the last 10 or so years.
Those bots wouldn’t pass the turing test, that’s for sure. One thing is pure spam like you’re describing, another is to be arguing with an AI (and losing) without even being aware that it’s an AI on the other side.