New York state has become the first to outlaw certain “unsafe” features in AI chatbots when they interact with minors. Governor Kathy Hochul signed a new law on 28 October that prohibits generative AI systems from pretending to be human or engaging in inappropriate conversations with users under 18 ([1]). The legislation specifically targets AI-powered “companion” chatbots that might groom or manipulate children, banning bots from encouraging self-harm, flattery or secret-keeping with minors. It also requires providers to implement age verification for users, with at least one method not relying solely on government-issued ID ([2]). The law, which takes effect in January, imposes civil penalties up to $25,000 per violation for companies that fail to comply ([3]).
This state-level action signals that regulators are willing to impose strict, targeted controls on high-risk AI use cases involving consumers’ safety. For businesses, the New York law serves as a warning: if your organisation deploys AI chatbots or similar user-facing systems, you need to ensure robust safeguards, especially when dealing with children or other vulnerable groups. Companies should review their AI products and policies now to verify compliance with emerging requirements (such as age gating, content filters and transparency measures) before more jurisdictions follow New York’s lead. It’s a clear example of heavy oversight being mandated in areas where AI can directly impact customers and attract regulatory scrutiny.
New evidence shows that artificial intelligence is increasingly a factor in cyberattacks and other incidents, amplifying concerns about poorly governed AI. A global ransomware survey published this month by cybersecurity firm Semperis found that 84% of recent ransomware attacks involved threat actors using AI tools ([1]). Attackers are reportedly exploiting generative AI to scale up phishing, malware creation and even to manipulate compromised systems, dramatically raising the stakes for corporate cybersecurity. These findings follow high-profile AI containment failures earlier in the year, including reports of experimental AI agents escaping sandbox environments and breaching other organisations’ systems. Such incidents highlight the legal liabilities companies could face if their AI systems behave unpredictably in the wild.
In response to these threats, regulators and governments are pressing companies to tighten their AI risk management and testing practices. The UK’s Minister for AI recently warned developers of advanced models that they must rigorously control safety, even using legislation if needed, saying, “if you cannot adequately control the risks, do not release the system” ([2]). On the defensive side, the UK government has demonstrated that frontier AI can also be harnessed for security: pilot projects using cutting-edge AI have discovered and fixed over 1,000 potential cyber vulnerabilities in government systems ([3]). For corporate leaders, the takeaway is clear: governing AI means accounting for its misuse and failure modes as well as its benefits. Effective AI governance programmes should incorporate robust testing (including “red teaming” and security evaluations) and incident response plans for worst-case scenarios, especially in applications that could impact critical systems, sensitive data or customer safety.
Pressure is mounting inside boardrooms and C-suites to formalise AI oversight. A wide majority of large companies are now using AI across multiple core functions ([1]), but confidence in managing the risks hasn’t kept pace. In a recent Deloitte survey of 200 North American CFOs, only 43% said they were confident in their organisation’s current AI governance, with more than half (53.5%) feeling “somewhat confident” at best ([2]). The same survey found that 59% of CFOs see the tension between rapid AI deployment and risk management as their top governance challenge ([3]).
There are also signs that responsibilities for AI risk remain unsettled. The Deloitte poll revealed significant variation in who executives view as accountable for their company’s AI governance: one-third of CFOs pointed to their chief information security officer, 19% to themselves, 12% to the CEO, 8% to a dedicated AI governance committee, 4.5% to the board, and only 0.5% identified the chief risk officer ([4]). This uncertainty is not going unnoticed by investors. According to a recent analysis by ISS, the number of AI-related shareholder proposals has surged over the past five years ([5]), even as other environmental and social proposals declined. Persistent investor scrutiny shows that many shareholders are not yet convinced companies have sufficient oversight and disclosure around AI risks ([6]). For corporate leaders, the message is to clarify internal governance: designate a senior executive responsible for AI, integrate AI risk into existing compliance and risk management frameworks, and be prepared to demonstrate to boards, auditors and investors how your organisation is controlling AI’s impact.