NYC Government Chatbot Under Fire for Providing Inaccurate Information on City Laws and Regulations

The New York City government's "MyCity" chatbot has come under scrutiny.

It has been found to disseminate incorrect answers regarding essential questions of local law and municipal policies. This revelation underscores the potential risks associated with the use of LLM (large language model) chatbots, which have been known to generate inaccurate information while presenting it as authoritative.

Launched as a pilot program in October, NYC's MyCity chatbot was introduced as a resource to assist business owners in accessing reliable information from over 2,000 NYC Business webpages and articles. The aim was to provide guidance on compliance with codes and regulations, available business incentives, and best practices to avoid violations and fines.

However, a recent report from The Markup and local nonprofit news site The City has revealed instances where the MyCity chatbot provided dangerously erroneous information on fundamental city policies. For instance, the chatbot incorrectly stated that NYC buildings are not required to accept Section 8 vouchers, contrary to official city government information which clearly mandates acceptance of Section 8 housing subsidies without discrimination.

Further testing conducted by BlueSky user Kathryn Tewson revealed additional inaccuracies, including misinformation regarding worker pay and work hour regulations, as well as industry-specific details such as funeral home pricing. Particularly alarming were instances where the chatbot provided erroneous information on the treatment of workplace whistleblowers and even made inaccurate statements regarding the necessity to pay rent.

The underlying cause of these inaccuracies lies in the token-based predictive models that power such chatbots. These models, including the Microsoft Azure-powered chatbot used by MyCity, rely on statistical associations across millions of tokens to predict the most likely next word in a sequence, without true comprehension of the underlying information being conveyed. Consequently, discrepancies between the provided information and actual facts can arise, leading to potentially harmful outcomes.

Despite being labeled as a "Beta" product with warnings about potential inaccuracies, the MyCity chatbot is marketed as a tool to provide official NYC business information and aid business owners in navigating government regulations. However, these inaccuracies pose significant risks to users who may unwittingly rely on the chatbot's responses for critical decision-making.

While Leslie Brown, spokesperson for the NYC Office of Technology and Innovation, maintains that the chatbot has provided thousands of people with accurate answers, concerns persist regarding its reliability. Reports of inaccuracies from business owners and industry professionals highlight the need for greater scrutiny and improvement of such chatbot systems.

The issues surrounding the MyCity chatbot underscore the importance of ensuring the accuracy and reliability of AI-powered systems, particularly those deployed by governments and corporations. With instances of misleading information from chatbots leading to legal disputes and potential harm to users, there is a pressing need for enhanced oversight and accountability measures to safeguard against such risks. As advancements continue in AI technology, it becomes imperative to prioritize accuracy and transparency to mitigate the potential negative impacts of AI-powered systems on society.