If your work, personal files, or life are integrated into the cloud through Google’s services, you may be at risk because of AI flagging. Recently in the news, manga artist Masahiro Itsosugi lost access to everything on his Google account due to it being terminated. Why was it terminated? AI flagged his account based on the manga content that he uploaded to his Google Drive. An appeal was ignored; and he suspects the entire review process was AI driven.
More recently I saw an offer for 10TB of cloud storage for $10 monthly (or something similar). I thought this was a good idea and considered using it as a backup in case all of my backups fall prey to a natural disaster (e.g. fire, tornado, flooding, etc.). However, in light of the fact one of the big players (Google) uses AI to monitor accounts and uploaded content, I thought twice about it and walked away. If a big player does it, so will other companies at some point. More importantly, I don’t like the idea of my personal files being monitored–if AI flags them as inappropriate, it can cost me my entire Google account and part of my life with it. I think that is too much of a risk.
We already have concerns about companies using our private data to make money; in fact there are laws to keep some of this in check, but it isn’t always a sure thing. Data breaches are an ever present thing in the IT world, and these represent security risks for when people are required to enter their personal information in order to access and use a service. What do we do when hackers build an AI that can be used to scrape, mine or search data that has been breached and released to the Dark Web?
More importantly, what do we do when AI is used to monitor our information and behavior, all to determine if we are fit for a particular purpose, or if we have committed a crime?
An incident in the last year brings this kind of system into question. In October of 2025, a boy at a Maryland high school had police surround him with guns drawn; this was to detain and handcuff him while they determined if he had a weapon on him. The assumed weapon was actually an empty bag of chips that had been falsely flagged by AI camera software as a gun. One officer was visibly frustrated on his body camera; human error was to blame in communication and escalation despite the alert being cancelled; the school’s county leaders called for the school to review their practices; others wanted accountability. More importantly, at the time of the news video, the system was active in at least fifty schools across the country. Find the video here.
Of note is also the “Advanced Impaired Driving Technology” provision that was signed into the 2021 Bipartisan Infrastructure Law. It requires passive monitoring systems in cars to detect and prevent drunk driving by shutting down the vehicle. At the time technology was not available to do such a thing; but we have come a long way in AI technology and monitoring software. In 2027 this monitoring will be required by Federal mandate in all new cars. In not so many terms, it means that the technology will monitor and determine your behavior based on every action that you make or take–whether you are aware of it or not. If it determines the driver is unfit to operate the vehicle, it could shut the vehicle down, prevent the vehicle from starting up or limit vehicle speed. This system is passive and the driver has little say in it. This could be a serious problem in conditions such as heavy traffic, city driving, obstructed roads, or poor weather conditions (among others).
AI monitoring software is fallible. In 2019, an AI traffic camera fined a Chinese driver a distracted driving ticket; as it turns out he was scratching his mouth and the software flagged it as a mobile phone. But what if such a system works as it is supposed to? If a person is bleeding, dizzy and half out of it, then they really shouldn’t be driving a vehicle. But what if they have a life threatening injury that creates an emergency where they must rush to a hospital that is an hour away? If the system operates as it is expected, it might flag the driver as unfit to operate a vehicle. The opportunity cost for such a system is in question when the cost can be a nuisance expense for an unwanted feature added to a vehicle’s retail cost, or as high as a human life.
Entering the news more recently are Flock cameras which have been put into Lowes, Menards and Home Depot parking lots. They are there for the typical reasons–scanning vehicles for specific data including license plate information. This information creates a retail security database that tracks shoplifters, banned individuals and then repeat offenders. Some of the data might also be shared with local law enforcement on stolen or wanted vehicles. However, these cameras are also appearing in private communities to track information on vehicles to prevent crime or aid in investigations.
The concern starts when these kinds of surveillance devices are used to gather information. This information is stored in a database. Normally only authorized individuals have access to this information, but as mentioned, data breaches can easily change this and release the information to the Dark Web. Further, AI monitoring software can be used to scrutinize, and while it might be helpful, it might also perform a false positive. When needing to be transparent for police, accept detainment or otherwise go through the proper channels (e.g. the court system) in order to deal with false positives, it becomes more than a “minor inconvenience” and becomes a time sink at worst, and a bad memory at minimum.
I may be jumping the gun on this, but AI software monitoring and flagging us is kind of invasive and creepy. We did this for hundreds of years without the technology, and even human error has brandished the law on the wrong person at points. AI software is still in its infancy and fallible; do we really need such a system monitoring us? What about when a company or the government decides to cross that line (because reasons) and monitor us for trends? No right to reasonable degree of privacy is a thing in public, so there is nothing preventing it. But with talk about stores replacing prices with digital labels, and testing adjusted pricing (on demand pricing) based on customer demand (e.g. times of the day and week), it would only be a matter of time before AI monitoring becomes normal.
The bottom line is that humans make errors and AI is seen as the robust solution to bridge that gap; but without human intervention (which is also fallible) AI’s mistakes can be left unchecked and cause major inconveniences, if not ruin lives. But there might come a time when it is normal for us to always be monitored; that information can be processed as the processing entity sees fit–the law only keeps actions in check if an illegal action is noticed and enforced; it basically goes both ways. We would be looking at a possible cyberpunk future where we accept all of this as the norm.
I, for one, am not in favor of the technology. It helps in some areas, and fails in others. If we have been getting along fine for hundreds of years, we can do so until we have perfected the technology. If Walmart self-checkout machines are anything to go by, we have a long way to go.
