AI can rip you off. Here’s how California lawmakers want to stop price discrimination

California lawmakers want to stop artificial intelligence from ripping you off. 

In recent weeks they introduced five bills to address the issue, making predictive pricing based on a customer’s personal information one of the most popular tech policy concerns in the Legislature this session.

Ride-sharing apps, travel companies, and retail giants such as Staples, Target and reportedly Amazon have engaged in the practice, which can set different prices for customers based on factors including internet browsing data or where they live. In one recent example published by SFGATE, a person in the Bay Area was offered a hotel room for $500 more than people in less affluent areas. 

The pricing isn’t based on supply or demand. It’s based on predictions made about your eagerness and desires, said researcher Justin Kloczko. In one recent instance he found that Lyft charged his wife $5 more than him for the same ride. Kloczko works at Consumer Watchdog, an advocacy group that cosponsored one of the bills.

“They’re literally trying to hack your brain. They’re trying to read your mind and what you want,” he said. “I think this is happening all the time, it’s just really hard to catch.”

The package of bills proposed by California lawmakers to regulate AI responds to a call by Speaker Robert Rivas following elections last fall to focus on measures to address the cost of living.

One bill introduced by Assemblymember Cecilia Aguiar-Curry, a Democrat representing Davis, would make it easier for the California attorney general to pursue lawsuits against companies that use a pricing algorithm trained on nonpublic competitor data. Another bill would ban use of algorithms that personalize prices based on perceived characteristics or personal data. 

One more, Senate Bill 52, would ban use of algorithms that set prices for rental properties and allow tenants to sue their landlord if they discover use of the technology. The proposal follows the filing of an antitrust lawsuit by eight states against RealPage, a Texas company whose software is used to set rental prices. A 2020 investigation by The Markup and The New York Times found that RealPage used faulty algorithms to automate tenant background checks, falsely accusing people of crimes and denying them a place to live.

State Sen. Melissa Hurtado, a Democrat representing Bakersfield, also reintroduced a bill to prevent algorithmic pricing. The Senate Judiciary committee criticized the previous version of the bill last year because the attorney general already has the power to file suits for violating state antitrust law.

And, finally, Assembly Bill 446 would make it unlawful for retailers to use personal information or make predictions about them based on appearance in order to change the price of goods sold in a retail store. In the rollout of the bill, author Assemblymember Chris Ward cited a $5 million settlement with Target following a lawsuit by the County of San Diego where customers saw higher prices on the Target app if they were in the parking lot. The San Diego Democrat fears that face recognition in tandem with electronic shelving in grocery stores could lead to a future where prices are made on the spot based on a person’s appearance or physical characteristics. In a letter to Kroger, the largest supermarket chain in the U.S., members of Congress also expressed concern about the potential for tech-enabled price gouging based on the time of day or weather.

Tweaking and targeting prices can lead to revenue growth of 2 to 5%, according to an initial Federal Trade Commission report released days before the end of the Biden administration. For example, a company could target first-time parents or car buyers with higher priced items or make inferences about a person based on their location or IP address.

Consumer protection is a primary goal of state lawmakers interested in regulating AI, according to a recent analysis by the National Conference of State Legislatures. As Congress fails to act and policy varies wildly from one presidential administration to another, state lawmakers are stepping up. The State of State Tech Policy report published in December by NYU’s Center for Social Media and Politics found a 163% increase in tech policy proposals by state lawmakers last year compared to 2023. That trend is driven by one-party control in the vast majority of state houses across the country.

In other legislation intended to protect customers, one bill would require disclosure if AI is used to alter the images you see on the real estate website Trulia, and another would require health insurance companies to disclose the number of claims they deny using AI every month. Class action lawsuits filed in 2023 allege that UnitedHealthcare and Cigna used AI to deny people health care services. UnitedHealthcare and Cigna deny those allegations. A recent analysis by the National Conference of State Legislatures found that lawmakers in about a dozen states proposed bills to regulate the use of AI in health care.

Between the pressures of a budget deficit, wanting to keep businesses in California, the governor’s future political ambitions, and debate over whether regulation will stifle innovation, lawmakers will have to predict whether Gov. Gavin Newsom may veto AI bills, which is probably part of the reason why a number of them focus on kitchen table issues like whether the tech can raise the price of eggs or housing.

When you look at the slate of 30 AI bills proposed by California lawmakers, there’s a clear focus on kitchen table issues such as how AI can facilitate price gouging or raise prices in grocery stores, said Vinhcent Le, who until recently was part of the Consumer Privacy Protection Agency board where he led efforts to create rules that require protections for Californians from AI. He now works at Tech Equity, a nonprofit that is a supporter of one of the bills to prevent algorithms from setting prices.

“I’m pretty excited about the bills we’ve seen so far but the big questions are where is the governor going to be on these bills and where will the federal government take action and potentially preempt in the policy space,” he said.