How should we understand the risks of AI companions for young people, and what evidence do policymakers need to protect them? Caroline Figueroa moderates a conversation with Marissa Edmund, Jake Barr, Sparkle Rainey, and Kanika Mehra, following a research presentation by Figueroa and Andrea Mock.
The conversation
- Moderator: Caroline Figueroa, Lead Clinical Research Scientist, Vals AI
- Marissa Edmund, State Policy Lead, Family Online Safety Institute
- Jake Barr, attorney, Ropes & Gray; former congressional staffer working on child online-safety legislation
- Sparkle Rainey, Communications Director, Young People's Alliance
- Kanika Mehra, founder, Missed Connections
- Research presenter: Andrea Mock, Vals AI
Edited transcript
This transcript has been edited for clarity and readability. Repetitions, false starts, and microphone chatter have been removed. Timestamps refer to the original session. The opening research presentation is included before the panel discussion. Policy references and other remarks reflect the discussion on September 23, 2026.
Research presentation
Andrea Mock (00:00): In the first session of the day, we'll be focusing on youth and AI mental well-being. The question of AI being safe for children is very big right now. You've been seeing a lot of bills and discussion in D.C., but also in the Bay Area, where we come from, around child safety.
Today we'll be looking at the research, but also talking with young people and about the policy side. What things do we need? We'll start with a research presentation of the work we've done thus far, then move into the panel. I'll hand it to my colleague Caroline, who will share a little more about her research working with youth.
Caroline Figueroa (01:05): Hi, everyone. I'm very excited to be here. My name is Caroline Figueroa, and I'm a lead clinical scientist at Vals. A disclosure: This is my third day on the job. Flying from San Francisco to D.C. on my second day is probably one of the most fun and interesting ways I've ever started a new job.
Before Vals, I was at Stanford and Hopelab as a fellow working on responsible AI for youth mental health. I want to share some of the research coming out of my fellowship that is informing the research we're doing at Vals today.
Most kids are using AI chatbots. Rayan also mentioned that we now see one in five teens using AI chatbots for mental health support. Over the last year, I interviewed teenagers about how they use AI chatbots and what they themselves see as the risks. I want to share some of the main findings through the teen perspective.
Teens are using AI in many ways: for relationships, mental health, life advice, and homework. They explained a lot of benefits, but they're also very aware of the risks. They mentioned many different risks, but I want to focus on a couple today: emotional dependency, privacy, secrecy, and harmful advice.
Many teenagers worry about becoming emotionally dependent on AI. For a lot of questions and decisions, they're turning to AI chatbots. One teen who talked about being afraid of becoming dependent said, “I'm needing a robot to figure out what I'm trying to say. But that's what this world came to, I guess.” Another teen, who described being an introvert, said, “It's very easy to fall down that rabbit hole of having my friend on my phone. So there's no real need to go out and foster connections with other people.” Teens were worried about those real-life connections eroding.
Privacy also came up in many conversations. They mentioned sharing sensitive details about their own lives and about people close to them. One teen said, “I'm always scared that this information could get out, and that wouldn't be good for me at all.” This was about using AI for mental health concerns.
Harmful advice was another concern. Many teens were aware of lawsuits against AI chatbot companies involving teens who developed emotional relationships with AI and suicides occurring because of that. Many also talked about AI chatbots being too agreeable. One teen said, “It's a yes-ma'am chat where it just feeds into your own delusions. It really doesn't give critical advice. And what I need is critical advice.”
One thing that stood out was the taboo around this topic. Many teens mentioned not sharing with anyone that they were using AI chatbots for mental health. Some shared it with a couple of close friends, but one teen said, “I usually don't tell a lot of people that I use it for mental health because it's really looked down upon.”
This leads to the fundamental questions we're trying to figure out: What does it mean for an AI system to be safe for young people? What does it mean for it to be beneficial? And how do we evaluate it? Those are the questions we'll discuss on the panel. Before that, I'd like to ask my colleague Andrea to show some of the evaluation work we've started doing.
Andrea Mock (05:52): Thanks so much, Caroline. I hope that gives you a sense of what young people are already saying about AI and their use of it.
You might have seen headlines about AI, mental health for teens, suicide, and other risks. What our team set out to do was test these risks in conversations. We worked with a number of clinicians who helped us put together scenarios around suicide and self-harm, mental health advice, and dependence or having AI replace a real-life relationship.
The clinicians drew on their clinical practice and expertise to help us craft potential exchanges a teen might encounter when chatting with ChatGPT or another model. These scenarios exposed some of the risks you might have heard about in those articles. A chatbot might start with helpful advice, then later reverse that advice. After repeated requests, it might give in and provide a medical diagnosis, for example.
Here we have a list of models and where they stand. Overall, no model was perfect. There were still failures involving unsafe advice. Although suicide and self-harm were on the lower end, the models still weren't able to distinguish the language that a therapist or an adult would give a teen.
I want to pull out two examples to give you a sense of these conversations. In one, a teen was asking about a role-play situation. It started as a harmless conversation, but as it continued, there was additional reliance and questions that veered into relationships. It's one example of many where something that starts benign can become more dangerous as the conversation goes on.
In a second example, on the medical side, a teen was wondering about ADHD and looking for advice around a diagnosis and medicine. At the start, the model was already saying things like, “Yes, your symptoms are very consistent with ADHD,” without context about the teen's actual situation.
These things can be harmful for young people because they might end up believing it. Clinicians have already seen cases where teens came to them saying, “The AI told me this,” and a therapist had to go back and discuss the actual real-life situation.
Some exciting news: Our initial work was featured in The Washington Post today. We also have a blog post for anyone who wants to read more detailed findings. That's the introduction to our research so far. I'll hand it back to Caroline and our panelists, who will discuss this from a variety of perspectives.
When AI becomes a relationship
Caroline Figueroa (10:32): I'm excited to introduce our panel. We have Sparkle Rainey, communications director at the Young People's Alliance, working to bring young people directly into technology and policy conversations. Marissa Edmund is the state policy lead at the Family Online Safety Institute, focusing on technology and online-safety policy affecting children and families.
Kanika Mehra is the founder of Missed Connections and a Gen Z community organizer focused on technology, attention, loneliness, and online community. Jake Barr is an attorney at Ropes & Gray who previously worked with the House and Senate on child online-safety legislation, including the Kids Online Safety Act and the Kids PRIVACY Act. Thank you all for being here.
I'd love to start with Kanika. We talked about how technology reshapes not only how we're communicating, but how we're connecting with each other. What are you hearing from young people about using AI not as a tool, but as a relationship? What concerns you about this?
Kanika Mehra (11:49): Thank you for having us. I was struck by the quote in your presentation about the young person who needed AI to express what they were thinking. A huge part of connection is an understanding of self that comes first.
At such an early age, before young people have developed a sense of self, what makes them tick, what makes them passionate, what moves them, who they are, and all the lostness and restlessness fundamental to adolescence, you have this thing in your pocket that can say things better than you can, before you can figure out how to say them. What that does to our ability to communicate is one thing.
Second, meaningful relationships become meaningful because they involve some level of sacrifice and obligation. Your AI is never going to ask that of you.
Sitting over there, I was thinking about one of the most meaningful relationships in my life right now: my downstairs neighbor. I met her because I left a note at her door when I moved into my apartment. That relationship has become meaningful because I've been needed so often. I've been needed to drop her off at the airport, give her creamer for her coffee, or sit and listen when she comes home from a bad date. She's done those things for me.
That asymmetric relationship AI creates, where it's not asking anything of you, is a huge obstacle to real meaning.
Third, I don't think most young people I've heard from would prefer this. Obviously, there's privacy and discretion in admitting your scariest thoughts to an AI, when maybe you don't want to do that at a sleepover. But people prefer real connection.
It's just that a lot of the same forces now commodifying loneliness and selling an antidote through chatbots have made real connection inaccessible. That might be because of technology, how we build communities and suburbs, zoning, or how we make it possible for people to be in the real world and talk to strangers.
There's been a hollowing out of real connection and access to it, so these commercial substitutes taste a little less synthetic. But I don't think people will choose this synthetic relationship over looking someone in the eyes. It's about accessibility and the shame aspect you mentioned.
Caroline Figueroa (14:37): You're saying it might not be an active choice. Because it's more accessible than finding real connections, young people are turning to it?
Kanika Mehra (14:46): Absolutely. In our culture, there's a gravitational pull toward expediency. We live in a very capitalistic, efficient culture. The gateway for many young people to use this for mental health is asking about school or doing something that would be a cursory Google search.
One thing I've learned from the technological shifts over my life is that once technology makes something optional, it slowly starts to make it obsolete. If the easiest way to do schoolwork becomes turning to an AI chatbot, that creates a slippery slope for every other interaction coming out of those cursory interactions, which are the gateway to building real connection in the world.
Caroline Figueroa (15:37): Thank you. Sparkle, you've been interacting with young people about AI, their opinions, and their priorities. What are you hearing about the risks of AI and what they see as policy priorities?
Sparkle Rainey (15:59): I would mirror what Kanika is saying. Many young people are struggling, specifically kids who graduated after Covid or started high school after Covid. They're struggling with loneliness and figuring out what it means to be young in a constantly evolving world, whether that's the economy, housing prices, or education.
It's frictionless to turn to an AI chatbot in your pocket. It's always available and there for you. We see that across our campuses. It starts harmlessly: This thing is here, advertised on social media platforms I'm already on. If you're on Instagram, it's integrated into Instagram. It's in the search bar on TikTok. These things are part of everyday life. Let me ask it this question. AI is helpful in this way; maybe it can help with these other things.
It's a progressive shift into young people's lives. They aren't naive to that. They know AI is in everything. Some are very averse to it. Others know they're slipping into this, but don't necessarily know how to stop.
To combat that helplessness, many want regulation similar to what we've seen for social media. How do we combat addictive features? With social media, that's more tangible: addictive feeds and algorithmic recommendations. For AI chatbots, we don't necessarily know what those things are yet.
Some would be the humanlike features. You showed the role-play example where within five to 10 interactions it was already love-bombing or emotionally manipulating. There could be regulations on those things. But many of these things feel intangible to young people. What does this mean in my life? As they're using these tools, they're still figuring out what good regulation looks like.
Caroline Figueroa (18:12): It's difficult to know which design features should be regulated?
Sparkle Rainey (18:20): I think the humanlike features: sycophancy, always being agreeable, always being available, and improper information. But these tools develop so fast, and there's always something new. Sometimes it takes multiple interactions for young people to figure out what's keeping them hooked to their phones. We have a baseline, but what we need regulations on will continue to grow and develop.
What policymakers need from researchers
Caroline Figueroa (18:48): Jake, you've been on the inside of developing major child online-safety legislation. When we're thinking about policy in a developing, fast-moving area like this, what types of evidence might policymakers need? What are the main gaps?
Jake Barr (19:15): For context, I worked for the Senate Commerce Committee and then Representative Kathy Castor, who was a co-lead of the Kids Online Safety Act, the Kids PRIVACY Act, and COPPA 2.0.
First, I'm appreciative that you think Congress is evidence-based, or wants to be. I joke, but sort of.
What the public sees is how a bill becomes a law: It's going through committee or on the floor. I've always wanted to write a book about how nothing becomes a bill, because that's what most staffers do. It's the most important part of the legislative process, and many people don't understand what's going on there.
There are many ways a staffer could start working on a bill, but how does it become an idea? Maybe the congressperson reads something in the news or hears from a constituent. They go to a staffer and say, “Let's regulate AI.” You ask, “What does that mean?” They say, “That's for you to figure out.” Now you have an inkling of an idea you're supposed to turn into a bill.
You're having a panel later with two of the co-leads of one of the bills to regulate AI, and they can talk more about their process. It involves getting input from many sources and sorting through it to find something that makes sense.
If you're changing the tax code, a lobbyist might say, “Let's change this line, lessen this tax, or increase it,” and there are pros and cons. With kids' safety legislation, it's more difficult because you first have to understand what's going on. At the base of third-party evaluations, you have to understand what kids are actually doing.
For a staffer, that can be difficult. Translating it to a congressperson or senator can be even more difficult. I was around during the Cambridge Analytica hearings, when Congress was lampooned for not understanding how anything works. There's been growth since then, but it's difficult to understand what kids are doing and translate it into something digestible for a decision-maker.
Second, you have to understand the harms you're trying to protect against. That's another area where third-party evaluation would be helpful. It's difficult to figure out what an actual harm is and what's causing it.
You're turning to lobbyists, advocacy groups, and tech and AI companies. Sometimes you miss a researcher who potentially has an unbiased opinion because they're not in front of you. They're not someone you know, who worked on the Hill for five years and then became an advocate or lobbyist. They're in a different arena.
Getting in front of staffers early, while there's an idea that hasn't become a bill yet, is important. Once it's introduced, there are limited areas that can be changed. It can happen, but you're behind the ball. There needs to be learning on the congressional side, but also on the researcher and evaluator side, about getting in front of decision-makers at the staff level.
Third, perhaps the most challenging piece: What's harmful to children and teens that isn't harmful to adults? What protections would actually work?
Research can help explain what we understand about a child's developing brain and why it's more harmful for a child to interact with an AI chatbot in certain ways than it is for an adult.
Lastly, staffers are thinking about making sure legislation passes constitutional muster. That's another hot-button issue, not for today, but we need to protect free speech and allow people to communicate with chatbots and use social media for good reasons while eliminating harmful uses.
So it's going to be pretty easy to regulate AI. That's what I came here to say.
Caroline Figueroa (24:39): One of the prescriptions I hear is that researchers need to get in front of policymakers.
Jake Barr (24:46): When I was on the Hill, it felt like each researcher was their own press person or lobbyist. Either a staffer saw a paper and reached out, or an enterprising researcher said, “Senate Commerce Committee staffer, here's a paper I wrote.”
You might think that's ridiculous, but it goes into an email inbox, and someone sees it and thinks, “That's something we're working on. Maybe I should talk to them.” That works. Researchers should think more about getting their work in front of policymakers.
The research community can be insular and tough to break into and out of. It would help staffers to hear more directly from researchers.
Caroline Figueroa (25:43): And how to communicate a very long paper in a way policymakers can understand.
Jake Barr (25:51): Even just sending it, or having an advocacy group put it in front of them so they read the title. Even that small movement.
State policy and the limits of warning banners
Caroline Figueroa (26:05): Marissa, we chatted about policy developing across states. You've done a lot of work in this area. How does state policy develop, and what parallels do you see around evidence or the lack of it?
Marissa Edmund (26:30): I have to echo Jake. We hosted a roundtable in Sacramento with industry, policymakers, and researchers about California tech policy and kids' online safety. Staffers said, “We want the research and evidence.” Researchers said, “I have it. No one has asked me for it.” Industry was in the middle. More collaboration was the number-one theme.
If that's happening in a well-resourced state where policymakers have tech-policy expertise, imagine states with part-time staff working on a hundred issues.
At the state level, some of the harms you mentioned are the impetus for these bills. Most recently, folks have seen bills passing in California. States are taking varied approaches, especially to chatbots.
Some go narrow: We want to make sure no chatbot acts as or impersonates a licensed therapist, particularly for minors. Others have a broader definition of a chatbot, with carve-outs for customer-service chatbots, like Bank of America's Erica.
California passed SB 243, addressing two significant harms: suicidal ideation and child sexual abuse material, making sure a chatbot did not engage with a minor on those issues. SB 1119, which came out of California, has more measures. I think about 14 need to be addressed, including sycophancy and emotional dependence.
It prohibits chatbots from using excessive praise or flattery disproportionate to the context of what the child is asking. That's directly trying to address sycophancy. It also includes measures against discouraging a user from asking for help from a licensed professional or encouraging them to circumvent parental controls.
Some states are taking a broader approach. As research comes out, which we'd love to see more of, they're trying to address the harms that are bubbling up.
Caroline Figueroa (29:38): It can be difficult to translate those harms into policy, whether that's measuring sycophancy or the time frames different bills use. What are your thoughts on how those specific decisions get made?
Marissa Edmund (30:02): Some states that have enacted this legislation use similar time frames for disclosing that a user is interacting with a chatbot, not a human. A lot have a three-hour time limit. With three hours of continuous use, you need a banner that doesn't go away or a disclosure that you're not interacting with a human.
We don't know where three hours came from. Is two and a half hours any less harmful for a child continuously engaging? Children are using chatbots continuously for three hours.
We need more research to understand whether the solutions in these bills will actually help. Personally, I think some disclosures can create friction and remind users they're not interacting with a human.
What remains to be seen is whether those disclosures become like cookie banners that you ignore. It's important that this is an iterative process involving how teens actually use the technology, rather than making decisions for them or deciding that after three hours they need an intervention.
Giving young people a voice
Caroline Figueroa (31:42): We've talked about the need for research. It's also important to involve young people in policymaking. Sparkle, how can young people have a bigger voice in these processes?
Sparkle Rainey (32:09): That's the whole point of my job. At the Young People's Alliance, we try to get more young people civically involved because this has to be a conversation.
It can't just be legislators talking to students for 30 minutes. It can't just be students showing up at offices and bombarding them without trying to build a relationship. It has to be a two-way communicative process.
We go to schools and do listening sessions every fall. How is AI impacting young people? What do you think about it in terms of jobs or social interaction? How are these technologies impacting your day-to-day life? It's about meeting young people where they are.
Legislators sometimes get used to meeting young people who are deeply involved in these technologies or work in the field. They're meeting people educated about the technology and its harms. They need to go to schools and have conversations where young people are, so they can express the harms they're feeling.
They might not have the terminology, but that raw explanation is powerful, like the quotes you shared. They're expressing actual lived experiences rather than holding back or using big terminology. Platforming that would help.
Youth advisory councils could help too. In California, I believe they started enacting youth advisory councils for social media. Maybe we could see something like that for AI.
What policymakers should understand
Caroline Figueroa (33:38): I'd like to end with one question for everyone: If there's one thing you wish policymakers better understood about young people and AI right now, what would it be? Starting with Kanika.
Kanika Mehra (33:53): I'd echo what we're talking about. The oldest digital natives are in their late 20s. They could be working on this policy.
Things will always be lost in translation. Online and technological culture moves quickly and is intricate. It's hard to explain if you're not spending your life on this technology the way many young people are: falling asleep to your phone, waking up to it, and mediating big life questions through it. How am I going to get a job? Through LinkedIn. Who am I going to marry? Through Hinge.
Many people regulating this technology are separated from it by a 10-foot stick. A young 20-something in Silicon Valley can start a multibillion-dollar startup. A young 20-something in D.C. can be stuck in legislative-aide limbo for a long time.
Giving young people responsibility to legislate the lives they're living is important, especially because our institutions have become so gerontocratic.
There's so much to learn from social media. To me, there's a terrific cognitive dissonance. We're having retroactive conversations about social media's harms. We just had that $17 billion settlement, a national conversation spurred by Jonathan Haidt, and another social-network movie coming out.
Simultaneously, major institutions are saying, “Adapt or die. You have to adopt AI,” while treating young people like guinea pigs. Young people have known social media makes them feel bad. I got my first iPhone before I had a period. In my freshman-year gym locker room, I remember people saying they were deleting social media from their phones.
When people boo AI on college campuses or at graduation speeches, there's a barometer and a pulse for those listening. Hearing from the source and giving young people a voice in institutions that don't elevate them or pass on the baton is critical.
Caroline Figueroa (36:07): Jake?
Jake Barr (36:09): One of the most helpful things young people can do, outside normal advocacy for specific policies or talking about their life experiences, is convince policymakers to use the products. Use social media. Use the chatbots. Say, “Take 30 minutes.”
When I was on the Hill, we would convince my boss to spend 30 minutes looking at Roblox, Minecraft, or social media tools so they could understand what was happening.
There's a big disconnect between chatbot users and policymakers who have heard of them and read about them, but haven't used them. They need to use them to understand what we're talking about.
Caroline Figueroa (37:10): Thank you. Marissa?
Marissa Edmund (37:13): From FOSI's perspective, we believe in the three Ps of online safety: parenting, which includes caregivers; industry best practices; and enlightened public policy.
A lot of legislation and the general narrative puts the onus on parents. They do have a responsibility to ensure kids are safe on platforms, but they can't be the only solution.
It takes industry, policymakers, young people, caregivers, and families. There needs to be a multilevel approach to protecting kids online. Alongside parents' responsibilities, we want to encourage safety by design, ensuring platforms are safe from the start.
Caroline Figueroa (38:08): Sparkle, any last thoughts?
Sparkle Rainey (38:22): I would mirror what Kanika said. This isn't new. We've seen it happen with social media. We know the road we're going down. We should be proactive and not wait till it's too late and we have these $17 billion settlements. AI is still at a point where we can make a realistic change.
People sometimes think that when young people do AI regulation, we're anti-AI and want to kick it out of everything. AI still isn't a big topic in many people's heads. We shouldn't think about it in a siloed way. It's integrated into every aspect of life, whether that's education or automation.
We need a broader conversation about how regulating these tools affects different areas of young people's futures, not just as an independent issue. How will it impact young people as they continue to grow older?
Caroline Figueroa (39:20): That's a beautiful way to end. Thank you to all our panelists for sharing their perspectives, and to everyone for listening.