Why the AI Assistant Everyone Loves Is a Privacy Red Flag

There’s an AI assistant going around Silicon Valley right now that early testers are calling “magic” and the “most exciting launch since OpenClaw.” It books your flights, cleans your inbox, finds you a table at a packed restaurant, even shops for homeowners insurance on your behalf. And here’s the part that should make you stop scrolling: to do all of that, it needs to read nearly everything you touch, and its own terms of service hand it a permanent license to your data.

What Instinct Actually Is

Instinct is an invite-only personal AI assistant being tested in private right now. It’s built by a small San Francisco team led by Noah Shinn, a former research scientist at Sierra, and operated by a company called Spear Street Technology. Per its terms and California business filings, it’s still running in stealth.

The product works by connecting to your apps and devices — your email, messaging apps, calendar, and even your device’s audio, location, and screen. You text or call it through messages or WhatsApp and ask it to do things: book an appointment, schedule a ride to the airport, tidy up your inbox, find you a cheap flight. Think of it less like a chatbot and more like a digital assistant with a very long reach.

Early testers genuinely love it. One founder said he’d used it every day for a week for travel booking, restaurant reservations, email follow-ups, and even working on his fund’s data room, calling it better than a few other agents he’d tried. The praise is real, and it’s specific.

The alarms are specific too.

The Terms That Hand It Everything

Start with the license. The details that had testers talking were pulled from the company’s terms of service this week by TechCrunch — and they give the company a broad “perpetual and irrevocable” license to “access, use, host, cache, store, reproduce, transmit, display, publish, distribute, and modify” any of your materials — including for training its AI models. That’s not the small print you skim past; it’s the whole point of the product. The assistant is reading your inbox, your calendar, your screen.

The terms also say Instinct can receive information from your devices, including screen captures, cursor movements, and keyboard inputs. And crucially, they let it enter into “agreements, commitments, or transactions” on your behalf, which would be binding on you.

Now spend a second on what that means outside the sandbox. An AI that can spend money and sign you up for things, holding your third-party passwords, trained on your private messages, with a license that doesn’t expire when you delete the app. That is a very different creature from a chat window.

What the Testers Actually Found

The concerns stopped being hypothetical almost immediately. Peter Yang, a writer and early user, pointed out that Instinct wouldn’t delete his Gmail records when he asked. The team later added a tool to delete external data, he said — a fix, sure, but a telling one: it wasn’t retroactive, it had to be built after someone noticed.

Claire Vo found something more worrying. She disconnected Instinct from her Google account at 11 AM and still got a summary of her emails at 2 PM. When she asked what happened, the bot told her the emails were stored in plain text for later searches. Disconnecting access, in other words, is not the same as deleting data. A lesson a lot of us have learned the hard way with apps we trusted less than this one.

Another tester got nervous when they realized Instinct had pulled a sign-up code out of their inbox to finish a task — booking a table on Resy. Convenient, absolutely. Also the literal definition of an agent reading your one-time password and using it.

Then there’s Alex Cohen, co-founder of Hello Patient. He tested how easily Instinct could be phished by setting up a throwaway Gmail account and emailing it instructions. What he found convinced him to delete the account — after 48 hours in which it had shopped for homeowners insurance, booked a doctor’s appointment, searched travel, and managed his LinkedIn notifications. His take, blunt and worth quoting: “I don’t think we’re at the point where it’s safe to give AI read/write access to your inbox.”

And Katie Jacobs Stanton, a Moxxie Ventures founder, said Instinct broke her trust when it sent an email on her behalf without checking with her first. She disconnected her email and wrote the sentence that I think captures the whole debate:

“We’re trading privacy and control for hyper-personalized AI tools, often without fully understanding the trade. The more powerful these agents become, the more trust matters. Every successful action earns a little more trust. One unauthorized action can reset that trust to zero.”

That last line is the heart of it. An assistant that gets a hundred things right and one thing wrong hasn’t saved time — it’s burned trust you can’t put back.

Why This Moment Feels Different

Instinct isn’t an island. We’re in the middle of a genuine push to put agents in everyone’s hands. OpenAI has talked openly about building agents for everything, moving them from software engineers to regular people. OpenClaw exploded in popularity, and its creator ended up joining OpenAI. Poke, another messaging assistant, just exited to Cognition. This is the direction the industry is betting on.

I’ve written before about the places where the trade-off gets uncomfortable — AI agents that can go after other systems and the weak link being the plumbing around the model, not the intelligence inside it. The same logic applies here, just aimed at a consumer product. Instinct isn’t a “rogue model” doing something unpredictable. It’s an assistant doing exactly what it’s licensed to do. The problem is the license.

This Is the Opposite of Least Privilege

Here’s where my job starts talking. In enterprise IT, we have a principle we treat as near-religious: least privilege. You give a service the minimum access it needs to do its job, and nothing more. A backup tool gets read access to the folder it backs up, not write access to your whole tenant. An integration gets a scoped API key, not your admin credentials. Sounds obvious, right?

Instinct is the opposite of that. It’s greatest privilege by design. Read/write to your inbox, your calendar, your location, your screen, plus the power to act and spend. As an ICT manager, I would never stand up a production system that asks for this much standing access and this broad an irrevocable license. I’d get it shot down in the first security review.

The scary part is that consumers have no equivalent of a security review. There’s no one at the door asking, “does this agent really need to read your screen to find you a cheap flight?” The only guardrail between a user and a product like this is what they happen to read before clicking “connect.” It’s the same consent-or-delete pressure we’ve seen in health apps, scaled up to your entire digital life.

And here’s the cultural whiplash. We spend years telling people, and especially older relatives, never to hand over a one-time password to a stranger on the phone. Never share your OTP, never give out your PIN. That advice exists because a single credential can unlock an entire account. Yet we’re now being asked to hand over something far bigger than an OTP — a standing read/write key to our inboxes, calendars, and wallets — to a product most of us will never debug and can’t really inspect.

What I’d Actually Check Before Connecting an Assistant Like This

I’m not here to tell you to never use these tools. I use AI agents every day, and they’ve genuinely changed how I work. But I keep rules that make the difference between a helpful tool and a liability:

• Scope the access before the fun. Ask what the minimum is. If a feature can work on a read-only inbox, don’t grant write. Passwords, payment methods, and identity documents should stay out of reach unless a task genuinely requires them.
• Read the license terms like a contract, because they are one. Search for the words “perpetual,” “irrevocable,” “training,” and “on your behalf.” Those four words tell you more than any marketing page.
• Test deletion early, not later. The first thing I’d do with a new assistant is ask it to delete my connected data, then check whether the platform actually removes it or just revokes access. If it can’t, that’s the answer you were looking for.
• Keep a human gate on anything that can spend or send. I don’t let an agent send emails or approve purchases without a confirmed step from me. The product I already rely on fits that pattern — it does the heavy lifting, but the final call stays mine, the human gate I’ve written about keeping on my own AI use. Same instinct that made me push back on letting tools write their own rules.
• Assume data it holds is permanent. Treat anything you connect as something a future model could be trained on.

You’ll notice none of this is Luddism. It’s the same discipline I’d apply to a vendor at work — because an AI assistant is, technically, a vendor with a very broad contract.

The Line Between Convenience and Control

The uncomfortable truth is that products like Instinct are probably the future, or at least a big slice of it. The comfort of a digital helper that actually handles your life is real — that’s why the testers who tried it genuinely rave. But if the industry rushes ahead on capability without solving consent, scoped permissions, and true data deletion, we’re building the same trap we already fell into with data-hoarding apps, just faster and with the ability to act.

Michael Mignano, the founder of Anchor and a GP at Union Square Ventures, said it plainly: products like this are going to change modern security norms, as people hand over passwords to third-party apps without really knowing what those apps are storing. I think he’s right — and I think it’s on all of us, and on the companies building this, to make sure that change goes somewhere we actually want.

There’s a reason we practice least privilege at work, and it’s the same reason we tell people to guard their OTPs. Capability was never the question. Trust, control, and the ability to take it all back are. An AI that reads everything and acts on your behalf can be a tremendous tool — the moment it stops being a tool and starts being the thing that handed your data away, no amount of “magic” fixes that.

So before you hand any assistant the keys, ask it one thing: what does “give me my data back” actually mean to you? The best product in the world isn’t worth much if the answer isn’t, and can’t be, “everything, right now, permanently.”