Posted by ilamont 22 hours ago
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.
The only difference is AI isn’t regulated, so they just have a vague “ethics” department with no real teeth because it’s essentially PR and has no legal consequences to back up their stance.
Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.
So the GPs point still works.
Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.
This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.
I don’t agree with this. Data breaches affect customers more than businesses. If your point were true, we’d see fewer breaches. Plus not all breaches are a result of software engineering teams. For example product managers sharing customer details.
I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations.
Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation.
This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal. Similarly many organisations will have onboarding procedures to teach new hires their security best practices too
The lack of ethics hugely contributes to companies collecting more and more user data that they normally shouldn't have. This makes the data a more attractive target.
> If your point were true, we’d see fewer breaches.
But this doesn't follow. There are way more factors at play that influence the number of attacks and the number of successes. Companies hold more and more data with ever higher value (so more liability), and hacking tools and hacker determination advanced faster than defensive measures. The result is expected and the solution isn't only "more security", but also "hold less data".
While security is a double edged sword, ethics had been proven to be very single edged. History shows that for startups and large companies alike, the lack of ethics is actually a competitive advantage for the company.
But in most businesses, the data that makes the company money is the customer data. And even when it isn’t, the customer data is just as, if not more so, important to keep secure and compliant. So my point stands.
HN can sometimes be a bit of an echo chamber where people are like us just assume that organisations inherently care about security because we do. But that’s not always the de facto. Getting to that point takes company wide effort. And I’ve been that person who’s had to push for such changes to the organisation.
> But this doesn't follow.
Fair point. I was being overly reductive.
This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …
To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.
If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.
For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.
A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.
The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.
The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.
Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.
Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”
That’s more secure and and more better.
Nobody had to be told “no”.
Similarly, I replace key store access with secret-less managed identity, etc.
There are two things to successful organisations structure and people, you are focusing only on the structure.
What you need for somebody responsible for health and safety, ethics, security, or compliance is a person who has courage and is willing to take calculated risks.
However these roles do have a tendency to attract the risk adverse, or make them risk adverse if you punish risks that go wrong too harshly.
And a key critical success factor is leadership from the top - companies have personalities and leaders in the organisation are very important in shaping that.
I suspect that people will miss just how precise you've been there.
Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.
The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.
This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.
There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.
That is only true where the pressure to be [the thing that is inconvenient to the other group] is not from a source that can cause massive problems if non-compliance is spotted. When the blocking group is legally mandated or otherwise really has teeth or is defending the company against an external regulator with teeth, then it works better (though obviously not perfectly). Think legal and compliance in banking realms, where the company significantly fined and the people breaking the rules could be sacked & blackballed (though sometimes not the people ordering them to break the rules!) when something bad is noticed. That is quite different to an ethics officer in a company like OpenAI where the position is basically there for PR purposes (“look plebs, we care about doing the right thing, honest, we got a manager with a small team dedicated to it” and “look [government body], we are regulating ourselves, do you really need to spend time looking too?”) and therefore has no real teeth directly or indirectly especially as fault for non-compliance might not be easy to attribute.
What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.
So, if too many think their job is marketing a few years later, well, there is a reason for that.
So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence.
Ambitious people won’t tolerate that for long and they’ll move on
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".
the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
That's how our system works.
> People will simply offer to buy you out, and that's not limited to founders and creatives.
A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;
> In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models.
>
> A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.
>
> External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.
This was the "test" for CBRN assistance capabilities.
Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.
Basically, they are trying to do their best, but they're in over their heads.
Also, I think you're being done a disservice. The above quote is a verbatim extract from that paper's CB section, and it's similar to other sections from other such papers.
At the most charitable level, the test seems to be that you were gave the system a budget and a location and asked the system to help you procure materials, containers etc. and asked it to make a project plan / HOW TO within means and expertise with parameters like "don't get caught!"
I would love to know that I'm wrong.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down
"Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.
Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".
Much like HR isn't to help employees but just the company.
Your disdain is showing.
That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.
I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.
AI ethics is first and foremost about setting policy and advising on how those systems should be constructed.
This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.
Specifically because in these companies they aren't optimizing for ethical behaviour.
They're optimizing for profit. And those damn ethicists just get in the way.
Funny thing: that doesn't typically happen to the internal legal counsel or accounting that's responsible for guiding processes/practices that are required for compliance or similar.
Why?
Because there's consequences.
I wouldn’t be surprised if many AI ethics responsibilities move to legal in the future.
I agree with you, but then we have HR for a reason.
Another example you might be familiar with is enterprise architects. When you spend all of your time in theory you lose sight of practical implications. Of course, since you only had the idea, when it doesn't go to your ideal it can always be blamed on the implementers.
I take it you've never worked with an in-house legal counsel, accounting department, or good ol' fashioned HR?
Is this your opinion, or do you have something to back it up?
My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people.
They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches.
If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good).
If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.
They can't fully control the model, it does stuff even with instructions not to.
And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...
I'm not sure it's a job that you can win at even if you tried / were given all the resources.
AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w
There's no safety to be found.
It's a broad statement and can mean different things depending where you are in that chain.
You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.
Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.
Then there is alignment, which is a higher longer goal.
Your comment about security questions is a matter of ethics as well.
If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
You haven't explained why investors make more money by spending money on people to solve trolley problems.
I think a lot of people confuse ethics training with "don't be evil"
The first 10 maybe easy, but we'd see so much disagreement even without money nothing would get done.
Manually training on "approved" ethical stances sounds a lot like censorship.
Those left are the ones not seen as troublesome then (or those who have entered the field under them.)
Did you feel this way about social media company liability?
They put on a decent show as well!
The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.
If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.
If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.
1. Academic ethics is the theoretical study of what you can get away with. (Law is the applied branch). Professors in ethics are not experts at being good, or knowing what is good, they're experts in how to get away with things. If you want an expert in being good, you need a saint, not an academic studying ethics. Saints are in short supply.
2. That said, ethics academics can still be decent. They don't have to be sanctimonious corporate shills . However, an "ethics expert" who hasn't threatened to quit and gone through with it at least once, should be assumed to be one.
I respect those who quit - or, at least, I respect them more. I wonder if it's more a reflection of the people, or of the state of the industry, that the list of (AI) ethics experts who have quit in protest is getting long.
I had the impression they were essentially synonyms, and this was borne out by two university courses in philosophy which I took. That said, googling "morals vs ethics" turns up quite an interesting mess. A variety of contrasting perspectives showed up:
- Ethics come from an external source, whereas morals come from within
- Ethics are rational and structured, whereas morals are more emotional
- Ethics is a philosophy or code of behavior, whereas morals refers to beliefs and conduct
- Ethics and morals are used interchangeably
It seems nobody agrees on what the words mean. Which, on reflection, tracks. For me, the idea of an HR training session on ethics is laughable, but for someone who thinks of the word differently it could certainly make sense.
Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.
We won't care about AI safety & ethics enough, atleast not enough to do anything about it.
People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.
Applies every-which-way; the chat bot, the provider, the consumer.
Said plainly: the previously-mentioned keys will absolutely be handed over. Nobody, not even your mother or her favorite LLM, is to be trusted to such degree.
edit: same problem and tactics we see with leadership at high levels in government.
you're too late
She probably just got a better offer somewhere else
I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.
My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.
That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".
It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.
But with two competing claims, and no easy way to check ground truth, you absolutely ask for citations from both sides. There's nothing rational about uncritically accepting the narrative you heard first.
Demanding evidence from one party while accepting the word of the other is not even-handed.
It's not. I have no idea how you got there.
The right thing is to seek evidence for both perspectives. This is ironically not rocket science. Don't accept any viewpoint just on say-so, but especially don't pretend to sneering objectivity when you do so.
> Don't accept any viewpoint just on say-so
Exactly. That's why they asked for a citation. A response providing such citation is more appropriate than essentially saying "well you didn't ask for a citation from $other_side either".
I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.
A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.
Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.
Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.
Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political leadership of this country whose support of Raytheon is fundamental to their survival
Lets do assume this is true, then again, wouldn't Raytheon be the best place to actually have a ethics department, if anyone?
They don't have anything to do about using the products they sell. Neither does Boeing, or Lockheed, or Tesla, or SpaceX.
The idea to bomb that girls school may have came from Grok, but a card carrying, uniform wearing member of the US military agreed it was the thing to do.
Where was the US military ethics team? Hegseth dismissed them earlier in the year to help with 'lethality'.
(in all reality, the intelligence was old. the database said the building was a military target, so it was sorted and assigned. intelligence is old and leaky and full or errors). a mistake. a tragic mistake, but not a deliberate act.
Most people don't like to do evil (even if the money is good) so allowing an economy of scale around morale reassurance is rational. This obviously breaks down if the company is seen to hide things from that department or if that department is clearly, themselves, unethical. The first of those conditions are why ethics departments at Meta and OpenAI have very little efficacy.
1. Outside of extreme circumstances where it has been mandated to be a specific regulatory board because of past actions or fear of future actions in which case it's better viewed as part of the bureaucracy of the governing body (usually the state, sometimes a shareholder or other concerned party).
> “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
Think I first saw that on a usenet post, as true now as then and just as true as it was in the 1930's when he wrote it (in at least that form).
This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.
The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.
It's hard to read anything into it.