Alix: [00:00:00] Hey there, I'm Alix Dunn, and welcome to Computer Says Maybe. This is our last episode of our series, Computer Says Kill, all about AI and militarization. Even though it's the last episode, I wanna be clear, the series doesn't stop here. You probably know that Computer Says Maybe is kind of a companion podcast for the Maybe Collective, our big, diverse network of friends and collaborators.

It's for basically anyone who wants to learn more and find paths toward action on technology politics issues. If you wanna learn more about what you might be able to do to help, we'll be hosting a community session in the coming few weeks, um, particularly on AI and militarization. If you aren't a member yet, head over to themaybe.org/join, and we'll keep you in the loop of community events and opportunities to get more involved in a wide variety of tech politics issues, but specifically as a follow-up from this series.

So if you've been listening to all these episodes, feeling a kind of heaviness [00:01:00] of these issues and don't know where to put it, um, that's kinda what motivated us to do this series, because we wanted to expand the circle of people that have gone down this rabbit hole so we can collaborate with more and more people on, I think, one of the most important issues of our time.

And when we started the series eight weeks ago, the relationship between AI companies like Anthropic or OpenAI or Palantir in the US military were kind of only just coming into public view. Our series took it a few layers deeper, beyond the headlines, to make sense of this history and the narrative myths and strategic plays that were made to recklessly expedite fundamentally unlawful AI systems into our military weapons and kill chains, mostly in the name of profit and power.

These last two months taught me a lot, and to be honest, it's made me even more enraged than I was when we started, and I'm just left struggling to wrap my head around how we arrived here. And it leaves me with a singular question, which we're hopefully partly going to answer in this episode, [00:02:00] which is, "What can we do?"

And as luck would have it, there are lots of friends of the show that are doing great work to get the world out of this mess, and in this episode, we're gonna talk to two of them. For this last episode, I'm joined by our first guest in the series and my friend Matt Mahmoudi, assistant professor in digital humanities at University of Cambridge and a researcher at Amnesty International, plus Marwa Fatafta, who is the MENA policy and advocacy director at Access Now.

Matt and Marwa played a huge role in shaping a joint statement on AI and warfare, which was just published and signed this week by over 250 people and organizations from civil society experts to lawyers to tech workers that are disgusted by where all of this is headed. And while the conversation might not have been the most hopeful, it was a really honest and necessary discussion about our roles, not just as professionals working in and against this corrupt industry, but as humans on the right side of history together in this fight.

So without further ado, my conversation with Matt and Marwa [00:03:00] about the joint statement released this week

Matt: I'm Matt Mahmoudi. I am a researcher and advisor on artificial intelligence and human rights at Amnesty's Technology and Human Rights program, as well as an assistant professor in digital humanities at the University of Cambridge.

Marwa: I'm Marwa Fatafta, and I work at Access Now as the Middle East and North Africa policy and advocacy director.

Alix: The two of you among a kind of tight-knit human rights and technology crew around the world have been crafting the statements to respond to what felt like a watershed moment a few months ago with the illegal invasion of Iran that led to the death of 168 people, um, at a school, and there was wild speculation 48, 72 [00:04:00] hours after that invasion began about the role of Anthropic and the role of large language models, these knowingly kind of crappy technologies being involved in making choices that led to a strike on a school.

And then, you know, what does that mean for the future of war when you have a rising authoritarian president in the US willing to basically copy Israel's playbook in countries around the world? I know we're in this kind of wild era of lack of enforcement, lack of accountability. But do you guys-- I, I feel like the first thing that would be nice to do, because there are so many details, but that I think are so critical and also these very, very bright lines, do you all want to walk us through what's in the statement specifically and maybe, like, I don't know, what are the top three or four points within it that you feel like capture the kind of spirit and, and focus of, of the statement?

Matt: So the statement in a nutshell is trying to get us all to call on tech companies and states to, and I'm quoting here, "Halt [00:05:00] the provision of AI systems for use in the military kill chain and to take all steps to ensure that other AI systems don't provide or contribute to violations of international law."

And by that we mean international humanitarian law, international human rights law. And to do that throughout the statement, we've basically pointed to some of the ways in which AI, and in particular generative AI models, have been used within the context of warfare, and also going back to some of the most gruesome instances in which AI was used for targeting and the perpetuation of, for example, international grave crimes against Palestinians.

So we point to this in, and I, I think I can broadly summarize this in four different points. One is, you know, the statement really is an attempt at, at warning against the sort of rapidly militarizing field of AI in general, and the ways in which these systems become embedded up in kill chains that allow for the acceleration of attacks, the undermining of [00:06:00] accountability, whilst also increasing the risk of international humanitarian law, international human rights law violations, and international crimes more broadly.

Um, the second point I'd say is that we argue here that there are no meaningful technical or procedural fixes right now in terms of bringing these AI systems in line with international law. So there's often going to be these retorts that people will have heard that, well, if there is just a human in the loop, or if we just provide more data into these systems, they'll be better.

But what we continue to fall back on is that The speed and scale and opacity, as well as the ongoing biases and, and questionable data sources that feed into this system make lawful and responsible use completely unrealistic and immaterial to the ways in which these systems function. The third point I'd say is that companies across the AI supply chain, not just on one [00:07:00] select portion of it, not just on the LLM, not just on the data aggregator side, you know, across the whole supply chain really have a duty to avoid causing, contributing to, or being directly linked to human rights abuses and international crime.

And what we're asking of them to do is just very basic, to respect the responsibilities under the UN Guiding Principles on Business and Human Rights, under the UNDP guidelines on human rights due diligence under conditions of heightened risk. And what that constitutes is effectively to exercise leverage.

US companies have the power to exercise leverage against actors that you're contracting with, where you can see that there are and where you know that there are human rights violations taking place. And so we're saying, you know, don't fulfill your contracts where you can see that these human rights risks that you've identified can't be mitigated, or even where you haven't been able to actually carry out a human rights [00:08:00] assessment because there wasn't an appropriate access to the data from your contracting party.

So we're saying companies have power, they know they have power, and they have a responsibility to exercise it. And the final point here is that we've looked at how our existing analysis of artificial intelligence systems that are used for international grave crimes, from large language models to target generation systems to biometric surveillance, cloud and data services, that these have already demonstrated such harm when being used by actors that are engaged in the commission of international crimes.

So we've seen, again, examples in Gaza, in the occupied Palestinian territory more broadly. We heard reporting of how AI targeting was being used in Iran. And we've seen biometric surveillance like facial recognition, emotion recognition, race detection lead to incredibly profoundly harmful outcomes. So if the component systems of these [00:09:00] larger systems are themselves wound up with such problematic human rights violations and violence, then the idea of having them put together into the kinds of applications that we're increasingly seeing in warfare, entire systems that build on these different tools, is something that we cannot and should not in any way entertain.

And so those are, those are the broad strokes of four sort of, I think, key points that the statement covers, and then we've got a number of demands that stem naturally from that structure.

Alix: I feel like, Marwa, do you wanna walk through the demands? I also I s-- in that last point you're making, Matt, I think a lot about Abebe Berhane's metaphor of you can't build a house with rotting wood, or you shouldn't.

Um, and there's something about this idea that you could build a tool that is as consequential as a weapon with technology that we all know just is not only an international human rights crime in and of itself, it's also kinda shitty. The thrust of this is not that we want [00:10:00] the technology to be better.

Um, uh, it's not that we want it to be more accurate. Um, it's that we don't want it to be used at all because it's essentially a mechanism by which the scale of war grows, the level of death grows, and that that in and of itself isn't acceptable. But I also, Marwa, would love to hear what you all are asking for different stakeholders to do in response to these conclusions that you draw on the statement.

Marwa: Yeah. I just wanna first, uh, comment on the remarks made by you and, and Matt, and also maybe share a, a bit of background what the trigger behind the statement was the Pentagon Anthropic drama, and also the interesting reactions that we've seen from the community, from civil society, and from some experts, and I think the statement also aims to debunk some of those myths that you guys discussed.

That one, indeed, those systems can function better. They are designed-- When AI is embedded in the kill chain, it is designed to accelerate [00:11:00] and escalate violence, so there's no way that you can insert a human in the loop or have better surveillance data sets to make targeting more lawful or within the bounds of, of international humanitarian law.

Also, the other myth that we saw with the Anthropic situation and the reprisal against it, and how some people celebrated Anthropic as, you know, the good guys in the story, but we see time and time again that companies don't have any leverage when they enter into military contracts with governments. And so all this talk in corporate human rights policies and public commitments about respecting human rights, about adhering to the UN Guiding Principles on Business and Human Rights, which tells companies that you have a responsibility to identify adverse human rights risks and mitigate them and remediate them when it comes to your services, policies, or [00:12:00] products.

And then when you are operating in a situation of armed conflict, that responsibility becomes even more heightened because there is a real risk of the company being complicit or contributing to or directly linked to international crimes such as genocide, war crimes, or crimes against humanity. And so the demands that we have in the statement, well, first, for companies, we're basically telling them that they should not basically enter into military contracts or contracts with armed groups full stop.

Because again, we know that when there's actual pressure, there's very little that these companies can do to mitigate human rights abuses or mitigate the risk of being involved or complicit in international crimes perpetrated by their government clients, and especially when those clients have a very established record of, uh, violating human rights law right, left, and center.

The second demand on companies is to also [00:13:00] refrain from selling, transferring, or exporting AI, uh, decision support systems for military kill chains and target acquisition or, or generation. We're saying that for lethal purposes, i.e., you're selling AI, LLMs, or cloud computing infrastructure for the purpose of identifying and tracking targets that are bombed and killed, and that should not happen full stop.

But then also for non-lethal purposes, and here companies can also get very, um, cheeky in terms of the transparency about how they sell or what they sell exactly and for what purposes. And we can see some technologies being sold as, you know, uh, civilian tech used for non-military purposes, such as LLMs used for translation or transcribing, where in reality we know that these have been used for also targeting, as we've seen in the case of Gaza, where Israel used Azure Cloud and the AI Suite in order to [00:14:00] store, transcribe, and also translate millions of hours of intercepted, recorded intercepted phone calls made by Palestinians in Gaza, which is then fed into their AI targeting systems to kill people And so we're trying to make that distinction, but then again, for the non-lethal purposes, those should not be sold or exported until there is genuine accountability, uh, meaningful human control, oversight, and transparency.

And then for states, and they have an obligation to abide by international law and to also ensure that the companies domiciled within their jurisdictions are also respecting and not violating international law, and there the demands are straightforward. The first is that they need to halt the use and export of AI embedded in the military kill chain or used in the conduct of targeting.

And then secondly, that there should be transparency about how AI is used in the conduct of hostilities. [00:15:00] Now, how those recommendations will be implemented, that's a long fight that not only civil society is involved in, but as you mentioned, also tech workers AI scholars, people who know these tools very well, also those of us who work with impacted communities, I think what the statement at the end tries to do is to bring lived reality of people and to put it at the center of why AI militarization needs to be stopped before, as you said, we live in this extremely dystopian world where people are killed by algorithms.

Alix: I wanna dig into this example, 'cause I remember I didn't know that LLMs were being used for translation kill list generation until you brought that to my attention, like a couple months ago when we were talking about this. I, I want people that maybe aren't following this as closely to, like, really understand the stakes of how this is being integrated, and this feels like one of the most relatable examples to someone who maybe has, like, dabbled in using one of these products.[00:16:00]

Like, we've all seen meeting transcriptions that transcribe a person or misattribute part of a conversation to another person. Like, I've seen, like, weird meeting transcripts where it's like, "This person said these things," and I'm like, "No, that person didn't say those things. This other person said those things."

It is really common that in transcription, massive errors are made by these technologies, and the idea, this is also adding a translation layer. So saying these conversations are happening in Arabic and then being translated into English by these technologies, and then that is summarized, and then that is generating kill lists, and then lethal force is used against people that are on these kill lists, and that the idea is that this is like an administrative technology.

It's not even like a lethal autonomous weaponry technology. It's just like, "Oh, we're being faster at office related work." And it's like, no, you're like using these technologies that we know are terrible in low risk settings in the highest risk setting imaginable. Sorry, I'm like now going on and on, but I like-- I, I wanted to [00:17:00] like dig a little bit more into that example because I feel like it so brings all of the stuff you're describing to the fore.

Marwa: Yeah, I mean, I'm, I'm glad you brought this up because also one thing that the statement is trying to do is to draw the line where it needs to be drawn. Because also with Anthropic's reaction, and generally speaking, the, the international consensus when it comes to AI in the military domain seems to be drawn at the lethal autonomous weapons.

When it comes to The use of LLMs, I don't know, like for translation, transcription, and God knows what else, I have a couple of points. One is that when you sell any technology, even if it's, I don't know, just a simple function technology to a government and to specifically to a ministry of defense that is engaged in military...

en-engaged in genocide as a matter of fact, then you know there's a high risk of abuse. So I think the onus here are [00:18:00] on the companies to ensure that their technologies are not being abused or linked to atrocity crimes, irrespective of what the technology is. And then in the context of Palestine specifically, I mean, it's not only people's everyday communications that has been datafied, pretty much everything, individual behavioral patterns, how they communicate, with who they communicate, but also social patterns.

I mean, if you look at how Israel used Lavender in order to identify if a person is a militant or not, and therefore adding them to the kill list and how they're killed when they enter, you know, they're monitored in real time, and they're... the bombs drop when they enter their family home. There is a, a datafication of Palestinians at a personal level, but also at a social and communal level.

And then what complicates things further is the fact that for the longest time, civil society have been telling tech companies, "You need to work better on [00:19:00] non-English language training for your AI systems," and, and so on and so forth. So one could imagine how terrible these translation or transcription technologies would be for not only Arabic language full stop, but like a Gazan dialect.

And there, I don't think Israel is particularly concerned about errors in translation. As a matter of fact, we know from The Guardian investigation that they actually used those intercepted phone calls in order to justify people's killing after the fact. So anything really that can justify their, the killing of people is good enough.

Whether it's true or not, accurate or not, that's another story. And finally, I also want to say that there is a political dimension here, and we've seen it not necessarily in the use of LLMs, but in other examples where that translation from Arabic to Hebrew, which I assume would be the language used here for translation It's [00:20:00] very interesting in terms of the choice of words and, uh, where, for instance, a particular word would be, um, like the word martyr, for instance, would be translated into terrorist.

So there are so many layers that can point to the fact of how dangerous these technologies are used or how dangerous they can be when they are deployed by a client known for violating international law and for the purpose of, uh, perpetrating, uh, international crimes.

Matt: This is in a lot of ways the bureaucratization of, of violence, and I think I, you know, gonna get into nerdy, nerdy mode and just quote Hannah Arendt real quick, who very spookily accurately said in On Violence, "In a fully developed bureaucracy," and I quote, "there is nobody left with whom one can argue, to whom one can present grievances, on whom the pressures of power can be exerted.

Bureaucracy is a form of government in which everybody is deprived of political freedom, of the power to act, for the rule by nobody is not no rule, and where all are [00:21:00] equally powerless, we have a tyranny without a tyrant." And I think this is, this is so accurate and i- in a way incredibly on point so early on, you know, shortly after the Holocaust in, in pointing out to the ways in which bureaucracy becomes the no-face exercise of violence in a way that becomes uncontestable.

And, you know, I'm veering slightly to the left field of my remit as a, as an amnesty human here today, but that is just to say I think that is what AI serve to do in many ways, right? In many instances, we're seeing these questions that are fundamentally about ethics, morals, human rights, uh, that are fundamentally about violence and the cost of violence, and that should rub us the wrong way, and that should be subject to human friction because it is a human friction that allows us to go, "No, this is where we draw the line."

But instead of that friction being felt upon human bodies and human minds and human souls, it's being increasingly automated and baked back into systems that make determinations without our involvement or with minimal [00:22:00] checklist, street-level bureaucrat levels of involvement. And I think that's where Hannah Arendt has really been spot on in predicting this AI moment.

Marwa: I have so many underlined, uh, so many, um, phrases on technology and violence in this book. It's an excellent, uh, reference indeed, so thanks a lot for bringing the

Alix: book. I mean, the modern human rights movement in a lot of ways came out of that time period as well. Um, and I'd love to, like, hear a little bit about...

Like, I get nervous that the current sort of network of human rights organizations that exist-- Like, I was so excited that you all did this because I think we need more Relevant pushback from experts that really understand these constraints that we've tried to build up as a society in response to these horrific atrocities, because we kind of nominally tried to reach some level of consensus about here's the boundaries of, you know, ethics and war, morality and [00:23:00] war, morality in terms of, like, how nation states should relate to individuals, how nation states should relate to each other, more contemporarily, how companies, um, should operate, you know, within, within these frameworks.

And it's felt like almost that the times are so big and moving so quickly that a lot of those levers that we spent decades trying to construct just feel so tiny in the face of everything that's happening. So do you all wanna talk a little bit about this approach of, like, does the world need another joint statement?

I think some people, some people could argue no, but I also think that, like, that's the primary way that the human rights movement expresses, you know, specificity of, like, where a line is and how we can translate a line in a particular moment, and, like, how to draw those lines. So, like, how do you all see the role of Access, the role of Amnesty, the role of other human rights organizations right now in terms of trying to bend this trajectory towards dystopia back to something that's more a society that we wanna live in?

Marwa: I'd say no, the world doesn't need another joint [00:24:00] statement, but what it needs is the right framing, because we need to really say things for what they are. And I mean, on a personal level, I share your fears, Alix, and I feel actually the world is just going on full speed without breaks into hell, and the international rules-based order, international law is being tested, and I do feel that we are at a make or break moment.

And yet sometimes it feels like those concerns that we see is being either dismissed or minimized. I mean, how is this not at the top of every state's agenda and also at the top of the UN agenda? In all discussions or most discussions around AI governance, when we talk about human rights, if we get to that stage, there are still no, no state that is, at least from what I see, is, that is willing to just discuss and unpack the use of AI in the military [00:25:00] domain And so this is exactly why I think it's important to have this statement and also to have over 200, I think we're at 226 signatories from civil society, human rights organizations, tech workers.

The statement aims to bring all these people that care about how these technologies are used and want to put this at the top of policy agenda for states and companies. Because even, you know, engaging with companies have also been simply dismissing civil society letters or requests for transparency, retaliating against their own tech workers' activism and calls for accountability and transparency.

So it's the start of a collective effort, in my opinion, to push the needle and to put pressure, more pressure on companies and states to halt the AI militarization, which is going at full speed.

Matt: I totally agree that human rights here serves as a sort of a lingua franca, right, through which we all communicate through our different walks of life, whether it's as tech [00:26:00] workers or as human rights researchers or as policy people, government folk.

I, I noticed just was it four or five days ago that a director for Android platform security, Rene Mayorhoffer, resigned at Google citing concerns related to the Pentagon deals with Google. And I think that becomes possible when human rights as a framework operates at the level of a lingua franca, where it both has a politicizing effect in the sense that it served to quote unquote radicalize groups into action and mobilize them into action, whilst also serving as a sort of a uncontestable legal analysis, right?

It's a legal analysis that can claim to be somewhat divorced from politics in the way that, that legal, you know, scholars would often argue. And in so doing, it provides a threshold of evidence that allows us to make determinations on certain realities related to certain products that can then be used by way of shorthand by everyone from, you know, the head of product [00:27:00] security at a tech company right down to a civil servant who is maybe trying to, to dissent.

And I think it's through engaging with that lingua franca that, that we unearth the hell that Marwa and, and Alix are both referring to here, which I think has been obfuscated by years of complacency. Like, under a complacent sort of veneer of liberal tech entrepreneurship for highly sophisticated complex problem-solving, we allowed so much to just go under the radar.

Under the veneer of voluntary, uh, agreements and sort of responsibility frameworks around AI development, so much went under the radar. It's not that we've suddenly ended up in a moment in which everything has gone from a zero to 100. It's the fact that, that this was building over a longer period of time.

We now have the masks off, on the one hand, because of certain developments in the sort of authoritarian practices being particularly in vogue at the moment. That's on [00:28:00] the one hand. The other hand is that we've now got a better language to be able to point these things out, that this was always about the concentration and consolidation of power.

And if it was about consolidation and concentration of power Then we need a way of being able to talk about those things which we now have, and we've seen the consequences of turning an eye on it. And I think the sort of what does a joint statement do? What does another joint statement do in addition to providing a lingua franca in this moment?

It also serves as a tool for posterity. I know this is not a satisfying answer. When tech workers can go back in particular courts, in future tribunals and otherwise and say, "We reached out to our company executives, and we said that we're concerned that their products are being used to commit human rights violations, perhaps even the crime of crimes, genocide, and they did nothing, and they responded nothing, and they were aware, and we now have a paper trail that shows that they've been made aware of the risks, and they didn't even commit to committing an assessment, a carry on an assessment, looking at the risks [00:29:00] involved in their products being used in particular contexts such as Palestine," that can be used.

That can be used in a court of law. It might not be now, it might not be in five years, it'll be in ten years, and on that basis alone, tech workers can go now and say, "Look, we don't wanna work for companies that end up being complicit in international grave crimes. We want you to be accountable now." So in that way, it serves this dual function of both being a lingua franca but also a tool for posterity that I think we need to cling onto as well.

Alix: Yeah. That feels right to me that it's, like, worth preserving and using, and I think the timeline of this is something that I keep coming back to, that, like, I want it to stop now. I want the people that have the audacity to basically drive our economy, our morality, our capability as a society off of a cliff, I want, I want them to be punished.

Like, I, and I, and I think that, like, you know, I, I thought the dust would settle and that [00:30:00] there would be accountability for people like Mark Zuckerberg, who knowingly sold ads to Myanmar generals to, uh, make it easier for them to essentially enable and drive a genocide against the Rohingya people in Myanmar.

It felt like a Rubicon was crossed when they knowingly sold ads to incite violence. The whole thing was just like an absolute perfect storm of human rights violation that, like, should have led to him being in jail. So fast-forward twelve years and you have this another crop of, like, sociopaths, and I, I want, I want something to happen to them, and I think that there's something about that arc that basically like this might be a hundred-year project.

And sometimes I get-- I feel really like hopeless and, and that we're kind of helpless in the face of the amount of power that's being used in these horrifying ways to sort of take us down this path that like I think any thinking person in the world knows this is a dark path. And if you laid the facts of this [00:31:00] statement on the table and walked someone through it that wasn't following this, they would be like, "Oh my God, that's horrifying.

Absolutely not." And I imagine that like the vast, vast majority of people in this world would understand that. That like because of that, that like this longer arc of commitment to these principles feels meaningful, and their inadequacy in the moment doesn't necessarily negate their value for, like, us as, like, a, as a society.

Marwa: Yeah. I, I just wanna say that norm building is a long and frustrating endeavor because you see the risks, you see the harms, and the desire for accountability and justice is also... I think it's a very dignifying human instinct. But again, to have states, to have companies act the way they should within the bounds of international law is another story.

So there's a lot of [00:32:00] handholding or even punching in order to get to that point, you know, where first you have to document the harm, you have to link the harm to the service, you have to link the harm to the companies, you have to do a lot of advocacy, a lot of engagement, create that paper trail that Matt was talking about to say, "Look, we have engaged, and here are the harms," in order, you know, for that light bulb to come out, and finally, states which are, at the end of the day, the main organs that make up international law to think, "Okay, we need to regulate AI in the military domain."

Because

the reason why we're here today is indeed decades of, um, lack of, of accountability and impunity, where companies feel pretty much free to think, "Okay, we've saturated the commercial market, so now let's look at the other lucrative market that is armed conflict." And I bet you that those tech executives are thinking extremely differently from the way we do when we [00:33:00] look at the state of the world.

We look at it with horror, seeing armed conflict and violence spreading across the world, where probably they see it as, uh, uh, business opportunities. And that's also what some of the Google tech workers said when they mentioned that Project Nimbus, for instance, with the Israeli government, was the blueprint to test whether these technologies can be used in other challenging national security contexts.

And in what better place can you test such cloud infrastructure and systems than Israel? And then, of course, we saw how Google dropped its or changed its AI principles to allow for the use of its technologies for surveillance and other purposes. And so it's business.

It generates a lot of profits, and as long as there are no clear regulations and accountability...

And I, I hope, and I kind of actually, uh, have a bit of faith that it won't take 100 years. It will take a few years, maybe a decade, maybe two. I think people [00:34:00] feel another red line has been crossed or a point of no going back There is the Myanmar, but I mean, there is a long list of egregious examples where tech companies have been complicit in, in horrible abuses and never, ever been held accountable for it.

And now entering armed conflict and genocide as the new, as a new market, I think there... Another line has been broken, and I think it's only a matter of time until civil society and even impacted communities will come knocking on their doors with lawsuits and, and other types of things. So I'm hopeful that it will take much shorter than, uh, than a century, but let's see.

Alix: I think you're right. I think also this sense of time has been warped. I don't know, this like intensification of our information environment. Like, I have deep faith that, like, a lot of these things will come back around to accountability. It will come back around to consequences. It will come back around to a rules-based order because the alternative is just, [00:35:00] like, unthinkable, and I don't think even the people that are arguing for it now actually want that reality.

So I, I guess, like, we've been talking a lot about what you're seeing, how it fits or doesn't fit in the human rights framework, very specifically, like, where these technologies should be walled off. A lot of that is what we don't want, and I'm wondering if there's like a... When you think about the human rights movement specifically or, or human rights law or like, if that space were, like, fit for purpose for this moment or if you were able to, like, build, like, the next generation architecture that would prevent these kinds of things from happening but also enable something else, like, what would you want?

Marwa: I mean, one thing I want is clear labeling and transparency when those technologies are being sold and exported for military use because, again, I think companies are absolutely benefiting from the, the increasingly blurring line between what's military and what's civilian. [00:36:00] And so you could say... And that was actually the, the first type of excuse that Google and Amazon gave when Project Nimbus came into being, saying, "Oh, we're not, we're not selling our technologies for sensitive military use."

And then they claim time and time again, and you can see if you type Project Nimbus and then Google spokesperson, you'll, you'll see the same press line repeated in every article, especially in the early years like 2021, 2022, that's saying, you know, we're selling basically technology for a government mostly for...

Well, they don't say mostly for civilian use, but they negate the use of the military use. And I think here, if a company wants to turn itself, a tech company wants to turn itself into an arms dealer, so be it, but then they need to be transparent about it. And then also I think once you have that transparency, then I think governments also need to come in with their expert controls and, and restrictions around how these technologies are being transferred and exported and then used.[00:37:00]

Maybe I should make a disclaimer that as much as I, I think those are at this point, like basic regulatory steps I don't have much faith in governments ensuring that they're, they're applied in a, in a international law binding manner, especially because we see how with actual weapons, governments are willing to break their own laws in order to give weapons to their own friends.

And so again, it becomes, like, an extremely politicized matter of who is selling what to who. But nevertheless, I think the-- this fog, or I should say this manufactured fog between what civilian and military needs to be clarified, and I think transparency is of paramount importance in terms of what we want to see in a positive sense.

Matt: I can come in here with a series of questions that I feel like are perhaps cop-outs on this. Uh, there's so much that we can do in human rights [00:38:00] terms which will often be framed as, as Marwa was saying as well, like, in terms of are there ways in which these tools can be transparent? But I think if we peel it back a little bit and go even broader, and I'm sort of cusping the tension between what a human rights organization is typically in the business of saying, so again, I'm gonna frame these questions.

What if technology companies built tools that at every step of the way were considered public goods? What if they weren't private? What if they were developed on the basis of mass surveillance, but on the basis of mass service? What if they weren't tax exempt, but actually helped build up revenues that could be used in ways that redistributed wealth at a time in which so many communities are falling short of being able to meet their basic needs?

There are so many ways in which tech company violations of international law and their commitment and contributions to international grave crimes are wound up with siphoning out the material wellbeing [00:39:00] of communities everywhere. It's not just what happens at the end of the weapon, but what needs to happen redistributionally in order for that weapon to be produced in the first place.

And so just thinking back to these questions, by being able to ask what are the futures that we're foregoing in giving them such power and such private power, and by assuming that they must be built in particular ways on the basis of, again, mass surveillance, we're giving them free roam to warp and bring into materiality a future that is in no way democratic, that is in no way in line with what we might want-- where we might want our imaginations to stretch.

So I'm gonna leave us with, with these questions instead for us to, to prompt what a human rights centric approach to tech actors might look like.

Alix: So you all spend a lot of time thinking about, writing about, mobilizing around the horrifying moral trajectory of basically all monopoly technologies that everyone around us uses constantly, including ourselves in some cases.[00:40:00]

There's this new program as an example where Anthropic is gonna pay A thousand full-time salaries-ish for, like, 1,000 people to go embed in nonprofit organizations who are, like, AI curious or, like, wanna work on integrating LLMs into parts of their operation. There's a normalization that happens and also a cognitive dissonance of, like, having very strong feelings about these companies while also being people in a world captured by monopolies that, like, you have to engage with these technologies, and you also sometimes have to partner with organizations that are dabbling and, like, don't have as clear a line or aren't as comfortable asserting, "Let's have our operational practices and technology choices align with our morals," but instead separate those two things that, like, operational maturity is, like, distinct from how we think about what is right and what is wrong in, in the technology industry per se.

How do you all, like, [00:41:00] think about that? And how would you encourage other people in nonprofit spaces to sort of deal with that duality of, like, hating in a lot of ways these companies and at the same time touching their products day in, day out?

Marwa: To build on Matt's power of reimagination, his last remark, I mean, one thing about AI where I feel we're being so conditioned to accept that AI deployment is inevitable, and it's the new reality that we have to reckon with and just accept, and that also includes civil society organizations where AI is here, so we might as well just start incorporating it or using it.

There are, of course, several questions, such as at what cost? Also, what are we normalizing in our spaces? And then also, what whitewashing operations are we participating in, thinking about this Anthropic, uh, fellowship or the payment of Civil society [00:42:00] staffers, I mean, coming to them at this point wh- where you know that the human rights community is struggling and is being choked financially to say, "Hey, here is money, but you need to use our AI."

I mean, this is a, just a plain marketing scheme. I would expect civil society to be smarter than that. At the end of the day, it's not like you making a company an enemy just for the sake of it. You want to engage with companies so they can correct their conduct. But absent that, I think we have an expression in Arabic, which I really like.

It's like you slap someone, and then you fix their cap, and I feel like they're-- we've been playing that game with s- with companies where, you know, we slap them with strong-worded letters and then at the same time, you know, we welcome them into our spaces. There is tension there, but I think sometimes it's an artificial one, sometimes it's, it's a, a necessary one that we have to recognize, and [00:43:00] also it's very context specific.

What I want to say is that, you know, there is a difference between engaging with Anthropic to make sure that they don't engage in military contracts, and there is a difference between accepting their money so you can use their AI. I hope civil society leaders are-- can see into that scheme and, um, not contribute knowingly or unknowingly in the normalization of right-abusing companies.

Similar to what the, uh, UN is doing, by the way. I mean, maybe Matt can talk about the effort we did last year on a personal level with, uh, the UN AI for Good summit, where you have Google and Amazon and, and Microsoft and Palantir, like all the, the companies that are possibly, you know, complicit in genocide are being platformed as a force for good.

And that cognitive dissonance sometimes really drives me mad in where if you make those arguments, you're simply dismissed as the [00:44:00] non-pragmatic actor. But that pragmatism or the pressure to be pragmatic and to let go of taking principled position is, I also think, why we got here in the first place.

That's all part of the of the normalization and, and having this ecosystem where different actors contribute in building it in different ways. I think we need to do a lot of, might sound harsh, but a bit of soul-searching, especially at these extremely, uh, trying times.

Matt: At every turn of the way, we have to understand that our role within the human rights community and civil society organizations should be to make the development of these products and the particular applications that are wound up with abuses of international law costly.

It is our role to make that costly. That, that should be our one line that we just adopt across the movement, and we continue to bring that point home. And, and we can do that in a number of different ways. One is who are we kidding ourselves when we go out and we decide that we should [00:45:00] get on the AI train, and the first thing that comes to mind is either to engage with OpenAI and Anthropic when there has been for decades a very vibrant free-libre open source software community who are probably very happy to come and build infrastructure for us, who have possibly already built infrastructure.

Like, why aren't we doing what we expect other institutions to do when it comes to procurement? And go out and look far wider than the immediate names that come to mind when it comes to the hype train, right? Yes, of course, the Free Libre Open Source software community has been gutted, but not in small part, of course, due to some of the funding problems and cutting that we've seen across the board in the last couple of years.

But, but also not in small part due to our lack of engagement with them and, and building infrastructure with them, wider ecosystems that are human rights enabling, which we've seen already do enormous things, right? Like the kinds of evidence preservation systems that are often [00:46:00] used in crisis response programs across some of the premier human rights organizations are developed by Free Libre Open Source software community members.

You know, one way in which we make it costly and stop larger tech companies from deriving off of, uh, the benefit that we provide them with our symbolic engagement with them by rubber-stamping them is by going elsewhere. We can also exercise leverage where we have preexisting commitments and, and contracts with tech companies.

I understand that a number of human rights organizations on the back of the revelations around Microsoft Azure processing Palestinian data for AI targeting are seriously reconsidering internally how they might develop, uh, independence from Microsoft and companies like it. And this is, this is made possible in large part because of the noise that's being generated on the back of the Azure revelations.

Without that, again, without that usage of the lingua franca of, of human rights to engage on and create a bit of a, a narrative shit storm [00:47:00] around Azure, we wouldn't necessarily be here, but we are, and that's costly. And another thing that, that we could do, and this is slightly, you know, this is hinged more on an existential question, is once and for all at this critical juncture, figuring out as human rights organizations and civil society organizations at large, whether we see ourselves as professionalized entities whose remits are very specific to a watchdog function that doesn't go beyond the original remit as described when said organization was founded however many decades ago, or whether we see ourselves as engaged in a struggle, and in a struggle that requires us to also think differently about the methods that we use.

And if, for example, that means that we lose out on some funding But it means that ultimately our labels and our-- the symbolic capital that we provide [00:48:00] can't be utilized in ways that rubber stamp these companies. And if it means that we can build a more conscious, intentionally connected, wider broad tent of tech workers, of human rights organizations, of policy folk, of what have you, who refuse to engage with those companies through using their products, then I also think we, we, we help chip away at how powerful and how broadly appealing those products are in the first place, because it becomes clear that they're speaking to themselves in some ways.

Those are conversations that I really do think are worth having We talked before about how short or long this fight might be. I love that Marwa was, was mentioning, you know, it might just be a couple of decades. How do we know? It's okay to sit in that discomfort of not knowing when it might be. Part of the just-- the paraphernalia around this moment, the weird, weird paraphernalia around this moment is also, I think, helping just elicit that sense of surrealism and the sense that something is, is off, and we [00:49:00] ought to shift gear.

And I'll stop there.

Alix: Thank you both so much, both for this conversation and also all the work that you did to make the statement come to life, and also for everything you're doing to kind of help the human rights movement be what everyone needs, um, in this very complicated time. Thank

Matt: you, Alix. Thank you, Alix.

Thanks for hosting these critical conversations.

Marwa: Can I make one thank you, a huge thank you, and this should go on record. A huge thank you to my colleague, Daniel Loifer, who has been the-- So he deserves a lot of applause and recognition for taking us to the finish line, because it's been, it's been a few months.

And so, so many thanks, Daniel, for taking care of this.

Matt: Huge thanks. And I think Daniel has now, like, an unprecedented overview of the internal policy lines of just about every human rights organization- ... and how they conflict or- ... are spurred on by one another.

Alix: Oh. Um, yes. Thank you,

Marwa: Daniel. Um, and also has kept good humor throughout, um, [00:50:00] which I feel like is not an easy- Yeah

easy thing to do.

Alix: If you're as moved by this conversation as I am and want to learn more about how to get involved, sign up for free to join the Maybe Collective in the show notes. As part of the Maybe Collective, you'll get access to member events, a community of like-minded critical progressives in the tech space, and our monthly roundup, where I'll be sharing my kind of no-holds-barred hot takes from time to time, along with resources from orgs and individuals that are doing great work.

You'll also find out every month about what's coming up in terms of events and what came last month. So join the Maybe Collective if you don't just want a podcast, you want a tech politics home. Thank you to Matt and Marwa for being fellow travelers navigating these dark times, and next week, we're in for a dose of more hope and more expertise that I find really calming and interesting for all kinds of issues.

Next week, we're gonna be releasing the first episode in another short series made in collaboration with Real ML. And for those of you who don't know about Real ML, it's an [00:51:00] organization that seeks to build movements by connecting the researchers, journalists, and activists who are looking to prevent the tech industry's most egregious We'll say outputs around the world.

Um, this collection of special episodes will bring together groups of practitioners who've participated in Real ML's real-life workshops, um, to share the impacts that their resistance work is having in their respective home contexts. So be on the lookout for that next week. And as always, thank you to our production team who worked to put this series together and the Real ML series together: Sarah Myles, Georgia Iacovou, Kushal Dev, Marion Wellington, Van Newman, and Zoe Trout

‍