Open Government in Action: Emerging Practices in Participatory Algorithm Design - July 29, 2024
>> Good afternoon, and welcome to today's session at the Open Government in Action: Emerging Practices and Participatory Algorithmic Design. This session will be open to the public, and is being recorded. A copy of who attended the event, as well as the agenda notes, will all be posted publicly on the open.usa.gov website. My name is Daniel York, and I serve as the Director of the U.S. Open Government Secretariat, housed under the U.S. General Services Administration.
The United States established Secretariat in September of 2023, so almost a year ago, so lead and oversee the domestic implementation of our membership in the global open government partnership, a voluntary global alliance between governments and civil society to bolster democracy through openness, transparency, and public engagement. The U.S. Secretariat leads the domestic OGP processes and development of the U.S. Open Government National Action Plans, conducts, engage in sessions like this here today with U.S. civil society members, as well as members of federal, state, and local governments, and actively collaborates with the American public on all Open Government themes. In this vein, we are very pleased to be co hosting today's event with the White House's Office of Science Technology Policy and the U.S. tech policy network as a part of our ongoing public series
on advancing Open Government government, including public engagement participation across the United States. We are very grateful to have our expert panelists and all of you joining us here today. And a reminder, the session is being recorded. And, again, will be made available. With that, I'd like to turn it over to Jennifer Anderson Lewis, who serves as the Senior Advisor for the Open Government and Tech Policy at the White House, and who leads efforts on mainstream Open Government principles at federal, state, local, tribal levels. Jen, over to you.
>> Thank you so much, Dan. And a special thank you to our friends at the U.S. Open Government Secretariat, for hosting today's virtual platform.
I'm really thrilled to welcome you today, and to serve as the overall moderator for today's session, which seeks to unpack the emerging practice of participatory algorithmic design, where the use of public participation and community engagement in the scoping, designing, and implementation of public sector algorithms. For those of you in the tech and data space, including our peers and our innovators at the state and local level, you know well the significant, the significance and the promise of machine learning and algorithmic tools in delivering for your communities. These are tools that are increasingly finding their way into all aspects of American life, from the ways in which students are directed to schools, to the ways in which health services are distributed. AI increasingly has the power to transform the way our public needs are identified, and the ways in which they are delivered. For those of you in the Open Government space, you know well that participatory and inclusive technological tools are required to not just ensure effective public service delivery, but to bolster democratic dividends, primarily community trust. We will begin today's session with a technical exchange, highlighting some pioneering efforts and experiences by leaders at the city, country, and federal level, in implementing participatory algorithmic design.
We encourage you to listen and learn, as we are doing at the federal level, and engage in the Q&A that will follow the exchange. We will then move into a moderated panel to discuss and explore evidence, research, and emerging practices, both nationally and globally, in participatory algorithm design. And finally, today's session will conclude with remarks and reflections by the White House OSTP's Principal Deputy Chief Technology Officer, Deirdre Mulligan. We are thrilled to be hosting this stellar line up of practitioners and experts for today's event, which represent a diversity of regions, levels of government, and institutions across this country. We are also so pleased to be joined by our partners at the Global Open Government Partnership to share comparative experiences across countries.
We really hope you enjoy today's event. And so without further ado, I'd like to pass this over to my colleague and co founder of the U.S. Tech Policy Network, Shannon Arvizu, who also serves as Senior Advisor to the Chief Data Officer at the U.S. Department of Commerce. Shannon, over to you. >> Thanks so much, Jen. And allow me to add my welcome on behalf of the U.S. Tech Policy Network,
which was launched earlier this year, by GSA and White House OSTP to provide a durable conduit between federal, state, local, tribal, and territorial levels on tech and data policy. I'm pleased to serve as a moderator of this technical exchange, which features groundbreaking efforts and participatory algorithmic design across the cities of San Jose, California, and San Antonio, Texas, from Allegheny County, Pennsylvania, and from the U.S. Census Bureau. We'll start with some brief presentations from each of our panelists, and then move to a moderated Q&A.
And I'm pleased to be joined today by Chelsea Palacio, who is the Public Information Officer for the City of San Jose, California. And in this role, she develops, designs, and implements strategic communications, outreach, and community stakeholder engagement plans for city IT programs in four languages, including 311 services and responsible utilization of artificial intelligence. Our second presentation today will be by Erin Dalton, who is the Director of the Allegheny County Department of Human Services, which works to strengthen families and communities through a network of social services, care, and support. Also joining us is Emily Royall, who services as the Smart City Administrator at the Information Technology Services Department at the City of San Antonio, Texas, where she manages the [inaudible] framework to shape the city's investments in emerging technology for delivering public services. Emily also serves as the Chair of the Cooperative Purchasing Committee of the Gov AI Coalition, and as Vice Chair for United and Smart, for United for Smart and Sustainable Communities, a joint initiative of the UN Habitat and ITU.
And last, but not least, we are joined by Michael Haas, who serves as the Senior Advisor for Data Access and Privacy for the U.S. Census Bureau, where he is responsible for outreach and engagement with the Census Bureau's data users on issues relating to the impact of privacy protection methodologies on the accessibility and usability of Census data. Welcome to you all. And with that, I will hand this over to Chelsea. >> Thank you, Shannon.
So, again, I am Chelsea Palacio, and I am the Public Information Officer for the City of San Jose IT Department. So, participatory algorithmic design, it's a technical way to say that the public is not only engaged, but involved in the process of design and implementation of new technologies. The City of San Jose has been meaning to integrate this idea into how we introduce new technologies for city services like AI based [inaudible] technologies. As we implement these technologies, including automated license plate readers, or ALPRs, and AI gunshot detection, initially public perception was not ideal. Initial perceptions refer to the city as a big brother, that the city is always watching, and AI is about privacy nightmare.
The city saw a critical need to expand public involvement in order to proceed with AI technologies in a responsible way. And so far, we have, we are seeing a more collaborative relationship with the public. As San Jose continues to establishing, to establish working partnerships with the community, residents are actively engaging to inform direction of the city's AI implementation, like asking the city to install more ALPRs to make neighborhoods safer, and how the city should use AI to make other city services more efficient. For participatory algorithmic design, public engagement is key.
It is important to open up the process so the community can be involved at the level they are most comfortable speaking to, whether it be sharing technical feedback on algorithmic design, or more general feedback on how the city uses technology. While implementing ALPRs and gunshot detection, San Jose hosted public meetings in our project of communities, which represent underserved areas such as crime, light, and violence. Meetings represented in multiple languages to ensure that residents can speak freely and provide input and direction without language barriers. So, you increase accountability and transparency. The city publishes feedback, as well as our data usage [inaudible], data usage reports, on our website.
San Jose is also exploring AI object detection, where we are actively engaging our communities to guide the scope on how the technology is being trained and potentially using our city services. Engagement has already provided direction on what objects the city will train AI to detect, as well as using that feedback, which has so far shifted the focus of our pilot to specific road conditions and hazards, like potholes and illegal dumping, to increase safety of motorists and pedestrians on our streets. Next slide, please. So, here you see a little bit of a before and after of our exploring of participatory algorithmic design. In the media, as well as from public engagement, we see people are confused with why are we using technology if they haven't done anything yet, the privacy nightmare quotes, as well as when it comes to after, people are fighting to actually have these devices in their neighborhoods. Next slide.
So, participatory algorithmic design is creating a positive cultural shift of how government works, planning public engagement from the start, ensuring meetings are communicative with enough notice for residents to attend, reducing language and accessibility barriers, and providing follow up to our residents who encourage ongoing engagement. This proactive approach is increasing trust and transparency in city, in the City of San Jose. The focus on public engagement promotes all voices to be heard from the start, resulting in staff and residents strengthening our city together. Next slide, please. So, this completes my case study.
And I'm excited to pass it onto Erin Dalton from Allegheny County. >> Great. Sorry. Thank you so much. I'm so glad to be here. Thank you for having me. So, just a little bit about the Department of Human Services.
We're responsible for child protection amongst a bunch of other important topics in human services. And so we have responsibility to work with and support our most vulnerable residents. Next slide, please. And we have amazing, integrated data.
So, just wanted to make sure folks were aware of that as we began. Next slide. So, I wanted to focus on Hello Baby as the use case for this work. And so we didn't set out to do the best participatory design of an algorithm, you know, in the world.
What we set out to do is to solve a very real human services problem. And you can see that on the screen, right? So, in over half of the cases where a child died or nearly died as a result of abuse or neglect in our county, there had not been a child welfare referral prior to that critical incident. That meant that we had really no opportunity to help support the family prior to the incident, and that's what we wanted to do. Next slide.
And it's not unique to Allegheny County. Across the country and the world, we're really trying to flip the paradigm on prevention, really prevent abuse and neglect, and support families early. But primarily what researchers and practitioners around the world have been paying attention to was what are those interventions? What do you give people to help bend the curve, whether that's nurse family partnership, or other home visiting, or direct aid? But we really sought to pay attention to first and foremost, who do you give that too, right? So, if we, if we can acknowledge that an identical program that is delivered to parents with double the risks has twice the effect, then you really need to be able to target and identify early those that have the highest adversities. And traditional programs just don't do a very good job at identifying those highest need families.
They serve people all over the map, not just the highest risk. Next slide, please. So, we, with partners, set out to build a model that did a great job of identifying those most vulnerable. Here are some of the factors and things that we predicted, you know, really using the language of adverse childhood experiences. Next slide. And we did what is relatively the easy thing, right? We built a model that could identify out of all babies born in Allegheny County, who is the most vulnerable? And then we sought to provide supports not just to the highest risk, to all families, but to tier those supports to those at moderate and high risk to use our resources for those most vulnerable.
So, you can see, you know, we have 23 times the likelihood of a home removal, and 10 times the likelihood of experiencing a postneonatal infant mortality. Next slide. So, that's Hello Baby.
Right? So, we built a better model to identify people to families to reach out to and offer those kind of tiered supports that I mentioned ever so briefly. What this means is that we had to get community consent to risk score every single baby born in Allegheny County, whether that's my child or any other child. Next slide. So, really if you want to be able to do, I don't know, at least on my screen, that's not coming out so great, but if you want to be able to do this sort of work, you've got to have a strong process in place and be sort of granted social license.
So, we, you know, we've been at this for a long time. We did the kinds of things on this screen, including funding independent ethical review, getting stakeholder input, competitive procurement, strong evaluation. But we did a number of other very special things to kind of get us social approval, social license to proceed.
Amongst those, and I'll just be brief, one of the most important, I think, was doing case reviews with families and with practitioners who were given the deidentified examples of people who were the highest need. Right? And everyone really agreed across those scenarios that we really should intervene with people, and they gave us insights into what would be helpful. We also partnered with researchers, giving them access to our community, to ask our family members and people in community what was most important to them.
And no surprise, a lot of what they told those researchers and us was that it mattered more what we gave to people and how we approached people than maybe how we got to their door. And then I'll just say the last, last two things. One, I think it is really important for government officials to have the big meeting, right? It's difficult to stand in front of 100 people plus and make a state for why this is important.
I think that's critical. And then I think it's also important that we, we're willing to make changes, and we make changes both to the algorithm and to the intervention. All right, that's a very brief bit about Hello Baby. I'm going to turn it over to Emily Royall from San Antonio.
>> Great. Thank you, Erin. Hi, everybody. I'm Emily Royall. I am the Smart City's Administrator for the City of San Antonio.
I manage our Smart City's program and division. Next slide, please. So, I want to flag that when I was asked by Jen Lewis to identify a case study for participatory algorithms, I looked closely at some of the projects we've executed under our Smart City's roadmap and our Smarter Together initiative.
We prototype and test emerging technologies and evaluate them for their capabilities to address business needs for the City of San Antonio. And we do that through a community driven, people centered framework. You can visit smartertogetheressay.com to check out all the awesome work under the Smart City's roadmap, this project being one of them. Next slide, please.
So, the business case for this problem, and we always set out our business cases first, rather than looking at specific technologies to pilot, we had a serious issue with communications around construction related to various bond packages that we had introduced in 217 that were being carried out through 2024. We had several small businesses that were impacted across corridors. As you can see here, right, that we're under major construction. And we really lacked an easy 24/7 updated way and method to communicate with our residents and small businessowners, and our visitors to some of these major corridors about construction, road closures, and really provide that real time information about upcoming road closures, and challenges they may experience navigating this particular area of town. Those pain points increased over time. As you can see, we started getting some negative Twitter reactions to the lack of communication about construction in downtown San Antonio.
Next slide, please. So, to fix this problem, we introduced an AI powered chat bot. And we partnered with a company called Hello Lamp Post. And really what the chat bot is intended to do is provide interactive up to date 24/7 information to people visiting our Broadway corridor to residents that were interested in the bond project, and also to our businessowners that were impacted by those bond projects. We identified about 120 businesses that would be impacted by the use of this tool.
And we provided signage at about 40 locations along this corridor as part of a pilot to test and evaluate the capabilities of AI to provide this real time information to the public. Next slide, please. The way the system works is that through either digital means or by scanning a QR code that is physically located at the area of construction or in small businesses themselves.
You can scan that code and access the chat bot through your SMS or through an actual mobile app that you don't have to download, it's live in the browser, and begin talking. And the AI will respond and answer your questions accordingly. This was huge, because previously we had in person meetings that were provided by our public works department for small businesses. We got counts of maybe eight to ten people joining those meetings weekly in person. And we were able to increase that traction of communication and reach to about 829 conversations over time, and resulting in about 3,000, almost 3,500 messages. Next slide, please.
Now, while there's a lot of efficiency and excitement around being able to provide these kinds of services with tools like AI, there are some significant drawbacks. In our initial release of this tool, the system hallucinated. And in one particular case, a user asked, where do I go if I have any questions about upcoming construction? And the AI actually responded, you should call our mayor directly and scan the live web to provide information about how to get in touch with our mayor. That is not the kind of solution we were looking for. And so we realized really early on that we needed to prevent the AI from accessing the web directly. But more importantly, we needed to dive much deeper into prompt engineering in order to set those boundaries and restrictions around how the AI could respond and provide those accurate responses to our residents.
Next slide, please. So, rather than shutting down the project and retracting the project, we instead pivoted towards what we would call a public prompt engineering model. And this is where we essentially invited the public to test the AI system with us. We turned off the AI tool. We created a sandbox environment for testing. And then we brought that AI chat bot to public events.
We invited the public to come on site. We activated local businesses along our Broadway corridor to be the site of public testing around artificial intelligence technology. And we identified all kinds of nuances to really help us redesign this chat bot to be more responsive to the needs of our residents and more accurate. Next slide, please. And in my final slide, I just want to highlight, you know, some key take aways for the consideration of the group today.
First of all, I think there is a misunderstanding about how cities are developing these AI systems. Typically, we're not building them in house. Rather, we're procuring them and partnering with vendors in order to purchase these technologies and deliver them as public services. As a consequence of that, it's very important for us to have vendors that are willing to open the black box, if you will, and allow us to partner with them and lab rate with them, by looking under the hood of these technologies, and being transparent with us about how these AI systems work and are engineers so that we can inform and influence them to be more responsive to our unique conditions on the ground. I also want to flag that prompt engineering is really critical to developing services that are responsive and realistic to the unique context in which they operate.
And that requires a lot of investment and time on behalf of cities to be able to provide that subject matter expertise in order to do that. And finally, you know, something we learned from this project is that that public testing is critical to build these robust services, certainly if they're going to be invested in the long term. But I do believe that that should occur regardless of the technology that we're exploring. So, in this case, it is artificial intelligence.
But arguably any emerging technology is one in which we should integrate public feedback and perspective into its development. Thank you very much. And I will pass that onto Michael Haas with the U.S. Census Bureau. >> Thank you so much, Emily. And good afternoon, everyone. I'm Michael Haas.
I'm a senior statistician for scientific communications at the U.S. Census Bureau. And I'm going to speak to you briefly about the critical role that stakeholder participation and engagement has in the design and implementation of the algorithms behind the 2020 Census Disclosure Avoidance System, the system that we use to protect the confidentiality of 2020 Census data, how we developed our statistical products. Next slide, please. So, the U.S. Census Bureau, like our peer statistical agencies around the world, have a dual mandate to produce quality statistics about the nation's people and economy, while also protecting the confidentiality of the information entrusted to us by those who fill out our censuses and surveys. Now, unfortunately, that kind of dual mandate is made rather challenging by what I call the triple trade off of official statistics.
And this kind of essentially boils down to the fact that the more statistics you publish, and the greater the granularity and accuracy of those statistics that you're publishing, the greater the disclosure of it, the greater the confidentiality threat of those statistics. And any statistical technique that you could use to protect confidentiality in the data that you want to publish and release, imposes a fundamental trade off between the degree of data protection on the one hand, and the resulting availability and accuracy of the statistics that you want to release on the others. So, you have this three dimensional trade off here. Next slide, please. And with these kind of three finite commodities, the protection of the data, the accuracy of the data, and the quantity of the data that you want to release, you can maximize on any two of those three dimensions, but only at a really profound cost to the third. And so there are some rather difficult choices and decisions that agencies have to make when kind of deciding, okay, how much protection is enough? How much accuracy is enough? How many statistics do I want to be releasing? Because these are all three interrelated.
Next slide, please. Now, as we were kind of approaching the 2020 Census, we realized that the confidentiality protections that we had been using for the prior Census, particularly the mechanisms that we used from 1990 until 2010, were becoming increasingly insufficient to counter the much more kind of vexing confidentiality threats of today with the proliferation of third party data and with much more powerful machine learning algorithms that could leverage that third party data in an act on what we were seeking to publish. So, we decided to develop a new approach for disclosure avoidance for the 2020 Census.
And the system that we built is kind of predicated on this mathematical framework known as differential privacy. Differential privacy is kind of like a privacy risk accounting system, if you will. And the way this works is like every individual that's reflected in a particular statistic contributes towards that statistic value. And every statistic that you publish is going to reveal or leak a small amount of private information in the process.
Differential privacy has this kind of accounting, this risk accounting framework, allows you to assess each individual's contribution to each statistic. And then through noise infusion to essentially limit how much private information about them will weaken the process. Next slide. Now, all disclosure avoidance methods, as I mentioned before, and the parameters of their implementation, impact the resulting fitness for use of the data in very different ways.
An agency has to be very careful and deliberate in their selection and implementation of disclosure avoidance methods to ensure that the resulting statistical products that we're releasing meet the needs of our data users, meet those priority use cases. And that requires subject matter expertise. It requires substantial research and evaluation.
But most importantly, and most importantly for today's conversation, it requires stakeholder communication and engagement. Next slide, please. So, in the context of the 2020 Census Disclosure Avoidance System, one of the primary algorithms that we use to protect confidentiality in our data is known as the top down algorithm. And I'm not going to go into the technical details of that. I can spend an entire afternoon doing that.
But I do have a link here to a technical paper, if you're interested. But I'm going to highlight there's one stage in the middle of the algorithm that's called the noisy measurement stage. And this is, this is really the heart of the confidentiality protections themselves. And during that [inaudible] measurement stage, we're essentially taking the confidential 2020 Census data and we're asking a whole lot of questions about it. We're saying, how many people with this combination of characteristics live in this particular area? How many people with that combination of characteristics live in that area? And we're asking this trillions and trillions of times. And to each result of those questions that we're asking, we are injecting, we're using a little bit of statistical noise or uncertainty into the results.
So, if the actual confidential count was three people have this combination of characteristics, as in a particular Census block, well, the result that you would get might be three, it might be two, it might be four. But we're asking this trillions of times. And that is how we're protecting confidentiality.
Next slide, please. The issue is I mentioned differential privacy is this accounting framework. Well, the parameters of how you design and implement your algorithms here are going to determine that triple trade off that I mentioned before. And those parameters, the kind of dials of kind of your implementation about the site and how much protection, how much accuracy, is all governed in the context of differential privacy through a parameter known as your privacy loss budget.
Your privacy loss budget kind of governs at the macro level, just how accurate and just how protected the resulting statistics are going to be. And then much like a, much like a monetary budget, your privacy loss budget you allocate to different queries that you're asking. The more privacy loss budget you allocate to a particular query, the more accurate that resulting statistic is going to be. The less privacy loss budget you allocate to that query, the more protected the statistic will be, but the more noisy, the more error will be built into that statistic. And there's no right or wrong set of allocations.
Essentially, you could allocate those privacy loss budget shares in very different ways, and you would get very different kind of resulting fitness for you. So, you might be privileging certain use cases over others, or making certain statistics more accurate at the cost of accuracy for others. And this was where kind of stakeholder engagement and participation in our decision making and design was critically important. The over, the four years between 2019 and 2023, we at the Census Bureau released eight sets of what we call demonstration data, where we essentially ran 2010 Census data through various iterations of our algorithms. And then allowed people to kind of play with the resulting protected statistics to see would these meet my use cases? Would these meet my needs if this was done with the 2020 Census data? And after each round, we got lots of feedback back from the public, and we used that feedback to tweak these allocations to improve the design of our algorithms. And the result was the parameter settings and algorithmic design that we used to actually protect the 2020 Census.
And with that, I believe I am at time. So, next slide, and I'll hand things back to Shannon. Thank you so very much. >> Thank you all. And I'd like to invite all of the panelists to bring their video back on. And while they're doing that, I'd also like to invite folks listening in.
Please send in your questions in the chat. We'll be turning our attention to audience generated questions in just a moment. But before then, I wanted to ask each of our panelists, based on your experience, what worked best so far? And what might you do differently in the future? So, Emily, I see you are here, present, so I'll pass it on over to you first. >> Sure. Thanks, Shannon. Well, I think, you know, for the City of San Antonio, you know, what's worked best for us is probably a confluence of a couple factors when it comes to AI and algorithmic projects in particular. One certainly is the integration of the public perspective, especially for anything public facing, you know, to elaborate a little bit on lessons learned from our public testing, we discovered things like, you know, our Spanish users, for example, had to navigate a couple English questions before they were able to indicate that they wanted the conversation to take place in Spanish.
And some of these basic UX types of information. But secondly, you know, really having a vendor that was able and willing to be transparent with us about the process of prompt engineering. So, the vendor that we worked with, Hello Lamp Post, was very open to us providing feedback in terms of what we were hearing through public testing, and collaborate together to really reengineer the algorithm to respond accordingly on the back end.
That process was critical to the success of this project in particular. And I believe the outcome was a chat bot that really was more responsive, and the opportunity for us to be able to receive all of the data on the back end of that to discover and uncover where people were asking questions that the bot wasn't able to respond to effectively, log that, analyze that in partnership with a transparent vendor that was providing us with the access to all of that data and information. >> That's great to hear.
Thank you. Erin, Chelsea, would you like to chime in here? >> Sure. I'll go ahead. Yeah, I mean, the Hello Baby example is a little bit different, right, in that the consequences and stakes are super high to either identifying people for supports, or people's kind of worst nightmares around identifying people for other purposes. One of the things that we took a lot of time with, but I don't think it had a ton of value in the end, was really working through technical aspects of the model with participants, potential participants, family members, community, trying to, developing, for example, slim models, models that were smaller that were theoretically easier to understand. When that wasn't really at the heart of what people really told us were their concerns, right? So, a lot of us in this work, I think, think a lot of it is about the modeling itself, and that if we just make it easier to understand, that will, that will solve some of the problems.
That wasn't our experience with our community. Mostly they were concerned with what that, what they might benefit from, from the service of being identifying early, and being supported, what they could be concerned about, and how we were going to mitigate that, how people would treat them as human beings and so on, and so while I think that work was really interesting, I don't think in our particular case, generated any additional kind of trust or understanding of the approach. At least for the people that we talked to. So, wanted to share that. >> That's really interesting.
Could you share what were some of the things that you heard were most important for them to provide input on, or things that they were most concerned about? >> Yeah, I mean, people said things, which you wouldn't be surprised, that said, you know, sort of, people come to our door all the time, or they offer us things. We don't so much care, if you will, how you get to the door. What we care about is what, how you treat us when you get there, and what you're going to do to support us.
You know, if you're offering us something that we don't need, then that's not going to be helpful. But things like diapers would be really helpful, meals for new parents. And when we were able to make good on the things that people wanted, and I think treat people with the respect that they deserve at that place, that was really what was of interest to them. >> Thank you. Chelsea, did you want to chime in? >> Yeah, I think for the City of San Jose, what really benefited us was that we work with a lot of nonprofit organizations within our community who had already built that trust with residents and community members.
So, partnering up with those organizations allowed the city to come in and really provide answers to those big questions that allowed us to build trust through a city lens as well. And with those nonprofit organizations, we were really able to know exactly what languages were needed for those community meetings to provide the interpreters that we needed. So, if it was going to be primarily Spanish speakers, we had a Spanish interpreter. If it was Vietnamese, providing a Vietnamese interpretation, to really allow people to provide feedback in the language that they're most comfortable with, which allows them to, you know, speak at ease, and speak exactly what's on their mind without having that language barrier.
Something that also really was, that continues to be a challenge, is that having a lot of public meetings can be hard. The timing, scheduling, and budget, while government employees, we work throughout the day, a lot of people in the community are working those same time frames, so being able to schedule those meetings outside of the regular working hours, so people can join meetings afterward, virtually, in person, making sure we're in those communities, as well as how would it help, how could we help them participate. So, if they need childcare, how can we incorporate childcare so that they can focus on providing feedback while the children are also being taken care of. So, there's a lot of factors that play into how we can do this efficiently, while receiving feedback that we need, while also making sure that we are making it easy for our community to do so. >> That's great.
And actually you prompted one of the questions in our chat, which was about how do you bring non technical public stakeholders up to speed, and also create the conditions for them to constructively participate in algorithmic design? I don't know if other folks want to chime in here in terms of things that you've done to bring non technical public stakeholders up to speed. >> I can chime in on that, actually. So, one of the encounters that we had at the Census Bureau is kind of historically decisions about kind of how you implement your statistical safeguards for confidentiality have been made behind closed doors that meet internally.
And we actually benefited substantially from the public engage these this go around. We have incredibly talented demographers and economists and statisticians inside, but they don't know the full spectrum of uses of our data, or how people on the ground are using these, and so they can't find of inform a lot of that decision making without kind of information from the public. And being able to crowdsource this to say demographers and city planners and tribal leaders and public health professionals, gave us this wealth of information to improve the design and implementation of our algorithms. But these were conversations that almost none of the stakeholder groups had any experience participating in before. Because, as I said, historically these are the types of things that have happened inside, inside the agency. And so what we had to do was in addition to trying to set up communication pathways with a very diverse set of stakeholders, we also had to really invest in a lot of education and information sharing outwards to those stakeholder groups so that they understood kind of what the design of our algorithm even looks so that they understood this trade off that had to come into play between privacy and confidentiality.
So, there was a huge educational component for us. And that was a challenge. Like we had a few missteps certainly at the beginning of the process. But we kind of iterated and asked for feedback not just on our algorithm design, but also asked for feedback on the materials that we were giving from that educational component, and use that feedback to then help better design the information we had to put out to bring these non technical stakeholders kind of at least up to the level where they can be constructively contributed to the conversation. >> Thank you. And one of the questions in the chat is around the very concept of participatory design and AI.
And wanted to know maybe Chelsea or Emily, do you have thoughts on how the concept of participatory design and algorithms emerged, or when did it first start to come across your radar? >> Well, I think participatory design in emerging technology has been around for a while. At least in the Smart Cities field. You know, we've seen a really big push, especially after the sidewalk labs case study and case side where there was a lack of communication about the use of residents' data for analyzing and generating efficiencies and optimalization of different types of technologies in a Smart City framework. I think we saw a really big push for integrating public perspective more and more across emerging technology. So, my view is that the conversation isn't so different from with AI than from other types of technologies, but we have seen some great examples of cities that have integrated public perspective in a really informed way really well.
One example I love pointing to is Paris instituted civic assemblies, when which they basically create an education opportunity for residents to become experts in particular field or a particular technology, and they compensate them to provide their lived experience inside over a certain period of time to shape policy around that technology. And then at the end of that process, they receive a credential. And that credential, in some cases, residents report using that to gain their own professional opportunities elsewhere. So, I think that's a great example of integrating, you know, what we need from residents, which is their lived experience and understanding so we can write policy or build a better service, but also treat that as, you know, something that deserves to be compensated, and that in exchange for their time, you know, they can receive a professional development opportunity in the process. And I think that can be applied across the board of different technologies, including AI. >> Love this concept of the civic assemblies, particularly in providing input in technological issues.
Erin or Emily, I'm curious, how do you budget for participatory design activities? I'm sorry, for Chelsea, or Erin, how would you, how do you budget currently, how are you thinking about budgeting in the future? >> Oh, go ahead. Sorry, Chelsea, I wasn't sure if you were going first. We just, we just make sure it's part of the overall budget for the initiative, right? And so if we're not willing to, if those costs are too much, then I guess the whole thing is too much, right? So, we just see it as a core part of the effort. And without which we would not be able to implement the effort for real. So, it is, you know, it's government funding primarily that we use to fund our initiatives. >> It's similar for the City of San Jose.
A lot of technology is tied to city services are already implemented, so for the City of San Jose, as we look into increasing our San Jose through our one services, and looking at using AI to check road conditions, that's, in a way, partnered with the DOT in how we help their services, fixing the roads. However, what's pretty new for today at San Jose is actually my job in general. I only set it four months ago, and I was tired as a public information officer for IT, which is [inaudible] to really, while most of the departments already have public information officers to promote their services, as IT saw the need to promote technology, even before that tech is implemented into city services, to really start off with that engagement even before city services are even brought up into the conversation. So, really promoting the idea of having someone dedicated to doing public outreach is something that's really important. And I'm excited to see where it goes.
>> Love that pioneering spirit. Bringing a public engagement professional like yourself to the IT team. Another question around how do you find public stakeholders? Do you piggyback onto existing public engagements, or do you start from scratch? And anyone who would like to chime in, please feel free. >> I'll start. So, we have a well established stakeholder engagement pathways. We have our federal advisory committees, we have various formal and informal stakeholder groups.
We do formal consultations with tribal nations. But we also have a fairly flexible ability to kind of do more targeted engagement when we need to. And so we started with our kind of formal mechanisms, and then branched out as we were hearing, like, oh, we're missing segments of the community over here, we're missing segments over there.
And as, again, this was an iterative process, so as we, as we were hearing from groups that were not kind of represented in our ongoing [inaudible] we would kind of establish new mechanisms for communicating with them. And it was very much an evolving process over a number of years, but ended up being a fairly active one. >> Yeah, I'll just, I'll add, I mean, we do similar things.
Of course we have traditional stakeholders who are part of advisory committees and things like that. And then definitely do specific outreach to people most impacted by whatever, you know, whatever we're doing. I think the other, I said this before, but other two really important things to do is to seek feedback from the people, especially advocates who you think will have the most concerns. So, you know, ACLU of Pittsburgh engaged very early on in the Hello Baby initiative and other similar approaches we've taken.
And then, again, I think like it's not fun, it's super hard, but I think the big meeting, the big public meeting is still, even though we can target more easily now [inaudible] kind of help us to do that, it's still a thing that that is necessary to put yourself through as a public official trying to do something important. >> Thank you for encouraging folks to be brave. With that, I want to thank all of our presenters. And I'm going to pass it on over to Jen for our panel discussion. >> Thank you so much, Shannon. And thanks to all our presenters on the last session.
There's never enough time to get to all the questions. Never enough time to unpack all the experiences. But we really appreciate that, those fantastic examples.
I'm very pleased to move us into this next section, which is a planner discussion with a wide range of experts in academics on lessons and emerging norms in this space. I'm thrilled to be joined by an outstanding panel, including Dr. Sheena Erete, who is Associate Professor at the College of Information Studies at the University of Maryland. Sheena is a researcher, educator, designer, and community advocate, whose research has focused on co designing, sociocultural technologies, practices, and policies, with community residents to amplify their local efforts in addressing issues such as violence, education, civic engagement, and health.
The objective of her work is to create more just and equitable outcomes and futures for those who have historically and who are currently facing structural oppression. Zoe Kahn is joining us from the UC Berkeley School of Information, where she is a Ph.D. candidate, and whose work uses qualitative methods to understand the perspectives and experiences of impacted communities and storytelling to influence the design of technical systems and the policies that surround their use.
She has conducted empirical studies in rural villages in Togo, rural ranching communities in the United States, and the people experiencing homelessness in the Bay Area. We're also joined by Min Kyung Lee, who is an Assistant Professor in the School of Information at the University of Texas at Austin. Min has conducted some of the first studies that empirically examine the social implication of algorithms emerging roles in management and governance in society. Looking at the impacts of algorithmic management on workers, as well as public perceptions of algorithmic fairness. She has also proposed a participatory framework that empowers community members to design matching algorithms for their own communities. We are also joined by Devansh Saxena, who is a Presidential Postdoctoral Fellow at Carnegie Mellon University in the Human Computer Interaction Institute, where he studies sociotechnical practices of decision making in the public sector, and examines how human AI interaction plays out in practice, where decisions are mediated by policies, nuances of professional practice, and algorithmic decision making.
And finally, we are joined by Tim Hughes, who is the Democracy and Participation Lead at the Open Government Partnership, and where he leads on deepening OGP's work around strengthening and reimagining democracy through Open Government, and especially through innovative approaches to participation and digital governance. So, welcome to all of you. I'm thrilled to have you join this session. And if I can make sure that everyone's videos are on, please, I'd like to invite all our panelists to turn their videos on. Fantastic. Let's jump right into it.
You all represent a wide diversity of academic institutions and expertise across the United States and globally. I'm wondering what norms we are seeing emerging in this space around open and participatory algorithmic design. And in particular, what lessons we are starting to glean. So, Sheena, I'd like to turn to you first, if you don't mind sharing with us what your thoughts are on what we're seeing coming out of this space. >> Yeah, so there are several things that have been emerging as norms. First, there have been extensive chatter about obviously inclusion and diversity.
We've talked a lot about transparency and how do we make algorithms more open and explainable to public. We've talked a lot about ethics and public engagement. And so this leads me to think about these questions about who's at the table when we're engaging in algorithmic design.
And so as we're, and as we're engaging in a participatory design of algorithms, who are we engaging, and can they attend, and what's the process and all of that. And so for the residence that they're attending at, also always try to pose the question about who's not there and why aren't they there. And so we really have to think about as we're bringing everyone to the table, even a deeper beyond the diversity and inclusion of like the participants who are helping us engage [inaudible] but also the diversity and inclusion and how does that reflect on the teams that are actually building the design of these technologies.
And so the question is like do they actually represent the diversity of the communities and the neighborhoods that are impacted by these algorithms. And so I'll give you an example. For example, I worked with street outreach workers in the City of Chicago across the city, and we really engaged very, very deeply. And it took extensive amount of time to build trust. And the street outreach work and the violence prevention work that they were doing on the ground, and this was much, much earlier than ChatGPT, so, you know, algorithms and AI is not as common, et cetera.
And so the question is, and so we worked with those street outreach workers to design tools that they could use on a daily basis. And the other question is like, that came up is how are we engaging them in meaningful ways? Can they hold us accountable? And so I think even as we think about local governments creating these technologies is are there mechanisms to try to hold, hold us accountable to that? And so, and then the last thing I think I would want to say is if we're engaging in, and we're trying to create means for them to be accountable, how are we thinking about issues around power? And how are we shifting power, such that the communities own their own data and are a part of the processes, not just at the point where we are creating the tools, but also all throughout the design, evaluation, and deployment of the algorithms? And I'll stop there. >> Thanks so much for those comments, Sheena. That issue of local empowerment, going beyond just the inclusion, but really using these tools to empower communities is such a part of what I think we'll hear, especially some of our global comparative experiences, what we're hearing on that stage. Min, I wonder if you might come in on this as well, because I know your research has focused on this nexus point.
>> Thank you. So, I wanted to talk about some of the benefits of participation in AI design. So, the speakers in the first case study session, they gave presentation on excellent cases where they included community members from the get go, and how they actually improved and redesigned the AI system, and which kind of earned their trust and kind of made the system to actually fit their local culture, et cetera.
So, that was an excellent example. I also wanted to highlight that the participation itself can actually give really good ideas on what to build. And not just how, but what to build to address their actual needs.
So, one example that I like to use is the work that we did with [inaudible] workers. Initially, we were doing interviews with them to understand the pain point. And our original thought was, okay, as a researcher designer, we are going to think about how to address their needs. But actually when we were engaging with them, they were giving us a lot of good ideas on how they want to benefit from their own data, what are analytics that they want to see in their dashboard, et cetera, so we quickly kind of reframed our studies to think about like how can we actually co design the initial conceptual idea itself through the worker participation [inaudible]? So, I think one benefit of the participation is that they actually have great ideas on what to build and how to build. I also wanted to highlight some benefits that we saw on participants themselves.
The participation is a laborious process, so I'm very grateful for their participation. But also it was very meaningful to see that the [inaudible] had an effect on them, positive effect on them. Like, for example, a lot of people use this occasion as a way to learn about AI. So, in some of the earlier work we did, a lot of, some participants expressed their skepticism that AI can do, for example, allocation work.
For them, allocation for the nonprofit donation work, it's work that involves care. And they thought they would strategically human work. But when they're actually doing it, and so how AI can be trained on their own data, et cetera, they started to say that actually think the AI can do this work. And then their perceived barrier in thinking of AI got a lot lower. The other kind of benefit that we saw is because they themselves have to think about their own work routine, or as a service, et cetera, it gave them an opportunity to reflect on their own work and practice.
So, a lot of people when they're leaving this study, they said, I kind of learned about this, my work, and now I'm going to talk to my manager to talk about this point. So, that was also kind of, I think when we are thinking of participation, we should really think about intentionally how to promote this positive impact on the participants. And the last thing that I will briefly touch is that because we think of, when we think about written design because of AI, the importance of new method and tools also can be, is important, and there is a lot of room for innovation, for them to give input on this. It can range from storyboards and comic board techniques to a more technical driven approach, where we build a tool where they can easily train with a learning model for themselves, so kind of the third lesson is you need to think about the new method. >> Thanks for those excellent points.
And really hands on, everything from how we are communicating and participating in accessible ways, in open ways, but also understanding that this is a two way street. There's clear benefits to governments from this participation, and from officials. Denosh [phonetic], I wonder if you might come in on this, because particularly on that point about how it changes participants, I know you've done some interesting research from the employee side, so I wonder if you might come in on that. >> Yeah, absolutely. So, I'm going to first piggyback off of Min's point about how citizens can help us, not just question the technology that we are talking about, but also help us design the right things. So, citizen engagement often occurs after a system has been developed.
And here, the goal is often to make the system more ethical or community centered. But in my recent work, what we are learning is that citizens can play a critical role during the very first step of the AI lifecycle. That is, when the [inaudible] social problem and convert it into a measurable statistical problem. We generally call that problem formulation. And this is where a lot of the [inaudible] issues become embedded that are significantly hard to fix at later stages. And so what we are doing is building simple sandboxes for community members to engage with.
This kind of builds off of something that Emily was talking about earlier. And we are learning that community members are able to really question the core objective of the tool itself, the data that it's using, and also how this is being used. And finally, these sandboxes also help create an informed mistrust, showing community members all the ways in which the AI system fails. And so, for instance, we build a simple AI crime mapping tool that was based on the city that the participants live in. And from the get go, that helped establish a degree of comfort, because they knew the city, they knew the neighborhoods, they knew, yeah, and so from the get go, community members question whether allocating more patrol cars actually help protect the community. So, the goal of a tool was misaligned with what the community wanted in the first place.
We also called the [inaudible] over policing. You send more patrol cars to the neighborhoods, you find more crime, and you send, and then you end up sending more patrol cars through those neighborhoods. And then they also highlighted that arrests do not really equal the likelihood for crime. Because an arrest does not really mean a person is guilty.
And based on your zip code, your likelihood for crime, your likelihood of getting arrested can be very high. So, what we saw was from the get go, just at the very first stage of the EI lifecycle, community members are really able to question the primary goal of the tool, what kind of data is being used, and the downstream impact that it can have on their own communities. So, that was the community perspective on this. I've also worked extensively with workers to understand their perspective on these tools and how they engage with them. And what we often end up finding is that workers have very good make and models of how their work actually goes on, and where these tools come in.
And if the tool is not [inaudible] in their original practice helps them accomplish their work in some ways. They end up finding ways to work around it, or giving the system. And they will socially learn from other workers on how to do this. And so a lot of my work is in child welfare. And you see that child welfare workers will continually find ways to [inaudible] systems to make them work for their clients, the families. And not just to meet the demands of the policy makers.
>> Thanks so much, Denosh [phonetic]. And that's just such an interesting connection between both sort of the design elements around being intentional and what communities need and the stakeholders, sort of the state for
2024-08-09 23:08
