Notebooks for All

Isabela: Hello!

Tony: Is this working? Do I need to bring it closer to my face? We are good. Cool.

Isabela: Awesome. I will be starting us off. That's good.

Tony: Hey, Jen, thank you for having us here.

Isabela: The gratitude is really necessary pair this will take a lot of people and expertise. I value you being here.

Jennifer: I'm having trouble finding the document for the presentation. maybe it is on the Webex.

Tony: Great!

Isabela: Wonderful. Can resume that in a little bit?

Tony: Thank you, Jen.

Isabela: You should have access to this document, resources for you in the future pair that's one thing I would add. Shared more about the project than I expected. Nonetheless, I'm super excited to talk to you about the Notebooks for All project, which includes this event but it's not the limits of this event. We will talk more about what we've been doing to better understand problems within notebook accessibility issues, what kind of remediation we've been doing, fixes, where we are going in the future without. First, I know we were introduced, my name is Isabela Presedo-Floyd, I use she/her pronouns, I look like out of a fluffy dog atop my head at all times. Thank you, I appreciate that. In terms of what I do on this project, I spend most of my time on the user research section, how will be set up these tests, what are we going to do? I'm here with Tony, he/they pronouns, a much larger dog than me. Under the hat, equally poofy. He handles a lot of our open-source expertise. He's really fantastic at it. Those are the perspectives will be talking from. Other people on the team include Jen and Patrick and Eric, who is in the audience. I don't know if we will hear from him today. A really awesome team with a lot of different skills. I will hand it over to Tony to explain a little more about space, maybe?

Tony: Thank you so much. That was a great intro. Thank you so much, Isabela. I kind of wanted to think about this in the scale of space here. There are trillions of galaxies out there. At least, that's what Google told me. I'm not an expert out there. Billions of devices, more important on the software side for us. Recently, there are millions of notebooks. This concept of a notebook, a scientific document is growing on our social network for code, Github, growing at the scale of thousands. But there is a significant scale that we are trying to address in this very small project.

Isabela: A lot of these notebooks are shared through so many different disciplines. Neither one of us are astronomers here today. But tons of languages, both spoken language, coding language, documents that are cutting across all of these different things. When they are inaccessible, this is not a small impact. We don't want anything to be inaccessible, I would say. This is impacting more people than we can accurately count right now. Our smallest picture, for the scope of this project is actually the notebook file itself. For those of you familiar with Jupyter, I'm talking about the file, not the notebook that is an interface. You can't hear the capitalization in my voice. What notebook is she talking about? Yeah, in this case, we have a circular flow chart. That's what happens when you're staring at a bunch of people and forgetting words, that has three boxes in it. Design on the lower left, if you go up and center, development, lower right is testing. They all connect in a circle. That's the cycle we were working with for this project. We have different expertise, trying to cycle through, what do we have with Jupyter right now and what we want with it Jupyter? Take it to the designer, how do we want to test this and how do we get that information to development? Going back and forth. Retesting those changes we had in the cycle. Of course, that sounds much more straightforward when I say it to you like that, cycles and practice are much messier. Those were the things we were focusing on, highlighting all of the different skills from those different people and creating a process that we could validate that these changes were helping people and centering feedback from actual disabled people instead of discussing what was going to work well and not. Tony is going to talk to everyone a little bit more about notebooks.

Tony: The only way I can do this is if I can yell. I have to cram a lot into very little. I'm going to be careful with everybody's ears here. Notebooks, right? Most people are familiar with the physical concept of a notebook where you write some stuff or draw some pretty pictures. That's exactly what scientists are doing. But they have an interface for it. It's not words, it something more than words, we have word processing documents that allow you to put computation in them. It changes the way that we can see and explore the world. These notebooks are designed to combine code and narrative. Code becoming so much more popular, we need everybody to understand how to talk about the two, writing code is a privilege. But software literacy should be accessible to everybody. That is our major concern here. These notebooks have a concept of them, of cells, take a bunch of cells and put them together, they can do magical things together. These cells contain these notes, narratives, people can put them together and tell their stories in really novel ways in a data-driven world. This technology is affecting millions and millions of data scientists and students, providers with Jupyter, Google, Microsoft, so on and so forth, they are used in documentation, testing, classrooms, the platform is consistently inaccessible across all things. As our user base grows, the number of people being excluded by this technology grow. We are trying to figure out in this Notebooks for All problem what makes an accessible computational document so that people can participate in the code world? I think that's your part, about getting people to participate.

Isabela: Is it? Sure.

Tony: Indeed!

Isabela: Are we talking about participating with our user testing?

Tony: You are doing user design and you brought all of these people in.

Isabela: This project has been years in the making, believe it or not. I'm so excited to be here, that this work is being done. All of us across different organizations, we are working together. There's a public Jupyter accessibility call. It took a few years, I think. But we figured out how to work together. It's been really awesome. That's part of why we have such a mixed group of people working on the project and what made it possible, we all had different resources we were able to work with to actually get paid to user testing, which has been something we were all aiming for but really only able to do together. That's been really awesome. Another aspect of that is that we wanted to make all of this public. That also means for those of you less aware with open sores, everything we are working on, code and design assets -- anyone can use and reuse as you see fit with open source. Getting participation hopefully from other people in the future. We want this resource to be useful to you.

Tony: Yes, indeed. Every single thing you mentioned includes a round of advocacy by advocating compensation, this work, you continue to do it. It's fabulous. One of the things that is hard -- we live in this notebook world, this concept of a notebook. We wanted a better analogy to bring what we are trying to work on here. The situation for us and for data scientists using Jupyter who have been the main clients we've had, they want everything. They want to fill their house with everything, now, if you try to move around the house, it's completely inaccessible. That's the place we got to. We have a lot of stuff and we don't know how to make it accessible. Some of the steps, some of the approaches and analogies we wanted to hear, how do we perceive a notebook as a house? If you use this for a classroom and people are together for an hour in the same space, this notebook can transform space and allow people to collaborate. These cells are rooms. When we started to do this and we were thinking about this house analogy, retrofitting accessibility is... A risk. It's an attrition problem. Either you have the resources to do it or you do not. We kind of found ourselves on that cusp. We are going to talk about what it was like trying to retrofit and understand what we have and what our new house might look like based on standards. Tell us about inviting people into our new house, full of stuff.

Isabela: Inviting people in is the user testing. We need activity -- we want you to come test this and complain, essentially. Tell us what is working, what isn't, that's kind of the process we are going through here. The image on this page is a list of GitHub issues, part of the user testing was writing up these issues so even if we do not get to fix them all, because there's more than we have time for but this project, that they are all documented now. Both the problems that we had and, in many cases, solutions we had discussed. Making sure this work does not get lost. It is documented somewhere. And we have these concrete issues but we also have larger takeaways. Perhaps some of the most interesting ones to me. Many of them kind of applied to more places than I thought. There is a link to the navigation test results, can you get year, can you do this? To get to this place on the notebook? I'm currently writing up once, to actually work on this event. That is a draft, sorry. I have some takeaways, we were testing when you have a notebook basically stuffed with all of this stuff. Existing space telescope notebooks we were able to test in as well. Real things people wanted to put in the notebook. Many of the fixes, people involved in accessibility work would consider the basics. A lot of the issues, there is no label. I have no idea what I'm doing here. There's no way -- no heading structure that makes sense. Those are very fixable. We know how to fix those things. Really anyone authoring a notebook can fix them on their own. Users really rely on familiarity. Some of them were like oh, I know what a notebook is. Some people had no idea what a notebook was but the no the code. If they know the code, they can backtrack. But figuring out what their expectations are, and the annoying but normal -- the best-case scenario we ever got, oh, yeah, I hate the way that works but I hate the way that works everywhere on the Internet. Like that's not really your problem. But that was interesting, Jen and I leaned into that. What could we do instead? Maybe we can start figuring out these things. That's your preview. I'm pretty sure I'm handing it back to Tony, though.

Tony: That's great. What could it be instead? Standard spaced, for any accessibility experts in the room or attending, every single notebook interface has a form, multiple forms. Each cell has a form in it. If I have 20 cells, I have 20 forms, not a single one of them have a submit button on any of the projects, anywhere. Massive failure. It started looking like yeah, we got to this technology out of need but the -- what is the meaning of this thing? This is actually a form. We could not have gotten to the sort of things if we didn't have the feedback from people's experiences and listen and interpret the results. The way that Jen and Isabela shared their work in an open-source way, it's super exemplary. User testing really benefited us. Another thing that benefited us, the community. Shout out again to Jen and Patrick for organizing the hackathon, having these advocacy events. Learning that we have this house. But it's part of a community, right? Part of community advocacy. We appreciate you all being here. Yeah.

Isabela: I think we have 2 minutes.

Tony: Oh, we started late? Okay, cool. Getting the house up to code. This is my favorite part. A short period of time to condense down hundreds of hours of work. Basically what we started with was making fixes to the existing situation and I lost my hammer and I started using my head for the nails instead. I was like, there has to be a better way. For guidelines, there are core principles and accessibility, we have something more. We do not have a banking application, we have this malleable object. We need to add more principles, it needs to be compromising, assistive and flexible. We cannot provide -- we have to clean up everything here. These notebooks become documents when we see them especially in the browser and they contain metadata. We started trying to apply analogies. I need a table in my house to have people over for dinner. One of the notebooks that we had from the space telescope folks had 80 cells in it. That's a lot of rooms in a house, a big McMansion. I need a lot of tables, need to navigate this place. By going toward HTML 5, a modern standard for the web, we can comply with more accessibility features, if we obey the semantics right and stop having opinions as scientists and let designers have opinions as designers, I promise everybody is going to do more of what they are good at.

Isabela: Opinions --

Tony: This project is an example of it, the interdisciplinary things happening at the Space Telescope Science Institute. I can't wait for the alt text talk. I'm not going to show a demo because we are lacking on time but how do you navigate these things? They are very unwieldy. But they look better on a screen reader, we are considering the legibility of it, proposing things like dyslexic font. Using code, highlighting syntaxes for colorblindness. And there's a table of contents. I was going to party but they're a bunch of links here. Check out some of the notebooks that we tested. There are links in this presentation here. This is an example of one of the notebooks, give a quick rip through it.

Isabela: This is not how it looks now, by the way.

Tony: But a better future, a better visual experience. Check out these on your own. Some of the things we used while we were working on this, screen readers, dev tools, equal access checker, standards and heuristics.

Isabela: It's important to know what we did not get to come out the next questions we want to answer, one of the big things, we worked with a small sample and it ended up being very visual-focused. We got a wide variety but I think that's important to note, there's a lot of research we could be doing there. We really focused on navigation and reading experiences. We didn't talk about the editing experience at all. That's another big problem and we wanted to make sure we could center on things, reading is also necessary to advocate -- it felt a little bit like a precursor. People can put almost anything in notebooks. The reason I bring that end, we tested on existing notebooks, things that people are really using but every day, I see people show up with the new things in notebooks that have never seen before. Awesome in one sense but keep in mind, if you are one of those people writing notebooks, give it a try with something. See what you can do to check its accessibility, a longer discussion. The retrofitting is difficult. A lot of cases where we were hitting their heads against walls, figuring out what to do with the existing product rather than starting from scratch has been an experience. It has felt like a lot of things from the outside. We are obviously going over time. That is evidenced in our --

Tony: Wait, what? You said we had 20 minutes. Okay, cool. We will leave some room for questions.

Isabela: That's probably my fault, not doing math right. I have not had to take a math class in many years.

Tony: That's what notebooks are for.

Isabela: No. The math still needs to be done right. But a lot of what we've already discussed but to reiterate, I cannot tell you how many accessibility projects have been on where this has been a struggle to make this happen -- it's sometimes tricky with Jupyter because we know it'll be inhospitable. We are inviting people to a house that is not up to code in many cases and we are just kind of holding it together. We've also found a lot of patterns across all of these notebooks. Things become a lot clearer. A lot of things that I didn't see before, kind of what I was talking about earlier with some of those annoying but normal, all of those kind of things, just kept coming up. Even with our limited sample and budget. Fighting that technical debt -- oh, every time we think we get a little closer, I think that resonates with people. It feels like two steps back some dates. Having resources for open user testing -- if anyone is interested in that, I would love to walk you through those resources and see what you can repurpose for yourself, what kind of testing structures do we have for you? All of the open resources. Tony has also been producing things that I know less about how to explain it well. I'm just really excited to see this process not only happen but for us to be able to document it so people can carry it on. The work is not done. I don't want to send that message at all. You want to wrap up with that one?

Tony: I just want to highlight how remarkable it is, what a small interdisciplinary and inclusive team accomplished in such a short period of time on very limited resources, it's challenging these spaces, which need more resources. We appreciate your advocacy and support in all of this stuff. Shout out to the team. I did not think we would get here. It seemed ambitious. We are at a good place and it would not have happened if a lot of special people did not get together. Isabela and I have been working together for... Four, five years?

Isabela: We from working with each other a lot for, like, three.

Tony: We've never been able to give this presentation together. Thank you. Hey.

Isabela: How do we want to take questions? I wasn't expecting that. I should also say, I do a lot of remote work. I'm not used to giving these talks in person. I'm used to staring at a chat silently for a few moments.

This is a question for Isabela. I'm sure that you can talk about this at length but I wanted to hear a little bit more about the modes of user testing that you use and what you've found most valuable, if there is anything that brought out any particular surprises?

Isabela: We really only used two methods. It was usability testing, which means we create a script -- I'm using that word loosely. A list of tasks we are going to ask people to do. In many cases, specific about the way we are asking it too. If we ask them to find this sell by number, by heading, thinking of those ways, that's the main design but we've also had these user interviews at the end as well, those are not completing tasks, in many cases, we do it as a reflection at the end, what happened, what did you expect? Those were the main two methods, mentioned in the write ups. If those terms aren't familiar with you, you might have to give it a quick look up. I'm trying to see if there's anything I want to say about that. Was that helpful?

Tony: Isabela has her scripts on our documentation so you can find links. The script and results. And some work in between is reproducible.

Isabela: Tony is showing an image for the documentation, which is linked in the markdown document that we shared. This is a write up that describes -- some more details about what the participants were using, what versions of things, if that's helpful to you as well. Thank you for asking.

You may have just answered my question. I was interested where you put the overlap guidelines. Were they under accessibility tips?

Tony: We haven't completed those yet.

Isabela: Like a mapping, one to one?

No, where the guidelines are.

Tony: All of the stuff, the content is in the GitHub repo. We have a document describing different things. Again, everything is a work in progress. These will inform you about some of the different states of the notebook in ways that we talk about the notebook in accessibility tips and how you can publish and there is an authoring -- the formatting, I'm sorry, aww.

Isabela: Yes, they all exist here. I can show you specifically where in the documentation as well. That's the main place where they live right now, on GitHub.

Patrick: If you have a question, raise your hand and I will bring the microphone to you.

This is a new thing for me and it is fascinating. My question, if I were to do this and get involved, having a lot of really good examples would be wonderful. It is part of your effort getting people together to create new notebooks from scratch that are following these best practices? Very deliberate, people that are invested and interested and all that? To have that kind of come alike, critical mass of examples for people to work from?

Isabela: Fantastic question. I realistically don't think we will have the time in this project to create new notebooks come at the main thing we've been looking at doing, updating existing space telescopes, following those kind of guidelines. But that's a dream I want. I just know we have some more things we need to wrap up in terms of what we've promised since that was not something we initially thought we were going to find. We were not looking specifically for authoring recommendations. But doing all of this navigation and content testing, we got a lot of feedback for those were actually things the authors could do. We expected to find things that would require development fixing and we really ended up finding that that's a big thing. But there's a lot of things authors are doing that would make a big difference as well, headings is an author twice. The developer cannot help you use good headings, right? At the moment, no, I don't have "this is a beautiful paragon of a notebook" type thing but that's what the checklist is starting to be. It's pretty long, I will say that. That's something I hope to have in the future. If anyone is interested in that, talk to us because I would be interested in making that happen.

Tony: That is my stretch goal for the project but I'm not very limber.

I

Patrick: Are there other questions?

Hey, I'm proud to say I'm one of the user testers. I don't know if I'm allowed to say that but it's one of the best experiences I have had. I don't know if Jen agrees with that because I was deafly the person saying what are we doing this for? Why are there no headings? It was a good experience for me because I'm an analyst for the federal government and we use Jupyter Notebooks all the time. My colleagues use it because I cannot use it, I should say. This is one of the reasons why I wanted to help. I don't really have a question but I want people, as many people as possible, to participate. You are not only helping Space Telescope Science Institute but blind people everywhere like me or analysts or people that want to become analysts, it's one thing to understand the data but if you can I've been read what your colleagues are reading and you can't edit or write notebooks? It sets your career back and it's really frustrating. I'm happy to participate in the user testing and I think it was a great experience and everyone did a great job. I want to plug in everyone that was able to do that.

Tony: Thank you, Beth.

Isabela: Thank you so much.

I have one question for you, can you slowly navigate to that accessible -- the accessibility tips for creating and publishing? Because... They will be interested in those links.

Tony: I'm going to do this on GitHub instead because it is more predictable. If you go to our link, it should be scattered in certain places for the event. Which one? Resources? Notebook authoring checklist. If you are at the top of the repository, you can also search for it -- there is no search unless you are signed in. Resources, hackathon, event. You will find the checklist. Thank you.

Jen: Thank you so much. That was really fun to watch. We are going to take a 12-minute break, if we could all get back here at 11:00, get some coffee, go to the bathroom, we will have more people entering. See you at 11:00!