Discourse Graphs and the Future of Science | Protocol Labs

Discourse Graphs and the Future of Science

Network Research

Feb 27, 2023

Interview between Tom Kalil, Chief Innovation Officer of Schmidt Futures, Dr. Evan Miyazono, Research Team Lead at Protocol Labs, and Dr. Matt Akamatsu, Assistant Professor of Biology at the University of Washington.

Tom Kalil: I wanted to talk to Evan and Matt because of their pioneering work on discourse graphs – a potentially new way of communicating and sharing scientific arguments. I think more scientists and funders of science should be aware of these ideas. I’m also interested in the use of graphs for fostering innovation and commercialization of research, such as work by Deep Science Ventures on outcomes graphs.

Below is a copy of the Q&A conducted over email between me, Evan, and Matt.

Individual answers are from Evan Miyazono (EM) and Matt Akamatsu (MA).

What is a discourse graph, and how are scientists beginning to use them?

EM: A discourse graph is a way of structuring and sharing scientific arguments. Each brief note is labeled as a question, claim, or evidence. These notes (nodes) are connected to each other (via edges) in a graph based on their relationships.

Discourse graph schema. Scientific arguments are decomposed into their constituent parts -- questions, claims, evidence -- and connected into a graph. Evidence can support or oppose a claim.

MA: Discourse graphs currently have two use cases:

1. Knowledge synthesis and sharing. A modular, reusable alternative to systematic reviews, discourse graphs capture the current lines of evidence that make up the state of knowledge in a given research subfield. Each evidence page contains enough context for the user to assess its validity and relevance, and are used to build novel claims that propose to answer a given research question. Multiple, opposing claims can coexist; since each claim is linked to the underlying evidence, each reader can assess the validity of each claim on its own merit. Discourse graphs give researchers the means to structure and reuse their literature searches, and importantly distribute the effort in generating and maintaining the database.

Example discourse graph for the question: Are bans an effective way to mitigate antisocial behavior in online forums? (Source: Joel Chan)

2. Open-source, modular research communication. Adapting the question/claim/evidence schema to ongoing research has allowed us to discretize the scientific research process for modular, collaborative research. At the end of the literature review, researchers pose hypotheses (testable claims) which motivate the collection of new data, leading to original results (new evidence) that support evolving claims. Each unit of new research lives in the discourse graph for context and reuse by other researchers, potential collaborators, science communicators, and funders. This enables gap analysis and facilitates discoverability of interesting problems within the field.

(Left) Schematic of discourse graph usage for original research. (Right) Illustrative historical example

MA: The Akamatsu Lab shares a discourse graph to pose new research questions, get up to speed on the current state of knowledge for that research question, and find an entry point for useful contributions. Lab members share evidence from articles, working hypotheses, requests for new experiments (called issues), and original results such that we can build on each others’ findings. Contributing to discourse graphs has accelerated students’ onboarding to our research field and given them a structured, discretized means of contributing to research.

(Left) Interactive bulletin board for new contributors to the Akamatsu lab discourse graph. Each element on the right column links to a live query within the discourse graph. (Right) Flow diagram of modular research sharing within the Akamatsu lab discourse graph

Researchers are generating and sharing their discourse graphs.

How did you get interested in discourse graphs?

EM: My team had invited Prof. Joel Chan to present his research on synthesis of academic literature. He started describing how he’s changed the way he structured his notes based on his research, and when he said, “I don’t give new students a stack of papers when they join my lab, I give them access to my notes graph” I formed the immediate conviction that this was the future of scientific collaboration. It was easy to see this as a tool to achieve the paradigm of collaboration presented in Michael Nielsen’s Reinventing Discovery.

MA: Our project is based on the hypothesis that new tools for collaboration can facilitate cultural and organizational change within scientific research communities. I connected with Joel Chan after trying networked note-taking tools to connect my notes for a collaborative project on SARS-CoV-2 cell biology during the pandemic. Our conversations led to the insight that discourse graphs could be applied to our ongoing research, which might facilitate the types of modular, rapid research contributions that I wished were possible for the COVID researcher community.

What do you think are the potential advantages of discourse graphs, relative to the journal article or preprint?

EM: A journal article is usually the culmination of 6-24 months of work. The initial idea or result, which would be usable to others in the field, often is generated in the initial months of the project. The promise of Discourse Graphs (or any robust, graph-based, notes schema) is that useful progress can be shared sooner, with sufficient context for collaborators or with the research community at large. Preprints increase the accessibility, but usually do not increase the availability of research to the extent that it could be said to accelerate science.

DGs provide the opportunity for modular peer review, with clear affordances to ask “does this evidence really support that claim?” or “is this evidence reproducible?” without needing to pass judgment on an entire body of work.

Journal articles and preprints are also required to demonstrate a complete scientific narrative, starting from a research question, generating a hypothesis, designing and running an experiment to test the hypothesis, and generating conclusions. Researchers must perform all steps to receive credit; discourse graphs could enable specialization of labor within the scientific system. While the complete narrative still has clear value, discourse & results graphs enable us to reward efforts both in whole and in part.

MA: In my field, journal articles have ballooned to include dozens of results and authors, taking several years from idea to publication. Collaboration is disincentivized because a single first author claims the majority of credit. Individual researchers collect individual results - why not attribute credit for each research contribution? Generating a discourse graph cites atomic units of work -- a single result or hypothesis -- so that researchers have an ongoing, accurate representation of their work and its impact. Journals’ “author contributions” sections have provided limited value because they are added manually post facto and do not track individual results or intellectual contributions.

While journal articles and preprints center around one or more claims, discourse graphs link to the underlying evidence, which makes it easier for other researchers to make their own claims from the aggregated evidence.

Journal articles tend to select submissions based on perceived novelty of claims, rather than rigor of results. With discourse graphs, we envision that results from a single experiment will gain strength over time by being reproduced by other researchers. Overlapping work will be mutually beneficial rather than adversarial. Hypotheses can be updated as new evidence becomes available.

Research output, rather than resembling single-use consumables, will instead be reused in a sustainable scientific ecosystem. These global changes result from the following local changes:

• Modular peer review – individual hypotheses can be challenged by others who present additional evidence from the literature or their lab notebooks. Peers can corroborate results by reproducing or flagging for irreproducibility, supporting constructive addition to the work rather than gatekeeping.

• Decentralized coordination – within groups, the transparency facilitates useful exchange of research between peers, rather than relying on PIs as the primary sources of plans and ideas. This decentralization raises the agency and responsibility of individual researchers, and facilitates the generation of novel organizational structures for science (e.g. DAOs).

• Constraints liberate – an open, structured conversation between researchers from idea to hypothesis to result to claim more closely reflects the intrinsic discovery process than the current first-author incentive system.

What do you view as the primary obstacles to the adoption of discourse graphs, such as the incentives for scientists to contribute?

EM: One obstacle for researchers is becoming comfortable sharing results and ideas sooner. Hopefully the sharing of progress immediately and continuously is viewed as an extension of the preprint revolution, rather than a new practice. We need the community to respect notes graphs as artifacts in the scientific literature, which means integrating them into existing workflows and allocating credit based on work done in notes graphs. The current culture in many fields is one of secrecy over openness, which needs to be inverted.

MA: Academic career incentives are an obstacle to adoption. DGs encourage researchers to make careful contributions that can be built upon, which may not correlate with current “first past the post” publication incentives. In DGs, impact is assessed by the usage and lasting power of individual results, rather than by article citations or journal reputation.

Many academic researchers will resist moving away from journal articles as career currency. That said, I imagine that DGs and journal articles can coexist for the foreseeable future; in our lab, we will publish some traditional journal articles and co-publish companion DGs, or alternately micropublish individual results or conclusions from our DG as connected modular preprint articles.

EM: Journals currently provide a valuable signal around recognizing certain research as important and making the results salient, and researchers will need new habits and tools to do this at least as well in a world where all science ideas are shared immediately and continuously. The eventual use of tools leveraging NLP over DGs to filter the most relevant contributions and then match and merge relevant results clearly has capacity to surpass the information salience provided by journals.

Another obstacle is adopting the subtle change in how they take notes. This should be close to the change of using Overleaf or Google Docs instead of emailing .doc attachments to each other, but it is a barrier. There are multiple emerging proof-of-concept tools for generating DGs. However, the user experience for teams is still early days and a bit rough around the edges.

Lastly, DGs do not carry universal context. Much of any single working DG will initially only have enough context for close collaborators to use it. That said, the addition of this context is often much of the beneficial aspects of writing up research results in journals, so we see this as an epistemological limitation on scientific communication made explicit by the graph format, rather than a limit of the format itself.

What role could different types of individuals and organizations play in accelerating the adoption of discourse graphs?

EM: Funders could give funding for research contingent on open publishing of structured notes for that research, in support of open science. However, it becomes more compelling if funders may eventually be more interested in the discourse graph as a research output.

MA: Funders could accept structured hypotheses within discourse graphs as substitutes for grant proposals. Funders can also support developers making discourse graph tools easier to use, and allow teams to hire “cybrarians” to assist in generating and maintaining a team’s DG.

EM: Publishers could partner in the development and usage of citation conventions. In the longer term, publishers could co-publish discourse graphs alongside or embedded within journal articles, and update their citation metrics to leverage the added value of discourse graphs.

MA: Researchers could pilot the use of team discourse graphs that are open or will be shared upon publication of resultant journal articles. They can provide user feedback to the developers improving the usability of discourse graph tools.

EM: Developers could build tooling for interoperability between different note-taking platforms to facilitate adoption; build tools to assist in knowledge synthesis.

What types of pilots should the research community try? How might we evaluate the efficacy of these pilots?

EM: A structured program could be developed for any of the above roles. Examples could include:

• A funder funds research collaborations that pilot discourse graphs for coordination & synthesis; individual teams pilot discourse graph use internally to measure if it noticeably results in more cross-pollination of research elements.
• A funder, professional organization, or institution makes a publication community and invites teams of researchers; this could be tied to funding opportunities.

These pilots should seek to show that discourse graphs are both (a) appealing to use and (b) more effective than other methods of collaboration.

How do discourse graphs relate to NSF’s support for knowledge networks?

EM: Knowledge networks provide unifying venues for data; discourse graphs represent the scientific process and its epistemological uncertainty in the format of a graph. In this way, we should expect discourse graphs to leverage and, in some ways, extend knowledge networks.

What role can large language models play in automating the process of building discourse graphs?

MA: We are making early progress using user-annotated papers as training data to help GPT-3 classify discourse graph elements of research papers. In this way, discourse graphs will help LLMs to extract relationships between claims from the existing academic literature. LLMs can also summarize the context for a given piece of evidence and tie a sequence of discourse nodes into a narrative, to help translate between discourse graph content and traditional articles for diverse audiences.