Usage
It is possible to complete and customize each step individually:
import pandas as pd
from bib_dedupe.bib_dedupe import prep, block, match, merge, export_maybe, import_maybe
# Load your bibliographic dataset into a pandas DataFrame
records_df = pd.read_csv("records.csv")
# Preproces records
records_df = prep(records_df)
# Block records
blocked_df = block(records_df)
# Identify matches
matched_df = match(blocked_df)
# Check maybe cases
export_maybe(records_df, matched_df=matched_df)
matched_df = import_maybe(matched_df)
# Merge
merged_df = merge(records_df, matched_df=matched_df)
Fields used by BibDeduper
Name |
Definition |
|---|---|
ID |
A unique ID |
ENTRYTYPE |
The type of publication (e.g., article, book, inproceedings) |
author |
The author(s) of the publication |
title |
The title of the publication |
year |
The year of publication |
journal |
The name of the journal in which the publication appeared |
volume |
The volume number of the publication |
number |
The issue number of the publication |
pages |
The page numbers of the publication |
doi |
The Digital Object Identifier (DOI) |
abstract |
The abstract |
search_set |
Distinct sets of papers (e.g., old_search), can be empty. * |
* The merge() function ensures that records from the same search_set are not merged. The match() function ensures that individual pairs (e.g., A-B, B-C) do not come from the same search_set. match() does not consider transitive relations (i.e., A-C could be from the same search_set). The cluster() and get_connected_components() functions (part of merge()) ensure that records are not merged if the component already contains a record from the same search_set.
Search updates
When updating a literature search, the old_search can be assumed to have no duplicates. To exclude a set of papers from deduplication, it is possible to pass a corresponding label to the search_set column.
Example data
Data from the example datasets can be loaded as follows:
from bib_dedupe.bib_dedupe import merge
from bib_dedupe.bib_dedupe import load_example_data
# Load example dataset
records_df = load_example_data("stroke")
# Get the merged_df
merged_df = merge(records_df)
# Save as csv
merged_df.to_csv("merged.csv", index=False)
Import file formats
BibDedupe can process any bibliographic data set once it is in a pandas DataFrame, and contains the columns listed above. For importing common bibliographic formats, use the standalone bibliography-io package. It can read BibTeX, RIS, CSV, Excel, EndNote, NBIB, and JSON files; Markdown reference-list parsing is also available with its optional Markdown backend. BibDedupe does not require bibliography-io to be installed.
Install bibliography-io with the Excel extra to enable its pandas DataFrame helper (and Excel file support):
python -m pip install "bibliography-io[excel]"
Use load_df() to load and normalize a supported file directly into a DataFrame. The format is inferred from the filename extension, so change the path to load a different format:
from bibliography_io import load_df
from bib_dedupe.bib_dedupe import merge
records_df = load_df("references.ris") # also accepts .bib, .csv, .xlsx, .enl, .nbib, and .json
merged_df = merge(records_df)
If the filename extension is ambiguous or does not match the file format, pass format explicitly:
records_df = load_df("references.txt", format="ris")
load_df() applies generic normalization unless a source is specified. Database exports can differ in their field conventions, so check that the loaded DataFrame contains BibDedupe’s required fields (including ID, ENTRYTYPE, author, title, and year); source-specific normalization may be needed for some exports. Markdown reference-list parsing is also available with the optional Markdown backend and requires a GROBID service managed by the caller. See the bibliography-io usage documentation for source options, the complete format support table, format limitations, and optional dependencies.