DedupEndNote: Details, lots of details ...

Details of the comparisons

1. Publication year: Are they at most 1 year apart?
2. Starting page or DOI: Are they the same?
3. Authors: Is the Jaro-Winkler similarity of the first 40 authors > 0.67
4. Title: Is the Jaro-Winkler similarity of (one of) the normalized titles > 0.89?
5. ISBN, ISSN or Journal: Are they the same (ISBN, ISSN) or similar (Journal)?

The following comparisons are used (in this order, chosen for performance reasons):

1. Publication year: Are they at most 1 year apart?

This 1 year margin was chosen because "ahead of print" publications and final publications often are not published in the same year. A margin larger than 1 year would make the program a lot slower (see Justification: Order of comparisons).

• Preprocessing: Publications years before 1800 are removed.
• Insufficient data: Records without a publication year are compared to all records unless they already have been identified as a duplicate.
• Special cases: Cochrane Reviews are compared for the same publication year only.

2. Starting page or DOI: Are they the same?

If the Starting and Ending page of at least one of the publications are more than 2 pages apart,
then: the DOIs are compared first. If the DOIs are different or one or both are absent, then the starting pages are compared,
otherwise: the starting pages are compared first. If the starting pages are different or one or both are absent, the DOIs are compared.

Meeting abstracts often get the DOI of the whole conference proceedings. Comparing them by DOI produces a lot of False Positives. Using this "more than 2 pages" choice circumvents this problem.

• Preprocessing:
  • Article number is treated as a starting page if starting page itself is empty or contains "-".
  • Starting pages are compared only for number: "S123" and "123" are considered the same.
  • In DOIs 'http://dx.doi.org/', 'http://doi.org/', ... are left out, so that only the part starting with "10." is compared.
• Insufficient data: If one or both DOIs are missing AND one or both of the starting pages are missing, the answer is YES. This is important because of PubMed ahead of print publications.
• Special cases: Cochrane Reviews: if both records have a DOI, only the DOIs are compared, otherwise the starting pages are compared.

3. Authors: Is the Jaro-Winkler similarity of the first 40 authors > 0.67?

Jaro-Winkler similarity: See this Wikipedia page for a definition and here for some examples.

• Preprocessing:
  • The author "Anonymous," and all Author Groups are skipped.
  • Only the first 40 authors are retained.
  • All diacritical signs/accents, supplemental characters (e.g. "£") and all characters in non-Latin scripts are removed
  • First names are reduced to initials ("Moorthy, Ranjith K." becomes "Moorthy RK").
  • Names with a space in the last name, are also compared in a transposed form ("Lofving Gupta, S." is compared as "Lofving Gupta S" and as "Gupta SL").
  • All authors from each record are joined by "; ". This is the string that is used in the comparison.
• Insufficient data: If one or both records have no authors,
  • if both have an ISBN AND (one or both have no DOI OR one or both have no starting page), the answer is YES
    Reason: Conference proceedings have no authors, the comparison by ISBN will happen in step 5.
  • if one or both have no DOI OR one or both have no starting page, the answer is NO,
    Reason: Too few fields to make a safe comparison.
• Special case: If one of the records is a reply, erratum or comment (see below), a higher Jaro-Winkler Similarity threshold is used.

4. Title: Is the Jaro-Winkler similarity of (one of) the normalized titles > 0.89?

The fields Original publication (OP), Short Title (ST), Title (TI) and sometimes Book section (T3, see below) are treated as titles.
Because the Jaro-Winkler similarity algorithm puts a heavy penalty on differences at the beginning of a string, the normalized titles are also reversed if the publication is longer than 1 page (i.e. except for meeting abtracts, replies, retraction notices, ...).

• Preprocessing:
  • The titles are normalized (converted to lower case, text between "<...>" removed, all diacritical signs/accents, supplemental characters (e.g. "£") and all characters in non-Latin scripts are removed, all characters which are not letters or numbers are replaced by a space character, ...).
  • Titles which can be split on ". ", ": " or "? " and have 2 parts of at least 50 characters, are also compared on these parts.
  • In the titles of retracted publications all parts which refer to the retraction are removed. "RETRACTED: Response of Breast Cancer Cells and Cancer Stem Cells to Metformin and Hyperthermia Alone or Combined (Retracted article. See vol. 20, 2025)". A publication is considered a retraction if the Title starts with "retracted", "removed" or "withdrawn", or contains "retracted article" (all case insensitive).
• Insufficient data: If one of the records is a reply, erratum or comment (see below), the titles are not compared / the answer is YES (but the Jaro-Winkler similarity of the authors should be > 0.75 and the comparison between the journals is more strict).

5. ISBN, ISSN or Journal: Are they the same (ISBN, ISSN) or similar (Journal)?

This rule is skipped if both records have the same DOI (that comparison was made in step 2).
The fields Journal / Book Title (T2), Alternate Journal (J2) and sometimes Book section (T3, see below) are treated as journals. All ISBNs, ISSns and journal titles (including abbreviations) in the records are used.

If both records have an ISBN, the ISBNs are compared (stop), if both have an ISSN, the ISSns are compared (stop), else the journal titles are compared.
Abbreviated and full journal titles are compared in a sensible way (see Examples of comparisons).

• Preprocessing:
  • ISBNs and ISSNs are normalized (dashes are removed, lowercased). For ISBN-10 the first 9 digits are used, for ISBN-13 the 9 digits starting at position 4.
  • Journal titles of the form "Zhonghua wai ke za zhi [Chinese journal of surgery]" or "Zhonghua wei chang wai ke za zhi = Chinese journal of gastrointestinal surgery" or "The Canadian Journal of Neurological Sciences / Le Journal Canadien Des Sciences Neurologiques" are split into 2 journal titles.
  • the journal titles are normalized (hyphens, dots and apostrophes are replaced with space, end part between round or square brackets is removed, initial article is removed, ...).

Remarks:

Comment:

a publication is considered a comment if the title (fields ST and TI) contains words as "comment" or "commentary".

Erratum:

a publication is considered am erratum if the title (fields ST and TI) contains "Correction", "Corrigendum" or "Erratum".

Reply:

a publication is considered a reply if the title (fields ST and TI) contains "reply", or contains "author(...)respon(...)", or is nothing but "response" (all case insensitive).

T3 field:

Especially EMBASE (OVID) uses this field for (1) Conference title (majority of cases), (2) an alternative journal title, and (3) original (non English) title. Case 1 (identified as containing a number or "Annual", "Conference", "Congress", "Meeting" or "Society") is skipped. All other T3 fields are treated as Journals and as titles.

Examples of comparisons

In the following table the results of EndNote's Find duplicates is compared to the comparisons in DedupEndNote. For these tests only one field was selected in EndNote in "Edit > Preferences > Duplicates", and "Ignore spacing and punctuation" was selected.

Field	Examples	EndNote finds duplicates	DedupEndNote Score
Starting page and article number	... ...	???	???
Title	90Y radioembolization using resin microspheres in patients with hepatocellular carcinoma and portal vein thrombosis 90Y RADIOEMBOLIZATION USING RESIN MICROSPHERES IN PATIENTS WITH HEPATOCELLULAR CARCINOMA AND PORTAL VEIN THROMBOSIS	Yes	1.00 == Yes
Title	Comments about Glisson's capsule phleboliths and portal vein thrombosis [1] COMMENTS ABOUT GLISSON CAPSULE PHLEBOLITHS AND PORTAL-VEIN THROMBOSIS	No	0.92 == Yes
Title	Transarterial chemoembolization and <sup>90</sup>y radioembolization for hepatocellular carcinoma: Review of current applications beyond intermediate-stage disease Transarterial Chemoembolization and Y-90 Radioembolization for Hepatocellular Carcinoma: Review of Current Applications Beyond Intermediate-Stage Disease	No	0.92 == Yes
Title	Epidemiology and diagnosis profile of digestive cancer in teaching hospital campus of lome: About 250 cases. [French] Epidemiology and diagnosis profile of digestive cancer in teaching Hospital Campus of Lome: about 250 cases	No	0.99 == Yes
Title	Post Splenectomy Outcome in beta-Thalassemia Post Splenectomy Outcome in β-Thalassemia	No	0.96 == Yes
Title	Letter: portal vein obstruction--which subset of patients could benefit the most? Authors' reply Letter: Portal vein obstruction - Which subset of patients could benefit the most?	No	0.97 == Yes *
Title	Title: Some diseases associated with ulcero- hemorrhagic colitis: complication or coincidence. [French] Original Title: Quelques maladies associees a la colite ulcero- Hemorragique: Complications ou coincidences Title: [Various diseases associated with ulcero-hemorrhagic colitis: complications or coincidences] Original Title: Quelques maladies associees a la colite ulcero-hemorragique: complications ou coincidences.	No	1.00 == Yes *
Title	Title: [HELLP in the second trimester in a patient with antiphospholipid syndrome] Original Title: HELLP kan ses i andet trimester ved antifosfolipidsyndrom. Title: HELLP kan ses i andet trimester ved antifosfolipidsyndrom	No	1.00 == Yes *
Title	Title: NFkappaB inhibition decreases hepatocyte proliferation but does not alter apoptosis in obstructive jaundice Reversed Title: ecidnuaj evitcurtsbo ni sisotpopa retla ton seod tub noitarefilorp etycotapeh sesaerced noitibihni BappakFN Title: NF kappa B inhibition decreases hepatocyte proliferation but does not alter apoptosis in obstructive jaundice Reversed title: ecidnuaj evitcurtsbo ni sisotpopa retla ton seod tub noitarefilorp etycotapeh sesaerced noitibihni B appak FN	No	1.00 == Yes *
Title	Title: Case report. Duplication of the portal vein: a rare congenital anomaly Reversed Title: ylamona latinegnoc erar a :niev latrop eht fo noitacilpuD .troper esaC Title: Duplication of the portal vein - A rare congenital anomaly Reversed title: ylamona latinegnoc erar A - niev latrop eht fo noitacilpuD	No	0.96 == Yes *
Title	Title: La sémantique de l'image radiologique. Intérêt du procédé de soustraction électronique en couleurs d'Oosterkamp en angiographie abdominale Reversed Title: elanimodba eihpargoigna ne pmakretsoO'd srueluoc ne euqinortcelé noitcartsuos ed édécorp ud têrétnI .euqigoloidar egami'l ed euqitnamés aL Title: INTERET DU PROCEDE DE SOUSTRACTION ELECTRONIQUE EN COULEURS D'OOSTERKAMP EN ANGIOGRAPHIE ABDOMINALE Reversed title: ELANIMODBA EIHPARGOIGNA NE PMAKRETSOO'D SRUELUOC NE EUQINORTCELE NOITCARTSUOS ED EDECORP UD TERETNI	No	0.91 == Yes *
Authors	Cobos Mateos, J. M.; Aguinaga Manzanos, M. V.; Casas Pinillos, M. S.; Gonzalez Conde, R.; Gonzalez Sanchez, J. A.; De Miguel Velasco, J. E.; Soleto Saez, E.; Suarez Mier, M. P. Mateos, J. M. C.; Manzanos, M. V. A.; Pinillos, M. S. C.; Conde, R. G.; Sanchez, J. A. G.; Velasco, J. E. D.; Saez, E. S.; Mier, M. P. S.	No	0.75 == Yes
Authors	Danilă, M.; Sporea, I.; Popescu, A.; şirli, R. Danila, M.; Sporea, I.; Popescu, A.; Sirli, R.	No	0.93 == Yes
Authors	Lv, Y.; Qi, X.; Xia, J.; Fan, D.; Han, G. Lv, Y.; Qi, X. S.; Xia, J. L.; Fan, D. M.; Han, G. H.	No	0.90 == Yes
Authors	[empty] Anonymous,	No	Yes
Journal	British journal of surgery Br J Surg	No	Similar == Yes
Journal	European Journal of Gastroenterology and Hepatology European Journal of Gastroenterology & Hepatology	No	Similar == Yes
Journal + ISSN	Japanese Journal of Cancer and Chemotherapy [ISSN: 2690-2692] Gan To Kagaku Ryoho [ISSN: 2690-2692]	No	Similar == Yes
Journal	JAMA Journal of the American Medical Association	No	Similar == Yes
Journal	The Lancet Haematology Lancet Haematol	No	Similar == Yes
Journal	Hepatology Hepatology International	No	Similar == Yes *
Journal	AJR Am J Roentgenol American Journal of Roentgenology	No	Similar == Yes
Journal	British journal of surgery Surgery	No	NOT similar == No

*: In these cases the comparison of DedupEndNote for this content for this field is not accurate. However, the comparison of the other fields for these records does not result in YES answers, so the records are ultimately not considered duplicates.

Enrichment of the deduplicated records

Only the first record of a set of duplicate records is copied to the output file.

When writing the output file, the following fields will be changed:

Author (AU)	if the (only) author is "Anonymous", the author is omitted
DOI (DO):	the DOIs of the removed duplicate records are copied to the saved record and deduplicated. The DOI field is important for finding the full text in EndNote. DOIs of the form "10.1038/ctg.2014.12", "http://dx.doi.org/10.1038/ctg.2014.12", ... are rewritten in the prescribed form "https://doi.org/10.1038/ctg.2014.12". DOIs of this form are clickable links in EndNote.
Journal name (T2)	if the saved record has no value for the Journal name (T2) but has a value for the Alternate Journal (J2), this J2 value is copied to the T2 field. This is relevant for the ClinicalTrials.gov records in Embase. missing Journal name (T2) for "Social Science Research Network" records from Embase.com are filled
Publication year (PY):	if the saved record has no value for its Publication year but one of the removed duplicate records has, the first not empty Publication year of the duplicates is copied to the saved record.
Starting page (SP) and Article Number (C7):	the article number is put in the Pages field (SP) if the Pages field is empty or does not contain a "-", overwriting the Pages field content. the article number field (C7) is omitted if the saved record has no value for its Pages field (e.g. PubMed ahead of print publications) but one of the removed duplicate records has, the first not empty pages of the duplicates are copied to the saved record. the Pages field gets an unabbreviated form: e.g. "482-91" is rewritten as "482-491". if the ending page is the same as the starting page, only the starting page is written ("192" instead of "192-192"). for Cochrane Reviews a missing review number ("CD...") is extracted from the DOI.
Title (TI):	If the publication is a reply / erratum / comment / retraction, the title is replaced with the longest title from the duplicates (e.g. "Reply from the authors" is replaced by "Coagulation parameters and portal vein thrombosis in cirrhosis Reply")

Justification

1. 1. General rule
2. 2. Comparisons
3. 2.1. The fields chosen
4. 2.2. The fields not chosen
5. 2.3. Similarity
6. 2.4. Order of comparisons
7. 2.5. Effect of insufficient data from one of two records
8. 3. Enriching the duplicate chosen

Performance

Data are from:

• [SRA] Rathbone, J., Carter, M., Hoffmann, T. et al. Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module. Syst Rev 4, 6 (2015). https://doi.org/10.1186/2046-4053-4-6
  The data sets are available at https://osf.io/dyvnj/
• [McKeown] McKeown, S., Mir, Z.M. Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Syst Rev 10, 38 (2021). https://doi.org/10.1186/s13643-021-01583-y
• [BIG_SET] Own test database for DedupEndNote on portal vein thrombosis (52,828 records, with 5078 records validated)

Name	Tool	True pos	False neg	Sensitivity	True neg	False pos	Specificity	Accuracy
SRA: Cytology screening (1856 rec)	EndNote X9	885	518	63.1%	452	1	99.8%	72.0%
	SRA-DM	1265	139	90.1%	452	0	100.0%	92.5%
	DedupEndNote	1361	59	95.8%	436	0	100.0%	96.8%
SRA: Haematology (1415 rec)	EndNote	159	87	64.6%	1165	4	99.7%	93.6%
	SRA-DM	208	38	84.6%	1169	0	100.0%	97.3%
	DedupEndNote	222	6	97.3%	1186	1	99.9%	99.5%
SRA: Respiratory (1988 rec)	EndNote X9	410	391	51.2%	1185	2	99.8%	80.2%
	SRA-DM	674	125	84.4%	1189	0	100.0%	93.7%
	DedupEndNote	768	18	97.7%	1202	0	100.0%	99.0%
SRA: Stroke (1292 rec)	EndNote X9	372	134	73.5%	784	2	99.7%	89.5%
	SRA-DM	426	81	84.0%	785	0	100.0%	93.7%
	DedupEndNote	497	8	98.4%	787	0	100.0%	99.4%
McKeown (3130 rec)	OVID	1982	90	95.7%	1058	0	100.0%	97.1%
	EndNote	1541	531	74.4%	850	208	80.3%	76.4%
	Mendeley	1877	195	90.6%	1041	17	98.4%	93.2%
	Zotero	1473	599	71.1%	1038	20	98.1%	80.2%
	Covidence	1952	120	94.2%	1056	2	99.8%	96.1%
	Rayyan	2023	49	97.6%	1006	52	95.1%	96.8%
	DedupEndNote	2023	33	98.4%	1074	0	100.0%	98.9%
BIG_SET (5082 rec)	DedupEndNote	3952	92	97.7%	1030	8	99.2%	98.0%

See Test results - details for a description of this test database BIG_TEST on portal vein thrombosis.

Perfomance table

Tool	Setting	Duplicates found	Duplicates to delete	After deduplication	% kept
EndNote	Author + Year + Title + Reference Type (default setting)	32,891	19,959	32,869	62%
EndNote	Author + Year + Title	32,920	19,976	32,852	62%
EndNote	Author + Year + Title + Secondary Title (Journal)	22,120	12,333	40,495	77%
DedupEndNote		38,617	24,357	28,471	54%

TODO: UPDATE examples

The False Positives are all conference abstracts:

• wrong DOI in second of 2 EMBASE (OVID) records: same authors, year, title, DOI, different starting page
  • Segovia, M. C., et al. (2019). "Combined multivisceral and renal transplant in a patient with JAK-2 mutation." Transplantation 103(7 Supplement 2): S143. DOI: 10.1097/01.tp.0000576288.84252.91
  • Segovia, M. C., et al. (2019). "Combined multivisceral and renal transplant in a patient with JAK-2 mutation." Neurology 92(15 Supplement 1). DOI: 10.1097/01.tp.0000576288.84252.91
• wrong DOI in second of 2 EMBASE (OVID) records: same authors, year, title, DOI, different starting page
  • Galvao, F. H., et al. (2019). "Intestinal and multivisceral transplantation at hospital dasclinicas da faculdade De medicina da universidade De Sao Paulo (HC-FMUSP)-Brazil." Transplantation 103(7 Supplement 2): S171. DOI: 10.1097/01.tp.0000576492.69414.80
  • Galvao, F. H., et al. (2019). "Intestinal and multivisceral transplantation at hospital dasclinicas da faculdade De medicina da universidade De Sao Paulo (HC-FMUSP)-Brazil." Neurology 92(15 Supplement 1). DOI: 10.1097/01.tp.0000576492.69414.80
• reversed title 3 seen as similar to reversed title of 1 and 2: same authors, year, title, starting page
  • Cool, J., et al. (2018). "TRENDS IN THE PREVALENCE OF PORTAL VEIN THROMBOSIS AND ASSOCIATED MORTALITY IN CIRRHOSIS: ANALYSIS OF A NATIONALLY REPRESENTATIVE INPATIENT COHORT." Gastroenterology 154(6): S1178-S1178. [Web of Science]
  • Cool, J., et al. (2018). "TRENDS IN THE PREVALENCE OF PORTAL VEIN THROMBOSIS AND ASSOCIATED MORTALITY IN CIRRHOSIS: ANALYSIS OF A NATIONALLY REPRESENTATIVE INPATIENT COHORT." Gastroenterology 154(6 Supplement 1): S-1178. DOI: 10.1016/S0016-5085%2818%2933901-5 [Embase OVID]
  • Cool, J., et al. (2018). "THE ASSOCIATION BETWEEN PORTAL VEIN THROMBOSIS AND OTHER VENOUS THROMBOEMBOLISM IN CIRRHOSIS: ANALYSIS OF A NATIONALLY REPRESENTATIVE INPATIENT COHORT." Gastroenterology 154(6): S1178-S1179. [Web of Science]

Mark mode

If you want to manually merge the records which are duplicates according to DedupEndNote, you can use Mark mode.

Because Zotero does not work with / does not show record numbers, Mark Mode is not relevant for Zotero users.

In Mark mode the ID of the first record of a set of duplicate records is copied to the label field ("LB") of all duplicate records. The input file is copied to the output file with the addition of the Label field if it is not empty. The original content of the Label field is overwritten! All other fields are copied as is (so no enriching of the output: prescribed form of DOI, ...).

After importing the results file into a new EndNote database, make the Label field visible. The IDs in the Label field refer to the IDs of the original EndNote database.

• To see the duplicates, search for "Label" "is greater than" 0, and sort on the label field.
• To manually merge, change in Preferences / Deduplicate the fields to deduplicate on to the Label field, deduplicate (in EndNote) and merge at will

Comparing the results of deduplication by EndNote itself and DedupEndNote:

• Import the result file into a new EndNote database
• Deduplicate by EndNote itself
• Select all records in the "Duplicate References" set, and mark them as Read
• Select the "All References" set, limit to the duplicates found by DedupEndNote (search for "Label" "is greater than" 0), and sort on the label field.
• The records marked as Unread were identified as duplicates by DedupEndNote, but not by EndNote itself.
• Select the "All References" set, limit to the duplicates not found by DedupEndNote (search for "Label" "is less than" 0).
• The records marked as Read were identified as duplicates by EndNote itself, but not by DedupEndNote.

Special cases

1. ClinicalTrials.gov records

• Reference Type: Journal Article
• Authors: (empty)
• Journal: https://clinicaltrials.gov
• Pages: the ClinicalTrials.gov ID (e.g. NCT06923007)
• URL: the first URL is for ClinicalTrials.gov (e.g. https://clinicaltrials.gov/study/NCT06923007)
• the other fields are from the first record in a deduplication set

Limitations

• Input file size: The maximum size of the input file is limited to 150MB.
• Input file format: The program only handles files in RIS format, not in XML or CSV format.
• Input file encoding: The program assumes that the input file is encoded as UTF-8.
• If authors AND (all) titles AND (all) journal names for a record use a non-Latin script, results for this record may be inaccurate.
• (when deduplicating one file:) The input file must be an export from ONE EndNote database: the ID fields are used internally for identifying the records, so they have to be unique. However, if the RIS file does not have an ID field in the first publication, DedupEndNote assumes the whole file has no ID fields and gives every publication an ID (starting with 1). This has only been tested on Zotero files!
• The program has been developed and tested for biomedical databases (PubMed, EMBASE, ...) and some general databases (Web of Science, Scopus). The data sets used were the results of biomedical queries. Deduplicating records from other databases is not guaranteed to work, and performance is often very poor, esp. for non journal articles (see Justification. 2.5. Effect of insufficient data from one of two records).
• The program uses a bibliographic point of view:
  • an article or conference abstract that has been published in more than one (issue of a) journal is not considered a duplicate publication.
  • Records for each publication year are compared to records from the same and the following year: a record from 2016 is compared to the records from 2015 (when treating the records from 2015) and from 2016 and 2017 (when treating the records from 2016). A PubMed ahead-of-print record from 2013 and a corresponding record from 2017 (when it was 'officially' published) will not be compared (and possibly deduplicated).
  • Bibliographic databases are not always very accurate in the starting page of a publication. Because starting page is part of the comparisons, DedupEndNote misses the duplicates when bibliographic databases don't agree on the starting page (and one or both records have no DOIs).

FAQ

1. Why only EndNote / Zotero RIS export files?
2. I don't use EndNote, but ...
3. I use groups, comments, ... in EndNote and don't want to lose them
4. I have deleted records in EndNote and don't want them to reappear when updating
5. I prefer PubMed records above EMBASE records
6. Is DedupEndNote perfect?

1. Why only EndNote / Zotero RIS export files?

2. I don't use EndNote, but ...

3. I use groups, comments, ... in EndNote and don't want to lose them

EndNote export files in RIS format do not contain grouping information!

If you have groupings, ... in your EndNote database before you started using DedupEndNote: use DedupEndNote in MarkMode, and deduplicate then within EndNote only on the Label field.

TODO: EXPAND

TODO: XML (formatting)

However, for new projects the following will work: using a MASTER Database and a WORKING Database
(The names of the EndNote databases and the RIS files are just examples)

1. create a MASTER database
2. import your first set of records from one or more bibliographic databases into the MASTER Database
3. (optionally: clean up the MASTER Database) if the MASTER Database can contain duplicate records, export the MASTER Database as a RIS file, deduplicate the file with DedupEndNote, import the deduplicated results into a NEW_MASTER Database EndNote file, remove the old MASTER Database, and make a copy of the NEW_MASTER database as your MASTER database (File > Save a Copy ...)
4. Make a copy of the MASTER Database as your WORKING Database EndNote file (File > Save a Copy ...)
5. edit, comment, delete, group, ... records only in the WORKING Database!

When you have new export files from the same or other bibliographic databases:

1. import these results into a new TEMP Endnote database, export that TEMP EndNote database into a RIS file NEW_RECORDS.txt (overwriting the file if it already exists), and remove the TEMP EndNote database
2. export the MASTER Database into a RIS file OLD_RECORDS.txt (overwriting the file if it already exists)
3. use DedupEndNote - deduplicate 2 files with these two files (results will be in NEW_RECORDS_deduplicated.txt)
4. import NEW_RECORDS_deduplicated.txt into the MASTER Database
5. import NEW_RECORDS_deduplicated.txt into the WORKING Database.
   And again: edit, comment, delete, group, ... records only in the WORKING Database!
   • because the deleted records from the WORKING Database are still present in the MASTER Database (and as a consequence also in OLD_RECORDS.txt), duplicates for these records in NEW_RECORDS.txt will be recognized by DedupEndNote and not reappear in NEW_RECORDS_deduplicated.txt
   • the existing groupings, edits, ... in the WORKING Database will not be overwritten.

   TODO: What about manual adds?

   TODO: What about updates of ahead-of-print publications?

4. I have deleted records in EndNote and don't want them to reappear when updating

See previous question

5. I prefer PubMed records above EMBASE records

DedupEndNote compares records in the order of the input file, after they are grouped on publication year. This makes it possible to influence the choice of duplicate within a duplicate set.

Suppose all records have the field Database Provider filled in with either "PubMed" or "EMBASE". If you order the orginal EndNote database on the field Database Provider in descending order before exporting to a RIS file, DedupEndNote will use a PubMed record if a duplicate set contains one or more PubMed records and one or more EMBASE records (except if one or more EMBASE records of this duplicate set has a publication year one year later than all PubMed records). TODO: Explain why "later"?

Your preference will probably not be the same as the alphabetical order of the names of database providers ("CINAHL", "Cochrane ...", "EMBASE", ...). You could however change this field in "1 - PubMed", "2 - EMBASE", "3 - Cochrane", ... and order your EndNote database on this field (EndNote will order numerically) before exporting to a RIS file, or use another field with similar content.

6. Is DedupEndNote perfect?

Of course not.

However, DedupEndNote tries to avoid at all costs to wrongly identify records as duplicates. The MarkMode was specifically developed for this, making comparison between results of deduplication by EndNote and by DedupEndNote feasible.

False positives

There was only 1 false positive found in the validation sets (11,474 validated records):

• Cytology screening (from SRA-DM, 1856 records, all validated)
• Haematology (from SRA-DM, 1415 records, all validated)
• Respiratory (from SRA-DM, 1988 records, all validated)
• Stroke (from SRA-DM, 1292 records, all validated)
• BIG SET (our own test database of 52,828 records, with 4923 records validated)

• Cool, J.; Rosenblatt, R.; Kumar, S.; Lucero, C.; Fortune, B.; Crawford, C. V.; Jesudian, A.: The Association Between Portal Vein Thrombosis and Other Venous Thromboembolism in Cirrhosis: Analysis of a Nationally Representative Inpatient Cohort, in: Gastroenterology 154 (6 Supplement 1), 2018, S-1179
• Cool, J.; Rosenblatt, R.; Kumar, S.; Lucero, C.; Fortune, B.; Crawford, C. V.; Jesudian, A.: TRENDS IN THE PREVALENCE OF PORTAL VEIN THROMBOSIS AND ASSOCIATED MORTALITY IN CIRRHOSIS: ANALYSIS OF A NATIONALLY REPRESENTATIVE INPATIENT COHORT, in: Gastroenterology 154 (6 Supplement 1), 2018, S-1178

False negatives

DedupEndNote will miss some duplicate records, e.g.:

1. Records have different starting pages and not 2 DOIs which are the same
2. The publication years of duplicate records can be more than 1 year apart. The validated subset of BIG_SET (4923 records) has 60 examples of duplicate records with different publication years: differences are 1 year (48), 2 years (8) and 4 years (4).
3. Journals titles are not identified as similar (e.g. "Jbr-btr" and "Journal Belge de Radiology" with different or no ISSNs, or "Am J Dig Dis" and "Aaierj.Dio.Dis." (from EMBASE, definitely a typo), or "J Can Assoc Radiol" and "Canadian Association of Radiologists Journal")
4. Replies: Identifying replies by "response" as one of the title words is not accurate enough (e.g. "Regulation of hepatic blood flow: The hepatic arterial buffer response revisited") and could wrongly identify duplicates. As a consequence "Incidence and Natural Course of Portal Vein Thrombosis in Cirrhosis Response" (Web of Science) is not marked as a duplicate of "Response to Senzolo and García-Pagán" (PubMed, EMBASE and Scopus) (https://doi.org/10.1038/ajg.2013.298).

After deduplication by DedupEndNote, deduplication with EndNote itself (default setting: Author, Year, Title) could be worthwhile. The case of different starting pages can only be solved by looking up the publication itself, the case of the missed similar journals by your brain.

The next results are for version 1.0.0 / these results should be updated for the latest version:
After deduplicating our test database of 52,828 records with DedupEndNote and importing the result file into a new EndNote database, deduplication with EndNote itself (default setting: Author, Year, Title) found 260 duplicates. After manual inspection, we found 118 records which were true duplicates. Some of the "reasons":

• different starting page: 25
• different starting page and DOI: 1
• different journals: 7, i.e. DedupEndNote couldn't be expected to recognize these journals as similar
• error in starting page: 1
• Database error in Journal name: 12
  Scopus sometimes uses the continuation title too early:
  • "Blood Vessels" (PubMed) vs "Journal of Vascular Research" (Scopus, continuation of "Blood Vessels")
  • "Acta Pathol Jpn" (PubMed, EMBASE) vs "Pathology International" (Scopus, continuation of "Acta Pathol Jpn")
  • "Acta Paediatr Jpn" (PubMed) and "Pediatrics International" (Scopus, continuation of "Acta Paediatr Jpn")
  • "American Journal of Pediatric Hematology/Oncology" (PubMed) and "Journal of pediatric hematology/oncology" (Cochrane, Scopus, continuation of "American Journal of Pediatric Hematology/Oncology")
  • "Z Kinderchir" (PubMed) and "European Journal of Pediatric Surgery" (Scopus, continuation of "Z Kinderchir")
  • "Cardiovasc Surg" (PubMed) and "Vascular" (Scopus, continuation of "Cardiovasc Surg")
  • "Int J Addict" (PubMed) and "Substance Use and Misuse" (Scopus, continuation of "Int J Addict")
  • "Aust Paediatr J" (PubMed) and "Journal of Paediatrics and Child Health" (Scopus, continuation of "Aust Paediatr J")

  2025-08-01: the change in version 1.0.2 (comparison 5 (ISSN and Journal names) is skipped when the DOIs are the same) solves some (a lot?) of these problems. But: the journal name used in the deduplicated file is the one from the first record in the duplicate set.

• DedupEndNote error in Journal name: 3, i.e. DedupEndNote would be expected to recognize these journals as similar
  • ("Jbr-btr" vs "Journal Belge de Radiologie")
  • "Langenbecks Arch Chir Suppl Kongressbd" vs "Langenbecks archiv fur chirurgie. Supplement. Kongressband. Deutsche gesellschaft fur chirurgie. Kongress"
  • "J Vasc Surg Venous Lymphat Disord" vs "Journal of Vascular Surgery: Venous and Lymphatic Disorders"
• DedupEndNote parsing error in Starting page: 1 ("S6-97-s6-99" vs "S697-s699", DedupEndNote used "6" in the first case instead of "697")

Some cases of duplicate records are "unsolvable" for programs? Take e.g.the following publication (https://www.nejm.org/doi/full/10.1056/NEJM199105303242207):

1. Cabot RC, Scully RE, Mark EJ, McNeely WF, McNeely BU, Podolsky DK, Lewandrowski KB. Case 22-1991: A 15-Year-Old Boy with Fever of Unknown Origin, Severe Anemia, and Portal-Vein Thrombosis. New England Journal of Medicine. 1991;324(22):1575-84.
2. Podolsky DK, Lewandrowski KB. Case records of the Massachusetts General Hospital. New England Journal of Medicine. 1991;324(22):1575-84.
3. Podolsky DK, Ferrucci JT, Ellis DS, Mark EJ, Pasternack MS, Huang PL, Lewandrowski KB, Dowling WJ, Goldfinger SE. A 15-YEAR-OLD BOY WITH FEVER OF UNKNOWN ORIGIN, SEVERE ANEMIA, AND PORTAL-VEIN THROMBOSIS - APPENDICITIS WITH PERIAPPENDICITIS AND FOREIGN-BODY GIANT-CELL REACTION CONSISTENT WITH PRIOR RUPTURE - PYLEPHLEBITIS WITH THROMBOSIS, ACUTE AND CHRONIC TRIADITIS, AND CHOLANGITIS. New England Journal of Medicine. 1991;324(22):1575-84.
4. Case records of the Massachusetts General Hospital. Weekly clinicopathological exercises. Case 22-1991. A 15-year-old boy with fever of unknown origin, severe anemia, and portal-vein thrombosis. N Engl J Med. 1991;324(22):1575-84.

"Cabot RC, Scully RE, Mark EJ, McNeely WF, McNeely BU" in (1) were the (associate) editors of the NEJM in 1991, "Podolsky DK, Lewandrowski KB" in (2) described the case, all authors in (3) discussed the case.

For developers

1. Validation
2. Logging and log levels