Deduplicate research events without deleting distinct sources
Deduplicate the event, not the evidence trail. Several files may describe one event, while one filing may contain several events. Keep a document identifier for each source and a separate event identifier for the underlying occurrence. This prevents repeated publication from looking like repeated progress.
W3C provenance provides background for representing relationships between objects and activities. HKEXnews is an example of a publication route where document identities matter. The event-key design below is an original manual convention, not a universal identifier standard.
Compare meaning before merging
Use the legal entity, event subject, event stage, and relevant date as matching fields. A title alone is weak: “project update” may appear repeatedly as the project advances. Likewise, a changed website date may merely signal a repost of an old release.
Ask whether the sources describe the same occurrence and status. Signing and completion are distinct event stages even when they concern the same transaction. A translation of the signing notice is another document for the signing event. A correction belongs in a version relationship, with any changed event facts recorded explicitly.
Hypothetical deduplication table
Assume fictional Brookline Storage publishes a transaction notice through several channels. This scenario is invented.
| Incoming item | Treatment | Reason |
|---|---|---|
| Original signing notice | New event and source | First inspected signing evidence |
| Official language companion | Same event; new source object | Different file, same occurrence |
| News repost of notice | Same event; secondary source | Distribution is not new progress |
| Corrected signing date | Link correction; inspect impact | Version relationship changes a field |
| Later completion notice | New stage linked to transaction | Different occurrence and status |
The output might be one transaction with signing and completion stages, each connected to its relevant source objects. It should not be five unrelated events. Nor should it be a single overwritten row that loses the correction and publication history.
Preserve uncertain matches
When two documents might refer to the same project but the identity is unclear, mark the relationship provisional. Keep both sources visible and write the missing matching evidence. A provisional duplicate should not be deleted merely to clean a dashboard or reduce counts.
If working manually, record a merge note: objects grouped, reason, date, and how to reverse the decision. If the match later proves wrong, split the event and retain the source relationships. Reversible grouping is especially useful when project names are reused across phases or subsidiaries.
Verify counts with a reconstruction test
Select one grouped event and list every source attached to it. Can you explain why each belongs and whether any source introduces a new stage? Then select one document and list the events it supports. This two-direction test exposes both overcounting and accidental loss of distinct events.
Count independent observations separately from distribution copies. Two official language versions can help language verification without becoming two independent confirmations. A separate regulator decision may add independent evidence, but only for what that decision actually establishes.
Limits and common mistakes
Event grouping is a research organization choice. It does not prove causation, legal equivalence, or the accuracy of the underlying statements. Record ambiguous scope and avoid overconfident merges when entity relationships remain unresolved.
Common errors include deduplicating by title only, deleting translations, treating every filing as one event, and merging signing with completion. Another is using a content hash as an event key: identical bytes identify copies, while distinct files can still describe the same event. Preserve both file-level and meaning-level relationships.
A final record check
Keep a reversible merge note when event counts matter. State which source IDs were grouped and which event stage they share. If a later passage shows that one document concerns a separate facility or subsidiary, split the group and retain the reason. A clean count is less useful than a count whose classification can be inspected and corrected.
FAQ
Should I delete a duplicate file?
Only after preserving its identity and relationship if it matters to the research trail. Deduplication should reduce repeated work without hiding where evidence came from.
Can one document belong to multiple events?
Yes. A filing can discuss several unrelated developments. Link the relevant passages to their events instead of forcing the entire document into one category.