Gedcom Import Purge

From TNG_Wiki
Jump to: navigation, search
Gedcom Import Purge
Summary Causes the Gedcom Import Process to delete Medialinks records that have been deleted from the source database, and to retain some Places records that otherwise would be purged.
Validation
Mod Updated 28 Aug 2020
Download link v12.0.0.5a.zip
TNG 12.0
Older versions can be downloaded from the Revision History.
Download stats
Author(s) Robin Richmond
Homepage this page
Mod Support My Mod Support form or TNG Community Forums
Contact Developer My Mod Support form
Latest Mod v12.0.0.5a
Min TNG V 10.1
Max TNG V 12.3
Files modified
admin_dataimport.php, admin_gedimport.php, gedimport_trees.php, gedimport_misc.php, admin_editmedia.php, js/mediafind.js, js/mediautils.js, micro_medialinks.php, ajx_updateorder.php, admin_importconfig.php, admin_updateimportconfig, English/data_help.php, English cust_text.php.


Installs: rradminbranches_ajx.php, rradminbranches_lib.php.


Installs shared files: rrshared_innermodmenu2.php, rrshared_modsettingsblocks2.php, img/rrshared_wikilogo.gif
Related Mods
Notes

Purpose of the Mod

This primary purpose of this mod is to make sure that old imported Medialinks do not hang around and disrupt TNG pages after they have been deleted from the source database. To recognize imported Medialinks, it flags all Medialinks that are created through a gedcom import. See the Purging Medialinks section below for details.

It also retains certain Places that are purged by the native TNG code.

It is important to realize that this mod purges Medialinks that were created by a previous Gedcom Import and that were flagged during that Gedcom Import. That is, the mod purges old Medialinks that were created after the mod was installed, but cannot recognize and purge old Medialinks that are already causing trouble when the mod is installed. See "[#[Pre-Existing_Medialinks|Pre-Existing Medialinks]] below.

Without this mod, the Gedcom Import Process:

  • Leaves all Medialinks intact, including those that have been removed from the source database,
  • May purge Place records or Places data that are not replaced by the Gedcom Import.

Details

Purging Places

[Show Details]

When the "pristine" (unchanged by a mod) Gedcom Import process starts - and "all current data" is being replaced - it deletes all People, Families, Events, Notes, Citations, Sources, and Repositories in the tree being loaded, assuming it will replace them with data in the Gedcom file.

It is more careful with Places, since Places can contain data that cannot be (or is oftent not) loaded from Gedcom. Thus, when it purges Place records, it retains records that contain latitude or longitude values or descriptive notes. Still, it ignores placelevels and medialinks to Places, both of which cannot be loaded from Gedcom files.

Consequently, the Gedcom Import Purge mod modifies the Places purge at the start of the Gedcom Import process so that it retains:

  • All Places when there are multiple trees, but just one Place list,
  • Places with a longitude, latitude, note, or placelevel value, and
  • Places with a Medialink.
[Hide details]

Purging Medialinks

[Show Details]
[Hide details]

When the Gedcom Import Process purges data from the tree being loaded, it does not delete any Medialinks, presumably because

  • Many TNG admins don't count on GEDCOM to load (all) of their media items and links; they manually add other media items and links, and
  • Gedcom does not support medialinks for places or cemeteries, and any such links in the database must have been created manually, using the TNG editors

However, as a result, Media links that have been deleted from the source database (that produced the Gedcom) will remain in TNG indefinitely, unless deleted manually

Consequently, the Gedcom Import Purge mod

  1. Installs a setup program that create a new database field named "createdbygedcom" for keeping track of Medialinks that are created by the Gedcom Import Process.
  2. Modifies the Gedcom Import process to:
    1. Use the new "createdbygedcom" Medialinks field to flag the Medialinks it creates and
    2. Purge (delete) Medialinks that are flagged as having been created by a Gedcom Import - but only if the user checks the new "Purge media links created by previous Gedcom imports" checkbox in the Gedcom Import kickoff form,
  3. Modifies the Medialinks section of the Edit Media form to display and edit the "createdbygedcom" flag. The flag is turned off when new Medialinks are created via TNG data entry
  • This mod creates a checkbox for each Medialink in a Media Edit page so that you can see whether a given medialink came from a Gedcom Import or from TNG data entry. The checkbox status can be changed, though you are not likely to want to change it, unless you have created a Medialink in your source database that duplicates a Medialink you have created through TNG data entry.
[Hide details]

Pre-Existing Medialinks

[Show Details]
[Hide details]

Medialinks that already exist before this mod is installed will not be purged by the mod for the same reason that they are not purged by the native TNG code. That is, when it is first installed, the mod cannot tell the difference between old Medialinks that were created by a previous Gedcom import and those that were created through TNG data entry. If you have not created Medialinks through TNG data entry, or if you are content to delete those medialinks, then, before running the first Gedcom import after you install the mod, you can clean up those old medialinks by using a tool such as phpMyAdmin

  1. To delete all medialinks in your tree, with a SQL statement such as DELETE tng_medialinks WHERE gedcom='xx' (where 'xx' is your tree abbreviation,
  2. If you have only one tree, you can more thoroughly clean up the Medialinks table by truncating it with the SQL statement TRUNCATE tng_medialinks

On the other hand, if your database has Medialinks that you do want to preserve, well, you have to be able to identify those Medialinks when you purge the table. If the only media items that you have added through TNG data entry are links to Places (which cannot be created through a Gedcom import), then you can use a variant of the DELETE statement suggested above:

  • Before TNGv12: DELETE tng_medialinks WHERE gedcom='xx' AND linktype!='L' AND linktype!=
  • IN TNGv12.0 and after: DELETE tng_medialinks WHERE gedcom='xx' AND linktype!='L'

If you have created other Medialinks, well, those medialinks cannot easily by identified in a purge. If you have created medialinks to only a very few different media items, then my best suggestion is to

  1. Launch the Media Edit page for those media items, and make note of their mediaIDs,
  2. Create a comma-delimted list of those mediaIDs.
  3. Paste that list in to this SQL query: DELETE tng_medialinks WHERE mediaID NOT IN "({your list})".
    • For example, if you only need to preserve Medialinks to media items with mediaIDs 350 and 580 the command would be <code style="white-space:nowrap">DELETE tng_medialinks WHERE mediaID NOT IN "(350,580)"
[Hide details]

Citation Medialinks

[Show Details]
[Hide details]

Until version 12, TNG's Gedcom Import process ignored Citation Medialinks, basically assuming (I presume) that the Gedcom medialinks were repeated in the relevant lower-level source and person or family records. In fact, because TNG did not handle citation medialinks, the "Modify Gedcom files for TNG" option in the Gedcom Converter mod specifically creates those lower-level medialinks when it encounters citation medialinks.

Now that the TNG Gedcom Import process does create citation medialinks, Gedcom files that have citation medialinks and lower-level medialinks generate redundant medialinks. In addition, (in TNGv12*) not all TNG program that display medialinks handle citation medialinks cleanly.

To handle redundant Citation Medialinks;

  1. If you use the Gedcom Converter,
    • You can remove Citation Medialinks through the Gedcom Converter option
      Suppress Citation Medialinks
    • If you want Citation Medialinks to be defined for lower-level records (i.e. the Event and the Person or Family) and your source Gedcom file does not already have such medialinks, then you can create them through the Gedcom Converter option
      Create Individual, Family, and Source medialinks from Citation Medialinks
  2. In the Gedcom Import kick-off page (whether or not you have used the Gedcom Converter) you can control the creation of Citation Medialinks through option (created by the Gedcom Purge mod)
    Supress the creation of Citation Medialinks
    • Of course, you would never want to use this option unless the Gedcom file that you import contains lower-level Medialinks that substitute for those Citation Medialinks

Note that neither the Gedcom Converter nor the Gedcom Import process can keep only the Citation Medialinks, and suppress the matching lower-level Medialinks.

[Hide details]

Related Mods

I do not know of any conflicts with other mods.

  • Admin Media Search - This mod changes the way that TNG displays all medialinks, and it handles citation medialinks, which are a significant concern of this mod.
  • Gedcom Import Mediatype is related only in that it also affects the Gedcom Import kickoff form and the Gedcom Import process. Aspects of that mods are coordinated with this mod, but there are 'no dependencies between them.
  • Inner Mod Menus This optional, but for me, very important, mod displays a drop-down menu at the right end of the Gedcom Import kickoff-form's "Inner Menu" (where the "Help for this Area" is at the left end). That drop-down menu provides links to the mod's Wiki article and to the mod's options.
  • If the optional Show Mod Names mod is installed, this mod will utilize its functionality.

Installation

Files Installed

This mod has two mod subfolders:

  1. gedcom_import_purge_v12.0.0.5a - The subfolder that you expect to exist for any mod that installs files. It contains
    • rrgedcomimportpurge_dbsetup.php This is ordinarily run once, from a link in the Mod Manager, to define the new Medialinks table field that keeps track of medialinks that are created by the Gedcom Import. The field is called createdfromgedcom.
  2. The shared folder RR-shared_mod_includes_v12.0.0.4, which contains files that support mod option management in many of my mods:
    • rrshared_modsettingsblocks4.php - Mod Settings Blocks
    • img/rrshared_wikilogo.gif - An icon that links to a mod's TNG Wiki page.

The shared folder is packaged with several mods, each of which may install some or all of its files. Once a file from the shared folder has been installed (by any mod), that file will not be overwritten or removed by the subsequent installation or un-installation of any mod (including this one), nor will its presence generate any Mod Manager errors.

If you unzip mod distribution files directly into your mods folder, then the presence of this second subfolder and the installation of its files should be invisible to you. But if you generally copy mod subfolders to your mods folder, you need to make sure to copy the folder RR-shared_mod_includes_v12.0.0.4 (well, unless you are confident that it is already in your mods folder because it was part of another mod).

[Show Installation Details]

Requirements

  • A working TNG installation.
  • An installed current version of the Mod Manager.
  • You should backup files listed in the panel on the right.

Procedure

  1. Remove and delete previous version of this mod.
  2. Backup the files updated by this mod. They are listed in the panel at the upper right.
  3. Download the .zip file, Extract its .cfg file to the mods folder.
  4. Follow the normal automated installation for Mod Manager, as shown in the example Mod Manager - Installing Config Files.

Problems?

  1. Try using the Mod Manager Remove capability
  2. Contact me through My Mod Support form.
[Hide Details]

Visualizations

In the Mod Manager, After Installation, showing the database setup hyperlink.
Gedcom import purge-modman.png

You must follow the hyperlink shown above to run the setup program that creates the database field through which TNG remembers which media links were created by a Gedcom Import.

The Database setup program

Gedcom import purge-createdb.png

The Gedcom Import Kickoff Form (Admin>>Import/Export) AFTER INSTALLATION:
Gedcom import purge-after1.jpg

New or changed elements on the Gedcom Import Kickoff form:

  1. The Inner Mod Menu, which pulls down from the "Mod Information" heading, is shown in its open state.
  2. A mod option determines whether the "Import media if present" checkbox should be checked. For the Gedcom Import Purge mod to be useful in a Gedcom Import, this checkbox should be checked.
  3. Two new checkboxes. The default status of each checkbox is determined by mod options.
    1. "Purge media links..." - The primary purpose of this mod.
    2. "Suppress the creation of citation medialinks" - It all depends on your Gedcom file and how you want to handle citation medialinks.
Message As the Gedcom Import starts:

This message is visible only if you choose the "Old style import" checkbox (or, if you have installed Gedcom Import Monitor, the "Scrolling progress listing").

Gedcom import purge-purge.png

Note that the Gedcom Import process will not purge any medialinks the first time you run it after installation, since the new database field hasn't yet been populated by flags that say that the Medialink was created by a Gedcom Import.

Media Links in the Admin>>Media>>Edit form

The Medialinks table in the Media Edit form now has a column with a checkbox for the "From Gedcom" flag. It is checked for Media items that were loaded through a Gedcom Import process, and defaults to unchecked (and usually remains that way) when Media items are added within TNG.
Gedcom import purge-after-medialinks.jpg

The Mod Options - in a Mod Settings Block

The Mod Settings Block for Gedcom Import Purge is at Admin>>Setup>>Import Settings. You can jump straight there from the "Change Mod Options" link in the Inner Mod Menu.
Gedcom import purge-after-modsettingsblock.jpg

Revision History

Mod Version TNG Version Date Note
12.0.0.5a 12.0-12.3 20Aug2020 Functional:
  1. Added a checkbox to the Gedcom Kickoff import form that can suppress the creation of citation medialinks.
  2. Does not purge places if there is more than one tree but only one places list, since purging places without regard to their use in other trees can break external reference to those places.

Visual & Internal:

  1. Added a document.ready function in admin_importconfig.php to move a footnote from the bottom of the page to a position right below the text it refers to. The insertion of the Gedcom Import mods made that footnote too far from where it needs to be.
  2. Fixed two missing %end:% tags that break installation in TNGv12.3
12.0.0.5 12.0+ 2Dec2019
  1. Modified the Place purge process so that it does not purge places on sites that have multiple trees and have one shared Places list.
  2. Added an Inner Mod Menu, and moved the options to an Mod Settings Block
  3. Added a form field in the Medialinks Admin>>Media>>Edit form so that admin users can set or unset the "createdfromgedcom" (aka "From Gedcom") flag.
  4. Enhanced the message that, at the beginning of the Gedcom Import process, displays the counts of places and medialinks that were purged, and medialinks that were not purged.
  5. The Gedcom Import process generates a list of the places that were purged and recreated, and purged and not recreated.
  6. Added the "Suppress the creation of citation medialinks" checkbox on the Admin Gedcom Import kickoff form.
  7. Added code to suppress citation medialinks when the checkbox is checked.
  8. Adds a Save buttons to the left of the BodyText field in the Edit Media form, so that the user doesn't always have to scroll to the top or bottom of the form to finish.
12.0.0.4 12.0+ 15May2018 No functional changes; made compatible with TNGv12.
10.1.0.3p 10.1-11.1 26Mar2017 Removed the second line from the cust_text.php target location search string
10.1.0.3 10.1-11.1 26Mar2017 A technical update that, mostly, just makes Show Mod Names optional, and avoids a installation conflict introduced by TNGv11.0.1. That is, this version omits the adminlib.php patch that was part of v2 of this mod, and changes this mod's database setup program so that it works independently of that adminlib.php patch (which was fixed in TNGv11.0.1).
10.1.0.2 10.1-11.0.1 25May2016 Removed a unneeded <script> element that incorrectly referenced an external file, and that - under rare circumstances I still don't understand - could cause the database setup program to log you out from your TNG session. Also fixes a similarly incorrect <script> element in adminlib.php.
10.1.0.1b 10.1-11.0 25Mar 2016 Fixed an error in a JavaScript warning. Cleaned up the code. Changed the Mod Parameters to strings rather than boolean values to be more tolerant of data entry errors.
10.1.0.1a 10.1-10.1.3 21Feb2016 No new end-user functionality. Now depends on Show Mod Names v2+.
10.1.0.1 10.1-10.1.3 6Feb2016 New mod.

Sites using this mod

If you download and install this mod, please add your site to the table below.

URL User Note Mod-Version TNG-Version User-language
Robin Richmond's Genealogy Database - admin function; not visible. Robin Richmond Mod developer 12.0.0.5a 12.3 English
Hooley Family Links Rick Hooley Public/Private See Here See Here EN
Pete's Research TBirdUK Public/Private 12.0.0.5 12.01+ English
Woking Family Tree Project TBirdUK Public/Private 12.0.0.5 12.01+ English
KKs Family History Kathy Kult Public/Private 12.0.0.4 13.0 English