Gedcom Import Purge

Gedcom Import Purge
Summary	Causes the Gedcom Import Process to delete Medialinks records that have been deleted from the source database, and to retain some Places records that otherwise would be purged.
Validation
Mod Updated	23 Sep 2023
Download link	v14.0.0.5h TNG 14.0 (See the Revision History)
Download stats
Author(s)	Robin Richmond
Homepage	this page
Mod Support	My Mod Support form or TNG Community Forums
Contact Developer	My Mod Support form
Latest Mod	14.0.0.5h
Min TNG V	10.1
Max TNG V	14.0.3
Files modified	admin_dataimport.php, admin_gedimport.php, gedimport_trees.php, gedimport_misc.php, admin_editmedia.php, js/mediafind.js, js/mediautils.js, micro_medialinks.php, ajx_updateorder.php, admin_importconfig.php, admin_updateimportconfig, English/data_help.php.
Related Mods	Gedcom Converter, Depends on Mod Settings Blocks
Notes

Purpose of the Mod

This mod could have been called "Gedcom Import Medialinks" because its primary purpose is to make sure that old no-longer-valid imported Medialinks do not hang around and disrupt TNG pages after they have been deleted from the source database. To recognize imported Medialinks, it flags all Medialinks that are created through a Gedcom import. See the Purging Medialinks section below for details.

This mod also retains certain Places that are purged by the native TNG code.

This mod and its functionality are meaningful only when a Gedcom import replaces "All current data" in a tree. Data can be loaded from Gedcom files incrementally; that is, multiple Gedcom files can be loaded into one tree. But incremental Gedcom imports can make relationships among objects in a tree very difficult to define and maintain. Thus replacing "All current data" is the most straightforward and common way of doing Gedcom imports in TNG.

Significantly, to replace "All current data" is something of a misnomer. It does mean to replace all People, Families, Events, Notes, Citations, Sources, and Repositories, by purging all such records from the tree, and then replacing them with whatever is in the Gedcom file. (In this context, to "purge" a record basically means to "delete the record, with the expectation that the deleted records will be reloaded from the Gedcom file".)

But "All current data" excludes Media items and Medialinks, because

• Any media item can be linked to an object (a Person, Family, Event, Citation, Source, Repository, or Place) in a different tree,
• Medialinks to places cannot be defined in a Gedcom file. If they exist, they must have been created with TNG data entry forms,
• Some TNG admins define Media items and Medialinks independently of Gedcom imports, that is, strictly through TNG data entry forms, and
• (Without this mod), there is no way to distinguish between Medialinks created through TNG data entry forms, and Medialinks created through Gedcom Imports.

Details

Without this mod, when the Gedcom Import Process starts (and "All current data" is being replaced), it

1. Purges all People, Families, Events, Notes, Citations, Sources, Repositories, and Citation Medialinks in the tree being loaded, assuming it will replace them with data in the Gedcom file,
2. Leaves all Media items intact,
3. Purges Place records that it believes do not contain data that cannot be replaced by Gedcom data, again, assuming that it can replace them with data in the Gedcom file, and
4. Leaves all Medialinks, except Citation Medialinks, intact.

This mod leaves actions #1 and #2 just above as they are, tweaks #3, and significantly changes #4.

Purging Medialinks   [Show Details]

[Hide details]

The native Gedcom import process does purge Citation Medialinks, because it purges all Citation, at which point all Citation Medialinks become invalid. But it does not purge other Medialinks, because we know that some Media items and Medialinks that we want to keep' may not be loaded from the Gedcom file. For instance:

• Many TNG admins don't count on GEDCOM to load all ofdel their media items and links; they use TNG data entry forms to add some Media items and Medialinks, which cannot be replaced by Gedcom data.
• Gedcom does not support Medialinks to Places, so any such links in the database must have been created with the TNG data entry forms, and cannot be replaced by Gedcom data.

However, as a result, Media links that have been deleted from the source database that produced the Gedcom file will remain in TNG indefinitely, unless deleted manually

Consequently, the Gedcom Import Purge mod

1. Installs a setup program that create a new database field in the Medialinks table to keep track of Medialinks that are created by the Gedcom Import process.
2. Modifies the Gedcom Import process to:
   1. Use the new 'Imported from Gedcom" field to flag the Medialinks it creates and
   2. Purge (delete) Medialinks that are flagged as having been created by a Gedcom Import,
3. Modifies the Medialinks section of the Edit Media form to display and edit the 'Imported from Gedcom' flag.
   • This mod creates a checkbox for each Medialink in a Media Edit page so that you can see whether a given mMdialink came from a Gedcom Import or from TNG data entry. The checkbox status can be changed, though you are not likely to want to change it.

[Hide details]

Purging Places   [Show Details]

The Gedcom Import is more careful with Places, since Places can contain data that cannot be (or is often not) loaded from Gedcom files. Thus, when the native Gedcom import purges Place records, it retains records that contain latitude or longitude values or descriptive notes. Still, it ignores placelevels and Medialinks to Places, both of which cannot be loaded from Gedcom files.

Consequently, the Gedcom Import Purge mod modifies the Places purge at the start of the Gedcom Import process so that it retains:

• All Places if there are multiple trees but just one Place list,
• Places with a longitude, latitude, note, or placelevel value, and
• Places that are pointed to by a Medialink.

[Hide details]

Pre-Existing Medialinks   [Show Details]

[Hide details]

Starting with the second Gedcom import after Gedcom Import Purge is installed, the mod prevents the Gedcom Import process from leaving invalid Gedcom-import-created medialinks behind. It does so by purging all medialinks that are flagged as having been created by a Gedcom import. It is safe to purge old medialinks in this circumstance because

1. Medialinks that were not created by a Gedcom import will not be purged, since thay have not been flagged, and
2. All valid medialinks that were purged should be re-created by the Gedcom Import process.

The mod does not

• Remove invalid Medialinksthat already existed when Gedcom Import Purge was installed, nor
• Prevent Medialinks that were defined through TNG data entry screens from being invalidated by a Gedcom import.

Both of these conditions will be addressed in a future version of Gedcom Import Purge through a new "Secondary Process". Until that feature is added to the mod, you will have to use "manual" SQL queries to delete problematic Medialinks. You can execute some of these queries with the TNG reporting tool. Others will need to be executed through a separate program such as sqlMyAdmin.

• If you have never created any Medialinksthrough TNG data entry, you can confidently remove invalid medialinks by removing 'all' medialinks:
  • If you have multiple trees, then use the SQL command
    DELETE tng_medialinks WHERE gedcom='xx' (where 'xx' is your tree abbreviation,
  • If you have only one tree, you can clean up your Medialinks table more thoroughly with the SQL command
    TRUNCATE tng_medialinks
• If you created Medialinks to Places through TNG data entry, but have not used TNG data entry to define any other Medialinks, you can clean up your Medialinks table with one of these SQL commands:
  • Before TNGv12: DELETE tng_medialinks WHERE gedcom='xx' AND linktype!='L' AND linktype!=""
  • IN TNGv12.0 and after: DELETE tng_medialinks WHERE gedcom='xx' AND linktype!='L'
• If you have entered Medialink sother than Place Medialinks through TNG data entry, well, your options are limited. It is extremely difficult for to define SQL query that will remove invalid Medialinks while also preserving Medialinks defined through data entry. If that's what you need to do, then, practically speaking, you'll need to make note of the Medialinks that you entered manually, issue one of the SQL commands above, and then redefine those Medialinks after you do the first Gedcom Import after you install the mod. (At that point, you may want to define those medialinks in your source database rather than in Gedcom.)

[Hide details]

Citation Medialinks   [Show Details]

[Hide details]

In TNG, a Citation Medialink is a link from a Media item to a Citation, and is defined in the Medialinks table. In Gedcom files, a Citation is a Source reference in an Event record inside a Person or Family Record. And a Citation Medialink is a Media item reference within a Source Citation. Thus, though TNG and Gedcom medialinks both define many-to-many relationships between Media items and Citations, those relations are defined in opposite directions in TNG and in Gedcom files.

When Family Tree Maker (FTM) Gedcom files contain media information, they define Citation Medialinks, and do not automatically link the related Media items to the People and Families in which the Citation Medialinks occur. But until version 12, TNG didn't support Citation Medialinks at all. That is, it did not import them, had no way to enter them through TNG data entry, and ignored them in Gedcom imports. Consequently - in the absence of a Gedcom converter - many Media items would not be represented in TNG Person Profiles or Family Charts. Thus, TNG Gedcom converters (all of which were initially created to handle invalid aspects) took note of Citation Medialinks, and created Person and Family medialinks to the Media items referenced by all of their Citation Medialinks.

Starting at some point within TNGv12 (not TNGv12.0, but an incremental version or two later), TNG's Gedcom Import did (and does) read Citation Medialinks from Gedcom file, and create its own Citation Medialinks. (In addition, TNG administrators can now define Citation Medialinks through the Media Links section of the Media Edit page, but that's not really relevant to this discussion.) Aside from the Media Edit page, Citation Medialinks are presented to users in two places:

1. In the Person Profile, as hyperlinked thumbnail images in Source Citations,
2. In the "Linked to" column of the end-user Media item search/results page, browsemedia.php, and
3. In the "Linked to" column of the administrative Media item search/results page, admin_media.php.

Note that the native deletion of Citation Medialinks was implemented after TNG began importing Citation Medialinks. Thus, there was a short period of time in which old Citiation Medialinks did hang around and (sometimes) get in the way after a Gedcom Import. Since that problem no longer exists, there is not much point in one of the Gedcom Import Purge mod's 'features'; that is, the checkbox on the Gedcom Import kick-off form that allows the user to suppress the creation of Citation Medialinks. At this point, the only reason I can think of to suppress the creation of Citation Medialinks is to suppress the Media item thumbnails in the Sources section of the Person Profile. (And that's not very compelling.)

[Hide details]

Other Mods

I do not know of any conflicts with other mods.

Related Mods

1. The Mod Settings Blocks mod packages this mod's options into what I call a "Mod Settings Block", which is a set of form fields that formatted and labeled so that it stands out among other options in an Admin>>Setting form. Mod Settings Blocks is a prerequisite for this mod; it must be installed before this mod can be installed.
2. An Inner Mod Menu, which provides handy links from TNG pages directly to mod documentation and a mod's Mods Settings Block. (Inner Mod Menus are visible only to TNG Admins; never to end users.)
3. The Mods Used feature, which captures a list of mods that affected the code used to display a TNG page. (This is analogous to the Mod Manager Analyzer feature that reports which mods affect a given TNG file, but that applies at run-time.
4. Gedcom Converter installs by the Gedcom converter program that creates (in the Gedcom file) the Person and Family Medialinks described above that are important when importing Gedcom files that do not have Person and Family Medialinks to the Media items associated with Citation Medialinks. (This applies to Family Tree Maker Gedcoms, and perhaps others.)
5. Admin Media Search changes the way that TNG displays Medialinks, and it handles Citation Medialinks, which are a significant concern of this mod.
6. Gedcom Import Mediatype is related only in that it also affects the Gedcom Import kickoff form and the Gedcom Import process. Aspects of that mods are coordinated with this mod, but there are no dependencies between them.

Installation

Files Installed

This mod includes a typical mod subfolder, with the same base name as the mod's .cfg file. The subfolder is atypical in that no files in it are installed to working TNG folders. However, it does contain critical files that contain the mod's language strings, which the mod does not add to standard cust_text.php files. Instead, the PHP code modified by this mod loads the language strings from files in the mod's languages/ subfolder. Read more about this technique.

This mod installs one file from the .fg file.
rrgedcomimportpurge_dbsetup.php is the utility program that creates the database field that is used to flag Gedcom-Import-created medialinks. The fieldname is createdfromgedcom and it is in the Medialinks table. This utility program is ordinarily run once, from a link in the Mod Manager, to define that database field. If you no longer intend to use the mod, the utility program can be run a second time to remove the createdfromgedcom field from the database.

[Show Standard Installation Process]

Requirements

• A working TNG installation.
• An installed current version of the Mod Manager.
• You should backup files listed in the panel on the right.

Procedure

1. Remove and delete previous version of this mod.
2. Backup the files updated by this mod. They are listed in the panel at the upper right.
3. Download the .zip file, Extract its .cfg file to the mods folder.
4. Follow the normal automated installation for Mod Manager, as shown in the example Mod Manager - Installing Config Files.

Problems?

1. Try using the Mod Manager Remove capability
2. Contact me through My Mod Support form.

[Hide Details]

Visualizations

In the Mod Manager, After Installation, showing the database setup hyperlink. You must follow the hyperlink shown above to run the setup program that creates the database field through which TNG remembers which media links were created by a Gedcom Import.

The Database setup program

The Gedcom Import Kickoff Form (Admin>>Import/Export) AFTER INSTALLATION:

New or changed elements on the Gedcom Import Kickoff form: The Inner Mod Menu, which pulls down from the "Mod Information" heading, is shown in its open state. This menu is visible only if you have installed the Inner Mod Menus mod. The "Import media if present" checkbox is native to TNG. This mod has an option that will check this checkbox by default. To re-create medialinks after they are purged, media information must be defined in the Gedcom file, and this flag must be checked. Leave this box unchecked if your Gedcom file does not have media information, or if you do not want to re-create purged medialinks. Two new checkboxes. The default status of each checkbox is determined by mod options. "Purge media links created by previous Gedcom imports." - Since the medialink purge is the primary purpose of the mod, you will probably always check this box. If you left the "Import media if present" checkbox unchecked, then you probably want to uncheck this box, too. "Suppress the creation of citation medialinks" - This is meaningful only if your Gedcom file contains media information 'and contains citation medialinks (that is, if it contains OBJE tags that are subordinate to SOUR tags in INDI or FAM records). But, as noted above, there's really not much point in suppressing citation medialinks. This checkbox is really just a legacy of a time when I though that it was important to suppress citation medialinks.

Message As the Gedcom Import starts: This message is visible only if you choose the "Old style import" checkbox (or, if you have installed Gedcom Import Monitor, the "Scrolling progress listing").

Remember that the Gedcom Import process will not purge any medialinks the first time you run it after installation, since the new database field hasn't yet been populated by flags that say that the Medialink was created by a Gedcom Import.

Media Links in the Admin>>Media>>Edit form The Medialinks table in the Media Edit form now has a column with a checkbox for the "From Gedcom" flag. It is checked for Media items that were loaded through a Gedcom Import process, and defaults to unchecked (and usually remains that way) when Media items are added within TNG.

The Mod Options - in a Mod Settings Block The Mod Settings Block for Gedcom Import Purge is at Admin>>Setup>>Import Settings. You can jump straight there from the "Change Mod Options" link in the Inner Mod Menu, which is shown in a visualization above. But, of course, the Inner Mod Menu will not be visible if you have not installed the Inner Mod Menus mod.

Language Strings

This mod includes a typical mod subfolder, with the same base name as the mod's .cfg file. The subfolder is atypical in that no files in it are installed to working TNG folders. However, it does contain critical files that contain the mod's language strings, which the mod does not add to standard cust_text.php files. Instead, the PHP code modified by this mod loads the language strings from files in the mod's languages/ subfolder. Read more about this technique.

Revision History

Mod Version	TNG Version	Date	Note
v14.0.0.5h	14	23Sep2023	Removed unneeded IMM code that Mod Settings Blocks 4
14.0.0.5g	14.0	20Mar2023	Upgrade to TNGv14
13.0.0.5e	13.0-13.0.1	2Nov2020	Tweaked on location's search text to accommodate a subtle change in TNGv13.0.1
13.0.0.5d	13	31Oct2020	Fixed the bug that prevented the database utility from deleting the new database field. Handled the SQL crashes that occurred in the Gedcom Import kickoff form the Media Edit page when the user had not yet created the new database field. The Gedcom Import kickoff form and the Cemetery edit page both now Display a message such as "The Gedcom Import Purge mod's database field has not been defined", and Provide a link to the Gedcom Import Purge field creation utility, and Work without the new database field (and without its functionality).
13.0.0.5c	13	26Oct2020	5c Fixed language strings in the database-field-creation utility 5b Fixed language strings in the mod settings block 5a Upgraded the mod to TNGv13. The mod now uses the Mod Settings Blocks and Inner Mod Menu mods rather than shared include file, and its language strings are in the mod subfolder.
12.0.0.5a	12.0-12.3	20Aug2020	Functional: Added a checkbox to the Gedcom Kickoff import form that can suppress the creation of citation medialinks. Does not purge places if there is more than one tree but only one places list, since purging places without regard to their use in other trees can break external reference to those places. Visual & Internal: Added a document.ready function in admin_importconfig.php to move a footnote from the bottom of the page to a position right below the text it refers to. The insertion of the Gedcom Import mods made that footnote too far from where it needs to be. Fixed two missing %end:% tags that break installation in TNGv12.3
12.0.0.5	12.0+	2Dec2019	Modified the Place purge process so that it does not purge places on sites that have multiple trees and have one shared Places list. Added an Inner Mod Menu, and moved the options to an Mod Settings Block Added a form field in the Medialinks Admin>>Media>>Edit form so that admin users can set or unset the "createdfromgedcom" (aka "From Gedcom") flag. Enhanced the message that, at the beginning of the Gedcom Import process, displays the counts of places and medialinks that were purged, and medialinks that were not purged. The Gedcom Import process generates a list of the places that were purged and recreated, and purged and not recreated. Added the "Suppress the creation of citation medialinks" checkbox on the Admin Gedcom Import kickoff form. Added code to suppress citation medialinks when the checkbox is checked. Adds a Save buttons to the left of the BodyText field in the Edit Media form, so that the user doesn't always have to scroll to the top or bottom of the form to finish.
12.0.0.4	12.0+	15May2018	No functional changes; made compatible with TNGv12.
10.1.0.3p	10.1-11.1	26Mar2017	Removed the second line from the cust_text.php target location search string
10.1.0.3	10.1-11.1	26Mar2017	A technical update that, mostly, just makes Show Mod Names optional, and avoids a installation conflict introduced by TNGv11.0.1. That is, this version omits the adminlib.php patch that was part of v2 of this mod, and changes this mod's database setup program so that it works independently of that adminlib.php patch (which was fixed in TNGv11.0.1).
10.1.0.2	10.1-11.0.1	25May2016	Removed a unneeded <script> element that incorrectly referenced an external file, and that - under rare circumstances I still don't understand - could cause the database setup program to log you out from your TNG session. Also fixes a similarly incorrect <script> element in adminlib.php.
10.1.0.1b	10.1-11.0	25Mar 2016	Fixed an error in a JavaScript warning. Cleaned up the code. Changed the Mod Parameters to strings rather than boolean values to be more tolerant of data entry errors.
10.1.0.1a	10.1-10.1.3	21Feb2016	No new end-user functionality. Now depends on Show Mod Names v2+.
10.1.0.1	10.1-10.1.3	6Feb2016	New mod.

Sites using this mod

If you download and install this mod, please add your site to the table below.

URL	User	Note	Mod-Version	TNG-Version	User-language
Robin Richmond's Genealogy Database - admin function; not visible.	Robin Richmond	Mod developer	14.0.0.5h	14.0	English
Hooley Family Links	Rick Hooley	Public/Private	See Here	See Here	EN
Pete's Research	TBirdUK	Public/Private	12.0.0.5	12.01+	English
Woking Family Tree Project	TBirdUK	Public/Private	12.0.0.5	12.01+	English
KKs Family History	Kathy Kult	Public/Private	14.0.0.5g	TNG 14.0.2	English
Dabbling in Genealogy	Jeff Rueger	Public/Private	13.0.0.5e	13.1.2	English