View source for Gedcom Import Purge

{{TNGmod
| mod_name        =  Gedcom Import Purge
|  mod_summary     = Causes the Gedcom Import Process to delete Medialinks records that have been deleted from the source database, and to retain some Places records that otherwise would be purged.
| mod_validation  = 
| mod_last_update = 2 Dec 2019
| download_link   =  [[Media:gedcom_import_purge_v12.0.0.5.zip|v12.0.0.5.zip]]{{Tv120}}
<div style='font-size:90%'>Older versions can be downloaded from the [[#Revision_History|Revision History]].</div>
| download_stats  =
| mod_author      = [[User:Robinrichm|Robin Richmond]]
| mod_url         = ''this page''
| mod_contact     = [http://www.robinrichmond.com/family/mod_support.php My Mod Support form]
| mod_support     = [http://www.robinrichmond.com/family/mod_support.php My Mod Support form] or [http://tng.community/ TNG Community Forums]
| mod_version     = v12.0.0.5
| min_TNG_ver     = 10.1
| max_TNG_ver     = at least 12.2
|   TNG_file_list   = admin_dataimport.php, admin_gedimport.php, gedimport_trees.php, gedimport_misc.php, admin_editmedia.php, js/mediafind.js, js/mediautils.js, micro_medialinks.php, ajx_updateorder.php,  admin_importconfig.php, admin_updateimportconfig, English/data_help.php, English cust_text.php. 
<br>Installs: rradminbranches_ajx.php, rradminbranches_lib.php.
<br>Installs shared files: rrshared_innermodmenu2.php, rrshared_modsettingsblocks2.php, img/rrshared_wikilogo.gif
| related_mods   =  [[Admin Media Search]], [[Gedcom Import Mediatype]], [[Gedcom Import Monitor]], [[Show Mod Names]]
| notes    =   
}}== Purpose of the Mod ==
This mod changes the Gedcom Import process to
# Add a flag (a new database field) to Medialinks that are created by Gedcom Imports,
# Optionally, purge Medialinks that have that flag, but retains Medialinks without the flag (i.e. that were created through TNG data entry), and
# Retain certain Places that are purged by the native code.
{| style="margin-right:0.5 em;" align="right" 
| __TOC__
|}
Without this mod, the Gedcom Import Process:
* Leaves ''all'' Medialinks intact, including those that have been removed from the source database,
* May purge Place records or Places data that is not replaced by the Gedcom Import.

== Details ==
=== Purging Places ===
<!-- *** BEGIN DOUBLE TOGGLE --><div class="mw-collapsible mw-customtoggle-places1 mw-customtoggle-places2" id='mw-customcollapsible-places1' style="text-decoration:underline;color:#0645ad;">&#91;Show Details&#93;</div>
<div class='mw-collapsible mw-collapsible-content mw-collapsed' id='mw-customcollapsible-places2' style="border:thin solid grey;"><!-- BEGIN TOGGLED CONTENT -->
When the "pristine" (unchanged by a mod) Gedcom Import process starts - and "all current data" is being replaced - it deletes all People, Families, Events, Notes, Citations, Sources, and Repositories in the tree being loaded, assuming it will replace them with data in the Gedcom file.

It is more careful with Places, since Places can contain data that cannot be (or is oftent not) loaded from Gedcom. Thus, when it purges Place records, it retains records that contain latitude or longitude values or descriptive notes. Still, it ignores placelevels and medialinks to Places, both of which cannot be loaded from Gedcom files.

Consequently, the Gedcom Import Purge mod modifies the Places purge at the start of the Gedcom Import process so that it retains:
* All Places when there are multiple trees, but just one Place list,
* Places with a longitude, latitude, note, or placelevel value, and
* Places with a Medialink.
<!-- BUTTON AT THE BOTTOM OF TOGGLED CONTENT --><div class="mw-customtoggle-places1 mw-customtoggle-places2" style="text-decoration:underline;color:#0645ad;text-align:right;">&#91;Hide details&#93;</div>
</div><!-- END TOGGLED CONTENT and END DOUBLE TOGGLE -->

=== Purging Medialinks ===
<!-- *** BEGIN DOUBLE TOGGLE --><div class="mw-collapsible mw-customtoggle-mlinks1 mw-customtoggle-mlinks2" id='mw-customcollapsible-mlinks1' style="text-decoration:underline;color:#0645ad;">&#91;Show Details&#93;</div>
<div class='mw-collapsible mw-collapsible-content mw-collapsed' id='mw-customcollapsible-mlinks2' style="border:thin solid grey;"><!-- BEGIN TOGGLED CONTENT -->
<!-- BUTTON AT THE TOP OF TOGGLED CONTENT --><div class="mw-customtoggle-mlinks1 mw-customtoggle-mlinks2" style="padding-left:2em;text-decoration:underline;color:#0645ad;float:right;">&#91;Hide details&#93;</div>
When the Gedcom Import Process purges data from the tree being loaded, it does not delete any Medialinks, presumably because
* Many TNG admins don't count on GEDCOM to load (all) of their media items and links; they manually add other media items and links, and
* Gedcom does not support medialinks for places or cemeteries, and any such links in the database must have been created manually, using the TNG editors
However, as a result, Media links that have been deleted from the source database (that produced the Gedcom) will remain in TNG indefinitely, unless deleted manually

Consequently, the Gedcom Import Purge mod
# Installs a setup program that create a new database field named "createdbygedcom" for keeping track of Medialinks that are created by the Gedcom Import Process.
# Modifies the Gedcom Import process to:
## Use the new "createdbygedcom" Medialinks field to flag the Medialinks it creates and
## Purge (delete) Medialinks that are flagged as having been created by a Gedcom Import - but only if the user checks the new "Purge media links created by previous Gedcom imports" checkbox in the Gedcom Import kickoff form,
# Modifies the Medialinks section of the Edit Media form to display and edit the "createdbygedcom" flag. The flag is turned off when new Medialinks are created via TNG data entry, and, from that point, can be changed, though it is not likely to be changed.
<!-- BUTTON AT THE BOTTOM OF TOGGLED CONTENT --><div class="mw-customtoggle-mlinks1 mw-customtoggle-mlinks2" style="text-decoration:underline;color:#0645ad;text-align:right;">&#91;Hide details&#93;</div>
</div><!-- END TOGGLED CONTENT and END DOUBLE TOGGLE -->

=== Citation Medialinks ===
<!-- *** BEGIN DOUBLE TOGGLE --><div class="mw-collapsible mw-customtoggle-cm1 mw-customtoggle-cm2" id='mw-customcollapsible-cm1' style="text-decoration:underline;color:#0645ad;">&#91;Show Details&#93;</div>
<div class='mw-collapsible mw-collapsible-content mw-collapsed' id='mw-customcollapsible-cm2' style="border:thin solid grey;"><!-- BEGIN TOGGLED CONTENT -->
<!-- BUTTON AT THE TOP OF TOGGLED CONTENT --><div class="mw-customtoggle-cm1 mw-customtoggle-cm2" style="text-decoration:underline;color:#0645ad;float:right;">&#91;Hide details&#93;</div>
Until version 12, TNG's Gedcom Import process ignored Citation Medialinks, basically assuming (I presume) that the Gedcom medialinks were repeated in the relevant higher-level source and person or family records. In fact, because TNG did not handle citation medialinks, the "Modify Gedcom files for TNG" option in the Gedcom Converter mod specifically creates those higher-level medialinks when it encounters citation medialinks.

Now that the TNG Gedcom Import process does create citation medialinks, Gedcom files that have citation medialinks ''and'' higher-level medialinks generate redundant medialinks. In addition, (as of TNGv12.2) not all TNG program that display medialinks handle citation medialinks cleanly.

Consequently, the Gedcom Import Purge mod 
# Installs a setup program that create a new database field named "createdbygedcom" for keeping track of Medialinks that are created by the Gedcom Import Process, and
# Modifies the Gedcom Import process to:
## Use the new "createdbygedcom" Medialinks field to flag the Medialinks it creates
## Purge (delete) Medialinks that are flagged as having been created by a Gedcom Import - if the user checks the new "Purge media links created by previous Gedcom imports" checkbox in the Gedcom Import kickoff form,
## Modify the Places purge so that it retains:
##* All Places when there are multiple trees, but just one Place list,
##* Places with a longitude, latitude, note, or placelevel value, or
##* Places with a Medialink.
## Ignore Citation Medialinks - if the user checks the new "Ignore Citation Medialinks" checkbox in the Gedcom Import kickoff form.

Some possible next steps:
# The Gedcom Gedcom Converter mod should implement an option ''not to create'' those higher-level medialinks, and/or an option to remove the citation medialinks if it creates higher-level medialinks.
# The Gedcom Import Purge mod should (perhaps) have an option to keep citation medialinks, and to ignore redundant higher-level medialinks.
#* (Doing so might require the converter to look ahead in the Gedcom file to determine whether such redundant links exist, so it may not be practical to implement this feature.)

Finally, note that it is common for Gedcom files to have direct links from media items to sources,people, and places, independently of citation medialinks. Thus, when I speak of suppressing or removing higher-level medialinks, I am referring to the redundant ones that duplicate a citation medialink, not to the direct medialinks.
<!-- BUTTON AT THE BOTTOM OF TOGGLED CONTENT --><div class="mw-customtoggle-cm1 mw-customtoggle-cm2" style="text-decoration:underline;color:#0645ad;text-align:right;">&#91;Hide details&#93;</div>
</div><!-- END TOGGLED CONTENT and END DOUBLE TOGGLE -->

== Related Mods ==
I do not know of any conflicts with other mods.
* [[Admin Media Search]] - This mod changes the way that TNG displays all medialinks, and it handles citation medialinks, which are a significant concern of this mod.
* [[Gedcom Import Mediatype]] and [[Gedcom Import Monitor]] are related only in that they also affect the Gedcom Import kickoff form and the Gedcom Import process. Aspects of those mods are ''coordinated'' with this mod, but there are '''no dependencies'' among them and this mod.
* If the ''optional'' [[Show Mod Names]] mod is installed, this mod will utilize its functionality. 

== Installation ==
=== Files Installed ===
This mod has two mod subfolders:
# '''gedcom_import_purge_v12.0.0.5''' - The subfolder that you expect to exist for any mod that installs files. It contains
#*'''rrgedcomimportpurge_dbsetup.php''' This is ordinarily run once, from a link in the Mod Manager, to define the new Medialinks table field that keeps track of medialinks that are created by the Gedcom Import.  The field is called '''createdfromgedcom'''.
# The shared folder '''RR-shared_mod_includes_v12.0.0.3''', which contains files that support mod option management in many of my mods:
#* '''rrshared_innermodmenu2.php''' - [https://tng.lythgoes.net/wiki/index.php?title=Inner_Mod_Menus Inner&nbsp;Mod&nbsp;Menus]
#* '''rrshared_modsettingsblocks2.php''' - [https://tng.lythgoes.net/wiki/index.php?title=Mod_Settings_Blocks Mod Settings Blocks]
#* '''img/rrshared_wikilogo.gif''' - An icon that links to a mod's TNG Wiki page.

The shared folder is packaged with several mods, each of which may install some or all of its files. Once a file from the shared folder has been installed (by any mod), that file will not be overwritten or removed by the subsequent installation or un-installation of any mod (including this one), nor will its presence generate any Mod Manager errors. 

If you unzip mod distribution files directly into your mods folder, then the presence of this second subfolder and the installation of its files should be invisible to you.  But if you generally ''copy'' mod subfolders to your mods folder, you need to make sure to copy the folder '''RR-shared_mod_includes_v12.0.0.2''' (well, unless you are confident that it is already in your mods folder because it was part of another mod).
{{RobinInstallationBoilerplate}}

== Visualizations ==

{| border="3" cellspacing="1" cellpadding="2" class="wikitable"
|-
| <span style="color: red">'''In the Mod Manager, After Installation'''</span>, showing the database setup hyperlink.
<div>[[Image:gedcom_import_purge-modman.png]]</div>
You must follow the hyperlink shown above to run the setup program that creates the database field through which TNG remembers which media links were created by a Gedcom Import. 
|-
| <span style="color: red">The Database setup program</span>
[[Image:gedcom_import_purge-createdb.png]]
|- 
| <span style="color: red">'''The Gedcom Import Kickoff Form (Admin>>Import/Export) AFTER INSTALLATION:'''</span>
<div>[[Image:gedcom_import_purge-after1.jpg]]</div>
New or changed elements on the Gedcom Import Kickoff form:
# The Inner Mod Menu, which pulls down from the "Mod Information" heading, is shown in its open state.
# A mod option determines whether the "Import media if present" checkbox should be checked. For the Gedcom Import Purge mod to be useful in a Gedcom Import, this checkbox should be checked.
# Two new checkboxes. The default status of each checkbox is determined by mod options.
## "Purge media links..." - The primary purpose of this mod.
## "Suppress the creation of citation medialinks" - It all depends on your Gedcom file and how you want to handle citation medialinks.
|}

{| border="3" cellspacing="1" cellpadding="2" class="wikitable"
| <span style="color: red;">'''Message As the Gedcom Import starts:'''</span>
This message is visible only if you choose the "Old style import" checkbox (or, if you have installed [[Gedcom Import Monitor]], the "Scrolling progress listing").
<div>[[Image:gedcom_import_purge-purge.png]]</div>
Note that the Gedcom Import process will not purge any medialinks the ''first'' time you run it after installation, since the new database field hasn't yet been populated by flags that say that the Medialink was created by a Gedcom Import.
|}

{| border="3" cellspacing="1" cellpadding="2" class="wikitable"
| <span style="color: red">'''Media Links in the Admin>>Media>>Edit form'''</span>
The Medialinks table in the Media Edit form now has a column with a checkbox for the "From Gedcom" flag. It is checked for Media items that were loaded through a Gedcom Import process, and defaults to unchecked (and usually remains that way) when Media items are added within TNG.
<br>[[Image:gedcom_import_purge-after-medialinks.jpg]]
|}

{| border="3" cellspacing="1" cellpadding="2" class="wikitable"
| <span style="color: red">'''The Mod Options - in a Mod Settings Block'''</span>
The Mod Settings Block for Gedcom Import Purge is at '''Admin>>Setup>>Import&nbsp;Settings'''. You can jump straight there from the "Change Mod Options" link in the Inner Mod Menu.<br>
[[Image:gedcom_import_purge-after-modsettingsblock.jpg]]
|}

== Revision History ==
{| border="0" cellspacing="1" cellpadding="2" class="wikitable"
! Mod Version !! TNG Version !! Date !! Note
|- style="vertical-align:top;"  <!-- ############################################################## --> 
| [[Media:gedcom_import_purge_v12.0.0.5.zip|12.0.0.5]]
| 12.0+
| 2 Dec 2019
| 
# Modified the Place purge process so that it does not purge places on sites that have multiple trees and have one shared Places list.
# Added an Inner Mod Menu, and moved the options to an Mod Settings Block
# Added a form field in the Medialinks Admin>>Media>>Edit form so that admin users can set or unset the "createdfromgedcom" (aka "From Gedcom") flag.
# Enhanced the message that, at the beginning of the Gedcom Import process, displays the counts of places and medialinks that were purged, and medialinks that were not purged.
# The Gedcom Import process generates a list of the places that were purged and recreated, and purged and not recreated.
# Added the "Suppress the creation of citation medialinks" checkbox on the Admin Gedcom Import kickoff form.
# Added code to suppress citation medialinks when the checkbox is checked.
# Adds a Save buttons to the left of the BodyText field in the Edit Media form, so that the user doesn't always have to scroll to the top or bottom of the form to finish.

|- style="vertical-align:top;"  <!-- ############################################################## -->
| [[Media:gedcom_import_purge_v12.0.0.4.zip|12.0.0.4]]
| 12.0+
| 15 May 2018
| No functional changes; made compatible with TNGv12.

|- style="vertical-align:top;"  <!-- ############################################################## -->
| [[Media:textplus_charts_v10.1.0.3p.zip|10.1.0.3p]]
| 10.1-11.1
| 26 Mar 2017
| Removed the second line from the cust_text.php target location search string

|- style="vertical-align:top;"  <!-- ############################################################## -->
| 10.1.0.3
| 10.1-11.1
| 26 Mar 2017
| A technical update that, mostly, just makes [[Show Mod Names]] optional, and avoids a installation conflict introduced by TNGv11.0.1. That is, this version omits the adminlib.php patch that was part of v2 of this mod, and changes this mod's database setup program so that it works independently of that adminlib.php patch (which was fixed in TNGv11.0.1).

|- style="vertical-align:top;"  <!-- ############################################################## -->
| 10.1.0.2
| 10.1-11.0.1
| 25 May 2016
| Removed a unneeded &lt;script> element that incorrectly referenced an external file, and that - under rare circumstances I still don't understand - could cause the database setup program to log you out from your TNG session.  Also fixes a similarly incorrect &lt;script> element in adminlib.php.

|- style="vertical-align:top;"  <!-- ############################################################## -->
| 10.1.0.1b
| 10.1-11.0
| 25 Mar 2016
| Fixed an error in a JavaScript warning. Cleaned up the code. Changed the Mod Parameters to strings rather than boolean values to be more tolerant of data entry errors.

|- style="vertical-align:top;"  <!-- ############################################################## -->
| 10.1.0.1a
| 10.1-10.1.3
| 21 Feb 2016
| No new end-user functionality. Now depends on [[Show Mod Names|Show Mod Names v2+]].

|- style="vertical-align:top;"  <!-- ############################################################## -->
| 10.1.0.1
| 10.1-10.1.3
| 6 Feb 2016
| New mod.
|}

== Sites using this mod ==
If you download and install this mod, please add your site to the table below.

{| border="0" cellspacing="1" cellpadding="2" class="wikitable"
|-
! URL
! User
! Note
! Mod-Version
! TNG-Version
! User-language
|-
| Robin Richmond's Genealogy Database - admin function; not visible.
| [[User:Robinrichm|Robin Richmond]]
| Mod developer
| 10.1.0.3
| 11.1.0
| English
|-
| [http://genealogy.oceanwharf.com Hooley Family Links]
| [[User:lastejas|Rick Hooley]]
| Public/Private
| [[User: lastejas|See Here]]
| [[User: lastejas|See Here]]
| EN
|-
| [https://SmeeFamily.com/ Pete's Research]
| [[User:TBirdUK|TBirdUK]]
| Public/Private
| 12.0.0.5
| 12.01+
| English
|-
| [https://WokingFamily.com/ Woking Family Tree Project]
| [[User:TBirdUK|TBirdUK]]
| Public/Private
| 12.0.0.5
| 12.01+
| English
|-
|}

[[Category:Mods for TNG v12]]
[[Category:Mods for TNG v11]]
[[Category:Mods for TNG v10]]
[[Category:admin_dataimport.php mods]]
[[Category:admin_gedimport.php mods]]
[[Category:gedimport_trees.php mods]]
[[Category:gedimport_misc.php mods]]
[[Category:admin_editmedia.php mods]]
[[Category:js/mediafind.js mods]]
[[Category:js/mediautils.js mods]]
[[Category:micro_medialinks.php mods]]
[[Category:ajx_updateorder.php mods]]
[[Category:admin_importconfig.php mods]]
[[Category:admin_updateimportconfig mods]]
[[Category:data_help.php mods]]
[[Category:English cust_text.php mods]]