Census Feature

From TNG_Wiki
Jump to navigation Jump to search
Caution The Census Feature Mod has been removed from further distribution pending repairs and simplifications. Existing users will still be supported thru the forums and direct communication with the author.

Attention Census Feature mod users, Ancestry has changed the way the DOM source file gets generated. When you attempt to import a transcript it results in a single blank line showing.

At this point, you should click the REMOVE BAD IMPORT (Only chance!) link to delete the imported census. v9002d / v9212d has been created to address this problem.
You should use the REMOVE BAD IMPORT (Only chance!) link to delete the records at that point, and then download the v9002d or v9212d zip files, remove the existing mod and install the updated mod

Caution


Ambox notice.png Please read the Upgrading section for instructions to upgrade to V9002 or v9212 of this mod.

The Census Feature FAQ contains answers to common questions.
Important: See the Note in the Interactive Viewer section on how to import family census that span two pages.

TNG 9.2.1

If you downloaded the TNG v9.2.1 upgrade/update after May 14 you need the v9.2.1.2 version of this mod.

TNG 9.0


Census Feature
Summary Adds census record data for Households on individual Info page
Validation
Mod Updated
Download link
Download stats
Author(s) Gerald 'Jerry' Leehan
Homepage Census Feature (This page)
Mod Support TNG Community Forums
Contact Developer
Latest Mod 9.0.0.2 for TNG v9
9.2.1.2 for TNG v9.2.1
if TNG v9.2.1 downloaded after May 14th
Min TNG V 9.0.0
Max TNG V 9.2.1
Files modified
See TNG Modules Impacted
Related Mods
Notes
Note that the mod zip file is 9MB because of the help images.

Purpose of the mod

This mod provides administrators with the ability to create census records and transcripts and store them in new tables that can be searched and displayed in TNG. It also creates a directory for census images.
The mod creates a display in the 'Census' tag on each census individual's TNG Info page that shows key fields from the census record for each member of the household. Transcripts of the census page containing the individual are created and linked to the census item.
Tools automate the linking of individual census records to people in the database using search techniques that consider alternate spellings and maiden and married names.
When census individuals cannot be located in the existing TNG People table, the administrator is given the option of automatically adding them along with any relationships that can be deduced from the census household data.
Ancestry.com members can automatically import census records.

Developer

This mod was developed by Gerald 'Jerry' Leehan with very significant contributions from Ken Roy.

V9.0.0.2 enhancements to this mod were made by Janice Nation and Ken Roy

Requirements

  • A working TNG installation.
  • A backup of all affected files (see the list in the summary in the upper right corner of the page)
  • An installed current version of the Mod Manager (is already included in TNG v9)
  • All templates/templateN folders be on your site
  • Both English and English-UTF8 folders be on your site
  • The CENS custom event type must be set to Accept.
    You must use the standard CENS - Census event and not a personally created census event (v9000 or v9211 only)
  • The CENS custom event date must be entered as YYYY only (v9000 or v9211 only)

Installation

  1. Download the appropriate zip file as shown in the summary in the upper right corner of the page.
  2. Extract the files and folder from the downloaded zip file into the mods folder or FTP the extracted files and folder to your mods folder.
    You MUST transfer the censusmod_vnnnn folder and all its contents to the TNG mods folder.
  3. Follow the normal automated installation for Mod Manager, as shown in the example Mod Manager - Installing Config Files to install the mod.
  4. Note - if you're using the tng.php WordPress-TNG Plugin, you might need to "Update the TNG Variables" in the WordPress Admin pane for the TNG Plugin to get this to work correctly.

First Time Post Install Actions

  1. Click the Create Reqd Tables if they do not exist button shown on the mod manager screen to add the census tables if they do not exist.
    This button will only create these tables if they do not already exist in your database.
  2. Click the Create Census directory if it doesn't exist button on the Mod Manager screen to create the census folder if it does not already exist.
  3. Click the Load initial Alternate Spelling Dictionary to load a dictionary of common alternate spellings used to improve searches.
    You can optionally, skip step 3 and create the Alternate Spelling table as you Assign IDs on new transcripts

Upgrading

  • Mod Upgrades: You should
  1. Create a backup of the Census Tables before you remove the mod and do the upgrade
  2. Create a backup of the extensions/cen_settings.inc file that contains your Mod Manager Edit changes to the Census Feature default options
  3. Remove (uninstall) the currently installed version of the mod but not the tables since they will contain your census data
  4. Download and extract the new version of the mod
  5. FTP the extracted .cfg file and the censusmod_v900n folder to your TNG mods folder (or extract on site)
  6. Install the new version of the mod which should leave your tables and directories intact.
  7. Delete the previous version of the mod
    For v9.0.0.2 or v9.2.1.2 of the mod you need to click the buttons in the Mod Manager screen for the Census Feature mod to
    1. Create the Census Summary table
    2. Update the Census Features Tables
  8. Note - if you're using the tng.php WordPress-TNG Plugin, you might need to "Update the TNG Variables" in the WordPress Admin pane for the TNG Plugin to get this to work correctly.
  9. Create a backup of the Census Tables after the upgrade before you make any changes
  10. Using the backup of the extensions/cen_settings.inc file Edit the Mod Manager option to set your preferred processing options.
  • TNG release upgrades: It is not necessary to remove (uninstall) the mod prior to installing the TNG upgrade. You will likely need to do a Clean up after the upgrade and then re-install the mod.

Uninstalling the mod

If you want to completely Remove (uninstall) this mod and all its tables, you should

  1. click the Drop Census Feature Tables button in the Mod Manager status screen for the mod to delete the census table before you
  2. click the Remove button to uninstall the mod and then
  3. click the Delete button to delete the config file from the mods folder.

If you are uninstalling the mod prior to an upgrade, you should only click the Remove button to uninstall the mod. You should not Drop Census Feature Tables nor Delete the mod since you will want to re-install the mod after the upgrade.

Other Languages

This MOD currently only supports English language and has extensive 'help' files containing images with English depicted on screens. Since the TNG Help screens are in the English language folder, if you run your site as UTF-8 then you need to keep both the English and English-UTF8 folders.

V9002 or V9212 has added a census_text.php and started converting several headings used in difference scripts to use $text variables, which should allow converting the displays for multi-language sites in the future. Not all hard-coded English text has been converted to $text variables at this time.

Restrictions

  • If you import a GEDCOM as a way of supporting your web site, you must define the Census events (or facts) in your desktop genealogy program
  • The image file name in the census folder must be in the following format dbid=6743&iid=4241561-00594.jpg that matches the transcript_file column in the cen_transhead and cen_data database tables
  • The household number must be the same for all members in a household.
    If a family crosses images, where the Head of household is at the bottom of one page, and part of the family is at the top of the next page, you must enter the same household number for both transcript sources when assigning household numbers.

See Census Feature restrictions removed for restrictions removed in v9002 or v9212 of the mod

Known Issues

For a list of issues fixed and restrictions removed in v9.0.0.2 of the mod, see Census Feature issues fixed in v9002

  • Template 5 width in some cases, like the 1830 and 1840 transcripts with household summary counts will generated table entries that exceed the page content width
You will need to increase the width for maintable class to something like 1100 px
  • The help files in Version 9.0.0.2 have not yet been updated to reflect the changes in this version.
  • A problem has been reported where the Census Feature tables appear to have been created with a collation sequence that is different from other collation sequence
You may need to check the collation sequence of the Census Feature tables after creating them to make sure that they match your existing collation sequence and change them accordingly before importing any census transcripts, since the MySQL manual indicates that new tables created would use the same collation sequence as the database

Feature List

  • Four new tables and two new directories are added to TNG to facilitate storage, manipulation, and display of Census data and images.
    (Note that if you are already using a census folder for a Census media collection for example, it will co-exist with this mod).
  • Records can be linked to individuals in the main TNG People table... see ADMIN tools
  • Information from the Census tables will display in the Census Description box of each individual in the census
    • Each individual in the household will be shown
      • Key census fields displayed
      • Links to each individual
      • Focus individual is highlighted
    • Links to full transcripts and to actual image data are provided
      • Image link will find images stored in the Census folder or look for them on Ancestry.com if they are not on site
      • The transcript display
        • Shows all individuals and all fields contained on each census page
        • Shows header data for the page... e.g. Township, City, State, Enumeration District, Page...
        • Links to preceding and succeeding transcripts if they exist and to the full image
        • Is editable by those with administrator privileges
  • A Census section is added to the TNG Administration Index page with tabs for key census administration procedures:
    • View
    • Add
    • Edit
    • Delete (v9002 or v9212 only)
    • Import from Ancestry
    • Update Transcript
    • Assign IDs
      After assigning Person IDs, v9002 or v9212 allows creating the 1830 or 1840 Summary count information (100 plus or Free Color counts are not yet implemented)
    • Image
  • Extensive pictorial 'Help' tutorials are provided for each Census Administration tab.
The help files in Version 9.0.0.2 have not yet been updated to reflect the changes in this versioon.
  • The census tables can be optimized, backed up or restored just like any other table.
  • The census tables' structure will be backup up together with all other tables. Note that restoring such backup after explicitly deleting the table and uninstalling the mod will restore the census table. So you should probably make a new table structure backup before uninstalling the mod.

Visualization of this mod

The following images show the visual effects of installing the mod.

Site Visitor

Visitors to your site will see the Census transcripts.

Census custom event

With the mod installed and census transcripts added the individual info page shows a transcript for the Household for the Census custom event

Census transcript display

Visitors can click the Image and Transcript links on the Census transcript to view the actual image, if you have loaded the image to your census folder using the correct file name, or from Ancestry.com if you did not load the image and they have a subscription that allows them access, and the additional formatted transcript information.

You can click on the [Collapse] link on the right to hide the screen captures or [Expand] to display them again.
Census image

The visitor can click the Image link on the Census image on Ancestry.com if they have a paid subscription that allows access.

Census Image on Ancestry
Census transcript

The visitor can click the Transcript link on the Census display on the Individual page to get a formatted transcript display.

V9002 or v9212
Census transcript formatted V2

V9000 and V9211
Census transcript formatted

TNG Admin

The TNG Administrator will see the following changes in the admin side.

You can click on the [Collapse] link on the right to hide the Admin screen captures or [Expand] to display it again.
Mod Manager screen for the Census Feature mod shows three buttons to be used after installing the mod for the first time only:
  • Create Census Tables if they do not exist
  • Create the census and transcriptsource folders if the do not exist
  • Load initial Alternate Spelling Dictionary
    Optionally, you can skip the load of the provided Alternate Spelling table and create your own Alternate Spelling Dictionary as you add Census Transcript records

V9002 or v9212 requires upgrade actions to:

  • Create the Census Summary table
  • Update the Census Feature table field definitions
 After mod is installed first time only
Utilities >> Back up, Restore & Optimize Table Data

The mod adds the

  • Alternate Spelling,
  • Census Data,
  • Census Links,
  • Census Summary (v2),
  • Transcript Heading

tables to the list of tables you can optimize, backup or restore.

With the mod installed
Admin

The mod adds a Census link in the left frame and button in the main frame of the TNG Admin screen.

Census option added to Admin
Admin >> Census

The mod adds a Census screen with

  • View,
  • Add,
  • Edit,
  • Delete (v2),
  • Import from Ancestry,
  • Update Transcript,
  • Assign IDs, and
  • Image

tabs

Census screen with various tabs
Admin >> Census>> Import from Ancestry

v9002 or v9212 of the mod allows selecting the

  • US Federal Census,
  • UK Census, or
  • Census of Canada

series or collection for import. The UK Census and Census of Canada collections are only supported using the Interactive Viewer. You may be able to import from AncestryLibrary using the Interactive Viewer, if your public library allows you to use your own laptop and connect to their network so that you can save the DOM Source using Firefox or SeaMonkey. Most public libraries only support Internet Explorer (IE) which does not support saving a DOM Source file.

Import from Ancestry v9002 or v9212

An Approach to Using

Interactive Viewer

The procedure for the Interactive Viewer is similar. Sorry, I have not had the time to update the wiki documentation and may not before I leave on Wednesday, but it should be as follows:

  1. - select the View Record from the Ancestry Search results screen
  2. - select the New Image viewer from the Ancestry search results screen, if there is a choice since they appear to default to the new viewer in a lot of cases now
  3. - Expand the Index icon in the lower left corner of the screen
  4. - right click on the index heading line
  5. - choose the Select All option
  6. - right click again on the screen
  7. - choose the View Selection Source which launches a DOM Source window
  8. - wait for the new window screen to be completely populated
  9. - select File, Save Page As
  10. - save the file using a name that makes sense to you, such as year_person_US_state_county_town.txt
  11. - FTP the saved file to your tng/transcriptsource folder
  12. - click the Refresh List button if you don't see the file
  13. - if importing different than US, you must go back to the top of the screen and reselect the Series
  14. - click the Interactive Viewer button

Note that if you are saving DOM Source from the interactive and standard (basic) viewer you should indicate in your .txt file name the source such as using _std or _int in the file name

Note that only the Interactive Viewer is supported for the UK Census and Census of Canada collections. The Interactive Viewer might also support importing from AncestryLibrary provided that you can use your own laptop to connect to their network so that you can save the DOM Source file in Firefox or SeaMonkey. Most public libraries only use Internet Explorer (IE) on their own computers which does not allow saving a DOM Source file.

Note that you cannot use the right-arrow to get the DOM Source for the next page, since the transcript_file will be the same. To get to the next page when a family is across 2 pages, you should

  • use the Census URL from the imported census
  • increment the last digit to the next number

For example, if I captured my grandfather's page for the 1940 US Census http://interactive.ancestry.com/2442/m-t0627-01472-00028

to get to my dad's page, I need to change the ending 8 in the URL to a 9 http://interactive.ancestry.com/2442/m-t0627-01472-00029

Previous instructions

If you have a paid subscription to Ancestry.com, you can find the Census images and save the transcripts so they can be processed by this mod. The following is an approach used.

  • Save the image to your census folder with a file name that uses part of the URL for the dbid= and iid=, for example
    dbid=6224&iid=4639366_00188
  • save the census image with a file name that allows you to find the image
  • save the transcript file with the same name but with a .txt extension
  • copy the transcript file to the TNG transcriptsource directory
  • copy the image file using the dbid= and iid= name to the census directory

Note: If you maintain your genealogy master database in a desktop program, the Census events (CENS) should be created in your desktop database using the census year only for the event.

See Using the Census Feature Mod for additional details.

In the event of a problem

Revision History

Version Release Date Contents
v9.0.0.2d
v9.2.1.2d 		28 Jul 2013
  • Fix UK DOM Source import to handle recent changes made by Ancestry
  • Fix US Interactive DOM Source import to handle recent changes made by Ancestry
  • Fix Canada DOM Source import to handle recent changes made by Ancestry and to handle 2 different formats of the 1871 census
  • Fix treeID and census event ID assignment in US Interactive DOM Source import which required Assign IDs be done a second time
  • Implemented option to not overlay census event location as an attempt to fix disappearing census events
v9.0.0.2c
v9.2.1.2c 		11 Jul 2013
  • Fix Image link and thumbnail display on Census Transcript
  • Fix UK DOM Source import to handle recent change made by Ancestry
  • Fix US Interactive DOM Source import to handle two different formats of 1850 and four different formats of 1940 censuses
  • fixed 1901 Census of Canada import keys to handle 2 different DOM Source formats
  • fixed conflict with Person Custom Events mod in personlib.php changes
v9.0.0.2b
v9.2.1.2b 		8 Jun 2013
  • Fix truncation of Incorporated place in Transcript Update
  • Fix US Standard DOM Source import to handle 50 lines in 1930 census
  • Fix US Interactive DOM Source import to handle two different formats of 1930 and 1940 censuses
v9.0.0.2a
v9.2.1.2a 		2 Jun 2013 Fix INSERT query for manual adds to use imageurl instead of image column name
You only need to re-download the v9002 or v9212 if you are using the Add tab to manually add census transcript records.
v9.0.0.2
v9.2.1.2 		28 May 2013
  • Support for the Ancestry Interactive Viewer Dom Source file thanks to the effort of Janice Nation.
The standard US viewer was left as an option to support previously saved transcriptsource files
  • Addition of the UK Census collection also thanks to Janice Nation. The UK Collection includes the England, Isle of Man, Channel Islands, and Wales censuses.
Note that manual add of censuses was not updated for these new series
  • Addition of the Census of Canada collection by Ken Roy and updated for the interactive viewer by Janice Nation
  • Several screens were updated to support the multi-language capability using a census_text.php file by Ken Roy.
Some custom text still remains in cust_text.php
  • Several messages were added to screens to help facilitate the import process
  • Selection of the Census event was added by Janice Nation which now allows other Census Custom events besides the CENS - Census event to be used
provided they contain census in the event description
  • Validation of the input transcript source was added by Ken Roy
  • Capability to directly assign a PersonID bypassing the search was added by Ken Roy
  • Transcript File was added as a direct search criterion to the AssignIDs screen and look4assignupdate.php
  • Summary count support for the 1830 and 1840 US Federal Census was added by Ken Roy
  • Mod Manager Edit Options were added to control the behavior of the mod, for example
    • whether USA or UK is added to census location
    • whether the import census process will estimate a birth date if none is in the census
    • whether to default the use of the Alternate Spelling to Yes or No
    • whether to delete the file after import
  • Restriction on using only YYYY for the Census Event date was removed
v9.2.1.1 17 May 2013 Updated for TNG V9.2.1 downloaded after May 14th
v9.0.0.1 16 March 2013 This version was only released to beta testers
  • added copyfile2 for censusthumb.jpg
  • added missing help file images - ancestryselectmenu.jpg and viewselectionsourcemenu.jpg
  • changed background:silver to background:lightyellow in transcript display
  • fixed negative dates
  • fixed missing header info in update and display
  • fixed too narrow input widths on some variables
  • fixed to support US Census from any Ancestry.co.uk or Ancestry.ca (Fix provided by Janice on the TNG Forum) but not from AncestryLibrary.com
  • added a Close Window link to several screens
  • fixed to display all lines for the 1850 and 1860 transcript display
  • fixed the link to the TNG Wiki article for the Census Feature mod
  • fixed to support index data updates note that user corrections are ignored and the original name is imported (Fix provided by Janice on the TNG Forum)
  • fixes error message for age display in look4assignupdate.php
22 February 2013 Initial release of the Census Feature mod

Sites using this mod

If you download and install this mod, please add your TNG site to the table below:

URL User Note Mod-Version TNG-Version User-language
Families of NW Pennsylvania Gerald 'Jerry' Leehan Mod Developer V9.0.0.0 TNG 9.0.2 EN
Our Roy and Boucher Family Ken Roy Worked with Jerry Leehan and Janice Nation to develop v9.0.0.2 and v9.2.1.2.
Now using the Census Plus International mod 		TNG 9.2.0 EN, FR
The Cook/Garza/Harms family pages Klaus Cook Registered only. Guest access login: guest pwd: guest
Now using the Census Plus International mod 		V9.0.0.0 TNG 9.2 EN, DE, ES
Our Family Histories Bryan S. Larson Tested the mod for Jerry
Now using the Census Plus International mod 		V9.2.1.2 TNG 9.2.1 EN
AppeltHaus Genealogy Lawrence Appelt Not a Public Site. V9.0.0.0 TNG 9.2.0 EN
Pimble One-Name Study Gerald A Cooke Not a Public Site.
Now using the Census Plus International mod 		V9.0.0.0 9.1.0 EN
The Dutton's of Vermont Darrin Dutton Sample Page V9.0.0.0 TNG 9.1.2 EN
Conjunctions Rosina Lippi sample page (still tinkering) V9.0.0.1 TNG V9.1.0 EN, DE
Whittlesey-Whittelsey Family History Willis S Whittlesey III Public Site V9.0.0.0 TNG V9.2.0 EN
Ancestral Stewart Family Tree Michael Stewart Public
Now using the Census Plus International mod 		V9.0.0.0 TNG V9.2.0 EN
Mueller-Diefenbach Families Bob Mueller Link is to an example of this mod. I'm affected by the known AncestryLibrary bug V9.0.0.0 TNG V9.2.0 EN, DE
Pages From Our Past Allen Prunty Semi-Private Site V9.0.0.0 TNG V9.2.0 EN
Mosley Families Martin Mosley Public Site, privacy restricted. User: tester password 3005Tester can be used to view mod in action V9.2.1.2 TNG V9.2.1 EN
Roebuck And Families Ancestry Website Don Roebuck Public Site, access to living is restricted. Using UK Census V9.0.0.2 TNG V9.2.0 EN

TNG Modules Impacted

The following TNG Modules are modified, and should be backed up prior to installing this mod:

  • customconfig.php
  • admin_backup.php
  • admin_optimize.php
  • admin_restore.php
  • admin_utilities.php
  • admin_genconfig.php
  • tabledefs.php -- should there be a change here?
  • admin_merge.php
  • getperson.php
  • personlib.php
  • admin_leftbanner.php
  • admin_main.php
  • treelib.php
  • css/genstyle.css
  • templates/templateN/css/templatestyle.css
  • your languages cust_text.php file(s)

The following modules are added by this mod to create and delete the Census tables:

  • usemakecensustables.php
  • dropcensustables.php

Related Links

Census Feature mod

Importing US Census with the Interactive Viewer

Importing UK Census

Using the Basic Viewer for US Census

Importing Census of Canada

Census Feature considerations when using TNG only

Census Feature considerations when importing GEDCOM

Retrieved from "https://tng.lythgoes.net/wiki/index.php?title=Census_Feature&oldid=78300"