Difference between revisions of "Changing to UTF-8"
Chris Moss (talk | contribs) (→The Places table: clarify) |
|||
| (46 intermediate revisions by 8 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | {{Languages|Changing to UTF-8}} | |
| − | + | {{TNGmod | |
| − | + | | mod_name = Change coding | |
| − | + | | mod_summary = Allows a simple change of character set and collation sequence for the TNG database | |
| − | + | | mod_validation = | |
| − | | | + | | mod_last_update = 23 Mar 2018 |
| − | |} | + | | download_link = [http://mossfamilytree.info/download.php?mod=changecoding&version=11.1.0.2 11.1.0.2] |
| + | | download_stats = [http://mossfamilytree.info/downloadstats.php?mod=changecoding show] | ||
| + | | mod_author = [[user:Chris Moss|Chris Moss]] | ||
| + | | mod_url = | ||
| + | | mod_support = [http://mossfamilytree.info/suggest.php contact author] | ||
| + | | mod_contact = [http://mossfamilytree.info/suggest.php contact author] | ||
| + | | mod_version = 11.1.0.2 | ||
| + | | min_TNG_ver = | ||
| + | | max_TNG_ver = 12.1.2 | ||
| + | | TNG_file_list = | ||
| + | | related_mods = | ||
| + | | notes = only available with English instructions | ||
| + | }} | ||
| + | Increasingly UTF-8 is being used on the web as it handles all character sets in use. But often a TNG site is uploaded from a local database which uses Windows 1252 (ANSI) or ISO-8859-1 which only handle some Western European languages. Converting the TNG site involves changing both the database and a number of settings within TNG. | ||
Notes on changing a TNG Site and database to UTF-8 provided by [[User:TheKiwi]] | Notes on changing a TNG Site and database to UTF-8 provided by [[User:TheKiwi]] | ||
| + | == Create a copy or backup of your database == | ||
| + | You can use phpMyAdmin to make a copy of your database (always make a copy in case something goes wrong!!!!!!!) | ||
| + | # In the databases list on the left click on the Database to select it. | ||
| + | # On the right side click on the Operations Tab. In the section called "Copy Database to" enter a name for your new database, and make sure that "Structure and Database" and "CREATE DATABASE before copying" are both checked, then click "Go". | ||
| − | + | When done this will have created a copy of your database. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Alternatively you can back up your database using any of the methods described in [[Database - Backup]]. | |
== Change database to UTF-8 == | == Change database to UTF-8 == | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | You need to run a script to change the structure of your database to UTF-8. The previously recommended script has some problems so a new mod has been created which is much simpler to use and works in a flexible way. | |
| − | + | Download the mod in the normal way, unzip, place it in your mods directory and click the install button. | |
| − | + | [[File:change_to_utf8.png|thumb|left|380px|After running the change coding mod]] | |
| + | When it is installed, click on the green "Installed" line and then hit the "Change database" button below. The conversion will take a number of seconds and when it is complete a summary will appear similat to that shown at the left. Note in this case that the default language is English but the language actually in use when the conversion was done is German. The "utf8_swedish_ci" collation is suitable for most languages (including German), but if you want a different one, then you can edit the options of the mod and put another collation, as long as it's consistent with the character set chosen. | ||
| − | + | If for any reason you want to change back to one of the other two supported character sets, ISO-8859-1 and ISO-8859-2, then choose the Edit options button on the mod and put one of these in the first box. In this case you will need to change the collation to latin1_swedish_ci or similar (there is no 'general' for the latin languages, the MySQL variant of ISO/8859). | |
| − | + | The mod alters all the appropriate settings for the different language folders, including cookies and session variables, so you shouldn't need to do anything else. Once you have changed the database it's ok to uninstall the mod as it shouldn't be needed again. But's ok to run the mod again if, for instance, an incompatible table has been installed into your database or the settings have been disturbed by an update. | |
| − | + | But '''don't''' try and import an ISO Gedcom file into a UTF-8 database (or vice versa). There will be no checking or conversion and you will end up with a mess if it contains accented characters. | |
| − | + | ===The Places table=== | |
| + | One particular issue that occurs in some cases is the table "tng_places" in which the main entry "place" is required to be unique. An earlier version of this mod used utf8_general_ci which didn't exactly correspond to latin1_swedish_ci in that characters with accents such as ä are treated as identical to a and it may therefore happen that the conversion of that table fails. | ||
| − | + | In the case of a truly multinational database it may be best to convert the whole database first to utf8_bin and then attempt a translation to utf8_general_ci, which will work on most tables but fail on tng_places (and possibly tng_countries). Having the tng_places table as utf8_bin will ensure that there are no clashes caused by the translation. Leaving the whole database as utf8_bin will make many other operations such as searches fail. Changing the collation only is generally quicker as no data is changed. | |
| − | + | == Make sure that you have all the necessary UTF-8 Encoded language files == | |
| + | In the languages directory, there are pairs of folders for each language, e.g. French and French-UTF8, which have all the strings used in the two different character sets. If you have deleted the -UTF8 directories to save space, then these should be restored before running the change script above. Most mods will also include changes to both of these sets, normally changing the cust_text.php file. | ||
| − | + | However if you made changes to cust_text.php file yourself then you need to make sure these changes are also in the corresponding -UTF8 version of cust_text.php. The steps below will help you do that on Macintosh and Windows computers | |
| − | + | === On Macintosh system === | |
| − | + | Here's what works on Macintosh. You need BBEdit - a free text editor. If you don't already have BBEdit, you can download it from [https://www.barebones.com/products/bbedit/download.html https://www.barebones.com/products/bbedit/download.html] | |
| − | + | # For each language folder open the file cust_text.php in BBEdit | |
| + | # After the file is open, go to the File menu and choose "Reopen using Encoding ------> Western (Windows Latin 1). | ||
| + | # At the bottom of the BBEdit window is a pop up menu should now say "Western (Windows Latin 1)". Click on this and choose "Unicode (UTF-8)". | ||
| + | # Save the file and place in the appropriate directory. | ||
| − | + | === On Windows system === | |
| − | + | You need to use a text editor that can handle saving files in UTF-8 format WITHOUT writing the BOM (Byte Order Mark) to the file such as Notepad++ ([http://notepad-plus-plus.org/download/ Current Version]). | |
| − | + | Load the cust_text.php file and save it specifying UTF-8 encoding without BOM, and place it in the corresponding folder (e.g. French-UTF8). | |
== Related links == | == Related links == | ||
| Line 134: | Line 76: | ||
*[[Setup - Language#Cust_Text|Cust Text in Setup - Language]] provides examples of overriding $text in both the text.php and admintext.php files | *[[Setup - Language#Cust_Text|Cust Text in Setup - Language]] provides examples of overriding $text in both the text.php and admintext.php files | ||
* [[TNG charset]] | * [[TNG charset]] | ||
| − | + | *[[Database_Collation_-_Explain_Choosing|Selecting your TNG Database Collation]] | |
| + | *[[UTF-8 translator]] - a utility for replicating the ISO or UTF parts of a mod using html entities | ||
[[Category:Charset]] | [[Category:Charset]] | ||
| − | [[Category: | + | [[Category:Programmer]] |
| + | [[Category: Mods for TNG v12]] | ||
| + | [[Category: Mods for TNG v11]] | ||
| + | [[Category: Mods for TNG v10]] | ||
Revision as of 09:08, 8 August 2018
| Change coding | |
|---|---|
| Summary | Allows a simple change of character set and collation sequence for the TNG database |
| Validation | |
| Mod Updated | 23 Mar 2018 |
| Download link | 11.1.0.2 |
| Download stats | show |
| Author(s) | Chris Moss |
| Homepage | |
| Mod Support | contact author |
| Contact Developer | contact author |
| Latest Mod | 11.1.0.2 |
| Min TNG V | |
| Max TNG V | 12.1.2 |
| Files modified | |
| Related Mods | |
| Notes | only available with English instructions |
Increasingly UTF-8 is being used on the web as it handles all character sets in use. But often a TNG site is uploaded from a local database which uses Windows 1252 (ANSI) or ISO-8859-1 which only handle some Western European languages. Converting the TNG site involves changing both the database and a number of settings within TNG.
Notes on changing a TNG Site and database to UTF-8 provided by User:TheKiwi
Create a copy or backup of your database
You can use phpMyAdmin to make a copy of your database (always make a copy in case something goes wrong!!!!!!!)
- In the databases list on the left click on the Database to select it.
- On the right side click on the Operations Tab. In the section called "Copy Database to" enter a name for your new database, and make sure that "Structure and Database" and "CREATE DATABASE before copying" are both checked, then click "Go".
When done this will have created a copy of your database.
Alternatively you can back up your database using any of the methods described in Database - Backup.
Change database to UTF-8
You need to run a script to change the structure of your database to UTF-8. The previously recommended script has some problems so a new mod has been created which is much simpler to use and works in a flexible way.
Download the mod in the normal way, unzip, place it in your mods directory and click the install button.
When it is installed, click on the green "Installed" line and then hit the "Change database" button below. The conversion will take a number of seconds and when it is complete a summary will appear similat to that shown at the left. Note in this case that the default language is English but the language actually in use when the conversion was done is German. The "utf8_swedish_ci" collation is suitable for most languages (including German), but if you want a different one, then you can edit the options of the mod and put another collation, as long as it's consistent with the character set chosen.
If for any reason you want to change back to one of the other two supported character sets, ISO-8859-1 and ISO-8859-2, then choose the Edit options button on the mod and put one of these in the first box. In this case you will need to change the collation to latin1_swedish_ci or similar (there is no 'general' for the latin languages, the MySQL variant of ISO/8859).
The mod alters all the appropriate settings for the different language folders, including cookies and session variables, so you shouldn't need to do anything else. Once you have changed the database it's ok to uninstall the mod as it shouldn't be needed again. But's ok to run the mod again if, for instance, an incompatible table has been installed into your database or the settings have been disturbed by an update.
But don't try and import an ISO Gedcom file into a UTF-8 database (or vice versa). There will be no checking or conversion and you will end up with a mess if it contains accented characters.
The Places table
One particular issue that occurs in some cases is the table "tng_places" in which the main entry "place" is required to be unique. An earlier version of this mod used utf8_general_ci which didn't exactly correspond to latin1_swedish_ci in that characters with accents such as ä are treated as identical to a and it may therefore happen that the conversion of that table fails.
In the case of a truly multinational database it may be best to convert the whole database first to utf8_bin and then attempt a translation to utf8_general_ci, which will work on most tables but fail on tng_places (and possibly tng_countries). Having the tng_places table as utf8_bin will ensure that there are no clashes caused by the translation. Leaving the whole database as utf8_bin will make many other operations such as searches fail. Changing the collation only is generally quicker as no data is changed.
Make sure that you have all the necessary UTF-8 Encoded language files
In the languages directory, there are pairs of folders for each language, e.g. French and French-UTF8, which have all the strings used in the two different character sets. If you have deleted the -UTF8 directories to save space, then these should be restored before running the change script above. Most mods will also include changes to both of these sets, normally changing the cust_text.php file.
However if you made changes to cust_text.php file yourself then you need to make sure these changes are also in the corresponding -UTF8 version of cust_text.php. The steps below will help you do that on Macintosh and Windows computers
On Macintosh system
Here's what works on Macintosh. You need BBEdit - a free text editor. If you don't already have BBEdit, you can download it from https://www.barebones.com/products/bbedit/download.html
- For each language folder open the file cust_text.php in BBEdit
- After the file is open, go to the File menu and choose "Reopen using Encoding ------> Western (Windows Latin 1).
- At the bottom of the BBEdit window is a pop up menu should now say "Western (Windows Latin 1)". Click on this and choose "Unicode (UTF-8)".
- Save the file and place in the appropriate directory.
On Windows system
You need to use a text editor that can handle saving files in UTF-8 format WITHOUT writing the BOM (Byte Order Mark) to the file such as Notepad++ (Current Version).
Load the cust_text.php file and save it specifying UTF-8 encoding without BOM, and place it in the corresponding folder (e.g. French-UTF8).
Related links
- Text_variables_-_explained provides information on how the $text variables are used
- Cust Text in Setup - Language provides examples of overriding $text in both the text.php and admintext.php files
- TNG charset
- Selecting your TNG Database Collation
- UTF-8 translator - a utility for replicating the ISO or UTF parts of a mod using html entities
