User:Daveh/ESO Update

The UESPWiki – Your source for The Elder Scrolls since 1995
Jump to: navigation, search

This page details the procedure of updating the UESP data after a major ESO update/release.

General[edit]

  • Run ESO patcher to get latest version.
  • Edit content3:/home/uesp/esolog/esolog.index and increment/note the log index.
  • Backup/delete the uespLog_backupData.txt file created by uespLogMonitor.
  • Export Game Data
  • Edit createNewExport.sh and update the top 6 lines with the relevant update numbers:
    VERSION="17"            # Game update number
    ISPTS="1"               # 1 for PTS, blank for Live
    LASTVERSION="16"        # The last game update
    LASTPTSVERSION="17pts"  # The last PTS game update
    MAKEPTSDIFF="1"         # 1 or blank, Create diff files from the last PTS update
    MAKEDIFF=""             # 1 of blank, Create diff files from the last Live update
  • Run createNewExport.sh
  • Fix/address any errors that occur during the script.
  • Update Log Parser
  • Edit the parseLog.php file.
  • Update the START_MINEITEM_TIMESTAMP and real-time timestamps.
  • Update the MINEITEM_TABLESUFFIX and SKILLS_TABLESUFFIX constants with the current version.
  • Ensure the TABLES_WITH_IDFIELD array contains values with the current version.
  • Optionally update the $IGNORE_LOGENTRY_BEFORE_TIMESTAMP1 field with the relevant timestamp.
  • Update esofiles.uesp.net
  • Create new directory files1:/shared/uesp/esofiles/update-XX/
  • Upload all ZIP files created in the export process.
  • Update esolog/esoCommon.php
  • Add API version to $APIVERSION_TO_GAMEUPDATE and $APIVERSION_TO_GAMEVERSION arrays.
  • Update the GetEsoItemTableSuffix() function with the latest update numbers.
  • Live: Update the GetEsoItemTableSuffix() function to return the correct current update number (usually done after item mining is completed).
  • Live: Update the GetEsoUpdateVersion() function to return the current update number (usually done after item mining is completed).
  • Update/add item types.
  • Update/add item special types.
  • Update set indexes:
  • Copy content from the Sets.php file generated by the data extraction script and copy into esoCommon.php replacing the existing $ESO_SET_INDEXES variable.
  • Skills and Champion Points
  • Only dump data on a character with no CPs/skills purchased and wearing no items that could affect the mined data.
  • /uespdump skills abilities VERSION
  • /uespdump skills types VERSION
  • /uespdump cp VERSION
  • Upload saved variable files.
  • Run log parser.
  • Edit createCpFits.php with the new table $TABLE_SUFFIX and run.
  • Edit fixSkillsSlotted.php with the new table $TABLE_SUFFIX and run.
  • Edit createSkillTree.php with the new table $TABLE_SUFFIX and run.
  • Edit fixDestructionSkills.php with the new table $TABLE_SUFFIX and run.
  • Edit fixPvpArtifactSkills.php with the new table $TABLE_SUFFIX and run.
  • Move old tables (minedSkills, skillTree, minedSkillLines, cpSkills, cpSkillDescriptions, cpDisciplines) to tableOLDVERSION and new tables to tables with no version suffix, for example:
     RENAME TABLE minedSkills TO minedSkillsXX; RENAME TABLE minedSkillsYY TO minedSkills;
  • Mine Item Data
  • Make sure you do this on a max level character (CP160) with no CP/skills purchased and not wearing any ability altering equipment.
  • PTS
  • Only the min/max item data are exported in PTS updates to minimize the time and storage required.
  • Run the following in-game commands:
        /umi level 1
        /umi type 1
        /umi autostart 1
  • After each automatic /reloadui copy the uespLog.lua saved variables to a new file for later manual upload.
  • Repeat the mining for max level/quality data.
        /umi level 50
        /umi type 370
        /umi autostart 1
  • Repeat the mining for potion data.
        /umi potion on
        /umi autostart 0
  • Item mining for PTS takes about 30 min.
  • Live
  • Run the following in-game commands (make sure level/type/item type/potion options are turned off):
        /umi autostart 1
  • Note that the entire process takes around 1 week of full time mining to complete.
  • Repeat the mining for potion data.
        /umi potion on
        /umi autostart 0
  • Edit createMinedItemSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Edit createSetSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Move old tables (minedItem, minedItemSummary, setSummary) to tableOLDVERSION and new tables to tables with no version suffix, for example:
     RENAME TABLE minedItem TO minedItemXX; RENAME TABLE minedItemYY TO minedItem;
  • PTS: Update Log Viewer
  • Edit viewLog.php
  • Add options/tables for viewing the new PTS version.
  • Update Maps
  • Upload any files in the NewMaps folder to content1/2/3:/home/uesp/www/esomap/ and files1:/shared/uesp/maps/esomap/.
  • (Optional) Edit maps_new.txt that contains a list of all new maps to add.
  • On db1 find the last user world ID by the SQL command: select max(id) from uesp_gamemap.world;
  • Edit createNewMap.sh with the first free world ID (+1 from last step) and update version and run.
  • Upload the newmaps.sql to db1.
  • Backup the current uesp_gamemap database: mysqldump --opt -u user -p uesp_gamemap > gamemap.sql
  • Run the SQL file: mysql -u user -p uesp_gamemap < newmaps.sql
  • Test the map to ensure new maps have been successfully added.
  • Update Styles
  • Use /ud GetHighestItemStyleId() in-game to get the last style ID.
  • Use /ud GetItemStyleName(#) in-game to get the style material.
  • Use /ud GetItemStyleMaterialLink(#) in-game to get the style material.
  • Use /uespstyle all, /uespstyle allid in-game to list all styles.
  • Edit uespLog.lua and update the uespLog.CRAFTSTYLENAME_TO_ITEMSTYLE and uespLog.CRAFTSTYLENAME_TO_MOTIFID arrays.
  • Edit uespLogTradeData.lua and update the uespLog.ALT_STYLE_ICON_DATA and uespLog.STYLE_ICON_DATA arrays.
  • Edit esolog:esoCommon.php and update the $ESO_ITEMSTYLE_TEXTS array.
  • Edit esochardata:viewCharData.class.php and update the $MOTIFS_FOR_MASTERWRITS array.
  • Update Traits
  • Edit esolog:esoCommon.php and update the $ESO_ITEMTRAIT##_FULLTEXTS, $ESO_ITEMTRAIT_DESCRIPTIONS, $ESO_ITEMTRANSMUTETRAIT_IDS, and $ESO_ITEMTRAIT##_TEXTS arrays.
  • Edit esolog:resources/esoitemlink.js and update the ESO_ITEM_TRAITS array.
  • Edit esolog:resources/esoItemSearchPopup.js and update the UESP.ESO_TRAIT_TYPES array (and select lists if needed).
  • Update/add other tables as needed.
  • esolog:esoCommon.php $ESO_ITEMTYPE_TEXTS
  • esolog:esoCommon.php $ESO_ITEMSPECIALTYPE_TEXTS and $ESO_ITEMSPECIALTYPE_RAW_TEXTS
  • esolog:esoCommon.php $ESO_SET_INDEXES
  • Achievements
  • /uespdump achievements VERSION
  • Upload log and run log parser.
  • Copy old esoAchievementData.php to esoAchievementDataXX.php
  • Run createAchievementData.php
  • Update Game Data
  • Collectibles
  • /uespminecollect
  • /ud uespLog.MineCollectibleTree()
  • Run createRuneBoxData.php (after mined items are updated) and update uespLog.RUNEBOX_COLLECTIBLE_IDS array in uespLog.lua with any new runebox/collectible pairs.
  • Upload log and run log parser.
  • Copy data out of tempData in the uespLog.lua saved variable file and re-format/paste into esoCollectibleData.php.
  • Upload log and run log parser.
  • Books
  • Perform this on a character that has unlocked Eidetic Memory/Shalidor's Library.
  • Run /ud uespLog.MineBookTree() in-game.
  • Run /ud uespLog.MineBooks() in-game (upload/parse log data).
  • Copy data from the tempData section of the saved variable file, re-format, and paste into esoBookCollection.php.
  • Upload books.sql to db1 and run.
  • Potions
  • Quests
  • Update the esoQuestData.php file from the Quests/Quest.php file created by the export script.
  • Upload quests.sql to db1 and run.
  • Recipes
  • Perform this on a PTS template to ensure all recipes are recorded correctly.
  • Run /ud uespLog.MineRecipeDataStart() from in-game.
  • Copy data from the tempData section of the saved variable file, re-format, and paste into esoRecipeData.php.
  • Update Icons
  • Upload icons to content3/files1:
    rsync -avz ./esomnf-XX/esoui/art/icons/ dave@content3.uesp.net:/home/uesp/www/eso/gameicons/esoui/art/icons/
    rsync -avz ./esomnf-XX/esoui/art/icons/ dave@files1.uesp.net:/shared/uesp/esogameicons/esoui/art/icons/
  • Repeat for the following folders:
  • esomnf/esoui/art/treeicons
  • Update file permissions:
    sudo chown dave:uespadmin -R /home/uesp/www/eso/gameicons/esoui/art/
    sudo chmod a+rx -R /home/uesp/www/eso/gameicons/esoui/art/
  • Update the API source code
  • Modify OS to use case insenstive filenames.
  • Set Windows registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive=dword:00000000.
  • Reboot after modifying registry key.
  • Use Cygwin with posix=1' set for all relevant drives in /etc/fstab.
  • Test case insenstivity before running script.
  • Modify EsoParseData.py and update the INPUT_GLOBAL_FILENAME, INPUT_LUA_PATH, and OUTPUT_PATH variables.
  • Run EsoParseData.py from a CygWin shell window.
  • If script aborts in error fix the folder security permissions in Windows and retry.
  • Edit makeApiDiff.sh to the current and previous API versions and run.
  • Upload the new API directory to content2:/home/uesp/www/eso/data/.
  • Copy api.html to the root index.html.
  • Replace the symbolic link current to the new API directory.
    ln -sfn 100025 current 
  • (Optional) Set Windows registry key obcaseinsensitive back 1.

Old[edit]

Live / PTS[edit]

Generally the procedure is identical for an update on PTS as one on Live except for Live updates the main databases are updated instead of temporary PTS ones.

  • Edit the export scripts with the new update number: XX for Live and XXpts for PTS
  • Extract game.mnf (2 minutes) and eso.mnf (several hours) data to new directory
  • Convert DDS to PNG:
    ./convertdds.bat ./esomnf-XX/art/
    ./convertdds.bat ./esomnf-XX/esoui/
    ./convertdds.bat ./gamemnf-XX/esoui/
  • Edit the copyNew.sh script to reflect the new update number and run.
  • Edit the makeDiff.sh script with the previous and current update numbers and run. For Live updates this is usually run twice, once against the last Live update and once against the last PTS update.
  • Edit esoCommon.php in the esolog project and add checks for the new version in the GetEsoItemTableSuffix() function.
  • Modify the table suffixes in the following esolog files:
    parseLog.php (MINEITEM_TABLESUFFIX, SKILLS_TABLESUFFIX)
    createSetSummary.php (TABLE_SUFFIX)
    createMinedItemSummary.php (TABLE_SUFFIX)
    createCpFits.php (TABLE_SUFFIX)
    createSkillTree.php (TABLE_SUFFIX)
    fixDestructionSkills.php (TABLE_SUFFIX)
  • Run /uespminecollect in game to update collectible data.
  • Update mined item data
  • PTS
  • Only the min/max item data are exported in PTS updates to minimize the time and storage required.
  • Run the following in-game commands:
        /umi level 1
        /umi type 1
        /umi autostart 1
  • After each automatic /reloadui copy the uespLog.lua saved variables to a new file for later manual upload.
  • Repeat the mining for max level/quality data.
        /umi level 50
        /umi type 370
        /umi autostart 1
  • Repeat the mining for potion data.
        /umi potion on
        /umi autostart 0
  • Item mining for PTS takes about 30 min.
  • Live
  • Run the following in-game commands (make sure level/type/item type/potion options are turned off):
        /umi autostart 1
  • Note that the entire process takes around 1 week of full time mining to complete.
  • Repeat the mining for potion data.
        /umi potion on
        /umi autostart 0
  • Dump ability data with /uespdump skills all XX. Before parsing logs make sure you update parseLog.php in the esolog project and include the minedSkillsXX table in the $TABLES_WITH_IDFIELD array.
  • Dump CP data with /uespdump cp XX.
  • Run createCpFits.php.
  • Run createMinedItemSummary.php and createSetSummary.php.
  • PTS
  • Modify esolog's viewLog.php to display the new PTS tables.
  • Add a ENABLE_XXPTS constant and the appropriate lines in the constructor that use it.
  • Add the minedItemXXpts, minedItemSummaryXXpts, setSummary17pts, and minedSkillsXXpts entries in the $RECORD_TYPES array.
  • Add the minedItemSummaryXXpts, setSummary17pts, and minedSkillsXXpts entries in the $SEARCH_DATA array.
  • Added the MakeMinedItemLinkXXpts and MakeMinedItemSummaryLinkXXpts functions.
  • Update the $SEARCH_TYPE_OPTIONS array to use the latest PTS tables.
  • Skill Updates
  • For a full update in-game run the command /uespdump skills character XX once for each class and each race along with Vampire/Werewolf to capture orderings and required levels of all skill lines.
  • Alternatively, you can use the previous update data for skill order and required levels. Edit saveEsoPlayerSkills.php with the prior update version and run to save data and then run restoreEsoPlayerSkills.php using the saved file from the previous step and the current update version.
  • Run createSkillTree.php (update table suffix first).
  • Run fixDestructionSkills.php.
  • Mine Skill Coefficients
  • In-game run /usc addall to add all skills that have numbers in their description.
  • Make sure character being used has no skill passives and no CPs purchased.
  • Equip a set of Light/Medium/Heavy and run /usc save. Equip/unequip items and repeat save at least 5 times per set (the more saved the more accurate the estimates but the larger the saved data). Sets used should be "normal" stat modifying sets and not sets with hidden damage modifiers (e.g., increases spell damage for Destruction skills by X).
  • /usc calc
  • /reloadui
  • Save (large) saved variable uespLog.lua file to a safe place.
  • In-game run /usc reset to reduce file size (coefficient data is saved to the log).
  • /reloadui and upload saved variable file to esolog.
  • Check for bad fits (check for R<0.95 directly in the database) and re-run coefficients with fixed add-on data as needed.
  • Maps
  • Edit CombineEsoMaps.py with the current update # and run.
  • Edit CreateEsoMapTiles.py with the current update # and run.
  • Edit CreateEsoMapTilesZoom11.py with the current update # and run.
  • Manually create maps.zip file from CombinedMaps output directory including maps diff file.
  • Upload new/updated map tiles to content1/2/3 and files1 servers.
  • Create New Maps
  • Create the text file maps_new.txt with the following CSV structure of one map per line (using an empty ,, and -1 if you don't know the parent map):
         [rawname], [Map Name], [parent rawname], [parent ID]
  • Find the current max world ID from the ESO map database:
        select max(id) from world;
  • Edit CreateDbStatementForNewMaps.py with the current update # and first available map ID (+1 from above query) and run.
  • Backup the current ESO map database on db1:
        mysqldump --opt -u root -p uesp_gamemap > esomap-XX.sql
  • Upload newmaps.sql to db1 and import into database:
        mysql -u root -p uesp_gamemap < newmaps-17pts.sql
  • Test map to ensure new maps are working.
  • Add links/locations to new maps as required.
  • Update map groupings file worldgrouplist.txt.
  • Update style and set index data in common code.

Live[edit]

  • Start ESO patcher.
  • Manually increase esolog index (esolog.index) and note.
  • Update uespLog addon.
  • Once you can login note time and timestamp and update parser code.
  • Update guild message related to uespLog.
  • Mine achievement data.
  • API data.
  • Clear eso item link image cache (content3:/home/uesp/esoItemImages/).
  • Book and Quest data.
  • Load ParseBook project in VC++ 2015
  • Modify input/output paths.
  • Run.
  • Copy SQL files to db1 and execute.
  • Update PHP files in esolog project.
  • Recipe Data:
  • /ud uespLog.MineRecipeDataStart()
  • Copy from tempData section.
  • Copy into esolog/esoRecipeData.php
  • Similarly with /ud uespLog.MineBookTree() and /ud uespLog.MineCollectibleTree().
  • Potion data (if updated)
  • Traits (new and updated)
  • Styles