Login

Blog: Torchlight 3 – Data fixup war stories

<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories.png" width="243" height="77" title="" alt="" /></div><div><p>   <!– –> <title>Gamasutra: Jill Sullivan’s Blog – Torchlight 3: Data Fixup War Stories </title>                  </p>
<div id="iribbon-container" class="content-body-wrapper"> <button id="iribbon-title" title="show/hide" class="inactive">Informa</button> </p>
<div id="iribbon-detail" class="ribbon-hide">
<div id="iribbon-left">
<p>Gamasutra is part of the Informa Tech Division of Informa PLC</p>
</p></div>
<div id="iribbon-right">
<p>This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC’s registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.</p>
</p></div>
</p></div>
</p></div>
<p>      </p>
<div class="container">
<div id="container-main" class="content-body-wrapper">   </p>
<div class="container bottom3">
<div class="span-7"> <a href="https://www.gamasutra.com"><img loading="lazy" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories.png" alt="Gamasutra: The Art & Business of Making Games" width="243" height="77" border="0"></a><img alt="spacer" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories.gif" width="27"> </div>
</p></div>
<div class="span-20">    </p>
<div class="span-20 last content_bg">
<div class="hide-phone">
<div class="span-4">
<div class="content_box_left">
<div class="leftcol">                              <!– </p>
<div class="header"><img loading="lazy" alt="arrow" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-1.png" width="22px" height="20px" /><a href="http://www.gamasutra.com/calendar/calendar.php">Calendar</a></div>
<div class="leftnav bottom2"> <a href="http://www.gamasutra.com/calendar/calendar.php"><strong>View All</strong></a>     <a href="http://www.gamasutra.com/calendar/calendar_submit.php"><strong>Submit Event</strong></a> </p>
<hr>
<ul> </ul>
</div>
<p> –>   </p>
<div class="bottom2"> <a href="https://www.gamasutra.com/advertise"> <img loading="lazy" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories.jpg" alt="Sponsor" border="0" class="whiteTop" width="177px" height="60px"> </a>
</div>
<p>   </p>
<div class="leftnav_network bottom2"> If you enjoy reading this site, you might also want to check out these UBM Tech sites: </p>
<hr>
<hr>
<p> <br class="clear">
</div>
<p>    </div>
<p> </div>
</div></div>
<div class="span-16 last">
<div class="content_box_middle">  </p>
<div class="page_item">
<div><a name="twitter_share" href="http://twitter.com?status=RT @gamasutra: Torchlight 3: Data Fixup War Stories http://www.gamasutra.com/blogs/JillSulli...23/375423/" target="_blank" rel="noopener noreferrer"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-1.gif" alt="Share on Twitter" border="0" height="20"></a> <span id="edit_post_link"></span>   <a href="https://www.gamasutra.com/blogs/rss/"><img loading="lazy" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-2.gif" width="15" height="15" border="0" alt="RSS"></a> </div>
<hr>
<div class="item_body mobile_image_transform"> <strong><i><small> The following blog post, unless otherwise noted, was written by a member of Gamasutras community.<br />The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company. </small></i></strong> </p>
<hr>
<p><img loading="lazy" alt height="338" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-2.png" width="600"></p>
<p><em>During the ongoing development of Torchlight III, we want players to get up-to-the-moment information about what’s going on with the game & dev team. This month’s Developer Update comes from Guy Somberg, Lead Programmer of Echtra, with help from Jill Sullivan, Senior Community Manager.</em></p>
<p><strong>Introduction</strong></p>
<p>In game development, not every problem that you solve is performance, features, or functionality that the players get to see.  Sometimes, you have something messy that needs to be fixed, and you just need to dive in and fix it so that work can get done.</p>
<p>There is a phrase that we use to talk about this sort of work: “yak shaving”.  Originally from a reference to the TV program “Ren and Stimpy”, it now refers to work that appears to be completely unrelated to the end goal, but which you have to accomplish in order to reach it.  For example – “I am trying to build a stone bridge over this creek.  I am shaving this yak so that I can trade the fur to a yarn maker, who in exchange will let me borrow the cart so that I can take it to the quarry to pick up some stones.”  Shaving the yak isn’t necessarily an important part of building that bridge, but you won’t be able to make any progress while the yak still has its coat!</p>
<p>This is a collection of just some of the times that we’ve had yak-shaving problems that needed to be “just fixed”.</p>
<p><strong>0D0D0A</strong></p>
<p>Source control systems are one of the fundamental tools that game developers (and, in fact, pretty much all developers) use.  It is a database containing the entire history of every file that makes up our game – source code, assets, sounds, you name it.</p>
<p>About three years ago, we switched our source control from one system to another.  It doesn’t matter from what to what.  The system we were using was breaking down under our load, so we needed a new one.  We did our homework, examined the alternatives, and made a call.</p>
<p>Now, when you transition source control systems, there are broadly speaking two ways to go about it.  The simpler way is to lock everybody out of source control, take a snapshot of the latest stuff, import it into the new system, tweak it to conform to the new system’s idea of how the universe should work, and then turn the new system on for people.  This has the advantage that it “just works”, but it loses all of the source history from before the changeover.  In these environments there is often a single moment in source history that says “Imported everything.  If you want history before this, go look in the other source control system.”  That’s fine, so long as the other system stays around or you otherwise have access to it, but often the commits from earlier are lost forever.</p>
<p>The more complex way of doing this is to actually import the history from the old system into the new one.  Most source control systems allow you to do this, but it is time-consuming, error prone, and still requires some manual intervention to conform to the idiosyncrasies of the new system.  Although it is more work up-front, it is invaluable in the end to have your entire source history available.</p>
<p>We opted for the history import, which – at least on the surface – seemed to go just fine.  We saw the history, we saw the files, and we were able to poke around and verify that everything looked right.  Some files had weird spacing issues, which didn’t seem like that big a deal.</p>
<p>But then we tried to compile, and it all crumbled down.  The Visual Studio compiler complained about “Mac line endings” and refused to compile anything.</p>
<p>What?!  Why would that be?</p>
<p>A bit of background here: when a computer wants to represent a character, it has to select an encoding.  The most common encoding in use today is called UTF-8, which can encode all English-language characters, common punctuation, or a number of control codes into a single byte of data.  (Using multiple bytes, you can encode data in just about any language, but that’s another discussion.)</p>
<p>Two of these control codes are the Carriage Return (CR) and the Line Feed (LF) characters, which hearken back to the days where computers were hooked up to automated typewriters rather than screens.  In those days, you would tell the printer carriage to go back to its home column by sending it a CR code, and you would have the paper roll to the next line by sending it an LF code.  Thus, if you wanted to start typing at the beginning of a new line, you would send the sequence CR LF.</p>
<p>When the switch to fancy graphical displays occurred, this CR LF convention remained for backward compatibility.  However, developers of new systems – like the sparkly new Apple Macintosh computer and the Unix system at Bell Labs – weren’t hindered by backward compatibility and were free to make different choices.</p>
<p>It turns out that the three most common systems in the world today all made different choices: DOS used CR LF, Macintosh used CR, and Unix used LF.  Windows inherited its line endings from DOS, and MacOS now uses LF (the same as Unix).</p>
<p>Over time, the differences sorted themselves out.  Software is generally able to operate in “text mode” and provide the user with whatever line endings they need for their system to render it correctly.  The details of these differences leak through every so often, but usually aren’t a big deal.</p>
<p>All of this background flashed through our minds when we saw the error about Mac line endings.  What was it talking about?  We develop on Windows, so all of the line endings should have been Windows (CR LF) line endings – or, at the very least a combination of Windows and Unix (LF) line endings.  Where were these lone CR characters coming from?</p>
<p>And then we remembered the weird spacing issues – all of our source code appeared to be double-spaced.  Where did those extra spaces come from?</p>
<p>This is where somebody had the idea to look at the file in a hex editor – a tool which allows us to see the binary representation of the text files by displaying each byte’s value in hexadecimal.  Ordinarily, on a file with Windows line endings, you expect to see a line of text, then a CR (13, or 0D in hexadecimal representation) and an LF (10, or 0A).  For some reason, on the broken lines, we saw a CR (0D), then another CR (0D), and then an LF (0A), giving us 0D0D0A.</p>
<p>Somehow, during the conversion process from one source control program to another, the conversion program decided that the file had Unix line endings, then went through and did a blind search/replace every LF with CR LF, even if it already had a CR!  That explained everything.  Our editor was perfectly happy to render the CR as a blank line, and knew how to convert the CRLF into a blank line, which was why our code appeared to be double-spaced.  Contrariwise, the Visual Studio compiler was happy to interpret the CR LF combo as a newline, but errored out on the preceding CR.</p>
<p>I fixed this by writing a little program in C++.  It would iterate over our source code directory, open every text file, find patterns of 0D0D0A and replace them with 0D0A.  We don’t expect to change source control systems again, so the code for this tool is lost to the sands of time.  (Ed. – Or, so we thought!  A drive containing the source for this program was discovered after this article was written, so we have uploaded the code to our repository for posterity.)</p>
<p>There were only two or three of us who worked on this particular issue, but you can get any of us to twitch a little just by saying “oh doh doa”.</p>
<p><strong>Octothorpe Fixer</strong></p>
<p>A couple of years ago our sound designer and our composer took a trip out to Bratislava, Slovakia to record a live orchestra for some of our music.  It was an awesome trip (so I’m told), and they got a lot done over the few days that they were there.</p>
<p>One of the outputs of this trip was a suite of content that we call “vzory” – Slovakian for “pattern”.  These are small orchestral chunks of music that can be combined in myriad ways to create new music, and are recorded in various combinations of keys and notes.  The end result is that we have a particular pattern in G, in G#, in F, in F#, etc.</p>
<p>Our composer did the natural thing – he spent a bunch of time cutting and organizing all of this content, doled it out into folders and files matching the note that they were recorded at, and then imported the whole suite into the audio tool that we use, FMOD Studio.  FMOD is hooked up to our source control system, and it happily added all of the new files and then checked them in.</p>
<p>So far so good.  Until people started to get mysterious warnings about filenames when they got the latest code and data through our source control system.  They were just warnings – they weren’t preventing anybody from working – but it was definitely something that we didn’t want to stick around.</p>
<p>These vzory tracks were tracked down as the culprit.  It turns out that our source control system doesn’t like it if you check in files with an octothorpe (‘#’, sometimes called a pound sign, hash mark, hash tag, number sign, or various other things) in the filename.  It will accept them, but complain loudly.  It turns out that our composer named the directory and matching files for the vzory tracks in the key of A sharp with the name “A#” – naturally!  (The other sharp keys were set up this way as well.)</p>
<p>The source control system was most displeased with this choice.</p>
<p>Renaming the files wasn’t enough, because FMOD keeps track of file and directory metadata in XML files – each one with a GUID (a sequence of letters, numbers, and dashes) as the filename.</p>
<p>Once again, code to the rescue.  This time it was a program written in C# (ironically) that would iterate over all of the files and subdirectories in the given path, find ones with an octothorpe in the name, and rename them to replace the ‘#’ with the word ‘sharp’.  So, ‘A#’ became ‘Asharp’.  Then it would iterate over the XML files in the path, find any that had an octothorpe in the file contents (which were therefore metadata about the files or directories that had been renamed), and replace the ‘#’ in that line with the word ‘sharp’.</p>
<p>Other than telling our folks “don’t do that”, there’s not much that we had to do to prevent this from happening again.  This time we kept the source code, so if we make that particular mistake again then the tool to fix it is ready at hand.</p>
<p><strong>POFixer</strong></p>
<p>Localization and internationalization are important parts of any game project.  We use the Unreal engine, which has a suite of localization tools built-in.  By using a particular data structure in our data files, Unreal can find all of the localized lines in the game.  We can then export them into a standardized format called a “portable object” (.po) file, used by the GNU gettext tools, among others.</p>
<p>This is a format that our translators have tools to deal with.  They grab the files, translate the lines, and send them back.  We can then import them to a particular locale and then Unreal will render the text.  All very neat, so long as you color within the lines and follow the way that Unreal expects you to work.</p>
<p>Naturally, we have built some of our own stuff which lives inside of Unreal’s systems and plays nicely with them, but is sufficiently “off to the side” that it is invisible to some of Unreal’s other systems.  One of those parts is the localized string system, which didn’t see any of our fancy assets.</p>
<p>We wrote a tool that makes them visible, and called it a day.  Our first big batch of localization went out to the translators.  We went to import it…only to find that none of our strings got imported!</p>
<p><strong>What happened?</strong></p>
<p>Unreal allows you to identify each localized string by a pair of text strings: a category and an entry within that category.  If you don’t provide either of those entries, it will generate them for you.  It turns out that the tool that we had written to make our assets visible to the translators generated a new category and entry for every localized text string every time it was run, which meant that every text line would get a different code every time we ran the import or the export.</p>
<p>Oh, dear.  We fixed the underlying problem and made the category/entry pairs be consistent across runs, but we had this massive drop of strings in all of the languages that was incompatible with the fixed-up data!  We had to figure out how to run a one-time fixup on these strings to make them match.</p>
<p>Fortunately, each string came with a lot of metadata about its context and provenance.  Much of this metadata did not change, or at least changed in a predictable fashion.  This metadata turned out to be enough that we could compare an imported line and a newly-exported line and match up the strings.</p>
<p>As before, writing some code was the answer here.  We wrote a program (C++ again) to read in the translated file (containing the old, incorrect category/entry pairs) and a newly-exported English-language file (containing the new, correct category/entry pairs), match up the metadata, and then write out a fixed version of the file containing the translated text with the corrected category/entry pairs.</p>
<p>Here is one situation where simply fixing up the data was insufficient.  We needed to solve the underlying problem first before we could write the tool to fix the data.</p>
<p><strong>Conclusion</strong></p>
<p>These problems all had a common theme: through some sequence of events – human error or machine error – a bunch of important files appeared that were all broken in some way.  Ultimately, to the people who are working with the data it doesn’t really matter why any of these things happened.  They just want to take their broken files and fix them.  It is always a worthwhile endeavour to figure out a root cause and prevent an issue from occurring again, but sometimes you just need to get to work.  All of the postmortem analysis and preventative work in the world won’t help people get their jobs done with the broken files that they already have.</p>
<p>The examples I have talked about here were all important work that needed to get done, but all three of those programs got run exactly once.  It turns out that many of the systems that we build are complex, and these sorts of issues crop up as a normal part or development as we discover some of the edge cases.</p>
<p>Sometimes, you need to write tools that you run exactly once, and that is not a problem – you just need to grit your teeth and shave that yak.</p>
<p>– Guy Somberg</p>
</p></div>
<hr>
<div class="hide-phone">
<h3>Related Jobs</h3>
<div class="stories_item">
<div class="thumb"><a href="https://jobs.gamasutra.com/job/java-software-developer-core-team-hamburg-34724"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-3.png" alt="innogames" width="120"></a></div>
</p></div>
<div class="stories_item">
<div class="thumb"><a href="https://jobs.gamasutra.com/job/ui-developer-brisbane-queensland-34738"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-4.png" alt="Gameloft Australia Pty Ltd " width="120"></a></div>
<div class="title"> <strong> Gameloft Australia Pty Ltd — Brisbane, Queensland, Australia <br />[12.22.20] </strong> <br /><a href="https://jobs.gamasutra.com/job/ui-developer-brisbane-queensland-34738">UI Developer</a> </div>
</p></div>
<div class="stories_item">
<div class="thumb"><a href="https://jobs.gamasutra.com/job/senior-mobile-game-developer-c-forge-of-empires-hamburg-34704"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-3.png" alt="innogames" width="120"></a></div>
</p></div>
<div class="stories_item">
<div class="thumb"><a href="https://jobs.gamasutra.com/job/senior-lead-engineer-boston-massachusetts-34701"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-1.jpg" alt="Crate Entertainment" width="120"></a></div>
</p></div>
</p></div>
<hr></div>
<p>  </div>
<p>
</div>
<p> </div>
<p> <br class="clear"> </div>
<p>   <!– </div>
<p> Extra Div –>       <br class="clear"></p></div>
<p> </div>
<p><br />
   <a href="http://www.omniture.com" title="Web Analytics"><img loading="lazy" src="https://www.sickgaming.net/blog/wp-content/uploads/2020/12/blog-torchlight-3-data-fixup-war-stories-3.gif" height="1" width="1" border="0" alt></a><br />
  <br />
  </p>
</div>

https://www.sickgaming.net/blog/2020/12/...r-stories/

xSicKxBot