From 91f5e2c479714325785ed4aceb98d224c7d3bc37 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Sun, 11 Mar 2012 01:14:11 -0500 Subject: [PATCH 01/15] added support for removing tiles that shouldn't exist with --check-tiles --check-tiles is now activated by shrinking maps better detection for --forcerender situtations Also cleaned up some docs and comments Moved tiledir creation to TileSet constructor --- docs/running.rst | 74 ++++++++++++++++----- overviewer.py | 2 - overviewer_core/tileset.py | 131 ++++++++++++++++++++++++++++++------- 3 files changed, 163 insertions(+), 44 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 2790402..0c37284 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -116,34 +116,57 @@ only have to use once-in-a-while. .. cmdoption:: --forcerender Forces The Overviewer to re-render every tile regardless of whether it - thinks it needs updating or not. This is similar to deleting your output - directory and rendering from scratch. + thinks it needs updating or not. It does no tile mtime checks, nor does it + check any last modification time of the world. It unconditionally renders + every tile that exists. - This is the default mode for first-time renders. This option overrides + The caveat with this option is that it does *no* checks, period. Meaning it + will not detect tiles that do exist, but shouldn't (this can happen if your + world shrinks for some reason. For that specific case, + :option:`--check-tiles` is actually more appropriate). + + This is the default mode for first-time renders. This option conflicts with :option:`--check-tiles` and :option:`--no-tile-checks` .. cmdoption:: --check-tiles - Forces The Overviewer to check each tile on disk and compare its - modification time to the modification time of the part of the world that - tile renders. This is slightly slower than the default, but can be useful if - there are some tiles that somehow got skipped. + Forces The Overviewer to check each tile on disk and check to make sure it + is up to date. This also checks for tiles that shouldn't exist and deletes + them. + + This is slightly slower than :option:`--no-tile-checks` due to the disk-io + involved in reading tile mtimes and reading the world's region file headers + for chunk mtimes, but can be useful if there are some tiles that somehow got + skipped. + + The caveat with this option is that it compares the tile mtimes on disk with + the chunk mtimes reported by Minecraft. Thus, if the Minecraft world was not + updated since the last render, tiles will not be detected as needing + updates. This means you cannot use this option in response to a changed + configuration setting. This option is the default when The Overviewer detects the last render was - interrupted midway through. This option overrides :option:`--forcerender` - and :option:`--no-tile-checks` + interrupted midway through. This option conflicts with + :option:`--forcerender` and :option:`--no-tile-checks` .. cmdoption:: --no-tile-checks - With this option, The Overviewer will not do any checking of tiles on disk - to determine what tiles need updating. Instead, it will look at the time - that the last render was performed, and render parts of the map that were - changed since then. This is the fastest option, but could cause problems if - the clocks of the Minecraft server and the machine running The Overviewer - are not in sync. + With this option, The Overviewer will determine which tiles to render by + looking at the saved last-render timestamp and comparing it to the + last-modified time of the chunks of the world. It builds a tree of tiles + that need updating and renders only those tiles. - This option is the default unless the condition for :option:`--forcerender` - or :option:`--check-tiles` is in effect. This option overrides + This option does not do *any* checking of tile mtimes on disk, and thus is + the cheapest option: only rendering what needs updating while minimising + disk IO. + + The caveat is that the *only* thing to trigger a tile update is if Minecraft + updates a chunk. Any other reason for needing to re-render a tile requires a + different mode to detect the needed update. It could also cause problems if + the system clock of the machine running Minecraft is not stable. + + **This option is the default** unless :option:`--forcerender` or + :option:`--check-tiles` is in effect. This option conflicts with :option:`--forcerender` and :option:`--check-tiles`. .. cmdoption:: -p , --processes @@ -154,6 +177,23 @@ only have to use once-in-a-while. This option can also be specified in the config file as :ref:`processes ` +.. cmdoption:: -v, --verbose + + Activate a more verbose logging format and turn on debugging output. This + can be quite noisy but also gives a lot more info on what The Overviewer is + doing. + +.. cmdoption:: -q, --quiet + + Turns off one level of logging for quieter output. You can specify this more + than once. One ``-q`` will suppress all INFO lines. Two will suppress all + INFO and WARNING lines. And so on for ERROR and CRITICAL log messages. + + If :option:`--verbose<-v>` is given, then the first ``-q`` will counteract + the DEBUG lines, but not the more verbose logging format. Thus, you can + specify ``-v -q`` to get only INFO logs and higher (no DEBUG) but with the + more verbose logging format. + .. _installing-textures: Installing the Textures diff --git a/overviewer.py b/overviewer.py index d8800ba..d7f72d0 100755 --- a/overviewer.py +++ b/overviewer.py @@ -402,8 +402,6 @@ dir but you forgot to put quotes around the directory, since it contains spaces. # create our TileSet from this RegionSet tileset_dir = os.path.abspath(os.path.join(destdir, render_name)) - if not os.path.exists(tileset_dir): - os.mkdir(tileset_dir) # only pass to the TileSet the options it really cares about render['name'] = render_name # perhaps a hack. This is stored here for the asset manager diff --git a/overviewer_core/tileset.py b/overviewer_core/tileset.py index b889286..8102328 100644 --- a/overviewer_core/tileset.py +++ b/overviewer_core/tileset.py @@ -178,10 +178,7 @@ class TileSet(object): This class does nothing with it except pass it through. outputdir is the absolute path to the tile output directory where the - tiles are saved. It is assumed to exist already. - TODO: This should probably be relative to the asset manager's output - directory to avoid redundancy. - + tiles are saved. It is created if it doesn't exist Current valid options for the options dictionary are shown below. All the options must be specified unless they are not relevant. If the @@ -211,12 +208,15 @@ class TileSet(object): partial interrupted render left off. 1 - For render-tiles, render all whose chunks have an mtime greater - than the mtime of the tile on disk, and their upper-tile - ancestors. + "check-tiles" mode. For render-tiles, render all whose chunks + have an mtime greater than the mtime of the tile on disk, and + their upper-tile ancestors. Also check all other upper-tiles and render any that have children with more rencent mtimes than itself. + + Also remove tiles and directory trees that do exist but + shouldn't. This is slower due to stat calls to determine tile mtimes, but safe if the last render was interrupted. @@ -270,6 +270,7 @@ class TileSet(object): self.regionset = regionsetobj self.am = assetmanagerobj self.textures = texturesobj + self.outputdir = os.path.abspath(outputdir) config = self.am.get_tileset_config(self.options.get("name")) self.config = config @@ -277,17 +278,42 @@ class TileSet(object): self.last_rendertime = config.get('last_rendertime', 0) if "renderchecks" not in self.options: + # renderchecks was not given, this indicates it was not specified + # in either the config file or the command line. The following code + # attempts to detect the most appropriate mode if not config: - # No persistent config? This is a full render then. + # No persistent config? + if os.path.exists(self.outputdir): + # Somehow there's no config but the output dir DOES exist. + # That's strange! + logging.warning( + "For render '%s' I couldn't find any persistent config, " + "but I did find my tile directory already exists. This " + "shouldn't normally happen, something may be up, but I " + "think I can continue...", self.options['name']) + logging.info("Switching to --check-tiles mode") + self.options['renderchecks'] = 1 + else: + # This is the typical code path for an initial render, make + # this a "forcerender" + self.options['renderchecks'] = 2 + logging.debug("This is the first time rendering %s. Doing" + + " a full-render", + self.options['name']) + elif not os.path.exists(self.outputdir): + # Somehow the outputdir got erased but the metadata file is + # still there. That's strange! + logging.warning( + "This is strange. There is metadata for render '%s' but " + "the output directory is missing. This shouldn't " + "normally happen. I guess we have no choice but to do a " + "--forcerender", self.options['name']) self.options['renderchecks'] = 2 - logging.debug("This is the first time rendering %s. Doing" + - " a full-render", - self.options['name']) elif config.get("render_in_progress", False): # The last render must have been interrupted. The default should be - # 1 then, not 0 + # a check-tiles render then logging.warning( - "The last render for %s didn't finish. I'll be " + + "The last render for '%s' didn't finish. I'll be " + "scanning all the tiles to make sure everything's up "+ "to date.", self.options['name'], @@ -304,9 +330,15 @@ class TileSet(object): ) self.options['renderchecks'] = 0 - # Throughout the class, self.outputdir is an absolute path to the - # directory where we output tiles. It is assumed to exist. - self.outputdir = os.path.abspath(outputdir) + if not os.path.exists(self.outputdir): + if self.options['renderchecks'] != 2: + logging.warning( + "The tile directory didn't exist, but you have specified " + "explicitly not to do a --fullrender (which is the default for " + "this situation). I'm overriding your decision and setting " + "--fullrender for just this run") + self.options['rednerchecks'] = 2 + os.mkdir(self.outputdir) # Set the image format according to the options if self.options['imgformat'] == 'png': @@ -375,6 +407,7 @@ class TileSet(object): This method returns an iterator over (obj, [dependencies, ...]) """ + # The following block of code implementes the changelist functionality. fd = self.options.get("changelist", None) if fd: logging.debug("Changelist activated for %s (fileno %s)", self, fd) @@ -492,7 +525,6 @@ class TileSet(object): return d - def _find_chunk_range(self): """Finds the chunk range in rows/columns and stores them in self.minrow, self.maxrow, self.mincol, self.maxcol @@ -569,6 +601,11 @@ class TileSet(object): logging.warning("Your map seems to have shrunk. Did you delete some chunks? No problem. Re-arranging tiles, just a sec...") for _ in xrange(curdepth - self.treedepth): self._decrease_depth() + logging.info( + "There done. I'm switching to --check-tiles mode for " + "this one render. This will make sure any old tiles that " + "should no longer exist are deleted.") + self.options['renderchecks'] = 1 def _increase_depth(self): """Moves existing tiles into place for a larger tree""" @@ -928,11 +965,17 @@ class TileSet(object): os.utime(tmppath, (max_chunk_mtime, max_chunk_mtime)) def _iterate_and_check_tiles(self, path): - """A generator function over all tiles that need rendering in the - subtree identified by path. This yields, in order, all tiles that need + """A generator function over all tiles that should exist in the subtree + identified by path. This yields, in order, all tiles that need rendering in a post-traversal order, including this node itself. - This method actually yields tuples in this form: + This method takes one parameter: + + path + The path of a tile that should exist + + + This method yields tuples in this form: (path, mtime, needs_rendering) path is the path tuple of the tile that needs rendering @@ -945,8 +988,9 @@ class TileSet(object): is a boolean indicating this tile does in fact need rendering. (Since this is a recursive generator, tiles that don't need rendering - are not propagated all the way out of the recursive stack, but the - immediate parent call needs to know its mtime) + are not propagated all the way out of the recursive stack, but are + still yielded to the immediate parent because it needs to know its + childs' mtimes) """ if len(path) == self.treedepth: @@ -967,7 +1011,16 @@ class TileSet(object): # If a tile has been modified more recently than any of its # chunks, then this could indicate a potential issue with # this or future renders. - logging.warning("I found a tile with a more recent modification time than any of its chunks. This can happen when a tile has been modified with an outside program, or by a copy utility that doesn't preserve mtimes. Overviewer uses the filesystem's mtimes to determine which tiles need rendering and which don't, so it's important to preserve the mtimes Overviewer sets. Please see our FAQ page on docs.overviewer.org or ask us in IRC for more information") + logging.warning( + "I found a tile with a more recent modification time " + "than any of its chunks. This can happen when a tile has " + "been modified with an outside program, or by a copy " + "utility that doesn't preserve mtimes. Overviewer uses " + "the filesystem's mtimes to determine which tiles need " + "rendering and which don't, so it's important to " + "preserve the mtimes Overviewer sets. Please see our FAQ " + "page on docs.overviewer.org or ask us in IRC for more " + "information") logging.warning("Tile was: %s", imgpath) if max_chunk_mtime > tile_mtime: @@ -975,20 +1028,23 @@ class TileSet(object): yield (path, None, True) else: # This doesn't need rendering. Return mtime to parent in case - # they do need to render + # its mtime is less, indicating the parent DOES need a render yield path, max_chunk_mtime, False else: # A composite-tile. - # First, recurse to each of our children render_me = False max_child_mtime = 0 + + # First, recurse to each of our children for childnum in xrange(4): childpath = path + (childnum,) # Check if this sub-tree should actually exist, so that we only # end up checking tiles that actually exist if not self.dirtytree.query_path(childpath): + # Here check if it *does* exist, and if so, nuke it. + self._nuke_path(childpath) continue for child_path, child_mtime, child_needs_rendering in \ @@ -1034,6 +1090,31 @@ class TileSet(object): # Nope. yield path, max_child_mtime, False + def _nuke_path(self, path): + """Given a quadtree path, erase it from disk. This is called by + _iterate_and_check_tiles() as a helper-method. + + """ + if len(path) == self.treedepth: + # path referrs to a single tile + tileobj = RenderTile.from_path(path) + imgpath = tileobj.get_filepath(self.outputdir, self.imgextension) + if os.path.exists(imgpath): + # No need to catch ENOENT since this is only called from the + # master process + logging.debug("Found an image that shouldn't exist. Deleting it: %s", imgpath) + os.remove(imgpath) + else: + # path referrs to a composite tile, and by extension a directory + dirpath = os.path.join(self.outputdir, *(str(x) for x in path)) + imgpath = dirpath + "." + self.imgextension + if os.path.exists(imgpath): + logging.debug("Found an image that shouldn't exist. Deleting it: %s", imgpath) + os.remove(imgpath) + if os.path.exists(dirpath): + logging.debug("Found a subtree that shouldn't exist. Deleting it: %s", dirpath) + shutil.rmtree(dirpath) + ## ## Functions for converting (x, z) to (col, row) and back ## From bc1118df35006d4b3dfa3494e0262af6a53de51c Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Sun, 11 Mar 2012 01:40:32 -0500 Subject: [PATCH 02/15] clarified docs around the tile check options --- docs/running.rst | 87 ++++++++++++++++++++++++++---------------------- 1 file changed, 48 insertions(+), 39 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 0c37284..d751b41 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -113,42 +113,6 @@ Options These options change the way the render works, and are intended to be things you only have to use once-in-a-while. -.. cmdoption:: --forcerender - - Forces The Overviewer to re-render every tile regardless of whether it - thinks it needs updating or not. It does no tile mtime checks, nor does it - check any last modification time of the world. It unconditionally renders - every tile that exists. - - The caveat with this option is that it does *no* checks, period. Meaning it - will not detect tiles that do exist, but shouldn't (this can happen if your - world shrinks for some reason. For that specific case, - :option:`--check-tiles` is actually more appropriate). - - This is the default mode for first-time renders. This option conflicts with - :option:`--check-tiles` and :option:`--no-tile-checks` - -.. cmdoption:: --check-tiles - - Forces The Overviewer to check each tile on disk and check to make sure it - is up to date. This also checks for tiles that shouldn't exist and deletes - them. - - This is slightly slower than :option:`--no-tile-checks` due to the disk-io - involved in reading tile mtimes and reading the world's region file headers - for chunk mtimes, but can be useful if there are some tiles that somehow got - skipped. - - The caveat with this option is that it compares the tile mtimes on disk with - the chunk mtimes reported by Minecraft. Thus, if the Minecraft world was not - updated since the last render, tiles will not be detected as needing - updates. This means you cannot use this option in response to a changed - configuration setting. - - This option is the default when The Overviewer detects the last render was - interrupted midway through. This option conflicts with - :option:`--forcerender` and :option:`--no-tile-checks` - .. cmdoption:: --no-tile-checks With this option, The Overviewer will determine which tiles to render by @@ -161,14 +125,59 @@ only have to use once-in-a-while. disk IO. The caveat is that the *only* thing to trigger a tile update is if Minecraft - updates a chunk. Any other reason for needing to re-render a tile requires a - different mode to detect the needed update. It could also cause problems if - the system clock of the machine running Minecraft is not stable. + updates a chunk. Any other reason a tile may have for needing re-rendering + is not detected. It could also cause problems if the system clock of the + machine running Minecraft is not stable. **This option is the default** unless :option:`--forcerender` or :option:`--check-tiles` is in effect. This option conflicts with :option:`--forcerender` and :option:`--check-tiles`. +.. cmdoption:: --check-tiles + + Forces The Overviewer to check each tile on disk and check to make sure it + is up to date. This also checks for tiles that shouldn't exist and deletes + them. + + This is functionally equivalent to :option:`--no-tile-checks` with the + difference that each tile is individually checked. It is therefore useful if + the tiles are not consistent with the last-render timestamp that is + automatically stored. This option was designed to handle the case where the + last render was interrupted -- some tiles have been updated but others + haven't, so each one is checked before it is rendered. + + This is slightly slower than :option:`--no-tile-checks` due to the + additonaly disk-io involved in reading tile mtimes from the filesystem + + Since this option also checks for erroneous tiles, **It is also useful after + you delete sections of your map, e.g. with worldedit, to delete tiles that + should no longer exist.** + + The caveat with this option is that it compares the tile mtimes on disk with + the chunk mtimes reported by Minecraft. Thus, it will *not* detect tiles as + needing an update if the Minecraft world hasn't changed. This means you + cannot use this option in response to a change in configuration setting. + + This option is automatically activated when The Overviewer detects the last + render was interrupted midway through. This option conflicts with + :option:`--forcerender` and :option:`--no-tile-checks` + +.. cmdoption:: --forcerender + + Forces The Overviewer to re-render every tile regardless of whether it + thinks it needs updating or not. It does no tile mtime checks, nor does it + check any last modification time of the world. It unconditionally renders + every tile that exists. + + The caveat with this option is that it does *no* checks, period. Meaning it + will not detect tiles that do exist, but shouldn't (this can happen if your + world shrinks for some reason. For that specific case, + :option:`--check-tiles` is actually more appropriate). + + This option is automatically activated for first-time renders. This option + conflicts with :option:`--check-tiles` and :option:`--no-tile-checks` + + .. cmdoption:: -p , --processes This specifies the number of worker processes to spawn on the local machine From 3fd9b8210881ff641e638689a7ea4a0f99ca5709 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Sun, 11 Mar 2012 01:55:34 -0500 Subject: [PATCH 03/15] more doc clarification about the render mode options --- docs/running.rst | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index d751b41..89a422d 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -110,8 +110,9 @@ The first render can take a while, depending on the size of your world. Options ------- -These options change the way the render works, and are intended to be things you -only have to use once-in-a-while. +The following three options change the way the checks for tiles to update, and +are intended to be things you only have to use in special situations. You should +not normally have to specify these options, the default is typically correct. .. cmdoption:: --no-tile-checks @@ -126,8 +127,9 @@ only have to use once-in-a-while. The caveat is that the *only* thing to trigger a tile update is if Minecraft updates a chunk. Any other reason a tile may have for needing re-rendering - is not detected. It could also cause problems if the system clock of the - machine running Minecraft is not stable. + is not detected. This means that changes in your render configuration will + not be reflected in your world except in updated chunks. It could also cause + problems if the system clock of the machine running Minecraft is not stable. **This option is the default** unless :option:`--forcerender` or :option:`--check-tiles` is in effect. This option conflicts with @@ -153,10 +155,10 @@ only have to use once-in-a-while. you delete sections of your map, e.g. with worldedit, to delete tiles that should no longer exist.** - The caveat with this option is that it compares the tile mtimes on disk with - the chunk mtimes reported by Minecraft. Thus, it will *not* detect tiles as - needing an update if the Minecraft world hasn't changed. This means you - cannot use this option in response to a change in configuration setting. + The caveats with this option are the same as for :option:`--no-tile-checks` + with the additional caveat that tile timestamps in the filesystem must be + preserved. If you copy tiles or make changes to them with an external tool + that modifies mtimes of tiles, it could cause problems with this option. This option is automatically activated when The Overviewer detects the last render was interrupted midway through. This option conflicts with @@ -165,14 +167,17 @@ only have to use once-in-a-while. .. cmdoption:: --forcerender Forces The Overviewer to re-render every tile regardless of whether it - thinks it needs updating or not. It does no tile mtime checks, nor does it - check any last modification time of the world. It unconditionally renders - every tile that exists. + thinks it needs updating or not. It does no tile mtime checks, and therefore + ignores the last render time of the world and the last modification time of + each chunk. It unconditionally renders every tile that exists. The caveat with this option is that it does *no* checks, period. Meaning it will not detect tiles that do exist, but shouldn't (this can happen if your world shrinks for some reason. For that specific case, - :option:`--check-tiles` is actually more appropriate). + :option:`--check-tiles` is actually the appropriate mode). + + This option is useful if you have changed a render setting and wish to + re-render every tile with the new settings. This option is automatically activated for first-time renders. This option conflicts with :option:`--check-tiles` and :option:`--no-tile-checks` From 6b77d545557b65d47cb9b8ba6bae471ae8e02d3e Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sat, 3 Mar 2012 23:42:05 -0500 Subject: [PATCH 04/15] Reimplement signs/POIs --- genPOI.py | 132 +++++++++++++++++ overviewer_core/assetmanager.py | 18 +-- overviewer_core/data/js_src/overviewer.js | 9 +- overviewer_core/data/js_src/util.js | 36 ++++- overviewer_core/data/js_src/views.js | 156 +++++++++++++++++++++ overviewer_core/data/web_assets/index.html | 1 + overviewer_core/settingsDefinition.py | 1 + overviewer_core/settingsValidators.py | 7 + 8 files changed, 346 insertions(+), 14 deletions(-) create mode 100755 genPOI.py diff --git a/genPOI.py b/genPOI.py new file mode 100755 index 0000000..6bb0123 --- /dev/null +++ b/genPOI.py @@ -0,0 +1,132 @@ +#!/usr/bin/python2 + +''' +genPOI.py + +Scans regionsets for TileEntities and Entities, filters them, and writes out +POI/marker info. + +A markerSet is list of POIs to display on a tileset. It has a display name, +and a group name. + +markersDB.js holds a list of POIs in each group +markers.js holds a list of which markerSets are attached to each tileSet + + +''' +import os +import logging +import json +from optparse import OptionParser + +from overviewer_core import logger +from overviewer_core import nbt +from overviewer_core import configParser, world + +helptext = """ +%prog --config=""" + +logger.configure() + +def handleSigns(rset, outputdir, render, rname): + + if hasattr(rset, "_pois"): + return + + logging.info("Looking for entities in %r", rset) + + filters = render['markers'] + rset._pois = dict(TileEntities=[], Entities=[]) + + for (x,z,mtime) in rset.iterate_chunks(): + data = rset.get_chunk(x,z) + rset._pois['TileEntities'] += data['TileEntities'] + rset._pois['Entities'] += data['Entities'] + + +def main(): + parser = OptionParser(usage=helptext) + parser.add_option("--config", dest="config", action="store", help="Specify the config file to use.") + + options, args = parser.parse_args() + if not options.config: + parser.print_help() + return + + # Parse the config file + mw_parser = configParser.MultiWorldParser() + mw_parser.parse(options.config) + try: + config = mw_parser.get_validated_config() + except Exception: + logging.exception("An error was encountered with your configuration. See the info below.") + return 1 + + destdir = config['outputdir'] + # saves us from creating the same World object over and over again + worldcache = {} + + markersets = set() + markers = dict() + + for rname, render in config['renders'].iteritems(): + try: + worldpath = config['worlds'][render['world']] + except KeyError: + logging.error("Render %s's world is '%s', but I could not find a corresponding entry in the worlds dictionary.", + rname, render['world']) + return 1 + render['worldname_orig'] = render['world'] + render['world'] = worldpath + + # find or create the world object + if (render['world'] not in worldcache): + w = world.World(render['world']) + worldcache[render['world']] = w + else: + w = worldcache[render['world']] + + rset = w.get_regionset(render['dimension']) + if rset == None: # indicates no such dimension was found: + logging.error("Sorry, you requested dimension '%s' for %s, but I couldn't find it", render['dimension'], render_name) + return 1 + + for f in render['markers']: + markersets.add((f, rset)) + name = f.__name__ + hex(hash(f))[-4:] + "_" + hex(hash(rset))[-4:] + try: + l = markers[rname] + l.append(dict(groupName=name, displayName = f.__doc__)) + except KeyError: + markers[rname] = [dict(groupName=name, displayName=f.__doc__),] + + handleSigns(rset, os.path.join(destdir, rname), render, rname) + + logging.info("Done scanning regions") + logging.info("Writing out javascript files") + markerSetDict = dict() + for (flter, rset) in markersets: + # generate a unique name for this markerset. it will not be user visible + name = flter.__name__ + hex(hash(flter))[-4:] + "_" + hex(hash(rset))[-4:] + markerSetDict[name] = dict(created=False, raw=[]) + for poi in rset._pois['TileEntities']: + if flter(poi): + markerSetDict[name]['raw'].append(poi) + #print markerSetDict + + with open(os.path.join(destdir, "markersDB.js"), "w") as output: + output.write("var markersDB=") + json.dump(markerSetDict, output, indent=2) + output.write(";\n"); + with open(os.path.join(destdir, "markers.js"), "w") as output: + output.write("var markers=") + json.dump(markers, output, indent=2) + output.write(";\n"); + with open(os.path.join(destdir, "baseMarkers.js"), "w") as output: + output.write("overviewer.util.injectMarkerScript('markersDB.js');\n") + output.write("overviewer.util.injectMarkerScript('markers.js');\n") + output.write("overviewer.collections.haveSigns=true;\n") + logging.info("Done") + +if __name__ == "__main__": + main() diff --git a/overviewer_core/assetmanager.py b/overviewer_core/assetmanager.py index 131d986..6407c1d 100644 --- a/overviewer_core/assetmanager.py +++ b/overviewer_core/assetmanager.py @@ -42,11 +42,6 @@ directory. self.outputdir = outputdir self.renders = dict() - # stores Points Of Interest to be mapped with markers - # This is a dictionary of lists of dictionaries - # Each regionset's name is a key in this dictionary - self.POI = dict() - # look for overviewerConfig in self.outputdir try: with open(os.path.join(self.outputdir, "overviewerConfig.js")) as c: @@ -65,13 +60,6 @@ directory. return dict() - - def found_poi(self, regionset, poi_type, contents, chunkX, chunkY): - if regionset.name not in self.POI.keys(): - POI[regionset.name] = [] - # TODO based on the type, so something - POI[regionset.name].append - def initialize(self, tilesets): """Similar to finalize() but calls the tilesets' get_initial_data() instead of get_persistent_data() to compile the generated javascript @@ -152,6 +140,12 @@ directory. global_assets = os.path.join(util.get_program_path(), "web_assets") mirror_dir(global_assets, self.outputdir) + # write a dummy baseMarkers.js if none exists + if not os.path.exists(os.path.join(self.outputdir, "baseMarkers.js")): + with open(os.path.join(self.outputdir, "baseMarkers.js"), "w") as f: + f.write("// if you wants signs, please see genPOI.py\n"); + + # create overviewer.js from the source js files js_src = os.path.join(util.get_program_path(), "overviewer_core", "data", "js_src") if not os.path.isdir(js_src): diff --git a/overviewer_core/data/js_src/overviewer.js b/overviewer_core/data/js_src/overviewer.js index 2f9d651..a8ad80b 100644 --- a/overviewer_core/data/js_src/overviewer.js +++ b/overviewer_core/data/js_src/overviewer.js @@ -29,7 +29,14 @@ overviewer.collections = { */ 'infoWindow': null, - 'worldViews': [] + 'worldViews': [], + + 'haveSigns': false, + + /** + * Hold the raw marker data for each tilest + */ + 'markerInfo': {} }; overviewer.classes = { diff --git a/overviewer_core/data/js_src/util.js b/overviewer_core/data/js_src/util.js index 2daa770..143ba5f 100644 --- a/overviewer_core/data/js_src/util.js +++ b/overviewer_core/data/js_src/util.js @@ -58,6 +58,11 @@ overviewer.util = { var coordsdiv = new overviewer.views.CoordboxView({tagName: 'DIV'}); coordsdiv.render(); + if (overviewer.collections.haveSigns) { + var signs = new overviewer.views.SignControlView(); + signs.registerEvents(signs); + } + // Update coords on mousemove google.maps.event.addListener(overviewer.map, 'mousemove', function (event) { coordsdiv.updateCoords(event.latLng); @@ -102,6 +107,7 @@ overviewer.util = { overviewer.map.setZoom(zoom); } + }); var worldSelector = new overviewer.views.WorldSelectorView({tagName:'DIV'}); @@ -116,15 +122,43 @@ overviewer.util = { // Jump to the hash if given overviewer.util.initHash(); + overviewer.util.initializeMarkers(); /* overviewer.util.initializeMapTypes(); overviewer.util.initializeMap(); - overviewer.util.initializeMarkers(); overviewer.util.initializeRegions(); overviewer.util.createMapControls(); */ }, + + 'injectMarkerScript': function(url) { + var m = document.createElement('script'); m.type = 'text/javascript'; m.async = false; + m.src = url; + var s = document.getElementsByTagName('script')[0]; s.parentNode.appendChild(m); + }, + + 'initializeMarkers': function() { + return; + + }, + + 'createMarkerInfoWindow': function(marker) { + var windowContent = '

' + marker.title.replace(/\n/g,'
') + '

'; + var infowindow = new google.maps.InfoWindow({ + 'content': windowContent + }); + google.maps.event.addListener(marker, 'click', function() { + if (overviewer.collections.infoWindow) { + overviewer.collections.infoWindow.close(); + } + infowindow.open(overviewer.map, marker); + overviewer.collections.infoWindow = infowindow; + }); + }, + + /** * This adds some methods to these classes because Javascript is stupid * and this seems like the best way to avoid re-creating the same methods diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index 70867e0..7a1921e 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -219,3 +219,159 @@ overviewer.views.GoogleMapView = Backbone.View.extend({ }); + + +/** + * SignControlView + */ +overviewer.views.SignControlView = Backbone.View.extend({ + /** SignControlView::initialize + */ + initialize: function(opts) { + $(this.el).addClass("customControl"); + overviewer.map.controls[google.maps.ControlPosition.TOP_RIGHT].push(this.el); + + }, + registerEvents: function(me) { + google.maps.event.addListener(overviewer.map, 'maptypeid_changed', function(event) { + overviewer.mapView.updateCurrentTileset(); + me.render(); + // hide markers, if necessary + // for each markerSet, check: + // if the markerSet isnot part of this tileset, hide all of the markers + var curMarkerSet = overviewer.mapView.options.currentTileSet.attributes.path; + console.log("sign control things %r is the new current tileset", curMarkerSet); + var dataRoot = markers[curMarkerSet]; + var groupsForThisTileSet = jQuery.map(dataRoot, function(elem, i) { return elem.groupName;}) + for (markerSet in markersDB) { + console.log("checking to see if markerset %r should be hidden (is it not in %r)", markerSet, groupsForThisTileSet); + if (jQuery.inArray(markerSet, groupsForThisTileSet) == -1){ + // hide these + console.log("nope, going to hide these"); + if (markersDB[markerSet].created) { + jQuery.each(markersDB[markerSet].raw, function(i, elem) { + //console.log("attempting to set %r to visible(%r)", elem.markerObj, checked); + elem.markerObj.setVisible(false); + }); + } + } else { + //make sure the checkbox is checked + //TODO fix this + //console.log("trying to checkbox for " + markerSet); + //console.log($("[_mc_groupname=" + markerSet + "]")); + } + + } + + }); + + }, + /** + * SignControlView::render + */ + render: function() { + + var curMarkerSet = overviewer.mapView.options.currentTileSet.attributes.path; + console.log(curMarkerSet); + //var dataRoot = overviewer.collections.markerInfo[curMarkerSet]; + var dataRoot = markers[curMarkerSet]; + + console.log(dataRoot); + + // before re-building this control, we need to hide all currently displayed signs + // TODO + + this.el.innerHTML="" + + var controlText = document.createElement('DIV'); + controlText.innerHTML = "Signs"; + + var controlBorder = document.createElement('DIV'); + $(controlBorder).addClass('top'); + this.el.appendChild(controlBorder); + controlBorder.appendChild(controlText); + + var dropdownDiv = document.createElement('DIV'); + $(dropdownDiv).addClass('dropDown'); + this.el.appendChild(dropdownDiv); + dropdownDiv.innerHTML=''; + + // add the functionality to toggle visibility of the items + $(controlText).click(function() { + $(controlBorder).toggleClass('top-active'); + $(dropdownDiv).toggle(); + }); + + + // add some menus + for (i in dataRoot) { + var group = dataRoot[i]; + console.log(group); + this.addItem({label: group.displayName, groupName:group.groupName, action:function(this_item, checked) { + console.log("%r is %r", this_item, checked); + console.log("group name is %r", this_item.groupName); + jQuery.each(markersDB[this_item.groupName].raw, function(i, elem) { + //console.log("attempting to set %r to visible(%r)", elem.markerObj, checked); + elem.markerObj.setVisible(checked); + }); + }}); + } + + iconURL = overviewerConfig.CONST.image.signMarker; + //dataRoot['markers'] = []; + // + for (i in dataRoot) { + var groupName = dataRoot[i].groupName; + if (!markersDB[groupName].created) { + for (j in markersDB[groupName].raw) { + var entity = markersDB[groupName].raw[j]; + var marker = new google.maps.Marker({ + 'position': overviewer.util.fromWorldToLatLng(entity.x, + entity.y, entity.z, overviewer.mapView.options.currentTileSet), + 'map': overviewer.map, + 'title': jQuery.trim(entity.Text1 + "\n" + entity.Text2 + "\n" + entity.Text3 + "\n" + entity.Text4), + 'icon': iconURL, + 'visible': false + }); + if (entity['id'] == 'Sign') { + overviewer.util.createMarkerInfoWindow(marker); + } + console.log("Added marker for %r", entity); + jQuery.extend(entity, {markerObj: marker}); + } + markersDB[groupName].created = true; + } + } + + + }, + addItem: function(item) { + var itemDiv = document.createElement('div'); + var itemInput = document.createElement('input'); + itemInput.type='checkbox'; + + // give it a name + $(itemInput).data('label',item.label); + $(itemInput).attr("_mc_groupname", item.groupName); + jQuery(itemInput).click((function(local_item) { + return function(e) { + item.action(local_item, e.target.checked); + }; + })(item)); + + this.$(".dropDown")[0].appendChild(itemDiv); + itemDiv.appendChild(itemInput); + var textNode = document.createElement('text'); + if(item.icon) { + textNode.innerHTML = '' + item.label + '
'; + } else { + textNode.innerHTML = item.label + '
'; + } + + itemDiv.appendChild(textNode); + + + }, +}); + diff --git a/overviewer_core/data/web_assets/index.html b/overviewer_core/data/web_assets/index.html index 0766727..e8010a0 100644 --- a/overviewer_core/data/web_assets/index.html +++ b/overviewer_core/data/web_assets/index.html @@ -17,6 +17,7 @@ + diff --git a/overviewer_core/settingsDefinition.py b/overviewer_core/settingsDefinition.py index 67fee81..5773dd2 100644 --- a/overviewer_core/settingsDefinition.py +++ b/overviewer_core/settingsDefinition.py @@ -75,6 +75,7 @@ renders = Setting(required=True, default=util.OrderedDict(), "rerenderprob": Setting(required=True, validator=validateFloat, default=0), "crop": Setting(required=False, validator=validateCrop, default=None), "changelist": Setting(required=False, validator=validateStr, default=None), + "markers": Setting(required=False, validator=validateMarkers, default=[]), # Remove this eventually (once people update their configs) "worldname": Setting(required=False, default=None, diff --git a/overviewer_core/settingsValidators.py b/overviewer_core/settingsValidators.py index f6fdd77..8aedc30 100644 --- a/overviewer_core/settingsValidators.py +++ b/overviewer_core/settingsValidators.py @@ -43,6 +43,13 @@ def checkBadEscape(s): fixed = True return (fixed, fixed_string) +def validateMarkers(filterlist): + if type(filterlist) != list: + raise ValidationException("Markers must specify a list of filters") + for x in filterlist: + if not callable(x): + raise ValidationException("%r must be a function"% x) + return filterlist def validateWorldPath(worldpath): _, worldpath = checkBadEscape(worldpath) From 10806381d909e7430caddbea6616b81643c85874 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Sun, 11 Mar 2012 13:10:02 -0400 Subject: [PATCH 05/15] even more doc tweaks --- docs/running.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 89a422d..7b9d26b 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -110,9 +110,10 @@ The first render can take a while, depending on the size of your world. Options ------- -The following three options change the way the checks for tiles to update, and -are intended to be things you only have to use in special situations. You should -not normally have to specify these options, the default is typically correct. +The following three options change the way The Overviewer determines which tiles +to update, and are intended to be things you only have to use in special +situations. You should not normally have to specify these options; the default +is typically correct. .. cmdoption:: --no-tile-checks @@ -168,8 +169,9 @@ not normally have to specify these options, the default is typically correct. Forces The Overviewer to re-render every tile regardless of whether it thinks it needs updating or not. It does no tile mtime checks, and therefore - ignores the last render time of the world and the last modification time of - each chunk. It unconditionally renders every tile that exists. + ignores the last render time of the world, the last modification times of + each chunk, and the filesystem mtimes of each tile. It unconditionally + renders every tile that exists. The caveat with this option is that it does *no* checks, period. Meaning it will not detect tiles that do exist, but shouldn't (this can happen if your From bfeae91e7834066d4c1a1f88e4a9f1332304d2a6 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 13:10:50 -0400 Subject: [PATCH 06/15] New docs on signs/markers --- docs/config.rst | 15 ++++++++++ docs/index.rst | 1 + docs/signs.rst | 75 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+) create mode 100644 docs/signs.rst diff --git a/docs/config.rst b/docs/config.rst index cac1d81..1affc01 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -443,6 +443,21 @@ values. The valid configuration keys are listed below. removed some tiles, you may need to do some manual deletion on the remote side. +.. _option_markers: + +``markers`` + This controls the display of markers, signs, and other points of interest + in the output HTML. It should be a list of filter functions. + + .. note:: + + Setting this configuration option alone does nothing. In order to get + markers and signs on our map, you must also run the genPO script. See + the :doc:`Signs and markers` section for more details and documenation. + + + **Default:** ``[]`` (an empty list) + .. _customrendermodes: Custom Rendermodes and Rendermode Primitives diff --git a/docs/index.rst b/docs/index.rst index 2447f22..5bbac6d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -181,6 +181,7 @@ Documentation Contents building running config + signs win_tut/windowsguide faq design/designdoc diff --git a/docs/signs.rst b/docs/signs.rst new file mode 100644 index 0000000..1a2177f --- /dev/null +++ b/docs/signs.rst @@ -0,0 +1,75 @@ +.. _signsmarkers: + +================= +Signs and Markers +================= + +The Overviewer can display signs, markers, and other points of interest on your +map. This works a little differently than it has in the past, so be sure to read +these docs carefully. + +In these docs, we use the term POI (or point of interest) to refer to entities and +tileentities. + + +Configuration File +================== + + +Filter Functions +---------------- + +A filter function is a python function that is used to figure out if a given POI +should be part of a markerSet of not. The function should accept one argument +(a dictionary, also know as an associative array), and return a boolean:: + + def signFilter(poi): + "All signs" + return poi['id'] == 'Sign' + +The single argument will either a TileEntity, or an Entity taken directly from +the chunk file. In this example, this function returns true only if the type +of entity is a sign. For more information of TileEntities and Entities, see +the `Chunk Format `_ page on +the Minecraft Wiki. + +.. note:: + The doc string ("All signs" in this example) is important. It is the label + that appears in your rendered map + +A more advanced filter may also look at other entity fields, such as the sign text:: + + def goldFilter(poi): + "Gold" + return poi['id'] == 'Sign' and (\ + 'gold' in poi['Text1] or + 'gold' in poi['Text2']) + +This looks for the word 'gold' in either the first or second line of the signtext. + +Since writing these filters can be a little tedious, a set of predefined filters +functions are provided. See the :ref:`predefined_filter_functions` section for +details. + +Render Dictionary Key +--------------------- + +Each render can specify a list of zero or more filter functions. Each of these +filter functions become a selectable item in the 'Signs' drop-down menu in the +rendered map. For example:: + + renders['myrender'] = { + 'world': 'myworld', + 'title': "Example", + 'markers': [allFilter, anotherFilter], + } + + + +.. _predefined_filter_functions: + +Predefined Filter Functions +=========================== + +TODO write some filter functions, then document them here + From aba0449477ec4f274a5810bfea202045a42efac9 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 13:30:49 -0400 Subject: [PATCH 07/15] Hide the 'Signs' control if the tileset has no signs --- overviewer_core/data/js_src/views.js | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index 7a1921e..c5bd291 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -242,6 +242,18 @@ overviewer.views.SignControlView = Backbone.View.extend({ var curMarkerSet = overviewer.mapView.options.currentTileSet.attributes.path; console.log("sign control things %r is the new current tileset", curMarkerSet); var dataRoot = markers[curMarkerSet]; + if (!dataRoot) { + // this tileset has no signs, so hide all of them + for (markerSet in markersDB) { + if (markersDB[markerSet].created) { + jQuery.each(markersDB[markerSet].raw, function(i, elem) { + elem.markerObj.setVisible(false); + }); + } + } + + return; + } var groupsForThisTileSet = jQuery.map(dataRoot, function(elem, i) { return elem.groupName;}) for (markerSet in markersDB) { console.log("checking to see if markerset %r should be hidden (is it not in %r)", markerSet, groupsForThisTileSet); @@ -278,10 +290,11 @@ overviewer.views.SignControlView = Backbone.View.extend({ console.log(dataRoot); - // before re-building this control, we need to hide all currently displayed signs - // TODO - this.el.innerHTML="" + + // if we have no markerSets for this tileset, do nothing: + if (!dataRoot) { return; } + var controlText = document.createElement('DIV'); controlText.innerHTML = "Signs"; From 5b168ebdbdfac9d3363677dd6c90cb962ffb6346 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 13:33:02 -0400 Subject: [PATCH 08/15] Removed console debugging output --- overviewer_core/data/js_src/util.js | 2 -- overviewer_core/data/js_src/views.js | 31 ---------------------------- 2 files changed, 33 deletions(-) diff --git a/overviewer_core/data/js_src/util.js b/overviewer_core/data/js_src/util.js index 143ba5f..0a1655f 100644 --- a/overviewer_core/data/js_src/util.js +++ b/overviewer_core/data/js_src/util.js @@ -273,7 +273,6 @@ overviewer.util = { var zoomLevels = model.get("zoomLevels"); var north_direction = model.get('north_direction'); - //console.log("fromWorldToLatLng: north_direction is %r", north_direction); // the width and height of all the highest-zoom tiles combined, // inverted @@ -440,7 +439,6 @@ overviewer.util = { // save this info is a nice easy to parse format var currentWorldView = overviewer.mapModel.get("currentWorldView"); currentWorldView.options.lastViewport = [x,y,z,zoom]; - //console.log("Updated lastViewport: %r" , [x,y,z,zoom]); window.location.replace("#/" + Math.floor(x) + "/" + Math.floor(y) + "/" + Math.floor(z) + "/" + zoom + "/" + w + "/" + maptype); }, 'updateHash': function() { diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index c5bd291..28ba007 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -3,13 +3,9 @@ overviewer.views= {} overviewer.views.WorldView = Backbone.View.extend({ initialize: function(opts) { - //console.log("WorldView::initialize()"); - //console.log(this.model.get("tileSets")); this.options.mapTypes = []; this.options.mapTypeIds = []; this.model.get("tileSets").each(function(tset, index, list) { - //console.log(" eaching"); - //console.log(" Working on tileset %s" , tset.get("name")); var ops = { getTileUrl: overviewer.gmap.getTileUrlGenerator(tset.get("path"), tset.get("base"), tset.get("imgextension")), 'tileSize': new google.maps.Size( @@ -57,7 +53,6 @@ overviewer.views.WorldSelectorView = Backbone.View.extend({ "change select": "changeWorld" }, changeWorld: function() { - //console.log("change world!"); var selectObj = this.$("select")[0]; var selectedOption = selectObj.options[selectObj.selectedIndex]; @@ -120,11 +115,8 @@ overviewer.views.CoordboxView = Backbone.View.extend({ overviewer.views.GoogleMapView = Backbone.View.extend({ initialize: function(opts) { - //console.log(this); this.options.map = null; var curWorld = this.model.get("currentWorldView").model; - //console.log("Current world:"); - //console.log(curWorld); var curTset = curWorld.get("tileSets").at(0); @@ -145,12 +137,6 @@ overviewer.views.GoogleMapView = Backbone.View.extend({ var mapOptions = {}; // - curWorld.get("tileSets").each(function(elem, index, list) { - //console.log("Setting up map for:"); - //console.log(elem); - //console.log("for %s generating url func with %s and %s", elem.get("name"), elem.get("path"), elem.get("base")); - - }); // init the map with some default options. use the first tileset in the first world this.options.mapOptions = { zoom: curTset.get("defaultZoom"), @@ -174,7 +160,6 @@ overviewer.views.GoogleMapView = Backbone.View.extend({ // register every ImageMapType with the map $.each(overviewer.collections.worldViews, function( index, worldView) { $.each(worldView.options.mapTypes, function(i_index, maptype) { - //console.log("registered %s with the maptype registery", worldView.model.get("name") + maptype.shortname); overviewer.map.mapTypes.set(overviewerConfig.CONST.mapDivId + worldView.model.get("name") + maptype.shortname , maptype); }); @@ -185,7 +170,6 @@ overviewer.views.GoogleMapView = Backbone.View.extend({ * Should be called when the current world has changed in GoogleMapModel */ render: function() { - //console.log("GoogleMapView::render()"); var view = this.model.get("currentWorldView"); this.options.mapOptions.mapTypeControlOptions = { mapTypeIds: view.options.mapTypeIds}; @@ -200,14 +184,11 @@ overviewer.views.GoogleMapView = Backbone.View.extend({ * Keeps track of the currently visible tileset */ updateCurrentTileset: function() { - //console.log("GoogleMapView::updateCurrentTileset()"); var currentWorldView = this.model.get("currentWorldView"); var gmapCurrent = overviewer.map.getMapTypeId(); for (id in currentWorldView.options.mapTypeIds) { if (currentWorldView.options.mapTypeIds[id] == gmapCurrent) { - //console.log("updating currenttileset"); this.options.currentTileSet = currentWorldView.model.get("tileSets").at(id); - //console.log(this); } } @@ -240,7 +221,6 @@ overviewer.views.SignControlView = Backbone.View.extend({ // for each markerSet, check: // if the markerSet isnot part of this tileset, hide all of the markers var curMarkerSet = overviewer.mapView.options.currentTileSet.attributes.path; - console.log("sign control things %r is the new current tileset", curMarkerSet); var dataRoot = markers[curMarkerSet]; if (!dataRoot) { // this tileset has no signs, so hide all of them @@ -256,13 +236,10 @@ overviewer.views.SignControlView = Backbone.View.extend({ } var groupsForThisTileSet = jQuery.map(dataRoot, function(elem, i) { return elem.groupName;}) for (markerSet in markersDB) { - console.log("checking to see if markerset %r should be hidden (is it not in %r)", markerSet, groupsForThisTileSet); if (jQuery.inArray(markerSet, groupsForThisTileSet) == -1){ // hide these - console.log("nope, going to hide these"); if (markersDB[markerSet].created) { jQuery.each(markersDB[markerSet].raw, function(i, elem) { - //console.log("attempting to set %r to visible(%r)", elem.markerObj, checked); elem.markerObj.setVisible(false); }); } @@ -284,12 +261,9 @@ overviewer.views.SignControlView = Backbone.View.extend({ render: function() { var curMarkerSet = overviewer.mapView.options.currentTileSet.attributes.path; - console.log(curMarkerSet); //var dataRoot = overviewer.collections.markerInfo[curMarkerSet]; var dataRoot = markers[curMarkerSet]; - console.log(dataRoot); - this.el.innerHTML="" // if we have no markerSets for this tileset, do nothing: @@ -319,12 +293,8 @@ overviewer.views.SignControlView = Backbone.View.extend({ // add some menus for (i in dataRoot) { var group = dataRoot[i]; - console.log(group); this.addItem({label: group.displayName, groupName:group.groupName, action:function(this_item, checked) { - console.log("%r is %r", this_item, checked); - console.log("group name is %r", this_item.groupName); jQuery.each(markersDB[this_item.groupName].raw, function(i, elem) { - //console.log("attempting to set %r to visible(%r)", elem.markerObj, checked); elem.markerObj.setVisible(checked); }); }}); @@ -349,7 +319,6 @@ overviewer.views.SignControlView = Backbone.View.extend({ if (entity['id'] == 'Sign') { overviewer.util.createMarkerInfoWindow(marker); } - console.log("Added marker for %r", entity); jQuery.extend(entity, {markerObj: marker}); } markersDB[groupName].created = true; From a757be510ab101d9311de7f502f4e366144f564d Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 13:48:54 -0400 Subject: [PATCH 09/15] Work around IE issue. This work around has the problem where the first tileset you view won't have the 'signs' control --- overviewer_core/data/js_src/views.js | 3 +++ 1 file changed, 3 insertions(+) diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index 28ba007..ea6a46c 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -216,6 +216,9 @@ overviewer.views.SignControlView = Backbone.View.extend({ registerEvents: function(me) { google.maps.event.addListener(overviewer.map, 'maptypeid_changed', function(event) { overviewer.mapView.updateCurrentTileset(); + + // workaround IE issue. bah! + if (typeof markers=="undefined") { return; } me.render(); // hide markers, if necessary // for each markerSet, check: From 9d97ff4e374b8978a94b8f63c4e0b7a638f72acf Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 14:17:04 -0400 Subject: [PATCH 10/15] Ensure sign checkboxes remain checked across tileset changes --- overviewer_core/data/js_src/views.js | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index ea6a46c..b4ffbb9 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -246,12 +246,10 @@ overviewer.views.SignControlView = Backbone.View.extend({ elem.markerObj.setVisible(false); }); } - } else { - //make sure the checkbox is checked - //TODO fix this - //console.log("trying to checkbox for " + markerSet); - //console.log($("[_mc_groupname=" + markerSet + "]")); + markersDB[markerSet].checked=false; } + // make sure the checkboxes checked if necessary + $("[_mc_groupname=" + markerSet + "]").attr("checked", markersDB[markerSet].checked); } @@ -297,6 +295,7 @@ overviewer.views.SignControlView = Backbone.View.extend({ for (i in dataRoot) { var group = dataRoot[i]; this.addItem({label: group.displayName, groupName:group.groupName, action:function(this_item, checked) { + markersDB[this_item.groupName].checked = checked; jQuery.each(markersDB[this_item.groupName].raw, function(i, elem) { elem.markerObj.setVisible(checked); }); From 729fcda9643254e06e5994ede9b64b65e8f559fb Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Sun, 11 Mar 2012 17:59:29 -0400 Subject: [PATCH 11/15] Display a spawn marker --- overviewer.py | 1 + overviewer_core/data/js_src/overviewer.js | 7 ++++- overviewer_core/data/js_src/util.js | 3 ++ overviewer_core/data/js_src/views.js | 26 ++++++++++++++++ overviewer_core/tileset.py | 2 ++ overviewer_core/world.py | 37 ++++++++++++++++++----- 6 files changed, 68 insertions(+), 8 deletions(-) diff --git a/overviewer.py b/overviewer.py index d7f72d0..96ed27a 100755 --- a/overviewer.py +++ b/overviewer.py @@ -406,6 +406,7 @@ dir but you forgot to put quotes around the directory, since it contains spaces. # only pass to the TileSet the options it really cares about render['name'] = render_name # perhaps a hack. This is stored here for the asset manager tileSetOpts = util.dict_subset(render, ["name", "imgformat", "renderchecks", "rerenderprob", "bgcolor", "imgquality", "optimizeimg", "rendermode", "worldname_orig", "title", "dimension", "changelist"]) + tileSetOpts.update({"spawn": w.find_true_spawn()}) # TODO find a better way to do this tset = tileset.TileSet(rset, assetMrg, tex, tileSetOpts, tileset_dir) tilesets.append(tset) diff --git a/overviewer_core/data/js_src/overviewer.js b/overviewer_core/data/js_src/overviewer.js index a8ad80b..00dd283 100644 --- a/overviewer_core/data/js_src/overviewer.js +++ b/overviewer_core/data/js_src/overviewer.js @@ -36,7 +36,12 @@ overviewer.collections = { /** * Hold the raw marker data for each tilest */ - 'markerInfo': {} + 'markerInfo': {}, + + /** + * holds a reference to the spawn marker. + */ + 'spawnMarker': null, }; overviewer.classes = { diff --git a/overviewer_core/data/js_src/util.js b/overviewer_core/data/js_src/util.js index 0a1655f..0e738da 100644 --- a/overviewer_core/data/js_src/util.js +++ b/overviewer_core/data/js_src/util.js @@ -63,6 +63,8 @@ overviewer.util = { signs.registerEvents(signs); } + var spawnmarker = new overviewer.views.SpawnIconView(); + // Update coords on mousemove google.maps.event.addListener(overviewer.map, 'mousemove', function (event) { coordsdiv.updateCoords(event.latLng); @@ -77,6 +79,7 @@ overviewer.util = { overviewer.mapView.updateCurrentTileset(); compass.render(); + spawnmarker.render(); // re-center on the last viewport var currentWorldView = overviewer.mapModel.get("currentWorldView"); diff --git a/overviewer_core/data/js_src/views.js b/overviewer_core/data/js_src/views.js index b4ffbb9..323c1fc 100644 --- a/overviewer_core/data/js_src/views.js +++ b/overviewer_core/data/js_src/views.js @@ -359,3 +359,29 @@ overviewer.views.SignControlView = Backbone.View.extend({ }, }); +/** + * SpawnIconView + */ +overviewer.views.SpawnIconView = Backbone.View.extend({ + render: function() { + // + var curTileSet = overviewer.mapView.options.currentTileSet; + if (overviewer.collections.spawnMarker) { + overviewer.collections.spawnMarker.setMap(null); + overviewer.collections.spawnMarker = null; + } + var spawn = curTileSet.get("spawn"); + if (spawn) { + overviewer.collections.spawnMarker = new google.maps.Marker({ + 'position': overviewer.util.fromWorldToLatLng(spawn[0], + spawn[1], spawn[2], overviewer.mapView.options.currentTileSet), + 'map': overviewer.map, + 'title': 'spawn', + 'icon': overviewerConfig.CONST.image.spawnMarker, + 'visible': false + }); + overviewer.collections.spawnMarker.setVisible(true); + } + } +}); + diff --git a/overviewer_core/tileset.py b/overviewer_core/tileset.py index 8102328..8f78c97 100644 --- a/overviewer_core/tileset.py +++ b/overviewer_core/tileset.py @@ -518,6 +518,8 @@ class TileSet(object): last_rendertime = self.max_chunk_mtime, imgextension = self.imgextension, ) + if (self.regionset.get_type() == "overworld"): + d.update({"spawn": self.options.get("spawn")}) try: d['north_direction'] = self.regionset.north_dir except AttributeError: diff --git a/overviewer_core/world.py b/overviewer_core/world.py index 12a5d0a..7323a4a 100644 --- a/overviewer_core/world.py +++ b/overviewer_core/world.py @@ -19,6 +19,8 @@ import os.path from glob import glob import logging import hashlib +import time +import random import numpy @@ -91,8 +93,21 @@ class World(object): if not os.path.exists(os.path.join(self.worlddir, "level.dat")): raise ValueError("level.dat not found in %s" % self.worlddir) - # Hard-code this to only work with format version 19133, "Anvil" data = nbt.load(os.path.join(self.worlddir, "level.dat"))[1]['Data'] + # it seems that reading a level.dat file is unstable, particularly with respect + # to the spawnX,Y,Z variables. So we'll try a few times to get a good reading + # empirically, it seems that 0,50,0 is a "bad" reading + retrycount = 0 + while (data['SpawnX'] == 0 and data['SpawnY'] == 50 and data['SpawnZ'] ==0 ): + logging.debug("bad level read, retrying") + time.sleep(random.random()) + data = nbt.load(os.path.join(self.worlddir, "level.dat"))[1]['Data'] + retrycount += 1 + if retrycount > 10: + raise Exception("Failed to correctly read level.dat") + + + # Hard-code this to only work with format version 19133, "Anvil" if not ('version' in data and data['version'] == 19133): logging.critical("Sorry, This version of Minecraft-Overviewer only works with the 'Anvil' chunk format") raise ValueError("World at %s is not compatible with Overviewer" % self.worlddir) @@ -163,7 +178,7 @@ class World(object): # location ## read spawn info from level.dat - data = self.data + data = self.leveldat disp_spawnX = spawnX = data['SpawnX'] spawnY = data['SpawnY'] disp_spawnZ = spawnZ = data['SpawnZ'] @@ -175,24 +190,32 @@ class World(object): ## clamp spawnY to a sane value, in-chunk value if spawnY < 0: spawnY = 0 - if spawnY > 127: - spawnY = 127 + if spawnY > 255: + spawnY = 255 # Open up the chunk that the spawn is in - regionset = self.get_regionset(0) + regionset = self.get_regionset("overworld") try: chunk = regionset.get_chunk(chunkX, chunkZ) except ChunkDoesntExist: return (spawnX, spawnY, spawnZ) + + def getBlock(y): + "This is stupid and slow but I don't care" + targetSection = spawnY//16 + for section in chunk['Sections']: + if section['Y'] == targetSection: + blockArray = section['Blocks'] + return blockArray[inChunkX, inChunkZ, y % 16] + - blockArray = chunk['Blocks'] ## The block for spawn *within* the chunk inChunkX = spawnX - (chunkX*16) inChunkZ = spawnZ - (chunkZ*16) ## find the first air block - while (blockArray[inChunkX, inChunkZ, spawnY] != 0) and spawnY < 127: + while (getBlock(spawnY) != 0) and spawnY < 256: spawnY += 1 return spawnX, spawnY, spawnZ From 8228ebbfffa354fd471e6ed6df74b223050ff4cf Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 15 Mar 2012 09:30:01 -0400 Subject: [PATCH 12/15] updated docs for the crop option to reference --check-tiles --- docs/config.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/config.rst b/docs/config.rst index 1affc01..a1370d7 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -387,15 +387,15 @@ values. The valid configuration keys are listed below. once it has been rendered once. For an expansion to the bounds, because chunks in the new bounds have - the same mtime as the old, tiles will not automatically be updated. You - may need to use :option:`--forcerender` to force those tiles to update. - (You can define the ``forcerender`` option on just one render) + the same mtime as the old, tiles will not automatically be updated, + leaving strange artifacts along the old border. You may need to use + :option:`--forcerender` to force those tiles to update. (You can use + the ``forcerender`` option on just one render by adding ``'forcerender': + True`` to that render's configuration) - For reductions to the bounds, because there is currently no mechanism to - detect tiles that shouldn't exist but do, old tiles may remain and will - not get deleted. The only fix for this currently is to delete that - render directory entirely and render it again with - :option:`--forcerender`. + For reductions to the bounds, you will need to render your map at least + once with the :option:`--check-tiles` mode activated so that tiles that + should no longer exist will be detected and deleted. ``forcerender`` This is a boolean. If set to ``True`` (or any non-false value) then this From 03d3e852fcf308e2c9aff94f36836cc8d372695e Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Thu, 15 Mar 2012 21:52:22 -0400 Subject: [PATCH 13/15] Backed out my level.dat reread code --- overviewer_core/world.py | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/overviewer_core/world.py b/overviewer_core/world.py index 7323a4a..0a5864e 100644 --- a/overviewer_core/world.py +++ b/overviewer_core/world.py @@ -97,14 +97,9 @@ class World(object): # it seems that reading a level.dat file is unstable, particularly with respect # to the spawnX,Y,Z variables. So we'll try a few times to get a good reading # empirically, it seems that 0,50,0 is a "bad" reading - retrycount = 0 - while (data['SpawnX'] == 0 and data['SpawnY'] == 50 and data['SpawnZ'] ==0 ): - logging.debug("bad level read, retrying") - time.sleep(random.random()) - data = nbt.load(os.path.join(self.worlddir, "level.dat"))[1]['Data'] - retrycount += 1 - if retrycount > 10: - raise Exception("Failed to correctly read level.dat") + # update: 0,50,0 is the default spawn, and may be valid is some cases + # more info is needed + data = nbt.load(os.path.join(self.worlddir, "level.dat"))[1]['Data'] # Hard-code this to only work with format version 19133, "Anvil" From bafc1fd93ace69aed304049a7305718019a0d0bf Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 15 Mar 2012 22:00:49 -0400 Subject: [PATCH 14/15] corrected the caveat note for the crop option --- docs/config.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/config.rst b/docs/config.rst index a1370d7..52c0cca 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -394,8 +394,13 @@ values. The valid configuration keys are listed below. True`` to that render's configuration) For reductions to the bounds, you will need to render your map at least - once with the :option:`--check-tiles` mode activated so that tiles that - should no longer exist will be detected and deleted. + once with the :option:`--check-tiles` mode activated, and then once with + the :option:`--forcerender` option. The first run will go and delete tiles that + should no longer exist, while the second will render the tiles around + the edge properly. + + Sorry there's no better way to handle these cases at the moment. It's a + tricky problem and nobody has devoted the effort to solve it yet. ``forcerender`` This is a boolean. If set to ``True`` (or any non-false value) then this From 74f029b9b41c95f01f4358160b532537edb7ba48 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 15 Mar 2012 23:35:03 -0400 Subject: [PATCH 15/15] added an faq about deleting parts of your map --- docs/config.rst | 4 +++- docs/faq.rst | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 1 deletion(-) diff --git a/docs/config.rst b/docs/config.rst index 52c0cca..be88405 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -358,6 +358,8 @@ values. The valid configuration keys are listed below. This is a where a specific texture pack can be found to be used during this render. It can be either a folder or a directory. Its value should be a string. +.. _crop: + ``crop`` You can use this to render a small subset of your map, instead of the entire thing. The format is (min x, min z, max x, max z). @@ -397,7 +399,7 @@ values. The valid configuration keys are listed below. once with the :option:`--check-tiles` mode activated, and then once with the :option:`--forcerender` option. The first run will go and delete tiles that should no longer exist, while the second will render the tiles around - the edge properly. + the edge properly. Also see :ref:`this faq entry`. Sorry there's no better way to handle these cases at the moment. It's a tricky problem and nobody has devoted the effort to solve it yet. diff --git a/docs/faq.rst b/docs/faq.rst index b7bd3eb..e6f1a0c 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -90,3 +90,61 @@ If you are seeing exorbitant memory usage, then it is likely either a bug or a subtly corrupted world. Please file an issue or come talk to us on IRC so we can take a look! See :ref:`help`. +.. _cropping_faq: + +I've deleted some sections of my world but they still appear in the map +----------------------------------------------------------------------- +Okay, so making edits to your world in e.g. worldedit has some caveats, +especially regarding deleting sections of your world. + +This faq also applies to using the :ref:`crop` option. + +Under normal operation with vanilla Minecraft and no external tools fiddling +with the world, Overviewer performs correctly, rendering areas that have +changed, and everything is good. + +Often with servers one user will travel reeeeally far out and cause a lot of +extra work for the server and for The Overviewer, so you may be tempted to +delete parts of your map. This can cause problems, so read on to learn what you +can do about it. + +First some explanation: Until recently (Mid May 2012) The Overviewer did not +have any facility for detecting parts of the map that should no longer exist. +Remember that the map is split into small tiles. When Overviewer starts up, the +first thing it does is calculate which tiles should exist and which should be +updated. This means it does not check or even look at tiles that should not +exist. This means that parts of your world which have been deleted will hang +around on your map because Overviewer won't even look at those tiles and notice +they shouldn't be there. You may even see strange artifacts around the border as +tiles that should exist get updated. + +Now, with the :option:`--check-tiles` option, The Overviewer *will* look for and +remove tiles that should no longer exist. So you can render your map once with +that option and all those extra tiles will get removed automatically. However, +this is only half of the solution. The other half is making sure the tiles along +the border are re-rendered, or else it will look like your map is being cut off. + +Explanation: The tiles next to the ones that were removed are tiles that should +continue to exist, but parts of them have chunks that no longer exist. Those +tiles then should be re-rendered to show that. However, since tile updates are +triggered by the chunk last-modified timestamp changing, and the chunks that +still exist have *not* been updated, those tiles will not get re-rendered. + +The consequence of this is that your map will end up looking cut-off around the +new borders that were created by the parts you deleted. You can fix this one of +two ways. + +1. You can run a render with :option:`--forcerender`. This has the unfortunate + side-effect of re-rendering *everything* and doing much more work than is + necessary. + +2. Manually navigate the tile directory hierarchy and manually delete tiles + along the edge. Then run once again with :option:`--check-tiles` to re-render + the tiles you just deleted. This may not be as bad as it seems. Remember each + zoom level divides the world into 4 quadrants: 0, 1, 2, and 3 are the upper + left, upper right, lower left, and lower right. It shouldn't be too hard to + navigate it manually to find the parts of the map that need re-generating. + +3. The third non-option is to not worry about it. The problem will fix itself if + people explore near there, because that will force that part of the map to + update.