December 21st, 2009
Yesterday was day 2 of the hackfest, and we did a number of improvements :
- Cities can now be searched in an efficient way, thanks to the usage of Nominatim. The “Administrative city” field has been turned into a small search engine that returns, through Javascript, the list of cities matching the request. We spent quite some time understanding the results of Nomimatim and how they relate to the objects in the PostGIS database generated by osm2pgsql. We still need to fix a few issues with this Javascript-based search engine.
- We used to identify cities by their name, but this of course doesn’t work since many cities have the same name around the world. So, now, rendering of city by administrative boundary are identified by the OSM id of the polygon in the PostGIS database. OCitySMap has been modified consequently, and jobs submitted through MapOSMatic are properly handled by maposmaticd.
- The language used to generate a map can now be selected before generating the map, and several other improvements have been made on the internationalization. The main topic remaining is to pre-select the languages that are useful depending on the country in which the city being rendered is in.
- The slippy map integration has been reworked to post-pone the loading of the tiles so that the homepage of MapOSMatic is much faster to load.
- The documentation to install MapOSMatic and OCitySMap has been further improved.
- Several bugs or tasks waiting in our bug tracker have been fixed, and some cleanup has been done on bugs that were already fixed by previous development
We expect to put the new version into production tomorrow, with the main achievement of having world-wide support, a better city search feature, and internalization for maps. Of course, we’ll need a lot of contributions for the internationalization support.
Posted in hackfests, maposmatic | No Comments »
December 20th, 2009
The hackfest started just yesterday around noon with the arrival of six crazy hackers who will spend four days working on MapOSMatic. For the first day, we spent some time setting up MapOSMatic on the laptop of all developers and then started working on different topics :
- A group started experimenting with Nominatim in order to improve the city lookup feature of the MapOSMatic website. This is very important for several reasons. First because for the moment our city lookup mechanism only finds the city if the entered name matches exactly the one in the database (spaces, dashes and accents included). And also because we have no way of letting the user choose between several cities of the same name, which is very important as we intend to put the worldwide database into production soon. The initial work we did is to add a small gateway to Nominatim in MapOSMatic (see this commit). The rest of the work is being worked on now.
- Another developer worked on the infrastructure to allow the adaptation of OCitySMap to other languages: how the street names should be sorted, what are the common prefixes for streets, places, roads, etc. See this commit, this commit, this commit, this commit, this commit and this commit. Support is only for fr_FR and en_GB at the moment, and only at the OCitySMap level. Nothing has (yet) been implemented to select this from the MapOSMatic website, but it’s obviously on the TODO-list for this hackfest.
- The documentation for installing OCitySMap and MapOSMatic has been improved and updated for newer distributions.
- The german translation contributed by malenki has been added to the website and now allows to access the MapOSMatic website in German. Contributions for other languages welcome, contact us for details.
The previous hackfest took place in August in Toulouse (south of France) where we enjoyed a very nice and warm whether, wearing shorts, eating tomatoes and cucumbers straight from the garden. It was just a perfect summer hackfest. Now, we’re in Méridon (near Paris), where snow has fallen since Friday and was falling again today, so we are enjoying a perfect winter hackfest, with a nice tartiflette yesterday evening ! 
Posted in hackfests, maposmatic | No Comments »
December 15th, 2009
The MapOSMatic application website was initially created in August 2009 by a group of free software enthousiasts during a one-week intensive hackfest. The same group of people is going to meet again for a 5 days hackfest next week and we plan to take advantage of this hackfest to work on the major issues of MapOSMatic :
- Put the world, daily-updated, OpenStreetMap database into production, so that cities from all over the world can be rendered and up-to-date data is used
- Improve the city selection, to make it fuzzy, and to support various countries
- Improve the city index generation to support different languages
- Improve the web site display of existing renderings
Of course, a lot more issues are registered in our task list and in our bug tracker, but we will very likely focus on the points mentioned above.
Posted in hackfests, maposmatic | No Comments »
November 27th, 2009
MapOSMatic is currently being featured on the main OpenStreetMap wiki page. More precisely, a MapOSMatic rendering of the french city Saint-Pierre-du-Perray has been slightly reworked, nicely mixing the map and the street index on the same picture, is featured as the « Image of the week ». Thanks to the nice contributors who did this!
The description below the picture says that we only cover France, which is true at the moment. But we have a database ready with updated data for the whole world, and expect it to make it publicly available soon. At least, guaranteed before the new year !
Posted in maposmatic | No Comments »
November 9th, 2009
Shortly after the public announcement of MapOSMatic.org, the french magazine SVM published an article on its website, entitled « MapOSMatic.org, on-demand creation of city maps ». Of course, many other websites also relayed the announcement of MapOSMatic.org, but SVM also announced it in its printed version, through a small interview of Thomas Petazzoni in its october issue :
Funnily, the main title of the magazine was about the supposedly wonderful new features of Windows Seven. Something most OpenStreetMap contributors probably don’t care about
Posted in maposmatic | No Comments »
November 7th, 2009
As the tests on the more powerful server didn’t show a dramatic increase of performances, we decided to stay with our existing server. After the addition of an index by David recently, I’ve done a few tests on rendering duration in different conditions :
- On the France-only database (our “production” database)
- On the France-only database, with the daily diff being applied on another database
- On the world database (our “development database”)
- On the world database, with the daily diff being applied to this database
For each case, I’ve done the rendering of three different french cities (Rennes, Colomiers and Sanguinet) in PDF, PNG and SVGZ formats, and I’ve done each test twice.
Rennes
- World-database, without daily udate process: 2 minutes 5 seconds, 1 minute 51 seconds
- World database, with daily update process: 3 minutes 3 seconds, 2 minutes 10 seconds
- France-only database, without daily update process: 1 minute 48 seconds, 1 minute 47 seconds
- France-only database, with daily update process on the world database: 3 minutes 54 seconds, 2 minutes 25 seconds
Colomiers
- World-database, without daily udate process: 37 seconds, 32 seconds
- World database, with daily update process: 1 minutes 5 seconds, 39 seconds
- France database, without daily update process: 31 seconds, 31 seconds
- France database, with daily update process on the world database: 51 seconds, 37 seconds
Sanguinet
- World-database, without daily udate process: 33 seconds, 39 seconds
- World database, with daily update process: 49 seconds, 56 seconds
- France database, without daily update process: 32 seconds, 36 seconds
- France database, with daily update process on the world database: 42 seconds, 40 seconds
Conclusions
First, we can conclude that when the daily update process is not running, the rendering time for a given city is only slightly longer on the world database than on the France-only database. So, the index added by David is very efficient and solved our main issue.
Secondly, when the daily update process is running, the rendering time increases up to 50% in the worst cases. It’s a lot, but the daily update process is very I/O intensive so we expected a large impact on the rendering time. However, the rendering times remain acceptable, since the rendering is done asynchronously on MapOSMatic.org.
Therefore, I think we will soon put into production the world database, which will be updated every day with the OpenStreetMap data.
Posted in database, maposmatic, ocitysmap | No Comments »
October 13th, 2009
In the course of finding some way to “optimize” support for the planet on our server, here is an index which speeds things up greatly for city name lookups:
create index admin_city_names on planet_osm_line (boundary,admin_level,name) where (boundary='administrative' and admin_level='8');
Read the rest of this entry »
Posted in database, maposmatic | No Comments »
October 12th, 2009
On the higher-end server running osm2pgsql with a 8G cache, the speedup ratio is at least 2.1 and seems to increase with the size of the diff. Considering the very few samples we have, it’s hard to tell the effect of the cache size increase. But the fact is that this higher-end server behaves better, a speedup factor between 2 and 2.75 can reasonably be expected. This would translate to a daily server downtime of around 4h for planet updates. Details follow.
Read the rest of this entry »
Posted in database, maposmatic | No Comments »
October 11th, 2009
Thomas already posted the results of the initial planet import on the higher end server (see the maposmatic-dev mailing-list): 52 hours, which corresponds to a speedup factor around 2 compared to our current server. Our first incremental diff import (with a 1.5G cache) shows a speedup factor of the same order, which means that an incremental diff import would require around 4h10 server downtime every day. Details follow.
We’re currently running another incremental diff import on the higher end server, this time with a 8G cache… Let’s hope it’s getting even better.
Read the rest of this entry »
Posted in database, maposmatic | No Comments »
October 7th, 2009
The main MapOSMatic.org website now include a box in the right column containing the five latest news coming from the blog. The intent is to make the blog contents more visible to MapOSMatic visitors and to include some more dynamic and fresh content on the main website.
Posted in maposmatic, website | No Comments »
October 6th, 2009
For testing purposes, we just got an access to a server that uses the latest Intel CPUs, 16 GB of RAM and a storage using RAID10 over 8 disks. We are currently importing the planet data into a PostgreSQL database and will then be able to test :
- The duration required to apply a daily diff
- The efficiency of MapOSMatic on the full planet
This will allow us to tell if the issue is really on the hardware side or if we must think and work on optimizations at different levels. However, such a high-end server is only given to us for a short period of time, and if the conclusion of the experiment is that such a server is needed, then we’ll have to find a way to fund a similar server.
Posted in maposmatic | No Comments »
October 2nd, 2009
We have a development server running on a database holding the whole planet (as of today). Besides requiring 9h/day to keep the DB updated, the following usage issues arise on that “planet” development instance:
- looking up an administrative name such as “Paris” takes ages (typically 2 to 5 minutes). This is not really great: the rendering jobs are queued, but such queries are not queued and will run in parallel. Besides, we should not ask the user to wait 2 to 5 minutes before the server confirms that it’s queueing her rendering
- when rendering (ie. ocitysmap), finding out the city limits/boundaries of a city takes ages too. This is probably due to the same “feature” as above, but this time the problem is less critical as we are in the “qeueued” side of the force
- in overall, the whole rendering process takes around 8 times longer with a DB holding the planet, than with a DB holding just France. This was measured on “Rennes”: 10mn rendering vs. around 1.25mn.
The “funny” thing is that both the “mapnik” and street index rendering don’t seem to be that slower, but this needs to be confirmed by a more thorough profiling. If this proves correct, we have to find out what makes PostGIS so slow for the first queries. Is this our queries ? The structure of the DB which is lacking a few indexes ? And, in that case, how come mapnik rendering + street index seem almost as fast: is this because the previous queries did put most things in cache ?
But what a great pleasure to be able to map NYC, Palo Alto, !…
-
-
Manhattan, NY
-
-
Palo Alto, CA
-
-
San Francisco, CA
Unfortunately, this will be our privilege, since we are decidedly not planning to open this to the public… for the time being, that is: with our current server (a “modest” dual core x86_64 2GHz / 2GB RAM). Donations or support for a new server are welcome !
Posted in maposmatic, ocitysmap | No Comments »
September 28th, 2009
By far the most important feature request we got, and the one we also consider the most important is to get world-wide support in MapOSMatic, and not only support for France. A large number of people volunteered to help us in translating and adapting the necessary parts of MapOSMatic. However, at the moment, we haven’t reached the point where the issue is on country-specific adaptations and translations.
Two weeks ago, David Decotigny started the import of planet.osm.bz2 into a new PostgreSQL database, so that we could start experimenting world-wide support. The import took around 4-5 days, and completed with a 75 GB PostgreSQL database.
Then, David worked on getting daily diffs applied to the database, so that we can keep it up-to-date with the contributions made on OpenStreetMap. Unfortunately, applying each daily diff takes around 9 hours on our server, which means that 9 hours per day, our server would be busy updating the database, something we don’t consider sustainable. Moreover, we tried to use MapOSMatic concurrently with the daily diff update and it did not work. See David’s email on the MapOSMatic list for further details.
At the moment, we are considering two solutions :
- Having two servers, used alternatively. But that means paying two dedicated servers all year long, which cost quite a lot for a non-profit project like MapOSMatic, and is quite annoying to maintain.
- Understand why the daily diff application process with osm2pgsql is so slow, and see if optimizations are possible. These optimizations would benefit to the entire OpenStreetMap community by allowing more people to play with OpenStreetMap data. Anyone having some PostgreSQL optimizations skills and some free time would definitely help in this area. I think we can provide an access to the server and database for testing purposes to motivated people.
Until this issue has been solved, we unfortunately won’t really be able to start with adding country selection support in MapOSMatic. Some people suggested to restrict ourselves to Europe, but Europe is only a third (in size) of the whole world, and no diffs are provided for Europe (even though they could probably be generated using osmosis). And getting to the whole world is more exciting !
Posted in maposmatic | No Comments »
September 28th, 2009
Since the public announcement of MapOSMatic that took place on September 10th, we did a small number of improvements :
- Due to the very high number of rendering requests we had on the announcement day, we had to implement a redirection mechanism. This redirection mechanism redirects the user to an existing rendering when such a rendering already exists for the given administrative city. Of course, this mechanism does not apply to bounding box renderings
- Add a check to make sure that the given bounding box was more or less in France. We had a small number of renderings done on an area outside of France, and the result was obviously empty, since our database is only loaded with OSM data for France at the moment. The check is not perfect (France is not exactly rectangular) but at least prevents rendering requests in Turkey or in the US
- Improve the log messages we got from the rendering daemon, so that failed renderings can be investigated
- Fix a bug that was causing some of the failed rendering, when the user selected an administrative city and finally decided to fall back to bounding box selection before validating the rendering
- Fix the bug that was causing most of the rendering problems, which was due to a not-robust-enough code to compute the city area. The current code still isn’t perfect (there are really strange unclosed city areas), but at least the renderings do not fail anymore
We’ve also received a large number of very interesting proposals and bug reports, don’t hesitate to join us to improve MapOSMatic !
Posted in maposmatic | No Comments »
September 22nd, 2009
Welcome to news.maposmatic.org, the news pages for MapOSMatic !
Please also visit wiki.maposmatic.org for documentation, info, and stuff !
Posted in maposmatic | No Comments »
