Archive for the ‘maposmatic’ Category

« Older Entries
Newer Entries »

MapOSMatic is back!

Tuesday, October 12th, 2010

The past couple of weeks have unfortunately been very difficult for MapOSMatic. The rapid growth of the OpenStreetMap database with the recent massive data imports drastically reduced the time we thought we had to migrate away from our server that we knew would not be able to host MapOSMatic and the full planet database for long. And so it happened, our disk was full.

We started by removing the previous renderings, more and more aggressively, to make room for the database updates and a few renderings, to maintain the service running. But this could not last forever, and we ended up having to stop the rendering queue.

Thanks to the FSF France, we were given access to a powerful server that could temporarily host the OSM planet database until we can find a more perennial solution. In the past two weeks, I spent as much time as I could (which was not much since I was relocating to the United States at the same time) setting up a PostGIS database on this new server. This database is now operational, and currently catching-up its replication lag from the latest OpenStreetMap data.

MapOSMatic is now hosted on two distinct servers. The first one is provided by Enix, a French hosting company. It hosts the rendering queue, the web service, this blog, the wiki, the statistics and of course the rendering results. With 1TB of disk storage available, we have enough room to keep a lot more renderings, so your maps will be available for a greater length of time (of course you still need to regenerate them to get up-to-date data). The second server, from the FSF France, hosts solely the PostGIS database, with one task only: keeping it as up-to-date as possible. For now, we are still on a daily updates schedule, but I’m working on moving to more frequent updates, eventually up to minutely updates.

With our apologies for the service downtime, we hope you’re as happy as we are to see MapOSMatic back on its feet again! All the rendering requests submitted while the queue was halted are now being processed, so don’t hesitate to go search for the maps you recently requested, they are probably available now! And stay tuned for some more updates, the new version of the MapOSMatic rendering engine is just around the corner…

Posted in maposmatic | Comments Off

Recent updates in MapOSMatic

Saturday, March 20th, 2010
Saint Laurent en Grandvaux, Jura, Franche Comté, France

Saint Laurent en Grandvaux, Jura, Franche Comté, France

MapOSMatic recently passed an important milestone in its history. A few weeks ago, we served our 10,000th rendered map (Saint-Laurent-en-Grandvaux, Jura, France) ! We are very pleased with this accomplishment and feel it is important for us to continue improving the MapOSMatic service for our users. We are working on several objectives in parallel:

  • improving the user experience on the website, adding new features and more translations;
  • reducing the rendering latency to produce maps faster for our users;
  • improving the maps and indexes themselves.

Today we rolled out on MapOSMatic a few changes that have been brewing and undergoing extended testing on our development setup, and as I write this post I am pleased to see that we are making good progress on these three main directions.

First and foremost, I think the most important update today comes from the addition of a Spanish translation of the MapOSMatic website (and the corresponding capability to generate maps and indexes internationalized for Spanish-speaking countries), contributed by Julio Costa Zambelli, Sebastian Borgwardt and Jean-Guilhem Cailton. We hope the availability of MapOSMatic in Spanish will help the Chile earthquake disaster response teams.

Next, we restored the bounding-box area input fields for specifying an exact zone to render by the coordinates of its top-left and bottom-right corners. We also added a new feature to the slippy map: by maintaining the Control key pressed, you can draw a rectangle directly on the map to precisely define the limits of the area you wish to render!

Small part of Seoul, South Korea

Small part of Seoul, South Korea

This update allows integrates better support for some languages :

  • The index of the streets and amenities, as well as the map and index titles are now rendered through the Pango library, which allows to correctly render Arabic, Korean or Japanese characters
  • The index is now rendered right-to-left for RTL languages such as arabic. The first column is on the right, the last column on the left, the street names are on the right while the square locations are on the left.
  • We also fixed a font problem with our Mapnik installation allowing for rendering of non-latin characters in the maps as well.
Small part of the index for Seoul, South Korea

Small part of the index for Seoul, South Korea

And finally, as advertised in a previous post about MapOSMatic improvements, we are now using a new version of the rendering daemon (the software in charge of processing the rendering requests queue and creating the maps and indexes). This new daemon lays the groundwork for future concurrent rendering and rendering jobs distribution which will help reducing the rendering latency, when completed. It also has better management of obsolete renderings, with the direct impact of no longer listing hundreds, if not thousands of maps we no longer have the rendered files for.

I would like to conclude this post by thanking, again, all the amazing contributors to MapOSMatic for their continuous feedback, ideas, bug reports and their celerity in providing translations updates when we add new features to MapOSMatic!

Posted in maposmatic, ocitysmap, website | No Comments »

Two months of MapOSMatic improvements

Monday, March 8th, 2010

Since the december 2009 hackfest (reported here, here and here) and the official announcement of the improvements early january 2010, MapOSMatic has seen a sustained flow of interesting improvements. This post compiles the most important improvements that we have done recently, and some details on future improvements.

OcitySMap

Support for other countries has been added to OcitySMap, the Python module that actually generates the maps and the indexes. This support allows OcitySMap to generate correct indexes for other countries, by transforming street names such as “Calle de la Vida” into “Vida (Calle de la)”. This was made possible through the major re-organization of OcitySMap for internalization that took place during the december 2009 hackfest. So far, we have support for french, italian, spanish, catalan, brasilian portuguese, arabic, russian, dutch, crotian and polish, thanks to many contributors !

From a coding point of view, OcitySMap now uses the psycopg2 Python module to query the PostgreSQL database instead of pygresql. We made this change because Django, the Web framework that we use for MapOSMatic, already uses psycopg2. Using the same Python module to query the PostgreSQL database sounded like a logical choice.

MapOSMatic

MapOSMatic, the web front-end for OcitySMap, has also seen a large number of improvements. First, many translations have been added: german, italian, catalan, russian, arabic, brasilian portuguese, crotian, polish and dutch. For the Arabic translation, we have added Right-to-Left support in our web site design. Again, all the translations come from new contributors to the project.

The website has also been completely re-designed :

  • A brand new CSS, with a much cleaner design
  • The map creation page is now separate from the homepage, for more clarity
  • The selector for the city name in the map creation page now provides an auto-suggestion mechanism
  • The job and maps page have been completely re-designed, with a better design than the old table-based approach. The maps are now classified by first letter, and a search engine allows to quickly find existing maps.
  • A RSS feed has been added to keep track of map renderings taking place on MapOSMatic.org.

See Maxime Petazzoni’s blog post for details on these design improvements.

Coming improvements

We have already implemented further improvements, that are currently being tested on our development server :

  • A new rendering daemon. When a map request is entered on MapOSMatic, the map rendering doesn’t take place synchronously, since it is a quite consuming work. Map rendering requests are processed asynchronously, one after the other, through a daemon called maposmaticd. This daemon has been completely rewritten to use more Python-ic approaches, to better support timeouts (renderings taking too much time) and in the future to support parallel renderings.
  • Use of Pango for text rendering. Until now, OcitySMap was directly using Cairo for rendering text in the street index. While this was working fine for latin characters, it didn’t work for arabic or korean characters (and probably other non-latin characters). Therefore, we have rewritten the code that generates the index to use Pango on top of Cairo, so that all characters are properly rendered. This change needs a little more work but should be ready for production soon.

We have also identified one of the major reasons for which the rendering requests are sometimes so long to process. One of our geographic SQL query is taking a very long time, due to the absence of optimization. We are working on this issue with PostGIS/PostgreSQL experts, and hope to get down from 4 minutes 38 seconds for getting the list of streets for Toulouse to something around 3.2 seconds. As you can expect, we are really looking forward to making these new changes available to the MapOSMatic community!

Posted in database, maposmatic, ocitysmap, website | No Comments »

Major improvements to MapOSMatic

Monday, January 4th, 2010

We have just sent the following announcement to OpenStreetMap lists, officially announcing the improvements made to MapOSMatic during the latest hackfest.

As a new year’s present, the MapOSMatic team is proud to announce that a new version of the maposmatic.org website has been put online, with major improvements over the initial version announced in September 2009.

For the record, MapOSMatic is a website that allows to generate city maps from OpenStreetMap data. Each map is divided into squares to easily find streets and is delivered with the corresponding street index.

The new MapOSMatic provides the following improvements :

  • Support for the whole world. Any location in the world can now be rendered on maposmatic.org.
  • OpenStreetMap database updated daily. Until now, the database had never been updated since the service was started in September 2009. Now, the geographic database used to render the maps is updated daily, providing maps with the latest contributions to OpenStreetMap. Each map contains the date at which it was generated.
  • Better city search engine. Thanks to Nominatim, we now provide a search engine that allows to find cities in a much more usable way: cities with the same name can be distinguished and the search works even when the city name is not completely correct.
  • Support for other languages. A few parts of the map rendering process is language-dependent and we now have the infrastructure to use language-dependent code. For the moment, we support English, French and Italian, but we are waiting for your contributions to support other languages. The website has also been translated to German and Italian.
  • Amenities in the index. In addition to the streets, we have added important amenities to the index: schools, town hall, post offices, places of worship, etc.

All these improvements are available now on http://www.maposmatic.org

You can follow the progress and improvements of MapOSMatic on our blog at http://news.maposmatic.org. MapOSMatic is of course free software, you can fetch its source code and contribute to the project, see http://savannah.nongnu.org/projects/maposmatic/.

Do not hesitate to send us your feedback, comments, suggestions and contributions to [email protected]

Cheers,

The MapOSMatic team.

Posted in hackfests, maposmatic | No Comments »

MapOSMatic hackfest, day 3

Tuesday, December 22nd, 2009

Yesterday was day 3 of the hackfest, and we of course spent our day working on MapOSMatic.

  • Support for amenities in index has been added. Now, in addition to the streets, you have the location of the town hall, police station and other public buildings, the schools, universities and other education buildings, and the places of worship. More improvements are needed in this area (translation of headings, support for amenities represented by polygons, fix incorrect grouping), but the general infrastructure is in place.
  • The main form to select the city has been further improved, with bug fixes, improved usability, etc. Said like this, it doesn’t sound like a big deal, but all this stuff is written in Javascript using the JQuery library and requires careful fine-tuning to be working. We easily spend a day on the same 20 lines of Javascript.
  • The language selection has been improved, with all french-speaking countries and all english-speaking countries added (we’re waiting for your contributions for other languages). The list of languages available is now automatically reduced to the languages spoken in the country in which the selected city is. For multi-lingual countries like Belgium, we have no way to know if the selected city is in the french-speaking area, the dutch-speaking area or the german-speaking area. Therefore, it is up to the user to choose between these languages. This selection changes how the streets are sorted and the list of prefixes that we consider before sorting.
  • The slippy map based on OpenLayers is now loaded only if bounding box mode is used. This allows a much quicker loading of the MapOSMatic homepage.
  • We have improved our server infrastructure to regularly update the coast lines, which are coming from a difference source than the planet OSM xml file. It allowed to fix rendering problems in island where we had ugly coast lines. These informations are now updated once a week.

We’re now starting day 4 of the hackfest. We will soon send out a notification to beta-testers so that they can test all these new features and report their comments and bugs.

Posted in hackfests, maposmatic | No Comments »

MapOSMatic hackfest, day 2

Monday, December 21st, 2009

Yesterday was day 2 of the hackfest, and we did a number of improvements :

  • Cities can now be searched in an efficient way, thanks to the usage of Nominatim. The “Administrative city” field has been turned into a small search engine that returns, through Javascript, the list of cities matching the request. We spent quite some time understanding the results of Nomimatim and how they relate to the objects in the PostGIS database generated by osm2pgsql. We still need to fix a few issues with this Javascript-based search engine.
  • We used to identify cities by their name, but this of course doesn’t work since many cities have the same name around the world. So, now, rendering of city by administrative boundary are identified by the OSM id of the polygon in the PostGIS database. OCitySMap has been modified consequently, and jobs submitted through MapOSMatic are properly handled by maposmaticd.
  • The language used to generate a map can now be selected before generating the map, and several other improvements have been made on the internationalization. The main topic remaining is to pre-select the languages that are useful depending on the country in which the city being rendered is in.
  • The slippy map integration has been reworked to post-pone the loading of the tiles so that the homepage of MapOSMatic is much faster to load.
  • The documentation to install MapOSMatic and OCitySMap has been further improved.
  • Several bugs or tasks waiting in our bug tracker have been fixed, and some cleanup has been done on bugs that were already fixed by previous development

We expect to put the new version into production tomorrow, with the main achievement of having world-wide support, a better city search feature, and internalization for maps. Of course, we’ll need a lot of contributions for the internationalization support.

Posted in hackfests, maposmatic | No Comments »

MapOSMatic hackfest, day 1

Sunday, December 20th, 2009

The hackfest started just yesterday around noon with the arrival of six crazy hackers who will spend four days working on MapOSMatic. For the first day, we spent some time setting up MapOSMatic on the laptop of all developers and then started working on different topics :

  • A group started experimenting with Nominatim in order to improve the city lookup feature of the MapOSMatic website. This is very important for several reasons. First because for the moment our city lookup mechanism only finds the city if the entered name matches exactly the one in the database (spaces, dashes and accents included). And also because we have no way of letting the user choose between several cities of the same name, which is very important as we intend to put the worldwide database into production soon. The initial work we did is to add a small gateway to Nominatim in MapOSMatic (see this commit). The rest of the work is being worked on now.
  • Another developer worked on the infrastructure to allow the adaptation of OCitySMap to other languages: how the street names should be sorted, what are the common prefixes for streets, places, roads, etc. See this commit, this commit, this commit, this commit, this commit and this commit. Support is only for fr_FR and en_GB at the moment, and only at the OCitySMap level. Nothing has (yet) been implemented to select this from the MapOSMatic website, but it’s obviously on the TODO-list for this hackfest.
  • The documentation for installing OCitySMap and MapOSMatic has been improved and updated for newer distributions.
  • The german translation contributed by malenki has been added to the website and now allows to access the MapOSMatic website in German. Contributions for other languages welcome, contact us for details.

The previous hackfest took place in August in Toulouse (south of France) where we enjoyed a very nice and warm whether, wearing shorts, eating tomatoes and cucumbers straight from the garden. It was just a perfect summer hackfest. Now, we’re in Méridon (near Paris), where snow has fallen since Friday and was falling again today, so we are enjoying a perfect winter hackfest, with a nice tartiflette yesterday evening ! :-)

Posted in hackfests, maposmatic | No Comments »

MapOSMatic hackfest next week

Tuesday, December 15th, 2009

The MapOSMatic application website was initially created in August 2009 by a group of free software enthousiasts during a one-week intensive hackfest. The same group of people is going to meet again for a 5 days hackfest next week and we plan to take advantage of this hackfest to work on the major issues of MapOSMatic :

  • Put the world, daily-updated, OpenStreetMap database into production, so that cities from all over the world can be rendered and up-to-date data is used
  • Improve the city selection, to make it fuzzy, and to support various countries
  • Improve the city index generation to support different languages
  • Improve the web site display of existing renderings

Of course, a lot more issues are registered in our task list and in our bug tracker, but we will very likely focus on the points mentioned above.

Posted in hackfests, maposmatic | No Comments »

MapOSMatic, image of the week on OpenStreetMap main wiki page !

Friday, November 27th, 2009

MapOSMatic is currently being featured on the main OpenStreetMap wiki page. More precisely, a MapOSMatic rendering of the french city Saint-Pierre-du-Perray has been slightly reworked, nicely mixing the map and the street index on the same picture, is featured as the « Image of the week ». Thanks to the nice contributors who did this!

Saint Pierre du Perray

The description below the picture says that we only cover France, which is true at the moment. But we have a database ready with updated data for the whole world, and expect it to make it publicly available soon. At least, guaranteed before the new year !

Posted in maposmatic | No Comments »

MapOSMatic.org in french magazine SVM

Monday, November 9th, 2009

Shortly after the public announcement of MapOSMatic.org, the french magazine SVM published an article on its website, entitled « MapOSMatic.org, on-demand creation of city maps ». Of course, many other websites also relayed the announcement of MapOSMatic.org, but SVM also announced it in its printed version, through a small interview of Thomas Petazzoni in its october issue :

maposmatic-svmFunnily, the main title of the magazine was about the supposedly wonderful new features of Windows Seven. Something most OpenStreetMap contributors probably don’t care about :-)

Posted in maposmatic | No Comments »

Performance testings, world database in production soon

Saturday, November 7th, 2009

As the tests on the more powerful server didn’t show a dramatic increase of performances, we decided to stay with our existing server. After the addition of an index by David recently, I’ve done a few tests on rendering duration in different conditions :

  • On the France-only database (our “production” database)
  • On the France-only database, with the daily diff being applied on another database
  • On the world database (our “development database”)
  • On the world database, with the daily diff being applied to this database

For each case, I’ve done the rendering of three different french cities (Rennes, Colomiers and Sanguinet) in PDF, PNG and SVGZ formats, and I’ve done each test twice.
Rennes

  • World-database, without daily udate process: 2 minutes 5 seconds, 1 minute 51 seconds
  • World database, with daily update process: 3 minutes 3 seconds, 2 minutes 10 seconds
  • France-only database, without daily update process: 1 minute 48 seconds, 1 minute 47 seconds
  • France-only database, with daily update process on the world database: 3 minutes 54 seconds, 2 minutes 25 seconds

Colomiers

  • World-database, without daily udate process: 37 seconds, 32 seconds
  • World database, with daily update process: 1 minutes 5 seconds, 39 seconds
  • France database, without daily update process: 31 seconds, 31 seconds
  • France database, with daily update process on the world database: 51 seconds, 37 seconds

Sanguinet

  • World-database, without daily udate process: 33 seconds, 39 seconds
  • World database, with daily update process: 49 seconds, 56 seconds
  • France database, without daily update process: 32 seconds, 36 seconds
  • France database, with daily update process on the world database: 42 seconds, 40 seconds

Conclusions

First, we can conclude that when the daily update process is not running, the rendering time for a given city is only slightly longer on the world database than on the France-only database. So, the index added by David is very efficient and solved our main issue.

Secondly, when the daily update process is running, the rendering time increases up to 50% in the worst cases. It’s a lot, but the daily update process is very I/O intensive so we expected a large impact on the rendering time. However, the rendering times remain acceptable, since the rendering is done asynchronously on MapOSMatic.org.

Therefore, I think we will soon put into production the world database, which will be updated every day with the OpenStreetMap data.

Posted in database, maposmatic, ocitysmap | No Comments »

Improve city name lookups

Tuesday, October 13th, 2009

In the course of finding some way to “optimize” support for the planet on our server, here is an index which speeds things up greatly for city name lookups:

create index admin_city_names on planet_osm_line (boundary,admin_level,name) where (boundary='administrative' and admin_level='8');

(more…)

Posted in database, maposmatic | No Comments »

Latest benchmarks for planet incremental diff imports on a higher end server

Monday, October 12th, 2009

On the higher-end server running osm2pgsql with a 8G cache, the speedup ratio is at least 2.1 and seems to increase with the size of the diff. Considering the very few samples we have, it’s hard to tell the effect of the cache size increase. But the fact is that this higher-end server behaves better, a speedup factor between 2 and 2.75 can reasonably be expected. This would translate to a daily server downtime of around 4h for planet updates. Details follow.
(more…)

Posted in database, maposmatic | No Comments »

First performance results on the higher-end server

Sunday, October 11th, 2009

Thomas already posted the results of the initial planet import on the higher end server (see the maposmatic-dev mailing-list): 52 hours, which corresponds to a speedup factor around 2 compared to our current server. Our first incremental diff import (with a 1.5G cache) shows a speedup factor of the same order, which means that an incremental diff import would require around 4h10 server downtime every day. Details follow.

We’re currently running another incremental diff import on the higher end server, this time with a 8G cache… Let’s hope it’s getting even better.

(more…)

Posted in database, maposmatic | No Comments »

Latest news now visible on the main website

Wednesday, October 7th, 2009

The main MapOSMatic.org website now include a box in the right column containing the five latest news coming from the blog. The intent is to make the blog contents more visible to MapOSMatic visitors and to include some more dynamic and fresh content on the main website.

Posted in maposmatic, website | No Comments »

« Older Entries
Newer Entries »