I have returned to the project and have spent the last 7 weeks rewriting the publication and simplification code, which is half of what this is. What I chose to do was fire the entire OSM road map, down to and including secondary roads, plus the entire rail network, and all the ferries for good measure, at my program and get it to work. Even if I don’t manage to handle the entire map, I hoped to get the simpler job of merging and presenting the combined bus and rail routes, as defined by their GTFS, under control.
The results are mixed as discussed below. I have good renders of the global road+rail system, as detailed in The Map itself, up to level 5, and I certainly have a very much improved approach than before, having twigged the plainly obvious that it’s all just a massive geojson simplification problem. The complete road tiles are however far too heavy to be used as vectors, they will have to be rendered. I am trying to get a “have your vector tile and eat it” approach where we can render PNGs for each tile, and generate rarefied vector tiles, by chucking out small stuff, presented transparently, so we can still interact with most of the map. I go to some effort to try and make the longest possible Linestrings of multi-line features to aid this. Much of what I am doing can be achieved simply with styling back the roads. But that too will fail in these dense areas once you get to the top 5 or so tiers. You cant show all the roads and railways in Germany, Switzerland and a 300km radius around about Munich at level 5, you’ll just get a messy blob, we need to thin these routes down.
The result at medium range, levels #9 to 5, gives me at least something to say for the vast areas of the map I have no GTFS for and the ability to make them interactive . It has already produced genuine discoveries for me the author (interpreter ?) , in particular the locations of ferries. The Irrawaddy delta turns out to be the densest ferry network on Earth. It is virtually invisible on the current render even at close range, as is any ferry that winds up two very windy narrow river banks. You can get a postal ferry all up the side of Norway to directly north of Finland. There are ferries in much of Russia, also right the way up the Amazon, but virtually none left in China which is a great shame if true.
I have placed these global road map tiles here http://buz-map.com:8080/ . They are preposterously large at the higher levels, in excess of 150k zipped, so over 10 megabytes unzipped, per tile!, and will probably crash your browser. I haven’t got a geojson to png working yet using mapnik plug-ins, but I haven’t really tried. Indeed right now I cant even bake level 3 and above without bumping up my RAM. They are nowhere near fit for purpose till we have this bake and shake PNG file and reduced vector tile approach. I obviously need to tone the styling right down in order that it can be used as an underlay for the bus map.
My plan now is to get back to the original job, that of showing people where we know there are bus, rail and ferry services. I was aware of routes and services being encoded into the map. Personally I don’t think this is feasible, and indeed when I attempted to use it to reduce the numerous industrial and agricultural railways in Russia I just got loads of routes and services of industrial railways. In the US, all those coal railways in Virginia have absolutely no obvious way of differentiating them. My issue about coding the services into the map is essentially the source of this data is in the GTFS, not the map. We are just going to end up having to synch the map up, mechanically, with the GTFS world, with potentially undesirable effects. I think we should have a Global GTFS repo, or GGTFS, and a method of assigning OSM features/ways to it, not the other way round, not writing into OSM, but writing into the G(lobal)GTFS and pointing it to something somewhere in OSM . Well that’s essentially what I am building anyway.
On my return to the doing the BuzMap proper stuff I will look to refit Graphhopper with the Open Route Service ecosystem. I don’t really fit into the Graphhopper group and hope to get a little more interaction from ORS. I have a number of things I will need help with, 3 easy, one “research”. Specifically
- hacking out Douglas Peucker simplification of the results, (trivial)
- fixing road “encoder” to use bus-able ways. (easy)
- providing a ferry encoder (trivial)
- determining preferred platforms/ways through station networks (very hard)
The railways turn out to be quite tricky. We can’t just pick any old railway line, we will end up shunting in and out of junction stations. Worse still, GTFS gives no indication of the gauge any service uses. So we don’t even know which network to use. I hacked something up to do this, it failed miserably once I threw Victorian railway networks at it. Glasgow in particular but think of Gare de Lyon and Gare de l’Est, you cant just do a radius search. It may be that the only short term practical solution is to knock up an editors app to allow subsequent manual assignment of these platform nodes. I suspect that inevitably someone is going to go have to go round and tweak massive Bahnhofs.
I haven’t done anything on the bus map proper, this post is just a way of getting me back onto the target. So apologies to Norway, Austria and Spain who have all directed me to their national data sets. I will be loading those countries along with the missing Switzerland and the half broken Finland before leaving Europe to tackle the US. I will produce an interim release and blog post with Europe better completed before moving to the US.
More news as I get it, all feedback gratefully received.
Recap of latest attempt
I am building a travellers map, and so wanted to fill out those places, most of the map that is, without GTFS. The hope was that I might be able to at least display a world map with something on it anywhere there might be a bus, even if I don’t have the timetable. It would balance the map out. It has moved from being a feasibility study to temper the original route map code, to a quest for a single usable tile at 0,0,0 to represent the entire planet’s transport infrastructure, ensuring that we have remote roads in Siberia and across the Arctic, a miniature version of Australia, a complete view of Africa etc. I want a render where I can see stuff above level 9 which is when we normally have to start styling stuff out or end up with an unreadable map. By level 5 on most renders we have lost just about everything. You need level 5 just to be able to get a handle on supersize countries or regions. Even on renders that still have major roads preserved at level #5 they don’t really make much impact. They have to be styled very thinly. On Google, to cope with the Ruhr and the US East Coast and still have detail in sparser regions, individual countries have to have different styling, it’s quite ugly and still doesn’t give the high level continental view I want. Only motorways and trunk roads can be rendered, we end up with a very messy map in Europe if we go any deeper. This can be misleading. France has a far more developed road network than Spain but these renders say the opposite. This issue of how individual countries chose to classify their roads is of course a general problem across the map, not just with roads.
These remote and sparse areas are relatively easy. The fun starts when you tackle a scaled down version of these super dense areas. Maharastra is also problematic and of course China. My map considers Chicago area to have the greatest major road density. In these areas, applying my solution of clustering vertices will rapidly and inevitably result in a honeycomb effect. This isn’t just major cities, the whole of the Ruhr and Low Countries becomes a triangulated lattice all too easily and the map becomes distinctly untrustworthy.
There are a number of simple things I have tried to mitigate this. The first is not to use the cluster root as the point to shift to, but the nearest point on the best line by grade then length. Indeed we should not merge any of the points on this “best” line through the cluster, thus keeping it’s detail. This solves most of the issue but it is all too easy to lose information in the quest for reduction of the stupendous amounts of data in the map. Line simplification needs to be performed piece-wise on each inter-nodal section, i.e don’t simplify out the junctions. If we lose junctions we will naturally end up having to merge onto more distant vertices as Douglas and Peucker will have zapped junction nodes on straitish lines. This can re-introduce “triangulism”, we can also end up getting gaps in the map. It’s remarkable how much information your eyes can get through. A tiny blank pixel on a line can make a lot of difference.
Another scourge I had to fight was “creaseism” as a result of the hierarchical nature of my tiling architecture which inherits geojson from child “tiers” of groups of layers and simplifies these tiles. Without a significant overlap you will end up clustering away form the edges, and end up with very visible major tile boundary creases. To reduce based on density I obviously need the context of the current region in order to determine if a feature should be reduced . This is my excuse for not using tippecanoe to date as I don’t fancy hacking that to make it happen.
And then there is “squarism”, which needs to be preserved. This is the business of trying to present the vast tract up the centre of the US and Canada that has a grid road system religiously aligned to the geographic co-ordinate system. At this point I leaned back and decided I was disappearing down a rabbit hole. We may be able to tackle these issues by applying some stuff called computer science and determining the dominant paths through a network, but for now I need to cut my losses and return to the bus map proper.