written by Jens, DL3SJB, email: dl3sjb [.at.] qsl [.dot.] net

Project team: DG8NGN, DL3SJB.

As DG8NGN and DL6NEK made the last link-map of the EU in 2003, it's about time to create a new more automated system, called “PR-Crawler reloaded 1.0”.

A what? Answer: The project's goal is to get a holistic view of the international packet-radio network's status as seen from IGATE using an automated way for obtaining the netword data. The objective is to get continuous updates, daily.

Why? There is no real source for holistic maps of the Packet Radio network available. If you find a map, chances are it is put together for regional areas and many times it isn't updated regularly. In an ever changing network, one would like to have an overview what parts of the network develop or die. Also, the resources and service offered on specific packet radio nodes are of interest for users. The idea - not new, if I may add - is to build something that both regularly scans the network and visualizes it - a Packet Radio Crawler.

But first things first: follow this link to view to the Packet Radio map as shown during the 2009 IPRT in Darmstadt:

http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-iprt2009.htm.

A more complete map was shown during the 2009 HAMRADIO in Friedrichshafen:

http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-adacom-hamradio-2009.htm.

As a spin-off, here's a Amateur Radio pager map (beta test phase) showing DL's actual pager network in July 2009 :

http://db0fhn.efi.fh-nuernberg.de/prmap/frmap-adacom.htm

NEWS NEWS NEWS: IPRT 2010 SPECIAL

Mailbox Map http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-mailbox.htm

Dynamic Network Map (updated daily): http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-dynamic.htm

How was this done? Answer: We decided to break this project down into 3 main jobs: visualization, data collection and data analysis. The easiest way to visualize map data these days is using Google Maps ©.

Main objetivce: Get the network's topology read in an automated way (Linux, Kernellink, Scripting)

In order to obtain a wholistic view of the packet radio network, two prerequisites need to be fulfilled:

1. Fast access to as many digipeaters as possible to obain local information such as active links, locator, etc
2. An automated method to 'crawl' all digipeaters on all SSIDs for later analysis of their services

An ideal way the bring both things together is 'Linux shell scripting', using a Kernellink into the AX25 network, and IGATE.

1. Shell scripting provides for batch job processing along with full access to all Linux text analysis tools such as 'grep', 'head, 'tail', etc.
2. IGATE provides fast access to more than 800 destinations world-wide.

The datacollection works in basically 7 step process

1. Get a list of all packet radio nodes currently available
2. Filter the list for all SSIDs available per digipeater
3. Obtain address of packet radio nodes QTH - OR - geo data (WGS84)
4. Convert address or geo data into decimal geo-data so 'Google maps' can work with it, take the best data available per packet radio node
5. Obtain neccessary data to visualize the network topology crawling all packet radio nodes on all SSIDs forwarded through the network protocol for local information.
6. Analyze the local data and convert it into a 'Google maps' compatible format
7. Start all over to update the network map

There are various obstacles to navigate around. The two biggest steps in the above list is 'crawling the AX25 net' and 'analyze the local data'. Softwarewise, they require the highest level of precise error handling in this project.

There are various services available on the AX25 net such as DX-Cluster, BBS, Convers, packet radio nodes router access. The 'crawler' needs to trigger information that allows for recognition of each service by unique patterns. Also, the crawler needs to connect more than 5000 services (approx. 800 destination with approx. 5-10 SSIDs, max. 16) and must not get locked should the AX25 frames not come as expected. So a stringend error handling (i.e. timeout) needs to take place. Other that that, the mechanism itself is easy. Connect all dests known to IGATE, ask for the required data (i.e. “l”), stored it locally and disconnect. Funny enough, it takes about 24 hrs to complete the crawling job, if done in a 'daisy chain' way. One can speed up the process by scaling the job into several. We did this with the May version of the crawler. Since that time, the crawling job itself takes about 4hrs. One thread takes about 10-20 kbps of bandwith on the local digipeater where the crawler takes off.

This I'd call the 'headacher'. Why? Think about it. There are a number of AX25 packet radio nodes implementations out there. All have their own way to 'present' the data, i.e. link information. In order to avoid 'runaway' network map data, one needs to find the UNIQUE pattern that allows to interpret the data. Luckily, it's possible. XNET and FLEXNET can be differentiated using Linux shell tools such as “cut”. Long story short: the job can be done. TNN is easy to recognize, after all.

There are various things one might be interested in:

1. What digipeaters are out there? Where are they?
2. Who's on-line? Who's not?
3. Who links to whom?
4. Which of those links are currently active, where are network islands?
5. How many XNET routers are out there? (.. FLEXNET, TNN …)
6. Who has direct IGATE access?
7. How many Pogsac masters and slaves are out there?
8. Who links over the internet, who does not?

The main job is to put the packet radio nodes on the map at the right spot and draw the currently live links between them.

The markers represent the location of the packet radio nodes on the map. The markers contain a number of information, such as geo data, suffix of nodes call, node call itself, linktype (with IGATE/RF only), etc.

All packet radio nodes that are to appear on the map are kept in one text file called 'markers.txt'. The files structure looks like this:

<latitude in dec>|<longitude in dec>|suffix|call|status|linktype|

where:

• suffix = suffix of digipeater's call (max 3 chars)
• call = digipeater's full call w/o SSID (max 6 chars)
• status = [P,N]: 'P' = present; 'N' = not present. ⇒Status of digipeater, meaning was it found online or offline during last crawling session
• linktype = [I,F]: 'I' = at least an IGATE link; 'F' = no IGATE link, radio only

Example:

(…)

48.6653791|9.0410710|SAO|DB0SAO|F|

(…)

The polylines represent to links between the digipeaters in a graphical way (as lines, right). The polylines contain a number of information, such as geo data of both source and destination, linktype (active (-,internetlink), dead, etc.

All links that are to appear on the map are kept in one text file called 'polylines.txt'. The files structure looks like this:

<latitude source in dec>|<longitude source in dec>|<latitude destination in dec>|<longitude destination in dec>|linktype|misc information|

where:

• linktype = [F,D,I]: 'F' = link was found active during last crawler run; 'I' = link was found active and run over internet; 'D' = link was found dead for more than x days
• misc information = all data that is useful for debugging, but not used on Google maps.

Example:

(…)

48.6653791|9.0410710|48.6947227|9.1180094|F|DB0SAO-DB0LEL|

(…)

Main job: throw data on a map that can be accessed by anyone at any time (Internet)

The easiest way to visualize map data these days is using Google Maps ©. The API provides a vast number of functions. If you want to learn more, follow this link: http://code.google.com/intl/de/apis/maps/documentation/reference.html. The underlaying elements used are 'markers' (= packet radio nodes, e.g. DB0SAO) and 'polylines' representing the links between the packet radio nodes (e.g. DB0HER↔DB0SAO on 23cm@19200 baud). We will add more elements as we go along, such as popup messages showing the status of the nodes. Don't expect things to happen in days or weeks. This is a fun project.

http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-iprt2009.htm Later a dynamic map with continuous updates is to follow, the latest by the 2009 Hamradio in Friedrichshafen.

Feedback is highly welcome. Please email to dl3sjb [.at.] qsl [.] net

Der Crawler ruft den Befehl “i” und “digiinfo” auf jedem Digipeater ab. Die Ausgabe enthaelt die manuell vom Sysop gepflegten Informationen zum Digipeater.

Datenerfassung:

Die Auswertung beginnt ab dem Schluesselwort “Callsign:”. Die vorherigen Zeilen werden verworfen. Es wird am Anfang jeder Zeile nach einem gueltigen Schluesselwort gesucht (Callsign:, Locator:, Digi:, Funkruf:, APRS:, Link:, Contact:, QTH-Info:) und diese dann ausgewertet. Wird kein Schluesselwort gefunden, wird die Zeile uebersprungen. Gross-/Kleinschreibung wird ignoriert. Einzelne Datenfelder bei Zeilen mit dem Schluesselwort “Digi:, Funkruf:, APRS: oder Link:” sind durch Leerzeichen getrennt. Pflichtangaben fuer das Projekt sind Callsign und Locator (10-stellig!). Alle weitere Datenfelder koennen mit ”-” freigelassen werden. Internetlinks (oder ähnliche) werden grundsätzlich nicht eingetragen.

Datenfelder:

Data fields:

QTH-Locator Ermittlung alternativ mit "Location"

Screenshots:

Digi-Datenfelder / APRS-Datenfelder / Funkruf-Datenfelder:

Link-Datenfelder:

Digi-/ APRS-/ Pager data fields:

Link-data field:

Beispiel:

Callsign: DB0FHN

Locator: JN59NK18HP

Frequenztabelle
Typ          TX         RX        Ant Pwr Azim OeW Pol kHz  kbit/s
------------------------------------------------------------------
Digi:          144.825    144.825 30m 15W   ND  -   V  12.5    1.2
Digi:          438.075    438.075 30m 15W   ND  -   V  25      9.6
Digi:          438.275    430.675 30m 15W   ND  -   V  20      1.2

Funkruf:       439.9875           30m 15W   ND  -   V  12.5    1.2

Link: DB0BBG  1299.150   1240.150 30m 15W 357°  20° H  50      9.6
Link: DB0VOX  2361.000   2396.000 30m  1W 234°  12° H  8000  10000
Link: DF0ANN  5600       5600     35m   -   -   -   H  -      1000
------------------------------------------------------------------

Contact: Jann Traschewski, DG8NGN Email: jann@gmx.de

QTH-Info: Hochschule Nuernberg