projects:prcrawler
Differences
This shows you the differences between two versions of the page.
|
projects:prcrawler [2014/01/12 10:56] jann created |
projects:prcrawler [2014/06/12 11:25] (current) jann |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | The Packet Radio Crawler is gathering data from the worldwide packet radio network to generate a daily map. | + | ====== Discontinued Project ====== |
| - | * [[projects:prcrawler|Packet Radio Crawler (by DL3SJB)]] | + | |
| + | http://db0fhn.efi.fh-nuernberg.de/prmap | ||
| + | |||
| + | ====== Project: Packet Radio Crawler reloaded 1.0 ====== | ||
| + | //written by Jens, DL3SJB, email: dl3sjb [.at.] qsl [.dot.] net// | ||
| + | |||
| + | Project team: DG8NGN, DL3SJB. | ||
| + | |||
| + | As DG8NGN and DL6NEK made the last link-map of the EU in 2003, it's about time to create a new more automated system, called “PR-Crawler reloaded 1.0”. | ||
| + | |||
| + | //A what?// Answer: The project's goal is to get a holistic view of the international packet-radio network's status as seen from IGATE using an automated way for obtaining the netword data. The objective is to get continuous updates, daily. | ||
| + | |||
| + | //Why?// There is no real source for holistic maps of the Packet Radio network available. If you find a map, chances are it is put together for regional areas and many times it isn't updated regularly. In an ever changing network, one would like to have an overview what parts of the network develop or die. Also, the resources and service offered on specific packet radio nodes are of interest for users. The idea - not new, if I may add - is to build something that both regularly scans the network and visualizes it - a Packet Radio Crawler. | ||
| + | |||
| + | But first things first: follow this link to view to the Packet Radio map as shown during the 2009 IPRT in Darmstadt: | ||
| + | |||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-iprt2009.htm | ||
| + | ]]. | ||
| + | |||
| + | A more complete map was shown during the 2009 HAMRADIO in Friedrichshafen: | ||
| + | |||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-adacom-hamradio-2009.htm | ||
| + | ]]. | ||
| + | |||
| + | As a spin-off, here's a Amateur Radio pager map (beta test phase) showing DL's actual pager network in July 2009 : | ||
| + | |||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/frmap-adacom.htm]] | ||
| + | |||
| + | NEWS NEWS NEWS: IPRT 2010 SPECIAL | ||
| + | |||
| + | Mailbox Map | ||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-mailbox.htm]] | ||
| + | |||
| + | Dynamic Network Map (updated daily): | ||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-dynamic.htm]] | ||
| + | |||
| + | //How was this done?// Answer: We decided to break this project down into 3 main jobs: visualization, data collection and data analysis. The easiest way to visualize map data these days is using Google Maps (c). | ||
| + | |||
| + | ==== Data Collection and Analysis ==== | ||
| + | Main objetivce: Get the network's topology read in an automated way (Linux, Kernellink, Scripting) | ||
| + | |||
| + | === The Motivation === | ||
| + | |||
| + | In order to obtain a wholistic view of the packet radio network, two prerequisites need to be fulfilled: | ||
| + | - Fast access to as many digipeaters as possible to obain local information such as active links, locator, etc | ||
| + | - An automated method to 'crawl' all digipeaters on all SSIDs for later analysis of their services | ||
| + | |||
| + | An ideal way the bring both things together is 'Linux shell scripting', using a Kernellink into the AX25 network, and IGATE. | ||
| + | - Shell scripting provides for batch job processing along with full access to all Linux text analysis tools such as 'grep', 'head, 'tail', etc. | ||
| + | - IGATE provides fast access to more than 800 destinations world-wide. | ||
| + | |||
| + | === The Approach === | ||
| + | |||
| + | The datacollection works in basically 7 step process | ||
| + | - Get a list of all packet radio nodes currently available | ||
| + | - Filter the list for all SSIDs available per digipeater | ||
| + | - Obtain address of packet radio nodes QTH - OR - geo data (WGS84) | ||
| + | - Convert address or geo data into decimal geo-data so 'Google maps' can work with it, take the best data available per packet radio node | ||
| + | - Obtain neccessary data to visualize the network topology crawling all packet radio nodes on all SSIDs forwarded through the network protocol for local information. | ||
| + | - Analyze the local data and convert it into a 'Google maps' compatible format | ||
| + | - Start all over to update the network map | ||
| + | |||
| + | === The Challenges === | ||
| + | There are various obstacles to navigate around. The two biggest steps in the above list is 'crawling the AX25 net' and 'analyze the local data'. Softwarewise, they require the highest level of precise error handling in this project. | ||
| + | |||
| + | == Crawling the AX25 network == | ||
| + | There are various services available on the AX25 net such as DX-Cluster, BBS, Convers, packet radio nodes router access. The 'crawler' needs to trigger information that allows for recognition of each service by unique patterns. Also, the crawler needs to connect more than 5000 services (approx. 800 destination with approx. 5-10 SSIDs, max. 16) and must not get locked should the AX25 frames not come as expected. So a stringend error handling (i.e. timeout) needs to take place. Other that that, the mechanism itself is easy. Connect all dests known to IGATE, ask for the required data (i.e. "l"), stored it locally and disconnect. Funny enough, it takes about 24 hrs to complete the crawling job, if done in a 'daisy chain' way. One can speed up the process by scaling the job into several. We did this with the May version of the crawler. Since that time, the crawling job itself takes about 4hrs. One thread takes about 10-20 kbps of bandwith on the local digipeater where the crawler takes off. | ||
| + | |||
| + | == Analyzing the Data == | ||
| + | This I'd call the 'headacher'. Why? Think about it. There are a number of AX25 packet radio nodes implementations out there. All have their own way to 'present' the data, i.e. link information. In order to avoid 'runaway' network map data, one needs to find the UNIQUE pattern that allows to interpret the data. Luckily, it's possible. XNET and FLEXNET can be differentiated using Linux shell tools such as "cut". Long story short: the job can be done. TNN is easy to recognize, after all. | ||
| + | |||
| + | There are various things one might be interested in: | ||
| + | - What digipeaters are out there? Where are they? | ||
| + | - Who's on-line? Who's not? | ||
| + | - Who links to whom? | ||
| + | - Which of those links are currently active, where are network islands? | ||
| + | - How many XNET routers are out there? (.. FLEXNET, TNN ...) | ||
| + | - Who has direct IGATE access? | ||
| + | - How many Pogsac masters and slaves are out there? | ||
| + | - Who links over the internet, who does not? | ||
| + | |||
| + | The main job is to put the packet radio nodes on the map at the right spot and draw the currently live links between them. | ||
| + | |||
| + | === Google compatible Data Structure === | ||
| + | == Markers == | ||
| + | The markers represent the location of the packet radio nodes on the map. The markers contain a number of information, such as geo data, suffix of nodes call, node call itself, linktype (with IGATE/RF only), etc. | ||
| + | |||
| + | All packet radio nodes that are to appear on the map are kept in one text file called 'markers.txt'. | ||
| + | The files structure looks like this: | ||
| + | |||
| + | **<latitude in dec>|<longitude in dec>|suffix|call|status|linktype|** | ||
| + | |||
| + | where: | ||
| + | |||
| + | * suffix = suffix of digipeater's call (max 3 chars) | ||
| + | * call = digipeater's full call w/o SSID (max 6 chars) | ||
| + | * status = [P,N]: 'P' = present; 'N' = not present. =>Status of digipeater, meaning was it found online or offline during last crawling session | ||
| + | * linktype = [I,F]: 'I' = at least an IGATE link; 'F' = no IGATE link, radio only | ||
| + | |||
| + | Example: | ||
| + | |||
| + | (...) | ||
| + | |||
| + | 48.6653791|9.0410710|SAO|DB0SAO|F| | ||
| + | |||
| + | (...) | ||
| + | |||
| + | |||
| + | == Polylines == | ||
| + | The polylines represent to links between the digipeaters in a graphical way (as lines, right). The polylines contain a number of information, such as geo data of both source and destination, linktype (active (-,internetlink), dead, etc. | ||
| + | |||
| + | All links that are to appear on the map are kept in one text file called 'polylines.txt'. | ||
| + | The files structure looks like this: | ||
| + | |||
| + | **<latitude source in dec>|<longitude source in dec>|<latitude destination in dec>|<longitude destination in dec>|linktype|misc information|** | ||
| + | |||
| + | where: | ||
| + | |||
| + | * linktype = [F,D,I]: 'F' = link was found active during last crawler run; 'I' = link was found active and run over internet; 'D' = link was found dead for more than x days | ||
| + | * misc information = all data that is useful for debugging, but not used on Google maps. | ||
| + | |||
| + | Example: | ||
| + | |||
| + | (...) | ||
| + | |||
| + | 48.6653791|9.0410710|48.6947227|9.1180094|F|DB0SAO-DB0LEL| | ||
| + | |||
| + | (...) | ||
| + | |||
| + | ==== Visualisation ==== | ||
| + | Main job: throw data on a map that can be accessed by anyone at any time (Internet) | ||
| + | |||
| + | The easiest way to visualize map data these days is using Google Maps (c). The API provides a vast number of functions. If you want to learn more, follow this link: [[http://code.google.com/intl/de/apis/maps/documentation/reference.html]]. The underlaying elements used are 'markers' (= packet radio nodes, e.g. DB0SAO) and 'polylines' representing the links between the packet radio nodes (e.g. DB0HER<->DB0SAO on 23cm@19200 baud). We will add more elements as we go along, such as popup messages showing the status of the nodes. Don't expect things to happen in days or weeks. This is a fun project. LOL | ||
| + | |||
| + | ===== Result ===== | ||
| + | |||
| + | [[http://db0fhn.efi.fh-nuernberg.de/prmap/prmap-iprt2009.htm]] | ||
| + | Later a dynamic map with continuous updates is to follow, the latest by the 2009 Hamradio in Friedrichshafen. | ||
| + | |||
| + | Feedback is highly welcome. Please email to dl3sjb [.at.] qsl [.] net | ||
| + | |||
| + | ====== Crawler Info ====== | ||
| + | |||
| + | Der Crawler ruft den Befehl "i" und "digiinfo" auf jedem Digipeater ab. Die Ausgabe enthaelt die manuell vom Sysop gepflegten Informationen zum Digipeater. | ||
| + | |||
| + | Datenerfassung: | ||
| + | |||
| + | Die Auswertung beginnt ab dem Schluesselwort "Callsign:". Die vorherigen Zeilen werden verworfen. Es wird __am Anfang__ jeder Zeile nach einem gueltigen Schluesselwort gesucht (Callsign:, Locator:, Digi:, Funkruf:, APRS:, Link:, Contact:, QTH-Info:) und diese dann ausgewertet. Wird kein Schluesselwort gefunden, wird die Zeile uebersprungen. Gross-/Kleinschreibung wird ignoriert. Einzelne Datenfelder bei Zeilen mit dem Schluesselwort "Digi:, Funkruf:, APRS: oder Link:" sind durch Leerzeichen getrennt. Pflichtangaben fuer das Projekt sind **Callsign** und **Locator** (__10-stellig!__). Alle weitere Datenfelder koennen mit "-" freigelassen werden. Internetlinks (oder ähnliche) werden grundsätzlich nicht eingetragen. | ||
| + | |||
| + | __**Datenfelder:**__ | ||
| + | |||
| + | ^Schluesselwort ^Info ^Beispiel 1 ^Beispiel 2 ^ | ||
| + | ^Callsign: |Rufzeichen mit/ohne SSID |DB0ABC |DB0XYZ-3 | | ||
| + | ^Locator: |10-stelliger Locator nach http://no.nonsense.ee/qthmap/ |JN59NK18HP |JN48MP49TR | | ||
| + | ^Digi: |Digi-Einstieg-Beschreibung (siehe Digi-Datenfeld) | | | | ||
| + | ^APRS: |APRS-Beschreibung (siehe APRS-Datenfeld) | | | | ||
| + | ^Funkruf: |Funkruf-Beschreibung (siehe Funkruf-Datenfeld) | | | | ||
| + | ^Link: |Link-Beschreibung (siehe Link-Datenfeld) | | | | ||
| + | ^Contact: |Freies Datenfeld fuer Kontaktinformationen |e@mail.de |Jann, DG8NGN | | ||
| + | ^QTH-Info: |Freies Datenfled fuer Standortinformationen |HS Nuernbg |TV Tower Nbg | | ||
| + | |||
| + | __**Data fields:**__ | ||
| + | |||
| + | ^Keywords ^Info ^Example 1 ^Example 2 ^ | ||
| + | ^Callsign: |callsign with or without SSID |DB0ABC |DB0XYZ-3 | | ||
| + | ^Locator: |10-digit locator, see http://qth.map.googlepages.com |JN59NK18HP |JN48MP49TR | | ||
| + | ^Digi: |node's user access description (see digi data field) | | | | ||
| + | ^APRS: |APRS-description (see APRS-data field below) | | | | ||
| + | ^Funkruf: |Pager description (see pager data field below) | | | | ||
| + | ^Link: |interlink-description (see link data field) | | | | ||
| + | ^Contact: |free text field for contact information |e@mail.de |Jann, DG8NGN | | ||
| + | ^QTH-Info: |free text field for contact information |HS Nuernbg |TV Tower Nbg | | ||
| + | |||
| + | |||
| + | QTH-Locator Ermittlung alternativ mit {{:projects:prcrawler:locator.zip|"Location"}} | ||
| + | |||
| + | Screenshots: | ||
| + | |||
| + | {{:projects:prcrawler:screenshot1.png|Location}} | ||
| + | |||
| + | {{:projects:prcrawler:screenshot2.png|Helpscreen Location}} | ||
| + | |||
| + | |||
| + | |||
| + | __**Digi-Datenfelder / APRS-Datenfelder / Funkruf-Datenfelder:**__ | ||
| + | |||
| + | ^Datenfeldnr. ^Datenfeld ^Formatierung ^Beispiel 1 ^Beispiel 2 ^ | ||
| + | |1 |Sendefrequenz |bis 99999.9999 MHz |10308MHz |144.8125 | | ||
| + | |2 |Empfangsfrequenz |bis 99999.9999 MHz |10308.0125 |438.1abc | | ||
| + | |3 |Antennenhoehe |Hoehe ueber Grund bis 999 Meter |30 |30m | | ||
| + | |4 |Leistung |in ERP bis 9999 W |15W |1 | | ||
| + | |5 |Antennenrichtung |Rundstrahler = ND, ansonsten ° |ND |234° | | ||
| + | |6 |Oeffnungswinkel |Rundstrahler = -, ansonsten ° |- |12 | | ||
| + | |7 |Polarisation |H=hori. V=vert. Z=zirkular |H |Z | | ||
| + | |8 |Bandbreite |bis 999999.99 kHz |10000 |6.25kHz | | ||
| + | |9 |Geschwindigkeit |bis 999999.99 kbit/s |20000 |1.2 | | ||
| + | |||
| + | __**Link-Datenfelder:**__ | ||
| + | |||
| + | ^Datenfeldnr. ^Datenfeld ^Formatierung ^Beispiel 1 ^Beispiel 2 ^ | ||
| + | |1 |Gegenstelle |Rufzeichen |DB0BBG |DB0VOX | | ||
| + | |2 |Sendefrequenz |bis 99999.9999 MHz |10308MHz |144.8125 | | ||
| + | |3 |Empfangsfrequenz |bis 99999.9999 MHz |10308.0125 |438.1abc | | ||
| + | |4 |Antennenhoehe |Hoehe ueber Grund bis 999 Meter |30 |30m | | ||
| + | |5 |Leistung |in ERP bis 9999 W |15 |.01W | | ||
| + | |6 |Antennenrichtung |Rundstrahler = ND, ansonsten ° |ND |234° | | ||
| + | |7 |Oeffnungswinkel |Rundstrahler = -, ansonsten ° |- |12 | | ||
| + | |8 |Polarisation |H=hori. V=vert. Z=zirkular |H |Z | | ||
| + | |9 |Bandbreite |bis 999999.99 kHz |10000 |6.25kHz | | ||
| + | |10 |Geschwindigkeit |bis 999999.99 kbit/s |20000 |1.2 | | ||
| + | |||
| + | |||
| + | __**Digi-/ APRS-/ Pager data fields:**__ | ||
| + | |||
| + | ^data field # ^data field ^formatting ^Example 1 ^Example 2 ^ | ||
| + | |1 |TX frequency |up to 99999.9999 MHz |10308MHz |144.8125 | | ||
| + | |2 |RX frequency |up to 99999.9999 MHz |10308.0125 |438.1abc | | ||
| + | |3 |ant height |heigth over ground up to 999 meters |30 |30m | | ||
| + | |4 |TX power |in ERP up to 9999 W |15W |1 | | ||
| + | |5 |ant bearing |isotropic = ND, all other in '°' |ND |234° | | ||
| + | |6 |beam width |isotropic = -, all other in '°' |- |12 | | ||
| + | |7 |polarisation |H=hori. V=vert. Z=circular |H |Z | | ||
| + | |8 |band width |up to 999999.99 kHz |10000 |6.25kHz | | ||
| + | |9 |speed |up to 999999.99 kbit/s |20000 |1.2 | | ||
| + | |||
| + | __**Link-data field:**__ | ||
| + | |||
| + | ^data field # ^data field ^formatting ^Example 1 ^Example 2 ^ | ||
| + | |1 |Opposite node |Callsign |DB0BBG |DB0VOX | | ||
| + | |2 |TX frequency |up to 99999.9999 MHz |10308MHz |144.8125 | | ||
| + | |3 |RX frequency |up to 99999.9999 MHz |10308.0125 |438.1abc | | ||
| + | |4 |ant height |heigth over ground up to 999 meters |30 |30m | | ||
| + | |5 |TX power |in ERP up to 9999 W |15W |1 | | ||
| + | |6 |ant bearing |isotropic = ND, all other in '°' |ND |234° | | ||
| + | |7 |beam width |isotropic = -, all other in '°' |- |12 | | ||
| + | |8 |polarisation |H=hori. V=vert. Z=circular |H |Z | | ||
| + | |9 |band width |up to 999999.99 kHz |10000 |6.25kHz | | ||
| + | |10 |speed |up to 999999.99 kbit/s |20000 |1.2 | | ||
| + | |||
| + | Beispiel: | ||
| + | |||
| + | <code> | ||
| + | Callsign: DB0FHN | ||
| + | |||
| + | Locator: JN59NK18HP | ||
| + | |||
| + | Frequenztabelle | ||
| + | Typ TX RX Ant Pwr Azim OeW Pol kHz kbit/s | ||
| + | ------------------------------------------------------------------ | ||
| + | Digi: 144.825 144.825 30m 15W ND - V 12.5 1.2 | ||
| + | Digi: 438.075 438.075 30m 15W ND - V 25 9.6 | ||
| + | Digi: 438.275 430.675 30m 15W ND - V 20 1.2 | ||
| + | |||
| + | Funkruf: 439.9875 30m 15W ND - V 12.5 1.2 | ||
| + | |||
| + | Link: DB0BBG 1299.150 1240.150 30m 15W 357° 20° H 50 9.6 | ||
| + | Link: DB0VOX 2361.000 2396.000 30m 1W 234° 12° H 8000 10000 | ||
| + | Link: DF0ANN 5600 5600 35m - - - H - 1000 | ||
| + | ------------------------------------------------------------------ | ||
| + | |||
| + | Contact: Jann Traschewski, DG8NGN Email: jann@gmx.de | ||
| + | |||
| + | QTH-Info: Hochschule Nuernberg | ||
| + | </code> | ||
projects/prcrawler.1389520616.txt.gz · Last modified: 2014/01/12 10:56 by jann
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
