Commit Graph

127 Commits

Author SHA1 Message Date
Adrian Schmutzler c83c4eb860 api/alfred: Mask public IPv6 addresses
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-05-11 23:53:02 +02:00
Adrian Schmutzler 6efd0bda59 api/alfred: Only use single hood for V1
This removes code to evaluate the position of a V1 router.

Distinction between Default and NoCoordinates is dropped.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-05-11 23:51:53 +02:00
Adrian Schmutzler 49cefe2767 api/alfred: Keep old hood if empty field is sent
This suppresses hood change events if hood is lost due to
disconnection.

Normally, disconnects should be indicated by other events like
online/offline.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-05-11 23:51:47 +02:00
Adrian Schmutzler 4c52271897 api/alfred: Treat empty hood element as V2
To get rid of V2 <-> V1 hood changes, an empty hood element is
treated as V2, while a missing one is treated as V1.

This requires a firmware update to work.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-05-11 23:49:50 +02:00
Adrian Schmutzler 0bf2312fe4 Global: Put hoods into table and use smallint for reference
This will reduce size of stats_hood and, more importantly,
make hood assignment independent of hood name changes:

Previouly: Name change = changing string in every place
Now: Name change = change of one table entry

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-11-23 13:14:50 +01:00
Adrian Schmutzler dd6d101ccd Global: Identify local routers and indicate their status
This detects local routers based on knowing their hood, but not
having the hood listed in hoodsv2 table.

This classification is performed when the routers' alfred data
is parsed. Thus, offline routers won't change.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-11-23 13:14:50 +01:00
Adrian Schmutzler c1c728f2a3 Global: Convert IPv4/IPv6 from char to numbers/binary
This is done for tables router_ipv6 and router_netif.

This is not done for table gw_netif (contains subnet masks).

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-11-23 13:14:50 +01:00
Adrian Schmutzler bc3460f2e0 Global: Change MAC address storage to use BIGINT
This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-11-23 13:14:41 +01:00
Adrian Schmutzler cf3517d9d2 api/alfred: Fix check for existing router and delete fragments
The router id "router_id" is only evaluated based on the
router_netif table. If the corresponding entry in the router
table is missing, an error occurs.

To deal with that, we now use the "olddata" variable for ifs,
which is initialized based on the router table. If nothing is
found there, we trigger delete_router to get rid of fragments
in other tables.

The latter is necessary, as we identify routers by MAC addresses
and thus old entries will keep to be a problem if just a new
entry is added to router_netif.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-07-03 14:10:15 +02:00
Adrian Schmutzler 6d492a3a25 api/alfred: Support Babel neighbor IP address inside tag
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-07-03 14:10:15 +02:00
Adrian Schmutzler c01a3017a2 api/alfred: Support loadavg5
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-07-03 14:10:15 +02:00
Adrian Schmutzler 323a3a000c api/alfred: Improve retrieval of L3 mac address from IPv6
Thanks to Fabian Blaese.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-07-03 14:10:15 +02:00
Adrian Schmutzler 5e60c70ca5 api/alfred: Introduce gateway field
This introduced a boolean gateway field, which is set based on
the gateway connections sent via alfred.

If a device provides no gateways which it is connected to, it is
assumed to respresent a gateway.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-07-03 13:45:29 +02:00
Adrian Schmutzler fc9a494078 routertools: Also delete from router_gw and router_stats_gw
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 13:12:39 +02:00
Adrian Schmutzler 8eb207e04e api/alfred: Suppress duplicate key errors for stats (workaround!)
Sometimes two queries want to insert the same entry into the
router stats table, although we check for that right before the
query is made. One can suppress this by using ON DUPLICATE KEY UPDATE
to ignore the second (redundant) entry.

This is no fix, but will work until this is examined more thoroughly.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:50:01 +02:00
Adrian Schmutzler a57d459c77 api/alfred: Disable router_rate_limit_list
With router_rate_limit_list, routers were not processed if the
time difference between calls was less than 5 minutes for the
same MAC address.

While this is generally not bad, there are some drawbacks:
- Not having been aware of this fact, we have established other
  mechanisms to dilute data density, which might have interfered
- With KeyXchangeV2, two gateways will send data with less than
  5 min. difference. As gateways are not connected, we know that
  we alternately receive newer and older data. With
  router_rate_limit_list, some of this data has been discarded
  before its "age" was evaluated. This caused an unwanted additional
  dilution of data which might have caused "offline" routers not
  being actually offline (for a short period)
- With KeyXchangeV1, if the second call was a little earlier, the
  a big share of the data would not be "new enough" and just be
  discarded
- With KeyXchangeV1, the same would happen for the order of records
  varying between alfred calls, were some records would have more and
  some less than 5 minutes time difference

To get rid of these issues, we remove router_rate_limit_list and
test whether the newer measures to dilute data are effective.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:49:34 +02:00
Adrian Schmutzler c77e04ee7f api/alfred: Evaluate utcnow() only once for the whole alfred dataset
Since the time is used as key in MySQL, this might help to solve key
race conditions where new_router_stats is entered at the same time.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:49:23 +02:00
Adrian Schmutzler d1fd5e2bf9 api/alfred: Optimize delete queries for IPv6
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:48:28 +02:00
Adrian Schmutzler 68dff475d2 api/alfred: Use individual transactions to delete router data
This patch uses smaller (but more) transactions to delete router
data. This makes lock more specific and thus prevents deadlocks
quite effectively.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:45:44 +02:00
Adrian Schmutzler d6b7ce8e1e config: Introduce specific stats deletion threshold for gw history
Value set to only 7 days to reduce database size, since these
data seem to be of little relevance.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:45:01 +02:00
Adrian Schmutzler c8178beedd gwinfo and scripts/deletestats: Delete old netif data
Netif information is deleted 48 hours after the MAC addresses
have changed.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-04-16 12:42:35 +02:00
Adrian Schmutzler c581915076 api/alfred: Remove router hardname name correction for old FW
This only affects 0.5.1 and older.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:46:52 +01:00
Adrian Schmutzler 5d7e00422e api/alfred: Check for new router data initially by system time
If we receive data from more than one gateway, there happens to
be a mix of older and newer data (since synchronization between
gateways seems to be not working).

To deal with that, we now only accept data where the router's
system time is newer than the value stored in the DB. To account
for time synchronization issues, we also accept data which is more
than one hour older.

This patch removes other checks for old data which are now obsolete.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:46:31 +01:00
Adrian Schmutzler d3ea76b648 scripts/deletestats: Use block-wise delete for more stats tables
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:46:20 +01:00
Adrian Schmutzler f6f699c8c7 api/alfred and router.html: Include routing_protocol
This only works for routers with updated nodewatcher.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:45:32 +01:00
Adrian Schmutzler 146d64da19 api/alfred and router.html: Retrieve and show traffic control status
This only works for routers with updated nodewatcher.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:44:44 +01:00
Adrian Schmutzler 00ba4ace1b routertools/filters/map.js: Implement babel cost by negative quality
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:31:58 +01:00
Adrian Schmutzler 89ad846375 routertools: Use float for neighbor and gateway quality
This enables support for BATMAN V.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:27:18 +01:00
Adrian Schmutzler f8ba13268a routertools: Parse gateway quality differently to catch wrong data
Since old routers send defective gateway data, some routers got
values like "false6" for quality. This is now caught and false
removed.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:25:16 +01:00
Adrian Schmutzler d298cc7762 Introduce v2 field in router table
If a router sends his hood, it is considered to be V2.

V2 hoods are highlighted on the statistics page.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-22 16:23:54 +01:00
Adrian Schmutzler 8d41ed9838 api/alfred and router.html: Introduce airtime stats
This only works for routers with updated nodewatcher.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-02 13:25:06 +01:00
Adrian Schmutzler 1cffed9f79 api/alfred and router.html: Introduce detailed client stats
This only works for routers with updated nodewatcher.

This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-02 13:14:35 +01:00
Adrian Schmutzler 907b3154a9 api/alfred: Introduce NoCoordinates hood
Previously, mesh routers without coordinates are assigned to the
default hood, even if they are meshing with routers from other
hoods.

To keep the default hood clean, we introduce the "NoCoordinates"
hood for all those routers.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-02 13:01:02 +01:00
Adrian Schmutzler 0ce3fa5c2f routertools: Set rx/tx to zero if router gets offline
This has to be done BEFORE the status is set, since otherwise
the selector won't work anymore.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-02 12:52:27 +01:00
Adrian Schmutzler 58cc2e167f scripts/deletestats: Do not evaluate status in stats deletion
Previously, old stats had been only deleted if the router was
online. This required a join with the router table, which caused
locking issues when writing to this table by alfred request.

Now, we just delete old stats ignoring the router state, except
for the router_stats table, which is smaller and thus remains
as before.

In addition, this patch logs the number of rows affected.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-02-02 01:00:44 +01:00
Adrian Schmutzler 804ce80472 scripts/deletestats.py: Delete netif stats after 21 days
This adds an additional config value to set the deletion threshold
for netif data.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-15 12:30:09 +01:00
Adrian Schmutzler 823858d981 scripts: Add deleteunlinked.py
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-15 12:30:09 +01:00
Adrian Schmutzler 2a515b94c7 config/routertools: Set minimum distance for data in router stats
This implement to different minimum distances in seconds for the
router stats in general and the netif stats in particular.

The values are chosen so that they are 30 secs. shorter than the
desired timespans of 5 and 10 minutes, to allow for fluctuation
in when data arrives.

This fixes the data density increase caused by V2 Hoods with two
gateways.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-15 12:30:08 +01:00
Adrian Schmutzler f12a3f5a3e api/alfred and router.html: Collect and show gateway information
This requires changes to the MySQL database!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-15 00:10:29 +01:00
Adrian Schmutzler c3adf5fd68 api/alfred: Only log reboot if uptime difference greater than 5 min
If we have two gateways per hood, uptime may vary and cause
frequent logged reboots.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-13 16:46:45 +01:00
Adrian Schmutzler 232e8fe6f0 scripts/calcglobalstats: Add marker at the end of logged durations
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2018-01-13 16:45:03 +01:00
Adrian Schmutzler 55dc7b4ad5 Rename net_if from router_neighbor table to netif
This includes various changes throughout the code. Additionally,
some fields are reordered.

This requires a change of the MySQL table router_neighbor!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 18:21:08 +01:00
Adrian Schmutzler 39d9ddc9d0 api/alfred: Update router data instead of DELETE/INSERT
Previously, I just deleted all entries and recreated them again
with INSERT. Although this is simple to write and actually includes
less queries, it causes a lot more write IO. Since most of the
neighbors and interfaces do NOT change frequently, it is worth the
extra effort to delete only those really gone since the last
update.

Only br-mesh will normally have assigned IPv6 addresses, thus
we just delete all IPv6 adresses of the other ones.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 18:21:08 +01:00
Adrian Schmutzler 55f81c4295 api/alfred: Change calculation of rx/tx after reboot
Previously, on restart the traffic of the last period before
the restart was reused.

Now, we use the logged traffic divided by the uptime.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 18:19:24 +01:00
Adrian Schmutzler 858f419e54 api/alfred and router.html: Show WiFi data for netifs
This shows information about WiFi parameters (e.g. channel).

Except the Tx-Power, data is only available if a firmware with
a corresponding nodewatcher update is present (version 44).

This requires a change of the MySQL table router_netif!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 18:08:41 +01:00
Adrian Schmutzler 7ef6f47c4e routertools: Reorder fields in INSERT queries for router stats
This is a cosmetic change to realign the INSERTs to the database.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 18:04:12 +01:00
Adrian Schmutzler 58ce32e322 Change router_stats_netif to use ids for netifs
This introduces a serious of changes to code and database.

This patch requires changes to the MySQL database.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-30 17:18:22 +01:00
Adrian Schmutzler a3ee0edead routertools/alfred: Replace empty hostname
Fixes #109

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-21 17:06:59 +01:00
Adrian Schmutzler 20e71afeb0 Provide possibility to ban routers
If routers are supposed to be removed from the Monitoring
permanently, they can now be banned based on their MAC address.

All admins can do that via the web interface.

ATTENTION: This requires a database update!

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-21 17:03:27 +01:00
Adrian Schmutzler 636a8d3baa routertools: Use custom file for full log, do not write to syslog
It is easier to evaluate a specific log which is not crowded with
messages from other programs

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2017-12-10 20:25:16 +01:00